Towards Smooth Video Composition
Authors: Qihang Zhang
Ceyuan Yang
Yujun Shen
Yinghao Xu
Bolei Zhou
Venue: ICLR 2023 poster (2023)
Keywords: video generation
generative adversarial network

Abstract: Video generation, with the purpose of producing a sequence of frames, requires synthesizing consistent and persistent dynamic contents over time. This work investigates how to model the temporal relations for composing a video with arbitrary number of frames, from a few to even infinite, using generative adversarial networks (GANs). First, towards composing adjacent frames, we show that the alias-free operation for single image generation, together with adequately pre-learned knowledge, bring a smooth frame transition without harming the per-frame quality. Second, through incorporating a temporal shift module (TSM), which is originally designed for video understanding, into the discriminator, we manage to advance the generator in synthesizing more reasonable dynamics. Third, we develop a novel B-Spline based motion representation to ensure the temporal smoothness, and hence achieve infinite-length video generation, going beyond the frame number used in training. We evaluate our approach on a range of datasets and show substantial improvements over baselines on video generation. Code and models are publicly available at \url{https://genforce.github.io/StyleSV}.

Published as a conference paper at ICLR 2023
TOWARDS SMOOTH VIDEO COMPOSITION
Qihang Zhang' Ceyuan Yang² Yujun Shen Yinghao Xu
Bolei Zhou
TTe Chinese University Hong Kong. Shhnghaii Al Laborator,, UUniversity of California, Los Angeles
Ann Group.
Figure 1: Generated video frames from the proposed model trained on Countryside, YouTube Driving (Zhang al., 2022), and Sky 3kyTimelapse (Xiong et al., 2018) (from top to bottom).
ABSTRACT
Video generation, with the purpose of producing sequence of frames, requires synthesizing consistent and persistent dynamic contents over time. This work investigates how model the temporal relations for composing video with arbitrary number of frames, from few to even infinite, using generative adversarial networks (GANs). First, towards composing adjacent frames, show that the alias-free operation for single image generation, together with adequately pre-leamndd knowledge, bring smooth frame transition without harming the per- frame quality. Second, through incorporating a temporal shift module (TSM). which is originally designed for video understanding, into the discriminator, we manage to advance the generator in synthesizing more reasonable dynamics. Third, we develop novel B-Spline based motion representation to ensure the temporal smoothness, and hence achieve infinite-length video generation, going beyond the frame number used training. We evaluate our approach on a range of datasets and show substantial improvements over baselines video generation. Code and models are publicly available at https: / ggnforceeeeeeeeeooooooooooooecee qi github. i //SSttyeSS
INTRODUCTION
Synthesizing images using generative adversarial network (GAN) (Goodfellow et al., 2014; Radford et al., 2016; Karras et al., 2019; 2020b; 2021; 2018) usually requires compose diverse visual concepts with fine details of single object and plausible spatial arrangement of different objects. Recent advances in GANs have enabled many appealing applications such customized editing (Goetschalckx et al., 2019; Shen et al., 2020; Jahanian et al., 2020; Yang et al., 2021) and animation (Qiu al., 2022; Alaluf et al., 2022). However, employing GANs for video generation remains challenging considering the additional requirement on the temporal dimension.
In fact, video is not simply stack of images. Instead, the contents in video frames should have a smooth transition over time, and the video may last arbitrarily long. Thus, compared to image synthesis, the crux of video synthesis lies in modeling the temporal relations across frames. We

---

Published as a conference paper at ICLR 2023
argue that the temporal relations fall into three folds regarding the time scale. First, when looking at a transient dynamic, we would focus more on the subtle change between neighbor frames and expect decent local motions, such as facial muscle movement and cloud drifting. Along with the duration getting longer, say a segment, more contents within the frame may vary. Under such a case, learning consistent global motion is vital. For example, in video of first-view driving, trees and buildings along the street should move backward together with the car running forward. Finally, for those extremely long videos, the objects inside are not immutable. It therefore requires the motion to be generalizable along the time axis in a continuous and rational sense.
This work targets smooth video composition through modeling multi-scale temporal relations with GANs, First, we confirm that, same as in image synthesis, the texture sticking problem (i.e., some visual concepts are bound to their coordinates) also exists in video generation, interrupting the smooth flow of frame contents. To tackle this obstacle, we borrow the alias-free technique (Karras et al., 2021) from single image generation and preserve the frame quality via appropriate pre- training. Then, to assist the generator in producing reasonable dynamics, we introduce temporal shift module (TSM) (Lin et al., 2019) into the discriminator as an inductive bias. That way, the discriminator could capture more information from the temporal perspective for real/fake classification, serving as a better guidance to the generator. Furthermore, we observe that the motion representation in previous work (Skorokhodov et al., 2022) suffers from undesired content jittering (see Sec. 2.4 for details) for super-long video generation. We identify the cause of such a phenomenon as the first-order discontinuity when interpolating motion embeddings. Towards this problem, we design a novel B-Spline based motion representation that could soundly and continuously generalize over time. A low-rank strategy is further proposed alleviate the issue that frame contents may repeat cyclically. We evaluate our approach on various video generation benchmarks, including YouTube driving dataset (Zhang et al., 2022), Sky Timelapse (Xiong et al., 2018), Taichi-HD (Siarohin et al., 2019b) and observe consistent and substantial improvements over existing alternatives. Given its simplicity and efficiency, our approach sets up a simple yet strong baseline for the task of video generation.
2
METHOD
We introduce the improvements made on the prior art StyleGAN-V (Skorokhodov et al., 2022) to set new baseline for video generation. We first introduce the default configuration (Config-A) in Sec. 2.1. We then make a comprehensive overhaul on it. Concretely, in Sec. 2.2 we confirm that the alias-free technique (Config-B) in single image generation, together with adequately pre-learned knowledge (Config-C), C) could result in a smooth transition of two adjacent frames. Sec. 2.3 shows that when the temporal information is explicitly modeled into the discriminator through the temporal shift module (Lin et al., 2019) (Config-D), the generator can produce significantly better dynamic content across frames. Although prior arts could already generate arbitrarily long videos, the cyclic jittering is observed as time goes by. We therefore propose a B B-Spline based motion representation (Config-E) E) to ensure the continuity, which together with a low-rank temporal modulation (Config- F) could produce much more realistic and natural long videos in Sec. 2.4.
2.1 PRELIMINARY
StyleGAN-V (Skorokhodov et al., 2022) introduces continuous motion representations and holistic discriminator for video generation. Specifically, continuous frames 4 could be obtained by feeding continuous / into a generator G(.):
7 = I=G mt);
where u and vi denote the content code and continuous motion representation. Concretely, content code sampled from standard gaussian distribution while the motion representation i, consists of two embeddings: time positional embedding of and interpolated motion embedding o!
To obtain the time positional embedding of we first randomly sample NA codes as a set of discrete time anchors A, i E [O.... NA- - 1] that share equal interval (256 frames in practice). Convolutional operation with ID kernel is then applied on anchor A; for temporal modeling, producing the corresponding features a, with timestamp t, For an arbitrary continuous t, its corresponding interval is first found with the nearest left and right anchor feature a; and 41+1, at
time 4 and 4441, that 4 4 E [0,..., "..,NA 2). For time positional embedding oh".
2

---

Published as a conference paper at ICLR 2023
Table 1: Performance evaluation on various configurations. Config-A is the baseline model, StyleGAN-V (Skorokhodov et al., 2022). Config-B and Config-C target fixing the texture sticking issue yet maintain the per-frame quality (Sec. 2.2), which is primarily evaluated by FID. Config-D aims to help the generator to produce more reasonable dynamics (Sec. 2.3), which is primarily evaluated by 16 and 12- Config-E and Config-F alleviate the discontinuity when interpolating motion embeddings (Sec. 2.4). For all metrics, lower number is better.
SkyTimelapse
YouTube Driving
Taichi-HD
Configuration
FID
FVD1a
FVD12s
FID
FVD16
FVD128
FID
FVD16
FVD128
A StyleGAN-V
40.8
73.9
248.3
28.3
449.8
460.6
33.8
152.0
267.3
B + Alias free
54.0
118.8
221.4
56.4
729.8
886.0
31.4
171.7
522.9
C + Image pretrain
52.2
73.5
230.3
15.6
272.8
447.5
20.8
104.3
314.2
D + TSM
49.9
49.0
135.9
19.2
207.2
221.5
26.0
84.6
176.2
E. B-Spline
63.5
64.8
185.3
21.8
281.6
375.2
25.55
82.6
169.7
F Low-rank
60.4
61.9
229.1
23.0
260.4
278.9
24.8
98.3
186.8
wave function is derived with learned frequency B, phase y and amplitude from the left anchor feature ay:
(a),) Mg(ar), Max)))
(2)
of =< sin(n t +),) a a ++
where Ma. Ma. M, are learnable MLPs. The interpolated motion embedding of"e could be obtained through linear interpolation between anchor feature aj and a1+1 based on
me - ti)at (t|+1 t)ai-1
(3)
411 4
The final motion representation U is the sum of these two terms:
01
of + me
(4)
Additionally, a sparse sampling strategy is proposed for efficient training. Namely, N, discrete time steps & and real video frames I, are sampled respectively. By feeding N; discrete time steps and one content code into generator, multiple synthesis 1 G(u, Ut,) are obtained. In order to distinguish real video frames from synthesized ones, a discriminator D(-) extracts feature yi for the individual frame, and then a fusion function θ would perform the temporal aggregation over all frames:
y ((y,) = O(D(T)), i= 0, N₂ 1.
(5)
The final discriminative logit / is computed by MLP M conditioned on time steps:
= M (y, ,,, (1).
(6)
the following context, we would make a comprehensive overhaul on it, leading to a new simple yet strong approach for video generation.
2.2 ALIAS-FREE OPERATIONS AND PRE-LEARNED KNOWLEDGE
When composing a few frames, they usually share similar visual concepts and micro motion gradually occurs. Therefore, the smooth transition between frames tends make the synthesis much more realistic. However, by examining the synthesized frames of StyleGAN-V, we find that texture sticking (Karras al., 2021) exists. To be specific, we track the pixels at certain coordinates video moves and the brush effect in Fig. 2a indicates that these pixels actually move little. Thus, the texture sticks to fixed coordinates.
Alias-free operations (Config-B). To overcome the texture sticking issue, we follow Style- GAN3 (Karras et al., 2021) to leverage the alias-free technique in the generator, as one could remove the unhealthy dependency shown in Fig. 2b. brush effect disappear, i.e., texture sticking significantly alleviated, with the aid of alias-free technique.
absolute pixel coordinates through decent signal processing, As is
3

---

Published as a conference paper at ICLR 2023
StyleGAN-V
Ours
time expansion
time expansion
Figure Visualization of the texture sticking issue. On the top are a few samples from continuous sequence of frames, while the bottom visualizes the varying content fixed position. We can observe the brush effect from baseline (Skorokhodov al., 2022), where the pixel values strongly bound to the coordinates. Instead, our approach achieves smoother frame transition between neighbor frames.
However, reported in Tab. I, using StyleGAN3 achieves lower performance in FID and FVD than before. One possible reason is that the training receipt is directly inherited from the original StyleGAN-V without any modification. Nevertheless, given that it may impair FID and FVD to some extent, we still take StyleGAN3 in for sake of its alias-free benefit.
Pre-learned knowledge (Config-C). As simply replacing the generator backbone with StyleGAN3 can deteriorate the synthesis, we propose to make the best of pre-learned knowledge to catch up with the performance. As specified ahove, the key challenge for video generation is model temporal relations between frames. However, this target is entangled with the urge to model single image distribution. We confirm that pre-training image level and then fine-tuning in video level can decouple these two targets well. As discovered in (Karras et al., 2019; Yang et al., 2021), GAN's s deep layers mainly control details, for example coloring, detailed refinements, and We hypothesize that these parts' knowledge can be pre-learned and re-used in the video generation learning process.
To be specific, we set N₂ as in Eq. (5), and train the whole network from scratch as the image pre-training stage. Then deep layers' weights are loaded the vidco fine-tuning stage. With pre- trained deep layers' weights, model sequential changes in structures which show natural movements. As shown Tab, 1, after adopting Config-C, the generated quality gets improved compared to Config-B. Now, with the alias- free technique, we achieve smooth frame transition without sacrificing much
the optimization process can focus early layers to learn to
quality.
2.3 EXPLICIT TEMPORAL REASONING IN DISCRIMINATOR
In an adversarial context, is essential to have a strong discriminator, assuring the sufficient training of the generator. In video generation task, the discriminator thus has to model the temporal relations of multiple frames to distinguish unnatural movements from real ones. However, a simple concatenation operation is adopted before: y yt. where yi denotes single frame's 's feature
extracted by the discriminator and y denotes the feature after temporal fusion.
Temporal modeling (Config-D). We therefore introduce an explicit temporal modeling approach, i.e. temporal shift module (TSM) (Lin al., 2019) that shows superiority standard video understanding area and, most importantly, introduces zero overhead. Concretely, at each convolution layer before temporal fusion, we have features from multiple frames: y E #x X i 0, N₁ 1. where n is the layer index, H and W are the feature map resolution and C' is the feature channel number. Then, TSM performs channel-wise swapping between adjacent frames:
v)(: Gl. w) 7
: Mi w) 7C
vi () +
:)),
where Ox(-) is a concatenation operation, and (x, y) are arbitrary coordinates. this way, quarter of the channels of single frame after temporal fusion contain information from
4

---

Published as a conference paper at ICLR 2023
StyleGAN-V
Ours
9.5 sec
10.5 sec
Figure 3: Visualization of the content jittering issue. As time goes by, can observe the unstable crossroad in the baseline model (Skorokhodov et al., 2022), caused by discontinuous motion representation. Instead, our approach alleviates such an issue with the proposed B-Spline motion representation, facilitating long video generation.
adjacent frames. Follow-up convolution kernels in deeper layers can perform efficient temporal reasoning based on this mixed representation. our ablation study, while FID gets worse slightly compared to Config-C. adopting explicit temporal modeling (Config-D) improves FVD16 and FVD 128 significantly, yielding over 100% improvement on YouTube Driving dataset.
2.4 TOWARDS UNLIMITED VIDEO GENERATION
Considering that the main metrics ie., FID, FVD1G and FVD12s 128 used before merely measuring the synthesis of relatively short videos (e.g., 128 frames usually cover around 6 seconds), we further investigate whether current configuration is able to produce arbitrarily long videos. Note that, by saying arbitrarily long, mention infinite length along time axis, but infinite content. Ideally, we could easily generate infinite frames by continuously sampling time t. The generator is supposed produce the corresponding synthesís. However, rollouting the video, conspicuous jittering phenomenon occurs periodically every 10 seconds. Concretely, as demonstrated in Fig. 3, the crossroad moves forward at the beginning, and then suddenly goes backwards.
Discontinuity of motion embeddings. As mentioned in Sec, 2.1, motion embedding or contains time positional embedding vf" and interpolated motion embedding v!"e, In particular, of could be obtained by a learned wave function on the left nearest anchor feature a while the motion embedding eite derives from the linear interpolation between the left and right anchor feature at and a1+1- This interpolation would result in the first-order discontinuity for both learned wave function and linear interpolation when getting through multiple anchors, as shown in Fig. 4a. Moreover, T-SNE (Van der Maaten & Hinton, 2008) is applied to investigate the motion embedding given long period. Fig. 5a confirms that such discontinuity would cause the drastic change, i.e., jittering content.
B-Spline based motion representations (Config-E). With assumption of the discontinuity of motion embeddings, we design a basic spline (B-Spline) based motion representations that could guarantee first-order numerical smoothness. To be specifie, B-Spline of order is piece- wise polynomial function of degree n 1 in a given variable. It is widely used in computer- and computer graphics for curve fitting, controlling a smooth curve with calculated weights w; on several control points. In particular, the numher of control points is determined by the order of the B-Spline. With pre-defined knot sequence [t+]. B-Spline with any order can be defined by means of the Cox-de Boor recursion formula (de Boor, 1971). The first-order B-Spline is then defined by:
design
(1) == =(²
St
(8)
otherwise.
Here i is the interval index. B-spline of higher orders is further defined recursively:
(i.++((t= = wi. (t. + (.)) (1)
(9)
where ls is the order and
t- 1
14t a l, otherwise.
(1.x(1) =
(10)
5

---

Published as a conference paper at ICLR 2023
Linear interpolation
b. B-spline controlled interpolation
W2
W1
Anchor feature
0 Interpolated motion code
W2 Interpolated weight
Figure 4: Comparison between linear interpolation and our B-Spline controlled interpolation. B-spline is able to smooth the interpolation between various anchor features, and hence improve the first-order numerical continuity.
We treat each anchor feature ag the control point and use (t, its corresponding weight given time t. Hence, B-Spline based anchor feature is defined as:
a(t) = Σ Bi,x(t)a
(11)
We further use a(t) instead of discretized a; to calculate time positional embedding vfe and interpolated motion embedding ozee defined in Eq. (2) and Eq. (3). Fig. 4b suggests the continuity property of B-Spline. Meanwhile, through T-SNE again, the B-Spline based motion representations become much smoother in Fig. 5b. As shown Fig. the crossroad gradually approaches the observer without any sudden jittering.
Low-rank temporal modulation (Config-F). After adopting B-Spline based motion embedding, the jittering phenomenon gets erased. However, we find that similar content periodically appears as time goes by (see Fig. A6a). This phenomenon implies that motion embedding represents visual concepts in an ill manner. We thus hypothesize that the new motion embedding might be endowed with a stronger capacity to represent various visual concepts. Inspired by recent works (Bau al., 2020; Wang et al., 2022a), we suppress the representation capacity through the low-rank strategy.
Original StyleGAN-based architecture incorporates styles into convolutional kernels via modulation trick (Karras et al., 2020a). When generating videos, similar technique is also applied with small modifications. Content embedding together with the motion embedding vn would be fed into an affine layer M, generating the style code to modulate the original weight W:
W W M(u e vt),
(12)
where @ stands for the concatenation operation. That is, motion and content embedding could equally contribute to the final style to change the visual concepts of frames. order suppress the representational capacity of motion embeddings, we first separate the motion and content codes and then replace the kernel regarding the motion code with a low-rank one:
W
Weo Mc)(u) + Wm. (m), where Wm. UV,
(13)
where U, V is tall matrix that guarantees Wmo is low-rank matrix. With such low-rank strategy, the capacity of motion embedding is suppressed. The repeating issue is alleviated in Fig. A6b.
In conclusion, generate short video (less than 10 seconds), Config-D is the best choice. In contrast, one can use Contig-F for long video generation.
3 EXPERIMENTS
We evaluate our method on multiple datasets and compare it against prior arts. Additionally, some good properties are also given to support the effectiveness of our approach.
Evaluation Metrics. Akin to previous literature, Frechet Inception Distance (FID) (Heusel et al., 2017) and Frechet Video Distance (FVD) (Unterthiner al., 2018) serve as the quantitative metrics

---

Published as a conference paper at ICLR 2023
StyleGAN-V
b. Ours
Figure 5: Visualization of motion embeddings using T-SNE (Van der Maaten & Hinton, 2008)- Each dot refers motion code U/ which represents single frame's motion. A set of progressively darkening dots construct one trajectory that represents a synthesized frame sequence. We can tell that there usually exist undesired twists in the motion trajectory of StyleGAN-V (Skorokhodov et al., 2022), indicating the discontinuity of motion embedding, On the contrary, our approach helps alleviate this issue, resulting in smoother transition throughout the entire video.
Quantitative comparison between approach and existing video generation methods on SkyTimelapse (Xiong et al., 2018). For all metrics, a smaller number is better.
Table 2:
Method
FID
FVD16
FVD
MoCoGAN (Tulyakov et al., 2018)
206.6
575.9
+ StylleGAN2 backhone
85.9
272.8
MoCoGAN-HD (Tian et al., 2021)
164.1
878.1
VideoGPT (Yan et al., 2021)
222.7
DIGAN (Yu al., 2022)
83.1
196.7
Long VideoGAN (Brooks et al., 2022)
116.5
152.7
TATS-base (Ge et al., 2022)
132.6
StyleGAN-V (Skorokhodov et al., 2022)
40.8
73.9
248.3
Ours
49.9
49.0
135.9
to evaluate the synthesis for image and video respectively. In terms of FVD, two temporal spans (covering consecutive 16 and 128 frames) are chosen to measure the synthesized videos.
Benchmarks. In Sec. 2, have already conducted comprehensive ablation studies on SkyTime- lapse (Xiong et al., 2018), Taichi-HD (Siarohin et al., 2019b), and YouTube Driving dataset (Zhang al., 2022). To be specific, SkyTimelapse contains over two thousand videos which in average last twenty seconds. Various types of scenes (daytime, nightfall, and aurora) are included. Similar to StyleGAN-V (Skorokhodov et al., 2022), resize the videos of SkyTimelapse to 256 256 resolution. Taichi-HI contains over three thousand videos recording a person performing Tai- chi. We also resize the videos 256 256 resolution. YouTube Driving dataset (Zhang et al., 2022) consists of 134 first-view driving videos with a total length of over 120 hours, covering up 68 cities, showing various conditions, from different weather, different regions, to diverse scene types. The driving scenes have tight geometrical constraints, with most of the objects (vehicles, buildings) following the rigid body constraint. We resize the videos 180 x 320 resolution. part of videos with countryside views in 320 x 640 resolution is also chosen to benchmark high resolution video generation. For training, we resize them to 256 x 256 and 512 x 512 resolution respectively. Additionally, we compare our approach against previous methods including MoCoGAN (Tulyakov et al., 2018) and its StyleGAN2 based variant, MoCoGAN-HD (Tian et al., 202 VideoGPT (Yan al., 2021), DIGAN (Yu et al., 2022), and StyleGAN-V (Skorokhodov al., 2022).
Training. We follow the training receipt of StyleGAN-V and train models on a server with A100 GPUs. In terms of various methods and datasets, we grid search the R₁ regularization weight, whose details are available in Appendix. In particular, performances MoCoGAN and its StyleGAN2 based variant, MoCoGAN-HD,, VideoGPT, and DIGAN are directly borrowed from StyleGAN-V.

---

Published as a conference paper at ICLR 2023
Table Quantitative comparison between our approach and existing video generation methods on Taichi-HD (Siarohin et al., 2019b), For all metrics, smaller number is better.
Method
FID
FVD1s
FVD128
MoCoGAN-HD (Tian et al., 2021)
144.7
DIGAN (Yu et al., 2022)
128.1
TATS-base (Ge et al., 2022)
94.6
StyleGAN-V (Skorokhodov et al., 2022)
33.8
152.0
267.3
Ours
26.0
84.6
176.2
Table 4: Quantitative comparison between our approach and StyleGAN-V (Skorokhodov al., 2022) on YouTube Driving dataset (Zhang et al., 2022). For all metrics, a smaller number is better.
Resolution
Method
FID
FVD1e
FVD128
StyleGAN-V
28.3
449.8
460.6
128
Ours
19.2
207.2
221.5
StyleGAN-V
14.6
262.4
285.4
512
Ours
14.5
116.0
139.2
3.1 MAIN RESULTS
As shown in Tab. 2, method significantly outperforms existing baselines SkyTimelapse in terms of FVD16 1 and FVD12s- On Taichi-HI, our method achieves consistent improvement in terms of FID, FVD16. 16. and FVD12s reported Tab. 3. We also compare our method with StyleGAN-V in challenging YouTube Driving dataset. at both 256 x 256 and 512 x 512 resolution. For 256 x 256 resolution Tab. 4, our method achieves better performance in both FID, FVD16. and FVD12s- For 512 x 512 resolution, StyleGAN-V has a comparable FID score but much worse FVD score. For both two resolutions on YouTube driving, we yield a strong performance of less than 50% FVD score than the prior art. As shown in Fig. 1, our method achieves an appealing visual effect, with smooth frame transition.
3.2 PROPERTIES
Good translation equivariance. We adopt alias-free technique from image generation for smooth frame transition. This design thus allows for geometric manipulation of the video with strong translation equivariance using human-specified translation matrix. We showcase several examples in Fig. 6 by feeding different translation matrix to the video generator. It is clear that videos can be successfully translated, and translation equivariance well maintained over time. This equivariance property improves human controllability over the video generation process.
No jittering phenomenon. As mentioned in Sec. 2.4, discontinuity exists in the motion representation in the prior method, around which jittering phenomenon appears in generated videos. It is caused by discretized anchor features with fixed interval lengths (256 frames in practice). Our method develops a novel B-Spline based motion represen- tation to ensure the temporal smoothness over the anchor
Table 5: FVD evaluated the frames around anchors.
FVD16
Config-D Config-F
207.2
269.1 (+61.9)
260.4
250.2 (-10.2)
move left
move down
move right
move up
Figure 6: Application of video 2D translation, where we are able to control the shift of the frame content, such as moving the setting sun left and moving the grassland down.
8

---

Published as a conference paper at ICLR 2023
features. To quantify the smoothness of generated videos, we sample 16 frames across the anchor (eight frames before and behind the anchor) and calculate the 11D1s metric (which we name "FID "ancho-" in YouTube Driving dataset. As reported in Tab. 5, Config-D suffers from severe performance drop, while Config-F with B-Spline based motion representation maintains the performance.
4 RELATED WORK
Synthesizing videos from latent space. Most of the video generation frameworks are built upon GANs (Goodfellow et al., 2014; Brock et al., 2019; Karras et al., 2019; 2020h; 2021; 2018; Radford al., 2016), owing their flexibility as well as strong performance in image generation. Prior arts (Tian et al., 2021; Saito et al., 2017: Tulyakov et al., 2018; Fox et al., 2021; Skorokhodov et al., 2022; Yu et al., 2022; Vondrick et al., 2016; Saito al., 2020; Clark et al., 2019; Wang et al., 2022b) adopt image generators to synthesize frames from a time-coheren sequence of latent codes generated by recurrent networks. To explicitly disentangle the motion from the content, MoCoGAN (Tulyakov et al., 2018) and TGAN (Saito et al., 2017) employ motion code and a content code as input noise for the generator, serving as common strategy for the later works. StyleGAN-V (Skorokhodov et al., 2022), which we use as strong baseline, and DIGAN (Yu et al., 2022) both use implicit neural-based representations for continuous video synthesis. Brooks et al. (2022) leverages a multi- resolution training strategy to prioritize the time axis and ensure long-term consistency. Another line of works (Yan et al., 2021; Kalchbrenner et al., 2017; Weissenborn et al., 2019; Rakhimov et al., 2020; Walker et al., 2021) 1) use autoregressive models et al. (2017) use a PixeICNN (Van den Oord et al., 2016) decoder to synthesize the next frame pixel by pixel in an autoregressive manner. VideoGPT (Yan al., 2021) built upon VQ-VAE adopts an autoregressive model to decompose videos into sequence of tokens. TATS-base (Ge et al., 2022) similarly leverages 3D- VQGAN decompose videos into tokens and further uses transformers to model the relationship between frame tokens. Additionally, some works using diffusion- based models also present promising results on video generation. Video diffusion (Ho al., 2022) models entire videos using 3D video architecture while Yang et al. (2022) uses an image diffusion model to synthesize frames with a recurrent network.
achieve video generation. Kalchbrenner
Synthesizing videos with guided conditions. A close line of research is video prediction, aiming to generate full videos from the observation of previous frames (Babaeizadeh et al., 2017; Kumar et al., 2019; Lee et al., 2018; Lue et al., 2020; Nash et al., 2022; Finn et al., 2016) or given conditions (Kim et al., 2020; 2021; Ha & Schmidhuber, 2018). These works typically employ the reconstruction losses to make the future frames predictable for the model. Video translation (Pan et al., 2019; Yang et al., 2018; Wang et al., 2018; Chan et al., 2019; Siarohin et al., 2019a; Ren et al., 2020; Siarohin et al., 2019b) is also a paradigm for video synthesis, which translates the given segmentation masks and keypoints into videos. Some (Li et al., 2022; Liu et al., 2021; Ren & Wang, 2022) focus on synthesizing videos of indoor and outdoor scenes for given camera trajectories, which need to model the 3D scene explicitly. Besides, Ho et al. (2022); Hong et al. (2022); Wu al. (2021a;b) explore text-to-video generation with diffusion- based models and transformers.
5 CONCLUSION
In this work, we set up a simple yet effective framework for video generation task. By introducing the alias-free operations together with pre-learned knowledge, per-frame quality is improved. Explicit temporal modeling in discriminator enhances the dynamic synthesis. In addition, a new B-Spline based motion representation with low-rank temporal modulation is proposed to improve infinite frame generation. Experimental results show substantial improvements on different datasets.
Discussion. Our approach has several limitations. Our current framework uses two latent codes to represent content and motion, but they often entangle with each other. Moreover, the frame quality is still far from satisfactory. The object shape is not consistent across consecutive frames, producing apparent artifacts. This could be further improved by introducing structural priors. Last, although could generate infinite videos by gradually changing content code, direct control of novel content over time is lacking. Explicit planning in the latent space, as used by Ge et al. (2022), could improve the quality of generated long videos.

---

Published as a conference paper at ICLR 2023
REFERENCES
Yuval Alaluf, Or Patashnik, Zongze Wu, Asif Zamir, Eli Shechtman, Dani Lischinski, and Daniel Cohen-Or. Third time's the charm? image and video editing with stylegan3. arXiv preprint 1rXiv:2201.13333 2022.
Mohammad Babaeizadeh, Chelsea Finn, Dumitru Erhan, Roy Campbell, and Sergey Levine. Stochastic variational video prediction. arXiv preprint 17100 1255, 2017.
David Bau, Steven Liu, Tongzhou Wang. Jun- Man Zhu, and Antonio Torralba. Rewriting deep generative model. In European conference on computer vision, pp. 351-369. Springer, 2020.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In Int. Conf. Learn. Represent., 2019.
Tim Brooks, Janne Hellsten, Miika Aittala, Ting-Chun Wang, Timo Aila, Jaakko Lehtinen, Ming- Yu Liu, Alexei A Efros, and Tero Karras. Generating long videos of dynamic scenes. arXiv preprint arXiv:2206.0344 03429, 2022.
Caroline Chan, Shiry Ginosar, Tinghui Zhou, and Alexei A Efros. Everybody dance now. In Int. Conf. Comput. Vis., 2019.
Aidan Clark, Jeff Donahue, and Karen Simonyan. Adversarial video generation on complex datasets. arXiv preprint arXiv: (xXiv:/907006571 2019.
Carl de Boor. Subroutine package for calculating with b-splines. Technical report, Los Alamos National Lab.(LAN.), Los Alamos, NM (United States), 1971.
Chelsea Finn, Ian Goodfellow, and Sergey Levine. Unsupervised learning for physical interaction through video prediction. Adv. Neural Inform. Process. Syst., 2016.
Gereon Fox, Ayush Tewari, Mohamed Elgharib, and Christian Theobalt. Stylevideogan: A temporal generative model using pretrained stylegan. arXiv preprint urXiv:2107.07224 2021.
songwei Ge, Thomas Hayes, Harry Yang, Xi Yin, Guan Pang, David Jacobs, Jia-Bin Huang, and Devi Parikh. Long video generation with time-agnostic vqgan and time-sensitive transformer. arXiv preprint arXiv:2204.036: 2022.
Lore Goetschalekx, Alex Andonian, Aude Oliva, and Phillip Isola. Ganalyze: Toward visual definitions of cognitive image properties. In Int. Conf. Comput. Vis., pp. 5744-5753, 2019.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Adv. Neural Inform. Process. Syst., 2014.
David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. In Adv. Neural Inform. Process. Syst., 2018.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. arXiv preprint arXiv:2204.03458 2022.
Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. arXiv preprint arXiv.2205.15868, 2022.
Ali Jahanian, Lucy Chai, and Phillip Isola. On the" steerability" of generative adversarial networks. Int. Conf. Learn. Represent., 2020.
Nal Kalchbrenner, Aäron Oord, Karen Simonyan, Ivo Danihelka, Oriol Vinyals, Alex Graves, and Koray Kavukcuoglu. Video pixel networks. In Int. Conf. Mach. Learn,, 2017.
10

---

Published as a conference paper at ICLR 2023
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. In Int. Conf. Learn. Represent., 2018.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In IEEE Conf. Comput. Vis. Pattern Recog., 2019.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8110-8119, 2020a.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In IEEE Conf. Comput. Vis. Pattern Recog., pp. 81 10-8119, 2020b.
Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. NIPS, 34:852-865 2021.
Seung Wook Kim, Yuhao Zhou, Jonah Philion, Antonio Torralba, and Sanja Fidler. Learning to Simulate Dynamic Environments with GameGAN. In IEEE Conf. Comput. Vis. Pattern Recog., 2020.
Seung Wook Kim, Jonah Philion, Antonio Torralba, and Sanja Fidler. DriveGAN: Towards a Controllable High-Quality Neural Simulation. In IEEE Conf. Comput. Vis. Pattern Recog., 2021.
Manoj Kumar, Mohammad Babaeizadeh, Dumitru Erhan, Chelsea Finn, Sergey Levine, Laurent Dinh, and Durk Kingma. Videoflow: conditional flow-based model for stochastic video generation. arXiv preprint arXiv: 1903.01400 2019.
Alex X Lee, Richard Zhang, Frederik Ebert, Pieter Abbeel, Chelsea Finn, and Sergey Levine. Stochastic adversarial video prediction. arXiv preprint arXiv:/804.0152: 2018.
Zhengqi Li, Qianqian Wang, Noah Snavely, and Angjoo Kanazawa. Infinitenature-zero Learning perpetual view generation of natural scenes from single images. arXiv preprint 2111: 2022.
Lin, Chuang Gan, and Song Han. Tsm: Temporal shift module for efficient video understanding. In Int. Conf. Comput. Vis., pp. 7083-7093, 2019.
Andrew Liu, Richard Tucker, Varun Jampani, Ameesh Makadia, Noah Snavely, and Angjoo Kanazawa. Infinite nature: Perpetual view generation natural scenes from a single image. In IEEE Conf. Comput. Vis. Patlern Recog., 2021.
Pauline Luc, Aidan Clark, Sander Dieleman, Diego de Las Casas, Yotam Doron, Albin Cassirer, and Karen Simonyan. Transformation based adversarial video prediction on large-scale data. arXiv preprint arXiv:2003.04033. 2020.
Charlie Nash, João Carreira, Jacob Walker, lain Barr, Andrew Jaegle, Mateusz Malinowski, and Peter Battaglia. Transframer: Arbitrary frame prediction with generative models. arXiv preprint 2022.
arXiv:2203.cm 09494, 2009.0994,
Junting Pan, Chengyu Wang, Xu Jia, Jing Shao, Lu Sheng, Junjie Yan, and Xiaogang Wang. Video generation from single semantic label map. In IEEE Conf. Comput. Vis. Pattern Recog., 2019.
Haonan Qiu, Yuming Jiang, Hang Zhou, Wayne Wu, and Ziwei Liu. Stylefacev: Face video gener- ation via decomposing and recomposing pretrained stylegan3. arXiv preprint urXiv:2208.07862. 2022.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In Ini. Conf. Learn. Represent., 2016.
Ruslan Rakhimov, Denis Volkhonskiy, Alexey Artemov, Denis Zorin, and Evgeny Burneev. Latent video transformer. arXiv preprint arXiv:2006. 10704, 2020.
Jian Ren, Menglei Chai, Sergey Tulyakov, Chen Fang, Xiaohui Shen, and Jianchao Yang. Human motion transfer from poses in the wild. In Eur. Conf. Comput. Vis., 2020.
11

---

Published as a conference paper at ICLR 2023
Xuanchi Ren and Xiaolong Wang. Look outside the room: Synthesizing consistent long-term 3d scene video from a single image. IEEE Conf. Comput. Vis. Pattern Recog., 2022.
Masaki Saito, Eiichi Matsumoto, and Shunta Saito. Temporal generative adversarial nets with singular value clipping. In Int. Conf. Comput. Vis., 2017.
Masaki Saito, Shunta Saito, Masanori Koyama, and Sosuke Kobayashi. Train sparsely, generate densely: Memory-efficien unsupervised training of high-resolution temporal gan. Int. J. Comput. Vis., 2020.
Yujun Shen, Ceyuan Yang, Xiaoou Tang, and Bolei Zhou. Interfacegan: Interpreting the disentangled face representation learned by gans. IEEE Trans. Pattern Anal. Mach. Intell., 2020.
Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. Animating arbitrary objects via deep motion transfer. In JEEE Conf. Comput. Vis. Pattern Recog. 2019a.
Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. First order motion model for image animation. In Adv. Neural Inform. Process. Syst., 2019b.
Ivan Skorokhodov, Sergey Tulyakov, and Mohamed Elhoseiny. Stylegan-v: A continuous video generator with the price, image quality and perks of stylegan2. In IEEE Conf. Comput. Vis. Pattern Recog., pp. 3626-3636, 2022.
Yu Tian, Jian Ren, Menglei Chai, Kyle Olszewski, Xi Peng, Dimitris N Metaxas, and Sergey Tulyakov. good image generator is what you need for high-resolution video synthesis. Int. Conf. Learn. Represent., 2021.
Sergey Tulyakov, Ming-Yu Yu Liu, Xiaodong Yang, and Jan Kautz. Mocogan: Decomposing motion and content for video generation. In IEEE Conf. Comput. Vis. Pattern Recog., 2018.
Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A metric & challenges. arXiv preprint arXiv:1812.017177 2018
Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Conditional image generation with pixelenn decoders. Adv. Neural Inform. Process. Syst., 2016.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dynamics. Adv. Neural Inform. Process. Syst., 2016.
Jacob Walker, Ali Razavi, and Aäron van den Oord. Predicting video with vqvae. arXiv preprint arXiv:2103.01950, 2021.
Sheng-Yo Yu Wang, David Bau, and Jun- Zhu. Rewriting geometric rules of gan. ACM Transactions on Graphics (TOG), (14 2022a.
Ting Ting-Chun Wang, Ming- Yu Liu, Jun-Yan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. Video-to-vides synthesis. arXiv preprint arXiv: (at:::888.06600 2018.
Yaohui Wang, Di Yang, Francois Bremond, and Antitza Dantcheva. Latent image animator: Learning to animate images via latent space navigation. arXiv preprini arXiv:2203.088 09044,, 2022b.
Dirk Weissenborn, Oscar Täckström, and Jakob Uszkoreit. Scaling autoregressive video models. arXiv preprint arXiv: (:XXv:1906.026344 2019.
Chenfei Wu, Lun Huang, Qianxi Zhang, Binyang Li, Lei Ji, Fan Yang, Guillerno Sapiro, and Nan Duan. Godiva: Generating open-domain videos from natural descriptions. arXiv preprint 2rXiv:2104.14806 202
Chenfei Wu, Jian Liang, Lei Ji, Fan Yang. Yuejian Fang. Daxin Jiang, and Nan Duan. NY"" uwa: Visual synthesis pre-trate are-training for neural visual world creation. arXiv preprint 2rX1v:2111.12417, 2021b.
12

---

Published as a conference paper at ICLR 2023
Wei Xiong, Wenhan Luo, Lin Ma, Wei Liu, and Jiebo Luo. Learning to generate time-lapse videos using multi-stage dynamic generative adversarial networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 2364-2373, 2018.
Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using vq-vae and transformers. arXiv preprint arXiv:2104 10157, 2021.
Ceyuan Yang, Zhe Wang, Xinge Zhu, Chen Huang, Jianping Shi, and Dahua Lin. Pose guided human video generation. In Eur. Conf. Comput. Vis., 2018
Ceyuan Yang, Yujun Shen, and Bolci Zhou. Semantic hierarchy emerges in deep generative representations for scene synthesis. Int. J. Comput. Vis., 12965:1151-1446, 2021.
Ruihan Yang. Prakhar Srivastava, and Stephan Mandt. Diffusion probabilistic modeling for video generation. arXiv preprint Xiv::203.0948.. 2022.
Sihyun Yu, Jihoon Tack, Sangwoo Mo, Hyumsu Kim, Junho Kim, Jung- Woo Ha, and Jinwoo Shin. Generating videos with dynamics-aware implicit generative adversarial networks. arXiv preprint 1.Xi::2202.1057,, 2022.
Qihang Zhang. Zhenghao Peng, and Bolei Zhou. Learning to drive by watching youtube videos: Action-conditions contrastive policy pretraining. In European Conference on Computer Vision, pp. 128.. Springer, 2022.
13

---

Published as a conference paper at ICLR 2023
APPENDIX
This appendix organized as follows. appendix A and appendix B present the implementation and dataset details respectively, followed by appendix D showing more visual results.
A IMPLEMENTATION DETAILS.
Our method is developed based on the official implementation of StyleGAN-V (Skorokhodov et al., 2022). We adopt syper-parameters the optimizer, the loss function, and the training script to ensure a fair comparison. Notably, the total number of seen images for sufficient data training is 25 million regardless of datasets. We evaluate once after training on every 2.5 million images, and report the result with highest FVD1e score.
Due to statistics variance of different datasets, we search training regularization term, iee. R₁ value, for each method and dataset. Empirically, we find that a smaller R1 value (e.g., 0.25) works well for pretraining stage (Config-C). While a larger R₁ value (e.g., 4) better suits video generation learning.
B DATASET DETAILS.
In this section, We introduce the datasets we use, briefly analyze their characteristics and challenges brought for video generation, and show sampled real video frames.
SkyTimelapse. SkyTimelapse (Xiong et al., 2018) is time-lapse dataset collected from the Internet showing dynamic sky scenes, such as the cloudy sky with moving clouds, and the starry sky with moving stars. It contains various conditions, for instance, different weather conditions (daytime, nightfall, dawn, starry night and aurora), different kinds of foreground object (cloudy sky, starry sky, and sunny sky), and different motion patterns of the sky. It also contains some other objects like trees, mountains, and buildings which further improve the visual diversity. However, since clouds fluids, there are very few constraints movement. Minor disturbances and deformations can make a generated video look realistic.
The whole dataset contains over thousand videos which in average last twenty seconds. We resize the videos to 256 x 256 following prior works. We sample several video clips in Fig. Al.
YouTube Driving. YouTube Driving Dataset (Zhang al., 2022) is crawled from YouTube which contains a massive amount of real-world driving frames with various conditions, from different weather, different regions, diverse scene types. To be specific, 134 videos with total length of over 120 hours are collected, covering up to 68 cities. Fig. A2 shows sampled frames. The videos are resized 256 256 resolution for training.
The driving scenes have tighter geometrical constraints, with most of the objects (vehicles, buildings) following the rigid body constraint. The 3D structure of the whole scene should be maintained without any deformation. Besides, different types of objects show different motion patterns. Roads and buildings recede at reasonable speed opposite to that ego car, while other cars show lane changes, stops, accelerations.
Countryside. We select a part of videos with countryside scenes from YouTube Driving dataset and resize them to 512 x 512 resolution for training. We use this dataset to benchmark the performance of generating high-resolution videos. Real samples are presented in Fig. A3.
Taichi-HD. Taichi-HD (Siarohin et al., 2019b) is a collection of YouTube videos recording single person performing Tai-chi. It contains 3049 training videos in total, with various foreground objects (pcople with different styles of clothing) and diverse background. It requires realistic human motion generation, which contains non-rigid motion modeling, which poses significant challenge for video generation.
14

---

Published as a conference paper at ICLR 2023
Table AI: User study result. We report the percentage of favorite users for the quality of set of single images, set of short videos, and set of long videos under models from three different settings.
YouTube Driving
Taichi-HD
image( %)
short video(%))
long video(%)
image( (%)
short video %6
long video (6)
Config-A
Config-D
62
60
12
52
66
36
Config-F
34
34
80
48
32
60
C USER STUDY
To further prove our method's efficacy, we conduct user study over baseline (Config-A), Config- D, and Config-F on YouTube Driving dataset and Taichi-HD dataset. We evaluate from three perspectives:
Single Image Quality. We provide user with collection of images randomly sampled from generated videos.
Short Video Quality. We provide user with collection of generated videos with consecutive 128 frames (up to about 5 seconds).
Long Video Quality. We provide user with a collection of generated videos with consecutive 500 frames (up 20 seconds).
As shown Tab. A1, terms single image quality, users prefer Config-D over other settings in both YouTube Driving and Taichi-HD dataset. The same trend occurs in short video quality assessment. With pre-learned knowledge and stronger discriminator with temporal shift module (TSM), Config-D reaps significantly higher number of votes. However, for long video generation, Config-F becomes the best choice for both YouTube Driving and Taichi-HI dataset. Unnatural jittering and discontinuous motion can drastically affect the users' video viewing experience. With B-Spline based motion embedding and low rank time modulation, Config-F is undoubtedly the best option for long video generation.
D MORE VISUAL RESULTS.
Texture Sticking phenomenon. Texture sticking exists in videos generated by prior methods. Our method leverage the alias-free technique in the generator to overcome this issue. Fig. A4 shows more comparison between our method and StyleGAN-V (Skorokhodov al., 2022).
Jittering phenomenon. Motion embedding designed in previous method is not first-order continuous, leading to jittering phenomenon when generating long videos. We develop B-Spline based motion embedding with smooth property which alleviates the problem. We show examples comparing video sequences generated by our method with StyleGAN-V (Skorokhodov et al., 2022) in Fig. A5. Our videos are smooth, without abnormal movements. In videos generated by StyleGAN-V (Skorokhodov et al., 2022), street lights and zebra crossings may suddenly change their direction of movement.
Repeating contents. With B-Spline based motion embedding, Config-E can compose smooth long videos. However, similar contents appear periodically as shown in Fig. A6a. B- B-Spline based motion embedding is endowed with so strong capacity that represents content and motion concepts simultaneously rather than motion concepts only. We suppress the representation capacity through the low-rank strategy. As shown in Fig. A6b, Config-F with low rank time modulation can generate progressively forward contents without the repeating phenomenon.
15

---

Published as a conference paper at ICLR 2023
Figure A1: Sampled video frames from SkyTimelapse (Xiong et al., 2018).
Figure A2: Sampled video frames from self-collected YouTube Driving dataset.
16

---

Published as a conference paper at ICLR 2023
Figure A3: Sampled video frames from self-collected Countryside dataset.
17

---

Published as a conference paper at ICLR 2023
StyleGAN-V
Ours
time expansion
time expansion
Figure A4: More examples texture sticking phenomenon. We concatenate slices in each frame the generated video (marked by red and green). We observe brush effect when texture sticks at specific location. Instead, our approach achieves more smooth frame transition between neighbor frames.
18

---

Published as a conference paper at ICLR 2023
StyleGAN-V
Ours
Figure A5: More examples on jittering phenomenon. Left three columns denote video frames produced by StyleGAN-V (Skorokhodov et al., 2022). Obviously, highlighted in red bounding boxes, the street lights and crossroad move unsteadily (e.g., gradually fading out first and suddenly fading in). On the contrary, our approach enables much more reasonable and continuous dynamics of certain objects (see green bounding boxes).
a. Config-E
b. Config-F
300
600
900
300
600
900
Figure A6: Examples on repeating contents. Left three columns denote video frames produced by Config-E the 300th, 600th, and 900th frame respectively. Similar contents (such as foreground objects, and scene layout) periodically appear. In contrast, after adopting low rank time modulation (Config-F), contents change gradually without repeating.
19


Reviews:
[{"review_id": "LrRc-XKwzeM", "reviewer": "Reviewer_pmsF", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes a method for smooth video synthesis. They build upon the SyleGan-V and progressively improve the baseline by proposing the alias-free operation, adding a temporal shift module, and incorporating the B-Spline based motion representation. It achieves improved temporal smoothness in the generated video without hurting the per-frame quality much. The experiments on SkyTimelapse and YouTube Driving datasets demonstrate the contributions of each component of the method. This work also considers long video generation which requires continuously sampling the temporal latent codes.", "summary_of_the_review": "The discussion on the worse per-frame FID score, and visual comparison on the same examples, and the best method among the Configs for short / long video will make this paper more complete.", "strength_and_weaknesses": "Strength\n+ They propose multiple techniques to siginficantly improve the video synthesis quality. Especially the FVD scores improved much compared to the prior arts.\n\n+ It resolves the brush effect which has been a problem in the previous methods, due to the generated contents being bound to the spatial coordinates in the image/frame.\n\n+ The visual quality of the generated video outperforms those of the previous methods.\n\n+ Long video generation is also considered, and is come up with novel methods to handle this.\n\nWeakness\n- While the proposed multiple techniques do improve over the baseline, the conclusion from the result is not very clear. In Table 1. Config-D shows the best FVD scores on the two datasets but not the methods with the best FID score are different over these two datasets. Why is the Config-E and F worse than D? What is the best Config for users?\n\n- While the paper argues the proposed method does not sacrifice the per-frame quality, the FID scores are worse than the baseline StyleGan-V. \n\n- The visual comparison (Figure 2, 3, A4, A5) shows the results on different sequence examples, which makes it hard to directly compare the performance between the baseline vs proposed method.\n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FouuKMkHGqw", "reviewer": "Reviewer_yrRK", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper improves the previous video generation model StyleGAN-V with several existing techniques and model-specific fixes. The proposed method shows better FVDs on the Sky Time-lapse dataset and a self-collected Driving dataset.", "summary_of_the_review": "I value some of the contributions of the paper such as adapting StyleGAN3 to video generation. However, I have multiple concerns regarding the experimental results as well as the conclusions drawn from them. I'm happy to hear from the authors about their thoughts on these concerns and reconsider my rating.", "strength_and_weaknesses": "[Strengths]\n\n1. The paper is very well-organized. The author gave a distinct description of the baseline StyleGAN-V model and clear discussions about the problems that existed in this model and how they propose to fix them. \n\n2. The motivations for adopting the temporal shift module (TSM) in the discriminator and pretraining the model on images first are reasonable, adapting techniques proposed in StyleGAN3 to video generation is also useful for the community. The experiments showing improvements brought by these modifications are relatively sound.\n\n[Weaknesses]\n\nThere are several flaws in the experiments of this paper:\n\n1. Only two landscape video datasets are used to evaluate the method. It is often the case that people are also interested in generating videos with human motions, which could imply different difficulties from landscapes. There are common video generation benchmark datasets for that purpose such as UCF-101, Tahich-HD, or FaceForensics where previous methods are evaluated on. However, none of these datasets is used in the paper. In addition, since the I3D model used to compute FVD scores is trained on the Kinetics dataset, I personally think that this metric will be more meaningful on the human motion datasets.\n\n2. Even though only two datasets are considered, the trends of the results on them are not well aligned. For instance, pretraining model on images helps on the driving dataset but not the sky dataset, while using TSM only improves the sky dataset. Is this because of the large variance? Probably more datasets could also help demystify these observations.\n\n3. For the main results, the authors adopt only those baselines used in the StyleGAN-V paper. However, since some of the results are focused on long video synthesis, the recently published methods on this problem should be considered [1, 2]. \n\n4. The most novel part of the method contains configurations E and F. The main goal of these two fixes is to resolve a known issue of StyleGAN-V which generates cyclic videos when extending to a longer duration. However, these fixes don't seem to improve the model in terms of quantitative evaluation. In addition, only a limited number of qualitative results (in the format of images) are given to demonstrate the improvements. Therefore, the claimed improvements are not convincing to me. Probably some metrics to detect the recurrence or human evaluation could better show the advantage of the proposed method.\n\nIn addition, the contribution of the proposed YouTube Driving dataset is not significate to me. There are multiple existing video datasets for driving or car dash cameras, e.g. BDD-100K [3] or Cityscapes[4]. There are also video generation works that are trained on these datasets. The paper lacks a comparison of the collected driving dataset with these existing datasets.\n\n[1] Long video generation with time-agnostic vqgan and time-sensitive transformer. ECCV 2022.\n\n[2] Generating long videos of dynamic scenes. NeurIPS 2022. \n\n[3] Bdd100k: A diverse driving dataset for heterogeneous multitask learning. CVPR 2020.\n\n[4] The cityscapes dataset for semantic urban scene understanding. CVPR 2016.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "5OCj9CaN1B", "reviewer": "Reviewer_U7vD", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper deals with generative modelling of videos, and specifically proposes a variety of improvements over StyleGAN-V. The improvements are intellectually appealing, and quite convincingly demonstrated. While Arxiv is starting to be full of video papers, many of which exceed the quality of this paper, they may not strictly speaking constitute prior art as even the NeurIPS decisions are not officially out yet. As such, I'm ok with the scientific novelty in this one. ", "summary_of_the_review": "This paper describes a clear step over StyleGAN-V, and as such I'm in favour of acceptance. Details are listed above. Obviously we need better quality metrics for video. To the best of my knowledge Sec. 2.2--2.4 are novel when compared to officially published work, but I may not know everything.", "strength_and_weaknesses": "Strengths:\n- Clear contributions\n- New ideas are explained well\n\nWeaknesses:\n- Prior art description (Sec 2.1) is confusing. What data is used to compute the anchor frame embeddings? I suspect the current description cannot be understood by anyone who doesn't already know how it works.\n- The result quality is still pretty low -- don't get me wrong, it's better than StyleGAN-V, but we definitely still see jittering in the longer videos and content warps around like crazy. These needs to be admitted and discussed in the paper.\n- \"Infinite\" video is a dubious concept. If the authors want to talk about it, then it needs to be defined. What constitutes an infinite video and what does not? Surely any interesting \"infinite\" video will need to introduce infinite amount of novel content over time? Otherwise it's hardly infinite -- although just ping ponging a short clip may technically be infinite, but that's hardly a productive definition. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2X-Jv2LHH_S", "reviewer": "Reviewer_xq3D", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed a GAN-based video generative model that is based upon StyleGAN-V while producing videos with significantly fewer artifacts. The paper first outlined several common artifacts seen in the videos produced by StyleGAN-V, and then provided an extensive set of targeted solutions:\n1. Alias-free techniques from StyleGAN3 is adopted to reduce the texture sticking artifacts at the cost of a slight drop in quantitative scores.\n2. To alleviate the quality drop, the author proposed pretraining the later layers on a image generation task and then finetuning the early layers on the actual video generation task.\n3. A *temporal shift module*, originally developed for video understanding, is incorporated into the discriminator, improving the quality further.\n4. To enable jitter-free long video generation, the motion latent space is instead modeled with a B-Spline.\n5. To reduce the periodic repeating artifact introduced by 4, a low-rank matrix is used to limit the effect of temporal features on the output frames.\n\nIn terms of quantitative results, the proposed method with improvement 1, 2, and 3 achieved the best video quality on an existing benchmark but inferior image quality. It however outperformed prior art in every aspect on a new, self-collected dataset. \n\nAdding improvement 4 and 5 will improve visual quality but reduce quantitative scores.", "summary_of_the_review": "I liked how the paper cast light on the practical issues in the previous works and then tacked them one-by-one, instead of focusing only on getting better numerical scores. This, as well as the techniques proposed in the paper, can serve as a good reference point for the future works. There are a few improvements that are not as well-justified as others, but it does not diminish other contributions.", "strength_and_weaknesses": "## Strength\n* The proposed method significantly improves upon the previous works in terms of video quality. What's more important, the paper conveyed a message that good quantitative scores does not always equal good visual quality perceived by humans. \n* The paper is easy to follow, and includes comprehensive quantitative experiments and good visualization.\n\n## Weaknesses\n* Most of the techniques used in the paper are not new. They are borrowed from related works.\n* The effect of setting F is not clear. There is no figures or videos showing what referred to as *similar content periodically appears*.\n* The new techniques introduced in the paper (setting E and F) significantly hurt the quantitative performance. Although claimed to have the benefit of reducing the jitter artifact, the non-visual evidence is weak (Table 4). It will be more persuasive if there are better evaluations, such as a user study.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster