Generating Long Videos of Dynamic Scenes
Authors: Tim Brooks
Janne Hellsten
Miika Aittala
Ting-chun Wang
Timo Aila
Jaakko Lehtinen
Ming-Yu Liu
Alexei A Efros
Tero Karras
Venue: NeurIPS 2022 Accept (2022)
Keywords: Video generation
GAN
generative model
dynamics
long videos

Abstract: We present a video generation model that accurately reproduces object motion, changes in camera viewpoint, and new content that arises over time. Existing video generation methods often fail to produce new content as a function of time while maintaining consistencies expected in real environments, such as plausible dynamics and object persistence. A common failure case is for content to never change due to over-reliance on inductive bias to provide temporal consistency, such as a single latent code that dictates content for the entire video. On the other extreme, without long-term consistency, generated videos may morph unrealistically between different scenes. To address these limitations, we prioritize the time axis by redesigning the temporal latent representation and learning long-term consistency from data by training on longer videos. We leverage a two-phase training strategy, where we separately train using longer videos at a low resolution and shorter videos at a high resolution. To evaluate the capabilities of our model, we introduce two new benchmark datasets with explicit focus on long-term temporal dynamics.

Generating Long Videos of Dynamic Scenes
Tim Brooks NVIDIA, UC Berkeley
Janne Hellsten NVIDIA
Miika Aittala NVIDIA
Ting-Chun Wang
Timo Aila NVIDIA
NVIDIA
Jaakko Lehtinen NVIDIA, Aalto University
Ming-Vo Yu Liu
Alexei A. Efros
Tero Karras
NVIDIA
UC Berkeley
NVIDIA
Abstract
We present video generation model that accurately reproduces object motion, changes in camera viewpoint, and new content that arises over time. Existing video generation methods often fail to produce new content as a function of time while maintaining consistencies expected in real environments, such as plausible dynam- ics and object persistence. A common failure case is for content to never change inductive biases provide temporal consistency, such single latent code that dictates content for the entire video. On the other extreme, without long-term consistency, generated videos may morph unrealistically be- tween different scenes. To address these limitations, we prioritize the time axis by redesigning the temporal latent representation and learning long-term consistency from data by training on longer videos. We leverage a two-phase training strategy, where we separately train using longer videos low resolution and shorter videos at a high resolution. To evaluate the capabilities of our model, we introduce two new benchmark datasets with explicit focus long-term temporal dynamics.
due to over-reliance
Introduction
Videos are data that change over time, with complex patterns of camera viewpoint, motion, deforma- tion and occlusion. In certain respects, vidcos are unbounded - they may last arbitrarily long and there is limit the amount of new content that may become visible over time. Yet videos that depict the real world must also remain consistent with physical laws that dictate which changes over time are feasible. For example, the camera may only move through 3D space along smooth path, objects cannot morph between each other, and time cannot go backward. Generating long videos thus requires the ability to produce endless new content while maintaining appropriate consistencies.
In this work, we focus on generating long videos with rich dynamics and new content that arises over time. While existing video generation models can produce "infinite" videos, the type and amount of change along the time axis is highly limited. For example, a synthesized infinite video of a person talking will only include small motions of the mouth and head. Moreover, common video generation datasets often contain short clips with little new content over time, which may inadvertently bias the design choices toward training on short segments or pairs of frames, forcing content in videos to stay fixed, or using architectures with small temporal receptive fields.
We make the time axis first-class citizen for video generation. To this end, we introduce two new datasets that contain motion, changing camera viewpoints, and entrances/exits of objects and scenery over time. We learn long-term consistencies by training on long videos and design temporal latent representation that enables modeling complex temporal changes. Figure iiluustrats the rich motion and scenery changes that our model capable of generating. See our webpage for video results. code, data and pretrained models.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Real
0.35
Ours StyleGAN-
Os
100
0.5s
Figure 1: We aim to generate videos that accurately portray motion, changing camera viewpoint, and new content that arises over time. Top: Our horseback riding dataset exhibits these types of changes as the horse moves forward in the environment. Middle: StyleGAN-V, video generation baseline, is incapable of generating new content over time; the horse fails to move forward past the obstacle, the scene does not change, and the video morphs back and forth within short window of motion. Bottom: Our novel video generation model prioritizes the time axis and generates realistic motion and scenery changes over long durations. The same videos can be viewed on the supplemental webpage.
of-te-of-the-ar
Our main contribution is a hierarchical generator architecture that employs a vast temporal receptive field and novel temporal embedding. We employ multi-rcsolution strategy, where first generate videos at low resolution and then refine them using a separate super- super-resolution network. Naively training long videos at high spatial resolution is prohibitively expensive, but find that the main aspects of a video persist at a low spatial resolution. This observation allows us to train with long videos at low resolution and short videos at high resolution, enabling
prioritize the time axis and ensure that long-term changes are accurately portrayed. The low-resolution and super-resolution networks are trained independently with RGB bottleneck in between. This modular design allows iterating on each network independently and leveraging the same super-resolution network for different low-resolution network ablations.
We compare our results to several recent video generative models and demonstrate statc-of-the-at performance in producing long videos with realistic motion and changes in content. Code, new datasets, and pre-trained models on these datasets will be made available.
2
Prior work
Video generation is challenging problem with long history. The classic early works, Video Textures (50) and Dynamic Textures (10), model videos as textures by analogy with image textures. That is, they explicitly assume the content to be stationary over time, e.g., fire burning, smoke rising, foliage falling, pendulum swinging, etc., and use non-parametric [50) or parametric [00 approaches model that stationary distribution. Although subsequent video synthesis works have dropped the "texture" moniker, much of the limitations remain similar - short training videos and models which produce little or new objects entering the frame during the video. Below summarize some of the more recent efforts on video generation.
Unconditional video generation. Many video generation works based on GANs including 47 60) and approaches that use recurrent networks to produce sequence of latent codes used to generate frames
early models that output fixed-length videos [1
12 55 56) MoCoGAN [56)
explicitly disentangles "motion" from "content" and keeps the latter fixed over the entire generated video. StyleGAN-V V [52] a recent state-of-the-art model we use as a primary baseline. Similar to MoCoGAN, StyleGAN-V employs global latent code that controls content of an entire video.
MoCoGAN-HD (55), which we also compare with, and Style VideoGAN [12 attempt to generate
videos by navigating the latent space of pretrained StyleGAN2 model [29] but struggle to produce

---

realistic motion. Unlike previous StyleGAN-based [28] video models, we prioritize the time axis in our generator through new temporal latent representation. temporal upsampling, and spatiotemporal modulated convolutions. We also compare with DIGAN 666 that employs an implicit representation to generate the video pixel by pixel.
Transformers are another class of models used for video generation 13
42
61
65
We com-
pare with TATS [13] that generates long unconditional videos with transformers, improving upon VideoGPT 655 Both TATS and VideoGPT employ a GPT-like autoregressive transformer 4 that represents videos as sequences of tokens. However, the resulting videos tend to accumulate error over time and often diverge or change too rapidly. The models are also expensive to train and deploy due their autoregressive nature over time and space. In concurrent work, promising results in generating diverse videos have also been demonstrated using diffusion- models [20]
Conditional video prediction. A separate line of research focuses on predicting future video frames conditioned on one or more real video frames [3] [3 34 36 39 41 or past frames accompanied by an action label 6 15 30 311. Some video prediction methods focus specifically on generating infinite scenery by conditioning on camera trajectory [37 44 and/or explicitly predicting depth (2, to then simulate a virtual camera flying through a 3D scene. Our goal, on the other hand, to support camera movement as well as moving objects by having the scene structure emerge implicitly.
Multi-resolution training.
Training at multiple scales is a common strategy for image generation Transformer-based video generators also employ a related two-phase
models 7 25
43
46
58).
setup 665
Saito al. 48 subsample frames at higher resolutions in their video generator
architecture to improve efficiency. A similar idea is also used in SlowFast [TT] networks where different network pathways are used for high and low frame rate video streams. Acharya al. M propose a multi-scale GAN for video generation that increases both spatial resolution and sequence length during training to produce a fixed-length video. In contrast, our multi-resolution approach designed to enable generating arbitrarily long videos with rich long-term dynamics by leveraging training of long sequences at low resolution.
3 Our method
Modeling the long-term temporal behavior observed in real videos presents us with two main challenges. First, we must use long enough sequences during training capture the relevant effects; using, c.g., pairs of consecutive frames fails to provide meaningful training signal for effects that occur over several seconds. Second, we must ensure that the networks themselves are capable of operating over long time scales; if, e.g., the receptive field of the generator spans only 8 adjacent frames, any two frames taken more than 8 frames apart will necessarily be uncorrelated with each other.
Figure p shows the overall design of our generator. We seed the generation process with a variable- length stream of temporal noise, consisting of 8 scalar components per frame drawn from i.i.d. Gaussian distribution. The temporal noise first processed by low-resolution generator to obtain sequence of RGB frames at 6 resolution that are then refined by separate super-resolutin network to produce the final frames at 256² resolution P The role of the low-resolution generator is to model major aspects of the motion and scene composition, which necessitates strong expressive power and large receptive field over time, whereas the super-resolution network is responsible for the more fine-grained task of hallucinating the remaining details.
Our two-stage design provides maximum flexibility in terms of generating long videos. Specifically, the low-resolution generator is designed to be fully convolutional over time, so the duration and time offset of the generated video can be controlled by shifting and reshaping the temporal noise, respectively. The super-resolution network, on the other hand, operates on a Trame-by-frane receives a short sequence of consecutive low-resolution frames and outputs a single high-resolution frame; each output frame is processed independently using a sliding window. The combination of fully-convolutional and framf processing enables us to generate arbitrary frames in arbitrary order, which is highly desirable for, e.g., interactive editing and real-time playback.
basis. It
We handle datasets with non aqu-r aspect ratio by shrinking all intermediate data accordingly. With 256x x 144 target resolution, for example, the low-resolution frames will have 64x36 36 resolution.
3

---

hightres video
Temporal noise
Downsample Augment
Low-res
Tempora noise
Sliding window
Latent
Super-res StyleGAN3
Conditional low-res video
video
Sliding window
Generated video
superres StyleGAN3
video
Generated high-res video
Augment
Real high-res video
Low-res
Augment
Real generated?
Super-nes StyleGANS
High-res
Real or generated?
(a) Generator overview
(b) I Low-res training
(c) Super-res training
Figure 2: Overview of our method. (a) To achieve long temporal receptive field and high spatial resolution, split our generator into two components: low-resolution generator, responsible modeling major aspects the motion and scene composition, and super-resolution network, responsible for hallucinating fine details. (b) The low-resolution generator (Section 3.1) employs wide temporal receptive field and is trained with sequences 128 frames at 64 resolution. (c) The super-resolution network (Section 3.2) is conditioned short sequences of low-resolution frames and trained to produce their plausible counterparts at 256 resolution.
The low-resolution and super-resolution networks modular with an RGB bottleneck in between. This greatly simplifies experimentation, since the networks are trained independently and can be used in different combinations during inference. We will first describe the training and architecture of the low-resolution generator in Section 3.1 and then discuss the super-resolution network in Section 3.2
3.1 Low-resolution generator
Figure p shows our training setup for the low-resolution generator. In each iteration, we provide the generator with fresh set of temporal noise to produce sequences of 128 frames (4.3 seconds at 30 fps). To train the discriminator, choosing a random video and a random interval of 128 frames within that video.
sample corresponding sequences from the training data by
We have observed that training with long sequences tends to exacerbate the issue of overfitting [26].
As the sequence length increases, we suspect that it becomes harder for the generator simultaneously model temporal dynamics at multiple time scales, but at the same time, easier for the discriminator to spot any mistakes. practice, we have found strong discriminator augmentation [26 to be necessary in order to stabilize the training. We employ DiffAug 69] using the same transformation for each frame in a sequence, as well as fractional time stretching between 2 x and 2x; see Appendix C.1
for details.
Architecture. Figure 3 illustrates the architecture our low rewolutiottion generator. Our main goal is to make the time axis a first-class citizen, including careful design of temporal latent representation, temporal style modulation, spatiotemporal convolutions, and temporal upsamples. Through these mechanisms, our generator spans vast temporal receptive field (5k frames), allowing it to represent temporal correlations at multiple time scales.
We employ style- style-based design, similar to Karras et al. 29 27,, that maps the input temporal noise into a sequence of intermediate latents {w:) used to modulate behavior each layer in the main synthesis path. Each intermediate latent is associated with specific frame, but it can significantly influence the scene composition and temporal behavior of several frames through hierarchical 3D) convolutions that appear in the main path.

---

Ter Temporal naise
Spatial input 1x4x3
Temporal lowpass filters
Temporal lowpass filters
(T/32)x4x3
-2000
4003
3300
-2000
-100
Mapping network Tx1x1
STu [T/16)x4x3
(frmme)
[T/8)x8*5
ST, [T/4)x8*5
Spatiotemporal/spetiall block
Sequence W, 0 Ost<?
(ST/S)
[T/2]x16x9
EMA norm
ST. Tw16x9
ST. Tx16x9
Leaky ReLU
Fixed
5, 32x18
EMA norm
1x1x1 Conv
Learned
5, 32x18
3x3x3 ModConv
Temporal downsample
5 64x.32
$, 64x36
Temporal upsample
Optional
EMA norm
2x Spatial upsample
Only present in ST block
1x1 MadConv
Leaky ReLU
Figure Low-resolution generator architecture, illustrated for 64x36 output. Left: The input temporal noise is mapped to a sequence of intermediate latents {w;} that modulate the interme- diate activations of the main synthesis path. Top right: To facilitate the modeling of long-term dependencies, we enrich the temporal noise by passing it through a series of lowpass filters whose temporal footprints range all the way from 100 5000 frames. Bottom right: The main synthesis path consists of spatiotemporal (ST) and sputial (S) blocks that gradually increase the resolution over time and space.
In order to reap the full benefits of the style-based design, is crucial for the intermediate latents to capture long-term temporal correlations, such as weather changes persistent objects. To this end, we adopt scheme where we first enrich the input temporal noise using a series of temporal lowpass filters and then pass it through celly-connected mapping network on a frame-by- by-ffrmee basis. The goal of the lowpass filtering provide the mapping network with sufficient long-term context across wide range of different time scales. Specifically, given stream of temporal noise =(t) E R$S we compute where { is set of 128 lowpass filters whose temporal footprint ranges from 100 to 5000 frames, and denotes convolution over time; see Appendix C.2 for details.
corresponding enriched representation 2t( E R L22888 as 2 fi Zj+
The main synthesis path starts by downsampling the temporal resolution of [w,} by 32x and concatenating it with a learned constant at 42 resolution. It then gradually increases the temporal and spatial resolutions through a series of processing blocks, illustrated in Figure [ right), focusing first on the time dimension (ST) and then the spatial dimensions (S). The first four blocks have 512 channels, followed by two blocks with 256, two with 128 and two with 64 channels. The processing blocks consist of the same basic building blocks as StyleGAN2 [29] and StyleGAN3 [27] with the addition of a skip connection; the intermediate activations are normalized before each convolution (27] and modulated (29) according to appropriately downsampled copy of {w, J. In practice, we employ bilinear upsampling (28) and use padding 227 for the time axis to eliminate boundary effects. Through the combination of our temporal latent representation and spatiotemporal processing blocks, our architecture is able to model complex and long term-term patterns across time.
For the discriminator, we employ an architecture that prioritizes the time axis via wide temporal recep- tive field, 3D spatiotemporal and ID temporal convolutions, and spatial and temporal downsamples; see Appendix
C.3 for details.
3.2 Super-resolution network
Figure 2 shows our training setup for the super-resolution network. Our video super-resolution network is straightforward extension of StyleGAN3 (27) for conditional frame generation. Unlike
5

---

(a) Mountain biking
(b) Horseback riding
(c) ACID
(d) SkyTimelapse
Figure 4: Example real frames from training datasets. We introduce first-person datasets of (a) mountain biking and (b) horseback riding videos that contain complex motion and new content over time. We also evaluate on existing datasets of (c) nature drone footage and (d) sky timelapse videos.
the low-resolution network that outputs a sequence of frames and includes explicit temporal operations, the super-resolution generator outputs single frame and only utilizes temporal information the input, where the real low-resolution frame and 4 neighboring real low-resolution frames before and after time are concatenated along the channel dimension to provide context. We remove the spatial Fourier feature inputs and resize and concatenate the stack of low-resolution frames to each layer throughout the generator. The generator architecture is otherwise unchanged from StyleGAN3, including the use of an intermediate latent code that is sampled per video. Low-resolution frames undergo augmentation prior to conditioning as part of the data pipeline, which helps ensure generalization to generated low-resolution images.
The super-res discriminator similar straightforward extension of the StyleGAN discriminator, with 4 low and high-resolution frames concatenated at the input. The only other change is the removal of the minibatch standard deviation layer that we found unnecessary in practice. Both low- and high- resolution segments of 4 frames undergo adaptive augmentation [26] where the same augmentation is applied to all frames at both resolutions. Low-resolution segments also undergo aggressive dropout (p = 0.9 probability of zeroing out the entire segment), which prevents the discriminator from relying too heavily on the conditioning signal; see Appendix [.] for details.
We find it remarkable that such a simple video super-resolution model appears sufficient for producing reasonably good high-resolution videos. We focus primarily on the low-resolution generator in experiments, utilizing a single super-resolution network trained per dataset. We feel that replacing this simple network with more advanced model from the video super-resolution literature [16] 24 49 54 is a promising avenue for future work.
Datasets
Most of the existing video datasets introduce little or new content over time. For example, talking head datasets 8 [45 62 63) show the same person for the duration of each video. UCF101 [53] portrays diverse human actions, but the videos are short and contain limited camera motion and little or no new objects that enter the videos over time.
To best evaluate our model, we introduce two new video datasets of first-person mountain biking and horseback riding (Figure b) that exhibit complex changes over time. Our new datasets include subject motion of the horse or biker, first-person camera viewpoint that moves through space, and new scenery and objects over time. The videos are available in high definition and were manually trimmed to remove problematic segments, scene cuts, text overlays, obstructed views, etc. The mountain biking dataset has 1202 videos with median duration of 330 frames at 30 fps, and the horseback dataset has 66 videos with median duration of 6504 frames also at 30fps. We have permission from the content owners to publicly release the datasets for research purposes. We believe new datasets will serve important benchmarks for future work.
We also evaluate our model on the ACID dataset motion but lacks other types of motion, well as the commonly used Sky AkyTimelapse dataset [67 (Figure that exhibits new content over time as the clouds pass by, but the videos are relatively homogeneous and the camera remains fixed.
38 (Figure 40) that contains significant camera

---

1.0
Color similarity
0.4
0.6
0.8
0.56
0.71
0.52
0.75
0.89
0.65
0,70
0.51
0.46
0.46
0.52
TATS
DIGAN
Dataset
MoCoGAN HD
0.2
0.06
0.0
64
128
0
64
128
64
128
0
64
128
128
Frame separation
Frame separation
Frame separation
separation
Frame separation
(a) Biking
(b) Horseback
(c) ACID
(d) Sky 256²
(c) Sky 1282
Figure 5: Color similarity (Eq. [ of real and generated videos function of frame separation,
reported as the mean (solid lines) and standard deviation (shaded regions) over 1000 random clips.
5 Results
We evaluate our model through qualitative examination of the generated videos (Section 5.1 ana- lyzing color change over time 5.22). computing the FVD metric (Section (.3). and ablating the key design choices (Section 5.44. We compare with StyleGAN-V [52) all datasets. Mountain biking, horseback riding and ACID 377 datasets contain videos with a 16x9 widescreen aspect ratio.
We train 256x1 14 resolution
these datasets to preserve the aspect ratio. Since StyleGAN-V is based on StyleGAN2 [29]. we can easily extend it to support non-square aspect ratios by masking real and generated frames during training. We found it necessary to increase the R1 y hyperparameter by 10x to produce good results with StyleGAN-V V on our new datasets that exhibit complex changes over time. We compare with MoCoGAN-HD (56), TATS [3] and DIGAN (66) using pre-trained models for the Sky Tiyelapse dataset at 128 resolution. For these comparisons, we train a separate super-resolution network output the frames 128 resolution, but use the same oow-resolution generator as in the 256² comparison.
5.1 Qualitative results
The major qualitative difference in results is that our model generates realistic new content over time, whereas StyleGAN-V continually repeats the same content. The effect is best observed by watching videos on the supplemental webpage and is additionally illustrated in Figure [ Scenery changes over time in real vidcos and our results the horse moves forward through space. However, the videos generated by StyleGAN-V tend to morph back to the same scene at regular intervals. Similar repeated content from StyleGAN-V V is apparent all datasets. For example, results on the webpage for the SkyTimelapse dataset show that clouds generated by StyleGAN-V repeatedly move back and forth. MoCoGAN-HD and TATS suffer from unrealistic rapid changes time that diverge, and DIGAN results contain periodic patterns visible in both space and time. Our model is capable of generating constant stream of new clouds.
As a further validation of our observations, we conducted a preliminary user study on Amazon Mechanical Turk. We created 50 pairs of videos for each of the 4 datasets. Each pair contained a random video generated by StyleGAN-V and one generated by our method, and we asked the participants which of them exhibited more realistic motion in forced-choice response. Each pair was shown to 10 participants, resulting in a total of 50x4x10 responses, Our method was preferred over 80% of the time for every dataset. Please Appendix A. for details.
5.2 Analyzing color change over time
To gain insight into how well different methods produce new content at appropriate rates, we analyze how the overall color scheme changes function of time. We measure color similarity as the intersection between RGB color histograms; this serves as a simple proxy for actual content changes and helps reveal the biases of different models. Let H (x, i) denote 3D color histogram function that computes the value of histogram bin i E [1..... 3 for the given image normalized so that Σ, (0, 1. Given video clip x = {a+} and frame separation t, we define the color similarity as
S(x, 1) = Σ, min (H(zo. i), H.HHO H(x.
(1)
7

---

Biking
Horsebuck
ACID
Sky 256
Sky 128
F2D
FVD1s
FVDra
FVD1s
FVD
FVD1s
2VD
FVD16
FVDras
FVD16
StyleGAN-V
533.3
353.7
427.0
319.2
112.4
91.5
151.2
48.4
MoCoGAN--HD
635.6
224.9
wia s
221.6
99.2
196.2
159.0
TATS
135.0
97.0
DIGAN
228.6
153.4
Ours
113.7
83.8
95.9
113.5
166.6
127.3
152.7
116.5
Ours
142.6
107.5
Table I: We compute FVD on segments of 128 and 16 frames (FVD 22 and FVD1e respectively), where lower is better. Left: Our model outperforms StyleGAN-V on horseback riding and mountain biking datasets both of which contain complex motion and new content over time. Our model underperforms StyleGAN-V on ACID and SkyTimelapse despite qualitative improvements and favorable user study ratings in Section[5.1 Right: Our model outperforms MoCoGAN-HD, TATS and DIGAN baselines on Sky Timelapse at 128 resolution on FVD128-
where S(æ, () indicates that the color histograms are identical between 2o and Tı. In practice, we set N = 20 and report the mean and standard deviation of S(,,t), t), measured on 1000 random video clips containing 128 frames each.
Figure 5 shows t-, as function of for real and generated videos on each dataset. The curves trend downward over time for real videos as content and scenery gradually change. StyleGAN-Y V and DIGAN are biased toward colors changing too slowly - both of these models include a global latent code that is fixed over the entire video. On the other extreme, MoCoGAN-HD and TATS are biased toward colors changing too quickly. These models use recurrent and autoregressive networks, respectively, both of which suffer from accumulating errors. Our model closely matches the shape of the target curve, indicating that colors in our generated videos change at appropriate rates.
Color change is a crude approximation of the complex changes over time in videos. In Appendix A.3 we also consider LPIPS [68] perceptual distance instead of color similarly and observe the same trends in most cases.
5.3 Fréchet video distance (FVD)
The commonly used Fréchet video distance (FVD) 55 attempts to measure similarity between real and generated video distributions. We find that FVD is sensitive to the realism of individual frames and motion over short segments, but that it does not capture long-term realism. For example, FVD essentially blind to unrealistic repetition of content over time, which is prominent in StyleGAN-V videos on all of our datasets. We found FVD to be most useful in ablations, i.c., when comparing slightly different variants of the same architecture.
FVD computes the Wasserstein-2 distance [59 between sets of real and generated features extracted from a pre-trained I3D action classification model [5]. Skorokhodov et al. [52) note that FVD is highly sensitive to small implementation differences, down to the level of image compression settings, and that the reported results are not necessarily comparable between papers (Appendix C in (52)) We report all FVD results using consistent evaluation protocol, ensuring apples-to-apples comparison. We separately measure FVD using 128- and 16-frame segments, denoted by FVD12a and FVD16. and sample 2048 random segments from both the dataset and generator in each case.
Table (left) reports FVD on all datasets for StyleGAN-V and our model. We outperform StyleGAN- V on horseback riding and mountain biking datasets that contain more complex changes over time, but underperform on ACID and slightly underperform on SkyTimelapse in terms of FVD128. However, this underperformance strongly disagrees with the conclusions from the qualitative user study in Section 5.1 We believe this discrepancy comes from StyleGAN-V producing better individual frames, and possibly better small-scale motion, but falling seriously short in recreating believable long-term realism and the FVD being sensitive primarily to the former aspects. Table I (right) reports FVD metrics on MoCoGAN-HD, TATS, DIGAN and our model for SkyTimelapse at 1282; we outperform all baselines in terms of FVD 128 this comparison.
8

---

F2D128
FVD16
FVD
FVD16
Ours (128 frames)
113.7
83.8
Ours
113.7
83.8
16 frames
163.6
108.5
0. Ix lowpass width
153.1
113.2
frames
396.8
169.4
10x lowpass width
217.9
126.5
(a) Ablation of training sequence length
(b) Ablation of temporal lowpass filter footprint
Table (a) Our model learns to generate realistic long videos by training on long videos; decreasing the sequence length used during training is consistently harmful. (b) The footprint of the temporal lowpass filters plays an important role in producing inputs to the low-resolution mapping network at appropriate temporal frequencies; changing the footprint by order of magnitude hurts performance.
FVD
Biking Ours
113.7
SR on reals
58.3
ACID
Ours
166.6
SR reals
68.8
(a) Mountain biking
(b) ACID
(c) Ablation
Figure 6: Evaluation of the super-resolution network. (a,b) Generated low-resolution frames and corresponding high-resolution frames produced by the super-resolution network. (c) The super- resolution network yields remarkably good FVD when provided with real low-resolution videos as input; the overall quality of our results is largely dictated by the low-resolution generator.
5.4 Ablations
Training long videos improves generation of long videos. Observing long vidcos during training helps our model learn long-term consistency, which is illustrated in Table that ablates the sequence length used during training of the low- wwwresolution generator. We found that the benefits of training with long videos only became evident after designing a generator architecture with appropriate temporal receptive field to utilize the rich training signal. Note that even though ablate aspects of the low-resolution generator, we still compute FVD using the final high-resolution videos produced by the nuper-resolution network.
Footprint of the temporal lowpass filters. Our temporal latent representation serves vital role in expanding the receptive field of our generator, modeling patterns over different time scales, and enabling the generation of new content over time. While we primarily leverage long training videos to learn long-term consistencies from data, the size of our temporal lowpass filters plays role in encouraging the low-resolution mapping network to learn correlations at appropriate time scales. Table 2 demonstrates the negative impact of using inappropriately sized filters. We find that our model performs well with the same filter configuration for all datasets, although it is possible that the ideal settings may vary slightly between datasets.
Effectiveness of the super-resolution network. Figure Gb.b shows examples of low-resolution frames generated by our model along with the corresponding high-resolution frames produced by our super-resolution network; we find that the super-resolution network generally performs well. To ensure that the quality of our results we further measure FVD when providing the rupertresolution network with real low-resolution videos as input in Figure GC. Indeed, FVD greatly improves in this case, which indicates that there are still significant gains to be realized by further improving the resoluoottiion generator.
dispropootiionatel limited by the super-resolution network,
5.5 Failure cases
Separate low- and super-resolution networks makes the problem computationally feasible, but may somewhat compromise the quality of the final heshluiso frames. We observed that "swirly" artifacts are most prominent the super-resolution output and not the low-resolution output. Our model also struggles with long-term consistency of small details (e.g., distant jumps in generated horseback riding videos) that begin to appear before quickly fading out. We believe these issues are due to limitations of our super-resolution network, and that improving the super-resolution network

---

would benefit the model in this regard. Another failure case we observed is difficulty preserving 3D consistency for scenes with very little motion, such as the ACID dataset. In cases where there is little motion, one may consider using an explicit 3D representation.
6 Conclusions
Video generation has historically focused on relatively short clips with little new content over time. We consider longer videos with complex temporal changes, and uncover several open questions and video generation practices worth reassessing - the temporal latent representation and generator architecture, the training sequence length and recipes for using long videos, and the right evaluation metrics for long-term dynamics.
We have shown that representations over many time scales serve as useful building blocks for modeling complex motions and the introduction of new content over time. We feel that the form of the latent space most suitable for video remains an open, almost philosophical question, leaving large design space to explore. For example, what the right latent representation model persistent objects that exit from video and re-enter later in the video while maintaining a consistent identity?
The benefits we find from training on longer sequences open up further questions. Would video generation benefit from even longer training sequences? Currently we train using segments of adjacent frames, but it might be beneficial use larger frame spacings cover longer time spans.
Quantitative evaluation of the results continues to be challenging. As we observed, FVD goes only a part of the way, being essentially blind to repetitive, even very implausible results. Our tests with how the colors and I.IPS distance change as a function of time partially bridge this gap, but we feel that this area deserves thorough, targeted investigation of its own. We hope work encourages further research into video generation that focuses on more complex and longer-term changes over time.
Negative societal impacts Our work falls within data-driven generative modeling, which, as a field, has well known potential for misuse with increasing quality improvements. The training of video generators is even more intensive computationally than training still image generators, increasing energy usage. Our project consumed 300MWh on an in-house cluster of V100 and A100 GPUs.
Acknowledgements We thank William Peebles, Samuli Laine, Axel Sauer and David Luebke for helpful discussion and feedback; Ivan Skorokhodov for providing additional results and in- sight into the StyleGAN-V baseline; Tero Kuosmanen for maintaining compute infrastructure; Elisa Wallace Eventing (https: / //www/ yout com/ c/WallaceEventingg and Brian Kennedy potttp://wwir youtube. com/ c/bkxc) for videos used to make the horseback riding and mountain biking datasets. Tim Brooks is supported by the National Science Foundation Graduate Research
Fellowship under Grant No. 2020306087.
References
[1] Dinesh Acharya, Zhiwu Huang, Danda Pani Paudel, and Luc Van Gool. Towards high resolution video generation with progressive growing of sliced wasserstein gans. CoRR, abs/1810.02419, 2018.
[2] Adil Kaan Akan, Sadra Safadoust, Erkut Erdem, Aykut Erdem, and Fatma Güney. Stochastic video prediction with structure and motion. CORR, abs/2203. 10528, 2022.
[3] Mohammad Babaeizadeh, Chelsea Finn, Dumitru Erhan, Roy H Camphell, and Sergey Levine. Stochastic variational video prediction. In Proc. ICLR, 2018
[4] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Proc. NewrIPS, 13:1877-1901, 2020.
[5] João Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In Proc. CVPR, pages 4724-4733, 2017.
[6] Silvia Chiappa, Sébastien Racaniere, Daan Wierstra, and Shakir Mohamed. Recurrent environ- ment simulators. In Proc. ICLR, 2017.
10

---

[7] Rewon Child. Very deep vaes generalize autoregressive models and can outperform them on images. arXiv preprint arXi::2011.10650 2020.
[8] Joon Son Chung, Arsha Nagrami, and Andrew Zisserman. Voxceleb2: Deep speaker recognition. In Interspeech, 2018.
[9] Aidan Clark, Jeff Donahue, and Karen Simonyan. Adversarial video generation on complex datasets. CoRR, abs/1907.06577 2019.
[10] Gianfranco Doretto, Alessandro Chiuso, Ying Nian Wu, and Stefano Soatto. Dynamic textures. International Journal of Computer Vision, 51(2)911-19, 2003.
[11] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6202-6211, 2019.
[12] Gereon Fox, Ayush Tewari, Mohamed Elgharib, and Christian Theobalt. Stylevideogan: A temporal generative model using a pretrained stylegan. CoRR, abs/2107.07224. 2021.
[13] Songwei Gc. Thomas Hayes, Harry Yang, Xi Yin, Guan Pang, David Jacobs, Jia-Bin Huang, and Devi Parikh. Long video generation with time-agnostic vggan and time-sensitive transformer. CoRR, abs/2204. 03638, 2022.
[14] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Proc. NIPS, 27, 2014.
[15]
David Ha and Jürgen Schmidhuber. World models. CoRR, abs/ 10122, 2018.
[16] Muhammad Haris, Gregory Shakhnarovich, and Norimichi Ukita. Recurrent back-projection network for video super-resolution. In Proc. CVPR, pages 3897-3906, 2019.
[17] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[18] Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky. Neural networks for machine learning lecture 6a overview of mini-batch gradient descent. Cited on, 14(8):2, 2012.
[19] Jonathan Ho. Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research, 23(47):1-33, 2022.
[20] Jonathan Ho, Tim Salimans, Alexey Gritsenko. William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. CoRR, abs/2204.03458, 2022.
[21] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European conference computer vision, pages 694-711. Springer, 2016.
[22] James F Kaiser. Nonrecursive digital filter design using the i_O-sinh window function. In Proc. 1974 IEEE International Symposium on Circuits & Systems, San Francisco DA, April, pages 20-23, 1974.
[23] Nal Kalchbrenner, Aäron Oord, Karen Simonyan, Ivo Danihelka, Oriol Vinyals, Alex Graves, and Koray Kavukcuoglu. Video pixel networks. In Proc. ICML, pages 1771-1779, 2017.
[24] Armin Kappeler, Seunghwan Yoo, Qiqin Dai, and Aggelos K Katsaggelos. Video super- resolution with convolutional neural networks. IEEE transactions on computational imaging, 2(2):109-122, 2016.
[25] Tero Karras, Timo Aila. Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. In Proc. ICLR, 2018
[26] Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. In Proc. NeurlPS, 2020.
[27] Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. In Proc. NeurIPS, 2021.
[28] Tero Karras, Samuli Laine, and Timo Aila. style- generator architecture for generative adversarial networks. In Proc. CVPR, pages 4401-4410 2019.
11

---

[29] Tero Karras, Samuli Laine, Mika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proc. CVPR. pages 8110-8119, 2020.
[30] Seung Wook Kim, Jonah Philion, Antonio Torralba, and Sanja Fidler. Drivegan: Towards a controllable high-quality neural simulation. In Proc. CVPR, pages 5820-5829, 2021.
[31] Seung Wook Kim, Yuhao Zhou, Jonah Philion, Antonio Torralba, and Sanja Fidler. Learning to Simulate Dynamic Environments with GameGAN. In Proc. CVPR, Jun. 2020.
[32] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXivv1412.6980, 2014.
[33] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
[34] Manoj Kumar, Mohammad Babaeizadeh, Dumitru Erhan, Chelsea Finn, Sergey Levine, Laurent Dinh, and Durk Kingma. Videoflow: conditional flow-based model for stochastic video generation. In Proc. ICIR, 2020.
[35] Tuomas Kynkäänniemi, Tero Karras, Miika Aittala, Timo Aila, and Jaakko Lehtinen. The role of imagenet classes in fréchet inception distance. arXiv preprint arXiv:2203. 06026,, 2022.
[36] Alex X Lee, Richard Zhang, Frederik Ebert, Pieter Abbeel, Chelsea Finn, and Sergey Levine. Stochastic adversarial video prediction. CoRR, abs/ 0011888441122. 2018.
[37] Andrew Liu, Richard Tucker, Varun Jampani, Ameesh Makadia, Noah Snavely, and Angjoo Kanazawa. Infinite nature: Perpetual view generation of natural scenes from a single image. In Proc. ICCV, 2021.
[38] Andrew Liu, Richard Tucker, Varun Jampani, Ameesh Makadia, Noah Snavely, and Angjoo Kanazawa. Infinite nature: Perpetual view generation of natural scenes from a single image. In Proc. CVPR, 2021.
[39] Pauline Luc, Aidan Clark, Sander Dieleman, Diego de Las Casas, Yotam Doron, Albin Cassirer, and Karen Simonyan. Transformation- adversarial video prediction on large-scale data. CoRR, abs/2003.04033 2020.
[40] Lars Mescheder, Sebastian Nowozin, and Andreas Geiger. Which training methods for gans do actually converge? In International Conference on Machine Learning (ICML). 2018.
[41] Charlie Nash, João Carreira, Jacob Walker, lain Barr, Andrew Jaegle, Mateusz Malinowski, and Peter Battaglia. Transframer: Arbitrary frame prediction with generative models. CoRR, hhs/2203.094944 2022.
[42] Ruslan Rakhimov, Denis Volkhonskiy, Alexey Artemov, Denis Zorin, and Evgeny Burnaev. Latent video transformer. CoRR, abs/2006. 10704, 2020.
[43] Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. Proc. NeurIPS, 32, 2019.
[44] Xuanchi Ren and Xiaolong Wang. Look outside the room: Synthesizing a consistent long-term
3d scene video from a single image. In Proc. CVPR, 2022.
[45] Andreas Rössler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies, and Matthias NieBner. Faceforensics: large-scale video dataset for forgery detection human faces. arXiv preprint /803.09/7 2018.
[46] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. arXiv:2104.07636 2021.
[47] Masaki Saito, Eicchi Matsumoto, and Shunta Saito. Temporal generative adversarial nets with singular value clipping. In Proc. ICCV, pages 2830-2839, 2017.
[48] Masaki Saito. Shunta Saito, Masanori Koyama, and Sosuke Kobayashi. Train sparsely, generate densely: Memory-efficiint unsupervised training of high-resolution temporal gan. International Journal of Computer Vision, 128 (500)00))66666666 2606, 2020.
[49] Mehdi S. M. Sajjadi, Raviteja Vemulapalli, and Matthew Brown. Frame-recurrent video super-resolution. In Proc. CVPR, 2018.
[50] Arno Schödl, Richard Szeliski, David Salesin, and Irfan Essa. Video textures. Proc. SIGGRAPH, page 489-498, 2000.
12

---

[51] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409. 1556, 2014.
[52] Ivan Skorokhodov, Sergey Tulyakov, and Mohamed Elhoseiny. Stylegan-v: A continuous video
generator with the price, image quality and perks of stylegan2. CoRR, abs/21 12. 14683, 2021.
[53] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: dataset of 101 human
actions classes from videos in the wild. In Proc. ICCV, 2013.
[54] Xin Tao, Hongyun Gao, Renjie Liao, Jue Wang, and Jiaya Jia. Detail-revealing deep video super-resolution. Proc. ICCV, 2017.
[55] Yu Tian, Jian Ren, Menglei Chai, Kyle Olszewski, Xi Peng, Dimitris N. Metaxas, and Sergey Tulyakov. A good image generator is what you need for high-resolution video synthesis. In Proc. ICLR, 2021.
[56] Sergey Tulyakov, Ming- Yu Liu, Xiaodong Yang, and Jan Kautz. Mocogan: Decomposing motion and content for video generation. Proc. CVPR, pages 1526-1535, 2018.
[57] Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphael Marinier, Marein Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. CoRR, abs/1812.01717 2018.
[58] Arash Vahdat and Jan Kautz. NVAE: deep hierarchical variational autoencoder. Proc. NeurIPS, 2020.
[59] Leonid Nisonovich Vaserstein. Markov processes over denumerable products of spaces, de- scribing large systems of automata. Problemy Peredachi Informatsii, 5(3):64-72, 1969.
[60] Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dynamics. In Proc. NIPS, 2016.
[61] Jacob Walker, Ali Razavi, and Aäron van den Oord. Predicting video with vqvae. CoRR, abs/2103.01950) 2021.
[62] Kaisiyuan Wang, Qianyi Wu, Linsen Song, Zhuoqian Yang, Wayne Wu, Chen Qian, Ran He, Yu Qiao, and Chen Change Loy. Mead: A large-scale audio-visual dataset for emotional talking-face generation. Proc. ECCV, 2020.
[63] Ting-Chun Wang, Arun Mallya, and Ming-Yu Yu Liu. One-shot free-view neural talking-head synthesis for video conferencing. In Proc. CVPR, 2021.
[64] Wei Xiong, Wenhan Luo. Lin Ma, Wei Liu, and Jiebo Luo. Learning to generate time-lapse videos using multi-stage dynamic generative adversarial networks. In Proc. CVPR. pages 2364-2373, 2018.
[65] Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using vq-vae and transformers. CoRR, abs/2104. 10152 2021.
[66] Sihyun Yu, Jihoon Tack, Sangwoo Mo, Hyunsu Kim, Junho Kim, Jung- Woo Ha, and Jinwoo Shin. Generating vidcos with dynamics-aware implicit generative adversarial networks. In Proc. ICLR, 2022.
[67] Jiangning Zhang, Chao Xu, Liang Liu, Mengmeng Wang, Xia Wu, Yong Liu, and Yunliang Jiang. Dtvnet: Dynamic time-lapse video generation via single still image. In Proc. ECCV, 2020.
[68] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreason- able effectiveness of deep features as perceptual metric. In CVPR, 2018.
[69] Shengyu Zhao, Zhijian Liu, Ji Lin, Jun- Zhu, and Song Han. Differentiable augmentation for data-cfficient gan training. Proc. NeurIPS, 2020.
13


Reviews:
[{"review_id": "u0jiGSW8zJ", "reviewer": "Reviewer_u8rP", "summary": "This paper takes the initiative to improve the problem of video generation in terms of the dynamics of the generated videos. The main strategy proposed by the author is to increase the real video lengths seen by the discriminator by reducing the resolution. The reduced resolution is compensated separately with a super-resolution network.\n\nThe authors provide some analysis of the color dynamics of the real dataset and the generated videos using different methods.\n", "questions": "1 - Fig. 5 is a little hard to read due to the mix of green and blue colors. However, it seems that although the color average between the proposed method is the closest to the real data, the variance is much lower. What does that imply?\n\n2 - I am wondering if the color similarity based on the frame separation is a good indicator/metric for better scene dynamics. Why did the authors select this metric?\n\n3 - I found this statement confusing `time offset of the generated video can be controlled by shifting and reshaping the temporal noise, respectively.` How shifting the noise, can decide the time offset?\n\n4 - I am wondering if a pre-trained super-resolution network (perhaps on image datasets) can perform nearly as well as what authors trained on the video examples. \n\n5 - What about the number of generated shots? Applying a shot detector on the real vs generated data, how much the distribution of the number of shots per clip is similar per class? What about the distribution of consecutive shots similarity? Do authors think it will be a complementary indication to the color examination?  \n", "limitations": "I believe the authors are honest about the negative societal impacts. I have no more points to add here.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1- The literature has overlooked video dynamics and temporal modeling for video generation in generative models. This research has selected the correct challenge in video generation to address.\n\n2- Great writing and supplementary materials.\n\n3- Decoupling the temporal dynamics from frame resolution has some novelties. However, it has been remotely mentioned by previous research but has never been explicitly modeled with two networks in two different stages.\n\nWeaknesses:\n\n1- Although I respect the authors' braveness and choice of problem, I believe the paper conveys such a message that the main reason for low dynamics in video generation models is data. Since we are not showing long videos to the discriminator, the generated videos do not have enough dynamics or look repetitive. Although I agree with this argument, IMO, the main challenge is modeling the temporal dynamics. `We can still (wisely) select shorter videos but with enough frames change.` But we need better structures to model the temporal dynamics. \n\n2- I appreciate the honest selection of the generated video samples in the supplementary materials. It is clear that in some scenarios, like horse riding, we see novel objects enter the scene, and the motion is not repetitive as the baselines. However, in some cases like `ACID: acid_grid.mp4, second row`,  we see a lot of distortions in the generated video. Although the method works better for high dynamics, it may not handle low dynamics. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "lDFytyh7Du1", "reviewer": "Reviewer_cBCU", "summary": "Brief Summary: The paper addresses the problem of video generation with a focus on longer time horizon videos which requires consistency. To this end, the authors introduce two new benchmark datasets on horse-riding and mountain biking. The key observation is that the main components for temporal consistency are preserved at lower spatial resolution. The authors therefore propose using a hierarchical architecture to first create long low-resolution video, followed by sliding windows to create higher resolution videos at shorter time-steps.\n\nThe authors have further performed a human evaluation on mechanical turk to find their method was preferred over 80% of the time.  ", "questions": "Q1. The idea of using low and high resolution for long and small temporal segments reminds me of SlowFast networks [Ref1], where the slow path samples sparsely with more channel dimension, while fast path samples densely with less channel dimension. Obviously, it is not 1-1 correspondence, but some kind of discussion could be useful (maybe in supplementary).\n\n[Ref1]: Feichtenhofer, Christoph, Haoqi Fan, Jitendra Malik, and Kaiming He. \"Slowfast networks for video recognition.\" In Proceedings of the IEEE/CVF international conference on computer vision, pp. 6202-6211. 2019.", "limitations": "I think authors have done a good job of highlighting failure cases as well as pointing out where FVD metric is not very predictive of human eval performance.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n\n1. The idea is simple yet elegant. Implementation-wise it is quite interesting to find that a straightforward extension of stylegan to videos where input images are simply concatenated leads to promising results. \n\n2. New datasets contributions are always welcome. The two contributed datasets on mountain biking and horseback riding could be useful for future research especially to evaluate temporal consistency.\n\n3. The authors have done a human eval which is very important in video generation, and found their method was preferred 80% of the time. \n\n4. The authors also show color-change as a heuristic for temporal which clearly show the benefits of hierarchical training.\n\n5. The visualizations provided in the supplementary are very cool!\n\nCons:\n1. For the user study, the authors should have provided \"equally same\" as an option, and used sanity check that \"equally same\" is picked when both videos are obtained from the same model. I am not sure if this could have created any issues in the assessment (my guess is the effect could be mild, but not negligible).\n\n2. I am wondering if a better evaluation for long-term consistency would be conditional generation, where first few and last few seconds of real video are provided. For instance, if a tree is seen at the last frame (fig-1, 10s), one could have a soft-metric that the tree be seen (albeit at lower resolution) at some intermediate frame. Could make the task on human evaluation easier as well given that they would be comparing more similar videos.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "IRgZ6I08Dg9", "reviewer": "Reviewer_xqQ3", "summary": "The paper discusses a method for generating long videos in which new content is introduced as the camera moves forwards. Training on such videos especially in high resolution is prohibitively expensive. One of the possible ways to mitigate the issue is to train a two stage architecture, in which dynamics is trained on low-res, followed by super-resolving the low-res long video with the second stage. To show advantages of their approach the authors collect two new datasets, in which the camera moves forward. According to the results, the method outperformed current works on these two datasets, and finishes second on previously available datasets. ", "questions": "Please see above.", "limitations": "Please see above", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths. The paper has some:\n\nS1. The paper shows that treating temporal noise is important. The idea of filtering it with a low-pass filter is interesting. This way only low-frequency, longer events are captured by the model.\nS2. Results are quite impressive on the two new datasets. Very impressive!\n\nThere also are weaknesses, unfortunately:\n\nW1. Subsampling in space and time has been proposed in the previous literature. It has been shown that training at high res both in space and time is prohibitively expensive. In [TGAN-V2] for example, they generate longer videos at low resolution and as they increase resolution with more generators they drop the frame rate. This way they achieved efficient training of 256^2 resolution--highest resolution of this paper--in 2018. Another similar idea was used in a discriminator of DVD-GAN [9]. They downsampled the resolution for video discriminator, and reduced frame-rate for image discriminator. [TGAN-V2] is not cited, DVD-GAN is cited but the similarities/differences are not discussed. \n\nW2. The proposed framework is reasonable and seems to be working well on the proposed datasets. The key new interesting idea is temporal filtering, the rest of the framework from the high level is known (see W1). I acknowledge the amount of effort the authors put into making it work, of course. \n\nW3. Other video datasets. The paper fails to report UCF, FaceForensics and others from stylegan-v & tats, arguing that these datasets contain less new content and camera movements. While it might be case, the method only reports better scores on their own proposed datasets. The problem with this is that it takes a lot of effort to tune each method to each dataset, making it not very clear if the proper and sufficient tuning for stylegan-v was made. If one compares to existing numbers on existing datasets, such as UCF, on which TATS and stylegan-v show reasonable performance, one can guarantee that the method is evaluated against the numbers in which proper time was invested. Otherwise it's always possible to select a dataset on which the method scores best.\n\nW4. Other resolutions. Stylegan-v and mocogan-hd show training on much higher resolutions, as high as 1024. While this method focuses on the temporal part of videos, it's not clear what's the resolution upper bound. It's important to understand that even for this work, since scaling GANs is a non-trivial task both in terms of computational resources and quality. \n\n\n[TGAN-V2] Saito, Masaki, et al. \"Train sparsely, generate densely: Memory-efficient unsupervised training of high-resolution temporal gan.\" International Journal of Computer Vision 128.10 (2020): 2586-2606.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "5mskrduqx0D", "reviewer": "Reviewer_3wBD", "summary": "The paper presents a video generation model that is capable of producing new content (e.g., new object or scenery), object motion, and changes in camera viewpoint over time for longer videos than prior works. In order to achieve this, temporal modeling is emphasized in the proposed architecture, unlike existing works in which frame quality is often prioritized. As their main contribution, the authors redesign the temporal latent representation and train the carefully-designed model, which has the capability to operate over long time scales with a vast temporal receptive field, on longer videos at a low resolution and shorter videos at a high resolution. Two new benchmark datasets are introduced to best evaluate the proposed model since there are no existing datasets with long enough videos. Evaluated on 4 datasets with different characteristics, the proposed model outperforms prior methods especially qualitatively on aspects including generating plausible dynamics and object persistence, producing new content while maintaining consistencies over time.\n", "questions": "1. According to the color similarity plots, the standard deviations of Dataset are quite high, and often higher than that of models. Does it tell anything? \n\n2. The “swirly” artifacts visible in some of the videos might be attributed to the multi-resolution strategy, and it was referred to as “RGB bottleneck” (line 305). Could you further explain this in detail?\n", "limitations": "The authors have discussed the limitations and potential negative societal impact. From the qualitative results, it seems that the model also struggles when objects in the scene interact with each other.\n", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n+ The paper has suggested several interesting insights and many useful practices for video generation. For example, a multi-resolution two-stage strategy might be a good solution for training and deployment for models to handle long videos; the low-resolution generator should be fully convolutional over time to learn long-term temporal correlations whereas the super-resolution generator can operate in a frame-by-frame basis. Videos with longer sequence length tend to exacerbate the issue of overfitting and therefore some strong augmentations might be required. \n\n+ The paper has provided a deep investigation on the metrics of video generation. Particularly, they propose to analyze color change over time as a simple way to diagnose potential bias captured by the different models. In addition, existing commonly-used metrics such as FVD and LPIPS are also discussed, and the authors have found these metrics to agree less with the qualitative results or user study results.\n\n+ The qualitative results are convincing. It was shown that the proposed model is capable of generating videos with rich motion and scenery changes. Existing methods are incapable of generating realistic long videos, and explanations and analysis presented in the paper are reasonable.\n\n+ Two new datasets are proposed which might be beneficial for future researchers. \n\n\n\nWeaknesses:\n- An in-depth failure case analysis is absent, which might be interesting to have for readers.\n\n- The proposed architecture seems to heavily depend on the correct data augmentation in use.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept