SSW-GAN: Scalable Stage-wise Training of Video GANs
Authors: Lluis Castrejon
Nicolas Ballas
Aaron Courville
Venue:  (2021)
Keywords: video generation
GANs
scalable methods

Abstract: Current state-of-the-art generative models for videos have high computational requirements that impede high resolution generations beyond a few frames. In this work we propose a stage-wise strategy to train Generative Adversarial Networks (GANs) for videos.  We decompose the generative process to first produce a downsampled video that is then spatially upscaled and temporally interpolated by subsequent stages. Upsampling stages are applied locally on temporal chunks of previous outputs to manage the computational complexity. Stages are defined as Generative Adversarial Networks, which are trained sequentially and independently. We validate our approach on Kinetics-600 and BDD100K, for which we train a three stage model capable of generating 128x128 videos with 100 frames.

Under review as a conference paper at ICLR 2021
SSW-GAN: SCALABLE STAGE-WISE TRAINING OF VIDEO GANS
Anonymous authors Paper under double-bling review
ABSTRACT
Current state-of-the-art generative models for videos have high computational requirements that impede high resolution generations beyond few frames. In this work we propose a stage-wise strategy to train Generative Adversarial Networks (GANs) for videos. We decompose the generative process first produce a downsampled video that is then spatially upscaled and temporally interpolated by subsequent stages. Upsampling stages are applied locally on temporal chunks of previous outputs to manage the computational complexity. Stages are defined Generative Adversarial Networks, which are trained sequentially and independently. We validate our approach on Kinetics-600 and ADD100K, for which we train a three stage model capable of generating 128x128 videos with 100 frames.
INTRODUCTION
The field of generative modeling has seen rapid developments the past few years. Current models such as GPT-3 (Brown et al. 2020)) or BigGAN (Brock et al. 2018) are capable of generating coherent long paragraphs and detailed high resolution images. Generative models for videos have high memory requirements that quickly scale with the output resolution and length. Prior works have therefore restricted the video dimensions by operating at low spatial resolution or by only considering
small number of frames to generate (Ranzato et al. 2014 Vondrick et al., 2016a: Tulyakov et al. 2018 Kalchbrenner et al. 2017).
In this work we investigate an approach to reduce the computational costs needed to generate long high resolution videos in the context of Generative Adversarial Networks (GANs). Current GAN approaches require large batch sizes and high capacity models (Clark et al., 2019 Brock et al. 2018). We propose to break down the generative process into a set of smaller generative problems or stages, each stage having reduced computational requirements. The first stage produces downsampled low-resolution video that then spatially upscaled and temporally interpolated by subsequent upscaling stages. Each stage is modeled GAN problem and stages are trained sequentially and independently.
Each stages only considers lower dimensional view of the video during training. The first stage is trained to produce full-length videos at reduced spatiotemporal resolution, while the upscaling stages are trained to upsample partial temporal windows on the previous generations. At inference time, the upscaling stages are applied on the full first stage output in convolutional fashion to generate full resolution videos.
Learning the upscaling stages on local views of the data reduces their computational requirements. By keeping fixed temporal window size, computational requirements scale only output resolution. independent of the final video length. However, upsampling stages consider limited field of view in time, which could negatively impact the temporal consistency of the full-size generation. To address this problem, we rely the first low-resolution generation to capture long-term temporal information, and use it to condition the upscaling stages. In particular, we introduce novel matching discriminator that ensures that outputs grounded to the low-resolution generation.
Our approach, named SSW-GAN, offers a novel way to decompose the training of large GAN models, inspired by other coarse-to-fine methods that have been explored in the context of images and videos (Denton et al. 2015; Karras et al.) 2017 Acharya et al. 2018). In contrast to previous methods, do train on full resolution inputs in upscaling stages, and instead impose global

---

Under review as a conference paper at ICLR 2021
temporal consistency through conditioning the complete but low resolution of the the first stage output. Our model thus provides significant computational savings, which allow for high quality high resolution generators capable of producing hundred of frames.
Our contributions can be summarized as follows:
We define stage-wise approach to train GANs for video in which stages are trained sequentially and show that solving this multi-stage problem is equivalent to modeling the joint data probability of samples and their corresponding downsampled views.
We empirically validate our approach on Kinetics-600 and BDD100K, two large- scale datasets with complex videos in real- world scenarios. Our approach matches or outperforms state-of-art approaches while requiring significantly less computational resources.
We use our approach to generate videos with 100 frames at high resolutions with high capacity models. To the best of our knowledge, our method is the first one to produce such generations.
RELATED WORK
Since Ranzato et al. (2014) and Srivastava et al. (2015a) proposed the first video generation models inspired by the language modeling literature, there have been many papers that have proposed different approaches to represent and generate videos (Luc et al. 2017 2018 Villegas et al. 2017a b Xue et al. 2016).
We propose model for class conditional video generation, and therefore our setup is closely related to stochastic generative video models. Autoregressive models (Larochelle & Murray| [001] Dinh et al., 2016 Kalchbrenner et al. 2017; |Red et al.) 2017)) Weissenborn et al. (0200 approximate the joint data distribution in pixel space without introducing latent variables. These models capture complex pixel dependencies without independence assumptions. However, inference in autoregressive models often requires full model forward pass for each output pixel, making them slow and not scalable to long high resolution videos, with state-of-the-an models requiring multiple minutes to generate a single batch of samples (Weissenborn et al. 2020).
Variational AutoEncoders (VAEs) define latent variable models and use variational inference methods to optimize lower bound on the empirical data likelihood (Kingma & Welling 2013 Rezende et al. 2014 Babaeizadeh et al. 2017). Models based on VRNNs (Chung et al. | Denton & Fergus
2018
Castrejon et al.] 2019) use per-frame latent variables and have greater modeling capacity.
Normalizing flows (NFs) defne bijective functions that map probability distribution over latent variable a tractable distribution over data (Rezende & Mohamed| [015] Kingma & Dhariwal 2018) Kumar et al. 2019). NFs are trained to directly maximize the data likelihood. The main disadvantage of NF' is that their latent dimensionality has to match that of the data, often resulting slow and memory-intensive models.
Autoregressive models, VAEs and NFs are trained by maximizing the data likelihood (or a bound) under the generative distribution. It has been empirically observed that such models often produce blurry results. Generative Adversarial Networks (GANs) on the other hand optimize a min-max game between a Generator and Discriminator trained to tell real and generated data apart (Goodfellow et al. 2014). Empirically, GANs usually produce better samples but might suffer from mode collapse. GAN models for video were first proposed in Vondrick et al. (2016b a): Mathieu et al. (2015). In recent work, SAVP (Lee et al. 2018) proposed to use the VAE-GAN (Larsen et al. 2015) framework for video. TGANx2 (Saito Saito 2018) improves upon TGAN (Saito al. [017] and proposes a video GAN trained on data windows, similar to our approach. However, unlike TGANx2. our model composed of multiple stages which are not trained jointly. MoCoGAN (Tulyakov et al. 2018) first introduced a dual discriminator architecture for video, with DVD-GAN (Clark et al. 2019) scaling up this approach to high resolution videos in the wild. DVD-GAN outperforms MoCoGAN and TGANv2, and is arguably the current state-of-the-arr in adversarial video generation. We propose multi-stage generative model approach with each stage defining an adversarial game and a model architecture based upon DVD-GAN. Recent work (Xiong al. 2018 Zhao et al. (2020) also proposes multi-stage models, but differently from our approach their stages model different semantic aspects of the generation, such as producing motion outline.
2

---

Under review as a conference paper at ICLR 2021
3 METHOD
We consider dataset of videos (X1,..,, Xn) where each x; = (x4.0- x;TT) is a sequence of ' frames X; [2 Each x; comes from data distribution Pa- Our goal is to learn a generative distribution Py such that Pg - Pd-
3.1 STAGE-WISE GENERATIVE PROCESS
While our approach can be generalized to multiple stages, for clarity of exposition, we describe here two-stage version of the model. We begin by introducing view on the data. Generally, f denotes an operation that produces a view of x which decreases its dimensionality, and in this work spatiotemporal resolution of x. Our goal is to model the joint probability distribution Pa(x, x!).
variable x' = f(x) representing
nearest neighbor downsampling operation that reduces the
We define generative model that approximates the joint data-view distribution according to the following factorization:
Pg ( x²) Pos (x|x²) Ppn (x').
(1)
Each Py, defines stage in our model. This formulation allows us to decompose the training of the stages into a sequential process.
Training Stage We consider the distribution Py1 in eq. I and solve a min-max game with the following value function:
Vi Di) Extep, log (x') Ex, ee, log(1 Di (z))))
where G1 and D1 the generator/dissrriinntt associated with the first stage and Pe, is noise distribution. This is the standard GAN objective. As shown in Goodfellow et al. (2014), the min-max game ming, maxp, Vi (G1 Di) has global minimum when Psi (x') Pa(
Training Stage?
We formulate min-max game with the following value function:
V2(G2, D2) = ExtnpaBxxnnn
(2(x, x')) ))] + Eginpay Exg~Pe, [log(1 (2(Gu((2,, & )),, (3)
where G2 and D2 are the generator and discriminator of the second stage. The min-max game ming, maxp, 1/2 (2 D2) has global minimum when the two joint distributions are equal, xa(x,x!) Paz (x(x') Po, (x') (Dumoulin et al. 2016; Donahue et al. 2016). It follows that pa(x = Poz (x(x') when Pq. (x') pa().). This stage only learns the parameters associated with the distribution Pyz. as Ps. is trained the previous stage. However, even if the distribution Py, does not match exactly the marginal data distribution,
model still aims learning distribution P9> such that Po. (x(x')) ppg (x') approximates the joint data distribution. Therefore, approach finds generative distribution Py such that Py (x, x²) xx(
3.2 TRAINING ON LOCAL VIEWS OF THE DATA
We have shown that we can learn the joint data distribution with stage-wise strategy. However, this approach does yet have any significant computational benefits. In its current form, the second stage which upscales the low-resolution view x' parametrizes generative distribution over the full resolution video x. We now proceed describe locality assumption that allows us reduce the output dimensionality of the upscaling stage and therefore also reduce its computational complexity.
We decompose video and its view x into a of corresponding overlapping temporal windows x (Xw, ..... xw.) and x' = (x.,,...,,,, x_.)) We now assume:
p(xw,) T p(xw,) Xx Vi,j i and p(x..) ) I p(xx,,) x ii
(4)
We therefore assume that an output window Xw, is independent of all other data views conditioned on the corresponding oowresooutionn window x Given this locality assumption, we can rewrite the conditional data and generative distribution as:
(x =
II pa( , (x(..) and Pga (x)
II Pos (S.,, (2/m.))
(5)
3

---

Under review as a conference paper at ICLR 2021
This allows to rewrite the value function of eq. as:
Exl--pu Bx Be (((() ((xx)) (loggloogoono)l D2(Xw, xxm)] + Egg
EPa -Po Eerppe 10 D2(G2(22, &w)),, &))1
(6)
Eq. 6 describes the second stage value function in which we now match distributions of uniformly sampled data windows. Because these window sizes are fixed length and smaller than the full video length, the second stage now models a lower dimensional generative distribution.
This assumption introduces a trade-off between the modeling capacity of the second stage and the dimensionality of its outputs. Local upsampling stages might have trouble maintaining temporal consistency, especially for high resolution details and textures that are not fully defined by the low resolution inputs. In practice we define data windows with overlapping frames to aid maintain consistency. This is equivalent to defining fully convolutional upscaling model with temporal field of view smaller than the full input sequence. At inference time, we run the model convolutionally over the entire input.
4 MODEL ARCHITECTURE
In this section, we describe how parametrize the different stages of our generative model. A full description provided in the Appendix.
Stage We follow the DVD-GAN architecture (Clark et al. 2019) for the first stage of our model. The DVD-GAN generator stacks units composed by ConvGRU layer (Ballas et al.) (2015), modeling the temporal information, and 2D-ResNet blocks that upsample the spatial resolution. DVD- builds upon the dual discriminator of MoCoGAN (Tulyakov al. (2018) and defines both spatial discriminator that randomly samples & full-resolution frames and discriminates them individually, and a temporal discriminator that processes spatially downsampled but full-length videos.
The upsampling stage is composed by conditional generator and three discrim- inators (spatial, temporal and matching). The conditional generator produces an upscaled version &,. of a low resolution video x To discriminate samples from real videos, this stage uses a spatial and temporal discriminator, following DVD-GAN. Additionally, we introduce a matching discriminator (MD). The goal of MD is to ensure that the generation X is valid upsampling of &w- Conditioning on X!, allows to impose global temporal consistency on the generation & at inference time. Without this discriminator, the upsampling generator could learn to ignore the low resolution input video. The conditional generator is trained jointly with the spatial, temporal and matching discriminators. We now describe our conditional generator and matching discriminator.
Upsampling Stage
Conditional Generator The conditional generator takes as input a lower resolution video x noise vector and optionally a class label y. and generates &w, an upsampled version of x Before feeding it the generator, we increase the duration of the low resolution video X to the same temporal duration as the generator output by repeating its frames. Our conditional generator (see Figure IT stacks units composed by one 3D-ResNet block and two 2D-ResNet blocks. We observe that given enough capacity, the conditional generator with 3D convolutions matches the performance of similar model trained with ConvGRUs, while 3D convolutions allow for parallel computation over the time axis (sce appendix C Spatial upsampling is performed gradually by progressively increasing the resolution of the generator blocks. To condition the generator we add residual connections (He) et al.) 20166] Srriasttvaa et al., 2015b) from the low-resolution video to the output of each generator unit. In particular, we sum nearest- -neihbo interpolations of the lower resolution input to each unit output. We do not use skip connections for units whose outputs have higher spatial resolution than the lower dimensional video input, i.e. we do not upscale the low resolution video.
Matching Discriminator (MD) MD uses architecture similar to that of the DVD-GAN temporal discriminator. It discriminates real or generated (Xw, x pairs. Xw is downsampled to the same dimensionality x and both inputs concatenated on the channel dimension. The matching discriminator ensures that the model generates valid upsampled versions of its inputs, and therefore grounds upscaling stages to the low resolution input.
4

---

Under review as a conference paper at ICLR 2021
Generator
Closs
Discriminator
Clook
Consons
Speria
3D
ResoNod
block
2 RestVed block
2D Discriminatod
Sulsamppe
Repeat
Discriminator
Concal
Matching
Disertminator
Figure I: Stage 2 parametrization For upsampling stages, we adopt the DVD-GAN Clark et al. 2019) base architecture and re-purpose be conditional on the low-resolution samples. This includes replacing ConvGRUs with 3D convolutions, adding skip connections to x and adding matching discriminator that discriminates real or generated (x Xw.) pairs.
5 EXPERIMENTS
We consider the Kinetics-600 dataset (Kay et al. 2017 Carreira al. 2018) compare our approach with prior art (Clark et al. 2019) in class conditional video generation. Kinetics-600 is a large scale dataset of around 500k Youtube videos depicting 600 action classes. The videos captured in the wild and exhibit lots of variability. We follow the same data preprocessing of Clark et al. (2019) and use version of the dataset collected on June 2018. For unconditional video generation, we use the BDD100K dataset (Yu et al., 2018). BDD100K contains 100k videos recorded from inside cars representing more than 1000 hours of driving under different conditions. We use the training set split of 70K videos to train models.
Defining proper evaluation metries for video generation is an open research question. We use metrics inspired by the image generation literature and adapted to video. On Kinetics, we report three metrics for comparison to previous works (Clark et al. 2019 Unterthiner et al. 2018): i) Inception Score (IS) given by Inflated 3D Convnet (Carreira & Zisserman 2017) network trained on Kinetics-400, ïi) Frechet Inception Distance on logits from the same 13D network, also known as Frechet Video Distance (FVD) (Unterthiner al. 2018), and iii) Frechet Inception Distance the last layer activations 13D network trained on Kinetics-600 (FID). On BDD100K we report the FVD and FID metrics as described before. We do not report IS since there are well-defined classes in BDD.
Each stage of our model follows the DVD-GAN (Clark et al., 2019) architecture at given frame resolution with the modifications described in Section 4 for the upsampling stages. We also the same hyperparameters unless otherwise specified. For the rest of the section we denote video dimensions by their output resolution DxD and number of frames F in the format DxD/F. We use up to 128 n Vidia GPUs to train our models. All models are trained with batch size of 512. Further details can found in the Appendix.
5.1 TWO-STAGE SSW-GAN
In this section we evaluate a two-stage SSW-GAN to empirically validate the proposed training approach. For all experiments we train the first SSW-GAN stage generate 32x32/25 videos, with a temporal subsampling of 8 frames. The second stage upsamples by a factor of 2 the temporal resolution and by a factor of 4 the spatial resolution. resulting 128x 128/50 videos. During training the second stage operates locally 128x 18/6 or 128x 128/12 videos, respectively. At inference time we run the model convolutionally over the first stage 25 frames output to generate 128x 128 videos of up 50 frames.
input windows of 3 or 6 frames and is therefore trained to generate
We compare SSW-GAN with DVD-GAN as a baseline on Kinetics-600, We perform this comparison for two DVD-GAN models trained on 128x128/12 12 and 128x 128/48 in Table Additionally, also ablate our model to evaluate its performance without the matching discriminator in Table [ All models are trained for 300K iterations on Kinetics (100K for model used in the ablation experiments) and 35k iterations on BDD 100K, unless otherwise specified. Samples for our model are shown in Fig. 2 and in the Appendix.
5

---

Under review as a conference paper at ICLR 2021
20
40 45
Figure 2: Randomly selected SSW-GAN 128x128/50 frame samples for Kinetics-600: We show random samples from Kinetics-6k 600 128x128/12 model unrolled to generate 50 frames. Each row shows frames from the same sample different timesteps. We observe the generations remain fairly consistent through time while the frame quality does not degrade.
1=0
1=2
1=4
==
! 8
== 16
1 32
40
48
SIII
Figure 3: Unrolling DVD-GAN We show samples from 128x128/6 DVD-GAN baseline (our re- implementation) trained Kinetics-600 and unrolled to generate 128x128/50 vidcos. While samples valid for the first frames, they become motionless and degrade thereafter.
Comparison with prior work We compare model trained to generate 128x 128/12 videos the reported metrics for DVD-GAN in Clark et al. (2019) for Kinetics-600 128x 128 models. Table shows that model has better IS score than its DVD-GAN equivalent 12 frame generations. On 48 frames outputs, SSW-GAN outperforms reaches similar IS score. However, our second stage model is only trained 128x 128/12 outputs, as it is unrolled and applied convolutionally over the first stage output to generate 48 frames. In contrast, DVD-GAN models do unroll well and tend produce samples that become motionless past its training horizon (see Fig. 3). SSW-GAN therefore matches DVD-GAN on 48-frame generations in IS on Kinetics-600 and outperforms it in FID, while requiring significantly less computational resources.
28x 28/48 DVD-GAN model in FID score and
Impact of the temporal window size for the second stage One of the modelling choices in SSW- GAN the temporal window length used in the upsampling stages. We compare the 128/128/1 12 model used in the previous section to 228x128/6 version of our model. The 128x model requires approximately twice the amount of GPU memory needed by the frames model, but we expect it to perform better due to the larger input window. As can be seen in Table [ the 12 frames
'ttage is trained on 32x32/25 videos. Stage 2 trained on randomly selected 32x32/6 windows of stage 1 to generate 128x128/12 outputs. During evaluation, the scores obtained for 128x128/12 are obtained by upsampling 6 randomly selected frames from the stage 1. Stage 2 can be unrolled on stage frames to generate 128 128/48 outputs.

---

Under review as a conference paper at ICLR 2021
Table 1:
Results on Kinetics-600 compare different versions of our two-stage model against the reported metrics for DVD-GAN models (Clark et al. 2019). Our model when unrolled is able to generate 48 frames of similar quality as 128x128 DVD-GAN model trained to generate 48 frames while having computational requirements close the 128x 28/12 version. Furthermore,
compare two versions of our model with the second stage trained on different window sizes. We verify that the model with the bigger window size performs better at the cost being more computationally expensive.
12 frames
48 frames
Model
IS (t)
FID (1)
FVD (L)
IS (1)
FID (L)
FVD (L)
DVD-GAN 12 frames
77.45
1.16
DVD-GAN 48 frames
81.41
28.44
Ours
Stage 2 trained on 6 frames
58.21
31.59
714.74
Ours Stage 2 trained on 12 frames
104.00
2.09
591.90
77.36
14.00
517.21
10
=
15
20
25
30
35
40
45
No MD Stage
No MD Stage
MD Stage
MD - Stage 2
Figure 4: Matching discriminator We show here random sample from our two-stage model on Kinetics with (MD) and without the matching discriminator (no MD). For each sample we show the output of the first stage and the corresponding second stage output. We observe that, while the MD model generates plausible local snippets at stage 2, it does not remain coherent, Our model generates coherent sample because is grounded in the low resolution input.
model obtains significantly higher scores when unrolled to generate 48 frames. We conclude that the window size defines trade-off between computational resources, since shorter temporal outputs require less computation, and sample quality, as using a short length reduces the temporal field of view of stage 2 and the local independence assumption becomes less accurate.
Matching discriminator ablation We now investigate the importance of the matching discrimina- tor in Table On 6 frame generations. SSW-GAN and SSW-GAN (No MD) obtain similar scores. then unroll both models to generate frames, and observe that SSW-GAN (No MD) generates valid local snippets but is inconsistent through time. Fig. 4 shows an example of the unrolled no MD generations in which this effect is clearly observable. In contrast, our model with a matching discriminator stays consistent through time. This is reflected in the metrics well. Both models perform similarly at 6 frames, as the model without the matching discriminator is able to generate valid 128x 1 1286 videos. However, when unrolled this model performs significantly worse than regular model, as the metrics are capturing the temporal inconsistencies. This justifies the use of a matching discriminator, as it ensures that upsampling stage outputs are grounded its inputs.
5.2 STREE-STAGE SSW-GAN
To show that our model scales with the video dimensionality, we train a three-stage model on the BDD dataset to generate 128x128/100 videos. We train the first stage output 25 frames 32x32 resolution with a temporal subsampling of 8 frames. The second stage upscales windows of 12 frames at 64x64 resolution with temporal subsampling of 4 frames (since doubling the framerate
7

---

Under review as a conference paper at ICLR 2021
Table Stage 2 128x128/6 comparison We report metries on Kinetics and BDD for our model with/out matching discriminator (MD). Both models perform similarly for 6 frames, corresponding to the training video length. However. adding the MD improves the 50-frame generations, as they remain grounded to the first stage output allowing for consistent videns beyond the training length.
6 Frames
50 Frames
Dataset
Model
IS (1)
FID (1)
FVD (L)
IS (1)
FID (L)
FVD (4)
DVD-GAN (our reimpl.)
41.2
1.60
841.5
N/A
N/A
N/A
Kinetics-600
SSW-GAN (No MD)
50.31
1.62
594.99
37.81
42.29
1037.79
SSW-GAN
48.44
1.06
565.95
49.44
31.87
790.97
SSW-GAN (No MD)
1.36
211.69
26.52
575.51
BDDIOOK
SSW-GAN
1.07
144.96
18.73
326.78
Table 3: Computational costs: We compare the required memory and training time for DVD-GAN model its SSW-GAN equivalent. Our model takes roughly half of the training time and requires 4x less GPU memory. Data DVD-GAN is based in our reimplementation of model.
Samples/GPU
Memory (MB)
Iteration time (s)
Training time
Stage 32x32/25 Stage 2 128/12 DVD-GAN 128x128/50 128/50
27897
3.45
12 days
29073
3.2
days
32203
2.975#4
41 days
of the first stage). The third stage is trained to upscale 12 frame windows at 128x resolution for final temporal subsampling of 2 frames. A full pass for single example in DVD-GAN 128x128/100 100 model would require more than 32GB of memory, beyond the limits of most commercial GPUs. In contrast, are able to train our model using batch size 512 with 128 GPUs. Samples from this model can be seen in Figure
6 CONCLUSIONS
We propose SSW-GAN, multi-stage video generator that outputs long high-resolution videos, while being more computationally efficient than equivalent single-stage GAN approaches. High capacity models trained with big amounts of data are key aspects for video generation. SSW-GAN is step towards scalable models and allows training SOTA video GANs with fewer resources.
(=
15
Figure 5: Random 128x128/100 BDD100K samples: We show samples from our three-stage BDD100K
model. Each row shows different sample over time. Despite the two stages of local upsampling, the frame quality does not degrade noticeably through time.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Dinesh Acharya, Zhiwu Huang, Danda Pani Paudel, and Luc Van Gool. Towards high resolution video generation with progressive growing of sliced wasserstein gans. arXiv preprint arXiv:/810.02419, 2018.
Georgy Ayzel, Tobias Scheffer, and Maik Heistermann. Rainnet vl. O: a convolutional neural network for radar-based precipitation noweasting. Geoscientific Model Development, 13(6):2631-2644 2020.
Mohammad Babacizadeh, Chelsea Finn, Dumitru Erhan, Roy H Campbell, and Sergey Levine Stochastic variational video prediction. arXiv preprint arXiv: 1252, 2017.
Nicolas Ballas, Li Yao, Chris Pal, and Aaron Courville. Delving deeper into convolutional networks for learning video representations. arXiv preprint wrXiv:1511.064320 2015.
Andrew Brock, Jeff Donahue, and Karen Simonyan, Large scale gan training for high fidelity natural image synthesis. arXiv preprini arXiv: 1096, 2018.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165, 2020.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6299-6308, 2017.
Joao Carreira, Eric Noland, Andras Banki-Horvath, Chloe Hillier, and Andrew Zisserman. A short note about kinetics-600. arXiv preprint 0rX8vv/808.0013 340, 2018
Lluis Castrejon, Nicolas Ballas, and Aaron Courville. Improved conditional vrans for video prediction. In Proceedings of the IEEE International Conference on Computer Vision, pp. 7608-7617, 2019.
Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron C Courville, and Yoshua Bengio. recurrent latent variable model for sequential data. In Advances in neural information processing systems, pp. 2980-2988. 2015.
Aidan Clark, Jeff Donahue, and Karen Simonyan. Efficient video generation on complex datasets. arXiv preprint
2019.
Emily Denton and Rob Fergus. Stochastic video generation with a learned prior. In International Conference on Machine Learning, pp. 1182- 1191, 2018.
Emily L Denton, Soumith Chintala, Rob Fergus, al. Deep generative image models using laplacian pyramid of adversarial networks. Advances in neural information processing systems, pp. 14886-1494 2015.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv:1605.08800 2016.
Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell. Adversarial feature learning. arXiv preprint 1777v.1605.00782, 2016.
Vincent Dumoulin, Ishmael Belghazi. Ben Poole, Olivier Mastropietro, Alex Lamb, Martin Arjovsky, and Aaron Courville. Adversarially learned inference. arXiv preprint urXiv:1606 00704, 2016.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farles Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances neural informa- tion processing systems. pp. 2672-2680,22 2014.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.

---

Under review as a conference paper at ICLR 2021
Nal Kalchbrenner, Aäron van den Oord, Karen Simonyan, Ivo Danihelka, Oriol Vinyals, Alex Graves, and Koray Kavukcuoglu. Video pixel networks. In Proceedings of the 34th International Conference Machine Learning- Volume 70, pp. 1771-1779 JMLR. org, 2017.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint 0rXiv:1710.101900 2017.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaaanarasinhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, al. The kinetics human action video dataset. arXiv preprint arXiv: 7000000550000 705.06650, 2017.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv: 1412.6980, 2014.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv:1312.61 14, 2013.
arXiv preprint
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible Ixl convolutions. In Advances in Neural Information Processing Systems, pp. 10215-10224, 2018.
Manoj Kumar, Mohammad Babacizadeh, Dumitru Erhan, Chelsea Finn. Sergey Levine, Laurent Dinh, and Durk Kingma. Vidcoflow: A flow-based generative model for video. arXiv preprint arXiv: 1903.01444 2(5), 2019.
Hugo Larochelle and lain Murray. The neural autoregressive distribution estimator. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pp. 29-37, 2011.
Anders Boesen Lindbo Larsen, Søren Kaac Sønderby, Hugo Larochelle, and Ole Winther. Au- toencoding beyond pixels using learned similarity metric. arXiv preprint 11Xhv:1512.09300, 2015.
Alex X Lee, Richard Zhang, Frederik Ebert, Pieter Abbeel, Chelsea Finn, and Sergey Levine. Stochastic adversarial video prediction. arXiv preprint arXiv: 00.88000.15,,,. 2018.
Pauline Luc, Natalia Neverova, Camille Couprie, Jakob Verbeek, and Yann LeCun. Predicting deeper into the future of semantic segmentation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 648 648-657, 2017.
Pauline Luc, Camille Couprie, Yann Lecun, and Jakob Verbeek. Predicting future instance seg- mentation by forecasting convolutional features. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 584-599, 2018.
Michael Mathieu, Camille Couprie, and Yann LeCun. Deep multi-scale video prediction beyond mean square error. arXiv preprint arXiv: 15110054000 2015.
MarcAurelio Ranzato, Arthur Szlam, Joan Bruna, Michael Mathicu, Ronan Collobert, and Sumit Chopra. Video (language) modeling: baseline for generative models of natural videos. arXiv preprint arxiv: 141 2014.
Scott Reed, Aäron van den Oord, Nal Kalchbrenner, Sergio Gómez Colmenarejo, Ziyu Wang, Yutian Chen, Dan Belov, and Nando de Freitas. Parallel multiscale autoregressive density estimation. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 2912-2921. JMLR. org, 2017.
Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. arXiv preprint arXiv:1505.n 05770, 2015.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. arXiv preprint arXiv: 1403:::400..8.. 2014.
Masaki Saito and Shunta Saito. Tganv2: Efficient training of large models for video generation with multiple subsampling layers. arXiv preprint 1rXiv:1811.09222 2018.
Masaki Saito, Eiichi Matsumoto, and Shunta Saito. Temporal generative adversarial nets with singular value clipping. In ICCV, 2017.
10

---

Under review as a conference paper at ICLR 2021
Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov. Unsupervised learning of video representations using Istms. In International conference on machine learning, pp. 852, 2015a.
Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway networks. arXiv preprint arXiv: 00387, 2015b.
Sergey Tulyakov, Ming-Yo Yu Liu, Xiaodong Yang. and Jan Kautz. Mocogan: Decomposing motion and content for video generation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1536-1535, 2018.
Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A metric & challenges. arXiv preprint arXiv:/8/2..01717, 2018.
Ruben Villegas, Jimei Yang, Seunghoon Hong, Xunyu Lin, and Honglak Lee. Decomposing motion and content for natural video sequence prediction. arXiv preprint arXiv:1706.0m 7063003000........ 70.7a.
Ruben Villegas, Jimei Yang. Yuliang Zou, Sungryull Sohn, Xunyu Lin, and Honglak Lee. Learning to generate long-term future via hierarchical prediction. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 3560-3569. JMLR. org, 2017b.
Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Anticipating visual representations from un- labeled video. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 98-106,
2016a.
Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dynamics. In Advances in Neural Information Processing Systems, pp. 613 621, 2016b.
Dirk Weissenborn, Oscar Täckström, and Jakob Uszkoreit. Scaling autoregressive video models. International Conference Learning Representations, 2020.
Wei Xiong, Wenhan Luo, Lin Ma, Wei Liu, and Jiebo Luo. Learning generate time-lapse videos using multi-stage dynamic generative adversarial networks. Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp.
2018.
Tianfan Xue, Jiajun Wu, Katherine Bouman, and Bill Freeman. Visual dynamics: Probabilistic future frame synthesis via cross convolutional networks. In Advances in neural information processing systems, pp. 91-99, 2016
Fisher Yu, Wenqi Xian, Yingying Chen, Fangchen Liu, Mike Liao, Vashisht Madhavan, and Trevor Darrell. Bdd100k: A diverse driving video database with scalable annotation tooling, arXiv preprint 1805:1805.04687, 2018
Long Zhao, Xi Peng, Yu Tian, Mubbasir Kapadia, and Dimitris N Metaxas. Towards image-to- video translation: A structure-aware approach via multi-stage generative adversarial networks. International Journal of Computer Vision, 2020.

---

Under review as a conference paper at ICLR 2021
A ADDITIONAL IMPLEMENTATION DETAILS
We use the Adam optimizer Kingma & Ba 2014) with learning rate À Ie10-4 and Ap 5e10-1 for the generator and the discriminator, respectively. The discriminator is updated twice for each generator update.
We use orthogonal initialization for all the weights in our model and use spectral normalization both in the generator and the discriminator. We only use the first singular value normalize the weights. Different from DVD-GAN, we do not use weight moving averages nor orthogonal penalities.
Conditional batch normalization layers use the input noise as the condition, concatenated with the class label when applicable. Features are normalized with per- rermframe mean and standard deviation.
To unroll a generator beyond its training temporal horizon, we apply it convolutionally over longer input sequences. We perform 200 "dummy" forward passes to recompute the per-timestep batch normalization statistics at test time.
All convolutions in our models use 3x3 or 3x3x3 filters with padding=1 and stride=1, for 2D or 3D convolutions respectively. All models were implemented PyTorch
B MODEL ARCHITECTURE DETAILS
For the rest of the section, we use B to denote the batch size, T for the number of frames or timesteps, for the number of channels, H for the height frame and is the width of the frame.
B.1 STAGE ARCHITECTURE
Our first stage model is based on a re-implementation of DVD-GAN Clark et (2019). In all our experiments, the first stage produces 32x32/25 outputs.
The generator is composed by a stack of units where each unit is comprised a ConvGRU layer and two 2D-ResNet upsampling blocks. We follow the nomenclature of [Brock] ct al. (2188; Clark et al. (2019) and describe our network using a base number of channels ch and the channel multipliers associated with each unit. Our stage- I generators is formed by 4 units with channel multipliers [8, 8, 4, 2]. The base number of channel is 128.
Generator
The first input of this network is of size BxTx(8xch)44x4. This input is obtained by first embedding the class label onto a 128 dimensional space, then concatenating the embedding to a 128 dimensional Bx(8xch)x4x4 tensor with a linear layer and
noise vector. This concatenation is mapped reshape, and then the final tensor is obtained by replicating the output of the linear layer T times.
The COnVGRU layer Ballas et al. 2015) follows the ConyGRU implementation of Clark al. (0099) and uses a ReLU non-linearity to compute the ConvGRU update.
The 2D ResNet blocks are of the normalization layers, ReLU activations and standard 2D convolutions. Before the first convolution operation and after the first normalization and activation, there is optional upsampling operation when increasing the resolution of the tensor. We use standard nearest neighbor upsampling. Except for the last unit, all units perform this upsampling operation. The conditional batch normalization layers receive the embedded class label (if applicable) and the input noise as a condition and map it to the corresponding gain and bias term of the normalization layer using a learned linear transformation. The 2D ResNet blocks process all frames independently by reshaping their input to be (B*T)xCXHx..
teatt-ooonnoom-act--ooo style. We conditional batch
The output of the last stack goes through final norm-relu-conv-tanhh block that maps the output tensor to RGB space with values in the [-1, 1] range.
Discriminator There are two discriminators, a 2D discriminator and a 3D discriminator, The 2D discriminator is composed of 2D ResNet blocks. Each ResNet block is formed by a sequence of relu-conv-relin col---el---oooo layers. There are no normalization layers in the discriminator. After the last conv in each block there is optional downsampling operation, which is implemented with average
12

---

Under review as a conference paper at ICLR 2021
pooling layers. The 2D discriminator receives input 8 randomly sampled frames from real or generated samples.
The 3D discriminator is equal to the 2D discriminator except that its first two layers are 3D ResNet blocks, implemented by replacing 2D convolutions with regular 3D convolutions. The 3D discrimina- tor receives input spatially downsampled (by factor of two) real generated sample. The 2D) blocks process different timesteps independently.
We concatenate the output of both discriminators and use geometric hinge loss. The loss is averaged over samples and outputs.
We use 128 base number of channel for both discriminators, with the following channel multipliers for each ResNet block: [16, 16, 8, 4, 2]
B.2 UPSAMPLING STAGE ARCHITECTURE
The upsampling stage models follow the same architecture as the first stage with the following modifications.
The generator units replace the ConvGRU layers with a Separable 3D convolution. We first convolve over the temporal dimension with ID temporal kernel of size 3 and then convolve over the spatial dimension with 2D 3x3 kernel. We empirically compare generators with ConyGRU and separable convolutions in section
Generator
We add residual connections at the end each 3D and 2D ResNet block to an appropriately resized version of x We use nearest neighbor spatial downsampling for this operation, and we nearest neighbor temporal interpolation increase the number of frames of x! We then map the residual to the appropriate number of channels using a linear 1x1 convolution. We do not add x residual connections to feature maps with spatial resolutions (HxW) greater than the resolution of x!
Diseriminator We reuse the same 2D and 3D discriminators as for the first stage. Additionally, we add matching discriminator that discriminates (xle, Xms;) pairs. The matching discriminator utilizes the same architecture the 3D discriminator. It receives input concatenation of x and downsampled version of Xw; match the resolution of x! We concatenate the outputs of all three networks and use geometric hinge loss as for the first stage discriminator. The overall loss is averaged over samples and output locations.
For 128x 128 generations on Kinetics, the generator uses 128 as base number of filters with the fol- lowing channel multipliers [8, 8, 4, 2, 1). All discriminators have 96 base channels and the following channel multipliers [1, 4, 8, 16, 16). All our models at 128x1 28 are two-stage models. We train models to upsample x sizes 32x32/3 or 32x32/6 to 128x 12/6 or 128x 128/12, respectively. Since we train our first stage for 32x32/25 outputs, our two-stage models can generate 128x outputs when unrolled.
For 128x 18x128 generations BDD100K, the generator uses 96 as the base number of channels with channel multipliers [8, 4, 4, 2, 2]. All discriminators have 64 base channels and channel multipliers [1, 2, 4, 8, 8, 16). Our 128x128 models upsample 64x64/6 x inputs to 28x122/12 12, and can generate outputs of up to 12/1000
13

---

Under review as a conference paper at ICLR 2021
1.2
0.5
2.6
0.4
0.2
0.2
23200
10000 00000
80000
320000 120000 140000
20050 400000
00/00
90000
100200
120000 140000
Rarabars
Figure Comparison of recurrent layers We compare two variants of the same generator, one with a single ConvGRU layer per generator block and one with separable 3D convolution per generator block. On the left we show the evolution of the FVD score during training, and on the right we show the Inception Score. Both scores are normalized to the [0, I] range where I is the highest score obtained by these models and the lowest. Both models have similar behaviour and computational costs, but the 3D convolution processes inputs in parallel.
C COMPARISON OF RECURRENT LAYERS
In this section we justify the change of the ConvGRU for Separable 3D convolutions in upsampling stages. In Figure O we compare the evolution of two metries (IS and FVD) during training for two variants the same two-stage model, one using ConvGRUs and one using separable convolutions. Both models show similar behavior during training and achieve similar final metrics. However, ConvGRUs perform sequential operations over time whereas 3D convolutions can be parallelized.
D ADDITIONAL SAMPLES
Additional samples can be found in .mp4 format along with this appendix in the supplementary materials file. These videos show multiple samples from our two-stage model for Kinetics and our three-stage model for BDD. For each sample, we show the output of each stage a row.
We have included 3 videos. One video shows samples from 128x 128/12 model unrolled to generate 128x 128/100 samples on BDD. Another video shows samples from our 128x 128/12 model unrolled to generate 228x128/57 samples on Kinetics. Finally, we include video with samples from the same model but without the matching discriminator (baseline no matching disc suffix filename).
We include some additional samples for Kinetics 128x128/11 model and BDD 128x 128/12 model below.
For all evaluations, sample from isotropic Gaussian with unit variance for of comparison and reproducibility. Samples for figures and the provided video files were produced by sampling with standard deviation a 0.5. We observed that noise samples with reduced variance produce higher quality samples but are slightly less diverse.
14

---

Under review as a conference paper at ICLR 2021
t-0 0
5
t=
10
15
20
25
2
30
35
40
45
Figure 7: Additional samples Kinetics 128x128/12 show additional samples from our two- stage Kinetics 128x128/12 model unrolled to generate 128x 128/50 videos. More samples can be found in the supplementary videos.
15

---

Under review as a conference paper at ICLR 2021
t= 15
= 25
t 35
t= 15
55
65
75
85
95
Figure 8: Additional samples for BDD 1281128/100 We show additional samples from our three-
stage BDD 188/12 12 model unrolled to generate 128x 128/100 videos.
16

---

Under review as a conference paper at ICLR 2021
E POWER SPECTRUM DENSITY
Some video generation models tend to produce blurry results over time. In this section we generate Power Spectrum Density (PSD) plots, following (Ayzel et al. 2020 assess whether our generations become blurrier over time.
We conduct this experiment on Kinetics for videos of 50 frames at 128x 128 pixels resolution. We our model with the first stage trained on 32x32/25 sequences and the second stage trained to generate 128x128/12 video snippets from 32x32/6 windows, and unrolled after training to produce 128x128/50 videos. We took 1800 random videos from the ground truth data (GT) and 1800 generations from our model. We compute the PSD at frames 1, 10, 25, and 50 of each video. For each set of 600 videos, we compute the average PSD across videos, on per frame basis. Finally, we use the three sets of 600 videos to compute the standard deviation and mean for the average PSD of the original data and our generations. Figure 2 shows the plots for different frame indices. We observe that our generations have a very similar PSD to that of the ground-truth data in all video frames. This indicates that our generations, while they might not be accurate, have very similar frequeney statisties as the ground-truth data. We do not observe any significant blurring over time, and this is reflected in the plots, showing that even for frame 50 the high frequency part of the PSD is very similar between the original data and our generations.
PSD Step
PSD Step 10
GT
10
Pred
1020
Pred
Power Spectrum
10a
Power Spectrum
10a
10a
10a
10
107
10a
10a
0
20
40
60
0
40
60
80
Spatial Frequency
Spatial Frequency
PSD Step 25
PSD Step 50
GT
1030
Pred
1000
Prdd
Power Spectrum
109
Power Spectrum
10a
10a
10a
107
107
10a
10f
60
Spatial Frequency
Spatial Frequency
Figure 9: Power Spectrum Density (PSD) plots for different time steps We show comparison of the PSD between the original data and our generations at different steps in the predictions. We observe that our generations have a similar PSD to that of the original data, even at the end of the generation, indicating that the generations do not blur over time significantly.
17

---

Under review as a conference paper at ICLR 2021
F UPSAMPLING VISUALIZATIONS
Here we show some examples of stage generations on Kinetics-600 for a 32x32/25 model, as well the corresponding 128x 128/50 generations from stage 2 trained upscale 32x32/6 windows to 128x 128/ 12 and unrolled over the whole first stage generation. Examples are shown in Figure TO in which, for each example, we show the stage generation on the top row and the corresponding stage generation in the lower row. We observe that the second stage adds details and refines the low resolution generation, but the same time keeps the overall structure of the low resolution generation and is properly grounded.
5
15
20
Figure 10: Pairs of samples from stage 1 and their corresponding stage 2 output We show few examples from our 128x 128112 two-stage model trained on Kinetics-600 and unrolled to generate 128x 128/50 videos. For each example, we show the first stage low resolution generation and the corresponding stage upsampling. Stage 2 outputs refine the details of the first stage generations but retain the overall scene structure.
18

---

Under review as a conference paper at ICLR 2021
G INFLUENCE OF MOTION ON THE RESULTS
In this section we analyze whether our model has different performance for categories with different motion characteristics.
We conduct this experiment for the our Kinetics-600 model trained to generate 32x32/25 videos in the first stage upscale 32x32/6 videos 128x 128/12 videos for the second stage, which we then subsequently unroll over the full first stage generation to obtain 128x 1 18/50 videos.
We randomly selected five categories of videos with high motion content (bungee jumping, capoeira, cheerlcading, kitesurfing and skydiving) and five categories with less dynamic videos (doing nails, cooking egg, crying, reading book and yawning). We then generated 1000 samples from our model for each category, and used all the available samples in the dataset to compute IS and FID scores per category.
Table 4 shows the IS and FID scores of each class, while Figure 11 and Figure 12 show samples from high motion and low motion categories, respectively. We do not observe significant trend that indicates that the model produces worse generations for high motion classes. Some low motion categories have high FID scores similar to the highest scores for the high motion categories, while on average the IS scores for the high motion categories are slightly better. We also do not notice qualitative difference the samples. Instead, there might be other factors (such as the amount of structure present in scene) that have greater impact the quality of the generations.
Table 4: Per category scores for classes with different amounts of motion (Kinetics-600) We report per- class IS and FID scores for randomly selected categories high motion and categories with low motion. We observe that there is a high variability in FID scores, with some classes with low motion having high scores well as some high motion classes. In IS scores there are few differences between the two groups, with the high motion group having slightly higher mean score.
50 frames
class
is (1)
fid (1)
# Videos
High Motion
Bungee Jumping 11.66
82.21
799
Capoeira
9,48
84.57
816
Cheerleading
12.10
116.84
982
Kitesurfing
11.36
108.81
648
Skydiving
5.99
90.25
983
Low Motion
Doing Nails
13.32
91.67
537
Cooking Egg
7.60
111.63
441
Crying
8.92
70.74
627
Reading Book
11.13
64.97
793
Yawning
9.71.
79.08
530
19

---

Under review as a conference paper at ICLR 2021
t=
==
==
== 15
20
25
30
I 35
40
45
Figure 11: Samples from Kinetics-600 classes with high motion content
t=
t= 10
t= 15
20
25
= 30
35
40
45
Figure 12: Samples from Kinetics-600 classes with low motion content
20

---

Under review as a conference paper at ICLR 2021
H JOINT TRAINING OF MULTIPLE STAGES
Our proposed method consists of multiple stages trained independently and sequentially. This allows us reduce the memory requirements, we do not have keep the intermediate activations to backpropagate through the full model, and allows us to define stages with larger memory requirements than feasible if the model was trained end-to-end. However, our approach does not require each stage to be trained independently. Furthermore, with end-to-end training we could fit an equivalent full model a single optimization round, thus reducing the training time, and potentially finding better solution. Additionally, we might not need to discriminate the output of intermediate stages nor use matching discriminators. this section we report some initial results investigating the joint training of the different stages.
use two-stage model trained on Kinetics-600. The first stage is trained to generate 32x32/6 videos, and the second stage is trained to upscale 32x32/3 snippets 64x64/6 videos. When unrolled over the full first stage generation, this model is capable of producing 64x64/12 videos. We only discriminate the output of the second stage, with a spatial and temporal discriminator. We don t use any matching discriminator in this experiment.
generations from this model. We observe that the output of the first stage is longer a valid low resolution video. However, the second stage learns to generate valid video snippets. In general we observe that the generations from this model do exhibit lot of motion. This could be due to the window size used in the experiment or to the joint training setup.
Figure 13 shows
potential avenue improve joint training would be use discriminators the first stage output and/or use a matching discriminator to discriminate pairs of first and second stage outputs.
Stage
Stage
Stage
Stage 2
Stuge
Stage
Stage
Stuge
Stage
Stage
Figure 13: 64x64/12 samples from the jointly trained model on Kinetics-600 We show unrolled samples from a variant of our model with all stages trained jointly. The output of the first stage is not a valid generation anymore, but the second stage learns to generate valid video snippets. Samples do not exhibit lot of motion.
21

---

Under review as a conference paper at ICLR 2021
I BROADER IMPACT
Here we discuss the broader ethical impacts of this work. SSW-GAN is generative model for video. As with other generative models, there a chance that similar methods the one propose might be used to create "deepfakes". Note however that it would require extensive follow-up work and that it would not be direct application of our methods. Furthermore, it is not possible know in advance what generation is going to look like, and therefore our model could produce results that could be nonsensical in bad taste.
At the same time, generative models for video have multiple positive applications. First, their generations can used to train better classifiers. They can also be used to facilitate content creation for visual artists. Furthermore, variants of our model could be used to better compress videos, which has advantages such as reduced bandwidth requirements to transmit video data.
Overall our model would require follow-up work to enable some of the positive and negative applications described, as its current form it is class-conditional generative model with main ability being that of generating samples similar to those used for its training.
22


Reviews:
[{"review_id": "zpYaN8vCLe", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The method shows promising results on on generating high duration (up to 100 frames) class conditional videos with convincing Inception scores, indicating quality similar to DVD-GAN, while consuming less memory and with better coherence. \nWhile the contribution of the paper is mainly to improve the DVD-GAN architecture to reduce training time and memory consumption, the reviewer believes that the paper would be a good contribution to the venue.\nBelow there are some questions on the methodology:\n1. Is there any way to tell does the matching discriminator actually only estimates the ability to upsample $\\hat{x}_w$ from the previous low resolution sample $x_w^l$? From the architecture it is not evident whether or not it only does this or it is also entangled with assessment of how good the low-resolution sample $x_w^l$  was. In other words, if the low resolution sample scores good (e.g. because it's the real-world data) but the upsampling does not match,  would the objective of the matching descriptor training still score it as a good upscaling?  Or is there any reason preventing from this type of behaviour? \n2. Although, as mentioned in the introduction, it may not be as big problem as for VAE-based models, the problem of blurring might exist for DVD-GAN-like models. It is written in the caption of Figure 5 that 'Despite the two stages of local upsampling, the frame quality does not degrade noticeably through time.’ Although the reviewer appreciates that previous work reported only IS/FID/FVD metrics and that defining proper evaluation metrics for generative models is an open question, it might be a good idea to show some other quantitative metrics such as power spectral density (PSD) plots similar to figure 5 from [1]. This would help get an idea how it compares to the real-world video in terms of blurring of the results. \n3. Given that the generation of videos is class-conditional, is it possible to show the metrics per class? Are the scores per class similar or does the method score better for larger classes or the classes with specific motion dynamics? \n\n[1] Ayzel et al (2020) RainNet v1.0: a convolutional neural network for radar-based precipitation nowcasting", "extra_scores": {}, "extra_text": {}}, {"review_id": "sa9JPWHZ-M", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Paper Contributions**\n\nThe paper proposes SSW-GAN, an adversarial generative model of video which proposes a new generator architecture along with splitting the training into multiple stages.\n\n**Strong points of the paper**\n\n* The results are very strong.\n* Prior work has focused on efficient decomposition of the discriminator, this work focuses on decomposition of the generator (effectively), and this is an extremely reasonable direction to take adversarial video research in.\n* The claim that this requires substantially less computational cost is grounded.\n\n**Weak points of the paper**\n\n* A major departure from prior work is training stage 1 of SSW-GAN separate of stage 2, but this is independent of the computational benefits present from the generator architecture innovations. It is missing an important ablation showing the results of training the stages jointly.\n* The generator architectures changes from prior work are quite concrete, but the high level description doesn't clearly reflect that.\n* Some of the comparisons and model descriptions are lacking details, which make it difficult to understand experiments.\n\n**Clearly state your recommendation (accept or reject) with one or two key reasons for this choice.**\n\nI believe this paper is an accept (7), however I think there are a number of places where the description of the architecture and experiments needs more detail, and for my final rating I would like to see these addressed in rebuttal. \n\nFurthermore, I think there is a key experiment missing. Were that to be added I would strongly consider moving to clear accept (8).\n\n**Supporting arguments for your recommendation.**\n\nThere are two key innovations in the SSW-GAN model:\n* A generator architecture which provides output to discriminators that does not require an entire high-resolution full-length sample to be generated (because between stage 1 and 2 you select only a window of frames to run stage 2 on).\n* Splitting the training into two phases, where at first you only train stage 1 and then you train stage 2.\n\nThese are very interesting ideas, since either one substantially reduces the training-cost of the video model (something aptly described in the paper), and not something strongly touched on by previous work. In addition, the metric scores of the SSW-GAN model are very nice, state of the art in almost all cases. For that reason I think the work in this paper is worthy of acceptance.\n\nHowever there are a couple points of clarification and increased description which I think are needed, and I would like to see the following points addressed in the rebuttal:\n\n* The core idea of judging fixed-length output upscaling as a generative problem seems like a very clear and reasonable idea. Most of the high-level description of the most omits this in place of a general “multi stage definition” of the SSW-GAN model. I think the introduction and abstract would benefit about being more clear with regards to this change.\n\n* In section 4, it is unclear if all four bold sections are trained independently or not. I think it would be good to be clear in the paper-structure which components are trained together.\n\n* In general, the paper is not super clear where upsampling (both in time and space) occurs. I think it would be good to describe this in the paper text and in the architecture figures.\n\n* I am not super clear on the comparisons between SSW-GAN and DVD-GAN in “comparison with prior work”. When you say \"our model trained to generate 128x128/12 videos “ is this the model which had a first stage trained on 32s32/25 and then you trained the second stage on input windows of 6 frames, generating 128x128/12, and took just single samples from that to compare?\n\n* Similarly, when you say “However, our model is only trained on 128x128/12 outputs, as it is unrolled and applied convolutionally over the first stage output to generate 48 frames.”, isn’t it the case that the first stage is trained on longer sequences?\n\n*  In Table 1, the numbers for DVD-GAN seem lifted from the paper, which I believe is using a Kinetics-600 trained I3D for metric calculation. This means the FID numbers are comparable, but in section 5 when describing IS you say you are using a Kinetics-400 trained I3D (like in FVD). Is this correct (in which case the numbers are not quite comparable) or is this is just mis-explained, and all numbers in Table 1 come from Kinetics-600 trained I3Ds?\n\n* I think the IS/FID metrics you pick in Table 1 are the better metrics, but could you also add FVD numbers? That would allow you to compare against TriVD-GAN [1], which outperforms DVD-GAN. I think this is important because that paper also discusses modifications which reduce the memory requirement of DVD-GAN.\n\nFinally, there is a major question this paper does not address: **is the two-stage training of SSW-GAN necessary, or can it be trained in a single pass** (but with the generator decomposition as described)? Doing so would be simpler, and also might potentially reduce the need for the Matching Discriminator, which the paper contains an ablation for, but I think needs a further ablation when the model is not trained in two stages.\n\nI believe a key experiment would be training an SSW-GAN architecture but in a single pass, the results of that experiment would mean quite a lot for interpreting the changes described in this paper. In particular, the paper's title and abstract focus on the multi-stage aspect of the model, but skim over the substantial change of making the generator architecture more modular and scalable. I think it would be very beneficial to understand how each of these changes independently effect the performance of the model.\n\nI do not think this ablation is required to maintain an accept rating, but including it would push my rating closer to clear accept, and be quite a strong addition to the paper's content.\n \n[1] https://arxiv.org/abs/2003.04035\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "oI6oCp8aam6", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a GAN-based model which generates videos in multiple stages. The main idea is the upsampling of the spatio-temporal resolution upon addition of a stage. This is the key feature of the proposed model allowing the model to generate videos of higher temporal resolution while using significantly less computational resources. \n\n**Strengths**\n+ The paper is clearly written\n+ The model performs competitively with relevant baselines with respect to quantitative metrics\n+ The evaluation of the model has been conducted on real world datasets\n+ Implementation details have been mentioned clearly\n\n**Weaknesses**\n- There have been earlier attempts for multi-stage video generation  [1,2]. However, the paper misses citations in this direction. Also, apart from condition for the generation, how is the proposed model different from the existing multi-stage ones?\n- The generated samples for Kinetics dataset are not temporally consistent and misses several details especially for smaller entities in video. To list a few: in Figure 2 row 4, the baby's face looks distorted and different in every frame; in Figure 3 row 2 and in Figure 2 row 1, the face of the person is completely incomprehensible.\n- The generated samples in the paper do not have a lot of perceived motion in them. How does the model perform when the input class is supposed to possess huge temporal variations?\n\nOverall, the paper presents a scalable way to generate video with higher temporal resolution. However, the generated results do not look realistic and lot of important details are missing in the generated samples. Therefore, my initial rating for this paper is 4. \n\n\n*References used in the review:*\n\n[1] Learning to generate time-lapse videos using multi-stage dynamic generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018\n\n[2] Zhao, L., Peng, X., Tian, Y., Kapadia, M. and Metaxas, D.N., Towards Image-to-Video Translation: A Structure-Aware Approach via Multi-stage Generative Adversarial Networks. International Journal of Computer Vision, 2020\n\n\n\n============================================**Post-Rebuttal Comments**==================================\n\nI appreciate the revisions and additional results presented by the authors. The authors have addressed my concerns as well as improved the clarity of the model description in the revised version of the paper. While that results are not perfect, I acknowledge that the problem of video generation is difficult and I believe such multi-stage model can motivate future methods in this direction of scalable video generation. Therefore, I would like to improve my score to 6 and would recommend acceptance of this paper.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "S_P6fy7E5qA", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Pros:\n1. A stage-wise approach to train GANs for video is defined to reduce the computational costs needed to generate long high resolution videos.\n2. The authors provide some quality results of the proposed approach.\n\nCons:\n1. The contribution of this paper is very limited. The authors just do some incremental improvement based on current GAN models, and the theoretical analysis for the stage-wise training approach is not enough. \n2. The experiments are not convincing. The authors only compared the baseline methods in the experiments. Besides, the proposed training strategy should be applied in different generation models based on GAN to show the effectiveness in different cases. \n3. This paper aims to reduce the computation cost of the model training, but do not achieve significant effect, which takes 23 days for model training.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "EBNqdQHe1k2", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe paper proposes a stage-wise training pipeline for training 128x128 resolution videos of up to 100 frames. It starts by generating low resolution and temporally downsampled videos, and upsample the results in a stage-wise manner. Experimental results on Kinetics-600 and BDD100K demonstrate that the network is effective in generating higher resolution videos.\n\nStrengths:\nThe idea is easy to understand. The paper is well written and easy to follow. Quantitative results show that the proposed method is superior than existing methods under some circumstances.\n\nWeaknesses:\n1.\tThe novelty is very low. Stage-wise and progressive training have been proposed for such a long time, they have been used everywhere. The way the authors use them don’t really exhibit anything novel to me.\n2.\tThe resolution of the outputs (128x128) is lower than prior works (e.g. DVD-GAN has 256x256 outputs). Since the paper claims the computation cost is lower, one would expect the model can generate higher resolution and much longer duration videos, but in fact it’s quite the opposite. To prove the effectiveness, I feel the authors need to show something higher than 256x256, say 512 or 1024 resolution. On the other hand, the hardware requirement is still high (128 GPUs) instead of some normal equipment that everyone can have, so I really don’t see any benefit of the model. If the authors can train DVD-GAN using only a handful of GPUs, that might also be a contribution, but it’s not the case now.\n3.\tOutput quality is reasonable, but still far from realistic. Recent GAN works have shown amazing quality in synthesized results, and the bar has become much higher than a few years ago. In that aspect, I feel there’s still much room for improvement for the result quality.\n\nOverall, given the limited novelty, low resolution output and still high hardware requirement, I’m inclined to reject the paper.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a GAN for video generation based on stagewise training over different resolutions, addressing scalability issues with previous approaches. Reviewers noted that the paper is clearly written, proposes a method that improves upon the DVD-GAN architecture by reducing training time and memory consumption, and has competitive quantitative results.

On the other hand, the more negative reviewers are concerned that the empirical improvements demonstrated are somewhat incremental, and that there is not much novelty as the proposed approach is similar to other methods that decompose the generation process into multiple stages at different temporal window lengths and/or spatial resolutions. The authors argue that these criticisms are subjective and non-actionable. I sympathize with their frustration, but an acceptance decision for a competitive conference like ICLR does involve some subjective judgment as to whether the method and/or results meet a high bar beyond mere correctness. For this submission that's a close call, but between the novelty/incrementality concerns and the other more minor issues raised by reviewers (e.g., missing frame-conditional evaluation) I believe this paper could benefit from another round of revisions and improvements and recommend rejection.

I hope the authors will consider improving the submission based on the reviewers' feedback and resubmitting to a future venue, as the paper certainly has merit. To this end I have a few concrete recommendations for the authors which could have flipped my recommendation to an accept if implemented:

* Report results in the frame-conditional setting for comparison with DVD-GAN and other methods that operate in this setting.
* Proofread the paper more thoroughly. I noticed several typos while skimming the paper, e.g. in the theory section, the second term of eq. 6 confusingly uses $\rho$ instead of $\log$. (Relatedly, given that appendix B.1 reports that the hinge loss is used, I'm not sure whether $\log$ is correct in the first place -- this probably deserves further explanation or correction.)
* Demonstrate/argue more convincingly (in one way or another) that SSW-GAN's improved efficiency really expands the frontier of what was possible before. It is true that the 128x128/100 video samples contain 2x as many total pixels as DVD-GAN's 256x256/12 samples, but this isn't a *strict* improvement as the spatial resolution is smaller, and a 2x difference leaves space for reviewers to reasonably wonder whether previous methods really couldn't have matched this if pushed. Some possible examples of this: show that SSW-GAN can generate longer 256x256 videos (a strict improvement over what was possible with DVD-GAN), or orders of magnitude longer (e.g., 1 minute) but still temporally coherent videos at 128x128, or videos with substantially improved subjective sample quality at the same (or higher) resolution.
* The paper notes that "DVD-GAN models do not unroll well and tend to produce samples that become motionless past its training horizon". If this were quantified, e.g. by additionally reporting IS/FID/FVD separately for different timestep ranges, it could make a more compelling argument in favor of SSW-GAN.
Decision: Reject