Adversarial Video Generation on Complex Datasets
Authors: Aidan Clark
Jeff Donahue
Karen Simonyan
Venue:  (2020)
Keywords: GAN
generative model
generative adversarial network
video prediction

Abstract: Generative models of natural images have progressed towards high fidelity samples by the strong leveraging of scale. We attempt to carry this success to the field of video modeling by showing that large Generative Adversarial Networks trained on the complex Kinetics-600 dataset are able to produce video samples of substantially higher complexity and fidelity than previous work.  Our proposed model, Dual Video Discriminator GAN (DVD-GAN), scales to longer and higher resolution videos by leveraging a computationally efficient decomposition of its discriminator. We evaluate on the related tasks of video synthesis and video prediction,  and achieve new state-of-the-art Fréchet Inception Distance for prediction for Kinetics-600,  as well as state-of-the-art Inception Score for synthesis on the UCF-101 dataset, alongside establishing a strong baseline for synthesis on Kinetics-600.

Under review as a conference paper at ICLR 2020
ADVERSARIAL VIDEO GENERATION ON COMPLEX DATASETS
A Anonymous authors Paper under double-blind review
ABSTRACT
Generative models of natural images have progressed towards high fidelity samples by the strong leveraging of scale. We attempt carry this success the field of video modeling by showing that large Generative Adversarial Networks trained on the complex Kinetics-600 dataset are able to produce video samples of substantially higher complexity and fidelity than previous work. Our proposed model, Dual Video Discriminator GAN (DVD-GAN), scales to longer and higher resolution videos leveraging computationally efficient decomposition of its discriminator. We evaluate on the related tasks of video synthesis and video prediction, and achieve new state-of-the-ar Fréchet Inception Distance for prediction for Kinetics- 600, as well as state-of-the-arr Inception Score for synthesis on the UCF-101 dataset, alongside establishing strong baseline for synthesis on Kinetics-600.
INTRODUCTION
Time
Figure 1: Selected frames from videos generated by a DVD-GAN trained Kinetics-600 at 256 128 x 128, and 64 x 64 resolutions (top to bottom).
Modern deep generative models can produce realistic natural images when trained on high resolution and diverse datasets (Brock et al. 2019 Karras et al. 2018; Kingma & Dhariwal 2018 Menick &
Kalchbrenner] 2019 Razavi et al. 2019). Generation of natural video an obvious further challenge for generative modeling, but that is plagued by increased data complexity and computational requirements. For this reason, much prior work on video generation has revolved around relatively simple datasets, or tasks where strong temporal conditioning information is available.
We focus on the tasks of video synthesis and video prediction (defined Section 2.1 and aim to extend the strong results of generative image models the video domain. Building upon the state-of-the-ang BigGAN architecture (Brock et al.) 2019). we introduce an efficient spatio temporal decomposition of the discriminator which allows to train Kinetics-600 complex dataset of

---

Under review as a conference paper at ICLR 2020
DM5 1g
52% 1g
Fee
Figure 2: Generated video samples with interesting behavior. In raster-scan order: On-screen generated text with further lines appearing.- b) Zooming in on an object. <) Colored detail from pen being left on paper. d) A generated camera change and return.
natural videos an order of magnitude larger than other commonly used datasets, The resulting model, Dual Video Discriminator GAN (DVD-GAN), is able to generate temporally coherent, high-resolution videos of relatively high fidelity (Figure
Our contributions are as follows:
We propose DVD-GAN scalable generative model of natural video which produces high-quality samples at resolutions to 256 x 256 and lengths up to 48 frames.
We achieve state the art for video synthesis on UCF-101 and prediction on Kinetics-600.
We establish cass-cooniitional video synthesis Kinetics-600 generative video modeling, and report DVD-GAN results as a strong baseline.
benchmark for
BACKGROUND
2.1 VIDRO SYNTHESIS AND PREDICTION
The exact formulation of the video generation task can differ in the type of conditioning signal provided. At one extreme lies unconditional video synthesis where the task is to generate any video following the training distribution. Another extreme is occupied by strongly-conditionee models, including generation conditioned on another vidco for content transfer (Bansal et al. 2018 Zhou al. 2019), per-frame segmentation masks (Wang al. 2018a), or pose information (Walker et al. 2017 Villegas et al. 2017b; Yang et al. (018)). In the middle ground there are tasks which are more structured than unconditional generation, and yet are more challenging from modeling perspective than
generation (which gets a lot of information about the generated video through its input). The objective of class-conditioaa video synthexis to generate a video of given category (e.g., "riding bike") while future video prediction is concerned with generation of continuing video given initial frames. These problems differ in several aspects, but share a common requirement of needing to generate realistic temporal dynamics, and in this work we focus these two problems.
2.2 GENERATIVE ADVERSARIAL NETWORKS
Generative Adversarial Networks (GANs) (Gooffellow et al. 2014) class of generative models defined by minimax game between a Discriminator 2 and a Generator G. The original objective was proposed by Goodfellow al. (2014). and many improvements have since been suggested, mostly targeting improved training stability (Arjovsky et al. 2017 Zhang et al.) 2018 Brock et al. Guirajani al. 2017 Miyato et 2018). We use the hinge formulation of the objective (Lim
2019
& Ye 2017 Brock et al.) 2019) which is optimized by gradient descent (p is the elementwise ReLU
function):
D: min z=data(x)
[(x))(x))]
D(x))
E zmp(())
[(L+ p(1 ((
G: max [D(G(=)) 9 z>p(=>
GANs have well-known limitations including a tendency towards limited diversity in generated samples (a phenomenon known mode collapse) and the difficulty of quantitative evaluation due
2

---

Under review as a conference paper at ICLR 2020
to the lack of explicit likelihood measure over the data. Despite these downsides, GANs have produced some of the highest fidelity samples across many visual domains (Karras et al. 2018 Brock
et al. 2019).
2.3 KINETICS-600
Kinetics is large dataset of 10-second high-resolution YouTuhe clips (Kay al. 2017: DeepMind 2018) originally created for the task of human action recognition. We use the second iteration of the dataset, Kinetics-600 (Carreira et al. 2018), which consists of 600 classes with at least 600 videos per class for total of around 500,000 videos TareeossKiiiiiiico Kinetics videos are diverse and unconstrained, which allows us train large models without being concerned with the overfitting that occurs on small datasets with fixed objects interacting in specified ways (Ebert et al. 2017 Blank et al., (005). Among prior work, the closest dataset (in terms of subject and complexity) which is consistently used is UCF-101 (Soomro et al. 2012). We focus Kinetics-600 because of its larger size (almost 50x more videos than UCF- 101) and its increased diversity (600 instead of 101 classes - not to mention increased intra-class diversity). Nevertheless for comparison with prior art we train UCF-101 and achieve a state-of-the-art Inception Score there. Kinetics contains many artifacts expected from YouTube, including cuts (as in Figure I). title screens and visual effects. Except when specifically described, we choose frames with stride (meaning we skip every other frame). This allows us to generate videos with more complexity without incurring higher computational cost.
To the best of our knowledge we are the first to consider generative modelling of the entirety of the Kinetics video datase although a small subset of Kineties consisting of 4,000 selected and stabilized videos (via a SIFT + RANSAC procedure) has been used in at least two prior papers (L) al., 20188 Balaji al. 2018). Due to the heavy pre-processing and stabilization present, well as the sizable reduction in dataset size (two orders of magnitude) we do not consider these datasets comparable the full Kinetics-600 dataset.
2.4 EVALUATION METRICS
Designing metrics for measuring the quality of generative models (GANs in particular) is an active area of research (Sajjadi et al. 2018 |Barratt & Sharma| 2018). In this work we report the two most commonly used metrics, Inception Score (IS) (Salimans et al. (2016) and Fréchet Inception Distance (FID) (Heusel et al.) 2017). The standard instantiation of these metrics is intended for generative image models, and uses an Inception model (Szegedy et al. 2016 for image classification or feature extraction. For videos, on Kinetics-600 (Carreira & Zisserman, 2017). Our Fréchet Inception Distance is therefore very similar the Fréchet Video Distance (FVD) (Unterthiner et al. (018)), although implementation is different and more aligned with the original FII metric More details are in Appendix A.4
use the publicly available Inflated 3D Convnet (I3D) network trained
3 DUAL VIDEO DISCRIMINATOR GAN
Our primary contribution is Dual Video Discriminator GAN of complex human actions built upon the state-of-the-ar BigGAN architecture (Brock et al. 2019) while introducing scalable, video-specific generator and discriminator architectures. An overview of the DVD-GAN architecture is given in Figure and a detailed description is in Appendix A.2 Unlike some of the prior work, our generator contains explicit priors for foreground, background or motion (optical flow); instead, we rely on a high-capacity neural network to learn this in a data-driven manner. While DVD-GAN contains sequential components (RNNs), is not autoregressive in time or in space. In other words, the pixels of each frame do not directly depend on other pixels in the video, as would be the case for augo-regressive models or models generating one frame at a time.
a generative video model
Generating long and high resolution videos is a heavy computational challenge: individual samples from Kinetics-600 (just 10 seconds long) contain upwards of 16 million pixels which need to be
Kinetics is occasionally pruned and so cannot give exact size.
In paraliel with the concurrent work Weissenborn et al. (011
We use 'avgpool' features (rather than logits) by default, our 13D model is trained on Kinetics-600 (rather than Kinetics-400), and we pre-calculate ground-truth statistics on the entire training set.
3

---

Under review as a conference paper at ICLR 2020
Generator
Disciminnatora
Frane
9 ResNet Block
BN
ResNet
Block
f
g ResNet Block
Death Timas
BN
0000
g ResNet Block
BN
ResNet
Block
Franec
XCepst
B
Class Conditional Batch
Feature Concstenation
Standard Gaussian Noise
ReLU Activation
Convolution with Kernel Size
Convolutional Gated Recurrent Unit
Bilinear x2 Upsampling
Average Pool x2 Downsampling
One-Hat Class Indicator
Figure 3: Simplified architecture diagram of (left) and Dz/Dr (right), More details A.2
generated in consistent fashion. This is particular challenge the discriminator, For example, generated video might contain an object which leaves the field of view and incorrectly returns with a different color. Here, the ability determine this video is generated only possible by comparing two different spatial locations across two (potentially distant) frames. Given video with length T, height H, and width W, diseriminators that process the entire video would have to process all H W x T pixels - limiting the size of the model and the size of the videos being generated.
3.1 DUAL DISCRIMINATORS
DVD-GAN tackles this scale problem by using two discriminators: Spatial Discriminator Ds and Temporal Discriminator Dr- Ds critiques single frame content and structure by randomly sampling k full-resolution frames and judging them individually. We use h 8 and discuss this choice in Section 4.3 Ds's final score is the sum of the per-frame scores. The temporal discriminator Dr must provide G with the learning signal to generate movement (something not evaluated by Ds). To make the model scalable, we apply a spatial downsampling function 6(-) to the whole video and feed its output Dr. We choose Φ be 2 average pooling, and discuss alternatives Section 4.3 This results in architecture where the discriminators do not process the entire video's 's worth of pixels, since Ds processes only H W pixels and DD only T 2 2. For 48 frame video at 128 x 128 resolution, this reduces the number of pixels to process per video from 786432 to 327680: a 58% reduction. Despite this decomposition, the discriminator objective still able to penalize almost all inconsistencies which would be penalized by discriminator judging the entire video. Dr judges any temporal discrepancies across the entire length of the video, and Dg can judge any high resolution details. The only detail the DVD-GAN discriminator objective is unable reflect is the temporal evolution of pixels within a 2 x 2 window. We have however not noticed this affecting the generated samples in practice. DVD-GAN's D5 similar the per-frame discriminator D, in MoCoGAN (Tulyakov et al., 2018). However MoCoGAN's analog of Dr looks at full resolution videos, whereas Ds is the only source learning signal for high-resolution details in DVD-GAN. For this reason, Dş is essential when @ is not the identity, unlike in MoCoGAN where the additional per-frame discriminator is less crucial.
3.2 RELATED WORK
Generative video modeling is a widely explored problem which includes work on VABs (Babaeizadeh
et al.
2018 Denton & Fergus 2018 Lee et al.) 2018 Hsieh et al. (2018) and recurrent models (Wang
et al.
2018b Finn et al. 2016 Wang et al. 2018c Byeon et al. 2018). auto-regressive models RRan
zato et al.
2014 Srivastava
al. 2015 Kalchbrenner et al. 2017] Weissenborn
al.] 2019).
normalizing flows (Kumar et al. 2019), and GANs (Mathieu et al. 2015; Vondrick et al. 2016 Saito
al.
2017
Saito & Saito 2018). Much prior work considers decompositions which model the
4

---

Under review as a conference paper at ICLR 2020
Figure 4: Each row is the first frame of 15 videos from a random class, all from the same check- point. The classes are: cooking scallops, changing wheel (not on bike), calculating, dribbling basketball.
Figure 5: All 48 frames (in raster-scan order) from 64 x 64 sample from watermelon cutting class.
texture and spatial consistency of objects separately from their temporal dynamics. One approach is to split & into foreground and background models (Vondrick et al. 2016 Spampinato et al. 2018). 18)
while another considers explicit or implicit optical flow motion either or D (Saito et al. [0177]
Ohnishi et al.) 2018). Other methods decompose the generator (or encoder) to treat concepts like pose, content and motion separately from one another (Denton et al., 2017, Villegas et al., (217 Similar to DVD-GAN, MoCoGAN (Tulyakov et al., 2018) discriminates individual frames in addition to discriminator which operates on fixed-length -ffamee slices of the whole video (where
< T). Though this potentially reduces the number of pixels to discriminate to (H W) + (K x H x W), Tulyakov et al. (2018) describes discriminating sliding windows, which increases the total number of
pixels. Other models follow this approach by discriminating groups of frames (Xie et al. 2018 Sun
et al.
2018 Balaji et al. 2018).
TGANv2 Saito & Saito 2018) proposes "adaptive batch reduction" for efficient training, operation which randomly samples subsets of videos within a batch and temporal subwindows within each video. This operation applied throughout TGANv2's G, with heads projecting intermediate feature maps directly to pixel space before applying batch reduction, and corresponding discriminators evaluating these lower resolution intermediate outputs. An effect of this choice is that TGANx2 discriminators only evaluate full-length videos at very low resolution. We show in Figure 6 that similar reduction in DVD-GAN's resolution when judging full videos leads to loss in performance. We expect further reduction (towards the resolution at which TGANv2 evaluates the entire length of video) lead to further degradation of DVD-GAN's quality. Furthermore, this method is not easily adapted towards models with large batch sizes divided across number of accelerators, with only small batch size per replica.
EXPERIMENTS AND ANALYSIS
A detailed description of our training setup is in Appendix A.3 Each DVD-GAN was trained on TPU pods (Google, 2018) using between 32 and 512 replicas with an Adam (Kigmma & Ba 2014) optimizer. Video Synthesis models are trained for around 300,000 learning steps. whilst Video Prediction models are trained for up to 1,000,000 steps. Most models took between 12 and 96 hours to train.
5

---

Under review as a conference paper at ICLR 2020
4.1 VIDEO SYNTHESIS
Our primary results concern the problem of Video Synthesis. We provide our results for the UCF-101 and Kinetics-600 datasets. With Kinetics-600 emerging as a new benchmark for generative video modelling, our results establish strong baseline for future work.
4.1.1 KINETICS-600 RESULTS
Table I: FID/IS for DVD-GAN Kinetics-600 Video Synthesis. We present the scores of the model taken at the point in training when the best FID was attained. The "No Truncation" columns contain the scores obtained without the truncation trick. The "With Truncation' columns contain the scores obtained the truncation level which results in the best Inception Score.
(# Frames Resolution)
No Truncation
With Truncation
FID (4)
IS (t)
FID (4)
IS (f)
12/64 x 64
0.85
53.81
7.13
187.23
12/128/ x 128
1.16
77.45
13.04
246.18
12 2/2266 256
2.05
62.78
10.17
162.44
48/ /64 x 64
13.75
104.09
47.86
264.12
48/128 128
28.44
81.41
45.79
188.32
In Table we show the main result of this paper: benchmarks for Class-Conditional Video Synthesis on Kinetics-600 In this regime, we train single DVD-GAN on all classes of Kinetics-600, supplying per-sample class information to both G and D. We consider a range of resolutions and video lengths, and measure Inception Score and Fréchet Inception Distance (FID) for each (as described in Section 2.4). We further measure each model along a truncation curve, which we carry out by calculating FID and IS statistics while varying the standard deviation of the latent vectors between and 1. There is no prior work with which to quantitatively compare these results (for comparative experiments see Section 4.1.2 and Section 4.2.T). p. but we believe these samples show level of fidelity not yet achieved in datasets complex as Kinetics- 600 (see samples from each row in Appendix D.Ip. D.Ip.. Because all videos are resized for the I3D network (to 224 224), it is meaningful to compare metrics across equal length videos at different resolutions. Neither IS nor FIDD are comparable across videos of different lengths, and should be treated as separate metrics.
Generating longer and larger videos is more challenging modeling problem, which is conveyed by the metries (in particular, comparing 12-frame videos across 64 x 64, 128 x 128 and 256 x 256 resolutions). Nevertheless, DVD-GAN is able to generate plausible videos at all resolutions and with length spanning up to seconds (48 frames). As can be seen in Appendix D.I smaller videos display high quality textures, object composition and movement. At higher resolutions, generating coherent objects becomes more difficult (movement consists of a much larger number of pixels), but high-level details of the generated scenes still extremely coherent, and textures (even complicated ones like a forest backdrop in Figure are generated well. It is further worth noting that the 48-frame models do not see more high resolution frames than the 12-frame model (due the fixed choice of k described in Section 3.T), yet nevertheless learn to generate high resolution images.
4.1.2 VIDEO SYNTHESIS ON UCF-
We further verify our results by testing the same model on UCF-101 (Soomro et al. 2012) smaller dataset 13,320 videos human actions across 101 classes that has previously been used for video synthesis and prediction (Saito et al.] 2017: Saito & Saito 2018 Tulyakov al. 2018). Our model produces samples with an IS of 27.38, significantly outperforming the state of the art (see Table for quantitative comparison and Appendix B. for more details).

---

Under review as a conference paper at ICLR 2020
Table 2: IS on UCF-101 without class conditioning.
Table 3: FVD on BAIR.
Method
IS (t)
Method
FVD (4)
VGAN (Vondrick al.
2016)
8.31 .09
SVP-FP
315.5
TGAN (Saito et al. 2017)
11.85 + .07
CDNA
296.5
MoCoGAN (Tulykkov et
2018
12.42 = .03
SV2P
262.5
ProgressiveVGAN (Acharya al. 2018
14.56 = .05
SAVP
116.4
TGANv2 (Saito Sarto 2018)
24.34 = .35
DVD-GAN-IP (ours)
109.8
DVD-GAN (ours)
27.38 = 0.53
Video Transformer
94
Table 4: DVD-GAN-FP's FVD scores on Video Prediction for frames of Kinetics-600 without frame skipping. The final row represents a Video Synthesis model generating 16 frames.
Method
Training Set FVD (L)
Test Set FVD ())
Video Transformer Weissenborn et al. 2019
170 5
DVD-GAN-FP
68.66
0.78
69.15 + 1.16
DVD-GAN
32.3 + 0.82
31.1 + 0.56
4.2 FUTURE VIDEO PREDICTION
Future Video Prediction is the problem of generating a sequence of frames which directly follow from one (or number) of initial conditioning frames. Both this and video synthesis require G to learn to produce realistic scenes and temporal dynamics, however video prediction further requires G to analyze the conditioning frames and discover elements the scene which will evolve over time. In this section, of an I3D network trained on Kinetics-400 as features. This allows for direct comparison to prior work. Our model, DVD-GAN-FP (Frame Prediction), is slightly modified facilitate the changed problem, and details of these changes are given in Appendix A.5
the Fréchet Video Distance exactly Unterthiner et al. (2018): using the logits
4.2.1 FRAME-CONDITIONAL KINETICS
For direct comparison with concurrent work autoregressive video models (Weissenborn et al., 2019) we consider the generation of 11 frames of Kinetics-600 at 64 x 64 resolution conditioned on 5 frames, where the videos for training not taken with any frame skipping. We show results for all these cases in Table 4 Our frame-conditional model DVD-GAN-FP outperforms the prior work on frame-conditional prediction for Kinetics. The final row labeled DVD-GAN corresponds to 16-frame cass-conditional Video Synthesis samples, generated without frame conditioning and without frame skipping. The FVD of this video synthesis model is notably better.
On the one hand, we hypothesize that the synthesis model has an easier generative task: it can choose to generate (relatively) simple samples for each class, rather than be forced to continue frames taken from videos which are class outliers, or contain more complicated details. On the other hand, certain portion of the FID/FVD metric undoubtedly comes from the distribution of objects and backgrounds present in the dataset, and so it seems that the prediction model should have handicap in the metric by being given the ground truth distribution of backgrounds and objects with which to continue videos. The synthesis model's 's improved performance on this task seems to indicate that the advantage of being able to select videos to generate is greater than the advantage having a ground truth distribution of starting frames. This result is un-intuitive, as the frame conditional model has access to strictly more information about the data distribution it is trying to recover compared to the synthesis model (despite the fact that the two models being trained by identical objective). This experiment favors the synthesis model for FVD, but we highlight that other models or other metrics might produce the opposite ordering.
7

---

Under review as a conference paper at ICLR 2020
45
45
43.0
37
40
40
39.5
89.0
3.0
35
37.5
34,2
3.0
35
2.5
2.5
30
30
2.0
0.0
2.0
2.0
26.4
1.5
15.
0.25
0.5
1.0
Crop
0.25
0.5
10
Crop
k
10 k-1
k-4
k-
k-
Figure 6: The effect of Φ in Dy (left two) and & in Dy (right two). FID similar for any choice of ø. while IS declines downsampling increases. Increasing k improves both with diminishing returns.
4.2.2 BAIR ROBOT PUSHING
We further test future video prediction on the single-class BAIR Robot Pushing Dataset (Ebert et al. 2017). dataset of stationary videos of a robot arm moving around a set of changing objects. In order for direct comparison with previous results reported in Unterthiner et al. (2018), we consider generating 15 frames conditioned on a single starting frame. Like on prediction with Kineties, we report FVD exactly in Unterthiner et al. (2018), with ground truth statistics and conditioning frames taken from the 256-video dev set. Results are reported in Table 3 Scores are taken from Unterthiner al. (2018).
outperforms all prior adversarial models trained on this dataset, but performs slightly worse than Video Transformer, concurrently developed autoregressive model Weissenborn et al. 2019). Samples from DVD-GAN-FP on BAIR given in Figure
4.3 DUAL DISCRIMINATOR INPUT
We analyze several choices for & (the number of frames per sample in the input to Dg) and (the downsampling function for Dr). We expect setting the identity or k T result in the best model, but we are interested in the maximally compressive A: and á that reduce discriminator input size (and the amount of computation), while still producing a high quality generator. For d we consider: x 2 and 4 x 4 average pooling, the identity (no downsampling). takes random half-sized crop the input video (as in Saito & Saito (2018)). Results can be seen in Figure For each ablation, on 12-frame clips of Kinetics-60M at 64 X 64 resolution for 100,000 steps. We report mean and standard deviation (via the
well as a which
train three identical DVD-GANs with different random initializations
bars) across each group for the whole training period. For k, consider 1, 2, 8 and 10 frames. We see diminishing effect as increases, so settle on k = 8. We note the substantially reduced IS of 4 downsampling opposed to 2 2, and further note that taking half-sized crops (which results in the same number of pixels input to Dr as 2 x pooling) is also notably worse.
CONCLUSION
We approached the challenging problem of modeling natural video by introducing a GAN capable of capturing the complexity of large video dataset. We showed that UCF-101 and frame-conditional Kinetics-60X quantitatively achieves the new state of the art, alongside qualitatively producing video synthesis samples with high complexity and diversity. We further wish to emphasize the benefit of training generative models on large and complex video datasets, such as Kinetics-600, and envisage the strong baselines we established on this dataset with DVD-GAN will be used as reference point by the generative modeling community moving forward. While much remains to be done before realistic vidcos be consistently generated in an unconstrained setting, we believe DVD-GAN is step in that direction.
REFERENCES
Dinesh Acharya, Zhiwu Huang, Danda Pani Paudel, and Luc Van Gool. Towards high resolution video generation with progressive growing of sliced Wasserstein GANs. arXiv:1810.02449, 2018.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein GAN, arXiv: /701 00000/0075.000 2017.
8

---

Under review as a conference paper at ICLR 2020
Mohammad Babaeizadeh, Chelsea Finn, Dumitru Erhan, Roy H Campbell, and Sergey Levine Stochastic variational video prediction. ICLR, 2018.
Yogesh Balaji, Martin Renqiang Min, Bing Bai, Rama Chellappa, and Hans Peter Graf. Tfgan: Improving conditioning for text-to-video synthesis. 2018.
Nicolas Ballas, Li Yao, Chris Pal, and Aaron Courville. Delving deeper into convolutional networks for learning video representations. 0rXiv:1511.064322 2015.
Aayush Bansal, Shugao Ma, Deva Ramanan, and Yaser Sheikh. Recycle-GAN: Unsupervised video retargeting. In ECCV, September 2018.
Shane Barratt and Rishi Sharma. A note on the Inception Score. arXiv:1801.01973. 2018.
Moshe Blank, Lena Gorelick, Eli Shechtman, Michal Irani, and Ronen Basri. Actions as space-time shapes. In ICCV, 2005.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. ICIR, 2019.
Wonmin Byeon, Qin Wang, Rupesh Kumar Srivastava, and Petros Koumoutsakos. Contextvp: Fully context-aware video prediction. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 753-769, 2018.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? A new model and the Kinetics dataset. In ICCV, 2017.
Joao Carreira, Eric Noland, Andras Banki-Horvath, Chloe Hillier, and Andrew Zisserman. A short note about Kinetics-600. 1rXiv:/808.01344 2018.
Harm De Vries, Florian Strub, Jérémie Mary, Hugo Larochelle, Olivier Pietquin, and Aaron C Courville. Modulating early visual processing by language. In NeurIPS, 2017.
DeepMind, 2018. Kinetics.
wwwp:reeearchh/penn-ooooo source/ open-souree source dwwasstss/kietiiieeeot Accessed: 2019.
https:// /ewp:/in/eeeidddddeemmm
Emily Denton and Rob Fergus. Stochastic video generation with learned prior. ICML, 2018.
Emily L Denton et al. Unsupervised learning of disentangled representations from video. In Advances in neural information processing systems, pp. pp. 4414-4423, 2017.
Vincent Dumoulin, Jonathon Shlens, and Manjunath Kudlur. A learned representation for artistic style. In ICLR, 2017.
Frederik Ebert, Chelsea Finn, Alex X Lee, and Sergey Levine. Self-supervised visual planning with temporal skip connections. arXiv:1710.05266 2017.
Chelsea Finn, Ian Goodfellow, and Sergey Levine. Unsupervised learning for physical interaction through video prediction. In Advances in neural information processing systems, pp. 64-72, 2016.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurIPS, 2014.
Google, 2018. Cloud tpu. https: http:://clooddggogll..c////s///p
Accessed: 2019.
Google, 2019. Fvd. http:p///ggtthhb..o//googlleeee tttee/mastee/fxreceet video distance, Accessed: 2019.
rcc/ wttpeaarrggggggooeaeeeooaoooeeee google-
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Improved training of Wasserstein GANs. In NewrlPS, 2017.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by two time-scale update rule converge to a local nash equilibrium. In NeurIPS, 2017.

---

Under review as a conference paper at ICLR 2020
Jun-Ting Hsieh, Bingbin Liu, De-An Huang, Li Fei-Fei, and Juan Carlos Niebles. Learning to decompose and disentangle representations for video prediction. In NewrlPS. 2018.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXivv/502.03166) 2015.
Nal Kalchbrenner, Aäron van den Oord, Karen Simonyan, Ivo Danihelka, Oriol Vinyals, Alex Graves, and Koray Kavukcuoglu. Video pixel networks. In ICML, 2017.
Tero Karras, Samuli Laine, and Timo Aila, style-based generator architecture for generative adversarial networks. /8/22.///2..0 04948, 2018.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman. The Kinetics human action video dataset. arXiv:1705.06950 2017.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. UrXiv:1412.6900 2014.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions, In NeurIPS, 2018.
Manoj Kumar, Mohammad Babaeizadeh, Dumitru Erhan, Chelsea Finn, Sergey Levine, Laurent Dinh, and Durk Kingma. VideoFlow: A flow-based generative model for video. arXiv:1903.01444 2019.
Alex X Lee, Richard Zhang, Frederik Ebert, Pieter Abbeel, Chelsea Finn, and Sergey Levine. Stochastic adversarial video prediction. arXiv:1804.01523, 2018.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXivv1607.06450. 2016.
Yitong Li, Martin Renqiang Min, Dinghan Shen, David Carlson, and Lawrence Carin. Video generation from text. In Shiroy-Second AAAI Conference on Artificial Intelligence, 2018.
Jae Hyun Lim and Jong Chul Ye. Geometric GAN. arXiv: 2017.
Michael Mathieu, Camille Couprie, and Yann LeCun. Deep multi-scale video prediction beyond mean square error, 1rXiv:15/1.05440) 2015.
Jacob Menick and Nal Kalchbrenner. Generating high fidelity images with subscale pixel networks and multidimensional upscaling. In ICLR, 2019.
Takeru Miyato and Masanori Koyama. cGANs with projection discriminator. arXiv:1802.0500 2018.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXi:: 2018.
Katsunori Ohnishi, Shohei Yamamoto, Yoshitaka Ushiku, and Tatsuya Harada. Hierarchical video generation from orthogonal information: Optical flow and texture. In AAAI, 2018
Marc' Aurelio Ranzato, Arthur Szlam, Joan Bruna, Michaël Mathicu, Ronan Collobert, and Sumit Chopra. Video (language) modeling: baseline for generative models of natural videos. arXiv: 2014.
Ali Razavi, Aaron den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with VQ-VAE-2. arXiv preprint (006.00990000444 2019.
Masaki Saito and Shunta Saito. TGANv2: Efficient training of large models for video generation with multiple subsampling layers. arXiv: /8/ 1.09245, 2018.
Masaki Saito, Eiichi Matsumoto, and Shunta Saito. Temporal generative adversarial nets with singular value clipping. In ICCV, 2017.
10

---

Under review as a conference paper at ICLR 2020
Mchdi SM Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly. Assessing generative models via precision and recall. In NewrIPS, 2018.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training GANs. NeurIPS, 2016.
Andrew M Saxe, James L McClelland, and Surya Ganguli, Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv: 1312.6120, 2013.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. UCF101: A dataset of 101 human actions classes from videos in the wild. ar.iv: 22220022 2012.
Concetto Spampinato, Sergio Palazzo, P D' D'Oro, Francesca Murabito, Daniela Giordano, and M Shah. VOS-GAN: Adversarial learning of visual-temporal dynamics for unsupervised dense prediction in videos. arXiv:1803.0900 2018.
Nitish Srivastava, Elman Mansimov, and Ruslan Salakhutdinov. Unsupervised learning of video representations using LSTMs. In ICML, 2015.
Ximeng Sun, Huijuan Xu, and Kate Saenko. A two-stream variational adversarial network for video generation. arXiv:/812.01033 2018.
Ilya Sutskever, James Martens, and Geoffrey E Hinton. Generating text with recurrent neural networks. In ICML, 2011.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jonathon Shlens, and Zbigniew Wojna. Re- thinking the Inception architecture for computer vision. CVPR, 2016.
Seiya Tokui, Kenta Oono, Shohei Hido, and Justin Clayton. Chainer: next-generation open source framework for deep learning. In Workshop on Systems for ML and Open Source Software at NeurlPS, 2015.
Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spa- tiotemporal features with 3D convolutional networks. In ICCV, 2015.
Sergey Tulyakov, Ming- Yu Liu, Xiaodong Yang, and Jan Kautz. MoCoGAN: Decomposing motion and content for video generation. In CVPR, 2018.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing ingredient for fast stylization. /607.088 2016.
Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski. and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. 1812v:1812.0011 2018.
Ruben Villegas, Jimei Yang, Seunghoon Hong, Xunyu Lin, and Honglak Lee. Decomposing motion and content for natural video sequence prediction. arXiv preprint arXiv:1706.0800 2a.
Ruben Villegas, Jimei Yang, Yuliang Zou, Sungryull Sohn, Xunyu Lin, and Honglak Lee. Learning to generate long-term future via hierarchical prediction. In ICMI., 2017b.
Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dynamics. In NeurlPS, 2016.
Jacob Walker, Kenneth Marino, Abhinav Gupta, and Martial Hebert. The pose knows: Video forecasting by generating pose futures. In ICCV, 2017.
Ting-Chun Wang, Ming-Yu Yu Liu, Jun-Yan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. Video-to-video synthesis. arXiv: 066000 2018a
Yunbo Wang, Zhifeng Gao, Mingsheng Long, Jianmin Wang, and Philip S Yu. Predrnn++: Towards a resolution the deep-in-time dilemma in spatiotemporal predictive learning. arXiv preprint arXiv: 1804.8044.0000 06300, 2018b
11

---

Under review as a conference paper at ICLR 2020
9 ResNet Block
D ResNet Block
BN
Figure 7: The residual blocks for and Ds/Dr, See Figure 3 3for the icons and A.2 for more detail.
Yunbo Wang, Lu Jiang, Ming- Yang, Li-Jia Li, Mingsheng Long, and Li Fei-Fei. Eidetic 3d 1stm: A model for video prediction and beyond. 2018c.
Dirk Weissenborn, Oscar Täckström, and Jakob Uszkoreit. Scaling autoregressive video models. arXiv:1906.02634. 2019.
Yuxin Wu and Kaiming He. Group normalization. ECCV, 2018.
You Xie, Erik Franz, Mengyu Chu, and Nils Thuerey. tempoGAN: A temporally coherent, volumetric GAN for super-resolution fluid flow. TOG, 2018
Ceyuan Yang, Zhe Wang, Xinge Zhu, Chen Huang, Jianping Shi, and Dahua Lin. Pose guided human video generation. ECCV, 2018.
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. vrXiv:1805008316 2018.
Yipin Zhou, Zhaowen Wang, Chen Fang, Trung Bui, and Tamara L Berg. Dance dance generation: Motion transfer for internet videos.
2019.
A EXPERIMENT METHODOLOGY
A.1 DATASET PROCESSING
For all datasets we randomly shuffle the training set for each model replica independently. Experi- ments the BAIR Robot Pushing dataset are conducted in the native resolution of 64 x 64, where UCF-101 we operate at (downsampled) 128 128 resolution. This is done by a bilinear resize such that the vidco's 's smallest dimension is mapped 128 pixels while maintaining aspect ratio (144 for UOFF-01). From this we take a random 128-pixel crop along the other dimension. We use the same procedure construct datasets of different resolutions for Kinetics-600. All three datasets contain videos with more frames than we generate, so we take a random sequence of consecutive frames from the resized output. For UCF-101, augmented the dataset by randomly performing left-right flips with probability 0.5.
A.2 ARCHITECTURE DESCRIPTION
Our model adopts many architectural choices from Brock et al. (2019) including our nomenclature for describing network width, which determined by the product of channel multiplier ch with constant for each layer in the network. The layer-wise constants for G are [8, 8, 8, 4, 2) for 64 x 64 for 128 128. The width of the i-th layer is given by the product of ch
videos and [8, and the i-th constant and all layers prior the residual network in G use the initial layer's 's multiplier and we refer to the product of that and ch as chg- eh in DVD-GAN is 128 for videos with 64 64 resolution and 96 otherwise. The corresponding ch lists for both Dr and D3 are [2, 4,8, 16, 16) for 64 x 64 resolution and [1, 2, 4, 8, 16, 16] for 128 X 128.
The input G consists of Gaussian latent noise (O, I) and learned linear embedding e(y) of the desired class y. Both inputs are -20-dimensional vectors. 9 starts by computing an affine transformation of [2; e(y) to a [4, 4, chuj -hhpped tensor (in Figure 3 this is represented
1 x convolution). [2; e(y) is used as the input to all condiionnal Batch Normalization layers throughout (the gray line in Figure
12

---

Under review as a conference paper at ICLR 2020
This is then treated as the input (at each frame would like to generate) to a Convolutional Gated Recurrent Unit (Ballas et al.] 2015 SSuskkeer et al. (011) whose update rule for input 2, and previous output h is given by the tollowing:
oW,, *3 [he-1; + by)
u = a (Wu ky [ + bu)
p(W. *3 [ 3 1 + be)
h2 = ② 4 (1 u1 0
u)
In these equations and p are the elementwise sigmoid and ReLU functions respectively, the * operator represents a convolution with a kernel of size n X n,, and the @ operator an elementwise used to represent feature concatenation. This RNN is unrolled once per
multiplication. Brackets frame. The output of this RNN is processed by two residual blocks (whose architecture is given by Figure 7.. The time dimension is combined with the batch dimension here, each frame proceeds through the blocks independently. The output of these blocks has width and height dimensions which doubled (we skip upsampling in the first block). This is repeated number of times, with the output of one RNN + residual group fed as the input to the next group, until the output tensors have the desired spatial dimensions. We do not reduce over the time dimension when calculating Batch Normalization statistics. This prevents the network from utilizing the Batch Normalization layers to pass information between timesteps.
The spatial discriminator Dş functions almost identically to BigGAN's discriminator, though an overview of the residual blocks given in Figure 7 for completeness. A score is calculated for each of the uniformly sampled & frames (we default to k 8) and the DDs output is the sum over per-frame scores. The temporal discriminator Dr has a similar architecture, but pre-processes the real or generated video with a 2 x 2 average-pooling downsampling function d. Furthermore, the first two residual blocks of Dr are 3-D, where every convolution is replaced with a 3-D convolution with kernel size of 3 x 3 x 3. The rest of the architecture follows BigGAN (Brock et al.) 2019).
A.3 TRAINING DETAILS
Sampling from DVD-GAN very efficient, as the core of the generator architecture is feed-forward convolutional network: two 64 x 64 48-frame videos can be sampled in less than 150ms single TPU core. The dual discriminator is updated twice for every update of G (Heusel et al.) 2017) and use Spectral Normalization (Zhang al. 2018) for all weight layers (approximated by the first singular value) and orthogonal initialization of weights (Saxe et al., (013). Sampling is carried out using the exponential moving average of G's weights, which is accumulated with decay 7 0.9999 starting after 20,000 training steps. The model is optimized using Adam (Kingma & Bal 2014) with batch size 512 and a learning rate of 1-104 and 5 10-4 for G and D respectively. Class conditioning in 'D (Miyato& & Koyama 2018) is projection- projection-baaed whereas G relies on Ilas-conditional Batch Normalization (loffe& & Szegedy 2015: De Vries al.) Dumoulin al. (007)): equivalent to standard Batch Normalization without a learned scale and offset, followed by an elementwise affine transformation where each parameter a function of the noise vector and class conditioning.
A.4 FID FOR KINETICS-600 SYNTIESIS
The FID we use for Synthesis on Kinetics-600 is calculated exactly as Fréchet Video Dis- tance (Unterthiner et al. 2018) except that we use a different feature network: an I3D trained on Kinetics-600 (as opposed to the network trained on Kinetics-400 in FVD) and features from the final hidden layer instead of the logits. This metric can be implemented as a small change from the publically available FVD code (Google, 2019) by changing the name of the hetp://ww.deeeeeeeeeennidddddd kinetissoo// and loading the tensor named
TE-Hub module to "RGG/inneptionnnidlllaae [3d//Logiis/AvgPool33 from the resulting graph.
A.5 ARCHITECTURE EXTENSION TO VIDEO PREDICTION
In order to provide results on future video prediction problems we describe a simple modification to DVD-GAN to facilitate the added conditioning. A diagram of the extended model is in Figure g
13

---

Under review as a conference paper at ICLR 2020
Gourrator
Diseriminaters
Piavel
Uve Ioarse
ResNet Block
BN
ResNet Block
BN
G0B
Block
100
6
Reshitt Block
BN
Block
Frereo
Conditoning Stack
FarFrove
Block
Figure 8: An architecture diagram describing the changes for the frame conditional model.
Three video samples from a prediction model trained on the BAIR robot pushing dataset. Each row is a separate video, the leftmost column is a (true) conditioning frame.
Figure 9:
Given conditioning frames, our modified DVD-GAN-FP passes each frame separately through a deep residual network identical to Ds. The (near) symmetric design of G and Dz's 's residual blocks mean that each output from a D-style residual block has a corresponding intermediate tensor in G of the same spatial resolution. After each block the resulting features for each conditioning frame are stacked in the channel dimension and passed through 3 x 3 convolution and ReLU activation. The resulting tensor used the initial state for the Convolutional GRU in the corresponding block in G. Note that the frame conditioning stack reduces spatial resolution while G increases resolution. Therefore the smallest features of the conditioning frames (which have been through the most layers) are input earliest in G and the larger features (which have been through less processing) are input to G towards the end. Dr operates on the concatenation the conditioning frames and the output of G. meaning that it does not receive any extra information detailing that the first C' frames special. However to reduce wasted computation we do not sample the first C frames for Ds on real generated data. This technically means that D3 will never see the first few frames from real videos at full resolution, but this was not an issue in our experiments. Finally, our video prediction variant does not condition any class information, allowing to directly compare with prior art. This is achieved by settling the class id of all samples to O.
B FURTHER EXPERIMENTS
B.1
UCF-101
UCF-101 (Soomro et al. 2012) is dataset of 13,320 videos of human actions across 101 classes that has previously been used for video synthesis and prediction (Saito et al.) 2017, Saito & Saito 2018 Tulyakov al. 2018). In this case, DVD-GAN is not conditioned our results comparable with prior work. This achieved by setting the class labels of all input samples to O. We report Inception Score (IS) calculated with C3D network (Tran al. 2015)
class labels to make
14

---

Under review as a conference paper at ICLR 2020
Figure 10: The first frames of interpolations between UCF-101 samples. Each row is a separate interpolation. Contrast with samples in Appendix D.2
for quantitative comparison with prior work This evaluation is performed by re-scaling the video to 128 x 128, normalizing the input features based on mean statistics of the ground truth dataset, then taking a 112 112 center crop and applying C3D. Our model produces samples with an IS of 27.38, significantly outperforming the state of the art (see Table The DVD-GAN architecture on UCF-101 is identical to the model used for Kinetics, and is trained on 16-frame 128 x 128 clips from UCF-101.
The lack of class information does hurt the performance of DVD-GAN, and training on UCF-101 with class labels leads to an improved model with an Inception Score of 32.97. This is directly comparable to Conditional TGAN Saito al. (0077) which achieved an IS of 15.83 and is close to the IS reported for the ground truth data 34.49). However we note than many more recent video generation papers do not test in this regime. is worth mentioning that our improved score is, at least partially, due to memorization of the training data. In Figure 10 we show interpolation samples from our best UCF-101 model. Like interpolations in Appendix D.2] we sample 2 latents (left and rightmost columns) and show samples from the linear interpolation in latent space along each row. Here we show 4 such interpolations (the first frame from each video). Unlike Kinetics-60M interpolations, which smoothly transition from one sample to the other, see abrupt jumps in the latent space between highly distinct samples, and little intra-video diversity between samples each group. It can be further seen that some generated samples highly correlate with samples from the training set.
We show this both as a failure of the Inception Score metric, the commonly reported value for class- conditional video synthesis on UCF-101, but also as strong signal that UCF-101 is not a complex or diverse enough dataset to facilitate interesting video generation. Each class is relatively small, and reuse of clips from shared underlying videos means that the intra-class diversity can be restricted to just handful of videos per class. This suggests the need for larger, more diverse and challenging datasets for generative video modelling, and we believe that Kinetics-600 provides a better benchmark for this task.
C MISCELLANEOUS EXPERIMENTS
Here we detail a number of modifications or miscellaneous results we experimented with which did not produce conclusive result.
We experimented with several variations of normalization which do not require calculating statistics
batch of data. Group Normalization (Wu & He 2018)) performed best, almost on a par with (but worse than) Batch Normalization. We further tried Layer Normalization (Lei Ba al. 2016). Instance Normalization (Ulyanov al. 2016). and normalization, but found that these significantly underperformed Batch Normalization.
We found that removing the final Batch Normalization in G, which occurs after the ResNet and before the final convolution, caused a catastrophic failure in learning. Interestingly, just
'We use the Chainer Tooui et al. 2015 implementation of Inception Score for C3D available https//gghuu..co//hhhtssssssaaa
15

---

Under review as a conference paper at ICLR 2020
removing the Batch Normalization layers within G's residual blocks still led good (though slightly worse) generative models. In particular, variants without Batch Normalization in the residual blocks often achieve significantly higher IS (up to 110.05 for 64 x 64 12 frame samples twice normal). But these models had substantially worse FID scores (1.22 for the aforementioned model) and produced qualitatively worse video samples.
Early variants of DVDDGGAN contained Batch Normalization which normalized over all frames of all batch elements. This gave G an extra channel to convey information across time. It took advantage of this, with the result being model which required batch statistics in order to produce good samples. We found that the version which normalizes over timesteps independently worked just as well and without the dependence on statistics.
Models based on the residual blocks of BigGAA--dep trained faster (in wall clock time) but slower with regards to metries, and struggled to reach the accuracy of models based on BigGAN's residual blocks.
D GENERATED SAMPLES
is difficult to accurately convey complicated generated video through still frames. Where provided, recommend readers view the generated videos themselves via the provided links. We refer to videos within these batches by row/column number where the video the Oth row and column is in the top left corner.
16

---

Under review as a conference paper at ICLR 2020
D.1 SYNTHESIS SAMPLES
C
Figure 11: The first frames from random batch of samples from DVD-GAN trained on 12 frames of 64 x 64 Kinetics-600. Full samples at http::::///diie.ggool..... wwwpoooll..ooff////// com/file/d/
17

---

Under review as a conference paper at ICLR 2020
Figure 12: The first frames from a random batch of samples from DVD-GAN trained on 48 frames of 64 64 Kinetics-600. Full samples at / www.coooffileeeee
18

---

Under review as a conference paper at ICLR 2020
Figure 13: The first frames from a random batch of samples from DVD-GAN trained on 12 frames of 128 x 128 Kinetics-600. Full samples at http:///ddvive.gggllll 4/165Yxuvv..vvooy-333 https:/ssshhhoooccooooooosssssra sharing
file/
19

---

Under review as a conference paper at ICLR 2020
Figure 14: The first frames from a random batch of samples from DVD-GAN trained on 48 frames of 128 128 Kinetics-600 Full samples at /tttts:://driieii www.oo....o.....l.....o..
cww/oooleeeeee wttpl/////iiietwssssaaaaaaaaaaaa The sample in row
d/1P8SsWEGP EGP 1, column 5 is stereotypical example of degenerate sample occasionally produced by DVD-GAN.
20

---

Under review as a conference paper at ICLR 2020
GaV' OVT NO PO NUS FRL FED
Figure 15: The first frames from a random batch of samples from DVD-GAN trained on 12 frames of 256 x 256 Kinetics-600. Full samples at /https://ddrvve wttp:://o/l.ii.....glll... wwwggl..coo//..... file/
d/ a/2erwewpeessso00ccc.eee/////rpp
21

---

Under review as a conference paper at ICLR 2020
Figure 16: The first frames from random batch of samples from DVD-GAN
UCF-101. Full samples at htttps://driee. http::////e..eee.e.lllll.eo www.ggoooggg...lllioeoooo com/ com/file/d/ trained on
IVVLF wtt.://wwww....ate..//////e////e
D.2 INTERPOLATION SAMPLES
We expect & to produce samples of higher quality from latents near the mean of the distribution (zero). This the idea behind the Truncation Trick (Brock al., 2019). Like BigGAN, we find that DVD-GAN is amenable to truncation. We also experiment with interpolations in the latent space and in the class embedding. In both cases, interpolations are evidence that & has learned a relatively smooth mapping from the latent space to real videos: this would be impossible for a network that has only memorized the training data, or which is only capable of generating a few exemplars per class. valid (and therefore should produce
Note that while all latent vectors along interpolation reasonable sample), at no point during training is asked to generate a sample halfway between two classes. Nevertheless is able to interpolate between even very distinct classes.
22

---

Under review as a conference paper at ICLR 2020
Figure 17: An example intra-class interpolation. Each column is separate video (the vertical axis is the time dimension). The left and rightmost columns are randomly sampled latent vectors and are generated under shared class. Columns in between represent videos generated under the same class across the linear interpolation between the two random samples. Note the smooth transition between videos at all six timesteps displayed here.
Figure 18: An example of class interpolation. As before, each column is sequence of timesteps of single video. Here, we sample single latent vector, and the left and rightmost columns represent generating a video of that latent under two different classes. Columns in between represent vidcos of that same latent generated across interpolation of the class embedding. Even though point has DVD-GAN been trained on data under an interpolated class, it nevertheless produces reasonable samples.
23


Reviews:
[{"review_id": "rklJIn8wYS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nSummary:\nThis paper presents a method for training a generative adversarial network on high resolution videos and complex datasets. They propose decomposing the discriminator in Adversarial Networks into a spatial and temporal discriminator similar to previous works, however, the temporal discriminator downsamples the input using average pooling before forwarding it through the network. In experiments, the presented method outperforms previous state-of-the-art methods in the used metrics. In addition, the videos generated from the Kinetic-600 dataset in a non-conditional setting are the most realistic looking up to date. \n\n\nPros:\n+ Best generated video quality out there (In the Kinetics-600 dataset)\n+ Numerically outperforms baselines in all datasets\n\nWeaknesses / comments:\n* What is the message of the paper?\n- It is not clear to me whether this paper is about a new method or if it’s about an empirical study of the network size for video prediction algorithms. If the paper is about a new method, then the novelty is lacking. Decomposing the discriminator into space and time discriminators was originally proposed by MoCoGAN. Therefore, the only difference I can see is that the input to the time discriminator passes through a downsampling function (average pooling). On the other hand, if the paper is about showing that a bigger video model generates better video, the paper is lacking a more in depth evaluation. It seems that the size (frame resolution and number of frames) is only done on the Kinetics-600 dataset (Table 1). However, it is not clear to me whether there is a conclusion to be made from these results about the model size. The FID seems to favor smaller models and IS favors bigger models? But the last (biggest) model does not increase the IS? I am not sure what to make of these results. In addition, there is no such results on the other datasets (UCF101 and BAIR) or on the frame conditioned experiments for Kinetics-600. Do the authors have any comments or clarifications on this?\n\n\n* Lacking proof that the network is not just overfitting to each class.\n- It seems that the authors train separate models for each video class. The generated videos are the best I have seen in such complicated dataset. The authors claim that the fact that the Kinetics-600 dataset is large automatically removes the concern of overfitting. However, I would like to see  evidence that this is true. Can the authors do something like sample a video and find its nearest neighbor in the training data and compare them to see if the model is simply memorizing the training set? Something like this could serve as evidence that the network is not simply memorizing the training data.\n\n\n* FVD Evaluation on Robot Push (marginal improvement based on FVD paper)\n- In Table 3, the authors present quantitative comparisons for frame-conditioned video prediction with state-of-the-art methods. The authors of the FVD paper state the following: \n\n“In Figure 5 it can be seen that when the difference in FVD is smaller than 50, the agreement with human raters is close to random (but never worse).”\n\nThe difference between DVD-GAN and SAVP / Video Transformer is below 50. Does this mean that this result may not significant? It seems that basically the top 3 methods are performing similarly. Can the authors comment on this?\n\n\n* Comparing 11 frame conditional generation with 16 frame unconditional generation (DVD-GAN)?\n- In Table 4, the authors are comparing DVD-GAN-FP, Video Transformer and DVD-GAN. DVD-GAN-FP clearly outperforms Video Transformer. However, the authors make a comment about the DVD-GAN result being much better than the other two and that “the synthesis model’s improved performance on this task seems to indicate that the advantage of being able to select videos to generate is greater than the advantage of having a ground truth distribution of starting frames.”. However, the DVD-GAN model generates 16 frames, and not 11 like the other 2. Is this comparison correct? Also the other 2 methods (DVD-GAN-FP and Video Transformer) are constrained by what they must predict given input frames, does this fact affect the comparison / conclusion by the authors?\n\n\n* Claim of diversity but no evidence in the video prediction setting (frame conditional generation).\n- The authors claim that their method generates diverse videos, however, there is no palpable evidence that this is true. Something like showing examples of different videos generated given the same condition should be good or per-frame evaluations like PSNR and SSIM with error bars of the N number of samples generated given the same condition could also serve as proof that the generation is diverse.\n\n\n* Video results only provided for Kinetics-600\n- It looks like the authors only provide actual video files for the un-conditional generative model for Kinetics-600. Could the authors provide videos of the frame conditioned experiments as well (UCF101, BAIR and Kinetics-600)?\n\n\n* Argument in paper: “The synthesis model’s improved performance on this task seems to indicate that the advantage of being able to select videos to generate is greater than the advantage of having a ground truth distribution of starting frames”\n- I do not fully agree with this argument. Optimally, a frame-conditional model should identify all objects/background observed in the input frame and object/background dynamics if multiple frames are given. Given the optimally identified factors, video generation/prediction should be much better or the same as a purely generative model. The fact that the experiments in this paper show that the conditional model is not as good as the non-conditional model is most likely due to the model and/or objective function and not the generation scenario. Do the authors have any comments on this?\n\n\nConclusion:\nThis paper does indeed present the best video generation up-to-date given the dataset difficulty. However, the paper itself has multiple issues as stated above. Please try to address these in the next revision.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "r1ge0nbaKS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a class-conditional GAN model for video generation called DVD-GAN. The generator uses a single latent variable and uses ConvGRU modules and ResNet blocks to generate N frames. The model uses a dual discriminator, with one discriminator that discriminates individual frames, i.e. an image discriminator, and one that discriminates the whole video. This is similar to the MoCoGAN model, with the main difference being that the video discriminator operates on a smaller resolution video, thus reducing the dimensionality of the input to discriminate. The model is used to generate videos after being trained on the large-scale Kinetics-600 dataset, which contains multiple examples and has a lot of variability across videos. The main contribution of the paper is to successfully train this large GAN model on the very large-scale Kinetics dataset. The samples from the model are very visually appealing and are qualitatively  superior to any previous video prediction model.\n\nWhile the paper mostly focuses on scaling up current models, it achieves significantly better qualitative results than previous models on a very challenging dataset, and therefore I believe it should be accepted as it is a significant advance in the field which probably will lead to follow-up work based on the model proposed here. \n\nHowever, there are a number of things that could be improved/minor comments:\n\n- Further details about the generator should be included in the main body of the paper, only having a figure to describe its architecture is not enough when the model and how to scale it up are key contributions.\n- The authors introduce a FID score for video which is similar to FVD, but FID is only used to report results in one experiment, while FVD is used for the rest of the experiments. Since the community uses FVD and there is a publicly available implementation of this metric, I'd suggest that the authors also include FVD scores in Table 1 to help reproduce the results. This is important since the FID metric is not explained thoroughly in the paper and small implementation details matter when using these metrics.\n- The related work section is missing many references to video prediction models, please add them to the paper. Some examples include:\nDecomposing motion and content for natural video sequence prediction. Villegas et al. ICLR 2017\nUnsupervised Learning of Disentangled Representations from Video. Denton and Birodkar. NIPS 2017\nPredrnn: Recurrent neural networks for predictive learning using spatiotemporal lstms. Wang et al. NIPS 2017 \nPredRNN++, ContextVP, ...\n- Additional metrics for the BAIR experiment, including LPIPS and SSIM. While FVD correlates well with human judgement, LPIPS does so as well and provides another evaluation of the model. Furthermore, metrics such as SSIM as used in SVG-LP can help better understand how well do the models cover the ground-truth sequence for given context frames.\n- Qualitative results for the BAIR experiment. Since most current models are not trained on Kinetics, qualitative samples for the BAIR dataset would help to qualitatively compare current methods to DVD-GAN.\n- In practice people have found that it is very difficult to train BigGAN-like models on images and videos. A common difficulty is that training diverges after a number of iterations, with the model starting to show mode collapse and big oscillations in terms of FID scores. Since the models are trained for a big number of iterations, have you observed these kind of issues with different hyperparameter configurations? If so, did you find any strategies to address it?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "HkeZSl0aKS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper tackles the problem of efficient video generation. The authors present a Dual-Video-Discriminator Generative Adversarial Network (DVD-GAN) composed of an image-level spatial discriminator and video-level temporal discriminator. DVD-GAN achieves state-of-the-art results when benchmarked against the FID, IS and FVD quantitative metrics. Compared to previous video generation works, DVD-GAN is the first model to present compelling qualitative results on the UCF-101 and Kinetics-600 dataset.\n\nMotivation:\nEfficient video generation using GANs remains a significant challenge as it exacerbates all the issues associated with image generation using GANs. There are also increased memory and computation costs due to the 3D nature of video and the requirement for temporal modelling.\n\nMain Contributions:\n1. The authors propose to spatially down-sample the input into the video-level temporal discriminator. \n2. The authors propose to temporally sub-sample video frames for the image-level spatial discriminator\n\nSecondary Contributions:\n1. The authors set a benchmark for video generation on the Kinetics-600 dataset. This is significant due to the scale and complexity of this dataset in comparison to other datasets in the video generation literature.\n\n----------------\nPros:\n+ The paper is fairly well-written and clear\n+ The paper recognizes and tackles a significant challenge in video generation when using GANs.\n+ The ablation studies and experiments offer clarity with regard to the performance of the proposed model on different datasets and with different sampling strategies.\n+ This paper presents compelling results for high-resolution (i.e. > 64x64) video generation on complex datasets.\n\n\nCons:\nThe main weakness of this paper is that it does not represent a significant advancement in our understanding of video generation using GANs. The novelty of DVD-GAN is also limited with respect to prior video GAN literature. DVD-GAN appears to be a straight-forward application of the BIG-GAN [5] family of models to video generation. Subsequent experiments and results are tailored for this particular model with little to no applicability to prior models or generalization. This is a significant weakness given that this paper's main contribution is a discriminator component for video GAN architectures.\n\nOther Observations:\n- Dual Video Discriminator GANs were introduced in MoCoGAN [1], the naming of this model may mislead readers into thinking that this is the first such discriminator architecture.\n- Subsampling of the discriminator has already been explored in TGANv2 [2].\n\nPoints of Improvement:\n- Given that the video discriminator is the main contribution, it should be benchmarked against previous video generator models in the literature such as TGAN [3], MoCoGAN [1] and TGANv2 [2]\n- Given that the reproducibility of these results (at this moment in time) is difficult outside of a few organisations, can the authors provide a more thorough exposition of the efficiency aspects of their proposed discriminator. Currently, the best argument for the efficiency claims is the 58% reduction in pixels processed per video but what does this mean concretely, with respect to memory, computation and time efficiency?\n\n\nFinal notes:\n- Authors should clear up the confusion with class-conditional vs unconditional UCF101 results by clearly labeling class-conditional vs unconditional results. Furthermore, they should provide a detailed section on how the UCF101 metrics were achieved, it is known that video GAN metrics are highly sensitive to even the decision of when to normalize inputs to the evaluation network (see Appendix B of [4])\n\n- As impressive as the results are on Kinetics-600, they ultimately are dampened if they cannot be reproduced or built upon. The original Kinetics-600 dataset (based on youtube clips) is no longer available as youtube has either blocked, geo-blocked or removed a significant portion of this dataset. In light of this, do the authors plan to provide access to this dataset or should we place less significance on the Kinetics results?\n\n-------\n\nCurrent Review Decision:\nAlthough this paper does present state-of-the-art results for all video generation datasets and carries out thorough experimentation to demonstrate this. The DVD-GAN model rides significantly on the backbone of the BIG-GAN model to achieve this. The Dual Video Discriminator by itself does not provide a novel contribution for a conference such as ICLR as it has already been proposed [1] and discriminator subsampling has been explored in different forms previously [2]. Crucially, the claims on efficiency are not sufficiently explored and it is unclear what they are and how they would translate to other video GAN models in the literature. As such, I lean towards rejecting this paper in its current form.\n\n\n----------------\n[1] - https://arxiv.org/abs/1707.04993\n[2] - https://arxiv.org/abs/1811.09245\n[3] - https://arxiv.org/abs/1611.06624\n[4] - https://arxiv.org/abs/1909.12400\n[5] - https://arxiv.org/abs/1809.11096\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses the tasks of video generation and prediction and shows impressive results on the datasets such as Kinetics-600. There is a reviewer disagreement on this paper. AC can confirm that all three reviewers have read the rebuttal and have contributed to a long discussion. The reviewers have raised the following concerns that were viewed as critical issues when making the final decision: R1 and R3 expressed the concerns regarding limited technical novelty of the proposed approach in light of the prior works, e.g. MoCoGAN and TGANv2. R3 suggests, that the proposed method shows advantage that might be due to the large computational resources available to train the model. Providing a comparison of the proposed model and the relevant baselines on the Kinetics dataset is desirable to access the benefits of the proposed approach (R1). 
AC also agrees with the R2 about the potential impact this work could have in the community. However, given that the reviewers have raised important concerns and have given suggestions, the paper needs too many revisions for acceptance at this time. We hope the reviews are useful for improving and revising the paper.
Decision: Reject