Adversarial Video Generation on Complex Datasets

1 INTRODUCTION Modern deep generative models can produce realistic natural images when trained on high-resolution and diverse datasets ( Brock et al. , 2019 ; Karras et al. , 2018 ; Kingma & Dhariwal , 2018 ; Menick & Kalchbrenner , 2019 ; Razavi et al. , 2019 ) . Generation of natural video is an obvious further challenge for generative modeling , but one that is plagued by increased data complexity and computational requirements . For this reason , much prior work on video generation has revolved around relatively simple datasets , or tasks where strong temporal conditioning information is available . We focus on the tasks of video synthesis and video prediction ( defined in Section 2.1 ) , and aim to extend the strong results of generative image models to the video domain . Building upon the state-of-the-art BigGAN architecture ( Brock et al. , 2019 ) , we introduce an efficient spatio-temporal decomposition of the discriminator which allows us to train on Kinetics-600 – a complex dataset of natural videos an order of magnitude larger than other commonly used datasets . The resulting model , Dual Video Discriminator GAN ( DVD-GAN ) , is able to generate temporally coherent , high-resolution videos of relatively high fidelity ( Figure 1 ) . Our contributions are as follows : • We propose DVD-GAN – a scalable generative model of natural video which produces high-quality samples at resolutions up to 256× 256 and lengths up to 48 frames . • We achieve state of the art for video synthesis on UCF-101 and prediction on Kinetics-600 . • We establish class-conditional video synthesis on Kinetics-600 as a new benchmark for generative video modeling , and report DVD-GAN results as a strong baseline . 2 BACKGROUND . 2.1 VIDEO SYNTHESIS AND PREDICTION . The exact formulation of the video generation task can differ in the type of conditioning signal provided . At one extreme lies unconditional video synthesis where the task is to generate any video following the training distribution . Another extreme is occupied by strongly-conditioned models , including generation conditioned on another video for content transfer ( Bansal et al. , 2018 ; Zhou et al. , 2019 ) , per-frame segmentation masks ( Wang et al. , 2018a ) , or pose information ( Walker et al. , 2017 ; Villegas et al. , 2017b ; Yang et al. , 2018 ) . In the middle ground there are tasks which are more structured than unconditional generation , and yet are more challenging from a modeling perspective than strongly-conditional generation ( which gets a lot of information about the generated video through its input ) . The objective of class-conditional video synthesis is to generate a video of a given category ( e.g. , “ riding a bike ” ) while future video prediction is concerned with generation of continuing video given initial frames . These problems differ in several aspects , but share a common requirement of needing to generate realistic temporal dynamics , and in this work we focus on these two problems . 2.2 GENERATIVE ADVERSARIAL NETWORKS . Generative Adversarial Networks ( GANs ) ( Goodfellow et al. , 2014 ) are a class of generative models defined by a minimax game between a Discriminator D and a Generator G. The original objective was proposed by Goodfellow et al . ( 2014 ) , and many improvements have since been suggested , mostly targeting improved training stability ( Arjovsky et al. , 2017 ; Zhang et al. , 2018 ; Brock et al. , 2019 ; Gulrajani et al. , 2017 ; Miyato et al. , 2018 ) . We use the hinge formulation of the objective ( Lim & Ye , 2017 ; Brock et al. , 2019 ) which is optimized by gradient descent ( ρ is the elementwise ReLU function ) : D : min D E x∼data ( x ) [ ρ ( 1−D ( x ) ) ] + E z∼p ( z ) [ ρ ( 1 +D ( G ( z ) ) ) ] , G : max G E z∼p ( z ) [ D ( G ( z ) ) ] . GANs have well-known limitations including a tendency towards limited diversity in generated samples ( a phenomenon known as mode collapse ) and the difficulty of quantitative evaluation due to the lack of an explicit likelihood measure over the data . Despite these downsides , GANs have produced some of the highest fidelity samples across many visual domains ( Karras et al. , 2018 ; Brock et al. , 2019 ) . 2.3 KINETICS-600 . Kinetics is a large dataset of 10-second high-resolution YouTube clips ( Kay et al. , 2017 ; DeepMind , 2018 ) originally created for the task of human action recognition . We use the second iteration of the dataset , Kinetics-600 ( Carreira et al. , 2018 ) , which consists of 600 classes with at least 600 videos per class for a total of around 500,000 videos.1 Kinetics videos are diverse and unconstrained , which allows us to train large models without being concerned with the overfitting that occurs on small datasets with fixed objects interacting in specified ways ( Ebert et al. , 2017 ; Blank et al. , 2005 ) . Among prior work , the closest dataset ( in terms of subject and complexity ) which is consistently used is UCF-101 ( Soomro et al. , 2012 ) . We focus on Kinetics-600 because of its larger size ( almost 50x more videos than UCF-101 ) and its increased diversity ( 600 instead of 101 classes – not to mention increased intra-class diversity ) . Nevertheless for comparison with prior art we train on UCF-101 and achieve a state-of-the-art Inception Score there . Kinetics contains many artifacts expected from YouTube , including cuts ( as in Figure 2d ) , title screens and visual effects . Except when specifically described , we choose frames with stride 2 ( meaning we skip every other frame ) . This allows us to generate videos with more complexity without incurring higher computational cost . To the best of our knowledge we are the first to consider generative modelling of the entirety of the Kinetics video dataset2 , although a small subset of Kinetics consisting of 4,000 selected and stabilized videos ( via a SIFT + RANSAC procedure ) has been used in at least two prior papers ( Li et al. , 2018 ; Balaji et al. , 2018 ) . Due to the heavy pre-processing and stabilization present , as well as the sizable reduction in dataset size ( two orders of magnitude ) we do not consider these datasets comparable to the full Kinetics-600 dataset . 2.4 EVALUATION METRICS . Designing metrics for measuring the quality of generative models ( GANs in particular ) is an active area of research ( Sajjadi et al. , 2018 ; Barratt & Sharma , 2018 ) . In this work we report the two most commonly used metrics , Inception Score ( IS ) ( Salimans et al . ( 2016 ) ) and Fréchet Inception Distance ( FID ) ( Heusel et al. , 2017 ) . The standard instantiation of these metrics is intended for generative image models , and uses an Inception model ( Szegedy et al. , 2016 ) for image classification or feature extraction . For videos , we use the publicly available Inflated 3D Convnet ( I3D ) network trained on Kinetics-600 ( Carreira & Zisserman , 2017 ) . Our Fréchet Inception Distance is therefore very similar to the Fréchet Video Distance ( FVD ) ( Unterthiner et al. , 2018 ) , although our implementation is different and more aligned with the original FID metric.3 More details are in Appendix A.4 . 3 DUAL VIDEO DISCRIMINATOR GAN . Our primary contribution is Dual Video Discriminator GAN ( DVD-GAN ) , a generative video model of complex human actions built upon the state-of-the-art BigGAN architecture ( Brock et al. , 2019 ) while introducing scalable , video-specific generator and discriminator architectures . An overview of the DVD-GAN architecture is given in Figure 3 and a detailed description is in Appendix A.2 . Unlike some of the prior work , our generator contains no explicit priors for foreground , background or motion ( optical flow ) ; instead , we rely on a high-capacity neural network to learn this in a data-driven manner . While DVD-GAN contains sequential components ( RNNs ) , it is not autoregressive in time or in space . In other words , the pixels of each frame do not directly depend on other pixels in the video , as would be the case for auto-regressive models or models generating one frame at a time . Generating long and high resolution videos is a heavy computational challenge : individual samples from Kinetics-600 ( just 10 seconds long ) contain upwards of 16 million pixels which need to be 1Kinetics is occasionally pruned and so we can not give an exact size . 2In parallel with the concurrent work of Weissenborn et al . ( 2019 ) . 3We use ‘ avgpool ’ features ( rather than logits ) by default , our I3D model is trained on Kinetics-600 ( rather than Kinetics-400 ) , and we pre-calculate ground-truth statistics on the entire training set . Generator G ResNet Block 3 3 1 3 3 1 D ResNet Block Class Conditional Batch Norm ReLU Activation Bilinear x2 Upsampling Feature Concatenation Convolution with Kernel Size k Average Pool x2 Downsampling k Standard Gaussian Noise Convolutional Gated Recurrent Unit One-Hot Class Indicator 1 1 Frame T Frame 0 G ResNet Block x2 G ResNet Block x2 G ResNet Block x2 x Depth Times 3 3 3 Per Frame D ResNet Block x Depth Times ℒS D ResNet Block x Depth Times ℒT Discriminators Figure 3 : Simplified architecture diagram of G ( left ) and DS /DT ( right ) . More details in A.2 . generated in a consistent fashion . This is a particular challenge to the discriminator . For example , a generated video might contain an object which leaves the field of view and incorrectly returns with a different color . Here , the ability to determine this video is generated is only possible by comparing two different spatial locations across two ( potentially distant ) frames . Given a video with length T , height H , and width W , discriminators that process the entire video would have to process all H ×W × T pixels – limiting the size of the model and the size of the videos being generated . 3.1 DUAL DISCRIMINATORS . DVD-GAN tackles this scale problem by using two discriminators : a Spatial Discriminator DS and a Temporal Discriminator DT . DS critiques single frame content and structure by randomly sampling k full-resolution frames and judging them individually . We use k = 8 and discuss this choice in Section 4.3 . DS ’ s final score is the sum of the per-frame scores . The temporal discriminator DT must provide G with the learning signal to generate movement ( something not evaluated by DS ) . To make the model scalable , we apply a spatial downsampling function φ ( · ) to the whole video and feed its output to DT . We choose φ to be 2× 2 average pooling , and discuss alternatives in Section 4.3 . This results in an architecture where the discriminators do not process the entire video ’ s worth of pixels , since DS processes only k × H ×W pixels and DT only T × H2 × W 2 . For a 48 frame video at 128× 128 resolution , this reduces the number of pixels to process per video from 786432 to 327680 : a 58 % reduction . Despite this decomposition , the discriminator objective is still able to penalize almost all inconsistencies which would be penalized by a discriminator judging the entire video . DT judges any temporal discrepancies across the entire length of the video , and DS can judge any high resolution details . The only detail the DVD-GAN discriminator objective is unable to reflect is the temporal evolution of pixels within a 2 × 2 window . We have however not noticed this affecting the generated samples in practice . DVD-GAN ’ s DS is similar to the per-frame discriminator DI in MoCoGAN ( Tulyakov et al. , 2018 ) . However MoCoGAN ’ s analog of DT looks at full resolution videos , whereas DS is the only source of learning signal for high-resolution details in DVD-GAN . For this reason , DS is essential when φ is not the identity , unlike in MoCoGAN where the additional per-frame discriminator is less crucial .

Summary: The paper proposes a class-conditional GAN model for video generation called DVD-GAN. The generator uses a single latent variable and uses ConvGRU modules and ResNet blocks to generate N frames. The model uses a dual discriminator, with one discriminator that discriminates individual frames, i.e. an image discriminator, and one that discriminates the whole video. This is similar to the MoCoGAN model, with the main difference being that the video discriminator operates on a smaller resolution video, thus reducing the dimensionality of the input to discriminate. The model is used to generate videos after being trained on the large-scale Kinetics-600 dataset, which contains multiple examples and has a lot of variability across videos. The main contribution of the paper is to successfully train this large GAN model on the very large-scale Kinetics dataset. The samples from the model are very visually appealing and are qualitatively  superior to any previous video prediction model.