Adversarial Video Generation on Complex Datasets

1 INTRODUCTION Modern deep generative models can produce realistic natural images when trained on high-resolution and diverse datasets ( Brock et al. , 2019 ; Karras et al. , 2018 ; Kingma & Dhariwal , 2018 ; Menick & Kalchbrenner , 2019 ; Razavi et al. , 2019 ) . Generation of natural video is an obvious further challenge for generative modeling , but one that is plagued by increased data complexity and computational requirements . For this reason , much prior work on video generation has revolved around relatively simple datasets , or tasks where strong temporal conditioning information is available . We focus on the tasks of video synthesis and video prediction ( defined in Section 2.1 ) , and aim to extend the strong results of generative image models to the video domain . Building upon the state-of-the-art BigGAN architecture ( Brock et al. , 2019 ) , we introduce an efficient spatio-temporal decomposition of the discriminator which allows us to train on Kinetics-600 – a complex dataset of natural videos an order of magnitude larger than other commonly used datasets . The resulting model , Dual Video Discriminator GAN ( DVD-GAN ) , is able to generate temporally coherent , high-resolution videos of relatively high fidelity ( Figure 1 ) . Our contributions are as follows : • We propose DVD-GAN – a scalable generative model of natural video which produces high-quality samples at resolutions up to 256× 256 and lengths up to 48 frames . • We achieve state of the art for video synthesis on UCF-101 and prediction on Kinetics-600 . • We establish class-conditional video synthesis on Kinetics-600 as a new benchmark for generative video modeling , and report DVD-GAN results as a strong baseline . 2 BACKGROUND . 2.1 VIDEO SYNTHESIS AND PREDICTION . The exact formulation of the video generation task can differ in the type of conditioning signal provided . At one extreme lies unconditional video synthesis where the task is to generate any video following the training distribution . Another extreme is occupied by strongly-conditioned models , including generation conditioned on another video for content transfer ( Bansal et al. , 2018 ; Zhou et al. , 2019 ) , per-frame segmentation masks ( Wang et al. , 2018a ) , or pose information ( Walker et al. , 2017 ; Villegas et al. , 2017b ; Yang et al. , 2018 ) . In the middle ground there are tasks which are more structured than unconditional generation , and yet are more challenging from a modeling perspective than strongly-conditional generation ( which gets a lot of information about the generated video through its input ) . The objective of class-conditional video synthesis is to generate a video of a given category ( e.g. , “ riding a bike ” ) while future video prediction is concerned with generation of continuing video given initial frames . These problems differ in several aspects , but share a common requirement of needing to generate realistic temporal dynamics , and in this work we focus on these two problems . 2.2 GENERATIVE ADVERSARIAL NETWORKS . Generative Adversarial Networks ( GANs ) ( Goodfellow et al. , 2014 ) are a class of generative models defined by a minimax game between a Discriminator D and a Generator G. The original objective was proposed by Goodfellow et al . ( 2014 ) , and many improvements have since been suggested , mostly targeting improved training stability ( Arjovsky et al. , 2017 ; Zhang et al. , 2018 ; Brock et al. , 2019 ; Gulrajani et al. , 2017 ; Miyato et al. , 2018 ) . We use the hinge formulation of the objective ( Lim & Ye , 2017 ; Brock et al. , 2019 ) which is optimized by gradient descent ( ρ is the elementwise ReLU function ) : D : min D E x∼data ( x ) [ ρ ( 1−D ( x ) ) ] + E z∼p ( z ) [ ρ ( 1 +D ( G ( z ) ) ) ] , G : max G E z∼p ( z ) [ D ( G ( z ) ) ] . GANs have well-known limitations including a tendency towards limited diversity in generated samples ( a phenomenon known as mode collapse ) and the difficulty of quantitative evaluation due to the lack of an explicit likelihood measure over the data . Despite these downsides , GANs have produced some of the highest fidelity samples across many visual domains ( Karras et al. , 2018 ; Brock et al. , 2019 ) . 2.3 KINETICS-600 . Kinetics is a large dataset of 10-second high-resolution YouTube clips ( Kay et al. , 2017 ; DeepMind , 2018 ) originally created for the task of human action recognition . We use the second iteration of the dataset , Kinetics-600 ( Carreira et al. , 2018 ) , which consists of 600 classes with at least 600 videos per class for a total of around 500,000 videos.1 Kinetics videos are diverse and unconstrained , which allows us to train large models without being concerned with the overfitting that occurs on small datasets with fixed objects interacting in specified ways ( Ebert et al. , 2017 ; Blank et al. , 2005 ) . Among prior work , the closest dataset ( in terms of subject and complexity ) which is consistently used is UCF-101 ( Soomro et al. , 2012 ) . We focus on Kinetics-600 because of its larger size ( almost 50x more videos than UCF-101 ) and its increased diversity ( 600 instead of 101 classes – not to mention increased intra-class diversity ) . Nevertheless for comparison with prior art we train on UCF-101 and achieve a state-of-the-art Inception Score there . Kinetics contains many artifacts expected from YouTube , including cuts ( as in Figure 2d ) , title screens and visual effects . Except when specifically described , we choose frames with stride 2 ( meaning we skip every other frame ) . This allows us to generate videos with more complexity without incurring higher computational cost . To the best of our knowledge we are the first to consider generative modelling of the entirety of the Kinetics video dataset2 , although a small subset of Kinetics consisting of 4,000 selected and stabilized videos ( via a SIFT + RANSAC procedure ) has been used in at least two prior papers ( Li et al. , 2018 ; Balaji et al. , 2018 ) . Due to the heavy pre-processing and stabilization present , as well as the sizable reduction in dataset size ( two orders of magnitude ) we do not consider these datasets comparable to the full Kinetics-600 dataset . 2.4 EVALUATION METRICS . Designing metrics for measuring the quality of generative models ( GANs in particular ) is an active area of research ( Sajjadi et al. , 2018 ; Barratt & Sharma , 2018 ) . In this work we report the two most commonly used metrics , Inception Score ( IS ) ( Salimans et al . ( 2016 ) ) and Fréchet Inception Distance ( FID ) ( Heusel et al. , 2017 ) . The standard instantiation of these metrics is intended for generative image models , and uses an Inception model ( Szegedy et al. , 2016 ) for image classification or feature extraction . For videos , we use the publicly available Inflated 3D Convnet ( I3D ) network trained on Kinetics-600 ( Carreira & Zisserman , 2017 ) . Our Fréchet Inception Distance is therefore very similar to the Fréchet Video Distance ( FVD ) ( Unterthiner et al. , 2018 ) , although our implementation is different and more aligned with the original FID metric.3 More details are in Appendix A.4 . 3 DUAL VIDEO DISCRIMINATOR GAN . Our primary contribution is Dual Video Discriminator GAN ( DVD-GAN ) , a generative video model of complex human actions built upon the state-of-the-art BigGAN architecture ( Brock et al. , 2019 ) while introducing scalable , video-specific generator and discriminator architectures . An overview of the DVD-GAN architecture is given in Figure 3 and a detailed description is in Appendix A.2 . Unlike some of the prior work , our generator contains no explicit priors for foreground , background or motion ( optical flow ) ; instead , we rely on a high-capacity neural network to learn this in a data-driven manner . While DVD-GAN contains sequential components ( RNNs ) , it is not autoregressive in time or in space . In other words , the pixels of each frame do not directly depend on other pixels in the video , as would be the case for auto-regressive models or models generating one frame at a time . Generating long and high resolution videos is a heavy computational challenge : individual samples from Kinetics-600 ( just 10 seconds long ) contain upwards of 16 million pixels which need to be 1Kinetics is occasionally pruned and so we can not give an exact size . 2In parallel with the concurrent work of Weissenborn et al . ( 2019 ) . 3We use ‘ avgpool ’ features ( rather than logits ) by default , our I3D model is trained on Kinetics-600 ( rather than Kinetics-400 ) , and we pre-calculate ground-truth statistics on the entire training set . Generator G ResNet Block 3 3 1 3 3 1 D ResNet Block Class Conditional Batch Norm ReLU Activation Bilinear x2 Upsampling Feature Concatenation Convolution with Kernel Size k Average Pool x2 Downsampling k Standard Gaussian Noise Convolutional Gated Recurrent Unit One-Hot Class Indicator 1 1 Frame T Frame 0 G ResNet Block x2 G ResNet Block x2 G ResNet Block x2 x Depth Times 3 3 3 Per Frame D ResNet Block x Depth Times ℒS D ResNet Block x Depth Times ℒT Discriminators Figure 3 : Simplified architecture diagram of G ( left ) and DS /DT ( right ) . More details in A.2 . generated in a consistent fashion . This is a particular challenge to the discriminator . For example , a generated video might contain an object which leaves the field of view and incorrectly returns with a different color . Here , the ability to determine this video is generated is only possible by comparing two different spatial locations across two ( potentially distant ) frames . Given a video with length T , height H , and width W , discriminators that process the entire video would have to process all H ×W × T pixels – limiting the size of the model and the size of the videos being generated . 3.1 DUAL DISCRIMINATORS . DVD-GAN tackles this scale problem by using two discriminators : a Spatial Discriminator DS and a Temporal Discriminator DT . DS critiques single frame content and structure by randomly sampling k full-resolution frames and judging them individually . We use k = 8 and discuss this choice in Section 4.3 . DS ’ s final score is the sum of the per-frame scores . The temporal discriminator DT must provide G with the learning signal to generate movement ( something not evaluated by DS ) . To make the model scalable , we apply a spatial downsampling function φ ( · ) to the whole video and feed its output to DT . We choose φ to be 2× 2 average pooling , and discuss alternatives in Section 4.3 . This results in an architecture where the discriminators do not process the entire video ’ s worth of pixels , since DS processes only k × H ×W pixels and DT only T × H2 × W 2 . For a 48 frame video at 128× 128 resolution , this reduces the number of pixels to process per video from 786432 to 327680 : a 58 % reduction . Despite this decomposition , the discriminator objective is still able to penalize almost all inconsistencies which would be penalized by a discriminator judging the entire video . DT judges any temporal discrepancies across the entire length of the video , and DS can judge any high resolution details . The only detail the DVD-GAN discriminator objective is unable to reflect is the temporal evolution of pixels within a 2 × 2 window . We have however not noticed this affecting the generated samples in practice . DVD-GAN ’ s DS is similar to the per-frame discriminator DI in MoCoGAN ( Tulyakov et al. , 2018 ) . However MoCoGAN ’ s analog of DT looks at full resolution videos , whereas DS is the only source of learning signal for high-resolution details in DVD-GAN . For this reason , DS is essential when φ is not the identity , unlike in MoCoGAN where the additional per-frame discriminator is less crucial .

Summary: This paper tackles the problem of efficient video generation. The authors present a Dual-Video-Discriminator Generative Adversarial Network (DVD-GAN) composed of an image-level spatial discriminator and video-level temporal discriminator. DVD-GAN achieves state-of-the-art results when benchmarked against the FID, IS and FVD quantitative metrics. Compared to previous video generation works, DVD-GAN is the first model to present compelling qualitative results on the UCF-101 and Kinetics-600 dataset.