A Good Image Generator Is What You Need for High-Resolution Video Synthesis

1 INTRODUCTION . Video synthesis seeks to generate a sequence of moving pictures from noise . While its closely related counterpart—image synthesis—has seen substantial advances in recent years , allowing for synthesizing at high resolutions ( Karras et al. , 2017 ) , rendering images often indistinguishable from real ones ( Karras et al. , 2019 ) , and supporting multiple classes of image content ( Zhang et al. , 2019 ) , contemporary improvements in the domain of video synthesis have been comparatively modest . Due to the statistical complexity of videos and larger model sizes , video synthesis produces relatively low-resolution videos , yet requires longer training times . For example , scaling the image generator of Brock et al . ( 2019 ) to generate 256 × 256 videos requires a substantial computational budget1 . Can we use a similar method to attain higher resolutions ? We believe a different approach is needed . There are two desired properties for generated videos : ( i ) high quality for each individual frame , and ( ii ) the frame sequence should be temporally consistent , i.e . depicting the same content with plausible motion . Previous works ( Tulyakov et al. , 2018 ; Clark et al. , 2019 ) attempt to achieve both goals with a single framework , making such methods computationally demanding when high resolution is desired . We suggest a different perspective on this problem . We hypothesize that , given an image generator that has learned the distribution of video frames as independent images , a video can be represented as a sequence of latent codes from this generator . The problem of video synthesis can then be framed as discovering a latent trajectory that renders temporally consistent images . Hence , we demonstrate that ( i ) can be addressed by a pre-trained and fixed image generator , and ( ii ) can be achieved using the proposed framework to create appropriate image sequences . ∗Work done while at Snap Inc. 1We estimate that the cost of training a model such as DVD-GAN ( Clark et al. , 2019 ) once requires > $ 30K . To discover the appropriate latent trajectory , we introduce a motion generator , implemented via two recurrent neural networks , that operates on the initial content code to obtain the motion representation . We model motion as a residual between continuous latent codes that are passed to the image generator for individual frame generation . Such a residual representation can also facilitate the disentangling of motion and content . The motion generator is trained using the chosen image discriminator with contrastive loss to force the content to be temporally consistent , and a patch-based multi-scale video discriminator for learning motion patterns . Our framework supports contemporary image generators such as StyleGAN2 ( Karras et al. , 2019 ) and BigGAN ( Brock et al. , 2019 ) . We name our approach as MoCoGAN-HD ( Motion and Content decomposed GAN for HighDefinition video synthesis ) as it features several major advantages over traditional video synthesis pipelines . First , it transcends the limited resolutions of existing techniques , allowing for the generation of high-quality videos at resolutions up to 1024 × 1024 . Second , as we search for a latent trajectory in an image generator , our method is computationally more efficient , requiring an order of magnitude less training time than previous video-based works ( Clark et al. , 2019 ) . Third , as the image generator is fixed , it can be trained on a separate high-quality image dataset . Due to the disentangled representation of motion and content , our approach can learn motion from a video dataset and apply it to an image dataset , even in the case of two datasets belonging to different domains . It thus unleashes the power of an image generator to synthesize high quality videos when a domain ( e.g. , dogs ) contains many high-quality images but no corresponding high-quality videos ( see Fig . 4 ) . In this manner , our method can generate realistic videos of objects it has never seen moving during training ( such as generating realistic pet face videos using motions extracted from images of talking people ) . We refer to this new video generation task as cross-domain video synthesis . Finally , we quantitatively and qualitatively evaluate our approach , attaining state-of-the-art performance on each benchmark , and establish a challenging new baseline for video synthesis methods . 2 RELATED WORK . Video Synthesis . Approaches to image generation and translation using Generative Adversarial Networks ( GANs ) ( Goodfellow et al. , 2014 ) have demonstrated the ability to synthesize high quality images ( Radford et al. , 2016 ; Zhang et al. , 2019 ; Brock et al. , 2019 ; Donahue & Simonyan , 2019 ; Jin et al. , 2021 ) . Built upon image translation ( Isola et al. , 2017 ; Wang et al. , 2018b ) , works on video-to-video translation ( Bansal et al. , 2018 ; Wang et al. , 2018a ) are capable of converting an input video to a high-resolution output in another domain . However , the task of high-fidelity video generation , in the unconditional setting , is still a difficult and unresolved problem . Without the strong conditional inputs such as segmentation masks ( Wang et al. , 2019 ) or human poses ( Chan et al. , 2019 ; Ren et al. , 2020 ) that are employed by video-to-video translation works , generating videos following the distribution of training video samples is challenging . Earlier works on GANbased video modeling , including MDPGAN ( Yushchenko et al. , 2019 ) , VGAN ( Vondrick et al. , 2016 ) , TGAN ( Saito et al. , 2017 ) , MoCoGAN ( Tulyakov et al. , 2018 ) , ProgressiveVGAN ( Acharya et al. , 2018 ) , TGANv2 ( Saito et al. , 2020 ) show promising results on low-resolution datasets . Recent efforts demonstrate the capacity to generate more realistic videos , but with significantly more computation ( Clark et al. , 2019 ; Weissenborn et al. , 2020 ) . In this paper , we focus on generating realistic videos using manageable computational resources . LDVDGAN ( Kahembwe & Ramamoorthy , 2020 ) uses low dimensional discriminator to reduce model size and can generate videos with resolution up to 512× 512 , while we decrease training cost by utilizing a pre-trained image generator . The high-quality generation is achieved by using pre-trained image generators , while the motion trajectory is modeled within the latent space . Additionally , learning motion in the latent space allows us to easily adapt the video generation model to the task of video prediction ( Denton et al. , 2017 ) , in which the starting frame is given ( Denton & Fergus , 2018 ; Zhao et al. , 2018 ; Walker et al. , 2017 ; Villegas et al. , 2017b ; a ; Babaeizadeh et al. , 2017 ; Hsieh et al. , 2018 ; Byeon et al. , 2018 ) , by inverting the initial frame through the generator ( Abdal et al. , 2020 ) , instead of training an extra image encoder ( Tulyakov et al. , 2018 ; Zhang et al. , 2020 ) . Interpretable Latent Directions . The latent space of GANs is known to consist of semantically meaningful vectors for image manipulation . Both supervised methods , either using human annotations or pre-trained image classifiers ( Goetschalckx et al. , 2019 ; Shen et al. , 2020 ) , and unsupervised methods ( Jahanian et al. , 2020 ; Plumerault et al. , 2020 ) , are able to find interpretable directions for image editing , such as supervising directions for image rotation or background removal ( Voynov & Babenko , 2020 ; Shen & Zhou , 2020 ) . We further consider the motion vectors in the latent space . By disentangling the motion trajectories in an unsupervised fashion , we are able to transfer the motion information from a video dataset to an image dataset in which no temporal information is available . Contrastive Representation Learning is widely studied in unsupervised learning tasks ( He et al. , 2020 ; Chen et al. , 2020a ; b ; Hénaff et al. , 2020 ; Löwe et al. , 2019 ; Oord et al. , 2018 ; Misra & Maaten , 2020 ) . Related inputs , such as images ( Wu et al. , 2018 ) or latent representations ( Hjelm et al. , 2019 ) , which can vary while training due to data augmentation , are forced to be close by minimizing differences in their representation during training . Recent work ( Park et al. , 2020 ) applies noisecontrastive estimation ( Gutmann & Hyvärinen , 2010 ) to image generation tasks by learning the correspondence between image patches , achieving performance superior to that attained when using cycle-consistency constraints ( Zhu et al. , 2017 ; Yi et al. , 2017 ) . On the other hand , we learn an image discriminator to create videos with coherent content by leveraging contrastive loss ( Hadsell et al. , 2006 ) along with an adversarial loss ( Goodfellow et al. , 2014 ) . 3 METHOD . In this section , we introduce our method for high-resolution video generation . Our framework is built on top of a pre-trained image generator ( Karras et al. , 2020a ; b ; Zhao et al. , 2020a ; b ) , which helps to generate high-quality image frames and boosts the training efficiency with manageable computational resources . In addition , with the image generator fixed during training , we can disentangle video motion from image content , and enable video synthesis even when the image content and the video motion come from different domains . More specifically , our inference framework includes a motion generatorGM and an image generator GI . GM is implemented with two LSTM networks ( Hochreiter & Schmidhuber , 1997 ) and predicts the latent motion trajectory Z = { z1 , z2 , · · · , zn } , where n is the number of frames in the synthesized video . The image generator GI can thus synthesize each individual frame from the motion trajectory . The generated video sequence ṽ is given by ṽ = { x̃1 , x̃2 , · · · , x̃n } . For each synthesized frame x̃t , we have x̃t = GI ( zt ) for t = 1 , 2 , · · · , n. We also define the real video clip as v = { x1 , x2 , · · · , xn } and the training video distribution as pv . To train the motion generator GM to discover the desired motion trajectory , we apply a video discriminator to constrain the generated motion patterns to be similar to those of the training videos , and an image discriminator to force the frame content to be temporally consistent . Our framework is illustrated in Fig . 1 . We describe each component in more detail in the following sections . 3.1 MOTION GENERATOR . The motion generator GM predicts consecutive latent codes using an input code z1 ∈ Z , where the latent spaceZ is also shared by the image generator . For BigGAN ( Brock et al. , 2019 ) , we sample z1 from the normal distribution pz . For StyleGAN2 ( Karras et al. , 2020b ) , pz is the distribution after the multi-layer perceptron ( MLP ) , as the latent codes within this distribution can be semantically disentangled better than when using the normal distribution ( Shen et al. , 2020 ; Zhu et al. , 2020 ) . Formally , GM includes an LSTM encoder LSTMenc , which encodes z1 to get the initial hidden state , and a LSTM decoder LSTMdec , which estimates n− 1 continuous hidden states recursively : h1 , c1 = LSTMenc ( z1 ) , ht , ct = LSTMdec ( t , ( ht−1 , ct−1 ) ) , t = 2 , 3 , · · · , n , ( 1 ) where h and c denote the hidden state and cell state respectively , and t is a noise vector sampled from the normal distribution to model the motion diversity at timestamp t. Motion Disentanglement . Prior work ( Tulyakov et al. , 2018 ) applies ht as the motion code for the frame to be generated , while the content code is fixed for all frames . However , such a design requires a recurrent network to estimate the motion while preserving consistent content from the latent vector , which is difficult to learn in practice . Instead , we propose to use a sequence of motion residuals for estimating the motion trajectory . Specifically , we model the motion residual as the linear combination of a set of interpretable directions in the latent space ( Shen & Zhou , 2020 ; Härkönen et al. , 2020 ) . We first conduct principal component analysis ( PCA ) on m randomly sampled latent vectors from Z to get the basis V. Then , we estimate the motion direction from the previous frame zt−1 to the current frame zt by using ht and V as follows : zt = zt−1 + λ · ht ·V , t = 2 , 3 , · · · , n , ( 2 ) where the hidden state ht ∈ [ −1 , 1 ] , and λ controls the step given by the residual . With Eqn . 1 and Eqn . 2 , we have GM ( z1 ) = { z1 , z2 , · · · , zn } , and the generated video ṽ is given as ṽ = GI ( GM ( z1 ) ) . Motion Diversity . In Eqn . 1 , we introduce a noise vector t to control the diversity of motion . However , we observe that the LSTM decoder tends to neglect the t , resulting in motion mode collapse , meaning that GM can not capture the diverse motion patterns from training videos and generate distinct videos from one initial latent code with similar motion patterns for different sequences of noise vectors . To alleviate this issue , we introduce a mutual information loss Lm to maximize the mutual information between the hidden vector ht and the noise vector t. With sim ( u , v ) = uTv/ ‖u‖ ‖v‖ denoting the cosine similarity between vectors u and v , we define Lm as follows : Lm = 1 n− 1 n∑ t=2 sim ( H ( ht ) , t ) , ( 3 ) where H is a 2-layer MLP that serves as a mapping function . Learning . To learn the appropriate parameters for the motion generator GM , we apply a multi-scale video discriminator DV to tell whether a video sequence is real or synthesized . The discriminator is based on the architecture of PatchGAN ( Isola et al. , 2017 ) . However , we use 3D convolutional layers in DV , as they can model temporal dynamics better than 2D convolutional layers . We divide input video sequence into small 3D patches , and classify each patch as real or fake . The local responses for the input sequence are averaged to produce the final output . Additionally , each frame in the input video sequence is conditioned on the first frame , as it falls into the distribution of the pre-trained image generator , for more stable training . We thus optimize the following adversarial loss to learn GM and DV : LDV = Ev∼pv [ logDv ( v ) ] + Ez1∼pz [ log ( 1−DV ( GI ( GM ( z1 ) ) ) ) ] . ( 4 )

Summary: This paper addresses the problem of video synthesis --- generating diverse, realistic videos. This paper's core idea is to leverage a fixed, pre-trained GAN model for image synthesis and train a motion generator to produce a sequence of latent vectors to generate image sequences (using the pretrained GAN and the generated latent vectors) are temporally coherent. The specific technical novelties lie in (1) predicting the motion residual and (2) adding contrastive image discriminator to ensure that generated contents in a video are similar. The paper provides an extensive set of experiments demonstrating the proposed method's effectiveness over the state-of-the-art video synthesis models.