Generating Videos with Dynamics-aware Implicit Generative Adversarial Networks

1 INTRODUCTION . Deep generative models have successfully synthesized realistic samples on various domains , including image ( Brock et al. , 2019 ; Karras et al. , 2020b ; 2021 ; Dhariwal & Nichol , 2021 ) , text ( Adiwardana et al. , 2020 ; Brown et al. , 2020 ) , and audio ( Dhariwal et al. , 2020 ; Lakhotia et al. , 2021 ) . Recently , video generation has emerged as the next challenge of deep generative models , and thus a long line of work has been proposed to learn the video distribution ( Vondrick et al. , 2016 ; Kalchbrenner et al. , 2017 ; Saito et al. , 2017 ; 2020 ; Tulyakov et al. , 2018 ; Acharya et al. , 2018 ; Clark et al. , 2019 ; Weissenborn et al. , 2020 ; Rakhimov et al. , 2020 ; Tian et al. , 2021 ; Yan et al. , 2021 ) . Despite their significant efforts , a substantial gap still exists from large-scale real-world videos . The difficulty of video generation mainly stems from the complexity of video signals ; they are continuously correlated across spatio-temporal directions . Specifically , most prior works interpret the video as a 3D grid of RGB values , i.e. , a sequence of 2D images , and model them with discrete decoders such as convolutional ( Tian et al. , 2021 ) or autoregressive ( Yan et al. , 2021 ) networks . However , such discrete modeling limits the scalability of generated videos due to the cubic complexity ( Saito et al. , 2020 ) and ignores the inherent continuous temporal dynamics ( Gordon & Parde , 2021 ) . Meanwhile , implicit neural representations ( INRs ; Sitzmann et al . ( 2020 ) ; Tancik et al . ( 2020 ) ) have emerged as a new paradigm for representing continuous signals . INR encodes a signal into a neural network that maps input coordinates to corresponding signal values , e.g. , 2D coordinates of images to RGB values . Consequently , INR amortizes the signal values of arbitrary coordinates into a compact neural representation instead of discrete grid-wise signal values , requiring a large memory proportional to coordinate dimension and resolution . In this respect , INRs have shown to be highly 1Videos are available at the project site https : //sihyun-yu.github.io/digan/ . effective at modeling complex signals such as 3D scenes ( Mildenhall et al. , 2020 ; Li et al. , 2021a ) . Furthermore , INR has intriguing properties from its compactness and continuity , e.g. , reduced data memory ( Dupont et al. , 2021a ) and upsampling to arbitrary resolution ( Chen et al. , 2021b ) . Several works utilize INRs for generative modeling , i.e. , samples are generated through INRs ( Chan et al. , 2021 ; Dupont et al. , 2021b ; Kosiorek et al. , 2021 ) . In particular , Skorokhodov et al . ( 2021a ) and Anokhin et al . ( 2021 ) exposed that INR-based image generative adversarial networks ( GANs ; Goodfellow et al . ( 2014 ) ) , which generate images as INRs , show impressive generation performance . Interestingly , they further merit various advantages of INRs , e.g. , natural inter- and extra-polation , anycost inference ( i.e. , control the trade-off of quality and cost ) , and parallel computation , which needs a non-trivial modification to apply under other generative model architectures . Inspired by this , we aim to design an INR-based ( or implicit ) video generation model by interpreting videos as continuous signals . This alternative view is surprisingly effective as INRs compactly encode the videos without 3D grids and naturally model the continuous spatio-temporal dynamics . While naı̈vely applying INRs for videos is already fairly effective , we found that a careful design of separately manipulating space and time significantly improves the video generation . Contribution . We introduce dynamics-aware implicit generative adversarial network ( DIGAN ) , a novel INR-based GAN architecture for video generation . Our idea is two-fold ( see Figure 2 ) : • Generator : We propose an INR-based video generator that decomposes the motion and content ( image ) features , and incorporates the temporal dynamics into the motion features.2 To be specific , our generator encourages the temporal coherency of videos by regulating the variations of motion features with a smaller temporal frequency and enhancing the expressive power of motions with an extra non-linear mapping . Moreover , our generator can create videos with diverse motions sharing the initial frame by conditioning a random motion vector to the content vector . • Discriminator : We propose a motion discriminator that efficiently detects unnatural motions from a pair of images ( and their time difference ) instead of a long sequence of images . Specifically , DIGAN utilizes a 2D convolutional network for the motion discriminator , unlike prior works that utilize computationally heavier 3D convolutional networks to handle the entire video at once . Such an efficient discriminating scheme is possible since INRs of videos can nonautoregressively synthesize highly correlated frames at arbitrary times . We demonstrate the superiority of DIGAN on various datasets , including UCF-101 ( Soomro et al. , 2012 ) , Tai-Chi-HD ( Siarohin et al. , 2019 ) , Sky Time-lapse ( Xiong et al. , 2018 ) , and a food class subset of Kinetics-600 ( Carreira et al. , 2018 ) datasets , e.g. , it improves the state-of-the-art results 2We remark that this decomposition can be naturally done under the INR framework , and one can effectively extend the implicit image GANs for video generation by adding a motion generator . of Fréchet video distance ( FVD ; Unterthiner et al . ( 2018 ) , lower is better ) on UCF-101 from 833 to 577 ( +30.7 % ) . Furthermore , DIGAN appreciates various intriguing properties , including , • Long video generation : synthesize long videos of high-resolution without demanding resources on training , e.g. , 128 frame video of 128×128 resolution ( Figure 1 ) • Time interpolation and extrapolation : fill in the interim frames to make videos transit smoother , and synthesize the out-of-frame videos ( Figure 4 , Table 2 ) • Non-autoregressive generation : generate arbitrary time frames , e.g. , previous scenes ( Figure 5 ) , and fast inference via parallel computing of multiple frames ( Table 3 ) • Diverse motion sampling : sample diverse motions from the shared initial frames ( Figure 6 ) • Space interpolation and extrapolation : upsample the resolution of videos ( Figure 7 , Table 4 ) and create zoomed-out videos while preserving temporal consistency ( Figure 8 ) To the best of our knowledge , we are the first to leverage INRs for video generation . We hope that our work would guide new intriguing directions for both video generation and INRs in the future . 2 RELATED WORK . Image generation . Image generation has achieved remarkable progress , with the advance of various techniques , including generative adversarial networks ( GANs ; Goodfellow et al . ( 2014 ) ) , autoregressive models ( Ramesh et al. , 2021 ) , and diffusion models ( Dhariwal & Nichol , 2021 ) . In particular , GANs have been considered as one of the common practices for image generation , due to the fast inference at synthesizing high-resolution images ( Brock et al. , 2019 ; Karras et al. , 2020b ; 2021 ) . Inspired by recent GAN architectures ( Karras et al. , 2020b ; Skorokhodov et al. , 2021a ) and training techniques , ( Zhao et al. , 2020 ; Karras et al. , 2020a ) , we extend these methods for video generation . Video generation . Following the success of GANs on images , most prior works on video generation considered the temporal extension of image GANs ( Vondrick et al. , 2016 ; Saito et al. , 2017 ; 2020 ; Tulyakov et al. , 2018 ; Acharya et al. , 2018 ; Clark et al. , 2019 ; Yushchenko et al. , 2019 ; Kahembwe & Ramamoorthy , 2020 ; Gordon & Parde , 2021 ; Tian et al. , 2021 ; Fox et al. , 2021 ; Munoz et al. , 2021 ) . Another line of works ( Kalchbrenner et al. , 2017 ; Weissenborn et al. , 2020 ; Rakhimov et al. , 2020 ; Yan et al. , 2021 ) train autoregressive models over pixels or discretized embeddings . Despite their significant achievement , a large gap still exists between generated results and large-scale realworld videos . We aim to move towards longer video generation by exploiting the power of implicit neural representations . We provide more discussion on related fields in Appendix C. Implicit neural representations . Implicit neural representations ( INRs ) have recently gained considerable attention , observing that high-frequency sinusoidal activations significantly improve continuous signal modeling ( Sitzmann et al. , 2020 ; Tancik et al. , 2020 ) . In particular , INR has proven its efficacy in modeling complex signals such as static ( Chen et al. , 2021b ; Park et al. , 2021 ; MartinBrualla et al. , 2021 ) and dynamic ( Li et al. , 2021a ; b ; Pumarola et al. , 2021 ; Xian et al. , 2021 ) 3D scenes , and 2D videos ( Chen et al. , 2021a ) . Unlike prior works on INRs focusing on modeling a single signal , we learn a generative model over 2D videos . Moreover , while prior works focus more on rendering visually appealing 3D scenes , we aim to learn the diverse motion dynamics of videos . Generative models with INRs . Following the success of single signals , several works utilize INRs for generative modeling . One line of works synthesizes the INR weights correspond to the signals via hypernetwork ( Skorokhodov et al. , 2021a ; Anokhin et al. , 2021 ; Chan et al. , 2021 ; Dupont et al. , 2021b ) . The other line of works controls the generated signals via input condition , i.e. , concatenate the latent code corresponding to the signal to the input coordinates ( Schwarz et al. , 2020 ; Kosiorek et al. , 2021 ) . In particular , Skorokhodov et al . ( 2021a ) and Anokhin et al . ( 2021 ) have demonstrated the effectiveness of INR-based generative models for image synthesis . We extend the applicability of INR-based GANs for video generation by incorporating temporal dynamics . 3 DYNAMICS-AWARE IMPLICIT GENERATIVE ADVERSARIAL NETWORK . The goal of video generation is to learn the model distribution pG ( v ) to match with the data distribution pdata ( v ) , where a video v is defined as a continuous function of images it for time t ∈ R , and an image it is a function of spatial coordinates ( x , y ) ∈ R2 . To this end , most prior works considered videos as the discretized sequence of images , i.e. , 3D grid of RGB values of size H ×W × T , and generated videos with discrete decoders such as convolutional ( Tian et al. , 2021 ) or autoregressive ( Yan et al. , 2021 ) networks . However , the discretization limits video generation hard to scale due to the cubic complexity of generated videos and ignores the continuous dynamics . Our key idea is to directly model the videos as continuous signals using implicit neural representations ( INRs ; Sitzmann et al . ( 2020 ) ; Tancik et al . ( 2020 ) ) . Specifically , we propose dynamics-aware implicit generative adversarial network ( DIGAN ) , an INR-based generative adversarial network ( GAN ; Goodfellow et al . ( 2014 ) ) for video generation . Inspired by the success of INR-based GANs for image synthesis ( Skorokhodov et al. , 2021a ; Anokhin et al. , 2021 ) , DIGAN extends the implicit image GANs for video generation by incorporating temporal dynamics . We briefly review INRs and INR-based GANs in Section 3.1 and then introduce our method DIGAN in Section 3.2 .

Summary: The paper proposes a video generation approach based on implicit neural representations (INRs). They construct a "dynamics-aware" GAN; where by the generator model used derives from the coordinate-based models used to capture continuous representations of images in prior  INR work. The discriminator architecture derives from the 2D discriminator traditionally used in image GAN research; they simply expand the input channels from the traditional 3 channels (i.e. rgb) used; to 7 input channels. This discriminator processes a stack of images from one video simultaneously, two frames from different time points in the video and their difference image.