Generating Videos with Dynamics-aware Implicit Generative Adversarial Networks

1 INTRODUCTION . Deep generative models have successfully synthesized realistic samples on various domains , including image ( Brock et al. , 2019 ; Karras et al. , 2020b ; 2021 ; Dhariwal & Nichol , 2021 ) , text ( Adiwardana et al. , 2020 ; Brown et al. , 2020 ) , and audio ( Dhariwal et al. , 2020 ; Lakhotia et al. , 2021 ) . Recently , video generation has emerged as the next challenge of deep generative models , and thus a long line of work has been proposed to learn the video distribution ( Vondrick et al. , 2016 ; Kalchbrenner et al. , 2017 ; Saito et al. , 2017 ; 2020 ; Tulyakov et al. , 2018 ; Acharya et al. , 2018 ; Clark et al. , 2019 ; Weissenborn et al. , 2020 ; Rakhimov et al. , 2020 ; Tian et al. , 2021 ; Yan et al. , 2021 ) . Despite their significant efforts , a substantial gap still exists from large-scale real-world videos . The difficulty of video generation mainly stems from the complexity of video signals ; they are continuously correlated across spatio-temporal directions . Specifically , most prior works interpret the video as a 3D grid of RGB values , i.e. , a sequence of 2D images , and model them with discrete decoders such as convolutional ( Tian et al. , 2021 ) or autoregressive ( Yan et al. , 2021 ) networks . However , such discrete modeling limits the scalability of generated videos due to the cubic complexity ( Saito et al. , 2020 ) and ignores the inherent continuous temporal dynamics ( Gordon & Parde , 2021 ) . Meanwhile , implicit neural representations ( INRs ; Sitzmann et al . ( 2020 ) ; Tancik et al . ( 2020 ) ) have emerged as a new paradigm for representing continuous signals . INR encodes a signal into a neural network that maps input coordinates to corresponding signal values , e.g. , 2D coordinates of images to RGB values . Consequently , INR amortizes the signal values of arbitrary coordinates into a compact neural representation instead of discrete grid-wise signal values , requiring a large memory proportional to coordinate dimension and resolution . In this respect , INRs have shown to be highly 1Videos are available at the project site https : //sihyun-yu.github.io/digan/ . effective at modeling complex signals such as 3D scenes ( Mildenhall et al. , 2020 ; Li et al. , 2021a ) . Furthermore , INR has intriguing properties from its compactness and continuity , e.g. , reduced data memory ( Dupont et al. , 2021a ) and upsampling to arbitrary resolution ( Chen et al. , 2021b ) . Several works utilize INRs for generative modeling , i.e. , samples are generated through INRs ( Chan et al. , 2021 ; Dupont et al. , 2021b ; Kosiorek et al. , 2021 ) . In particular , Skorokhodov et al . ( 2021a ) and Anokhin et al . ( 2021 ) exposed that INR-based image generative adversarial networks ( GANs ; Goodfellow et al . ( 2014 ) ) , which generate images as INRs , show impressive generation performance . Interestingly , they further merit various advantages of INRs , e.g. , natural inter- and extra-polation , anycost inference ( i.e. , control the trade-off of quality and cost ) , and parallel computation , which needs a non-trivial modification to apply under other generative model architectures . Inspired by this , we aim to design an INR-based ( or implicit ) video generation model by interpreting videos as continuous signals . This alternative view is surprisingly effective as INRs compactly encode the videos without 3D grids and naturally model the continuous spatio-temporal dynamics . While naı̈vely applying INRs for videos is already fairly effective , we found that a careful design of separately manipulating space and time significantly improves the video generation . Contribution . We introduce dynamics-aware implicit generative adversarial network ( DIGAN ) , a novel INR-based GAN architecture for video generation . Our idea is two-fold ( see Figure 2 ) : • Generator : We propose an INR-based video generator that decomposes the motion and content ( image ) features , and incorporates the temporal dynamics into the motion features.2 To be specific , our generator encourages the temporal coherency of videos by regulating the variations of motion features with a smaller temporal frequency and enhancing the expressive power of motions with an extra non-linear mapping . Moreover , our generator can create videos with diverse motions sharing the initial frame by conditioning a random motion vector to the content vector . • Discriminator : We propose a motion discriminator that efficiently detects unnatural motions from a pair of images ( and their time difference ) instead of a long sequence of images . Specifically , DIGAN utilizes a 2D convolutional network for the motion discriminator , unlike prior works that utilize computationally heavier 3D convolutional networks to handle the entire video at once . Such an efficient discriminating scheme is possible since INRs of videos can nonautoregressively synthesize highly correlated frames at arbitrary times . We demonstrate the superiority of DIGAN on various datasets , including UCF-101 ( Soomro et al. , 2012 ) , Tai-Chi-HD ( Siarohin et al. , 2019 ) , Sky Time-lapse ( Xiong et al. , 2018 ) , and a food class subset of Kinetics-600 ( Carreira et al. , 2018 ) datasets , e.g. , it improves the state-of-the-art results 2We remark that this decomposition can be naturally done under the INR framework , and one can effectively extend the implicit image GANs for video generation by adding a motion generator . of Fréchet video distance ( FVD ; Unterthiner et al . ( 2018 ) , lower is better ) on UCF-101 from 833 to 577 ( +30.7 % ) . Furthermore , DIGAN appreciates various intriguing properties , including , • Long video generation : synthesize long videos of high-resolution without demanding resources on training , e.g. , 128 frame video of 128×128 resolution ( Figure 1 ) • Time interpolation and extrapolation : fill in the interim frames to make videos transit smoother , and synthesize the out-of-frame videos ( Figure 4 , Table 2 ) • Non-autoregressive generation : generate arbitrary time frames , e.g. , previous scenes ( Figure 5 ) , and fast inference via parallel computing of multiple frames ( Table 3 ) • Diverse motion sampling : sample diverse motions from the shared initial frames ( Figure 6 ) • Space interpolation and extrapolation : upsample the resolution of videos ( Figure 7 , Table 4 ) and create zoomed-out videos while preserving temporal consistency ( Figure 8 ) To the best of our knowledge , we are the first to leverage INRs for video generation . We hope that our work would guide new intriguing directions for both video generation and INRs in the future . 2 RELATED WORK . Image generation . Image generation has achieved remarkable progress , with the advance of various techniques , including generative adversarial networks ( GANs ; Goodfellow et al . ( 2014 ) ) , autoregressive models ( Ramesh et al. , 2021 ) , and diffusion models ( Dhariwal & Nichol , 2021 ) . In particular , GANs have been considered as one of the common practices for image generation , due to the fast inference at synthesizing high-resolution images ( Brock et al. , 2019 ; Karras et al. , 2020b ; 2021 ) . Inspired by recent GAN architectures ( Karras et al. , 2020b ; Skorokhodov et al. , 2021a ) and training techniques , ( Zhao et al. , 2020 ; Karras et al. , 2020a ) , we extend these methods for video generation . Video generation . Following the success of GANs on images , most prior works on video generation considered the temporal extension of image GANs ( Vondrick et al. , 2016 ; Saito et al. , 2017 ; 2020 ; Tulyakov et al. , 2018 ; Acharya et al. , 2018 ; Clark et al. , 2019 ; Yushchenko et al. , 2019 ; Kahembwe & Ramamoorthy , 2020 ; Gordon & Parde , 2021 ; Tian et al. , 2021 ; Fox et al. , 2021 ; Munoz et al. , 2021 ) . Another line of works ( Kalchbrenner et al. , 2017 ; Weissenborn et al. , 2020 ; Rakhimov et al. , 2020 ; Yan et al. , 2021 ) train autoregressive models over pixels or discretized embeddings . Despite their significant achievement , a large gap still exists between generated results and large-scale realworld videos . We aim to move towards longer video generation by exploiting the power of implicit neural representations . We provide more discussion on related fields in Appendix C. Implicit neural representations . Implicit neural representations ( INRs ) have recently gained considerable attention , observing that high-frequency sinusoidal activations significantly improve continuous signal modeling ( Sitzmann et al. , 2020 ; Tancik et al. , 2020 ) . In particular , INR has proven its efficacy in modeling complex signals such as static ( Chen et al. , 2021b ; Park et al. , 2021 ; MartinBrualla et al. , 2021 ) and dynamic ( Li et al. , 2021a ; b ; Pumarola et al. , 2021 ; Xian et al. , 2021 ) 3D scenes , and 2D videos ( Chen et al. , 2021a ) . Unlike prior works on INRs focusing on modeling a single signal , we learn a generative model over 2D videos . Moreover , while prior works focus more on rendering visually appealing 3D scenes , we aim to learn the diverse motion dynamics of videos . Generative models with INRs . Following the success of single signals , several works utilize INRs for generative modeling . One line of works synthesizes the INR weights correspond to the signals via hypernetwork ( Skorokhodov et al. , 2021a ; Anokhin et al. , 2021 ; Chan et al. , 2021 ; Dupont et al. , 2021b ) . The other line of works controls the generated signals via input condition , i.e. , concatenate the latent code corresponding to the signal to the input coordinates ( Schwarz et al. , 2020 ; Kosiorek et al. , 2021 ) . In particular , Skorokhodov et al . ( 2021a ) and Anokhin et al . ( 2021 ) have demonstrated the effectiveness of INR-based generative models for image synthesis . We extend the applicability of INR-based GANs for video generation by incorporating temporal dynamics . 3 DYNAMICS-AWARE IMPLICIT GENERATIVE ADVERSARIAL NETWORK . The goal of video generation is to learn the model distribution pG ( v ) to match with the data distribution pdata ( v ) , where a video v is defined as a continuous function of images it for time t ∈ R , and an image it is a function of spatial coordinates ( x , y ) ∈ R2 . To this end , most prior works considered videos as the discretized sequence of images , i.e. , 3D grid of RGB values of size H ×W × T , and generated videos with discrete decoders such as convolutional ( Tian et al. , 2021 ) or autoregressive ( Yan et al. , 2021 ) networks . However , the discretization limits video generation hard to scale due to the cubic complexity of generated videos and ignores the continuous dynamics . Our key idea is to directly model the videos as continuous signals using implicit neural representations ( INRs ; Sitzmann et al . ( 2020 ) ; Tancik et al . ( 2020 ) ) . Specifically , we propose dynamics-aware implicit generative adversarial network ( DIGAN ) , an INR-based generative adversarial network ( GAN ; Goodfellow et al . ( 2014 ) ) for video generation . Inspired by the success of INR-based GANs for image synthesis ( Skorokhodov et al. , 2021a ; Anokhin et al. , 2021 ) , DIGAN extends the implicit image GANs for video generation by incorporating temporal dynamics . We briefly review INRs and INR-based GANs in Section 3.1 and then introduce our method DIGAN in Section 3.2 .

Summary: The paper present a new method to video generation. They built on top of the INR-GAN framework and extend it to temporal domain. They, therefore, get continuous time and space image generation--something all other video methods lack (technically one should be able to interpolate motion codes of existing CNN-based works, but spatially none of them is continuous to my knowledge). They model a video as a function $f(x,y,t)$. They condition this functions on motion codes and on content codes. The upper layers get information from the content, the lower layers get information from both content and motion. They further introduce a discriminator that is conditioned on the time difference between frames. It allows them to train on longer videos.