Generating Videos with Dynamics-aware Implicit Generative Adversarial Networks
Authors: Sihyun Yu
Jihoon Tack
Sangwoo Mo
Hyunsu Kim
Junho Kim
Jung-Woo Ha
Jinwoo Shin
Venue: ICLR 2022 Poster (2022)
Keywords: video generation
implicit neural representations
generative adversarial networks

Abstract: In the deep learning era, long video generation of high-quality still remains challenging due to the spatio-temporal complexity and continuity of videos. Existing prior works have attempted to model video distribution by representing videos as 3D grids of RGB values, which impedes the scale of generated videos and neglects continuous dynamics. In this paper, we found that the recent emerging paradigm of implicit neural representations (INRs) that encodes a continuous signal into a parameterized neural network effectively mitigates the issue. By utilizing INRs of video, we propose dynamics-aware implicit generative adversarial network (DIGAN), a novel generative adversarial network for video generation. Specifically, we introduce (a) an INR-based video generator that improves the motion dynamics by manipulating the space and time coordinates differently and (b) a motion discriminator that efficiently identifies the unnatural motions without observing the entire long frame sequences. We demonstrate the superiority of DIGAN under various datasets, along with multiple intriguing properties, e.g., long video synthesis, video extrapolation, and non-autoregressive video generation. For example, DIGAN improves the previous state-of-the-art FVD score on UCF-101 by 30.7% and can be trained on 128 frame videos of 128x128 resolution, 80 frames longer than the 48 frames of the previous state-of-the-art method.

Published as a conference paper at ICLR 2022
GENERATING VIDEOS WITH DYNAMICS- IMPLICIT GENERATIVE ADVERSARIAL NETWORKS
Sihyun Yu"! Jihoon Tack*1, Sangwoo Mo*..,
Hyunsu Kim", Junho Kim", Jung-Woo Ha², Jinwoo Shin' Korea Advanced Institute of Science and Technology (KAIST), 2NAVER Al Lab {sihyun. yu, jihoontack, swmo, (awwoos)kkaitt..... ac. (hyunsu1125.. kim, jhkim.ai, jungwoo. ha hnaveerccoopp
ABSTRACT
In the deep learning era, long video generation of high-quality still remains chal- lenging due to the spatio-temporal complexity and continuity of videos. Existing prior works have attempted model video distribution by representing videos as 3D grids of RGB values, which impedes the scale of generated videos and neglects continuous dynamics. of implicit neural representations (INRs) that encodes a continuous signal into parameterized neural network effectively mitigates the issue. By utilizing INRs of video, propose dynamics-and
this paper, we found that the recent emerging paradigm
implicit generative adversarial network (DI- GAN), a novel generative adversarial network for video generation. Specifically, we introduce (a) INR-based video generator that improves the motion dynam- ics by manipulating the space and time coordinates differently and (b) a motion discriminator that efficiently identifies the unnatural motions without observing the entire long frame sequences. We demonstrate the superiority of DIGAN un- der various datasets, along with multiple intriguing properties, e.g., long video synthesis, video extrapolation, and non- non-autoregressive video generation. For ex- ample, DIGAN improves the previous state-of-the-art FVD score on UCF-101 by 30.7% and can be trained on 128 frame videos of 128 x 128 resolution, 80 frames longer than the 48 frames of the previous state-of- ttateoof-the-an method."
INTRODUCTION
Deep generative models have successfully synthesized realistic samples on various domains, includ- ing image (Brock et al., 2019; Karras et al., 2020b; 2021; Dhariwal & Nichol, 2021), text (Adi- wardana al., 2020; Brown al., 2020), and audio (Dhariwal al., 2020; Lakhotia et al., 2021). Recently, video generation has emerged as the next challenge of deep generative models, and thus a long line of work has been proposed to learn the video distribution (Vondrick al., 2016; Kalch- brenner et al., 2017; Saito et al., 2017; 2020; Tulyakov et al., 2018; Acharya et al., 2018; Clark et al., 2019; Weissenborn et al., 2020; Rakhimov et al., 2020; Tian et al., 2021; Yan et al., 2021).
Despite their significant efforts, substantial gap still exists from large-scale real-world videos. The difficulty of video generation mainly stems from the complexity of vidco signals; they are continu- ously correlated across spatio-temporal directions. Specifically, most prior works interpret the video as a 3D grid of RGB values, i.e., a sequence of 2D images, and model them with discrete decoders such convolutional (Tian et al., 2021) autoregressive (Yan al., 2021) networks. However, such discrete modeling limits the scalability of generated videos due to the cubic complexity (Saito al., 2020) and ignores the inherent continuous temporal dynamics (Gordon & Parde, 2021).
Meanwhile, implicit neural representations (INRs; Sitzmann et al. (2020); Tancik et al. (2020)) have emerged new paradigm for representing continuous signals. INR encodes signal into neural network that maps input coordinates to corresponding signal values, e.g., 2D coordinates of im- ages RGB values. Consequently, INR amortizes the signal values of arbitrary coordinates into compact neural representation instead of discrete grid-wise signal values, requiring a large memory
Videos available at the project site httt:ss:/sihhuueeee github. 1o/digan/. Equul contribution.

---

Published as a conference paper at ICLR 2022
Follow the direction, and the next line at the end
4
5
J
Figure 1: 128 frame video of 128 128 resolution generated by DIGAN on the Tai-Cii-H dataset. DIGAN can train these videos with 4 NVIDIA V100 GPUs, while the prior state-of-the-an method, DVD-GAN, uses more than 32 (up to 512) TPUs for training 48 frame videos of the same resolution.
proportional to coordinate dimension and resolution. In this respect, INRs have shown to be highly effective modeling complex signals such as 3D scenes (Mildenhall et al., 2020; Li et al., 2021a). Furthermore, INR has intriguing properties from its compactness and continuity. e.g., reduced data memory (Dupont et al., 202 a) and upsampling to arbitrary resolution (Chen et al., 2021b).
Several works utilize INRs for generative modeling, i..., samples are generated through INRs (Chan al., 2021; Dupont et al., 2021b; Kosiorek et al., 2021). In particular, Skorokhodov et al. (2021a) and Anokhin et al. (2021) exposed that based image generative adversarial networks (GANs; Goodfellow et al. (2014)), which generate images as INRs, show impressive generation performance. Interestingly, they further merit various advantages of INRs, e-g., natural inter- and estra-polation, anycost inference (i.e., control the trade-off of quality and cost), and parallel computation, which needs non-trivial modification apply under other generative model architectures.
Inspired by this, we aim to design an INR-based (or implicit) video generation model by interpret- ing videos as continuous signals. This alternative view is surprisingly effective as INRs compactly encode the videos without 3D grids and naturally model the continuous spatio-temporal dynamics. While naïvely applying INRs for videos is already fairly effective, we found that careful design of separately manipulating space and time significantly improves the video generation.
Contribution. We introduce dynamics-aware implicit generative adversarial network (DIGAN), novel INR- based GAN architecture for video generation. Our idea is two-fold (see Figure 2):
Generator: We propose an INR-based video generator that decomposes the motion and content (image) features, and incorporates the temporal dynamics into the motion features." 2 To be spe- cific, our generator encourages the temporal coherency of videos by regulating the variations of motion features with a smaller temporal frequency and enhancing the expressive power of mo- tions with an extra non-linear mapping. Moreover, our generator can create videos with diverse motions sharing the initial frame by conditioning a random motion vector to the content vector.
Discriminator: We propose a motion discriminator that efficiently detects unnatural motions from pair of images (and their time difference) instead of long sequence of images. Specif- ically, DIGAN utilizes a 2D convolutional network for the motion discriminator, unlike prior works that utilize computationally heavier 3D convolutional networks to handle the entire video at once. Such an efficient discriminating scheme is possible since INRs of videos can non- autoregressively synthesize highly correlated frames at arbitrary times.
We demonstrate the superiority of DIGAN on various datasets, including UCF-101 (Soomro et al., 2012), Tai-Chi-HD (Siarohin al., 2019), Sky Time-lapse (Xiong et al., 2018), and food class subset of Kinetics-600 (Carreira et al., 2018) datasets, e.g., it improves the state-of-the-art results
We remark that this decomposition can he naturally done under the INR framework, and one can effectively extend the implicit image GANs for video generation by adding a mation generator.
2

---

Published as a conference paper at ICLR 2022
of Fréchet video distance (FVD; Unterthiner et al. (2018), lower is better) UCF-101 from 833 to 577 (+30.7%). Furthermore, DIGAN appreciates various intriguing properties, including.
Long video generation: synthesize long videos of high-resolution without demanding resources on training, e.g., 128 frame video of 128x x 128 resolution (Figure 1)
Time interpolation and extrapolation: fill in the interim frames to make videos transit smoother, and synthesize the out-of out-of-frame videos (Figure 4, Table 2)
Non-anloregressive generation: generate arbitrary time frames, e.g., previous scenes (Figure 5), and fast inference via parallel computing of multiple frames (Table 3)
Diverse motion sampling: sample diverse motions from the shared initial frames (Figure 6) Space interpolation and extrapolation: upsample the resolution of videos (Figure 7, Table 4) and create zoomed-out videos while preserving temporal consistency (Figure 8)
To the best of our knowledge, we are the first to leverage INRs for video generation. We hope that our work would guide new intriguing directions for both video generation and INRs in the future.
2 RELATED WORK
Image generation. Image generation has achieved remarkable progress, with the advance of various techniques, including generative adversarial networks (GANs; Goodfellow al. (2014)), autoregres- sive models (Ramesh et al., 2021), and diffusion models (Dhariwal & Nichol, 2021). In particular, GANs have been considered as one of the common practices for image generation, due to the fast inference at synthesizing high-resolution images (Brock et al., 2019; Karras et al., 2020b; 2021). Inspired by recent GAN architectures (Karras et al., 2020b; Skorokhodov et al., 2021a) and training techniques, (Zhao al., 2020; Karras et al., 2020a), we extend these methods for video generation.
Video generation. Following the success of GANs on images, most prior works on video generation considered the temporal extension of image GANs (Vondrick et al., 2016; Saito et al., 2017; 2020; Tulyakov et al., 2018; Acharya et al., 2018; Clark et al., 2019; Yushchenko et al., 2019; Kahembwe & Ramamoorthy, 2020; Gordon & Parde, 2021; Tian et al., 2021; Fox et al., 2021; Munoz et al., 2021). Another line of works (Kalchbrenner et al., 2017; Weissenborn et al., 2020; Rakhimov et al., 2020; Yan et al., 2021) train autoregressive models over pixels discretized embeddings. Despite their significant achievement, a large gap still exists between generated results and large-scale real- world videos. We aim to move towards longer video generation by exploiting the power of implicit neural representations. We provide more discussion on related fields in Appendix C.
Implicit neural representations. Implicit neural representations (INRs) have recently gained con- siderable attention, observing that high-frequency sinusoidal activations significantly improve con- tinuous signal modeling (Sitzmann et al., 2020; Tancik et al., 2020). particular, INR has proven its efficacy in modeling complex signals such static (Chen et al., 2021 Park et al., 2021; Martin- Brualla et al., 2021) and dynamic (Li al., 2021a;b; Pumarola et al., 2021; Xian et al., 2021) 3D scenes, and 2D videos (Chen et al., 2021a). Unlike prior works on INRs focusing on modeling single signal, we learn a generative model over 2D videos. Moreover, while prior works focus more on rendering visually appealing 3D scenes, aim to learn the diverse motion dynamics of videos.
Generative models with INRs. Following the success of single signals, several works utilize INRs for generative modeling. One line of works synthesizes the INR weights correspond to the signals via hypernetwork (Skorokhodov et al., 2021a; Anokhin et al., 2021; Chan et al., 2021; Dupont et al., 202. The other line of works controls the generated signals via input condition, Le., concatenate the latent code corresponding to the signal to the input coordinates (Schwarz et al., 2020; Kosiorek al., 2021). In particular, Skorokhodov et al. (2021a) and Anokhin et al. (2021) have demonstrated the effectiveness of INR-based generative models for image synthesis. We extend the applicability of INR-based GANs for video generation by incorporating temporal dynamics.
DYNAMICS-AWAN IMPLICIT GENERATIVE ADVERSARIAL NETWORK
The goal of video generation is to learn the model distribution pG (v) to match with the data distri- bution Paana (v), where a video v is defined as a continuous function of images i, for time E R. and an image i, is function of spatial coordinates (x, y) IR.2. To this end, most prior works considered
3

---

Published as a conference paper at ICLR 2022
Content vedtor 2,
Video INR
Video INR
Image 6
Content weight of upper layers
2D grid (2)
G
Content weight of 1 layer My
Dj
/Fake
Real
Time 4
Ey
Mation weightt
myy
2D gnd (0%
Real
Motion vector Pg
Dio
/Fake
Frequendes 0, 20.0
Time 0g
Image 6g
(a) Generator
(b) Discriminator
Figure 2: Illustration of the (a) generator and (b) discriminator of DIGAN. The generator creates a video INR weight from random content and motion vectors, which produces an image that corre- sponds to the input 2D grids { y)} and time t. Two discriminators determine the reality of each image and motion (from a pair of images and their time difference), respectively.
videos as the discretized sequence of images, i.., 3D grid of RGB values of size H x W x T, and generated videos with discrete decoders such as convolutional (Tian et al., 2021) or autoregressive (Yan et al., 2021) networks. However, the discretization limits video generation hard to scale due to the cubic complexity of generated videos and ignores the continuous dynamics.
Our key idea is to directly model the videos continuous signals using implicit neural representa- tions (INRs; Sitzmann et al. (2020); Tancik et al. (2020)). Specifically, we propose dynamics-aware implicit generative adversarial network (DIGAN), an INR- INR-based generative adversarial network (GAN; Goodfellow et al. (2014)) for video generation. Inspired by the success of INR-based GANs for image synthesis (Skorokhodov al., 2021a; Anokhin et al., 2021), DIGAN extends the implicit image GANs for video generation by incorporating temporal dynamics. We briefly review INRs and INR-based GANs in Section 3.1 and then introduce our method DIGAN Section 3.2.
3.1 GENERATIVE MODELING WITH IMPLICIT NEURAL REPRESENTATION
Consider a signal v(:) Rm -> I of a coordinate mapping to the corresponding signal values, egg., videos vzz y, (r, g. b) with m, 3, where (I,, y, f) are space-time coordinates and (r,g, b) are RGB values. Without loss of generality, we assume that the range of coordinates for signals are [0, e.g., [0, 1]8 for videos. INR aims to directly model the signal with neural network v(= Rm + R" parametrized by d, e.g., using a multi-layer perceptron (MLP). Recently, Tancik al. (2020) and Sitzmann et al. (2020) found that sinusoidal activations with a high-frequency input ox, i.e., (ax) ) for a >> 1, significantly improve the modeling of complex signals like 3D scenes (Mildenhall et al., 2020) when applied on the first layer (or entire layers). Here, one can decode the INR v-: 6) as standard discrete grid interpretation of signals by computing the values of pre- defined grid of input coordinates, e.g., {( #=1 W-1+ {=1)} C [0, 1 for videos of size H x W x T.
Leveraging the power of INRs, several works utilize them for generative models, Lee., the generator G(-) maps latent ~ p(z) from given prior distribution p(2) to an INR parameter that corresponds to the generated signal v(:: Φ), e.g., Chan et al. (2021). Remark that INR-based generative models synthesize the function of coordinates, unlike previous approaches that directly predict the outputs of such a function, e.g., 2D grid of RGB values for image synthesis. Hence, IaRebased generative models only need to synthesize fixed size parameter Φ which reduces the complexity of the generated outputs for complicated signals. It is especially important for higher- dimensional signals than 2D images, e.g., video synthesis of grids require cubic complexity.
G(z)
Specifically, Skorokhodov et al. (2021a) and Anokhin et al. (2021) propose GAN frameworks for training INR-based (or implicit) model for image generation, i.e., joint training of the discriminator D(-) to distinguish the real and generated samples, while the generator G(-) aims to fool the discrim- inator:3 3 In particular, they present a new generator to synthesize weights of INRs and employ con- ventional convolutional GAN discriminator architectures to identify the fake images decoded from INRs. Remarkably, they have shown comparable results with of GANs while meriting intriguing properties, e.g., inter- and extra-polation and fast inference (Skorokhodov al., 2021
By gradually improving their abilities from two-player game, the generator converges to the data distribu- tion, iee, PG Rd Piu. where pG is distribution of G(z) for ~ p(2) (Mescheder et al., 2018).
4

---

Published as a conference paper at ICLR 2022
t=1
3
5=5
t=7
:=
t t= 11
t t= 13
t t= 15
UCF-101
1=1 1
1=3 3
t=
t=
t= 9
==
11
t= 13
t= 15
(b) Kinetics-food
Figure 3: Generated video results of DIGAN on UCF-101 and Kinetics-food datasets.
3.2 INCORPORATING TEMPORAL DYNAMICS FOR IMPLICIT GANS
Our method, DIGAN, an INR-based video GAN that incorporates the temporal dynamics into the INRs. Recall that the video INRs only differ from the image INRs by extra time coordinate, ie., an input becomes a 3D coordinate (I, y, t) from a 2D coordinate (I,z); y); hence, one can utilize the implicit image GANs for video generation by only expanding the input dimension of INRs by one (for the time coordinate). However, we found that a careful generator and discriminator design notably improves the generation quality and training efficiency. We provide the overall illustration of DIGAN in Figure 2, and explain the details of DIGAN in the remaining section.
Generator. A naïve extension of implicit image GANs for synthesizing video INRs vaz, y, t; ()) is to utilize them intactly, only modifying the first layer of INRs to handle the extra time coordinate (we assume that the INRs follow the MLP structure). This approach synthesizes the entire INR parameter é at once, which overlooks the difference in space and time of videos, e,g., the smooth change of frames over the time direction.
alleviate this issue, we first notice that the spatial and temporal terms of video INRs can be easily decomposed. Specifically, the output of the first layer (without sinusoidal activations) of video INRs can be interpreted as w + By Wyy + a,w,t + b, where Wx, wy: Wt, b are weights and biases of the first layer and az; Ty: Ot coordinates x; y, f. Here, note that only the term oww,t differs from the image INRs, so the outputs (of the first layer) of video INRs can be viewed frame (at f = 0) determined by the content parameter Φ1 == á \ [w,}. Inspired by this space-time decomposition view, we incorporate the temporal dynamics into the motion parameter wi
are frequencies of
continuous trajectory over time & from the initial
To be specific, we propose three components to improve the motion parameter w. considering the temporal behaviors. First, we use smaller tmmeffreuunccy o, than praceffreueeciis oz, Jy since the video frames change relatively slowly over time compared to the spatial variations covering diverse objects in images. It encourages the videos to coherent over time. Second, we sample latent vector for motion diversity zM ~ Pas (ZM) ) in addition to the original content (image) latent vector 21 ~ pr(2)), Here, the content parameter Φ1 = Gr(2)) is generated as the prior implicit image GANs, but the motion parameter w, G RM) is conditioned motion vectors; since possible motions of a video often depend on the content. Finally, we apply a non-linear mapping (.) on top of the motion features at time t to give more freedoms to motions, i..., fa (w,t). These simple modifications further improves the generation quality (Section 4.1) while creates diverse motions (Section 4.2).
both content and
5

---

Published as a conference paper at ICLR 2022
Table 1: IS, FVD, and KVD values of video generation models on (a) UCF-101, (b) Sky, (c) TaiChi, and (d) Kinetics-food food datasets. 1 and J. imply higher and lower values are better, respectively. Sub- scripts denote standard deviations, and bolds indicate the best results. "Train split" and "Train+test split" denote whether the model is trained with the train split (following the setup in Saito et al. (2020)) or with the full dataset (following the setup Tian al. (2021)), respectively.
(a) UCF-101
(b) Sky
Method
IS
FVD (1)
Method
FVD (L)
KVD (L)
Train split
MoCoGAN-HD
183.6#52
13.9
DIGAN (ours)
114.6:443
6,8
VGAN
8.31 09
TGAN
11.85
MoCoGAN
12.42:1.07
(c) TaiChi
ProgressiveVGAN
14.56.105
Method
FVD (L)
KVD (L)
LDVD-GAN
22.91
VideoGPT
24.69+330
MoCoGAN-HD
144.7:60
25.4
TGANx2
28.87
1209=28 655
DIGAN (ours)
128.1
20.6
DIGAN (ours)
29.71
(d) Kinetics-foom
Train split
Method
FVD (1)
KVD (4)
DVD-GAN
27.38
MoCoGAN-HII
32.36
838
MoCoGAN-HD
430.4:1299
276.0+:507
DIGAN (ours)
32.70=.35
577=21
DIGAN (ours)
313.3
183.0:40.3
Discriminator. As the generated videos should be natural in both images and motions, prior works on video GANs are commonly equipped with two discriminators: an image discriminator D, and motion (or video) discriminator DM (Clark al., 2019; Tian et al., 2021).* For the image dis- criminator, one can utilize the well-known 2D convolutional architectures from the image GANs. However, the motion discriminator needs an additional design, e.g., 3D convolutional networks, where inputs are a sequence of images (e.g.. the entire videos). The 3D convolutional motion dis- criminators are the main computational bottleneck of video GANs, and designing efficient video discriminator has been widely investigated (Saito et al., 2020; Kahembwe & Ramamoorthy, 2020).
Instead of the computationally expensive 3D architectures, tional video discriminator. Here, we emphasize that video INRs can efficiently generate two frames of arbitrary times t1, i2 unlike autoregressive models which require generating the entire sequence. Utilizing this unique property of INRs, we adopt the image discriminator to distinguish the triplet consists of pair of images and their time difference (ir, 1t2, At) for At == it tty by expanding the input channel from 3 to 7. Intriguingly, this discriminator can learn the dynamics without ob- serving the whole sequence (Section 4. 4). Moreover, we note that the two frames of INRs are highly correlated due to their continuity; the discriminator can focus on the artifacts in the motion.
propose efficient 2D convolu-
EXPERIMENTS
We present the setups and main video generation results in Section 1. We then exhibit the intriguing properties of DIGAN in Section 4.2. Finally, we conduct ablation studies in Section 4.3.
4.1 VIDEO GENERATION
Model. We implement DIGAN upon the INR-GAN (Skorokhodov et al., 202 architecture, an basebased GAN for image generation. The content generator G, is identical the INR-GAN gen- erator, but the motion generator Ga added. We set the spatial frequencies og 0g v10 following the original configuration of INR-GAN, but we found that using a smaller value for the temporal frequency Tr performs better; we use o, 0.25 for all experiments ff We use the same discriminator of INR-GAN for both image and motion discriminators but only differ for the input channels: 3 and See Appendix A. for details.
'While video discriminator alone can learn the video distribution (Vondrick al., 2016), recent works often use both diseriminators since such separation empirically performs better (Tulyakov al., 2018).

---

Published as a conference paper at ICLR 2022
Table FVD values of generated videos inter- and extra-polated over time. All models are trained
on 16 frame videos of 128 128 resolution. The videos are interpolated to 64 frames (i.e., 4x finer) and extrapolated 16 more frames. We measure FVD with 512 samples for Sky, since the test data size becomes less than 2,048.
Interpolation
Extrapolation
Method
Sky
TaiChi
Kinetics-food
Sky
TaiChi
Kinetics-food
MoCoGAN-HD
402.2:118.9
249.011227 241.6.17.5
102
303.2:14.3
337.8
877.81226
DIGAN (ours)
32..2.120.5
722.2
224.3
289.3 15.6
69..7.14.11
DIGAN (ours) McCoGAN-HO
t=8 8
t=
16
t= 24
t= 32
t= 40
48
56
1 64
Figure 4: Generated videos of MoCoGAN-HD and DIGAN, trained on 16 frame videos of 128x 128 resolution on the Sky dataset. Yellow box indicates the extrapolated results until 64 frames.
Table 3:
Time (sec) for generating a 128x 128 video for VideoGPT, MoCoGAN-HD, and DIGAN. Bolds indicate the best results.
Real
Generated
Video length
Method
16
32
64
VideoGPT
~00
~
1=3
1=7
MoCoGAN-HD
0.154
0.303
0.612
Figure Forward and backward prediction results of DIGAN. Yellow box indicates the given frame
DIGAN (ours)
0.069
0.132
0.260
Datasets and evaluation. We conduct the experiments on UCF-101 (Soomro et al., 2012). Tai-Chi- HD (TaiChi; Siarohin et al. (2019)), Sky Time-lapse (Sky; Xiong et al. (2018)), and food class subset of Kinetics-600 (Kinetics-foods Carreira et al. (2018)) datasets. All models are trained on 16 frame videos of 128 x 128 resolution unless otherwise specified. Specifically, we use the consecutive 16 frames for UCF-101, Sky, and Kinetics-food, but stride 4 (i.e., skip 3 frames after the chosen frame) for TaiChi to make motion more dynamic. Following prior works, we report the Inception score (IS; Salimans et al. (2016)), Fréchet video distance (FVD; Unterthiner al. (2018), and kernel video distance (KVD; Unterthiner et al. (2018)). We average 10 runs for main results and 5 runs for analysis with standard deviations. See Appendix A.2 for more details.
Baselines. We mainly compare DIGAN with prior works on UCF-101, a commonly used benchmark dataset for video generation. Specifically, we consider VGAN (Vondrick al., 2016), TGAN (Saito et al., 2017), MoCoGAN (Tulyakov et al., 2018), Progressive
(Acharya et al., 2018), LDVD- GAN (Kahembwe & Ramamoorthy, 2020), VideoGPT (Yan et al., 2021), TGANx2 (Saito et al., 2020), DVD-GAN (Clark et al., 2019), and MoCoGAN-HD (Tian et al., 2021). where the values are collected from the references. For other experiments, we compare DIGAN with the state-of-the method, MoCoGAN-HD, tested using the official code. See Appendix B for details.
Main results. Figure and Table present the qualitative and quantitative video generation results of DIGAN, respectively. DIGAN can model various video distributions, including unimodal videos like Sky and TaiChi and multimodal videos like UCF-101 and Kinetics-food. In particular, Figure presents that DIGAN produces reasonably good videos for challenging multimodal videos. Also, Table exhibits that DIGAN significantly outperforms the prior work on all datasets, e.g., improves the FVD of MoCoGAN-HD from 833 to 577 (+30.7%) on UCF-101. We remark that the Fréchet inception distance (FID; Heusel et al. (2017)), a metric for image quality, of DIGAN is similar to the MoCoGAN-HD. Thus, the FVD gains of DIGAN come from better dynamics modeling.

---

Published as a conference paper at ICLR 2022
Mation
Motion
Difference
1= 3
9
==
15
Figure 6: Videos sampled from two random motion vectors. The first two rows are generated videos, and the third row is the pixel difference between the videos (yellow implies more differences).
Table 4: FVD values of videos upsam- pled from 128x 128 to 256 reso- lution x larger) on TaiChi dataset.
Method
FVD (L)
Nearest
180.6=3.1
Bilinear
236.7=6.7
Bicubic
175.9
Video
Nearest
Bilinear
Bicubic
DIGAN (ours)
Figure 7: Videos upsampled from 128x 128 to 512x512 resolution (4x larger) on TaiChi dataset.
DIGAN (ours)
156.7=6.2
4.2 INTRIGUING PROPERTIES
Long video generation. The primary advantage of DIGAN an effective generation of long and high-quality videos, leveraging the compact representations of INRs. We also remark that DIGAN can be efficiently trained on the long videos, as the motion discriminator of DIGAN only handles pair of images instead of long image sequences. To verify the efficacy of DIGAN, we train model using 128 frame videos 128x 128 resolution from the TaiChi dataset. Figure shows that DIGAN produces long and natural motion with reasonable visual quality. To our best knowledge, we are the first to report 128 frame videos of this quality. See Appendix F for more results.
Time interpolation and extrapolation. DIGAN can easily interpolate (i.., fill in interim frames) or extrapolate (i.e., create out-of-frame videos) videos over time by controlling input coordinates. The videos inter- extra-polated by DIGAN more natural than those from discrete generative mod- els, as INRs model videos continuously. Table shows that DIGAN outperforms MoCoGAN-HD on all considered inter- and extra- potta-polation scenarios. In particular, DIGAN is remarkably effective for time extrapolation as INRs regularize the videos smoothly follow the scene flows defined by previous dynamics. Figure 4 shows that DIGAN can even extrapolate 4x longer frames while MoCoGAN-HD fails on the Sky dataset. See Appendix G for more results.
Non-autoregressive generation. DIGAN can generate samples of arbitrary time: it enables DIGAN to infer the past or interim frames from future frames or parallelly compute the entire video at once. is impossible for many prior video generation approaches that sample the next frame conditioned on the previous frames in an autoregressive manner. Figure 5 visualizes the forward and backward prediction results of DIGAN on the TaiChi dataset, conditioned on the initial frame indicated by the yellow box. Here, we find the content and motion latent codes by projecting the initial frames of t = (6, 7, and predict the frames of E {3...., 11). DIGAN well predicts both past and future motions, e,g., slowly lowering arms. DIGAN can also infer the interim frames from past and future frames, as shown in Appendix E.2. On the other hand, Table 3 shows the generation speed of DIGAN is much faster than its competitors, VideoGPT and MoCoGAN-HI works, we remark that DIGAN can compute multiple frames in parallel; generation can be N times further faster under N number of GPUs. For more analysis about the efficiency, Appendix I.
Different from prior
8

---

Published as a conference paper at ICLR 2022
(=1 1=6 1=11
16
[-1 3-3 1 - 1
13
Figure 8: Zoomed-out samples. Red boxes indicate the original frames.
Figure 9: Samples linearly interpolated INR weights
di over X, i.e. (1 A)Φ1 + XΦ2 TaiChi dataset.
Table 5: Ablation study of the generator compo- nents: smaller frequency oN motion vector 2M. and non-linearity by MLP f-)()).
D logits over conditioned time differencrs
Logits D (rea))
1.9
2.4g
1.8
2.7
Freq.
Motion Za
MLP fa(())
FVD (L)
3.3
33.
686#25
1.0
0.6
0.4
0.0
Time diference conditioned the direiminattr
640.22
/
585=27
Figure 10: Discriminator logits for far image
577
pair (4g, &1), conditioned over different At.
Diverse motion sampling. DIGAN can generate diverse videos from the initial frame by controlling the motion vectors. Figure 6 shows the vidcos from two random motion vectors on the Sky dataset. Note that the shape of clouds moves differently, but the tree in the left below stays. The freedom of variations conditioned on the initial frame depends on datasets, as discussed in Appendix D.
Space interpolation and extrapolation. DIGAN also inter- and extra-polate videos in the space directions. Figure 7 and Table show the qualitative and quantitative results for space interpolation (i.e., upsampling) the TaiChi dataset. DIGAN produces 4x higher resolution videos without ad-hoc training tricks, outperforming naïve heuristics such as bicubic interpolation. Figure 8 visual- izes the space extrapolation (i.e., zoom-out) results on various datasets. DIGAN creates out-of-box scenes while preserving temporal consistency. See Appendix H for more results.
INR weight interpolation. Interpolation of INR parameter sampled from DIGAN produces seman- tically meaningful videos. Figure 9 visualizes the videos from the linearly interpolated INR param- eters on the TaiChi dataset. The videos smoothly vary over interpolation, e.g., cloth color changes from white to blue. It is not obvious as the INR weights lie in a structured, high-dimensional space.
4.3 ABLATION STUDIES
We conduct the ablation studies on the components of DIGAN. Table 5 shows that all the proposed generator components: smaller frequency o,, motion vector 2M and non-linear MLP mapping far contribute to the generation performance measured by FVD scores on the UCF-101 dataset. We note that the motion vector ΣM and MLP fas( (-) remarkably affect FVD when applied solely, the all- combination result is saturated. On the other hand, Figure 10 verifies that the motion discriminator considers the time difference At a given pair of images. Specifically, we provide a far image pair (io, i1), i.e. the first and the last frames, with the time difference At. The motion discriminator thinks the triplet (ép, é1, At) is fake At and real At ss 1, as the real difference Namely, the discriminator leverages At identify whether the input triplets real or fake.
1.
5 CONCLUSION
We proposed DIGAN, an implicit neural representation (INR) based generative adversarial network (GAN) for video generation, incorporating the temporal dynamics of videos. Extensive experiments verified the superiority of DIGAN with multiple intriguing properties. We believe our work would guide various directions in video generation and INR research in the future.

---

Published as a conference paper at ICLR 2022
ETHICS STATEMENT
Video generation has potential threats of creating videos for unethical purposes, e.g., fake propa- ganda videos of politicians or sexual videos of any individuals. The generated videos, often called DeepFake, have arisen an important social problem (Westerlund, 2019). To tackle the issue, there have been enormous efforts in detecting fake videos (e.g.. Guera Delp (2018)). Here, we claim that the generation and detection techniques should be developed in parallel, rather than prohibit- ing the generation research itself. This is because the advance technology is inevitable, and such prohibition only promotes the technology hide in the dark, making them hard to pick out.
In this respect, the generative adversarial network (GAN) a neat solution as it naturally trains both generator and discriminator (or detector). Notably, the discriminator trained by GAN can effectively detect the fake samples created by other generative models (Wang et al., 2020). Our proposed video generation method, DIGAN, is also built upon the GAN framework. In particular, a discriminator that identifies the fake videos without observing long sequences. We believe our proposed discriminator would be step towards designing an efficient DeepFake detector.
introduced
REPRODUCIBILITY STATEMENT
We describe the implementation details of the model in Appendix A. , and details of the datasets and evaluation in Appendix A.2. We also provide our code in the supplementary material.
ACKNOWLEDGMENTS AND DISCLOSURE OF FUNDING
SY thanks Jaeho Lee, Younggyo Seo, Minkyu Kim, Soojung Yang. Seokhyun Moon, Hin-Hwa& Kim, and anonymous reviewers for their helpful feedbacks the early version of the manuscript. SY also acknowledges Ivan Skorokhodov for providing the implementation of INR-GAN. This work was mainly supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2021-0-02068, Artificial Intelligence Artificial Intelligence Graduate School Program (KAIST)). This
Innovation Hub; No.2019-0-0007 work was partly experimented on the NAVER Smart Machine Learning (NSML) platform (Sung al., 2017; Kim et al., 2018). This work was partly supported by KAIST-NAVER Hypercreative Al Center.
REFERENCES
Dinesh Acharya, Zhiwu Huang, Danda Pani Paudel, and Lue Van Gool. Towards high resolu- tion video generation with progressive growing of sliced Wasserstein GANs. arXiv preprint arXiv:1810.024199 2018.
Daniel Adiwardana. Minh-Thang Luong, David R So. Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gauray Nemade, Yifeng Lu, et al. Towards a human-like open- domain chatbot. arXiv preprint a7Xiv:2001.09977, 2020.
Ivan Anokhin, Kirill Demochkin, Taras Khakhulin, Gleb Sterkin, Victor Lempitsky, and Denis Ko- rzhenkov. Image generators with conditionally-nndeeenden pixel synthesis. In IEEE Conference on Computer Vision and Pattern Recognition, 2021.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International Conference on Machine Learning, 2017.
Mohammad Babaeizadeh, Chelsea Finn, Dumitru Erhan, Roy Campbell, and Sergey Levine. Stochastic variational video prediction. In International Conference on Learning Representa- fions, 2018.
Mohammad Babacizadeh, Mohammad Taghi Saffar, Suraj Nair, Sergey Levine, Chelsea Finn, and Dumitru Erhan. FitVid: Overfitting pixel-level video prediction. arXiv preprint arXiv:2106. 13195, 2021.
10

---

Published as a conference paper at ICLR 2022
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019.
Tom Brown, Benjamin Mann, Nick Ryder. Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranay Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems, 2020.
Joao Carreira, Eric Noland, Andras Banki-Horvath, Chloe Hillier, and Andrew Zisserman. A short note about Kinetics-600. arXiv preprint arXiv: 01340, 2018.
Eric R Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein. Pi-GAN: Periodic implicit generative adversarial networks for 3D-aware image synthesis. In IEEE Conference on Computer Vision and Pattern Recognition, 2021.
Hao Chen, Bo He, Hanyu Wang, Yixuan Ren, Ser Nam Lim, and Abhinav Shrivastava. NeRV: Neu- ral representations for videos. Advances in Neural Information Processing Systems, 34, 202
Yinbo Chen, Sifei Liu, and Xiaolong Wang. Learning continuous image representation with local implicit image function. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021b.
Aidan Clark, Jeff Donahue, and Karen Simonyan. Adversarial video generation on complex datasets.
arXiv preprini arXiv: orXiv:1907.000 (en::0907007777 06671, 2019.
Emily Denton and Vighnesh Birodkar. Unsupervised learning of disentangled representations from video. In Advances in Neural Information Processing Systems, 2017.
Emily Denton and Rob Fergus. Stochastic video generation with a learned prior. In International Conference on Machine Learning. 2018.
Terrance DeVsies and Graham W Taylor. Improved regularization of convolutional neural networks with Cutout. arXiv preprini
2017.
Prafulla Dhariwal and Alex Nichol. Diffusion models beat GANs on image synthesis. arXiv preprint arXiv:2105.05233, 2021.
Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. Jukebox: A generative model for music. arXiv preprint arXiv:2005.0034, 2020.
Emilien Dupont, Adam Golinski, Milad Alizadeh, Yee Whye Teh, and Amaud Doucet. COIN: Compression with implicit neural representations. arXiv preprint arXiv:2103.03/23, 2021a.
Emilien Dupont, Yee Whye Teh, and Arnaud Doucet. Generative models distributions of func- tions. Xiviv preprint arXiv:2 2021b.
Chelsea Finn, Ian Goodfellow, and Sergey Levine. Unsupervised learning for physical interaction through video prediction. In Advances Neural Information Processing Systems, 2016.
Gereon Fox, Ayush Tewari, Mohamed Elgharib, and Christian Theobalt. StyleViceoGAN: A tem- poral generative model using a pretrained StyleGAN. arXiv preprint arXiv:2107.07224 2021.
Jean-Yves Franceschi, Edouard Delasalles, Mickaël Chen, Sylvain Lamprier, and Patrick Gallinari. Stochastic latent residual video prediction. In International Conference Machine Learning, 2020.
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 2014.
Cade Gordon and Natalic Parde. Latent neural differential equations for video generation. In NeurIPS 2020 Workshop on Pre-registration in Machine Learning, 2021.

---

Published as a conference paper at ICLR 2022
David Guera and Edward J. Delp. Deepfake video detection using recurrent neural networks. IEEE International Conference on Advanced Video and Signal Based Surveillance, 2018.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.
GANs trained by a two time-scale update rule converge a local nash equilibrium. In Advances in Neural Information Processing Systems, 2017.
Jun-Ting Hsieh, Bingbin Liu, De-An Huang, Li Fei-Fei, and Juan Carlos Niebles. Learning to de- compose and disentangle representations for video prediction. In Advances in Neural Information Processing Systems, 2018.
Emmanuel Kahembwe and Subramanian Ramamoorthy, Lower dimensional kernels for video dis- criminators. Neural Networks, 132:506-520, 2020.
Nal Kalchbrenner, Aäron Oord, Karen Simonyan, Ivo Danihelka, Oriol Vinyals, Alex Graves, and Koray Kavukcuoglu. Video pixel networks. In International Conference on Machine Learning, 2017.
Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei- Fei. Large-scale video classification with convolutional neural networks. In IEEE Conference Computer Vision and Pattern Recognition, 2014.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. International Conference on Learning Representations, 2017.
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. In Advances in Neural Information Processing Systems, 2020a.
Tero Karras, Samuli Laine, Mika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of StyleGAN. In JEEE Conference Computer Vision and Pattern Recognition, 2020b.
Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. arXiv preprint arXiv:2 12423, 2021.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv:1705.06950 2017.
Hanjoo Kim, Minkyu Kim, Dongjoo Sco, Jinwoong Kim, Heungseok Park. Soeun Park, Hyunwoo Jo, KyungHyun Kim, Youngil Yang, Youngkwan Kim, et al. Nsml: Meet the MLaaS platform with a real-world case study, arXiv preprint arXiv:1810.0999 2018.
Adam R Kosiorek, Heiko Strathmann, Daniel Zoran, Pol Moreno, Rosalia Schneider, Soña Mokrá, and Danilo J Rezende. NeRF-VAE: A geometry aware 3D scene generative model. International Conference on Machine Learning, 2021.
Manoj Kumar, Mohammad Babaeizadeh, Dumitru Erhan, Chelsea Finn, Sergey Levine, Laurent Dinh, and Durk Kingma. Videoflow: A conditional flow-hased model for stochastic video gener- ation. In International Conference on Learning Representations, 2020.
Kushal Lakhotia, Evgeny Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexci Baevski, Adelrahman Mohamed, et al. Generative spoken language modeling from audio. arXiv preprint arXiv:2 2021.
Alex X Lee, Richard Zhang, Frederik Ebert, Pieter Abbeel, Chelsea Finn, and Sergey Levine. Stochastic adversarial video prediction. arXiv preprint arXiv:1804.01523 2018.
Wonkwang Lee, Whie Jung, Han Zhang, Ting Chen, Jing Yu Koh, Thomas Huang, Hyungsuk Yoon, Honglak Lee, and Seunghoon Hong. Revisiting hierarchical approach for persistent long-term video prediction. In International Conference on Learning Representations, 2021.
12

---

Published as a conference paper at ICLR 2022
Tianye Li, Mira Slavcheva, Michael Zollhoefer, Simon Green, Christoph Lassner, Changil Kim, Tan- ner Schmidt, Steven Lovegrove, Michael Goesele, and Zhaoyang Lv. Neural 3D video synthesis. arXiv preprint arXfv:2103.02599, 2021a.
Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space- time view synthesis of dynamic scenes. In IEEE Conference on Computer Vision and Pattern Recognition, 2021b.
Yuchen Liu, Zhixin Shu, Yijun Li, Zhe Lin, Federico Perazzi, and Yun-Yuan Kung. Content-aware GAN compression. In JEEE Conference on Computer Vision and Pattern Recognition, 2021.
Pauline Luc, Aidan Clark, Sander Dieleman, Diego de Las Casas, Yotam Doron, Albin Cassirer, and Karen Simonyan. Transformation- -aaaed adversarial video prediction on large-scale data. arXiv preprint arXiv:2003. 04035, 2020.
Ricardo Martin Bruuala, Noha Radwan, Mehdi SM Sajjadi, Jonathan T Barron, Alexey Dosovit- skiy, and Daniel Duckworth. NeRF in the wild: Neural radiance fields for unconstrained photo collections. In IEEE Conference on Computer Vision and Pattenn Recognition, 2021.
Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for GANs do actually converge? In International Conference on Machine Learning, 2018.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes neural radiance fields for view synthesis. European Conference on Computer Vision, 2020.
Andres Munoz, Mohammadreza Zolfaghari, Max Argus, and Thomas Brox, Temporal shift GAN for large scale video generation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2021.
Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Martin-Brualla. Deformable neural radiance fields. In IEEE International Conference on Computer Vision, 2021.
Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francese Moreno-Noguer. D-NeRF: Neu- ral radiance fields for dynamic scenes. In IEEE Conference Computer Vision and Pattern Recognition, 2021.
Ruslan Rakhimov, Denis Volkhonskiy, Alexey Artemov, Denis Zorin, and Evgeny Burnaev. Latent video transformer. arXiv preprint 10X7022006.10704 2020.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. arXiv preprint arXiv:2102. 12092, 2021.
Masaki Saito, Eiichi Matsumoto, and Shunta Saito. Temporal generative adversarial nets with sin- gular value clipping. In IEEE International Conference on Computer Vision, 2017.
Masaki Saito, Shunta Saito, Masanori Koyama, and Sosuke Kobayashi. Train sparsely, generate densely: Memory--fficient unsupervised training of high-resolution temporal GAN. International Journal of Computer Vision 128:2586-2600 2606, 2020.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training GANs. Advances Neural Information Processing Systems, 2016.
Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger. GRAF: Generative radiance fields for 3D-aware image synthesis. In Advances in Neural Information Processing Systems, 2020.
Aliaksandr Siarohin, Stéphane I Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. First order motion model for image animation. In Advances in Neural Information Processing Systems, 2019.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, 2020.
13

---

Published as a conference paper at ICLR 2022
Ivan Skorokhodov. Savva Ignatyev, and Mohamed Elhoseiny. Adversarial generation of continuous images. In IEEE Conference on Computer Vision and Pattern Recognition, 2021a.
Ivan Skorokhodov, Sergey Tulyakov, and Mohamed Elhoseiny. StyleGAN-V: A continuous video generator with the price, image quality and perks of StyleGAN2. arXiv preprint arXiv:21 14683, 2021b.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. UCF101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv::212.00000 2012.
Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov. Unsupervised learning of video representations using LSTMs. In International Conference on Machine Learning, 2015.
Nako Sung, Minkyu Kim, Hyunwoo Jo, Youngil Yang, Jingwoong Kim, Leonard Lausen, Youngk- wan Kim, Gayoung Lee, Donghyun Kwak, Jung-Wer Woo Ha, and Sunghun Kim. NSML: machine learning platform that enables you to focus on your models. In arXiv, 2017.
Christian Szegedy, Vincent Vanhoucke. Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In IEEE Conference on Computer Vision and Pattern Recognition, 2016.
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. arXiv preprint arXiv:2006.1077 2020.
Yu Tian, Jian Ren, Menglei Chai, Kyle Olszewski, Xi Peng, Dimitris N Metaxas, and Sergey Tulyakov. A good image generator is what you need for high-resolution video synthesis. In International Conference on Learning Representations, 2021.
Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spa- tiotemporal features with 3D convolutional networks. In IEEE International Conference on Com- puter Vision, 2015.
Sergey Tulyakov. Ming- Yu Liu, Xiaodong Yang, and Jan Kautz. MoCoGAN: Decomposing motion and content for video generation. In IEEE Conference on Computer Vision and Pattern Recogni- tion, 2018.
Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphael Marinier, Marein Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. arXiv preprint 11200:181..01717, 2018
Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learn- ing. In Advances in Neural Information Processing Systems, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems, 2017.
Ruben Villegas, Jimei Yang, Seunghoon Hong, Xunyu Lin, and Honglak Lee. Decomposing mo- tion and content for natural video sequence prediction. In International Conference Learning Representations, 2017.
Ruben Villegas, Arkanath Pathak, Harini Kannan, Dumitru Erhan, Quoe V Le, and Honglak Lee. High fidelity video prediction with large stochastic recurrent neural networks. In Advances in Neural Information Processing Systems, 2019.
Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dynamics. In Advances in Neural Information Processing Systems, 2016.
Sheng- Yu Wang, Oliver Wang, Richard Zhang, Andrew Owens, and Alexei A Efros. -Ne-generated images are surprisingly easy to spot... for now. IEEE Conference on Computer Vision and Pattern Recognition, 2020.
14

---

Published as a conference paper at ICLR 2022
Dirk Weissenborn, Oscar Täckström, and Jakob Uszkoreit, Scaling autoregressive video models. In International Conference on Learning Representations, 2020.
Mika Westerlund. The emergence of deepfake technology: A review. Technology Innovation Man- agement Review, 9:40-53, 11/2019 2019.
Wenqi Xian, Jia-Bin Huang, Johannes Kopf, and Changil Kim. Space-time neural irradiance fields for free-viewpoint video. In IEEE Conference on Computer Vision and Pattern Recognition, 2021.
Wei Xiong, Wenhan Luo, Lin Ma, Wei Liu, and Jiebo Luo. Learning to generate time-lapse videos using multi-stage dynamic generative adversarial networks. In IEEE Conference on Computer Vision and Pattern Recognition, 2018.
Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. VideoGPT: Video generation using VQ-VAE and transformers. arXiv preprint arXiv:2104110157 2021.
Vladyslav Yushchenko, Nikita Araslanov, and Stefan Roth. Markov decision process for video gen- eration. Proceedings of the IEEE/CVE International Conference on Computer Vision Work- shops, 2019.
Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, and Song Han. Differentiable augmentation for data-efficient GAN training. Advances in Neural Information Processing Systems, 2020.
15

---

Published as a conference paper at ICLR 2022
A IMPLEMENTATION DETAILS
A.1 MODEL DETAILS
Following the original configuration of INR-GAN (Skorokhodov et al., 2021a), we set the same spatial frequencies og = By = v10 and StyleGAN2 (Karras et al., 2020b) discriminator. We use small temporal frequency d₁ 0.25 encourage temporal coherence. Here, we remark that the number of frames used in most experiments (16) is much smaller than the image resolution (128), which also affects choosing the Ø* In this respect, choosing a larger a, (but still smaller than ax, Ty) can boost the performance for longer videos, e.g., we use d; = 0.5 for training on 128 frame videos.
In all experiments, we sample At == jt tzl by subtracting the values from two different beta distributions ( ~ Beta(2, 1), /2 ~ (eta 2) for both real and generated videos. Such distributions can sample AL. diversely (e.g., measured with 10,000 samples, 36.5%, 42.7%, and 20.8% of A/ is in the interval of [0, 31 [ 3). and 1 1]. respectively), and indeed worked well in our experiments.
Like in INR-GAN, we also use a progressive multi-layer perception (MLP) with a factorized mul- tiplicative modulation layers as implicit neural representation architecture. For non-linear mapping fas (-), we use the 2-layer MLP with leaky ReLU activation and, no bias are applied for motion vectors. We also apply DiffAug (Zhao et al., 2020) to mitigate overfitting from the limited number of videos like in Tian al. (2021). Specifically, we use all augmentations proposed from DiffAug except CutOut Vrees & Taylor, 2017), and use the same augmentation to the same video. All other hyperparameters are identical to StyleGAN2 except for RS regularization coefficient y: we use y 1 in all experiments. With these setups. all the experiments processed with 4 NVIDIA V100 32GB GPUs where it takes at most ~4.4 days to complete.
We note that standard generative adversarial networks (GANs; Goodfellow et al. (2014)) and implicit GANs are in a dual relation as the former samples the input latent while the latter samples the network parameters, which are combined to compute the final outputs. However, the generator of implicit GAN resembles the one of StyleGAN2 in practice since StyleGAN2 injects the input latent to the intermediate layers through the mapping network and modulates the weights with them.
A.2 DATASET AND EVALUATION DETAILS
Datasets. In what follows, we describe datasets that we used for the evaluation of our method. All videos are first pre-processed to video clips of consecutive 16 frames, unless otherwise specified.
UCF-101 (Soomro et al., 2012) is a 101-class video action dataset total of 13,320 videos with 320x240 resolution. Each video clip center-cropped to 240x240 and resized into 128x 128 resolution. We conducted two different experiments for a fair comparison: training the model with the train split of 9,357 videos or with all 13,320 videos without the split (following the setup of prior state-of-the-an art baselines (Clark et al., 2019; Tian et al., 2021)).
TDD (Siarohin al., 2019) is video dataset total of 280 long videos of people doing Tai-Chi. We use the official link for downloading and cropping the dataset as the rectangle videos of 128 128 resolution. We use 16 frame video clips of stride 4 (i.e., skip 3 frames after the chosen frame) for dynamic motion. We use all data without the split on training. We exclude the following videos due to the broken link: 8xSkbMUpegs, RCiy2FYViEg, 1MMbu9-Mejc, ceoe2f2648u, 6jHyn420KLk, VMSqVTE90hk, XmwGRXYO FEE, DnOmNZmAh2k,, VhprHat04dk, KYdyIdusDOg, EaEzVfhn07e, _1745tFFMCO, ytT4iu7h-AS A8, SujMrSyHO_8, JdiIQg47Wec aAwbJ and _XRyc2kiTIM.
Sky Tame-lapse (Xiong et al., 2018) is a collection of sky time-lapse total of 5,000 videos. We use the same data pre-processing following the official link. We use the train split for training the model and test split for the evaluation, following the setups in prior works.
Kinetics-600 (Carreira et al., 2018) is large-scale 600-class video action dataset, consists of a total of 495,547 videos. We sub-sampled a food subclass in the dataset to train the model, where we follow the list of such a subclass from Weissenbom et al. (2020), namely: (baking,
'tttps: http:/::/ggtttbooo/// com A1 wtwaaaaaiiaarraiinnnaaoaaaaaaaaa orttrrrrrrtiittttore------- Phttps:// // thub. com wei xiong-ur/ndga
16

---

Published as a conference paper at ICLR 2022
barbequing, breading, cooking, cutting, pancake, vegetables, meat, cake, sandwich, pizza, sushi, tea, peeling, fruit, eggs, and salad. We use train split for the model training and use the validation set for the evaluation. Note that only use these classes, different from Weissenborn et al. (2020) to train the model with the whole dataset.
Evaluation metries. We follow the prior setups for evaluation for fair comparison. We use the C3D network (Tran et al., 2015) pre-trained on Sports- (MMMaaa (Karpathy et al., 2014) and fine-tuned on UCF-101 datasets for the Inception score (IS; Salimans et al. (2016)). We use the official TGAN (Saito et al., 2017) implementation for computing IS: the score is evaluated over 10,000 generated videos. 7 We note this metric does not use the Inception network (Szegedy et al., 2016) have been used in evaluating IS in image generation, but only the formula for evaluation is identical.
For Fréchet video distance (FVD; Unterthiner et al. (2018)), and kernel video distance (KVD; Un- terthiner al. (2018)), use the I3D network trained Kinetics-400 (Kay al., 2017). All eval- uations are done by averaging 10 runs scores computed from 2,048 sampled real and generated videos, following the setup in TGANv2 (Saito et al., 2020).
Forward and backward prediction. For the forward and backward prediction in Figure 5, one should project the given frame into the latent code. To this end, we follow StyleGAN2 projection procedure Karras et al. (2020a) and optimize for 20,000 iteration.
Generation time. To fairly compare the generation time with the baselines in Table 3, we utilize the same machine and stop other processes. We used Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz and Titan XP GPU for the measurement.
Space extrapolation. We generate the video by 2D grid of range [-0.25, 1.25
B BASELINES
In this section, we explain video generation baselines we used for evaluating DIGAN at a high level.
VGAN (Vondrick et al., 2016) extends image generative adversarial network (GAN; (Goodfel- low et al., 2014)) by replacing 2D spatial architecture with 3D spatio-temporal architecture.
TGAN (Saito et al., 2017) separates the spatial (image) generator and temporal (latent dynamics) build on Wasserstein GAN (Arjovsky et al., 2017) model for image generation.
MoCoGAN (Tulyakov al., 2018) proposes GAN disentangle videos by the content of single latent and motion of stochastic latent trajectory and generate these two components.
Progressive VGAN (Acharya et al., 2018) progressively generates videos both in spatial and temporal directions, like pregressive image generation in ProgressiveGAN (Karras et al., 2017).
LDVD-GAN (Kahembwe & Ramamoorthy, 2020) proposes efficient video discriminator for training video GANs by considering its kernel dimension to low.
VideoGPT (Yan al., 2021) compresses videos as sequence of discrete latent vectors via vector-quantized variational auto-encoder (van den Oord et al., 2017), then trains autoregressive Transformer model (Vaswani al., 2017) with these latent sequences.
TGANx2 (Saito al., 2020) designs new video GAN of computationally efficient generator and discriminator by proposing sub-modules on each component.
DVD-GAN (Clark et al., 2019) uses spatial and temporal discriminators, where the input of the temporal discriminator is spatially down-sampled to reduce the computational bottleneck.
MoCoGAN-HD (Tian al., 2021) leverages a pre-trained image generator to additionally train a motion generator on the image latent space to synthesize the video with those two generators.
thub.
17

---

Published as a conference paper at ICLR 2022
C ADDITIONAL RELATED WORK
Video prediction. A related but distinct area, video prediction, aims to forecast future video frames from early frames (Srivastava al., 2015; Finn al., 2016; Denton & Birodkar, 2017; Babaeizadeh Villegas et al., 2019; Kumar et al., 2020;
et al., 2018; Denton & Fergus, 2018; Lee et al., 2018;
Franceschi al., 2020; Luc et al., 2020; Lee et al., 2021). By restricting the distribution conditioned on initial frames, video prediction models often show better visual quality than video generation (Babaeizadeh et al., 2021). Also, video prediction models employ image-to-ime they only need to modify the initial frames regarding the motion, while video generation models use teent-to-image architectures. While our primary focus video generation, one can utilize our model for video prediction by projecting initial frames into the corresponding latent codes.
architectures as
Video synthesis with motion and content decomposition. For better video generation or prediction with controllability, several works have proposed methods decompose the motion and contents. Specifically, they model videos with a content vector and a sequence of motion vectors from different subspaces, incorporating distinct motion and content encoders for prediction (Villegas al., 2017; Hsieh et al., 2018) or generators for a generation (Tulyakov et al., 2018; Tian et al., 2021; Munoz al., 2021). Similarly, utilize generators, but the motion is represented the weight of implicit neural representations rather than sequence of latents.
single vector of
Concurrent work. concurrent work, StyleGAN-V (Skorokhodov et al., 2021b), also focuses on generative modeling of videos by interpreting videos as continuous signals. Moreover, it shares many similar ideas with DIGAN, e.g., motion discriminator without 3D convolution networks. However, it has core difference to DIGAN; DIGAN treats videos as spatiotemporally continuous signals, while StyleGAN-V V extends StyleGAN2 (Karras et al., 2020b) by interpreting videos as temporally continuous signals (but not spatially continuous), i.e., video frames are discrete 2D grids.
18

---

Published as a conference paper at ICLR 2022
D LATENT DYNAMICS OF MOTION FEATURES
Resolution 32
Resolution
Resolution
UCF-101 Resolution 64
Resolution 32
Resolution 128
(b) Sky
Resolution
Resolution
Resolution
8
(c) TaiChi
Figure 11: Latent trajectories of the motion features on UCF-101, Sky, and TaiChi datasets. Resolu- tion denotes the location of motion features injected into the progressive generator; lower-resolution controls the high-level semantics, and higher-resolution controls the low-level variations. Dot colors gradually changes from blue (t 0) red (t = 1), and 5 random motion vectors are sampled. The features are projected onto 2D space via principal component analysis (PCA) for visualization.
Figure 11 visualizes the latent dynamics of motion features on various datasets. Note that the free- dom of motion features vary for different datasets: UCF-101 allows high-level (resolution 32) and low-level (resolution 128) variations, Sky only allows low-level variations, and TaiChi does not permit the variations much. Intuitively, the next move of TaiChi is mostly determined by the prior frames: the movement follows the pre-defined gestures. Sky permits some variations as shown in Figure 6, but the wind direction determines the global motion. UCF-101 has the largest freedom of motion as the dataset contains videos various actions and diverse objects.
19

---

Published as a conference paper at ICLR 2022
E MORE EXAMPLES ON VIDEO PREDICTION
MORE BACKWARD PREDICTION E.1
Real
Real
Generated
Generated
1=3
t= 5
1=7
t=
[=
11
1=3
t= 5
1=7
1=
9
1=11
Figure 12: More examples on forward and backward prediction results of DIGAN. Yellow box indicates the given frame.
E.2 INTERMEDIATE SCENE PREDICTION
Real
Generated
Real
Generated
Real
Generated
1=3
1=5
7
1= t= 13
1=
Figure 13: Intermediate scene (i.e., frame) prediction results of our method, DIGAN, The yellow boxes indicate the given initial and last frames.
We show that our proposed method, DIGAN, can predict the intermediate frames between the two frames the different time steps. To this end, we find the inverse content and motion latent codes by projecting two given frames into the generator (we follow the StyleGAN2 projection procedure (Karras et al., 2020b) and optimize 20,000 iteration for the projection). As shown in Figure 13, DIGAN can well predict the intermediate dynamics the given frames are somewhat far apart. This result implies that DIGAN indeed has learned the dynamics of the ground truth distribution.
20

---

Published as a conference paper at ICLR 2022
Real
StyleGAN2
DIGAN (ours)
t=1 t=
t=5
t=7
t=
t=
t= 13
15
Figure 14: Comparison of intermediate scene (i.e., frame) prediction results by StyleGAN2 and our method, DIGAN. The yellow boxes indicate the given initial and last frames.
Table 6: SSIM of intermediate scenes predicted by StyleGAN2 and method, DIGAN.
SSIM (1)
StyleGAN2 latent interpolation DIGAN (ours)
0.5639
0.6753
We also report structural similarity index measure (SSIM) to quantitatively compare predicted videos from DIGAN and linearly interpolated image sequences in StyleGAN2 latent space. For StyleGAN2, use the official StyleGAN2 inversion method, namely, we project each of the initial and final frames to the StyleGAN2 latent space trained on the Taichi dataset. After that, we lin- early interpolate on these two projected latent vectors. As shown in Table 6. DIGAN shows better prediction than StyleGAN2 interpolation. Moreover, as shown in Figure 14, one can observe that DIGAN preserves semantics like background across the temporal direction, while the StyleGAN2 latent interpolation cannot.
21

---

Published as a conference paper at ICLR 2022
F MORE EXAMPLES FOR LONG VIDEO GENERATION
Follow arrow direction, move to next at the
Figure 15: Additional 128 frame videos of 128x128 resolution by DIGAN, on the TaiChi dataset.
22

---

Published as a conference paper at ICLR 2022
MORE EXAMPLES FOR TIME EXTRAPOLATION G
McCoG
DIGAN (ours)
1=1
24
32
Figure 16: Additional time extrapolation videos of MoCoGAN-HD and DIGAN, trained on 16 frame videos of 128 x 128 resolution on the Sky dataset. Yellow box indicates the extrapolated frames.
H MORE EXAMPLES FOR SPACE POLATION
UCF-101
Sky
Taichi
Kinetics-Fie food
t=1
t=3 3
t=5
t=
15
Figure 17:
Additional zoomed-out samples. Red boxes indicate the original frames.
23

---

Published as a conference paper at ICLR 2022
1 EFFICIENCY OF DIGAN
In addition to the computational efficiency on inference (in Table 3), we provide the analysis of our method on efficiencies of diverse aspects, including computation, energy, memory, and time efficiency. Here, we mainly compare our method with MoCoGAN-HD (Tian et al., 2021), which is an energy-efficient (i.e., shorter GPU days for training) and quality, All experiments are performed under the same machine (NVIDIA V100 32GB GPUs).
method on the generation
Computational efficiency. We report floating point operations per second (FLOPS)* of video GAN generators (following Liu al. (2021)). The DIGAN generator requires 147.9 GFLOPS to generate 16 frame video of 128x 128 resolution, 4.6 times smaller than 682.3 GFLOPS of the MoCoGAN-HD generator.
Energy efficiency on training (GPU days for training). The FVD value of DIGAN trained on UCF-101 for 8 GPU days achieves 689-224, which is better than 838 of MoCoGAN-HD trained for 16 GPU days. This implies that DIGAN is (at least times) more energy-efficient than MoCoGAN-HD.
Memory efficiency on training. We report the memory size per GPU to load 16 frame videos of 128 x 128 resolution of batch size for training video GANs. DIGAN allocates 9.7GB memory, 2.9 times smaller than 28.0GB of MoCoGAN-HD.
Time efficiency on inference. Per each GPU, DIGAN discriminators can (forward) compute 195.6 video clips/sec, 3.2 times larger than 60.3 video clips/sec of MoCoGAN-HD discrimina- tors with 128 frame videos of 128x x 128 resolution and batch size of 16. It confirms that both the generator and discriminator of our method are much more time-efficient; recall that we already demonstrated that our generator is 2.3 times faster per each GPU in Table 3.
In summary, DIGAN is more computation-effiiiin times), memory-efficient on training (> 2.9 times), and time-efficient on inference (> 3.2 times) un- der each machine setup above. Such gaps stem from the fact that our method utilizes: (a) INR-based generator without consuminnm autoregressive modeling and (b) 2D convolutional discriminator rather than computationally heavy 3D convolutional discriminator. Finally, emphasize that DI- GAN can synthesize multiple frames of a video in parallel (which is impossible under prior autore- gressive or convolutional network-based methods), resulting in N times improved time-efficiency generate a single video under N number of GPUs. It can be particularly important for synthesizing extremely long videos; since it requires tremendous time if utilizing prior methods that are impossi- ble to compute frames in parallel. Like this, the superiority of computational efficiency of DIGAN can be further dramatically improved in the presence of multiple GPUs.
(> 4.6 times), energy on training (> 2
EFFECT OF CONTENT VECTOR FOR GENERATING MOTION
Table Effect of the motion vector 21 for generating the motion. We report the mean and standard deviation of the FVD values over 10 runs
UCF-101
TaiChi
Motion only Motion + content (ours)
596
147.9149
577=21
128.1444
We provide the result when only the motion vector is used as input (i.e., without the content vector); we report the FVD values of the models on the UCF-101 and the TaiChi dataset. The result in Table 7 confirms that the consideration of content vectors improves the result, as video motions often depend on the content (i.e., initial frame determines the possible future motions).
'https://gi thtbe////ggtbtuuuoeeee com http://c///cccccooeeeeereee Fvcore
24

---

Published as a conference paper at ICLR 2022
K CLASS-CONDITIONAL GENERATION OF DIGAN
Table 8: IS, FVD, and KVD values video generation models on the UCF-101 dataset. f and imply higher and lower values better, respectively. Subscripts denote standard deviations, and bolds indicate the best results.
Method
IS (t)
FVD ()) KVD (L)
DVD-GAN
32.97.1.7
TSB
42.79
DIGAN (ours)
59.68
465=12 39.6=29
We provide both qualitative and quantitative results of DIGAN on a class-conditional setup to com- pare several recent works on video synthesis that reported class-conditional generation results, such as DVD-GAN (Clark et al., 2019) and TSB (Munoz et al., 2021). Specifically, we report class- conditional generation results trained on UCF-101 in Table 8 and the project page," ? which demon- strate the superiority of DIGAN even on the conditional setup compared to existing baselines.
Phttps://si hatp::::////hyuuyyuuuuu thub. i /
25


Reviews:
[{"review_id": "j1ITy-ANVAh", "reviewer": "Reviewer_Tj68", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## General Thoughts\nThe paper is interesting and the method propose definitely shows promise. Applying INRs to this task seems like an excellent direction, and avoids many computational problems of video generation. I would have loved to see a more compelling set of visual examples to highlight the possibilities of the technique, and a potential comparison to other works that have trained on larger portions of Kinetics-600.\n\n## Strengths\n- The proposed alternative has several positive poperties, such as non-autoregessive generation.\n- Decomposition of spatial and temporal factors in the INR\n- Good variety in datasets tested\n- Strong improvements in quantitative metrics\n- INRs seem like a very good approach for this task, love the concept\n\n## Weaknesses\n- A few typos and writing could be improved (last paragraph of pages 4 and 5 for example)\n- Would love to see an ablation study showing that including the motion + content vector as input to the motion generator (instead of just the motion vector) improves performance\n- The idea of comparing two frames and their time difference is nice, but its power is only truly leveraged if many different $\\Delta t$ are shown to the discriminator - is this the case? How are you handling this during training?\n- Would love to see more results of the effects of sampling different motion vectors - Figure 6 is not compelling enough to believe the claim that the presented motion vector setup is beneficial. There is only one example in the anonymized webpage.\n- Why was only TaiChi used for the interpolation task? This technique should handle interpolation much better than its counterparts, and showing more results there seems like a winning move. Why were the other datasets omitted?\n- Fig 5 is not very compelling, would modify or add more examples. Would have loved to see more examples in the appendix or webpage.\n- Comparisons to MoCoGAN-HD's 1024x1024 videos would have been nice, to really test the extrapolation abilities of INRs\n\n\n## Questions\n- What is Lambda in figure 9? Can't find it in the main text\n- Figure 10: compelling, but I'm left to wonder if the discriminator does this even with a fake pair of frames: For a fake video, when the frames are far apart, we'd like the discriminator to still say that the input is fake, right?\n- Discriminator input channels increased from 3 to 7 -> why do this instead of siamese nets to compare the two frames? \n\n\n", "summary_of_the_paper": "This paper leverages the implicit neural representations paradigm to build generative adversarial networks for video generation. Implicit neural representations encode continuous signals into parametrized neural networks, and the authors claim that this mitigates the issue related to the inefficient modeling of videos as 3d tensors of RGB values. Their GAN includes an INR-based video generator, and a motion discriminator that is more efficient at identifying unnatural motions. Their approach yields FVD improvements in various existing datasets.", "summary_of_the_review": "UPDATE: The provided rebuttal answers most of my concerns, and my confidence about the strength of the paper has increased. I am updating my recommendation to 8 - accept.\n\nThe paper discusses a very relevant concept in video generation: utilizing INRs to improve performance and efficiency. The authors show strong quantitative results and a few architectural novelties. The work is solid, but there is a worrying lack of compelling qualitative examples to back up certain claims. If the authors can provide more examples, I believe this could be a strong addition to ICLR.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "b58Y0uH08xA", "reviewer": "Reviewer_ErkP", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper presents a good submission, is well written, well motivated and evaluated. \n\nStrengths:\n1. Idea of representing videos as continuous functions. I believe this is an important idea not addressed in the past\n2. The idea of temporally conditioned video discriminator. A heavy weight discriminator is a typical problem in video synthesis, as it makes training of the whole pipeline longer. I wonder if a similar discriminator can be used to make previous models be able to generate longer sequences.  \n3. Results and applications. The reported results show significant numerical improvements over the state-of-the-art methods. What's more interesting, similar to other INR methods, the present work can extrapolate videos in space and time. It can also do video in beetwening. \n\nWeaknesses:\n1. The generator part is adaptation of the INR-GAN to the temporal domain. The extension is simple.\n2. The paper doesn't discuss the limitations. I wonder if the method can be trained on extremely long sequences, or perhaps can generate a couple of tens of seconds at inference time? The provided website contain a limited of 2 seconds long examples and two more 4 seconds long examples, which look like interpolation. \n3. Consider two frames of a video, for example, when the hand is up and the other one when the hand is down. Since the action can be performed fast (or slow), there can be arbitrary many (or few) frames between these two frames. It's not clear if this adds any difficulties for the video discriminator, since the conditioning can be seen as incorrect. Could the authors comment?\n4. I wonder if the authors thought of any metric that show that the proposed motion is motion indeed, rather than interpolation between two or more frames. For example, in Fig. 12 the generated results look like interpolations between two frames. See the example on the top, the right hand grows, not moves. One could have an experiment, in which indeed an interpolation in StyleGAN space is made and then compared to these results.\n\nUPDATE:\n\nI read the authors' response. It seems to me, that both the generator and discriminator work because of the relative simplicity of commonly used benchmarks for video synthesis. It means that a dataset can be well described by a pair of frames sampled many times from the data.   UCF-101 is a way more challenging dataset (too challenging) for current video synthesis methods. This is, however, not an issue of this paper but of the field in general. \n\nIt seems odd to me, that authors showed only 2 examples of long video generation on TaiChi (where the method seems to work well) and 100 examples on UCF-101 where the method works very poorly, generates repetitive motions, generates videos that don't make sense (yet better than state-of-the-art). How representative are the two examples? I had the same question in my initial review. The authors had two 4-second examples, now they have 2 8-second example which doesn't bring much difference.\n\nSo I believe my initial evaluation was correct, the paper is slightly more on the positive side of the bar.", "summary_of_the_paper": "The paper present a new method to video generation. They built on top of the INR-GAN framework and extend it to temporal domain. They, therefore, get continuous time and space image generation--something all other video methods lack (technically one should be able to interpolate motion codes of existing CNN-based works, but spatially none of them is continuous to my knowledge). They model a video as a function $f(x,y,t)$. They condition this functions on motion codes and on content codes. The upper layers get information from the content, the lower layers get information from both content and motion. They further introduce a discriminator that is conditioned on the time difference between frames. It allows them to train on longer videos. \n", "summary_of_the_review": "I believe, the paper is a good piece of work and is above the bar.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "AjMrYejTN9b", "reviewer": "Reviewer_vhpn", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1. This work proposes a simple extension of INRs to video generation.\nINRs have thus far remained in the realm of image generation. To my knowledge, this is the first work to extend this approach to video.\nIn hindsight, this may seem like a trivial extension, but the introduction of INRs to video generation provides a solution to many hard problems in video generation work.\n\n2. As a result of being the first work to present INRs as a viable method for tackling video generation, it benefits from the efficiency advantages inherent to using INRs for videos, which include but are not limited to: long horizon video generation (can predict up 128 frames in future at a resolution of 128x128), time interpolation and extrapolation (can predict interim frames and also predict far out into the future), non-autoregressive generation (can generate frames for specific time points of interest without generating intermediate frames), space interpolation and extrapolation (can up sample and zoom out of generated video)\n\n3. The work is well motivated, written and benchmarks across all the standard video generation datasets. It also includes the relevant ablation studies. It is also timely, given the need for more efficient video GAN models.\n\n\n\nWeaknesses:\n1. This work suffers from one fatal flaw. Some of its experiments train the proposed model on the full dataset, not the training split as in prior work, yet it presents results comparing performance to models trained on only the training split of the dataset. I have to admit that I missed this as a reviewer of the prior work that was submitted and accepted to ICLR last year, but since it was not explicitly stated, it was assumed that the prior work followed proper experimentation procedure when benchmarking against prior art. Particularly for UCF101, the results where models train on the full 13320 videos should be explicitly differentiated from models that train on the \"trainlist01\" training split of UCF101 comprised of only 9537 videos. To allow for honest and accurate comparison to the majority of prior work, I would strongly encourage the authors to benchmark models trained on the training data split for the related datasets, in particular, the \"trainlist01\" split of the UCF101 dataset. They should also highlight and mark (e.g. star) results that come from models trained on a full dataset, that compare their results to prior art that only trains on a split of that dataset.\n\n2. This work is lacking when it comes to experiments comparing the compute, memory, time or energy efficiency of the proposed approach. This would greatly strengthen the proposed contribution since it is so naturally amenable to great gains in efficiency across all four metrics.", "summary_of_the_paper": "The paper proposes a video generation approach based on implicit neural representations (INRs). They construct a \"dynamics-aware\" GAN; where by the generator model used derives from the coordinate-based models used to capture continuous representations of images in prior  INR work. The discriminator architecture derives from the 2D discriminator traditionally used in image GAN research; they simply expand the input channels from the traditional 3 channels (i.e. rgb) used; to 7 input channels. This discriminator processes a stack of images from one video simultaneously, two frames from different time points in the video and their difference image.", "summary_of_the_review": "This is good work, a simple idea extended from the image to the video domain. It suffers a fatal flaw that prevents me from recommending it for acceptance. \n\nIn its current form, this paper is \"marginally below the acceptance threshold\". If the issues with benchmarking are resolved, then I will recommend it for acceptance. If additional experiments exploring the efficiency of the proposed approach are provided, then I will strongly recommend it for acceptance.\n\n\n------\n\n***EDIT****\n\nThe authors have sufficiently addressed all of my concerns (and those of the other reviewers it seems).\nI strongly recommend this paper for acceptance to ICLR22, it is a meaningful step in a promising direction for long-horizon GAN-based video generation.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7HCIFG0umhF", "reviewer": "Reviewer_3svg", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well-written, easy to follow, and provides experiments on multiple datasets.\n#### Strengths\n- The paper provides a comparison to recent methods on IS, FVD, and KVD evaluation metrics. This helps to understand better the performance of the introduced model on video generation.\n- Introducing an INR-based design for long-term video generation seems a good idea.\n- Non-autoregressive generation. This is the feature I like the most. Being able to generate arbitrary time or predict past or future frames. \n\n#### Weaknesses\n - Decomposing signal into motion and content is interesting however this is not new and previously studied by MoCoGAN[1] and Temporal Shift GAN [2] (which is not cited in the paper). We don't see any discussion in the related works on decomposition and similarity between works. \n- Although extending INR-based image generator to video generator is interesting, it's a trivial modification and not significantly novel.\n-  The paper states that \"the proposed generator encourages the temporal coherency of videos by regulating the\nvariations of motion features and enhancing the expressive power of motions with an extra nonlinear mapping\", however, this is not very clear that how is this done in the design (explicitly or implicitly)? Is there any specific term to control the temporal coherency of video in the generator? How do you \"regulate variations of motion features\"? I couldn't find the details in the paper.\n- There is no information on computational complexity, training time, and the number of GPUs used for training. \n- I really don't think that the discriminator design works for long-term video generation. Without seeing a good amount of video frames how do you expect the discriminator to evaluate the action? let's assume that in the long-term video a person doing something in the first 4 seconds and then goes out of the scene and another person starts doing something else! Then only two frames and temporal difference can't help the generator to create better motions. I think this discriminator works if the video is short and there is no repetition or high similarity in the video frames. To evaluate, a good experiment would be to generate random videos from UCF101 dataset and try to evaluate the performance with the metric introduced in [3] or a similar metric (S3) used in [2]. Or at least qualitatively show long videos from UCF101.\n- Missing ref [4]\n\n\n\n#### Ref:\n1-MoCoGAN: Decomposing Motion and Content for Video Generation, Sergey Tulyakov, Ming-Yu Liu, Xiaodong Yang, Jan Kautz\\\n2-Temporal Shift GAN for Large Scale Video Generation, Andres Munoz, Mohammadreza Zolfaghari, Max Argus, Thomas Brox\\\n3-Classification Accuracy Score for Conditional Generative Models, Suman Ravuri, Oriol Vinyals\\\n4-StyleVideoGAN: A Temporal Generative Model using a Pretrained StyleGAN, Gereon Fox, Ayush Tewari, Mohamed Elgharib, Christian Theobalt", "summary_of_the_paper": "The paper introduces an INR-based design that utilizes an MLP network to encode spatio-temporal dynamics of video. The authors also propose an efficient discriminator to detect unnatural motions. The design is an extension of INR-GAN (Skorokhodov et al., 2021) architecture. The paper claims that the method obtains SOTA performance on the UCF101 dataset. ", "summary_of_the_review": "Video generation is a very hard problem and the authors try to introduce a more efficient approach for this problem. I believe INR-based video generation is a valuable extension of the image generator but this is not significant. Also, the experiments cannot fully support the design choices for the discriminator.  Additionally, the paper misses some related works (and discussion on differences) and details on computational complexity and training time. I don't think the paper is ready to be accepted to ICLR. However, I'm open to discussions and will consider the author's response.\n\nUpdate:\nThe author's response address almost all my questions, however, I'm not convinced about the discriminator choice and the author's explanation is not clear and convincing. This is also confirmed by their qualitative results on long videos such as the UCF101 video (64 frames). The result is much worse than DVD-GAN (Clark et al 2019). \nI believe this work has positive aspects in terms of efficiency and ability to generate frames in parallel. Therefore, I'm increasing my rating from reject to \"marginally below the acceptance threshold\".", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work tackles video generation using implicit representations, and demonstrates that using these representations enables improvements to long-term coherence of the generated videos.

Reviewers praised the writing, the thorough experimental evaluation, and the strong quantitative results. Some concerns were raised about a lack of discussion of relevant related work, novelty/significance, model architecture, and a lack of qualitative examples, many of which the authors have tried to address during the discussion phase. Several reviewers raised their ratings as a result.

Personally I certainly believe that exploring implicit representations for video is important, and I know of no published prior work in this direction, which amplifies the potential significance of this work. Even if results are qualitatively worse than previous work in some ways, this exploration is still valuable and worth publishing.

While the paper ultimately received one reject rating, another reviewer chose to champion this work and award it the highest possible rating. Combined with the other positive reviews, this provides plenty of convincing evidence for me to recommend acceptance. That said, given the rating spread, I would like to encourage the authors to consider the reviewers' comments further as they prepare the final version of the manuscript. Especially providing more qualitative results would be a welcome addition.
Decision: Accept (Poster)