A Good Image Generator Is What You Need for High-Resolution Video Synthesis
Authors: Yu Tian
Jian Ren
Menglei Chai
Kyle Olszewski
Xi Peng
Dimitris N. Metaxas
Sergey Tulyakov
Venue: ICLR 2021 Spotlight (2021)
Keywords: high-resolution video generation
contrastive learning
cross-domain video generation

Abstract: Image and video synthesis are closely related areas aiming at generating content from noise. While rapid progress has been demonstrated in improving image-based models to handle large resolutions, high-quality renderings, and wide variations in image content, achieving comparable video generation results remains problematic. We present a framework that leverages contemporary image generators to render high-resolution videos. We frame the video synthesis problem as discovering a trajectory in the latent space of a pre-trained and fixed image generator. Not only does such a framework render high-resolution videos, but it also is an order of magnitude more computationally efficient. We introduce a motion generator that discovers the desired trajectory, in which content and motion are disentangled. With such a representation, our framework allows for a broad range of applications, including content and motion manipulation. Furthermore, we introduce a new task, which we call cross-domain video synthesis, in which the image and motion generators are trained on disjoint datasets belonging to different domains. This allows for generating moving objects for which the desired video data is not available. Extensive experiments on various datasets demonstrate the advantages of our methods over existing video generation techniques. Code will be released at https://github.com/snap-research/MoCoGAN-HD.

Published as a conference paper at ICLR 2021
A GOOD IMAGE GENERATOR IS WHAT YOU NEED FOR HIGH-RESOLUTION VIDEO SYNTHESIS
Yu Tian's ; Jian Ren", Menglei Chaif, Kyle Olszewski", Xi Peng',
Dimitris N. Metaxas', Sergey Tulyakov² 'Rttgers University, SSnap Inc., University of Delaware {yt219, rutgers. edu, (iren, mchai, kolszewski, Soullyakrr@snapchhee com
ABSTRACT
Image and video synthesis are closely related areas aiming at generating content from noise. While rapid progress has been demonstrated in improving image- based models to handle large resolutions, high-quality renderings, and wide vari- ations in image content, achieving comparable video generation results remains problematic. We present a framework that leverages contemporary image gen- erators to render high-resolution videos. We frame the video synthesis problem as discovering trajectory in the latent space of a pre-trained and fixed image generator. Not only does such a framework render high-resolution videos, but it also is an order of magnitude more computationally efficient. We introduce mo- tion generator that discovers the desired trajectory, in which content and motion are disentangled. With such representation, our framework allows for broad range of applications, including content and motion manipulation. Furthermore, we introduce new task, which call cross doos-nn video synthesis, in which the image and motion generators are trained on disjoint datasets belonging to different domains. This allows for generating moving objects for which the desired video data is not available. Extensive experiments on various datasets demonstrate the advantages of our methods over existing video generation techniques. Code will be released at
INTRODUCTION
Video synthesis seeks to generate a sequence of moving pictures from noise. While its closely related counterpart-image synthesis- seen substantial advances in recent years, allowing for synthesizing high resolutions (Karras et al., 2017). rendering images often indistinguishable from real ones (Karras et al. 2019)) and supporting multiple classes of image content (Zhang et al.) 20099), contemporary improvements in the domain video synthesis have been comparatively modest. Due to the statistical complexity of videos and larger model sizes, video synthesis produces relatively low-resolution videos, yet requires longer training times. For example, scaling the image generator of Brock et al. (2019) to generate 256 x 256 videos requires substantial computational budge [ Can we use a similar method to attain higher resolutions? We believe a different approach is needed.
There are desired properties for generated videos: (i) high quality for each individual frame, and (ii) the frame sequence should be temporally consistent, i.e. depicting the same content with plausible motion. Previous works (Tulyakov al.] 2018 Clark et al., 2019) attempt to achieve both goals with a single framework, making such methods computationally demanding when high resolution is desired. We suggest a different perspective on this problem. We hypothesize that, given an image generator that has learned the distribution of video frames as independent images, video can be represented as sequence of latent codes from this generator. The problem of video synthesis can then be framed as discovering latent trajectory that renders temporally consistent images. Hence, we demonstrate that (i) can be addressed by a pre-trained and fixed image generator, and (ii) can be achieved using the proposed framework to create appropriate image sequences.
Work done while at Snap Inc.
We estimate that the cost of training a model such as DVD-GAN (Clark al. 2019 once requires $30K.

---

Published as a conference paper at ICLR 2021
To discover the appropriate latent trajectory, we introduce motion generator, implemented via two recurrent neural networks, that operates on the initial content code to obtain the motion representa- tion. We model motion residual between continuous latent codes that are passed to the image generator for individual frame generation. Such a residual representation can also facilitate the disentangling of motion and content. The motion generator is trained using the chosen image dis- criminator with contrastive loss to force the content to be temporally consistent, and patch-based multi-scale video discriminator for learning motion patterns. Our framework supports contemporary image generators such as StyleGAN2 (Karras et al. 2019) and BigGAN (Brock et al. 2019).
We name our approach as MoCoGAN-HD (Motion and Content decomposed GAN for High- Definition video synthesis) as it features several major advantages over traditional video synthesis pipelines. First, transcends the limited resolutions of existing techniques, allowing for the gen- eration of high-quality videos at resolutions up to 1024 x 1024. Second, as we search for latent trajectory in an image generator, our method is computationally more efficient, requiring an order of magnitude less training time than previous video-based works (Clark al. 2019). Third, as the image generator is fixed, it can be trained on a separate high-quality image dataset. Due to the dis- entangled representation of motion and content, our approach can learn motion from a video dataset and apply it to an image dataset, even in the case of two datasets belonging to different domains. It thus unleashes the power of image generator to synthesize high quality videos when a do- main (e.g., dogs) contains many high-quality images but no corresponding high-quality videos (see Fig. 4. In this manner, our method can generate realistic videos of objects has never seen moving during training (such as generating realistic pet face videos using motions extracted from images of talking people). We refer this new video generation task as cross-domain video synthesis. Finally, we quantitatively and qualitatively evaluate our approach, attaining state-of sate-of-the-asd th--hhe-rtt art performance on each benchmark, and establish challenging new baseline for video synthesis methods.
RELATED WORK
Video Synthesis. Approaches to image generation and translation using Generative Adversarial Networks (GANs) Goodfellow al., 2014) have demonstrated the ability to synthesize high qual- ity images (Radford et al. 2016] Zhang et al.] 2019: Brock et al., 2019 Donahue & Simonyan, 2019 Jin al. 2021). Built upon image translation (Isola et al.
2017 Wang et al. 2018b). works
on video-to-video translation (Bansal et al. 2018 Wang et al. 2018a) are capable of converting an input video to a high- video generation, in the unconditional setting, is still a difficult and unresolved problem. Without the strong conditional inputs such as segmentation masks (Wang et al., 2019) or human poses (Chan eet al., 2019 Ren et al., 2020) that are employed by tideo-to-vided translation works, generating videos following the distribution of training video samples is challenging. Earlier works on GAN- based video modeling, including MDPGAN (Yushchenko et al. 2019). VGAN (Vondrick et al., 2016 TGAN (Saito et al.) 2017). MoCoGAN (Tulyakov et al. 2018), ProgressiveVGAN (Acharya et al. 2018). TGANv2 (Saito et al. 2020) show promising results on low- rew-resouuiinn datasets. Re- cent efforts demonstrate the capacity to generate more realistic videos, but with significantly more computation (Clark et al. 2019 Weissenborn al., 2020). In this paper, we focus on generating realistic videos using manageable computational resources. LDVDGAN (Kahembwe & Ramamoor- thy 2020) uses low dimensional discriminator to reduce model size and can generate videos with resolution up to 512* x 512, while we decrease training cost by utilizing a pre-trained image genera- tor. The high-quality generation is achieved by using pre-trained image generators, while the motion trajectory is modeled within the latent space. Additionally, learning motion in the latent space al- lows us to easily adapt the video generation model to the task of video prediction (Denton et al. 2017, in which the starting frame is given (Denton & Fergus) 2018 Zhao et al. 2018 Walker et al.) 2017 Villegas et al.) 2017bb al |Babaeizadeh et 2017] et al. 2018 Byeon et al. | 2018). by inverting the initial frame through the generator (Abdal et al. 2020). instead of training extra image encoder (Tulyakov et al. 2018 Zhang et al. 2020).
output in another domain. However, the task of high-fidelity
Interpretable Latent Directions. The latent space of GANs is known to consist of semantically meaningful vectors for image manipulation. Both supervised methods, either using human annota- tions or pre-trained image classifiers (Goetschalekx et al. 2019 Shen et al., 2020), and unsupervised methods (Jahanian et al. 2020 Plumerault et al.) 2020), are able to find interpretable directions for image editing, such as supervising directions for image rotation or background removal (Voynov &
2

---

Published as a conference paper at ICLR 2021
Pre-tmined Generator
Generated Video
Multiply Add
Image Discriminator
Augment
Generator
%;
Dj
Positive
Content
Negative
LSTM
Motion Trajectory
O
Cosine similasity
Contrastive Learning
Video
Discriminator
Gi
LSTMae
h
Real videos
LSTMac
8
0 Zn
G1
Given an initial latent code 21, a trajectory 6t, and PCA basis V. the motion generator Gm encodes 21 using LSTMMen to get the initial hidden state and uses LSTMa to estimate hidden states for future frames. The image generator G1 synthesizes images using the predicted latent codes. The discriminator Dv is trained on both real and generated video sequences. Right: For each generated video, the first and subsequent frames are sent to an image discriminator D1. An encoder-like network F calculates the features of synthesized images used to compute the contrastive loss Coontr with positive (same image content, but different augmentation, shown in blue) and negative pairs (different image content and augmentation, shown in red).
Figure Left:
Babenko 2020 Shen & Zhou 2020). We further consider the motion vectors in the latent space. By Eiientangiing the motion trajectories in an unsupervised fashion, we are able to transfer the motion information from a video dataset an image dataset in which temporal information is available.
Contrastive Representation Learning is widely studied in unsupervised learning tasks (He et al.
Chen et al. 2020aa bj Hénaff et al.) [0200] Löwe al. 2019; Oord al. [0118] Misra & Maaten Related inputs, such as images (Wu et al. 2018) or latent representations (Hjelm et al,, which can vary while training due data augmentation, are forced be close by minimizing differences in their representation during training. Recent work (Park et al. 2020) applies noise- contrastive estimation (Gutmann & Hyvärinen 2010) to image generation tasks by learning the correspondence between image patches, achieving performance superior to that attained when using cycle-consistency constraints (Zhu et al., 20177; Yi et al.) 2017). On the other hand, we learn an image discriminator to create videos with coherent content by leveraging contrastive loss (Hadsell ct al. 2006) along with an adversarial loss (Goodfellow et al. 20144).
2020
2020
2019
METHOD
In this section, we introduce our method for high-resolution video generation. Our framework is built on top of a pre-trained image generator (Karras et al., 2020a by Zhao et al. [020aa b). which helps to generate high-quality image frames and boosts the training efficiency with manageable com- putational resources. In addition, with the image generator fixed during training, we can disentangle video motion from image content, and enable video synthesis even when the image content and the video motion come from different domains.
More specifically, our inference framework includes motion generator GM and an image generator G1. Gm is implemented with two LSTM networks (Hochreiter Schmidhuber 1997) and predicts the latent motion trajectory Z (Z1, Z2, Znn ). where n is the number of frames in the synthe- sized video. The image generator G1 can thus synthesize each individual frame from the motion trajectory. The generated video sequence À is given by v X2, ). For each synthe- sized frame N,, we have X; = (z, for = 1,2, ... n. We also define the real video clip as {x,, X2, ,x,) and the training video distribution Pe-
To train the motion generator GM to discover the desired motion trajectory, we apply a video dis- criminator to constrain the generated motion patterns to be similar those of the training videos, and an image discriminator to force the frame content to be temporally consistent. Our framework is illustrated in Fig. [ We descrihe each component in more detail in the following sections.
3

---

Published as a conference paper at ICLR 2021
3.1 MOTION GENERATOR
The motion generator GM predicts consecutive latent codes using an input code E₁ E 2, where the latent space 2 is also shared by the image generator. For BigGAN (Brock et al. 2019), we sample z from the normal distribution P.- For StyleGAN2 (Karras et al. 2020b), Ps is the distribution after the multi-layer perceptron (MLP), the latent codes within this distribution can be semantically disentangled better than when using the normal distribution (Shen et al. 2020 Zhu et al. 2020
Formally, G' includes an LSTM encoder La which encodes 21 to get the initial hidden state, and LSTM decoder LSTMaee, which estimates n - 1 continuous hidden states recursively:
h1, C1 = LSTMMen(Z1)),
(1)
h; = LSTM (h.-1,c.1))) t 2,3. . .
where and с denote the hidden state and cell state respectively, and e, is a noise vector sampled from the normal distribution to model the motion diversity at timestamp f.
Motion Disentanglement. Prior work (Tulyakov et al. 2018) applies h, as the motion code for the frame to be generated, while the content code is fixed for all frames. However, such design quires a recurrent network to estimate the motion while preserving consistent content from the latent vector, which is difficult learn in practice. Instead, we propose to use sequence of motion resid- uals for estimating the motion trajectory. Specifically, we model the motion residual as the linear combination of a set of interpretable directions in the latent space (Shen & Zhou 2020; Härkönen iet al. 2020). We first conduct principal component analysis (PCA) on m randomly sampled latent vectors from Z to get the basis V. Then, we estimate the motion direction from the previous frame 2:-1 to the current frame 2₁ by using h, and V as follows:
2: = Zt-1 + X A-h V, t 2.3, ...
(2)
where the hidden state h, E [-1, 1), and x controls the step given by the residual. With Eqn. and Eqn. 2 we have GM = [21, 22, } and the generated video V is given as v (GM()Z)))
Motion Diversity, In Eqn. [ we introduce noise vector to control the diversity of motion. How- ever, we observe that the LSTM decoder tends neglect the ét, resulting in motion mode collapse, meaning that GM cannot capture the diverse motion patterns from training videos and generate dis- tinct videos from one initial latent code with similar motion patterns for different sequences of noise vectors. To alleviate this issue, we introduce mutual information loss 2m to maximize the mutual information between the hidden vector h, and the noise vector ét. With sim(u, (u, v) = u v [ |v] denoting the cosine similarity between vectors u and v, we define f. as follows:
sim (Hb)). 4),
(3)
n
where H is 2-layer MLP that serves as mapping function.
Learning. To learn the appropriate parameters for the motion generator Gm, we apply a multi-scale video discriminator Dy to tell whether video sequence is real synthesized. The discriminator is based on the architecture of PatchGAN (Isola et al. 2017). However, we use 3D convolutional layers in Dv, they model temporal dynamics better than 2D convolutional layers. We divide input video sequence into small 3D patches, and classify each patch as real or fake. The local responses for the input sequence averaged produce the final output. Additionally, each frame in the input video sequence is conditioned on the first frame, as it falls into the distribution of the pre-trained image generator, for more stable training. We thus optimize the following adversarial loss to learn GM and Dv:
LaY = Exep, (log Dv(v)] + Fzzyp, [log(1 - Dy(GG(GM(2)))))-
(4)
3.2 CONTRASTIVE IMAGE DISCRIMINATOR
As our image generator is trained, we may image generator that is trained on given domain, egg. images of animal faces (Choi et al. 2020), and leam the motion generator parameters using videos from different domain, such videos of human facial expressions (Nagrani al.,
4

---

Published as a conference paper at ICLR 2021
With Eqn. alone, however, lack the ability to explicitly constrain the generated images 1 to possess similar quality and content as the first image X1, which is sampled from the image space of the image generator and thus has high fidelity. Hence, we introduce contrastive image discriminator Dr, which is illustrated in Fig. T to match both image quality and content between X and 1//11-
2017].
Quality Matching. To increase the perceptual quality, we train Dj and Gm adversarially by for- warding X, into the discriminator Dj and using X real sample and Xtjt>1 as the fake sample.
LD. = [log (r(Gi(z))) [Ez (m.)))) [log(1 Dr(G)(z)))]
(5)
Content Matching. To learn content similarity between frames within a video, we use the im- age discriminator as a feature extractor and train it with form of contrastive loss known as In- foNCE (Oord et al. 2018). The goal that pairs of images with the same content should be close together in embedding space, while images containing different content should be far apart.
Given a minibatch of N generated videos {v(") ((2),... ((()), we randomly sample one frame t (ib),
from each video: {x(²) ;x('
xx()), and make two randomly augmented versions (x(1*), x
for each frame x resulting in 2N samples. (x(5 x; x(()) are positive pairs, as they share the same (+-)
content. (x(()) ( are all negative pairs for j.
Let F' be an encoder network, which shares the same weights and architecture of Dr, but excluding the last layer of Dj and including 2-layer MLP a projection head that produces the representation of the input images. We have a contrastive loss function Lcoutr, which is the cross-entropy computed across 2N augmentations follows:
Goants ΣΣ log
exp(sim( (xx) (10))
F(X:
1)/r)
(6)
ΣS Es=o (exp(simxsimmm Fxx
F(x
((x(5)))))
where sim(..-) is the cosine similarity function defined in Eqn. 1 jivl] E {0,1} is equal iff j * i, and a temperature parameter empirically set to 0.07. We use momentum decoder mechanism similar that of MoCo (He et al. 2020) by maintaining memory bank delete the oldest negative pairs and update the new negative pairs. We apply augmentation methods including translation, color jittering, and cutout (DeVries & Taylor 2017) synthesized images. With the positive and negative pairs generated on-the-fly during training, the discriminator can effectively focus on the content of the input samples.
The choice of positive pairs in Eqn. 6 specifically designed for cross-domain video synthesis, as videos of arbitrary content from the image domain is not available. In the case that images and videos are from the same domain, the positive and negative pairs are easier to obtain. We randomly select and augment two frames from real video to create positive pairs sharing the same content, while the negative pairs contain augmented images from different real videos.
Aside from Lcontr, we also adopt the feature matching loss (Wang et al. 2018b) Li between the generated first frame and other frames by changing the L1 regularization to cosine similarity.
Full Objective. The overall loss function for training motion generator, video discriminator, and image discriminator is thus defined as:
min (max LDv + max LD,) + (m + XeLe) + min()euntrFoontt)
Gx Dy
Dj
GM
Dr
where Ams Acantr. and d are hyperparameters to balance losses.
EXPERIMENTS
In this section, we evaluate the proposed approach several benchmark datasets for video genera- tion. We also demonstrate cross-domain video synthesis for various image and video datasets.
4.1 VIDEO GENERATION
We conduct experiments three datasets including UCF-101 (Soomro et al. 2012). FaceForen- sics (Rössler et al. 2018), and Sky Time-lapse (Xiong et al. 2018) for unconditional video synthesis.
We use StyleGAN2 as the image generator. Training details be found in Appx. B
5

---

Published as a conference paper at ICLR 2021
Table 1: IS and FVD on UCF-101,
Table 2: FVD, ACD, and Human Preference on FaceForensics.
Method
IS (1)
FVD (4)
Method
FVD (4)
ACD (L)
VGAN
8.31 + .09
TGAN
11.85 .07
GT
9.02
0.2935
MoCoGAN
12,42 +
TGANx2
58.03
0,4914
ProgressiveVGAN
14.56 -
Ours
53.26
0.3300
LDVD-GAN
22.91 1.99
TGAN22
26.60 #
1209 = 28
Method
Human Preference (%)
DVD-GAN
27.38 .53
Ours TGANx2
73.6 26.4
Ours
33.95 + .25 700 + 24
UCF-101 is widely used in video generation. The dataset includes 13, 320 videos of 101 sport categories. The resolution of each video is 320 x 240. To process the data, we crop a rectangle with size of 240 240 from each frame video and resize it to 256 x 256. We train the motion generator to predict 16 frames. For evaluation, we report Inception Score (IS) (Saito et al. 2020)
on 10, 000 generated videos and Fréchet Video Distance (FVD) (Unterthiner et al. (018) on 2, 048 videos. The classifier used to calculate IS is C3D network (Tran et al.) 2015) that is trained on the Sports- IM dataset (Karpathy et al. 2014) and fine-tuned on UCF-101, which is the same model used in previous works (Saito al. 2020 Clark al. 2019).
The quantitative results are shown in Tab. I Our method achieves state-of-the-art results for both IS and FVD, and outperforms existing works by a large margin. Interestingly, this result indicates that a well-trained image generator has learned to represent rich motion patterns, and therefore can be used synthesize high-quality videos when used with a well-trained motion generator.
FaceFiressscs a dataset containing news videos featuring various reporters. We use all the images from 704 training videos, with resolution of 256 256, to learn image generator, and sequences of 16 consecutive frames to train motion generator. Note that our network can generate even longer continuous sequences, e.g. 64 frames (Fig. [2 in Appx.), though only 16 frames are used for training.
We show the FVD between generated and real video clips (16 frames in length) for dif- ferent methods in Tab. [2] Additionally, we use the Average Content Distance (ACD) from MoCoGAN (Tulyakov et al., 2018) to evalu- ate the identity consistency for these human face videos. We calculate ACD values over 256 videos. We also report the two metrics for ground truth (GT) videos. To get FVD of GT videos, we randomly sample two groups of real videos and compute the score. Our method achieves better results than TGANx2 (Saito al. (0220). Both methods have low FVD val- ues, and can generate complex motion patterns close to the real data. However, the much lower ACD value of our approach, which is close GT, demonstrates that the videos it synthesizes have much better identity consistency than the videos from TGANx2. Qualitative examples Fig. iillustrate different motions patterns learned from the dataset. Furthermore, we perform perceptual experiments using Amazon Mechanical Turk (AMT) by presenting pair of videos from the two methods to users and asking them to select more real- istic one. Results in Tab. inddicate our method outperforms TGANx2 in 73.6% of the comparisons.
Figure 2: Example generated videos from a model trained on FaceForensics. We can generate natu- ral and photo-realistic videos with various motion patterns, such as eye blink and talking. Four ex- amples show frames 2, 7, 11, and 16.
Sky Time-Lapse is a video dataset consisting of dynamic sky scenes, such as moving clouds. The number of video clips for training and testing is 35, 392 and 2, 815, respectively. We resize images to 128 x 128 and train the model to generate 16 frames. We compare our methods with the two recent approaches of MDGAN (Xiong al. 2018) and DTVNet (Zhang et al.) 2020), which are specifically designed for this dataset. In Tab. 3 we report the FVD for all three methods. It is clear that approach significantly outperforms the others. Example sequences shown in Fig.

---

Published as a conference paper at ICLR 2021
Following DTVNet (Zhang et al. 2020), evaluate the proposed model for the task of video prediction. We use the Peak Signal-to-Nee Noise Ratio (PSNR) and Structural Similarity (SSIM) (Wang et al. 2004) evaluation metrics measure the frame quality the pixel level and the structural similarity between synthesized and real video frames. Evaluation performed on the testing set. We select the first frame X1 from each video clip and project it the latent space of the image generator (Abdal et al. 2020) to get ž1. We use ž as the starting latent code for motion generator get 16 latent codes, and interpolate them get 32 latent codes synthesize a video sequence. where the first frame is given by (21)). For fair comparison, we also use Gi as the starting frame for MDGAN and DTVNet calculate the metrics with ground truth videos. In addition, we calculate the PSNR and SSIM between and (2)) as the upper bound for all methods, which we denote as Up-B. Tab. shows the video prediction results, which demonstrate that our method's performance is superior to those of MDGAN and DTVNet. Interestingly, by simply interpolating the motion trajectory, we can easily generate longer video sequence, egg. from 16 to 32 frames, while retaining high quality.
Table 3:
Evaluation Sky Time-lapse for video synthesis and prediction.
Method
FVD (4)
PSNR (t)
SSIM (t)
Up-B
25.367
0.781
MDGAN
840.95
13.840
0.581
t t= 2
t= 10
t= 14
t= 16
DTVNet
451.14
21.953
0.531
Ours
77.77
22.286
0.688
Figure 3: Sample generated frames at several time steps (f) for the Sky Time-lapse dataset.
1=2
1=4 4
i 6
1 8
1 - 10
1 = 12
1 = 14
16
Figure& 4: Example sequences for cross-domaie video generation. First Row: (FFHQ. VoxCeleb). Second Row: (LSUN-Church, TLVDB). Third Row: (AFHQ-Dop VoxCeleb). Fourth Row: (Ani- meFaces, VoxCeleb). Images in the first and second rows have resolution of 256 x 256, while the third and fourth rows have a resolution of 512 512.
4.2 CROSS-DOMAIN VIDEO GENERATION
To demonstrate how our approach can disentangle motion from image content and transfer motion patterns from one domain to another, we perform several experiments on various datasets. More specifically, Dog (Choi et al. 2020), AnimeFaces (Branwen 2019), and LSUN-Church Yu et al. 2015 datasets,
the StyleGAN2 model, pre-trained on the FFHQ (Karras al.) 2019). AFHQ-
the image generators. We learn human facial motion from VoxCeleb (Nagrani al. 2020 and
7

---

Published as a conference paper at ICLR 2021
time-lapse transitions in outdoor scenes from TLVDB (Shih al. 2013). In these experiments, a pair such as (FFHQ. VoxCeleb) indicates that we synthesize videos with image content from FFHQ and motion patterns from VoxCeleb. We generate videos with resolution of 256 x 256 and 1024 1024 for FFHQ, 512x51 512 for AFHQ-Dog and AnimeFaces, and 256 x 256 for LSUN-Church. Qualitative examples for (FFHQ, VoxCeleb), (LSUN-Church, TLVDB), (AFHQ-Dog, VoxCeleb), and (Anime- Faces, VoxCeleb) are shown in Fig. 4 depicting high-quality and temporally consistent videos (more videos, including results with BigGAN as the image generator, are shown in the Appendix).
We also demonstrate how the motion and content are disentangled in Fig. and Fig. 6 which portray generated videos with the same identity but performing diverse motion patterns, and the same motion applied to different identities, respectively. We show results from (AFHQ-Dog, VoxCeleb) (first two rows) and (AnimeFaces, VoxCeleb) (last two rows) these two figures.
10
12
10
12
Figure 5: The first and second row (also the third and fourth row) share the same initial content code but with different motion codes.
Figure 6: The first and second row (also the third and fourth row) share the same motion code but with different content codes.
4.3 ABLATION ANALYSIS
We first report IS and FVD in Tab. for UCF-101 using the following methods: w/o Eqn. uses h instead of estimating the residual as in Eqn. [ w/o Dj omits the contrastive image discriminator Dj and uses the video discriminator Dy only for learning the motion generator; w/o Dy omits Dv during training; and Fu8 and Full-256 indicate that we generate videos using our full method with resolutions of 128 x 128 and 256 x 256, respectively. We resize frames for all methods to 128 x 128 when calculating IS and FVD. The full method outperforms all others, proving the importance of each module for learning temporally consistent and high-quality videos.
We perform further analysis of our cross cross- domain video generation on (FFHQ, VoxCeleb). We com- pare our full method (Full) with (Eqn. (6) from Di, and w/o L. indicates that we omit the mutual information loss (Eqn. (3) for the motion generator. The results in Tab. ddemonsstttttttee that Lcontr beneficial for learning videos with coherent content, as employing Coontr results in lower ACD values and higher human preferences. Lm also contributes generating higher quality videos by mitigating motion synchronization. To validate the motion diversity, we show pairs of 9 randomly generated videos from the two methods users and ask them choose which one has superior motion diversity, including rotations and facial expressions. User preference suggests that using L. increases motion diversity.
variants. w/o frontr denotes that we omit the contrastive loss
Table 4: Ablation study UCF-101.
Table 5: Ablation study on (FFHQ, VoxCeleb).
Method
IS (t)
FVD (L)
Method
w/o Coontr
w/o Lm
Full
w/o Eqn. 2
28.20
790.87
ACD (L)
0.5328
0.5158
0.4353
w/o D1
33.22
796.67
Method
Human Preference (%)
w/o Dy
33.84
867.43
Full-128
32.36
838.09
Full vs w/o Leantr Full vs w/o L.
68.3 / 31.7 64.4 / 35.6
Full-256
33.95
700.00
8

---

Published as a conference paper at ICLR 2021
4.4 LONG SEQUENCE GENERATION
Due to the limitation of computational resources, we train MoCoGAN-HD to synthesize 16 consec- utive frames. However, we can generate longer video sequences during inference by applying the following two ways.
Motion Generator Unrolling. For motion generator, we can run the LSTM decoder for more steps to synthesize long video sequences. In Fig. 7 we show a synthesized video example of 64 frames using the model trained the FaceForensiis dataset. Our method is capable synthesize videos with more frames than the number of frames used for training.
Figure 7: The generation of 64 64-frame video using model trained with 16-frame on FaceForenssics.
Motion Interpolation. We can do interpolation on the motion trajectory directly to synthesize long
videos. Fig. 8 shows an interpolation example of 32 framee (AFHQ- Dog, VoxCeleb) dataset.
Figure 8: The generation a 32-frame video on (AFHQ-Dog. VoxCeleb) by doing the interpolation on motion trajectory.
5 CONCLUSION
In this work, we present novel approach video synthesis. Building on contemporary advances in image synthesis, we show that good image generator and our framework are essential ingredients to boost video synthesis fidelity and resolution. The key is to find a meaningful trajectory in the im- age generator's 's latent space. This is achieved using the proposed motion generator, which produces a sequence of motion residuals, with the contrastive image discriminator and video discriminator. This disentangled representation further extends applications of video synthesis content and mo- tion manipulation and cross- domain video synthesis. The framework achieves superior results on variety of benchmarks and reaches resolutions unattainable by prior state-of the--arttt techniques.

---

Published as a conference paper at ICLR 2021
REFERENCES
Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan+#: How to edit the embedded images? In CVPR, 2020.
Dinesh Acharya, Zhiwu Huang, Danda Pani Paudel, and Lue Van Gool. Towards high resolution video generation with progressive growing of sliced wasserstein gans, arXiv:1810.02419, 2018.
Mohammad Babaeizadeh, Chelsea Finn, Dumitru Erhan, Roy H Campbell, and Sergey Levine. Stochastic variational video prediction. ICLR, 2017.
Aayush Bansal, Shugao Ma, Deva Ramanan, and Yaser Sheikh. Recycle-gan: Unsupervised video retargeting. In ECCV, 2018.
Gwern Branwen. Making anime faces with stylegan. 2019.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In ICLR, 2019.
Wonmin Byeon, Qin Wang, Rupesh Kumar Srivastava, and Petros Koumoutsakos. Contextvp: Fully context-aware video prediction. In ECCV, 2018.
Caroline Chan, Shiry Ginosar, Tinghui Zhou, and Alexei A Efros. Everybody dance now. In ICCV, 2019.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. ICML, 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self- supervised models strong sumi-supervised learners. arXiv:2006. 10.220006/0029 10029, 2020b.
Yunjey Choi, Youngjung Uh. Jaejun Yoo, and Jung-Wer Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In CVPR, 2020.
Aidan Clark, Jeff Donahue, and Karen Simonyan. Adversarial video generation complex datasets. arXiv, 2019.
Emily Denton and Rob Fergus. Stochastic video generation with a learned prior. In ICML, 2018.
Emily L Denton et al. Unsupervised learning of disentangled representations from video. In Ad- vances neural information processing systems, pp. 4414-4423, 2017.
Terrance DeVries and Graham W Taylor. Improved regularization convolutional neural networks with cutout. 1rXiv:/708.04552. 2017.
Jeff Donahue and Karen Simonyan. Large scale adversarial representation learning. In NeurlPS, 2019.
Frederik Ebert, Chelsea Finn, Alex X Lee, and Sergey Levine. Self-supervise visual planning with temporal skip connections. arXiv preprint arXiv:1710.0526. 2017.
Lore Goetschalckx, Alex Andonian, Aude Oliva, and Phillip Isola. Ganalyze: Toward visual defini- tions of cognitive image properties. In ICCV, 2019.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurlPS, 2014.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In AISTATS, 2010.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In CVPR, 2006.
Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris. Ganspace: Discovering interpretable gan controls. 02545 2020.
10

---

Published as a conference paper at ICLR 2021
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Olivier J Hénaff, Aravind Srinivas, Jeffrey De Fauw, Ali Razavi, Carl Doersch, SM Eslami, and Aaron van den Oord. Data-efficient image recognition with contrastive predictive coding. In ICML, 2020.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In NeurlPs, 2017.
R Devon Hjelm, Alex Fedorov, Samuel Lavoie- Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In ICLR, 2019.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 1997.
Jun-Ting Hsieh, Bingbin Liu, De-An Huang, Li F Fei-Fei, and Juan Carlos Niebles. Learning to decompose and disentangle representations for video prediction. In NewrIPS, 2018.
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In CVPR, 2017.
Ali Jahanian, Lucy Chai, and Phillip Isola. On the" "sterrability" of generative adversarial networks. In 2020.
Qing Jin, Jian Ren, Oliver J. Woodford, Jiazhuo Wang, Geng Yuan Yuan, Yanzhi Wang, and Sergey Tulyakov. Teachers do more than teach: Compressing image-to-image models. arXiv preprint arXiv:2103.03467, 2021.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In ECCV, 2016.
Emmanuel Kahembwe and Subramanian Ramamoorthy. Lower dimensional kernels for video dis- criminators. Neural Networks, 132:506-520, 2020.
Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei- Fei. Large-scale video classification with convolutional neural networks. In CVPR, 2014.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. arXiv: 10196, 2017.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, 2019.
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. arXiv:2006.06676 2020a.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of stylegan. In CVPR, 2020b.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. 1rXiv:1412.6980. 2014.
Sindy Löwe, Peter O' O'Connor, and Bastiaan Veeling. Putting an end to end-to-end: Gradient-isolated learning of representations. In NeurlPS, 2019.
Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invarian representa- tions. In CVPR, 2020.
A. Nagrani, J. S. Chung, and A. Zisserman. Voxceleb: large-scale speaker identification dataset. In INTERSPEECH. 2017.
Arsha Nagrani, Joon Son Chung, Weidi Xie, and Andrew Zisserman. Voxceleb: Large-scale speaker verification in the wild. Computer Speech & Language, 2020.
11

---

Published as a conference paper at ICLR 2021
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. wrXiv:187.033745 2018.
Taesung Park, Alexei A. Efros, Richard Zhang, and Jun-Yan Zhu. Contrastive learning for unpaired image-to-image translation. In ECCV, 2020.
Antoine Plumerault, Hervé Le Borgne, and Céline Hudelot. Controlling generative models with continuous factors of variations. In /CLR, 2020.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In ICLR, 2016.
Jian Ren, Menglei Chai, Sergey Tulyakov, Chen Fang, Xiaohui Shen, and Jianchao Yang. Human motion transfer from poses in the wild. arXiv:2004.03142, 2020.
Andreas Rössler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies, and Matthias Nießner. Faceforensics: A large-scale video dataset for forgery detection in human faces. arXiv:1803.09119, 2018.
Masaki Saito, Eiichi Matsumoto, and Shunta Saito. Temporal generative adversarial nets with sin- gular value clipping. In (CCVV,017,
Masaki Saito, Shunta Saito, Masanori Koyama, and Sosuke Kobayashi. Train sparsely, generate densely: Memoer-efficient unsupervised training of high-resolution temporal gan. IJC,, 2020.
Yujun Shen and Bolei Zhou. Closed-form factorization of latent semantics in gans. arXiv:2007.06600. 2020.
Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou, Interpreting the latent space of gans for semantic face editing. In CVPR, 2020.
Yichang Shih, Sylvain Paris, Frédo Durand, and William T Freeman. Data-driven hallucination of different times of day from a single outdoor photo. TOG, 2013.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv: 1556, 2014.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild.
2012.
Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spa- tiotemporal features with 3d convolutional networks. In ICCV, 2015.
Sergey Tulyakov, Ming-Yo Yu Liu, Xiaodong Yang, and Jan Kautz. Mocogan: Decomposing motion and content for video generation. In CVPR, 2018.
Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. arXiv:1812.01177. 2018.
Ruben Villegas, Jimei Yang, Seunghoon Hong, Xunyu Lin, and Honglak Lee. Decomposing motion and content for natural video sequence prediction. In ICLR, 2017a.
Ruben Villegas, Jimei Yang, Yuliang Zou, Sungryull Sohn, Xunyu Lin, and Honglak Lee. Learning to generate long-term future via hierarchical prediction. In ICML, 2017b.
Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dynamics. In NewrlPS, 2016.
Andrey Voynov and Artem Babenko. Unsupervised discovery of interpretable directions in the gan latent space. In ICML, 2020.
Jacob Walker, Kenneth Marino, Abhinav Gupta, and Martial Hebert. The pose knows: Video fore- casting by generating pose futures. In ICCV, 2017.
12

---

Published as a conference paper at ICLR 2021
Ting- Chg--Chn Wang, Ming- Yu Liu, Jun- Yan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, and Bryan Catan- zaro. Video-to-video synthesis. In NewrlPS, 2018a.
Ting-Chun Wang, Ming-Yu Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional gans. In CVPR, 2018b.
Ting-Chie Chun Wang. Ming- Yu Liu, Andrew Tao, Guilin Liu, Jan Kautz, and Bryan Catanzaro. Few- shot video-to-video synthesis. In NeurlPS, 2019.
Zhou Wang, Alan C Bovik. Hamid R Sheikh, and Ecro Simoncelli. Image quality assessment: from error visibility to structural similarity. 2004.
Dirk Weissenborn, Oscar Täckström, and Jakob Uszkoreit. Scaling autoregressive video models. In ICLR, 2020.
Zhirong Wu, Yuanjun Xiong, Yu Stella, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In CVPR, 2018.
Wei Xiong, Wenhan Luo, Lin Ma, Wei Liu, and Jiebo Luo. Learning to generate time-lapse videos using multi-stage dynamic generative adversarial networks. CVPR, 2018.
Zili Yi, Hao Zhang, Ping Tan, and Minglun Gong. Dualgan: Unsupervised dual learning for image- to-image translation. In ICCV, 2017.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv:1506.033365 2015.
Vladyslav Yushchenko, Nikita Araslanov, and Stefan Roth. Markov decision process for video generation. In Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 0-0, 2019.
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. In ICML, 2019.
Jiangning Zhang, Chao Xu, Liang Liu, Mengmeng Wang, Xia Wu, Yong Liu, and Yunliang Jiang. Dtvnet: Dynamic time-lapse video generation via single still image. In ECCV, 2020.
Long Zhao, Xi Peng, Yu Tian, Mubbasir Kapadia, and Dimitris Metaxas. Learning forecast and refine residual motion for image-to-videe generation. In Proceedings of the European conference computer vision (ECCV), pp. 387-403, 2018.
Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, and Song Han. Differentiable augmentation for data-efficient gan training. arXiv:2006. 10738, 2020a.
Zhengli Zhao, Zizhao Zhang, Ting Chen, Sameer Singh, and Han Zhang. Image augmentations for gan training. 0tti:200000055955 2020b.
Jiapeng Zhu, Yujun Shen, Deli Zhao, and Bolei Zhou. In-domain gan inversion for real image editing. In ECCV, 2020.
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In ICCV, 2017.
13

---

Published as a conference paper at ICLR 2021
A ADDITIONAL DETAILS FOR THE FRAMEWORK
A.1 ADDITIONAL DETAILS FOR THE MOTION GENERATOR
To use StyleGAN2 Karras et al. 2020b) as the image generator, we randomly sample 1, 0000 000 latent codes from the input space and send them to the 8-layer MLPs to get the latent codes in the space of W. Each latent code is a 512-dimension vector. We perform PCA these 1. 0000, 000 latent codes and select the top 384 principal components to form the matrix V E R384x512 which used model the motion residuals in Eqn. 2 The LSTTM encoder and the LSTTM decoder in the motion generator both have an input size of 512 and a hidden size of 384. The noise vector & in Eqn. I is also 512-dimension vector, and the network H in Eqn. 3 is 2-layer MLPs with 512 hidden units in each of the two fully-connected layers.
For BigGAN (Brock et al. 2019), we sample the latent code directly from the space of 2.
A.2
ADDITIONAL. DETAILS FOR THE DISCRIMINATORS
A.2.1 VIDEO DISCRIMINATOR
The input images for the video discriminator Dy are processed at two scales. We downsample the output images from the image generator to the resolution of 128 128 and 64 64. For in- domain video synthesis, the input sequences for Dy have the shape of 6 (n (n-11 1) x 128 128 and 6 x (n 1) 64 x 64, where is the sequence length used for training. For each of the (n 1) subsequent frames, we concatenate the RGB channels of both the first frame and that subsequent frame, resulting in 6-channel input. For cross-domain video synthesis, the input sequences for Dv have the shape of 3 x n x 128 x 128 and x n x 64 x 64, as the concatenation of the first frame will make the discriminator aware the domain gaps. Details for Dy are shown in Tab. 6
Table 6: The network architecture for video discriminator.
Operation
Kernel
Strides
# Channels
Norm Type
Nonlinearity
Conv3d
4x4
2
64
Leaky ReLU (0.2)
Conv3d
4x4
128
InstanceNorm3àd
Leaky ReLU (0.2)
Conv3d
4x4
2
256
InstanceNorm3d
Leaky ReLU (0.2)
Conv3d
4x4
512
InstanceNorm3d
Leaky ReLU (0.2)
Conv3d
4x4
1
A.2.2 IMAGE DISCRIMINATOR
The image discriminator Dj has an architecture based on that of the BigGAN discriminator, except that we remove the self-attention layer. The feature extractor F used for contrastive learning has the same architecture as Dj, except that it does not include the last layer of D₃ but has two additional fully connected (FC) layers the projection head. The number of hidden units for these two FC layers are both 256.
Here describe in more detail the image augmentation and memory bank techniques used for conducting contrastive learning.
Image Augmentation. We perform data augmentation images to create positive and negative pairs. We normalize the images to [-1,1] 1] and apply the following augmentation techniques.
Affine. We augment each image with an affine transformation defined with three random parameters: rotation a, E (10000, 180), translation as O(--0.1,01), and scale as (0.95,, 1.05).
Brightness. We add random value as (.(((05 0.5) all channels of each image.
Color. We add a random value a ~ u((-0.5, 0.5) to one randomly-selected channel of each image.
14

---

Published as a conference paper at ICLR 2021
Cutout (De Vries & Taylor 2017). We mask out pixels in random subregion of each image to 0. Each subregion starts at a random point and with size (am am W), where O' (0, 0.25) and (H, W) is the image resolution.
Flipping. We horizontally flip the image with the probability of 0.5.
Memory Bank. It has been shown that contrastive learning benefits from large batch-sizes and negative pairs (Chen et al. (020b). To increase the number of negative pairs, we incorporate the memory mechanism from MoCo (He al. 2020), which designates a memory bank to store negative examples. More specifically, we keep an exponential moving average of the image discriminator, and its output of fake video frames are buffered as negative examples. We use memory bank with a dictionary size of 4, 096.
B MORE DETAILS FOR EXPERIMENTS
Image Generators. We train the unconditional StyleGAN2 models from scratch on the UCF-101 FaceForensics, Sky lapse, and AFHQ-Dog datasets. We train the image generators with the official Tensorflow code and select the checkpoints that obtain the best Fréchet inception distance (FID) (Heusel et al., 2017) score to be used as the image generators. The FID score of each image generator is shown in Table 7 For FFHQ, AnimeFaces, and LSUN-Church, we simply use the released pre-trained models.
We also train an unconditional BigGAN model on the FFHQ dataset using the public PyTorch code We train model with resolution 128 x 128 and select the last checkpoint the image generator.
Table 7: FID of our trained StyleGAN2 models on different datasets.
UCF-101
FaceForensics
Sky Time-lapse
AFHQ-Dog
FID
45.63
10.99
10.80
7.85
Training Time. We train each image generator for UCF-101, FaceForensics, Sky Time-lapse, and AFHQ-Dog in less than 2 days using 8 Tesla V100 GPUs. For FFHQ, AnimeFaces, and LSUN- Church, we use the released models with no training cost. The training time for video generators ranges from 1.5 ~ 3 days depending on the datasets (Due to the memory issue, the training for generating videos with resolution 1, 020 x 1, 024 was done on 8 Quadro RTX 8000, with 5 days). The total training time for all the datasets is 1.5 5 days and the estimated cost for training Google Cloud $0.3K.
Implementation Details. We implement our experiments with PyTorch 1.3.1 and also tested them with PyTorch 1.6. We use the Adam optimizer (Kingma & Ba 2014) with learning rate of 0.0001 for GM. Dv. and D1 in all experiments. In Eqn. tasks and use a smaller X 0.2 for cross-domain video generation, as improves the content consistency. In Eqn. 7 could potentially lead to a performance boost. For TGANv2, we use the released code [4 to train the models on UCF-101 and FaceForensics using 8 Tesla V100 with 16GB of GPU memory.
set À 0.5 for conventional video generation
set A. Acontr A Grid searching on these hyper-parameters
Video Prediction. For video prediction, we predict consecutive frames, given the first frame x from test video clip as the input. We find the inverse latent code é1 for x by minimizing the following objective:
21 arg min |x] G1(21)ll2 + Avgs [Fgg - Fugy (21(2)))12
(8)
21
where Avgg the weight for perceptual loss (Johnson et al. traction model (Simonyan & Zisserman 2014). We set Ävgg I and optimize Eqn. g for 20, 000 iterations. We take 21 as the input to our model for video prediction.
2016). Fves is the VGG feature ex-
15

---

Published as a conference paper at ICLR 2021
AMT Experiments. We present details the AMT experiments for different experimental settings and datasets. For each experiment, we run 5 iterations to get the averaged score.
FaceForensics, Ours vs TGANx2. We randomly select 300 videos from each method and ask users to select the better one from a pair of videos.
Sky Time-lapse, Ours vs DTVNet. We compare our method with DTVNet on the video prediction task. The testing set of Sky Time-lapse dataset includes 2, 815 short video clips. Considering that many of these video clips share similar content and are sampled from 148 long videos, select 148 short videos with different content for testing. For these videos, we perform inversion (Eqn. 8) on the first frame to get the latent code and generate videos. For DTVNet, we use the first frame directly imput produce their results. We ask users to chose the one with better video quality from pair of videos generated by our method and DTVNet. The results shown Tab. demonstrate the clear advantage of our approach.
Table 8: Human evaluation experiments on Sky Time-lapse dataset.
Method
Human Preference (%)
Ours DTVNet
77.3 22.7
FFHQ, Full vs w/o Leonte- We randomly sample 200 videos generated by each method and ask users to select the more realistic one from pair of videos.
FFHQ, Full vs w/o Lm- For each method, we use the same content code z to generate 9 videos with different motion trajectories, and organize them into a 3 grid. To conduct AMT experiments, we randomly generate 50 3 x 3 videos for each method and ask users to choose the one with higher motion diversity from pair of videos.
Cross-Domain Video Generation. We provide more details on the image and video datasets.
Image Datasets:
FFHQ (Krraa et al. 2019 consists of 70, 000 high-quality face images at 1024 x 1024 resolution with consıderable variation in terms of age, ethnicity, and background.
AFHQ-Dog (Choi et al. 2020) contains 5, 239 high-quality dog images at 512 x 512 resolution with both training and testing sets.
AnimeFaces (Branwen, 2019) includes 2, 232, 462 anime face images at 512 512 resolution.
L.SUN-Churd Yuu et al. 256 resolution.
2015) includes 126, 227 in-the-wild church images at 256 x
Video Datasets:
VoxCeleb (Nagrani et al. 2020) consists of 22, 496 short clips of human speech, ex- tracted from interview videos uploaded to YouTube.
TLVDB (Shih et al.] 2013) includes 463 time-lapse videos, covering a wide range of landscapes and cityscapes.
For the video datasets, we randomly select 32 consecutive frames from training videos and select every other frame to form a 16-frame sequence for training.
C MORE VIDEO RESULTS
In this section, we provide more qualitative video results generated by our approach. We show the thumbnail from each video in the figures. Full resolution videos are in the supplementary material. We also provide an HTML page to visualize these videos.
UCF-101. In Fig. 9 show videos generated our approach the UCF-101 dataset.
FaceForessics. In Fig. 10, we show the generated videos for FaceForensics. In Fig. and Fig. 12 show that approach generate long consecutive results, 32 and 64 frames respectively,
16

---

Published as a conference paper at ICLR 2021
when trained with 16-frame clips. In Fig. 13 we demonstrate that approach generate diverse motion patterns using the same content code. In Fig. 14 we apply the same motion codes with different content to get the synthesized videos.
Sky Time-lapse. Fig. I3 shows the generated videos for the Sky Time-lapse dataset.
(FFHQ, VoxCeleb). Fig. 16 Fig. and Fig. 18 present the generated videos that have motion
patterns from VoxCeleb and content from FFHQ, with resolutions of 128 x 128, 256 x 256, and
1024 1024, respectively. We use BigGAN the generator for Fig. 16 and StyleGAN2 for Fig. and Fig. 18
(AFHQ-Dog, VoxCeleb). Fig. 19 presents the generated videos that have motion patterns from
VoxCeleb and content from AFHQ-Dog. The videos have a resolution of 512 x 512. In Fig. 20 we show the interpolation between every two frames get longer sequences.
(AnimeFaces, VoxCeleb). Fig. shows the generated videos that have motion patterns from Vox- Celeb and content from AmimeFaces The videos have resolution of 512 x 512.
(LSUN-Church, TLVDB). Fig. 22 presents the generated videos that have time-lapse changing style from TLVDB and content from LSUN-Church
Figure 9: Example videos generated by our ap- proach on the UCF-101 dataset.
Figure 10: Example videos generated by our ap- proach on the FaceForensics dataset.
Dir
La
Lan
Lat
L
Figure 11: The generated videos on the Face- Forensics dataset consisting frames.
Figure 12: The generated videos on the Face-
Forensics dataset consisting of frames.
17

---

Published as a conference paper at ICLR 2021
Figure 13: Each row is synthesized using the same content code to generate diverse motion patterns. Please the corresponding supple- mentary video for a better illustration.
Figure 14:
Each row is synthesized with the same motion trajectory but different content codes. Please the corresponding supplemen- tary video for a better illustration.
Figure 15: Example videos generated by our approach on the Sky Time-lapse dataset. The videos have resolution of 128 128.
Figure 16: Cross-domain video generation for (FFHQ, Vox). The videos have a resolution of 128 128.
Figure 17: Cross-domain video generation for
Figure 18: Cross-domain video generation for (FFHQ. Vox). The videos have resolution of 1024 x 1024.
(FFHQ, Vox). The videos have resolution of 256 x 256.
18

---

Published as a conference paper at ICLR 2021
Figure 19: Cross-domain video generation for (AFHQ-Dog, Vox), The videos have resolution of 512 x 512.
Figure 20: Cross-domain video generation for (AFHQ-Dog, Vox). We interpolate every two frames to get 32 sequential frames. The videos have resolution of 512 512.
Figure 21: Cross-domain video generation for (AnimeFaces, Vox). The videos have resolu- tion of 512 512.
Figure 22: Cross-domain video generation for (LSUN-Church, TLVDB). The videos have a res- olution of 256 256.
19

---

Published as a conference paper at ICLR 2021
D MORE ABLATION ANALYSIS FOR MUTUAL INFORMATION LOSS Lm
In addition Tab. 5 we perform another ablation experiment to show how mutual information loss Lm improves motion diversity by considering the following setting. We random sample content code 21 16 frames. We average the generated 1 100 videos (they share the same first frame) to get one mean- video, which contains 16 frames. For example, for the last frame the mean-video, it obtained by averaging all the last frames from the 100 generated videos. We also calculate the per-pixel standard deviation (std) for each averaged frame in the mean-video. More blurry frames and higher per-pixel std indicate the 100 synthetic videos contain more diverse motion.
and use it as an input synthesize 100 videos, where each video contains
We evaluate the settings Full and w/o Lm (without using the mutual information loss) by running the above experiments for 50 times, e.g., sampling for 50 times. Across the 50 trials, for Full model, the mean and std of the pir-pixel std for the 16th frame (the last frame a generated video) 0.233 + 0.036, which is significantly higher than that of the w/o L. model (0.126 + 0.025). In Fig. 23 we show examples of the last frame from the mean-video and the images with per-pixel std (See supplementary material for the whole videos). Our Full model has more diverse motion as the averaged frame is more blurry and the per-pixel std is higher. Note that StyleGAN2 enables noise inputs for extra randomness, we disable it in this ablation study.
The last frame of the mean-video and per-pixel std of w/o L. model. Row 2 and 4: The last frame of the mean-video and per-pixel std of the Full model. The Full model has a more blurry mean-video and higher per-pixel std, which indicates more diverse motion.
Figure 23: Row and 3:
E LIMITATIONS
Our framework requires well-trained image generator for frame synthesis. In order to synthesize high-quality and temporally coherent videos, ideal image generator should satisfy two require- ments: R1. The image generator should synthesize high-quality images, otherwise the video dis- criminator can easily tell the generated videos as the image quality is different from the real videos. R2. The image generator should be able to generate diverse image contents include enough mo- tion modes for sequence modeling.
Example of R1. UCF-101 is challenging dataset even for the training of image generator. In Tab. 7 the StyleGAN2 model trained on UCF-101 has FID 45.63, which is much worse than the others. We hypothesis the reason is that UCF-101 dataset has many categories, but within each category, it includes relatively small amount of videos and these videos share very similar content. achieve
Such observation is also discussed in DVDGAN (Clark al. 2019). Although
20

---

Published as a conference paper at ICLR 2021
tfte-of-thee-rr performance on UCF-101 dataset, the quality of the generated videos is not good as other datasets (Fig. 9. and the quality of synthesized videos is still not close to real videos.
Example of R2. We test method on BAIR Robot Pushing Dataset (Ebert al. 2017). We train a 64 x 64 StyleGAN2 image generator with using the frames from BAIR videos. The image generator has FID 6.12. Based the image generator, can synthesize 16 frames. An example of synthesized video is shown in Fig. 24 (more videos are in the supplementary materials). We see our method successfully model shadow changing, the robot arm moving, but it struggles to decouple the robot arm from some small objects in the background, which we show analysis follows.
train video generation model that
Figure 24: A synthesized video using BAIR dataset. Note the background changing of the first frame (upper-left) and the last frame (bottom-right).
E. ANALYSIS OF THE INFORMATION CONTAINED IN PCA COMPONENTS.
Inspired by previous work (Härkönen al. 2020), we further investigate the latent space of the image generator by considering the information contained in each PCA component. Fig. 25 shows the percentage of total variance captured by top PCA components. The image generator on BAIR compresses most of the information few components. Specially, the top 20 PCA components captures 85% of the variance. In contrast, the latent space the image generator trained on FFHQ (and FFHQ 1024 for high-resolution image synthesis) uses 100 PCA components capture 85% information. This implies the BAIR generator models the dataset in a low-dimension space, and such generator increases the difficulty for fully disentangling all the objects in images for manipulation.
100
% of Total Variance
80
60
40
BAIR
20
FFHQ
FFHQ 1024
85%
0
10
20
30
40
50
60
70
80
90 100
# of PCA Components
Figure 25: Percentage of variations captured by top PCA components different models.
21

---

Published as a conference paper at ICLR 2021
Moreover, visualize the video synthesis results by moving along the top 20 PCA components. Let V; denote the hth PCA component. Given content code 21, we synthesize a 5-frame video clip by using the following sequence as input: {=1 2V,
V4, 21, 21 V1, 21 2V, In Fig. 26 show the video synthesis results by moving along the top 20 PCA directions. can be seen that: 1) changing the later components (the 8th and later rows) of BAIR only make small changes; 2) the first 7 components of BAIR have entangled semantic meaning, while the components in FFHQ have more disentangled meaning (2nd row, rotation; 20th row, smile). This indicates the image generator of BAIR may not cover enough (disentangled) motion modes, and it might be hard for the motion generator fully disentangle all the contents and motion with only few dominating PCA components, while for the image generator trained on FFHQ, it is much easier for disentangling foreground and background.
22

---

Published as a conference paper at ICLR 2021
Figure 26: Visualization of top 20 principle components of BAIR (left) and FFHQ (right).
23


Reviews:
[{"review_id": "UVVkCcB8ylF", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "An interested outsider trying to exercise best judgment here.\n\nCertainly interesting idea with great results. The way this paper approaches video synthesis is to decouple that into two steps: 1) learning motion (residuals between the latent codes of consecutive frames) from a sequence of past motions, and 2) synthesizing a new image using the learnt latent code. Would you agree with this layman's assessment?\n\nReasons to accept the paper:\n* The decoupling idea is new and interesting from an interested outsider's perspective.\n* Evaluation is pretty comprehensive and the results are decent.\n\nI have a few clarification questions though.\n* The related work section claims that \"In this paper, we focus on generating realistic videos using manageable computational resources\" Do you have results to show that your approach is computationally more efficient?\n* Intuitively, why do you think future video frames can be predicted? That is, fundamentally, why do you think motion can be learnt?If I understand you correctly, you basically claim that given a sequence of motion from the training data, we should be able to learn the subsequent motion. Isn't it that there could be many kinds of subsequent motion given a sequence of past motions? I could understand how you are using it in the context of image synthesis, i.e., as long as the predicted motion can be used to generate realistic images you'd be fine with it, but precisely predicting the next frames seems to suggest that there is one fixed motion given the past motion sequence.\n* The whole motion diversity thing feels quite hand-wavy and a bit empirical to me. It feels like a hack. Why is it that motion needs to be diverse? In fact, how could you precisely quantify diverse?\n* Equation 4 (and the last paragraph of Section 3.1) shows that the motion generation is evaluated with the loss using the image generated from the image generator. Do you have a more direct way of quantifying the quality of the learnt motion? Very minor though.", "extra_scores": {}, "extra_text": {}}, {"review_id": "LdvS6MrHsNz", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\nThis paper proposes a method to disentangle content and motion from videos for high-resolution video synthesis. The proposed method consists of a motion generator, pre-trained generator, image discriminator, and video discriminator. The motion generator predicts the latent motion trajectory z, which is residually updated over time. Then the image generator produces each individual frame from the motion trajectory. For training, five types of loss functions are combined. In experiments, video generation by the proposed method is performed on UCF-101, FaceForensics, and Sky time-Lapse datasets. Also, cross-domain video generation and more ablation studies were conducted to show the effectiveness of the proposed method.\n\nOverall,  about this paper, I am leaning on the positive side. I summarize both the strength and weakness of this paper that I felt.\n\nStrength\n\nThe main benefits of this paper are covering high-resolution video synthesis and disentangling motion and contents. The proposed method was tested on the various datasets and showed experiments about cross-domain video synthesis. Also, ablation studies were performed to check the effects of each loss function.\n\n\nWeakness\n\n\nAs mentioned in the Introduction, the second desired property for generated video is temporal coherency.  The motion generator may be helpful to find a motion trajectory that makes a temporally consistent video. However, it is unclear whether the meaning of \"temporarily constant video\" means that the content of the video is consistent over frames in video or there is no flickering effect in the video. Also, there seems to be a lack of experiments to directly verify the temporal coherency.\n\nAbout the paragraph “Motion Disentanglement” in section 3.1, how to decide variable “m” for PCA? Also, what is the reason why using motion residual is helpful to motion disentanglement?\n\n\n\nFor evaluation, is there a reason for using different evaluation metrics for each dataset? Also, for each dataset, methods for comparison are different from each other. Is there a special reason to use different methods for comparisons? For, UCF101, IS and FVD were used while FVD and ACD were used for the FaceForensis dataset. Also, FVD, PSNR, and SSIM were used for Sky Time-lapse dataset. It is better to conduct a comparative experiment with the same metric and the same compared methods for all datasets.\n\n\nThere are qualitative results in section4.2 about cross-domain video generation. Are there results that have been verified quantitatively?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "7oyJcP0-dxT", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary**\nThis paper addresses the problem of video synthesis --- generating diverse, realistic videos. This paper's core idea is to leverage a fixed, pre-trained GAN model for image synthesis and train a motion generator to produce a sequence of latent vectors to generate image sequences (using the pretrained GAN and the generated latent vectors) are temporally coherent. The specific technical novelties lie in (1) predicting the motion residual and (2) adding contrastive image discriminator to ensure that generated contents in a video are similar. The paper provides an extensive set of experiments demonstrating the proposed method's effectiveness over the state-of-the-art video synthesis models.\n\n**Strength**\n+ The quantitative and visual results are extensive. The performance over existing methods across multiple datasets is significant. The paper also provides an ablation study (Table 4, 5) highlighting the importance of individual components.\n+ The capability of cross-domain video synthesis could be beneficial, particularly when high-quality video datasets are difficult to collect.\n+ The appendix and the supplementary material provide extensive details about the experimental settings and results that would help reproduce the results.\n\n\n**Weakness**\n- When discussing the difference over [Tulyakov et al. 2018], the paper states “…applies h_t as the motion code for the frame to be generated, while the content code is fixed for all frames. However, such a design requires a recurrent network to estimate the motion while preserving consistent content from the latent vector, … difficult to learn in practice”. I do not fully understand why this is the case. It would be clearer if the paper can explain why such a design causes difficulty in learning and why the proposed design could alleviate such problems.\n\n- For motion diversity, why maximizing the mutual information between the hidden vector and the noise vector can prevent mode collapse?\n\n- It seems to me that the proposed method can only handle 1) “subtle” motion, such as facial expressions and 2) short video sequences (e.g., 16 frames). One can see the problem in the synthesized results for UCF-101: inconsistent motion, changing color, or object disappearing over time. It would be interesting to videos with a longer duration (by running the LSTM over many time steps). \n\nIn sum, this is a paper with an interesting idea and extensive experiments. While the results are still not perfect and seem to handle subtle motion, the quantitative and qualitative evaluation show clearly improved results over the previous state-of-the-art.  ", "extra_scores": {}, "extra_text": {}}, {"review_id": "7N3CMd5F1G-", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This article proposes to retrofit a pretrained image generator for video generation. \n\n______\nPros:\n- Novel objective formulation that controls for motion diversity, disentanglement and content matching.\n- High resolution video generation at 1024x1024.\n- Cross domain video generation (i.e. use pretrained image generator from one domain on the motion from another).\n- Using a pretrained image encoder lowers the training cost and improves efficiency.\n- Cost estimates for training the proposed models on commodity hardware are provided.\n\n\n\n\n______\nCons:\n- Prior art missing:\n  - Proposed approach seems very related to DrNet and a discussion on this is not included (https://papers.nips.cc/paper/7028-unsupervised-learning-of-disentangled-representations-from-video)\n  - Proposed approach is missing prior art on efficient video GAN training - (https://www.sciencedirect.com/science/article/abs/pii/S0893608020303397)\n\n\n_________\nQuestions:\n- Are the LSTM encoder and decoder implemented in practice by the same neural network component?\n  - If they are separate components, why not use just one LSTM for both encoding and decoding (the noise vector could be 0 for the first timestep)?\n- How will such an approach handle occlusions, during video. A benchmark on the BAIR pushing dataset would be useful to help clarify this.\n- \"Additionally, each frame in the input video sequence is conditioned on the first frame, as it falls into the distribution of the pretrained image generator, for more stable training.\"\n  - please clarify this statement\n- How does this work relate to DrNET?\n - Unsupervised Learning of Disentangled Representations from Video - https://papers.nips.cc/paper/7028-unsupervised-learning-of-disentangled-representations-from-video\n - This is a very related piece of work and it should be included and discussed in the related works section. \n - DrNET seems capable of challenging most the main claims made in this paper.\n- How does the proposed approach perform on more complex datasets such as Kinetics-600?\n  - It doesn't appear that there is much change in the morphology of video objects temporally, this can be seen on samples from more complex datasets such as UCF-101. \n- What is the baseline performance of the model trained on the UCF dataset of videos, if you do not do any temporal modelling but instead tile the first image temporally to make a video ?\n  - I suspect that for more complex video with complex motion, there is little in the way of motion modelling happening and the proposed model is simply copying and jittering the initial image.\n  - This should be investigated in the paper so as to ascertain the limits of this approach. \n \n_______\nMissing References:\n-  MDP-GAN: Markov Decision Process for Video Generation - https://openaccess.thecvf.com/content_ICCVW_2019/html/HVU/Yushchenko_Markov_Decision_Process_for_Video_Generation_ICCVW_2019_paper.html\n  - Related prior art for video generation\n- LDVD-GAN: Lower Dimensional Kernels for Video Discriminators - https://www.sciencedirect.com/science/article/abs/pii/S0893608020303397\n  - Related prior art for video generation\n  - Relevant to discussion on high resolution video generation as it held the previous state-of-the-art at 512x512 video generation.\n  - Table 1 is missing the LDVD-GAN result of 22.91 +/- .19\n  - Relevant to the discussion on video GAN efficiency since LDVD-GAN is a single-GPU model.\n- DrNET: Unsupervised Learning of Disentangled Representations from Video - https://papers.nips.cc/paper/7028-unsupervised-learning-of-disentangled-representations-from-video\n  - Related prior art for video generation\n  - Related prior art for proposed disentanglement and training approach.\n_______\nOverall this work tries to take on video modelling from a different angle with some success but also leaves many questions unanswered due to gaps in the experimental design and related work (discussed above). I will increase my score should these questions and the other concerns be addressed.\n\n\n__________________________________________________________________________________________________________________________________________\n**Updated review** following revisions:\n- A5: For future reference, I believe the kinetics dataset is now downloadable from a google drive folder: https://github.com/activitynet/ActivityNet/issues/28#issuecomment-602838701\n\n- A6: I am not convinced by your hypothesis in Appendix (Section E) as to why UCF-101 is so challenging. Surely if UCF-101 is not diverse enough, then a model based on stylegan should perform well on it. In-fact I would argue that it is the opposite, that the UCF-101 is very diverse for such a small dataset.\n  - Observe that the datasets compared against UCF-101 in Table 7 are not that diverse, there is a dataset of just faces (FaceForensics), of just sky time-lapses and of just dogs. These are all uni-modal datasets. On the other hand, compare just a subset of the 101 classes in UCF-101; Horse Riding, Military Parade, Baseball Pitch, Billiards Shot, Brushing Teeth,...\n\n  - I would argue that the limitation of your approach (and that of DVDGAN [Clark et al., 2019]) on this dataset stems from the fact that UCF-101 is a small but very diverse dataset. On average, just over 100 samples per class.\n\n\nOverall, the authors have adequately addressed my questions and concerns. They also appear to have done so for all the other reviewers too.\n\nMy recommendation is to **accept** this work for publication to ICLR 2021.\n\nI recommend that the authors open-source their code and pretrained models. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All the reviewers are positive about the paper; R2 and R3 voted for clear accept. Overall, all the reviewers feel that evolution is comprehensive and the results are decent. There is a novel objective formulation that controls for motion diversity, disentanglement and content matching, outperforming existing methods across multiple datasets. High-res videos at 1024x1024 are generated and there is cross-domain video generation. Many good questions were raised by the reviewers, and they were addressed in details in the rebuttal. In particular, the question about subtle motion and short video sequences was raised (which was the concern that the AC had). The AC agrees with the reviewers that the paper warrants a publication. Please address the questions raised by the reviewers in the final version.

Decision: Accept (Spotlight)