VideoGen: Generative Modeling of Videos using VQ-VAE and Transformers
Authors: Yunzhi Zhang
Wilson Yan
Pieter Abbeel
Aravind Srinivas
Venue:  (2021)
Keywords: video generation
vqvae
transformers
gpt

Abstract: We present VideoGen: a conceptually simple architecture for scaling likelihood based generative modeling to natural videos. VideoGen uses VQ-VAE that learns learns downsampled discrete latent representations of a video by employing 3D convolutions and axial self-attention. A simple GPT-like architecture is then used to autoregressively model the discrete latents using spatio-temporal position encodings. Despite the simplicity in formulation, ease of training and a light compute requirement, our architecture is able to generate samples competitive with state-of-the-art GAN models for video generation on the BAIR Robot dataset, and generate coherent action-conditioned samples based on experiences gathered from the VizDoom simulator. We hope our proposed architecture serves as a reproducible reference for a minimalistic implementation of transformer based video generation models without requiring industry scale compute resources. Samples are available at https://sites.google.com/view/videogen

Under review as a conference paper at ICLR 2021
VIDEOGEN: GENERATIVE MODELING OF VIDEOS US- ING VQ-VAE AND TRANSFORMERS
Anonymous authors Paper under double-blind review
ABSTRACT
We present VideoGen: conceptually simple architecture for scaling likelihood based generative modeling natural videos. VideoGen uses VQ-VAE that learns learns downsampled discrete latent representations of a video by employing 3D convolutions and axial self-attention. A simple GPT-like architecture then used to autoregressively model the discrete latents using spatio-temporal position encod- ings, Despite the simplicity in formulation, ease training and a light compute requirement, our architecture is able to generate samples competitive with state-of- the-art GAN models for video generation on the BAIR Robot dataset, and generate coherent action-condittoned samples based experiences gathered from the ViZ. Doom simulator. We hope our proposed architecture serves as reproducible reference for minimalistic implementation of transformer based video generation models without requiring industry scale compute resources. Samples are available https: sites. google com/ wem/vieww/eee ntww//iieogee
INTRODUCTION
Figure 1: 64 64 samples for BAIR and ViZDoom environments generated by VideoGen
Deep generative models of multiple types (Goodfellow et al., 2014; van den Oord et al., 2016b; Dinh et al., 2016) have seen incredible progress in the last few years on multiple modalities including natural images (van den Oord et al., 20 Zhang al., 2019; Brock et al., 2018; 88; Kingma & Dhariwal, 2018; Ho et al., 2019;; Karras et al., 2017; 2019; Van Den Oord et al., 2017; Razavi et al., 2019; Vahdat & Kautz, 2020; Ho et al., 2020: Chen al., 2020), audio waveforms conditioned on language features (van den Oord et al., 2016a; Oord et al., 2017: Binikowsii et al., 2019), natural language in the form of text (Radford et al., 2019; Brown et al., 2020), and music generation (Dhariwal al., 2020). These results have been made possible thanks to fundamental advances in deep learning architectures (He al., 2015; van den Oord al., 2016b;c; Vaswani et al., 2017: Zhang et al., 2019; Menick & Kalchbrenner, 2018) as well as the availability of compute resources (Jouppi et al., 2017: Amodei & Hernandez, 2018) that are more powerful than a few years ago. However, one notable modality that has not seen the same level of progress in generative modeling is high fidelity natural videos. The complexity of natural videos requires modeling correlations across both space and time with much higher input dimensions, thereby presenting natural next challenge for current deep generative models. The complexity of the problem also demands more compute resources which can be considered one important reason for the slow progress in generative modeling of videos.
is useful to build generative models of videos, both conditional and unconditional, as it implicitly solves the problem of video prediction and forecasting. Video prediction (Kalchbrenner et al., 2017:

---

Under review as a conference paper at ICLR 2021
Sønderby al., 2020) can be seen as learning a generative model of future frames conditioned on the past frames. Architectures developed for video generation can be useful in forecasting applications for autonomous driving, such as predicting the future in more semantic and dense abstractions like segmentation masks (Luc et al., 2017). Finally, building generative models of the world around us is considered one way measure understanding of physical common sense (Lake al., 2015).
Multiple classes of generative models have been shown to produce strikingly good samples such as autoregressive models (van den Oord al., 2016b;e; Menick & Kalchbrenner, 2018; Radford et al., 2019; Chen et al., 2020), generative adversarial networks (GANs) (Goodfellow et al., 2014; Radford et al., 2015), variational autoencoders (VAEs) (Kingma Welling, 2013; Kingma et al., 2016; Vahdat & Kautz, 2020), Flows (Dinh et al., 2014; 2016; Kingma & Dhariwal, 2018), vector quantized VAE (VQ-VAE) (Van Den Oord et al., 2017: Razavi et al., 2019), and lately diffusion and score matching models (Sohl-Dickstel) et al., 2015; Song & Ermon, 2019; Ho et al., 2020). These different generative model families have their tradeoffs: sampling speed, sample diversity, sample quality, ease of training, compute requirements, and ease of evaluation.
To build a generative model for videos, we first make a choice between likelihood-based and adversarial models. Likelihood-basee understood, easy to optimize across a range of batch sizes, and easy to evaluate. Given that videos already present hard modeling challenge due to the nature of the data, we believe likelihood-basee models present fewer difficulties in the optimization and evaluation, hence allowing us to focus on the architecture modeling. Among rikelihood-base models, autoregressive models that work on discrete data in particular have shown great success and have well established training recipes and modeling architectures.
models are convenient to train since the objective is well
Second, we consider the following question: Is it better to perform autoregressive modeling in a downsampled latent space without spatio-temporal redundancies compared to modeling at the atomic level of all pixels across space and time? Below, we present our reasons for choosing the former: Natural images and videos contain lot of spatial and temporal redundancies and hence the reason we use image compression tools such as JPEG (Wallace, 1992) and video codecs such as MPEG (Le Gall, 1991) everyday. These redundancies can be removed by learning a denoised downsampled encoding of the high resolution inputs. For example, 4x downsampling across spatial and temporal dimensions results in 64x downsampled resolution so that the computation of powerful deep generative models is spent on these more fewer and useful bits. As shown in VQ-VAE (Van Den Oord al., 2017), even a lossy decoder can transform the latents to generate sufficiently realistic samples. Furthermore, modeling in the latent space downsampled across space and time instead of the pixel space improves sampling speed and compute requirements due to reduced dimensionality.
The above line of reasoning leads to our proposed model: VideoGen, simple video generation architecture that a minimal adaptation of VQ-VAE and GPT architectures for videos. VideoGen employs 3D convolutions and transposed convolutions (Tran et al., 2015) along with axial atten- tion (Clark et al., 2019; Ho et al., 2019b) for the autoencoder in VQ-VAE in order to be able to learn downsampled set of discrete latents. These latents are then autoregressively generated by a GPT-like (Radford et al., 2019; Child et al.. 2019; Chen et al., 2020) architecture. The latents are then decoded to videos of the original resolution using the decoder of the VQ-VAF.
Our results are highlighted below:
On the widely benchmarked BAIR Robot Pushing dataset (Ebert et al., 2017), VideoGen can generate realistic samples that are competitive with existing methods such as DVD-GAN (Clark et al., 2019), achieving an FVD of 112 when benchmarked with real samples, and an FVD* (Razavi et al., 2019) of 94 when benchmarked with reconstructions.
VideoGen can easily be adapted for action conditional video generation. We present qualitative
results on the BAIR Robot Pushing dataset and Vizdoom simulator (Kempka et al., 2016).
3. We present ablations showing that employing axial attention blocks the VQ-VAE and spatio- temporal position encodings in the Transformer
helpful design choices in VideoGen.
4. Our results are achievable with a maximum of 8 Quadro RTX 6000 GPUs (24 GB memory), significantly lower than the resources used in prior methods such as DVD-GAN (Clark al., 2019) (32 to 512 16GB TPU (Jouppi et al., 2017) cores).
Modeling long sequences is a challenge for transformer based architectures due quadratic memory complexity of the attention matrix (Child et al., 2019).
2

---

Under review as a conference paper at ICLR 2021
2 BACKGROUND
2.1 VQ-VAE
The Vector Quantized Variational Autoencoder (VQ-VAE) (Van Den Oord et al., 2017) is a model that learns to compress high dimensional data points into discretized latent space and reconstruct them. The encoder E(x) + h first encodes into a series of latent vectors / which is then discretized by performing a nearest neighbors lookup in codebook of embeddings C { K of size K. The decoder D(e) + å then learns to reconstruct x from the quantized encodings. The VQ-VAE is trained using the following objective:
L
= | D(e)ll +
||g][F[( [((x) ell2 + [sggle] - E(x))1
Ezmmin
where sy refers to a stop-gradient. The objective consists of reconstruction loss Lreom, codebook loss Loodebook. and a commitment loss Leommit- The reconstruction loss encourages the VQ-VAE to learn good representations accurately reconstruct data samples. The codebook loss brings codebook embeddings closer to their corresponding encoder outputs, and the commitment loss is weighted by hyperparameter / and prevents the encoder outputs from fluctuating between different code vectors. An alternative replacement for the codebook loss described in Van Den Oord et al. (2017) is to use EMA update which empirically shows faster training and convergence speed. In this paper, we use the EMA update when training the VQ-VAE.
2.2 GPT
GPT and Image-GPT (Chen et al., 2020) are class of autoregressive transformers that have shown tremendous success modelling discrete data such as natural language and high dimensional images.
These models factorize the data distribution p(z) according to p(x) = -II p(x,(=<))) through masked self-attention mechanisms and are optimized through maximum likelihood. As in Vaswani et al. (2017), the architectures follow the standard design of employing multi-head self-attention blocks followed by pointwise MLP feedforward blocks.
3 VIDEOGEN
Our primary contribution is VideoGen, a new method to model complex video data in computation- ally efficient manner. overview of our method is shown in Fig 2.
Learning Latent Codes In order to learn set of discrete latent codes, we first train a VQ-VAE on the video data. The encoder architecture consists of a series of 3D convolutions that downsample over space-time, followed by attention residual blocks. Each attention residual block is designed as shown Fig 3, where we use LayerNorm (Ba et al., 2016), and axial attention layers follow Ho et al. (2019b).
The architecture for the decoder is the reverse of the encoder, with attention residual blocks followed by series of 3D transposed convolution that upsample over space-time. The position encodings are learned spatio-temporal embeddings that are shared between all axial attention layers the encoder and decoder.
Learning a Prior The second stage our method is to learn a prior over the latents. The prior is learned by training transformer model over the VQ-VAE latents. We follow the iGPT architecture with added dropout after the feedforward and attention block layers for regularization.
Although the VQ-VAE is trained unconditionally, conditional prior. We use two types of conditioning:
generate conditional samples by training
We concatenate a conditional vector before every feedforward block in the transformer. This conditioning method is primarily used for frame conditioning, where the conditioned frame encoded into a conditioning vector by a ResNet (He et al., 2016) backbone and then concatenated.
Concatenation:
3

---

Under review as a conference paper at ICLR 2021
'lepe
CenrdD 2014
Discree Laben
Turradd accenca
Figure 2: We break down the training pipeline into two sequential stages: training VQ-VAE (Left) and training autoregressive transformer in the latent space (Right). The first stage similar to the original VQ- VAE training procedure. During the second stage, VQ-VAE encodes video data to latent sequences training data for the prior model. For inference, first sample latent sequence from the prior, and then use VQ- (Q-VAE to decode the latent sequence to a video sample.
Position Encodling
Norm
Corv 1xtx1
Norm ReLU
Axial Attention
h
ReLU
3x3x3
ReLU
Figure 3: Architecture of the attention residual block in the VQ-VAE as replacement for standard residual blocks.
Conditional Norms: Similar to conditioning methods used in GANs, we parameterize the gain and bias in the transformer Layer Normalization (Ba et al., 2016) layers affine functions of the conditional vector. This conditioning method is used for action conditioning.
EXPERIMENTS
In the following section, we evaluate our method and design experiments answer the following questions:
Can we generate high-fidelity samples from complex video datasets with limited compute?
What architecture design choices in the VQ-VAE and transformer help the most?
4.1 TRAINING DETAILS
All image data is scaled to [-0.5, 0.5) before training. For VQ-VAE training, we use random restarts for embeddings, and codebook initialization by copying encoder latents as described in Dhariwal et al. (2020). In addition, we found VQ-VAE training to more stable (less codebook collapse) when using Normalized MSE for the reconstruction loss, where MSE loss is divided by the variance of the dataset. For all datasets, we train on 64 x 64 videos sequence length 16. More training details can be found in Appendix A.
4

---

Under review as a conference paper at ICLR 2021
b
6
0
0
0
0
0
0
b
6
6
6
6
O
o
1
10
10
10
10
10
10
d
d
d
d
@
9
9
9
8
9
4
4
Figure 4: Moving MNIST samples conditioned on a single given frame (red).
4.2
MOVING MNIST
For Moving MNIST, the VQ-VAE downsamples factor of 4 over space-time (64x total reduction), and contains two residual layers with no attention. We use codebook of 512 codes, each 64-dim embeddings. To learn the single-frame conditional prior, train conditional transformer with 384 hidden features, heads, 8 layers, and ResNet-18 single frame encoder. Fig 4 shows several different generated trajectories conditioned on single frame.
Figure 5: VQ- VAE reconstructions for BAIR Robot Pushing. The original videos contained in green boxes and reconstructions in blue.
3
Figure 6: Samples for BAIR Robot Pushing. (Top) shows samples conditioned single frame. (Bottom) shows samples conditioned on a single frame and action sequence. Although scenarios are different in each trajectory, they all follow similar action pattern. The conditional single frames indicated by red boxes, and the sampled subsequent frames by blue.
5

---

Under review as a conference paper at ICLR 2021
Table Comparing FVD and FVD* values for BAIR on different transformer architectures. C= single-frame conditional, unconditional. The Axial Attention model follows the transformer ar- chitectures described in
Table 1: FVD on BAIR
al. (2019b). GPT and GPT Small follow the same architecture, but GPT Small containing half the number of layers GPT. FVD* is computed similar to FVD, but using reconstructed dataset examples instead of the original dataset examples
2
FVD (1)
SV2P
262.5
LVT
125.8
SAVP
116.4
VideoGen (ours)
112.5
DVD-GAN-FP
109.8
bits/dim
FVD
FVD*
TrVVD-GAN-PP-
103.3
Video Transformer
94 2
Axial (C)
3,94
170.1
133.3
GPT Small (U)
4.53
230.4
187.0
GPT (U)
4.08
191.7
164.2
GPT (C)
2.95
112.5
94.2
4.3 BAIR ROBOT PUSHING
factor of 4 space, and 2 over time (32x reduction), with 4 attention residual blocks. We use a codebook of 2048 codes, each 256-dim embeddings. The transformer prior has hidden size of 384 and 22 layers. For single-frame conditioning, we jointly train a ResNet-34 encoder that encodes the frame to a 512-dim vector.
For BAIR, the VQ-VAE downsamples
Qualitatively, Fig 5 shows VQ-VAE reconstructions on BAIR. Fig shows samples primed with single frame as well as frame and action- conditiooe samples. We can see that our method is able to generate good, realistic looking samples, and generalize to more out of distribution examples as shown in the case of samples for different priming frames with the same actions.
Quantitatively, Table 12 shows FVD results on BAIR, comparing our method with prior work. Although our method does not achieve state of the art, it is able to produce very realistic samples. To have a more thorough understanding of the model performance, we compute two extra metries: (1) FVD of reconstructed VQ-VAE data examples, and (2) an adjusted FVD* metric similar to Razavi et al. (2019), where FVD calculated between samples and a VQ- VQ-VAE VAE reconstructed validation set (as opposed to the actual validation set). Table 2 shows that the adjusted FVD* of our single-frame conditional model is 94.2, coming very close the Video Transformer performance for FVD. This suggests that the FVD sample performance of method is primarily bounded by the VQ-VAR reconstruction quality, as opposed to the transformer prior.
4.4 VIZDOOM
For ViZDoom, we use the same VQ-VAE and transformer architectures as for the BAIR dataset, with the exception that the transformer is trained without single-frame conditioning. We collect the training data by training a policy in each ViZDoom environment, and collecting rollouts of the final trained policies. The total dataset size consists of 1000 episodes of length 100 trajectories, split into an 8:1:1 train validataion / test ratio. We experiment on the Health Gathering Supreme and Battle2 ViZDoom environments, training both unconditional and action-conditioned priors. Fig and Fig 8 show samples from priors trained in each dataset and can see that the VQ-VAE and transformer are able to capture complex 3D camera movements and environment interactions. In addition, action-conditioned samples are visually consistent with the input action sequence and show a diverse range of backgrounds and scenarios under different random generations for the same set of actions.
FSV2P (Babaeizadeh al., 2017). SAVP (Lee al., 2018). DVD-GAN-FP (Clark et al., 2019), Video Transformer (Weissenborn et al., 2019), Latent Video Transformer (LVT) (Rakhimov et al., 2020), and TrIVD- GAN (Luc et al., 2020) are our baselines and we use FVD to compare different models given that adversarial models do not have likelihood metric while the likelihood loss in VideoGen is on the discrete latents and not directly comparable pixel level likelihond based models such as Video Transformer.

---

Under review as a conference paper at ICLR 2021
Figure Samples for ViZDoom health gathering supreme environment. (Top) shows unconditionally generated samples. (Bottom) shows samples conditioned on the same action sequence (turn right and go straight).
Figure Samples for ViZDoom battle2 environment. (Top) shows unconditionally generated samples. (Bottom) shows three samples conditioned on the same action sequence (moving forward and right).
4.5 ABLATIONS
In this section, we perform ablations on various architectural design choice for VideoGen.
Does attention in the VQ-VAE help? We remove the axial attention layers from the VQ-VAE and compare with the original architecture shown in Table 2. Empirically, incorporating axial attention into the VQ- VAE architecture improves reconstruction (NMSE) performance, and has much better reconstruction FVD compared using no attention. Fig 5 shows a qualitative result of the VQ-VAE architecture with axial attention module.
Are other forms of attention better for the transformer? In addition to training transformer with standard self-attention, we test our method using an Axial Transformer (Ho et al., 2019b) as a replacement. Table shows an FVD comparison between a single-frame conditional Axial Transformer, and the standard GPT model using full self-attention. Using full attention achieves better FVD performance than an Axial Transformer. We hypothesize that this may be due to the full attention transformer being better able to capture single-frame conditional information than the Axial Transformer.
Can a smaller transformer be used? Computational efficiency is primary advantage method, where we can first use the VQ-VAE to downsample by space-time before learning an autoregressive prior. Lower resolution latents allow to train larger and more expressive prior to learn complex data distributions. In order to better understand why these advantages are important, ablation transformer size demonstrate that training larger transformer produces

---

Under review as a conference paper at ICLR 2021
Table 4: Ablation on positional encodings
Table 3: Ablation on attention in VQ-VAE. FVD
with reconstructed examples
Position Encoding
bits/dim (t)
FVD (4)
Sin-Cosine
3.94
232
VQ-VAE Architecture
NMSE (L)
FVD ())
None
5.96
1697
No Attention 0.014
156
Temporal Only
5.84
1360
With Attention
0.010
125
Spatial Only
4.82
300
Spatial + Temporal
4.53
230
significantly better result. Table 2 shows the results of training smaller unconditional transformer on BAIR, where it achieves both a worse bits per dim and FVD (230) compared to the larger transformer (192).
Do learned spatiotemporal positional encodings help? Finally, we study the effects of using learned spatiotemporal positional encodings when training the GPT model. Table 4 shows ablations on comparing different positional encodings. We see that adding in both space and time broadcasting aspects is crucial to better learn the data, and achieve lower FVD. Using sine-cosine encodings performs roughly the same as spatiotemporal encodings, with a slightly worse FVD.
RELATED WORK
Video Prediction The problem of video prediction (Srivastava et al., 2015) is quite related to video generation in that the latter is one way to solve the former. Plenty of methods have been proposed for video prediction on the BAIR Robot dataset (Finn et al., 2016; Ebert et al., 2017; Babaeizadeh et al., 2017; Denton et al., 2017; Denton & Fergus, 2018; Lee et al., 2018) where the future frames predicted given the past frame(s) and (or) action(s) of robot arm moving across multiple objects thereby benchmarking the ability of video models to capture object-robot interaction, object permanance, robot motion, etc. Translating videos to videos is another paradigm to think about video prediction with a prominent example being vid2vi Wang et al. (2018). The vid2vid framework uses automatically generated supervision from more abstract information such as semantic segmentation (Lue et al., 2017 7) masks, keypoints, poses, edge detectors, etc to further condition the GAN based video translation setup.
Video Generation Most modern generative modeling architectures allow for easy adaptation of unconditional video generation to conditional versions through conditional batch-norm (Brock et al., 2018), concatenation (Salimans et al., 2017; van den Oord et al., 2016c), etc. Video Pixel Net- works (Kalchbrenner al., 2017) propose a convolutional LSTM based encoding of the past frames to be able to generate the next frame pixel by pixel autoregressively with PixeICNN (van den Oord et al., 201 6c) decoder. The architecture serves both as a video generative as well as predictive model, optimized through log-likelihood loss at the pixel level. Subscale Video Transformers (Weissenborn et al., 2019) extend the idea Subscale Pixel Networks (Menick & Kalchbrenner, 2018) for video generation at the pixel level using the subscale ordering across space and time. However, the sampling time and compute requirements are large for these models. In the past, video specific architectures have been proposed for GAN based video generation with primitive results by Vondrick et al. (2016). Recently, DVD-GAN proposed by Clark et al. (2019) adopts a BigGAN like architecture for videos with disentangled (axial) non-local (Wang et al., 2017) blocks across space and time. They present wide range of results, unconditional, past frame(s) conditional, and class conditional video generation. However, the DVD-GAN architecture is hard to replicate without industry scale compute resources. Furthermore, training DVD-GAN without a clean open source implementation is difficult due to instabilities often encountered in GAN training. Other examples of prior work with video generation of GANs include Saito et al. (2017), Tulyakov et al. (2018), Acharya et al. (2018), Yushchenko et al. (2019). In addition, Saito Saito (2018) and Kahembwe & Ramamoorthy (2020) propose more scalable and efficient GAN models for training on less compute. Our approach builds on top of VQ- VAEE (Van Den Oord et al., 2017) by adapting it for video generation. A clean architecture with VQ- VAE for video generation has not been presented yet and hope VideoGen is useful from that standpoint. While VQ- VAE-2 (Razavi et al., 2019) proposes using multi-scale hierarchical latents, the pipeline inherently more complicated. For simplicity, ease of reproduction and presenting the
8

---

Under review as a conference paper at ICLR 2021
first VQ- VAE based video generation model with minimal complexity, we stick with the single scale setup.
6 CONCLUSION
We have presented VideoGin,, new video generation architecture adapting VQ-VAE and Transformer models typically used for image generation to the domain of videos with minimal modifications. We have shown that VideoGen is able to synthesize videos that are competitive with state-of-the-arr GAN based video generation models by requiring orders of magnitude fewer resources. We have also presented ablations key design choices used in VideoGen which we hope is useful for future design of architectures video generation. We hope that VideoGen serves as a simple baseline that easy reproduce and build upon for future research in this challenging topic.
REFERENCES
Dinesh Acharya, Zhiwu Huang, Danda Pani Paudel, and Luc Van Gool. Towards high resolution video generation with progressive growing of sliced wasserstein gans. arXiv preprint arXiv: 181.00440 2018.
Dario Amodei and Danny Hernandez. Ai and compute. Heruntergeladen von https:///blog. openai. com/aiand-compute, 2018.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.
Mohammad Babaeizadeh, Chelsea Finn, Dumitru Erhan, Roy H Campbell, and Sergey Levine. Stochastic variational video prediction. arXiv preprint arXiv: /252, 2017.
Mikolaj Binkowski, Jeff Donahue, Sander Dieleman, Aidan Clark, Erich Elsen, Norman Casagrande, Luis C Cobo, and Karen Simonyan. High fidelity speech synthesis with adversarial networks. arXiv preprint arXiv: 1909. 1646, 2019.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: 1009: 2018
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165, 2020.
Mark Chen, Alec Radford, Rewon Child, Jeff Wu, Heewoo Jun, Prafulla Dhariwal, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In Proceedings of the 37th International Conference on Machine Learning, 2020.
Rewon Child, Scott Gray, Alec Radford, and llya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.100 2019.
Aidan Clark, Jeff Donahue, and Karen Simonyan. Adversarial video generation on complex datasets, 2019.
Emily Denton and Rob Fergus. Stochastic video generation with learned prior. arXiv preprint arXiv:/802.077 2018.
Emily L Denton et al. Unsupervised learning of disentangled representations from video. In Advances in neural information processing systems, pp. 4414-4423, 2017.
Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. Jukebox: A generative model for music. arXiv preprint 200400000500 2020.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. arXiv preprint arXiv: 14 10.85 16, 2014.

---

Under review as a conference paper at ICLR 2021
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using Real NVP. arXiv preprint arXiv:1605.08880 2016.
Frederik Ebert, Chelsea Finn, Alex X Lee, and Sergey Levine. Self-supervised visual planning with temporal skip connections. arXiv preprint arXiv:1710.05268, 2017.
Chelsea Finn, Ian Goodfellow, and Sergey Levine. Unsupervised learning for physical interaction through video prediction. In Advances in neural information processing systems, pp. 64-72, 2016.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural informa- tion processing systems. pp. 2672-2680, 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. arXiv preprint arXiv:1512.03338; 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Jonathan Ho. Xi Chen, Aravind Srinivas, Yan Duan, and Pieter Abbeel. Flow++: Improving flow- based generative models with variational dequantization and architecture design. arXiv preprint 1002000502200225... 2019a.
Jonathan Ho. Nal Kalchbrenner, Dirk Weissenborn, and Tim Salimans. Axial attention in multidi- mensional transformers. arXiv preprint arXiv:1912.12188 2019b.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising difftusion probabilistic models. arXiv preprint arXiv:2006. 1239, 2020.
Norman P Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, et al. In-datacenter performance analysis of
tensor processing unit. In Proceedings of the 44th Annual International Symposium Computer Architecture, pp. 2017.
Emmanuel Kahembwe and Subramanian Ramamoorthy. Lower dimensional kernels for video discriminators. Neural Networks, 132:506-5221 2020.
Nal Kalchbrenner, Aäron Oord, Karen Simonyan, Ivo Danihelka, Oriol Vinyals, Alex Graves, and Koray Kavukcuoglu. Video pixel networks. In International Conference on Machine Learning, pp- 1771-1779. PMLR, 2017.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv: 10196, 2017.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4401-4410, 2019.
Michat Kempka, Marek Wydmuch, Grzegorz Runc, Jakub Toczek, and Wojciech Jaskowski. Vizdoom: A doom-based research platform for visual reinforcement learning. In 2016 IEEE Conference on Computational Intelligence and Games (CIG), pp. 1-8. IEEE, 2016.
Diederík Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible Ixl convolutions.
arXiv preprint arXiv: 1807 0007900339.000 2018
Diederik P Kingma and Max Welling. Auto-encoding variational Bayes. Proceedings of the 2nd International Conference on Learning Representations, 2013.
Diederik P. Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Im- proving variational inference with inverse autoregressive flow. In Advances in Neural Information Processing Systems, 2016.
Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 15066266):1332--2 2015.
10

---

Under review as a conference paper at ICLR 2021
Didier Le Gall. Mpeg: A video compression standard for multimedia applications. Communications of the ACM, 34(4))46-58, 1991.
Alex X Lee, Richard Zhang, Frederik Ebert, Pieter Abbeel, Chelsea Finn, and Sergey Levine. Stochastic adversarial video prediction. arXiv preprint arXiv: 01520,1233, 2018.
Pauline Luc, Natalia Neverova, Camille Couprie, Jakob Verbeek, and Yann LeCun. Predicting deeper into the future of semantic segmentation. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017.
Pauline Luc, Aidan Clark, Sander Dieleman, Diego de Las Casas, Yotam Doron, Albin Cassirer, and Karen Simonyan. Transformation-basee adversarial video prediction large-scale data. arXiv preprint urXiv:2003.0403: 2020.
Jacob Menick and Nal Kalchbrenner. Generating high fidelity images with subscale pixel networks and multidimensional upscaling. arXiv preprint arXiv: /0 2018.
Aaron van den Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Koray Kavukcuoglu, George den Driessche, Edward Lockhart, Luis C Cobo, Florian Stimberg, et al. Parallel wavenet: Fast high-fidelity speech synthesis. arXiv preprint arXiv:
2017.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.0644 2015.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 188):9, 2019.
Ruslan Rakhimov, Denis Volkhonskiy, Alexey Artemov, Denis Zorin, and Evgeny Burnaev. Latent video transformer. arXiv preprint arXiv:2006. 10704, 2020.
Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. In Advances in Neural Information Processing Systems, pp. 14876, 2019.
Masaki Saito and Shunta Saito. Tganv2: Efficient training of large models for video generation with multiple subsampling layers. arXiv preprint arXiv:1811009245 2018.
Masaki Saito, Eicchi Matsumoto, and Shunta Saito. Temporal generative adversarial nets with singular value clipping. In Proceedings of the IEEE international conference on computer vision, pp. 2830-2839, 2017.
Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P Kingma. Pixelenn++: Improving the pixelenn with discretized logistic mixture likelihood and other modifications. arXiv preprint urXiv:/701.000 2017.
Jascha Sohl-Dickstein, Eric A Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. arXiv preprint arXiv: 153300355,, 2015.
Casper Kaae Sønderby, Lasse Espeholt, Jonathan Heek, Mostafa Dehghani, Avital Oliver, Tim Salimans, Shreya Agrawal, Jason Hickey, and Nal Kalchbrenner. Metnet: A neural weather model for precipitation forecasting. arXiv preprint 1XXi::2003.12144 2020.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems, pp. 11918-11930 2019.
Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov. Unsupervised learning of video representations using Istms. In International conference on machine learning, pp. 843-852, 2015.
Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spa- tiotemporal features with 3d convolutional networks. In Proceedings of the IEEE international conference on computer vision, pp. 1489-44977 2015.
Sergey Tulyakov, Ming- Yu Liu, Xiaodong Yang. and Jan Kautz. Mocogan: Decomposing motion and content for video generation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1226-1535. 2018
11

---

Under review as a conference paper at ICLR 2021
Arash Vahdat and Jan Kautz. Nvae: A deep hierarchical variational autoencoder. arXiv preprint arXiv:2007.03898, 2020.
Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint arXiv:1609.03499. 2016a.
Aaron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. International Conference on Machine Learning (ICML), 2016b.
Aaron van den Oord, Nal Kalchbrenner, Oriol Vinyals, Lasse Espeholt, Alex Graves, and Ko- ray Kavukcuoglu. Conditional image generation with pixelenn decoders. arXiv preprint arXiv:1606.05338,, 2016c.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. In Advances in Neural Information Processing Systems, pp. 6306-6315, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. arXiv preprint arXiv:1706.03762. 2017.
Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dynamics. In Advances in neural information processing systems, pp. 613-621, 2016.
Gregory K Wallace. The jpeg still picture compression standard. IEEE transactions on consumer
electronics, 38(1)xxviiixxx> -xxiv,, 1992.
Ting-Chun Wang, Ming-Yu Yu Liu, Jun-Yan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. Video-to-video synthesis. arXiv preprint arXiv:1808.06601 2018.
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. arXiv preprint 17X1v:1711.07771, 2017.
Dirk Weissenborn, Oscar Täckström, and Jakob Uszkoreit. Scaling autoregressive video models. arXiv preprint urXiv:1906.026:: 2019.
Vladyslav Yushchenko, Nikita Araslanov, and Stefan Roth. Markov decision process for video generation. In Proceedings of the IEEE International Conference Computer Vision Workshops, pp. 0-0, 2019.
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. In International Conference on Machine Learning, pp. 7354-7363. PMLR, 2019.
12

---

Under review as a conference paper at ICLR 2021
A ARCHITECTURE DETAILS AND HYPERPARAMETERS
A.1
VQ-VAE ENCODER AND DECODER
Table 5: Hyperparameters of VQ- VAE encoder and decoder models for Moving MNIST, ViZDoom
(HGS Health Gathering Supreme), and BAIR
Moving MNIST
ViZDoom (HGS)
BAIR ViZDoom (Battle2)
Input size
16 x 64 x 64
16 x 64 x 64
16 x 64 x 64
Latent size
4 16 x 16
16 16
x 16 16
3 (commitment loss coefficient)
0.25
0.25
0.25
Batch size
64
64
Learning rate
7 x 10
7 x 10
7 x 10-4
Hidden units
240
240
240
Residual units
128
128
128
Residual layers
2
Uses attention
No
Yes
Yes
Codebook size
512
2048
2048
Codebook dimension
64
256
256
Encoder filter size
3
3
Upsampling conv filter size
Training steps
20k
75K
75k
A.2 PRIOR NETWORKS
Table 6: Hyperparameters of prior networks for Moving MNIST, ViZDoom (HGS), BAIR and ViZDoom (Battle2).
Moving MNIST
ViZDoom (HGS)
BAIR
ViZDoom (Battle2)
Input size
4 x 16 x 16
4 x 16 x 16
8 x 16 x 16
8 x 16 x 16
Conditional sizes
1 1 x 64
60
3 3 64,
315
Batch size
32
32
32
32
Learning rate
3 x 10
3 x 10
3 x 10
3 x 10
Vocabulary size
512
2048
2048
2048
Attention heads
4
4
Attention layers
8
22
22
22 None
Resnet depth
18
None
34
Resnet units
512
None
512
None
Dropout
0.1
0.1
0.1
0.1
Training steps 40k
80k
80k
80k
80k
13


Reviews:
[{"review_id": "l1u1J_91-ts", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: Authors propose to model video by combining a VQ-VAE encoder-decoder model and a GPT model for the prior.\n\nThe primary contribution as stated by the authors: \"Our primary contribution is VideoGen, a new method to model complex video data in a computationally efficient manner\"\n___________\nPros:\n-\nAn interesting model and an ablation of its components.\n\n___________\nCons:\n-\n- The primary contribution is stated but not validated. The claim is a new method to model complex video efficiently.\n - There is no experiments and/or benchmarks validating this claim anywhere in the paper.\n - There is work on efficiency in the video generation field that is neither cited nor benchmarked against.\n   - TGANv2 (https://link.springer.com/article/10.1007%2Fs11263-020-01333-y) and LDVD-GAN (https://www.sciencedirect.com/science/article/abs/pii/S0893608020303397) come to mind.\n   -  \"Computational efficiency is a primary advantage to our method, where we can first use the VQ-VAE to downsample by space time before learning an autoregressive prior\" - TGANv2, LDVD-GAN and DVD-GAN also do this .\n\n- Some questionable highlights:\n - \"VideoGen can generate realistic samples that are competitive with existing methods such as DVD-GAN\"\n   - A very weak highlight because several existing methods already do this better as shown in Table 1. and DVD-GAN is not the state-of-the-art for this benchmark as shown in the same table.\n - \" VideoGen can easily be adapted for action conditional video generation\" \n   - This is applicable to every video generation model\n -  \"Our results are achievable with a maximum of 8 Quadro RTX 6000 GPUs (24 GB memory),\nsignificantly lower than the resources used in prior methods such as DVD-GAN\" \n   - This claim is not experimentally validated. DVD-GAN is also trainable on 8 Quadro RTX 6000 GPUs (24 GB memory). I would go further to argue that DVD-GAN would train faster and result in a higher performance than VideoGen. I would like to see a head to head benchmark or at the very least the wall clock time for training both the GPT prior and the VQ-VAE encoder-decoders.\n\n- Selective Citation: The video generation and prediction field has been around for a long time now. It is hard to believe that the authors can manage to find and cite every relevant (un)published paper by google and deepmind authors yet they fail to find work published by other groups in this field. They then go on to talk about the slow progress in the field of video generation without acknowledging all the work being done in this field. The following statements highlight this: \n - \"However, one notable modality that **has not seen the same level of progress** in generative modeling is high fidelity natural videos. \"\n - \" The complexity of the problem also demands more compute resources which can be considered as one important reason for the **slow progress** in generative modeling of videos.\"\n\n\n- Missing References to published articles (related to the previous point)\n   - TGAN: Temporal GAN - ICCV 2017 (First appeared on Arxiv - Nov 2016) - https://openaccess.thecvf.com/content_iccv_2017/html/Saito_Temporal_Generative_Adversarial_ICCV_2017_paper.html\n   - MoCoGAN - CVPR 2018  (First appeared on Arxiv - Jul 2017) - https://openaccess.thecvf.com/content_cvpr_2018/html/Tulyakov_MoCoGAN_Decomposing_Motion_CVPR_2018_paper.html\n   -  Progressive Video GAN - Masters Thesis (First appeared on Arxiv - Oct 2018) - https://arxiv.org/abs/1810.02419\n   - MDP-GAN: Markov Decision Process for Video Generation - ICCV 2019 (First appeared on Arxiv - Sep 2019) - https://openaccess.thecvf.com/content_ICCVW_2019/html/HVU/Yushchenko_Markov_Decision_Process_for_Video_Generation_ICCVW_2019_paper.html\n   - TGANv2: Train Sparsely, Generate Densely -  Journal of Computer Vision 2020 - (First appeared on Arxiv - Nov 2018) - https://link.springer.com/article/10.1007%2Fs11263-020-01333-y\n   - LDVD-GAN: Lower Dimensional Kernels for Video Discriminators - Journal of Neural Networks 2020 -  (First appeared on Arxiv - Dec 2019) - https://www.sciencedirect.com/science/article/abs/pii/S0893608020303397\n- If we were to include unpublished preprints on arxiv in this area, this list would at least double in size.\n\n\n \n___________\nSpecific Points:\n- \"However, one notable modality that has not seen the same level of progress in generative modeling is high fidelity natural videos. The complexity of **natural videos** requires modeling correlations across both space and time with much higher input dimensions, thereby presenting a natural next challenge for current deep generative models\" \n - The only natural video dataset benchmarked on is BAIR, the rest are all synthetic. Please benchmark on other datasets of natural video such as UCF101 and Kinetics-600 which also have comparative benchmarks at similar spatio-temporal resolutions. \n\n- \"Can we generate high-fidelity samples from complex video datasets with limited compute?\" \n   - Please address and expand on this point. It is currently left unanswered.\n\n___________\nCurrent recommendation: Rejection\n-\nAll in all, this paper is lacking in novelty and does not do a good job of convincing readers of its primary contributions. The ablation studies provide for the most interesting insights with regard to this work. The BAIR evaluations show that the proposed model is more expensive and has a lower performance than many existing models. The claims of efficiency are also questionable given that the vqvae prior is notoriously expensive to train for image models, let alone video models and there is no head to head comparison or wall clock benchmark to demonstrate otherwise. Lastly, the very selective referencing of work situated around google and deepmind while ignoring related and highly relevant (and famous) work from scientists in other institutes is detrimental to research in this field. I am happy to update my review and score if these issues are addressed. But this work in it's current form is not publishable at any conference.", "extra_scores": {}, "extra_text": {}}, {"review_id": "1GswAMpjGpL", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "After rebuttal: \nAuthors' responses do not address any of my concerns, and I completely agree with other reviewers regarding lack of clarity, evaluation, and novelty. The current form of the paper is not ready to be published. I decrease my score to reject. \n--------------------------------------\nSummary:\nThis paper presents a model combining VQ-VAE and GPT-like autoregressive model. Authors claim that the proposed model is light and easy to train compare to other generative models. The experiments on Moving MNIST, the BAIR Robot datasets, and ViZDoom show that the model can produce high quality and coherent action-conditioned samples. \n--------------------------------------\nPros:\n+ The model generates realistic and high quality video frames. \n+ The proposed model is evaluated on diverse scenarios: the synthetic, robotics, and simulator-based dataset with action conditioned or unconditioned. \n+ Section 4.5 ablations are very helpful. \n--------------------------------------\nCons:\n1. *Limited comparisons:*\n    1. Quantitative comparisons on Moving MNIST and ViZDoom are missing. \n    2. Qualitative results are shown on all datasets but without any comparisons with other models. Since this paper has limited quantitative comparisons and no qualitative comparisons, it is difficult to judge the performance of the model. \n    3. I believe [1] is very related to the proposed model. Comparing with [1] is required. \n2. *Justifying the claim about the light model:* \nAuthors claim that the proposed model requires light compute resources than other models. I understand the proposed model used significantly lower resources than others. The question is: is it due to less memory requirement or computations? To justify this claim, a comparison of model size, latency, and FLOPs with competing models, such as DVD-GAN-FP  [Clark2019], Video Transformer [Weissenborn2019], and Flow-based model [1] are necessary. \n3. *Scaling-up with the proposed model:*\nAuthors claim that the proposed model is efficient due to autoregressive modeling on downsampled latent space. Other generative models like DVD-GAN-FP  [Clark2019] and Video Transformer [Weissenborn2019] provide an evaluation on Kinetics-600. Can the proposed model generate comparable or better video frames on more realistic datasets?\n4. *Clarity:*\n    1. In Figure 6 caption, authors claim that '(Bottom) shows samples conditioned on a single frame and action sequence. Although scenarios are different in each trajectory, they all follow a similar action pattern.' Does it mean that the samples are conditioned on the same action sequence for different video sequences? I don't think the examples are following a similar action pattern. It requires more explanations. \n    2. Mistakes and typos. \n        - I cannot find the result without the axial attention layers from the VQ-VAE in Table 2 (section 4.5 first paragraph).\n        - The references of the models in Table 1 are missing. \n        - Page 6 last sentence, 5 -> Fig 5.\n    3. It would be easier to read the table 2 with more detailed caption. \n        - a description of Axial\n        - a difference between GPT and GPT Small\n        - a description of bits/dim, the difference between FVD and FVD*\n    4. Descriptions of experimental setup are missing. The results can not be easily reproduced. \n        - Vizdoom data generation process\n        - The number of samples in training/validation/test sets\n        - The resolutions of videos\n        - Any proprocessing (if there is any)\n        - The architecture of GPT Small compare to GPT\n5. *Missing references:* [1,2]\n--------------------------------------\n[1] Kumar, et al., VideoFlow: A Conditional Flow-Based Model for Stochastic Video Generation, ICLR 2020.\n[2] Franceschi, et al., Stochastic Latent Residual Video Prediction, ICML 2020.", "extra_scores": {}, "extra_text": {}}, {"review_id": "1ColeZ9fok0", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**[Quick summary ahead]**\nThough I ended up with a negative score, I like the overall idea and intuition, as well as some of the analysis. However, without a clear distinction from prior arts, it is hard to call an interesting idea a contribution. Meanwhile, the analysis and ablation are less meaningful before other major problems fixed. Consequently, I do not list the pros of this work at this moment, since it is still unclear to me.\n\nMy main concerns are (corresponding to the 1-5 in the main review below):\n    1. Requires a clear justification of the main contribution and tradeoffs, as well as a clear comparison with prior arts other than quality metrics (i.e., FVD) alone.\n    2. The absence of a descriptive analysis regarding the performance or behavioral differences from prior arts.\n    3. Writing problems in the methodology section.\n    4. A better clarification of the main novelty.\n    5. (Optional) Compare with a classical video generative model by modifying the architecture of CNNs and LSTM with VQ-VAE and GPT.\n\n\n**[Main review]**\n1. The paper lacks clarity in its positioning in terms of quality-efficiency tradeoff. In the experiment section, the quantitative results, as well as the authors' arguments (i.e., \"Although our method does not achieve state of the art\") agree that the performance is still not compatible with the state-of-the-art models. Meanwhile, in the abstract and conclusion, the authors mention: \"our architecture is able to generate samples competitive with state-of-the-art GAN models.\" I am confused about such a situation. Could the authors make it less vague and consistent?\nI understand this paper is not generation-quality-oriented. But when the method does not achieve state-of-the-art and claims it pose certain kinds of tradeoffs, then the authors are responsible for seriously quantifying and comparing the tradeoffs and show that the tradeoffs are significant and preferable.\nHere, I list the tradeoffs claimed by the authors:\n    - A. Simplicity in the formulation:\nNot discussed in this paper. This is a very subjective statement but still needs to be carefully justified and compared against prior arts in the paper.\n    - B. Ease of training:\nNot discussed in this paper. What kind of ease? Training stability and convergence (then the authors should report a learning curve)? Hyperparameters agnostic (then the authors should report robustness against different hyperparameters)? Easy to reproduce (this point is awkward in all sense)? The authors should make it specific.\n    - C. Light compute requirement:\n        - (i) Only compared with DVD-GAN, which is well-known in training with super-large batch size (512), at the end of page 2. Do the authors try to evaluate the quality of DVD-GAN with the same amount of GPU resource (by reducing batch size and number of parameters) as VideoGen? Furthermore, optimally, the authors should compare with DVD-GAN using the same architectural design of VideoGen, as the architectures of VQ-VAE and GPT are not the main contribution of this paper.\n        - (ii) The computational resource should be measured with GPU-days, not the number of GPUs. There are plenty of quick workarounds to reduce memory requirements, like reducing the batch size and the number of features. The real problem is whether the models can converge to a given performance with the same GPU-days.\n        - (iii) What about the computational resource used by SAVP and Video Transformer listed in Table 1? Do VideoGen achieves a better tradeoff against those methods?\nI believe the authors should take it more seriously on measuring the tradeoffs, especially when the tradeoffs are the main contributions.\n2. To be honest, I am not very sensitive to the FVD score numbers (I know the definition, but not familiar with how large the perceptual differences are with given values). But the 146 FVD from VideoGen doesn't seem very close to its opponents with 116, 109 and 94 FVD. Especially the Video Transformer has only two points of variance, which may imply the degradation from 94 to 146 is quite a large number.\nFurthermore, the background occupies a large area and is pretty static on the BAIR dataset. It is intuitively sound that, at least to me, such a performance difference is visually significant. Could the authors clarify the overall perceptual or behavioral differences between VideoGen from the other methods? I believe such a descriptive comparison and analysis are pretty common and should be presented in this paper.\n3. The methodology section is very vague, nearly poorly written. The method is supposed to be the main contribution of this work, it is a bit hard to understand why this section is written carelessly.\n    - A. \"In order to learn a set of discrete latent codes.\" What is the shape and design of the latent code? Does it consider a temporal temporal dimension or just a flat code? The authors should specify the basic properties here, even if it is the same as what the authors have mentioned in the background section. The background is not a part of the proposed method.\n    - B. \"The prior is learned by training a transformer model over the VQ-VAE latents.\" I would not call this a clear explanation of how a model is applied. What are the inputs and outputs? How are the latents used as sequential data? The authors do provide Figure 2 to illustrate the idea, but Section 3 itself should be self-contained, and figures should be a complementary explanation of descriptive statements or mathematical forms. In fact, I believe the caption of Figure 2 should be presented in Section 3, but with more details.\n    - C. \"Conditional Norms.\" Where is the citation to this component? There are multiple different implementations of a conditional normalization layer. The authors even do not specify the type of normalization. It can be called a conditional batch normalization if it normalizes across a batch, or an adaptive instance normalization if it normalizes across channel dimensions within an instance.\n4. Methodology-wise, the main contribution is more like partitioning a standard video generative model into two stages training, (a) reconstruction, and (b) diversity modeling. The VQ-VAE and GPT are only a change of backbone architecture design instead of the real contribution. I would recommend the authors rethink how they present their main contributions, instead of abusing the name of other well-known models.\n\n5. (Continued 4.) Technically speaking, though the partitioning is reasonable, separating a jointly-optimizable model into a two-stage training pipeline is a bit awkward, and obviously responsible for a certain level of quality degradation.\nI would recommend (not essential) the authors to have a baseline with the same architecture as VideoGen, but the VQ-VAE and GPT modules are jointly optimized (though expected to have a smaller batch size). I would be surprised if such a baseline does not perform better than VideoGen.\n\n\n**[Minor comments]**\n\n1. (Typo) Line 2-3 in the abstract, \"learns learns\"\n2. The name of the model/method is too generic. It will be problematic for future papers referring to the proposed method. I would recommend the authors to make it more specific to the main feature or novelty of the proposed method.\n3. Figure 2 is too sparse, while the texts in the figure are not friendly for reading.\n4. The iGPT (I suppose is image-GPT), in Section 3 is never defined.", "extra_scores": {}, "extra_text": {}}, {"review_id": "0fKLH55RRIC", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a generative model to synthesize videos using VQ-VAEs. The scheme works in latent space by using embeddings for video sequences learnt by the VQ-VAE. For inference, an autoregressive transformer prior for video sequences is learnt, which upon sampling from and sending to the VQ-VAE decoder, generates unconditional (or conditional) samples of video. To learn video embeddings, the paper uses a 3D convolutional network, with an extra dimension for time. \n\nPros:\n- Simple, principled setup\n- Architectural novelties for videos (3D CNN, transformer prior) \n\nCons:\n- I feel that the development is slightly incremental, compared with the original VQ-VAE work. \n- Not enough analysis of latent space. For example, the original VQ-VAE work looks at a few experiments where the scene is traversed by moving 'forward' and 'right' (Figure 7 in [1]). I would have hoped that we had some experiments that show the virtues of working in latent space. \n- Codebook collapse: it would be nice to have some more analysis of this component of the model. \n- Other comments on analysis: There are many components in the setup, many of which need some discussion and analysis for this kind of work such as axial attention, the transformer model, latent spaces, etc. \n- How does this model perform on larger image sequences, and larger number of timesteps?\n- Other tasks: This work looks at the task of video generation. But there are many other areas of practical application where we can benefit from modeling video sequences. How does, for example, the model work with image segmentation, or tracking?\n\nOverall: The work is interesting, but does not seem to have sufficient novelty other than having a different architecture design than used in the original VQ-VAE work. That being said, there's a lot to learn for practitioners if the authors were to put up a detailed write up on architectures and experiments. \n\n[1] VQ-VAE: https://arxiv.org/abs/1711.00937\n\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper focuses on the problem of high quality video generation. It approaches the problem by extending VQ-VAE to videos, where a GPT is used to model the low dimensional representation of the VAE. As agreed upon by the authors and the reviewers, the proposed method is simple and produces interesting results. 

Based on all the reviews and the subsequent discussions, it seems that the reviewers' comments were mostly not addressed and they maintain their stance with regards to the paper's technical novelty, empirical justification of the paper's claims (specifically the claim on computational efficiency), and the rigorous comparison with prior art. The authors themselves make it clear that technical novelty was not the main driving force in this paper. However, in this case, it would be expected that the major claims of the paper be very clearly justified (especially with experiments and analysis) and comparison with other methods be more thorough. It seems that these latter two points remain in the latest revision of this paper. Since the paper shows promise, the authors are recommended to take the reviewers' comments and suggestions into consideration to produce a stronger and more thorough submission in the future.
Decision: Reject