Predicting Video with VQVAE
Authors: Jacob C Walker
Ali Razavi
Aaron van den Oord
Venue:  (2021)
Keywords: Generative Models
Video Generation
Video Forecasting
Autoregressive Models
VQVAE
Computer Vision

Abstract: In recent years, the task of video prediction---forecasting future video given past video frames---has attracted attention in the research community. In this paper we propose a novel approach to this problem with Vector Quantized Variational AutoEncoders (VQ-VAE). With VQ-VAE we compress high-resolution videos into a hierarchical set of multi-scale discrete latent variables. Compared to pixels, this compressed latent space has dramatically reduced dimensionality, allowing us to apply scalable autoregressive generative models to predict video. In contrast to previous work that has largely emphasized highly constrained datasets, we focus
on very diverse, large-scale datasets such as Kinetics-600. We predict video at a higher resolution, 256$\times$256, than any other previous method to our knowledge. We further validate our approach against prior work via a crowdsourced human evaluation.

Under review as a conference paper at ICLR 2021
PREDICTING VIDEO WITH VQVAE
Anonymous authors
Paper under double-bling blind review
ABSTRACT
In recent years, the task of video prediction- forecasting future video given past video frames-has attracted attention in the research community. In this paper we propose novel approach to this problem with Vector Quantized Variational AutoEncoders (Q--AE). With VQ- VAE we compress high-resolution videos into a hierarchical set of multi-scale discrete latent variables. Compared to pixels, this compressed latent space has dramatically reduced dimensionality, allowing us to apply scalable autoregressive generative models to predict video. In contrast previous work that has largely emphasized highly constrained datasets, we focus very diverse, large-scale datasets such as Kinetics-600 We predict video at a higher resolution, 256 x 256, than any other previous method to our knowledge. We further validate our approach against prior work via a crowdsourced human evaluation.
INTRODUCTION
When it comes to real-world image data, deep generative models have made substantial progress. With advances in computational efficiency and improvements in architectures, it is now feasible to generate high resolution, realistic images from vast and highly diverse datasets (Brock et al. 2019
Razavi al. 2019 Karras et al., (2077). Apart from the domain of images, deep generative models have also shown promise in other data domains such as music (Dieleman et al. 20188 Dhariwa et al.
2020), speech synthesis (Oord al. 2016), 3D voxels (Liu et 20188 Nash & Williams 2017) and
text (Radford et al. (0119). One particular fledgling domain is video.
While some work in the area of video generation (Clark al.) 2020) Vondrick et al. 2016 Saito & Saito, 2018) has explored vidco synthesis-generating generating videos with no prior frame information
many approaches actually focus on the task of video prediction conditioned on past frames (Ranzato
et al. 2014; Srivastava et al.) 2015 Patraucean et al.
2015: Mathieu et al., 2016 Lee et al. [0188]
Babaeizadeh al. 2018 Oliu al. 2018 Xiong et al. 2018 Xue et al., 2016 Finn et al. 2016 Luc
2020). It can be argued that video synthesis is a combination of image generation and video
prediction. In other words, one could decouple the problem of video synthesis into unconditional
image generation and conditional video prediction from a generated image. Therefore, specifically focus on video prediction in this paper. Potential computer vision applications of video forecasting include interpolation, anomaly detection, and activity understanding. More generally, video prediction also has more general implications for intelligent systems the ability to anticipate the dynamics of the environment. The problem is thus also relevant for robotics and reinforcement learning (Finn) ct al. 2016 Ebert et al. 2017 Oh et al. 2015 Ha & Schmidhuber 2018 Racanire et al.) 2017).
Approaches toward video prediction have largely skewed toward variations of generative adversarial networks (Mathieu et al.] 2016 Lee et al. 2018 Clark et al.] (0020] Vondrick et al.) 20166] Luc al. 2020). In comparison. we are aware of only a relatively small number of approaches which propose variational autoencoders (Babacizadch et al. 2018 20u et al. [2016) Denton & Fergus 2018).
autoregressive models (Kalchbrenner et al. [017]] Weissenborn al. (222). or flow based approaches (Kumar et al. (2020). There may be number of reasons for this situation. One is the explosion in the dimensionality of the input space, A generative model of video needs to model not only one image but tens of them in coherent fashion. This makes it difficult to scale up such models to large datasets or high resolutions. In addition, previous work (Clark et al. 2020) suggests that video prediction may be fundamentally more difficult than video synthesis; synthesis model can generate simple samples from the dataset while prediction potentially forces the model to forecast conditioned on videos that outliers in the distribution. Furthermore, most prior work has focused on datasets with low scene

---

Under review as a conference paper at ICLR 2021
4th Frame
9th Frame
16th Frame
Figure 1: In this paper we predict video at high resolution (256 x 256) using compressed latent representation. The first frames are given as conditioning. We predict the 12, of which (9th and 16th) we show on the right. All frames shown here have been compressed by VA-.AE. Videos licensed under CC-BY. Atribution for videos in this paper can be found in supplementary material. Best seen in video in the supplementary material.
diversity such as Moving MNIST (Sivastavaa et al. 2015), KTH (Schuldt et al. 2004). or robotic arm datasets (Finn et 2016 Ebert al.) 2017). While there have been attempts synthesize video at high resolution (Clark et al. 2020), we know of no attempt- eetempt-excluding excluding flow based approaches- to predict video beyond resolutions of 64x64.
In this paper we address the large dimensionality of video data through compression. Using Vector Quantized Variational Autoencoders (VQ-VAE) (van den Oord et 2017). we can compress video into a space requiring only 1.3% of the bits expressed in pixels. While this compressed encoding is lossy, we can still reconstruct the original video from latent representation with high degree of fidelity. Furthermore, representation into a hierarchy of encodings, separating high-level, global information from details such fine texture small motions. Instead of training generative model directly on pixel space, we can instead model this much more tractable discrete representation, allowing us to train much powerful models, use large diverse datasets, and generate at high resolution. While most prior work has focused on GANs, this discrete representation can also be modeled by likelihood-baseed models. Likelihood models in concept do suffer from mode-collapse, instability in training, and lack of diversity of samples often witnessed in GANs (Denton & Fergus 2018 Babaeizadeh et al.) 2018 Razavi et al. 2019). In this paper, we propose PixelCNN augmented with causal convolutions in time and spatiotemporal self-attention model this space of latents. In addition, because the latent representation decomposed into a hierarchy, we can exploit this decomposition and train separate specialized models at different levels of the hierarchy.
leverage the modularity of VQ-VAE and decompose our latent
Our paper makes four contributions. First, we demonstrate the novel application VQ-VAD to video data. Second, we propose a set of spatiotemporal PixeICNNs to predict video by utilizing the latent representation learned with VQ-VAE. Third, we explicitly predict video at higher resolution than ever before. Finally, demonstrate the competitive performance of our model with crowdsoureed human evaluation.
BACKGROUND
2.1 VECTOR QUANTIZED AUTOENCODERS
VQ-VAEs (van den Oord al. 2017) are autoencoders which learn a discrete latent encoding for input data œ. First, the output of non-lineat encoder (z),,, implemented by neural network, is passed through discretization bottleneck. 2²(x) mapped via nearest-neighbor into a quantized codebook e E RKxD where D is the dimensionality of each vector e, and K is the number of categories the codebook. The discretized representation thus given by:
2g [x(x) Cb where k argmin, ( E;l12
2

---

Under review as a conference paper at ICLR 2021
Top Layer
Top + Bottom Layer
Original Frame
Here we demonstrate the compression capability of VQ-VAE The top and bottom rows represent two different frames within the same video. The top layer retains most of the global information, while the bottom layer adds fine detail. Videos licensed under CC-BY. Attribution for videos in this paper can be found in the supplementary material
Figure 2:
Equation T is not differentiable; however, (van den Oord et al. 2017) notes that copying the gradient of (x) to (a:) is suitable approximation similar the straight-through estimator (Bengio et al. 2013). A decoder D, also implemented by neural network, then reconstructs the input from (z(x). The total loss function for the VQ-VAE is thus:
L = -=(g(()(())) - x11 + [|gg[2)()) (112g()))) sg/cil12
(2)
Where sg a stop gradient operator, and is a parameter which regulates the rate of code change. As in previous work (van den Oord et al. 2017 Razavi et al. (2019), replace the second term in equation 2 and learn the codebook E R²x XD via an exponential moving average of previous values during training:
Σ =c (1) - 7), e,0) m
N(9 = N((-1) * x m (1
m( m (t-1)
(3)
N(O
Where is a decay parameter and n is the numbers vectors in 2g (x) in a batch that will map to Cx-
2.2 PIXELCNN MODELS
PixeICNN and related models have shown promise in modeling wide variety of data domains
(van den Oord et al.) 2016) Oord et al., 2016 Kalchbrenner et al. [2017] Weissenborn et al. 2020).
they explicitly optimize negative lig-likelihood. They exploit the fact that the joint probability distribution input data x can be factored into product
These autoregressive models are likelihood-basse
of conditional distributions for each dimension of the data:
(x = II Po (2.)2 =
(4)
Where is the full dimensionality of the data. This factorization is implemented by neural network, and the exact set of conditional dependencies is determined by the data domain. Image pixels may depend regions above and to the left of them (van den Oord al. 20166), while temporal dimensions may depend on past dimensions (Oord et al. 2016; Kalchbrenner et al.) 2017 Weissenborn et al. 2020).
3

---

Under review as a conference paper at ICLR 2021
Layer
Top Layer
Top Layer
MM
Encoder
Quantize
Decoder
Conditioning Frames
Concalonate
Bontom
I syer
VQ-VAE Model
Predicting Video
Figure Here we show an overview of our approach. On the left we show the process of compressing video with VQ-VAE. On the right we show the process of generating video with the latents. The top conditional prior model is PixeICNN with causal convolutions to incorporate at each point in space-time. The bottom conditional prior model is simply 2D PixeICNN which generates slice by slice. It is conditioned with a convolutional tower which incorporates window of time slices from the top latents and past bottom latents. The slices outside of this window colored grey in this diagram. Blue arrows represent conditioning, green arrows generation, and pink feed-forward decoding. Videos licensed under CC-BY. Attribution for videos in this paper can be found in the supplementary material.
past information
METHOD
Our approach consists of two main components. First, compress video segments into discrete latent representation using a hierarchical VQ-VAE. We then propose multi-stage autoregressive model based the PixeICNN architecture, exploiting the low dimensionality of the compressed latent space and the hierarchy of the representation.
3.1 COMPRESSING VIDEO WITH VQ-VAE
Similar to (aazavi et al. 2019), we use VQ- VAE to compress video hierarchical fashion. This multi-stage composition of the latent representation allows decomposition of global, high level information from low-level details such as edges or fine motion. For image information (Razavi et al. 2019) this approach confers number of advantages. First, the decomposition allows latent codes to specialize at each level. High level information can be represented an even more compressed manner, and the total reconstruction error is lower. In addition, this hierarchy leads modular generative model. We can then develop a generative model that specializes in modeling the high-level, global information. We can then train separate model, conditioned on global information, that fills in the details and models the low-level information further down the hierarchy. In this paper, we adopt the terminology of (Razavi et 2019) and call the set of high-level latents the top layer and the low-level latents the bottom layer.
naturally
Consistent with the experimental setup previous work in video prediction, deal with 16-frame videos. Most of the videos in our training dataset are 25 frames per second. We use frames at 256 256 resolution. The full video voxel is thus 256 x 256 x 16. Using residual blocks with 3D convolutions, we downsample the video spatiotemporally. At the bottom layer, the video is downsampled to quantized latent space of 64 64 reducing the spatial dimension by and the temporal dimension by 2. Another stack of blocks reduces all dimensions by 2, with top layer of 32 32 4. Each of the voxels in the layer is quantized into 512 codes with a different codebook for both layers.
The decoder then concatenates the bottom layer and the top layer after upsampling using transposed convolutions. From this concatentation input, the decoder deterministically outputs the full 256 256 x 16 video. Overall, we reduce a 256 x 256 x 16 x 3 x space down to a 64 64 x x (512)) 32 x 32 151() space, a greater than 98% reduction in bits required.
4

---

Under review as a conference paper at ICLR 2021
Table 1: Human Evaluation on Kinetics-600
Prefer Video VQ-VAE Prefer (Luc et al. 2020)
Indifferent
65.7%
12.8%
21.5%
During training, we randomly mask out the bottom layer in the concatenated input to the decoder. Masking encourages the model to utilize the top latent layer and prevent codebook collapse.
3.2 PREDICTING VIDEO WITH PIXELCNNS
With VQ-VAF, our 256 x 256 x 16 video is now decomposed into a hierarchy of quantized latents 64 x 64 x 8 and 32 32 4. Previous autoregressive approaches involved full pixel videos at 64 x 64 x 16 (Weissenborn et al., 2020) or 64 x 64 x 20 (Kalchbrenner et al., (017). Our latent representation is thus well within the range of tractability for these models. Furthermore, given the hierarchy, we can factorize our generative model into coarse-to-fine fashion. We denote the model of the top layer the top prior and model of the bottom layer the bottom prior. Because we are focusing on video prediction, we emphasize that both are still conditioned on series of input frames. While previous work used 5 frames and predicted 11 (Weissenborn et al. 2020: Clark et al.) 2020), the power-of-twe design of our architecture leads to condition 4 and predict 12. When conditioning our prior models on these frames, we need not use a large stack directly on the original images but save memory and computation by training a smaller stack of residual layers their latent representation, compressing these conditional frames into a small latent space of 32 x 32 and 64 x 64 x 2.
We first model the top layer with conditional prior model. Our prior model is based on a PixelCNN with multi-head self attention layers (Chen et al. 2018). We adapt this architecture by extending the PixeICNN into time; instead of a convolutional stack over square image, we use a comparable 3D) convolutional stack over the cube representing the prior latents. The convolutions are masked in the same way as the original PixeICNN in space-at at each location in space, the convolutions only have access to information the left and above them. In time, present and future timesteps are masked out, and convolutions only have access to previous timesteps. While spatiotemporal convolutions can be resource intensive, we can implement most of this functionality with 2D convolutions separately in the ı L plane and the y plane. We take 1D convolutions in the horizontal and vertical stacks in the original PixelCNN (van den Oord et al., 2016) and add the extra dimension of time to them, making them 2D. Our only true 3D convolution We use multi-head attention layers analogous to (Razavi et al. 2019); this time the attention is applied to 3D voxel instead of 2D layer in (Razavi et al.) 2019). Attention is applied every five layers. During sampling we can generate voxels left-to-right, top-to-bottom within each temporal step as in the original PixeICNN. Once final step generated, can generate the next step conditioned on the previous generated steps.
the first layer before the addition of gated stacks.
Once we have set of latents from the top layer, and generate the final bottom layer. Because the bottom layer has a higher number of dimensions and relies on local information, we don't 't necessarily need a 3D PixeICNN. Instead, we use 2D PixeICNN with multi-head self attention every five layers analogous to (Razavi et al., 2019). We implement a 3D conditional stack, however, that takes in a window of time steps from the top layer as well as a window of past generated time steps in the bottom layer. The window sizes we used were 4 and 2 respectively. This conditional stack is used as conditioning to the 2D PixelCNN at the current timestep.
condition our bottom conditional prior model
4 RELATED WORK
Video Prediction and Synthesis: In the last few years, the research community has focused spotlight on the topic of video generation- -ether in the form of video synthesis or prediction. Early approaches involved direct, deterministic pixel prediction (Ranzato al. 2014 Srivastava et al.) 2013; Oh et al. 2015 Patraucean et al.] 2015). Given the temporal nature video, such approaches often incorporated L.S.Ms. These papers usually applied their deterministic models datasets such
5

---

Under review as a conference paper at ICLR 2021
Table 2: FVD Scores Kinetics-600. Lower is better.
Method
FVD Score (4)
Video Transformer (64 x 64) WWeissenoor al. 2020)
170 5
DVD-GAN-FP (64 x 64) (Clark et al. 2020
69.15 + 1.16
TRIVD-GAN-FP (64 x 64) (Luc et 2020
25.74 + 0.66
Video VQ- (64 X 64)
64.30 = 2.04
Video VQ-VAE FVD* (64 64)
54.30 + 3.49
Video VQ-VAE (256 x 256)
129.85 + 1.64
Video VQ-VAE FVD* (256 256)
82.45 + 1.16
as moving MNIST characters (Srivastava et al. 2015); because of their deterministic nature, rarely were they successfully applied to more complex datasets. Given this situation, researchers started to adapt popular models for image generation to the problem starting with generative adversarial
models (Mathieu et al. 2016; Vondrick et al. 2016; Lee et al. 2018 Baaaeizzadhh et al. 2018 Clark
et al. 2020 Saito & Saito 2018 Luc et al. 2020; Xiong et al. 2018), variational autoencoders Xue
al.
2016), and autoregressive models (Kalchbrenner al. | et al. 2020). Others
stepped aside from the problem of full pixel prediction and instead predicted pixel motion (Finn et al.
2016.
Walker al. 2016; Jia et al., 2016) or decomposition of pixels and motion (Denton & Fergus
2018:
Gao et al. 2019: ang et al. 2018 Hao et al. [0188] Li et al.) 2018a Tulyakov et al. 2018
Villegas et al. 2017a). Finally, some have proposed hierarchical approach based on structured
information generating video conditioned on text (Li et al. 2018b), semantic segments (Luc et al.
2017
2018 or human pose (Walker al. 2017 Villegas et al. 2017b
Compressing Data with Latents: The key element in our video prediction framework is compression representing videos through lower dimensional latents. We apply the framework of VQ- VAE (van den Oord et al. 2017] Razavi et al. 2019) which has been successfully applied to compress image and sound data. Related VQ- other researchers have explored hierarchies of latents for generation of images (Fauw et al.) 2019) and music (Dieleman et al. 2018).
Autoregressive Models: The foundation of our model is based on PixeICNN (van den Oord (ct al., 2016). Distinct from implicit likelihood models such as GANs and approximate methods such VAEs, the family of PixeICNN architectures have shown promise in modeling variety of data domains including images (van den Oord et al.) 2016), sound (Oord et al.) 2016), and video (Kalchbrenner et al.) [017]] Weissenborn et al. 2020). In line with our paper, recent work with these models has shifted toward decomposing autoregression through hierarchies (Menick & Kalchbrenner
2019: Reed et al. (017) and latent compression (van den Oord et al., 2017 Razavi et al. 2019
Dhariwa et al. 2020).
EXPERIMENTS
In this section, we evaluate our model quantitively and qualitatively on the Kinetics-600 dataset (Carreira et al. 2018). This dataset of videos is very large and highly diverse, consisting of hundreds of thousands of videos selected from YouTube across 600 actions. While most previous work has focused on more constrained datasets, only few (Clark et al., 2020 Luc et al. 22200] Weissenborn et al. (2200) have attempted to scale to larger size and complexity. We train our top and bottom models for around 1000000 iterations with total batch sizes of 512 and 32 respectively. Our VQ- VAE model was trained batch size of 16 for 1000000 iterations.
5.1 QUALITATIVE EVALUATION
While the Kinetics- 600 dataset is publicly available for use, the individual videos in the dataset may not be licensed for display in academic paper. Therefore, in this paper, apply our model trained on Kinetics-60k to videos licensed under Creative Commons from the YFCC100m dataset (Thomee) et al. 2015). In figure 4 we show some selected predictions. We find that approach is able to

---

Under review as a conference paper at ICLR 2021
Final Conditioning Frame
9th Frame
16th Frame
Figure 4: Selected prediction results. The first 4 frames are given as conditioning. We predict the next 12, two of which (9th and 16th) show on the right. All frames shown here have been compressed by VQ-VAE. Videos licensed under CC-BY. Autribution for videos in this paper can be found in the supplementary material. Best seen in video in the supplementary material.
7

---

Under review as a conference paper at ICLR 2021
model camera perspective, parallax, inpainting, deformable human motions, and even aspects of crowd motion across a variety of different visual contexts.
5.2 QUANTITATIVE EVALUATION
Quantitative evaluation of generative models of images, especially across different classes of models, an open problem (This al. 2016). It is even less explored in the realm of video prediction. One proposed metric used on larger datasets such as Kinetics-600 is the Fréchet Video Distance (Unterthiner et al. 2018). As no previous approach has attempted 256 256 resolution,
downscale
our videos to 64 x 64 for proper comparison against prior work. We also compute FVD on the full resolution as a baseline for future work. We use 2-fold cross-validation over 39000 samples to compute FVD. We show our results in Table 7 We find performance exceeds (Clark et al. 2020) but not necessarily (Luc et al. 2020 We also find that comparing the VQ-VAE samples to the reconstructions, not the onginal videos, leads to an even better score (shown by FVD*). This result is similar the results on images for VQ-VAE VAE (Razavi al.) 2019). As GAN- approaches are explicitly trained on classifier (discriminator) based losses, FVD metric based on neural-network classifier-may favor GAN-based approaches versus log-likelinne liglliil based models even if the quality of the samples comparablc. Given the possible flaws in this metric, also conduct human evaluation similar to (Vondrick et al. 2016). We had 15 participants compare up to 30 side-by-side videos generated from our approach and that of (Luc al. 2020). Each video had at least 13 judgements. For each comparison, both models used the exact set of conditioning frames and had a resolution of 64x64. Participants could choose preference for either video, they could choose indifference-meanniog the difference in quality between two videos is too close to perceive. We show our results in Table Out of total of 405 judgements, participants preferred ours 65.7% of the time, (Luc et al., (2020) 12.8%, and 21.5% were judged to be too close in quality. Even though (Luc al., (2020) has a much lower FVD score, we find that our participants had stronger preference for samples generated from our model.
6 CONCLUSION
In this paper have explored the application of VQ-VAE towards the task of video prediction. With this learned compressed space, we can utilize powerful autoregressive models to generate possible future events in video higher resolutions. We show that we are also able to achieve level of performance comparable to contemporary GANs.
REFERENCES
Mohammad Babaeizadeh, Chelsea Finn, Dumitru Erhan, Roy H. Campbell, and Sergey Levine. Stochastic variational video prediction. In International Conference on Learning Representations, 2018.
Yoshua Bengio, Nicholas Lonard, and Aaron C. Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. CoRR, abs/1308.345 2013.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. In ICLR. OpenReview.net, 2019.
Joo Carreira, Eric Noland, Andras note about kinetics-600. CoRR, abss/100 340,,, 2018.
Chloe Hillier, and Andrew Zisserman. A short
Xi Chen, Nikhil Mishra, Mostafa Rohaninejad, and Pieter Abbeel. Pixelsnail: An improved autore- gressive generative model. In Jennifer G. Dy and Andreas Krause (eds.), ICML, volume 80 of Proceedings of Machine Learning Research, pp. 863-871. PMLR, 2018.
Aidan Clark, Jeff Donahue, and Karen Simonyan. Adversarial video generation on complex datasets, 2020.
Emily Denton and Rob Fergus. Stochastic video generation with a learned prior. In Jennifer Dy and Andreas Krause (eds.). International Conference Machine Learning, volume 80 of Proceedings
8

---

Under review as a conference paper at ICLR 2021
of Machine Learning Research, pp. 1114-1111111 1184,, Stockholmsmssan, Stockholm Sweden, 10-15 Jul 2018. PMLR.
Prafulla Dhariwa, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radfor, and Ilya Sutskever. Jukebox: A generative model for music. arXiv preprint arXiv:2005.00344/ 2020.
Sander Dieleman, Aron van den Oord, and Karen Simonyan. The challenge realistic music generation: modelling audio at scale. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicol Cesa-Bianchi, and Roman Garnett (eds.), NeurlPS, pp. 8000-8010, 2018.
Frederik Ebert, Chelsea Finn, Alex X. Lee, and Sergey Levine. Self-supervised visual planning with temporal skip connections. In CoRL, volume 78 Proceedings of Machine Learning Research, pp. 344-356. PMLR, 2017.
Jeffrey De Fauw, Sander Dieleman, and Karen Simonyan. Hierarchical autoregressive image models with auxiliary decoders. CoRR, abs/1903.00 04933, 2019.
Chelsea Finn, Ian Goodfellow, and Sergey Levine. Unsupervised learning for physical interaction through video prediction. D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 64-72. Curran Associates, Inc., 2016.
Hang Gao, Huazhe Xu, Qi-Zhi Cai, Ruth Wang, Fisher Yu, and Trevor Darrell. Disentangling propagation and generation for video prediction. In The IEEE International Conference on Computer Vision (ICCV), October 2019.
David Ha and Jrgen Schmidhuber. World models. 2018. doi: 21.228//enndo.120776 cite arxiv:1803.
Zekun Hao, Xun Huang, and Serge Belongie. Controllable video generation with sparse trajectories. In CVPR, 2018
Yunseok Jang, Gunhee Kim, and Yale Song. Video prediction with appearance and motion conditions. In Jennifer Dy and Andreas Krause (eds.). Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 2234,2234, Stockholmsmssan, Stockholm Sweden, 10-15 Jul 2018. PMLR.
Xu Jia, Bert De Brabandere, Tinne Tuytelaars, and Luc Van Gool. Dynamic filter networks. In Daniel D. Lee, Masashi Sugiyama, Ulrike Luxburg, Isabelle Guyon, and Roman Gamett (eds.), Advances in Neural Information Processing Systems, 667-675, 2016.
Nal Kalchbrenner, Aron van den Oord, Karen Simonyan, Ivo Danihelka, Oriol Vinyals, Alex Graves, and Koray Kavukcuoglu. Video pixel networks. In Doina Precup and Yee Whye Teh (eds.), ICML, volume 70 of Proceedings of Machine Learning Research, pp. 1771-1779. PMLR, 2017.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. 2017.
Manoj Kumar, Mohammad Babaeizadeh, Dumitru Erhan, Chelsea Finn, Sergey Levine, Laurent Dinh, and Durk Kingma. Videoflow: conditional flow-based model for stochastic video generation. In ICL.R OpenReview.net, 2020.
Alex Lee, Richard Zhang, Frederik Ebert, Pieter Abbeel, Chelsea Finn, and Sergey Levine. Stochastic adversarial video prediction. arXiv preprint arXiv:180400155: 1804.0004..2.
2018.
Yijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Xin Lu, and Ming-Hsuan Yang. Flow-grounded spatial-temporal video prediction from still images. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu. and Yair Weiss (eds.), ECCV (9), volume 11213 of Lecture Notes in Computer Science, pp. 609-625. Springer, 2018a. ISBN 778-3-030-01240-3.
Yitong Li, Martin Rengiang Min, Dinghan Shen, David E. Carlson, and Lawrence Carin. Video generation from text. In Sheila A. Mcllraith and Kilian Q. Weinberger (eds.), AAAI, pp. 7065-7072. AAAI Press, 2018b.

---

Under review as a conference paper at ICLR 2021
Shikun Liu, C. Lee Giles, and Alexander Ororbia. Learning hierarchical 3d shapes. In 3DV, pp. $42-551. IEEE Computer Society, 2018. ISBN 978-1-5386-4422
model of
Pauline Luc, Natalia Neverova, Camille Couprie, Jakob Verbeek, and Yann LeCun. Predicting deeper into the future of semantic segmentation. In ICCV, pp. 648 657. IEEE Computer Society, 2017.
ISBN 978- 14-388--103244
Pauline Luc, Camille Couprie, Yann LeCun, and Jakob Verbeck. Predicting future instance seg- mentation by forecasting convolutional features. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (eds.), ECCV (9), volume 11213 of Lecture Notes in Computer Science, pp. 593-608. Springer, 2018. ISBN 978-3-030-01140-3..
Pauline Luc, Aidan Clark, Sander Dieleman, Diego de Las Casas, Yotam Doron, Albin Cassirer, and Karen Simonyan. Transformation-bast adversarial video prediction on large-scale data. CoRR, abs/2003.04035. 2020.
Michal Mathieu, Camille Couprie, and Yann LeCun. Deep multi-scale video prediction beyond mean square error. In Yoshua Bengio and Yann LeCun (eds.), ICLR (Poster), 2016.
Jacob Menick and Nal Kalchbrenner. Generating high fidelity images with subscale pixel networks and multidimensional upscaling. In ICLR. OpenReview.net, 2019.
Charlie Nash and Christopher K. I. Williams. The shape variational autoencoder: A deep generative model of part-segmented 3d objects. Comput. Graph. Forum, 36(5):1-12, 2017.
Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard L.. Lewis, and Satinder P. Singh. Action- conditional video prediction using deep networks in atari games. In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett (eds.), NeurlPS, pp. 2863-2871, 2015.
Marc Oliu, Javier Selva, and Sergio Escalera. Folded recurrent neural networks for future video prediction. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (eds.), ECCV (14). volume 11218 of Lecture Notes in Computer Science, pp. 745-761. Springer, 2018. ISBN 978-3-030-01164-9,
Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. 2016. cite 1xxiv:6609.03499
Viorica Patraucean, Ankur Handa, and Roberto Cipolla. Spatio- video autoencoder with differentiable memory, 2015.
Sbastien Racanire, Theophane Weber, David P. Reichert, Lars Buesing, Arthur Guez, Danilo Jimenez Rezende, Adri Puigdomnech Badia, Oriol Vinyals, Nicolas Heess, Yujia Li, Razvan Pascanu, eengination-augmentte
Peter W. Battaglia, Demis Hassabis, David Silver, and Daan Wierstra. agents for deep reinforcement learning. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. N. Vishwanathan, and Roman Garnett (eds.), Neurips, pp. 5690-5701. 2017.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. 2019.
Marc' Aurelio Ranzato, Arthur Szlam, Joan Bruna, Michal Mathieu, Ronan Collobert, and Sumit Chopra. Video (language) modeling: a baseline for generative models of natural videos. CoRR, abs/ Abbs/441.66644 2014.
Ali Razavi, Aron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with v.- In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' d'Alch Buc, Emily B. Fox, and Roman Garnett (eds.), NeurIPS, pp. 14837-14847, 2019.
Scott E. Reed, Aron van den Oord, Nal Kalchbrenner, Sergio Gomez Colmenarejo, Ziyu Wang, Yutian Chen. Dan Belov, and Nando de Freitas. Parallel multiscale autoregressive density estimation. In Doina Precup and Yee Whye Teh (eds.), ICML, volume 70 of Proceedings of Machine Learning Research, pp. 2912-2921. PMLR, 2017.
10

---

Under review as a conference paper at ICLR 2021
Masaki Saito and Shunta Saito. Tganv2: Efficient training of large models for video generation with multiple subsampling layers. CoRR, abs/1881109245, 2018.
Christian Schuldt, Ivan Laptev, and Barbara Caputo. Recognizing human actions: local svm approach. In Pattern Recognition, 2004, ICPR 2004. Proceedings of the 17th International Conference on, volume 3, pp. 32-36. IEEE, 2004.
Nitish Srivastava, Elman Mansimov, and Ruslan Salakhutdinov. Unsupervised learning of video representations using lstms. In Francis R. Bach and David M. Blei (eds.), ICML, volume 37 of JMLR Workshop and Conference Proceedings, pp. 843-852. JMLR.org. 2015,
Lucas Theis, Aron van den Oord, and Matthias Bethge. A note the evaluation of generative models. In Yoshua Bengio and Yann LeCun (eds.), JCLR, 2016.
Bart Thomee, David A. Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. The new data and new challenges in multimedia research. CoRR, abs/1503.01817, 2015.
Sergey Tulyakov, Ming- Yu Liu, Xiaodong Yang. and Jan Kautz. Mocogan: Decomposing motion and content for video generation. In CVPR, pp. 1526-1535. IEEE Computer Society, 2018.
Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphal Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. CoRR, abs/1812.01717, 2018.
Aron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In Maria-Florinn Balean and Kilian Q. Weinberger (eds.), ICML, volume 48 of JMLR Workshop and Conference Proceedings, pp. 1747-1756. JMLR.org, 2016.
Aron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. N. Vishwanathan, and Roman Garnett (eds.), NeurlPS, pp. 6306 6315, 2017.
Ruben Villegas, Jimei Yang, Seunghoon Hong. Xunyu Lin, and Honglak Lee. Decomposing motion and content for natural video sequence prediction. In ICLR (Poster). OpenReview.net, 2017a.
Ruben Villegas, Jimei Yang, Yuliang Zou, Sungryull Sohn, Xunyu Lin, and Honglak Lee. Learning to generate long-term future via hierarchical prediction. In Doina Precup and Yee Whye Teh (eds.), ICML, volume 70 of Proceedings of Machine Learning Research, pp. 3560-3569. PMLR, 2017b.
Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dynamics. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett 2016.
(eds.), NeurlPS, pp.
Jacob Walker, Carl Doersch, Abhinav Gupta, and Martial Hebert. An uncertain future: Forecasting from static images using variational autoencoders. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.), ECCV (7), volume 9911 of Lecture Notes in Computer Science, pp. 835-851. Springer, 2016. ISBN -6-44-31
Jacob Walker, Kenneth Marino, Abhinav Gupta, and Martial Hebert. The pose knows: Video forecasting by generating pose futures. In ICCV, pp. 3361- IEEE Computer Society, 2017. ISBN 978-1-5386-1032.9)
Dirk Weissenborn, Oscar Tekstrm, and Jakob Uszkoreit. Scaling autoregressive video models. In ICLR. OpenReview.net, 2020.
Wei Xiong, Wenhan Luo, Lin Ma, Wei Liu, and Jiebo Luo. Learning to generate time-lapse videos using multi-stage dynamic generative adversarial networks. In CVPR, pp. 2364 2373. IEEE Computer Society, 2018
Tianfan Xue, Jiajun Wu, Katherine L. Bouman, and Bill Freeman. Visual dynamics: Probabilistic future frame synthesis via cross convolutional networks. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett (eds.), NeurlPS, pp. 91-99, 2016.
11

---

Under review as a conference paper at ICLR 2021
A
APPENDIX
Table VQ-VAE Architecture Details
Parameter
Value
Input Size
x56x256x 16x3
Latent layers
x4x, 64x64x8
(commitment loss coefficient)
0.25
Batch size
16
Hidden units
128
Residual units
32
Layers
2
Codebook size
512
Codebook dimension
64
First Stage Encoder twh-Conv Filter Size
488
First Stage Encoder twh-Conv Filter Stride
244
Second Stage Encoder twh-Conv Filter Size
444
Second Stage Encoder twh-Conv Filter Stride
222
Upsampling twh-Conv Filter Size
4
Upsampling twh-Conv Filter Stride Training steps
2 2 4
1000000
12

---

Under review as a conference paper at ICLR 2021
Table 4: PixeICNN Prior Details
Parameter
Top Prior
Bottom Prior
Input size
32x32x3
64x64
Batch size
512
32
Hidden units
512
512
Residual units
1024
1024
Layers
40
20
Attention layers
8
4
Attention heads
8
Conv Filter size
3
5
Dropout
0.5
0.0
Training steps
1016000
950000
Table 5: Top Prior Conditioning Stack
Parameter
Values
Input size Hidden units
32x32 (upsampled to 64x64x2), 64x64x2
512
Residual units
128
Layers
4
Table 6: The conditioning frames 256x256 4x3 are compressed using our VQ-VAE into 32x and 64x64x2 space. These two layers are then concatenated, downsampled back down to x3xx2x256 by convolutional layer, tiled to 32x32x xx266, and finally fed through another convolutional layer to 32x x122 before being fed through four residual blocks.
Table 7: Bottom Prior Conditioning Stack
Parameter
Values
Input size
32x.32x4, 64x64x4
Hidden units
1024
Residual Blocks
20
Table 8: Let be the timestep be modeled by the bottom prior. The 32x32x4 top layer is upsampled to 64x64 x8 8x 1024 through series of three convolutional layers with kernel sizes (4, 3. ()-4(3,4,4)-633 3, 3) and strides (2, 1, 1),>(1 1, 2, 2) -x11 1, 1) respectively. From this output, the nth slice is chosen input. For the bottom layer, input the past 4, 3, ...n timesteps fed into a series convolutional layers at size (4, 3. 3) and stride (2, I, 1) each. This downsamples to layer of size 64x64. This is concatenated with the output from the top layer and fed into conditioning stack identical to the one described in (Razavi et al. 2019)
13


Reviews:
[{"review_id": "ZybVqcjppc1", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\nThe paper proposes a modification of hierarchical VQ-VAE for video prediction task. To model temporal dependency, the encoder and pixelCNN in original VQ-VAE are extended with 3D convolutions. The proposed method is evaluated on a large-scale video dataset, Kinetics-600 dataset.\n \nPros\n- The paper is generally well-written and easy to follow.\n- The idea of employing hierarchical and quantized representation to model complex variations in video sounds reasonable. The overall algorithm seems to be a reasonable extension of VQ-VAE to videos. \n\nConcerns & Suggestions\n- Although the idea is reasonable, the paper simply extends the hierarchical VQ-VAE for images to videos with modification to model temporal dependency using 3D convolutions. Thus, the technical contribution of the paper is limited. \n- The paper inherits the pixel-wise autoregressive model for modeling prior distribution. However, the pixelCNN is based on highly serialized computations, and generally suffers from a very long inference time. It would be interesting to see the analysis of inference time and some justifications that pixelCNN fits the video prediction at a large-scale.  \n- The arguments on worse FVD performance require further justifications; it is unclear why the likelihood-based method has a disadvantage over the adversarial methods in terms of FVD, as the FVD is a model-neutral metric. \n- The paper presents a human study for qualitative comparisons. However, the number of both participants and presented videos are too small to draw meaningful conclusions. \n\n--- post rebuttal update ----\n\nI appreciate the authors for their response. However, the arguments on FVD are still not quite convincing (i.e., FVD still has reasonable correlation with actual generation quality; if the perceptual metric is inappropriate, then the authors should have tried other metrics. Also, the authors did not address other concerns such as concerns on computation time, scalability, small-scale human evaluation, etc. I maintain my score to rejection of this paper.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "W7cuK7g3jf9", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### SUMMARY\n\nThe authors propose to use a VQVAE-2 setup for video prediction. In particular, they propose a hierarchical discrete latent variable model that compresses videos into a latent space. An autoregressive model is then used to model dynamics in this latent space, which has reduced dimensionality, and can be used together with the VQVAE decoder to predict video.  Empirical results show that this model is comparable to SOTA GAN models and a human evaluation suggests that humans have a preference for the \nVQVAE generations.\n\n### STRENGTHS AND WEAKNESSES\n\n[+] Good empirical results\n[-] Reduced novelty\n[-] Weak empirical section - only one dataset, limited comparison to baselines, missing details\n[-] Some claims are not properly justified\n\n\n### DETAILED COMMENTS\n\nThe main positive aspect of the paper are its results. The authors show that, for the Kinetics-600 dataset, their approach performs similarly to SOTA GAN methods based on the FVD metric. Further, they conduct a human evaluation that indicates that their generations might be preferable to those from GANs.\n\nHowever, this empirical evaluation is quite limited. The authors only show results for the Kinetics-600 dataset. It is true that this is one of the largest scale video datasets and that it is quite challenging. However, the authors could have shown results for other datasets (UCF101, Kitti/Cityscapes/BDD100K, etc.) that offer different trade-offs between complexity and amount of available data (note that for example BDD100K is in the same order of magnitude as Kinetics in terms of data available) and show the performance of their method on those datasets. Furthermore, that would allow a more direct comparison to some previous baselines including non-GAN approaches. The authors could also show different metrics beyond FVD)to assess the performance of their method - video prediction evaluation is an open research question and some metrics are known to have shortcomings, but given that there is no ideal metric then having multiple metrics could help have a better understanding on the model strengths and weaknesses. \n\nAnother issue is the missing details in terms of architectural choices, optimization hyperparameters, computational requirements, training time, etc. These are important to be able to replicate the results and to assess the potential impact of the method, as many video generation/prediction models have large computational requirements that have prevented a wider adoption of some of these methods.\n\nOne of the main shortcomings of the paper is its novelty. The paper is a straightforward application of the VQVAE2 model to 4D tensors (video) and for video prediction. The original VQVAE paper already shows some results on video, contrary to the claim that \"this is the first application of VQVAE to video data\". Further, using autoregressive models for video is a common approach (Video Pixel Networks, Weissenborn et al. 2020), and using an autoregressive model on the latent space of a discrete latent variable model is common in all VQVAE approaches. There is merit in showing that this setup works for video prediction, but nothing about the setup is novel.\n\nFinally, there are some claims in the paper that are factually or arguably incorrect. These have not directly influenced my score, but I believe they should be addressed should the paper be accepted. First the authors claim \"we know of no attempt to predict video at resolutions beyond 64x64, except for flow models\". Clark et al., Castrejon et al. and even the codebase for Denton and Fergus, Lee et al. all have results/models for video prediction at 128x128.  Second, the authors claim that most video prediction methods have skewed towards using GANs. This is arguably not true, I believe most GAN models perform video *generation* (Vondrick, Tulyakov) and most current video *prediction* models are based on VAEs (Denton, Babaeizadeh, Lee, Castrejon, Villegas, etc.). This is due to the fact that usually it is hard to get GANs to cover all modes and usually they perform better for unconstrained generation rather that conditional generation/prediction. In any case, this is debatable and as such it should be clarified or toned down in the paper. Some of the other unjustified claims include \"this model being the first application of VQVAE to video data\" as discussed.\n\n\n### SCORE\n\nOverall I think this is a borderline paper. I think the empirical results are good and might be of interest to the community. However, the paper has important shortcomings - it has reduced novelty, there are important missing details, and the empirical evaluation is weak. Therefore in its current form I vote for rejection.\n\n### POST-REBUTTAL UPDATE\nThe authors' response did not address my concerns. Given that most current evaluations metrics for video generation/prediction have some shortcomings (including human evaluations), it makes more sense to include a wide range of metrics that showcase the strengths and weaknesses of a method rather than to argue against their inclusion in the paper. Additionally, the authors failed to mention very relevant prior work (Latent Video Transformer). \n\nTherefore I do not think this submission in its current form should be accepted and I have reduced my rating to a 4.", "extra_scores": {}, "extra_text": {}}, {"review_id": "LJV2gzcG6fl", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper proposed a new approach for the video prediction task. The proposed model is built upon VQ-VAE, which has shown promising image generation qualities. The authors proposed to use VQ-VAE to perform compression (dimensionality reduction), which in return allows them to perform video prediction for high-resolution videos -- 256*256.\n\nPros:\n1. The proposed framework is valid, and the equations in the paper describing the model are correct. There is no fundamental error. \n2. The paper is well-written and easy to understand. \n3. Qualitative results are promising.\n\nCons:\n1. The biggest issue of this paper is the lack of novelty. In general, this paper is an incremental work based on VQ-VAE. It is a novel application, but the theoretical novelty is minimal. \n2. The second major issue of this paper is that the quality of the evaluation is quite weak. Overall, there are two tables (table 1 and table 2) and one figure. Table 1 provides a human evaluation of the proposed Video VQ-VAE with baseline models. However, only one baseline model is compared. \n3. The caption of table 5 and 7 are given as table 6 and table 8, which makes it very confusing to understand the content.\n4. In section 5.2, the author mentioned that FVD results for full-resolution as a baseline are given in Table 7. I do not think this table is included in the paper.\n5. Since the proposed model is built upon autoregressive models and VAE models, one important evaluation metric would be likelihood (or ELBO). There have been quite a few video prediction papers based on VAE models (such as \"Probabilistic Video Generation Using Holistic Attribute Control\" from ECCV2018), a thorough comparison with these model would provide more insight in the effectiveness of the proposed model.\n6. Compressing data with latents is listed as an important novelty of the model. However, I am not convinced this is the case. Any embedding, whether it's a probabilistic encoder in VAE or deterministic encoder, will have the compressing effect. Actually, almost all of the previous time-series papers rely on the dimensionality reduction step.\n7. In section 3.2, the authors claim that \"the power-of-two design of our architecture leads us to condition on 4 and predict 12,\" while previous models condition on 5 and predict 11. This is listed as one of the benefits of the proposed model. However, most of the time-series models, no matter whether they are sequential latent variable models, or autoregressive models, or even simply an LSTM model, it is not a hard constrain on how many steps they can predict into the future. Theoretically, they can all predict infinite future steps. It is true that the quality of the prediction will decrease over time. But if the authors want to show that the Video VQ-VAE is superior from this perspective, experimental comparisons need to be provided to back up this statement.", "extra_scores": {}, "extra_text": {}}, {"review_id": "GkkdxxEJ6E4", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper introduces a model combining VQ-VAE and autoregressive generative model for video prediction. The multi-level discrete latent variables help to predict higher resolution videos. The experiment on Kinetics-600 shows that the the model is able to predict 256x256 resolution videos. \n-------------------------------------------------\nPros:\n+ High resolution video prediction: I believe there is no existing video prediction model that predicts high resolution videos like 256x256. \n+ The paper covers most of related references. \n-------------------------------------------------\nCons:\n1. *Lack of contribution.*\nHierarchical VAE is not new in the field [1,2]. The proposed approach combines two existing method (Pixel CNNs and VQ-VAE) for higher-resolution video prediction. Although, this paper claims that the model is able to predict higher-resolution video outputs, the quality of the prediction is not clearly evaluated in the paper. Regarding the quality evaluation, see Cons 3. for details. \n\n2. *Missing reference* in terms of high-resolution video prediction: [3]\n\n3. *Limited evaluation and comparisons.*\nThe introduced approach is evaluated on one dataset (Kinetics-600) using FVD and a human study. Other works, such as [Luc20], [Clark20], and [Weissenborn20], have shown prediction results on multiple datasets using multiple metrics. \nFor instance, UCF-101 is used by [Luc20] and [Clark20], and the BAIR robot pushing dataset is used by [Weissenborn20]. The KITTI driving dataset and Human 3.6M datasets are used by [1] for higher resolution video prediction. \nI understand that Kinetics-600 contains many and diverse videos, and some of previously used datasets are in small resolution such as the BAIR robot pushing dataset. However, \n    1. to understand the capability of the proposed approach, evaluating on more datasets is necessary such as the KITTI driving dataset or UCF-101. \n    2. In addition, there is no best way so far to quantitatively evaluate the video quality with one measure. I suggest to provide evaluation with more metrics. SSIM and LPIPS are other common metrics together with FVD. \n    3. It is not easy to judge the output quality by just looking at the prediction in Figure 4. Adding the ground truth next to the prediction output would be very helpful. \n    4. Also, providing more output videos in appendix/supplementary  materials will be useful. \n-------------------------------------------------\nMinor comments:\n- Table 2 caption needs a description of FVD*. \n-------------------------------------------------\n[1] Klushyn, et al., Learning Hierarchical Priors in VAEs, NeurIPS 2019.\n[2] Zhao, et al., Learning Hierarchical Features from Generative Models, ICML 2017.\n[3] Villegas, et al., High Fidelity Video Prediction with Large Stochastic Recurrent Neural Networks, NeurIPS 2019.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: While this paper was perceived as being fairly well written, the level of novelty and the evaluation were seen as weak by many reviewers. The aggregate opinions across reviewers is just too low to warrant an acceptance rating by the AC. The AC recommends rejection.
Decision: Reject