Autoregressive Latent Video Prediction with High-Fidelity Image Generator
Authors: Younggyo Seo
Kimin Lee
Fangchen Liu
Stephen James
Pieter Abbeel
Venue: ICLR 2022 Submitted (2022)
Keywords: video prediction
autoregressive models

Abstract: Video prediction is an important yet challenging problem; burdened with the tasks of generating future frames and learning environment dynamics. Recently, autoregressive latent video models have proved to be a powerful video prediction tool, by separating the video prediction into two sub-problems: pre-training an image generator model, followed by learning an autoregressive prediction model in the latent space of the image generator. However, successfully generating high-fidelity and high-resolution videos has yet to be seen. In this work, we investigate how to train an autoregressive latent video prediction model capable of predicting high-fidelity future frames with minimal modification to existing models, and produce high-resolution (256x256) videos. Specifically, we scale up prior models by employing a high-fidelity image generator (VQ-GAN) with a causal transformer model, and introduce additional techniques of top-$k$ sampling and data augmentation to further improve video prediction quality. Despite the simplicity, the proposed method achieves competitive performance to state-of-the-art approaches on standard video prediction benchmarks with fewer parameters, and enables high-resolution video prediction on complex and large-scale datasets. Videos are available at the anonymized website https://sites.google.com/view/harp-anonymous

Under review as a conference paper at ICLR 2022
AUTOREGRESSIVE LATENT VIDEO PREDICTION WITH HIGH-FIDELITY IMAGE GENERATOR
Anonymous authors Paper under double-blind review
ABSTRACT
Video prediction is important yet challenging problem; burdened with the tasks of generating future frames and learning environment dynamics. Recently, autore- gressive latent video models have proved to be powerful video prediction tool, by separating the video prediction into two sub-problems: pre-training an im- age generator model, followed by learning the latent space of the image generator. However, successfully generating high- fidelity and high-resolution videos has yet to be seen. In this work, investigate how to train an autoregressive latent video prediction model capable of predicting high-fidelity future frames with minimal modification existing models, and pro- duce high-resolution (256x256) videos. Specifically, we scale up prior models by employing high-fidelity image generator (VQ-GAN) with a causal transformer model, and introduce additional techniques of top-k sampling and data augmen- tation to further improve video prediction quality. Despite the simplicity, the pro- posed method achieves competitive performance state-oftte of-the-art art approaches standard video prediction benchmarks with fewer parameters, and enables high- resolution video prediction on complex and large- datasets. Videos are avail- able at the anonymized website https://sitesggogg wtwpge/eww/weee///aaaaaooyeeeeee
autoregressive prediction model in
INTRODUCTION
Time
Ground Truth
Predicted
Figure 1: Seleted 256 256 video sample generated by HARP RoboNet (Dasari et al., 2019).
Video prediction can enable agents to learn useful representations for predicting the future conse- quences of the decisions they make, which is crucial for solving the tasks that require long-term planning, including robotic manipulation (Finn & Levine, 2017; Kalashnikov et al., 2018) and au- tonomous driving (Levinson et al., 201 Xu et al., 2017). Despite the recent advances in improving the quality of vidco prediction (Finn et al., 2016; Lotter et al., 2017; Liang al., 2017; Babaeizadeh et al., 2018; Denton & Fergus, 2018; Lee et al., 2018; Bycon et al., 2018; Kumar et al., 2020; Weis- senborn al., 2020; Babaeizadeh al., 2021), learning an accurate video prediction model remains

---

Under review as a conference paper at ICLR 2022
notoriously difficult problem and requires a lot of computing resources, especially when the inputs are video sequences with high-resolution (Castrejon et al., 2019; Villegas et al., 2019; Clark et al,, 2019; Luc et al.. 2020; Walker et al., 2021). This is because the video prediction model should excel at both tasks of generating high-fidelity images and learning the dynamics of environments, though each task itself is already very challenging problem.
Recently, autoregressive latent video prediction methods (Rakhimov et al., 2021; Yan et al., 2021) have been proposed to improve the efficiency of video prediction, by separating video prediction into two sub-problems: first pre-training an image generator (e.g., VQ-VAE; Oord et al. 2017), and then learning the autoregressive prediction model (Weissenborn et al., 2020; Chen et al., 2020) in the latent space of the pre-trained image generator. However, the prior works are limited in that they only consider relatively low-resolution videos (up to 128 x 128 pixels) for demonstrating the efficiency of the approach; it is questionable that such experiments can fully demonstrate the benefit of operating in the latent space of image generator instead of -igeeiinnniional pixel-channel space.
In this paper, we present High-fideligy AutoRerresivve latent video Prediction (HARP), which scales up the previous autoregressive latent video prediction methods for high-fidelity video prediction. The main principle for the design of HARP is simplicity: we improve the video prediction quality with minimal modification to existing methods. First, for image generation, we employ a high- fidelity image generator, i.e., vector-quantizer generative adversarial network (VQ-GAN; Esser et al. 2021). This improves video prediction by enabling high-fidelity image generation (up to 256 x 256 pixels) on various video datasets. Then a causal transformer model (Chen et al., 2020), which operates on top of discrete latent codes, is trained to predict the discrete codes from VQ-GAN, and autoregressive predictions made by the transformer model are decoded into future frames at inference time. Moreover, motivated by the sampling techniques widely-used in language generation for making coherent and diverse predictions, we propose to utilize top-k sampling (Fan et al., 2018) that draws the next discrete code from the k-most probable codes. Since the number of discrete codes the autoregressive model has to predict very large, egg., 6,400 codes on KITTI driving dataset (Geiger et al., 2013), we find that discarding rare discrete codes helps the model predict diverse but high-quality videos, without any change to the training procedure.
We highlight the main contributions of this paper below:
We show that our autogressive latent video prediction model, HARP, can predict high- high-resolution (256 x 256 pixels) future frames on simulated robotics dataset (i.e.. Meta-Wort World; Yu et al. 2020) and large-scale real-world robotics dataset (i.e., RoboNet; Dasari et al. 2019).
We show that HARP can leverage the image generator pre-trained ImageNet (Deng et al., 2009) for training a high-resolution video prediction model on complex, large-scale Kinetics- 600 dataset (Carreira al., 2018), significantly reducing the training cost.
HARP achieves competitive or superior performance to prior state-of-the-ar video prediction models with large end-to-end networks on widely-used BAIR Robot Pushing (Ebert et al., 2017) and KITTI driving (Geiger et al., 2013) video prediction benchmarks.
We also show that the pre-trained representations of HARP can be useful for learning multi-task imitation learning agent Meta- World MT50 benchmark (Yu et al., 2020).
2 RELATED WORK
Video prediction. Video prediction aims to predict the future frames conditioned on images (Michalski et al., 2014: Ranzato et al., 2014: Srivastava et al., 2015; Vondrick et al., 2016; Lotter et al.. 2017), texts (Wu et al., 2021b), and actions (Oh et al., 2015; Finn et al., 2016), which would be useful for several applications, e.g., model-based RL (Hafner et al., 2019; Kaiser et al., 2020; Rybkin et al., 2021), and simulator development (Kim et al., 2020; 2021). Various video prediction models have been proposed with different approaches, including generative adversarial networks (GANs; Goodfellow et al. 2014) known to generate high-fidelity images by introducing adversarial discriminators that also considers temporal or motion information (Aigner & Körner, 2018; Jang et al., 2018; Kwon & Park, 2019; Clark et al., 2019; Luc et al., 2020), latent video prediction mod- els that operates on the latent space (Babaeizadeh et al., 2018; Denton & Fergus, 2018; Lee et al., 2018; Villegas al., 2019; Wu et al., 2021a; Babaeizadeh al., 2021), and autoregressive video
2

---

Under review as a conference paper at ICLR 2022
X
X
Raster scan ordering st'
Inference
Causal Transformer
Causal Transformer
Top-k sampling
xx)
x(x+11)
Target
Figure 2: Illustration of our approach. We first train VQ-GAN model that encodes frames into discrete latent codes. Then the discrete codes are flattened following the raster scan order, and a causal transformer model is trained to predict the next discrete codes in autoregressive manner. At inference time, we use top-k sampling that only draws the next discrete code from top-k probable codes, in order predict diverse but high-quality videos.
prediction models that operates on pixel space by predicting the next pixels in an autoregressive way (Kalchbrenner et al., 2017; Reed et al., 2017; Weissenborn et al., 2020).
Autoregressive latent video prediction. Most closely related our work are autoregressive la- tent video prediction models that separate the video prediction problem into image generation and dynamics learning. Walker al. (2021) proposed learn a hierarchical VÂ-VÄE (Razavi al., 2019) that extracts multi-scale hierarchical latents then train SNAIL blocks (Chen et al., 2018) that predict hierarchical latent codes, enabling high-fidelity video prediction. However, this involves complicated training pipeline and vidco-specific architecture, which limits its applicability. As simple alternatives, Rakhimov et al. (2021); Yan et al. (2021) proposed to first learn a VQ-VAE (Oord et al., 2017) and train causal transformer with 3D self-attention (Weissenborn et al., 2020) and factorized 2D self-attention (Child et al., 2019), respectively. These approaches, however, are limited in that they only consider low-resolution videos. We instead present simple high-resolution video prediction method that incorporates the strengths of both prior approaches.
PRELIMINARIES
We consider the standard video prediction framework where the goal is predict the future frames conditioned on the initial frames of a video. Specifically, conditioned on the first e frames of a video X<e (Xy, X1 ..., Xc-1), aim leam video prediction model that predicts the future frames Xe:T = (xe. [x....., xT-1). where X2 E xx No. the frame at timestep t. Optionally, one can also consider conditioning the prediction model actions (ag, -.., ar- that the agents in the video would take, iee., action-conditioned video prediction.
Autoregressive video prediction model. Motivated by the recent success of pixel-level autore- gressive models on image generation (Menick & Kalchhrenner, 2018), Weissenborn et al. (2020) introduced an autoregressive video prediction model that approximates the distribution x x Not, the joint distribution over in pixel-channel space. Specifically, given video x E IR ?
video
pixels conditioned on the first frames is modelled as the product of channel intensities Nch and all No = T - H W pixels except N. c H W pixels of conditioning frames:
Neh-1
p(XeT xxe)
II
II
p(x* Xm(<))) (
where ı raster-scan ordering over pixels from the video (we refer to Weissenborn (2020) for more details on the case where π is the combination of a subscale and raster-scan ordering since we only utilize raster-scan ordering for our approach), Xx(<() is all pixels before Xx(i)- x the
k-th channel intensity of the pixel (x))), and <k is all channel intensities before x5 x (1()

---

Under review as a conference paper at ICLR 2022
Vector quantized variational autoencoder. VQ-VAE (Oord et al., 2017) consists of an encoder that compresses images into discrete representations, and a decoder that reconstructs images from these discrete representations. Both encoder and decoder share codebook of prototype vectors which are also learned throughout training. Formally, given an image x E TR # XNec, the encoder E' encodes x into feature map ze(x) E IR "i'x that consists of a series of latent vectors (x) (2) E IRAV= where n' is a raster-scan ordering of the feature map ==(x) of size |+'| HI' W". Then ze (x) is quantized to discrete representations (r) E >= based on the distance of latent vectors Σx (i) (xr) to the prototype vectors in the codebook C = {ex [ as follows:
(z(2) = (Cn(2,,1> (o(..)) Co(2,111)) where q(z, i) = arg min (o)))) - ex l2.
(2)
[e(
where [K] is the set {1,... K Then the decoder G learns to reconstruct from discrete repre- sentations (z) The VQ- VAE is trained by minimizing the following objective:
LuqUAE (2r) = | ((2,(,))) lag [=x(x)] 2g (x)) + 3- ||ag (3g(2)) - ze (x)))
(3)
Erecon
Lendabaaic
Leoess
where the operator sg refers to a stop- hop-gradient operator, Lrecon is reconstruction loss for learning representations useful for reconstructing images, Leodaboox is a codebook loss to bring codebook rep- resentations closer to corresponding encoder outputs h, and Leomit is commitment loss weighted by to prevent encoder outputs from fluctuating frequently between different representations.
Vector quantized generative adversarial network. VQ-GAN (Esser et al., 2021) is variant of VQ-VAD that (a) replaces the Lrecon in (3) by a perceptual loss LIPIPS (Zhang et al., 2018), and (b) introduces an adversarial training scheme where a patch-level discriminator D (Isola et al., 2017) is trained to discriminate real and generated images by maximizing following loss:
LaaN (z) = [log (zz) + log(1 - (+)(=
(4)
Then, the objective for training the VQ-GAN model defined as:
min max Ez=p(a)) [(CLiIeS + Leodabook + Leomntt) + A Laau).
(5)
E,G.C D
where x Ve, [Cryps) Va, [au +-8 is an adaptive weight, VGL is the gradient of the inputs to the last layer of the decoder Gr, and = 10-6 6 is a scalar introduced for numerical stability.
METHOD
We present HARP, video prediction model capable of predicting high-fidelity future frames. Our method is designed to fully exploit the benefit of autoregressive latent video prediction model that separates the video prediction into image generation and dynamics learning. Specifically, we con- sider the combination of (a) the recently introduced high-fidelity image generator (Esser et al., 2021) and (b) autoregressive latent video prediction model (Oord et al., 2017; Rakhimov et al., 2021; Walker et al., 2021; Yan et al., 2021) that operates on top of the pre-trained image generator. The full architecture of HARP is illustrated in Figure 2.
4.1 HIGH-FIDELITY IMAGE GENERATOR
We consider the VQ-GAN model (Esser et al., 2021) that has proven to be effective for high- resolution image generation as image generator (see Section 3 for the formulation of VQ-GAN). Similar to the motivation of Tian et al. (2021) that utilizes a pre-trained image generator in the context of video synthesis, we first pre-train the image generator then freeze the model throughout training to improve the efficiency of learning video prediction models. The notable difference to prior work that utilize 3D convolutions to temporally downsample the video for efficiency (Yan et al., 2021) is that our image generator operates on single images; hence our image generator solely focus on improving the quality of generated images. Importantly, this enables us to utilize the VQ-GAN model pre-trained generator on the target datasets, which can significantly reduce the training cost of high-resolution video prediction model, Also, the representations from video prediction model can be easily transferred to downstream tasks that require fine- grained control at each timestep, e.g., imitation learning (see Section 5.3 for supporting experimental results multi-task imitation learning).
a wide range of natural images, egg., ImageNet, without training the image
4

---

Under review as a conference paper at ICLR 2022
Figure 3: 256 x 256 future frames predicted by HARP on action-free Meta- World (Yu al., 2020). Using videos from 50 different asks, we train single model to predict the 11 frames conditioned on the first frame without leveraging task-specific information (e.g., task index). We observe that the model can accurately predict the high-resolution future frames of diverse tasks. More videos predicted by HARP are available in Appendix D.
4.2 AUTOREGRESSIVG LATENT VIDEO PREDICTION MODEL.
To leverage the VQ-GAN model for video prediction, we utilize the autoregressive latent video pre- diction architecture that operates on top the discrete codes extracted from video x. Specifically, we extract the discrete codes z(x)
(z(x1)...., (xxx)) using the pre-trained VQ-GAN, where 2xx,) (1(x,,1) 2(x.2), ..., ((x+.(=(I)) ) is the discrete code extracted from the frame x, as (2). Then, instead modelling the distribution of video p(x) in the pixel-channel space as in (I), learn the distribution of the video in the discrete latent representation space:
Na-1
p(z(Xe:T (xce)
II 2=0
where N₂ -T-C) C) H' W" is the total number of codes from X.:. While the specific imple- mentation for modelling p(z(x)) differs in prior works (Oord et al., 2017; Rakhimov et al., 2021; Walker et al., 2021; Yan et al., 2021), due to its simplicity, tecture (Yan et al., 2021) where the output logits from input codes are trained to predict the next discrete codes. We remark that our approach is also compatible with other architectures.
utilize the causal transformer archi-
4.3 ADDITIONAL TECHNIQUES
Top-k sampling. To improve the video prediction quality of latent autoregressive models whose outputs are sampled from the probability distribution over a large number discrete codes, we uti- lize the top-k sampling (Fan et al., 2018) that randomly samples the output from the top-k probable discrete codes. By preventing the model from sampling rare discrete codes from the long-tail of probability distribution and predicting future frames conditioned such discrete codes, find that top-k sampling improves video prediction quality, especially given that the number of discrete encodings required for future prediction very large, e.g.. 2,560 on RoboNet (Dasari et al., 2019) up to 6,400 on KITTI dataset (Geiger et al., 2013) in our experimental setup.
Data augmentation. We also investigate how data augmentation can useful improving the performance of autoregressive latent video prediction models. Since the image generator model is not trained with augmentation, we utilize a weak augmentation avoid the instability coming from aggressive transformation of input frames, iee. translation augmentation that moves the input images by m pixels along the or Y direction.
5

---

Under review as a conference paper at ICLR 2022
Context Frames
t=
==
10
Ground Truth
Predicted
Ground Truth
Predicted
Figure 4: 256 256 future frames predicted by HARP on action- action-conditioned RoboNet (Dasari et al., 2019). The model predicts the next ten frames conditioned on the first two frames and the future ten actions. We observe that our model can predict the movement of robot arm surrounded by various objects of different colors and shapes. More videos predicted by HARP are available in Appendix E.
5 EXPERIMENTS
We design our experiments to investigate the following:
Can HARP predict high-resolution future frames (up 256 256 pixels) various video datasets with different characteristics?
How does HARP compare ofate-of-thh-art methods with large end-to-end networks standard video prediction benchmarks in terms of quantitative evaluation?
How does the proposed techniques affect the performance of HARP?
Can HARP be transferred to solve multi-task imitation learning tasks?
5.1 HIGH-RESOLLTION VIDEO PREDICTION
Implementation. We utilize up to 8 Nvidia 2080Ti GPU and 20 CPU cores for training each model. For training VQ-GAN (Esser et al., 2021), we first train the model without discriminator loss Laax, and then continue the training with the loss following the suggestion of the authors. For all experiments, VQ-GAN downsamples each frame into 16 16 latent codes, i.e. by factor of 4 for frames of size 64 x 64 frames, and 16 for frames of size 256 x 256. For training transformer model, the VQ-GAN model is frozen that its parameters are not updated. We use Sparse Transformers (Child et al., 2019) as our transformer architecture to accelerate the training. As for hyperparameter, k 10 for sampling inference time, but data augmentation for high-resolution video prediction experiments. We report more detailed implementation details in Appendix A.
Meta- experiments. To demonstrate that our method can predict high-resolution videos (256 256 pixels), we first action-free Meta-World dataset (Yu et al., 2020) consisting of 2,500 demonstrations from 50 different robotics manipulation tasks, which we collected using the deter- ministic scripted policies'. Specifically, we train a single model to predict future videos of all 50 tasks, without leveraging task-specific information, such as task index. For evaluation, we use 10% of the demonstrations as held-out test dataset. As shown in Figure 3, our model can accurately predict the high-resolution future frames of diverse tasks, capturing all the small details. This shows that our model can effectively learn all the information of multiple tasks required for predicting fu- ture frames. We also remark that such representations can be useful to improve the performance of imitation learner (see Section 5.3 for supporting experimental results).
RoboNet experiments. Now we investigate how our model works on large-scale, real-world RoboNet dataset (Dasari et al., 2019) consisting of more than 15 million frames. While prior works successfully trained a video prediction model with 64 x 64 videos (Wu et al., 202 la; Babaeizadeh
'The dataset available https//sshoturrlatacmmk

---

Under review as a conference paper at ICLR 2022
Context Frames
1
t
f- 8
t 10
Ground Truth
Predicted
Ground Truth
Prodicted
Figure 5: 256 x 256 future frames predicted by HARP trained on Kinteics-600 dataset (Carreira et al., 2018) using ImageNet pre-traines VQ-GAN model. Videos are licensed under CC-BY and at- tribution can be found Appendix G. More videos predicted by HARP are available Appendix F.
et al., 2021), we show that our model can predict high-resolution 256 x 256 videos even with fewer number of parameters than the one used in the prior works for predicting 64 x 64 videos. Specifically, we first train a VQ-GAN model with 91.5M parameters, and then train a 12-layer causal transformer model with 74.2M parameters that predicts future 10 frames conditioned on first two frames and future ten actions. Total number of parameters is 165.7M, which is smaller than 303.3M of FitVid (Babaeizadeh et al., 2021) that predicts 64 64 videos. Figure and Figure 4 show the predicted frames on the held-out test video, where the model predicts the high-resolution future frames where robot arm is moving around various objects of different colors and shapes.
Kinetics-600 experiments using ImageNet pre-trained VQ-GAN. Finally, we consider a very complex, large-scale Kinetics-600 dataset (Carreira et al., 2018) consisting of more than 400,000 videos, which requires a large amount of computing resources for training even on 64 x 64 reso- lution (Clark et al., 2019; Luc et al., 2020). To avoid the prohibitively expensive training cost of high-resolution video prediction models on this dataset and fully exploit the benefit of employing high-fidelity image generator, we utilize the VQ-GAN model pre- pre-trained on ImageNet dataset (Deng al., 2009). As only train the transformer model for video prediction, this enables us to train a high-resolution video prediction model in a very efficient manner. Specifically, we train the trans- former model for 60,000 steps training dataset, which takes less than day using our machine. As shown in Figure 5, our model can predict future frames on the test videos", which demonstrates that leveraging the large image generator pre-trained on a wide range of natural images can be promising recipe for efficient video prediction on high-resolution, large-scale video datasets.
5.2 COMPARATIVE EVALUATION ON STANDARD BENCHMARKS
Datasets. For quantitative evaluation, we first consider the BAIR robot pushing dataset (Ebert et al., 2017) consisting of roughly 40k training and 256 test videos. We consider action-free setup, hence video prediction models should be stochastic for predicting the diverse possible movement of a robot arm and objects. Following the setup in prior works (Clark et al., 2019; Weissenborn et al., 2020; Luc et al., 2020; Yan et al., 2021), we predict 15 future frames conditioned on one frame. We also evaluate our method on KITTI driving dataset (Geiger et al., 2013), where the training and test datasets are split by following the setup in Lotter et al. (2017). As the KITTI dataset relatively small-scale compared good testbed for investigating the effect of data augmentation. For hyperparameters, we use ls = 10 for both datasets and data augmentation with m is only applied KITTI as there was no sign of overfitting on BAIR dataset. For fair comparison, we follow the setup of Villegas et al. (2019), where (i) a model is trained to predict future ten frames conditioned on five frames and evaluated to predict future 25 frames conditioned on five frames, and (ii) test dataset consists of 148 video clips constructed by extracting 30- (0-frame clips and skipping every 5 frames.
other datasets, i.e., 57 training videos, it provides
Phttp::/github.coo cwt.s/oo/////////cc///o//ooo viss/aming-tranffrrmma transformers
We collected videos covered by CC-BY license, which available put the frames on paper.

---

Under review as a conference paper at ICLR 2022
Table 1:
Quantitative evaluation on (a) BAIR Robot Pushing (Ebert et al., 2017) and (b) KITTI driving dataset (Geiger et al., 2013), We observe that HARP can achieve competitive performance to state-of-the-art methods with large end-to-end networks on these benchmarks.
(a) BAIR Robot Pushing
(b) KITTI
Method'
Params
FVD (L)
Method
Params
FVD (4)
LPIPS (L)
LVT
50M
125.8
SVG
298M
1217.3
0.327
SAVP
53M
116.4
GHVAE
599M
552.9
0.286
DVD-GAN-FP
109.8
FitVid
302M
884.5
0.217
VideoGPT
82M
103.3
HARP (ours)
89M
482.9
0.191
THVD-GAN-FP
103.3
Video Transformer
373M
94.0
Not available
FitVid
302M
93.6
HARP (ours)
89M
99.3
Table 2: FVD scores of HARP with varying (a) the number of codes use for top-k sampling, (b)
number of layers, and (c) magnitude m of data augmentation.
(a) Effects of k
(b) Effects of layers
(c) Effects of m
Dataset
k
FVD (1)
No top-k 100
104.4
Dataset
Layers
FVD (L)
Dataset
FVD (L)
BAIR
103.6
BAIR
111.8
980.1
10
99.3
99.3
497.0
KITTI
No top-k 100
578.1
520.1
482.9
KITTI
557.7
KITTI
482.9
523.4
10
482.9
Metries. We use two evaluation metrics: Learned Perceptual Image Patch Similarity (LPIPS; Zhang et al. 2018), frame-wise metric designed to better represent the human perceptual simi- larity of two frames compared to traditional metrics (Wang et al., 2004; Huynh-Thu & Ghanbari, 2008), and Frèchet Video Distance (FVD; Unterthiner al. 2018), dynamics-based evaluation metric known to be better correlated with the human evaluation compared to frame-wise evaluation metrics. FVD is computed by comparing the summary statisties of I3D network trained Kinetics- 400 dataset (Carreira & Zisserman, 2017), and LPIPS is computed using the features from AlexNet (Krizhevsky et al., 2012). For comparison with the scores reported in prior works, we exactly follow the evaluation setup in Villegas et al. (2019) and Babaeizadeh et al. (2021) that samples 100 future videos for each ground-truth test video, then reports the best score over 100 videos for LPIPS, and the score using all videos for FVD, with the batch size of 256 for BAIR and 148 for KITTI.
Results. Table shows the performances of our method and basclines on test sets of BAIR Robot Pushing and KITTI driving dataset. We observe that model achieves competitive superior performance to state-of-the-a methods with large end-to-end networks, e.g., HARP outperforms FitVid with 302M parameters on KITTI driving dataset. Our model successfully extrapolates to seen number of future frames (i.e., 25) instead of 10 future frames used in training on KITTI dataset. This implies that transformer-based video prediction models can also predict arbitrary number of frames at inference time. In the case of BAIR dataset, HARP achieves the similar performance of FitVid with 302M parameters, even though our method only requires 89M parameters. We provide videos predicted by HARP on BAIR and KITTI datasets in Appendix C.
Analysis. We investigate how the top-k sampling, number layers, and magnitude m of data augmentation affect the performance. Table 2a shows that smaller À leads to better performance, implying that the proposed top-k sampling is effective for improving the performance by discarding rare discrete codes that might degrade the prediction quality inference time. As shown in Ta- ble 2b, we observe that more layers leads to better performance on BAIR dataset, which implics our model can be further improved by scaling up the networks. Finally, we find that (i) data augmen- tation on KITTI dataset is important for achieving strong performance, similar to the observation of Babaeizadeh al. (2021), and (ii) too aggressive augmentation leads worse performance. We provide the learning curves with and without augmentation in Appendix B.
Baselines are SVG (Villegas et al., 2019). GHVAE (Wu et al., 2021a), FitVid (Babaeizadeh et al., 2021), LVT (Rakhimov et al., 2021), SAVP (Lee et al., 2018), DVD-GAN-FP (Clark et al., 2019), VideoGPT (Yan al., 2021), THYD-GAN-FP (Lue et al., 2020), and Video Transformer (Weissenhorn al., 2020).
8

---

Under review as a conference paper at ICLR 2022
10
8
10
Predicted Ground Truth
Predicted Groumd Truth
(a) RoboNet
Kinetics-600
Figure 6: Failure cases our experiments. (a) Interaction with the objects is ignored. (b) The model repeats the first frame while a person is moving right in the ground-truth frames.
5.3 FINE-TUNING HARP FOR MULTI-TASK IMITATION LEARNING
Setup. In order to demonstrate that the pre-trained representations from HARP can be useful for solving downstream tasks, we evaluate the imitation learning performance of fine-tuned HARP on MT-50 benchmark from Meta-World. Specifically, we take the pre-trained HARP model (see Fig- ure for the video predictions from this model), and fine-tune the model to predict expert actions by introducing a policy network on top of the transformer model. For comparative evaluation, we consider three baselines: (a) VQ- Transformer, which shares the same architecture with HARP but trained from scratch, (b) CNN-LSTM, which extracts features using convolutional neural networks (CNN) and LSTM networks, and (c) of LSTM networks. For training and evaluation, we use the same training and test dataset used for video prediction experiments. We report the average success rate over 10 trials for each task. More details are available in Appendix A.
that utilizes transformer networks instead
Results. Table shows the performance of imitation learn- ing policies on MT50 test environments. We first observe that VQ-Transformer, which has same architecture with HARP but trained from scratch, completely fails to solve the tasks. This shows the difficulty of training useful represen- tations with fixed discrete codes as inputs. However, fine- tuned HARP model successfully outperforms other baselines because pre-trained representations contain useful information for long-term reasoning. This demonstrates that video predic- tion with HARP can be an effective self-supervised learning scheme for solving various control tasks.
Table 3: Average success rate on
MT-50 test environments. The sults report the mean and standard deviation over five runs.
Method
Success rate
VQ- Transformer
0.0.L 0.0
CNN-LSTM
0.41 - 0.057
NNN-Transformer
0.50 0.055
HARP (ours)
0.58 0.038
DISCUSSION
In this work, present HARP, high-fidelity autoregressive latent video prediction model. By employing a high-fidelity image generator and utilizing top-k sampling at inference time, HARP can predict high-resolution future frames, and achieve competitive performance to state-of-the-art video prediction methods with large end-to-end networks. We also show that HARP can leverage the image generator pre-trained on wide range of natural images for video prediction, similar to the approach in the context of video synthesis (Tian et al., 2021). We hope this work inspires more investigation into leveraging a pre-trained image generator for video prediction, which can significantly reduce the cost for training a high-resolution video prediction model by building on the recent success of high-fideity image generation (Oord et al., 2017; Razavi et al., 2019; Esser et al., 2021: Child, 2020; Ho al., 2020; Karras et al., 2020; Dhariwal & Nichol, 2021).
Finally, we report the failure cases of video prediction with HARP and discuss the possible ex- tensions resolve the issue. A common failure case for video prediction RoboNet dataset is ignoring the interaction between a robot arm and objects. For example, in Figure 6a, our model ignores the objects and only predicts the movement of a robot arm. On the other hand, common failure case for Kinetics-600 is a degenerate video prediction, where a model just repeats the con- ditioning frame without predicting the future, as shown in Figure 6b. These failure cases might be resolved by training more larger networks similar to the observation in the field of natural language processing, e.g., GPT-3 (Brown et al., 2020), or might necessitate new architecture for addressing the complexity of training autoregressive latent prediction models on video datasets.

---

Under review as a conference paper at ICLR 2022
ETHICS STATEMENT
While video prediction can be useful for various applications including robotic manipulation and autonomous driving, might be misused by malicious users for unethical purposes, e.g. fake videos accusing politicians or sexual videos of any individuals. As our work introduces a method for generating more high-resolution future frames, our method may improve the chance of such videos being recognized as real videos. For this reason, in addition to developing a video prediction method that generates more realistic frames, important to aware of potential problems and develop a method to detect generated videos (Gentine et al., 2018).
REPRODUCIBILITY STATEMENT
We describe the implementation and evaluation details in Section 5 and Appendix A. We also pro- vide our code in the supplementary material.
REFERENCES
Sandra Aigner and Marco Körner. Futuregan: Anticipating the future frames of video se- quences using spatio-temporal 3d convolutions in progressively growing gans. arXiv preprint arXiv:1810.01325,, 2018.
Mohammad Babaeizadeh, Chelsea Finn, Dumitru Erhan, Roy H Campbell, and Sergey Levine. Stochastic variational video prediction. In International Conference on Learning Representa- fions, 2018.
Mohammad Babaeizadeh, Mohammad Taghi Saffar, Suraj Nair, Sergey Levine, Chelsea Finn, and Dumitru Erhan. Fitvid: Overfitting in pixel-level video prediction. arXiv preprint arXiv:2106.133 131955 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165, 2020.
Wonmin Byeon, Qin Wang, Rupesh Kumar Srivastava, and Petros Koumoutsakos. Contextvp: Fully context-aware video prediction. In Proceedings of the European Conference on Computer Vision (ECCV), 2018.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.
Joao Carreira, Eric Noland, Andras note about kinetics-666 -600. arXiv preprint
Chloe Hillier, and Andrew Zisserman. A short 2018.
Lluis Castrejon, Nicolas Ballas, and Aaron Courville. Improved conditional vrans for video predic- tion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019.
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and llya Sutskever. Generative pretraining from pixels. In International Conference Muchine Learning, 2020.
Xi Chen, Nikhil Mishra, Mostafa Rohaninejad, and Pieter Abbeel. Pixelsnail: An improved autore- gressive generative model. In International Conference Machine Learning, 2018.
Rewon Child. Very deep vaes generalize autoregressive models and can outperform them images. In International Conference on Learning Representations, 2020.
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv: 10509, 2019.
Aidan Clark, Jeff Donahue, and Karen Simonyan. Adversarial video generation on complex datasets. arXiv preprint arXiv: 1907006977 2019.
10

---

Under review as a conference paper at ICLR 2022
Sudeep Dasari and Abhinav Gupta. Transformers for one-shot visual imitation. In Conference Robot Learning. 2020.
Sudeep Dasari, Frederik Ebert, Stephen Tian, Suraj Nair, Bermadette Bucher, Karl Schmeckpeper, Siddharth Singh, Sergey Levine, and Chelsea Finn. Robonet: Large-scale multi-robot learning. In Conference on Robot Learning, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE conference on computer vision and paltern recognition, 2009.
Emily Denton and Rob Fergus. Stochastic video generation with a learned prior. In International Conference on Machine Learning. 2018.
Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis. arXiv preprint 1rXiv:2105.05233 2021.
Frederik Ebert, Chelsea Finn, Alex X Lee, and Sergey Levine. Self-supervised visual planning with temporal skip connections. In Conference on Robot Learning, 2017.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nilion, 2021.
Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguisties, 2018.
Chelsea Finn and Sergey Levine. Deep visual foresight for planning robot motion. In 2017 IEEE International Conference Robotics and Automation (ICRA). IEEE, 2017.
Chelsea Finn, Ian Goodfellow, and Sergey Levine. Unsupervised learning for physical interaction through video prediction. In Advances neural information processing systems, 2016.
Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets roboties: The kitti dataset. The International Journal of Robotics Research, 2013.
Pierre Gentine, Mike Pritchard, Stephan Rasp. Gael Reinaudi, and Galen Yacalis, Could machine learning break the convection parameterization deadlock? Geophysical Research Letters, 2018.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Infor- mation Processing Systems, 2014.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International Conference on Machine Learning, 2019.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, 2020.
Quan Huynh-Thu and Mohammed Ghanbari. Scope of validity of psnr in image/video quality as- sessment. Electronics letters, 2008
Phillip Isola, Jun Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2017.
Yunseok Jang, Gunhee Kim, and Yale Song. Video prediction with appearance and motion condi- tions. In International Conference on Machine Learning. 2018
Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, et al. Model-based reinforcement learning for atari. In International Conference on Learning Representations, 2020.
11

---

Under review as a conference paper at ICLR 2022
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation. In Conference on Robot Learning, 2018.
Nal Kalchbrenner, Aäron Oord, Karen Simonyan, Ivo Danihelka, Oriol Vinyals, Alex Graves, and Koray Kavukcuoglu. Video pixel networks. In International Conference on Machine Learning, 2017.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2020.
Seung Wook Kim, Yuhao Zhou, Jonah Philion, Antonio Torralba, and Sanja Fidler. Learning to simulate dynamic environments with gamegan. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2020.
Seung Wook Kim, Jonah Philion, Antonio Torralba, and Sanja Fidler. Drivegan: Towards a control- lable high-quality neural simulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. Advances in Neural Information Processing Systems, 2012.
Manoj Kumar, Mohammad Babaeizadeh, Dumitru Erhan, Chelsea Finn, Sergey Levine, Laurent Dinh, and Durk Kingma. Videoflow: A flow-based generative model for video. In International Conference on Learning Representations, 2020.
Yong-Hoon Kwon and Min-Gyu Park.
Predicting future frames using retrospective cycle gan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
Alex X Lee, Richard Zhang, Frederik Ebert, Pieter Abbeel, Chelsea Finn, and Sergey Levine. Stochastic adversarial video prediction. arXiv preprint orXiv:/804.01522. 2018.
Jesse Levinson, Jake Askeland, Jan Becker, Jennifer Dolson, David Held, Soeren Kammel, Zico Kolter, Dirk Langer, Oliver Pink, Vaughan Pratt, et al. Towards fully autonomous driving: Sys- tems and algorithms. In IEEE intelligent vehicles symposium (IV), 2011.
Xiaodan Liang, Lisa Lee, Wei Dai, and Eric P Xing. Dual motion gan for future-flow embedded video prediction. In Proceedings the IEEE International Conference on Computer Vision, 2017.
the
William Lotter, Gabriel Kreiman, and David Cox. Deep predictive coding networks for video pre- diction and unsupervised learning. In International Conference on Learning Representations. 2017.
Pauline Luc, Aidan Clark, Sander Dieleman, Diego de Las Casas, Yotam Doron, Albin Cassirer, and Karen Simonyan. Transformation based adversarial video prediction on large-scale data, arXiv preprint arXiv:2003.04033 2020.
Jacob Menick and Nal Kalchbrenner. Generating high fidelity images with subscale pixel networks and multidimensional upscaling. International Conference on Learning Representations, 2018.
Vincent Michalski, Roland Memisevic, and Kishore Konda. Modeling deep temporal dependencies with recurrent grammar cells. In Advances in Neural Information Processing Systems, 2014.
Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard Lewis, and Satinder Singh. Action-conditional video prediction using deep networks in atari games. In Advances in Neural Information Process- ing Systems, 2015.
Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learn- ing. In Advances in Neural Information Processing Systems, 2017.
Ruslan Rakhimov, Denis Volkhonskiy, Alexey Artemov, Denis Zorin, and Evgeny Burnaev. Latent video transformer. In International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, 2021.
12

---

Under review as a conference paper at ICLR 2022
MarcAurelio Ranzato, Arthur Szlam, Joan Bruna, Michael Mathieu, Ronan Collobert, and Sumit Chopra. Video (language) modeling: a baseline for generative models of natural videos. arXiv preprint arXiv:1412.6604, 2014.
Ali Razavi, Aaron den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. In Advances in Neural Information Processing Systems, 2019.
Scott Reed, Aäron Oord, Nal Kalchbrenner, Sergio Gómez Colmenarejo, Ziyu Wang, Yutian Chen, Dan Belov, and Nando Freitas. Parallel multiscale autoregressive density estimation, Interna- tional Conference on Machine Learning, 2017.
Oleh Rybkin, Chuning Zhu, Anusha Nagabandi, Kostas Daniilidis, Igor Mordatch, and Sergey Levine. Model-based reinforcement learning via latent-space collocation. In International Con- ference Machine Learning, 2021.
Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov. Unsupervised leamiing of video representations using lstms. In International Conference on Machine Learning, 2015.
Yu Tian, Jian Ren. Menglei Chai, Kyle Olszewski, Xi Peng. Dimitris N Metaxas, and Sergey Tulyakov. A good image generator what you need for high-resolution video synthesis. In International Conference on Learning Representations, 2021.
Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models video: A new metric challenges. arXiv preprint arXiv:1812.01777,, 2018.
Ruben Villegas, Arkanath Pathak, Harini Kannan, Dumitru Erhan, Quoc V Le, and Honglak Lee. High fidelity video prediction with large stochastic recurrent neural networks. Advances in Neural Information Processing Systems, 2019.
Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dynamics. In Advances in Neural Information Processing Systems, 2016.
Jacob Walker, Ali Razavi, and Aäron van den Oord. Predicting video with vqvae. arXiv preprint arXiv:2103.01950, 2021.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Ecro Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions image processing, 2004.
Dirk Weissenborn, Oscar Täckström, and Jakob Uszkoreit. Scaling autoregressive video models. In International Conference on Learning Representations, 2020.
Bohan Wu, Suraj Nair, Roberto Martin-Martin, Li Fei-Fei, and Chelsea Finn. Greedy hierarchi- cal variational autoencoders for large-scale video prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021a.
Chenfei Wu, Lun Huang, Qianxi Zhang, Binyang Li, Lei Ji, Fan Yang, Guillermo Sapiro, and Nan Duan. Godiva: Generating open-domain videos from natural descriptions. arXiv preprint
arXiv:2104. 14806, 2021b.
Huazhe Xu, Yang Gao, Fisher Yu, and Trevor Darrell. End-to-end learning of driving models from large-scale video datasets. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2017.
Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using vq-vac and transformers. arXiv preprint arXiv:2104.1010 (0157,202) 2021.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: benchmark and evaluation for multi-task and meta reinforcement learning. In Conference Robot Learning, 2020.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and patlern recognition, 2018
13

---

Under review as a conference paper at ICLR 2022
A EXPERIMENTAL SETUP
A.1 DATASETS
Meta-Worid. Meta-World (Yu al., 2020) is robotics manipulation simulator that supports 50 different tasks. For our experiments, we collect 50 demonstrations for each task using the determin- istic scripted policies.', hence we use total 2500 demonstrations. For evaluation, we construct the held-out test dataset with 10% of the demonstrations. To improve the visibility of rendered frames, we adjust the camera view the camera using the publicly available source codes. We provide the dataset we used in our experiments..
RoboNet. RoboNet (Dasari et al., 2019) is a large-scale real-world robotics dataset consisting of more than 160,000 high-resolution videos. Since there no for RoboNet dataset, follow the setup in Wu et al. (2021a) for constructing a held-out test dataset of size 256. Following the setup in Wu al. (2021a); Babaeizadeh et al. (2021), we train a video prediction model to predict ten future frames conditioned on two initial frames and ten future actions. For preprocessing the frames, we resize the original frames 256 256 resolution frames without cropping. For downloading the dataset, we utilize the publicly available script."
Kinetics-600. Kinetics-600 dataset is large-scale video dataset consisting of more than 400,000 videos of total 600 action classes. Following the setup in Clark et al. (2019); Luc et al. (2020), train a video prediction model to predict future 11 frames conditioned on the first five frames. For downloading the dataset, we use the publicly available repository!
BAIR Robot Pushing. BAIR Robot Pushing dataset (Ebert et al., 2017) 7) consists of 43,264 training and 256 test videos. While BAIR dataset contains the information of actions robots take, common setup for evaluation on BAIR dataset is action-free (Clark et al., 2019; Weissenborn et al., 2020; Luc et al., 2020; Yan al., 2021), where video prediction model is trained to predict future 15 frames conditioned on the initial frame. For downloading and preprocessing the dataset, we utilize the publicly available script." !!
KITTI driving dataset. KITTI driving dataset (Geiger al., 2013) is the dataset that contains a large number of high-resolution driving videos. However, for video prediction, we follow the setup in Lotter al. (2017) and utilize 57 training videos and 3 test videos for evaluation. To avoid utilizing too similar video clips from the test dataset for evaluation, we follow the setup in Villegas al. (2019) where 30-frame clip extracted with the interval of frames, which constructs test of size 148. For comparison with baselines, following the setup in Villegas et al. (2019), we train a model to predict future ten frames conditioned on five frames and evaluate the model to predict future 25 frames conditioned on five frames. For downloading and preprocessing the dataset, we utilize the publicly available script. 12
we use the of 9e3863d in http://gthhb..ommrrookgggggnnnnn
7 hhtp::/shorturl, .ttaaurx..////axMM
We use the list of videos available at tttp:///hhhoooooooe.eeeeee///eie
filenames.txt
"o www.///wioonuddttinniiiiteeeeeee
httpps//githuh.com. com/wilson /
Phupp://github... "hanp://gituuh.oo//////aa////// com.etaal/lolllllcceettttttt
14

---

Under review as a conference paper at ICLR 2022
A.2 IMPLEMENTATION DETAILS OF HARP
VQ-GAN. The training of HARP consists of two-stages. First, for training a VQ-GAN model (Esser al., 2021) we use the publicly available source code from the authors Following the sug- gestion of the authors, we trained a VQ-GAN model without a discriminator loss until it converges, then resume the training with discriminator loss for total (300000, 500000, 150000, 30000) train- ing steps with batch size of {24, 24, 320. 96} on Meta-World, RoboNet, BAIR Robot Pushing, and KITTI driving dataset, respectively. For Kinetics-600 dataset, we leverage the publicly avail- able VQ-GAN model pre-trained on ImageNet without training a VQ-GAN model from scratch. The size of codebook in (2) {8192, 8192, 1024, 1024, 1024] for Meta-World, RoboNet, Kinetics-600, BAIR Robot Pushing, and KITTI dataset, respectively.
Causal transformer. Then we train a 12 12-layer Sparse Transformers (Child et al., 2019) to predict the discrete codes from VQ-GAN models in an autoregressive manner, by building the publicly available source code of VideoGPT (Yan et al., 2021). For conditioning on initial frames, we utilize the same architecture in Yan al. (2021) that utilizes ResNet architecture extract the downsampled feature map. We utilize ResNet-18 architecture for all experiments. We train the model until it converges on Meta- Meta-World, BAIR, and KITTI datasets, but we cannot find the sign of overfitting on large-scale datasets of RoboNet and Kinetics-600, Specifically, we train the model for {50000, 80000, 100000, 85000, 30000) training steps on Meta-World, RoboNet, Kinetics-600, BAIR Robot Pushing, and KITTI dataset, respectively. We use the data augmentation with the magnitude of m on KITTI driving dataset, which has shown to be very effective for improving the performance (see Appendix B) for a learning curve with and without augmentation).
Inference with top-k sampling. We utilize top-k sampling (Fan et al., 2018) to improve the video prediction quality. For all datasets, we utilize k = 10. One major limitation of autoregressive prediction model is slow inference time. We utilize the implementation of Yan et al. (2021) that caches the previous key, values and utilize them for fast inference. In order to enable our model to extrapolate to unseen length of frames at evaluation time on KITTI dataset (e.g., the model has to predict 25 frames instead of 10 frames the model is trained to predict), we first predict T frames Xc.- I conditioned on initial frames Xi:e. then predict the next frames by (a) keeping Xise as conditioning frames and (b) giving the predicted last T 1 frames as inputs to the causal transformer model. We repeat this process until predicting all 25 future frames.
A.3 IMPLEMENTATION DETAILS OF ACTION-CONDITIONN CONDITIONED HARP.
In order to predict future frames conditioned on future ac- tions on RoboNet dataset (Dasari et al., 2019), we condi- tion the prediction on actions by adding the action embed- dings to the embeddings of discrete codes. Specifically, we introduce a linear layer that processes raw actions to action embeddings with the same dimension of token em- beddings, then add the action embeddings of time step t ++1 to token embeddings used for predicting the tokens of time step + 1. At inference time, the inference pro- cedure is exactly same as HARP except that future action embedding is added to the token embedding, We find that this simple modification to the original architecture en- ables HARP to predict future frames conditioned ac- tions. We provide the illustration of action-conditioned HARP in Figure
a
Linear
4
Causal Transformer
z(x:)
2 ((x+1)
Figure Action HARP.
14 "http://hcibox.u uni- heidd/hergg.ee///888888222 164 6d4c31a192/ 15 "hhps///ithuh.com// www.e///wwwwwooaainoeeeeeee
15

---

Under review as a conference paper at ICLR 2022
A.4 IMPLEMENTATION DETAILS OF HARP FINE-TUNING.
In order to fine-tune the HARP model for solving multi-task imitation learning tasks on Meta- World (Yu et al., 2020), first pre-train model to predict future 11 frames conditioned on the first frame using the dataset from all 50 tasks (see Appendix A. for the details on the dataset). Then we fine- tune the model predict expert actions by introducing a two-layer policy network on top of the causal transformer model. Specifically, we train behavioral cloning policy to minimize the mean squared error between the predicted actions from a layer and ground-truth expert actions. In order to further improve the performance of all methods, we follow the idea of Dasari & Gupta (2020) to learn a inverse dynamics predictor that predicts the action given two consecutive frames. We train all methods for 20,000 steps with the batch size of 20 and data augmentation of magnitude m = 4. For CNN-based methods, we use ResNet-50 for feature extractor, and use 4-layer LSTM 16 for CNN-LSTM, and 12-layer causal transformer for CNN- ANN-Transformer.
B EFFECTS OF AUGMENTATION
Figure 8 shows the test error during the training of HARP KITTI driving dataset (Geiger et al., 2013). One can see that the test eπror from a model trained without augmentation increases after initial ~ 2500 training steps, which is sign of overfitting. However, observe that the test error from a model trained with data augmentation keeps decreasing throughout training until it converges, which shows the effectiveness of data augmentation for learning video prediction models, similar to the observation in Babaeizadeh et al. (2021). One notable detail here is that HARP overfits to KITTI training dataset with much fewer number of parameters (i.e., 89M) when compared to FitVid that utilizes 303M parameters.
5.5
No augmentation Augmentation
5.0
Test Error
4.5
3.0
2800
5000
7500
10000
12500
15000
Training steps
Figure 8: Test loss during the training of HARP with and without data augmentation on KITTI driving dataset (Geiger al., 2013). We observe that the model easily overfits the training dataset, and data augmentation helps mitigate the overfitting problem.
16We try more deeper networks to match the number of trainable parameters, but we find that deeper LSTM networks are very unstable to train. Hence we search over the layers of {2, 4, 8, 16) and report the best results achieved with 4-layer L.STM.
16

---

Under review as a conference paper at ICLR 2022
C VIDEO PREDICTIONS ON BAIR AND KITTI
We provide the future frames predicted by HARP on BAIR Robot Pushing (Ebert et al., 2017) and KITTI driving dataset (Geiger et al., 2013). The model is trained predict future 15 frames conditioned on the first frame on BAIR dataset. In the case of KITTI dataset, the model is trained predict future 10 frames conditioned on the five frames, and evaluated predict future 25 frames conditioned on the five frames.
19 Contedt Frame
5
10
15
IF Contest Frame
1-5
30
c= 15
Ground
Groune
Predicted
Predicted
La Currest Frrte
1=1
==5
t= 10
15
If Conteat Frame
t=1
-
17
25
Crmund Trahh
Ground
Prediced
Predicted
Figure 9: 64 64 future frames predicted by HARP on BAIR dataset (up) and KITTI dataset (bottom). We observe that our model can predict diverse future frames with stochastic sampling on BAIR dataset, and can deal with the partial observability the KITTI dataset.
D MORE VIDEO PREDICTIONS ON META-WORLD.
We provide more videos predicted by HARP on Meta- World dataset (Yu al., 2020). The model is trained to predict future 11 frames conditioned on the first frame.
Figure 10: 64 x 64 future frames predicted by HARP on Meta-World dataset.
17

---

Under review as a conference paper at ICLR 2022
E MORE VIDEO PREDICTIONS ON ROBONET
We provide more videos predicted by HARP on RoboNet dataset (Dasari et al., 2019). The model trained predict future frames conditioned the initial frames and future ten actions.
Context Frames
t=1
1=2
t=4
1=6
==8
t= 10
Ground Truth
Predicted
Ground Truth
Predicted
Context Frames
8 1
f=2
t=4
1=6
1=8
1=
10
Ground Truth
Predicted
Ground Truth
Predicted
Figure 11: 256 x 256 future frames predicted by HARP on RoboNet dataset.
18

---

Under review as a conference paper at ICLR 2022
F
MORE VIDEO PREDICTIONS ON KINETICS-600
Context Frames
1
1= 2
t= 4
1= 6
1= 8
t=
10
Ground Truth
Predicted
Ground Truth
Predicted
Context Frames
t= 1=2
t= 4
f= 6
t= 10
Ground Truth
Predicted
Ground Truth
Predicted
Figure 12: 256 256 future frames predicted by HARP on Kineties dataset.
19

---

Under review as a conference paper at ICLR 2022
G ATTRIBUTION
Figure (top):
"Windsurfing Session 2013 Iballaa Moreno" by morenotwins. Accessible here
Figure (bottom):
"How to fry egg in 5 simple steps" by What's Gaby Cooking. Accessible here
Figure (right):
"Prepare fruit cutting to pets #Shorts 4" by AP STUDIO. Accessible here
Figure 12 (first):
"Windsurfing" by Dmitry Rudnev. Accessible here
Figure 12 (second):
"Eric Cornelissen Windsurfing September 11, 2017" by Ron Van Dijk. Accessible here
Figure 12 (third);
"Smart way cut Fruits #6#" by MR. BEING SMART. Accessible here
Figure 12 (fourth):
"How make the Perfect Crunchy Half Cakes Kangumu IIIl Jinsi ya kupika Half Cakes za kupasuka." by Ayleen's 's Food & Vlogs, Accessible here
20


Reviews:
[{"review_id": "zCX9VP8I5uL", "reviewer": "Reviewer_6x6m", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed method is an interesting contribution for the community, since it reduces the complexity of learning video prediction models with appealing results. The paper contains a simple, actionable and effective methodology which is supported by the experiments. High-resolution ($256 \\times 256$) forecasting is one of the main advantage of HARP as it is rare in the literature, especially using reasonable computing power. The empirical evaluation partly illustrates the benefits of the method, thanks to results matching the state of the art with limited parameter count. Overall, the paper is well written and easy to read, and experiments seem to be reproducible.\n\nHowever, I find the contribution of this paper to be insufficient for two main reasons.\n\n### Novelty\n\nThe first limitation deals with the novelty of the method. As acknowledged by the authors, the proposed two-stage training has already been proposed in prior works: Rakhimov et al. (2021) and Yan et al. (2021). While these articles are recent, the reviewed paper is not concurrent and clearly builds on them. Indeed, to my understanding, the proposed method amounts to instantiating e.g. VideoGPT (Yan et al., 2021) with another more powerful image generator of the same nature (VQGAN instead of VQVAE) and the same transformer architecture. This significantly reduces the originality of the approach.\n\n### Experiments\n\nThe second limitation lies in the experimental evaluation which does not compensate the lack of novelty, thereby limiting the significance of the overall contribution. The issues are multiple and described below.\n\nRegarding the forecasting experiments on Meta-World, RoboNet and Kinetics-600, presented results seem satisfying but there lacks some point of comparison in the paper to assess their significance. This could consist in performance comparison with another baseline (like Video transformer which was tested on Kinetics in the original article or another autoregressive model like VideoGPT) or, because of scalability or computational power issues, a more detailed analysis of the computational advantages of HARP against the other baselines. The only provided comparison is on parameter count vs. a single other model, which is insufficient (cf. also the forthcoming remarks on parameter count). I do believe that HARP is computationally appealing, but a factual comparison is needed to confirm this impression.\n\nRegarding the forecasting experiments on BAIR and KITTI, the presented results do not allow us to unambiguously conclude about the advantages of HARP. The only comparison with a similar model (VideoGPT) is not conclusive, since the parameter count and performance of both models are similar. More generally, there are fairness issues in the comparisons that prevent the reader to correctly assess the performance of HARP. Indeed, top-$k$ sampling and data augmentation are beneficial to the model but it is unclear how the other models could benefit from them as well and how this would change the performed comparison. For example, the authors may discuss the impact of data augmentation on the considered baselines, as well as the links of top-$k$ sampling with temperature reduction in generative models like in VideoFlow (Kumar et al., 2021) which could similarly improve the performance of the other models.\n\nFinally, I fear that the sole indication of parameter count to evaluate the efficiency of a model may be insufficient for these comparisons, as it is an imperfect proxy of the actual capacity of computational requirements of a model (unless the numbers of parameters are orders of magnitude different). More information like GPU, memory and training time requirements for the baselines would circumvent this issue. Ideally, a study on the influence of the number of parameters on the performance of HARP and selected baselines would greatly improve the comparison.\n\nLess importantly, I would optionally suggest the authors to complement the numerical evaluation with the standard PSNR metric, like e.g. Denton et al. (2018) and Lee et al. (2018). Despite its flaws, PSNR is complementary to the perceptual scores FVD and LPIPS as it can better reveal dynamics prediction errors. Considering a synthetic dataset like Moving MNIST, that remains challenging for long-term prediction, would also help in this regard to confirm the ability of transformers to accurately predict dynamics on pretrained representations as proposed in the paper.", "summary_of_the_paper": "This paper introduces a new video prediction model, HARP, whose training operates in two stages: pretraining an image autoencoder on the frames with discrete latent states, then learning a transformer predictor on the resulting latent space instead of the pixel space. The authors complement this procedure with data augmentation during training and a modification of latent code sampling during inference in order to improve the performance of the model. This experimentally results in an efficient training procedure, with a model that is demonstrated to be able to generate high-resolution videos and match or outperform the state of the art in the domain on multiple datasets and tasks.", "summary_of_the_review": "Considering the two important limitations described above, I find this paper to be below the acceptance threshold. Given the limited technical novelty, the empirical evaluation should be stronger for this paper to be accepted. I believe that such an improvement could make it ready for publication.\n\nI understand that the suggested improvements are significant and it may be difficult for the authors to properly address them during the rebuttal. Nonetheless, I would be pleased to increase my score, should my concerns or misunderstandings be clarified during the discussion.\n\n### Post-Rebuttal Update\n\nI acknowledge the authors' response and would like to thank them for their answers. As stated by the authors themselves, the necessary improvements could not have been implemented during the rebuttal, which is understandable. Therefore, I choose to maintain my score.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "uRgHWX4C5yX", "reviewer": "Reviewer_pKCo", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n\n* The paper is well written and easy to read and the proposed framework is conceptually simple.\n* Authors show qualitative results on 256x256 images which is quite challenging. \n* The reported results on KITTI are strong.\n\n**Weaknesses**\n\n* The main technical novelty seems to be in the usage of VQ-GAN model (instead of the VQ-VAE model) to produce latent codes. But there is no ablation that quantifies the impact of this design choice. The authors should replace VQ-GAN with the standard VQ-VAE model and report the improvements (if any).\n* The paper claims that the prior related works are limited to generation of 128x128 images. But it is unclear what design decisions allow HARP to generate 256x256 resolution images. In terms of complexity, the proposed HARP should be similar to LVT (Rakhimov et al 2021) and VideoGPT (Yan el atl 2021) since both rely on attention-based architectures in latent space. Is it that the prior works could generate 256x256 images but with poor fidelity due to the reliance on VQ-VAE (instead of VQ-GAN)? This is simple to verify by just replacing the VQ-GAN with VQ-VAE and reporting the results on 256x256 images.\n* The authors should report their quantitative results on Kinetics600 64x64 to provide another point of comparison. Additionally, they should report their numbers on Kinetics600 256x256 with and without ImageNet pre training as a future baseline. \nFurther, [Clark et al 2019] report results on high resolution Kinestics600 in Table 1 of their paper, which should be compared to.\n* The final comparison in Table 2 is reported using top-k sampling, which is unfair to both LVT and VideoTransformer as top-k sampling is readily applicable to these models as well. After removing top-k sampling, the performance of HARP is comparable to VideoGPT.\n* What is “number of layers” in the Results Section? Is it the number of layers in VQ-GAN or the 1-D Transformer?\n* How was HARP tuned? The datasets in Section 5.2 only talk about the train and test split but no validation split?\n* Can the authors provide some ablations on the codebook size? Since it is specifically stated that “given that the number of discrete encodings required for future prediction is very large, e.g., 2,560 on RoboNet (Dasari et al., 2019) up to 6,400 on KITTI dataset (Geiger et al., 2013) in our experimental setup.”. How were these codebook numbers arrived at?\n", "summary_of_the_paper": "The paper proposes an autoregressive latent video prediction model with two components:\n* An image generation model (VG-GAN) is trained to compress an image into 2-D latent codes.\n* A 1-D autoregressive transformer generates the next latent code given the flattened history of the previous latent codes, i.e (H*W*T). This latent code is then transformed into the next frame of the sequence by the decoder of the VQ-GAN model.\n* Quantitative Results are shown on 64x64 Images on the BAIR Robot Pushing dataset and KITTI driving dataset.\n* Some performance improvement is shown using top-k sampling on the BAIR dataset and KITTI dataset and data-augmentation (pixel-translation) on the KITTI dataset.\n* Qualitative Results are shown on high-resolution images (256x256) on the RoboNet, MetaWorld and Kinetics600 datasets.\n", "summary_of_the_review": "My initial rating is reject. The technical novelty of the paper is arguably only the VQ-GAN encoder. The authors should show more empirical evidence that this component is crucial for their results on video prediction. This can be done via ablations and comparisons with prior work (as described in my weaknesses section)", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "9ikn_nBC_Sf", "reviewer": "Reviewer_3gJ8", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths:** I think this work fits well into the recent literature on generative models, which divides the learning process of perceptually highly redundant data (such as images, audio, or videos) into a two-stage procedure (e.g., [1, 2, 3]), where the first stage is trained to achieve maximum compression while still yielding perceptually close representations, and the actual generative model is then trained on this compressed representation in a second stage. This has the distinct advantage that the generative model does not have to \"relearn\" the compression itself (which is particularly difficult in mode-covering approaches such as AR likelihood models) and can focus on salient high-level features. However, this also reveals a weakness of the present work: the \"compressibility\" is only exploited for single frames, but not for the temporal dimension, which is particularly wasteful for video data. \n\n**Weaknesses:** The novelty of _HARP_ over previous works is quite limited: There is a large body of work such as [4,5,6] that uses a two-stage procedure for video generation.  The main advantage over VideoGPT is that HARP does not rely on a hierarchical representation and uses a VQGAN instead of a VQVAE. Also, I don't quite understand why top-k sampling is \"sold\" so much; it is an established technique in dealing with autoregressive models. What about other pruning techniques like nucleus sampling or temperature rescaling of logits? In addition, the work should also qualitatively demonstrate sensitivity to the specific value of $k$. Finally, a quantitative analysis of the \"high resolution\" experiments ($256 \\times 256$ px) is missing.\n\n**Mixed Comments:**\n- The technical details regarding the baseline models in Sec. 5.3 are very sparse. What does \"on top of the transformer model\" mean? How exactly does the CNN+Transformer model differ from the VQ+Transformer model? Is it possible to train only the policy network while fixing the (pre-trained) transformer model?\n- Does the VQGAN's codebook collapse affect training, and if so, how is it taken into account? \n- How long does it take to sample the different models? How does it compare to other approaches?\n- What is $m$ in Sec. 5.2?\n- Sec. 5.2 and Tab.2 heavily rely on the fact that HARP uses less parameters than some other models, but does not explicitly state why this is useful. See also [7] on this \"efficiency misnomer\".\n\n__References__\n- [1]: Esser, Patrick, Robin Rombach, and Bjorn Ommer. \"Taming transformers for high-resolution image synthesis.\" _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 2021.\n- [2]: Iashin, Vladimir, and Esa Rahtu. \"Taming Visually Guided Sound Generation.\" _arXiv preprint arXiv:2110.08791_ (2021).\n- [3]: Mentzer, Fabian, et al. \"High-fidelity generative image compression.\" _arXiv preprint arXiv:2006.09965_ (2020).\n- [4]: Yan, Wilson, et al. \"VideoGPT: Video Generation using VQ-VAE and Transformers.\" _arXiv preprint arXiv:2104.10157_ (2021).\n- [5]: Rakhimov, Ruslan, et al. \"Latent video transformer.\" _arXiv preprint arXiv:2006.10704_ (2020).\n- [6]: Dorkenwald, Michael, et al. \"Stochastic Image-to-Video Synthesis using cINNs.\" _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 2021.\n- [7]: Dehghani, Mostafa, et al. \"The Efficiency Misnomer.\" _arXiv preprint arXiv:2110.12894_ (2021).\n", "summary_of_the_paper": "This paper proposes a very simple method (dubbed _HARP_) for video synthesis: By relying on a pretrained VQGAN which encodes (and decodes) each frame seperately to (and from) a highly compressed representation, a (sparse) transformer model can be trained as an autoregressive generative model on the sequence of discrete image representations. The paper shows that despite its simplicity, the proposed approach can compete with the state of the art and produce high quality video at $256 \\times 256$ resolution. As a side contribution, it is shown that the proposed model can be fine-tuned towards multi-task imitation learning.", "summary_of_the_review": "The main advantage of the paper is its simplicity, and that is what distinguishes it from other approaches, i.e. it is simpler, but achieves better or close to SOTA results. The demonstration that such a simple approach can give (near) top results is definitely interesting.\nHowever, the novelty value compared to previous approaches is very limited. All in all, I'm leaning towards a rejection because I'm not convinced that the results of the paper warrant a full conference publication and might be better suited for a workshop, but I'm curious to see what the other reviewers think.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7ssN3aVcS8i", "reviewer": "Reviewer_3FiN", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n* The paper is well-written and easy to follow.\n* The idea is simple and clear.\n* It is interesting to see an ImageNet pre-trained VQ-GAN can be used for predicting Kinetics-600 videos\n\nWeaknesses:\n* Loss functions are missing.\n* Though using fewer parameters, the proposed method, HARP, doesn’t outperform baselines on the BAIR dataset. As in table 2(b), simply increasing the transformer layers can improve the performance, it is interesting to see if HARP can outperform the baselines by increasing the parameters to the same level (~300M).\n* The quantitative evaluation of video prediction is only conducted on BAIR and KITTI datasets, as KITTI is a small-scale dataset, it’s better to evaluate HARP with more datasets.\n* There are domain gaps between ImageNet images and Kinetics-600 videos, how does HARP deal with such domain gaps? Is finetuning needed? Also in the paper, the predicted videos are only shown with some selected categories. How about the performance on the whole Kinetics-600 datasets? (e.g. examples on more categories and/or FVD scores on the whole dataset) ", "summary_of_the_paper": "This paper introduces HARP for high-resolution video prediction. Specifically, they leverage a pre-trained image generator (VQ-GAN) and fix it in the video prediction stage. The predicted codes are generated by a progressive causal transformer. Other techniques, like top-k sampling and data augmentation, are further employed to improve the diversity and quality. Experiments show it outperforms baselines on several datasets.", "summary_of_the_review": "This paper introduces a new method for high-resolution video prediction, however, the experiments are not sufficient, I believe more experiments are needed to justify the effectiveness of the proposed method.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "22EyPvpodh", "reviewer": "Reviewer_JjT4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n   + The paper is well written and easy to follow.\n   + The idea of leveraging pre-trained image generator for high-fidelity frame outputs while using transformer to focus on learning video dynamics is well motivated.\n   + Experiments on a wide range of video prediction benchmarks demonstrate the proposed method can generate visually good results and performs well compared to state-of-the-art methods on this challenging problem.\n\nWeaknesses:\n   + While I appreciate the good results achieved by the proposed method, my major concern is that the novelty of it is somewhat limited. The idea of combining image generator with latent space auto-regressive model has been widely explored in previous works ([1], [2], [3], [4]). At the high level, the idea of the proposed method is similar to [1], only with VQ-VAE replaced by VQ-GAN. It is not surprising that the visual quality of the prediction results improves with a better image generator and I don't feel that it provides much novelty by itself.\n   + I also feel that the paper lacks in-depth discussion on the relation between the proposed method and other relevant works. In particular, the overall framework followed by the proposed method, including the top-k sampling strategy, is very similar to that of [5]. Yet there is no discussion on [5] except mentioning of the use of the same VQGAN model. Can the proposed method be considered a direct extension of [5] from image generation to video generation? Is there any fundamental challenge in doing that extension? and is there any fundamental change that the author incorporated to address such challenge? \n\n[1] Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using\nvq-vae and transformers. arXiv preprint arXiv:2104.10157, 2021.\n\n[2] Dirk Weissenborn, Oscar Tackstr ¨ om, and Jakob Uszkoreit. Scaling autoregressive video models. In ¨\nInternational Conference on Learning Representations, 2020.\n\n[3] Ruslan Rakhimov, Denis Volkhonskiy, Alexey Artemov, Denis Zorin, and Evgeny Burnaev. Latent\nvideo transformer. In International Joint Conference on Computer Vision, Imaging and Computer\nGraphics Theory and Applications, 2021.\n\n[4] Yu Tian, Jian Ren, Menglei Chai, Kyle Olszewski, Xi Peng, Dimitris N Metaxas, and Sergey\nTulyakov. A good image generator is what you need for high-resolution video synthesis. In\nInternational Conference on Learning Representations, 2021.\n\n[5] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image\nsynthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.    ", "summary_of_the_paper": "This paper presents a video prediction method that combines a transformer-based auto-regressive model with a pre-trained image generator for high-resolution frame generation. The image generator is based on VQGAN. The auto-regressive model is trained to predict the discrete VQGAN's latent codes which is then used by the image generator to generate patches in the video. Experiments on different video prediction benchmarks demonstrate the effectiveness of the proposed method for high-resolution (256x256) video prediction. ", "summary_of_the_review": "\nIn general, I found the proposed method effective in generating visually good video prediction results. However, I found the technical contribution of the paper somewhat limited. Each technical component is not new and the way of combining them has also been explored before, which limit the novelty value of the paper. Please see the previous box for more strengths and weaknesses details.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission proposes to learn a causal transformer model over a pretrained VQ-GAN representation to generate videos. While the paper is well written and clear, proposing a simple idea, the novelty of this method is not well explained compared to pre-existing publications see ([reviews JjT4](https://openreview.net/forum?id=K-hiHQXEQog&noteId=22EyPvpodh), [3gJ8](https://openreview.net/forum?id=K-hiHQXEQog&noteId=9ikn_nBC_Sf), [6x6m](https://openreview.net/forum?id=K-hiHQXEQog&noteId=zCX9VP8I5uL), and [pKCo](https://openreview.net/forum?id=K-hiHQXEQog&noteId=uRgHWX4C5yX)) especially since it's lacking [ablation](https://openreview.net/forum?id=K-hiHQXEQog&noteId=uRgHWX4C5yX) or comparative (see [reviews 6x6m](https://openreview.net/forum?id=K-hiHQXEQog&noteId=zCX9VP8I5uL), and[pKCo](https://openreview.net/forum?id=K-hiHQXEQog&noteId=uRgHWX4C5yX)) experiments. 
The authors have expressed their consideration of reviewers requests but will not satisfy them in time for this conference. Therefore I am currently recommending this submission for rejection.
Decision: Reject