FitVid: High-Capacity Pixel-Level Video Prediction
Authors: Mohammad Babaeizadeh
Mohammad Taghi Saffar
Suraj Nair
Sergey Levine
Chelsea Finn
Dumitru Erhan
Venue: ICLR 2022 Submitted (2022)
Keywords: video prediction
self supervised learning
unsupervised learning
robotics

Abstract: An agent that is capable of predicting what happens next can perform a variety of tasks through planning with no additional training. Furthermore, such an agent can internally represent the complex dynamics of the real-world and therefore can acquire a representation useful for a variety of visual perception tasks. This makes predicting the future frames of a video, conditioned on the observed past and potentially future actions, an interesting task which remains exceptionally challenging despite many recent advances. Existing video prediction models have shown promising results on simple narrow benchmarks but they generate low quality predictions on real-life datasets with more complicated dynamics or broader domain. There is a growing body of evidence that underfitting on the training data is one of the primary causes for the low quality predictions. In this paper, we argue that the inefficient use of parameters in the current video models is the main reason for underfitting. Therefore, we introduce a new architecture, named FitVid, which is capable of fitting the common benchmarks so well that it begins to suffer from overfitting -- while having similar parameter count as the current state-of-the-art models. We analyze the consequences of overfitting, illustrating how it can produce unexpected outcomes such as generating high quality output by repeating the training data, and how it can be mitigated using existing image augmentation techniques. As a result, FitVid outperforms the current state-of-the-art models across four different video prediction benchmarks on four different metrics. 

Under review as a conference paper at ICLR 2022
FITVID: HIGH-CAPACITY PIXEL-LEVEL VIDEO PREDICTION
Anonymous authors
Paper under double-blind review
ABSTRACT
An agent that is capable of predicting what happens next can perform a variety of tasks through planning with additional training. Furthermore, such an agent can internally represent the complex dynamics of the real-world and therefore can acquire a representation useful for many other visual perception tasks. This makes predicting the future frames of a video, conditioned on the observed past and potentially future actions, an interesting task which remains exceptionally challenging despite many recent advances. Existing video prediction models have shown promising results on simple narrow benchmarks but they generate low quality predictions on real-life datasets with more complicated dynamics or broader domain. There growing body of evidence that underfitting on the training data is one of the primary causes for the low quality predictions. In this paper, we argue that the inefficient use of parameters in the current video models is the main reason for underfitting. Therefore, which is capable of fitting the common benchmarks so well that begins to suffer from overfitting while having similar parameter count as the current state-of-the art models. We analyze the consequences of overfitting, illustrating how it can produce unexpected outcomes such generating high quality output by repeating the training data, and how it can be mitigated using existing image augmentation techniques. As a result, FitVid outperforms the current of-te-of-the-art models across four different video prediction benchmarks on four different metrics.
introduce new architecture, named FitVid,
INTRODUCTION
Context Frame
t=5
t= 10
Ground Truth
Predicting what happens next cornerstone of intelli- gence and one of the key capabilities of humans, which we heavily rely on to make decisions in everyday life (Bubic et al., 2010). This capability enables us to anticipate future events and plan ahead to perform temporally extended tasks. While the machine learning literature has studied wide range of prediction problems, one of the most direct challenges is to predict raw sensory inputs. In particu- lar, prediction of future visual inputs conditioned on a context of past observations - i.c., pixel-level video pre- diction - encapsulates the challenges of visual perception, modeling of physical events, and reasoning about uncer- tain behaviors. Video prediction can be formulated as a self-supervised problem, enabling us use substantial amount of unlabeled data to provide autonomous systems with powerful predictive capabilities as well as learning rich representations for downstream tasks. Already, video models have been successfully deployed in applications such as robotics (Finn & Levine, 2017; Zhang et al., 2019). simulation (Kim et al., 2021; 2020), compression (Duan et al., 2020; Sulun & Tekalp, 2021) and video synthesis from a single frame (Endo et al., 2019; Nam et al., 2019).
Predicted
Figure 1: FitVid predicts high qual- ity images of the future given the first few frames. Here, the visual detail on the pushed object is preserved ef- feetively, and even the shadow of the robot's 's arm is predicted correctly. In this example, FitVid is trained on 15M video frames of 7 diffedrent robots from RoboNet (Dasari et al., 2019).
Despite recent advances in generative models in many domains, such as images (Karras et al., 2017; Vahdat Kautz, 2020) and text (Devlin et al., 2018; Brown et al., 2020), video prediction still considered to be challenging. Current state-of-the-arr methods are limited to low-resolution videos (typically 64x64 (Villegas et al., 2019) and a maximum of 256 256 (Kim et al., 2021)), usually in narrow domain, such single human walking Ionescu et al. (2014), objects in a stationary setting Ebert et al. (2017). Even then the quality of predicted frames tends to drop substantially after little 10 seconds into the future (Saxena et al., 2021).
robotic arm pushing

---

Under review as a conference paper at ICLR 2022
A growing body of evidence suggests that underfitting is one of the primary reasons for low quality predictions. For example, Villegas et al. (2019) demonstrate how scaling the model, by adding more parameters, can substantially improve the prediction quality. Similarly, Castrejon et al. (2019) argue that blurry video prediction of variational methods is a sign of underfitting, exhibiting an improved test and train performance the network capacity increases. Wu et al. (2021a) also observed monotonic improvement as the number of modules in a hierarchical architecture increases. While scaling up models is common to address underfitting, comes the cost of more computation, memory, and integration risks (Wolf et al., 2019) as well as more complicated training regimes (Wu et al., 202 la).
In this paper we take step back and address underfitting by instead finding an architecture that uses its parameters more efficiently. We propose FitVid, a model that with the same parameter count as current state-of-the-arr models - can fit the current video prediction benchmarks successfully. In fact, VidVid fits these benchmarks so well that in many cases we find it actually begins to suffer from overfitting. To the best of our knowledge, this is the first time a video model reports substantial overfitting on theses benchmarks. Importantly, we also find that simple existing image augmentation techniques can mitigate this overfitting, leading to models that can both fit the training set and generalize well to held-out videos.
As a result, FitVid achieves state-of satte-of-the-art -hrt on four challenging video datasets across a wide range of metrics. Furthermore, while each of the components of our proposed model are not novel on their own, we find that FitVid can be trained a significantly simpler training recipe. Prior works on video prediction, particularly those that make of variational methods to provide for stochasticity, typically require a number of delicate design decisions to train successfully; curriculum training (Oh al., 2015; Lee et al., 2018; Cenzato al., 2019), learned prior (Denton & Fergus, 2018; Castrejon et al., 2019) and annealing of the weight on the VAE KL-divergence penalty (Babaeizadeh et al., 2018). In contrast, we show that our method actually fits the training data well without any such components, training directly via optimizing the evidence lower bound with minimal hyperparameters.
2 RELATED WORK
Video prediction (Ranzato et al., 2014; Srivastava al., 2015) has been formulated in different ways such as generating videos from a single image (Endo et al., 2019; Nam et al., 2019; Dorkenwald et al., 2021; Yang et al., 2021; Hu et al., 2021) or no image (Vondrick al., 2016; Saito & Saito, 2018; Clark et al., 2019), text to video generation (Wu et al., 2021b), video-to-viee -ovecooo translation (Wang et al., 2019a: 2018) and data-driven simulation (Kaiser et al., 2019; Kim et al., 2021; 2020). In this paper, our focus is on conditional video prediction, which is to predict the future frames of video conditioned on a few initial context frames and possibly the future actions of the agents (Finn et al., 201 6a; Babaeizadeh et al., 2018). Conditional video prediction has number of applications, including model-basee reinforcement learning from pixels (Hafner et al., 2018; 2019; Kaiser et al., 2019; Rafailov et al., 2020) and robotics (Boots al., 2014; Kalchbrenner et al., 2017a; Ebert et al., 2017: 2018a;b; Xie et al., 2019; Paxton et al., 2019; Nair & Finn, 2020; Nair et al., 2020).
Initially, video prediction was tackled using deterministic models (Walker et al., 2015; Jia et al., 2016; Xue et al., 2016; Walker et al., 2016; Liang et al., 2017: Byravan & Fox, 2017; Vondrick & Torralba, 2017; Van Amersfoort et al., 2017: Liu al., 2017: Chen et al., 2017: Lu al., 2017). Later on, given the common randomness and partial observability in the real-life situations, various stochastic models were proposed to capture the stochasticity of the future. Generative adversarial networks (GANs) (Goodfellow et al., 2014) are used for video predictions (Mathicu et al., 2015; Lee et al., 2018; Clark et al., 2019; Luc al., 2020; Hong et al., 2021) as sell as autoregressive models (Kalchbrenner et al., 2017b; Reed et al., 2017; Weissenborn et al., 2019). Flow based generative models (Dinh et al., 2014; 2016) are also adopted for video prediction (Kumar et al., 2019).
In this paper we focus on Variational Auto-Encoders (VAEs) (Kingma & Welling, 2014) which are widely used for conditional video prediction (Shu et al., 2016; Babaeizadeh et al., 2018; Denton & Fergus, 2018; Wichers et al., 2018; Villegas et al., 2019; Franceschi et al., 2020; Castrejon et al., 2019; Yan et al., 2021; Rakhimov et al., 2020; Lee et al., 2021; Wu et al., 2021a; Walker et al., 2021). Recently, it has been demonstrated that underfitting on training data plays major role in low quality blurry predictions in VAE based models. Villegas et al. (2019); Castrejon et al. (2019); Wu et al. (2021a) all reported improved prediction quality as the network capacity increases. In this paper, are interested in addressing underfitting without increasing the capacity of the current models, and instead, find more expressive architecture which uses its capacity more efficiently.
2

---

Under review as a conference paper at ICLR 2022
Squesze
LSTM Sep Connection
30333
18
+m
In km
Se
3/3 daumd
2000 BIE 2U mei
in 20oil +0
11
2016 Dr Tel
250 S0 SW1
siedh
Bnc 1
In Un
mm
Im
In
2000
100
Figure 2: The FitVid architecture. In the figure. (bn) is Patch-normalization (Ioffe & Szegedy, 2015), (swish) is the activation (Ramachandran et al., 2017). (s&e) is Squeeze and Excite (Hu et al., 2018) and (N N) is a convolutional layer with kernel size N N.N. The strides are always one, except when down-sampling which has stride of two. For up-sampling we nearest neighbour. The number under each box shows the number of filters while the top numbers indicate the input size. To model the dynamics, we use two layers of LSTMs (Hochreiter & Schmidhuber, 1997).
THE FITVID MODEL
Striving for simplicity, we propose the FitVid model for stochastic video prediction, convolutional non-hierarchical variational model with fixed prior. In this section we will describe the proposed model and arcitecture which is also visualized Fig 2. The appendix includes more architecture details and the code is available in Supplementary Material.
3.1 VARIATIONAL VIDEO PREDICTION WITH FIXED PRIOR
Following prior work (Finn et al., 2016a; Babaeizadeh et al., 2018; Denton & Fergus, 2018 8; Rubinstein, 1997: Oprea al., 2020; Wu al., 2021a), we define the problem of pixel-level video prediction as follows: given the first frame of video X<e = X4. Xi...., Xc-1. our goal is to predict the future frames by sampling from p(xe:T] x<<e). Optionally, the predictive model can be conditioned on additional given information a, such as the actions that the agents in the video are planning to take. This is typically called action-conditioned video prediction.
Variational video prediction (Babaeizadeh et al., 2018) follows the variational onto-encoder (Kingma & Welling, 2014) formalism by introducing a set of latent variables to capture the inherent stochasticity of the problem. We factorize the likclihood model to I po (x. |x Z5t) which is parametrized in autoregressive manner time; i.e. at each timestep i the video frame x, and the latent variables 2, are conditioned on the past latent samples and frames.
For inference, we need to compute marginalized distribution over the latent variables z, which is intractable. To overcome this problem, we use variation inference (Jordan et al., 1999) by defining an amortized approximate posterior q(z)x) II. (2,l2.c, xx+) that approximates the posterior distribution p(z]x). The approximated posterior is commonly modeled with an in- ference network (z(x) that outputs the parameters a conditionally Gaussian distribution N (x)(x),, (x)). This network can be trained using the reparameterization trick (Kingma & Welling, 2014) by optimizing the variational lower bound (Kingma & Welling, 2014; Rezende et al., 2014): [(x) Egg (mpx) [log Po (xxr |x z)] + 3DDL (z(xx)))lp(z)) where DKL is the Kullback-Leible divergence between the approximated posterior and the fixed prior p(z) = ~ I).
The hyper-paramete represents the trade-off between minimizing frame prediction error and fitting the prior (Higgins et al., 2016b;a; Denton & Fergus, 2018). For Fit Vid, we set 1.
3.2 THE PROPOSED ARCHITECTURE
Encoder and decoder. Following the recent advances image generation, we use similar residual encoding and decoding cells as NVAE (Vahdat & Kautz, 2020). Each cell includes convolutional
3

---

Under review as a conference paper at ICLR 2022
layers with cattc-normalizattion (loffe & Szegedy, 2015) and swish (Ramachandran et al., 2017; Elfwing et al., 2018; Hendrycks & Gimpel, 2016) as the activation function, followed by Squeeze and Excite (Hu al., 2018). The encoder is made of four encoding blocks with two cells in each block. There is down-sampling after each encoder block using a strided convolution of size three in the spatial dimensions. The decoder also consists of four decoding blocks with two cells in each block, and nearest neighbour up-sampling after each block. The number of filters in each encoding block doubled while the number of filters in each decoding block is halved from the previous one. There is a residual skip connection between the encoder and the decoder after each cell which are fixed to the output from the last context frame. The statisties for batch- noncanoomaliitiionn is averaged across time.
Dynamics model. Similar to Denton & Fergus (2018), the encoded frame h, is used to predict h using two layers of LSTMs (Hochreiter Schmidhuber, 1997). Likewise, ((xx(x<r)) is also modeled using a single layer of LSTMs with hę+1 as the input that outputs the parameters of a conditionally
Gaussian distribution N (iJs(( a (x)). During the training, z is sampled from (Zz,) xxx)) while at the inference time is sampled from the fixed prior (O, I). The input to the model is always the
ground-truth image (which is usually referred to as teacher-forcing (Goodfellow et al., 2016; Cenzato et al., 2019)). At inference time, the predicted image in the previous time-step is used as input to predict the next frame.
Data augmentation. We find that Vid can fit even very large datasets very well. However, in many cases, even for datasets that prior models struggle to fit effectively, we find that Fit actually begins suffer from overfitting (read Section 4 and 5). To prevent the model from overfitting we use data augmentation. To the best of our knowledge, this is the first use of augmentation in video prediction, perhaps because prior sate-of-the-ard models tend underfit already and therefore would not benefit from it. Given the rich literature in image augmentation, we augment the videos using RandAugment (Cubuk et al., 2020). We randomize the augmentation per video but keep the randomization constant for frames of a single video. RandAugment substantially improves the overfitting, however not entirely, as it can be seen in Figure 3. We improve the augmentation by selecting a random crop of the video before resizing it to the desired resolution at the training time, called RandCrop. The combination of RandCrop and RandAugment successfully prevents the overfitting, leading to models that both fit the training set and generalize well to held-out videos.
What FitVid does not need. Prior works on variational video prediction (Finn et al., 2016b; Babaeizadeh et al., 2018; Lee et al., 2018; Villegas et al., 2019; Wu et al., 2021a), generally require a range of additional design decisions for effective training. Common design parameters include using curriculum training, commonly by scheduled sampling (Bengio al., 2015), to mitigate distributional shift between training and generation time (Oh et al., 2015; Finn et al., 201 Lee al., 2018; Cenzato et al., 2019); heuristically tuning 3 in L(x) to balance the prediction fitting the prior (Denton & Fergus, 2018) by annealing it over the course of training (Babaeizadeh et al., 2018; Lee et al., 2018) or learned priors (Denton & Fergus, 2018; Villegas al., 2019; Castrejon et al., 2019; Wu et al., 2021a). Each of these design choices introduces hyperparameters, tuning burden, and additional work when applying a model to a new task. FitVid does not require any of these details: we simply train Vid by optimizing [(x) using Adam (Kingma & Ba, 2014).
4 EXPERIMENTS
To evaluate FitVid, we test it on four different real-world datasets and compare its performance with prior state-of-the-as methods, with comparable parameter count, using four different metrics. Our main goal is demonstrate that FirVid can in fact fit these datasets well and illustrate how augmentation can prevent FitVid from overfitting, resulting in state-of-the-art prediction performance. Please visit the anonymized website sites .oogle.. hoom/view/fftte generated videos. The code to reproduce these experiments is available in Supplementary Material,
to see samples of
4.1 EXPERIMENTATION SETUP
Datasets: To test FitVid, we use four datasets that cover a variety of real-life scenarios. We use the Human3.6M, 6M dataset (Ionescu al., 2014), which consists of actors performing various actions in a room to study the structured motion prediction. We also use the KITTI dataset (Geiger et al., 2013b 3b) to evaluate VitVid's ability to handle partial observability and dynamic backgrounds. For
4

---

Under review as a conference paper at ICLR 2022
Table 1: The empirical comparison between FitVid (with 302M parameters), GHVAE (Wu et al., 2021a) (with 599M parameters) and SVG (Villegas et al., 2019) (with 298M parameters). To prevent FitVid from overfitting, where Vid achieved state-of-the-ar result while the red color highlights otherwise.
augmentation for Human3.66 3.6M and KITTI The green color highlights
RoboNet
FVDL
PSNR
SSIMT
LPIPS.
Human3.6M
FVD
PSNR1
SSIM?
LPIPS4
GHVAE
95.2
24.7
89.1
0.036
Skip Frame
SVG
123.2
23.9
87.8
0.060
FirVid (ours)
62.5
28.2
89.3
GHVAE
355.2
26.7
94.6
0.018
SVG
0.060
FilVid (ours)
154.7
36.2
97.9
0.012
KITTI
FVDJ
P
SSIMF
I.PISS.
Skip Frame
GUIVAE
552.9
15.8
51.2
0.286
SVG
1217.3
15.0
41,9
0.327
SVG
429.9
23.8
88.9
FiLVid (ours)
884.5
17.1
49.1
0.217
FitVid (ours)
385.9
27.1
95.1
0.026
both datasets, we followed the pre-processing and testing format proposed by Wu et al. (2021a) and Villegas et al. (2019), which predicts 25-frames conditioned the previous five at 64 resolution.
To evaluate FitVid in action-conditioned setting, we use RoboNet dataset (Dasari et al., 2019). This large dataset includes more than 15 million video frames from 7 different robotic arms pushing objects in different bins. It contains wide range of conditions, including different viewpoints, objects, tables, and lighting. Prior video prediction methods have a tendency to badly underfit on this dataset (Dasari et al., 2019). We follow the same setup as Wu et al. (2021a) and randomly select 256 videos for testing. Similar to Wu et al. (2021a), we train FitVid to predict next ten frames given two context frames well the ten future actions.
Finally, to compare FitVid to a wider range of prior work, we use the BAIR robot pushing dataset (Ebert et al., 2017), which is widely-used benchmark in the video prediction literature. We follow the evaluation protocol of Rakhimov et al. (2020), which predicts the next 16 frames given only one context frame and no actions. Given the high stochasticity of robotic arm movement in BAIR, and the variety of unseen objects in the test set, it is great benchmark for evaluating the model's s ability to generate diverse outputs.
Metrics:
We evaluate our method and prior models across four different metrics: Structural Similarity Index Measure (SSIM) (Wang et al., 2004), Peak Signal-to-noise Ratio (PSNR) (Huynh-Thu Thu & Ghanbari, 2008). Learned Perceptual Image Patch Similarity (LPIPS) (Zhang et al., 2018) and Fréchet Video Distance (FVD) (Unterthiner et al., 2018). FVD measures the overall visual quality and temporal coherence without reference to the ground truth video. PSNR, SSIM, and LPIPS measure pixel-wise similarity to the ground-truth with LPIPS most accurately representing human perceptual similarity. Given the stochastic nature of video prediction benchmarks, we follow the standard stochastic video prediction evaluation protocol (Babaeizadeh et al., 2018; Villegas et al., 2019; Wu et al., 2021a): we sample 100 future trajectories per video and pick the best one as the final score for PSNR, SSIM and LPIPS. For FVD, we use all 100 with a batch size of 256.
4.2 RESULTS
First, we compare FitVid GHVAE (Wu al., 2021a) and SVG (Villegas al., 2019). GHVAE is a hierarchical variational video prediction model trained in greedy manner. SVG is large-scale variational video prediction model with learned prior and minimal inductive bias. We chose these two baseline because they both investigated overfitting by scaling the model, and achieve state-of-the-
Comparisons:
results. However, SVG reported no overfitting even for their biggest model ((M= = 3, K = 5)) with 298M parameters (Villegas et al., 2019) while GHVAE (with 599M paramteres) reported "some overfitting" on smaller datasets (Wu et al., 202 la). At the same time, both of these models share similar architecture to FitVid.
Table contains the results of these experiments. As can be seen in this table, FitVid outperforms both SVG and GHVAE across all metries in Robonet and Human3.6. M. In KITTI, FitVid also consistently outperforms SVG while either improves or closely matches the performance of GHVAE which has more than twice as parameters. For qualitative results, Figures 4, 5 and 6.
Comparison to non-variational methods: To provide a comprehensive comparison with prior methods, including non-varian voriaaiiai models, test FitVid on BAIR dataset (Ebert et al., 2017). As can be seen in Table 2, FitVid outperforms most of the previous models in this setting while performing comparably to Video Transformer (Weissenborn et al., 2019) with 373M parameters.
5

---

Under review as a conference paper at ICLR 2022
5.0
1e-1
Effect of Augmentation
Table 2: Comparison between FitVid and different methods for video prediction on action-free BAIR dataset (Ebert et al., 2017).
Augmentation
Rand/ugment
4.5
RandCrop
Evoluation LPIPS
Randkugment + RandCrop
4.0
BAIR
FVDJ
3.5
SV2P (Babaeizadeh et al., 2018)
262.5
LVT (Rakhimov ct al., 2020)
125,8
3.0
SAVP (Lee et al., 2018)
116.4
2.50
DVD-GAN-FP (Clark al., 2019)
109.8
50000
100000
150000
200000
Training Iteration
VideoGPT (Yan et al., 2021)
103.3
TrIVD-GAN-FP (Luc et al., 2020)
103.3
Figure 3: Effecc of augmentation meth- ods overfitting on KITTI. Both Ran- dAugment and Random Crop improve the overfitting but the combination of two gets the best results.
cINNs (Dorkenwald et al., 2021)
99.3
CCVS (Moing et al., 2021)
99.0
Video Transformer (Weissenborn et al., 2019)
94.0
FitVid (ours)
93.6
5 ANALYSIS
In this section, take closer look at the results from Section 4, to analyse the consequences of overfitting and the effect of regularization on the current benchmarks.
On Human3.66 3.6 as video prediction benchmark: Human3. 3.6M (Ionescu al., 2014) is com- mon benchmark in video prediction literature (Finn et al., 2016a; Babaeizadeh et al., 2018; Villegas et al., 2017; Wang et al., 2019b; Villegas et al., 2019; Lin et al., 2020; Franceschi al., 2020; Guen & Thome, 2020; Wu et al., 2021a;c) which we also use to evaluate FitVid (Figure 6). At the first glance, it seems that the model generating extremely detailed and human-like motions conditioned on the given context pose. However, on closer inspection, we observe that the human subject in the predicted video is changing. In fact, FitVid replaces the unseen human subject into a training subject which is particularly evident from the clothing. Actually, we can find similar video clips from the training data for each one of the predicted videos (see Figure 6). These frames are not exactly the same, but they look notably similar. This observation indicates that:
1. The model can generalize to unseen frames, poses and subjects since the test context frames are new and unseen. FitVid correctly distinguished the subject from the background, detects the human pose and continues the video from there.
2. The model memorized the motion and the appearance of the training subjects. The model morphs the test human subject into a training one, and then plays a relevant video from the memory.
This means that FitVid fails to generalize to a new subject, while still generalizing to unseen frames. Given that the Human3. 6M has five training and two test subjects (Finn et al., 2016a; Villegas et al., 2017) this may not be surprising.
Nevertheless, this observation shows how the current low- resolution setup for Human3.66 3.6M is not suitable for large-scale video prediction. In fact, after this observation, traced the same behaviour in other video prediction literature and, unfortunately, it seems this is common and overlooked issue. For example, the same phenomena can be seen in Figure 6 from Franceschi al. (2020) that shows changing the test to a training subject by Struct-VRNN (Minderer et al., 2019) and the proposed method by Franceschi al. (2020) (note the changed shirt). For others examples, see Figure 7 of Villegas et al. (2019) and Figure 5 of Villegas et al. (2017), See Figure 15 for copy of these figures.
Overfitting and regularization: As mentioned in Section 1, there is considerable evidence that current video prediction models tend to underfit when trained on large datasets (Villegas et al., 2019; Castrejon al., 2019; Wu et al., 2021a). Wu al. (2021a), which is the current state-of-the-e tf-the-art model with 599 million parameters, reported "some overfitting" on smaller datasets such as Human3.66 3.6M and KITTI. However, we observe severe and clear overfitting with FitVid, despite having only 302 million parameters. Figure 7 visualizes the training and evaluation LPIPS metric while training FitVid on Human3.6M. 6M, without augmentation. This graph demonstrates that the training keeps getting better while the test quality starts to get worse after 155 iterations. We also observed similar behaviour on KITTI, as can be seen in Figure 7b. These results clearly shows that FitVid is overfitting on Human 6M and KITTI, indicating that FitVid is using its parameters efficiently. As mentioned

---

Under review as a conference paper at ICLR 2022
1st Context Fraie t=1
(=4
t=7
1=10
1st Context Frame
t=
(=4
t=7
10
Trush
Predecsd
Figure 4: FitVid on action-conditione RoboNet (Dasari et al., 2019). The model is conditioned on the first two frames and predicting the next frames given the future actions of the robotic arm. These figures demonstrate how the predicted movements of the arm closely follows the ground truth given that the future actions is known. The model also predicts detailed movements of pushed objects (visible in the left example) as well as filling in the previously unseen background with some random objects (look at the object that appear behind the robotic arm in the right). Also, to demonstrate failure case, notice the wrong predictions of robot's fingers in the right example. See Figure for more frames from these video samples.
25
Ground
Figure 5: FitVid on KITTI dataset (Geiger et al., 2013b). As it can be seen in this figure, the model generates high quality prediction of the future in dynamic scene. Note how in the top example FitVid keeps predicting the movement of the shadow on the ground till it gets out of the frame. After that, the model still brings the background closer in each frame, implying driving forward. We noticed that the quality of predictions drop substantially faster when there are more objects the scene e.g. the driving scenes inside a city as can be seen in the right example. This indicates the model still fails to generalize to more complex scenes with more moving subjects, See Figure 12 for more frames.
Frame
t-7
t-11
1st Context
Ground Trush
Predcted
Clocst
Figure 6: FitVid Human3.6N, (Ionescu al., 2014). This figures demonstrates extremely detailed and human-like motions predicted by FitVid, conditioned on the given context frames. However, closer inspection,
be seen that the human subject in the video is changing, from the test subject to training subject. This is particularly evident from the cloths. This phenomena indicates that, although Fit Vid is capable of generalizing the frames out of training distribution, however, morphs the human subject into familiar one from the training set and then plays the video from the memory. In fact, we can find similar videos in the training set as visualized in the last row. The highlighted frame is the one used for finding the closest training video. Check Figure 13 for more predicted frames.
More videos can be found at sites
7

---

Under review as a conference paper at ICLR 2022
Human3.6M
KITTI
Robonet
Robonet
4.0
1c-
1e-1
1.0
1e-1 (scaled mode
3.5
Eval
0.9
3.0
Train
0.8
2.5
0.7
Eval Train
Eval
Eval
2.0
Train
0.6
Train
1.5
2-
1.0
0.5
0.3
08.00
0.50 1.00
8.0
0.0
1.0
2.0
Training Iteration 1e5
Training g Iteration le5
Training Iteration 1e5 (c)
Training Reeation 1e5
(b)
(b)
[d)
Figure 7: Overfitting of FitVid without augmentation. This figure visualizes the training and evaluation metrics on (a) Human3.66 6. (Ionescu et al., 2014), (b) KITTI (Geiger et al., 2013b) and (c) Robonet (Dasari al., 2019), without augmentation. As it can be seen, in all cases, FitVid overfits on the training data except for Robonet. This is evident from the evaluation measurement going up while the training keeps decreasing. In case of Robonet, FitVid with 302M parameters did not overfit but a scaled version of the model with 600M parameters did, as can be seen in (d). y-axis is LPIPS. r-axii the training iteration. The plots are smoothed with an average rolling window of size ten. The shadows are the raw non-smoothed values.
in Section 3.2, to address overfitting, quality, as reported in Table 1.
augmentation which results in ofateoofttheaar prediction
Overfitting on Robonet:
We did not observe any overfitting RoboNet, which is expected given the fact that RoboNet is much larger compared to the other benchmarks. Trying to find a model that can overfit on RoboNet, we test a scaled version of FitVid with 500M parameters which still smaller compared to GHVAE with 599M parameters and reported no overfitting on this dataset. This scaled version of Vid overfits on RoboNet, as demonstrated in Figure 7d. Note that we did not use this scaled version in the reported numbers of Table 1, which is generated using the 302M version. Our goal here was demonstrate that a scaled version of Fit can also use its parameters more efficiently, compared to prior models, leading to overfitting on even bigger datasets such as RoboNet.
Effect of Augmentation on SVG: There dis- crepancy between the input data for training the models in Section 4. FitVid is trained with aug- mentation while the baselines are trained without any augmentation which raises a question; can the better performance of FitVid be explained only by the augmentation? In other words, do the previous methods benefit from augmentation too? To answer this question, we retrain SVG with and without augmentation. As demonstrated in Table 3, SVG performs worse if trained with augmented data, sup- porting the claim that it is underfitting to the raw data. As a result, this experiment provides more support for Fit truly overfitting on these datasets and therefore benefiting from augmentation. Please note that we included our SVG results without augmentation too, as we could not perfectly reproduce the numbers reported by Villegas et al. (2019).
Table 3: SVG (Villegas et al., 2019) with and without augmentation. This table shows that SVG does not benefit from augmentation be- cause underfitting to the original data (Vil- legas et al., 2017).
Human3.6M
FVD1
PSNR1
SSIMT
LPIPS
Without With
389.55
27.4
93.7
0.041
429.25
23.0
0.094
KITTI
FVDJ
PSNR1
SSIMT
LPIPS
Without
1612.62
14.8
38.7
0.330
With
2051.67
14.4
0.333
Ablation Study In order to figure out how much each design element in FitVid is contributing to its performamce, we ablate it and check its perfor- mance on RoboNet. In this study we try to cover a variety of ablations including encoder/decoder ab- lations, adding or removing modeling features such as learned prior as well various skip connection implementations. As can be seen in Table 4, skip
Table 4:
Ablation study of FitVid on RoboNet. This table demonstrates how much each design choice in Vid contributes to its performance.
I.PPSS
PSNR1
SSIM
Baseline (Fi(Vid)
0.024
28.2
0.89
No Skip Connections
0.088
22.3
0.75
Nom residual Connections
0.023
28.4
0.90
Deterministic
0.025
0.90
connections are crucial component for archiving high prediction performance. This table also demon- strates that, while modeling sotchasticity import- nat even in action conditioned settings, Fit does
With Leamed Prior
0.024
28.2
0.90
ReL. activation
0.062
23,2
0.81
No Batch Norm
0.771
6.0
0.00
No Squeeze and Excite
0.124
19.4
0.68
not benefit from a learned prior. For more ablation studies, please look at Section A.
8

---

Under review as a conference paper at ICLR 2022
Zero-shot Real Robot Performance: Prior work indicate that improved video prediction translates to better performance in the down- stream tasks (Wu et al., 2021a; Babaeizadeh al., 2020). However, in these works, the train- ing and test distribution are the same and there is almost domain shift from training to testing. In this section, we are interested in investigat- ing whether Vid is capable of generalizing to similar but visually different task with no training data for this new domain. Therefore, we setup a real-robot experiment, with Franka Emika Panda robot arm, in which the goal is to push specific object to a predetermined goal position. We train Vid on RoboNet and use cross-entropy method (CEM) (Rubinstein, 1997; Chua et al., 2018) for planning (please see Ap- pendix for details). As can be seen in Table 5, this agent is unable to generalize to the new domain, achieving worse performance than random agent. This may not be surprising given the fact that the videos in RoboNet have entirely different robots and visuals, although the robots are performing the same task (i.e. pushing objects in a bin using a robotic arm)
Table 5: Zero-shot real robot performance. We use FitVid for planning future actions of real robot pushing an object to goal location with no training data from our setup. We train the model on visually different data (RoboNet) and the data from closer domain (from Wu et al. (2021a)) with and without augmentation. While unable to directly adapt from RoboNet to the new domain, the results illustrate that fine-tuning on similar data and augmentation improve FitVid's performance.
Training Data
Success Rate
Baseline (random actions)
28%
RoboNet
17%
RohoNet Wu al. (2021a)
56%
RoboNet+ + Augmented Wu et al. (2021a)
78%
We then try to bring the training and test domain closer to each other by fine-tuning FitVid on the data from Wu et al. (2021a). This data contains 5000 autonomously collected videos of a Franka Emika Panda robot arm pushing objects around which look more similar to our setup compared to RoboNet, but still contain different lighting, camera angle, and target objects. This time, we observe that FitVid is relatively successful at generalizing to the new domain, succeeding in 56% of the trials. Finally, we find that adding data augmentation to the fine-tuning improves the generalization ability of the model, achieving 78% success rate. These results illustrate that while large distribution shift adaptation (RoboNet) remains difficult, by using data augmentation Fit is capable of adapting to relatively new domain (from Wu et al. (2021a) a) data).
6 CONCLUSION
We propose FitVid, a simple and scalable variational video prediction model that can attain a significantly better fit to current video prediction datasets even with a similar parameter count as prior models. In fact, while prior methods generally suffer from underfitting applying FitVid actually results in overfitting. We therefore propose using a set of existing image augmentation techniques that prevent overfitting in video prediction settings, leading to state-of-the-art results across a range of prediction benchmarks.
these datasets, naïvely
To the best of our knowledge, this is the first time model reports, not only not underfitting, but rather substantial overfitting on these benchmarks. This is particularly important because underfitting is usually cited one the main reasons for low quality predictions of the future frames. We demonstrate that simple existing image augmentation techniques can be used to prevent the mode from overfitting, resulting high quality images. In fact, while each of the components of our proposed model are not novel on their own, Vid outperforms the current tftte-of-the-all art models across four different video prediction benchmarks on four different metrics. Moreover, the fit attained by Vid is so good that a number of previously used benchmarks and metrics can be fooled, resulting in undesired outcomes, which are often overlooked the video prediction literature.
There are many ways that FitVid can be expanded. As mentioned in the text, one of the interesting features of our proposed method is that it is simple. It non-hierarchical, convolutional, with no attention mechanism, no curriculum learning, and no training scheduling, Any of these features can potentially improve the results of Vid in order to generate even higher quality images. Given the simplicity of Vid, it can be casily built upon. Another interesting direction would be to introduce new training- aware metrics for video prediction and generation to signal when a model is generating high quality videos by repeating the training data.
9

---

Under review as a conference paper at ICLR 2022
ETHICS STATEMENT
Videos are an abundant source of visual information about our physical world. They contain information about objects, humans and how they interact with each other. The goal of video prediction is to foremost learn representation of the world, usable for downstream tasks by an agent. Second, is to predict what happens next, conditioned on the past and the future intents, which can be used for planning. Despite many recent advances in this field, the present day models are still relatively low-quality and limited narrow domains which makes their applications limited. However, if improved, video prediction or representations learned by video prediction, can be major step forward toward fully autonomous self-learning agents. This paper, we believe, takes an important step towards this goal by pushing tfte-of-thhearr forward and simplifying it in meaningful way. However, our model is vulnerable to the bias in the training data and if adopted widely, this can skew research in certain directions. For example, our results may lead higher quality models which can be scaled to generate even higher quality results. These models will be harder to design and train and require more computational power, and potentially can be biased. Finally, the underlying techniques for video prediction can be misused for generating nigh-quallin videos that are misleading, depicting deliberately false situationn or persons, similar to the phenomenon of deepfakes (Korshunov & Marcel, 2018). However, we believe that our experiments are conducted in relatively specific narrow settings and conditions which will likely not generalize broadly. Specifically, it is especially challenging to generate high-quality realistic-looking videos even with such state of the art methods.
REPRODUCIBILITY STATEMENT
The anonymized implementation of the model, as well as the code train and evaluate FitVid is attached in the Supplementary Material. This code can be used to regenerate the results reported in this paper. The approximated computation cost of rerumning these experiments is available in Section 1. Appendix also includes details of implementation such as the training and evaluation details in Section A.2 as well as augmentation details in Algorithm 3. The detailed architecture of the encoder networks can be found in Table 9, decoder in Table 11, and the dynamic model in Table 10 of the Appendix. The hyper-parameters used are reported in Table 8. Table 13 includes the license of the benchmark datasets.
REFERENCES
Mohammad Babaeizadeh, Chelsea Finn, Dumitru Erhan, Roy H Campbell, and Sergey Levine. Stochastic variational video prediction. ICI.R, 2018.
Mohammad Babaeizadeh, Mohammad Taghi Saffar, Danijar Hafner, Harini Kannan, Chelsea Finn, Sergey Levine, and Dumitru Erhan. Models, pixels, and rewards: Evaluating design trade-off's in visual model-based reinforcement learning. arXiv preprint arXiv:2012.04600 2020.
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. arXiv preprint 1eXiv:150.00009) 2015.
Byron Boots, Arunkumar Byravan, and Dieter Fox. Learning predictive models of a depth camera & manipulator from raw execution traces. In 2014 IEEE International Conference on Robutics and Automation (ICRA). pp. 4021-4028, 2014.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman- MMilee,, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URI. http: /tstt::/ggttttii com/ googl e jax.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Gretehen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165, 2020.
Andreja Bubic, D Yves Von Cramon, and Ricarda I Schubotz. Prediction, cognition and the brain. Frontiers in human neuroscience, 4:25, 2010.
10

---

Under review as a conference paper at ICLR 2022
Arunkumar Byravan and Dieter Fox. Se3-nets: Learning rigid body motion using deep neural networks. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 173-180, 2017.
Lluis Castrejon, Nicolas Ballas, and Aaron Courville. Improved conditional vmnss for video prediction. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pp. 7608-7617, 2019.
Alberto Cenzato, Alberto Testolin, and Marco Zorzi. On difficulty of learning and predicting the long-term dynamics of bouncing objects. arXiv preprint 19X.v:197..1344 2019.
Baoyang Chen, Wenmin Wang, and Jinzhuo Wang. Video imagination from a single image with transformation generation. In Proceedings of the Thematic Workshops of ACM Multimedia 2017, pp. 358- 366, 2017.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems, pp. 4754 4765, 2018.
Aidan Clark, Jeff Donahue, and Karen Simonyan. Adversarial video generation complex datasets. arXiv preprint 1007::999000066 2019.
Ekin Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 702-703, 2020.
Sudeep Dasari, Frederik Ehert, Stephen Tian, Suraj Nair, Bernadette Bucher, Karl Schmeckpeper, Siddharth Singh, Sergey Levine, and Chelsea Finn. Robonet: Large-scale multi-robot learning. In CoRI., 2019.
Emily Denton and Rob Fergus. Stochastic video generation with a learned prior. arXiv preprint arXiv: 1802 1802.07687, 2018.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint wrXiv:1810.04805, 2018.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. arXiv preprint arXiv:1410.88. 2014.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint 1605vv0605.08000 2016.
Michael Dorkenwald, Timo Milbich, Andreas Blattmann, Robin Rombach, Konstantinos G Derpanis, and Bjom Ommer. Stochastic image-to-video synthesis using cinns. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3742 3753, 2021.
Lingyu Duan, Jiaying Liu, Wenhan Yang, Tiejun Huang, and Wen Gao. Video coding for machines: A paradigm of collaborative compression and intelligent analytics. IEEE Transactions on Image Processing, 29: 8680-8695, 2020.
Frederik Ebert, Chelsea Finn, Alex Lee, and Sergey Levine. Self-Supervised Visual Planning with Temporal Skip Connections. Conference on Robot Learning (CoRL), 2017.
Frederik Ebert, Chelsea Finn, Alex X Lee, and Sergey Levine. Self-supervised visual planning with temporal skip connections. In Conference Robot Learning (CoRL), 2017.
Frederik Ebert, Sudeep Dasari, Alex X Lee, Sergey Levine, and Chelsea Finn. Robustness via retrying: Closed-loop robotic manipulation with self-supervised learning. In Conference on Robot Learning (CoRL), 2018a.
Frederik Ebert, Chelsea Finn, Sudeop Dasari, Annic Xic, Alex Lec, and Sergey Levine. Visual foresight: Model-based deep reinforcement learning for vision-based robotic control. arXiv preprint arXiv:1812.000 2018b.
Stefan Elfwing, Eiji Uchibe, and Kenji Doya. Sigmoid-weighted linear units for neural network function approximation in reinforcement learning, Neural Networks, 107:3-11, 2018
Yuki Endo, Yoshihiro Kanamori, and Shigeru Kuriyama. Animating landscape: self sepf-supe learning of decoupled motion and appearance for single-image video synthesis. arXiv preprint arXiv: 1910.0719,, 2019.
Chelsea Finn and Sergey Levine. Deep visual foresight for planning robot motion. In 20/7 IEFF International Conference on Robotics and Automation (ICRA), pp. 2786-2793. IEFF, 2017.
11

---

Under review as a conference paper at ICLR 2022
Chelsea Finn, Ian Goodfellow, and Sergey Levine. Unsupervised learning for physical interaction through video prediction. Advances in neural information processing systems, pp. 64-72, 201
Chelsea Finn, Xin Yu Tan, Yan Duan, Trevor Darrell, Sergey Levine, and Pieter Abbeel. Deep spatial autoen- coders visuomotor learning. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pp. 512-519. IEFF, 2016b.
Jean-Y ves Franceschi, Edouard Delasalles, Mickaël Chen, Sylvain Lamprier, and Patrick Gallinari. Stochastic latent residual video prediction. arXiv preprint 200. pp. 3233-3246, 2020.
Andreas Geiger, Philip Lenz. Christoph Stiller. and Raquel Urtasun. Vision meets robotics: The kitti dataset. International Journal Robotics Research (IJRR). 2013a.
Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets roboties: The kitti dataset. The International Journal of Robotics Research, 32(11)::221-133) 1237,, 2013b.
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT press Cambridge, 2016.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. arXiv preprint arXiv: 406466661, 2014.
Vincent Le Guen and Nicolas Thome. Disentangling physical dynamics from unknown factors for unsupervised video prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11474-1 1484, 2020.
Danijar Hafner, Timothy Lillicrap. Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. arXiv preprint 1e.iv::811.005555 2018.
Danijar Hafner, Timothy Lillicrap. Jimmy Ba, and Mohammad Norouzi. Dream control: Leamnngg behaviors by latent imagination, arXiv preprint arXiv:1912.0166) 2019.
Jonathan Heek, Anselm Levskaya, Avital Oliver, Marvin Ritter, Bertrand Rondepierre, Andreas Steiner, and Mare van Zec. Flax: A neural network library and ecosystem for JAX, 2020. URL httttt/p//gghttt com/ google/ flax.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint 1066:00606 2016.
Irina Higgins, Loic Matthey, Xavier Glorot, Arka Pal, Benigno Uria, Charles Blundell, Shakir Mohamed, and Alexander Lerehner. Early visual concept learning with unsupervised deep learning. arXiv preprint 1rXivv1606.0553 2016a.
Irina Higgins, Loic Matthey, Arka Pal. Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. International Conference on Learning Representations (ICLR), 2016b.
Sepp Hochreiter and Jürgen Schmidhuher. Long short-term memory. Neural computation, 133-173--) 1997.
Kibeom Hong, Youngjung Uh, and Hyeran Byun. Arrowgan: Learning to generate videos by learning arrow of time. arXiv preprint arXiv:2101 100.:1000033010 2021.
Jie Hu, Li Shen, and Gang Sun. Squeeze and eeddeexiiaiion networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1132-7141, 2018.
Qiyang Hu, Adrian Wälchli, Tiziano Portenier, Matthias Zwicker, and Paolo Favaro. Learning to take directions one step at a time. In 2020 25th International Conference on Pattern Recognition (ICPR), pp. 739-746. IEBE, 2021.
Quan Huynh-Thu and Mohammed Ghanbari. Scope of validity of psnr in image/video quality assessment. Electronics letters, 2008
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448 456. PMLR, 2015.
Catalin lonescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3. 6m: Large scale datasets and predictive methods for 3d human sensing in natural environments. IEFE transactions on pattern analysis and machine intelligence, 36(7), 2014.
12

---

Under review as a conference paper at ICLR 2022
Xu Jia, Bert De Brabandere, Tinne Tuytelaars, and Lue V Gool. Dynamic filter networks. In Advances in neural information processing systems, pp. 667-675, 2016.
Michael I Jordan, Zoubin Ghahramani, Tommi S Jaakknla, and Lawrence K Saul, An introduction to variational methods for graphical models. Machine learning, 37(2):183-233, 1999.
Eukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blažej Osiński, Roy H Camphell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, al. Model based reinforcement learning for atari. In International Conference on Learning Representations. 2019.
Nal Kalchbrenner, Aäron Oord, Karen Simonyan, Ivo Danihelka, Oriol Vinyals, Alex Graves, and Koray Kavukcuoglu. Video pixel networks. In International Conference on Machine Learning, pp. 1771-1779, 2017a.
Nal Kalchbrenner. Aäron van den Oord, Karen Simonyan, Ivo Danihelka, Oriol Vinyals, Alex Graves, and Koray Kavukcuoglu. Video pixel networks. International Conference on Machine Learning (ICML). 2017b.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv: 10196, 2017.
Seung Wook Kim, Yuhao Zhou, Jonah Philion, Antonio Torralba, and Sanja Fidler. Learning to simulate dynamic environments with gamegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1231 1240, 2020.
Scung Wook Kim, Jonah Philion, Antonio Torralba, and Sanja Fidler. Drivegan: Towards controllable high-quality neural simulation, 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412: 1412.6980, 2014.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. International Conference on Learning Representations (ICLR), 2014.
Pavel Korshunov and Sébastien Marcel. Deepfakes: a new threat to face recognition? assessment and detection. arXiv preprint arXiv: 1e1.. 2018.
Manoj Kumar, Mohammad Babaeizadeh, Dumitru Erhan, Chelsea Finn, Sergey Levine, Laurent Dinh, and Durk Kingma. Videoflow: A flow-based generative model for video. arXiv preprint arXiv:1903.01434 2019.
Alex X Lee, Richard Zhang. Frederik Ebert, Pieter Abbeel, Chelsea Finn, and Sergey Levine. Stochastic adversarial video prediction. arXiv preprint / 2018.
Wonkwang Lee, Whie Jung, Han Zhang, Ting Chen, Jing Yu Koh, Thomas Huang. Hyungsuk Yoon, Honglak Lee, and Seunghoon Hong. Revisiting hierarchical approach for persistent long-term video prediction. arXiv preprint arXiv:2104 146667,, 2021.
Xiaodan Liang, Lisa Lee, Wei Dai, and Eric Xing. Dual motion gan for future-flow embedded video prediction. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1744-1752, 2017.
Xue Lin, Qi Zou, Xixia Xu, Yaping Huang, and Tian. Motion-aware feature enhancement network for video prediction. IEEE Transactions on Circuits and Systems for Video Technology, 2020.
Ziwei Liu, Raymond A Ych, Xiaoou Tang, Yiming Liu, and Aseem Agarwala. Vidco frame synthesis using deep voxel flow. In Proceedings of the IEEE International Conference on Computer Vision, pp. 4463 4471, 2017.
Chaochao Lu, Michacl Hirsch, and Bernhard Scholkopf. Flexible spatio-temporal networks for video prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 6523-6531, 2017.
Pauline Luc, Aidan Clark, Sander Dieleman, Diego de Las Casas, Yotam Doron, Albin Cassirer, and Karen Simonyan. Transformation- based adversarial video prediction on large-scale data. arXiv preprint arXiv:2003.00 04035, 2020.
Michael Mathieu, Camille Couprie, and Yann LeCun. Deep multi-scale video prediction beyond mean square error. arXiv preprint arXiv:/S11.05:0 2015.
Matthias Minderer. Chen Sun, Ruben Villegas. Forrester Cole, Kevin Murphy, and Honglak Lee. Unsupervised learning of object structure and dynamies from vidoos. arXiv preprint arXiv: 0068078899,... 2019.
Guillaume Le Moing, Jean Ponce, and Cordelia Schmid. Cevs: Context-aware controllable video synthesis. arXiv preprint N0t.:2007.0003: 2021.
13

---

Under review as a conference paper at ICLR 2022
Suraj Nair and Chelsea Finn. Hierarchical foresight: Self-supervised learning of long-horizon tasks via visual subgoal generation. ICLR, 2020.
Suraj Nair, Mohammad Babacizadch, Chelsea Finn, Sergey Levine, and Vikash Kumar. Trass: Time reversal self-supervision. In 2020 IEEF International Conference on Robotics and Automation (ICRA), pp. 115-121, 2020.
Seonghyeon Nam, Chongyang Ma, Menglei Chai, William Brendel, Ning Xu, and Seon Joo Kim. End-to-end time-lapse video synthesis from a single outdoor image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1409 1418, 2019.
Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard I. Lewis, and Satinder Singh. Action-conditional video prediction using deep networks atari games. In Advances neural information processing systems, pp. 2863-2871, 2015.
Sergiu Oprea, Pablo Martinez- Gonzalezz Alberto Garcia, John Alejandro Castro- Castro-Vargas, Sergio Orts- Escolano, Jose Rodriigez and Antonis Argyros. A review on deep learning techniques for video prediction. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
Chris Paxton, Yotam Barny,, Kapil Katyal, Raman Arora, and Gregory D Hager. Visual rohot task planning. In 2019 International Conference Robotics and Automation (ICRA), pp. 8832-8838, 2019.
Rafael Rafailov, Tianhe Yu, Aravind Rajeswaran, and Chelsea Finn. Offline reinforcement learning from images with latent space models. arXiv preprint ar%iv 2020.
Ruslan Rakhimov, Denis Volkhonskiy, Alexey Artemov, Denis Zorin, and Evgeny Burnaev. Latent vidco transformer. arXiv preprint arXiv:2006. 107044 2020.
Prajit Ramachandran, Barret Zoph, and Quae Le.. Searching for activation functions. arXiv preprint arXiv:1710.05941,, 2017.
MarcAurelio Ranzato, Arthur Szlam. Joan Bruna, Michael Mathieu, Ronan Collobert, and Sumit Chopra. Video (language) modeling: baseline for generative models of natural vidcos. arXiv preprint arXiv: 2014.
Scott F. Reed, Aäron van den Oord, Nal Kalchhrenner, Sergio Gomez Colmenarejo, Ziyu Wang, Yutian Chen, Dan Belov, and Nando de Freitas. Parallel multiscale autoregressive density estimation. International Conference on Machine Learning (ICML), 2017.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International conference on machine learning, pp. 1278 1286. PMLR, 2014.
Reuven Rubinstein. Optimization of computer simulation models with rare events. European Journal of Operational Research, 99( 111212 1997.
Masaki Saito and Shunta Saito. Tganv2: Efficient training of large models for video generation with multiple subsampling layers. 2018.
Vaibhav Saxena, Jimmy Ba, and Danijar Hafner. Clockwork variational autocncoders for vidco prediction. arXiv preprint 102.52210..09533 2021.
Rui Shu, James Brofos, Frank Zhang, Hung Hai Bui, Mohammad Ghavamzadeh, and Mykel Kochender- fer. Stochastic video prediction with conditional density estimation. In ECCV Workshop on Action and Anticipation for Visual Learning, volume 2, 2016.
Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov. Unsupervised leamning of video representations using Istms. In International Conference on Machine Learning, 2015.
Serkan Sulun and Murat Tekalp. Can learned frame prediction compete with block motion compensation for video coding? Signal, Image and Video Processing, 15(2):401440, 2021.
Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Syl- vain Gelly. Towards accurate generative models of video: A new metrie & challenges. arXiv preprint 2018.
Arash Vahdat and Jan Kautz. Nvae: A deep hierarchical variational autoencoder. arXiv preprint arX2v22007.03888, 2020.
14

---

Under review as a conference paper at ICLR 2022
Joost Van Amersfoort, Anitha Kannan, Marc' Aurelio Ranzato, Arthur Szlam, Du Tran, and Soumith Chintala. Transformation- basondd---ddd-eee models of vidco sequences. arXiv preprint arXiv:170..08433 2017.
Ruben Villegas, Jimei Yang, Yuliang 7ou, Sungryull Sohn, Xunyu I Lin, and Honglak I Lee. Learning to generate long-term future via hierarchical prediction. In international conference on machine learning, pp. 3560-3569.
PMLR, 2017.
Ruben Villegas, Arkanath Pathak, Harini Kannan, Dumitru Erhan, Quoe V Le, and Honglak Lee. High fidelity video prediction with large stochastic recurrent neural networks. In Advances in Neural Information Processing Systems, pp. 81-91, 2019.
Carl Vondrick and Antonio Torralba. Generating the future with adversarial transformers. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1020- 1028, 2017.
Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dynamics. arXiv preprint 2016.
Jacob Walker, Abhinav Gupta, and Martial Hebert. Dease optical flow prediction from static image. In Proceedings of the IEEF International Conference on Computer Vision, pp. 2443-2451, 20155
Jacob Walker, Carl Doersch, Abhinav Gupta, and Martial Hebert. An uncertain future: Forecasting from static images using variational autoencoders. In European Conference Computer Vision, pp. 835-851. Springer, 2016.
Jacob Walker, Ali Razavi, and Aäron van den Oord. Predicting vidco with vqvae. wrX1v:2033.01900 2021.
arXiv preprint
Ting-Chun Wang, Ming-Yu Yu Liu, Jun-Yan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. Video-to-video synthesis. arXiv preprini 1808006681, 2018.
Ting-Chun Wang. Ming-Yo Yu Liu, Andrew Tao, Guilin Liu, Jan Kautz, and Bryan Catanzaro. Few-shot video-to- video synthesis. arXiv preprint arXiv: 1910. 2019a.
Yunbo Wang, Jianjin Zhang, Hongyu Zhu Mingsheng Long, Jianmin Wang, and Philip Yu. Memory in memory: A predictive neural network for learning higher-order non -on-stationariity from spatiotemporal dynamics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9154 9162, 2019b.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from emor visibility to structural similarity. IEEE transactions on image processing, 2004.
Dirk Weissenbom, Oscar Täckström, and Jakob Uszkoreit, Scaling autoregressive vidco models, arXiv preprint arXiv: 1et.:/9090024444 2019.
Nevan Wichers, Ruben Villegas. Dumitru Erhan, and Honglak Lee. Hierarchical long-term video prediction without supervision. International Conference on Machine Learning (ICML), 2018.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's 's transformers: State-of-the-art natural language processing. arXiv preprint 1rX00:1910003371,, 2019.
Bohan Wu, Suraj Nair, Roberto Martin-Martin. Li Fei-Fei, and Chelsea Finn. Greedy hicrarchical variational autocncoders for large-scalc video prediction. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 2328-23,8, 2021a.
Chenfei Wu, Lun Huang, Qianxi Zhang, Binyang Li, Lei Ji, Fan Yang. Guillermo Sapiro, and Nan Duan. Godiva: Generating open-domain videos from natural descriptions, 2021b.
Haixu Wu, Zhiyu Yao, Mingsheng Long, and Jianmin Wan. Motionmn: A flexible model for video prediction with spacetime-varying motions. arXiv preprint 100.02203.224, 2021c.
Annic Xie, Frederik Ebert, Sergey Levine, and Chelsea Finn. Improvisation through physical understanding: Using novel objects tools with visual foresight. Robotics: Science and Systems (RSS), 2019.
Tianfan Xue, Jiajun Wu, Katherine Bouman, and Bill Freeman. Visual dynamics: Probabilistic future frame synthesis via cross convolutional networks. In Advances in neural information processing systems, pp. 91-99, 2016.
Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using va- and transformers. arXiv preprint arXiv:2 104.00577 2021.
15

---

Under review as a conference paper at ICLR 2022
Fu-En Yang, Jing-Cheng Chang, Yuan-Hlao Lee, and Yu-Chiang Frank Wang. Dual-migan: Stochastic and deterministic motion transfer for image-to-video synthesis. In 2020 25th International Conference on Pattern Recognition (ICPR), pp. 6764 6771. IEEE, 2021.
Marvin Zhang, Sharad Vikram, Laura Smith, Pieter Abbeel, Matthew Johnson. Sergey Levine. Solar: Deep structured representations for model-based reinforcement learning. In International Conference on Machine Learning, pp. 7444-7453. PMLR, 2019.
Richard Zhang. Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In CVPR, 2018.
16

---

Under review as a conference paper at ICLR 2022
A APPENDIX
A.1 MORE ABLATION STUDIES
In order to demonstrate how the design decisions behind FitVid's architecture affect its performance, we conduct ablation studies. In the first study, replace FitVid's encoder and decoder blocks with the blocks from SVG, while keeping everything else same, and report prediction performance on RoboNet Table From this table, it can seen that the new encoder and decoder blocks are much more efficient at modeling the vidcos, resulting in improved prediction metrics. Please note that the numbers reported for SVG are diffcrent from Table I, because used FitVid 's dynamic model in this study.
In another study. analyze the effect of the number of encoding and decoding blocks the performance of one to three and measure the prediction quality on RoboNet. As
the model. very number blocks it can be seen in Table 7, it's not just about making the architecture bigger, but that we are actually benefiting from the particular design of our encoder and decoder i.c. having exactly two encoding blocks and two decoding blocks is the optimal spot for achieving the best results in reasonable number of training iterations while maintaining the number of parameters the ballpark previous work.
A.2 IMPLEMENTATIO DETAILS
this section we describe the details FitVid's architecture as well as training. Algorithm and Algorithm describe the high-level training and prediction process respectively, Look at Table for the used hyper-parametor We used the same set of hyper-parameters across all experiments. As mentioned in Section 3.2, all of the hyper- parameters are fixed during the training and there is scheduling. Table 9 and Table 11 include the detailed architecture of the encoder and the decoder. Table 10 describes the structure of dynamic and posterior networks. Finally, Table describes the augmentations details.
A.2.1 COMPUTATION RESOURCES
We implement VitVid using Flax (Heek et al., 2020) library for JAX (Bradbury et al., 2018). We train FitVid on 4x4 TPUs (32 co-processors), Each training step (with global batch size of 128 or local batch size of 4) takes 1020 milliscconds (i.c. 0.98 step per second). In parallel, evaluate the model every 1000 training steps on single V100 GPU which takes about 850 milliseconds for batch size 128. The models trained for one million training iterations which takes ~ 12 days to complete.
Algorithm 1: VidVid training.
Algorithm 2: FitVid prediction.
Input: Number of context frames
Input: Context frames
Data: Training frames X and actions ag-r
Input:
All actions a0:77
/7 Encode all frames
Output:
for L <- 0 to T do
Sa f- 0 // initializa states
hr, C. +, Encoder(x.)
C + --ncoder(xx)) // Get last skips.
end
for f + to do
// Prediction
// Encode Erame he, C, <,, Encoder(x,)
Sp. Sa + 0,0 Iritialize states for < to T do
samp from prior
// Approximate posterior
2t ~ (O,
(Hi, (1 ). Posterior((h+41] 8p) ~ (.)
11 predict future state
h + Dynamic((hh, a., 2c). s.)
Predict fucure stase
11 becode frame
52 + Dynamic([h,, a, 24). sa)
R. <- Decoder(hh, C.)
end
end
// Decode all frames
10 for 4 to T do
// Use last availsble skip connection R Decoder(hic Cc)
11
12 end
// Opt imize ELBO
+ Xx x11 Der. Dx. <- Adamm (w, 2)
17

---

Under review as a conference paper at ICLR 2022
Table Effect FitVid's encoder and decoder on prediction performance. As it can seen this paper, by replacing SVG's encoder and decoder with FitVid's encoder and decoder, the prediction quality improves.
Encader
Decoder
FVDJ
IPIPS.
SVG
SVG
155.2
FirVid
SVG
113.1
0,025
76.9
FitVid
FIVid
64.5
0.018
Table 7: Effect of the number of encoding and decoding blocks on the performance of Fit As can be seen this table, more encoder and decoder blocks do not essentially result in better prediction quality and having two blocks in both encoder and decoder is a good balance between number of paramters and the performance.
Encoder
Decoder Blocks
FVDL
LPIPS.
Blocks
121.3
0.022
73.3
0.019
64.5
0,018
176.6
0.028
88.8
0.019
83.9
0.019
Table 8: Syper-paramettrs used training FitVid, We used the same set of hyper-parameters across all experiments. As mentioned in Section 3.2, all of the hyper-paramters are fixed during the training and there is no scheduling.
Hyper-parameter
Value
Optimizer (Adam(Kingmm Ba, 2014))
Learning Rate
le-3
Btach Size
128
0.9
0.999
Gradient Clipping (12)
100.0
Model
B
Lateot dimennsnn
10
Encoder (h) dimension
128
LSTM size
256
1st Context Frame t=1
5-5
10
t-
1st Context Frame
t=1
5=5
Z- 10
t- 15
Gound Truhh
Figure 8: FirVid on BAIR robot pushing dataset (Ebert et al., 2017) with no actions. The model is conditioned only on the first frame and predicting the 16 frames. Given that the future actions of the robotic arm is unknown, the prediction can diverge substantially from the ground truth video. However, the model predicts movements for the objects whenever the arm pushes the object in an imaginary scenario. It also fills the background with random objects.
18

---

Under review as a conference paper at ICLR 2022
Table 9: FitVid Encoder Architecture. We are using the same encoding cells as NVAE (Vahdat & Kautz, 2020). The strides are always x except when down-sampling which has strides of 2 (bn) is natch-normalization (loffe & Szegedy, 2015), (swish) is the activation (Ramachandran et al., 2017), (s&e) is Squeeze and Excite (Hu al., 2018). There is skip connection from the beginning of each cell to the end of it. In these skip connections, the number of input filters will be matched by the output using x1 convolution.
Cell
Inpui Size
Pre
Kermel
Filters
Post
Down Sampling
1-1
64x
bn swish
3x3
64
1-1
64
bn + swish
3x
64
s&c
1-2
61
bn swish
1-2
64x
64
bu swish
3x3
64
s&ee
Yes
32x
swish
2.1
32x32
ha swish
3x3
128
s&c
2-2
32x
bn + swish
3x3
2-2
32
bn swish
3x3
128
8&c
Yos
3-1
16x 16
bn + swish
3x3
256
16x
bn swish
3-2
16x 16
bu swish
3x3
256
swish
4-1
8
bn + swish
3x3
512
8x8
bu swish
3x3
s&c
4-2
8x8
bu swish
3X3
512
4-2
8x8
bn swish
3x3
512
s&c
Table 10: FitVid Dynamics Architecture. We are using a similar dynamics as Denton & Fergus (2018). The encoded output first averaged across spatial dimension and then decoded into h-size using fully connected layer. Then, the dynamics are modeled by two LSTM layers. Finally, the output mapped and reshaped to an image tensor before passing to the decoder. The posterior uses the exact same architecture except that only has one LSTM layer.
Cell
Input Size
Layer
Size
Post
8
spatial average flatten
dense
256
append a and
128+
LSTM
256
256
LSIM
256
256
dense
8x8x 512
sigmoid reshape
19

---

Under review as a conference paper at ICLR 2022
Table 11: FitVid Encoder Architecture. We are using the same encoding and decoding cells as NVAE (Vahdat & Kautz, 2020). The strides are always 1x1. 1. For up-sampling we use nearest neigh- bour. (bn) is natch-normalization (loffe & Szegedy, 2015), (swish) is the activation (Ramachandran et al., 2017), (s&e) is Squeeze and Excite (Hu et al., 2018). There is skip connection from the also skip connections from each encoder block to
beginning of each cell to the end of it. There the corresponding decoder block (look at Figure 2). In these skip connections, the number of input filters will be matched by the output using 1x1 convolution.
Cell
Inpur Size
Pre
Kernel
Filters
Post
Up Sampling
1-1
8
bn
1x1
2048
swish
1-1
8x8
ba swish
1x1
512
bu s&e
1-2
8x8
bn
1x1
2048
1-2
8x8
bn swish
5x5
2048
1-2
8x8
bn swish
1x1
512
bn + site
Yes
2-1
16x16
bn bn swish
1x1
1024
2.1
16x 16
5x
1024
2-1
16x16
bn + swish
1x1
s&t
2-2
16x16
bn bm swish
1x1
1024
2/2
16x16
5x5
1024
2-2
16x16
bn swish
1x1
256
sit
Yes
3-1
32x32
bn swish
1x1
512
5x5
3-1
32x32
bn swish
1x1
128
bn s&e
32x32
bn ba swish
1x1
512
3-2
32x32
5x5
512
3-2
32x32
ba + swish
1x1
128
+ site
Yes
4.1
64x64
bn ba swish
1x
256
4-1
61x61
5x5
256
swish
4-2
64x64
bn 1x1
256
4-2
61x64
ba swish
5x5
256
4-2
64x64
bm swish
1x1
sike
64x64
1x1
sigmoid
Table 12: To prevent FitVid from overfitting we use augmentation. First, at training time, we select a random crop of the video before resizing it to the desired resolution (64x64) at the training time, called RandCrop. This processes crops all the frames of a given video include a minimum of C' percent of the frame's 's height. Then we use RandAugment (Cubuk et al., 2020) to improve the augmentation. We use the same augmentation configuration for all the datasets. Per video, we use the same randomization across all the frames.
Hyper-parameter
Value
Algorithm 3: Video Augmentation.
RandAugment
Input: Video x
No. of transformations
Input: Number of RandAugment transformations N
Magnitude M
Input: RandAugmentt magnitude M
Transformations
Input: RandCrop crop height minimum ratio C'
+ RandCrop(xx C) x
identity - auto citlll_oontssst-t equalize sotate solarize- color posterize contrast brightness sharpness shear
for i <- to N do TanooseRRaddmmTaanffoomnnmm
x f(x,
translate_ x translate_y
end
return
RandCrop
Crop height minimum ratio
0.8
Table 13: Used datasets and their licenses.
Dataset
Reference
License
RoboNet
Dasari et al. (2019)
MIT
KITTI
Geiger et (2013a)
Creative Commons
1.6man3.6M BAIR robot pushing
lonescu et al. (2014)
License
Ebert et al. (2017)
GitHub Default
20

---

Under review as a conference paper at ICLR 2022
Real Robot Domain
Kitchen Items
Cleaning Tools
Office Supplies
Figure 9: Zero-shot robot domain. We evaluate FitVid planning tasks Franka Emika Panda involving kitchen, cleaning, and office items. We did not collect any training data for this task. Instead the model is trained RoboNet and fine-tuned augmented data from Wu et al. (2021a).
A.3 ROBOT EXPERIMENT DETAILS
A.3.1 ENVIRONMENT
The rohot environment consists of Franka Emika Panda robot operating over hin which contains various objects. The robot's 's observations are 64 x RGB images, and its action space consists DOF delta position control of the end effector with action magnitudes in the range [-10cm,10cm)
A.3.2 DATA
The data used for finetuning the model used in robot experiments was taken directly from Wu et al. (2021a). This data consists of different viewpoint, lighting conditions, and target objects than what is used in evaluation.
A.3.3 EVALUATION DETAILS
During evaluation, agent is specified to complete an object pushing task by a goal image. The agent has S0 timesteps to complete the task, and trial measured as successful if the majority the object overlaps with its target position at some point in the episode. Each method is evaluated over 18 trials, of which consist of objects "office" setting, consist of objects in "kitchen" setting, and 6 consist of objects in "cleaning" setting (See Figure 9).
execute task, the agent performs visual model predictive control using the cross entropy method (CEM). Specifically, the agent takes frame, and predicts trajectorics of 10 time-steps for 200 different sampled action sequences. Trajectories are ranked according to their negative mean squared error to the goal image, averaged across all 10 time-steps. The action distribution refits the top 20 actions, and repeats for 3 iterations of CEM. Afterwards the best sequence of 10 actions is stepped in the environment in open loop faction. The process repeats 5 times until the end of the episode.
21

---

Under review as a conference paper at ICLR 2022
Initial State
Goal
State
Push tape to tape dispenser
Push brush towards sponge
Push donut next to cupcake
Figure 10: Example tasks for zero-shot object pushing using robotic arm. The goal in each trial is to push the a specific object to predetermined goal location. The trial is considered successful, if the robot pushes least half of the object overlaps with its goal location at any point in the episode.
22

---

Under review as a conference paper at ICLR 2022
More detailed video from Figure 4 which illustrates FirVid on action-condiin conoiniioo RoboNer (Dasari et al., 2019). The model is conditioned on the first two frames and predicting the next ten frames given the future actions of the robotic arm. These figures demonstrate how the predicted movements of the arm close|p ly follows the ground truth given that the future actions is known. The model also predicts detailed movements of the pushed objects (visible in the top example) as well as filling in the previously unseen background with some random objects (look the object that appear behind the robotic arm in the bottom example). Also notice the wrong predictions of robots fingers in the bottom example.
t= 1=10
t
t=
t= 6
t=
t=
t=3
t=2
t=1 1
1st Context Frame
Figure 11:
Ground Truth
Groome mub
23

---

Under review as a conference paper at ICLR 2022
Figure 12: More detailed video from Figure 5 which illustrates FitVid on KITTI dataset (Geiger et al. 2013b), As it can be seen in this figure. the model generates high quality prediction of the future in a dynamic scene. Note how in the top example Vid keeps predicting the movement of the shadow on the ground till it moves out of the frame. After that, the model keeps pushing the background closer in each frame, implying driving forward. We noticed that the quality of predictions drop substantially faster when there are more objects in the scene e.g. the driving scenes inside city as can be seen in the bottom example. This indicates the model still fails to generalize to more complex scenes with more moving subjects.
t
2-
t=
t-
t=17
15
1=
1=11
t-
t=7
t=5
t-3
1st Context Frame t=1
Grount
Gruuie Trull
24

---

Under review as a conference paper at ICLR 2022
Figure 13: More detailed vidco from Figure which illustrates FitVid on Human3 6M (lonescu et al., 2014). This figure demonstraiee extremely detailed and human- like motions predicted by FitVid, conditioned on the given context frames. However, on closer inspection, it can be seen that the human subject in the video is changing, from the test subject to a training subject. This particularly evident from the cloths. This phenomena indicates that, although FitVid is capable of generalizing to the frames out of training distribution. however, it morphs the human subject into a familiar one from the training set and then plays the video from the memory.
t-
2-
t=
t-
t=17
1=
1=
1=11
t
t=7
t=5
t=3
1st Context Frame 1=1
Predcctee
Gruuit Tuue
25

---

Under review as a conference paper at ICLR 2022
Figure 14: More detailed video from Figure which illustrates FitVid on BAIR robot pushing dataset (Ebert ct. 2017) with no actions. The model is conditioned only on the first frame and is predicting the next frames. Given that the future actions of the robotic arm is unknown, the prediction can diverge substantially from the ground truth video. However, the model predicts movements for the objects whenever the arm pushes the object in an imaginary scenario. It also fills the background with random objects.
t= 15
t=
t=
t=
t=7
t=5
t=3
t=1
Context Frame
Predictsd
Predicted
Stound Truth
Ground Truth
20

---

Under review as a conference paper at ICLR 2022
Copy of Figure 6 from Franceschi et al. (2020)
14
t=
19
34
t
Gronnd 1ruth
S-VRNN
Ours (Best)
(Worst)
Copy of Figure 7 from Villegas et al. (2019)
t=l1
t=1
t=17
t=20
t=23
1-26
Groundtruth
Biggest model
(Ours)
Copy of Figure 5 from Villegas et al. (2017)
Predicted frames (ours)
Groundtruth Conv LSTM
Figure 15: These are copies of Figure 6 from Franceschi et (2020), Figure from Villegas et al. (2019) and Figure 5 from Villegas et al. (2017). The proposed methods in these papers are also changing the human test subject into training subject (moslt visible in the changed shirt.). This seems be common issue which is typically overlooked in the video prediction literature.
27


Reviews:
[{"review_id": "l5HrVw1_jMU", "reviewer": "Reviewer_3nyU", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is easy to follow and is clearly written. The strengths can be summarized as:\n+ The results look strong and are much better than state-of-the-art across a wide range of datasets and metrics. The qualitative results look particularly good.\n+ The FitVid model is much easier to train and requires no tricks or extra designs to train as in previous works. It can be trained simply from scratch using Adam.\n+ A main argument about the FitVid is that it does not suffer from underfitting issues and is even able to overfit. The ablation studies are extensive and indeed support this argument.\n\nWeaknesses\n- It is not clear how the FitVid model is different from previous models, and why it is able to overcome the underfitting issues while the previous work could not. Even without the data augmentation, FitVid seems to be able to overfit already. Why? More discussion should be devoted to this topic. \n- Minor grammar issues. For example: \"To provide more performance comparison more prior methods...\"\n\nIt would also be great if the authors could provide some thoughts on this issue:\n* Experiments on RoboNet show strikingly good results in Fig4. As stated in the figure caption -- \"The model also predicts detailed movements of the pushed objects (visible in the left example) as well as filling in the previously unseen background with some random objects (look at the object that appear behind the robotic arm in the right)\".   It seems miraculous to me that the model is able to figure out the random objects in the background and their appearance after a few frames. Usually such tasks require understanding of the physical scene (occlusion, object geometry), physics and even 3D rendering. It seems that the proposed model is able to figure out all these very high-level understanding of the scene using a relatively straightforward network design and no extra mechanisms. Could the authors shed some light on how this is achieved?", "summary_of_the_paper": "This paper presents a new network architecture called FitVid to perform the task of video prediction, i.e. the task of predicting future frames from previous frames. Previous methods for this tasks usually suffer from \"underfitting\", while FitVid is able to overfit on major benchmarks without increasing the number of parameters, thanks to a better model architecture. What is more, FitVid is much easier to train than previous work, without using any bells and whistles in training. As a result, FitVid achieves state-of-the-art on four challenging video datasets across a wide range of metrics.", "summary_of_the_review": "I think the paper could provide more insights on why the FitVid network is able to overfit without bells and whistles using even fewer params than previous methods. It would make the paper even stronger. \n\n\n-------------------------------------------------------------------- Post rebuttal\nI have carefully read the reviews and the rebuttal. The main concerns revolve around a lack of novelty. The authors rebut by providing a detailed study of the proposed components. I think the rebuttal addresses well why carefully designed training schedules are no longer necessary, which might be useful to the community. Novelty-wise, it is still a bit limited, and the overall impression to me it is a bit empirical. But given the strong performance, I am keeping my original rating.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "TkNMIdG-_g", "reviewer": "Reviewer_jPAY", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "(1). The paper tries to argue that previous papers suffer from underfitting are mainly a result of inefficient usage of model parameters. It may provide new insight for researchers to design models for video prediction.\n\n(2). The writing of the proposed model is clear and easy to follow.", "summary_of_the_paper": "This paper discusses a new framework named FITVID to handle the problem of video prediction. FitVid is built on existing modules like Sikp connection, Sequeeze and Excite, and LSTMs. FitVid only needs a simple training strategy and begins overfitting the datasets. Data augmentation is also adopted to handle the problem of overfitting. ", "summary_of_the_review": "However, the reviewer does have some concerns on the paper.\n(1). A major concern of the paper is about the model's novelty. The reviewer has doubts on the argument that a new combination of existing techniques (BN, LSTM, S&E, skip connection) for the task of video prediction is significant enough to publish in ICLR. It will be important to show what is the design motivation of the combination and what it can outperform existing models on the task of video prediction.\n\n(2). A comparison of components of different models is needed to help clarify the novelty of the proposed method. It will be also easier for readers to understand why previous models suffer from underfitting and how the model is more data efficient.\n\n(3). Although the authors provide experimental results on 4 datasets. However, most of them contain only two simple baselines and the results on these datasets are not consistent. It will be more convincing if the authors can provide results on future and counterfactual predictions of the video dynamics on the CLEVRER[A] dataset and compare the proposed model' performance with recent stronger backbones like transformer [B], GNN[C] and Differentiable Physics Engine [D].\n\n[A]. Yi K, Gan C, Li Y, et al. Clevrer: Collision events for video representation and reasoning[J]. arXiv preprint arXiv:1910.01442, 2019. \n[B]. D. Ding, F. Hill, A. Santoro, and M. Botvinick. Object-based attention for spatio-temporal reasoning: Outperforming neuro-symbolic models with flexible distributed architectures. arXiv 2020.\n[C]. Chen Z, Mao J, Wu J, et al. Grounding physical concepts of objects and events through dynamic visual reasoning[J]. arXiv 2021.\n[D]. Ding M, Chen Z, Du T, et al. Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Language[J]. arXiv  2021. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Tft8NZs3vuw", "reviewer": "Reviewer_exmV", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n+ Scale is an important aspect of the system. The paper did a good job on investigating the overfit / underfit effect in their model.  Fig 6, analysis on Human 3.6M, Fig7, etc.\n+ The experiments are conducted across 4 datasets under different domain which indicates the proposed method generalizes to some extent. \n+ Though the generalization ability remains further exploration, a robot experiment based on MPC shows that their prediction model can be deployed in downstreaming tasks. \n\nWeakness:\n- Lack of Intuition. The author proposes an Unet like architecture that is claimed to use its parameters more efficiently. However, The intuition behind it is very limited to me. Why is theirs better? Where does the improvement come from? (they also mention they share similar architecture with baselines on Page5 Sec4.2 Para1)  Without intuition, the architecture design looks arbitrarily stacking blocks with some skip connections. I acknowledge that video predictions at pixel-level are extremely challenging and many have proposed their idea like only predicting transformation / hierarchical output / foreground-background factorization, etc.  But this work seems brute-force generates pixels with skip layers. I don’t see why it uses parameters more efficiently.  I look forward to authors justifying their idea. \n- Limited Novelty. Though the paper has achieved some empirical contribution (manage to overfit and improve results on large-scale datasets), the technical contribution is limited. Their architecture is a UNet-based VAE. Data augmentation is well known to prevent overfitting and their particular augmentation is also based on prior work.  \n- Some heterogeneous metrics. Tabel1 does not report results on BAIR pushing dataset while Table2 only reports on BAIR pushing dataset on one metric. I wonder the reason behind the inconsistency. 1) can table 1 include results for BAIR? 2) can table 2 include other datasets? 3) can other metrics be included? E.g. report the best of 100 samples to incorporate with random sample. \n- Their qualitative results on the website look very similar to baselines (GHVAE) for me.\n- Claim on Sec 5 about generalizing to unseen frames. It is a weird claim about video prediction as the unseen future itself is output. The model copy-pasting something from train set does not indicate generalization.  I would just call it overfitting/memorizing train set. (okay w claim 2 but not claim 1) \n- Is the figure 6 results with or without data augmentation? If it’s without, will data augmentation alleviate overfit? If it is with data augmentation, direct comparison (either quantitatively or qualitatively) is expected.\n\n\n--------- after rebuttal -----\nI'd like to thank authors for their detailed response and additional experiments/ablation that they ran. Some of my concerns are resolved (point 3,5,6). For point 4 (qualitative results), I cannot find qualitative comparison on the website anymore. I think it was there by the time I reviewed but not 100% sure. So I check the qualitative results in papers and on GHVAE's website. Most of figures in the paper only shows ground truth with their results w/o comparison except fig 15. For seqs on their and GHVAE website, I'm not convinced about the qualitative improvement, esp not the same seq. Quantitative results are important but qualitative comparison complements paper's arguments. For point 1,2, I appreciate the extra ablations by the authors. There are more empirical results (e.g. switching enc/dec respectively, hyperparam sweep) but the overall novelty is still limited. I'll keep my original score. ", "summary_of_the_paper": "The paper studies conditional video prediction. In particular, it focuses on the problem of current models underfitting and not scaling to datasets. The paper proposes an architecture that is claimed capable of using parameters more efficiently in order to overfit. Then, data augmentation is introduced to improve performance for generalization. They evaluated their method against baselines on 4 datasets (Human 3.6M, KITTI, Robonet, BAIR pushing dataset). They also showed experiments to support overfitting/underfitting claims. \n", "summary_of_the_review": "The paper focuses on an important problem on scaling video prediction and manage to overfit to large-scale data as the first step followed by improving generalization by data augmentation. However, it is not clear to me why and how their proposed method is crucial to achieving this. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "DqWtPDoUzHY", "reviewer": "Reviewer_7Y5j", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nThe paper discovers the underfitting phenomenon which is a novel but the basic problem for the video prediction task. The FitVid's architecture is simple but effective. The author uses several existing methods to overcome the underfitting problem and achieves state-of-the-art performance on four challenging datasets. I really appreciate that the experiments of this paper are very solid.\n\nWeakness:\nThe FitVid's architecture has no additional novelty and just combines the existing methods, such as SE, UNet, and LSTM. The paper did not analyze the effectiveness for each part of the FitVid. Why do previous methods not behave well? The explicit explanation of this problem is important and needs to be explored. I'm looking forward to your reply to this question.\n\n\n====After rebuttal====\nI have read the revised paper and rebuttal comments. I do really appreciate the author's effort to clarify the novelty of the combination of each module. However, I still hope that the proposed combination could be intuitively inspired by the overfit phenomena rather than you just design it and use a large number of experiments to vary its effectiveness. ", "summary_of_the_paper": "The paper proposes a simple and scalable variational video prediction model FitVid, which attains a better fit to video prediction datasets even with a similar parameter count as prior models. The author has observed that previous methods suffer from underfitting on these datasets, directly applying FitVid actually results in overfitting. The FitVid uses a set of existing image augmentation techniques to prevent overfitting so that it can achieve state-of-the-art results on several prediction benchmarks. FitVid's architecture is based on SE-UNet LSTM, which seems to be the common backbone for the stochastic video prediction task. ", "summary_of_the_review": "The paper proposes a simple and scalable variational video prediction model FitVid, which could overcome the underfitting problem better.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a  variational video prediction model FitVid and attains a better fit to video prediction datasets.  The draft was reviewed by four experts in the field and received mixed scores (1 borderline accept, 3 reject). The reviewers raised their concerns on lack of novelty, unconvincing experiment, and the presentation of this paper. For a video prediction model, fitting a dataset is quite important. But AC agrees with the reviewer jPAY. It will be more exciting to build a causal model of the world and enable it to perform future and counterfactual prediction (e.g,  CLEVRER).  The authors are encouraged to consider the reviewers' comments when revising the paper for submission elsewhere.
Decision: Reject