Year: 2018

Abstract:
Being able to predict what may happen in the future requires an in-depth understanding of the physical and causal rules that govern the world. A model that is able to do so has a number of appealing applications, from robotic planning to representation learning. However, learning to predict raw future observations, such as frames in a video, is exceedingly challenging—the ambiguous nature of the problem can cause a naively designed model to average together possible futures into a single, blurry prediction. Recently, this has been addressed by two distinct approaches: (a) latent variational variable models that explicitly model underlying stochasticity and (b) adversarially-trained models that aim to produce naturalistic images. However, a standard latent variable model can struggle to produce realistic results, and a standard adversarially-trained model underutilizes latent variables and fails to produce diverse predictions. We show that these distinct methods are in fact complementary. Combining the two produces predictions that look more realistic to human raters and better cover the range of possible futures. Our method outperforms prior works in these aspects.

heading: 1 INTRODUCTION
text: When we interact with objects in our environment, we can easily imagine the consequences of our actions: push a ball and it will roll; drop a vase and it will break. The ability to imagine future outcomes provides an appealing avenue for learning about the world. Unlabeled video sequences can be gathered autonomously with minimal human intervention, and a machine that learns to predict future events will gain an in-depth and functional understanding of its environment. This leads naturally to the problem of video prediction—given a sequence of context frames, and optionally a proposed action sequence, generate the pixels of the future frames. Once trained, such a model could be used to determine which actions can bring about desired outcomes (Finn et al., 2016; Ebert et al., 2017). Unfortunately, accurate and naturalistic video prediction remains an open problem.
One major challenge in video prediction is the ambiguous nature of the problem. While frames in the immediate future can be extrapolated with high precision, the space of possibilities diverges beyond a few frames, and the problem becomes multimodal by nature. Methods that use deterministic models and loss functions unequipped to handle this inherent uncertainty, such as mean-squared error (MSE), will average together possible futures, producing blurry predictions. Prior works have explored stochastic models for video prediction (Babaeizadeh et al., 2018; Denton & Fergus, 2018), using the framework of variational autoencoders (VAEs) (Kingma & Welling, 2014). These models predict possible futures by sampling latent variables. During training, they optimize a variational lower bound on the likelihood of the data in a latent variable model. However, the posterior is still a pixel-wise MSE loss, corresponding to the log-likelihood under a fully factorized Gaussian distribution. This makes training tractable, but causes them to still make blurry and unrealistic predictions when the latent variables alone do not adequately capture the uncertainty.
Another relevant branch of recent work has been generative adversarial networks (GANs) (Goodfellow et al., 2014) for image generation. Here, a generator network is trained to produce images that are indistinguishable from real images, under the guidance of a learned discriminator network trained to classify images as real or generated. The discriminator operates on patches or entire images, and is thus capable of modeling the joint distribution of pixels. Although this overcomes the limitations of pixel-wise losses, GANs are notoriously susceptible to mode collapse, where latent random variables are often ignored by the model, especially in the conditional setting. This makes them difficult to apply to generation of diverse and plausible futures, conditioned on context frames.
To address these challenges, we propose a model that combines both adversarial losses and latent variables to enable realistic stochastic video prediction. Our model consists of a video prediction network that can sample multiple plausible futures by sampling time-varying stochastic latent variables and decoding them into multiple frames. At training time, an inference network estimates the distribution of these latent variables, and video discriminator networks classify generated videos from real. The full training objective is the variational lower bound used in VAEs combined with the adversarial loss used in GANs. This enables us to capture stochastic posterior distributions of videos while also modeling the spatiotemporal joint distribution of pixels. VAEs with no adversarial losses are also capable of modeling joint distributions, provided that the generator model doesn’t assume any factorization of the pixels. This is the case of pixel-autoregressive models, such as Pixel Video Networks (Kalchbrenner et al., 2017), though training and inference with these models are impractically slow. In this work, we take a different approach and we instead focus on the losses.
The primary contribution of our work is an stochastic video prediction model based on VAE-GANs. To our knowledge, this is the first stochastic video prediction model that combines an adversarial loss with a latent variable model trained via the variational lower bound. Our experiments show that the VAE component greatly improves the diversity of the generated images, while the adversarial loss attains prediction results that are substantially more realistic than state-of-the-art methods, as shown in Fig. 1. We further present a comparison of various types of prediction models and losses, including VAE, GAN, and VAE-GAN models, and analyze the impact of these choices on prediction realism, diversity, and accuracy.

heading: 2 RELATED WORK
text: Recent developments in expressive generative models based on deep networks has led to impressive developments in video generation and prediction. Earlier approaches to prediction focused on models that generate pixels directly from the latent state of the model using both feed-forward (Ranzato et al., 2014; Mathieu et al., 2016) and recurrent (Oh et al., 2015; Xingjian et al., 2015) architectures. An alternative to generating pixels is transforming them by applying a constrained geometric distortion to a previous frame (Finn et al., 2016; De Brabandere et al., 2016; Xue et al., 2016; Byravan & Fox, 2016; Vondrick & Torralba, 2017; van Amersfoort et al., 2017; Liu et al., 2017; Chen et al., 2017; Lu et al., 2017; Walker et al., 2015; 2016; Liang et al., 2017). Aside from the design of the generator architecture, performance is strongly affected by the training objective. Simply minimizing MSE loss can lead to strong results for deterministic synthetic videos (Oh et al., 2015; Chiappa et al., 2017). However, on real-world videos that contain uncertainty, this loss can result in blurry predictions, as the model averages futures to avoid incurring a large MSE loss (Mathieu et al., 2016).
Incorporating uncertainty is critical for addressing this issue. One approach is to model the full joint distribution using pixel-autoregressive models (van den Oord et al., 2016; Kalchbrenner et al., 2017; Reed et al., 2017), though training and inference are impractically slow. Another approach is to train a latent variable model, such as in variational autoencoders (VAEs) (Kingma & Welling, 2014). Conditional VAEs have been used for prediction of optical flow trajectories (Walker et al.,
2016), single-frame prediction (Xue et al., 2016), and recently for stochastic multi-frame video prediction (Babaeizadeh et al., 2018; Denton & Fergus, 2018). While these models can model distributions over possible futures, the prediction distribution is still fully factorized over pixels, which still tends to produce blurry predictions.
Adversarial losses (Goodfellow et al., 2014) for image generation can produce substantially improved realism. However, these networks tend to be difficult to train and are susceptible to mode collapse. A number of prior works have used adversarial losses for deterministic video prediction (Mathieu et al., 2016; Vondrick & Torralba, 2017; Villegas et al., 2017; Lu et al., 2017; Zhou & Berg, 2016; Bhattacharjee & Das, 2017). Several prior works have also sought to produce unconditioned video generations (Vondrick et al., 2016; Saito et al., 2017; Tulyakov et al., 2018) and conditional generation with input noise (Chen et al., 2017; Tulyakov et al., 2018; Wang et al., 2018). We show that a GAN model with input noise can indeed generate realistic videos, but fails to adequately cover the space of possible futures. In contrast, our method, which combines latent variable models with an adversarial loss, produces videos that are both visually plausible and diverse.
Prior works have combined VAEs and GANs to produce stochastic and realistic predictions. Walker et al. (2017) predicts videos of humans by decomposing the problem into a VAE that predicts stochastic future poses and a GAN that generates videos conditioned on those poses and an image. VAE-GANs, which jointly optimize the VAE and GAN losses, have shown promising results for unconditional and conditional image generation (Larsen et al., 2016; Bao et al., 2017; Zhu et al., 2017), but have not been applied to video prediction. The video prediction setting presents two important challenges. First, conditional image generation can handle large appearance changes between the input and output, but suffer when attempting to produce large spatial changes. The video prediction setting is precisely the opposite—the appearance remains largely the same from frame to frame, but the most important changes are spatial. Secondly, video prediction involves sequential prediction, where it’s increasingly difficult to predict farther into the future. Our approach is the first to use VAE-GANs in a recurrent setting for stochastic video prediction.

heading: 3 VIDEO PREDICTION WITH STOCHASTIC ADVERSARIAL MODELS
text: Our goal is to learn a stochastic video prediction model that can predict videos that are diverse and perceptually realistic, and where all predictions are plausible futures for the given initial image. In practice, we use a short initial sequence of images (typically two frames), though we will omit this in our derivation for ease of notation. Our model consists of a recurrent generator network G, which is a deterministic video prediction model that maps an initial image x0 and a sequence of latent random codes z0:T−1, to the predicted sequence of future images x̂1:T . Intuitively, the latent codes encapsulate any ambiguous or stochastic events that might affect the future. At test time, we sample videos by first sampling the latent codes from a prior distribution p(zt), and then passing them to the generator. We use a fixed unit Gaussian prior, N (0, 1). The training procedure for this includes elements of variational inference and generative adversarial networks. Before describing the training procedure, we formulate the problem in the context of VAEs and GANs.

heading: 3.1 VARIATIONAL AUTOENCODERS
text: Our recurrent generator predicts each frame given the previous frame and a random latent code. The previous frame passed to the generator is denoted as x̃t−1 to indicate that it could be a ground truth frame xt−1 (for the initial frames) or the last prediction x̂t−1. The generator specifies a distribution p(xt|xt−1, zt−1), parametrized as a fixed-variance Laplacian distribution with mean x̂t = G(xt−1, zt−1). The likelihood of the data p(x1:T |x0) cannot be directly maximized, since it involves marginalizing over the latent variables, which is intractable in general. Thus, we instead maximize the variational lower bound of the log-likelihood. We approximate the posterior with a recognition model q(zt|xt:t+1), which is parametrized as a conditionally Gaussian distribution N (µzt , σ2zt), represented by a network E(xt:t+1). The encoderE is conditioned on adjacent frames xt and xt+1 in order to have temporally local latent variables zt that capture the ambiguity for only that transition, a sensible choice when using independent and identically distributed Gaussian priors. Another choice is to use temporally correlated latent variables, which would require a stronger prior (e.g. as in Denton & Fergus (2018)). For simplicity, we opted for the former.
During training, the latent code is sampled from q(zt|xt:t+1). The generation of each frame can be thought of as the reconstruction of frame x̂t+1, where the ground truth frame xt+1 (along with xt) is encoded into a latent code zt, and then it (along with the last frame) is mapped back to x̂t+1. Since the latent code has ground truth information about the frame being reconstructed, the model is encouraged to use it during training. This is a conditional version of VAEs, where the encoder and decoder are conditioned on the previous frame (xt or x̂t). To allow back-propagation through the encoder, the reconstruction term is rewritten using the re-parametrization trick,
L1(G,E) = Ex0:T ,zt∼E(xt:t+1)|T−1t=0 [∑T t=1 ||xt −G(xt−1, zt−1)||1 ] . (1)
To enable sampling from the prior at test time, a regularization term encourages the approximate posterior to be close to the prior distribution, LKL(E) = Ex0:T [∑T
t=1 DKL(E(xt−1:t)||p(zt−1))
] . (2)
The VAE optimization involves minimizing the following objective, where the relative weighting of λ1 and λKL is determined by the (fixed) variance of conditional likelihood p(xt|xt−1, zt−1),
G∗, E∗ = argmin G,E λ1L1(G,E) + λKLLKL(E). (3)

heading: 3.2 GENERATIVE ADVERSARIAL NETWORKS
text: Without overcoming the problem of modeling pixel covariances, it is likely not possible to produce sharp and clean predictions. Indeed, as shown in our experiments, the pure VAE model tends to produce blurry futures. We can force the predictions to stay on the video manifold by matching the distributions of predicted and real videos. Given a classifier D that is capable of distinguishing generated videos x̂1:T from real videos x1:T , the generator can be trained to match the statistics of the real data distribution using the binary cross-entropy loss,
LGAN(G,D) = Ex1:T [logD(x1:T )] + Ex1:T ,zt∼p(zt)|T−1t=0 [log(1−D(G(x0, z0:T−1)))]. (4) The overloaded notationG(x0, z0:T−1) indicates the generated sequence x̂1:T . The classifier, which is not known a priori and is problem-specific, can be realized as a deep discriminator network that can be adversarially learned,
G∗ = argmin G max D LGAN(G,D). (5)
This is the setting of GANs. In the conditional case, a per-pixel reconstruction term LGAN1 is added to the objective, which is analogous to L1, except that the latent codes are sampled from the prior.

heading: 3.3 STOCHASTIC ADVERSARIAL VIDEO PREDICTION
text: The VAE and GAN models provide complementary strengths. GANs use a learned loss function through the discriminator, which learns the statistics of natural videos. However, GANs can suffer from the problem of mode collapse, especially in the conditional setting (Pathak et al., 2016; Isola et al., 2017; Zhu et al., 2017). VAEs explicitly encourage the latent code to be more expressive and meaningful, since the learned encoder produces codes that are useful for making accurate predictions at training time. However, during training, VAEs only observe latent codes that are encodings of ground truth images, and never train on completely randomly drawn latent codes, leading to a potential train and test mismatch. GANs, however, are trained with randomly drawn codes.
Our stochastic adversarial video prediction (SAVP) model combines both approaches, shown in Fig. 2. Another term LVAEGAN is introduced, which is analogous to LGAN except that it uses latent codes sampled from q(zt|xt:t+1) and a video discriminator DVAE. The objective of our SAVP model is G∗, E∗ = argmin
G,E max D,DVAE
λ1L1(G,E) + λKLLKL(E) + LGAN(G,D) + LVAEGAN(G,E,DVAE). (6)

heading: 3.4 NETWORK ARCHITECTURES
text: The generator is a convolutional LSTM (Xingjian et al., 2015) that predicts pixel-space transformations between the current and next frame, with additional skip connections with the first frame as done in SNA (Ebert et al., 2017). At every time step, the network is conditioned on the current frame and latent code. After the initial frames, the network is conditioned on its own predictions. The conditioning on the latent codes is realized by concatenating them along the channel dimension to the inputs of all the convolutional layers of the convolutional LSTM. We note that the warping component of this generator assumes that the frames in the videos can be described as transformations of pixels, which is the case for the datasets that we consider. And although the generator used in this work is based on SNA, any video generator (including the one from Denton & Fergus (2018)) could be used with our losses. The encoder is a feed-forward convolutional network that, at every time step, encodes a pair of images xt and xt+1 into µzt and log σzt .
The video discriminator is a feed-forward convolutional network with 3D filters, based on SNGAN (Miyato et al., 2018) but with the filters “inflated” from 2D to 3D. The network takes in a spatiotemporal cube of all the predicted pixels and outputs a single logit. The ground-truth context frames are not provided to the network. We found that spectral normalization in the discriminator and conditioning only on the predicted frames were important for a stable training. We also found that image discriminators that operate on single frames were not necessary. See Fig. 8 and Appendix A for additional details.

heading: 3.5 DISCUSSION OF RELATED VAE MODELS
text: Aside from the adversarial losses, the VAE component of our model is related to prior work on stochastic video prediction. Although the variational losses are the same, there are differences on encoding the posterior distribution and sampling the latent variables at training and test time.
The inference network of Babaeizadeh et al. (2018) estimates a single distribution q(z|x1:T ) by using a feed-forward network that encodes the entire video sequence at once. At test time, the latent variable is sampled from a unit Gaussian prior. They propose two variants for sampling. The latent is sampled once for the entire sequence in the time-invariant case or at every time step in the time-variant case.
On the other hand, the inference network of Denton & Fergus (2018) estimates a time-varying distribution q(zt|x1:t+1) by using a recurrent network that encodes all the frames up to the next frame. They propose two versions for the prior. The prior is either a fixed unit Gaussian distribution or a time-varying distribution p(zt|x1:t), which is learned and estimated by a recurrent network. In both cases, the latent variable is sampled at every time step.
In contrast, our inference network estimates a single distribution q(zt|xt:t+1) by using a feedforward network that encodes the current and next frames. Unlike both prior works, the posterior distribution is temporally local and is conditioned on only two adjacent frames. The latent variables are sampled at every time step and, like the time-variant SV2P and fixed-prior SVG, the prior is a fixed unit Gaussian distribution for every time step.

heading: 4 EXPERIMENTS
text: Our experimental evaluation studies the realism, diversity, and accuracy of the videos generated by our approach and prior methods, and evaluates the importance of various design decisions, including the form of the reconstruction loss and the presence of the variational and adversarial objectives. Evaluating the performance of stochastic video prediction models is exceedingly challenging: not only should the samples from the model be physically realistic and visually plausible given the context frames, but the model should also be able to produce diverse samples that match the conditional distribution in the data. This is difficult to evaluate precisely: realism is not accurately reflected with simple metrics of reconstruction accuracy, and the true conditional distribution in the data is unknown, since real-world datasets only have a single future for each initial sequence. Below, we discuss the metrics that we use to evaluate realism, diversity, and accuracy. No single metric alone provides a clear answer as to which model is better, but considering multiple metrics can provide us with a more complete understanding of the performance and trade-offs of each approach.

heading: 4.1 EVALUATION METRICS
text: Realism: comparisons to real videos using human judges. The realism of the predicted videos is evaluated based on a real vs. fake two-alternative forced choice (2AFC) test. Human judges on Amazon Mechanical Turk (AMT) are presented with a pair of videos—one generated and one real—and asked to identify the generated, or “fake” video. We use the implementation from (Zhang et al., 2016), modified for videos. Each video is 10 frames long and shown over 2.5 seconds. For each method, we gather 1000 judgments from 25 human judges. Each human evaluator is provided with 10 training trials followed by 40 test trials. A method that produces perfectly realistic videos would achieve a fooling rate of 50%. Diversity: distance between samples. Realism is not the only factor in determining the performance of a video prediction model: aside from generating predictions that look physically plausible and realistic, a successful model must also adequately cover the range of possible futures in an uncertain environment. We compute diversity as the average distance between randomly sampled video predictions, similar to Zhu et al. (2017). Distance is measured in the VGG feature space (pretrained on ImageNet classification), averaged across five layers, which has been shown to correlate well with human perception (Dosovitskiy & Brox, 2016; Johnson et al., 2016; Zhang et al., 2018). Accuracy: similarity of the best sample. One weakness of the above metric is that the samples may be diverse but still not cover the feasible output space. Though we do not have the true output distribution, we can still leverage the single ground truth instance. This can be done by sampling the model a finite number of times, and evaluating the similarity between the best sample and the ground truth. This has been explored in prior work on stochastic video prediction (Babaeizadeh et al., 2018; Denton & Fergus, 2018), using PSNR or SSIM as the evaluation metric. In addition to these, we use cosine similarity in the pretrained VGG feature space.

heading: 4.2 DATASETS
text: We evaluate on two real-world datasets: the BAIR action-free robot pushing dataset (Ebert et al., 2017) and the KTH human actions dataset (Schuldt et al., 2004). See Appendix B.2 for additional results on the action-conditioned version of the robot pushing dataset. BAIR action-free. This dataset consists of a randomly moving robotic arm that pushes objects on a table. This dataset is particularly challenging since (a) it contains large amounts of stochasticity due to random arm motion, and (b) it is a real-world application, with a diverse set of objects and large cluttered scene (rather than a single frame-centered object with a neutral background). The frame resolution is 64 × 64. We condition on 2 frames and train to predict the next 10 frames. We predict 10 future frames for the 2AFC experiments and 28 future frames for the other experiments. KTH. This dataset consists of a human subject doing one of six activities: walking, jogging, running, boxing, hand waving, and hand clapping. For the first three activities, the human enters and leaves the frame multiple times, leaving the frame empty with a mostly static background for multiple frames at a time. The sequences are particularly stochastic when the initial frames are all empty since the human can enter the frame at any point in the future. As a preprocessing step, we center-crop each frame to a 120 × 120 square and then resize to a spatial resolution of 64 × 64. We condition on the first 10 frames and train to predict the next 10 frames. We predict 10 future
frames for the 2AFC experiments and 30 future frames for the other experiments. For each sequence, subclips of the desired length are randomly sampled at training and test time.

heading: 4.3 METHODS: ABLATIONS AND COMPARISONS
text: We compare the following variants of our method, in our effort to evaluate the effect of each loss term. Videos, code, and models are available at our website1.
Ours, SAVP. Our stochastic adversarial video prediction model, with the VAE and GAN objectives.
Ours, GAN-only. An ablation of our model with only a conditional GAN, without the variational autoencoder. This model still takes a noise sample as input, but the noise is sampled from the prior during training. This model is broadly representative of prior stochastic GAN-based methods.
Ours, VAE-only. An ablation of our model with only a conditional VAE, with the reconstruction L1 loss but without the adversarial loss. This model is broadly representative of prior stochastic VAE-based methods.
Ours, deterministic. A deterministic ablation of our model with the reconstruction L1 loss but without the VAE nor the GAN objectives. The model uses the same generator architecture but without the latent variables.
1https://video-prediction.github.io/video_prediction
We also compare to prior stochastic VAE-based methods Stochastic Variational Video Prediction (SV2P) (Babaeizadeh et al., 2018) and Stochastic Video Generation (SVG) (Denton & Fergus, 2018), both of which use the reconstruction L2 loss and no adversarial loss.

heading: 4.4 EXPERIMENTAL RESULTS
text: We show qualitative results on the BAIR and KTH datasets in Fig. 3 and Fig. 4, respectively. For the quantitative results, we evaluate the realism, diversity, and accuracy of the predicted videos.
Does our method produce realistic results? In Fig. 5, variants of our method are compared to prior work. On the BAIR action-free dataset, our GAN variant achieves the highest fooling rate, whereas our proposed SAVP model, a VAE-GAN-based method, achieves a fooling rate that is roughly halfway between the GAN and VAE models alone. The SV2P method (Babaeizadeh et al., 2018) does not achieve realistic results. The VAE-based SVG-LP method (Denton & Fergus, 2018) achieves high realism, similar to our VAE variant, but substantially below our GAN-based variants. On the KTH dataset, our SAVP model achieves the highest realism score, substantially above our GAN variant. Among the VAE-based methods without adversarial losses, our VAE-only model outperforms SV2P and SVG-FP (Denton & Fergus, 2018) in terms of realism. Does our method generate diverse results? We measure diversity by taking the distance between random samples. Diversity results are also shown in Fig. 5. For a qualitative visualization of diversity, see Appendix B.4. While the GAN-only approach achieves realistic results, it shows lower diversity than the VAE-based methods. This is an example of the commonly known phenomenon of
mode-collapse, where multiple latent codes produce the same or similar images on the output (Goodfellow, 2016). Intuitively, the VAE-based methods explicitly encourage the latent code to be more expressive by using an encoder from the output space into the latent space during training. This is verified in our experiments, as the VAE-based variants, including our SAVP model, achieve higher diversity than our GAN-only models on both datasets. On the KTH dataset, our VAE variant and VAE-based SVG-FP method (Denton & Fergus, 2018) both achieve significantly higher diversity than all the other methods. Although the VAE-based SV2P methods (Babaeizadeh et al., 2018) mode-collapse on the KTH dataset, we note that they did not evaluate on this dataset, and as such, their method could benefit from different hyperparameters that are better suited for this dataset. Does our method generate accurate results? Following recent work on VAE-based video prediction (Babaeizadeh et al., 2018; Denton & Fergus, 2018), we evaluate on full-reference metrics by sampling multiple predictions from the model. We draw 100 samples for each video, find the “best” sample by computing similarity to the ground truth video, and show the average similarity across the test set as a function of time. The results on the BAIR and KTH datasets are shown in Fig. 14 and Fig. 15, respectively. We test generalization ability by running the model for more time steps than it was trained for. Even though the model is only trained to predict 10 future frames, we observe graceful degradation over time.
While PSNR and SSIM (Wang et al., 2004) are commonly used for video prediction, these metrics are not necessarily indicative of prediction quality. In video prediction, structural ambiguities and geometric deformations are a dominant factor, and SSIM is not an appropriate metric in such situations (Sampat et al., 2009; Zhang et al., 2018). This is particularly noticeable with the SV2P method, which achieves high PSNR and SSIM scores, but produces blurry and unrealistic images. Furthermore, we additionally trained our VAE and deterministic variants using the standard MSE loss L2 to understand the relationship between the form of the reconstruction loss and the metrics. The general trend is that models trained with L2, which favors blurry predictions, are better on PSNR and SSIM, but models trained with L1 are better on VGG cosine similarity. See Appendix B.1 for quantitative results comparing models trained with L1 and L2. In addition, we expect for our GAN-based variants to underperform on PSNR and SSIM since GANs prioritize matching joint distributions of pixels over per-pixel reconstruction accuracy.
To partially overcome the limitations of these metrics, we also evaluate using distances in a deep feature space (Dosovitskiy & Brox, 2016; Johnson et al., 2016), which have been shown to correspond better with human perceptual judgments (Zhang et al., 2018). We use cosine similarity between VGG features averaged across five layers. Otherwise, a model trained for it would unfairly and artificially achieve better similarities by exploiting potential flaws on that metric. Our VAE variant, along with SVG (Denton & Fergus, 2018), performs best on this metric. Although our SAVP model
improves on diversity and realism, it also performs worse in accuracy compared to pure VAE models (both our own ablation and SVG). This is to be expected, since accuracy and realism are at odds with each other. This tradeoff has recently been proved and it holds even for similarity distances based on VGG features (Blau & Michaeli; Blau et al., 2018). Among the VAE-based methods, SV2P (Babaeizadeh et al., 2018) achieves the lowest VGG similarity.
On stochastic environments, such as in the BAIR action-free dataset, there is correlation between diversity and accuracy of the best sample: a model with diverse predictions is more likely to sample a video that is close to the ground truth. This relation can be seen in Fig. 5 and Fig. 14 for the robot dataset, e.g. our SAVP model is both more diverse and achieves higher similarity than our GAN-only variant. This is not true on less stochastic environments. We hypothesize that the KTH dataset is not as stochastic when conditioning on 10 frames, as evidenced the similarities between the predictions from the deterministic and stochastic models. This would explain why our GAN variant and SV2P achieve modest similarities despite achieving low diversity on the KTH dataset.
Does combining the VAE and GAN produce better predictions? The GAN alone achieves high realism but low diversity. The VAE alone achieves lower realism but increased diversity. Adding the GAN to the VAE model increases the realism without sacrificing diversity, at only a small or no cost in realism on stochastic datasets. This is consistent with Zhu et al. (2017), which showed that combining GAN and VAE-based models provides benefits in the case of image generation. To our knowledge, our method is the first to extend this class of models to the video prediction setting, and the first to illustrate that this leads to improved realism with a degree of diversity comparable to the best VAE models in stochastic environments. The results show that this combination of losses is the best choice for realistic coverage of diverse stochastic futures.

heading: 5 CONCLUSION
text: We develop a video prediction model that combines latent variables trained via a variational lower bound with an adversarial loss to produce a high degree of visual and physical realism. VAE-style training enables our method to make diverse stochastic predictions, and our experiments show that the adversarial loss is effective at producing predictions that are more visually realistic according to human raters. Evaluation of video prediction models is a major challenge, and we evaluate our method, as well as ablated variants that consist of only the VAE or only the GAN loss, in terms of a variety of quantitative and qualitative measures, including human ratings, diversity, and accuracy of the predicted samples. Our results demonstrate that our approach produces more realistic predictions than prior methods, while preserving the sample diversity of VAE-based methods.

heading: A NETWORKS AND TRAINING DETAILS

heading: A.1 NETWORK DETAILS

heading: A.1.1 GENERATOR.
text: Our generator network, shown in Fig. 8, is inspired by the convolutional dynamic neural advection (CDNA) model proposed by Finn et al. (2016). The video prediction setting is a sequential prediction problem, so we use a convolutional LSTM (Hochreiter & Schmidhuber, 1997; Xingjian et al., 2015) to predict future frames. We initialize the prediction on the initial sequence of ground truth frames (2 or 10 frames for the BAIR and KTH datasets, respectively), and predict 10 future frames. The model predicts a sequence of future frames by repeatedly making next-frame predictions and feeding those predictions back to itself. For each one-step prediction, the predicted frame is given by a compositing layer, which composes intermediate frames with predicted compositing masks. The intermediate frames include the previous frame, transformed versions of the previous frame, and a frame with pixels directly synthesized by the network. The transformed versions of the frame are produced by convolving in the input image with predicted convolutional kernels, allowing for different shifted versions of the input. In more recent work, the first frame of the sequence is also given as one of the intermediate frames (Ebert et al., 2017).
To enable stochastic sampling, the generator is also conditioned on time-varying latent codes, which are sampled at training and test time. Each latent code zt is an 8-dimensional vector. At each prediction step, the latent code is passed through a fully-connected LSTM to facilitate correlations in time of the latent variables. The encoded latent code is then passed to all the convolutional layers of the main network, by concatenating it along the channel dimension to the inputs of these layers. Since they are vectors with no spatial dimensions, they are replicated spatially to match the spatial dimensions of the inputs.
We made a variety of architectural improvements to the original CDNA (Finn et al., 2016) and SNA (Ebert et al., 2017) models, which overall produced better results on the per-pixel loss and similarity metrics. See Fig. 11 for a quantitative comparison of our deterministic variant (without
VAE or GAN losses) to the SNA model on the BAIR action-conditioned dataset. Each convolutional layer is followed by instance normalization (Ulyanov et al., 2016) and ReLU activations. We also use instance normalization on the LSTM pre-activations (i.e., the input, forget, and output gates, as well as the transformed and next cell of the LSTM). In addition, we modify the spatial downsampling and upsampling mechanisms. Standard subsampling and upsampling between convolutions is known to produce artifacts for dense image generation tasks (Odena et al., 2016; Zhao et al., 2017; Niklaus et al., 2017). In the encoding layers, we reduce the spatial resolution of the feature maps by average pooling, and in the decoding layers, we increase the resolution by using bilinear interpolation. All convolutions in the generator use a stride of 1. In the case of the action-conditioned dataset, actions are concatenated to the inputs of all the convolutional layers of the main network, as opposed to only the bottleneck.

heading: A.1.2 ENCODER.
text: The encoder is a standard convolutional network that, at every time step, encodes a pair of images xt and xt+1 into µzt and log σzt . The latent variable zt is sampled at every time step and the same encoder network with shared weights is used at every step. The encoder architecture consists of three convolutional layers, followed by average pooling of all the spatial dimensions. Two separate fullyconnected layers are then used to estimate µzt and log σzt , respectively. The convolutional layers use instance normalization, leaky ReLU non-linearities, and stride 2. This encoder architecture is the same one used in BicyleGAN (Zhu et al., 2017) except that the inputs are pair of images, concatenated along the channel dimension.

heading: A.1.3 DISCRIMINATOR.
text: The discriminator is a 3D convolutional neural network that takes in all the images of the video at once. We use spectral normalization and the SNGAN discriminator architecture (Miyato et al., 2018), except that we “inflate” the convolution filters from 2D to 3D. The two video discriminators, D and DVAE, share the same architecture, but not the weights, as done in BicycleGAN (Zhu et al., 2017).

heading: A.2 TRAINING DETAILS
text: Our generator network uses scheduled sampling during training as in Finn et al. (2016), such that at the beginning the model is trained for one-step predictions, while by the end of training the model is fully autoregressive. We trained all models with Adam (Kingma & Ba, 2015) for 300000 iterations, linearly decaying the learning rate to 0 for the last 100000 iterations. The same training schedule was used for all the models, except for SVG, which was trained by its author. Our GAN-based variants used an optimizer with β1 = 0.5, β2 = 0.999, learning rate of 0.0002, and a batch size of 16. Our deterministic and VAE models (including SNA and SV2P from prior work) used an optimizer with β1 = 0.9, β2 = 0.999, learning rate of 0.001, and a batch size of 32.
We used λ1 = 100 for our GAN-based variants, and λ1 = 1 for all the other models. For our VAEbased variants, we linearly anneal the weight on the KL divergence term from 0 to the final value λKL during training, as proposed by Bowman et al. (2016), from iterations 50000 to 100000. We used a relative weighting of λKL/λ1 = 0.001 for the BAIR robot pushing datasets, and λKL/λ1 = 0.00001 for the KTH dataset. This hyperparameter was empirically chosen by computing similarity metrics on the validation set.

heading: B ADDITIONAL EXPERIMENTS

heading: B.1 COMPARISON OF PIXEL-WISE L1 AND L2 LOSSES
text: We train our deterministic and VAE variants with the L1 and L2 losses to compare the effects of these reconstruction losses on the full-reference metrics used in this work. The pixel-wise L1 loss assumes that pixels are generated according to a fully factorized Laplacian distribution, whereas the L2 loss corresponds to a fully factorized Gaussian distribution. See Fig. 9 and Fig. 11 for quantitative results on the action-free and action-conditioned BAIR datasets, respectively.
The general trend is that models trained with the L2 loss tend to generate blurry predictions but achieve higher PSNR scores than equivalent models trained with the L1 loss. This is because PSNR and L2 are closely related, the former being a logarithmic function of the latter. The opposite is true for the VGG cosine similarity metric, which has been shown to correspond better with human perceptual judgments (Zhang et al., 2018). Models trained with L1 significantly outperforms equivalent models trained with L2. On the SSIM metric, models trained with L1 achieve roughly the same or better similarities than models trained with L2. Although both losses are pixel-wise losses, the choice between L1 and L2 have a significant impact on the quality of our predicted videos.

heading: B.2 RESULTS ON ACTION-CONDITIONED BAIR ROBOT PUSHING DATASET
text: We use the same dataset as the one in the main paper, except that we use the robot actions. Each action is a 4-dimensional vector corresponding to Cartesian translations and a value indicating if the gripper has been closed or opened. As in the action-free dataset, we condition on the first 2 frames of the sequence and train to predict the next 10 frames. In this dataset, the video prediction model is now also conditioned on a sequence of actions a0:T−1, in addition to the initial frames. The
generator network is modified to take an action at at each time step, by concatenating the action to the inputs of all the convolutional layers of the main network, similar to how the latent code zt is passed in (but without the additional fully-connected LSTM).
We show the realism and diversity results in Fig. 10 and the accuracy results in Fig. 11. In addition to the methods compared in the action-free dataset, we also compare to SNA (Ebert et al., 2017), an action-conditioned deterministic video prediction model. The results indicate that our VAE model significantly outperforms prior methods on the full-reference metrics, and that our models significantly outperforms the model by Babaeizadeh et al. (2018) both in terms of diversity of predictions and realism.

heading: Tulyakov et al.

heading: B.3 EXAMPLE GENERATIONS OF AN UNCONDITIONAL GAN
text: We consider the motion and content decomposed GAN (MoCoGAN) model (Tulyakov et al., 2018) as a representative unconditional GAN method from prior work, and use it to generate videos from the BAIR robot pushing dataset. We use their publicly available code and show qualitative results in Fig. 12. The results show the variant that uses patch-based discriminators, since this one achieved higher realism than the variant that uses image-based discriminators. Since this prior work demonstrates competitive results in comparison to other prior unconditional GAN methods, we chose it as the most representative recent example of purely GAN-based video generation for this comparison. MoCoGAN produces impressive results on various applications related to human action, which are focused on a single actor in the middle of the frame. However, it struggles on videos in the robot pushing domain where multiple entities are moving at a time, i.e. the robot arm and the objects it interacts with.

heading: B.4 QUALITATIVE VISUALIZATION OF DIVERSITY
text: We show a qualitative visualization of diversity in Fig. 13 by averaging multiple samples.

Review summaries:
The paper introduces a generative model for video prediction. The originality stems from a new training criterion which combines a VAE and a GAN criteria. At training time, the GAN and the VAE are trained simultaneously with a shared generator; at test time, prediction conditioned on initial frames is performed by sampling from a latent distribution and generating the next frames via an enhanced conv LST . Evaluations are performed on two movement video datasets classically used for benchmarking  this task - several quantitative evaluation criteria are considered.