Stochastic Adversarial Video Prediction

1 INTRODUCTION . When we interact with objects in our environment , we can easily imagine the consequences of our actions : push a ball and it will roll ; drop a vase and it will break . The ability to imagine future outcomes provides an appealing avenue for learning about the world . Unlabeled video sequences can be gathered autonomously with minimal human intervention , and a machine that learns to predict future events will gain an in-depth and functional understanding of its environment . This leads naturally to the problem of video prediction—given a sequence of context frames , and optionally a proposed action sequence , generate the pixels of the future frames . Once trained , such a model could be used to determine which actions can bring about desired outcomes ( Finn et al. , 2016 ; Ebert et al. , 2017 ) . Unfortunately , accurate and naturalistic video prediction remains an open problem . One major challenge in video prediction is the ambiguous nature of the problem . While frames in the immediate future can be extrapolated with high precision , the space of possibilities diverges beyond a few frames , and the problem becomes multimodal by nature . Methods that use deterministic models and loss functions unequipped to handle this inherent uncertainty , such as mean-squared error ( MSE ) , will average together possible futures , producing blurry predictions . Prior works have explored stochastic models for video prediction ( Babaeizadeh et al. , 2018 ; Denton & Fergus , 2018 ) , using the framework of variational autoencoders ( VAEs ) ( Kingma & Welling , 2014 ) . These models predict possible futures by sampling latent variables . During training , they optimize a variational lower bound on the likelihood of the data in a latent variable model . However , the posterior is still a pixel-wise MSE loss , corresponding to the log-likelihood under a fully factorized Gaussian distribution . This makes training tractable , but causes them to still make blurry and unrealistic predictions when the latent variables alone do not adequately capture the uncertainty . Another relevant branch of recent work has been generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ) for image generation . Here , a generator network is trained to produce images that are indistinguishable from real images , under the guidance of a learned discriminator network trained to classify images as real or generated . The discriminator operates on patches or entire images , and is thus capable of modeling the joint distribution of pixels . Although this overcomes the limitations of pixel-wise losses , GANs are notoriously susceptible to mode collapse , where latent random variables are often ignored by the model , especially in the conditional setting . This makes them difficult to apply to generation of diverse and plausible futures , conditioned on context frames . To address these challenges , we propose a model that combines both adversarial losses and latent variables to enable realistic stochastic video prediction . Our model consists of a video prediction network that can sample multiple plausible futures by sampling time-varying stochastic latent variables and decoding them into multiple frames . At training time , an inference network estimates the distribution of these latent variables , and video discriminator networks classify generated videos from real . The full training objective is the variational lower bound used in VAEs combined with the adversarial loss used in GANs . This enables us to capture stochastic posterior distributions of videos while also modeling the spatiotemporal joint distribution of pixels . VAEs with no adversarial losses are also capable of modeling joint distributions , provided that the generator model doesn ’ t assume any factorization of the pixels . This is the case of pixel-autoregressive models , such as Pixel Video Networks ( Kalchbrenner et al. , 2017 ) , though training and inference with these models are impractically slow . In this work , we take a different approach and we instead focus on the losses . The primary contribution of our work is an stochastic video prediction model based on VAE-GANs . To our knowledge , this is the first stochastic video prediction model that combines an adversarial loss with a latent variable model trained via the variational lower bound . Our experiments show that the VAE component greatly improves the diversity of the generated images , while the adversarial loss attains prediction results that are substantially more realistic than state-of-the-art methods , as shown in Fig . 1 . We further present a comparison of various types of prediction models and losses , including VAE , GAN , and VAE-GAN models , and analyze the impact of these choices on prediction realism , diversity , and accuracy . 2 RELATED WORK . Recent developments in expressive generative models based on deep networks has led to impressive developments in video generation and prediction . Earlier approaches to prediction focused on models that generate pixels directly from the latent state of the model using both feed-forward ( Ranzato et al. , 2014 ; Mathieu et al. , 2016 ) and recurrent ( Oh et al. , 2015 ; Xingjian et al. , 2015 ) architectures . An alternative to generating pixels is transforming them by applying a constrained geometric distortion to a previous frame ( Finn et al. , 2016 ; De Brabandere et al. , 2016 ; Xue et al. , 2016 ; Byravan & Fox , 2016 ; Vondrick & Torralba , 2017 ; van Amersfoort et al. , 2017 ; Liu et al. , 2017 ; Chen et al. , 2017 ; Lu et al. , 2017 ; Walker et al. , 2015 ; 2016 ; Liang et al. , 2017 ) . Aside from the design of the generator architecture , performance is strongly affected by the training objective . Simply minimizing MSE loss can lead to strong results for deterministic synthetic videos ( Oh et al. , 2015 ; Chiappa et al. , 2017 ) . However , on real-world videos that contain uncertainty , this loss can result in blurry predictions , as the model averages futures to avoid incurring a large MSE loss ( Mathieu et al. , 2016 ) . Incorporating uncertainty is critical for addressing this issue . One approach is to model the full joint distribution using pixel-autoregressive models ( van den Oord et al. , 2016 ; Kalchbrenner et al. , 2017 ; Reed et al. , 2017 ) , though training and inference are impractically slow . Another approach is to train a latent variable model , such as in variational autoencoders ( VAEs ) ( Kingma & Welling , 2014 ) . Conditional VAEs have been used for prediction of optical flow trajectories ( Walker et al. , 2016 ) , single-frame prediction ( Xue et al. , 2016 ) , and recently for stochastic multi-frame video prediction ( Babaeizadeh et al. , 2018 ; Denton & Fergus , 2018 ) . While these models can model distributions over possible futures , the prediction distribution is still fully factorized over pixels , which still tends to produce blurry predictions . Adversarial losses ( Goodfellow et al. , 2014 ) for image generation can produce substantially improved realism . However , these networks tend to be difficult to train and are susceptible to mode collapse . A number of prior works have used adversarial losses for deterministic video prediction ( Mathieu et al. , 2016 ; Vondrick & Torralba , 2017 ; Villegas et al. , 2017 ; Lu et al. , 2017 ; Zhou & Berg , 2016 ; Bhattacharjee & Das , 2017 ) . Several prior works have also sought to produce unconditioned video generations ( Vondrick et al. , 2016 ; Saito et al. , 2017 ; Tulyakov et al. , 2018 ) and conditional generation with input noise ( Chen et al. , 2017 ; Tulyakov et al. , 2018 ; Wang et al. , 2018 ) . We show that a GAN model with input noise can indeed generate realistic videos , but fails to adequately cover the space of possible futures . In contrast , our method , which combines latent variable models with an adversarial loss , produces videos that are both visually plausible and diverse . Prior works have combined VAEs and GANs to produce stochastic and realistic predictions . Walker et al . ( 2017 ) predicts videos of humans by decomposing the problem into a VAE that predicts stochastic future poses and a GAN that generates videos conditioned on those poses and an image . VAE-GANs , which jointly optimize the VAE and GAN losses , have shown promising results for unconditional and conditional image generation ( Larsen et al. , 2016 ; Bao et al. , 2017 ; Zhu et al. , 2017 ) , but have not been applied to video prediction . The video prediction setting presents two important challenges . First , conditional image generation can handle large appearance changes between the input and output , but suffer when attempting to produce large spatial changes . The video prediction setting is precisely the opposite—the appearance remains largely the same from frame to frame , but the most important changes are spatial . Secondly , video prediction involves sequential prediction , where it ’ s increasingly difficult to predict farther into the future . Our approach is the first to use VAE-GANs in a recurrent setting for stochastic video prediction . 3 VIDEO PREDICTION WITH STOCHASTIC ADVERSARIAL MODELS . Our goal is to learn a stochastic video prediction model that can predict videos that are diverse and perceptually realistic , and where all predictions are plausible futures for the given initial image . In practice , we use a short initial sequence of images ( typically two frames ) , though we will omit this in our derivation for ease of notation . Our model consists of a recurrent generator network G , which is a deterministic video prediction model that maps an initial image x0 and a sequence of latent random codes z0 : T−1 , to the predicted sequence of future images x̂1 : T . Intuitively , the latent codes encapsulate any ambiguous or stochastic events that might affect the future . At test time , we sample videos by first sampling the latent codes from a prior distribution p ( zt ) , and then passing them to the generator . We use a fixed unit Gaussian prior , N ( 0 , 1 ) . The training procedure for this includes elements of variational inference and generative adversarial networks . Before describing the training procedure , we formulate the problem in the context of VAEs and GANs . 3.1 VARIATIONAL AUTOENCODERS . Our recurrent generator predicts each frame given the previous frame and a random latent code . The previous frame passed to the generator is denoted as x̃t−1 to indicate that it could be a ground truth frame xt−1 ( for the initial frames ) or the last prediction x̂t−1 . The generator specifies a distribution p ( xt|xt−1 , zt−1 ) , parametrized as a fixed-variance Laplacian distribution with mean x̂t = G ( xt−1 , zt−1 ) . The likelihood of the data p ( x1 : T |x0 ) can not be directly maximized , since it involves marginalizing over the latent variables , which is intractable in general . Thus , we instead maximize the variational lower bound of the log-likelihood . We approximate the posterior with a recognition model q ( zt|xt : t+1 ) , which is parametrized as a conditionally Gaussian distribution N ( µzt , σ2zt ) , represented by a network E ( xt : t+1 ) . The encoderE is conditioned on adjacent frames xt and xt+1 in order to have temporally local latent variables zt that capture the ambiguity for only that transition , a sensible choice when using independent and identically distributed Gaussian priors . Another choice is to use temporally correlated latent variables , which would require a stronger prior ( e.g . as in Denton & Fergus ( 2018 ) ) . For simplicity , we opted for the former . During training , the latent code is sampled from q ( zt|xt : t+1 ) . The generation of each frame can be thought of as the reconstruction of frame x̂t+1 , where the ground truth frame xt+1 ( along with xt ) is encoded into a latent code zt , and then it ( along with the last frame ) is mapped back to x̂t+1 . Since the latent code has ground truth information about the frame being reconstructed , the model is encouraged to use it during training . This is a conditional version of VAEs , where the encoder and decoder are conditioned on the previous frame ( xt or x̂t ) . To allow back-propagation through the encoder , the reconstruction term is rewritten using the re-parametrization trick , L1 ( G , E ) = Ex0 : T , zt∼E ( xt : t+1 ) |T−1t=0 [ ∑T t=1 ||xt −G ( xt−1 , zt−1 ) ||1 ] . ( 1 ) To enable sampling from the prior at test time , a regularization term encourages the approximate posterior to be close to the prior distribution , LKL ( E ) = Ex0 : T [ ∑T t=1 DKL ( E ( xt−1 : t ) ||p ( zt−1 ) ) ] . ( 2 ) The VAE optimization involves minimizing the following objective , where the relative weighting of λ1 and λKL is determined by the ( fixed ) variance of conditional likelihood p ( xt|xt−1 , zt−1 ) , G∗ , E∗ = argmin G , E λ1L1 ( G , E ) + λKLLKL ( E ) . ( 3 )

Summary: The paper introduces a generative model for video prediction. The originality stems from a new training criterion which combines a VAE and a GAN criteria. At training time, the GAN and the VAE are trained simultaneously with a shared generator; at test time, prediction conditioned on initial frames is performed by sampling from a latent distribution and generating the next frames via an enhanced conv LST . Evaluations are performed on two movement video datasets classically used for benchmarking  this task - several quantitative evaluation criteria are considered.