Stochastic Latent Residual Video Prediction

1 INTRODUCTION . Being able to predict the future of a video from a few conditioning frames in a self-supervised manner has many applications in fields such as reinforcement learning ( Gregor et al. , 2019 ) or robotics ( Babaeizadeh et al. , 2018 ) . More generally , it challenges the ability of a model to capture visual and dynamic representations of the world . Video prediction has received a lot of attention from the computer vision community . However , most proposed methods are deterministic , reducing their ability to capture video dynamics , which are intrinsically stochastic ( Denton & Fergus , 2018 ) . Stochastic video prediction is a challenging task which has been tackled by recent works . Most state-of-the-art approaches are based on image-autoregressive models ( Denton & Fergus , 2018 ; Babaeizadeh et al. , 2018 ) , built around Recurrent Neural Networks ( RNNs ) , where each generated frame is fed back to the model to produce the next frame . However , performances of their temporal models innately depend on the capacity of their encoder and decoder , as each generated frame has to be re-encoded in a latent space . Such autoregressive processes induce a high computational cost , and strongly tie the frame synthesis and temporal models , which may hurt the performance of the generation process and limit its applicability ( Gregor et al. , 2019 ; Rubanova et al. , 2019 ) . An alternative approach consists in separating the dynamic of the state representations from the generated frames , which are independently decoded from the latent space . In addition to removing the aforementioned link between frame synthesis and temporal dynamics , this is computationally appealing when coupled with a low-dimensional latent-space . Moreover , such models can be used to shape a complete representation of the state of a system , e.g . for reinforcement learning applications ( Gregor et al. , 2019 ) , and more interpretable than autoregressive models ( Rubanova et al. , 2019 ) . Yet , these State-Space Models ( SSMs ) are more difficult to train as they require non-trivial latent state inference schemes ( Krishnan et al. , 2017 ) and a careful design of the dynamic model ( Karl et al. , 2017 ) . This leads most successful SSMs to only be evaluated on small or artificial toy tasks . In this work , we propose a novel stochastic dynamic model for the task of video prediction which successfully leverages structural and computational advantages of SSMs that operate on low-dimensional latent spaces . The dynamic component determines the evolution through residual updates of the latent state , conditioned on learned stochastic variables . This formulation allows us to implement an efficient training strategy and process in an interpretable manner complex high-dimensional data such as videos . This residual principle can be linked to recent advances relating residual networks and Ordinary Differential Equations ( ODEs ) ( Chen et al. , 2018 ) . This interpretation opens new perspectives such as generating videos at different frame rates , as demonstrated in our experiments . Overall , this approach outperforms current state-of-the-art models on the task of stochastic video prediction , as demonstrated by comparisons with competitive baselines on representative benchmarks . 2 RELATED WORK . Video synthesis covers a range of different tasks , such as video-to-video translation ( Wang et al. , 2018 ) , super-resolution ( Caballero et al. , 2017 ) , interpolation between frames ( Jiang et al. , 2018 ) , unconditonal generation ( Tulyakov et al. , 2018 ) , or video prediction , which is the focus of this paper . Deterministic models . Inspired by prior sequence generation models using RNNs ( Graves , 2013 ) , a number of video prediction methods ( Srivastava et al. , 2015 ; Villegas et al. , 2017 ; Wichers et al. , 2018 ) rely on LSTMs ( Hochreiter & Schmidhuber , 1997 ) , or , like Ranzato et al . ( 2014 ) and Jia et al . ( 2016 ) , on derived networks such as ConvLSTMs ( Shi et al. , 2015 ) taking advantage of Convolutional Neural Networks ( CNNs ) . Indeed , computer vision approaches are usually tailored to high-dimensional video sequences and propose domain-specific techniques as they often use pixel-level transformations and optical flow ( Shi et al. , 2015 ; Walker et al. , 2015 ; Finn et al. , 2016 ; Jia et al. , 2016 ; Vondrick & Torralba , 2017 ; Liang et al. , 2017 ; Liu et al. , 2017 ; Lotter et al. , 2017 ; Lu et al. , 2017a ; Fan et al. , 2019 ) that help to produce high-quality predictions . Such predictions are , however , deterministic , thus hurting their performance as they fail to generate sharp long-term video frames ( Babaeizadeh et al. , 2018 ; Denton & Fergus , 2018 ) . Following Mathieu et al . ( 2016 ) , some works proposed to use an adversarial loss ( Goodfellow et al. , 2014 ) on the predictions of their model to sharpen the generated frames ( Vondrick & Torralba , 2017 ; Liang et al. , 2017 ; Lu et al. , 2017a ; Xu et al. , 2018 ) . Nonetheless , adversarial losses are notoriously hard to train , and lead to mode collapse , preventing diversity of generations . Stochastic and image-autoregressive models . Some approaches rely on exact likelihood maximization , using pixel-level autoregressive generation ( van den Oord et al. , 2016 ; Kalchbrenner et al. , 2017 ) or normalizing flows through invertible transformations between the observation space and a latent space ( Kingma & Dhariwal , 2018 ; Kumar et al. , 2019 ) . However , they require careful design of complex temporal generation schemes manipulating high-dimensional data , thus inducing a prohibitive temporal generation cost . More efficient continuous models rely on Variational Auto-Encoders ( VAEs ) ( Kingma & Welling , 2014 ; Rezende et al. , 2014 ) for the inference of lowdimensional latent state variables . Except Xue et al . ( 2016 ) who learn a one-frame-ahead VAE , they model sequence stochasticity by incorporating a random latent variable per frame into a deterministic RNN-based image-autoregressive model . Babaeizadeh et al . ( 2018 ) integrate stochastic variables into the ConvLSTM architecture of Finn et al . ( 2016 ) . Concurrently with He et al . ( 2018 ) , Denton & Fergus ( 2018 ) , with Castrejon et al . ( 2019 ) in a follow-up , use a prior LSTM conditioned on previously generated frames in order to sample random variables that are fed to a predictor LSTM . Finally , Lee et al . ( 2018 ) combine the ConvLSTM architecture and this learned prior , adding an adversarial loss on the predicted videos to sharpen them at the cost of a diversity drop . Yet , all these methods are image-autoregressive , as they feed their predictions back into the latent space , thus tying the frame synthesis and temporal models and increasing their computational cost . Concurrently to our work , Minderer et al . ( 2019 ) propose to use the autoregressive VRNN model ( Chung et al. , 2015 ) on learned image key-points instead of raw frames . While this change could mitigate the aforementioned problems , the extent of such mitigation is unclear . We follow a complementary approach by directly proposing a dynamic model that is state-space and acts on a small latent state , tackling these issues . State-space models . Many latent state-space models have been proposed for sequence modelization ( Bayer & Osendorfer , 2014 ; Fraccaro et al. , 2016 ; 2017 ; Krishnan et al. , 2017 ; Karl et al. , 2017 ; Hafner et al. , 2019 ) , usually trained by deep Variational Inference ( VI ) . These methods , which use locally linear temporal transition functions or RNN-based dynamics , are designed for and tested on low-dimensional data , as learning such models on complex data is challenging , or focus on control or planning tasks . In contrast , our fully latent method is the first one to be successfully applied to complex high-dimensional data such as videos , thanks to a temporal model based on residual updates of its latent state . It thus falls within the scope of a recent trend linking differential equations with neural networks ( Lu et al. , 2017b ; Long et al. , 2018 ) , leading to the integration of ODEs , that are seen as continuous residual networks , in neural network architectures ( Chen et al. , 2018 ) . However , the latter work and follow-ups ( Rubanova et al. , 2019 ; Yıldız et al. , 2019 ) are either limited to low-dimensional data , prone to overfitting or unable to handle stochasticity within a sequence . Another line of works considers stochastic differential equations ( SDEs ) with neural networks ( Ryder et al. , 2018 ; De Brouwer et al. , 2019 ) , but are limited to continuous Brownian noise , whereas video prediction additionally requires to model punctual stochastic events . 3 MODEL . We consider the task of stochastic video prediction , consisting in approaching , given a number of conditioning video frames , the distribution of possible future frames given this conditioning . 3.1 LATENT RESIDUAL DYNAMIC MODEL . Let x1 : T be a sequence of T video frames . We model their evolution by introducing latent variables y that are driven by a dynamic temporal model . Each frame xt is then generated from the corresponding latent state yt only , making the dynamics independent from the previously generated frames . We propose to model the transition function of the latent dynamic of y with a stochastic residual network . State yt+1 is chosen to deterministically depend on the previous state yt , conditionally to an auxiliary random variable zt+1 . These auxiliary variables encapsulate the randomness of the video dynamics . They have a learned factorized Gaussian prior that depends on the previous state only . The model is depicted in Figure 1a , and defined as follows : y1 ∼ N ( 0 , I ) , zt+1 ∼ N ( µθ ( yt ) , σθ ( yt ) I ) , yt+1 = yt + fθ ( yt , zt+1 ) , xt ∼ G ( gθ ( yt ) ) , ( 1 ) where µθ , σθ , fθ and gθ are neural networks , and G ( gθ ( yt ) ) is a probability distribution parameterized by gθ ( yt ) . In our experiments , G is a normal distribution with fixed diagonal variance and mean gθ ( yt ) . Note that y1 is assumed to have a standard Gaussian prior , and , in our VAE setting , will be inferred from conditioning frames for the prediction task , as shown in Section 3.3 . The residual update rule takes inspiration in the Euler discretization scheme of differential equations . The state of the system yt is updated by its first-order movement , i.e. , the residual fθ ( yt , zt+1 ) . Compared to a regular RNN , this simple principle makes our temporal model lighter and more interpretable . Equation ( 1 ) , however , differs from a discretized ODE because of the introduction of the stochastic discrete-time variables z . Nonetheless , we propose to allow the Euler step size ∆t to be smaller than 1 , as a way to make the temporal model closer to a continuous dynamics . The updated dynamics becomes , with 1∆t ∈ N to synchronize the step size with the video frame rate : yt+∆t = yt + ∆t · fθ ( yt , zbtc+1 ) . ( 2 ) For this formulation , the auxiliary variable zt is kept constant between two integer time steps . Note that a different ∆t can be used during training or testing . This allows our model to generate videos at an arbitrary frame rate since each intermediate latent state can be decoded in the observation space . This ability enables us to observe the quality of the learned dynamic as well as challenge its ODE inspiration by testing its generalization to the continuous limit in Section 4 . In the following , we consider ∆t as a hyperparameter . For the sake of clarity , we consider that ∆t = 1 in the following ; generalizing to smaller ∆t is straightforward as Figure 1a remains unchanged .

Summary: Contributions: this submission proposes a video pixel generation framework with the goal to decouple visual appearance and dynamics. The latent dynamics are modeled with a latent residual dynamics model. Empirical evaluations on moving MNIST show that the proposed residual dynamics model outperform MLP or GRU. On more challenging KTH and BAIR datasets, the proposed method achieves on par or better quantitative performance with previous methods, and have nice qualitative results on content "swap" and dynamics interpolation.