Year: 2019

Abstract:
Video prediction is a challenging task: models have to account for the inherent uncertainty of the future. Most works in the literature are based on stochastic imageautoregressive recurrent networks, raising several performance and applicability issues. An alternative is to use fully latent temporal models which untie frame synthesis and dynamics. However, no such model for video prediction has been proposed in the literature yet, due to design and training difficulties. In this paper, we overcome these difficulties by introducing a novel stochastic temporal model. It is based on residual updates of a latent state, motivated by discretization schemes of differential equations. This first-order principle naturally models video dynamics as it allows our simpler, lightweight, interpretable, latent model to outperform prior state-of-the-art methods on challenging datasets.

heading: 1 INTRODUCTION
text: Being able to predict the future of a video from a few conditioning frames in a self-supervised manner has many applications in fields such as reinforcement learning (Gregor et al., 2019) or robotics (Babaeizadeh et al., 2018). More generally, it challenges the ability of a model to capture visual and dynamic representations of the world. Video prediction has received a lot of attention from the computer vision community. However, most proposed methods are deterministic, reducing their ability to capture video dynamics, which are intrinsically stochastic (Denton & Fergus, 2018).
Stochastic video prediction is a challenging task which has been tackled by recent works. Most state-of-the-art approaches are based on image-autoregressive models (Denton & Fergus, 2018; Babaeizadeh et al., 2018), built around Recurrent Neural Networks (RNNs), where each generated frame is fed back to the model to produce the next frame. However, performances of their temporal models innately depend on the capacity of their encoder and decoder, as each generated frame has to be re-encoded in a latent space. Such autoregressive processes induce a high computational cost, and strongly tie the frame synthesis and temporal models, which may hurt the performance of the generation process and limit its applicability (Gregor et al., 2019; Rubanova et al., 2019).
An alternative approach consists in separating the dynamic of the state representations from the generated frames, which are independently decoded from the latent space. In addition to removing the aforementioned link between frame synthesis and temporal dynamics, this is computationally appealing when coupled with a low-dimensional latent-space. Moreover, such models can be used to shape a complete representation of the state of a system, e.g. for reinforcement learning applications (Gregor et al., 2019), and more interpretable than autoregressive models (Rubanova et al., 2019). Yet, these State-Space Models (SSMs) are more difficult to train as they require non-trivial latent state inference schemes (Krishnan et al., 2017) and a careful design of the dynamic model (Karl et al., 2017). This leads most successful SSMs to only be evaluated on small or artificial toy tasks.
In this work, we propose a novel stochastic dynamic model for the task of video prediction which successfully leverages structural and computational advantages of SSMs that operate on low-dimensional latent spaces. The dynamic component determines the evolution through residual updates of the latent state, conditioned on learned stochastic variables. This formulation allows us to implement an efficient training strategy and process in an interpretable manner complex high-dimensional data such as videos. This residual principle can be linked to recent advances relating residual networks and Ordinary Differential Equations (ODEs) (Chen et al., 2018). This interpretation opens new perspectives such as generating videos at different frame rates, as demonstrated in our experiments. Overall, this approach outperforms current state-of-the-art models on the task of stochastic video prediction, as demonstrated by comparisons with competitive baselines on representative benchmarks.

heading: 2 RELATED WORK
text: Video synthesis covers a range of different tasks, such as video-to-video translation (Wang et al., 2018), super-resolution (Caballero et al., 2017), interpolation between frames (Jiang et al., 2018), unconditonal generation (Tulyakov et al., 2018), or video prediction, which is the focus of this paper.
Deterministic models. Inspired by prior sequence generation models using RNNs (Graves, 2013), a number of video prediction methods (Srivastava et al., 2015; Villegas et al., 2017; Wichers et al., 2018) rely on LSTMs (Hochreiter & Schmidhuber, 1997), or, like Ranzato et al. (2014) and Jia et al. (2016), on derived networks such as ConvLSTMs (Shi et al., 2015) taking advantage of Convolutional Neural Networks (CNNs). Indeed, computer vision approaches are usually tailored to high-dimensional video sequences and propose domain-specific techniques as they often use pixel-level transformations and optical flow (Shi et al., 2015; Walker et al., 2015; Finn et al., 2016; Jia et al., 2016; Vondrick & Torralba, 2017; Liang et al., 2017; Liu et al., 2017; Lotter et al., 2017; Lu et al., 2017a; Fan et al., 2019) that help to produce high-quality predictions. Such predictions are, however, deterministic, thus hurting their performance as they fail to generate sharp long-term video frames (Babaeizadeh et al., 2018; Denton & Fergus, 2018). Following Mathieu et al. (2016), some works proposed to use an adversarial loss (Goodfellow et al., 2014) on the predictions of their model to sharpen the generated frames (Vondrick & Torralba, 2017; Liang et al., 2017; Lu et al., 2017a; Xu et al., 2018). Nonetheless, adversarial losses are notoriously hard to train, and lead to mode collapse, preventing diversity of generations.
Stochastic and image-autoregressive models. Some approaches rely on exact likelihood maximization, using pixel-level autoregressive generation (van den Oord et al., 2016; Kalchbrenner et al., 2017) or normalizing flows through invertible transformations between the observation space and a latent space (Kingma & Dhariwal, 2018; Kumar et al., 2019). However, they require careful design of complex temporal generation schemes manipulating high-dimensional data, thus inducing a prohibitive temporal generation cost. More efficient continuous models rely on Variational Auto-Encoders (VAEs) (Kingma & Welling, 2014; Rezende et al., 2014) for the inference of lowdimensional latent state variables. Except Xue et al. (2016) who learn a one-frame-ahead VAE, they model sequence stochasticity by incorporating a random latent variable per frame into a deterministic RNN-based image-autoregressive model. Babaeizadeh et al. (2018) integrate stochastic variables into the ConvLSTM architecture of Finn et al. (2016). Concurrently with He et al. (2018), Denton & Fergus (2018), with Castrejon et al. (2019) in a follow-up, use a prior LSTM conditioned on previously generated frames in order to sample random variables that are fed to a predictor LSTM. Finally, Lee et al. (2018) combine the ConvLSTM architecture and this learned prior, adding an adversarial loss on the predicted videos to sharpen them at the cost of a diversity drop. Yet, all these methods are image-autoregressive, as they feed their predictions back into the latent space, thus tying the frame synthesis and temporal models and increasing their computational cost. Concurrently to our work, Minderer et al. (2019) propose to use the autoregressive VRNN model (Chung et al., 2015) on learned image key-points instead of raw frames. While this change could mitigate the aforementioned problems, the extent of such mitigation is unclear. We follow a complementary approach by directly proposing a dynamic model that is state-space and acts on a small latent state, tackling these issues.
State-space models. Many latent state-space models have been proposed for sequence modelization (Bayer & Osendorfer, 2014; Fraccaro et al., 2016; 2017; Krishnan et al., 2017; Karl et al., 2017; Hafner et al., 2019), usually trained by deep Variational Inference (VI). These methods, which use locally linear temporal transition functions or RNN-based dynamics, are designed for and tested on low-dimensional data, as learning such models on complex data is challenging, or focus on control or planning tasks. In contrast, our fully latent method is the first one to be successfully applied to complex high-dimensional data such as videos, thanks to a temporal model based on residual updates of its latent state. It thus falls within the scope of a recent trend linking differential equations with neural networks (Lu et al., 2017b; Long et al., 2018), leading to the integration of ODEs, that are seen as continuous residual networks, in neural network architectures (Chen et al., 2018). However, the latter work and follow-ups (Rubanova et al., 2019; Yıldız et al., 2019) are either limited to low-dimensional data, prone to overfitting or unable to handle stochasticity within a sequence. Another line of works considers stochastic differential equations (SDEs) with neural networks (Ryder
et al., 2018; De Brouwer et al., 2019), but are limited to continuous Brownian noise, whereas video prediction additionally requires to model punctual stochastic events.

heading: 3 MODEL
text: We consider the task of stochastic video prediction, consisting in approaching, given a number of conditioning video frames, the distribution of possible future frames given this conditioning.

heading: 3.1 LATENT RESIDUAL DYNAMIC MODEL
text: Let x1:T be a sequence of T video frames. We model their evolution by introducing latent variables y that are driven by a dynamic temporal model. Each frame xt is then generated from the corresponding latent state yt only, making the dynamics independent from the previously generated frames.
We propose to model the transition function of the latent dynamic of y with a stochastic residual network. State yt+1 is chosen to deterministically depend on the previous state yt, conditionally to an auxiliary random variable zt+1. These auxiliary variables encapsulate the randomness of the video dynamics. They have a learned factorized Gaussian prior that depends on the previous state only. The model is depicted in Figure 1a, and defined as follows: y1 ∼ N (0, I), zt+1 ∼ N ( µθ(yt), σθ(yt)I ) , yt+1 = yt + fθ(yt, zt+1), xt ∼ G ( gθ(yt) ) , (1)
where µθ, σθ, fθ and gθ are neural networks, and G ( gθ(yt) ) is a probability distribution parameterized by gθ(yt). In our experiments, G is a normal distribution with fixed diagonal variance and mean gθ(yt). Note that y1 is assumed to have a standard Gaussian prior, and, in our VAE setting, will be inferred from conditioning frames for the prediction task, as shown in Section 3.3.
The residual update rule takes inspiration in the Euler discretization scheme of differential equations. The state of the system yt is updated by its first-order movement, i.e., the residual fθ(yt, zt+1). Compared to a regular RNN, this simple principle makes our temporal model lighter and more interpretable. Equation (1), however, differs from a discretized ODE because of the introduction of the stochastic discrete-time variables z. Nonetheless, we propose to allow the Euler step size ∆t to be smaller than 1, as a way to make the temporal model closer to a continuous dynamics. The updated dynamics becomes, with 1∆t ∈ N to synchronize the step size with the video frame rate:
yt+∆t = yt + ∆t · fθ ( yt, zbtc+1 ) . (2)
For this formulation, the auxiliary variable zt is kept constant between two integer time steps. Note that a different ∆t can be used during training or testing. This allows our model to generate videos at
an arbitrary frame rate since each intermediate latent state can be decoded in the observation space. This ability enables us to observe the quality of the learned dynamic as well as challenge its ODE inspiration by testing its generalization to the continuous limit in Section 4. In the following, we consider ∆t as a hyperparameter. For the sake of clarity, we consider that ∆t = 1 in the following; generalizing to smaller ∆t is straightforward as Figure 1a remains unchanged.

heading: 3.2 CONTENT VARIABLE
text: Some components of video sequences can be static, such as the background or shapes of moving objects. They may not impact the dynamics; we therefore model them separately, in the same spirit as Denton & Birodkar (2017) and Yingzhen & Mandt (2018). We compute a content variable w that remains constant throughout the whole generation process and is fed together with yt into the frame generator. It enables the dynamical part of the model to focus only on movement, hence being lighter and more stable. Moreover, it allows us to leverage architectural advances in neural networks, such as skip connections (Ronneberger et al., 2015), to produce more realistic frames.
This content variable is a deterministic function cψ of a fixed number k < T of frames x (k) c :
x(k)c = xi1 , . . . ,xik , w = cψ ( x(k)c ) = cψ ( xi1 , . . . ,xik ) , xt ∼ G ( gθ(yt,w) ) . (3)
During testing, x(k)c are the last k conditioning frames (usually between 2 and 5).
This content variable is not endowed with any probabilistic prior, contrary to the dynamic variables y and z. Hence, the information it contains is not constrained in the loss function (see Section 3.3), but only architecturally. To prevent temporal information from leaking in w, we propose to uniformly sample these k frames within x1:T during training. We also design cψ as a permutation-invariant function (Zaheer et al., 2017), which is done by using an MLP fed with the sum of individual frame representations, similarly to Santoro et al. (2017).
This absence of prior and its architectural constraint allows w to contain as much non-temporal information as possible, while preventing it from containing dynamic information. On the other hand, due to their strong standard Gaussian priors, y and z are encouraged to discard unnecessary information. Therefore, y and z should only contain temporal information that could not be captured by w.
Note that this content variable can be removed from our model, yielding a more classical deep state-space model. An experiment in this setting is presented in Appendix E.

heading: 3.3 VARIATIONAL INFERENCE AND ARCHITECTURE
text: Following the generative process depicted in Figure 1a, the conditional joint probability of the full model, given a content variable w, can be written as:
p(x1:T , z2:T ,y1:T | w) = p(y1) T−1∏ t=1 p(zt+1 | yt)p(yt+1 | yt, zt+1) T∏ t=1 p(xt | yt,w), (4)
where p(yt+1 | yt, zt+1) = δ ( yt + fθ(yt, zt+1)− yt+1 ) and δ is the Dirac delta function centered on 0, according to the expression of yt+1 in Equation (1). Thus, in order to optimize the likelihood of the observed videos p(x1:T | w), we need to infer latent variables y1 and z2:T . This is done by deep Variational Inference using the inference model parameterized by φ and shown in Figure 1b, which comes down to consider a variational distribution qZ,Y defined and factorized as follows:
qZ,Y , q(z2:T ,y1:T | x1:T ,w) = q(y1 | x1:k) T∏ t=2 q(zt | x1:t)δ ( yt−1 + fθ(yt−1, zt)− yt ) . (5)
This yields the following evidence lower bound (ELBO), whose full derivation is given in Appendix A:
log p(x1:T | w) ≥ E(z̃2:T ,ỹ1:T )∼qZ,Y T∑ t=1 log p(xt | ỹt,w)−DKL ( q(y1 | x1:k) ∥∥ p(y1)) − E(z̃2:T ,ỹ1:T )∼qZ,Y
T∑ t=2 DKL ( q(zt | x1:t) ∥∥ p(zt | ỹt−1)) , L(x1:T ;w, θ, φ). (6) The sum of KL divergence expectations implies to consider the full past sequence of inferred states for each time step, due to the dependence on conditionally deterministic variables y2:T . However, optimizing L(x1:T ;w, θ, φ) with respect to model parameters θ and variational parameters φ can be done efficiently by sampling a single full sequence of states from qZ,Y per example, and computing gradients by backpropagation (Rumelhart et al., 1988) trough all inferred variables, using the reparametrization trick (Kingma & Welling, 2014; Rezende et al., 2014). We classically choose q(y1 | x1:k) and q(zt | x1:t) to be factorized Gaussian so that all KLDs can be computed analytically. We include an `2 regularization term on residuals fθ which stabilizes the temporal dynamics of the residual network, as noted by Behrmann et al. (2019) and Rousseau et al. (2019). Given a set of videos X , the full optimization problem, where L is defined as in Equation (6), is then given as:
arg max θ,φ,ψ ∑ x∈X E x (k) c L ( x1:T ; cψ ( x(k)c ) , θ, φ ) − λ · E(z2:T ,y1:T )∼qZ,Y T∑ t=2 ∥∥fθ(yt−1, zt)∥∥2 . (7)
Figure 1c depicts the full architecture of our temporal model, corresponding to how the model is applied during testing. The first latent variables are inferred with the conditioning framed and are then predicted with the dynamic model. In contrast, during training, each frame of the input sequence is considered for inference, which is done as follows. Firstly, each frame xt is independently encoded into a vector-valued representation x̃t, with x̃t = hφ(xt). y1 is then inferred using an MLP on the first k encoded frames x̃1:k. Each zt is inferred in a feed-forward fashion with an LSTM on the encoded frames. Inferring z this way experimentally performs better than, e.g., inferring them from the whole sequence x1:T ; we hypothesize that this follows from the fact that this filtering scheme is closer to the prediction setting, where the future is not available.

heading: 4 EXPERIMENTS
text: This section exposes the experimental results of our method on three standard stochastic video prediction datasets.1 We compare our method with state-of-the-art baselines on stochastic video prediction. Furthermore, we qualitatively study the dynamics and latent space learned by our model. Training details are described in Appendix C.
The stochastic nature and novelty of the task of stochastic video prediction make it challenging to evaluate (Lee et al., 2018): since videos and models are stochastic, comparing the ground truth and a predicted video is not adequate. We thus adopt the common approach (Denton & Fergus, 2018; Lee et al., 2018) consisting in, for each test sequence, sampling from the tested model a given number (here, 100) of possible futures and reporting the best performing sample against the true video. We report this discrepancy for three commonly used metrics: Peak Signal-to-Noise Ratio (PSNR, higher is better), Structured Similarity (SSIM, higher is better), and Learned Perceptual Image Patch Similarity (LPIPS, lower is better) (Zhang et al., 2018). PSNR tends to promote blurry predictions, as it is a pixel-level measure derived from the `2 distance, but greatly penalizes errors in predicted positions of objects in the scenes. SSIM is a similarity metric between image patches. LPIPS is a learned distance between activations of deep CNNs trained on image classification tasks, and have been shown to better correlate with human judgment on real images. While these three metrics are computed frame-wise, the recently proposed Fréchet Video Distance (FVD, lower is better) (Unterthiner et al., 2018) aims at directly comparing the distribution of predicted videos with the ground truth distribution through the representations computed by a deep CNN trained on action
1Code, video samples, and datasets are available at https://sites.google.com/view/srvp/.
recognition tasks. It has been shown, independently from LPIPS, to better correlate with human judgment than PSNR and SSIM. We treat all four metrics as complementary, as they capture different modalities. PSNR challenges the dynamics of the predicted videos, while SSIM rather compares local frame patches but loses some dynamics information. LPIPS and FVD both measure the realism of the predictions compared to the ground truth. FVD considers videos as a whole, making it more capable of detecting temporal inconsistencies. On the other hand, the frame-wise LPIPS metric penalizes more the temporal drifts of videos, since it directly compares each predicted and ground truth frame.
We present experimental results on a simulated dataset and two real-world datasets, that we briefly present in the following and detail in Appendix B. The corresponding numerical results can be found in Appendix D. For the sake of concision, we only display a handful of qualitative samples in this section, and refer to Appendix H for additional samples. We compare our model against several state-of-the-art models: SV2P (Babaeizadeh et al., 2018), SVG (Denton & Fergus, 2018) and SAVP (Lee et al., 2018). All baseline results were obtained with pretrained models released by the authors. Note that we use the same neural architecture as SVG for our encoders and decoders in order to perform fair comparisons with this method, which is the closest to ours among the state of the art. Unless specified otherwise, our model is tested with the same ∆t as in training (see Equation (2)).
Stochastic Moving MNIST (SM-MNIST). This dataset consists of one or two MNIST digits (LeCun et al., 1998) moving linearly and randomly bouncing on walls with new direction and velocity sampled randomly at each bounce (Denton & Fergus, 2018). As SV2P and SAVP were not tested on this dataset (in particular, with no pretrain model, code or hyperparameters), we only report scores for SVG as state-of-the-art model on SM-MNIST.
Figure 2a shows quantitative results with two digits. Our model outperforms SVG on both PSNR and SSIM; LPIPS and FVD are not reported as they are not relevant for this synthetic task. Decoupling dynamics from image synthesis allows our method to maintain temporal consistency despite highuncertainty frames where crossing digits become indistinguishable. For instance in Figure 3, the digits shape changes after they cross in the SVG prediction, while our model predicts the correct digits. To evaluate the predictive ability on a longer horizon, we perform experiments on the classic deterministic version of the dataset (Srivastava et al., 2015). We show the results up to t + 95 in
Figure 2b. We can see that our model better captures the dynamics of the problem compared to SVG as its performance decreases significantly less, even at a long-term horizon.
We also compare to two alternative versions of our model in Figure 2, where the residual dynamic function is replaced by an MLP or a GRU network (Cho et al., 2014). Our residual model outperforms both versions on the stochastic, and especially on the deterministic version of the dataset, showing its intrinsic advantage at modeling dynamics. Finally, on the deterministic version of Moving MNIST, we compare to an alternative where z is entirely removed, resulting in a temporal model very close to the one presented in Chen et al. (2018). The loss of performance of this alternative model is significant, especially in SSIM, showing that our stochastic residual model offers a substantial advantage even when used in a deterministic environment.
KTH Action dataset (KTH). This dataset is composed of real-world videos of people performing a single action per video in front of different backgrounds (Schüldt et al., 2004). Uncertainty lies in the appearance of subjects, the actions they perform and how they are performed.
We outperform on this dataset every considered baseline for each metric, as depicted in Figure 4 and Table 2. In some videos, the subject only appears after the conditioning frames, requiring the model to sample the moment and location of the subject appearance, as well as its action. This critical case is illustrated in Figure 5. There, SVG fails to even generate a moving person; only SAVP and our model manage to do so, and our best sample is closer to the subject’s poses compared to SAVP. Moreover, the worst and a random sample of our model demonstrate that it captures the diversity of the dataset by making a person appear at different time steps and with different speeds. An additional experiment on this dataset is included in Appendix G, studying the influence of the encoder and decoder architecture on SVG and our model.
Finally, Table 2 compares our method to its MLP and GRU alternative versions, leading to two conclusions. Firstly, it confirms the structural advantage of residual dynamics observed on Moving MNIST. On one hand, MLP better captures dynamics than GRU on KTH according to PSNR and SSIM, but loses in terms of realism according to LPIPS and FVD. On the other hand, the residual version shows a slight dynamics improvement with respect to both MLP and GRU, while substantially pushing further prediction realism. Secondly, all three versions of our model (residual, MLP, GRU) outperform prior methods. Therefore, this improvement is due to their common inference method, latent nature and content variable, strengthening our motivation to propose a non-autoregressive model.
BAIR robot pushing dataset (BAIR). This dataset contains videos of a Sawyer robotic arm pushing objects on a tabletop (Ebert et al., 2017). It is highly stochastic as the arm can change its direction at any moment. We achieve similar or better results compared to state-of-the-art models, as Figure 6 and Table 3 shows, and second-best PSNR behind SV2P, but the latter produces very blurry samples, which can be seen in Appendix H, yielding prohibitive LPIPS and FVD scores. In contrast, we achieve the highest SSIM overall, as well as state-of-the-art LPIPS and competitive FVD among these models. Note that we could not add VideoFlow to our experiments, due to the unavailability of pretrained models and numerical results. However, compared to PSNR, SSIM and LPIPS results reported by Kumar et al. (2019) for BAIR (the only tested dataset and metrics in their paper), our model appears to behave better than VideoFlow, which is on par with SAVP on these metrics.
Varying frame rate in testing. We challenge the ability of our model to use a different Euler step size than the one used in training (see Equation (2)). Figures 4 and 6 include corresponding results with a halved ∆t. Prediction performances remain stable while generating twice as many frames (cf. Appendix F for further discussion). Our model is thus robust to the refinement of the Euler approximation, showing the quality of the learned dynamic which is close to continuous. In particular, this shows that our model learned a dynamic driven by a piecewise ODE, i.e., the learned dynamic of each interval between two consecutive frames is an ODE, as a constant z is given on such interval. This can be used to generate frames at a higher frame rate than the training videos without supervision. We show in Figure 7 and Appendix F frames generated at a double and quadruple frame rate on BAIR and KTH. Both figures show smooth intermediate generated frames.
Disentangling dynamics and content. Let us show that the proposed model actually separates content from dynamics as discussed in Section 3.2. To this end, two sequences xs and xt are drawn
from the BAIR test set. While xs is used for extracting our content variable ws, dynamic states yt are inferred with our model from xt. New frame sequences x̂ are finally generated from the fusion of the content vector and the dynamics. This results in a content corresponding to the first sequence xs while moving according to the dynamics of the second sequence xt, as observed in Figure 8. More samples for BAIR and KTH can be seen in Appendix H.
Interpolation of dynamics. Our state-space structure allows us to learn semantic representations in yt. To highlight this feature, we test whether two Moving MNIST trajectories can be interpolated by linearly interpolating their inferred latent initial conditions. We begin by generating two trajectories xs and xt of a single moving digit. We infer their respective latent initial conditions ys1 and y t 1. We then use our model to generate frame sequences from latent initial conditions linearly interpolated between ys1 and y t 1. If it learned a meaningful latent space, the resulting trajectories should also be smooth interpolations between the directions of reference trajectories xs and xt, and this is what we observe in Figure 9. Additional examples can be found in Appendix H.

heading: 5 CONCLUSION
text: We introduce a novel dynamic latent model for stochastic video prediction which, unlike prior imageautoregressive models, decouples frame synthesis and dynamics. This temporal model is based on residual updates of a small latent state that is showed to perform better than RNN-based models. This endows our method with several desirable properties, such as temporal efficiency and latent space interpretability. We experimentally demonstrate the performance and advantages of the proposed model, which outperforms prior state-of-the-art methods for stochastic video prediction. This work is, to the best of our knowledge, the first to propose a latent dynamic model scaling for video prediction. The proposed model is also novel with respect to the recent line of work dealing with neural networks and ODEs for temporal modeling; it is the first such residual model to scale to complex stochastic data such as videos.
We believe that the general principles of our model (state-space, residual dynamic, static content variable) can be generally applied to other models as well. Interesting future works include replacing the VRNN model of Minderer et al. (2019) in order to model the evolution of key-points, or leveraging the state-space nature of our model in model-based reinforcement learning.

heading: A EVIDENCE LOWER BOUND
text: We develop in this section the computations of the variational lower bound for the proposed model.
Using the original variational lower bound of Kingma & Welling (2014) in Equation (8):
log p(x1:T | w) ≥ E(z̃2:T ,ỹ1:T )∼qZ,Y log p(x1:T | z̃2:T , ỹ1:T ,w)−DKL ( qZ,Y ∥∥ p(y1:T , z2:T | w)) (8) = E(z̃2:T ,ỹ1:T )∼qZ,Y log p(x1:T | z̃2:T , ỹ1:T ,w)−DKL ( q(y1, z2:T | x1:T )
∥∥ p(y1, z2:T )) (9)
= E(z̃2:T ,ỹ1:T )∼qZ,Y T∑ t=1 log p(xt | ỹt,w)−DKL ( q(y1, z2:T | x1:T ) ∥∥ p(y1, z2:T )), (10)
where:
• Equation (9) is given by the forward and inference models factorizing p and q in Equations (4) and (5) and illustrated by, respectively, Figures 1a and 1b:
– the z variables and y1 are independent from w to p and q; – the y2:T variables are deterministic functions of y1 and z2:T with respect to p and q;
• Equation (10) results from the factorization of p(x1:T | y1:T , z1:T ,w) in Equation (4).
From there, by using the integral formulation of DKL:
log p(x1:T | w)
≥ E(z̃2:T ,ỹ1:T )∼qZ,Y T∑ t=1 log p(xt | ỹt,w)
+ ∫ · · · ∫ y1,z2:T q(y1, z2:T | x1:T ) log p(y1, z2:T ) q(y1, z2:T | x1:T ) dz2:T dy1
(11)
= E(z̃2:T ,ỹ1:T )∼qZ,Y T∑ t=1 log p(xt | ỹt,w)−DKL ( q(y1 | x1:T ) ∥∥ p(y1)) + Eỹ1∼q(y1 | x1:T ) [∫ · · · ∫ z2:T q(z2:T | x1:T , ỹ1) log p(z2:T | ỹ1) q(z2:T | x1:T , ỹ1) dz2:T ] (12) = E(z̃2:T ,ỹ1:T )∼qZ,Y T∑ t=1 log p(xt | ỹt,w)−DKL ( q(y1 | x1:k)
∥∥ p(y1)) + Eỹ1∼q(y1 | x1:k) [∫ · · · ∫ z2:T q(z2:T | x1:T , ỹ1) log p(z2:T | ỹ1) q(z2:T | x1:T , ỹ1) dz2:T ] (13)
= E(z̃2:T ,ỹ1:T )∼qZ,Y T∑ t=1 log p(xt | ỹt,w)−DKL ( q(y1 | x1:k) ∥∥ p(y1)) + Eỹ1∼q(y1 | x1:k) ∫ · · · ∫ z2:T T∏ t=2 q(zt | x1:t) T∑ t=2 log p(zt | ỹ1, z2:t−1) q(zt | x1:t) dz2:T  (14) = E(z̃2:T ,ỹ1:T )∼qZ,Y T∑ t=1 log p(xt | ỹt,w)−DKL ( q(y1 | x1:k)
∥∥ p(y1)) − Eỹ1∼q(y1 | x1:k)DKL ( q(z2 | x1:t)
∥∥ p(z2 | ỹ1)) + Eỹ1∼q(y1 | x1:k)Ez̃2∼q(z2 | x1:2)∫ · · · ∫
z3:T T∏ t=3 q(zt | x1:t) T∑ t=3 log p(zt | y1, z̃2:t−1) q(zt | x1:t) dz3:T
, (15)
where:
• Equation (13) follows from the inference model of Equation (5), where y1 only depends on x1:k; • Equation (14) is obtained from the factorizations of Equations (4) and (5).
By iterating Equation (15)’s step on z3, . . . ,zT and factorizing all expectations, we obtain:
(16) log p(x1:T | w)
≥ E(z̃2:T ,ỹ1:T )∼qZ,Y T∑ t=1 log p(xt | ỹt,w)−DKL ( q(y1 | x1:k) ∥∥ p(y1)) − Eỹ1∼q(y1 | xc) ( Ez̃t∼q(zt | x1:t) )T t=2 T∑ t=2 DKL ( q(zt | x1:t) ∥∥ p(zt | ỹ1, z̃1:t−1)), (17)
and we finally retrieve Equation (6) by using the factorization of Equation (5): log p(x1:T | w)
≥ E(z̃2:T ,ỹ1:T )∼qZ,Y T∑ t=1 log p(xt | ỹt,w)−DKL ( q(y1 | x1:k) ∥∥ p(y1)) − E(z̃2:T ,ỹ1:T )∼qZ,Y
T∑ t=2 DKL ( q(zt | x1:t) ∥∥ p(zt | ỹt−1)). (18)

heading: B DATASETS DETAILS

heading: B.1 STOCHASTIC MOVING MNIST (SM-MNIST)
text: This dataset consists in one or two train MNIST digits (LeCun et al., 1998) of size 27× 27 moving linearly within a 64× 64 frame and randomly bounce against its border, sampling a new direction and velocity at each bounce (Denton & Fergus, 2018). We use the same settings as Denton & Fergus (2018), train all models on 15 timesteps and condition them at test time on 5 frames. Note that we adapted the dataset to sample more coherent bounces: the original dataset computes digit trajectories that are dependent on the chosen framerate, unlike our corrected version of the dataset. We consequently retrained SVG on this dataset, obtaining comparable results as those originally presented by Denton & Fergus (2018). Test data were produced by generating 5000 samples with a different digit for each sequence coming from the MNIST test set.

heading: B.2 KTH ACTION DATASET (KTH)
text: This dataset is composed of real-world 64× 64 videos of 25 people performing one of six actions (walking, jogging, running, boxing, handwaving and handclapping) in front of different backgrounds (Schüldt et al., 2004). Uncertainty lies in the appearance of subjects, the action they perform and how it is performed. The training set is formed with actions from 20 people, the remaining five being used for testing. Training is performed by sampling sub-sequences of size 20 in the train set. The test set is composed of 1000 randomly sampled sub-sequences of size 40.

heading: B.3 BAIR ROBOT PUSHING DATASET (BAIR)
text: This dataset contains 64× 64 videos of a Sawyer robotic arm pushing objects on a tabletop (Ebert et al., 2017). It is highly stochastic as the arm can change its direction at any moment. Training is performed on 12 frames and testing is done with two conditioning frames on the provided test set, consisting of 256 sequences of 30 frames.

heading: C TRAINING DETAILS

heading: C.1 SPECIFICATIONS
text: We used Python 3.7.4 and PyTorch 1.2.0 (Paszke et al., 2017) to implement our model. Each model was trained on a Nvidia GPUs with CUDA 10 in mixed-precision training with the help of Apex.2

heading: C.2 ARCHITECTURE
text: Encoder and decoder architecture. Both gθ and hφ are chosen to have different architectures depending on the dataset. We used the same architectures as in Denton & Fergus (2018): a DCGAN
2https://github.com/nvidia/apex.
discriminator and generator architecture (Radford et al., 2016) for Moving MNIST, and a VGG16 (Simonyan & Zisserman, 2015) architecture (mirrored for hφ) for BAIR and KTH. In both cases, the output of hφ (i.e., x̃) is a vector of size 128, and gθ and hφ weights are initialized using a centered normal distribution with a standard deviation of 0.02.
For the Moving MNIST dataset, the content variable w is obtained directly from x̃ and is thus a vector of size 128. For KTH and BAIR, we supplement this vectorial variable with skip connections from all layers of the encoder gθ that are then fed to the decoder hφ to handle complex backgrounds. For Moving MNIST, the number of frames k used to compute the content variable is 5; for KTH, it is 3; for BAIR, it is 2.
LSTM architecture. The LSTM used for all datasets has a single layer of LSTM cells with a hidden state size of 256.
MLP architecture. All MLPs used in inference (with parameters φ) have three linear layers with hidden size 256 and leaky ReLU activations. All MLPs used in the forward model (with parameters θ) have four linear layers with hidden size 512 and leaky ReLU activations. Weights of fθ, in particular, are orthogonally initialized with a gain of 1.41, while the other MLPs are initialized with default weight initialization of PyTorch.
Sizes of latent variables. The sizes of the latent variables in our model are the following: for Moving MNIST, y and z have size 20; for KTH and BAIR, y and z have size 50.
Euler step size All models but those trained on KTH are trained with ∆t = 1. Models on KTH are trained with ∆t = 12 .

heading: C.3 OPTIMIZATION
text: Loss function. All models are trained using the Adam optimizer (Kingma & Ba, 2015) with learning rate 3× 10−4 and λ = 1. The batch size for Moving MNIST and BAIR is chosen to be 128, and the batch size for KTH is chosen to be 100.
Following (Higgins et al., 2017), we use β = 1 (cf. Equation (7)), except for the Moving MNIST dataset where the β factor in front of the KL on z (last term of Equation (6)) is equal to 2.
Variance of the observation. The variance ν used in the observation probability distribution G ( gθ(y) ) = N ( gθ(y), νI ) is chosen as follows:
• for Moving MNIST, ν = 1;
• for KTH, ν = 4× 10−2;
• for BAIR, ν = 12 .
Number of optimization steps. The number of optimization steps is the following for the different datasets:
• Moving MNIST (stochastic): 1 000 000 steps with additional 100 000 steps where the learning rated is linearly decreased to 0;
• Moving MNIST (deterministic): 800 000 steps with additional 100 000 steps where the learning rated is linearly decreased to 0;
• KTH: 200 000 steps, the final model being chosen among several checkpoints as the one having the best evaluation score (which differs from the test score as we extract from the train set an evaluation set);
• BAIR: 250 000 steps, the final model is chosen as for KTH.

heading: D ADDITIONAL NUMERICAL RESULTS
text: Tables 1 to 3 present, respectively, numerical results for PSNR, SSIM and LPIPS averaged over all time steps for our methods and considered baselines on the SM-MNIST, KTH and BAIR datasets, corresponding to Figures 2, 4 and 6.

heading: E PENDULUM EXPERIMENTS
text: We test the ability of our model to model the dynamics of a common dataset used in the literature of state-space models (Karl et al., 2017; Fraccaro et al., 2017), Pendulum (Karl et al., 2017). It consists of noisy observations of a dynamic torque-controlled pendulum; it is stochastic as the information of this control is not available. We test our model, without the content variable w, in the same setting as DVBF (Karl et al., 2017) and KVAE (Fraccaro et al., 2017) and report the corresponding ELBO scores in Table 4. The encoders and decoders for all methods are MLPs.
Our model outperforms DVBF and is merely beaten by KVAE. This can be explained by the nature of the KVAE model, whose sequential model is not learned using a VAE but a Kalman filter allowing
exact inference in the latent space. On the contrary, DVBF is learned, like our model, by a sequential VAE, and is thus much closer to our model than KVAE. This result then shows that the dynamic model that we chose in the context of sequential VAEs is more adapted on this dataset than the one of DVBF, and achieve results close to a method taking advantage of exact inference using adapted tools such as Kalman filters.
F INFLUENCE OF THE EULER STEP SIZE
Table 5 details the numerical results of our model trained on BAIR with ∆t = 1 and tested with different values of ∆t. It shows that, when refining the Euler approximation, our model can improve its performance in a setting that is unseen during training. Results stabilize when ∆t is small enough, showing that the model is close to the continuous limit.
Tables 6 and 7 details the numerical results of our model trained on KTH with, respectively, ∆t = 1 and ∆t = 12 , and tested with different values of ∆t. They show that if ∆t is chosen too high when training (here, ∆t = 1), the model drops in performance when refining the Euler approximation. We assume that this phenomenon arises because the Euler approximation used in training is too rough, making the model adapt to a very discretized dynamic that cannot be transferred to smaller Euler step sizes. Indeed, when training with smaller step size, (here, ∆t = 12 ), results in the training settings are equivalent while results obtained with a lower ∆t are now much closer, if not equivalent, to the nominal ones. This shows that the model learned a continuous dynamic if learned with a small enough step size.
Note that the loss of performance using a higher ∆t in testing than in training, like in Table 7, is expected as it corresponds to loosening the Euler approximation compared to training. However, even in this adversarial setting, our model maintains state-of-the-art results, demonstrating the quality of the learned dynamic as it can be further discretized if needed at the cost of a reasonable drop in performance.

heading: G AUTOREGRESSIVITY AND IMPACT OF ENCODER AND DECODER ARCHITECTURE
text: Figure 10 exposes the numerical results on KTH of our model and SVG for different choices of architectures: DCGAN and VGG.
Since DCGAN is a less powerful architecture than VGG, results of each method with VGG are expectedly better than those of the same method with DCGAN. Moreover, our model outperforms SVG for any fixed choice of encoder and decoder architecture, which is coherent with Figure 4.
We observe, however, that the difference between a method using VGG and its DCGAN counterpart differs depending on the model. Ours shows more robustness to changing of encoder and decoder architecture, as it loses much less performance than SVG when switching to a less powerful architecture. Indeed, while the difference in LPIPS is similar for both models (as expected from a score evaluating the realism of produced frames), the loss of SVG is significantly larger than our loss in terms of SSIM, and in particular PNSR. This shows that reducing the capacity of the encoders and decoders of SVG not only hurts its ability to produce realistic frames as expected but also substantially lowers its ability to learn a good dynamic. We assume that this phenomenon is caused by the autoregressive nature of SVG, which makes it reliant of the performance of its encoders and decoders. This supports our motivation to propose a non-autoregressive model for stochastic video prediction.

heading: H ADDITIONAL SAMPLES
text: This section includes some additional samples corresponding to experiments described in Section 4.

heading: H.1 STOCHASTIC MOVING MNIST
text: We present in Figures 11 to 14 additional samples from SVG and our model on SM-MNIST.
In particular, Figure 13 shows SVG changing a digit shape in the course of a prediction even though it does not cross another digit, whereas ours maintain the digit shape. We assume that this advantage of ours comes from the latent nature of the dynamic of our model and the use in our of a static content variable that is prevented from containing temporal information. Indeed, even when the best sample
from our model is not close from the ground truth of the dataset, like in Figure 14, the shapes of the digits are still maintained by our model.

heading: H.2 KTH
text: We present in Figures 15 to 19 additional samples from SV2P, SVG, SAVP and our model on KTH, with additional insights.

heading: H.3 BAIR
text: We present in Figures 20 to 22 additional samples from SV2P, SVG, SAVP and our model on BAIR, with additional insights.

heading: H.4 OVERSAMPLING
text: We present in Figure 23 additional examples of video generation at a doubled frame rate by our model.

heading: H.5 CONTENT SWAP
text: We present in Figures 24 to 28 additional examples of content swap as in Figure 8.
H.6 INTERPOLATION IN THE LATENT SPACE
We present in Figures 29 and 30 additional examples of interpolation in the latent space between two trajectories.

Review summaries:
Contributions: this submission proposes a video pixel generation framework with the goal to decouple visual appearance and dynamics. The latent dynamics are modeled with a latent residual dynamics model. Empirical evaluations on moving MNIST show that the proposed residual dynamics model outperform MLP or GRU. On more challenging KTH and BAIR datasets, the proposed method achieves on par or better quantitative performance with previous methods, and have nice qualitative results on content "swap" and dynamics interpolation.