Title: STOCHASTIC LATENT RESIDUAL VIDEO PREDICTION

Abstract: Video prediction is a challenging task: models have to account for the inherent uncertainty of the future. Most works in the literature are based on stochastic imageautoregressive recurrent networks, raising several performance and applicability issues. An alternative is to use fully latent temporal models which untie frame synthesis and dynamics. However, no such model for video prediction has been proposed in the literature yet, due to design and training difficulties. In this paper, we overcome these difficulties by introducing a novel stochastic temporal model. It is based on residual updates of a latent state, motivated by discretization schemes of differential equations. This first-order principle naturally models video dynamics as it allows our simpler, lightweight, interpretable, latent model to outperform prior state-of-the-art methods on challenging datasets.

1. INTRODUCTION:

Being able to predict the future of a video from a few conditioning frames in a self-supervised manner has many applications in fields such as reinforcement learning (Gregor et al., 2019) or robotics (Babaeizadeh et al., 2018) . More generally, it challenges the ability of a model to capture visual and dynamic representations of the world. Video prediction has received a lot of attention from the computer vision community. However, most proposed methods are deterministic, reducing their ability to capture video dynamics, which are intrinsically stochastic (Denton & Fergus, 2018) .

Stochastic video prediction is a challenging task which has been tackled by recent works. Most state-of-the-art approaches are based on image-autoregressive models (Denton & Fergus, 2018; Babaeizadeh et al., 2018) , built around Recurrent Neural Networks (RNNs), where each generated frame is fed back to the model to produce the next frame. However, performances of their temporal models innately depend on the capacity of their encoder and decoder, as each generated frame has to be re-encoded in a latent space. Such autoregressive processes induce a high computational cost, and strongly tie the frame synthesis and temporal models, which may hurt the performance of the generation process and limit its applicability (Gregor et al., 2019; Rubanova et al., 2019) .

An alternative approach consists in separating the dynamic of the state representations from the generated frames, which are independently decoded from the latent space. In addition to removing the aforementioned link between frame synthesis and temporal dynamics, this is computationally appealing when coupled with a low-dimensional latent-space. Moreover, such models can be used to shape a complete representation of the state of a system, e.g. for reinforcement learning applications (Gregor et al., 2019) , and more interpretable than autoregressive models (Rubanova et al., 2019 ). Yet, these State-Space Models (SSMs) are more difficult to train as they require non-trivial latent state inference schemes (Krishnan et al., 2017) and a careful design of the dynamic model (Karl et al., 2017) . This leads most successful SSMs to only be evaluated on small or artificial toy tasks.

In this work, we propose a novel stochastic dynamic model for the task of video prediction which successfully leverages structural and computational advantages of SSMs that operate on low-dimensional latent spaces. The dynamic component determines the evolution through residual updates of the latent state, conditioned on learned stochastic variables. This formulation allows us to implement an efficient training strategy and process in an interpretable manner complex high-dimensional data such as videos. This residual principle can be linked to recent advances relating residual networks and Ordinary Differential Equations (ODEs) (Chen et al., 2018) . This interpretation opens new perspectives such as generating videos at different frame rates, as demonstrated in our experiments. Overall, this approach outperforms current state-of-the-art models on the task of stochastic video prediction, as demonstrated by comparisons with competitive baselines on representative benchmarks.

2. RELATED WORK:

Video synthesis covers a range of different tasks, such as video-to-video translation (Wang et al., 2018) , super-resolution (Caballero et al., 2017) , interpolation between frames (Jiang et al., 2018) , unconditonal generation (Tulyakov et al., 2018) , or video prediction, which is the focus of this paper.

Deterministic models. Inspired by prior sequence generation models using RNNs (Graves, 2013) , a number of video prediction methods (Srivastava et al., 2015; Villegas et al., 2017; Wichers et al., 2018) rely on LSTMs (Hochreiter & Schmidhuber, 1997) , or, like Ranzato et al. (2014) and Jia et al. (2016) , on derived networks such as ConvLSTMs (Shi et al., 2015) taking advantage of Convolutional Neural Networks (CNNs). Indeed, computer vision approaches are usually tailored to high-dimensional video sequences and propose domain-specific techniques as they often use pixel-level transformations and optical flow (Shi et al., 2015; Walker et al., 2015; Finn et al., 2016; Jia et al., 2016; Vondrick & Torralba, 2017; Liang et al., 2017; Liu et al., 2017; Lotter et al., 2017; Lu et al., 2017a; Fan et al., 2019) that help to produce high-quality predictions. Such predictions are, however, deterministic, thus hurting their performance as they fail to generate sharp long-term video frames (Babaeizadeh et al., 2018; Denton & Fergus, 2018) . Following Mathieu et al. (2016) , some works proposed to use an adversarial loss (Goodfellow et al., 2014) on the predictions of their model to sharpen the generated frames (Vondrick & Torralba, 2017; Liang et al., 2017; Lu et al., 2017a; Xu et al., 2018) . Nonetheless, adversarial losses are notoriously hard to train, and lead to mode collapse, preventing diversity of generations.

Stochastic and image-autoregressive models. Some approaches rely on exact likelihood maximization, using pixel-level autoregressive generation (van den Oord et al., 2016; Kalchbrenner et al., 2017) or normalizing flows through invertible transformations between the observation space and a latent space (Kingma & Dhariwal, 2018; Kumar et al., 2019) . However, they require careful design of complex temporal generation schemes manipulating high-dimensional data, thus inducing a prohibitive temporal generation cost. More efficient continuous models rely on Variational Auto-Encoders (VAEs) (Kingma & Welling, 2014; Rezende et al., 2014) for the inference of lowdimensional latent state variables. Except Xue et al. (2016) who learn a one-frame-ahead VAE, they model sequence stochasticity by incorporating a random latent variable per frame into a deterministic RNN-based image-autoregressive model. Babaeizadeh et al. (2018) integrate stochastic variables into the ConvLSTM architecture of Finn et al. (2016) . Concurrently with He et al. (2018) , Denton & Fergus (2018) , with Castrejon et al. (2019) in a follow-up, use a prior LSTM conditioned on previously generated frames in order to sample random variables that are fed to a predictor LSTM. Finally, Lee et al. (2018) combine the ConvLSTM architecture and this learned prior, adding an adversarial loss on the predicted videos to sharpen them at the cost of a diversity drop. Yet, all these methods are image-autoregressive, as they feed their predictions back into the latent space, thus tying the frame synthesis and temporal models and increasing their computational cost. Concurrently to our work, Minderer et al. (2019) propose to use the autoregressive VRNN model (Chung et al., 2015) on learned image key-points instead of raw frames. While this change could mitigate the aforementioned problems, the extent of such mitigation is unclear. We follow a complementary approach by directly proposing a dynamic model that is state-space and acts on a small latent state, tackling these issues.

State-space models. Many latent state-space models have been proposed for sequence modelization (Bayer & Osendorfer, 2014; Fraccaro et al., 2016; 2017; Krishnan et al., 2017; Karl et al., 2017; Hafner et al., 2019) , usually trained by deep Variational Inference (VI). These methods, which use locally linear temporal transition functions or RNN-based dynamics, are designed for and tested on low-dimensional data, as learning such models on complex data is challenging, or focus on control or planning tasks. In contrast, our fully latent method is the first one to be successfully applied to complex high-dimensional data such as videos, thanks to a temporal model based on residual updates of its latent state. It thus falls within the scope of a recent trend linking differential equations with neural networks (Lu et al., 2017b; Long et al., 2018) , leading to the integration of ODEs, that are seen as continuous residual networks, in neural network architectures (Chen et al., 2018) . However, the latter work and follow-ups (Rubanova et al., 2019; Yıldız et al., 2019) are either limited to low-dimensional data, prone to overfitting or unable to handle stochasticity within a sequence. Another line of works considers stochastic differential equations (SDEs) with neural networks (Ryder

y 1 y 2 y 3 x 1 x 2 x 3 z 2 z 3

(a) Generative model p. et al., 2018; De Brouwer et al., 2019) , but are limited to continuous Brownian noise, whereas video prediction additionally requires to model punctual stochastic events.

y 1 y 2 y 3 x 1 x 2 x 3 z 2 z 3 (b) Inference model q. x 1 x 2 x 1 x 2 µ θ , σ θ z 2 LSTM y 1 y 2 x 1 x 2 µ z φ , σ z φ w g θ g θ h φ h φ f θ µ θ , σ θ z 3 f θ µ z φ , σ z φ h φ x 3 x 3 y 3 g θ LSTM LSTM x 3 µ y φ , σ y φ q q p p q (c)

3. MODEL:

We consider the task of stochastic video prediction, consisting in approaching, given a number of conditioning video frames, the distribution of possible future frames given this conditioning.

3.1. LATENT RESIDUAL DYNAMIC MODEL:

Let x 1:T be a sequence of T video frames. We model their evolution by introducing latent variables y that are driven by a dynamic temporal model. Each frame x t is then generated from the corresponding latent state y t only, making the dynamics independent from the previously generated frames.

We propose to model the transition function of the latent dynamic of y with a stochastic residual network. State y t+1 is chosen to deterministically depend on the previous state y t , conditionally to an auxiliary random variable z t+1 . These auxiliary variables encapsulate the randomness of the video dynamics. They have a learned factorized Gaussian prior that depends on the previous state only. The model is depicted in Figure 1a , and defined as follows:

EQUATION

where µ θ , σ θ , f θ and g θ are neural networks, and G g θ (y t ) is a probability distribution parameterized by g θ (y t ). In our experiments, G is a normal distribution with fixed diagonal variance and mean g θ (y t ). Note that y 1 is assumed to have a standard Gaussian prior, and, in our VAE setting, will be inferred from conditioning frames for the prediction task, as shown in Section 3.3.

The residual update rule takes inspiration in the Euler discretization scheme of differential equations. The state of the system y t is updated by its first-order movement, i.e., the residual f θ (y t , z t+1 ).

Compared to a regular RNN, this simple principle makes our temporal model lighter and more interpretable. Equation (1), however, differs from a discretized ODE because of the introduction of the stochastic discrete-time variables z. Nonetheless, we propose to allow the Euler step size ∆t to be smaller than 1, as a way to make the temporal model closer to a continuous dynamics. The updated dynamics becomes, with 1 ∆t ∈ N to synchronize the step size with the video frame rate:

y t+∆t = y t + ∆t • f θ y t , z t +1 .

(2)

For this formulation, the auxiliary variable z t is kept constant between two integer time steps. Note that a different ∆t can be used during training or testing. This allows our model to generate videos at an arbitrary frame rate since each intermediate latent state can be decoded in the observation space. This ability enables us to observe the quality of the learned dynamic as well as challenge its ODE inspiration by testing its generalization to the continuous limit in Section 4. In the following, we consider ∆t as a hyperparameter. For the sake of clarity, we consider that ∆t = 1 in the following; generalizing to smaller ∆t is straightforward as Figure 1a remains unchanged.

3.2. CONTENT VARIABLE:

Some components of video sequences can be static, such as the background or shapes of moving objects. They may not impact the dynamics; we therefore model them separately, in the same spirit as Denton & Birodkar (2017) and Yingzhen & Mandt (2018) . We compute a content variable w that remains constant throughout the whole generation process and is fed together with y t into the frame generator. It enables the dynamical part of the model to focus only on movement, hence being lighter and more stable. Moreover, it allows us to leverage architectural advances in neural networks, such as skip connections (Ronneberger et al., 2015) , to produce more realistic frames.

This content variable is a deterministic function c ψ of a fixed number k < T of frames x (k) c :

EQUATION

c are the last k conditioning frames (usually between 2 and 5). This content variable is not endowed with any probabilistic prior, contrary to the dynamic variables y and z. Hence, the information it contains is not constrained in the loss function (see Section 3.3), but only architecturally. To prevent temporal information from leaking in w, we propose to uniformly sample these k frames within x 1:T during training. We also design c ψ as a permutation-invariant function (Zaheer et al., 2017) , which is done by using an MLP fed with the sum of individual frame representations, similarly to Santoro et al. (2017) . This absence of prior and its architectural constraint allows w to contain as much non-temporal information as possible, while preventing it from containing dynamic information. On the other hand, due to their strong standard Gaussian priors, y and z are encouraged to discard unnecessary information. Therefore, y and z should only contain temporal information that could not be captured by w.

Note that this content variable can be removed from our model, yielding a more classical deep state-space model. An experiment in this setting is presented in Appendix E.

3.3. VARIATIONAL INFERENCE AND ARCHITECTURE:

Following the generative process depicted in Figure 1a , the conditional joint probability of the full model, given a content variable w, can be written as:

p(x 1:T , z 2:T , y 1:T | w) = p(y 1 ) T -1 t=1 p(z t+1 | y t )p(y t+1 | y t , z t+1 ) T t=1 p(x t | y t , w), (4)

where p(y t+1 | y t , z t+1 ) = δ y t + f θ (y t , z t+1 ) -y t+1 and δ is the Dirac delta function centered on 0, according to the expression of y t+1 in Equation (1). Thus, in order to optimize the likelihood of the observed videos p(x 1:T | w), we need to infer latent variables y 1 and z 2:T . This is done by deep Variational Inference using the inference model parameterized by φ and shown in Figure 1b , which comes down to consider a variational distribution q Z,Y defined and factorized as follows:

q Z,Y q(z 2:T , y 1:T | x 1:T , w) = q(y 1 | x 1:k ) T t=2 q(z t | x 1:t )δ y t-1 + f θ (y t-1 , z t ) -y t . (5)

This yields the following evidence lower bound (ELBO), whose full derivation is given in Appendix A:

log p(x 1:T | w) ≥ E ( z 2:T , y 1:T )∼q Z,Y T t=1 log p(x t | y t , w) -D KL q(y 1 | x 1:k ) p(y 1 ) -E ( z 2:T , y 1:T )∼q Z,Y T t=2 D KL q(z t | x 1:t ) p(z t | y t-1 ) L(x 1:T ; w, θ, φ). (6)

The sum of KL divergence expectations implies to consider the full past sequence of inferred states for each time step, due to the dependence on conditionally deterministic variables y 2:T . However, optimizing L(x 1:T ; w, θ, φ) with respect to model parameters θ and variational parameters φ can be done efficiently by sampling a single full sequence of states from q Z,Y per example, and computing gradients by backpropagation (Rumelhart et al., 1988 ) trough all inferred variables, using the reparametrization trick (Kingma & Welling, 2014; Rezende et al., 2014) . We classically choose q(y 1 | x 1:k ) and q(z t | x 1:t ) to be factorized Gaussian so that all KLDs can be computed analytically.

We include an 2 regularization term on residuals f θ which stabilizes the temporal dynamics of the residual network, as noted by Behrmann et al. (2019) and Rousseau et al. (2019) . Given a set of videos X , the full optimization problem, where L is defined as in Equation ( 6), is then given as:

arg max θ,φ,ψ x∈X   E x (k) c L x 1:T ; c ψ x (k) c , θ, φ -λ • E (z 2:T ,y 1:T )∼q Z,Y T t=2 f θ (y t-1 , z t ) 2   . (7)

Figure 1c depicts the full architecture of our temporal model, corresponding to how the model is applied during testing. The first latent variables are inferred with the conditioning framed and are then predicted with the dynamic model. In contrast, during training, each frame of the input sequence is considered for inference, which is done as follows. Firstly, each frame x t is independently encoded into a vector-valued representation x t , with x t = h φ (x t ). y 1 is then inferred using an MLP on the first k encoded frames x 1:k . Each z t is inferred in a feed-forward fashion with an LSTM on the encoded frames. Inferring z this way experimentally performs better than, e.g., inferring them from the whole sequence x 1:T ; we hypothesize that this follows from the fact that this filtering scheme is closer to the prediction setting, where the future is not available.

4. EXPERIMENTS:

This section exposes the experimental results of our method on three standard stochastic video prediction datasets. 1 We compare our method with state-of-the-art baselines on stochastic video prediction. Furthermore, we qualitatively study the dynamics and latent space learned by our model.

Training details are described in Appendix C.

The stochastic nature and novelty of the task of stochastic video prediction make it challenging to evaluate (Lee et al., 2018) : since videos and models are stochastic, comparing the ground truth and a predicted video is not adequate. We thus adopt the common approach (Denton & Fergus, 2018; Lee et al., 2018) consisting in, for each test sequence, sampling from the tested model a given number (here, 100) of possible futures and reporting the best performing sample against the true video. We report this discrepancy for three commonly used metrics: Peak Signal-to-Noise Ratio (PSNR, higher is better), Structured Similarity (SSIM, higher is better), and Learned Perceptual Image Patch Similarity (LPIPS, lower is better) (Zhang et al., 2018) . PSNR tends to promote blurry predictions, as it is a pixel-level measure derived from the 2 distance, but greatly penalizes errors in predicted positions of objects in the scenes. SSIM is a similarity metric between image patches. LPIPS is a learned distance between activations of deep CNNs trained on image classification tasks, and have been shown to better correlate with human judgment on real images. While these three metrics are computed frame-wise, the recently proposed Fréchet Video Distance (FVD, lower is better) (Unterthiner et al., 2018) aims at directly comparing the distribution of predicted videos with the ground truth distribution through the representations computed by a deep CNN trained on action recognition tasks. It has been shown, independently from LPIPS, to better correlate with human judgment than PSNR and SSIM. We treat all four metrics as complementary, as they capture different modalities. PSNR challenges the dynamics of the predicted videos, while SSIM rather compares local frame patches but loses some dynamics information. LPIPS and FVD both measure the realism of the predictions compared to the ground truth. FVD considers videos as a whole, making it more capable of detecting temporal inconsistencies. On the other hand, the frame-wise LPIPS metric penalizes more the temporal drifts of videos, since it directly compares each predicted and ground truth frame.

We present experimental results on a simulated dataset and two real-world datasets, that we briefly present in the following and detail in Appendix B. The corresponding numerical results can be found in Appendix D. For the sake of concision, we only display a handful of qualitative samples in this section, and refer to Appendix H for additional samples. We compare our model against several state-of-the-art models: SV2P (Babaeizadeh et al., 2018) , SVG (Denton & Fergus, 2018) and SAVP (Lee et al., 2018) . All baseline results were obtained with pretrained models released by the authors. Note that we use the same neural architecture as SVG for our encoders and decoders in order to perform fair comparisons with this method, which is the closest to ours among the state of the art. Unless specified otherwise, our model is tested with the same ∆t as in training (see Equation ( 2)).

Stochastic Moving MNIST (SM-MNIST).:

This dataset consists of one or two MNIST digits (LeCun et al., 1998) moving linearly and randomly bouncing on walls with new direction and velocity sampled randomly at each bounce (Denton & Fergus, 2018) . As SV2P and SAVP were not tested on this dataset (in particular, with no pretrain model, code or hyperparameters), we only report scores for SVG as state-of-the-art model on SM-MNIST.

Figure 2a shows quantitative results with two digits. Our model outperforms SVG on both PSNR and SSIM; LPIPS and FVD are not reported as they are not relevant for this synthetic task. Decoupling dynamics from image synthesis allows our method to maintain temporal consistency despite highuncertainty frames where crossing digits become indistinguishable. For instance in Figure 3 , the digits shape changes after they cross in the SVG prediction, while our model predicts the correct digits. To evaluate the predictive ability on a longer horizon, we perform experiments on the classic deterministic version of the dataset (Srivastava et al., 2015) . We show the results up to t + 95 in Figure 5 : Conditioning frames and corresponding ground truth, best samples from SVG, SAVP and our method, and worst and random samples from our method, for an example of the KTH dataset. Samples are chosen according to their LPIPS with respect to the ground truth. SVG fails to make a person appear unlike SAVP and our model, and the latter better predicts the pose of the subject.

Figure 2b . We can see that our model better captures the dynamics of the problem compared to SVG as its performance decreases significantly less, even at a long-term horizon.

We also compare to two alternative versions of our model in Figure 2 , where the residual dynamic function is replaced by an MLP or a GRU network (Cho et al., 2014) . Our residual model outperforms both versions on the stochastic, and especially on the deterministic version of the dataset, showing its intrinsic advantage at modeling dynamics. Finally, on the deterministic version of Moving MNIST, we compare to an alternative where z is entirely removed, resulting in a temporal model very close to the one presented in Chen et al. ( 2018). The loss of performance of this alternative model is significant, especially in SSIM, showing that our stochastic residual model offers a substantial advantage even when used in a deterministic environment.

KTH Action dataset (KTH).:

This dataset is composed of real-world videos of people performing a single action per video in front of different backgrounds (Schüldt et al., 2004) . Uncertainty lies in the appearance of subjects, the actions they perform and how they are performed.

We outperform on this dataset every considered baseline for each metric, as depicted in Figure 4 and Table 2 . In some videos, the subject only appears after the conditioning frames, requiring the model to sample the moment and location of the subject appearance, as well as its action. This critical case is illustrated in Figure 5 . There, SVG fails to even generate a moving person; only SAVP and our model manage to do so, and our best sample is closer to the subject's poses compared to SAVP. Moreover, the worst and a random sample of our model demonstrate that it captures the diversity of the dataset by making a person appear at different time steps and with different speeds. An additional experiment on this dataset is included in Appendix G, studying the influence of the encoder and decoder architecture on SVG and our model. Finally, Table 2 compares our method to its MLP and GRU alternative versions, leading to two conclusions. Firstly, it confirms the structural advantage of residual dynamics observed on Moving MNIST. On one hand, MLP better captures dynamics than GRU on KTH according to PSNR and SSIM, but loses in terms of realism according to LPIPS and FVD. On the other hand, the residual version shows a slight dynamics improvement with respect to both MLP and GRU, while substantially pushing further prediction realism. Secondly, all three versions of our model (residual, MLP, GRU) outperform prior methods. Therefore, this improvement is due to their common inference method, latent nature and content variable, strengthening our motivation to propose a non-autoregressive model.

BAIR robot pushing dataset (BAIR). This dataset contains videos of a Sawyer robotic arm pushing objects on a tabletop (Ebert et al., 2017) . It is highly stochastic as the arm can change its direction at any moment. We achieve similar or better results compared to state-of-the-art models, as Figure 6 and Table 3 shows, and second-best PSNR behind SV2P, but the latter produces very blurry samples, which can be seen in Appendix H, yielding prohibitive LPIPS and FVD scores. In contrast, we achieve the highest SSIM overall, as well as state-of-the-art LPIPS and competitive FVD among these models. Note that we could not add VideoFlow to our experiments, due to the unavailability of pretrained models and numerical results. However, compared to PSNR, SSIM and LPIPS results reported by Kumar et al. (2019) for BAIR (the only tested dataset and metrics in their paper), our model appears to behave better than VideoFlow, which is on par with SAVP on these metrics.

Varying frame rate in testing. We challenge the ability of our model to use a different Euler step size than the one used in training (see Equation ( 2)). Figures 4 and 6 include corresponding results with a halved ∆t. Prediction performances remain stable while generating twice as many frames (cf. Appendix F for further discussion). Our model is thus robust to the refinement of the Euler approximation, showing the quality of the learned dynamic which is close to continuous. In particular, this shows that our model learned a dynamic driven by a piecewise ODE, i.e., the learned dynamic of each interval between two consecutive frames is an ODE, as a constant z is given on such interval. This can be used to generate frames at a higher frame rate than the training videos without supervision. We show in Figure 7 The generated video keeps the same background as the bottom left frames, while the robotic arm moves accordingly to the top frames.

Ref 1 Ref 2 Rec 1:

Rec 2 Interpolation Time Figure 9 : From left to right, x s , x s (reconstruction of x s by the VAE of our model), results of the interpolation in the latent space between x s and x t , x t and x t . Each trajectory is materialized in shades of grey in the frames. from the BAIR test set. While x s is used for extracting our content variable w s , dynamic states y t are inferred with our model from x t . New frame sequences x are finally generated from the fusion of the content vector and the dynamics. This results in a content corresponding to the first sequence x s while moving according to the dynamics of the second sequence x t , as observed in Figure 8 . More samples for BAIR and KTH can be seen in Appendix H.

Interpolation of dynamics. Our state-space structure allows us to learn semantic representations in y t . To highlight this feature, we test whether two Moving MNIST trajectories can be interpolated by linearly interpolating their inferred latent initial conditions. We begin by generating two trajectories x s and x t of a single moving digit. We infer their respective latent initial conditions y s 1 and y t 1 . We then use our model to generate frame sequences from latent initial conditions linearly interpolated between y s 1 and y t 1 . If it learned a meaningful latent space, the resulting trajectories should also be smooth interpolations between the directions of reference trajectories x s and x t , and this is what we observe in Figure 9 . Additional examples can be found in Appendix H.

5. CONCLUSION:

We introduce a novel dynamic latent model for stochastic video prediction which, unlike prior imageautoregressive models, decouples frame synthesis and dynamics. This temporal model is based on residual updates of a small latent state that is showed to perform better than RNN-based models. This endows our method with several desirable properties, such as temporal efficiency and latent space interpretability. We experimentally demonstrate the performance and advantages of the proposed model, which outperforms prior state-of-the-art methods for stochastic video prediction. This work is, to the best of our knowledge, the first to propose a latent dynamic model scaling for video prediction. The proposed model is also novel with respect to the recent line of work dealing with neural networks and ODEs for temporal modeling; it is the first such residual model to scale to complex stochastic data such as videos.

We believe that the general principles of our model (state-space, residual dynamic, static content variable) can be generally applied to other models as well. Interesting future works include replacing the VRNN model of Minderer et al. (2019) in order to model the evolution of key-points, or leveraging the state-space nature of our model in model-based reinforcement learning.

A EVIDENCE LOWER BOUND:

We develop in this section the computations of the variational lower bound for the proposed model.

Using the original variational lower bound of Kingma & Welling (2014) in Equation ( 8):

EQUATION

)

where:

• Equation ( 9) is given by the forward and inference models factorizing p and q in Equations ( 4) and ( 5) and illustrated by, respectively, Figures 1a and 1b: the z variables and y 1 are independent from w to p and q; the y 2:T variables are deterministic functions of y 1 and z 2:T with respect to p and q;

• Equation ( 10) results from the factorization of p(x 1:T | y 1:T , z 1:T , w) in Equation ( 4).

From there, by using the integral formulation of D KL :

EQUATION

where:

• Equation ( 13) follows from the inference model of Equation ( 5), where y 1 only depends on x 1:k ;

• Equation ( 14) is obtained from the factorizations of Equations ( 4) and ( 5).

By iterating Equation ( 15)'s step on z 3 , . . . , z T and factorizing all expectations, we obtain:

(16) log p(x 1:T | w) ≥ E ( z 2:T , y 1:T )∼q Z,Y T t=1 log p(x t | y t , w) -D KL q(y 1 | x 1:k ) p(y 1 ) -E y1∼q(y1 | xc) E zt∼q(zt | x1:t) T t=2 T t=2 D KL q(z t | x 1:t ) p(z t | y 1 , z 1:t-1 ) , (17)

and we finally retrieve Equation ( 6) by using the factorization of Equation ( 5):

EQUATION

B DATASETS DETAILS

B.1 STOCHASTIC MOVING MNIST (SM-MNIST):

This dataset consists in one or two train MNIST digits (LeCun et al., 1998) of size 27 × 27 moving linearly within a 64 × 64 frame and randomly bounce against its border, sampling a new direction and velocity at each bounce (Denton & Fergus, 2018) . We use the same settings as Denton & Fergus (2018) , train all models on 15 timesteps and condition them at test time on 5 frames. Note that we adapted the dataset to sample more coherent bounces: the original dataset computes digit trajectories that are dependent on the chosen framerate, unlike our corrected version of the dataset. We consequently retrained SVG on this dataset, obtaining comparable results as those originally presented by Denton & Fergus (2018) . Test data were produced by generating 5000 samples with a different digit for each sequence coming from the MNIST test set.

B.2 KTH ACTION DATASET (KTH):

This dataset is composed of real-world 64 × 64 videos of 25 people performing one of six actions (walking, jogging, running, boxing, handwaving and handclapping) in front of different backgrounds (Schüldt et al., 2004) . Uncertainty lies in the appearance of subjects, the action they perform and how it is performed. The training set is formed with actions from 20 people, the remaining five being used for testing. Training is performed by sampling sub-sequences of size 20 in the train set. The test set is composed of 1000 randomly sampled sub-sequences of size 40.

B.3 BAIR ROBOT PUSHING DATASET (BAIR):

This dataset contains 64 × 64 videos of a Sawyer robotic arm pushing objects on a tabletop (Ebert et al., 2017) . It is highly stochastic as the arm can change its direction at any moment. Training is performed on 12 frames and testing is done with two conditioning frames on the provided test set, consisting of 256 sequences of 30 frames.

C TRAINING DETAILS C.1 SPECIFICATIONS:

We used Python 3.7.4 and PyTorch 1.2.0 (Paszke et al., 2017) to implement our model. Each model was trained on a Nvidia GPUs with CUDA 10 in mixed-precision training with the help of Apex.2 

C.2 ARCHITECTURE:

Encoder and decoder architecture. Both g θ and h φ are chosen to have different architectures depending on the dataset. We used the same architectures as in Denton & Fergus (2018) : a DCGAN discriminator and generator architecture (Radford et al., 2016) for Moving MNIST, and a VGG16 (Simonyan & Zisserman, 2015) architecture (mirrored for h φ ) for BAIR and KTH. In both cases, the output of h φ (i.e., x) is a vector of size 128, and g θ and h φ weights are initialized using a centered normal distribution with a standard deviation of 0.02.

For the Moving MNIST dataset, the content variable w is obtained directly from x and is thus a vector of size 128. For KTH and BAIR, we supplement this vectorial variable with skip connections from all layers of the encoder g θ that are then fed to the decoder h φ to handle complex backgrounds. For Moving MNIST, the number of frames k used to compute the content variable is 5; for KTH, it is 3; for BAIR, it is 2.

LSTM architecture. The LSTM used for all datasets has a single layer of LSTM cells with a hidden state size of 256.

MLP architecture. All MLPs used in inference (with parameters φ) have three linear layers with hidden size 256 and leaky ReLU activations. All MLPs used in the forward model (with parameters θ) have four linear layers with hidden size 512 and leaky ReLU activations. Weights of f θ , in particular, are orthogonally initialized with a gain of 1.41, while the other MLPs are initialized with default weight initialization of PyTorch.

Sizes of latent variables. The sizes of the latent variables in our model are the following: for Moving MNIST, y and z have size 20; for KTH and BAIR, y and z have size 50.

Euler step size All models but those trained on KTH are trained with ∆t = 1. Models on KTH are trained with ∆t = 1 2 .

C.3 OPTIMIZATION:

Loss function. All models are trained using the Adam optimizer (Kingma & Ba, 2015) with learning rate 3 × 10 -4 and λ = 1. The batch size for Moving MNIST and BAIR is chosen to be 128, and the batch size for KTH is chosen to be 100.

Following (Higgins et al., 2017) , we use β = 1 (cf. Equation ( 7)), except for the Moving MNIST dataset where the β factor in front of the KL on z (last term of Equation ( 6)) is equal to 2.

Variance of the observation. The variance ν used in the observation probability distribution G g θ (y) = N g θ (y), νI is chosen as follows:

• for Moving MNIST, ν = 1;

• for KTH, ν = 4 × 10 -2 ;

• for BAIR, ν = 1 2 .

Number of optimization steps. The number of optimization steps is the following for the different datasets:

• Moving MNIST (stochastic): 1 000 000 steps with additional 100 000 steps where the learning rated is linearly decreased to 0;

• Moving MNIST (deterministic): 800 000 steps with additional 100 000 steps where the learning rated is linearly decreased to 0;

• KTH: 200 000 steps, the final model being chosen among several checkpoints as the one having the best evaluation score (which differs from the test score as we extract from the train set an evaluation set);

• BAIR: 250 000 steps, the final model is chosen as for KTH. 

D ADDITIONAL NUMERICAL RESULTS:

Tables 1 to 3 present, respectively, numerical results for PSNR, SSIM and LPIPS averaged over all time steps for our methods and considered baselines on the SM-MNIST, KTH and BAIR datasets, corresponding to Figures 2, 4 and 6 .

E PENDULUM EXPERIMENTS:

We test the ability of our model to model the dynamics of a common dataset used in the literature of state-space models (Karl et al., 2017; Fraccaro et al., 2017) , Pendulum (Karl et al., 2017) . It consists of noisy observations of a dynamic torque-controlled pendulum; it is stochastic as the information of this control is not available. We test our model, without the content variable w, in the same setting as DVBF (Karl et al., 2017) and KVAE (Fraccaro et al., 2017) and report the corresponding ELBO scores in Table 4 . The encoders and decoders for all methods are MLPs.

Our model outperforms DVBF and is merely beaten by KVAE. This can be explained by the nature of the KVAE model, whose sequential model is not learned using a VAE but a Kalman filter allowing Table 3 : Numerical results (mean and 95%-confidence interval, when relevant) with respect to PSNR, SSIM, LPIPS, and FVD for tested methods on the BAIR dataset. Bold scores indicate the best performing method for each metric and, where appropriate, scores whose means lie in the confidence interval of the best performing method. 19.85 ± 0.46 0.8243 ± 0.0111 0.0591 ± 0.0048 exact inference in the latent space. On the contrary, DVBF is learned, like our model, by a sequential VAE, and is thus much closer to our model than KVAE. This result then shows that the dynamic model that we chose in the context of sequential VAEs is more adapted on this dataset than the one of DVBF, and achieve results close to a method taking advantage of exact inference using adapted tools such as Kalman filters.

Models:



F INFLUENCE OF THE EULER STEP SIZE:

Table 5 details the numerical results of our model trained on BAIR with ∆t = 1 and tested with different values of ∆t. It shows that, when refining the Euler approximation, our model can improve its performance in a setting that is unseen during training. Results stabilize when ∆t is small enough, showing that the model is close to the continuous limit.

Tables 6 and 7 details the numerical results of our model trained on KTH with, respectively, ∆t = 1 and ∆t = 1 2 , and tested with different values of ∆t. They show that if ∆t is chosen too high when training (here, ∆t = 1), the model drops in performance when refining the Euler approximation. We assume that this phenomenon arises because the Euler approximation used in training is too rough, making the model adapt to a very discretized dynamic that cannot be transferred to smaller Euler step sizes. Indeed, when training with smaller step size, (here, ∆t = 1 2 ), results in the training settings are equivalent while results obtained with a lower ∆t are now much closer, if not equivalent, to the nominal ones. This shows that the model learned a continuous dynamic if learned with a small enough step size.

Note that the loss of performance using a higher ∆t in testing than in training, like in Table 7 , is expected as it corresponds to loosening the Euler approximation compared to training. However, even in this adversarial setting, our model maintains state-of-the-art results, demonstrating the quality of the learned dynamic as it can be further discretized if needed at the cost of a reasonable drop in performance. Since DCGAN is a less powerful architecture than VGG, results of each method with VGG are expectedly better than those of the same method with DCGAN. Moreover, our model outperforms SVG for any fixed choice of encoder and decoder architecture, which is coherent with Figure 4 .

We observe, however, that the difference between a method using VGG and its DCGAN counterpart differs depending on the model. Ours shows more robustness to changing of encoder and decoder architecture, as it loses much less performance than SVG when switching to a less powerful architecture. Indeed, while the difference in LPIPS is similar for both models (as expected from a score evaluating the realism of produced frames), the loss of SVG is significantly larger than our loss in terms of SSIM, and in particular PNSR. This shows that reducing the capacity of the encoders and decoders of SVG not only hurts its ability to produce realistic frames as expected but also substantially lowers its ability to learn a good dynamic. We assume that this phenomenon is caused by the autoregressive nature of SVG, which makes it reliant of the performance of its encoders and decoders. This supports our motivation to propose a non-autoregressive model for stochastic video prediction.

H ADDITIONAL SAMPLES:

This section includes some additional samples corresponding to experiments described in Section 4.

H.1 STOCHASTIC MOVING MNIST:

We present in Figures 11 to 14 additional samples from SVG and our model on SM-MNIST.

In particular, Figure 13 shows SVG changing a digit shape in the course of a prediction even though it does not cross another digit, whereas ours maintain the digit shape. We assume that this advantage of ours comes from the latent nature of the dynamic of our model and the use in our of a static content variable that is prevented from containing temporal information. Indeed, even when the best sample Figure 13 : Additional samples for the SM-MNIST dataset (cf. Figure 11 ). SVG fails to maintain the shape of a digit, while ours is temporally coherent. Figure 15 : Conditioning frames and corresponding ground truth, best samples from SVG, SAVP and our method, and worst and random samples from our method, for an example of the KTH dataset. Samples are chosen according to their LPIPS with respect to the ground truth. On this specific task (clapping), all methods but SV2P (which produce blurry predictions) perform well, even though ours stays closer to the ground truth.

from our model is not close from the ground truth of the dataset, like in Figure 14 , the shapes of the digits are still maintained by our model. Figure 16 : Additonal samples for the KTH dataset (cf. Figure 15 ). In this example, the shadow of the subject is visible in the last conditioning frames, foreshadowing its appearance. This is a failure case for SVG and SAVP which only produce an indistinct shadow, whereas SAVP and our model make the subject appear. Yet, SAVP produces the wrong action and an inconsistent subject in its best sample, while ours is correct. 

:

Code, video samples, and datasets are available at https://sites.google.com/view/srvp/.

https://github.com/nvidia/apex.

Figures and Tables:

Model and inference architecture on a test sequence. The transparent block on the left depicts the prior, and those on the right correspond to the full inference performed at training time.

Figure 1: (a), (b) Proposed generative and inference models. Diamonds and circles represent, respectively, deterministic and stochastic states. (c) Corresponding architecture with two parts: inference on conditioning frames on the left, generation for extrapolation on the right. h θ and g θ are deep CNNs, and other named networks are Multilayer Perceptrons (MLPs).

Figure 2: Mean PSNR and SSIM scores with respect to t for all tested models on the SM-MNIST dataset, with their 95%-confidence intervals. Vertical bars mark the length of train sequences.

Figure 4: PSNR, SSIM and LPIPS scores with respect to t for all tested models on the KTH dataset.

Figure 6: PSNR, SSIM and LPIPS scores with respect to t for all tested models on the BAIR dataset.

(a) Cropped KTH sample. (b) Cropped BAIR sample.

Figure 7: Generation examples at doubled frame rate, using a halved ∆t compared to training. Frames including a bottom red dashed bar are intermediate frames.

Figure 8: Video (bottom right) generated from the dynamic latent state y inferred with a video (top) and the content variable w computed with the conditioning frames of another video (bottom left).The generated video keeps the same background as the bottom left frames, while the robotic arm moves accordingly to the top frames.

Figure 10: PNSR, SSIM and LPIPS scores with respect to t on the KTH dataset for SVG and our model with two choices of encoder and decoder architecture for each: DCGAN and VGG.

Figure11: Conditioning frames and corresponding ground truth and best samples with respect to PSNR from SVG and our method, and worst and random samples from our method, for an example of the SM-MNIST dataset.

Figure 14: Additional samples for the SM-MNIST dataset (cf. Figure 11). This example was chosen in the worst 1% test examples of our model with respect to PSNR. Despite this adversarial criterion, our model maintains temporal consistency as digits are not deformed in the course of the video.

Figure17: Additonal samples for the KTH dataset (cf. Figure15). This example is a failure case for each method: SV2P produce blurry frames, SVG and SAVP are not consistent (change of action or subject appearance in the video), and our model produces a ghost image at the end of the prediction on the worst sample only. Ground Truth SV2P SVG SAVP Ours (Best) Ours (Worst) Ours (Random)Figure18: Additonal samples for the KTH dataset (cf. Figure15). Our model is the only one to make a subject appear in the ground truth.

Figure 22: Additonal samples for the KTH dataset (cf. Figure 20).

Figure 22: Additonal samples for the KTH dataset (cf. Figure 20).

(a) KTH sample, centered on the subject. (b) BAIR sample, centered on the robot arm.

Figure 23: Generation examples at doubled frame rate, using a halved ∆t compared to training. Frames including a bottom red dashed bar are intermediate frames.

Numerical results (mean and 95%-confidence interval) for PSNR and SSIM for tested methods on the two-digits Moving MMNIST dataset. Bold scores indicate the best performing method and, where appropriate, scores whose means lie in the confidence interval of the best performing method.

Table content: <table><tr><td>Models</td><td/><td>Stochastic</td><td colspan="2">Deterministic</td></tr><tr><td/><td>PSNR</td><td>SSIM</td><td>PSNR</td><td>SSIM</td></tr><tr><td>SVG</td><td colspan="4">14.45 ± 0.06 0.7070 ± 0.0021 12.93 ± 0.05 0.6245 ± 0.0022</td></tr><tr><td>Ours</td><td colspan="4">16.90 ± 0.09 0.7789 ± 0.0025 16.49 ± 0.06 0.7808 ± 0.0020</td></tr><tr><td colspan="5">Ours -MLP 16.55 ± 0.09 0.7693 ± 0.0024 14.32 ± 0.06 0.6895 ± 0.0023</td></tr><tr><td colspan="5">Ours -GRU 15.81 ± 0.08 0.7463 ± 0.0023 13.16 ± 0.05 0.6318 ± 0.0022</td></tr></table>

Numerical results (mean and 95%-confidence interval, when relevant) for PSNR, SSIM, LPIPS, and FVD for tested methods on the KTH dataset. Bold scores indicate the best performing method for each metric and, where appropriate, scores whose means lie in the confidence interval of the best performing method. Ours -GRU 29.13 ± 0.38 0.8590 ± 0.0060 0.0790 ± 0.0039 237Ours -MLP 29.49 ± 0.38 0.8626 ± 0.0061 0.0825 ± 0.0042 290

Table content: <table><tr><td>Models</td><td>PSNR</td><td>SSIM</td><td>LPIPS</td><td>FVD</td></tr><tr><td>SV2P</td><td colspan="4">28.18 ± 0.39 0.8141 ± 0.0068 0.2049 ± 0.0080 636</td></tr><tr><td>SAVP</td><td colspan="4">26.51 ± 0.36 0.7560 ± 0.0083 0.1120 ± 0.0058 376</td></tr><tr><td>SVG</td><td colspan="4">26.99 ± 0.33 0.8291 ± 0.0074 0.1083 ± 0.0058 265</td></tr><tr><td>Ours</td><td colspan="4">29.69 ± 0.37 0.8697 ± 0.0057 0.0736 ± 0.0036 224</td></tr></table>

ELBO score for DVBF, KVAE and our model on the Pendulum dataset. The bold score indicates the best performing method.

Table content: <table><tr><td/><td>PSNR</td><td>SSIM</td><td>LPIPS</td><td>FVD</td></tr><tr><td>SV2P</td><td colspan="4">20.39 ± 0.42 0.8169 ± 0.0110 0.0912 ± 0.0063 955</td></tr><tr><td>SAVP</td><td colspan="4">18.44 ± 0.40 0.7886 ± 0.0117 0.0634 ± 0.0048 152</td></tr><tr><td>SVG</td><td colspan="4">18.95 ± 0.41 0.8057 ± 0.0116 0.0609 ± 0.0046 253</td></tr><tr><td>Ours</td><td colspan="4">19.64 ± 0.45 0.8211 ± 0.0110 0.0610 ± 0.0048 181</td></tr></table>

Numerical results for PSNR, SSIM, and LPIPS on BAIR of our model trained with ∆t = 1 and tested with different values of ∆t.

Table content: <table><tr><td>Step size ∆t</td><td>PSNR</td><td>SSIM</td><td>LPIPS</td></tr><tr><td>∆t = 1</td><td colspan="3">19.64 ± 0.45 0.8210 ± 0.0110 0.0612 ± 0.0048</td></tr><tr><td>∆t = 1 2 ∆t = 1 3 ∆t = 1 4 ∆t = 1 5</td><td colspan="3">19.76 ± 0.44 0.8235 ± 0.0110 0.0597 ± 0.0047 19.82 ± 0.45 0.8245 ± 0.0111 0.0593 ± 0.0048 19.83 ± 0.46 0.8242 ± 0.0111 0.0593 ± 0.0049</td></tr></table>

Numerical resuls for PSNR, SSIM, and LPIPS on KTH of our model trained with ∆t = 1 and tested with different values of ∆t.

Table content: <table><tr><td>Step size ∆t</td><td>PSNR</td><td>SSIM</td><td>LPIPS</td></tr><tr><td>∆t = 1</td><td colspan="3">29.76 ± 0.38 0.8681 ± 0.0057 0.0737 ± 0.0057</td></tr><tr><td>∆t = 1 2 ∆t = 1 3 ∆t = 1 4 ∆t = 1 5</td><td colspan="3">29.05 ± 0.42 0.8539 ± 0.0066 0.0882 ± 0.0050 29.05 ± 0.42 0.8509 ± 0.0069 0.0924 ± 0.0055 28.98 ± 0.42 0.8496 ± 0.0069 0.0939 ± 0.0056 28.95 ± 0.42 0.8490 ± 0.0070 0.0948 ± 0.0057</td></tr></table>

Numerical results for PSNR, SSIM, and LPIPS on KTH of our model trained with ∆t = 1 2 and tested with different values of ∆t.

Table content: <table><tr><td/><td/><td/><td/><td colspan="3">Step size ∆t</td><td/><td>PSNR</td><td/><td/><td/><td>SSIM</td><td/><td/><td/><td/><td>LPIPS</td><td/><td/><td/><td/></tr><tr><td/><td/><td/><td/><td colspan="2">∆t = 1</td><td/><td colspan="13">28.80 ± 0.41 0.8495 ± 0.0068 0.0994 ± 0.0057</td><td/><td/></tr><tr><td/><td/><td/><td/><td colspan="2">∆t = 1 2 ∆t = 1 3 ∆t = 1 4 ∆t = 1 5</td><td/><td colspan="13">29.69 ± 0.37 0.8697 ± 0.0057 0.0736 ± 0.0036 29.52 ± 0.38 0.8656 ± 0.0059 0.0777 ± 0.0041 29.43 ± 0.39 0.8633 ± 0.0061 0.0790 ± 0.0042 29.35 ± 0.39 0.8615 ± 0.0062 0.0810 ± 0.0045</td><td/><td/></tr><tr><td/><td>34</td><td/><td/><td/><td colspan="3">SVG-LP -DCGAN SVG-LP -VGG</td><td>0.95</td><td/><td/><td/><td/><td/><td/><td>0.18</td><td/><td/><td/><td/><td/><td/></tr><tr><td/><td>30 32</td><td/><td/><td/><td colspan="2">Ours -DCGAN Ours -VGG</td><td/><td>0.90</td><td/><td/><td/><td/><td/><td/><td>0.14 0.16</td><td/><td/><td/><td/><td/><td/></tr><tr><td>PSNR</td><td>26 28</td><td/><td/><td/><td/><td/><td>SSIM</td><td>0.85</td><td/><td/><td/><td/><td/><td>LPIPS</td><td>0.10 0.12</td><td/><td/><td/><td/><td/><td/></tr><tr><td/><td>24</td><td/><td/><td/><td/><td/><td/><td>0.80</td><td/><td/><td/><td/><td/><td/><td>0.08</td><td/><td/><td/><td/><td/><td/></tr><tr><td/><td>20 22</td><td/><td/><td/><td/><td/><td/><td>0.75</td><td/><td/><td/><td/><td/><td/><td>0.04 0.06</td><td/><td/><td/><td/><td/><td/></tr><tr><td/><td>10</td><td>15</td><td>20</td><td>25</td><td>30</td><td>35</td><td>40</td><td>10</td><td>15</td><td>20</td><td>25</td><td>30</td><td>35</td><td>40</td><td>0.02</td><td>10</td><td>15</td><td>20</td><td>25</td><td>30</td><td>35</td><td>40</td></tr><tr><td/><td/><td/><td/><td>t</td><td/><td/><td/><td/><td/><td/><td>t</td><td/><td/><td/><td/><td/><td/><td/><td>t</td><td/><td/></tr></table>

H.2 KTHWe present in Figures 15 to 19 additional samples from SV2P, SVG, SAVP and our model on KTH, with additional insights.H.3 BAIRWe present in Figures 20 to 22 additional samples from SV2P, SVG, SAVP and our model on BAIR, with additional insights.H.4 OVERSAMPLINGWe present in Figure23additional examples of video generation at a doubled frame rate by our model.

Table content: <table><tr><td>Ground</td><td>Truth</td></tr><tr><td/><td>SV2P</td></tr><tr><td/><td>SVG</td></tr><tr><td/><td>SAVP</td></tr><tr><td>Ours</td><td>(Best)</td></tr><tr><td>Ours</td><td>(Worst)</td></tr><tr><td>Ours</td><td>(Random)</td></tr></table>

Peer review (iclr2020):
Summary: The paper proposes a video prediction method based on State-Space Models. The paper describes two main contributions: 1. By learning dynamics in the latent state space, the method avoids the high computational cost and accumulating image reconstruction errors of autoregressive models that condition on generated frames. 2. To model dynamics, the paper proposes a residual update rule inspired by Euler’s method to solve ODEs. According to this rule, the update to the latent state y_t is modeled as an additive residual f(y_t, z_{t+1}). This has the advantage that the step size of the discretization can be adjusted freely, e.g. between training and inference. The paper provides extensive experimental comparison of their model to the SVG and SAVP models on several standard datasets. The paper further contains experiments illustrating features of the model such as disentangling dynamics and content, and interpolation of dynamics in the latent space. Decision: The paper is written clearly and the mathematical treatment and experiments appear rigorous. The idea of predicting video using state-space models is interesting and promising. However, as described below, the paper overstates its novelty and falls short of showing the advantages of the method beyond incremental improvements on frame-wise image quality metrics. I therefore suggest rejection in its current version. Supporting arguments and suggestions: 1. The idea to use fully latent models for video prediction, to untie frame synthesis and dynamics, is not new and the paper does not fully cite this literature. For example, [1] and [2] perform unsupervised, non-autoregressive video prediction. The differences to these models should be discussed. 2. The advantages of the residual update rule are not made clear enough. The parallels to the ODE literature seem tenuous. The main advantage described in the paper is the ability to synthesize videos at different frame rates, but interpolation over such short time horizons is not a hard problem. At least, the paper should compare to existing methods for frame interpolation. Apart from interpolation (variable step size), it appears that the update rule could be changed from y_{t+1} = y_t + f(y_t, z_{t+1}) to y_{t+1} = f(y_t, z_{t+1}) without impact to the model. How is it different from the standard VRNN formulation [3]? More experiments to show the advantage of the proposed update rule would be helpful. 3. Some of the experiments seem like interesting starting points but do not support general claims. For example, Fig 2 (b) shows that the proposed dynamics model is better than an MLP or GRU on deterministic Moving MNIST, but is this also true on real datasets, which have much more complex dynamics? Similarly, the interpolation in Figure 9 is intriguing, but it would be helpful to describe and test how this ability is useful for applications of the predictive model. 4. The comparisons use frame-wise metrics of image quality (PSNR, SSIM, LPIPS). Even though they are common in the literature, these metrics are unsuitable for comparing long video sequences due to their stochasticity. The metrics are probably dominated by relatively uninteresting features such as the quality of the static background. Metrics for comparing entire videos exist (e.g. FVD [4]) and should be used. Even better, the paper should demonstrate the usefulness of the model for downstream tasks such as reinforcement learning, although I understand that this may be out of scope. Minor comments: - As far as I know, the correct term for error terms is residual, not residue. - What do the error bars in the figures show? Please add this information to the figure legends. [1] Wichers et al., 2018, https://arxiv.org/pdf/1806.04768.pdf [2] Minderer et al., 2019, https://arxiv.org/abs/1906.07889 [3] Chung et al, 2015, https://arxiv.org/abs/1506.02216 [4] Unterthiner et al., 2018, https://arxiv.org/abs/1812.01717