Stochastic Latent Residual Video Prediction
Authors: Jean-Yves Franceschi
Edouard Delasalles
Mickael Chen
Sylvain Lamprier
Patrick Gallinari
Venue:  (2020)
Keywords: stochastic video prediction
variational autoencoder
residual dynamics

Abstract: Video prediction is a challenging task: models have to account for the inherent uncertainty of the future. Most works in the literature are based on stochastic image-autoregressive recurrent networks, raising several performance and applicability issues. An alternative is to use fully latent temporal models which untie frame synthesis and dynamics. However, no such model for video prediction has been proposed in the literature yet, due to design and training difficulties. In this paper, we overcome these difficulties by introducing a novel stochastic temporal model. It is based on residual updates of a latent state, motivated by discretization schemes of differential equations. This first-order principle naturally models video dynamics as it allows our simpler, lightweight, interpretable, latent model to outperform prior state-of-the-art methods on challenging datasets.

Under review as a conference paper at ICLR 2020
STOCHASTIC LATENT RESIDUAL VIDEO PREDICTION
Anonymous authors
Paper under double-blind review
ABSTRACT
Video prediction is a challenging task: models have to account for the inherent uncertainty of the future. Most works in the literature are based on stochastic image- autoregressive recurrent networks, raising several performance and applicability issues. An alternative is use fully latent temporal models which untie frame synthesis and dynamics. However, no such model for video prediction has been proposed in the literature yet, due design and training difficulties. In this paper, we overcome these difficulties by introducing a novel stochastic temporal model. It is based on residual updates of latent state, motivated by discretization schemes of differential equations. This first-order principle naturally models video dynamics as allows our simpler, lightweight, interpretable, latent model to outperform prior state-of titee-of-hh-aas art methods on challenging datasets.
INTRODUCTION
Being able to predict the future of video from a few conditioning frames in sulf-supervieed manner has many applications in fields such as reinforcement learning (Gregor et al., 2019) or robotics (Babaeizadeh al., 2018). More generally, it challenges the ability of model to capture visual and dynamic representations of the world. Video prediction has received a lot of attention from the computer vision community. However, most proposed methods are deterministic, reducing their ability to capture video dynamics, which are intrinsically stochastic (Denton & Fergus, 2018).
Stochastic video prediction is challenging task which has been tackled by recent works. Most state-on tiite-of-thee thethe- approaches are based on image-autorege autre-ruutoreressiveee models (Denton & Fergus, 2018; Babacizadeh et al., 2018), built around Recurrent Neural Networks (RNNs), where each generated frame is fed back to the model to produce the next frame. However, performances of their temporal models innately depend on the capacity of their encoder and decoder, as each generated frame has to be re-encoded in a latent space. Such autoregressive processes induce high computational cost, and strongly tie the frame synthesis and temporal models, which may hurt the performance of the generation process and limit its applicability (Gregor et al., 2019; Rubanova et al., 2019).
An alternative approach consists in separating the dynamic of the state representations from the generated frames, which are independently decoded from the latent space. In addition to removing the aforementioned link between frame synthesis and temporal dynamics, this is computationally appealing when coupled with low-dimensional latent-space. Moreover, such models shape complete representation of the state a system, e.g. for reinforcement learning applications (Gregor et al., 2019), and more interpretable than autoregressive models (Rubanova et al., 2019). Yet, these State-Space Models (SSMs) are more difficult to train as they require non-trivial latent state inference schemes (Krishnan et al., 2017) and careful design of the dynamic model (Karl al., 2017). This leads most successful SSMs to only be evaluated on small or artificial toy tasks.
be used to
In this work, we propose a novel stochastic dynamic model for the task of video prediction which suc- cessfully leverages structural and computational advantages of SSMs that operate on oow-dimensional latent spaces. The dynamic component determines the evolution through residual updates of the latent state, conditioned on learned stochastic variables. This formulation allows to implement an efficient training strategy and process in an interpretable manner complex high-dimensional data such as videos. This residual principle can be linked to recent advances relating residual networks and Ordinary Differential Equations (ODEs) (Chen et al., 2018). This interpretation opens new perspectives such as generating videos at different frame rates, demonstrated in experiments. Overall, this approach outperforms current oftee-of-the-ar models on the task of stochastic video prediction, as demonstrated by comparisons with competitive baselines representative benchmarks.

---

Under review as a conference paper at ICLR 2020
2
RELATED WORK
Video synthesis covers a range of different tasks, such as video-to-video translation (Wang et al., 2018), super-resolution (Caballero et al., 2017), interpolation between frames (Jiang al., 2018), unconditonal generation (Tulyakov et al., 201 8), or video prediction, which is the focus of this paper.
Deterministic models. Inspired by prior sequence generation models using RNNs (Graves, 2013). a
number of video prediction methods (Srivastava et al., 2015; Villegas et al., 2017; Wichers et al., 2018) rely on LSTMs (Hochreiter & Schmidhuber. 1997), or, like Ranzato et al. (2014) and Jia et al. (2016), on derived networks such as ConvLSTMs (Shi et al., 2015) taking advantage of Convolutional Neural usually tailored to high-dimensional
Networks (CNNs). Indeed, computer vision approaches video sequences and propose domain-specific techniques as they often use pixel-level transformations and optical flow (Shi et al., 2015; Walker et al., 2015; Finn et al., 2016; Jia et al., 2016; Vondrick & Torralba, 2017; Liang et al., 2017; Liu et al., 2017; Lotter et al., 2017: Lu et al., 7a; Fan et al., 2019) that help produce high-quality predictions. Such predictions are, however, deterministic, thus hurting their performance as they fail to generate sharp long-term video frames (Babaeizadeh et al.. 2018; Denton & Fergus, 2018). Following Mathieu et al. (2016). some works proposed to use an adversarial loss (Goodfellow et al., 2014) on the predictions of their model to sharpen the generated frames (Vondrick& Torralba, 2017; Liang et al., 2017; Lu et al., 7a; Xu et al., 2018) 8). Nonetheless, adversarial losses are notoriously hard to train, and lead to mode collapse, preventing diversity of generations.
Stochastic and image-autoregreesive models. Some approaches rely on exact likelihood max- imization, using pixel-level autoregressive generation (van den Oord et al., 2016; Kalchbrenner et al., 2017) or normalizing flows through invertible transformations between the observation space and a latent space (Kingma& & Dhariwal, 2018; Kumar et al., 2019). However, they require careful design of complex temporal generation schemes manipulating nigh-dimensional data, thus induc- ing a prohibitive temporal generation cost. More efficient continuous models rely on Variational Auto-Encoders (VAEs) (Kingma& & Welling, 2014; Rezende et al., 2014) for the inference of low- dimensional latent state variables. Except Xue et al. (2016) who learn a one-frame-ahead VAE, they model sequence stochasticity by incorporating random latent variable per frame into deterministic RNN-based auttrerruutreeesivvee model. Babacizadeh et al. (2018) integrate stochastic variables into the ConvLSTM architecture of Finn et al. (2016). Concurrently with He et al. (2018), Denton & Fergus (2018), with Castrejon et al. (2019) in a follow-up. use a prior LSTM conditioned on previously generated frames in order sample random variables that fed predictor LSTM. Finally, Lee et al. (2018) combine the ConvL.TTM architecture and this learned prior, adding an adversarial loss on the predicted videos to sharpen them at the cost of a diversity drop. Yet, all these methods are enneee-,utoregresssive, as they feed their predictions back into the latent space, thus tying the frame synthesis and temporal models and increasing their computational cost. Concurrently to our work, Minderer et al. (2019) propose to use the autoregressive VRNN model (Chung et al., 2015) on learned image key-points instead of raw frames. While this change could mitigate the aforementioned problems, the extent of such mitigation is unclear. We follow a complementary approach by directly proposing a dynamic model that is state-space and acts on a small latent state, tackling these issues.
State-space models. Many latent state-space models have been proposed for sequence modelization (Bayer & Osendorfer, 2014; Fraccaro al., 2016; 2017: Krishnan al., 2017: Karl et al., 2017; Hafner et al., 2019), usually trained by deep Variational Inference (VI). These methods, which use locally linear temporal transition functions or RNN- dynamics, are designed for and tested on low-dimensional data, as learning such models on complex data is challenging, or focus on control or planning tasks. In contrast, our fully latent method is the first one to be successfully applied to complex high-dimensional data such as videos, thanks to temporal model based on residual updates of its latent state. It thus falls within the scope of a recent trend linking differential equations with neural networks (Lu et al., 2017b; Long et al., 2018), leading to the integration of ODEs, that are seen as continuous residual networks, in neural network architectures (Chen et al., 2018). However, the latter work and follow-ups ups (Rubanova et al., 2019; Yıldız et al., 2019) are either limited to low-dimensional data, prone to overfitting or unable to handle stochasticity within a sequence. Another line of works considers stochastic differential equations (SDEs) with neural networks (Ryder
2

---

Under review as a conference paper at ICLR 2020
90
98
(a) Generative model p.
Po ag
22
23
LSTM
L.STM
LSTTM
22
2Fg
Model and inference architecture on a test sequence. transparent block on left depicts the prior, and those on the right correspond the full inference performed (b) Inference model at training time.
Figure (a), (b) Proposed generative and inference models. Diamonds and circles represent, respectively, deterministic and stochastic states. (c) Corresponding architecture with two parts: inference on conditioning frames on the left, generation for extrapolation on the right. hg and go are deep CNNs, and other named networks are Multilayer Perceptrons (MLPs).
et al., 2018; De Brouwer et al., 2019), but are limited to continuous Brownian noise, whereas video prediction additionally requires to model punctual stochastic events.
MODEL
We consider the task of stochastic video prediction, consisting in approaching, given number of conditioning video frames, the distribution possible future frames given this conditioning.
3.1 LATENT RESIDUAL. DYNAMIC MODEL.
Let U1.T be sequence of T video frames. We model their evolution by introducing latent variables that are driven by dynamic temporal model. Each frame x, is then generated from the corresponding latent state Ye only, making the dynamics independent from the previously generated frames.
We propose to model the transition function of the latent dynamic of y with a stochastic residual network. State Ut+1 is chosen deterministically depend on the previous state yt. conditionally to an auxiliary random variable 2++1- These auxiliary variables encapsulate the randomness of the video dynamics. They have learned factorized Gaussian prior that depends on the previous state only. The model is depicted in Figure la, and defined as follows:
21 ~ (o, I); Ze+1 (No (o (y:)) Yc+1 = yt + So(ye: 2,+1), a, ~ g (go(y)))
(1)
where p4o, fo and are neural networks, and 9 (go(t:)) is probability distribution parameterized by go (y).. In our experiments, G is a normal distribution with fixed diagonal variance and mean go gy())) Note that y is assumed to have standard Gaussian prior, and, in our VAE setting, will be inferred from conditioning frames for the prediction task, shown in Section 3.3.
The residual update rule takes inspiration in the Euler discretization scheme of differential equations. The state of the system yı is updated by its first-order movement, i.c., the residual f3 *++1). Compared regular RNN, this simple principle makes our temporal model lighter and more interpretable. Equation (1), however, differs from a discretized ODE because of the introduction of the stochastic discrete-ting time variables z. Nonetheless, we propose to allow the Euler step size At to be smaller than 1, as way to make the temporal model closer to continuous dynamics. The updated dynamics becomes, with Ar E N to synchronize the step size with the video frame rate:
U++A y: At fo zw(+1)
(2)
For this formulation, the auxiliary variable z, is kept constant between two integer time steps. Note that different At be used during training or testing. This allows model generate videos
3

---

Under review as a conference paper at ICLR 2020
an arbitrary frame rate since each intermediate latent state be decoded in the observation space. This ability enables us to observe the quality of the learned dynamic as well as challenge its ODE inspiration by testing its generalization to the continuous limit in Section 4. In the following, consider At as a hyperparameter. For the sake of clarity, we consider that At in the following: generalizing smaller A/ is straightforward Figure la remains unchanged.
3.2 CONTENT VARIABLE
Some components of vidco sequences can be static, such as the background or shapes of moving objects. They may not impact the dynamics; therefore model them separately, the same spirit as Denton & Birodkar (2017) and Yingzhen & Mandt (2018). We compute a content variable w that remains constant throughout the whole generation process and is fed together with yr into the frame generator. It enables the dynamical part of the model to focus only on movement, hence being lighter and more stable. Moreover, allows us leverage architectural advances in neural networks, such as skip connections (Ronneberger et al., 2015), to produce more realistic frames.
This content variable a deterministic function Co of a fixed number it of frames x (k),
(k)
(k) Ct
Cu (x.,.
Biy.) Ct ~ G w)).
Dipy
Cip
During testing, x are the last & conditioning frames (usually between and 5).
This content variable is not endowed with any probabilistic prior, contrary to the dynamic variables and z. Hence, the information contains not constrained in the loss function (see Section 3.3), but only architecturally. To prevent temporal information from leaking in we propose to uniformly sample these & frames within x 1.7 during training. We also design cø as a permutation-invarian function (Zaheer et al., 2017), which is done by using MLP fed with the sum of individual frame representations, similarly to Santoro et al. (2017).
This absence of prior and its architectural constraint allows to contain as much non-temporal information as possible, while preventing from containing dynamic information. On the other hand, due to their strong standard Gaussian priors, y and are encouraged to discard unnecessary information. Therefore, y and z should only contain temporal information that could not be captured by w.
Note that this content variable can be removed from our model, yielding a more classical deep state-space model. An experiment in this setting presented in Appendix E.
3.3 VARIATIONAL INFERENCE AND ARCHITECTURE
Following the generative process depicted in Figure 1a, the conditional joint probability of the full model, given content variable w, can be written as:
T
p(aEer #2:T; Vur w) = ((y)) ) II plze+ gt)p(ue yr, 2e+1) II p( yt. w),
(4)
where p(ye+1 yt, x;-1) 5(y) + fo(y), z(-1) y0+1) and 8 the Dirac delta function centered on O, according to the expression yt+1 1 in Equation (1). Thus, order optimize the likelihood of the observed videos p(#1.T w), we need to infer latent variables and 22. This is done by deep Variational Inference using the inference model parameterized by and shown in Figure 1b, which comes down to consider variational distribution q2,Y defined and factorized as follows:
T
qz,y 2 g((2/T B æir,, w) - ((ak) II (( x11)00yr-11 + fo(y ze) yr). (5)
t=2
4

---

Under review as a conference paper at ICLR 2020
This yields the following evidence lower bound (ELBO), whose full derivation is given in Appendix A:
logp(E1:7 w) 2 (E D1 r2
log p(æ:
Ye: w) (9 x1*4) p(31))
T
B( M
DKL (q(z)
p(zz (1-1)) - L(11:) w. 0. (6)
The sum of KL divergence expectations implies to consider the full past sequence of inferred states for each time step, due to the dependence However, optimizing L(æ1:r; w, 0, .) with respect to model parameters and variational parameters be done efficiently by sampling single full sequence of states from qz,Y per example, and computing gradients by backpropagation (Rumelhart et al., 1988) trough all inferred variables, using the reparametrization trick (Kingma & Welling, 2014; Rezende et al., 2014). We classically choose 4311 æ1:s) and (z) x1:t) to be factorized Gaussian so that all KLDs can be computed analytically.
conditionally deterministic variables y2:r-
We include an & regularization term on residuals fa which stabilizes the temporal dynamics of the residual network, as noted by Behrmann et al. (2019) and Rousseau et al. (2019). Given a set of videos X, the full optimization problem, where L is defined in Equation (6), is then given as:
E ( C ( Go ( - .. (x2. M.r)-oz
Σ #EX
0.00
Ell 2:)112 (7)
arg max 0,
Figure Ic depicts the full architecture of our temporal model, corresponding how the model is applied during testing. The first latent variables are inferred with the conditioning framed and are then predicted with the dynamic model. In contrast, during training, each frame of the input sequence is considered for inference, which is done as follows. Firstly, each frame æ; is independently encoded into a vector- valuee representation G,, with 3. = h (ar,)) y/1 is then inferred using an MIP on the first & encoded frames Ti.k. Each 2, is inferred in a feed-forward fashion with an LSTM on the encoded frames. Inferring = this way experimentally performs better than, c.g., inferring them from the whole sequence #1:7; hypothesize that this follows from the fact that this filtering scheme is closer to the prediction setting, where the future is not available.
4 EXPERIMENTS
This section exposes the experimental results of our method on three standard stochastic video prediction datasets.' We compare our method with state-of-the-art baselines on stochastic video prediction. Furthermore, we qualitatively study the dynamics and latent space learned by our model. Training details are described in Appendix C.
The stochastic nature and novelty of the task of stochastic video prediction make it challenging to evaluate (Lee et al., 2018): since videos and models are stochastic, comparing the ground truth and a predicted video is not adequate. We thus adopt the common approach (Denton & Fergus, 2018; Lee et al., 2018) consisting in, for each test sequence, sampling from the tested model a given number (here, 100) of possible futures and reporting the best performing sample against the true video. We report this discrepancy for three commonly used metrics: Peak Signal-to-Noise Ratio (PSNR, higher is better). Structured Similarity (SSIM, higher better), and Learned Perceptual Image Patch Similarity (LPIPS, lower is better) (Zhang et al., 2018). PSNR tends to promote blurry predictions, as it is pixel-level measure derived from the &a distance, but greatly penalizes errors in predicted positions of objects in the scenes. SSIM is similarity metric between image patches. LPIPS is a learned distance between activations of deep CNNs trained on image classification tasks, and have been shown to better correlate with human judgment real images. While these three metrics are computed frime-wise, the recently proposed Fréchet Video Distance (FVD, lower is better) (Unterthiner al., 2018) aims at directly comparing the distribution of predicted videos with the ground truth distribution through the representations computed by a deep CNN trained on action
Code, video samples, and datasets are available httph://///// /ttee:///iieeggee com/ /vi ew/ srvp/.
5

---

Under review as a conference paper at ICLR 2020
SVGL2
SVG-LP
Oup
Oup 99l0
(a) Stochastic Moving MMNIST.
(b) Deterministic Moving MMNIST.
Figure 2: Mean PSNR and SSIM scores with respect to t for all tested models the SM-MNIST
dataset, with their 95%-ccnniee intervals. Vertical bars mark the length of train sequences.
t-1
t=5
t=6
t=8
t-10
T-12
t-14
t-16
t-18
t=28
t-22
t-24
Ground Truth
S
5
3 S
5
<
5
5
5
S
SVG
2 5
25
6
6
62
36
3/
3
5
35
3
5
$
Figure 3: Conditioning frames and corresponding ground truth and best samples with respect to PSNR from SVG and our method for an example of the SM-MNIST dataset.
recognition tasks. It has been shown, independently from LPIPS, better correlate with human judgment than PSNR and SSIM. We treat all four metrics as complementary, as they capture different modalities. PSNR challenges the dynamics of the predicted videos, while SSIM rather compares local frame patches but loses some dynamics information. LPIPS and FVD both measure the realism the predictions compared to the ground truth. FVD considers videos of detecting temporal inconsistencies. On the other hand, the frame-wise LPIPS metric penalizes more the temporal drifts of videos, since it directly compares each predicted and ground truth frame.
whole, making it more capable
We present experimental results on simulated dataset and two real-world datasets, that we briefly present the following and detail in Appendix The corresponding numerical results can be found in Appendix D. For the sake of concision, we only display handful of qualitative samples in this section, and refer to Appendix H for additional samples. We compare our model against several state-of-the of-te-o-the-arr art models: SV2P (Babaeizadeh et al., 2018), SVG (Denton & Fergus, 2018) () and SAVP (Lee et al., 2018). All baseline results were obtained with pretrained models released by the authors. Note that we use the same neural architecture SVG for our encoders and decoders in order to perform fair comparisons with this method, which the closest to ours among the state of the art. Unless specified otherwise, model is tested with the same At in training (see Equation (2)).
Stochastic Moving MNIST (SM-MNIST). This dataset consists of one or two MNIST digits (LeCun et al.. 1998) moving linearly and randomly bouncing on walls with new direction and velocity sampled randomly at each bounce (Denton & Fergus, 2018). As SV2P and SAVP were not tested on this dataset (in particular, with pretrain model, code or hyperparameters) we only report scores for SVG as state-of-the-art model on SM-MNIST.
Figure 2a shows quantitative results with two digits. Our model outperforms SVG on both PSNR and SSIM; LPIPS and FVD are not reported as they are not relevant for this synthetic task. Decoupling dynamics from image synthesis allows our method to maintain temporal consistency despite high- uncertainty frames where crossing digits become indistinguishable. For instance in Figure 3, the digits shape changes after they cross in the SVG prediction, while our model predicts the correct digits. To evaluate the predictive ability on a longer horizon, we perform experiments on the classic deterministic version of the dataset (Srivastava al., 2015). We show the results up to f 95 in

---

Under review as a conference paper at ICLR 2020
SVG.LP SAVP
16:00
SV2P
000
Curs
PENS
Oure
S983
1PPIS
16:2
6:90
Figure 4: PSNR, SSIM and LPIPS scores with respect to t for all tested models the KTH dataset.
Ground Truch
Ours Ours (handon) (verst:) (Best)
Figure 5: Conditioning frames and corresponding ground truth, best samples from SVG, SAVP and method, and worst and random samples from method, for example of the KTH dataset. Samples are chosen according to their LPIPS with respect to the ground truth. SVG fails to make person appear unlike SAVP and our model, and the latter better predicts the pose of the subject.
Figure 2b. We can see that our model better captures the dynamics of the problem compared to SVG as performance decreases significantly less, even at long-term horizon.
We also compare to two alternative versions of our model in Figure 2, where the residual dynamic function is replaced by an MLP or a GRU network (Cho al., 2014). Our residual model outperforms both versions the stochastic, and especially on the deterministic version of the dataset, showing its intrinsic advantage at modeling dynamics. Finally, on the deterministic version of Moving MNIST, we compare alternative where is entirely removed, resulting in temporal model very close the one presented in Chen et al. (2018). The loss of performance of this alternative model is significant, especially in SSIM, showing that stochastic residual model offers substantial advantage when used in a deterministic environment.
KTH Action dataset (KTH). This dataset composed of real-world videos of people performing single action per video in front of different backgrounds (Schüldt et al., 2004). Uncertainty lies in the appearance of subjects, the actions they perform and how they are performed.
We outperform on this dataset every considered baseline for each metric, as depicted in Figure and Table In some videos, the subject only appears after the conditioning frames, requiring the model to sample the moment and location of the subject appearance, well as its action. This critical case is illustrated in Figure There, SVG fails to even generate moving person; only SAVP and model manage to do and our best sample is closer to the subject's poses compared SAVP. Moreover, the worst and a random sample of our model demonstrate that it captures the diversity of the dataset by making person appear different time steps and with different speeds. An additional experiment on this dataset is included in Appendix G, studying the influence of the encoder and decoder architecture on SVG and model.
7

---

Under review as a conference paper at ICLR 2020
SAYF
0890
0.000
1PTP5
0.00
0.71
0.6
Figure 6: PSNR, SSIM and LPIPS scores with respect to t for all tested models the BAIR dataset.
(a) Cropped KTH sample.
(b) Cropped BAIR sample.
Figure 7: Generation examples doubled frame rate, using halved At compared training. Frames including bottom red dashed bar are intermediate frames.
Finally, Table compares our method its MLP and GRU alternative versions. leading to two conclusions. Firstly, it confirms the structural advantage of residual dynamics observed on Moving MNIST. On one hand, MIP better captures dynamics than GRU on KTTH according to PSNR and SSIM, but loses in terms of realism according to LPIPS and FVD. On the other hand, the residual version shows slight dynamics improvement with respect to both MLP and GRU, while substantially pushing further prediction realism. Secondly, all three versions of our model (residual, MLP, GRU) outperform prior methods. Therefore, this improvement is due to their common inference method, latent nature and content variable, strengthening our motivation propose non non-autoregressive model.
BAIR robot pushing dataset (BAIR). This dataset contains videos a Sawyer robotic arm pushing objects tabletop (Ebert et al., 2017). It is highly stochastic the arm can change its direction at any moment. We achieve similar or better results compared to state-of-the-art models, as Figure 6 and Table 3 shows, and second-best PSNR behind SV2P, but the latter produces very blurry samples, which can be seen in Appendix H. yielding prohibitive LPIPS and FVD scores. In contrast, achieve the highest SSIM overall, well as state-of-the-art LPIPS and competitive FVD among these models. Note that we could not add VideoFlow to our experiments, due to the unavailability of pretrained models and numerical results. However, compared to PSNR, SSIM and LPIPS results reported by Kumar et al. (2019) for BAIR (the only tested dataset and metrics in their paper), our model appears to behave better than VideoFlow, which is on par with SAVP these metries.
Varying frame rate in testing. We challenge the ability of our model to different Euler step size than the one used in training (see Equation (2)). Figures 4 and include corresponding results with halved At. Prediction performances remain stable while generating twice as many frames (cf. Appendix F for further discussion). Our model is thus robust to the refinement of the Buler approximation, showing the quality of the learned dynamic which close continuous. In particular, this shows that our model learned a dynamic driven by a piecewise ODE, i.e., the learned dynamic of each interval between two consecutive frames is an ODE, as a constant is given on such interval. This can be used to generate frames at a higher frame rate than the training videos without supervision. We show in Figure and Appendix frames generated double and quadruple frame rate on BAIR and KTH.. Both figures show smooth intermediate generated frames.
Disentangling dynamics and content. Let us show that the proposed model actually separates content from dynamics as discussed in Section 3.2. To this end, two sequences af and x are drawn
8

---

Under review as a conference paper at ICLR 2020
t-11
t-13
t-15 t-17
t-19
Dynanic
content
Figure 8: Video (bottom right) generated from the dynamic latent state y inferred with a video (top) and the content variable computed with the conditioning frames of another video (bottom left). The generated video keeps the same background as the bottom left frames, while the robotic arm moves accordingly to the top frames.
Ref
Rec
Interpolation
Rec
Ref
Time
Figure 9: From left to right, x', Th (reconstruction of x by the VAE of our model), results of the interpolation the latent space between x and æ., 2 and w². Each trajectory is materialized in shades of grey in the frames.
from the BAIR test set. While x" is used for extracting our content variable w", dynamic states y are inferred with our model from æ². New frame sequences E are finally generated from the fusion of the content vector and the dynamics. This results in a content corresponding to the first sequence x while moving according to the dynamics of the second sequence x", observed in Figure 8. More samples for BAIR and KTH can be seen in Appendix H.
Interpolation of dynamics. Our state-space structure allows us learn semantic representations in y:: To highlight this feature, we test whether two Moving MNIST trajectories can be interpolated by linearly interpolating their inferred latent initial conditions. We begin by generating two trajectories xx and x² of a single moving digit. We infer their respective latent initial conditions yi and yţ. We then use our model to generate frame sequences from latent initial conditions linearly interpolated between yi and y it learned meaningful latent space, the resulting trajectories should also be smooth interpolations between the directions of reference trajectories x" and æ², and this is what we observe in Figure 9. Additional examples can be found in Appendix H.
CONCLUSION
We introduce novel dynamic latent model for stochastic video prediction which, unlike prior image- autoregressive models, decouples frame synthesis and dynamics. This temporal model is based on residual updates of small latent that is showed to perform better than RNN-based models. This endows our method with several desirable properties, such as temporal efficiency and latent space interpretability. We experimentally demonstrate the performance and advantages of the proposed model, which outperforms prior state-of-ther -hee-artt methods for stochastic video prediction. This work is, the best of our knowledge, the first to propose latent dynamic model scaling for video prediction. The proposed model is also novel with respect to the recent line of work dealing with neural networks and ODEs for temporal modeling; it is the first such residual model to scale to complex stochastic data such videos.
We believe that the general principles our model (state-space, residual dynamic, static content variable) be generally applied to other models well. Interesting future works include replacing the VRNN model Minderer et al. (2019) in order to model the evolution of key- cey-toints, or leveraging the state-space nature of our model in model-based reinforcement learning.

---

Under review as a conference paper at ICLR 2020
REFERENCES
Mohammad Babacizadeh, Chelsea Finn, Dumitru Erhan, Roy H. Campbell, and Sergey Levine. Stochastic variational video prediction. In International Conference on Learning Representations. 2018.
Justin Bayer and Christian Osendorfer. Learning stochastic recurrent networks. arXiv preprint arXiv:1411.7610 2014.
Jens Behrmann, Will Grathwohl, Ricky T. Q. Chen, David Duvenaud, and Joern-Henrik Jacobsen. Invertible residual networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 573-582, Long Beach, California, USA, June 2019. PMLR.
Jose Caballero, Christian Ledig, Andrew Aitken, Alejandro Acosta, Johannes Totz, Zehan Wang, and Wenzhe Shi. Real-time video super-resolution with spatio-temporal networks and motion compensation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2848-2857, July 2017.
Lluis Castrejon, Nicolas Ballas, and Aaron Courville. Improved conditional VRNNs for video prediction. arXiv preprint arXiv:1904. 12165,, 2019.
Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David K. Duvenaud. Neural ordinary differential equations. Samy Bengio, Hanna Wallch, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianche and Roman Garnett (eds.). Advances in Neural Information Processing Systems 31, pp. 6571-6583. Curran Associates, Inc., 2018.
Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). pp. 1724- 1734, Doha, Qatar, October 2014. Association for Computational Linguistics.
encoder-decoder
Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron C. Courville, and Yoshua Bengio. A recurrent latent variable model for sequential data. In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Gamett (eds.), Advances in Neural Information Processing Systems 28, pp. 2980-2988. Curran Associates, Inc., 2015.
Edward De Brouwer, Jaak Simm, Adam Arany, and Yves Moreau. GRU-ODE-Bayee Continuous modeling of aporadically-observe time series. arXiv preprint arxiv: /205. 2019.
Emily Denton and Rob Fergus. Stochastic video generation with a learned prior. In Jennifer Dy and Andreas Krause (eds.). Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1174-1183, Stockholmsmässan, Stockholm, Sweden, July 2018. PMLR.
Emily L. Denton and Vighnesh Birodkar. Unsupervised learning of disentangled representations from video. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna Wallach, Rob Fergus, S. N. Vishwanathan, and Garnett Roman (eds.), Advances Neural Information Processing Systems 30, pp. 4414-4423. Curran Associates, Inc., 2017.
Frederik Ebert, Chelsea Finn, Alex X. Lee, and Sergey Levine. Self-supervised visual planning with temporal skip connections. In Sergey Levine, Vincent Vanhoucke, and Ken Goldberg (eds.), Proceedings of the 1st Annual Conference Robot Learning, volume 78 of Proceedings of Machine Learning Research, pp. 346-356. PMIR, November 2017.
Hehe Fan, Linchao Zhu, and Yi Yang. Cubic LSTMs for video prediction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 8263-8270, 2019.
Chelsea Finn, Ian Goodfellow, and Sergey Levine. Unsupervised learning for physical interaction through video prediction. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Garnett Roman (eds.), Advances in Neural Information Processing Systems 29, pp. 64-72. Curran Associates, Inc., 2016.
10

---

Under review as a conference paper at ICLR 2020
Marco Fraccaro, Søren Kaae Sønderby, UIrich Paquet, and Ole Winther. Sequential neural models with stochastic layers. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg. Isabelle Guyon, and Garnett Roman (eds.), Advances in Neural Information Processing Systems 29, pp. 2199-2207. Curran Associates, Inc., 2016.
Marco Fraccaro, Simon Kamronn, Ulrich Paquet, and Ole Winther. A disentangled recognition and nonlinear dynamics model for unsupervised learning. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna Wallach, Rob Fergus. S. V. N. Vishwanathan, and Garnett Roman (eds.), Advances in Neural Information Processing Systems 30, pp. 3601-3610. Curran Associates, Inc., 2017.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Zoubin Ghahramani, Max Welling, Corinna Cortes, Neil D. Lawrence, and Kilian Q. Weinberger (eds.), Advances in Neural Information Processing Systems 27, pp. 2672 2672-2680. Curran Associates, Inc., 2014.
Alex Graves. Generating sequences with recurrent neural networks. arXiv preprint 1308.v3308.08. 2013.
Karol Gregor, George Papamakarios, Frederie Besse, Lars Buesing, and Theophane Weber. Temporal difference variational auto-encoder. In International Conference on Learning Representations, 2019.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 2555-2565, 2565, Long Beach, California, USA, June 2019. PMLR.
Jiawei He, Andreas Lehrmann, Joseph Marino, Greg Mori, and Leonid Sigal. Probabilistic video generation using holistic attribute control. In Vittorio Ferrari, Martial Hebert, Cristian Sminchis- escu, and Yair Weiss (eds.). The European Conference on Computer Vision (ECCV), pp. 466-483. Springer International Publishing, September 2018.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. 3 3-VAE: Learning basic visual concepts with a constrained variational framework. International Conference Learning Representations, 2017.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 98):
1780, 1997.
Xu Jia, Bert De Brabandere, Tinne Tuytelaars, and Lue V Gool. Dynamic filter networks. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Garnett Roman (eds.), Advances in Neural Information Processing Systems 29, pp. 667-675. Curran Associates, Inc., 2016.
Huaizu Jiang, Deqing Sun, Varun Jampani, Ming-Hsuan Yang, Erik Learned-Miller, and Jan Kautz. Super slomo: High quality estimation multiple intermediate frames for video interpolation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9000 0000-9008, June 2018.
Nal Kalchbrenner, Aäron van den Oord, Karen Simonyan, Ivo Danihelka, Oriol Vinyals, Alex Graves, and Koray Kavukcuoglu. Video pixel networks. In Doina Precup and Yee Whye Teh (eds.). Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1771-1779, International Convention Centre, Sydney, Australia, August 2017. PMI.R
Maximilian Karl, Maximilian Soelch, Justin Bayer, and Patrick der Smagt. Deep variational Bayes filters: Unsupervised learning of state space models from raw data. In International Conference on Learning Representations), 2017.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference Learning Representations, 2015.
11

---

Under review as a conference paper at ICLR 2020
Diederik P. Kingma and Max Welling. Auto-encoding variational Bayes. In International Conference on Learning Representations, 2014.
Durk P. Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1xl convolutions. In Samy Bengio, Hanna Wallch, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 10215-10224. Curran Associates, Inc., 2018.
Rahul G. Krishnan, Uri Shalit, and David Sontag. Structured inference networks for nonlinear state space models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, pp. 2101-2109, 2017.
Manoj Kumar, Mohammad Babaeizadeh, Dumitru Erhan, Chelsea Finn, Sergey Levine, Laurent Dinh, and Durk Kingma. VideoFlow: A flow-based generative model for video. arXiv preprint arXiv:/903.014:: 014444, 2019.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86( 86(11):228--2244, November 1998.
Alex X. Lee, Richard Zhang, Frederik Ebert, Pieter Abbeel, Chelsea Finn, and Sergey Levine. Stochastic adversarial video prediction. arXiv preprint wwX.iv:880400122. 2018.
Xiaodan Liang, Lisa Lee, Wei Dai, and Eric Xing. Dual motion GAN for future-flow embedded video prediction. In JEEE International Conference Computer Vision (ICCV), pp. 1762-1770, October 2017.
Ziwei Liu, Raymond A. Yeh, Xiaoou Tang, Yiming Liu, and Aseem Agarwala. Video frame synthesis using deep voxel flow, In IEEE International Conference on Computer Vision (ICCV), pp. 4473-4481, October 2017.
Zichao Long, Yiping Lu, Xianzhong Ma, and Bin Dong. PDE-net: Learning PDEs from data. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 221666 Stockholmsmässann, Stockholm Sweden, July 2018. PMLR.
William Lotter, Gabriel Kreiman, and David Cox. Deep predictive coding networks for video prediction and unsupervised learning. International Conference Learning Representations, 2017.
C. Lu, M. Hirsch, and B. Schölkopf. Flexible Conference Computer Vision and Pattern Recognition (CVPR), pp. 2137-2145, July 2017a.
networks for video prediction. IEEE
Yiping Lu, Aoxiao Zhong, Quanzheng Li, and Bin Dong. Beyond finite layer neural networks: Bridging deep architectures and numerical differential equations. arXiv preprint 1rXiv:/710.100 10121, 2017b.
Michael Mathieu, Camille Couprie, and Yann LeCun. Deep multi-scale video prediction beyond mean square error. In International Conference on Learning Representations, 2016.
Matthias Minderer, Chen Sun, Ruben Villegas, Forrester Cole, Kevin Murphy, and Honglak Lec. Un- supervised learning of object structure and dynamics from videos. arXiv preprint 1906v 07889, 2019.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Vito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. In NIPS 2017 Autodiff Workshop, 2017.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep con- volutional generative adversarial networks. International Conference on Learning Representations, 2016.
MarcAurelio Ranzato, Arthur Szlam, Joan Bruna, Michael Mathieu, Ronan Collobert, and Sumit Chopra. Video (language) modeling: a baseline for generative models of natural videos. arXiv preprint arXiv: 2014.
12

---

Under review as a conference paper at ICLR 2020
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In Eric P. Xing and Tony Jebara e, Pro- ceedings of the 31st International Conference Machine Learning, volume 32 of Proceedings of Machine Learning Research, pp. 1278-1286, Bejing, China, June 2014. PMLR.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Nassir Navab, Joachim Hornegger, William M. Wells, and Alejandro F. Frangi (eds.), Medical Image Computing and Computer-Assisted Intervention MICCAI 2015, pp. 234-241, Cham, 2015. Springer International Publishing.
François Rousseau, Lucas Drumetz, and Ronan Fablet. Residual networks as flows of diffeomor- phisms. Journal of Mathematical Imaging and Vision, May 2019.
Yulia Rubanova, Ricky TQ Chen, and David Duvenaud. Latent ODEs for iregullrly-samplee time series. arXiv preprint 19XXvv1907.03300 2019.
David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. Neurocomputing: Foundations of Research, chapter Learning Representations by Back-propagating Errors, pp. 696-699. MIT Press, Cambridge, MA, USA, 1988.
Tom Ryder, Andrew Golightly, A. Stephen McGough, and Dennis Prangle. Black-box variational inference for stochastic differential equations. In Jennifer Dy and Andreas Krause (eds.), Proceed- ings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 4423 4432, Stockholmsmässan, Stockholm Sweden, July 2018 PMLR.
Adam Santoro, David Raposo, David G Barrett, Matcusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. A simple neural network module for relational reasoning. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna Wallach, Rob Fergus, S. V. N. Vishwanathan, and Gamett Roman (eds.), Advances in Neural Information Processing Systems 30, pp. 4967-4976. Curran Associates, Inc., 2017.
Christian Schüldt, Ivan Laptev, and Barbara Caputo. Recognizing human actions: A local SVM approach. In Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004., volume 3. pp. 32 36, August 2004.
Xingjian Shi, Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wai-kin Wong, and Wang-chun WOO. Convolutional LSTM network: A machine learning approach for precipitation nowcasting, In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 28, pp. 802 -10. Curran Associates, Inc., 2015.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. International Conference on Learning Representations, 2015.
Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov. Unsupervised learning of video representations using LSTMs. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 843-852, Lille, France, July 2015. PMLR.
Sergey Tulyakov, Ming-Yt Yu Liu, Xiaodong Yang, and Jan Kautz. MoCoGAN: Decomposing motion and content for video generation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1526-1535, June 2018
Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphaël Marinier, Marein Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. arXiv preprint 1./2.0/// 2018.
Aäron van den Oord, Nal Kalchbrenner, Lasse Espeholt, Koray Kavukcuoglu, Oriol Vinyals, and Alex Graves. Conditional image generation with PixeICNN decoders. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Garnett Roman (eds.), Advances in Neural Information Processing Systems 29, pp. 4790 4798. Curran Associates, Inc., 2016.
13

---

Under review as a conference paper at ICLR 2020
Ruben Villegas, Jimei Yang, Seunghoon Hong, Xunyu Lin, and Honglak Lee. Decomposing motion and content for natural video sequence prediction. International Conference on Learning Representations, 2017.
Carl Vondrick and Antonio Torralba. Generating the future with adversarial transformers. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2992-3000, July 2017.
Jacob Walker, Abhinav Gupta, and Martial Hebert. Dense optical flow prediction from a static image. In The IEEE International Conference Computer Vision (ICCV), pp. 2015.
December
Ting-Chun Wang, Ming-Yo Yu Liu, Jun-Yan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. Video-to-video synthesis. In Samy Bengio, Hanna Walleh, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 1144-1156. Curran Associates, Inc., 2018.
Nevan Wichers, Ruben Villegas, Dumitru Erhan, and Honglak Lee. Hierarchical long-term term video prediction without supervision. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume of Proceedings of Machine Learning Research, pp.
Stockholmsmässan, Stockholm Sweden, July 2018. PMLR.
Jingwei Xu, Bingbing Ni, and Xiaokang Yang. Video prediction via selective sampling. In Samy Bengio, Hanna Wallch, Hugo Larochelle, Kristen Grauman, Nicolò Cesa- Desa-Bianchhi, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 1705-1715. Curran Associates, Inc., 2018.
Tianfan Xue, Jiajun Wu, Katherine L. Bouman, and Bill Freeman. Visual dynamies: Probabilistic future frame synthesis via cross convolutional networks. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg. Isabelle Guyon, and Garnett Roman (eds.), Advances in Neural Information Processing Systems 29, pp. 91-99. Curran Associates, Inc., 2016.
Çağatay Yildiz, Markus Heinonen, and Harri Lähdesmäki. ODE Deep generative second order ODEs with Bayesian neural networks. arXiv preprint arXiv: 2019.
Li Yingzhen and Stephan Mandt. Disentangled sequential autoencoder. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 5670-5679, Stockholmsmässan, Stockholm Sweden, July 2018. PMLR.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Ruslan R Salakhutdinov, and Alexander J Smola. Deep sets. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna Wallach, Rob Fergus, S. V. N. Vishwanathan, and Garnett Roman (eds.), Advances in Neural Information Processing Systems 30, pp. 3391-3401. Curran Associates, Inc., 2017.
Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metrie. JEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 586-593 June 2018.
A EVIDENCE LOWER BOUND
We develop in this section the computations of the variational lower bound for the proposed model.
Using the original variational lower bound of Kingma & Welling (2014) in Equation (8):
log p( u)
2 ((ur log p(trr Be.r, Wr.r, w) DRL. (qz,y || p(y):T, Z2.T w))
(8)
Eugur (ear) Maz log ((p1(
Ser, Uur, un) DKL. (g(v), (2: m1.T) || p(v1, 22.T))
= -Erar (og.
Z2.T log p Dr, w) DKI Z2,7 x || p(y), (27)) (10)
14

---

Under review as a conference paper at ICLR 2020
where:
Equation (9) is given by the forward and inference models factorizing and in Equations (4)
and (5) and illustrated by, respectively, Figures la and lb:
the variables and y1 independent from to p and q
the y2:7 variables are deterministic functions of y1 and #2:7 with respect to p and q;
Equation (10) results from the factorization of p(æ1:r y1.r, 21:7, w) in Equation (4).
From there, by using the integral formulation of DKL:
log p(p w)
2 B(zer (ir)))))))))
log p( Dr, w)
1=1
(1I)
+ *
p(y1, #2.T)
(v), 22.7
dz2.r dy
q 22.r (1:r)
log p
Dr, w)
(Kl(1 (1rT) p(y1))
(12)
pl #2:T 51 ((
+ Egnne(en (1rr)
((22TT Wi.7, Ø1) log
d22.7
a1.r, 91)
#2.7
= -E - (ir))
log p( Tr, w) (Kl( x p(y1))
t=1
(13)
f
a(22:T a..r 01) log p(z2:r W1
+ Egrwy(tn wiek)
d22.7
((22:T FI.T, (1.r,
22/T
= (i
log p(
Ür. w) DKI (x p(31))
1m11
(14)
pizz J1 22.111 dz2r
+ Egg ~ mis)
Il a(z x1a))
log
g(zz x
2m
2m
B(aam:00)))02 logp(z; Tr. w) DKL x1ik) (((1))
Egrsalnn aaa) DKL (1(22 mi:e) || p(z2 Øi))
(15)
+ Bin mns) E (< m11))
p(zx yı, Ze.t dzyT
I a(z v1:)) Σ
log g(z1 x1a))
23.7 E33
1=3
where:
Equation (13) follows from the inference model of Equation (5), where yn only depends on
Equation (14) is obtained from the factorizations of Equations (4) and (5).
By iterating Equation (15)'s step on ,3...., and factorizing all expectations, we obtain:
(16)
log p( w)
15

---

Under review as a conference paper at ICLR 2020
2 E (Zur Wur)oqzz.
log p(
Dr. w)
DKL (a(v) 1x1x) p
(17)
- Migli mz) Bi
@1:e]]
Σ
DRL (a(ze mns) || p(zz 5í 1));
1=2
t=2
and we finally retrieve Equation (6) by using the factorization of Equation (5): log p(11:T w)
2 B (ur))
log p( De, w)
DKL (g(y1 x1:x) [(())))
1=1
(18)
DKL (gl( xi1)) p(zz 1)).
B DATASETS DETAILS
B.1 STOCHASTIC MOVING MNIST (SM-MNIST)
This dataset consists in one or two train MNIST digits (LeCun et al., 1998) of size 27 x 27 moving linearly within a 64 X 64 frame and randomly bounce against its border, sampling new direction and velocity at each bounce (Denton & Fergus, 2018). We use the same settings as Denton & Fergus (2018), train all models 15 timesteps and condition them test time frames. Note that we adapted the dataset to sample more coherent bounces: the original dataset computes digit trajectories that dependent on the chosen framerate, unlike our corrected version of the dataset. We consequently retrained SVG on this dataset, obtaining comparable results as those originally presented by Denton & Fergus (2018). Test data were produced by generating 5000 samples with different digit for each sequence coming from the MNIST test set.
B.2 KTH ACTION DATASET (KTH)
This dataset is composed real-world 64 64 videos of 25 people performing one six actions (walking, jogging, running, boxing, handwaving and handclapping) in front of different backgrounds (Schüldt et al., 2004). Uncertainty lies in the appearance of subjects, the action they perform and how it is performed. The training set is formed with actions from 20 people, the remaining five being used for testing. Training performed by sampling sub-sequences of size 20 in the train set. The test set composed of 1000 randomly sampled sub-sequences of size 40.
B.3 BAIR ROBOT PUSHING DATASET (BAIR)
This dataset contains 64 x 64 videos Sawyer robotic arm pushing objects on a tabletop (Ebert et al., 2017). It is highly stochastic as the arm can change its direction at any moment. Training is performed on 12 frames and testing is done with two conditioning frames on the provided test set, consisting of 256 sequences of 30 frames.
C TRAINING DETAILS
C.I SPECIFICATIONS
We used Python 3.7.4 and PyTorch 1.2.0 (Paszke al., 2017) implement model. Each model was trained on a Nvidia GPUs with CUDA 10 in mixed-precision training with the help of Apex.
C.2 ARCHITECTURE
Encoder and decoder architecture. Both go and hs chosen to have different architectures depending on the dataset. We used the same architectures as Denton & Fergus (2018): DCGAN
Phtpps:/ ://gi thub. cwm/nvid/iiooooo aapex.
16

---

Under review as a conference paper at ICLR 2020
discriminator and generator architecture (Radford al., 2016) for Moving MNIST, and VGG16 (Simonyan & Zisserman, 2015) architecture (mirrored for h2) for BAIR and KTH. both cases, the output of h (i.e., I) is a vector of size 128, and 9o and h weights initialized using centered normal distribution with a standard deviation 0.02.
For the Moving MNIST dataset, the content variable is obtained directly from E. and is thus vector size 128. For KTH and BAIR, we supplement this vectorial variable with skip connections from all layers of the encoder go that then fed the decoder he handle complex backgrounds. For Moving MNIST, the number of frames k used to compute the content variable is 5; for KTH, it is 3; for BAIR, it 2.
LSTM architecture. The LSTT used for all datasets has single layer of LSTM cells with hidden state size of 256.
MLP architecture. All MLPs used in inference (with parameters Φ) have three linear layers with hidden size 256 and leaky ReLU activations. All MLPs used in the forward model (with parameters Ø) have four linear layers with hidden size 512 and leaky ReLU activations. Weights of fa. in particular, orthogonally initialized with gain of 1.41, while the other MLPs initialized with default weight initialization of Py PyToch.
Sizes of latent variables. The sizes the latent variables our model are the following: for Moving MNIST, y and z have size 20; for KTH and BAIR, y and have size 50.
Euler step size All models but those trained on KTH are trained with Al 1. Models on KTH are trained with At
2-
C.3 OPTIMIZATION
Loss function. All models are trained using the Adam optimizer (Kingma & Ba, 2015) with learning rate 3 x 10-4 and = 1. The batch size for Moving MNIST and BAIR chosen to be 128, and the batch size for KTH is chosen to be 100.
Following (Higgins et al., 2017), we use
(cf. Equation (7)), except for the Moving MNIST
dataset where the 3 factor in front of the KL on (last term of Equation (6)) is equal to 2.
Variance of the observation. The variance v used in the observation probability distribution G (ga(z)) No (I) chosen as follows:
for Moving MNIST, // == 1;
for KTH, / = X 10 10-2, 2-
for BAIR, // 1/4
Number of optimization steps. datasets:
The number optimization steps is the following for the different
Moving MNIST (stochastic): 1 000 000 steps with additional 100 000 steps where the learning rated is linearly decreased to O;
Moving MNIST (deterministic): 800 000 steps with additional 100 000 steps where the learning rated is linearly decreased to 0;
KTH: 200 000 steps, the final model being chosen among several checkpoints the one having the best evaluation score (which differs from the test score as we extract from the train evaluation set);
BAIR: 250 000 steps, the final model is chosen for KTH.
17

---

Under review as a conference paper at ICLR 2020
Table 1: Numerical results (mean and 95%--confidnnc interval) for PSNR and SSIM for tested methods on the two-digits Moving MMNIST dataset. Bold scores indicate the best performing method and, where appropriate, scores whose means lie in the confidence interval of the best performing method.
Stochastic
Deterministic
Models
PSNR
SSIM
PSNR
SSIM
SVG
14.45 + 0.06
0.7070 + 0.0021
12.93 = 0.05
0.6245 + 0.0022
Ours
16.90 + 0.09
0.7789 0.0025
16,49
0.06
0.7808 + 0.0020
Ours MLP
16.55 + 0.09
0.7693 0.0024
14.32 + 0.06
0.6895 0.0023
Ours GRU
15.81 +: 0.08
0.7463 + 0.0023
13.16 + 0.05
0.6318 + 0.0022
Table Numerical results (mean and 95%-confidence interval, when relevant) for PSNR, SSIM. LPIPS, and FVD for tested methods on the KTH dataset. Bold scores indicate the best performing method for each metric and, where appropriate, scores whose means lie in the confidence interval of the best performing method.
Models
PSNR
SSIM
LPIPS
FVD
SV2P
28.18 + 0.39
0,8141 + 0.0068
0.2049 + 0.0080
636
SAVP
26.51 + 0.36
0.7560 = 0.0083
376
SVG
26.99 0.33
0.8291 + 0.0074
0.1083 0.0058
265
Ours
29.69 + 0.37
0.8697 + 0.0057
1 0.0736 0.0036
224
Ours GRU
29.13 #
0.8590 + 0.0060
0.0790
0.0039
237
Ours - MLP
29.49 + 0.38
0.8626 + 0.0061
0.0825 + 0.0042
290
D ADDITIONAL NUMERICAL RESULTS
Tables to present, respectively, numerical results for PSNR, SSIM and LPIPS averaged over all time steps for methods and considered baselines on the SM-MNIST, KTH and BAIR datasets, corresponding to Figures 2, and 6.
E PENDULUM EXPERIMENTS
We the ability of our model model the dynamics of common dataset used in the literature of state-space models (Karl et al., 2017; Fraccaro et al., 2017), Pendulum (Karl et al., 2017). It consists of noisy observations of dynamic torque- torque- pendulum; it stochastic as the information of this control is not available. We test our model, without the content variable w. in the same setting as DVBF (Karl et al., 2017) and VVAE (Fraccaro et al., 2017) and report the corresponding ELBO scores in Table 4. The encoders and decoders for all methods are MIPs.
Our model outperforms DVBF and merely beaten by KVAE This can be explained by the nature of the KVAE model, whose sequential model is not learned using VAE but Kalman filter allowing
Table 3: Numerical results (mean and 95%--cnffidene interval, when relevant) with respect to PSNR, SSIM, LPIPS, and FVD for tested methods on the BAIR dataset. Bold scores indicate the best performing method for each metric and, where appropriate, scores whose means lie in the confidence interval the best performing method.
Models
PSNR
SSIM
LPIPS
FVD
SV2P
20.39
0.42
0.8169 + 0.0110
0.0912 + 0.0063
955
SAVP
18.44 + 0.40
0.7886
0.0117
0.0634 + 0.0048
152
SVG
18.95 + 0.41
0.8057
+ 0.0116
0.0609 +: 0.0046
253
Ours
19.64 + 0.45
0.8211
0.0110
0.0610 + 0.0048
181
18

---

Under review as a conference paper at ICLR 2020
Table 4: ELBO score for DVBF, KVAE and our model on the Pendulum dataset. The bold score indicates the best performing method.
Score
DVBF
KVAE
Ours
ELBO
798.56
807.02
806.12
Table 5: Numerical results for PSNR, SSIM, and LPIPS on BAIR of our model trained with At- and tested with different values of At.
Step size At
PSNR
SSIM
LPIPS
At =
19.64 + 0.45
0.8210 + 0.0110
0.0612 0.0048
At 19.76 = 0.44
0.8235 0.0110
0.0597 = 0.0047
At 19.82 0.45
0,8245 0.0111
0.0593
0.0048
AI 19.83 + 0.46
0.8242 + 0.0111
0.0593
0.0049
At
19.85 = 0.46
0.8243 = 0.0111
0.0591
0.0048
exact inference in the latent space. On the contrary, DVBF is learned, like our model, by a sequential VAE, and is thus much closer to model than KVAE. This result then shows that the dynamic model that we chose in the context of sequential VAEs is more adapted on this dataset than the one of DVBF, and achieve results close method taking advantage of exact inference using adapted tools such as Kalman filters.
F INFLUENCE OF THE EULER STEP SIZE
Table 5 details the numerical results of our model trained on BAIR with Al = and tested with different values of Av. It shows that, when refining the Euler approximation, our model can improve its performance in a setting that is unseen during training. Results stabilize when At is small enough, showing that the model is close to the continuous limit.
Tables 6 and 7 details the numerical results of our model trained on KTH with, respectively, At 1 and At 2. and tested with different values of At. They show that At is chosen too high when training (here, At = 1). the model drops in performance when refining the Euler approximation. We assume that this phenomenon arises because the Euler approximation used training too rough, making the model adapt to very discretized dynamic that cannot be transferred smaller Euler step sizes. Indeed, when training with smaller step size, (here, At = }) results in the training settings equivalent while results obtained with lower At the nominal ones. This shows that the model learned a continuous dynamic learned with a small enough step size.
much closer, if not equivalent, to
Note that the loss of performance using higher At testing than in training, like in Table 7, is expected it corresponds loosening the Euler approximation compared training. However, even in this adversarial setting, our model maintains state-of-the-ar results, demonstrating the quality of the learned dynamic it can be further discretized if needed the cost of reasonable drop in performance.
Table Numerical resuls for PSNR, SSIM, and LPIPS on KTH our model trained with At 1 and tested with different values of A/.
Step size At
PSNR
SSIM
LPIPS
At
29.76 + 0.38
0.8681 +: 0.0057
0.0737
0.0057
At
29,05 = 0.42
0.8539 + 0.0066
0.0882
0.0050
At
29.05 + 0.42
0.8509 + 0.0069
0.0924
0.0055
At
28.98 0.42
0.8496 + 0.0069
0.0939
0.0056
I
AL
28.95 + 0.42
0.8490 + 0.0070
0.0948 0.0057
19

---

Under review as a conference paper at ICLR 2020
Table Numerical results for PSNR, SSIM, and LPIPS on KTH of our model trained with At and tested with different values of At.
Step size At
PSNR
SSIM
LPIPS
At
28.80 + 0.41
0.8495 + 0.0068
0.0994
0.0057
At
29.69 + 0.37
0.8697 + 0.0057
0.0736
0.0036
At
29.52 0.38
0.8656 = 0.0059
0.0777
0.0041
29.43 = 0.39
0.8633 = 0.0061
0.0790
0.0042
At
29.35 0.39
0.8615 + 0.0062
0.0810 0.0045
SUG-EP YOR
0J0
SVC-LP- DOGAN
VING
0.10
DOGAN
0,00
15
0.83
0.12
0.00
0.38
0.50
10
15
20
30
111
15
25
30
35
331
22
25
Figure 10: PNSR, SSIM and LPIPS scores with respect to on the KTH dataset for SVG and our model with two choices of encoder and decoder architecture for each: DCGAN and VGG.
G AUTOREGRESSIVITY AND IMPACT OF ENCODER AND DECODER ARCHITECTURE
Figure 10 exposes the numerical results on KTH of our model and SVG for different choices of architectures: DCGAN and VGG.
Since DCGAN is less powerful architecture than VGG, results of each method with VGG expectedly better than those of the same method with DCGAN. Moreover, our model outperforms SVG for any fixed choice of encoder and decoder architecture, which coherent with Figure 4.
We observe, however, that the difference between a method using VGG and its DCGAN counterpart differs depending on the model. Ours shows more robustness changing of encoder and decoder ar- chitecture, as it loses much less performance than SVG when switching to less powerful architecture. Indeed, while the difference in LPIPS is similar for both models (as expected from score evaluating the realism of produced frames), the loss of SVG is significantly larger than our loss in terms of SSIM, and in particular PNSR. This shows that reducing the capacity of the encoders and decoders of SVG not only hurts its ability to produce realistic frames expected but also substantially lowers its ability to learn a good dynamic. We assume that this phenomenon is caused by the autoregressive nature of SVG, which makes it reliant of the performance of its encoders and decoders. This supports our motivation to propose a non-autoregressive model for stochastic video prediction.
H ADDITIONAL. SAMPLES
This section includes some additional samples corresponding to experiments described in Section
H.1 STOCHASTIC MOVING MNIST
We present in Figures 11 14 additional samples from SVG and model SM-MNIST.
In particular, Figure 13 shows SVG changing a digit shape in the course of prediction even though does not cross another digit, whereas maintain the digit shape. We assume that this advantage of ours comes from the latent nature of the dynamic of our model and the use our of a static content variable that is prevented from containing temporal information. Indeed, even when the best sample
20

---

Under review as a conference paper at ICLR 2020
2
2
21
Ground Truth
/
5
12
12
122
122
12
SVG
2
5
/
/2
1
1
1
12
12
12
6
b
OuR (Best)
2
12
2
2
2
/
2
/
2
2
Outs (Random) (Wors)
2
/
2
2
2
/
/
2
2
/
Ours
2
2
2
2
/o2
2
2
2
2 7
Figure 11: Conditioning frames and corresponding ground truth and best samples with respect PSNR from SVG and our method, and worst and random samples from our method, for an example of the SM-MNIST dataset.
3
Ground Truth
#
4/
43
4
3
3
2
43
1/
V3
43
23
3
3
2
30
30
Ours (Best)
1/3
4
43
46
46
40
3
&
4
Our (Randon)) (Wors.)
4
4/
3
4/
3
4
3
3
4/
4
3
4
#
Oues
3
3
3
3
3
4
4
4
4
Figure 12: Additional samples for the SM-MNIST dataset (ef. Figure 11).
20
Ground Truth
20
2
2
2
2
2
3
3
2
O
O
O
20
20
2
7
3
3
3
O
O
Ours (Best)
20
20
2
2
2
2
2
O
O
O
O
O
Ours (Randenn) (Wosi))
20
2 O
O
2
O
2
O
o
2
O
2
o
2
2
20
20
6
2
Ours
Figure 13: Additional samples for the SM-MNIST dataset (cf. Figure 11). SVG fails to maintain the shape of digit, while ours is temporally coherent.
21

---

Under review as a conference paper at ICLR 2020
g
Ground Truth
0
9p
0
82
88
5
8
80
8
69
60
8
8
8
4
80
8
0
8
80
O
Ours (Beat)
O 8
8
7
8
8
8
8
(Wors))
8
8
8
8
8
8
0
8
Ours (Random)
8
8
8
8
8
Figure 14: Additional samples for the SM-MNIST dataset (cf. Figure 11). This example chosen in the worst 1% test examples of our model with respect to PSNR. Despite this adversarial criterion, model maintains temporal consistency as digits not deformed in the course of the video.
Ground (Random) (Wors)) (Bei) SAVP SVG SV2P Truch
1
f
1
Ours
t
t
t
Ours
Our
Figure 15: Conditioning frames and corresponding ground truth, best samples from SVG, SAVP and our method, and worst and random samples from our method, for an example the KTH dataset. Samples are chosen according to their IPIPS with respect the ground truth. On this specific task (clapping), all methods but SV2P (which produce blurry predictions) perform well, even though ours stays closer to the ground truth.
from our model not close from the ground truth of the dataset, like in Figure 14, the shapes of the digits are still maintained by our model.
H.2 KTH
We present in Figures 15 to 19 additional samples from SV2P, SVG, SAVP and our model on KTH, with additional insights.
H.3 BAIR
We present Figures 20 22 additional samples from SV2P, SVG, SAVP and our model on BAIR, with additional insights.
H.4 OVERSAMPLING
We present in Figure 23 additional examples of video generation at a doubled frame rate by our model.
22

---

Under review as a conference paper at ICLR 2020
Ground
SAVP SVG SV2P Thuth
Ours
(Randon)) (Woes)) (Bes)
Ours
Ours
Figure 16: Additonal samples for the KTH dataset (cf. Figure 15). In this example, the shadow of the subject is visible in the last conditioning frames, foreshadowing its appearance. This is failure case for SVG and SAVP which only produce an indistinet shadow, whereas SAVP and our model make the subject appear. Yet, SAVP produces the wrong action and an inconsistent subject in its best sample, while ours is correct.
Ground (Random) (Wors)) (Bes)) SAVP SVG SV2P Trith
Outs
Outs
Oues
Figure 17: Additonal samples for the KTH dataset (cf. Figure 15). This example is a failure case for each method: SV2P produce blurry frames, SVG and SAVP are not consistent (change of action or subject appearance in the video), and our model produces ghost image at the end of the prediction on the worst sample only.
Ground SAVP SVG SV2P Truth
Our
Ours (Random) (Wors)) (Beat)
Outs
Figure 18: Additonal samples for the KTH dataset (cf. Figure 15). Our model is the only one to make a subject appear in the ground truth.
23

---

Under review as a conference paper at ICLR 2020
Ground
Ours (Random)) (Warst) (Best) SAVP SVG SV2P Troth
Ours
Ous
Figure 19: Additonal samples for the KTH dataset (cf. Figure 15). The subject in this example is boxing, which is the most challenging action in the dataset as all methods are far from the ground truth.
Ground SVG SV2P Truth
SAVP
Ows Ours (Randon) (Worst) (Box)
Figure 20: Conditioning frames and corresponding ground truth, best samples from SVG, SAVP and our method, and worst and random samples from our method, for an example of the BAIR dataset. Samples are chosen according their LPIPS with respect to the ground truth.
Ground SVIP Thith
SVG
SAVP
(Best)
Onrs (Randon)) (Wors))
Figure 21: Additonal samples for the KTH dataset (cf. Figure 20).
24

---

Under review as a conference paper at ICLR 2020
Ground Truth
SV2P
SVG
SAVP
Ours (Randoon) (Wons) (Bea)
Figure 22: Additonal samples for the KTH dataset (ef. Figure 20).
(a) KTH sample, centered the subject.
(b) BAIR sample, centered the robot arm.
Generation examples doubled frame rate, using halved At compared to training. Frames including bottom red dashed bar are intermediate frames.
Figure 23:
H.S
CONTENT SWAP
We present in Figures 24 to 28 additional examples of content swap in Figure 8.
H.6
INTERPOLATION IN THE LATENT SPACE
We present in Figures 29 and 30 additional examples of interpolation in the latent space between two trajectories.
Swap Dynamic
Content
Figure 24: Video (bottom right) generated from the combination of dynamic variables (y, x) inferred with a video (top) and the content variable (w) computed with the conditioning frames another video (bottom left).
25

---

Under review as a conference paper at ICLR 2020
Dynamic
Conteat
Swap
Figure 25; Additional example of content swap (cf. Figure 24).
Dynumic
Content
Swap
Figure 26: Additional example of content swap (ef. Figure 24). In this example, the extracted content the video background, which successfully transferred to the target video.
Swap Dynamic
Content
Figure 27: Additional example of content swap (ef. Figure 24). In this example, the extracted content the video background and the subject appearance, which are successfully transferred to the target video.
Soap Dynumic
Content
Figure 28: Additional example of content swap (cf. Figure 24). This example shows a failure case of content swapping.
(a) Ref
(b) Rec
(c) Interpolation
(d) Rec
(e) Ref 2
Figure 29: From left to right, x", x² (reconstruction of x by the VAE of our model), results of the interpolation in the latent space between æ² and x², Øt and æ². Each trajectory is materialized in shades of grey in the frames.
(a) Ref 1
(b) Rec
(c) Interpolation
(d) Rec
(e) Ref 2
Figure 30:
Additional example interpolation in the latent space between two trajectories (cf.
Figure 29).
26


Reviews:
[{"review_id": "r1xt2GQpFH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a video prediction model which explicitly decouples frame synthesis and motion dynamics. This is a very subtle change (compared to the current models) that can result in higher quality predictions.\n\nFirst of all, the paper is extremely well written. It provides clear motivations and goals, as well as an impressively comprehensive related work that discusses their shortcomings. The experiments are comprehensive and provide good support for the claims. And finally, the appendix presents additional visualization and information. \n\nOn the main proposed method, it is a very subtle but reasonable change. Therefore, my suggestion to the authors is to provide a more thorough comparison with existing methods specifically SVG (Denton 2018) since the models share a lot of similarities. It is also quite similar to PlaNet (Hafner 2019). This is where the paper can be improved.\n\nFor the experiments, although they are quite comprehensive, there is still room for improvement. First, none of the metrics used are good evaluation metrics for frame prediction (I know they are quite common but that doesn't make them good) as they do not give us an objective evaluation in the sense of the semantic quality of predicted frames, specially for long videos. It really helps if the authors present additional quantitative evaluation to show that the predicted frames contain useful semantic information with metrics such as FVD and Inception score. Second, a pretribulation study is required to see where the improvements are coming from. Is it from a different architecture or the separation of dynamics?  Finally, a website with generated videos really helps for qualitative comparison! \n\nOverall, this is a well-written paper with clear motivations and goals. I find the impact of the paper to be marginal (given the quality difference with already existing models) which can be improved by emphasizing more on other aspects such as disentanglement.  ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "r1gsleMpKS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Contributions: this submission proposes a video pixel generation framework with the goal to decouple visual appearance and dynamics. The latent dynamics are modeled with a latent residual dynamics model. Empirical evaluations on moving MNIST show that the proposed residual dynamics model outperform MLP or GRU. On more challenging KTH and BAIR datasets, the proposed method achieves on par or better quantitative performance with previous methods, and have nice qualitative results on content \"swap\" and dynamics interpolation.\n\nAssessment:\n- To my knowledge, the proposed model is novel for video generation.\n- The proposed method is supported with strong quantitative results and qualitative analysis, ablation on Moving MNIST shows that the proposed latent residual dynamics model outperforms MLP and GRU baselines.\n- The authors might be interested in related work on video generation with decoupled appearance and dynamics models, such as [1]. It would also be interesting to see evaluation on more challenging datasets, such as Human3.6M.\n- Question: how does the proposed inference framework make sure to decouple appearance with dynamics? Can y_i not encode the appearance information?\n\n\n[1] Minderer et al., Unsupervised Learning of Object Structure and Dynamics from Videos. NeurIPS 2019.\n\n-----------------------------\nPost rebuttal:\nThank you for your answers to my questions and the updated manuscript. My questions have been addressed and the additional results further confirm the performance of the proposed method. Therefore I recommend weak accept of the submission.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "S1efBB1AYS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe paper proposes a video prediction method based on State-Space Models. The paper describes two main contributions:\n\n1. By learning dynamics in the latent state space, the method avoids the high computational cost and accumulating image reconstruction errors of autoregressive models that condition on generated frames.\n\n2. To model dynamics, the paper proposes a residual update rule inspired by Euler’s method to solve ODEs. According to this rule, the update to the latent state y_t is modeled as an additive residual f(y_t, z_{t+1}). This has the advantage that the step size of the discretization can be adjusted freely, e.g. between training and inference. \n\nThe paper provides extensive experimental comparison of their model to the SVG and SAVP models on several standard datasets. The paper further contains experiments illustrating features of the model such as disentangling dynamics and content, and interpolation of dynamics in the latent space.\n\nDecision:\n\nThe paper is written clearly and the mathematical treatment and experiments appear rigorous. The idea of predicting video using state-space models is interesting and promising. However, as described below, the paper overstates its novelty and falls short of showing the advantages of the method beyond incremental improvements on frame-wise image quality metrics. I therefore suggest rejection in its current version.\n\nSupporting arguments and suggestions:\n\n1. The idea to use fully latent models for video prediction, to untie frame synthesis and dynamics, is not new and the paper does not fully cite this literature. For example, [1] and [2] perform unsupervised, non-autoregressive video prediction. The differences to these models should be discussed.\n\n2. The advantages of the residual update rule are not made clear enough. The parallels to the ODE literature seem tenuous. The main advantage described in the paper is the ability to synthesize videos at different frame rates, but interpolation over such short time horizons is not a hard problem. At least, the paper should compare to existing methods for frame interpolation. Apart from interpolation (variable step size), it appears that the update rule could be changed from y_{t+1} = y_t + f(y_t, z_{t+1}) to y_{t+1} = f(y_t, z_{t+1}) without impact to the model. How is it different from the standard VRNN formulation [3]? More experiments to show the advantage of the proposed update rule would be helpful.\n\n3. Some of the experiments seem like interesting starting points but do not support general claims. For example, Fig 2 (b) shows that the proposed dynamics model is better than an MLP or GRU on deterministic Moving MNIST, but is this also true on real datasets, which have much more complex dynamics? Similarly, the interpolation in Figure 9 is intriguing, but it would be helpful to describe and test how this ability is useful for applications of the predictive model.\n\n4. The comparisons use frame-wise metrics of image quality (PSNR, SSIM, LPIPS). Even though they are common in the literature, these metrics are unsuitable for comparing long video sequences due to their stochasticity. The metrics are probably dominated by relatively uninteresting features such as the quality of the static background. Metrics for comparing entire videos exist (e.g. FVD [4]) and should be used. Even better, the paper should demonstrate the usefulness of the model for downstream tasks such as reinforcement learning, although I understand that this may be out of scope.\n\nMinor comments:\n\n- As far as I know, the correct term for error terms is residual, not residue.\n- What do the error bars in the figures show? Please add this information to the figure legends.\n\n[1] Wichers et al., 2018, https://arxiv.org/pdf/1806.04768.pdf\n[2] Minderer et al., 2019, https://arxiv.org/abs/1906.07889\n[3] Chung et al, 2015, https://arxiv.org/abs/1506.02216\n[4] Unterthiner et al., 2018, https://arxiv.org/abs/1812.01717", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a method for learning a latent dynamics model for videos. The main idea is to learn a latent representation and model the dynamics of the latent features via residual connection motivated by ODE. The architectural choice of residual connection itself is not new as many prior works have employed "skip connections" in hidden representations but the notion of connecting this with ODE and factoring time as input into the residual function seems a new idea. The experimental results show the promise of the proposed method on moving MNIST, KTH, and BAIR datasets. The experiments on different frame rates are also nice.  In terms of weakness, the evaluation is performed on relatively simple domains (e.g., moving MNIST and KTH) with static backgrounds and the improvement on BAIR dataset (which is not considered as a difficult benchmark) in terms of FVD is not as clear. For the BAIR dataset, it's unclear how the proposed method will handle the interactions between the robot arm and background objects due to the modeling assumption (i.e., static background). In this sense, content swap results on BAIR dataset look quite anecdotal, and the significance is limited. For improvement, I would suggest adding evaluations on other challenging domains, such as Human 3.6M (where human motions are much more uncertain compared to KTH) and other Robot datasets with more complex robot-object interactions. Overall, the paper proposes an interesting architecture with promising results on relatively simple datasets, but the advantage over existing SOTA methods on challenging benchmarks is unclear yet.

Decision: Reject