Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: Summary:

The paper proposes a video prediction method based on State-Space Models. The paper describes two main contributions:

1. By learning dynamics in the latent state space, the method avoids the high computational cost and accumulating image reconstruction errors of autoregressive models that condition on generated frames.

2. To model dynamics, the paper proposes a residual update rule inspired by Euler’s method to solve ODEs. According to this rule, the update to the latent state y_t is modeled as an additive residual f(y_t, z_{t+1}). This has the advantage that the step size of the discretization can be adjusted freely, e.g. between training and inference. 

The paper provides extensive experimental comparison of their model to the SVG and SAVP models on several standard datasets. The paper further contains experiments illustrating features of the model such as disentangling dynamics and content, and interpolation of dynamics in the latent space.

Decision:

The paper is written clearly and the mathematical treatment and experiments appear rigorous. The idea of predicting video using state-space models is interesting and promising. However, as described below, the paper overstates its novelty and falls short of showing the advantages of the method beyond incremental improvements on frame-wise image quality metrics. I therefore suggest rejection in its current version.

Supporting arguments and suggestions:

1. The idea to use fully latent models for video prediction, to untie frame synthesis and dynamics, is not new and the paper does not fully cite this literature. For example, [1] and [2] perform unsupervised, non-autoregressive video prediction. The differences to these models should be discussed.

2. The advantages of the residual update rule are not made clear enough. The parallels to the ODE literature seem tenuous. The main advantage described in the paper is the ability to synthesize videos at different frame rates, but interpolation over such short time horizons is not a hard problem. At least, the paper should compare to existing methods for frame interpolation. Apart from interpolation (variable step size), it appears that the update rule could be changed from y_{t+1} = y_t + f(y_t, z_{t+1}) to y_{t+1} = f(y_t, z_{t+1}) without impact to the model. How is it different from the standard VRNN formulation [3]? More experiments to show the advantage of the proposed update rule would be helpful.

3. Some of the experiments seem like interesting starting points but do not support general claims. For example, Fig 2 (b) shows that the proposed dynamics model is better than an MLP or GRU on deterministic Moving MNIST, but is this also true on real datasets, which have much more complex dynamics? Similarly, the interpolation in Figure 9 is intriguing, but it would be helpful to describe and test how this ability is useful for applications of the predictive model.

4. The comparisons use frame-wise metrics of image quality (PSNR, SSIM, LPIPS). Even though they are common in the literature, these metrics are unsuitable for comparing long video sequences due to their stochasticity. The metrics are probably dominated by relatively uninteresting features such as the quality of the static background. Metrics for comparing entire videos exist (e.g. FVD [4]) and should be used. Even better, the paper should demonstrate the usefulness of the model for downstream tasks such as reinforcement learning, although I understand that this may be out of scope.

Minor comments:

- As far as I know, the correct term for error terms is residual, not residue.
- What do the error bars in the figures show? Please add this information to the figure legends.

[1] Wichers et al., 2018, https://arxiv.org/pdf/1806.04768.pdf
[2] Minderer et al., 2019, https://arxiv.org/abs/1906.07889
[3] Chung et al, 2015, https://arxiv.org/abs/1506.02216
[4] Unterthiner et al., 2018, https://arxiv.org/abs/1812.01717
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: The paper proposes a video prediction model which explicitly decouples frame synthesis and motion dynamics. This is a very subtle change (compared to the current models) that can result in higher quality predictions.

First of all, the paper is extremely well written. It provides clear motivations and goals, as well as an impressively comprehensive related work that discusses their shortcomings. The experiments are comprehensive and provide good support for the claims. And finally, the appendix presents additional visualization and information. 

On the main proposed method, it is a very subtle but reasonable change. Therefore, my suggestion to the authors is to provide a more thorough comparison with existing methods specifically SVG (Denton 2018) since the models share a lot of similarities. It is also quite similar to PlaNet (Hafner 2019). This is where the paper can be improved.

For the experiments, although they are quite comprehensive, there is still room for improvement. First, none of the metrics used are good evaluation metrics for frame prediction (I know they are quite common but that doesn't make them good) as they do not give us an objective evaluation in the sense of the semantic quality of predicted frames, specially for long videos. It really helps if the authors present additional quantitative evaluation to show that the predicted frames contain useful semantic information with metrics such as FVD and Inception score. Second, a pretribulation study is required to see where the improvements are coming from. Is it from a different architecture or the separation of dynamics?  Finally, a website with generated videos really helps for qualitative comparison! 

Overall, this is a well-written paper with clear motivations and goals. I find the impact of the paper to be marginal (given the quality difference with already existing models) which can be improved by emphasizing more on other aspects such as disentanglement.  
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: Contributions: this submission proposes a video pixel generation framework with the goal to decouple visual appearance and dynamics. The latent dynamics are modeled with a latent residual dynamics model. Empirical evaluations on moving MNIST show that the proposed residual dynamics model outperform MLP or GRU. On more challenging KTH and BAIR datasets, the proposed method achieves on par or better quantitative performance with previous methods, and have nice qualitative results on content "swap" and dynamics interpolation.

Assessment:
- To my knowledge, the proposed model is novel for video generation.
- The proposed method is supported with strong quantitative results and qualitative analysis, ablation on Moving MNIST shows that the proposed latent residual dynamics model outperforms MLP and GRU baselines.
- The authors might be interested in related work on video generation with decoupled appearance and dynamics models, such as [1]. It would also be interesting to see evaluation on more challenging datasets, such as Human3.6M.
- Question: how does the proposed inference framework make sure to decouple appearance with dynamics? Can y_i not encode the appearance information?


[1] Minderer et al., Unsupervised Learning of Object Structure and Dynamics from Videos. NeurIPS 2019.

-----------------------------
Post rebuttal:
Thank you for your answers to my questions and the updated manuscript. My questions have been addressed and the additional results further confirm the performance of the proposed method. Therefore I recommend weak accept of the submission.
review_title: Official Blind Review #3

Meta-review: The paper proposes a method for learning a latent dynamics model for videos. The main idea is to learn a latent representation and model the dynamics of the latent features via residual connection motivated by ODE. The architectural choice of residual connection itself is not new as many prior works have employed "skip connections" in hidden representations but the notion of connecting this with ODE and factoring time as input into the residual function seems a new idea. The experimental results show the promise of the proposed method on moving MNIST, KTH, and BAIR datasets. The experiments on different frame rates are also nice.  In terms of weakness, the evaluation is performed on relatively simple domains (e.g., moving MNIST and KTH) with static backgrounds and the improvement on BAIR dataset (which is not considered as a difficult benchmark) in terms of FVD is not as clear. For the BAIR dataset, it's unclear how the proposed method will handle the interactions between the robot arm and background objects due to the modeling assumption (i.e., static background). In this sense, content swap results on BAIR dataset look quite anecdotal, and the significance is limited. For improvement, I would suggest adding evaluations on other challenging domains, such as Human 3.6M (where human motions are much more uncertain compared to KTH) and other Robot datasets with more complex robot-object interactions. Overall, the paper proposes an interesting architecture with promising results on relatively simple datasets, but the advantage over existing SOTA methods on challenging benchmarks is unclear yet.

Decision: Reject