Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a method called Temporal Abstract Latent Dynamics (TALD). TALD is built up on RSSM (Hafner et al. 2019) but with hierarchical dynamics. The experiments are conducted on moving MNIST, GQN 3D Mazes, and KTH. Results are qualitatively better than other methods in term of maintaining long-term consistent prediction. Quantitative comparison is reported only on KTH dataset (Figure 5). Written presentation is clear and easy to understand.

Pros:
- The idea of modeling hierarchical latent variables for long-term video prediction is novel and interesting.
- TALD qualitatively and quantitatively (on KTH) outperforms other methods, such as RSSM (direct baselines), VTA, SVG.
- The experiments of removing one level of hierarchical latent (by replacing the posterior) is particularly interesting which proved that TALD can abstract different semantic information at different level its hierarchical representation.

Cons: 
- The comparisons on moving MNIST and GQN Mazes are only qualitatively (which may be biased to the selected sequences). The supplementary provides two more sequences (1 from moving MINIST and 1 from GQN Mazes) with qualitative results, but not quantitative results on these benchmarks. It should be better to report some sort of quantitative comparisons and evaluated over the full dataset instead (as done with KTH).
review_title: A good paper, novel idea, OK experiments

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper presents a recurrent network for long-term video prediction, which learns hierarchical latent states in space-time. Although it shows strong experimental results, I have the following concerns about the novelty of the proposed model and the completeness of the experiments: 
1. To me, the proposed TALD architecture can be seen as a combination of two previous models, RSSM and VTA, with small extensions. While I understand that they have many differences in technical details, such as the number of layers and the dependencies between the latent states, the key insight behind these approaches is similar. 
2. Hierarchical latent variables have long been used for video prediction, as in Wichers et al. (2018) and Xu et al. (2018). I think the authors should make more comparisons with these similar works both in terms of methods and experimental results.
[Xu et al., 2018] PredCNN: Predictive Learning with Cascade Convolutions
3. For both Moving MNIST and KTH datasets, the proposed model tasks as input 36 context frames, which is not a common practice of most previous models, including those from Wichers et al. (2018) and Denton et al. (2018). Given that the temporal dynamics is relatively simple in these datasets, it may not be necessary to use so much context information. What would happen if we reduce the number of input frames?
4. The authors put the descriptions of Figure 1 in Section 4.1 and the descriptions of Figure 2 in Section 4.3, which caused me some reading difficulties at the very beginning of the paper.
review_title: Well-experimented paper with limited novelty 

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper tackles the problem of long term video prediction. It proposes a novel sequential hierarchical VAE-based model called Temporal Abstract Latent Dynamics (TALD), that can keep long term consistency by having different levels of the hierarchy update at different frequencies. 
Empirically, the authors validate that the higher levels of the hierarchy do indeed encode for slower changing features in the videos. They also show that using temporal abstraction allows TALD to produce better predictions of long sequences (hundreds of frames) and outperform prior models in that setting.

################################################


Strong points:

- The paper is clear and easy to follow. 

- It deals with an important topic, as long-term consistency is a challenge in many settings for sequential deep learning models.

- The idea behind the model is simple and appealing.

- Thorough experimental results validate that the model behaves as expected.


Concerns:

- Mainly, evaluation of performance seems inadequate:

All of the considered models (including the proposed one) are of stochastic nature. They are trained to provide plausible sequence, and not retrieve the exact ground-truth sequence. Yet, the proposed evaluation metric ignores the stochastic aspect, assuming the models should be able to correctly retrieve the ground-truth sequence given the initial frames. 
Those metrics might be ill-suited, as small positional shift due to the stochasticity can propagate through time and lead to huge drops in performances even if the digits are correctly conserved the entire sequence and the trajectories mostly correct. This actually shows up when comparing Figure 1 (where all models have scores comparable to random frames at the end)  and Figure 4 that shows that the digits mostly remain unchanged.
Why weren't stochastic variants of PSNR, LPIPS and SSIM (as used in the original publication for SVG-LP), or Frechet Video Distance (maybe on subsampled frames of the sequences) considered instead? 

Also, the long-term consistency is mostly evaluated by human observation on samples. Maybe, a digit detector (and a wall/ground texture detector) could be used to provide a quantitative evaluation?

################################################

Score motivation:

The paper tackle an important problem with a well-motivated novel model. However, I have concerns about the quantitative evaluation methods.

################################################

Minor remarks:

- How are the initial states h_0 computed? This information might be needed for reproducibility.


The authors might also want to acknowledge (or compare to) some recent works in video generation:

- Stochastic Adversarial Video Prediction
Alex X. Lee, Richard Zhang, Frederik Ebert, Pieter Abbeel, Chelsea Finn, Sergey Levine

- VideoFlow: A Conditional Flow-Based Model for Stochastic Video Generation
Manoj Kumar, Mohammad Babaeizadeh, Dumitru Erhan, Chelsea Finn, Sergey Levine, Laurent Dinh, Durk Kingma

- Stochastic Latent Residual Video Prediction
Jean-Yves Franceschi, Edouard Delasalles, Mickael Chen, Sylvain Lamprier, Patrick Gallinari

- Scaling Autoregressive Video Models
Dirk Weissenborn, Oscar Täckström, Jakob Uszkoreit
review_title: A sequential hierarchical VAE for long-term video prediction. Quantitative evaluation could be improved.

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors present build on the recurrent state space model by Haffner et al. (2019) to include a hierarchy of latent spaces. The idea is that the lower levels model short-term changes while the higher levels of the hierarchy model longer temporal dependencies.

Pros:
1. The paper is largely well-written and easy to follow.
2. The experiments do show that the proposed method is able to capture the long-term trends better than the chosen baselines for the chosen datasets. 
3. Experiments showing different amounts of information present at different levels of the latent space are interesting and show that desired hierarchical structure in the latent space is achieved.

Cons:
1. In light of the paper "Improved conditional VRNNs for video prediction" by Castrejon et al., I do not believe there is enough novelty in this paper in terms of the latent space architecture, as this paper also applies a hierarchical latent space model for video prediction. The main contributions are in the analysis of the hierarchical architectures using information-theoretic measures for three datasets, which I believe are not sufficient for accepting this paper. The authors should clarify the main differences in the proposed method and the paper by Castrejon et al.

2. Related to the above point, the datasets for the experiments should be stronger as this paper's novelty is mainly empirical. The KTH action dataset is the closest to a real-world dataset, but is still simplistic with just one moving object and a fixed background. The authors should consider experimenting with more challenging datasets. This is necessary to understand the limitations of the proposed method. 

UPDATE AFTER AUTHOR RESPONSE:
I appreciate the authors' response and for clarifying the contributions. However I still feel that the experiments and datasets are too simple and not realistic. I am increasing my rating to reflect this.
review_title: An interesting paper with a need for stronger experiments

Meta-review: The paper studies the properties and advantages of temporal abstraction in hierarchical latent variable-based video prediction approaches, producing interesting results on various simulated environments and the KTH action dataset. 

The two key drawbacks of this paper are: limited visual complexity of datasets used for evaluation (a real video dataset like BAIR pushing would have really helped), and lack of comparison (conceptually or empirically) to relevant prior work including those raised by various reviewers, plus see more below.

Aside from this, does the paper claim to propose a new model, or perform an empirical study to evaluate an existing model class, or both? The answer to this question is not always clear from the paper and this confusion is reflected both in the reviews and reviewer discussions and also in the authors' own responses to these.

Some potentially relevant works that don't find mention in this paper (aside from those pointed out by reviewers):
- NAOMI: Non-Autoregressive Multiresolution Sequence Imputation
https://arxiv.org/pdf/1901.10946.pdf
- Long-Horizon Visual Planning with Goal-Conditioned Hierarchical Predictors
https://arxiv.org/pdf/2006.13205.pdf 

I do think these are all relatively easily fixed shortcomings, and I urge the authors to fix them in future versions.
Decision: Reject