Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents a method for hierarchical representation learning of spatiotemporal features in long-term video prediction. The proposed method is called: Dynamic Latent Hierarchy (DLH). The method distinguishes between features that are changing and those that are not changing in the video sequence. DLH is able to handle multiple objects moving at different speeds and differentiate moving objects from a static environment. The advantages of DLH include: long-term video prediction, improved modeling of stochasticity, and dynamic, efficient latent structure. DLH outperforms baseline approaches on Moving MNIST, KTH Action, and DML Mazes datasets. 
strength_and_weaknesses: Strengths:
* I really enjoyed reading this paper. It is well written and the ideas are easy to follow. 
* The problem is well-motivated and the literature review does a good job at contextualizing the paper in prior work.
* The methods section was presented with intuition for the design choices and notation. This intuition was then supported empirically.
* An attempt was made to interpret the hierarchical levels semantically in the experiments.
* Given the considered datasets, the experiments section is constructed thoughtfully and the results are convincing.
* The figures are informative and effectively illustrate the benefits of the proposed approach. Particularly, the intuition provided by the results in Fig. 5 was helpful in understanding how the method works.

Weaknesses:
* Overall, the datasets considered are fairly uncluttered and simplistic. The video examples do not highlight the capabilities of DLH to handle many objects at many different speeds (e.g., in crowded urban scenes). The data also does not showcase what happens when the background is not stationary and there are moving objects. I would recommend considering a more complicated, dynamic dataset, for example, from the autonomous driving setting (i.e., Waymo Open Dataset [1], NuScenes [2], or KITTI [3]). The simplicity of the toy Moving Ball dataset is also underscored by the results of Table 2, where the full capacity of the hierarchical model is not necessary to model the data. Although it is great to see that the model can dynamically adapt to use less of the latent space when the underlying data distribution is simpler, it would be compelling to see how the full latent space would be used in a more complex setting.
* Although there is a discussion on the importance of stochasticity, this capability is only explored in a toy-dataset setting with random color changes. No discussion of multimodality in the distribution is included. Could DHL handle multimodal outputs (e.g., multiple equally valid possibilities for the future)? This would again be relevant to more complex video datasets (e.g., in the urban setting), where given observations of a person walking straight, they could choose to continue walking straight or turn in the future.
* It would be helpful in Fig. 4 or in an appendix to show the outputs of the baseline approaches for comparison.
* In Fig. 5, it is not entirely clear that levels 1 and 2 for the KTH Action dataset are disentangled.
* It would be interesting to benchmark against a deterministic video prediction method in Table 1 to see if the considered datasets are sufficiently stochastic to warrant modeling of stochasticity.
* Is there a way to report a measure of statistic significance of the proposed method's metric performance over the baselines in Table 1?
* In Fig. 6, it is not very clear to me what is wrong with some of the highlighted frames output by the CW-VAE. Is the issue that the ball reduces in size for those frames? Why were the other baseline results not shown?
* Further explanation for the values in Table 3 would be helpful.

[1] Sun, Pei, et al. "Scalability in perception for autonomous driving: Waymo open dataset." CVPR, 2020.

[2] Caesar, Holger, et al. "nuScenes: A multimodal dataset for autonomous driving." CVPR, 2020.

[3] Geiger, Andreas, et al. "Vision meets robotics: The KITTI dataset." IJRR, 2013.

Some typos and minor points of confusion are listed below:

1. I am not sure I fully followed the diagrams in Fig. 2. Are there temporal indices missing from the states?
2. Are there hierarchical levels and $\psi$ parameters missing in the 'Estimating' paragraphs in Sec. 2.2 and similarly dropped indexes in $p(e \mid s)$ in the paragraph before Eq. 3?
3. In Eq. 3, $\Lambda$ is not defined.
4. Missing period at the end of Eq. 3.
5. In Sec. 2.3, I am having some trouble understanding the notation. Should $q(\bf{e}^{n+1} \mid \bf{e}^n = 0) = 0$ be $q(\bf{e}^{n+1} = 1 \mid \bf{e}^n = 0) = 0$?
6. It would be helpful to derive Eq. 8 from Eq. 7 in the appendix for completeness.
7. Fig. 3 was referenced much earlier than it appears.
8. Am I correct in understanding that at the first hierarchical level $e^1 = 1$ always?
9. In Fig. 4, it should be made clear whether the 30 past context frames are included in the visualization or only the 100 predicted frames are shown.
10. In Sec. 3, temporal abstraction paragraph, the sentence "Temporal abstraction models ..." has a grammatical typo, is a bit long, and is missing a period at the end.
11. Missing periods at the end of table captions. Unclear what the * symbol signifies.
12. In Sec. 4.3, I did not fully understand what it means that "DLH learns transition between progressively slower features in the higher levels of its hierarchy". Does this mean that minor variations in the scene are faster features than location changes of the view?
13. The references should be proofread (e.g., to ensure the year is not entered twice in a citation, the conference venue is listed instead of ArXiv when available, the confererence name formatting is consistent, etc.).
clarity,_quality,_novelty_and_reproducibility: The overall presentation and quality of the paper was quite high. Although the method builds on components from existing approaches, the dynamic hierarchical architecture for video prediction appears to be novel. Aside from the points of clarification listed above, I found the paper to be clear. There does not appear to be code provided in the supplemental material, which hurts reproducibility. 
summary_of_the_review: Overall, this is a good, clearly written paper that proposes a reasonable approach for video prediction that outperforms the considered baselines across several datasets. I am currently inclined to accept it. I encourage the authors address the first two weaknesses listed above, in particular, regarding the complexity of the datasets used for evaluation.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a hierarchical VAE for video prediction. The latent structure of the model is formulated as a mixture of gaussians. The authors show that the proposed model is competitive to some previous hierarchical video VAE models on relatively simple datasets.
strength_and_weaknesses: **Strengths:**

[+] Clear method formulation and good empirical results on relatively simple datasets.


**Weaknesses:**

[-] The model is fairly similar to previous approaches that use hierarchical latent variables in time and/or space.
[-] Results are on relatively simple datasets, with newer video prediction methods being capable of generating sequences for long horizons on more complex datasets (e.g. Kinetics)

clarity,_quality,_novelty_and_reproducibility: * Clarity: The model is presented clearly.

* Quality: The experiments show that the model can improve upon previous hierarchical VAEs, but more work is needed to compare it to newer more performant methods on more complex datasets.

* Novelty: While the formulation is novel, the idea of hierarchical video VAEs has been extensively studied (see citations in the paper) and overall it is not clear that the more involved formulation presented in this paper will have practical impact.

* Reproducibility: The paper contains enough details to be reproduced.
summary_of_the_review: Overall the method presented in this paper is not that different from other previous hierarchical VAE models, and while it shows some improvements over these methods, in general it is unclear if this model will be of interest to the community compared to newer methods (FitVid, diffusion video models) that can generate long video sequences on complex datasets and have simpler latent spaces.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents a new video prediction model in the form of neural networks, which combines the advantages of hierarchical clockwork models (from CW-VAE) and jumpy recurrent modeling controlled by binary indicators (from VTA). 
- Major: It provides a novel graphical model for spatiotemporal prediction. To avoid the unnecessary accumulation of prediction errors and improve computational efficiency, the authors propose to disentangle the temporal dynamics into hierarchical latent states and predict whether the states would change or remain static at each future timestep. 
- Minor: The proposed model is shown effective on four datasets, improving previous benchmarks including CW-VAE and VTA.
strength_and_weaknesses: Strength: 
1. This paper is clearly written and easy to follow. 
2. By considering modular representations, this paper has proposed a simple yet effective hierarchical framework to model the mixed spatiotemporal dynamics in video sequence at different timescales, and also achieves great quantitative results.
3. Detailed validations are performed on multiple datasets. In addition to the significant improvement in SSIM/PSNR, in Fig 5, the authors also tried to demonstrate the advantages of the proposed method in decoupling different spatiotemporal dynamics from the qualitative visualization.

Weaknesses:
1. My main concern lies in the novelty of the proposed approach, which seems to be a simple extension of the hierarchical model from Zakharov et al. (2022). If I understand correctly, the biggest difference between them is the binary indicator e_t^n, which is used to detect event (change or static). However, similar techniques has been well explored by Kim et al. (2019). The technical contributions of the model are not convincing to me and require further clarifications. 
2. I also have some doubts about the connections between the example explained in Fig 1 and the method of "nested timescales" proposed in Sec 2.3. I understand that the authors attempted to describe the hierarchical state organization, but in Fig 1, it seems that the dynamics of the panda has nothing to do with that of the airplane. Therefore, if Fig 1 correctly states the motivation of the method, it should learn the dynamics at different timescales in parallel state transition branches, instead of the hierarchical one. In other words, in Eq (5), the indicator of state changes at layer n (e_t^n=0) should not be completely determined by e_t^{n-1}=0. 
3. Although in Sec 2.3, the method is claimed to reduce the computational complexity, no corresponding empirical evidence was given in Sec 4 that could support the efficiency of the proposed model. 
4. In my view, the numbers of state levels that are used to model complex scenes (such as KTH or even datasets with more complex visual dynamics) and simple scenes (such as Moving MNIST) may be significantly different. How should the hierarchies be determined for different scenarios? In other words, how many levels of latent states should be selected for different datasets? As shown in Table 1 and Fig 5 for selecting 3 levels throughout the first three datasets, can the conclusions be extended to general scenarios? If not, can the authors provide some reference schemes for selecting the number of levels. Besides, it would improve the fairness of the comparison to implement a VTA model that also employs a 3-level architecture with similar number of parameters. 
5. What does * indicate in Table 1?
6. In Figs 4-5, I strongly encourage the authors to include more qualitative comparisons for long-term video prediction with existing benchmarks. The proposed method is expected to generate future frames with higher visualization quality. In addition, it would be nice if the authors could give clearer textual explanations in Figs 4-5. I am confused on what each image sequence represents...Are those in the first row the ground-truth images, or does each row represent a different prediction sequence by the model based on the same input sequence? 

Other suggestions (NOT weaknesses):
1. If the experimental conditions allow, it is essential to show the model performance on more real-world datasets with large spatiotemporal uncertainty, such as RoboNet or BAIR. The authors may consider the action-conditioned video prediction setup because it would be interesting to analyze the correspondence between the learned indicators and the given action signals. 
2. In addition to using the best prediction among 100 prediction samples, the authors may evaluate the models using the average performance of the 100 samples as well as the worst cases.

clarity,_quality,_novelty_and_reproducibility: This paper is clearly written but has insufficient novelty for the proposed method. Please see the comments above for more details.
summary_of_the_review: Overall, it is a good paper that improves the existing benchmarks on multiple video prediction datasets. However, it needs significant improvement, specifically in the following aspects: 
1. The technical contribution of the paper. Although the proposed approach is new in terms of the graphical model shown in Fig 1, the key insight of combining hierarchical latent representations and the jumpy state transitions for modeling video dynamics adaptively at different timescales has been explored by existing work with similar techniques (using binary indicators for the state changes). 
2. More explanation of the connection between the motivations shown in Fig 1 and the proposed approach.
3. Additional experiments on model efficiency and more qualitative comparisons for long-term prediction should be included to understand the effectiveness of the approach.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes Dynamic Latent Hierarchy, a latent-variable video prediction model. This model can learn a hierarchy of latent variables where the latent variable at each level of the hierarchy, operates at a different timescale. Unlike related work, ClockWork VAE’s, (Saxena et al. 2021), whether or not a latent variable at a certain level is activated is not hard coded. Instead, the paper proposes to cast this as a Bernoulli latent variable and learn it using variational inference.

Improvements are shown on KTH, DML Mazes and Moving Mnist on SSIM and PSNR over 3 competing methods. Qualitiative experiments on KTH show that the higher levels get activated lesser as compared to the lower levels. Similarly, on a synthetic moving ball dataset, when the color is changed stochastically, on higher levels of stochasticity, the emission probability is higher.

strength_and_weaknesses: **Strengths**

The problem is well motivated as stated in the introduction. In a video, it is expected that latent variables operate at different timescales. Providing this as an inductive bias in video modeling, could help in video prediction and could lead to interesting insights.

**Weaknesses**

I have some concerns with the novelty, clarity and experiments in the paper. See below for detailed feedback and for actionable suggestions.

clarity,_quality,_novelty_and_reproducibility: **Novelty**

[Extremely Major:] The paper heavily relies on Variational Predictive Routing [Zakharov et al, 2021] in several places, but the paper does not describe what exactly is novel over the VPR setting. More specifically, what are the changes or adaptations made to the VPR model in this paper? In my opinion, the paper should relegate the descriptions of the VPR model and focus only on their contributions in the Dynamic Latent Hierarchy section. The paper should also offer comparisons to VPR as a baseline, showing that their proposed modifications lead to superior performance.

**Clarity and Quality**

The algorithm is somewhat confusing partially because in some locations, the dependencies are omitted, and in others, they are not. I have posted my understanding of the algorithm from reading it a few times, with some questions. Updating the draft, in response to these questions, can improve the clarity of the paper.

* First, the posterior over the latent variables $q(s_t^n)$ is inferred in Eq 8b). From Section 2.2, apparently there is a dependency across levels in a top-down fashion, from higher to lower levels. I assume there is also a dependency across timesteps? Please update Eq 8b) to reflect both these dependencies.
* At every level and timestep, the algorithm compares the KL between the posterior $q(s_t^n | s_t^{>n}, x_t)$ and $P(s_{t-1}^n)$ with the KL between KL between the posterior $q(s_t^n | s_t^{>n}, x_t)$ and $P_{\theta}(s_t^n | s_{<t}^n)$. If the first term is higher, then the inferred latent variable $q(e_t^n = 1)$ or 0 otherwise. Is this the same prior as in Eq 8b) or a different prior? Are these also the dependencies across levels in this prior?
* Is the prior over the latent emission variables conditioned on the previous emission variables? Equation 1 describes that $e_t^n$ is dependent on $e_t^{<n}$ while eq 8c) omits these dependencies. If it is conditioned on the sampled emission variables, then how is backpropagation done through this discrete sampling operation, while minimizing the ELBO?
* In Section 2.3, if $e_t^{n-1}=0$, then $e_t^{n}$ is also set to 0. Also, latent discrete variable, $q(s_t^n)$ is set to the $q(s_{t-1}^n)$. However, since $q(s^{n+1})$ was conditioned on the previous value of $q(s_t^n)$? Is this unchanged or kept the same? What is the rationale behind this?
* Figure 2a is confusing because there are no superscripts. Superscripts are necessary to show the evolution of the states over time.
* The dependencies across $x$, $c$, $d$ and $s$ should be made very clear in Section 2.5. For example. how does $s$ depend on the temporal state?
* Is $x_t^0$ the image and $c_t^0$ correspond to the output image?
* Are the parameters shared across all levels? Since each level $l$ has only dependencies to the levels above them, how is this "variable length" in terms of levels handled?

**Reproducibility**
It would be nice if some pesudocode of the training loop is provided. It would make it easier to digest and understand.

**Experiments**

* How were the hyperparameters of DHL tuned, for example, the number of levels.? On which validation set and on which metric?
* How does the computational efficiency and number of parameters compare to the baselines?
* Could the authors also plot the per-frame LPIPS, SSIM? It would also be nice to report the FVD Scores? What are the star superscripts in Table 1?
* Figure 5 demonstrates reconstructed frames retrieved by sampling the different hierarchical levels of the model. What does this exactly mean? Given a video, first the posterior latent distribution is inferred and then this is replaced with sampling from the prior. Is that correct?
* In Figure 5, on the bottom most subplot, it makes sense that L1 is set to 1, since the authors explicitly set it to be 1 in Section 2.3. So its not super meaningful.
* What is average hierarchical depth and how is it computed?
* Table 3 specifies different levels but not which specific level.

summary_of_the_review: I've provided detailed feedback above with respect to my concerns regarding the paper. I'm open to adjusting my rating if authors can clarify my concerns.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents deep hierarchical latent model of video, where different layers update on different time scales. 
- The paper is not very novel compared to other hierarchical VAE and jumpy state transition approaches and its contribution is marginal, especially given recent development of video models and that the results are not that strong.
- Some of the claims of the paper are not well supported.
- Writing can be improved both of the model and the experiments.
justification_for_why_not_higher_score: - The paper is not very novel compared to other hierarchical VAE and jumpy state transition approaches and its contribution is marginal, especially given recent development of video models and that the results are not that strong.
- Some of the claims of the paper are not well supported.
- Writing can be improved both of the model and the experiments.
Decision: Reject