Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposed a method for video prediction called Clockwork Variational Autoencoders. The method adapts Clockwork-RNNs to sequential VAEs and defines a hierarchy of latents operating at different temporal scales. This allows the model to generate long term videos.
main_review: **Originality**

The idea of having different temporal scales in sequential models has been proposed before in Chung et al. or Koutnik et al. However, the adaptation of this idea to sequential VAEs for video prediction is not straightforward and therefore novel.

**Clarity**

The paper is well written. The problem statement, the model description and the experimental section are clear.

**Quality**

The technical contribution of the paper is sound. The experimental section of the paper is extensive and adequately highlights the properties of the proposed model, not only comparing to different baselines but also investigating different aspects of the model such as the utilization of the different levels of latents. 

**Significance**

Overall the paper is clearly written, presents a novel approach for video prediction specifically designed to model long-term videos, and the experimental section properly supports the validity of the model. I believe the community will be interested in this work.

**Review Updates**

After reading the rest of the reviews and the authors' response, I stand by my original rating and argue for the acceptance of the paper.
limitations_and_societal_impact: The authors discuss some of the limitations of their model but do not acknowledge any potential negative societal impact of their work. Video prediction models in general might be used with malicious intents (for example "deepfakes"), and might reflect biases found in the training data. Please include a discussion on potential negative impacts.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: In this paper, the authors tackle the problem of long-term video generation. To this end, they propose a hierarchical latent variational autoencoder: The Clockwork Variation Autoencoder. This model incorporates a hierarchy of latents separated at different frame rates. This frameworks allows the VAE to model dependencies in the video at different timescales. The authors evaluate their approach on a variety of datasets: a new benchmark from Minecraft, KTH, Moving MNIST, and GQN Mazes. The paper presents promising qualitative and quantitative results with the proposed approach.
main_review: Strengths:

1. The paper is well written and easy to understand. It flows well and properly structured.

2. The problem is relevant and important to the community: There has been a cornucopia of video modeling papers, but relatively few have addressed the problem of long-term coherence in video generation. This is a fairly unexplored problem relevant to other fields such as Reinforcement Learning.

3. The quantitative and qualitative results on the chosen datasets is promising.

Weaknesses:

1. Choice of metrics. This is an issue common with video modeling papers in general, but metrics such as SSIM, PSNR, and
LPIPS assume a single ground truth future video. They assume the problem of future video generation is a deterministic problem
when more stochastic enviroments (i.e. real-world videos from UCF101, Kinetics, etc) clearly have multiple valid outcomes. Video
prediction should be treated as a generative model that estimates the probability distribution of future video. Metrics such as Log-likelihood (or approximations to this), and even crowdsourced human ratings would set a precedent for future work on more complex datasets.

2. The datasets are simplistic. From a computer vision standpoint, the datasets featured have very low visual complexity.
From an event standpoint, KTH, Moving MNIST, and GQN Mazes are domains with very low levels of uncertainty.


Overall:

The paper has a few flaws. The authors choose simple datasets for evaluation and use flawed metrics that may not be the best foundation for future work. However, they explore an important, underexplored problem in video generation. Modeling long-term dependencies in many domains-text, images, and video-is non-trivial. In spite of these issues, I believe this paper is a step in the right direction for research in video generation.

limitations_and_societal_impact: The authors have addressed any limitations or social impact.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes Clockwork-VAEs, a hierarchical latent dynamics video prediction model. CW-VAE uses different clock speeds at each level to model dependencies occurring at different frequencies. This allows it to perform well on long-term video prediction. The paper presents results on four diverse datasets, including using a new dataset (MineRL) for long term video prediction.

main_review: The proposed architecture is intuitive, simple, and achieve strong results. Experiments are also fairly comprehensive, and I appreciate the visualizations provided. I think this model and the proposed use of the MineRL dataset are valuable contributions for the community, and I lean towards acceptance.

## Strengths

* The proposed CW-VAE automatically adapts to the frame rate of the dataset, which is a neat feature to sidestep annoying hyperparameter tuning across different datasets.
* The quantitative results are strong, and the ablation experiments are comprehensive. I particularly appreciate the efforts in sections 3.6, 3.8 and 3.9 which examine different levels of CW-VAE, to show that the proposed hierarchy does indeed encode information of differing content and temporal speeds.
* The model itself is intuitive and simple, yet appears to achieve strong results. Training time and resources are also modest and comparable to prior work.


## Weaknesses

* The experiments seem to be conducted on fairly simple datasets, which have minimal moving objects. The only dataset with more than one moving object is Moving MNIST, which is somewhat of a toy dataset. Experiments on more complex datasets such as KITTI would be valuable. It would be interesting to see how the different hierarchies perform with faster / slower moving cars, in particular, or how it operates when there are multiple moving objects in the video.
* While the quantitative metrics seem strong, it is also valuable to conduct a human study over the generated results, especially for long term generation results on MineRL, which is a new dataset.
* There is missing comparison to prior work on stronger hierarchical models, such as [1].
* The results in the paper are presented on 64x64 images, which seem rather small. How does CW-VAE perform if scaled up to larger video resolutions, such as 128x128 or 256x256?

## Concerns / questions

* Are results presented in Figures F.1 and G.1 randomly sampled, or cherry picked? If they are cherry picked, please also share some randomly sampled results.
* How is the video diversity of CW-VAE generated videos? There is a claim in the paper (L198) that VTA lacks diversity on MineRL. Please substantiate CW-VAE’s improved diversity with qualitative results for the different models, or with some measure of video diversity (e.g. LPIPS of predicted video frames at long horizons) when sampling different values from the Gaussian prior.
* MineRL seems like a natural setting for action-conditioned video prediction. It would be interesting to hear the authors’ thoughts on whether they attempted to include the action condition in CW-VAE, or if there is a natural way to do so.

## Presentation

There are some minor typos in the paper:
* L169: “factors training faster” -> “factors *train* faster”


[1] Lee, W., Jung, W., Zhang, H., Chen, T., Koh, J. Y., Huang, T., ... & Hong, S. (2020, September). Revisiting Hierarchical Approach for Persistent Long-Term Video Prediction. In International Conference on Learning Representations.


*Post rebuttal*

* The authors have addressed most of my concerns. I think this is a meaningful paper and I would be happy to accept it. I have raised my score accordingly.
limitations_and_societal_impact: Limitations of CW-VAE and potential directions for future work are provided in Section 4 of the main paper. There does not seem to be any potential negative societal impact that would directly arise from this work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents a video prediction model named Clockwork VAE, which leverages hierarchical latent variables and temporal abstraction to learn long-term dependency. It also proposes the new Minecraft benchmark for long-term video prediction. The model is compared with existing video prediction methods on four datasets, and achieves remarkable long-term prediction results.

main_review: Pros: 
1. The overall idea of the proposed model is reasonable. It uses a hierarchy of latent variables with a temporally abstract structure to capture long-term video dynamics. It also uses the clockwork recurrent transitions to accommodate different sequence speeds. All the above network designs are integrated in a VAE framework and can be trained in an end-to-end manner. 
2. Impressive qualitative results for long-term prediction.

Cons: 
1. The idea of combining temporal abstraction, clockwork transition (state copy/update), and variational models has been introduced by VTA. Though a clear difference is that CW-VAE can be equipped with more than two levels of temporal abstraction (that can better capture hierarchical dynamics), it would be good if the authors could further clarify other differences between these two approaches.
2. Another concern of mine is about the completeness of the experiments. It seems that VTA is the most important baseline model, so it would be a good choice if the datasets used by VTA can also be included in this paper. Besides, RSSM is mostly used for model-based RL, where long-term prediction is less important due to the use of re-planning and model predictive control. Since none of the compared approaches were originally designed for long-term prediction, it would be good to include other existing models in the experiments, such as the work from Wichers et al. (2018), which also uses hierarchical latent variables to facilitate long-term prediction. 

A minor concern: The authors are encouraged to present the quantitative results of the four metrics and to give more description and analysis in the main text, rather than in the supplementary material. Also, from TableE.1, the performance of the proposed model is not much improved compared with RSSM, especially on the MineRL Navigate dataset.
limitations_and_societal_impact: Yes.
review_title: null

Meta-review: This paper proposes a novel architecture for long term video generation. Reviewers agree that the approach is novel, it focuses on an important and under-explore problem of video generation and that the experimental section supports the value of the contribution.
Decision: Accept (Poster)