Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary: This paper proposes a hierarchical framework for long-term video prediction. The structure is firstly predicted in the form of semantic map. It lies in a categorical structure space which is easier to predict. Then the authors translate the predicted semantic map to a real video sequence in a frame-by-frame manner. The proposed model is "surprisingly successful" for long-term video prediction, as claimed by the authors (thousands frames). 

Pros:

+ clarity: This paper is overall easy to understand. The proposed method, which firstly predicts the categorical map and then translate  the map to video frame, is very straightforward and clear. The motivation behind such kind of design is convincing, i.e., to explicitly reduce the modelling difficulty of the prediction module.

+ significance: I have carefully checked the prediction results provided by the authors. I think the predicted video sequences are convincing and appealing, which are of >1000 frames. The video content does not freeze even after thousands frames. The visual quality on the cityscape dataset looks very promising.

Cons:

- originality: The proposed framework (first predicting semantic label then translating) is a simple extension compared to the previous work[1,2]. The model used in this work is a typical combination of LSTM and VAE, where the uncertainty is modeled with Gaussian Distribution and KL-divergence loss. For the second part, the auto-encoder based translation model, along with adversarial training scheme, is also commonly used in the image transfer task. Overall, the novelty of the proposed method is limited in my point of view. 

- quality: Regarding this aspect, my major concern lies in the lack of the ablation study. In the experiment, the authors present sufficient and extensive results compared with other methods. However, the contribution of each module in the proposed method is completely unknown. Which part is the most important? Which part is the key to long-term video prediction? A comprehensive analysis about this part is highly recommended.

[`1] Predicting deeper into the future of semantic segmentation.
[2] Predicting future instance segmentation by forecasting convolutional features.
review_title: Appealing visual results but with limited novelty and unspecific contribution.

final_score: rating: 6: Marginally above acceptance threshold
review_content: ###Summary###


The paper proposes the hierarchical video prediction model.
Specifically, the model first generates a sequence of semantic segmentation maps.
And then it generate a sequence of future frames corresponding to the semantic segmentation maps.
Since the sequence is generated stochastically, it can generate various futures given same condition.
With this hierarchical method, they could successfully generated thousands of future frames.


###Pros###


-
By using low-level information for modeling of dynamics, the proposed method generates realistic images not suffering from severe error amplification issue.



###Questions###

-
It is not clearly pointed why this method successfully generates long-term sequences.
Is it the result of learning the approximate posterior distribution of the following semantic segmentation map, or the result of the hierarchical generation mechanism?

-
Is the modeling motion only using the LSTM or conditional CNN inferior to this method? (deterministic one, no modeling for the approximate posterior distribution)

-
Regarding the equation 4, have you tried training the model without applying the teacher-forcing?
Which means, training with the generated semantic segmentation maps as the condition.
review_title:   

final_score: rating: 7: Good paper, accept
review_content: This paper proposes a VAE based hierarchical model for video prediction. The model employs recurrent model to predict intermediate representations (in the form of label maps) and these representations are mapped to pixel level information, i.e., videos. The paper presents an interesting idea of using representations that do not use any domain knowledge. The authors demonstrate the value of modeling temporal evolution of these representations which enables long term video prediction.

**Strengths**
+ The paper is clearly written and the implementation details are clearly described 
+ The model outperforms relevant baselines quantitatively
+ The model is able to generate realistic and temporally consistent samples
+ The evaluation of the model is thorough and performs better consistently across challenging datasets.


**Weaknesses**
- How important is the warped optical flow in the video discriminator?
- How is $\tau'$ decided? Does it have any impact on the performance or computation requirements of the model? 
- The role of $G_{edge}$ is unclear. An ablation study would be useful. 
- Given the complexity of the task, it would be good to report the time required to train the models.

Overall, the paper presents impressive results on long-term video prediction and the evaluation of the paper is thorough. However, some ablations studies are required to convince the importance of each component used in the overall model. Therefore, my initial rating for this paper is 6.


=======================================**Post-rebuttal Comments**===============================


I appreciate the revisions and additional results reported by the authors. The authors have addressed the concerns raised by me in the revision. While I agree with R1 that novelty of the method is limited, after considering the reviews collectively, I believe this paper presents very impressive results given the fact that the problem is challenging. Therefore, I would like to improve my final rating to 7.

review_title: Some ablations missing

final_score: rating: 6: Marginally above acceptance threshold
review_content: ---- Summary ----

The paper extends video-to-video translation model of (Wang’18) to video prediction by first generating a sequence of segmentation masks and then translating them into videos. Variational video prediction is used to generate a sequence of segmentation masks. The model produces impressive high-resolution and long-horizon results, and is extensively evaluated on Kitti, Cityscapes, and dancing data, outperforming some previously proposed methods.

---- Decision ----

The paper proposes a relevant method for hierarchical video prediction with impressive high-resolution and long-horizon results. As the paper notes, this sets a new standard for video prediction methods, and will likely spur more research into achieving similar results without the labeled data requirement. I am willing to accept the paper provided the author’s response clarifies my questions.

---- Strengths ----

The paper presents a modern version of Villegas’17a, powered by the advances in probabilistic video prediction and generative adversarial networks. The proposed model significantly outperforms prior work, scaling to high-resolution images (256x256) and long horizons (2500 frames). 

---- Weaknesses ----

A weakness in the experimental setup is that the compared baselines are not controlled for the number of parameters. In particular, SVG-extend, which also works well on Kitti data, seems to contain the same number of parameters as the segmentation prediction network of the proposed model. It is also unclear how the baselines were tuned and what was the range of considered hypeparameters for all methods. More generally, it would be good for the paper to present some simple setting in which SVG-extend works well to analyze where hierarchical prediction is most important. 

---- Additional comments ----

There is a number of minor inaccuracies in the paper: 

The authors of Wichers’18 are cited as “Ruben Wichers, Nevan Villegas”. The real authors of this paper are Nevan Wichers and Ruben Villegas.

Page 2, “We employ the sequence model based on VAE (Denton & Fergus, 2018)”. Denton and Fergus did not invent VAEs, neither they invented sequential VAEs. You likely want to cite the original VAE papers (Kingma’14, Rezende’14), or the sequential VAE papers (Chung’15, Fraccaro’15). Somewhat strangely, none of these papers are cited elsewhere in the paper either. If you do want to cite Denton & Fergus, the sentence would need to be “We employ the sequence model based on VAE proposed by Denton & Fergus (2018)””.

Page 3, “we skip hidden representations of the encoder to the decoder at every time step during testing to handle longterm dynamics in structure”. Would be great to cite some papers that do also that, e.g. Villegas’17a or Finn’16.

“Extension to object boundary prediction”. This paragraph seems to be hastily written and full of incorrect statements. The G is not an autoencoder (and not a denoising autoencoder either). It predicts e from s, maybe just call it a boundary prediction network? The conditional GAN objective does not maximize p(e,s). In fact, p(s) is a constant. Instead, it tries to match p(e|s), but not by maximizing likelihood, but by minimizing an approximation to Jensen-Shannon divergence (see Goodfellow’14).

Eq (1) is not a variational lower bound for beta <> 1.
review_title: Review 1: reasonable paper with good results

Meta-review: This paper proposes a new implementation of a previously proposed two-stage process for video prediction: first predict future segmentation maps, then map them to video frames. Combined with other advances in video prediction and image generation, this simple idea is shown empirically to work very well, producing video predictions up to many hundreds of frames into the future in real stochastic settings with unprecedented quality. Strong ablation studies over the course of the review process further serve to confirm the value of various design choices involved in the implementation. 
Decision: Accept (Poster)