Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper extended the flow-based generative model for stochastic video prediction. The proposed model takes an advantage of the flow-based models which provide exact latent-variable inference, exact log-likelihood evaluation, and efficiency. The paper used the autoregressive model and the multi-scale Glow architecture. The experiments on the stochastic movement dataset (synthetic) and the BAIR Robot push dataset show the performance improvement against other state-of-the-art stochastic video generation models (SV2P and SAVP-VAE). 

The main contribution in this paper is the use of flow-based models for video prediction, and it is the first work in this direction. The major idea sounds and the paper is clearly written. 

Below is my concerns and the feedback. 

It looks like the low-temperature sampling is important to achieve the better scores for prediction. Can the low-temperature sampling trick be applied for SV2P and SAVP-VAE as well? If then, how is the performance difference compare to the proposed model?

The authors reported the best possible values of PSNR, SSIM and VGG perceptual metrics by choosing the video closest to the ground-truth. However, I believe this evaluation does not present the benefit of the stochastic models. The better comparison I believe is to report the median/mean with the range between best and worst values. 

The BAIR robot push dataset is with a pretty limited setting: a small robot and/or object motion between frames and a small variation of the background between videos. It would be interesting to see more dynamic scenarios such as driving or human motion scenes. 
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: The paper "VideoFlow: A Conditional Flow-Based Model for Stochastic Video " proposes a new model for video prediction from a starting sequence of conditionning frames. It is based on a state-space model that encodes successive frames in a continuous hierarchical state, with contraints on trajectories of the codes in this state. 

I like the invertible NN framework the model relies on. It allows to avoid variational autoencoding of frames via invertible deterministic transforms. Learning the dynamics of the video is therefore easier, since there is no need of any stochastic inference process.    However, is there no risk of high latent vacancy in the representation space? Uncertainty of stochastic inference usually helps filling the space by considering larger areas of codes than deterministic process. Also, since at each step, the next code is conditionned by the whole past sequence of codes, besides the increasing complexity induced, I am wondering if such a model is able to efficiently encode the dynamics and the stochasticity of the video. In fact, a given z_t does not encode any dynamics nor uncertainty at that point, only the image (it cannot since it is fully determined via the invertible function from the image). Imagine that at a given point, two very different scenarios can follow, with very different following frames. In that case, how could the next state could encode these two different futures with a simple gaussian in the space ? Also,  it would be useful to compare the model with a version where the invertible frame encoder and the sequential model would be learned separately, to better understand what the model really does during training. A study of the impact of the hierarchy depth would also be useful.  

Also, an additional real-world dataset would be useful for really assessing the performance of the model, since BAIR is known to be fully random and the past does not highly impact the future. A possible dataset would be KTH. Other baselines could also be considered, notably the famous approach from  [Denton et al., 2017]. 

At last, the clarity of some parts could be improved. Notably the description of the sequential model in the space, whih is succintly given in the appendix.   


review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper presents a stochastic model based on Glow for conditional video generation. The major novelty of this work is to introduce the flow-based models to video modeling and learn the video dynamics via the dependencies of the latent variables. The general idea is reasonable and the proposed model is technically correct, but I have the following concerns mainly about the originality and the experiments.  

**Above all, most of the text in Section 3 is very similar (or exactly the same) to the text of the Glow paper (the background section).**

— Significance and originality —
a.1) In Section 1, the authors discussed some possible application scenarios of video prediction models, e.g. learning from unlabeled data and being used for downstream tasks. However, all models mentioned here, including [Mathieu et al. 2016] and [Finn et al. 2016], are deterministic models. Thus, in what way can the stochastic model proposed in this paper be used in real applications?

a.2) VideoFlow can be viewed as an extension of the Glow model. There are two problems. First, the originality is limited. I don’t think modeling the temporal dependencies of the latent variables with a convolutional network is a significant contribution to the conditional flow-based methods. Second, this paper is not self-contained. After reading Section 4.2, I have to check the previous literature to find the objective function, the network details, or the training procedure.

— Experiments —
b.1) Throughout the experiments, the VideoFlow model is mainly compared with two stochastic video prediction models that were probably proposed by the same research group. If it is possible, the authors might include other stochastic models such as the SVG-LP [Denton & Fergus 2018], and at least one deterministic model such as the E3D-LSTM [Wang et al. 2019] as well.
[Wang et al. 2019] E3D-LSTM: A Model for Video Prediction and Beyond.

b.2) The evaluation metric bits-per-pixel was not directly optimized by the previous video generation/prediction models. Thus, the comparisons in Table 2 might be unfair.

b.3) Since training the Glow model requires a huge computational cost, how is the training efficiency of the VideoFlow model compared with other stochastic video generation models?

— Other —
c.1) In Section 4, it is not clear what the temporal border effect means?

AFTER REBUTTAL:
Though the overall novelty is still not fully convincing, this paper may shed some insights into video generation by introducing flow-based models to this topic. I have increased my score from 3 to 6.
review_title: Official Blind Review #3

Meta-review: The authors explore the use of flow-based models for video prediction. The idea is interesting. The paper is well-written. It is a good paper worthwhile presenting in ICLR.

For final version, we suggest that the authors can significantly improve the experiments: (1) report results on human motion datasets; (2) include the results by the FVD metric. 

Decision: Accept (Poster)