Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: Summary:
This paper proposes a new LSTM architecture called LH-STM (and Double LH-STM). The main idea deals with having a history selection mechanism to directly extract what information from the past. The authors also propose to decompose the history and update in LH-STM into two networks called Double LH-STM. In experiments, the authors evaluate and compare their two architectures with previously proposed models. They show that their architecture outperforms previous in the PSNR, SSIM and VIF metrics.


Pros:
+ New architecture that can use all computed state history in a sequence
+ Outperforms previous methods in the used metrics

Weaknesses / comments:
- Larger number of parameters in comparison to ContextVP-4
In Table 1, the authors present a comparison to previous works and the respective number of parameters used for most of the methods. It is mentioned in the paper that Single LH-STM uses an architecture similar to ContextVP-4. Since the architectures are similar and the proposed method is added to this architecture, can the authors make sure that Context VP-4 and Single LH-STM have around the same number of parameters? This can give a more direct comparison in performance to make sure that the parameter boost is not the reason for performance boost.


- Why retrain for longer sequences?
The authors have experiments where they attempt to predict past 40 frames into the future. However, they mention that for this experiment, they train another network that takes in 64x64 pixels. Optimally, they should just let the 128x128 network predict past 40 frames. Can the authors comment on why they don’t just do this? Is long-term prediction limited to how many previous states you can store in the GPU?


- PSNR, SSIM, and VIF could be biased to blurriness and perfect background reconstruction
It has been shown before that PSNR and SSIM can be biased to blurriness and perfectly copying the background (Villegas et al., 2017b). Therefore, these metrics should be complemented with other metrics such as actual humans look at the videos and evaluated them for realism. The fact that these metrics look good in this paper can be due to blurriness. There is clear evidence of blurry predictions in the comparison Figures in the main paper and supplementary material. In addition these metrics, and VIF can also be biased to perfectly copying the background. I suggest the authors separate the data into videos with large motion and little motion similar to Villegas et al., 2017a. This way we can better evaluate this method on how well it predicts videos with large motion in comparison with videos where copying the background is enough.


- Testing for 80 frame sequences is not very meaningful in this dataset.
The KTH dataset contains the action categories of running, walking, and jogging which most of these videos do not go up up 40 frames while the person still being in the frame. Therefore, after frame 40, the method is just required to copy the background and it will look like the future is perfectly predicted.. Can the authors clarify if they only tested on handwave, handclap and boxing? Handwave, handclap and boxing are the only categories that will still have a human present in the video at frame 80.


- No video files are provided.
Finally, this method does not provide any videos to better evaluate the proposed network. Looking at videos is necessary to observe temporal consistency and blurriness happening in the video. Or humans appearing and disappearing randomly.


Conclusion:
The proposed method is novel and interesting, but the experimental section has many issues as discussed above. If the authors successfully address these issues, I am willing to increase my score.

review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: The paper proposes a type of recurrent neural network module called Long History Short-Term Memory (LH-STM) for longer-term video generation. This module can be used to replace ConvLSTMs in previously published video prediction models. It expands ConvLSTMs by adding a "previous history" term to the ConvLSTM equations that compute the IFO gates and the candidate new state. This history term corresponds to a linear combination of previous hidden states selected through a soft-attention mechanism. As such, it is not clear if there are significant differences between LH-STMs and previously proposed LSTMs with attention on previous hidden states. The authors propose recurrent units that include one or two History Selection (soft-attention) steps, called single LH-STM and double LH-STM respectively. The exact formulation of the double LH-STM is not clear from the paper.  The authors then propose to use models with LH-STM units for longer term video generation. They claim that LH-STM can better reduce error propagation and better model the complex dynamics of videos. To support the claims, they conduct empirical experiments where they show that the proposed model outperforms previous video prediction models on KTH (up to 80 frames) and the BAIR Push dataset (up to 25 frames).

Overall I believe there are serious flaws with the paper that prevent acceptance in its current form.

First, I believe the paper starts from the wrong assumption, namely that current video prediction models are limited by their capacity to limit the propagation of errors and to capture complex dynamics. Instead, it is well known that the main difficulty for longer term video prediction is to manage the increasing uncertainty in future outcomes. Stochastic models such as SVG-LP or SAVP are currently the state-of-the-art in video generation, with deterministic models not being able to generate more than a few non-blurry frames of video. While the authors mention that they do not focus on future uncertainty here, it is not clear how the proposed model helps to generate better longer-term videos when it does not deal with what actually makes long-term video generation difficult. In addition, it's misleading to claim that current models produce high quality generations for "only one or less than ten frames", especially without defining high quality. Models such as SVG [1] or SAVP[2] can produce non-blurry videos for 30-100 frames for the BAIR dataset, for example. 

The experiments are missing 1) SVG as a baseline, 2) metrics that correlate with human perception such as LPIPS or FVD [3] and 3) qualitative samples that compare to stochastic models. Deterministic models can achieve very high PSNR/MSE/SSIM scores but produce very bad samples, as these scores are maximized by blurry predictions that conflate all possible future outcomes. This is highly apparent when looking at samples, and metrics that correlate better with human perception are usually better to compare video prediction methods. Comparisons to SAVP are found in Table 1 and 3 but there are no figures comparing samples from this model to the proposed model. The samples from the proposed model on the BAIR Push dataset for example (found in the appendix) are of significant lower quality than those reported from SAVP or SVG, and at the same time they are not longer-term than the predictions from these models. Consequently, the experimental section does not correctly assess how this model can generate better longer-term prediction than current models and it also does not give an accurate assessment of the model with respect to the current state-of-the-art.

To sum up, the paper does not adequately address how the proposed model allows for longer-term video generation. It is missing critical qualitative comparisons to state-of-the-art models such as SVG and it is unclear how the proposed model is different from a ConvLSTM with attention on previous hidden states.

[1] Stochastic Video Generation with a Learned Prior. E.Denton and R. Fergus. ICML 2018
[2] Stochastic Adversarial Video Prediction. Lee et al. Arxiv 2018
[3] Towards Accurate Generative Models of Video: A New Metric & Challenges. Unterthiner et al. Arxiv 2018


--- Post-discussion update ---
The authors have addressed a number of points raised by the reviewers and I'm raising my score to a weak reject from a reject. There are important remaining issues with the experimental section and the conclusions reached from their results, and therefore I still think the paper is below the acceptance bar.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper presents a new RNN unit based on ConvLSTM for long-term video prediction. The proposed method is technically correct but lacks enough originality. I tend to reject this paper due to the following three reasons:

1. The major novelty of this paper is the LH-STM unit, which applies a temporal attention approach to historical hidden states. This module is very similar to the Recall gate of the E3D-LSTM [Wang et al. 2018b]. Besides, the Double LH-STM looks like an incremental extension of the Single LH-STM. As mentioned, it is technically correct, and yet has limited novelty for an ICLR paper.

2. The authors mainly compared the proposed network with the Context-VP model in the experiments (Fig 5, Fig 8, and Table 3), which is not enough. As far as I know, there are other existing methods for long-term video prediction, e.g. [Denton et al. 2017]. 

3. Another problem of the experiments is that Lee et al. [2018] proposed a stochastic model for video prediction, but the authors only compared the LH-STM with its deterministic version.

4. In Figure 11, there is no significant improvement by using LH-STM. Also, the authors might include more compared models on the pushing dataset.
review_title: Official Blind Review #2

Meta-review: The paper proposes a new recurrent unit which incorporates long history states to learn longer range dependencies for improved video prediction. This history term corresponds to a linear combination of previous hidden states selected through a soft-attention mechanism and can be directly added to ConvLSTM equations that compute the IFO gates and the new state. The authors perform empirical validation on the challenging KTH and BAIR Push datasets and show that their architecture outperforms existing work in terms of SSIM, PSNR, and VIF.
The main issue raised by the reviewers is the incremental nature of the work and issues in the empirical evaluation which do not support the main claims in the paper. After the rebuttal and discussion phase the reviewers agree that these issues were not adequately resolved and the work doesn’t meet the acceptance bar. I will hence recommend the rejection of this paper. Nevertheless, we encourage the authors improve the manuscript by addressing the remaining issues in the empirical evaluation.
Decision: Reject