Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper discusses a new framework named FITVID to handle the problem of video prediction. FitVid is built on existing modules like Sikp connection, Sequeeze and Excite, and LSTMs. FitVid only needs a simple training strategy and begins overfitting the datasets. Data augmentation is also adopted to handle the problem of overfitting. 
main_review: (1). The paper tries to argue that previous papers suffer from underfitting are mainly a result of inefficient usage of model parameters. It may provide new insight for researchers to design models for video prediction.

(2). The writing of the proposed model is clear and easy to follow.
summary_of_the_review: However, the reviewer does have some concerns on the paper.
(1). A major concern of the paper is about the model's novelty. The reviewer has doubts on the argument that a new combination of existing techniques (BN, LSTM, S&E, skip connection) for the task of video prediction is significant enough to publish in ICLR. It will be important to show what is the design motivation of the combination and what it can outperform existing models on the task of video prediction.

(2). A comparison of components of different models is needed to help clarify the novelty of the proposed method. It will be also easier for readers to understand why previous models suffer from underfitting and how the model is more data efficient.

(3). Although the authors provide experimental results on 4 datasets. However, most of them contain only two simple baselines and the results on these datasets are not consistent. It will be more convincing if the authors can provide results on future and counterfactual predictions of the video dynamics on the CLEVRER[A] dataset and compare the proposed model' performance with recent stronger backbones like transformer [B], GNN[C] and Differentiable Physics Engine [D].

[A]. Yi K, Gan C, Li Y, et al. Clevrer: Collision events for video representation and reasoning[J]. arXiv preprint arXiv:1910.01442, 2019. 
[B]. D. Ding, F. Hill, A. Santoro, and M. Botvinick. Object-based attention for spatio-temporal reasoning: Outperforming neuro-symbolic models with flexible distributed architectures. arXiv 2020.
[C]. Chen Z, Mao J, Wu J, et al. Grounding physical concepts of objects and events through dynamic visual reasoning[J]. arXiv 2021.
[D]. Ding M, Chen Z, Du T, et al. Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Language[J]. arXiv  2021.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper studies conditional video prediction. In particular, it focuses on the problem of current models underfitting and not scaling to datasets. The paper proposes an architecture that is claimed capable of using parameters more efficiently in order to overfit. Then, data augmentation is introduced to improve performance for generalization. They evaluated their method against baselines on 4 datasets (Human 3.6M, KITTI, Robonet, BAIR pushing dataset). They also showed experiments to support overfitting/underfitting claims. 

main_review: Strength:
+ Scale is an important aspect of the system. The paper did a good job on investigating the overfit / underfit effect in their model.  Fig 6, analysis on Human 3.6M, Fig7, etc.
+ The experiments are conducted across 4 datasets under different domain which indicates the proposed method generalizes to some extent. 
+ Though the generalization ability remains further exploration, a robot experiment based on MPC shows that their prediction model can be deployed in downstreaming tasks. 

Weakness:
- Lack of Intuition. The author proposes an Unet like architecture that is claimed to use its parameters more efficiently. However, The intuition behind it is very limited to me. Why is theirs better? Where does the improvement come from? (they also mention they share similar architecture with baselines on Page5 Sec4.2 Para1)  Without intuition, the architecture design looks arbitrarily stacking blocks with some skip connections. I acknowledge that video predictions at pixel-level are extremely challenging and many have proposed their idea like only predicting transformation / hierarchical output / foreground-background factorization, etc.  But this work seems brute-force generates pixels with skip layers. I don’t see why it uses parameters more efficiently.  I look forward to authors justifying their idea. 
- Limited Novelty. Though the paper has achieved some empirical contribution (manage to overfit and improve results on large-scale datasets), the technical contribution is limited. Their architecture is a UNet-based VAE. Data augmentation is well known to prevent overfitting and their particular augmentation is also based on prior work.  
- Some heterogeneous metrics. Tabel1 does not report results on BAIR pushing dataset while Table2 only reports on BAIR pushing dataset on one metric. I wonder the reason behind the inconsistency. 1) can table 1 include results for BAIR? 2) can table 2 include other datasets? 3) can other metrics be included? E.g. report the best of 100 samples to incorporate with random sample. 
- Their qualitative results on the website look very similar to baselines (GHVAE) for me.
- Claim on Sec 5 about generalizing to unseen frames. It is a weird claim about video prediction as the unseen future itself is output. The model copy-pasting something from train set does not indicate generalization.  I would just call it overfitting/memorizing train set. (okay w claim 2 but not claim 1) 
- Is the figure 6 results with or without data augmentation? If it’s without, will data augmentation alleviate overfit? If it is with data augmentation, direct comparison (either quantitatively or qualitatively) is expected.


--------- after rebuttal -----
I'd like to thank authors for their detailed response and additional experiments/ablation that they ran. Some of my concerns are resolved (point 3,5,6). For point 4 (qualitative results), I cannot find qualitative comparison on the website anymore. I think it was there by the time I reviewed but not 100% sure. So I check the qualitative results in papers and on GHVAE's website. Most of figures in the paper only shows ground truth with their results w/o comparison except fig 15. For seqs on their and GHVAE website, I'm not convinced about the qualitative improvement, esp not the same seq. Quantitative results are important but qualitative comparison complements paper's arguments. For point 1,2, I appreciate the extra ablations by the authors. There are more empirical results (e.g. switching enc/dec respectively, hyperparam sweep) but the overall novelty is still limited. I'll keep my original score. 
summary_of_the_review: The paper focuses on an important problem on scaling video prediction and manage to overfit to large-scale data as the first step followed by improving generalization by data augmentation. However, it is not clear to me why and how their proposed method is crucial to achieving this.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a simple and scalable variational video prediction model FitVid, which attains a better fit to video prediction datasets even with a similar parameter count as prior models. The author has observed that previous methods suffer from underfitting on these datasets, directly applying FitVid actually results in overfitting. The FitVid uses a set of existing image augmentation techniques to prevent overfitting so that it can achieve state-of-the-art results on several prediction benchmarks. FitVid's architecture is based on SE-UNet LSTM, which seems to be the common backbone for the stochastic video prediction task. 
main_review: Strengths:
The paper discovers the underfitting phenomenon which is a novel but the basic problem for the video prediction task. The FitVid's architecture is simple but effective. The author uses several existing methods to overcome the underfitting problem and achieves state-of-the-art performance on four challenging datasets. I really appreciate that the experiments of this paper are very solid.

Weakness:
The FitVid's architecture has no additional novelty and just combines the existing methods, such as SE, UNet, and LSTM. The paper did not analyze the effectiveness for each part of the FitVid. Why do previous methods not behave well? The explicit explanation of this problem is important and needs to be explored. I'm looking forward to your reply to this question.


====After rebuttal====
I have read the revised paper and rebuttal comments. I do really appreciate the author's effort to clarify the novelty of the combination of each module. However, I still hope that the proposed combination could be intuitively inspired by the overfit phenomena rather than you just design it and use a large number of experiments to vary its effectiveness. 
summary_of_the_review: The paper proposes a simple and scalable variational video prediction model FitVid, which could overcome the underfitting problem better.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a new network architecture called FitVid to perform the task of video prediction, i.e. the task of predicting future frames from previous frames. Previous methods for this tasks usually suffer from "underfitting", while FitVid is able to overfit on major benchmarks without increasing the number of parameters, thanks to a better model architecture. What is more, FitVid is much easier to train than previous work, without using any bells and whistles in training. As a result, FitVid achieves state-of-the-art on four challenging video datasets across a wide range of metrics.
main_review: The paper is easy to follow and is clearly written. The strengths can be summarized as:
+ The results look strong and are much better than state-of-the-art across a wide range of datasets and metrics. The qualitative results look particularly good.
+ The FitVid model is much easier to train and requires no tricks or extra designs to train as in previous works. It can be trained simply from scratch using Adam.
+ A main argument about the FitVid is that it does not suffer from underfitting issues and is even able to overfit. The ablation studies are extensive and indeed support this argument.

Weaknesses
- It is not clear how the FitVid model is different from previous models, and why it is able to overcome the underfitting issues while the previous work could not. Even without the data augmentation, FitVid seems to be able to overfit already. Why? More discussion should be devoted to this topic. 
- Minor grammar issues. For example: "To provide more performance comparison more prior methods..."

It would also be great if the authors could provide some thoughts on this issue:
* Experiments on RoboNet show strikingly good results in Fig4. As stated in the figure caption -- "The model also predicts detailed movements of the pushed objects (visible in the left example) as well as filling in the previously unseen background with some random objects (look at the object that appear behind the robotic arm in the right)".   It seems miraculous to me that the model is able to figure out the random objects in the background and their appearance after a few frames. Usually such tasks require understanding of the physical scene (occlusion, object geometry), physics and even 3D rendering. It seems that the proposed model is able to figure out all these very high-level understanding of the scene using a relatively straightforward network design and no extra mechanisms. Could the authors shed some light on how this is achieved?
summary_of_the_review: I think the paper could provide more insights on why the FitVid network is able to overfit without bells and whistles using even fewer params than previous methods. It would make the paper even stronger. 


-------------------------------------------------------------------- Post rebuttal
I have carefully read the reviews and the rebuttal. The main concerns revolve around a lack of novelty. The authors rebut by providing a detailed study of the proposed components. I think the rebuttal addresses well why carefully designed training schedules are no longer necessary, which might be useful to the community. Novelty-wise, it is still a bit limited, and the overall impression to me it is a bit empirical. But given the strong performance, I am keeping my original rating.
review_title: null

Meta-review: This paper proposes a  variational video prediction model FitVid and attains a better fit to video prediction datasets.  The draft was reviewed by four experts in the field and received mixed scores (1 borderline accept, 3 reject). The reviewers raised their concerns on lack of novelty, unconvincing experiment, and the presentation of this paper. For a video prediction model, fitting a dataset is quite important. But AC agrees with the reviewer jPAY. It will be more exciting to build a causal model of the world and enable it to perform future and counterfactual prediction (e.g,  CLEVRER).  The authors are encouraged to consider the reviewers' comments when revising the paper for submission elsewhere.
Decision: Reject