Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes the Motion-Aware Unit (MAU) for video prediction. This module attempts to model the temporal information in videos by enlarging the temporal receptive field and aggregating it through an attention mechanism. The proposed module is employed in the bottleneck part of an autoencoder where the spatial and temporal information are combined at different MAU layers. The proposed method is compared to previous works on several benchmarks for next frames prediction and early action recognition showing state-of-the-art results.
main_review: The proposed method shows interesting results on both the video prediction and early action recognition tasks. Although the idea of enlarging the temporal receptive field when dealing with sequence forecasting is not new [1], in the video prediction task, this idea is not already investigated. However, the method introduced is incremental as the main contribution stands on the aggregation of spatial and temporal features using an attention-based mechanism.

Figure 1 on the right is barely comprehensible; I suggest making it clearer. In line 110, the L1 and L2 terms are not introduced before; it would be better to state what they are clearly.

Many important details regarding the model and the experiments are missing.
- Which encoder and decoder backbones do you use for the experiments?
- Are the encoder and the decoder comparable with respect to previous works in the results reported in the Caltech Pedestrian, KITTI, and TownCentreXVID datasets?
- What is the dimensionality of the internal features of the model?
- Does such dimensionality change in the different benchmarks you used?
- How does such dimensionality impact the efficiency of the model?
- Is your architecture fully convolutional?

Lots of details for reproducing the results are also missing (training parameters, ...).

It would be useful to visualize the attention scores of the MAU module in order to better understand the behavior of the proposed model when it generates new frames.

References:

[1] Attention is All you Need, Vaswani et al, NIPS 2017.
limitations_and_societal_impact: The authors do not highlight any limitations of the proposed method. Moreover, I do not find any potential negative societal impact regarding this work.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a method for video prediction that relies on a newly proposed Motion-Aware Unit (MAU). MAU acts as a temporal receptive field based on an attention mechanism that is used to aggregate features from previous frames in order to predict future frames. The aggregate features are combined with “content” features extracted from the last observed frame and the decoder is connected to the encoder via an “Information recalling scheme” that directly passes the last observed frame features from each layer of the encoder to each layer of the decoder. In experiments, the authors show superior performance against the baselines pixel based metrics and a perceptual metric.
main_review: Strengths:
+ A Motion-Aware Unit (MAU) that enables direct retrieval of previous frames extracted features using attention.
+ Combination layers that aggregate motion and content information by choosing important information from each information stream.
+ Outperforms baselines

Weaknesses:

- Evaluation (Table5 and Table 6):
As far as I can tell, the experimental setup is described in Table 1. However, when I read the captions on top of Table 5 and 6, I see that the authors describe something different. They indicate that for the Caltech and TownCentreXVID experiments their test setups are (10->1) and (4->1), respectively. This is completely different to what Table1 says. The test setups in Table 1 are (10 -> 10) and (10->10). If we assume that they mean train setup, it is the same story. The train setups in Table 1 are (10->1) and (10->10). Can the authors clarify this? I would also suggest the authors include a sequence level evaluation metric such as Fretchet Video Distance. Frame level evaluations do not usually tell the full story video prediction since we need to also measure if the dynamics are also plausible. FVD repo: https://github.com/google-research/google-research/blob/master/frechet_video_distance/frechet_video_distance.py. Finally, the use of evaluation metrics are not consistent throughout the experiments. I understand that perceptual evaluations such as LPIPs may not make sense for the Bouncing MNIST experiments, but they are applicable to both Caltech and TownCentreXVID experiments. However, I only see perceptual evaluations on the latter. Can the authors explain why this is the case?

- Ablations:
The authors have provided ablations of \tau for the number of frames considered in the temporal receptive field. However, there are other pieces in the architecture/formulation that would be good to see how much they contribute to performance. For example, what happens in Equation 5 if we simply let T_{AMI} = T_{attn} or T_{AMI} = T^k_{t-1}? T_{attn}. Another place where an ablation is necessary is  Equation 7. What if we let S^k_t = W_{ts} * T_{AMI} + \gamma S^k-1_t? Or can we visualize U_s to see if the network simply decided to set U_s=0 during training? This would be informative to decide whether the current complexity is necessary or not.

- TownCentreXVID experiments:
Table 6 shows that the proposed method outperforms the baselines. However, it is very difficult to tell from Figure 4 whether the proposed method is simply copying the last observed frame or making an actual prediction of the future. From very closely inspecting MAU against the groundtruth, it looks like what MAU outputs is what seems to be the frame before the Grouthtruth. It has what seems to be a temporal progression. However, as I said before, it’s hard to tell without previous image context. In addition, the folder in the supplementary material where visual results should be for this dataset is empty. The author’s should have provided a baseline where the last observed frame is copied and evaluate it to check the numbers against the proposed method and make sure it’s not doing the same thing.

- Video results:
Papers on video prediction usually provide video results since it’s very difficult to tell the quality of the prediction from frames on a pdf. This would have clarified a lot of the concerns mentioned above.


- Limitations of this work?
The authors have not mentioned any limitations of this work, and also marked N/A in the limitations question of the checklist. Why are there no limitations? Did the proposed method completely solve the problem of video prediction? Does the method learn perfect features for the action recognition/anticipation task or other sequence downstream tasks? From reading the paper, I see there are a lot of limitations, some of which we should be able to see if we have videos to observe. Also the action recognition/anticipation experiments (quality of learned features verification) accuracy is really low. In the future, the authors should have a paragraph or two describing what is failing in their method and what needs to be done next so that other researchers can pick up from there or get inspired by the work. That’s what academia is all about. 

Suggestions:
- Action recognition experiments against recognition methods: 
The provided action recognition experiments are great to show that the proposed method learns something useful from predicting future frames. I also suggest the authors train a network to do action recognition from scratch and compare against it. It would give readers a good idea about the gap between self-supervised and fully-supervised feature learning for this task.

- Action recognition without “recalling” in the prediction training:
The “recalling” connections from encoder to decoder are basically skip connections which have been used by previous works to get better frame prediction and make learning easier. However, these skip connections could have a negative impact on feature learning since the network can just copy a large portion of the pixels in the input image. I wonder if the action recognition performance improves if the recalling connections are removed because the features have to represent the entire image during prediction.

Conclusion:
I really like the formulation of this method as it is simple and seems that it should work well. However, the experimental section has a lot of issues that I cannot look past because I like the method. Because of this, I am at the borderline/leaning towards rejection of this paper. I am looking forward to the author's response to hopefully address my doubts/concerns.


#####################################
##### Post rebuttal comments ############
#####################################

The authors addressed all my concerns. Therefore, I have decided to increase my score to accept.

limitations_and_societal_impact: They have not addressed their limitations or societal impact of their work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: In this work, the authors introduce a Motion-Aware Unit (MAU) for video prediction. Basically, they discover the previous motion and aggregate it via attention module and fusion module. In the attention module, they compute correlation between spatial states and use it as attention for aggregation the previous temporal states. In the fusion module, they design spatial and temporal update gates for fusing attentive temporal state and current spatial state.
main_review: 1. Originality: The novelty is relatively limited. Attentively discovering historical motion has been widely investigated in online action recognition [Skeleton-Based Online Action Prediction Using Scale Selection Network, TPAMI2020] and detection [Learning to Discriminate Information for Online Action Detection, CVPR2020]
2. Quality: The motivation, model design and experiments basically make sense.
3. Clarity: It is an OK paper for me. 
4. Significance: It is probably useful for online action recognition and detection, etc. 
limitations_and_societal_impact: Please refer to Main Review for the detailed comments.
1 Novelty is limited. The design is not quite new, based on the fact that attention for motion learning has been widely used in video understanding.
2 By the way, temporal shift module [TSM: Temporal Shift Module for Efficient Video Understanding, ICCV2019] is a popular mechanism for early action recognition. It would be interesting to see how it works in Table 7.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a Motion-Aware Unit (MAU) for video prediction tasks. The MAU includes two modules, the attention module and the fusion module. The attention module aims to learn the correlation between the current spatial state and previous spatial states. The learnt attention map can be used for augmenting the motion information (AMI). The fusion module is used for aggregating the augmented motion information (AMI) and current spatial state. The authors have demonstrated superior performance compared to state-of-the-arts.
main_review: Apart from the good results presented in the paper, I have some questions about the paper as below:
1) P2 L67, the authors mentioned the good performance on the early action recognition task, but I have not found early action related experiment presented in the paper.
2) P4 L147, the authors clarify that they aim to quantify the correlation between different temporal states. But for **2nd line in eq.3**, how the Hadamard Product operation between two spatial states is able to quantify the correlation between matrices? 
3) Do SUM and MUL in Figure 1 represent summation and multiplication? Needs an explanation here in the caption and main text.
4)  **2nd and 3rd lines in eq.3**, I suppose the objective of the 2nd line is to calculate the correlation between one state $t$ and all previous temporal states in window $\tau$ based on this state. If so, the subscription of $q_i$ is confused, it should be $q_t$.
5) In the experiment part, are backbones in encoder and decoder the same as a backbone in previous approaches? This needs description in the experiment part e.g., what backbones are used in encoder and decoder as well as for other approaches.
limitations_and_societal_impact: The authors have not stated negative parts. Current video prediction tasks are still low resolution (max $128\times 128$), which might not be suitable for real-life applications. Future work can be extended based on high resolution.
review_title: null

Meta-review: There was a robust discussion between the reviewers on the merits of this work. The author response helped clarify many of the things that the reviewers asked for. All in all, I feel this work should be accepted at NeurIPS. While the reviewers as a whole felt that there is some lack of novelty in the work, and had some minor constructive feedback (e.g. more justification for not performing multi-step prediction in their TownCentreXVID experiments), the conclusion is that the contributions are sufficient and interesting enough to warrant acceptance. 
Decision: Accept (Poster)