Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper tackles the video action recognition task focusing on temporal modeling. An Alignment-guided Temporal Attention (ATA) module with similarity-based feature-level alignment is proposed to enlarge the mutual information among consecutive frames, and theoretical analysis shows that increasing mutual information in neighboring frames' feature representations can increase task-relevant information and benefit the model performance. Experiments are conducted on two video action recognition datasets, i.e., Kinetics-400 and Something-Something V2, and show that the proposed ATA module can be a general plug-in module to apply on any image backbones for video recognition task.



strengths_and_weaknesses: - Strengths:
The theoretical analysis is a plus, showing that increasing mutual information in neighboring frames' feature representations can increase task-relevant information and bring performance improvement.

- Weaknesses:
1. the proposed ATA module contains one alignment component and one de-alignment component. Though it is motivated in L183 that “To keep the original location-based information and facilitate subsequent spatial operations, a recovery with de-alignment as in Eq. (7) is thus needed.", how will the de-alignment component affect the performance?

2. The main results shown in Table 1 and Table 2 do not show significant improvement compared to other models, without convincing justification for the relative inferior results. Some explanations like in L214 that"Our approach can achieve better performance if using pretrained models on larger datasets, e.g., JFT-300M."  just show the anticipation without the real result support.

3. For the ablation study of ATA module in different locations shown in Table 3, the ATA module is inserted in an increasing order. How about the independent performance of inserting ATA module into different locations?



questions: 1. For the results in Table 4, why are the results in the last line highlighted? The numbers in that line are not the best compared to others.

2. It is better to mark these symbols from equations (17) and (18) into the Figure 5 for easy understanding.
limitations: The authors mention the privacy issue of using these video action recognition models at certain circumstances, e.g. hotels and homes,
in the part of potential negative societal impacts. While the general solution of restricting the model usage permission is pointed, the specific policies still depend on multiple factors, e.g. social science, etc.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper presents a method for video action recognition. It discusses the problems of factorized (2D+1D) and joint (3D) spatial-temporal operations and proposes Alignment-guided Temporal Attention (ATA) for temporal modeling. ATA achieves strong performance while maintaining low computational overheads. What’s more, this paper theoretically proves that the frame-by-frame alignment increases mutual information between neighboring frame representations, thereby potentially including more task-relevant information. Extensive experiments show the effectiveness and generality of ATA, which not only achieves satisfying results on multiple video action recognition benchmarks, but also can act as a general plug-in to extend any image backbone for video learning tasks.
strengths_and_weaknesses: Strengths:
The paper is well organized and the writing is clear. The figures are well-understood. The experiments seem to be limited but convincing. 
Weaknesses:
This paper doesn’t propose a model which is original and innovative enough. Because the main contribution is the aligned temporal attention operation, however the alignment operation seems to be as same as the alignment method in reference [4].

questions: 1.	The idea of aligned temporal attention is reasonable, however, the alignment method seems to be not innovative enough. It seems that the pixel alignment method in this paper is as same as that in reference [4]. Could you please clarify the difference of these two alignments?
2.	Could you please explain more about the Figure 1 (a)? How can the Task & representations shared information (illustrated by red line) be obtained? Why is it a trapezoid? Is the Task-relevant information obtained from RGB videos?
3.	It seems that the paper doesn’t cite the SOTA results to make the comparison, for example the articles below should be cited and compared.
-	Liu Z, Ning J, Cao Y, et al. Video swin transformer. CVPR. 2022.
-	Li K, Wang Y, Gao P, et al. Uniformer: Unified transformer for efficient spatiotemporal representation learning. ICLR, 2022.
4.	When calculating the alignment with Kuhn-Munkres Algorithm, cosine similarity is employed. Why the cosine similarity is chosen? Could you give some ablation studies based on the choice of similarity?

limitations: 1.	This paper indicates that the aligned temporal attention can mine more task-relevant information from the aligned pixels. However, the spatial-temporal self-attention structure can model the attention among all the pixels along the time axis. Consequently, it seems that the alignment doesn’t have much significance.
2.	It seems that the results don’t have a huge improvement. For example, in Table 1, there is no large difference between the results of MViTv1-B, ViViT-L, Mformer-HR and ATA for Top-1 Acc.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper tackles the problem of modeling temporal context from the input frames for video action recognition task. It discusses the current limitations of existing approaches based on factorized and joint convolutions. As a result, it proposes to use frame-by-frame alignment to potentially increase mutual information of frame representations. The proposed Alignment-guided Temporal Attention (ATA) is studied on two widely used action recognition datasets, Kinetics400 and Something-Something V2 showing improvement over baselines models. 
strengths_and_weaknesses: **Strengths**:
1) The paper is well-written with nice and useful visualizations. The motivation for the proposed method is clear. 
2) The discussion of the limitations of the current standard convolutional approaches for temporal modeling is thorough. The theoretical results help to better understand the internal benefits of the method.
3) Ablation study shows the improvement over simple averaging and another attention-based method. 
4) Combination of ATA with TimrSformer achieves state-of-the-art results, outperforming even larger variants of TimeSformer (TimrSformer-L and TimrSformer-HR).

**Weaknesses**:
1) I would like to see more discussion and comparison with attention-based methods? How does frame-by-frame alignment help in comparison to global attention? The complexity of the Kuhn-Munkres Algorithm is also cubic as in global attention. So what are the benefits?


questions: 1) Why there is an increase in computational complexity over baseline TimeSformer with 8 frames. Is it just using a larger number of input frames (32)?
2) Would be more beneficial to have a more detailed comparison and discussion with different attention methods like vanilla attention and some factorized and more efficient versions of attention. 
limitations: I find the discussion of the limitations is not sufficient and detailed.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper introduce a novel alignment-guided transformer architecture for video action recognition. Specifically, the proposed method makes use of KMA to obtain the alignment matrix. Then 1D temporal operation is further used to extract better video representation. The proposed method achieves competitive results on mutiple action recognition datasets.
strengths_and_weaknesses: +the core idea of using KMA to align spatial information for temporal operation is very interesting and novel to my best knowledge.

+The paper is overall well-written.

+The mathematical derivation is solid, and the results are strong.

+The proposed method is robust and can achieve consistent performance improvement using different backbones.


-The analysis on what has been learned by the model is not thorough enough, see questions section

-some details should be presented with more clarity see limitation section


questions: Q1.Have the reviewers investigates how the temporal sampling rate affects the proposed method?There are two ways to examine this.
1. If the input is temporally downsampled by x2 x4 x8, how does this affect the model performance
2. the ATA is done on every other frame, what is the performance difference

Q2. Does the model learn better motion representation? This can be done by conducting the arrow of time experiment as in [A,B]
[A] Xie, Saining, et al. "Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in video classification." Proceedings of the European conference on computer vision (ECCV). 2018.
[B] Zhou, Bolei, et al. "Temporal relational reasoning in videos." Proceedings of the European conference on computer vision (ECCV). 2018.

Q3. How does the global and local motion affects the method. According to the authors, the model performs better on ssthv2. Is this because the videos in ssthv2 have more global motion. It would be great to see some analysis on this.
limitations: 1. This paper needs some additional analysis on what has been learned by the proposed method. See questions raised in Questions section.
2. This paper should better describe the efficiency part. Though the proposed method is parameter free, how ever it does incur quite some additional computational cost when comparing with Timesformer backbone as shown in Table2. The authors should better tune the tone to help readers understand the additional computational cost
review_title: null

Meta-review: metareview: Paper was reviewed by four reviewers, receiving: 2 x Borderline Rejects and 2 x Weak Accepts. Importantly, post rebuttal, [1mVh] mentioned upgrading the rating from Borderline Reject to Borderline Accept (though this is not reflected in final ratings). The general concerns raised by the reviewers included, limited improvements over the baselines and lack of certain ablations / comparisons. Much of these concerns have been addressed by various new experiments and discussions provided during the rebuttal period. [1mVh], [Ppgh] and [Fgtq] have all acknowledged that their concerns were largely resolved. [Shks], who remained the only negative reviewer, did not participate in the discussion nor acknowledged reading the author responses. AC has gone through the responses to comments of [Shks] and found them partially convincing. Overall, given the overall positive assessment of the reviewers and the generality of the proposed approach that can be combined with variety of architectures, the work will make a fine contribution to NeurIPS. 
confidence: Less certain
Decision: Accept