Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a second-order pooling method, called temporal-attentive covariance pooling (TCP), for summarizing temporal features and generating video representation. Specifically, TCP involves three components: 1) a temporal convolution before covariance pooling that introduces learnable parameters for temporal modeling; 2) a spatial and channel attention to further enhance the convolutional features; 3) matrix power normalization to utilize geometry of covariance space. Experiments on multiple video datasets show that TCP outperforms the standard global average pooling and simple covariance pooling.


After author rebuttal:
The author responses address most of my major concerns: (1) Contribution/novelty: after carefully reviewing the paper, the review from other reviewers and the author's rebuttals, I agree to recognize the contribution of TCP for introducing a solid framework that successfully applies the second-order pooling method to boost video action recognition. (2) TSA design: although the method is not intuitive to me, the comparison results provided in the author rebuttal justify the effectiveness of the proposed design.
In all, I'll raise my rating to "6: Marginally above the acceptance threshold".
main_review: My major concern of this paper is its limited technical contribution -- most components of TCP are not novel and have been widely studied in prior work.

1) TCP aims to improve the vanilla covariance pooling by introducing order-aware aggregation along temporal dimension. However, simply adding a temporal convolution before covariance pooling is not sufficient for achieving this goal. First, temporal convolution can only capture local temporal dependencies within a small sliding window. The video-level temporal aggregation is still unachievable because the covariance features are still pooled by simple average operation. Moreover, using temporal convolution to capture local temporal dependencies is not novel and has been widely used for video recognition.

2) The proposed temporal-based spatial attention and channel attention are not novel as well. Both spatial attention [1] and channel attention [2] have been widely used for video recognition, and their combination is also presented in prior work, for example [3]. Moreover, the motivation of using $X_{l-2}, X_{l-1}$ as query and key for $X_l$ is not clear. $X_{l-2}$ may have very different data distribution with $X_{l}$, which makes the alignment of the generated attention map and the feature map $X_{l}$ inaccurate.

3) According to Table 1, the contribution of matrix power normalization is the most significant in TCP. However, matrix power normalization is proposed and used for vision tasks in prior work and it is simply applied to TCP in this paper.

Minor issue: The terms "intra-frame" and "inter-frame" in Section 3.2 is misleading or even incorrect. Correlation within the same frame $X_l^T X_l$ should be intra-frame covariance instead of inter-frame covariance, and vice versa.

In all, although the proposed TCP achieves superior accuracy compared with simple average pooling and covariance pooling, the novelty of TCP is limited and more careful designs to improve the temporal aggregation of covariance features are needed. In addition, considering the extra computation and parameter introduced in computing covariance pooling, spatial/channel attention and matrix power normalization, the performance gain of TCP is not that significant, especially for 3D networks that already consider local temporal dependencies.

[1] Wang, Xiaolong, et al. "Non-local neural networks."
[2] Feichtenhofer, Christoph. "X3d: Expanding architectures for efficient video recognition."
[3] Woo, Sanghyun, et al. "Cbam: Convolutional block attention module."

limitations_and_societal_impact: The paper didn't discuss its limitations and potential negative societal impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents a temporal-attentive covariance pooling (TCP) method for action recognition, which consists of a temporal attention module, a temporal covariance pooling, and a fast matrix power normalization of covariance. TCPNet is then devised by inserting TCP at the top of existing deep architectures. The experiments are conducted on 6 action recognition benchmarks to evaluate the proposed model.
main_review: Strengths:

++ The paper is clearly written and well organized.

++ Extensive experiments are conducted on 6 benchmarks.

Weaknesses:

-- The technical contribution of this work is somewhat limited. The main idea of temporal-attentive covariance pooling seems to be a simple combination of self-attention mechanism and deep covariance pooling which have been widely used for action recognition [7, 17]. The combination of these techniques could be a contribution. But to me, the technical contribution is somewhat incremental for a NeurIPS paper.

-- In Table 4(a), the authors only compared the proposed TCP to one simple pooling method (GAP). It is better to include more feature aggregation baselines (such as ActionVLAD[18] and TLE[7]) for performance comparison.

-- In section 4, the authors should compare the proposed TCPNet with the state-of-the-art methods, such as [a] (81.0% on Kinetics-400, 53.3% on S-S V1) and [b] (50.6% on S-S V1).

[a] Heng Wang, Du Tran, Lorenzo Torresani, and Matt Feiszli. "Video modeling with correlation networks." In CVPR, 2020.

[b] Swathikiran Sudhakaran, Sergio Escalera, and Oswald Lanz. "Gate-shift networks for video action recognition." In CVPR, 2020.

-- It is better to show some qualitative results of the learnt temporal attention.

limitations_and_societal_impact: I do not find any significant aspect about its limitation and potential negative societal impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes temporal attentive covariance pooling module for video recognition. This module extracts seconder-order information from video feature tensors to improve the conventional global average pooling layer and it works in a plug-and-play fashion. Temporal-based channel attention and spatial attention are also designed to handle complex temporal dynamics. Extensive results are conducted to show effectiveness of the proposed module. 
main_review: Originality: This paper extends second-order pooling, which is originally developed for image recognition, to the field of video recognition. Instead of directly appling covariance pooling on per-frame basis, this paper proposes temporal-based spatial and channel attention mechanisms to model complex temporal dynamics. Besides, the temporal coviarance pooling scheme takes into acount both intra- and inter-frame covariances. From this point of view, the novelty of this paper is OK.

Quality: Generally, the overall quality of this paper reaches or is slightly above the bar. The motivation of this paper is reasonable  and the performance of the proposed TCP is validated via extensive experiments.

Clarity: It is clear for readers to follow the main idea of this paper.  The draft is well organized and the presentation of this paper is good despite of the following small issues: 
1. In eq.5 and eq.6, in my opinion, the first part denotes intra-frame covariance and the second part denotes inter-frame covariance. The terms in the text are also misleading (e.g., line-170, C_{l,l} should be intra-frame covariance, C_{l,l-1} should be inter-frame covariance).
2. There are works which perform local feature integration to generate a compact representation from video feature tensors, such as ActionVLAD [1], NeXtVLAD [2], AttentionClusters[3]. Having some discussions on them or making some comparisons with them is better.

[1] Girdhar, Rohit, et al. "Actionvlad: Learning spatio-temporal aggregation for action classification." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017.

[2] Lin, Rongcheng, Jing Xiao, and Jianping Fan. "Nextvlad: An efficient neural network to aggregate frame-level features for large-scale video classification." Proceedings of the European Conference on Computer Vision (ECCV) Workshops. 2018.

[3] Purely Attention Based Local Feature Integration for Video Classification
X Long, G De Melo, D He, F Li, Z Chi, S Wen, C Gan - IEEE TPAMI2020

Significance:  The evaluation results show superiority of TCP, compared to various global pooling methods. The TCP module is also verified to be applicable for various 2D CNN, 3D CNN backbones. The proposed TCA and TSA modules are also shown to be helpful. But the evaluation part should be improved:
1. Matrix Power Normalization is a n existing technique, therefore, in Table.1, the "+Matrix Power Normalization" is better moved to the upper part of this table and it can viewed as a stronger baseline compare to Plain GCP.  The Major contribution of this paper is TCA, TSA and TC-COV, which bring gain of 1.6% in total. 
2. The hyper-parameter k should be studied. 
3. Plain GCP + Matrix Power Normalization, which is a straightforward extension from existing state-of-the-art image covariance pooling, should better be compared in Table 4(a) to make the gain of the proposed TCP more clear.  The baseline of GAP in table 4(a) is kind of weak. 

Overall, I think this is a nice work and I am willing to raise the score if my concerns can be addressed in the rebuttal phase. 
limitations_and_societal_impact: None
review_title: null

Meta-review: All three reviewers provided a rating of "6: Marginally above the acceptance threshold" for this submission. Reviewers qjEJ and SHTG initially raised concerns about the technical novelty of the proposed temporal-attentive covariance pooling (TCP). However, the authors provided a response that convinced both reviewers that there are significant differences between the TCP and related mechanisms introduced in prior works. Reviewer qjEJ pointed out the somewhat unintuitive design of TSA but concurs that the approach is empirically effective. Finally, Reviewer zp2B requested a few additional studies (such the comparison to local feature aggregation methods and the ablation on k), which were presented in the response by the authors. The ACs agree with the recommendation of accepting the paper.
Decision: Accept (Poster)