Temporal-attentive Covariance Pooling Networks for Video Recognition
Authors: Zilin Gao
Qilong Wang
Bingbing Zhang
Qinghua Hu
Peihua Li
Venue: NeurIPS 2021 Poster (2021)
Keywords: Video recogniton
Temporal covariance pooling
Temporal-based attention

Abstract: For video recognition task, a global representation summarizing the whole contents of the video snippets plays an important role for the final performance. However, existing video architectures usually generate it by using a simple, global average pooling (GAP) method, which has limited ability to capture complex dynamics of videos. For image recognition task, there exist evidences showing that covariance pooling has stronger representation ability than GAP. Unfortunately, such plain covariance pooling used in image recognition is an orderless representative, which cannot model spatio-temporal structure inherent in videos. Therefore, this paper proposes a Temporal-attentive Covariance Pooling (TCP), inserted at the end of deep architectures, to produce powerful video representations. Specifically, our TCP first develops a temporal attention module to adaptively calibrate spatio-temporal features for the succeeding covariance pooling, approximatively producing attentive covariance representations. Then, a temporal covariance pooling performs temporal pooling of the attentive covariance representations to characterize both intra-frame correlations and inter-frame cross-correlations of the calibrated features. As such, the proposed TCP can capture complex temporal dynamics. Finally, a fast matrix power normalization is introduced to exploit geometry of covariance representations. Note that our TCP is model-agnostic and can be flexibly integrated into any video architectures, resulting in TCPNet for effective video recognition. The extensive experiments on six benchmarks (e.g., Kinetics, Something-Something V1 and Charades) using various video architectures show our TCPNet is clearly superior to its counterparts, while having strong generalization ability. The source code is publicly available.

Temporal-attentive Covariance Pooling Networks for Video Recognition
Zilin Gaot, Qilong Wang', Bingbing Zhang', Qinghua Hu', Peihua Lit*
School of Information and Communication Engineering, Dalian University of Technology
*College of Intelligence and Computing, Tianjin University
gzl@mail dlut. edu. en, qlwanget ju. edu, en, icyzhangemail www.llt...ooo .uutteddu.cmm huginghua@= ju, edu, en, peihuall@dlus edu, on
Abstract
For video recognition task, global representation summarizing the whole contents of the video snippets plays an important role for the final performance. However, existing video architectures usually generate it by using a simple, global average pooling (GAP) method, which has limited ability to capture complex dynamics of videos. For image recognition task, there exist evidences showing that covari- ance pooling has stronger representation ability than GAP. Unfortunately, such plain covariance pooling used in image recognition is orderless representative, which cannot model pemporl structure inherent in videos. Therefore, this paper proposes Temporal-attentive Covariance Pooling (TCP), inserted the end of deep architectures, to produce powerful video representations. Specifi- cally, our TCP first develops temporal attention module to adaptively calibrate spatio-temporal features for the succeeding covariance pooling, approximatively producing attentive covariance representations. Then, temporal covariance pool- ing performs temporal pooling of the attentive covariance representations to char- acterize both intra-frame correlations and inter-frame coss-coorrllttions of the calibrated features. As such, the proposed TCP can capture complex temporal dynamics. Finally, fast matrix power normalization introduced exploit geometry of covariance representations. Note that our TCP is model-agnostic and can be flexibly integrated into any video architectures, resulting in TCPNet for effective video recognition. The extensive experiments on six benchmarks (e.g., Kinetics, Something SeeethinngSomethngg VI and Charades) using various video architec- tures show our TCPNet is clearly superior to its counterparts, while having strong generalization ability. The source code is publicly available.
Introduction
Video recognition aims to automatically analyze the contents of vidcos (c.g.. events and actions), and has wide range of applications, including intelligent surveillance, multimedia retrieval and recommendation. The great progress of deep learning, especially convolutional neural networks (CNNs) 48
remarkably improves performance of video recognition. Since videos in the wild involve complex dynamics caused by appearance changes and variation of visual tempo, powerful video representations potentially provide high performance of video recognition. However, most of the existing video recognition architectures based on 3D CNNs [53 43 2
52 22
54 [0] or 2D
CNNs 35 pooling (GAP). Such GAP simply computes the first-order statistics of convolution features in temporal orderless manner, which discards richer statistical information inherent in features and has limited ability to capture complex dynamics of videos.
usually generate the final video representations by leveraging global average
Corresponding author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Temporal-based
Temporal
Classifier
CNN Backbone
Channel Attention
Matrix Power Normalization
Covariance Pooling
Temporal-based
Spatial Attention
Temporal Attention Module
Temporal-attentive Covariance Pooling
X
GAP(X, ....
BN
g
X
x,
Srea
GAP(X, -X,,)
X,
X
X
Temporal-based Channel Attention
Temporal-based Spatial Attention
Figure I: Overview our TCPNet for video recognition. Its core a Temporal-attentive Covariance Pooling (TCP) integrated at the end of deep recognition architectures to produce powerful video rep- resentations. Specifically, our TCP consists of temporal attention module for adaptively calibrating spatio-temporal features, temporal covariance pooling to characterize mntra-frame correlations and inter-frame cross cross-correlation. of the calibrated features in temporal manner, and a fast matrix power normalization for use of geometry of covariances. Refer to Section 3.2 for details.
There are a few researches studying global high-order pooling for improving performance of video recognition 4S 29 7 Sanin et [45] propose spatio-temporal covariance descriptors as video representations, where covariance descriptors are built with hand-crafted features of candidate regions and a subset of the optimal descriptors are learned by LogitBoost classifiers. However, such method is designed based on the idea of the classical shallow architectures, without making the most of the merits deep learning paradigm. In (2], two tensor-based feature representations (namely SCK and DCK) are introduced for action recognition, where multiple subsequences and CNN classifier scores are used to compute tensor representations while a support vector machine trained for classification (see Figures 2&4 and the third paragraph of Section 5.2). Under end-to-end deep architectures, Diba et al. 77 propose a temporal linear encoding (TLE) method for video classification. Specifically, TLE adopts a bilinear pooling (37] or compact bilinear pooling [15) to aggregate the features output by 2D and 3D CNNs as the entire video representations. TLE shows better performance than the original fully-connected (FC) layer, but it fails fully exploit temporal information and geometry of covariance representations, limiting effectiveness of covariance pooling for video recognition. Recently, there also exist evidences showing that global covariance (second- order) pooling can significantly improve performance of deep CNNs in various image-related tasks, such fine-grained visual recognition [37 15), large-scale image classification [31 30 60) and
visual question answering 13 (68) However, these methods aim to generate image representations, without considering important temporal information inherent in videos. Meanwhile, naive usage of image-related covariance pooling (called plain covariance pooling in this work) technique for video-related tasks cannot make full use of the advantage of covariance pooling (63).
As discussed previously, though global covariance pooling shows the potential to improve recognition performance, aforementioned works suffer from the following drawbacks: (1) Shallow approach [45) fails to make the most of the merits of deep learning paradigm; (2) Plain covariance pooling meth- ods (3 15 31 30. 60) as well TLE [ cannot adequately model complex temporal dynamics, and in the meanwhile TLE considers no geometry of covariances. To overcome these issues, this paper proposes Temporal-attentivy Covariance Pooling (TCP), which be integrated into existing deep architectures to produce powerful video representations. Different from the plain covariance pooling methods which perform orderless aggregation along temporal dimension, our TCP presents temporal pooling of attentive covariance representations to capture complex temporal dynamics. Specifically, as illustrated in Figure I our TCP consists of temporal attention module, a temporal covariance pooling, and fast matrix power normalization of covariances. The temporal attention module is developed to adaptively calibrate apatio-temporal features for the subsequent covariance pooling. The covariance pooling of the calibrated features can be approximatively regarded as producing attentive covariance representations, having the ability to handle complex dynamics. Then, temporal pooling of the attentive covariance representations (i.e., temporal covariance pooling) characterizes both intra-frame correlations and inter-frame cross-correlations the calibrated features, which can fully harness the complex temporal information. The fast matrix power normalization [30] is finally

---

introduced to exploit geometry of covariance spaces. Note that our TCP is flexibly integrated into existing 2D or 3D CNNs in a plug-and-pla manner, resulting an effective video recognition architecture (namely TCPNet).
and can be
The overview of our TCPNet is shown in Figure T To verify its effectiveness, extensive experiments are conducted on six video benchmarks (i.e..
[66]. Kinetics-400) [2). Something- Something V1 [9], Charades [46), UCF101 49) and HMDB51 (23) using various deep architectures (e.g., TSN 159 X3D 10 and TEA (33 The contributions of our work are summarized as follows.
(1) We propose an effective TCPNet for video recogmition, where a model-agnostic Semporal-attenter Covariance Pooling (TCP) integrated at the end of existing deep architectures can produce powerful video representations. (2) To our best knowledge, our TCP makes the first attempt characterize both intra-fraut tara-ffame and inter-frame correlations of the calibrated spatio-temporal features in temporal manner, which approximatively performs temporal pooling of the attentive covariance representations. The proposed TCP provides compelling alternative to GAP for improving recognition performance of deep video architectures. (3) We perform comprehensive study of global covariance pooling for deep video recognition architectures, and conduct extensive experiments six challenging video benchmarks. The results show our TCPNet has strong generalization ability while performing favorably against state-of-the-arts.
2 Related Works
2.1 Deep Architectures for Video Recognition
Video data usually consists of appearance information and temporal motion. Two-stream architec- tures [47 [2] are specially developed for separately capturing the two kind of information. They obtain competitive performance, but suffer from huge computational cost due to offline computation of optical flow. Extending 2D convolutions to 3D ones, 3D CNNs achieve state-of-the-arr performance by taking only RGB inputs 53 43 (66] [54] 11 10, which can simultaneously learn appearance and temporal features. Furthermore, a family of attention models based on idea of non-local [6]
64 3 21
are proposed to further improve performance of 3D CNNs by adaptively calibrating spatio-temporal features. Recently, integration of temporal modeling with 2D CNNs achieves promising results in 33 39 40), where 2D convolutions extract appearance features while temporal modules (e.g., channel shift, temporal difference, and temporal convolution) are used to capture motion information. However, these video recognition architectures usually generate the final video representations by leveraging a global average pooling (GAP), which has the limited ability capture complex dynamics of videos. Some works exploit classical encoding methods (i.e., VLAD (26) and Fisher vector (44 to replace GAP for aggregating spatio-temporal features [18] 57) Different from aforementioned methods, we propose temporal-attentive covariance pooling (TCP) to replace GAP for producing powerful video representations in deep architectures.
more efficient way [35 27 9 50
2.2 Deep Covariance (Second-order) Pooling
Global covariance (second-order) pooling has been successfully applied to various image-related
tasks, such as image classification (37 25 15 Bi 30
16 60) and visual question answering [13] 68
These works show global covariance pooling is much better than the original GAP, and appropriate use of geometry of covariances plays a key role for achieving high performance 31). However, these works have no ability to capture temporal information inherent in video data, and naive usage of them cannot fully exploit advantages of covariance pooling
[X]
For modeling videos based on deep second-
order pooling, TLE (7] makes an attempt to use bilinear pooling (37] and CBP [15) for aggregating spatio-temporal features in an end-to-end manner. However, this method makes no full use of temporal information and geometry of covariance representations. Besides, some researches [17] [7] (34] integrate local second-order statistics into deep architectures for improving performance of video recognition. Specifically, AP [17] introduces second-orde -oder statistics by designing an element- multiplication of mauttppe-bbanch based on rank-1 approximation. Sharing similar idea, ABM [7]]
and TBN (34] apply element-wise multiplication to various frames for capturing motion information. Different from above works, our proposed TCP performs a global temporal pooling of attentive covariance representations, which can handle complex temporal dynamics and exploit geometry of covariances. Besides, our TCPNet has the ability to fully harness the merits of deep learning paradigm, compared to shallow approach [45)
3

---

3 Proposed Method
In this section, first briefly introduce the plain covariance pooling for modeling videos. Then, describe detail our proposed temporal-attentin covariance pooling (TCP), which consists of temporal attention module, temporal covariance pooling, and fast matrix power normalization. Finally, we instantiate our TCP with integration of existing deep video architectures, resulting in temporal-attenne attentive covariance pooling networks (TCPNet).
3.1 Plain Covariance Pooling for Modeling Videos
For the existing video architectures, global average pooling (GAP) is widely used to generate the final video representations. Suppose we have total of I. feature tensors output by some deep architecture. Let H, W and C' be spatial height, width and channel number of one feature tensor. We denote by X₁ E NxC the matrix obtained from the (-th feature tensor where N = H x W and each row X is a C-dimensional feature. GAP computes the final video representation PeAP E IRICC' as
1 LN
PGAP
XI,
where L indicates temporal resolution. As shown in Eqn. from statistical view, GAP estimates first-order moment while higher-order statistics containing richer information are discarded.
Compared to GAP, covariance pooling can capture richer statistics by modeling correlation among the features. For video data, plain covariance pooling generates video representation PPCP by aggregating the features as
N Σ 1=1 ==1
PCCP
1
Σ
X
X1,
Σ
X X
(2)
LN
where 7 indicates the matrix transpose. Although PPCP in Eqn. 2 has better representation ability than PCAP, both of them perform orderless aggregation along temporal dimension based simple average operation, lacking the ability to handle complex temporal dynamics. In addition, naive use of PCC only obtains small gains (see Section geometry of covariance spaces is neglected.
3.2
Temporal-attentive Covariance Pooling (TCP)
To overcome the drawbacks of plain covariance pooling, we propose temporal-attentive covariance pooling (TCP) to capture richer statistics of ppatio-tempone features and handle complex dynamics.
3.2.1 Temporal Covariance Pooling
We regard the video signal as stationary stochastic process 38 and propose to model its statistics by the second-order moments in a learnable manner. Specifically, we perform a temporal-attent attentive covariance pooling PTCP for summarizing the statistics of the videos:
PrCp = CP ,...,,,..... C....., CL.., C = N
X,
(3)
where (-y(-) indicates a temporal pooling of frame-wise covariance representations (Ci I-I..... In this work, we achieve (.) with temporal convolution of covariances followed by average pooling:
1 TP CPC Cr,..., Ct)) -
Σ TC-COV, (Xi), I=1
(4)
where TC-COV, (Xt) performs temporal convolution of covariance X, with kernel size of k. By taking 3 as example, achieve (C-COVx=aal (Xx) by considering the temporal correlations of intra- and inter- frames in a sliding window manner. Thus, TC-COV k= (Xi) is computed as
TC-COV x= (X1) W x X X W-1 + w/ W. + WT X141 W1
(5)
intra-framc covariance pooling
X
w x X
inter-frame cross-covarianse pooling

---

where (.).e((1.00 are set of learnable parameters. For Eqn. () the first part of terms is combination of frame-wise covariances learned based on geometry-awarr projections [20]. aiming to characterize intra-frame correlations. Meanwhile, the second part of terms is a combination for learnable cross-covariances between all pairs of different frames, aiming to capture inter-frame correlations. In this work, we model the distributions of stochastic process describing the video signals by second-order moments, Along this line, we can further exploit higher-order (e.g.. third- order) moments [5] in learnable manner for more accurate distribution modeling, which is left for future work.
3.2.2 Temporal Pooling of Attentive Covariances
Although Eqn. (5) performs a temporal convolution on covariances, both intra-frame covariances (e.g., Ct, X X1, 1/N is omitted for simplicity) and inter-frame cross-covariances (e.g.. Ct,-1 x11) computed in specific manner, having limited ability handle complex temporal dynamics. Therefore, we can calculate attentive covariances (or cross-covariances) to adaptively calibrate importance of covariance representations to account for dynamic changes. However, element- wise attention for covariances (c.g., s Ct,1 or S @ Cr,) will break up their geometry. In this work, we approximatively compute attentive covariances by developing temporal attention module to calibrate the features before temporal covariance pooling:
TC-COV x= (Ci)
W6 Ct, Wo +.. W C1.1+1 W1
(6)
intra-frame covariance pooling
inter-frame cross-covariance pooling
where C₁ X7 X,, and the features X can be adaptively calibrated by temporal canporal-based spatial attention SASA and a temporal remporal-based based channel attention fTCA, i.e.,
X (X @ fTSA ( X,, X:)) @ (X1 Xi), where @ and @ represent element-wise addition and element-wise multiplication, respectively. Both STSA and Srea are functions of X, and its two preceding frames (i.e., X1-2 and X1-1). Next, we will describe how to compute frSA and fTCA.
(7)
Computation of fISA Given X X and X we compute frSA as
BN (=000--(Xx-)))= where BN means batch normalization while a softmax operation. @ indicates matrix multiplication. dn. Φh and Φ are linear transformations implemented by convolutions. As shown in Eqn. (8). we extract spatial importance of individual features of X, by considering relationship among three consecutive frames based on the idea of attention (55). where X1-1, X and X₁ indicates queries, keys and values, respectively. In this way, we model long-range dependencies of spatial features while attending to the temporal relations based the inherent smoothness between adjacent frames.
STSA ( X,
@ (o(X))),
(8)
Computation of STOA We calibrate the feature channels of I-th frame by considering temporal difference. Specifically, we formulate fICA as
1
STC (X X,,, (GAP(XX - -x,))) + 7 g - -X,-2),
where GAP indicates global average pooling, and g is achieved by two FC layers followed by Sigmoid function.
After some arrangement, our TCP in Eqn. (6) can be efficiently implemented by a temporal convolu- tion of the calibrated features with kernel size of x 1 X K. followed by a covariance pooling:
PTCP L. (0xxxxxx(Xx)) [TCx
(10)
where TC1xIxI (X.) performs temporal convolution on X, with kernel size of 1 K. Finally, let us compare GAP, plain covariance pooling and our TCP from the view of stochastic process. Since the video signals are high- nigh-dimensional, spanning temporal and spatial dimensions, their distribution is very complex. GAP (resp. plain covariance pooling) only considers mean and variances of one single time, while our TCP models two different times, capturing temporal dynamics. In addition, as the features possess spatial structure, various features (either nearby or long-range features) may function differently in the interplay of different times, but GAP and plain covariance pooling neglect this interplay. In contrast, TCP can attend to this by using idea of attention.
5

---

3.2.3 Fast Matrix Power Normalization
Our TCP produces a covariance representation for each video. It is well known that the space of covariances is Riemannian manifold 42) which has geometrical structure. The appropriate use of geometry plays a key role for the effectiveness of covariance representations. Under deep architectures, recent works [31 60] show matrix power normalization approximatively yet effectively utilizes geometry of covariance space, and is superior to its counterparts (e.g., Eug-Eunlidenn metrics [37 and element-wise power normalization [25]). Here we exploit fast matrix power normalization method [30] for efficiency and effectiveness. Specifically, for each covariance representation PTCP output by our TCP, we compute its approximate matrix square roo as follows:
Iteration: Q = 10--1(31 R&- Qk 1): Rs = -
RR--QQ- -3I-R&-100
R) 1 (11)
2
where Qo = Prep and Ro = I is the identity matrix. After K' iterations, we have PTCP s QK-
Finally, Picp is used as video representation for classification, In this paper, K is set to 3 throughout all the experiments. Note that Eq. (TT) can be efficiently implemented on GPU due involving of only matrix multiplications, and more details can refer to 30
3.3 Teneoral-attentive Covariance Pooling Networks (TCPNet)
In this subsection, instantiate our TCP with existing deep video architectures, which yield powerful TCPNet for video recognition. As shown Figure | given any specific video architecture (e.g.. TSN (59] with 2D ResNet-50 22 or 3D CNNs (10), TCPNet inserts the proposed TCP after the last convolution layer, which replaces the original GAP to produce the final video representations for 1 X convolutions between the last convolution layer and our TCP, which reduce dimension of convolution features and control size of video representations. Our TCP is lightweight: taking 9256-dimensional TCP (default setting) as an example, parameters and FLOPs of our TCP are 3.3M and 1.2G for 8 frames of 224 x 224 inputs, respectively, Compared backbones of 2D ResNet-50 (24.3M and 33G) and 2D ResNet- 152 (59.0M and 91.5G), our TCP brings about extra 13.6% (5.6%) parameters and 3.6% (1.3%) FLOPs, respectively.
classification. Note that we introduce
4 Experiments
To evaluate our proposed method, we conduct experiments on six video benchmarks, including Mini- V1 (S-S VI) [19 Charades (66). UCF101 49 and HMDB51 23 We first describe implementation details of our
Kinetics-208 Mini-K200) [66], Kinetics-400 (K-400) (2]. Something
method, and then make ablation studies Mini- K200. Furthermore, we compare with counterparts and -fate-of-the-arts on K-400, and finally show generalization of our TCPNet on remaining datasets.
(GAA)
ParamateriGA)
Method
Top-1 (%)
Top-5 (%)
24.
Parameter (M)
GAP (Baseline)
71.9
90.6
Koc 190
Plain GCP
72.1
90.8
75.4
Plain GCP MPN
74.5
91.8
28-2
+fTCA
75.1
92.2
+ITSA
75.6
92.5
+TC-COV K=
76.1
92.7
66
:24
190
726
Feature Dimension
Table 1: Impact of various key components.
Figure 2: Effect of feature dimension.
Implementation Details. To construct our TCPNet, integrate the proposed TCP with several video architectures, including TSN (59] with 2D CNNs (i.e., ResNet-50 (22 and ResNet-155 152 (22)), X3D-
M [10) and 2D CNNs with temporal modeling (e.g.. TEA (33). ResNet-50 (R50) and ResNet-152
(R152) are pre-trained on ImageNet-I IK and ImageNet- 1 IK, respectively. As suggested in [3]] (0), discard the last downsamping operation when covariance representations are computed. Here we describe the settings of syper-parameters on K200 and K-400. For training our TCPNet
'The matrix square root is special case of matrix power normalization with the power of 1/2, usually achieving the best performance.

---

Method
Dim.
Top-1 1 (%)
Top-5 (%)
Method
Backbone
Input
Top-1 (%)
GAP
2048
71.9
90.6
TBN 34
2D RI8
112
69.5
BONN
16384
72.7
91.4
MARS
3D RX101
112
72.8
CBP 15
8256
72.5
91.5
V4D 70
3D R50
224
80.7
TLE (BCNN-A) 7
8256
72.2
91.0
BAT
2D R550
224
70.6
TLE (CBP-A) 7
8256
72.2
90.6
iSQRT (30
8256
74,5
91.8
RMS 28
3D Slow50
224
78.6
iSQRT-A
8256
72.0
90.5
CGNL 69
3D R101
224
79.5
iSQRT-B
8256
74.3
91.2
TCPNet (Ours)
2D R50
112
78.3
TCP (Ours)
8256
76.1
92.7
TCPNet (Ours)
2D R50
224
80.7
Table 2: Comparison of various global pooling methods on
Table 3: Comparison of state-of-the-arts on Mini-K200.
with 2D CNNs, we adopt the same data augmentation strategy as [59]. and number of segments is set to 8 or 16. A dropout with rate of 0.5 is used for the last FC layer. TCPNet is optimized by mini-batch stochastic gradient descent (SGD) with batch size of 96, a momentum of 0.9 and weight decay of e--4. The whole networks are trained within 50 epochs, where initial learning rate is 0.015 and decay by 0.1 every 20 epochs. For training our TCPNet with X3D-M, we process the images followed by [10]. and 16 frames sampled imputs. The SGD with cosine training strategy is used to optimize the network parameters within 100 epochs, and the initial learning rate set to 0.1. For training our TCPNet with TEA, first train the backbone of TEA (i.e. Res2Net50 [14]) with covariance pooling on ImageNet- -K, and then fine-tune tune TCPNet with TEA using the same settings as [3]. During inference, the strategies of crop x clip and crops x 10 clips used for ablation study and comparison with state-of- of-the-arts. The training settings for the remaining datasets will be specified in the following parts. All programs are implemented by Pytorch and run PC equipped with four NVIDIA Titan RTX GPUs.
4.1 Ablation Study on Mini-K200
For efficiency, we make ublation studies on Mini-K200 with input size of 112 x 112, and TSN with R50 used as backbone model. We first assess the effects of feature dimension and key components on our TCP, and then compare with different global pooling methods and state-of-the-arts.
Effeet of Feature Dimension As described in section 3.3 we introduce 1 x 1 x convolutions before our TCP to reduce dimension of convolution features, controlling the size of covariance representations. Here assess the effect of feature dimension on recognition accuracy. Note that we do not use temporal attention module and temporal pooling for climinating their effects and guaranteeing generality. Specifically, the feature dimension varies from 64 to 256, and the corresponding results are shown in Figure where GAP is compared as a baseline. From Figure we that global covariance pooling (GCP) consistently outperforms GAP. With increase of feature dimension, performance saturates at dimension of 128, but after that parameters sigmificantly increase. GCP with dimension of 128 is better than GAP by 2.6%, and brings extra 22 parameters. For efficiency and performance, we set feature dimension to 128 throughout the following experiments, producing 8256-dimensional covariance representations after matrix triangulation.
Impact of Key Components Our TCP involves several key components, including temporal attention module, temporal covariance pooling TC-COV, x (x = and к for 8 and 16 frame inputs) and matrix power normalization. To evaluate their effects, we give the results of our TCP under various configurations. As shown in Table [ plain covariance pooling obtains little improvement over the original GAP and matrix power normalization (MPN) brings about 2.4% gains. Note that Plain GCP + MPN is strong baseline established by our work. Although promising performance is achieved by this baseline, the temporal- based channel attention fIAA achieves 0.6% gains over plain covariance pooling with matrix power normalization, while temporal- further brings 0.5% gains. Furthermore, TC-COV, k=5 achieves 0.5% improvement by performing temporal covariance pooling. Our TCP finally obtains the best results (76.1% 17), outperforming GAP and plain covariance pooling by 4.2% and 4.0% respectively. These results show the effectiveness of each key component in TCP.
spatial attention frsA
7

---

(a) GAP vs. TCP (Ours) under various architectures.
(b) Comparison with various 3D CNNs.
Method
Pooling
Top-1
Top-5
Method
Backbone
Frx Views
Top-
Top-S
GAP
70.6
89.2
B3D [2]
3D Inc.
64xN/A
71.1
89.3
TSN
59 +R50
TCP
75.3 (1 4.7)
91.7 (12.5)
R(2+1)D
3D R34
16x10
72.0
90.0
GAP
75.9
92.1
Non-local 61
3D R101
32x30
76.0
92.1
TSN
59 +9152
TCP
78.3 21:4.4)
93.7 (t1.6)
ABM 71
3D Inc.
150x1
72.6
N/A
GAP
76.0
92.3
ConNet
R50
32x10
77.2
N/A
X3D-M
TCP
71.0 1.0)
92.7()0.4)
SlowFast
3D R50
64x
77.0
92.6
GAP
75.0
91.8
X3D 10
X3D-M
16x30
76.0
92.3
TEA 33
RR50
TCP
76.8 (11.8)
92.9(11.1)
TCPNet (Ours)
X3D-M
16x
77.0
92.7
(c) Comparison with various 2D CNNs (8 frames).
(d) Comparison with various 2D CNNs (16 frames).
Method
Backbone
Frx Views
Top-1
Top-5
Method
Backhone
Frx Views
Top-1
Top-5
TSN 33
2D R50
8x10
70.6
89.2
STM 27
2D R50
16x30
73.7
91.6
TPN 67
2D R50
8x10
73.5
N/A
TSM 35
2D R50
16x10
74.7
N/A
TEA
2D R50
8x30
75.0
91.8
TEA 1
2D R50
16x30
76.1
92.5
2D R50
8x30
75.8
N/A
TEINet
39
2D R50
16x30
76.2
92.5
GTA 2
2D R50
8x30
75.9
92.2
TAM 40
2D R50
16x
76.9
92.9
SmallBig
32
2D R50
8x30
76.3
92.5
bL-TAM 9
2D R50
24x9
73.5
91.2
TCPNet (Ours)
TEA R:50
8x 30
76.8
92.9
TCPNet (Ours)
TEA R50
16x 30
77.2
93.1
TCPNet (Ours)
TSN R152
8x 30
78,3
93.7
TCPNet (Ours)
TSN R152
16x 30
79.3
94.0
Table 4: Comparison (in %) of state-of-the-arts on K-400 dataset under various configurations.
Comparison of Various Global Pooling Methods To further verify the effectiveness of our TCP. we compare with eight kinds of global pooling methods, including GAP, BCNN 37,, CBP [5]. iSQRT 330. TLE (BCNN-A & CBP-A) 77, iSQRT-A and iSQRT-B. Specifically, compute GAP, } in temporal orderless manner. For BCNN-A, CBP-A and iSQRT-A, we first perform average pooling along temporal dimension to
BCNN, CBP and iSQRT by using all features X2..X.....
obtain X = Σn X; upon which we BCNN, CBP and iSQRT. For iSQRT-B, we first apply iSQRT to each frame X,, and then perform average pooling for all covariances. The results of all methods presented in Table 2 based which
draw the following conclusions. First, all GCP methods outperform GAP. showing that GCP can produce more powerful video representations. also be that iSQRT performs significantly better than BCNN and CBP, clearly suggesting that the advantage exploiting geometry of covariance spaces. Second, BCNN, CBP and iSQRT achieve better results than TLE and ISQRT-A/SSQRTA respectively; this suggests that the simple strategy to exploit temporal information 7 brings little gains. Finally, our TCP achieves the best results, which outperforms GAP and iSQRT by 4.2% and 1.6% in Top-1 accuracy, respectively.
Comparison with State-of-the-arts At the end of this part, we compare our TCPNet with several tfte-oof-the-arts on Mini-K200 -200. The compared results are given in Table 3 When 112x 112 images are used as inputs, our TCPNet outperforms TBN (34) and MARS [4] by 8.8% and 5.5%, respectively. When size of input images increases to 224x224, our TCPNet obtains 80. 80.7%
Top-1 accuracy, performing best among all competing methods. Notably, for backbone of 2D R50. TCPNet improves BAT (3] over 10. %. Finally, TCPNet with 2D backbone is superior to CGNL (69] and RMS [2]]
while obtaining the same result with V4D 77) note that all three latter exploit 3D backbone models.
4.2 Comparison with Sfate-of-the-arre on Kinetics-400
In this subsection, we compare our method with state-of-the-art on K-400 under various con- figurations. As shown Table (a), our TCP consistently outperforms the original GAP un- der video architectures of TSN, X3D-M and TEA, which span all types of popular video archi- tectures. Particularly, TCP achieves larger improvement if less temporal information learned by backbone models. Table A (b) compares different 3D CNNs, where our TCPNet achieves

---

Method
Backbone
Top-1 (%)
Method
Backbone
Fr
UCFI01
HMDB5I
TSN [59]
2D R50
19.7
TSN 35
R50
8
91.7
64.7
TSM
135
2D R50
45.6
TLE
2D Inc.
15
95.6
71.1
TPN 66
2D R50
49.0
TEN 34
2D R34
8
93.6
69.4
TSM NeXIVLAD 36
Inc.-Res
41.9
AVLAD!
18
VGG-16
25
92.7
66.9
TSM PPAC A1
Inc.-Res
43.7
PPAC 41
2D R152
20
94.9
69.8
GSM [5]
Inc.-V3
49.0
ECO 72
Inc,+3D RTS
8
91.7
65.6
TSN + TCP (Ours)
R50
38.3 (118.6)
TSM 35
2D R50
8
95.9
73.5
TSM TCP (Ours)
2D R50
46.9 (11.3)
TSM 35
95.2
72.0
TPN TCP (Ours)
2D R50
49.6 (10.6)
ECO 72
Inc,+3D R18
16
92.8
68.5
Table 5: Comparison on S-S.
TEA 33
2D R50
16
96.9
73.3
STM 27
2D R50
16
96.2
72.2
Method
Backbone
Frames
mAP (%)
STC 6
3D RX101
16
92.3
65.4
Non-local 61
3D RI01
128
39.5
ART 58
3D R18
94.3
70.9
FlowFast 65
3D R50-NL
64
38.0
B3D 2
3D Inc.
64
95.4
74.5
LFB 64
3D RIO1-NI.
42.5
ABM 7
3D Inc.
64
95.1
72.7
STRG 62
3D R101-NL
32
39.7
TCPNet (Ours)
TSN R50
8
95.1
72.5
TimeCeption 24
3D RI01
128
41.1
TCPNet (Ours)
TEA R50
96.4
76.7
SlowFast 11
3D RI01-NL
128
42.5
Table 7: Results (% in Top-1) on UCF101 and
TCPNet (Ours)
3D RIO1-NL
32
42.6
HMDB51. ERR-iimpeementaion. Σ: RGB++low.
Table Comparison on Charades.
AVLAD: ActionVLAD
comparable performance with state-of-the-arts. Particularly, our TCPNet shows similar performance with SlowFast and CorrNet, uses less input frames. As listed in T and Table TCPNet based on TEA with input of 8 frames outperforms other methods, when R50 is used. For input of 16 frames, TCPNet based on TEA superior to all the compared methods by a clear margin. When R152 is employed, TCPNet obtains wate-of-the-ar results for inputs of both 8 and 16 frames.
4 (d), our
4.3 Generalization to Other Benchmarks
We further verify the generalization ability our TCP on S-S VI, Charades, UCF101 and HMDB51. For S-S V1, integrate TCP into three video architectures, i.e., TSN [59], TSM B5 and TPN [67]. Specifically, we train our TCPNet based on TSN and TSM following the settings in [35 with 8 frames and test with 1 clip x crop. while TCPNet based on TPN is trained using the same settings as (67]. As shown in T our TCP brings 18.6%, 1.3% and 0.6% gains for TSN, TSM and TPN, respectively. Based on TSM backone,
TCP with frame inputs is superior NeXtVLAD and PPAC with 20 frame inputs by 5.0% and 3.2%, respectively. Besides, TCP outperforms GSM with backbone of Inception-V3 by 0.6%. For Charades dataset, fine-tune TCPNet with backbone of 3D R101-NL (61] pre-trained on K-400 using learning rate of 0.3 within 50 epochs. The results of different methods terms of mean Average Precision (mAP) are presented in Table 6 where our TCPNet obtains 42.6% in terms of mAP, outperforming all compared methods. On UCF101 and HMDB51, TCPNet is fine-tuned based on TSN and TEA with backbone of 2D R50 with initial learning rate of le-3 within 15 epochs. The results in top-1 accuracy are shown in Table 7 from it we can see that our TCPNet clearly outperforms TSN. Besides, TCPNet based on TSN superior to other encoding methods (i.e., TLE, TBN, Action VLAD and PPAC) by large margin. Lastly, our TCPNet with TEA performs competitively on UCF101 and achieves state-of-tte of-the--rt results on more challenging HMDB51. These results clearly show our TCP has good generalization ability.
Conclusion
In this paper, we propose a temporal-attentie covariance pooling for generating powerful video representations, which can be seamlessly integrated with existing deep models and result in an effective video recognition architecture (namely TCPNet). Extensive experimental comparisons using various deep video architectures demonstrate the effectiveness our TCPNet, and show our TCP is compelling alternative to GAP in popular deep architectures for effective video recognition. By considering complex temporal dynamics and geometry of covariances, our TCP is also superior to its covariance pooling counterparts by clear margin. In future, we will study more powerful global pooling methods (e.g., third-order pooling or mixture model) for more effective video signal modeling, and apply our proposed TCP to other deep architectures, c.g., vision Transformers g

---

Acknowledgements
Our work was supported by National Natural Science Foundation of China (Grant Numbers 6197 6087,086, 61806140, 61471082, 61872118, 61925602), Natural Science Foundation of Tianjin City (Grant No. 20ICONIC1530) and CCF-Baidu Open Fund (No. 2021PP150020000
References
[1] Vincent Arsigny, Pierre Fillard, Xavier Pennec, and Nicholas Ayache. Fast and simple calculus on tensors in the Bog-Euclidean framework. In MICCAI, 2005.
[2] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? A new model and the Kinetics dataset. In CVPR, 2017.
[3] Lu Chi, Zehuan Yuan, Yadong Mu, and Changhu Wang. Non-local neural networks with grouped bilinear attentional transforms. In CVPR, 2020.
[4] Nieves Crasto, Philippe Weinzaepfel, Karteek Alahari, and Cordelia Schmid. MARS: aattoo-augmented RGB stream for action recognition. In CVPR, 2019.
[5] Lieven De Lathauwer. Signal processing based multilinear algebra. Katholieke Universiteit Leuven,
1997.
[6] Ali Diba, Mohsen Fayyaz, Vivek Sharma, M. Mahdi Arzani, Rahman Yousefzadeh, Juergen Gall, and Luc Van Gool. Spatio-temporal channel correlation networks for action classification. In ECCV, 2018.
[7] Ali Diba, Vivek Sharma, and Luc Van Gool. Deep temporal linear encoding networks. In CVPR, 2017.
[8] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghami, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. In ICLR, 2021.
[9] Quanfu Fan, Chun-Fu Fu Richard Chen, Hilde Kuehne, Marco Pistoia, and David Cox. More less: Learning efficient vidco representations by big-little network and depthwise temporal aggregation. In NeurlPS, 2019.
[10] Christoph Feichtenhofer. X3D: Expanding architectures for efficient video recognition. In CVPR, 2020.
[11] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In ICCV, 2019.
[12] Christoph Feichtenhofer, Axel Pinz, and Andrew Zisserman. Convolutional two-stream network fusion for video action recognition. In CVPR, 2016.
[13] Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, and Marcus Rohrbach. Multimodal compact bilinear pooling for visual question answering and visual grounding. In EMNLP, 2016.
[14] Shanghua Gao, Ming-Ming Cheng, Kai Zhao, Xin-Yo Yu Zhang, Ming-Hsuan Yang, and Philip HS Torr. Res2Net: A new multi-scale backbone architecture. IEEE TPAMI, 43(6):652-662, 2021.
[15] Yang Gao, Oscar Beijbom, Ning Zhang, and Trevor Darrell. Compact bilinear pooling. In CVPR, 2016.
[16] Zilin Gao, Jiangtao Xie, Qilong Wang, and Peihua Li. Global second-order pooling convolutional networks. In CVPR, 2019.
[17] Rohit Girdhar and Deva Ramanan. Attentional pooling for action recognition. In NeurIPS, 2018.
[18] Rohit Girdhar, Deva Ramanan, Abhinav Gupta, Josef Sivic, and Bryan Russell, ActionVLAAD Learning spatio-temporal aggregation for action classification. In CVPR, 2017.
[19] Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller Freitag. Florian Hoppe, Christian Thurau, Ingo Bax, and Roland Memisevic. The "something something" video database for learning and evaluating visual common sense. In ICCV, 2017.
[20] Mehrtash Tafazzoli Harandi, Mathieu Salzmann, and Richard I. Hartley. Dimensionality reduction on SPD manifolds: The emergence of geometry-aware methods. IEEE TPAMI, 40(1):48-62, 2018
[21] Bo He, Xitong Yang. Zuxuanu Wu, Hao Chen, Ser-Nam Lim, and Abhinav Shrivastava. GTA: Global temporal attention for video action understanding, arXiv preprint arXiv:2012.08510, 2020.
[22] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[23] Email Hilde, JhuanRainer Hueihan, hagen Stiefel, and Serre Thomas. HMDB: A large video database for human motion recognition. In /CCV, 2011.
[24] N. Hussein, E. Gavves, and A. W. M. Smeulders. Timeception for complex action recognition. In CVPR, 2019.
10

---

[25] Catalin Ionescu. Orestis Vantzos, and Cristian Sminchisescu. Matrix backpropagation for deep networks with structured layers. In ICCV, 2015.
[26] Hervé Jégou, Florent Perronnin, Matthijs Douze, Jorge Sánchez, Patrick Pérez, and Cordelia Schmid. Aggregating local image descriptors into compact codes. IEEE TPAMI, 340)):704-11766, 2012.
[27] Boyuan Jiang, Mengmeng Wang. Weihao Gan, Wei Wu, and Junjie Yan. STM: Spatiotemporal and motion encoding for action recognition. In ICCV, 2019.
[28] Jinhyung Kim, Seunghwan Cha, Dongyoon Wee. Soonmin Bae, and Junmo Kim. Regularization on spatio-temporally smoothed feature for action recognition. In CVPR. 2020.
[29] Piotr Koniusz, Lei Wang, and Anoop Cherian. Tensor representations for action recognition. IEEE TPAMI, 2020.
[30] Peihua Li, Jiangtao Xie. Qilong Wang. and Zilin Gao. Towards faster training of global covariance pooling networks by iterative matrix square root normalization. In CVPR, 2018.
[31] Peihua Li, Jiangtao Xic, Qilong Wang, and Wangmeng Zuo. second-ord order information helpful for large-scale visual recognition? In ICCV, 2017.
[32] Xianhang Li, Yali Wang, Zhipeng Zhou, and Yu Qiao. SmallBigNet: Integrating core and contextual views for video classification. In CVPR, 2020.
[33] Yan Li, Bin Ji, Xintian Shi. Jiangue Zhang, Bin Kang, and Limin Wang. TEA: Temporal excitation and aggregation for action recognition. In CVPR, 2020.
[34] Yanghao Li, Sijie Song, Yuqi Li, and Jiaying Liu. Temporal bilincar networks for video action recognition. In AAAI, 2018.
[35] Ji I Lin, Chuang Gan, and Song Han. TSM: Temporal shift module for efficient video understanding. In ICCV, 2019.
[36] Rongcheng Lin, Jing Xiao, and Jianping Fan. NeXtVLAD: An efficient neural network to aggregate frame-level features for large-scale vidco classification. In ECCV Workshops, 2018.
[37] T. Lin, A. RoyChowdhury, and S. Maji. Bilinear CNN models for fine-grained visual recognition. In ICCV, 2015.
[38] Georg Lindgren, Holger Rootzén, and Maria Sandsten. Stationary stochastic processes for scientists and engineers. CRC press, 2013.
[39] Zhaoyang Liu, Donghao Luo, Yabiao Wang, Limin Wang, and Tong Lu, TEINet: Towards an efficient architecture for video recognition. In AAAI, 2020.
[40] Zhaoyang Liu, Limin Wang. Wayne Wu, Chen Qian, and Tong Lu. TAM: Temporal adaptive module for video recognition. In ICLR, 2021.
[41] Xiang Long, Gerard De Melo, Dongliang He, Fu Li, Zhizhen Chi. Shilei Wen, and Chuang Gan. Purely attention based local feature integration for video classification. IEEE TPAMI, 2020.
[42] Xavier Pennec, Pierre Fillard, and Nicholas Ayache. A Riemannian framework for tensor computing. IJC,, 66(1):41-66, 2006.
[43] Zhaofan Qiu, Ting Yao, and Tao Mei. Learning spatio-temporal representation with pseudo-3D residual networks. ICCV, 2017.
[44] Jorge Sánchez, Florent Perronnin, Thomas Mensink, and Jakob Verbeek. Image classification with the Fisher Vector: Theory and practice. IJV,, 105(3):222-245, 2013.
[45] Andres Sanin, Conrad Sanderson, Mehrtash Tafazzoli Harandi, and Brian C. Lovell. Spatio-temporal covariance descriptors for action and gesture recognition. In WACV, 2013.
[46] Gunnar Sigurdsson, G Varol. Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. Hollywood homes: Crowdsourcing data collection for activity understanding. In ECCV, 2018.
[47] Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recognition in videos. In NIPS, 2014.
[48] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. ICLR, 2015.
[49] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. UCF101: A dataset of 101 human actions classes from videos in the wild, arXiv preprint orXiv: 12220002, 2012.
[50] Swathikiran Sudhakaran, Sergio Escalera, and Oswald Lanz. Gate-shift networks for video action recogni- tion. In CVPR, 2020.
[51] Swathikiran Sudhakaran, Sergio Escalera, and Oswald Lanz. Gate-shift networks for video action recogni- tion. In CVPR. 2020.
[52] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR, 2015.
11

---

[53] Du Tran, Lubomir D. Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spatiotem- poral features with 3D convolutional networks. In /CCV, 2015.
[54] Tran, Heng Wang. Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri, A closer look spatiotemporal convolutions for action recognition. CVPR, 2018.
[55] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NewrlPS, 2017.
[56] Heng Wang. Du Tran, Lorenzo Torresani, and Matt Feiszli. Video modeling with correlation networks. In CVPR. 2020.
[57] Lei Wang. Piotr Koniusz, and Du Q. Huynh. Hallucinating IDT descriptors and I3D optical flow features for action recogmition with CNNs. In ICCV, 2019.
[58] Limin Wang. Wei Li, Wen Li, and Lue Van Gool. Appearance-and-rclttoon rnd-relation networks for video classification. In CVPR. 2018
[59]
Limin Wang. Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang. and Lue Van Gool. Temporal segment networks for action recognition videos. IEEE TPAMI, 41(11):2740 2755, 2019.
[60] Qilong Wang, Jiangtao Xic, Wangmeng Zuo, Lei Zhang, and Peihua Li. Deep CNNs meet global covariance pooling: Better representation and generalization, IEEE TPAMI, 2020.
[61]
Xiaolong Wang. Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In CVPR, 2018
[62]
Xiaolong Wang and Abhinav Gupta. Vidcos as space-time region graphs. In ECCV, 2018,
[63]
Thomas Winterbottom, Sarah Xiao, Alistair Mcl can, and Noura Al Moubayed. Trying hilinear pooling in
Video-QA. arXiv preprint arXiv:20/2.02285 2020.
[64]
Chao-Yuan Wu, Christoph Feichtenhofer, Haoqi Fan, Kaiming He, Philipp Krähenbühl, and Ross Girshick.
Long-term feature banks for detailed video understanding. In CVPR, 2019.
[65] Chao- Yuan Wu, Ross Girshick, Kaiming He, Christoph Feichtenhofer, and Philipp Krahenbuhl. A multigrid method for efficiently training video models. In CVPR, 2020.
[66] Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy. Rethinking spatiotemporal feature learning: Speed-accuracy trade-off's in video classification. In ECCV, 2018.
[67] Ceyuan Yang, Yinghao Xu, Jianping Shi, Bo Dai, and Bolei Zhou. Temporal pyramid network for action recognition. In CVPR, 2020.
[68] Zhou Yu, Jun Yu, Jianping Fan, and Dacheng Tao, Multi-modal factorized bilinear pooling with co-attention learning for visual question answering. In ICCV, 2017.
[69] Kaiyu Yue, Sun Ming. Yuchen Yuan, Feng Zhou, Errui Ding, and Fuxin Xu. Compact generalized non-local network. NeurIPS, 2018.
[70] Shiwen Zhang, Sheng Guo, Weilin Huang, Matthew R Scott, and Limin Wang. V4D: 4D convolutional neural networks for video-level representation learning. In ICLR, 2020.
[71] Xinqi Zhu, Chang Xu, Langwen Hui, Cewu Lu, and Dacheng Tao. Approximated bilinear modules for temporal modeling. In ICCV,
2019. 2019.
[72] Mohammadreza Zolfaghari, Kamaljeet Singh, and Thomas Brox. ECO: Efficient convolutional network for online video understanding. In ECCV, 2018.
12


Reviews:
[{"review_id": "p92WiQ_FTp", "reviewer": "Reviewer_SHTG", "summary": "This paper presents a temporal-attentive covariance pooling (TCP) method for action recognition, which consists of a temporal attention module, a temporal covariance pooling, and a fast matrix power normalization of covariance. TCPNet is then devised by inserting TCP at the top of existing deep architectures. The experiments are conducted on 6 action recognition benchmarks to evaluate the proposed model.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n++ The paper is clearly written and well organized.\n\n++ Extensive experiments are conducted on 6 benchmarks.\n\nWeaknesses:\n\n-- The technical contribution of this work is somewhat limited. The main idea of temporal-attentive covariance pooling seems to be a simple combination of self-attention mechanism and deep covariance pooling which have been widely used for action recognition [7, 17]. The combination of these techniques could be a contribution. But to me, the technical contribution is somewhat incremental for a NeurIPS paper.\n\n-- In Table 4(a), the authors only compared the proposed TCP to one simple pooling method (GAP). It is better to include more feature aggregation baselines (such as ActionVLAD[18] and TLE[7]) for performance comparison.\n\n-- In section 4, the authors should compare the proposed TCPNet with the state-of-the-art methods, such as [a] (81.0% on Kinetics-400, 53.3% on S-S V1) and [b] (50.6% on S-S V1).\n\n[a] Heng Wang, Du Tran, Lorenzo Torresani, and Matt Feiszli. \"Video modeling with correlation networks.\" In CVPR, 2020.\n\n[b] Swathikiran Sudhakaran, Sergio Escalera, and Oswald Lanz. \"Gate-shift networks for video action recognition.\" In CVPR, 2020.\n\n-- It is better to show some qualitative results of the learnt temporal attention.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I do not find any significant aspect about its limitation and potential negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8 hours"}, "extra_text": {}}, {"review_id": "48DUbD_TFe1", "reviewer": "Reviewer_zp2B", "summary": "This paper proposes temporal attentive covariance pooling module for video recognition. This module extracts seconder-order information from video feature tensors to improve the conventional global average pooling layer and it works in a plug-and-play fashion. Temporal-based channel attention and spatial attention are also designed to handle complex temporal dynamics. Extensive results are conducted to show effectiveness of the proposed module. ", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: This paper extends second-order pooling, which is originally developed for image recognition, to the field of video recognition. Instead of directly appling covariance pooling on per-frame basis, this paper proposes temporal-based spatial and channel attention mechanisms to model complex temporal dynamics. Besides, the temporal coviarance pooling scheme takes into acount both intra- and inter-frame covariances. From this point of view, the novelty of this paper is OK.\n\nQuality: Generally, the overall quality of this paper reaches or is slightly above the bar. The motivation of this paper is reasonable  and the performance of the proposed TCP is validated via extensive experiments.\n\nClarity: It is clear for readers to follow the main idea of this paper.  The draft is well organized and the presentation of this paper is good despite of the following small issues: \n1. In eq.5 and eq.6, in my opinion, the first part denotes intra-frame covariance and the second part denotes inter-frame covariance. The terms in the text are also misleading (e.g., line-170, C_{l,l} should be intra-frame covariance, C_{l,l-1} should be inter-frame covariance).\n2. There are works which perform local feature integration to generate a compact representation from video feature tensors, such as ActionVLAD [1], NeXtVLAD [2], AttentionClusters[3]. Having some discussions on them or making some comparisons with them is better.\n\n[1] Girdhar, Rohit, et al. \"Actionvlad: Learning spatio-temporal aggregation for action classification.\" Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017.\n\n[2] Lin, Rongcheng, Jing Xiao, and Jianping Fan. \"Nextvlad: An efficient neural network to aggregate frame-level features for large-scale video classification.\" Proceedings of the European Conference on Computer Vision (ECCV) Workshops. 2018.\n\n[3] Purely Attention Based Local Feature Integration for Video Classification\nX Long, G De Melo, D He, F Li, Z Chi, S Wen, C Gan - IEEE TPAMI2020\n\nSignificance:  The evaluation results show superiority of TCP, compared to various global pooling methods. The TCP module is also verified to be applicable for various 2D CNN, 3D CNN backbones. The proposed TCA and TSA modules are also shown to be helpful. But the evaluation part should be improved:\n1. Matrix Power Normalization is a n existing technique, therefore, in Table.1, the \"+Matrix Power Normalization\" is better moved to the upper part of this table and it can viewed as a stronger baseline compare to Plain GCP.  The Major contribution of this paper is TCA, TSA and TC-COV, which bring gain of 1.6% in total. \n2. The hyper-parameter k should be studied. \n3. Plain GCP + Matrix Power Normalization, which is a straightforward extension from existing state-of-the-art image covariance pooling, should better be compared in Table 4(a) to make the gain of the proposed TCP more clear.  The baseline of GAP in table 4(a) is kind of weak. \n\nOverall, I think this is a nice work and I am willing to raise the score if my concerns can be addressed in the rebuttal phase. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "None", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "48"}, "extra_text": {}}, {"review_id": "36Ug6pHww2q", "reviewer": "Reviewer_qjEJ", "summary": "The paper proposes a second-order pooling method, called temporal-attentive covariance pooling (TCP), for summarizing temporal features and generating video representation. Specifically, TCP involves three components: 1) a temporal convolution before covariance pooling that introduces learnable parameters for temporal modeling; 2) a spatial and channel attention to further enhance the convolutional features; 3) matrix power normalization to utilize geometry of covariance space. Experiments on multiple video datasets show that TCP outperforms the standard global average pooling and simple covariance pooling.\n\n\nAfter author rebuttal:\nThe author responses address most of my major concerns: (1) Contribution/novelty: after carefully reviewing the paper, the review from other reviewers and the author's rebuttals, I agree to recognize the contribution of TCP for introducing a solid framework that successfully applies the second-order pooling method to boost video action recognition. (2) TSA design: although the method is not intuitive to me, the comparison results provided in the author rebuttal justify the effectiveness of the proposed design.\nIn all, I'll raise my rating to \"6: Marginally above the acceptance threshold\".", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "My major concern of this paper is its limited technical contribution -- most components of TCP are not novel and have been widely studied in prior work.\n\n1) TCP aims to improve the vanilla covariance pooling by introducing order-aware aggregation along temporal dimension. However, simply adding a temporal convolution before covariance pooling is not sufficient for achieving this goal. First, temporal convolution can only capture local temporal dependencies within a small sliding window. The video-level temporal aggregation is still unachievable because the covariance features are still pooled by simple average operation. Moreover, using temporal convolution to capture local temporal dependencies is not novel and has been widely used for video recognition.\n\n2) The proposed temporal-based spatial attention and channel attention are not novel as well. Both spatial attention [1] and channel attention [2] have been widely used for video recognition, and their combination is also presented in prior work, for example [3]. Moreover, the motivation of using $X_{l-2}, X_{l-1}$ as query and key for $X_l$ is not clear. $X_{l-2}$ may have very different data distribution with $X_{l}$, which makes the alignment of the generated attention map and the feature map $X_{l}$ inaccurate.\n\n3) According to Table 1, the contribution of matrix power normalization is the most significant in TCP. However, matrix power normalization is proposed and used for vision tasks in prior work and it is simply applied to TCP in this paper.\n\nMinor issue: The terms \"intra-frame\" and \"inter-frame\" in Section 3.2 is misleading or even incorrect. Correlation within the same frame $X_l^T X_l$ should be intra-frame covariance instead of inter-frame covariance, and vice versa.\n\nIn all, although the proposed TCP achieves superior accuracy compared with simple average pooling and covariance pooling, the novelty of TCP is limited and more careful designs to improve the temporal aggregation of covariance features are needed. In addition, considering the extra computation and parameter introduced in computing covariance pooling, spatial/channel attention and matrix power normalization, the performance gain of TCP is not that significant, especially for 3D networks that already consider local temporal dependencies.\n\n[1] Wang, Xiaolong, et al. \"Non-local neural networks.\"\n[2] Feichtenhofer, Christoph. \"X3d: Expanding architectures for efficient video recognition.\"\n[3] Woo, Sanghyun, et al. \"Cbam: Convolutional block attention module.\"\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper didn't discuss its limitations and potential negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}]

Author rebuttal:

Meta-review: All three reviewers provided a rating of "6: Marginally above the acceptance threshold" for this submission. Reviewers qjEJ and SHTG initially raised concerns about the technical novelty of the proposed temporal-attentive covariance pooling (TCP). However, the authors provided a response that convinced both reviewers that there are significant differences between the TCP and related mechanisms introduced in prior works. Reviewer qjEJ pointed out the somewhat unintuitive design of TSA but concurs that the approach is empirically effective. Finally, Reviewer zp2B requested a few additional studies (such the comparison to local feature aggregation methods and the ablation on k), which were presented in the response by the authors. The ACs agree with the recommendation of accepting the paper.
Decision: Accept (Poster)