Temporal-attentive Covariance Pooling Networks for Video Recognition

1 Introduction . Video recognition aims to automatically analyze the contents of videos ( e.g. , events and actions ) , and has a wide range of applications , including intelligent surveillance , multimedia retrieval and recommendation . The great progress of deep learning , especially convolutional neural networks ( CNNs ) [ 48 , 52 , 22 , 61 ] , remarkably improves performance of video recognition . Since videos in the wild involve complex dynamics caused by appearance changes and variation of visual tempo , powerful video representations potentially provide high performance of video recognition . However , most of the existing video recognition architectures based on 3D CNNs [ 53 , 43 , 2 , 11 , 54 , 10 ] or 2D CNNs [ 35 , 27 , 33 , 39 ] usually generate the final video representations by leveraging a global average pooling ( GAP ) . Such GAP simply computes the first-order statistics of convolution features in a temporal orderless manner , which discards richer statistical information inherent in spatio-temporal features and has limited ability to capture complex dynamics of videos . ∗Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . There are a few researches studying global high-order pooling for improving performance of video recognition [ 45 , 29 , 7 ] . Sanin et al . [ 45 ] propose spatio-temporal covariance descriptors as video representations , where covariance descriptors are built with hand-crafted features of candidate regions and a subset of the optimal descriptors are learned by LogitBoost classifiers . However , such a method is designed based on the idea of the classical shallow architectures , without making the most of the merits of deep learning paradigm . In [ 29 ] , two tensor-based feature representations ( namely SCK and DCK ) are introduced for action recognition , where multiple subsequences and CNN classifier scores are used to compute tensor representations while a support vector machine is trained for classification ( see Figures 2 & 4 and the third paragraph of Section 5.2 ) . Under end-to-end deep architectures , Diba et al . [ 7 ] propose a temporal linear encoding ( TLE ) method for video classification . Specifically , TLE adopts a bilinear pooling [ 37 ] or compact bilinear pooling [ 15 ] to aggregate the features output by 2D and 3D CNNs as the entire video representations . TLE shows better performance than the original fully-connected ( FC ) layer , but it fails to fully exploit temporal information and geometry of covariance representations , limiting effectiveness of covariance pooling for video recognition . Recently , there also exist evidences showing that global covariance ( secondorder ) pooling can significantly improve performance of deep CNNs in various image-related tasks , such as fine-grained visual recognition [ 37 , 15 ] , large-scale image classification [ 31 , 30 , 60 ] and visual question answering [ 13 , 68 ] . However , these methods aim to generate image representations , without considering important temporal information inherent in videos . Meanwhile , naive usage of image-related covariance pooling ( called plain covariance pooling in this work ) technique for video-related tasks can not make full use of the advantage of covariance pooling [ 63 ] . As discussed previously , though global covariance pooling shows the potential to improve recognition performance , aforementioned works suffer from the following drawbacks : ( 1 ) Shallow approach [ 45 ] fails to make the most of the merits of deep learning paradigm ; ( 2 ) Plain covariance pooling methods [ 37 , 15 , 31 , 30 , 60 ] as well as TLE [ 7 ] can not adequately model complex temporal dynamics , and in the meanwhile TLE considers no geometry of covariances . To overcome these issues , this paper proposes a Temporal-attentive Covariance Pooling ( TCP ) , which can be integrated into existing deep architectures to produce powerful video representations . Different from the plain covariance pooling methods which perform orderless aggregation along temporal dimension , our TCP presents a temporal pooling of attentive covariance representations to capture complex temporal dynamics . Specifically , as illustrated in Figure 1 , our TCP consists of a temporal attention module , a temporal covariance pooling , and a fast matrix power normalization of covariances . The temporal attention module is developed to adaptively calibrate spatio-temporal features for the subsequent covariance pooling . The covariance pooling of the calibrated features can be approximatively regarded as producing attentive covariance representations , having the ability to handle complex dynamics . Then , temporal pooling of the attentive covariance representations ( i.e. , temporal covariance pooling ) characterizes both intra-frame correlations and inter-frame cross-correlations of the calibrated features , which can fully harness the complex temporal information . The fast matrix power normalization [ 30 ] is finally introduced to exploit geometry of covariance spaces . Note that our TCP is model-agnostic and can be flexibly integrated into existing 2D or 3D CNNs in a plug-and-play manner , resulting in an effective video recognition architecture ( namely TCPNet ) . The overview of our TCPNet is shown in Figure 1 . To verify its effectiveness , extensive experiments are conducted on six video benchmarks ( i.e. , Mini-Kinetics-200 [ 66 ] , Kinetics-400 [ 2 ] , SomethingSomething V1 [ 19 ] , Charades [ 46 ] , UCF101 [ 49 ] and HMDB51 [ 23 ] ) using various deep architectures ( e.g. , TSN [ 59 ] , X3D [ 10 ] and TEA [ 33 ] ) . The contributions of our work are summarized as follows . ( 1 ) We propose an effective TCPNet for video recognition , where a model-agnostic Temporal-attentive Covariance Pooling ( TCP ) integrated at the end of existing deep architectures can produce powerful video representations . ( 2 ) To our best knowledge , our TCP makes the first attempt to characterize both intra-frame and inter-frame correlations of the calibrated spatio-temporal features in a temporal manner , which approximatively performs temporal pooling of the attentive covariance representations . The proposed TCP provides a compelling alternative to GAP for improving recognition performance of deep video architectures . ( 3 ) We perform a comprehensive study of global covariance pooling for deep video recognition architectures , and conduct extensive experiments on six challenging video benchmarks . The results show our TCPNet has strong generalization ability while performing favorably against state-of-the-arts . 2 Related Works . 2.1 Deep Architectures for Video Recognition . Video data usually consists of appearance information and temporal motion . Two-stream architectures [ 47 , 12 ] are specially developed for separately capturing the two kind of information . They obtain competitive performance , but suffer from huge computational cost due to offline computation of optical flow . Extending 2D convolutions to 3D ones , 3D CNNs achieve state-of-the-art performance by taking only RGB inputs [ 53 , 2 , 43 , 66 , 54 , 11 , 10 ] , which can simultaneously learn appearance and temporal features . Furthermore , a family of attention models based on idea of non-local [ 61 , 64 , 3 , 21 ] are proposed to further improve performance of 3D CNNs by adaptively calibrating spatio-temporal features . Recently , integration of temporal modeling with 2D CNNs achieves promising results in a more efficient way [ 35 , 27 , 9 , 50 , 33 , 39 , 40 ] , where 2D convolutions extract appearance features while temporal modules ( e.g. , channel shift , temporal difference , and temporal convolution ) are used to capture motion information . However , these video recognition architectures usually generate the final video representations by leveraging a global average pooling ( GAP ) , which has the limited ability to capture complex dynamics of videos . Some works exploit classical encoding methods ( i.e. , VLAD [ 26 ] and Fisher vector [ 44 ] ) to replace GAP for aggregating spatio-temporal features [ 18 , 57 ] . Different from aforementioned methods , we propose a temporal-attentive covariance pooling ( TCP ) to replace GAP for producing powerful video representations in deep architectures . 2.2 Deep Covariance ( Second-order ) Pooling . Global covariance ( second-order ) pooling has been successfully applied to various image-related tasks , such as image classification [ 37 , 25 , 15 , 31 , 30 , 16 , 60 ] and visual question answering [ 13 , 68 ] . These works show global covariance pooling is much better than the original GAP , and appropriate use of geometry of covariances plays a key role for achieving high performance [ 25 , 31 ] . However , these works have no ability to capture temporal information inherent in video data , and naive usage of them can not fully exploit advantages of covariance pooling [ 63 ] . For modeling videos based on deep secondorder pooling , TLE [ 7 ] makes an attempt to use bilinear pooling [ 37 ] and CBP [ 15 ] for aggregating spatio-temporal features in an end-to-end manner . However , this method makes no full use of temporal information and geometry of covariance representations . Besides , some researches [ 17 , 71 , 34 ] integrate local second-order statistics into deep architectures for improving performance of video recognition . Specifically , AP [ 17 ] introduces second-order statistics by designing an element-wise multiplication of multiple-branch based on rank-1 approximation . Sharing similar idea , ABM [ 71 ] and TBN [ 34 ] apply element-wise multiplication to various frames for capturing motion information . Different from above works , our proposed TCP performs a global temporal pooling of attentive covariance representations , which can handle complex temporal dynamics and exploit geometry of covariances . Besides , our TCPNet has the ability to fully harness the merits of deep learning paradigm , compared to shallow approach [ 45 ] . 3 Proposed Method . In this section , we first briefly introduce the plain covariance pooling for modeling videos . Then , we describe in detail our proposed temporal-attentive covariance pooling ( TCP ) , which consists of a temporal attention module , a temporal covariance pooling , and a fast matrix power normalization . Finally , we instantiate our TCP with integration of existing deep video architectures , resulting in temporal-attentive covariance pooling networks ( TCPNet ) . 3.1 Plain Covariance Pooling for Modeling Videos . For the existing video architectures , global average pooling ( GAP ) is widely used to generate the final video representations . Suppose we have a total of L feature tensors output by some deep architecture . Let H , W and C be spatial height , width and channel number of one feature tensor . We denote by Xl ∈ RN×C the matrix obtained from the l-th feature tensor where N = H ×W and each row xl , n is a C-dimensional feature . GAP computes the final video representation pGAP ∈ R1×C as pGAP = 1 LN L∑ l=1 N∑ n=1 xl , n , ( 1 ) where L indicates temporal resolution . As shown in Eqn . ( 1 ) , from statistical view , GAP estimates first-order moment while higher-order statistics containing richer information are discarded . Compared to GAP , covariance pooling can capture richer statistics by modeling correlation among the features . For video data , plain covariance pooling generates video representation PPCP ∈ RC×C by aggregating the features as PPCP = 1 LN L∑ l=1 N∑ n=1 xTl , nxl , n = 1 L L∑ l=1 1 N XTl Xl , ( 2 ) where T indicates the matrix transpose . Although PPCP in Eqn . ( 2 ) has better representation ability than pGAP , both of them perform orderless aggregation along temporal dimension based on simple average operation , lacking the ability to handle complex temporal dynamics . In addition , naive use of PPCP only obtains small gains ( see Section 4 ) as geometry of covariance spaces is neglected .

Summary: The paper proposes a second-order pooling method, called temporal-attentive covariance pooling (TCP), for summarizing temporal features and generating video representation. Specifically, TCP involves three components: 1) a temporal convolution before covariance pooling that introduces learnable parameters for temporal modeling; 2) a spatial and channel attention to further enhance the convolutional features; 3) matrix power normalization to utilize geometry of covariance space. Experiments on multiple video datasets show that TCP outperforms the standard global average pooling and simple covariance pooling.   After author rebuttal: The author responses address most of my major concerns: (1) Contribution/novelty: after carefully reviewing the paper, the review from other reviewers and the author's rebuttals, I agree to recognize the contribution of TCP for introducing a solid framework that successfully applies the second-order pooling method to boost video action recognition. (2) TSA design: although the method is not intuitive to me, the comparison results provided in the author rebuttal justify the effectiveness of the proposed design. In all, I'll raise my rating to "6: Marginally above the acceptance threshold".