Temporal-attentive Covariance Pooling Networks for Video Recognition

1 Introduction . Video recognition aims to automatically analyze the contents of videos ( e.g. , events and actions ) , and has a wide range of applications , including intelligent surveillance , multimedia retrieval and recommendation . The great progress of deep learning , especially convolutional neural networks ( CNNs ) [ 48 , 52 , 22 , 61 ] , remarkably improves performance of video recognition . Since videos in the wild involve complex dynamics caused by appearance changes and variation of visual tempo , powerful video representations potentially provide high performance of video recognition . However , most of the existing video recognition architectures based on 3D CNNs [ 53 , 43 , 2 , 11 , 54 , 10 ] or 2D CNNs [ 35 , 27 , 33 , 39 ] usually generate the final video representations by leveraging a global average pooling ( GAP ) . Such GAP simply computes the first-order statistics of convolution features in a temporal orderless manner , which discards richer statistical information inherent in spatio-temporal features and has limited ability to capture complex dynamics of videos . ∗Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . There are a few researches studying global high-order pooling for improving performance of video recognition [ 45 , 29 , 7 ] . Sanin et al . [ 45 ] propose spatio-temporal covariance descriptors as video representations , where covariance descriptors are built with hand-crafted features of candidate regions and a subset of the optimal descriptors are learned by LogitBoost classifiers . However , such a method is designed based on the idea of the classical shallow architectures , without making the most of the merits of deep learning paradigm . In [ 29 ] , two tensor-based feature representations ( namely SCK and DCK ) are introduced for action recognition , where multiple subsequences and CNN classifier scores are used to compute tensor representations while a support vector machine is trained for classification ( see Figures 2 & 4 and the third paragraph of Section 5.2 ) . Under end-to-end deep architectures , Diba et al . [ 7 ] propose a temporal linear encoding ( TLE ) method for video classification . Specifically , TLE adopts a bilinear pooling [ 37 ] or compact bilinear pooling [ 15 ] to aggregate the features output by 2D and 3D CNNs as the entire video representations . TLE shows better performance than the original fully-connected ( FC ) layer , but it fails to fully exploit temporal information and geometry of covariance representations , limiting effectiveness of covariance pooling for video recognition . Recently , there also exist evidences showing that global covariance ( secondorder ) pooling can significantly improve performance of deep CNNs in various image-related tasks , such as fine-grained visual recognition [ 37 , 15 ] , large-scale image classification [ 31 , 30 , 60 ] and visual question answering [ 13 , 68 ] . However , these methods aim to generate image representations , without considering important temporal information inherent in videos . Meanwhile , naive usage of image-related covariance pooling ( called plain covariance pooling in this work ) technique for video-related tasks can not make full use of the advantage of covariance pooling [ 63 ] . As discussed previously , though global covariance pooling shows the potential to improve recognition performance , aforementioned works suffer from the following drawbacks : ( 1 ) Shallow approach [ 45 ] fails to make the most of the merits of deep learning paradigm ; ( 2 ) Plain covariance pooling methods [ 37 , 15 , 31 , 30 , 60 ] as well as TLE [ 7 ] can not adequately model complex temporal dynamics , and in the meanwhile TLE considers no geometry of covariances . To overcome these issues , this paper proposes a Temporal-attentive Covariance Pooling ( TCP ) , which can be integrated into existing deep architectures to produce powerful video representations . Different from the plain covariance pooling methods which perform orderless aggregation along temporal dimension , our TCP presents a temporal pooling of attentive covariance representations to capture complex temporal dynamics . Specifically , as illustrated in Figure 1 , our TCP consists of a temporal attention module , a temporal covariance pooling , and a fast matrix power normalization of covariances . The temporal attention module is developed to adaptively calibrate spatio-temporal features for the subsequent covariance pooling . The covariance pooling of the calibrated features can be approximatively regarded as producing attentive covariance representations , having the ability to handle complex dynamics . Then , temporal pooling of the attentive covariance representations ( i.e. , temporal covariance pooling ) characterizes both intra-frame correlations and inter-frame cross-correlations of the calibrated features , which can fully harness the complex temporal information . The fast matrix power normalization [ 30 ] is finally introduced to exploit geometry of covariance spaces . Note that our TCP is model-agnostic and can be flexibly integrated into existing 2D or 3D CNNs in a plug-and-play manner , resulting in an effective video recognition architecture ( namely TCPNet ) . The overview of our TCPNet is shown in Figure 1 . To verify its effectiveness , extensive experiments are conducted on six video benchmarks ( i.e. , Mini-Kinetics-200 [ 66 ] , Kinetics-400 [ 2 ] , SomethingSomething V1 [ 19 ] , Charades [ 46 ] , UCF101 [ 49 ] and HMDB51 [ 23 ] ) using various deep architectures ( e.g. , TSN [ 59 ] , X3D [ 10 ] and TEA [ 33 ] ) . The contributions of our work are summarized as follows . ( 1 ) We propose an effective TCPNet for video recognition , where a model-agnostic Temporal-attentive Covariance Pooling ( TCP ) integrated at the end of existing deep architectures can produce powerful video representations . ( 2 ) To our best knowledge , our TCP makes the first attempt to characterize both intra-frame and inter-frame correlations of the calibrated spatio-temporal features in a temporal manner , which approximatively performs temporal pooling of the attentive covariance representations . The proposed TCP provides a compelling alternative to GAP for improving recognition performance of deep video architectures . ( 3 ) We perform a comprehensive study of global covariance pooling for deep video recognition architectures , and conduct extensive experiments on six challenging video benchmarks . The results show our TCPNet has strong generalization ability while performing favorably against state-of-the-arts . 2 Related Works . 2.1 Deep Architectures for Video Recognition . Video data usually consists of appearance information and temporal motion . Two-stream architectures [ 47 , 12 ] are specially developed for separately capturing the two kind of information . They obtain competitive performance , but suffer from huge computational cost due to offline computation of optical flow . Extending 2D convolutions to 3D ones , 3D CNNs achieve state-of-the-art performance by taking only RGB inputs [ 53 , 2 , 43 , 66 , 54 , 11 , 10 ] , which can simultaneously learn appearance and temporal features . Furthermore , a family of attention models based on idea of non-local [ 61 , 64 , 3 , 21 ] are proposed to further improve performance of 3D CNNs by adaptively calibrating spatio-temporal features . Recently , integration of temporal modeling with 2D CNNs achieves promising results in a more efficient way [ 35 , 27 , 9 , 50 , 33 , 39 , 40 ] , where 2D convolutions extract appearance features while temporal modules ( e.g. , channel shift , temporal difference , and temporal convolution ) are used to capture motion information . However , these video recognition architectures usually generate the final video representations by leveraging a global average pooling ( GAP ) , which has the limited ability to capture complex dynamics of videos . Some works exploit classical encoding methods ( i.e. , VLAD [ 26 ] and Fisher vector [ 44 ] ) to replace GAP for aggregating spatio-temporal features [ 18 , 57 ] . Different from aforementioned methods , we propose a temporal-attentive covariance pooling ( TCP ) to replace GAP for producing powerful video representations in deep architectures . 2.2 Deep Covariance ( Second-order ) Pooling . Global covariance ( second-order ) pooling has been successfully applied to various image-related tasks , such as image classification [ 37 , 25 , 15 , 31 , 30 , 16 , 60 ] and visual question answering [ 13 , 68 ] . These works show global covariance pooling is much better than the original GAP , and appropriate use of geometry of covariances plays a key role for achieving high performance [ 25 , 31 ] . However , these works have no ability to capture temporal information inherent in video data , and naive usage of them can not fully exploit advantages of covariance pooling [ 63 ] . For modeling videos based on deep secondorder pooling , TLE [ 7 ] makes an attempt to use bilinear pooling [ 37 ] and CBP [ 15 ] for aggregating spatio-temporal features in an end-to-end manner . However , this method makes no full use of temporal information and geometry of covariance representations . Besides , some researches [ 17 , 71 , 34 ] integrate local second-order statistics into deep architectures for improving performance of video recognition . Specifically , AP [ 17 ] introduces second-order statistics by designing an element-wise multiplication of multiple-branch based on rank-1 approximation . Sharing similar idea , ABM [ 71 ] and TBN [ 34 ] apply element-wise multiplication to various frames for capturing motion information . Different from above works , our proposed TCP performs a global temporal pooling of attentive covariance representations , which can handle complex temporal dynamics and exploit geometry of covariances . Besides , our TCPNet has the ability to fully harness the merits of deep learning paradigm , compared to shallow approach [ 45 ] . 3 Proposed Method . In this section , we first briefly introduce the plain covariance pooling for modeling videos . Then , we describe in detail our proposed temporal-attentive covariance pooling ( TCP ) , which consists of a temporal attention module , a temporal covariance pooling , and a fast matrix power normalization . Finally , we instantiate our TCP with integration of existing deep video architectures , resulting in temporal-attentive covariance pooling networks ( TCPNet ) . 3.1 Plain Covariance Pooling for Modeling Videos . For the existing video architectures , global average pooling ( GAP ) is widely used to generate the final video representations . Suppose we have a total of L feature tensors output by some deep architecture . Let H , W and C be spatial height , width and channel number of one feature tensor . We denote by Xl ∈ RN×C the matrix obtained from the l-th feature tensor where N = H ×W and each row xl , n is a C-dimensional feature . GAP computes the final video representation pGAP ∈ R1×C as pGAP = 1 LN L∑ l=1 N∑ n=1 xl , n , ( 1 ) where L indicates temporal resolution . As shown in Eqn . ( 1 ) , from statistical view , GAP estimates first-order moment while higher-order statistics containing richer information are discarded . Compared to GAP , covariance pooling can capture richer statistics by modeling correlation among the features . For video data , plain covariance pooling generates video representation PPCP ∈ RC×C by aggregating the features as PPCP = 1 LN L∑ l=1 N∑ n=1 xTl , nxl , n = 1 L L∑ l=1 1 N XTl Xl , ( 2 ) where T indicates the matrix transpose . Although PPCP in Eqn . ( 2 ) has better representation ability than pGAP , both of them perform orderless aggregation along temporal dimension based on simple average operation , lacking the ability to handle complex temporal dynamics . In addition , naive use of PPCP only obtains small gains ( see Section 4 ) as geometry of covariance spaces is neglected .

Summary: This paper proposes temporal attentive covariance pooling module for video recognition. This module extracts seconder-order information from video feature tensors to improve the conventional global average pooling layer and it works in a plug-and-play fashion. Temporal-based channel attention and spatial attention are also designed to handle complex temporal dynamics. Extensive results are conducted to show effectiveness of the proposed module.