Feature Integration and Group Transformers for Action Proposal Generation

1 INTRODUCTION . Owing to the fast development of digital cameras and online video services , the rapid growth of video sequences encourages the research of video content analysis . The applications of interest include video summarization ( Yao et al. , 2015 ; 2016 ) , captioning ( Chen et al. , 2019a ; Chen & Jiang , 2019 ) , grounding ( Chen et al. , 2019b ) , and temporal action detection ( Gao et al. , 2019 ; Zhang et al. , 2019 ) . The temporal action detection task is an important topic related to several video content analysis methods , and it aims to detect the human-action instances within the untrimmed long video sequences . Like the image object detection task , the temporal action detection can be separated into a temporal action proposal generation ( TAPG ) stage and an action classification stage . Recent studies ( Escorcia et al. , 2016 ; Buch et al. , 2017b ; Lin et al. , 2018 ; Liu et al. , 2019 ; Lin et al. , 2019 ; 2020 ) demonstrate that the way to pursue the proposal quality clearly improves the performance of two-stage temporal action detectors . To this end , a temporal action proposal generator is demanded to use a limited number of proposals for capturing the ground-truth action instances in a high recall rate , hence reducing the burden of the succeeding action classification stage . One popular way to tackle the TAPG task is to generate the proposals via the estimations of boundary and actioness probabilities . The boundary probability is usually factorized as the starting and ending for an action instance . Rather than directly estimating the actioness boundary as the existing methods , we leverage the actioness estimation and the additional background estimation in a bi-directional temporal manner to co-estimate the action boundaries . The background means existing no actions . This sort of boundary estimation derived from the observation that the features for describing the long-time actioness/background are more consistent along the temporal dimension than the short-time starting/ending . Therefore , estimating the boundary with the actioness and background features allows us to estimate the proposal boundaries of much less false-positive , hence obtaining the high-quality proposal candidates for further scoring . In practice , we estimate an action starting boundary as the time of descending background with simultaneous ascending actioness . In contrast , an action ending boundary occurs with the ascending background with descending actioness . Figure 1 illustrates our action-boundary co-estimation mechanism . This paper introduces an effective temporal action proposal generator , i.e. , FITS , which aims to provide the action proposals , that precisely and exhaustively cover the human-action instances . By considering the two essential TAPG issues , namely the feature representation and the scoring mechanism , and the above-mentioned action-boundary co-estimation , our attention-based FITS model comprises Feature Integration ( FI ) module and Transformer-driven Scorers ( TS ) module for dealing with these considerations . Precisely , our FI module enhances the common TAPG two-stream features ( Simonyan & Zisserman , 2014 ; Wang et al. , 2016 ; Xiong et al. , 2016 ) by concerning the feature interaction . The previous TAPG methods usually directly concatenate the appearance stream and motion stream features for usage . In contrast , we were inspired by the non-local attention mechanisms ( Wang et al. , 2018 ; Hsieh et al. , 2019 ) to extend such a long-range attention mechanism for integrating the two-stream features . As a result , our experiments show the robustness of the integrated features by reducing their mutual feature discrepancies . More importantly , to score the temporal action proposals for discriminating high-quality ones , we devise a novel transformer-driven scoring mechanism . The TS mechanism leverages the temporal contextual supports over the feature representations to obtain the self-attended representations and then associates these self-attended representations to co-estimate the action boundaries . The experiments show the retrieved action proposals containing much less false-positive ones . Figure 2 overviews our temporal action proposal model , termed as FITS network . To sum up , our main contributions include i ) We introduce the novel feature integration module to integrate the two-stream features by reducing their feature discrepancies via non-local-style attention and obtaining robust representation . ii ) We devise the novel transformer-driven scorers module to co-estimate the transformer-driven self-attended representations , which leverage long-range temporal contextual supports . Hence , we are able to retrieve high-quality temporal action proposals . iii ) The extensive experiments demonstrate that the proposed FITS model achieves significantly better performance than current state-of-the-art TAPG methods . 2 RELATED WORK . Feature Representation . As a de facto trend , instead of using the handcrafted features , the neuralnetwork-based features are widely employed for addressing the action classification task . These popular neural network approaches include the two-stream networks ( Simonyan & Zisserman , 2014 ; Feichtenhofer et al. , 2016 ; Wang et al. , 2016 ) , which separately represent the appearance feature and the motion feature , and 3D networks ( Tran et al. , 2015 ; Carreira & Zisserman , 2017 ; Qiu et al. , 2017 ; Xu et al. , 2017 ) , which directly represent a video sequence as the spatio-temporal feature . In this paper , we use the action recognition model ( Wang et al. , 2016 ; Xiong et al. , 2016 ) to extract two-stream features for representing each untrimmed video sequence . Attention Mechanism . The attention mechanism is the process of selectively focusing on a few relevant things in comparison with everything . Fields like natural language processing and computer vision , broadly leverage such an attention mechanism . For instances , Bahdanau et al . ( 2015 ) enable their model to focus on searching a group of related words from the input sentences for predicting the target words , Xu et al . ( 2015 ) introduce the soft and hard attention to generate image captions , and LFB ( Wu et al. , 2019 ) introduces long-term feature banks to analyze videos , Hsieh et al . ( 2019 ) employ non-local attention ( Wang et al. , 2018 ) to capture long-range dependencies and channel attention ( Hu et al. , 2018 ) to re-weight the channel-level feature maps . Since Vaswani et al . ( 2017 ) introduce a self-attention mechanism , called Transformer , for exploring the intra-attention and interattention to address the machine translation task , the Transformer-based models show their advantages to tackle various tasks , such as object detection ( Carion et al. , 2020 ) , action recognition ( Girdhar et al. , 2019 ) , image generation ( Parmar et al. , 2018 ) , and image captioning ( Cornia et al. , 2020 ) . We notice that the work of Girdhar et al . ( 2019 ) that classifies the human actions by first use the RPN ( Ren et al. , 2015 ) to localize the human body parts as several frame crops , and then encodes these crops via Transformers for the subsequent classification . In contrast , a TAPG task discriminates whether a given video segment , composed of several video frames , covering any action event . Our model extends the merits of these attention efforts for constructing a robust action proposal generator . Temporal Action Proposal Generation . We categorize the TAPG methods into Anchor-based ( Gao et al. , 2017 ; Heilbron et al. , 2016 ; Shou et al. , 2016 ) and Probability-based ( Lin et al. , 2020 ; 2019 ; 2018 ; Zhao et al. , 2017 ) . The former focuses on designing several multi-scale anchor boxes to cover action instances , while the latter estimates the temporal location probabilities of the action instances . Besides , some methods ( Gao et al. , 2018 ; Liu et al. , 2019 ; Gao et al. , 2020 ) also explore the way to integrate the above-mentioned two categories for precisely localizing the temporal boundaries . In anchor-based methods , the S-CNN ( Shou et al. , 2016 ) and Heilbron et al . ( 2016 ) respectively evaluate anchors via C3D network and sparse learning , and TURN ( Gao et al. , 2017 ) suggests regressing the temporal boundaries of action instances . The probability-based work , TAG ( Zhao et al. , 2017 ) , generates action proposals via a temporal watershed algorithm to merge contiguous temporal locations of high actioness probabilities . BSN ( Lin et al. , 2018 ) generates proposals as well as their confidence by formulating the probabilities of boundaries and actioness . BMN ( Lin et al. , 2019 ) proposes a boundary-matching mechanism to evaluate the confidence among densely distributed proposals . DBG ( Lin et al. , 2020 ) uses the maps of dense boundary confidence and completeness to further score boundaries for all action proposals . BC-GNN ( Bai et al. , 2020 ) employs a Graph Neural Network to model the relationship between the actioness and boundaries . Zhao et al . ( 2020 ) propose loss terms to regularize the feature consistency between the actioness and boundaries . In sum , the anchor-based methods focus on the anchor-box design and usually lack the flexible temporal boundaries for various action instances . The existing probability-based methods may generate the action proposals , but merely rely on the actioness ( Zhao et al. , 2017 ) or the boundary information ( Lin et al. , 2019 ) , or lack the association between the actioness and the boundaries of action instances ( Lin et al. , 2020 ; 2018 ) . By contrast , we introduce the Transformer-driven scorers module to score proposals by explicitly associating the self-attended representations during scorers training . Combining with our feature integration module for retrieving the robust feature representation , the experiments show that our model outperforms the existing leading TAPG methods . 3 METHOD . TAPG Formulation . Given a video sequence X = { xn } Nn=1 of N frames , we assume it includes I action instances . The TAPG task aims at generating a proper set of action proposals that can be used to detect the underlying human actions in X . We denote an action proposal as p = ( s , e ) to indicate that the proposal p starts at the sth frame and ends at the eth frame of X . Analogous to the object proposals for detection , action proposals are generic and class agnostic . Let the I ground-truth actions of X be Y = { ( s∗ , e∗ ) i = ( s∗i , e∗i ) } Ii=1 . An action proposal ( s , e ) is said to be matched to some ground-truth action ( s∗ , e∗ ) if their time-interval IoU ( in terms of frames ) is greater than a specified threshold τ . Considering a set P of proposals , its goodness to X can be explicitly measured by the number of matched action proposals . 3.1 FEATURE INTEGRATION . We decompose each video sequence X into a set of , say , T consecutive video segments , denoted as V = { vt } Tt=1 , where v is called a snippet . Each snippet v is then represented with two-stream features . To account for videos of various lengths , we adopt the setting in BSN ( Lin et al. , 2018 ) to sample single-stream features over the temporal ( frame ) dimension to consistently obtain T snippets per video sequence . Precisely , each video sequence X is represented by an appearance feature tensor A ∈ RC×T and a motion feature tensor M ∈ RC×T . To enhance the representation power of each snippet , we enrich the representation by three-step operations : co-attention , mutual-excitation , and aggregation . Before we describe the steps , we first define a basic convolutional layer φ by φ ( X ; f , o ) = ReLU ( W ∗X + b ) , ( 1 ) where X , f , o respectively denotes the input feature , the filter size , and the number of the output filters . ReLU , ∗ , W , b means the activation , convolution operation , the weights and bias of φ. Co-attention . We consider the non-local function ( Hsieh et al. , 2019 ; Wang et al. , 2018 ) to explore the snippet-level correlations between the two-stream cues . Hence we enable our model to learn to emphasize the temporal correlations of human-action descriptions between the two-stream cues . Given a video sequence X , its appearance tensor A ∈ RC×T and motion tensor M ∈ RC×T , the proposed two-stream co-attention is achieved by carrying out the following feature re-adjustment : Â = Φ ( A ; M ) ⊕A , M̂ = Φ ( M ; A ) ⊕M , ( 2 ) where Â , M̂ ∈ RC×T , the non-local function Φ ( · ) yields T × T snippet-level feature correlations for feature adjustment that conditioning on the other feature , and ⊕ denotes the element-wise addition . Mutual-excitation . We further enhance the two-stream features by re-weighting over the channel dimension C. To this end , we introduce a mutual-excitation mechanism to re-weight each singlestream feature concerning the channel-attention from the other stream . A convolution layer φ of filter size 1× 1 over the channel dimension is applied to re-organize the two-stream features . With the adjusted features , we construct a two-stream mutual-excitation function Ψ to extract the channel attention vector from one stream to excite the feature of the other stream . That is , we have Ã = Ψ ( φ ( M̂ ; 1× 1 , C ) ) φ ( Â ; 1× 1 , C ) , M̃ = Ψ ( φ ( Â ; 1× 1 , C ) ) φ ( M̂ ; 1× 1 , C ) , ( 3 ) where Ã , M̃ ∈ RC×T , the mutual-excitation function Ψ ( · ) is generalized from squeeze-and-excitation ( Hu et al. , 2018 ) for describing channel-attention , and symbolizes the element-wise multiplication . Aggregation . Inspired by the inception block ( Szegedy et al. , 2015 ; 2016 ) , we further enrich the feature representation concerning the multi-scale temporal contexts before integrating the two-stream features . Given the two-stream features Ã and M̃ , we employ the convolution layer and max-pooling layer to respectively map each of the two-stream features in two different temporal contexts by Ā1 = φ ( Ã ; 1×3 , C′ ) , Ā2 = φ ( ρ ( Ã ) ; 1×3 , C′ ) , M̄1 = φ ( M̃ ; 1×3 , C′ ) , M̄2 = φ ( ρ ( M̃ ) ; 1×3 , C′ ) , ( 4 ) where Ā1 , Ā2 , M̄1 , M̄2 ∈ RC ′×T , the max-pooling function ρ ( · ) using the filter of size 1× 3 . We then concatenate all features followed by another convolution layer to unify them . Formally , the aggregated feature , say F , is generated by F = φ ( Ā1 ‖ Ā2 ‖ M̄1 ‖ M̄2 ; 1× 3× 4 , C ′′ ) , ( 5 ) where the notation ‖ symbolizes the concatenation over an augmented last dimension , namely , ( Ā1 ‖ Ā2 ‖ M̄1 ‖ M̄2 ) ∈ RC ′×T×4 , and F ∈ RC′′×T is squeezed over the augmented last dimension .

Summary: This paper tackles the problem of temporal action proposal generation (TAPG). The authors address the problem from two perspectives: features wise and score fusion wise. They use non-local blocks to integrate appearance features and motion features together. For score fusion, they propose transformer based module to incorporate long range temporal information. The proposed method is evaluated on two benchmark datasets and achieved state-of-the art performance.