Where and when to look? Spatial-temporal attention for action recognition in videos

1 INTRODUCTION . An important property of human perception is that one does not need to process a whole scene in its entirety at once . Instead , humans focus attention selectively on parts of the visual space to acquire information where and when it is needed , and combine information from different fixations over time to build up an internal representation of the scene ( Rensink , 2000 ) , which can then be used for interpretation or decision making . In computer vision and natural language processing , over the last couple of years , attention models have proved similarly important . Particularly for the tasks where interpretation or explanation requires only a small portion of the image or video . Examples include visual question answering ( Lu et al. , 2016 ; Xu & Saenko , 2016 ; Xiong et al. , 2016 ) , activity recognition ( Sharma et al. , 2015 ; Girdhar & Ramanan , 2017 ; Li et al. , 2018b ) , and natural machine translation ( Bahdanau et al. , 2015 ) . These models have also provided a level of interpretability , by visualizing regions selected or attended over for a particular task or decision . In particular , for video action classification , a proper attention model can help answer the question of where and when it needs to look at the image evidence to draw a classification decision . It intuitively explains which part the model attends to when making a particular decision , which is very helpful in real applications , e.g. , medical AI systems or self-driving cars . In this paper , we propose a novel spatio-temporal attention mechanism that is designed to address these challenges . Our attention mechanism is efficient , due to its space- and time- separability , and yet flexible enough to enable encoding of effective regularizers ( or priors ) . As such , our attention mechanism consists of spatial and temporal components shown in Fig . 1 . The spatial attention component , that attenuates frame-wise CNN image features , consists of the saliency mask ; regularized to be discriminative and spatially smooth . The temporal component consists of a uni-modal soft attention mechanism that aggregates information over the near-by attenuated frame features before passing it into Convolutional LSTM for class prediction . Contributions : In summary , the main contributions of this work are : ( 1 ) We introduce a simple yet effective spatial-temporal attention for video action recognition , which consists of the saliency mask for spatial attention learned by ConvNets and temporal attention learned by convolutional LSTM . ( 2 ) We introduce three different regularizers , two for spatial and one for temporal attention components , to improve performance and interpretability of our model ; ( 3 ) We demonstrate the efficacy of our model for video action recognition in three public datasets and explore the importance of our modeling choices through ablation experiments ; ( 4 ) Finally , we qualitatively and quantitatively show that our spatio-temporal attention is able to localize discriminative regions and important frames , despite being trained in a purely weakly-supervised manner with only classification labels . 2 RELATED WORK . 2.1 NETWORK INTERPRETATION . Various methods have been proposed to try to explain neural networks ( Zeiler & Fergus , 2014 ; Springenberg et al. , 2014 ; Mahendran & Vedaldi , 2016 ; Zhou et al. , 2016 ; Zhang et al. , 2016 ; Simonyan et al. , 2013 ; Ramprasaath et al. , 2016 ; Ribeiro et al. , 2016 ; 2018 ; Chang et al. , 2018 ) in various of ways , including visualizing the gradients , perturbing the inputs , and bridging relations with other well-studied systems . Visual attention is also one way that tries to explain which part of the image is responsible for the network ’ s decision ( Li et al. , 2018a ; Jetley et al. , 2018 ) . Besides the explanation , Li et al . ( 2018a ) build up an end-to-end model to provide supervision directly on these explanations , specifically network ’ s attention . 2.2 VISUAL ATTENTION FOR VIDEO ACTION RECOGNITION . For video action recognition , visualizing which part of the frame and which frame of the video sequence that the model was attending to provides valuable insight into the model ’ s behavior . Sharma et al . ( 2015 ) develop an attention-driven LSTM by highlighting important spatial locations for action recognition . Girdhar & Ramanan ( 2017 ) introduce an attention mechanism based on a derivation of bottom-up and top-down attention as low-rank approximations of bilinear pooling methods . However , these work only focus on the crucial spatial locations of each image , without considering temporal relations among different frames in a video sequence . To alleviate this shortcoming , visual attention is incorporated in the motion stream ( Wang et al. , 2016b ; Li et al. , 2018b ; Du et al. , 2018 ) . However , the motion stream only employs the optical flow frames generated from consequent frames , can not consider the long-term temporal relations among different frames in a video sequence . Moreover , motion stream needs additional optical flow frames as input , which imposes burden due to additional optical flow extraction , storage and computation and is especially severe for large datasets . Torabi & Sigal ( 2017 ) propose an attention based LSTM model to hightlight frames in videos , but spatial information is not used for temporal attention . An end-to-end spatial and temporal attention model is proposed in ( Song et al. , 2017 ) for human action recognition , but additional skeleton data is needed . 3 SPATIAL-TEMPORAL ATTENTION MECHANISM . Our overall model is an Recurrent Neural Network ( RNN ) that aggregates frame-based convolutional features across the video to make action predictions as shown in Fig . 1 . The convolutional features are attended over both spatially , in each frame , and subsequently temporally . Both attentions are soft , meaning that the effective final representation at time t of an RNN , used to make the prediction , is a spatio-temporally weighted aggregation of convolutional features across the video along with the past hidden state from t − 1 . The core novelty is the overall form of our attention mechanism and the additional terms of the loss function that induce sensible spatial and temporal attention priors . 3.1 CONVOLUTIONAL FRAME FEATURES . We use the last convolutional layer output extracted by ResNet50 or ResNet101 ( He et al. , 2016 ) , pretrained on the ImageNet ( Deng et al. , 2009 ) dataset and fine-tuned for the target dataset , as our frame feature representation . We acknowledge that more accurate feature extractors ( for instance , network with more parameters such as ResNet-152 or higher performance networks such as DenseNet ( Huang et al. , 2017 ) or SENet ( Hu et al. , 2018 ) ) and optical flow features will likely lead to better overall performance . Our primary purpose in this paper is to prove the efficacy of our spatial-temporal attention mechanism . Hence we kept the features relatively simple . 3.2 SPATIAL ATTENTION WITH IMPORTANCE MASK . We apply an importance mask Mi to the i-th image features Xi to obtain attended image features by element-wise multiplication : X̃i = Xi Mi , ( 1 ) for 1 ≤ i ≤ n. This operation attenuates certain regions of the feature map based on their estimated importance . Here we simply use three convolutional layers to learn the importance mask ( please refer to Appendix B.2 for network architecture details ) . Fig . 2 illustrates our spatial attention mechanism . However , if left uncontrolled , an arbitrarily structured mask could be learned , leading to possible overfitting . We posit that , in practice , it is often useful to attend to a few important larger regions ( e.g. , objects , elements of the scene ) . To induce this behavior , we encourage smoothness of the mask by introducing total variation loss on the spatial attention , as will be described in Sec . 3.4 . 3.3 TEMPORAL ATTENTION . Inspired by attention for neural machine translation ( Bahdanau et al. , 2015 ) , we introduce the temporal attention mechanism which generates energy for each attended frame X̃i at each time step t , eti = Φ ( Ht−1 , X̃i ) , ( 2 ) where Ht−1 represents the ConvLSTM hidden state at time t−1 that implicitly contains all previous information up to time step t− 1 , X̃i represents the i-th frame masked features . Φ = ΦH ( Ht−1 ) + ΦX ( X̃i ) , where ΦH and ΦX are feed-forward neural networks which are jointly trained with all other components of the proposed system . This temporal attention model directly computes soft attention weight for each frame at each time t as shown in Fig . 3 . It allows the gradient of the cost function to be backpropagated through . This gradient can be used to train the entire spatial-temporal attention model jointly . The importance weight wti for each frame is : wti = exp ( eti ) ∑n i=1 ( exp ( eti ) ) ( 3 ) for 1 ≤ i ≤ n , 1 ≤ t ≤ T . This importance weighting mechanism decides which frame of the video to pay attention to . The final feature map Yt to the ConvLSTM is a weighted sum of the feature from all of the frames as the ConvLSTM cell inputs : Yt = 1 n n∑ i=1 wtiX̃i ( 4 ) where X̃i denotes the i-th masked frame of each video , n represents the total number of frames for each video . For RNN , instead of using conventional LSTM ( Graves , 2013 ) , we use Convolutional LSTM ( ConvLSTM ) ( Shi et al. , 2015 ) instead . The drawback of conventional LSTM is its use of full connections in the input-to-state and state-to-state transitions in which no spatial information is encoded . In contrast , each input , cell output , hidden state , gate are 3D tensors whose last two dimensions are spatial dimensions which can preserve spatial information , which is more suitable for image inputs . We use the following initialization strategy for the ConvLSTM cell state and hidden state for faster convergence : C0 = gc ( 1 n n∑ i=1 X̃i ) , H0 = gh ( 1 n n∑ i=1 X̃i ) ( 5 ) where gc and gh are two layer convolutional networks with batch normalization ( Ioffe & Szegedy , 2015 ) . We calculate the average hidden states of ConvLSTM over time length T , H = 1 T T∑ i=1 Hi ( 6 ) and send it to a fully connected classification layer for the final video action classification . 3.4 LOSS FUNCTION . Considering the spatial and temporal nature of our video action recognition ; we would like to learn ( 1 ) a sensible attention mask for spatial attention , ( 2 ) reasonable importance weighting scores for different frames , and ( 3 ) improve the action recognition accuracy at the same time . Therefore , our loss function L : L = LCE + λTVLTV + λcontrastLcontrast + λunimodalLunimodal , ( 7 ) where LCE is the cross entropy loss for classification , LTV represents the total variation regularization ( Rudin et al. , 1992 ) ; Lcontrast represents the mask and background contrast regularizer ; and Lunimodal represents unimodality regularizer . λTV , λcontrast and λunimodal are the weights for corresponding regularizers . The total variation regularizationLTV of the learnable attention mask encourages spatial smoothness of the mask and is defined as : LTV = n∑ i=1 ∑ j , k |M j+1 , ki −M j , k i |+ ∑ j , k |M j , k+1i −M j , k i | ( 8 ) where Mi is the mask for the i-th frame , and M j , k i is entry at the ( j , k ) -th spatial location of the mask . Different from the total variation of the mask of using L2 loss in Dabkowski & Gal ( 2017 ) , we use L1 loss instead . The contrast regularization Lcontrast of learnable attention mask is to suppress the irrelevant information and highlight important information : Lcontrast = n∑ i=1 ( −1 2 Mi Bi + 1 2 Mi ( 1−Bi ) ) ( 9 ) where Bi = I { Mi > 0.5 } represents the binarized mask , I is the indicator function applied elementwise . The unimodality regularizer Lunimodal encourages the temporal attention weights to be unimodal , biasing against spurious temporal weights . This stems from our observation that in most cases only one activity would be present in the considered frame window , with possible irrelevant information on either or both sides . Here we use the log concave distribution to encourage the unimodal pattern of temporal attention weights : Lunimodal = T∑ t=1 n−1∑ i=2 max { 0 , wt , i−1wt , i+1 − w2t , i } ( 10 ) where T represents the ConvLSTM time sequence length and n is the number of frames for each video . More details on this log concave sequence please refer to Appendix A .

Summary: The paper propose an end-to-end technique that applies both spatial and temporal attention. The spatial attention is done by training a mask-filter, while the temporal-attention use a soft-attention mechanism.  In addition the authors propose several regularization terms  to directly improve attention. The evaluated datasets are action recognition datasets, such as HMDB51, UCF10, Moments in Time, THUMOS’14. The paper reports SOTA on all three datasets.