Human Action Recognition Based on Spatial-Temporal Attention

1 INTRODUCTION . Human action recognition is focused by more and more people in recent years , researchers have developed many different methods of human action recognition . For example , there are some models based on CNN ( Carreira & Zisserman , 2017 ) , independent subspace analysis ( Le et al. , 2011 ) , Restricted Boltzmann machine ( RBM ) ( Nie et al. , 2015 ) and RNN ( Zhu et al. , 2016 ) . Some of them achieve pretty great performances in human action recognition.The methods in human action recognition can be split into two main categories . One is the traditional machine learning , another is the deep learning . As for traditional machine learning , the improved Dense Trajectories ( iDT ) ( Wang & Schmid , 2013 ) method performs pretty well before the rapid development of deep learning . Because of the low speed of the iDT , there are some improved methods based on iDT . Stacked Fisher Vectors ( SVF ) ( Peng et al. , 2014 ) is thought as one of the best improvement of IDT . Actually , many new deep learning methods of action recognition would use iDT as one part of their networks to optimize their models . As for deep learning , there are lots of methods proposed recently which are classical and useful . There are three main deep learning methods which are based on CNN , RNN or both of them . With respect to CNN , in order to remind the temporal messages in videos , 3D-CNN ( Ji et al. , 2013 ) has been obtained good performance which adds the temporal information to CNN . Long-term Temporal Convolutions networks ( Varol et al. , 2018 ) is demonstrated with increased temporal extents , it improves the performance in action recognition . Two-stream CNN ( Feichtenhofer et al. , 2016b ) is made up of two deep networks , one is used as temporal networks and another is used as spatial networks . Pose-based CNN ( Chéron et al. , 2015 ) uses RGB frames and optical flow to recognize actions . Two-stream Inflated 3D ConvNet ( I3D ) ( Carreira & Zisserman , 2017 ) gives a state-of-art performance in human action recognition , it uses a more advanced architecture , the model is trained on bigger data sets . Based on I3D model , Choutas et al . ( 2018 ) proposes a new methods which is potion and motion representation , this improves the performance when they combine it with I3D . As the development of CNN on human action recognition , many more complex and efficient architectures have been come up with . The pure RNN-based models are usually used on skeleton data . This kind of data do not need CNN to extract features , just because the human in these videos is usually abstracted to the skeleton . There are many methods based on RNN for action recognition in skeleton data . Hierarchical Recurrent Neural Network ( Du et al. , 2015 ) divides human skeleton into five parts , then it uses RNN as the main networks in action recognition . Zhu et al . ( 2016 ) use Regularized Deep LSTM Networks to learn co-occurrence features of skeleton data . Part-Aware LSTM ( Shahroudy et al. , 2016 ) also divides human body joints into five groups in skeleton videos . Song et al . ( 2017 ) apply spatial and temporal attention mechanism in those skeleton videos , and the spatial attention on their models can be thought as the weight of human joints . They propose a method to calculate the temporal attention in skeleton data sets , and in this paper we will use it as the temporal attention of our model . Other popular method uses RNNs as the main networks in RGB videos . However , as we all know it , CNN has achieved great performance on human action recognition both on images and videos ( Carreira & Zisserman , 2017 ) ( Feichtenhofer et al. , 2016a ) , because it has great capability in dealing with images . So in this kind of model , CNNs are used to preprocess the video clips by extracting the convolutional features . The Long-term recurrent convolutional networks model ( Donahue et al. , 2015 ) is a classical architecture , it combines RNNs and CNNs organically . Based on CNN and RNN , Sharma et al . ( 2016 ) propounds a better model with visual attention mechanism , this model can focus on the important parts in each frame , it improves the accuracy of recognition . LatticeLSTM ( Sun et al. , 2017 ) is put forward for action recognition by learning independent hidden state transitions of memory cells for individual spatial locations . Although RNN is good at sequences , researchers think that ’ s not enough . So attention mechanism ( Vaswani et al. , 2017 ) is proposed , it can improve the performance of RNN networks.The attention in videos includes spatial attention and temporal attention . The spatial attention in each frame shows the saliency of every part , but it ’ s not very easy to generate attention automatically in images . Inspired by the success of attention in machine translation , Xu et al . ( 2015 ) come up with the attention mechanism including soft attention and hard attention in single still image , then their model generates the caption of the image by using soft or hard attention . But their attention mechanism still only works in still images until Sharma et al . ( 2016 ) apply their soft attention mechanism to human action recognition in videos . In consideration of many useless frames that have nothing to do with the action in a video , it ’ s really necessary to select the key frames and clean the useless frames out . Then computers will be able to spend less resources and get better results . Temporal Segment Networks ( Wang et al. , 2016 ) combines a sparse temporal sampling strategy and video-level supervision , this model is efficient and effective . The temporal attention mechanism in skeleton data ( Song et al. , 2017 ) pays different levels of attention to the outputs of different frames . In this paper , for the purpose that our model can focus on the important part in the frame and pay more attention to more relevant frames in videos , based on the soft attention mechanism ( Xu et al. , 2015 ) ( Sharma et al. , 2016 ) , we combine the temporal attention mechanism ( Song et al. , 2017 ) which has used in skeleton videos into our models . By using LSTM and spatial-temporal attention mechanism , we obtain satisfactory results in UCF11 , UCF101 , and HMDB51 . In detail , in this paper , we choose Vgg19 ( Simonyan & Zisserman , 2014 ) which is pretrained on the ImageNet ( Deng et al. , 2009 ) to extract the features in video frames , then we use LSTM as our models ’ main networks . The architecture of our model is illustrated in Fig . 1 . It shows the progress of how our model dealing with the data . In the summary , we make three contributions in this paper as follows : • We propose a model with spatial and temporal attention in human action recognition in RGB videos . In our model , the attention of two dimensionality is combined with each other successfully . • Compare with the baseline , we achieve the improvement of accuracy , which is about 7 % in UCF11 and 15 % HMDB51 . Besides , our results is satisfactory when compared with other methods . In our paper , in section 2 , we will expound our method in detail . Then we plan to show the details of the experiment and also the results in section 3 . In the end of this paper , we make a conclusion of our model in section 4 . 2 SPATIAL-TEMPORAL ATTENTION MECHANISM . In this section , we will demonstrate our model in detail . First we are going to introduce how our model encode the videos . Then we will present how our model figure out the spatial attention in each frame and select the key frames in the videos , later we will show the progress of the decoder and the production of results . Finally we plan to introduce the loss function chosen in our model . 2.1 ENCODER OF THE MODEL . If we put the RGB video frames into the LSTM straightly , we would not get a satisfactory results . So we decide to encode these frames by using CNN like many other models . As showed in Fig . 2 , in our model , we use the Vgg19 network pre-trained in Imagenet to extract the features in each video frame . After this step , we will get the feature cube which has the shape L×L×D ( 14× 14× 512 ) , for the purpose of easier calculation in next step , we flat the feature cube from 3D to 2D of shape L2 ×D ( 196× 512 ) . The flat feature cube can be expressed as follows , Xt = [ xt,1 , xt,2 , ... , xt , i , ... ] ( 1 ) where the t denotes the time which means the frame number in the videos , for example , if we extract twenty images in each video evenly , that means t should be in the range of 1 to 20 . The i means the localization in each single image which has been divided into 14×14 parts , so the i should be in the range of 1 to 196 . 2.2 SPATIAL ATTENTION MECHANISM . The overall architecture is showed by Fig.3 , in the left of the figure , α = { α1 , α2 , ... , αt , ... } donates the attention in the frames in each time t ( the range of t is 1 to T ) , so αt is a matrix of shape 196 which is L2 . In this paper , we adopt the soft attention mechanism that is demonstrated by Xu et al . ( 2015 ) and Sharma et al . ( 2016 ) , which is shown in Fig . 4 . The soft attention mechanism can be formulated as follows , et , i =W T h × ht−1 +Wx ×Xt + b αt , i = exp ( et , i ) ∑L2 j=1 exp ( et , j ) ( 2 ) where the ht−1 is the hidden state of LSTM in previous time t− 1 , and the Xt is the feature matrix in present frame t. After the et , i is calculated , the model will use the softmax function to compute αt , i , it is the attention weight in the frame t and the ith parts of the frame . αt , i is the probability that means the importance of the ith part for the action recognition in the frame . After computing the spatial attention , the model will weight the feature as the input of LSTM formulated in Eq . 3 . Yt = L2∑ i=1 αt , ixt , i ( 3 ) In previous section , we have already known the meaning of variable xt , i , after the attention weights each parts in time t , Yt is computed as the input of LSTM in time t. This spatial weighting will make our model focus on the important parts in the frame rather than treat each part as the same materiality . The initialization of the memory state and the hidden state of LSTM we adopt refers to the strategy used by Xu et al . ( 2015 ) . It can be formulated as follows , c0 = finit , c ( 1 T T∑ t=1 ( 1 L2 L2∑ i=1 xt , i ) ) h0 = finit , h ( 1 T T∑ t=1 ( 1 L2 L2∑ i=1 xt , i ) ) ( 4 ) The finit , c ( ) and finit , h ( ) are both the multilayer perceptrons . T is the number of the total frames . The c0 and h0 will be used as the initial state of LSTM , in addition to this , these two values will be also used for calculated α1 , the spatial attention of the first frame .

Summary: The paper proposes a spatio-temporal attention weighting mechanism in LSTM, applied to the task of human action recognition. VGG19 based frame features are fed to LSTM, soft attention is calculated based on previous works and temporal attention is predicted using another small neural network. The features are weighted by these attentions and eventually the network is trained with a regularized cross entropy loss. Empirical results are given on three datasets for action recognition, UCF11, UCF101 and HMDB51.