## 1 Introduction  
Human action recognition in videos is one of the fundamental challenging tasks in computer vision. Since studies in images have had great success, researchers are paying more attention to video. In the literature, recognizing actions plays a vital role in several applications, such as multimedia contexts [1, 2], video surveillance [3, 4], video streaming [5, 6, 7], health care systems [8] and smart indoor security systems [9, 10, 11]. Recently, human action recognition has achieved great success in depth video and sensor [12, 13, 14, 15, 16, 17, 18]. In this work, we focus on RGB video sequences from single camera. Unlike action recognition in images, action recognition in videos relies on motion dynamics in addition to visual appearance. Traditional approaches use hand-crafted features, and then input the aggregated features to the classifier. In contrast, the recent surge of deep neural networks (DNNs) achieve a significant advancement in extracting features. Among them, convolutional neural networks (CNNs) have shown a great ability to produce a rich representation of the input and achieve promising results on several challenging tasks, including image classification, object detection, machine translation, and tracking [19, 20, 21]. For this task, a direct approach to recognize action is to arm the convolutional operation with temporal information such as 3D convolutional networks [22, 23]. Although they acheive great performance, these works are still missing much temporal information. Recurrent neural networks (RNNs), another branch of DNNs, are a class of artificial neural networks, where the recurrent architecture allows the networks to exhibit dynamic temporal behavior. Since the video frames are sequential, RNNs might be more suitable for this kind of task. Thus, many works [24; 25] have combined RNN with CNN for action recognition and have shown good results. Specifically, LSTM [26] has been proven by [PERSON] et al. [27], [PERSON] et al. [28] and [PERSON] et al. [29] to be effective for action recognition task from video sequences. In [25], they stack three LSTM layers and get great performance, however, they only fed the bottom hidden states of LSTM to top layers.Baradel et al. [30] are aware of weakness of this simply structure, thus, they embed the input feature maps with spatial attention model and change the structure of LSTM to fit the different input. Moreover, these changes improve the performance and then [30] outperform [25]. Attention mechanism mentioned in above method has obtained many interests from computer vision community and received a great achievement in caption generation, neural machine translation and image recognition. By attention mechanism, a model can learn where should be paid attention to, for example, in Figure 2, the boy is juggling soccer, attention mechanism leads the model to focus on the leg and football which are important factors to classify the video into \"soccer_juggling\". The attention mechanism due to great performance of catching region of interest is also introduced to action recognition [25; 27; 31]. For example, [PERSON] et al. [25] transferred the attention mechanism on the spatial domain to action recognition. Although the above DNN architectures with attention mechanisms have achieved great performance, we believe that attention mechanisms in video are not the same as that in images. For instance, in the task of image caption, the model with attention mechanisms learns to pay attention to the part of image from the previous state, which focus on the other part of image. However, images are different from videos, which consist of many frames. Therefore, the regions where activities are happening will be changing, and the interesting areas in previous frames might not be the main portions in next video frame. In addition, the intermediate feature vectors are limited as they are generated by only looking at previous frames in order.