# Multi-Term Attention Networks for Skeleton-Based Action Recognition
## 3 Related Work  
Due to the presence of highly accurate depth sensor and attitude estimation algorithm, the skeleton information of human action is easy to obtain [11]. Therefore, considering the advantage of the skeleton information, skeleton-based action recognition method is gaining popularity in recent years. In this section, we reviewed the literature, which is closely related to the methods proposed in this paper. Skeleton information is a kind of time sequence for position coordinates of skeleton joints. According to this, a straightforward idea is applying recurrent neural networks for extracting actions from time sequences, such as RNN [12], LSTM [13], and Gator recursion unit (GRU) [14]. Researchers have tried to apply these methods to skeleton-based action recognition. A hierarchical RNN network [5] was proposed to classify actions based on skeleton data. An advanced LSTM network [13] that fully connected and added the regularization scheme is designed to obtain the high-level temporal features of skeleton information. A novel gating mechanism was introduced by [PERSON] et al. [15] to advance the reliability of the learning sequence in LSTM, which can adjust the effect of updating long-term context information stored in the storage. The methods above are helpful when dealing with short skeleton sequences. However, these methods still have some drawbacks--(i) the dependencies between poses are not considered, which lead the low action recognition accuracy when dealing with long skeleton sequences. (ii) the position relation dependencies of joints are not considered when dealing with single pose in skeleton sequences. To overcome the drawback, (i) attention mechanism was introduced for achieving human action recognition, which was widely used in deep-learning-based machine translation [16]. [PERSON] et al. [9] introduced the attention mechanism on an RNN-based model to recalibrate the temporal attention of the frames in the skeleton sequence. [PERSON] et al. [17] used the attention mechanism for skeleton-based action recognition, and the spatio-temporal features can be obtained by assigning different weights to each joint and assigning different weights to each frame. [PERSON] et al. [18] allocated the most informative frames through the attention mechanism and discarded the ambiguous frames in the sequence to recognize human actions. Due to the model with the attention mechanism being able to build the distribution of frames in a skeleton sequence, different weights can be assigned to different frames, and the dependency of frames in a sequence can be extracted to increase action recognition accuracy. As a solution of drawback (ii), there are two classes of methods to extract the spatial features of human actions from the skeleton information. One class is to design handcrafted features to capture the dynamics of joints. These methods are capturing the moving of joints according to the handcrafted features, such as the covariance matrixes of joint trajectories [19], the relative positions of joints [20], or rotations and translations between body parts [21]. However, these methods are limited by the specific positional relationship in the action, which not only leads to a decrease in expressive power and a reduction in accuracy, but also is not easy to transfer to different environment. The other class is to apply CNNs for to extract the position relation dependencies of joints. For example, [PERSON] et al. [9] proposed a framework called MAN combining CNNs and RNNs to extract spatio-temporal features in skeletons and calibrate the features with an attention mechanism. [PERSON] et al. [22] proposed a CNN-based framework for action classification, which helps to determine the importance of skeleton joints and rearrange them automatically. [PERSON] et al. [6] introduced an algorithm that uses deep neural networks to extract spatio-temporal features for 3D action recognition based on the skeleton. The above works are dedicated to exploring how to extract effective features from skeleton data.