# Multi-Term Attention Networks for Skeleton-Based Action Recognition
## 1 Introduction  
Human Action Recognition (HAR) has attracted the attention of research communities in the computer vision area in recent years. Researchers have made significant achievements in action recognition by video-based methods. Generally, RGB videos andRGB-D videos are the most used for this task. For the methods based on RGB videos [1], researchers mainly extracted features from video frames directly by image processing techniques, and then applied various machine learning algorithms for modeling. However, color information in the video is susceptible to external factors that are not related to human actions, such as shooting environments, illumination conditions, and clothing textures [2]. Depth information in the video is less influenced by noises because of appearance independence and light independence, which can be obtained in real-time by depth cameras such as Microsoft Kinect [3]. Besides, RGB-D video-based methods consider the spatial position relationship of the human body, which can help to extract more spatial features in action recognition. Based on that, more and more researchers [4] apply depth information for action recognition. Skeleton-based methods are also widely applied for action recognition in recent years. Comparing with the RGB-D videos that are still containing the noise from the video background, skeleton data only contains the position coordinates of joints, so it avoids not only the influence of the variation of illumination and appearance but also the influence of the background. Meanwhile, benefit with applying the depthinformation of the human joints in skeleton data, skeleton-based methods are able to extract depth features from human actions compared to RGB-based methods. On the other hand, the skeleton-based methods are more lightweight than RGB-D video-based methods. Skeleton datasets take up less memory than RGB-D video datasets, and the cost for training and deploying skeleton-based action recognition methods is lower. Due to the spatial position relationship is already included in the skeleton sequence, skeleton-based action recognition [5] was initially implemented using only Recurrent Neural Networks (RNNs). Later, some researchers [6] tried to obtain spatial features by adding Convolutional Neural Networks (CNNs). It is worth noting that straightforward RNN algorithms [7] generally extracted only the long-term temporal dependence of human actions, which is unable to solve problems that the time taken by different objects for completing the same action is different. In actual situations, the time scales for different people to complete the same action may vary greatly. Take the example of \"stand up\" and \"put on a shoe\", the samples of the first action have almost the same time scale, and the recognition accuracy is higher; the difference between the time scale of the second action samples is larger, and the classification accuracy is lower. This difference will affect the accuracy of action recognition to a certain extent. Therefore, the temporal difference in actions should be considered as a feature to improve the accuracy of action recognition. The problem is that the general action recognition method can only extract single-term temporal features, and the ability to model the spatio-temporal dynamics of actions with different time scale is limited. This will result in lower accuracy. To resolve this problem, we propose a new general deep learning framework called Multi-Term Attention Networks (MTANs, see the supplementary materials) for skeleton-based action recognition, which is inspired by the Ensemble Temporal Sliding Long Short Term Memory networks (Ensemble TS-LSTM) [8] and Memory Attention Networks (MAN) [9]. An overview of the proposed framework is shown in Figure 1. The MTANs consists of three Multi-Term Attention Recurrent Neural Network (MTA-RNN) and a Spatio-Temporal Convolutional Neural Network (ST-CNN). Firstly, the MTA-RNN is trained by three skeleton coordinate sequence subsets aligned X, Y, and Z-axis respectively to obtain temporal features.