# Multi-Term Attention Networks for Skeleton-Based Action Recognition
## 4 Methods
### Multi-Term Attention Recurrent Neural Network  
The Multi-Term Attention Recurrent Neural Network (MTA-RNN) is introduced for time series processing, it consists of the Multi-Term Temporal Sliding LSTM (MT-TS-LSTM) and the AttentionRecalibration Module (ARM). Figure 2 shows the structure of the Multi-Term Attention Recurrent Neural Networks.  
In MTA-RNN, the input skeleton data is 3D joint coordinates, which is a series of multi-frame sequences constituting the action. It is divided into three subsets aligned with the X, Y, and Z-axis, respectively, and is trained with MTA-RNN. The MT-TS-LSTM is introduced in MTA-RNN for extracting various temporal features and fuse them according to weights, and the ARM is employed to enhance the key temporal features.  
For the input skeleton sequences, a multi-LSTM module based on temporal sliding is used to capture temporal features information for input action sequences at different terms, and the temporal feature is calibrated by the ARM.  
We process the input data in the MTA-RNN module to better obtain the temporal features of skeleton information. A sequence of multi-frame 3D joint coordinates is divided into three skeleton coordinate sequence subsets aligned X, Y, and Z-axis. By learning from the three channels, the temporal feature information in three dimensions: X, Y, and Z-axis can be obtained. The input of MTANs is set as \(I=\{X,Y,Z\}\in\mathbb{R}^{(T\times N\times 3)}\), where \(T\) represents the number of frames included in an action, and \(N\) represents the number of human joints included in the input data. \(X\in\mathbb{R}^{(T\times N)}\), \(Y\in\mathbb{R}^{(T\times N)}\), \(Z\in\mathbb{R}^{(T\times N)}\), which is the set of X coordinate, Y coordinate and Z coordinate of \(N\) joints along \(T\) frames. The following will take \(X\) as an example to explain the MTA-RNN. \(X\in\mathbb{R}^{(T\times N)}\) can be expressed as:  
\[X=\begin{bmatrix}x_{1}(1)&\cdots&x_{N}(1)\\ \vdots&\ddots&\vdots\\ x_{1}(T)&\cdots&x_{N}(T)\end{bmatrix} \tag{1}\]  
The design of the MTA-RNN is based on residual block \(X=X+F(X)\), in which \(F(X)\) is a recalibrated function. Taking original residual block in ResNets as the reference, the MTA-RNN is constructed by  
Figure 2: The structure of Multi-Term Attention Recurrent Neural Network (MTA-RNN), TS-LSTM: Temporal Sliding Long Short Term Memory. FC: Fully Connected layer  
mapping with the transformation from input \(X_{i}\) to output \(X_{o}\) to capture the richer temporal information, and the formula is as follows:  
\[X_{o}=X_{i}+F(X_{i}), \tag{2}\]  
where \(X_{o}\) is the final output of MTA-RNN, and \(F(X_{i})\) is the temporal feature recalibrated by the ARM. As is shown in Figure 2, \(F(X_{i})\) is based on two branches: \(F_{MT}(X_{i})\) and \(F_{A}(X_{i})\). The MT-TS-LSTM and the ARM are integrated into a unified framework, and two branches \(F_{MT}(X_{i})\) and \(F_{A}(X_{i})\) in the network respectively represent the temporal information of MT-TS-LSTM after fusion and attention weight. Therefore, the recalibrated temporal feature is as follows:  
\[F(X_{i})=F_{MT}(X_{i})\odot F_{A}(X_{i}), \tag{3}\]  
where \(\odot\) denotes the element-wise multiplication, \(F_{A}(X_{i})\) is the weight of \(F_{MT}(X_{i})\) to recalibrate temporal information, and \(F_{MT}(X_{i})\in\mathbb{R}^{(T\times K)}\) and \(F_{A}(X_{i})\in\mathbb{R}^{(T\times K)}\).