# Multi-Term Attention Networks for Skeleton-Based Action Recognition
## 4 Methods
### Multi-Term Attention Recurrent Neural Network
#### 4.1.2 Attention Recalibration Based on Fusion Features  
After MTA-RNN obtains the multi-scale temporal features captured by the long-term, medium-term, and short-term TS-LSTM networks, respectively, we fuse the three different scale features by weights. It hopes to obtain robust temporal features from multi-scale information.  
In order to find reasonable weights for temporal features and obtain better experimental results, we set multiple sets of different weights and analyzed by the Analytic Hierarchy Process (AHP) ([10]). According to the experiment, \(p_{L}=3\), \(p_{M}=2\), and \(p_{S}=1\) are selected as the weight of \(F_{L}(X_{i})\), \(F_{M}(X_{i})\), \(F_{S}(X_{i})\), respectively. This process is to better allocate the three LSTMs with different terms, so as to obtain more effective fusion temporal features. This laid the foundation for the attention recalibration of the fusion features later. Then their weighted average is calculated, which is the final output \(F_{MT}(X_{i})\):\[F_{MT}(X_{i})= (p_{L}+p_{M}+p_{S})^{-1} \tag{8}\] \[(p_{L}F_{L}(X_{i})+p_{M}F_{M}(X_{i})+p_{S}F_{S}(X_{i})).\]  
The action features of different time scales are considered after weighted fusion processing, which is the uniqueness of our method compared to others. \(F_{MT}(X_{i})\in\mathbb{R}^{(T\times K)}\) summarizes the temporal information in the MTA-RNN for the skeleton joints across the sequence, which can be expressed as:  
\[F_{MT}(X_{i})=\begin{bmatrix}x_{1}(1)&\cdots&x_{k}(1)&\cdots&x_{K}(1)\\ \vdots&\ddots&\vdots&&\vdots\\ x_{1}(t)&\cdots&x_{k}(t)&\cdots&x_{K}(t)\\ \vdots&&\vdots&\ddots&\vdots\\ x_{1}(T)&\cdots&x_{k}(T)&\cdots&x_{K}(T)\end{bmatrix} \tag{9}\]  
where \(x(t)\) is the row vecach frame has different effects on whether the action can be correctly classified. Most frames in the sequence provide contextual information, and only keyframes contain the discriminative and important information, which play a crucial role in action recognition. Based on this observation, we add the attention mechanism in MTA-RNN called ARM to pay different levels of attention to different frames in the fusion temporal features.  
To recalibrate the fusion temporal features information \(F_{MT}\), the attention weight \(F_{A}\) is exploited as \(F(X_{i})=F_{MT}(X_{i})\odot F_{A}(X_{i})\). The attention recalibration scheme is able to capture global frame-wise depenence across \(T\) frame. Firstly, summing up every row vector of \(X\) in the Equation (9), through the average pooling operation, a vector of \(T\times 1\) is generated:  
\[X_{p}=\left[\frac{1}{K}\sum_{k=1}^{K}x_{k}(1),\cdots,\frac{1}{K}\sum_{k=1}^{K} x_{k}(T)\right]^{\mathrm{T}}\in\mathbb{R}^{(T\times 1)}. \tag{10}\]  
As shown in Figure 2, the under branch of MTA-RNN is the structure of the attention module. Two FC layers provide the non-linear interaction between frames, and a dimensionality-reduction layer and a dimensionality-increasing layer are added to carry out denoising and excitation operations respectively, and thus enhance the discriminant ability of features. The output of the attention branch \(F_{A}(X_{i})\) is:  
\[F_{A}=\sigma(W_{2}\theta(W_{1}X_{i}))\in\mathbb{R}^{(T\times K)}, \tag{11}\]  
where \(\theta(\cdot)\) denotes ReLU activation function, \(\sigma(\cdot)\) refers to the sigmoid function, \(W_{1}\) is the parameter of dimensionality-reduction layer, and \(W_{2}\) is the parameter of dimensionality-increasing layer. Finally, by the element-wise multiplication of \(F_{MT}\) and \(F_{A}\), \(F(X_{i})\) is obtained.  
As shown in Figure 2, to calculate the output feature map of the MTA-RNN, \(F(X_{i})\) is updated by an FC layer as \(X_{FC}\gets FC(X_{i})\in\mathbb{R}^{(T\times K)}\). \(F_{MT}\) and \(F_{A}\) can be jointly learned during training, which is similar to the residual block.  
Similarly, the output of Y-axis and Z-axis temporal features is obtained as \(Y_{o}\), \(Z_{o}\) based on \(Y_{i}\) and \(Z_{i}\) in its corresponding MTA-RNN, and the final output is \(O=\{X_{o},Y_{o},Z_{o}\}\in\mathbb{R}^{(T\times N\times 3)}\).