# Multi-Term Attention Networks for Skeleton-Based Action Recognition
## 5 Experiments and Analysis
### Ablation Experiment
#### 5.3.2 Effectiveness of Each Module in MTA-RNN  
For multiple modules in MTA-RNN, we also conducted experiments one by one, including MT-TS-LSTM, ARM, and the necessity of dividing the input data into three subsets aligned with X, Y and Z axes.  
The results of experiments using different structures on the NTU RGB+D dataset are shown in Table 3. \"1C (one channel)\" means the input is not divided into three subsets aligned with the X, Y, and Z-axis and trained together, and \"3C (three channel)\" means the input is divided into three subsets aligned with the X, Y, and Z-axis and trained separately. \"Single\" indicates that there is only a single long-term TS-LSTM in MTA-RNN, while \"Multiple\" applies multi-term TS-LSTM, including long-term, medium-term and short-term TS-LSTM. The last setting is whether to use the ARM module to recalibrate the temporal features. We apply ResNet-18 in the ST-CNN for all experiments because compared with the deeper residual network like ResNet-50, ResNet-18 can achieve similar classification accuracy but relatively faster training speed.  
As shown in Table 3, it is easy to see that: (1) The overall accuracy of one channel is lower than that of three channel, which indicates that modeling by three subsets aligned with the X, Y, and Z-axis respectively is better than modeling by all skeleton coordinate data. (2) The accuracy of applying a single long-term TS-LSTM is lower than that of multi-term TS-LSTM, which indicates that using multi-term TS-LSTM in the MTA-RNN can get better temporal dependence between actions. (3) The addition of the ARM module makes the model get higher recognition accuracy. This situation reveals the effectiveness of attention recalibration for temporal features. Based on the above experiments, it well validates the superiority of MTANs, and proved that the training of three subsets aligned with the X, Y, and Z-axis, the selection of MT-TS-LSTM and the addition of ARM are indispensable.  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Method** & **CS Acc. (\%)** & **CV Acc. (\%)** \\ \hline MTA-RNN (no ST-CNN) & 81.64 & 89.78 \\ ST-CNN (no MTA-RNN) & 74.19 & 80.11 \\ MTANs (MTA-RNN and ST-CNN) & 84.74 & 93.23 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Comparison of ablation experiment results on the NTU RGB+D dataset, which demonstrated the performance of MTA-RNN (Multi-Term Attention Recurrent Neural Network) and ST-CNN (Spatio-Temporal Convolutional Neural Network). CS: Cross-Subject; CV: Cross-View.  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Method** & **CS Acc. (\%)** & **CV Acc. (\%)** \\ \hline
1C-Single & 72.11 & 79.47 \\
1C-Single + ARM & 75.20 & 85.65 \\
1C-Multiple & 77.66 & 83.52 \\
1C-Multiple + ARM & 78.66 & 84.54 \\
3C-Single & 81.41 & 89.84 \\
3C-Single + ARM & 83.01 & 90.66 \\
3C-Multiple & 81.96 & 91.15 \\
3C-Multiple + ARM (MTA-RNN) & 84.74 & 93.23 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Comparison of ablation experiment results on the NTU RGB+D dataset, which demonstrated the performance of multi-channel, multi-term TS-LSTM and the ARM in MTA-RNN. ARM: Attention Recalibration Module. 1C: one channel. 3C: three channel. MTA-RNN: Multi-Term Attention Recurrent Neural Network.  
In order to better show the effect of the ARM model, we visualized the attention weight to analyze which frames play a role in the result of action recognition. Figure 4 shows temporal attention weights and the visual skeleton diagram of keyframes. In the above attention weight heatmap, the green color from light to dark represents its temporal attention weight from low to high. we can see that the keyframes (frames with top 3 weights) \"lift the hand\", \"move hand left\" and \"move hand right\" play a key role in action recognition.