# Multi-Scale Adaptive Aggregate Graph Convolutional Network for Skeleton-Based Action Recognition
## 3 Multi-Scale Adaptive Aggregate Graph Convolutional Network
### Spatial-Temporal-Channel Attention Module  
The attention mechanism is a resource allocation mechanism, which redistributes the originally evenly allocated resources according to the importance of the focus. Inspired by [42; 43], here we propose an attention mechanism, which contains three modules of spatial attention, temporal attention and channel attention mask.  
Spatial attention module: In the spatial dimension, the movement range and dynamics of each joint in different actions are different. We can use the attention mechanism to help the model adaptively capture the dynamic correlation of each joint in the spatial dimension based on different levels of attention. By using the spatial relationship of features to generate the spatial attention mask \(M_{s}\), average pooling can learn from the target object and calculate spatial statistics [19; 44], and maximum pooling can extract another important clue of object features to infer more refined attention. Using both types of pooling can improve the network representation ability [42]. It is formulated as  
\[M_{s}=\sigma(f_{s}(Avgpool(f_{in})\|Maxpool(f_{in}))) \tag{5}\]  
where \(f_{in}\in R^{C\times T\times N}\) is the input feature map, \(C,T,N\) represent the number of channels, frames and joints, respectively; \(Avgpool(\cdot)\) and \(Maxpool(\cdot)\) represent the average-pooling and max-pooling operation; \(\|\) represents the connection operation; \(f_{s}\) represents the 2D convolution operation; and \(\sigma\) represents the \(\hat{sigmoid}\) activation function. \(M_{s}\in R^{1\times 1\times N}\), as shown in the attention module in Figure 1; we calculate the product of the attention map \(M_{s}\) and the input feature map, and then the input is connected with the result to refine.  
\[f_{1}=f_{in}\otimes M_{s}+f_{in} \tag{6}\]  
where \(\otimes\) denotes element-wise multiplication.  
Temporal attention module: In the time dimension, there is a correlation between the joint points of different frames, and the correlation between frames of different actions is also different. Similarly, we can use the attention mechanism to adaptively give them different weights. It is formulated as  
\[M_{t}=\sigma(f_{t}(Avgpool(f_{1});Maxpool(f_{1}))) \tag{7}\]  
\[M_{t}=\sigma(f_{t}(Avgpool(f_{1});Maxpool(f_{1}))) \tag{8}\]  
where \(M_{t}\in R^{1\times T\times 1}\), and other symbols are similar to the spatial attention module.  
Channel attention module: according to [19], each layer of the convolutional network has multiple convolution kernels, and each convolution kernel corresponds to a feature channel. The channel attention mechanism allocates resources between each convolution channel. We first perform feature compression through the squeeze operation, use global average-pooling to generate the statistics of each channel and then utilize the excitation operation to explicitly model the correlation between the feature channels through parameters and generate weights representing importance for each feature channel. Finally, we reweight the feature map, which is weighted to the previous feature channel by channel through multiplication. Similarly, we can formulate it as  
\[M_{C}=\sigma(\delta(Avgpool(f_{2})W_{1})W_{2}+\delta(Maxpool(f_{2})W_{1})W_{2}) \tag{9}\]  
\[f_{out}=f_{2}\otimes M_{c}+f_{2} \tag{10}\]where \(W_{1}\) and \(W_{2}\) are learnable weights; \(W_{1}\) is used for dimensionality reduction; \(W_{2}\) is used to calculate attention weights; and \(\delta\) represents the \(ReLU\) activation function.