# Multi-Grained Temporal Clip Transformer for Skeleton-Based Human Activity Recognition
## 3 Method
### Multi-Grained Temporal Clip Attention Module  
The researchers of [40] revealed that in the visual cortex, the local receptive field (RF) sizes of neurons in the same area are different, which enables the neurons to collect multi-scale spatial information in the same processing stage. Furthermore, the RF sizes of neurons in the visual cortex are not fixed, but modulated by the stimulus. We employed an adaptive approach to generate attention weights for temporal motion features at different granularities, assigning varying importance weights to short-term motion features across scales before ultimately fusing them.  
Based on the above analysis, we propose the multi-grained temporal clip attention (MTCA) module, as shown in Figure 4. Specifically, the input feature \(\mathbf{X}\in\mathbb{R}^{C_{1}\times T\times V}\) is segmented into \(K\) clips along the channel. These \(K\) features are then processed by ICSA modules with varying granularity to capture short-term motion information at different granularities. Finally, the outputs from the ICSA modules are reshaped for further processing, as follows:  
\[[\mathbf{X}^{1},\mathbf{X}^{2},\ldots,\mathbf{X}^{K}]=\mathbf{Split}(\mathbf{ X})\in\mathbb{R}^{C_{1}/K\times T\times V} \tag{12}\]  
\[\mathbf{X}_{ICSA}^{i}=\text{ICSA}_{n=i}(\mathbf{X}^{i})\in\mathbb{R}^{C_{1} \times T_{0}\times V_{0}} \tag{13}\]  
\[\mathbf{X}_{ICSA}^{i}\in\mathbb{R}^{C_{1}\times T_{0}\times V_{0}}\rightarrow \mathbb{R}^{C_{1}\times T\times V} \tag{14}\]  
where \(\mathbf{X}^{i}\) is obtained by splitting \(\mathbf{X}\) along the channel dimension, with \(i=\{1,2,3,\ldots,K\}\), and \(\mathbf{X}_{ICSA}^{i}\) indicates the output features from the i-th ICSA module.  
Figure 4: The architecture of the multi-grained temporal clip attention (MTCA) module, where MTCA consists of \(K\) ICSA modules and a multi-grained feature fusion (MGFF) module. MGFF module is designed to dynamically fuse short-term motion features with various granularities from \(K\) ICSA modules.  
Next, as shown in Figure 4 for MGFF, we aggregate the output features from the ICSA modules with \(K\) different granularities through summation, followed by an average pooling operation to generate a global representation \(\mathbf{X}_{ICSA}^{global}\in\mathbb{R}^{C_{1}\times 1\times 1}\).  
Subsequently, \(\mathbf{f}_{1}\) is applied on \(\mathbf{X}_{ICSA}^{global}\) for channel reduction, and the result is sequentially passed through \(\mathbf{f}_{2}\) and a softmax function to adjust the weights of features at different granularities, obtaining weight vector \(\mathbf{W}\in\mathbb{R}^{KC_{1}\times 1\times 1}\). Finally, \(\mathbf{W}\) is divided along the channel dimension using a Split operation to obtain weight vectors with different granularities.  
\[\mathbf{X}_{ICSA}^{global}=\mathrm{GAP}\Bigg{(}\sum_{i=1}^{K}\mathbf{X}_{ICSA }^{i}\Bigg{)} \tag{15}\]  
\[\mathbf{W}=\mathrm{Softmax}\Bigg{(}\mathbf{f}_{2}\bigg{(}\mathbf{f}_{1} \bigg{(}\mathbf{X}_{ICSA}^{global}\bigg{)}\bigg{)}\Bigg{)}\in\mathbb{R}^{KC_{1} \times 1\times 1} \tag{16}\]  
\[\mathbf{W}_{1},\mathbf{W}_{2},\ldots,\mathbf{W}_{K}=\mathrm{Split}(\mathbf{W} )\in\mathbb{R}^{C_{1}\times 1\times 1} \tag{17}\]  
where \(\mathbf{W}_{i}\) represents the weight vector corresponding to the \(i\)-th granularity, \(\mathbf{f}_{1}\) and \(\mathbf{f}_{2}\) denote \(1\times 1\) 2D convolutional operations. Then, a dot-product operation is applied between \(\mathbf{W}\) and \(\mathbf{X}_{ICSA}\) to reweight the output features from different ICSA modules. Finally, to strengthen the training stability, a residual connection is incorporated to obtain the output \(\mathbf{X}_{MTCA}\in\mathbb{R}^{C_{1}\times T\times V}\):  
\[\mathbf{X}_{MTCA}=\sum_{i=1}^{K}\mathbf{W}_{i}\odot\mathbf{X}_{ICSA}^{i}+ \mathbf{X} \tag{18}\]