# Multi-Grained Temporal Clip Transformer for Skeleton-Based Human Activity Recognition
## 3 Method
### Intra-Clip Self-Attention  
By kinematic analysis of human action categories, it is observed that inter-joint correlations across consecutive frames is important for the HAR. For instance, in \"long jump\" action, arm movements in frame \(t\) exhibit strong dynamical coupling with leg motions of frame \(t+i\) at various temporal resolutions, demonstrating the necessity of cross-frame joint feature extraction. Therefore, it is very useful to extract the related features of different joints between adjacent frames and the sub-action clip information of short time intervals at various temporal resolutions.  
Based on the above analysis, we propose the intra-clip self-attention (ICSA) module to effectively capture short-term motion information, as shown in Figure 3. Specifically, the encoded skeleton feature \(\tilde{\textbf{X}}\in\mathbb{R}^{C_{1}\times T\times V}\) is first partitioned into \(T_{0}\) clips. Each clip is then flattened by reshaping the tensor, as follows:  
\[\tilde{\textbf{X}}=[\textbf{X}_{1},\textbf{X}_{2},\dots,\textbf{X}_{T_{0}}], \ \ \textbf{X}_{i}\in\mathbb{R}^{C_{1}\times n\times V} \tag{5}\]  
\[\tilde{\textbf{X}}\in\mathbb{R}^{C_{1}\times T_{0}\times n\times V}\to \mathbb{R}^{C_{1}\times T_{0}\times V_{0}} \tag{6}\]  
where \(T_{0}=T/n\), \(V_{0}=V\times n\), and \(n\) indicates the total frames within a clip.  
Subsequently, the flattened tensor \(\tilde{\textbf{X}}\) is projected into \(\textbf{X}_{p}\) through a \(1\times 1\) 2D convolution layer. Then, the query (**Q**), key (**K**), and value (**V**) are obtained by splitting \(\textbf{X}_{p}\) along the channel dimension using the Split operation, as follows:  
\[\textbf{X}_{p}=\text{Conv2D}_{1\times 1}(\tilde{\textbf{X}})\in\mathbb{R}^{3C_{1} \times T_{0}\times V_{0}} \tag{7}\]  
\[\textbf{Q},\textbf{K},\textbf{V}=\text{Split}(\textbf{X}_{p})\in\mathbb{R}^{C _{1}\times T_{0}\times V_{0}} \tag{8}\]  
Figure 3: The structure of the intra-clip self-attention (ICSA) module.  
To model the intrinsic correlations between joints, we introduce a learnable parameter matrix \(\mathbf{A}\in\mathbb{R}^{T_{0}\times V_{0}\times V_{0}}\) to encode the topology information of the human body. Additionally, a learnable parameter \(\beta\) is employed to dynamically reweigh the correlation matrix. The output \(\mathbf{X}_{attn}\) is then computed by multiplying the correlation matrix with the corresponding value \(\mathbf{V}\), as follows:  
\[\mathbf{A}_{tot}=\text{Tanh}\bigg{(}\frac{\mathbf{Q}\mathbf{K}^{T}}{\sqrt{C}} \bigg{)} \tag{9}\]  
\[\mathbf{X}_{attn}=\mathbf{V}(\beta\times\mathbf{A}_{tot}+\mathbf{A})\in \mathbb{R}^{C_{1}\times T_{0}\times V_{0}} \tag{10}\]  
Finally, we employ \(H\) multi-head self-attention mechanisms to capture richer and more complex dependencies among joints.  
\[\mathbf{X}_{ICSA}=\text{Concat}(\mathbf{x}_{attn}^{1},\mathbf{x}_{attn}^{2}, \ldots,\mathbf{x}_{attn}^{H})\in\mathbb{R}^{C_{1}\times T_{0}\times V_{0}} \tag{11}\]  
In the following, a feed-forward network (FFN) and two residual connections are utilized to improve the model's representation capacity and training stability.