# Multi-Grained Temporal Clip Transformer for Skeleton-Based Human Activity Recognition
###### Abstract  
Skeleton-based human activity recognition is a key research topic in the fields of deep learning and computer vision. However, existing approaches are less effective at capturing short-term sub-action information at different granularity levels and long-term motion correlations, which affect recognition accuracy. To overcome these challenges, an innovative multi-grained temporal clip transformer (MTC-Former) is proposed. Firstly, based on the transformer backbone, a multi-grained temporal clip attention (MTCA) module with multi-branch architecture is proposed to capture the characteristics of short-term sub-action features. Secondly, an innovative multi-scale spatial-temporal feature interaction module is proposed to jointly learn sub-action dependencies and facilitate skeletal motion interactions, where long-range motion patterns are embedded to enhance correlation modeling. Experiments were conducted on three datasets, including NTU RGB+D, NTU RGB+D 120, and InHARD, and achieved state-of-the-art Top-1 recognition accuracy, demonstrating the superiority of the proposed MTC-Former. human activity recognition; temporal clip; deep learning; self-attention; skeleton 20 models demonstrate strong capabilities in processing skeleton data and capturing spatiotemporal correlations within skeleton sequences. [PERSON] et al. [11] introduced a multi-scale aggregation strategy to broaden the receptive field and addressed the biased weight issue. They found that when handling rapidly changing sub-actions, the short-term features may lack sufficient granularity. [PERSON] et al. [14] proposed a local-global GCN combined with a dynamic frame weighting module for joint global temporal modeling, relying on a fixed skeleton topology and predefined adjacency relationships, which limited its ability to model long-range periodic actions (e.g., dance sequences). [PERSON] et al. [19] introduced hierarchical temporal partitioning and aggregation to capture long-term dependencies, whereas global temporal relationships were constrained by the fixed-window mechanism of transformers. Despite the remarkable advancements made by the aforementioned methods, these methods solely focus on spatial dependencies among joints within individual frames, while neglecting the complex spatiotemporal correlations across consecutive frames, thereby failing to handle both short-term sub-action features and long-term motion correlations. To address these problems, [PERSON] et al. [18] proposed a segment partitioning approach and employed self-attention mechanisms to model short-term sub-action information within the intra-clip. [PERSON] et al. [20] introduced a temporal-local spatial-temporal attention mechanism (TL-STA) designed to model inter-joint correlations across sequential frames. [PERSON] et al. [22] utilized physical topology constraints of the human skeleton to fully explore dependency between joints within a tuple to capture local and global temporal motion information. However, most aforementioned approaches failed to account for the inherent variations in temporal dynamics across different actions, thereby lacking in human motion domain knowledge-embedded feature extraction and interpretability. In actuality, the domain knowledge of human motion plays a critical role in skeleton-based activity recognition, including the dynamics of human motion, behavioral habit patterns, and spatial-temporal contextual information. As shown in Figure 1, the activities of \"clapping\" and \"long jumping\" differ in terms of motion phases, motion patterns, and motion speeds. Consequently, selecting an appropriate clip partition strategy is crucial for extracting short-term sub-action information. To address this, an innovative multi-grained temporal clip transformer (MTC-Former) employing self-attention mechanisms is proposed to model correlations among joints, thereby improving activity recognition accuracy by integrating both fine-grained details and contextual information.