###### Abstract  
Graph neural networks (GNNs) are extensively utilized to capture the spatial-temporal relationships among human body parts for skeleton-based action recognition. However, due to the inefficient information propagation caused by redundant sampling of video frames in the temporal domain, we focus on refining temporal graphs through key frame selection. To this end, we propose a multi-stage key frame selection (MSKFS) method, aiming to find the most representative frames as the graph nodes to learn compact temporal graph representations of human skeletons for action recognition. The MSKFS progressively selects key frames in two stages: (1) salient posture frame selection based on the global dynamics of body parts and (2) key frame refinement and alignment according to intra-frame correlations. The first stage captures the most salient information and aligns the corresponding information of skeleton sequences within the same category. The second stage enriches the subtle information for the integrity of the information derived by the salient frames. Moreover, variational inference is applied to differentiate the key frame refinement and alignment procedure, allowing the end-to-end optimization of arbitrary graph-based models to represent the obtained compact graph for skeleton-based action recognition. Our MSKFS method achieves state-of-the-art performances on two challenging action recognition datasets. c 2024 1 2024 1 2024 1 2024 1  
Aticle  
Key Frame Selection for Temporal Graph Optimization of Skeleton-Based Action Recognition  
Aticle  
Aticle  
Key Frame Selection for Temporal Graph Optimization of Skeleton-Based Action Recognition  
Aticle  
Key Frame Selection for Temporal Graph Optimization of Skeleton-Based Action Recognition  
recognition, such as refining graph topology structures [2; 3; 4; 5; 6; 7; 8; 9; 10], reasoning the relations between the partial and the whole graphs [11], improving the robustness of noise interference resistance [12; 13] and efficient computing [14; 15]. Among them, graph topology refinement methods mainly focus on revealing the latent spatial relationships between body parts beyond only reasoning the physical relations. Differently, we mainly focus on optimizing the temporal structure of the spatial-temporal graph that represents the skeleton-based action. In this paper, we consider two main problems in the temporal modeling of skeleton-based actions with GNNs. One problem of modeling temporal relations in human actions stems from the inefficiency of information propagation through frame-level features that are highly redundant. The presence of numerous redundant and misleading frames within each skeleton sequence poses a challenge to GNNs that aim to learn discriminative representations of temporal relations, and these redundant frames might even lead to incorrect predictions [14; 15]. We also consider another problem of temporal relation representation with GNNs, i.e., the performance might be affected by the dramatically variant lengths of different skeleton sequences. For example, the action \"drink water\" in the NTU RGB+D 60 dataset [16] takes 66 frames for Subject No. 2 and 150 frames for Subject No. 3. Generally, two main methods are employed to conquer this problem by fixing the lengths of different sequences: zero or loop padding and random sampling. Intuitively, padding increases the computation expense and introduces a lot of irrelevant frames. Random sampling makes the training procedure unstable for the robustness of GNNs [17]. We focus on graph optimization in the temporal domain and the uniformity of the length of the sequence by selecting a certain number of key frames. To tackle the above challenges, we propose a multi-stage key frame selection (MSKFS) method, which optimizes the temporal graph by automatically discovering a few representative frames from the skeleton sequence to recognize the skeleton-based actions.