## 3 Multi-Stage Key Frame Selection
### Salient Posture Frame Selection  
We conduct salient posture frame selection by choosing some exclusive positions by global dynamics of human body parts. An example of salient posture frame selection (action \"hand waving\") is illustrated in Figure 2. We first divide the skeleton graph in each frame into five body parts (i.e., \"trunk\", \"left arm\", \"right arm\", \"left leg\" and \"right leg\") and denote \(\mathbf{X}_{t}^{s}\in\mathbb{R}^{\mathbb{C}\times V_{s}}\) as the \(s\)-th body part feature, where \(\sum\limits_{s=1}^{5}V_{s}=V\). We conduct average pooling across all the axes of each body part feature \(\mathbf{X}_{t}^{s}\) to obtain the part global pose dynamics \(\mathbf{p}^{s}=[p_{1}^{s},p_{2}^{s},\cdots,p_{N}^{s}]^{\top}\), where  
\[p_{t}^{s}=\text{AverPooling\_2D}(\mathbf{X}_{t}^{s})\in\mathbb{R}^{1}. \tag{1}\]  
The Savitzky-Golay (S-G) filter [30] is applied to remove the high-frequency noise from the \(\mathbf{p}^{s}\) to obtain the representative dynamics:  
\[\mathbf{p}=\mathbb{S}-\mathbb{G}(\mathbf{p}^{s},\beta_{t},\beta_{o}). \tag{2}\]  
where \(\mathbf{p}=[p_{1},p_{2},\cdots,p_{N}]^{\top}\). The \(\beta_{t}\) represents the length of the filter window, and \(\beta_{o}\) is the order of the fitted polynomial. Afterwards, we select the body part with the largest variance to remove the unnecessary low-frequency information,  
\[\hat{s}=\underset{s}{\text{argmax}}\text{Var}(\mathbf{p}^{s}). \tag{3}\]  
Figure 3 shows a pair of the representative dynamics of the body part \"left leg\" of an action \"jump up\" displayed by two subjects as an example. The action is composed of five phases: stand (before jump), squat (before jump), jump, squat (after jump) and stand (after jump). We find that the changes in actions conducted by the two subjects are consistent, thus extracting representative frames of the corresponding phases according to  
the dynamics can help to align the inter-class actions and generate temporal scale-invariant representations. From the figure, we also observe that the phase of standing upright is dispensable for the \"jump up\" action recognition, and the other phases are more necessary and discriminative. Frames at time steps with high volatility in the dynamics can just satisfy the aforementioned characteristics. Encouraged by the above observations, we select salient posture frames by simply finding the peaks and troughs of the representative dynamics calculated by the comparison of neighboring values, which is formulated as  
\[\mathbf{t}_{\mathrm{s}}=\mathrm{FindingPT}(\mathbf{p},\beta_{d},\beta_{p}), \tag{4}\]  
where \(\mathbf{t}_{\mathrm{s}}\in\mathbb{R}^{L_{1}}\) is the indices of the selected salient posture frames, and \(\beta_{d}\), \(\beta_{p}\) are the hyperparameters of distance and prominence, respectively. Specifically, the distance \(\beta_{d}\) represents the minimum horizontal time steps between the sampled peaks or troughs. If there is distance between two neighboring peaks or troughs, the smaller one will be removed. The prominence \(\beta_{p}\) is the minimum vertical distance between the peak or trough and the surrounding baseline. The sequence of salient posture frames is generated by extracting the corresponding selected frames of the video according to the chronologically sorted indices. Figure 1: Overview of our MSKFS. (**a**) Architecture of the proposed method. The skeleton data involved in video frames are inputted into the modules of salient posture frame selection and key frame refinement and alignment, where the output salient posture frames of the former module are regarded as the prior for refining and aligning the selected frames of the latter module. The skeleton structures in the selected frames can thus be constructed as a spatial–temporal graph and fed into a GNN to generate the final predicted action category. (**b**) Data flow through three main modules of our method. In the salient posture frame selection, the salient postures are selected from the uniformly sampled frames from the input sequence.