## 3 Multi-Stage Key Frame Selection
### Key Frame Refinement and Alignment  
In the second stage of our MSKFS, we adaptively augment and refine the key frames based on the selected salient posture frames in the first stage. In addition to the discriminative information carried by the salient posture frames, we still need more detailed information to represent some local continuous variants (i.e., high-order features) of the motion and periodic trends that are important features of time signals. We believe that frames containing the aforementioned characteristics share some similarities with the salient posture frames in certain spaces. For example, frames representing local continuous variants are similar in appearance, and the most representative variants are the neighborhoods of the salient posture frames. Accordingly, we can learn to map the joint-level features of  
Figure 3: Example of the dynamics of body part “left leg” in action “jump up”. The left part of the figure is a pair of two skeleton sequences that are performed by Subject No. 2 and Subject No. 4. The right part represents a pair of two dynamics in the body part “left leg”. Figure 2: Example data flow of the proposed salient posture frame selection based on the global dynamics of body parts. The global dynamics of different body parts are plotted with curves in different colors. The salient postures can be reflected by dynamic changes. the frames into a common feature space and select the final key frames according to the similarity measurement in this space. In this paper, we use the variational inference to estimate the posterior of the selected key frames conditioned on the features that carry information related to the similarity measurement. There are two benefits as follows. First, the reparameterization trick enables the key frame selection operation differentiable for optimization with gradient descent. Second, the variational inference allows the knowledge to constrain the distribution of the temporal graph node, which improves the generalization and robustness of the proposed model. To be specific, a spatial graph convolution layer is first utilized to learn the frame-level representation \(g_{t}\). At time step \(t\), the human bodies are regarded as a graph where vertices represent body joints and edges characterize their connections. The graph feature of the \(v\)-th vertex at time \(t\) is calculated by  
\[g_{t}=\text{AverPooling\_1D}(\text{ReLU}(A\textbf{X}_{t}^{\top}\textbf{W})), \tag{5}\]  
where \(\textbf{W}\) is a learnable weight matrix, \(A\in\{0,1\}^{V\times V}\) represents the adjacency matrix and \(\text{AverPooling\_1D}(\cdot)\) means the average pooling operation along the axis of body joints. Given the frame-level graph features, we then use a multi-head attention operation to learn the similarity-measurement-based feature \(\textbf{s}_{t}\) for the key frame refinement and alignment. For the \(t\)-th frame, \(i\)-th head attention operation is formulated as  
\[\textbf{s}_{t}[i]=f_{v_{i}}([\textbf{g}_{t_{t}},\textbf{g}_{t}])\cdot\text{ Softmax}(\frac{f_{k_{i}}([\textbf{g}_{t_{s}},\textbf{g}_{t}])^{\top}f_{q_{i}}( \textbf{g}_{t})}{n_{\text{head}}}), \tag{6}\]  
where \(f_{q_{i}}(\cdot),f_{k_{i}}(\cdot)\) and \(f_{v_{i}}(\cdot)\) are linear functions, and \(n_{\text{head}}\) denotes the number of attention heads. Thus, we have the similarity-measurement-based feature \(\textbf{s}_{t}\) by concatenating the features calculated by different heads of attention operations:  
\[\textbf{s}_{t}=[\textbf{s}_{t}[1];\textbf{s}_{t}[2];\ldots;\textbf{s}_{t}[n_{ \text{head}}]]. \tag{7}\]  
We learn the posterior of each key frame conditioned on the input human skeleton sequence:  
\[q(w_{t}|\mathcal{X})=\text{Bernoulli}(\pi_{t}), \tag{8}\]  
where  
\[w_{t}=\left\{ \begin{array}{ll}0,&\text{if }L+\pi_{t}<0,\\ 1,&\text{otherwise},\end{array} \right. \tag{9}\]  
is an indicator variable representing whether to select the frame at \(t\) as the key frame, and \(L\sim\text{Logistic}\), i.e.,  
\[ \begin{array}{ll}L=\log U-\log(1-U),\\ U\sim\text{Uniform}(0,1).\end{array}