# Whole-Body Keypoint and Skeleton Augmented RGB Networks for Video Action Recognition
## 3 Proposed Method
### Overview of Clip-Level Training and Prediction  
To directly feed the classifier with a whole video is infeasible for our computer due to limited hardware resources. Therefore, normal strategies usually divided the video v into a set of clips \(\{c^{(1)},c^{(2)},c^{(3)},\ldots,c^{(i)},\ldots\}\), and the classifier was fed with clips instead of a whole video. Given a clip classifier \(\mathbf{g}\), the clip-level prediction can be denoted as \(\mathbf{g}(c^{(1)};\mathcal{W})\), and video-level prediction can be formulated as follows:  
\[Y_{v}=H\big{(}\mathbf{g}(c^{(1)};\mathcal{W});\mathbf{g}(c^{(2)};\mathcal{W}); \ldots;\mathbf{g}(c^{(n)};\mathcal{W})\big{)}, \tag{1}\]  
where \(\mathbf{g}(c^{(1)};\mathcal{W})\) is the function that represents the classifier \(\mathbf{g}\) with weights \(\mathcal{W}\). We can infer from Equation (1) that the classification result of the given video is obtained by aggregating the clip-level results through function \(H\).  
The clip-level training and prediction course is shown in Figure 2. The classifier \(\mathbf{f}\) operates on fixed-length clips of \(F\) frames with spatial resolution \(H\times W\), and makes a prediction of classification probabilities.  
Our goal is to train the RGB classifier \(\mathbf{f}:\mathbb{R}^{F\times 3\times H\times W}\rightarrow[0,1]^{C}\) to possess more robust classification ability by focusing on body information when fed with clips that contain a noisy background. For this purpose, we first obtain a well-trained classifier \(\mathbf{f}_{WKS}\) on the WKS labeled dataset, which is then used as the teacher network for training the RGB network based on the distillation strategy. We use \(f_{WKS}\) to indicate the feature map of the target distillation layer; it is localized at the high layer of classifier \(\mathbf{f}_{WKS}\) and produced by fully concerning the body information. Normally, the extraction course from the input RGB frames to \(f_{WKS}\) should consist of two steps: locating the keypoints of input frames, and operating the classification function \(\mathbf{f}_{WKS}\). In our method, this course is greatly simplified by using distillation, and high-level features \(f_{WKS}\) can be directly produced from RGB frames. Similarly, we use \(f_{distill}\) to indicate the feature map of the distillation layer in the RGB network. During the training process, body information capturing ability is transferred to the RGB network as \(f_{distill}\) gradually approximates to \(f_{WKS}\). Finally, at inference, our RGB  
Figure 2: Framework of clip-level training and prediction. The architecture of our proposed RGB network is shown in the dotted box, which includes a 3D Conv module and Swin transformer module. The dotted arrows show the backpropagation path.  
network can distinguish the actions with the ability of concerning body information and do not require any pose detection processes.  
Moreover, transformers lack some of the inductive biases inherent to CNNs, and therefore do not generalize well when trained on small datasets [46]. Distillation is an effective method for improving the performance of transformers [47]. Based on these considerations, our proposed transformer-based framework will further benefit from this distillation training method.