# Whole-Body Keypoint and Skeleton Augmented RGB Networks for Video Action Recognition
## 3 Proposed Method
### Evaluation of Distillation  
We denote the Swin-B-based network and the Swin-S-based network as 3D-ConSwin-B and 3D-ConSwin-S, respectively. The evaluation on the UCF-101 and HMDB-51 datasets are listed in Table 2. All accuracies are obtained from split-1 of UCF-101 and HMDB-51. We compared Swin transformer-based models to advanced 3D CNNs, such as 3D-ResNext-101 [51], 3D-DenseNet-121 [51], and 3D-ResNet-152 [51], and our network achieved competitive results. Note that all our models are pretrained on Kinetics-400 and fine-tuned on the target dataset. We observed that a single stream of the WKS network or RGB network does not show remarkable recognition ability, but the combination of the two streams achieved an impressive accuracy of 94.1%. By utilizing the distillation method, the accuracy on UCF-101 is effectively boosted by approximately 2.1% and 1.4%, respectively, for 3D-ConSwin-B and 3D-ConSwin-S.  
We conducted an experiment to explore how the distillation method changes the RGB network, as illustrated in Figure 5. For deep learning, we found that what kinds of features are extracted for recognizing is very uncertain due to its \"black box\" characteristic. Therefore, extracted features may not be very discriminative to target action, such as in video v_ApplyLipstick_g04_c01, the RGB network may take the face of a female as a key feature and lead to confusion between categories of Apply Lipstick and Apply Eye Makeup, as both categories have a female face. However, if the hand pose and face keypoints are clearly marked, the network may capture the discriminative features of hand and lip overlapping, thus, making more precise classification based on these features. It can be observed in Figure 5, after distilled pose capturing ability to RGB network, some easily confused categories for the RGB network, such as Blow Dry Hair and Brushing Teeth, Nunchucks, and Archery which can be better distinguished by using pose features. With distillation, WKS augmented the RGB network which is able to classify these categories as more accurate. A comparison of the confusion matrices is shown in Figure 6, which illustrates the confusion matrices of twenty categories having the lowest accuracy on the UCF-101 dataset; 3D-ConSwin-B(RGB) and 3D-ConSwin-B(RGB) with distillation are presented at the top and the bottom, respectively. We observed that after distillation, some confused categories have improved accuracy. As the distilled RGB network can extract features like the WKS network, it focuses more on the pose and pose motion of the input clips. Parts of the categories that share similar backgrounds are easily confused, but with the benefit of WKS labeling, they can be more effectively classified. For example, the category of FrontCrawl only obtains an accuracy of 45.9% with the RGB network, it is easily confused with the category of BreastStroke. They both have swimming pools as their background, but after distillation, the accuracy is improved to 59.5% because the pose is represented more clearly. \begin{table}
\begin{tabular}{c c c c} \hline \hline  & **Architecture** & \multicolumn{3}{c}{**Top-1 Accuracy (\%)**} \\ \cline{3-4}  & & **UCF-101** & **HMDB-51** \\ \hline  & 3D-ConSwin-B (RGB) & 90.9 & 63.7 \\  & 3D-ConSwin-S (RGB) & 90.5 & 63.6 \\ w/o & 3D-ResNext-101 & 90.7 & 63.8 \\ distillation & 3D-DenseNet-121 & 87.6 & 59.6 \\  & 3D-ResNet-152 & 89.6 & 62.4 \\ \hline w/ & 3D-ConSwin-B (RGB) & 93.0 & 66.7 \\ distillation & 3D-ConSwin-S (RGB) & 91.9 & 65.1 \\ \hline
3D-ConSwin-B (HKS) & 89.6 & 62.9 \\
3D-ConSwin-B (RGB) + 3D-ConSwin-B (HKS) & 94.1 & 69.3 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Evaluation results on UCF-101 and HMDB-51. Figure 5: Comparison of the possibility of the Top 5 categories when the RGB network, WKS network, and WKS augmented RGB network are fed with clips of v_BrushingTeeth_g04_c03, v_Archery_g06_c01, v_ApplyLipstick_g04_c01, and v_Haircut_g02_c01, respectively. The original images and WKS labeled images are shown in the first line and the second line, respectively.