# Whole-Body Keypoint and Skeleton Augmented RGB Networks for Video Action Recognition
## 5 Conclusions and Discussion  
We introduced a new data modality, which is marked with WKS labels, strengthening the difference between the body and background. We also introduced a novel framework that attempts to use whole-body labeled images for model training, and transfers this recognition ability to the RGB network. The experimental results show that the data provided more refined body information for classification, and by distilling the high-level semantic features based on refined body information to the RGB model, the original RGB model can capture more comprehensive action information. Irrelevant information such as human clothing and appearance can be further filtered, which makes the model well-suited for the classification of actions that focus on pose. Some categories which shared a similar background can be classified more accurate by focusing on pose features, such as FrontCrawl and BreastStroke. Inspired by the success of transformers for vision tasks, we designed an architecture that takes advantage of both the 3D CNNs and Swin transformer to extract spatiotemporal features, which has demonstrated advanced performance. By analyzing the confidence of the clip-level results, we designed an aggregation method to assign more rational weights to each clip output. The experimental results demonstrate that our methods achieve favorable accuracy on the UCF-101 and HMDB-51 datasets. Of note, with limited hardware resources, we temporarily have no ability to perform our experiment on large datasets, such as Kinetics, therefore, we will investigate the performance of our strategy on them in the future. Conceptualization, Z.G. ; Funding acquisition, S.Y. ; Project administration, S.Y. ; Writing--original draft, Z.G. All authors have read and agreed to the published version of the manuscript. \begin{table}
\begin{tabular}{c c c c c} \hline \hline  & \multirow{2}{*}{**Method**} & \multirow{2}{*}{**Pretrain Dataset**} & \multicolumn{2}{c}{**Top-1 Accuracy (\%)**} \\ \cline{3-5}  & & & **UCF-101** & **HMDB-51** \\ \hline \multirow{8}{*}{One-Stream} & C3D [30] & Sports-1M & 82.3 & 51.6 \\  & Temporal Segment Network (RGB) [22] & ImageNet & 85.2 & 51.0 \\  & 3D-ResNet-152 [51] & Kinetics & 89.6 & 62.4 \\  & R(2+1)D [20] & Sports-1M & 93.6 & 66.6 \\  & 3D CNN Architecture [52] & None & 87.4 & - \\  & CoViAR [12] & ILSVRC 2012-CLS & 90.4 & 59.1 \\ \cline{2-5}  & Ours (3D-ConSwin-B-base) & Kinetics & 92.2 & 66.0 \\  & Ours (3D-ConSwin-B-full) & Kinetics & 93.4 & 67.2 \\ \hline \multirow{4}{*}{Multi-Stream} & Two-stream CNN [29] & ImageNet & 88.0 & 59.4 \\  & 13D [3] & ImageNet + Kinetics & 97.8 & 80.9 \\ \cline{1-1}  & ResNet101 BERT [21] & Kinetics & 97.9 & 83.6 \\ \cline{1-1}  & DMC-Net [11] & ILSVRC 2012-CLS & 92.3 & 71.8 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Comparison with existing methods on UCF-101 and HMDB-51. **Funding:** The authors disclose the following financial support for the research, authorship, and/or publication of this article. This work was supported by the Major Special Science and Technology Project of Sichuan Province, grant number 2018 GZDZX0024 and the Sichuan Science and Technology Program, grant number 2020 YFG0288. **Institutional Review Board Statement:** Not applicable. **Informed Consent Statement:** Not applicable. **Data Availability Statement:** The data that support the findings of this study are openly available at [[https://www.crcv.ucf.edu/data/UCF101.php](https://www.crcv.ucf.edu/data/UCF101.php)]([https://www.crcv.ucf.edu/data/UCF101.php](https://www.crcv.ucf.edu/data/UCF101.php)), accessed on 3 May 2022 and [[https://serve-lab.clps.brown.edu/resource/hmdb-a-large-human-motion-database/#Downloads](https://serve-lab.clps.brown.edu/resource/hmdb-a-large-human-motion-database/#Downloads)]([https://serve-lab.clps.brown.edu/resource/hmdb-a-large-human-motion-database/#Downloads](https://serve-lab.clps.brown.edu/resource/hmdb-a-large-human-motion-database/#Downloads)), accessed on 3 May 2022, reference number [48,49], respectively.