# Whole-Body Keypoint and Skeleton Augmented RGB Networks for Video Action Recognition
## 3 Proposed Method
### Confidence-Weighting Aggregation Function  
We introduce the CWAF, a simple but effective method to adaptively assign weights for each output according to their confidence. The inspiration comes from the observation that more credible input is expected to produce more distinct probabilities over classes. A comparison of the output as the classifier which makes a right or wrong classification is shown in Figure 4. These examples were chosen from the videos v_Biking_g02_c02 and v_Haircut_g02_c04, which are classified correctly and incorrectly, respectively. We can easily determine that category probabilities corresponding to a correct recognition represent minor fluctuations, resulting in a more reliable output. To assign weights for each clip-level result, we first need to design an estimator \(\mathbf{h}(\mathbf{f}(c^{(i)}))\in[0,1]\) to score the confidence; thus, we propose a confidence scoring network. The confidence of output \(\mathbf{f}(c^{(i)})\) is mainly determined by the relative difference among the highest class scores and others. To reduce the distribution from the absolute positions of the values, we directly sort the output values in order from large to small, keeping only the relative difference information. We designed the confidence scoring network to be composed of multiple layers of a fully connected neural network. Several architectures have been examined regarding the balance of speed and accuracy, and a simple four-layer network has been found to be completely competent for this task. For a video that consists of clips \(\{c^{(1)}\}\), \(c^{(2)}\), \(c^{(3)}\),..., \(c^{(i)}\}\), we estimate each output \(\mathbf{f}(c^{(i)})\) with confidence scoring network and produce scores \(\{s_{1},s_{2},s_{3},\)..., \(s_{i}\}\) to represent confidence of clip-level results. Video-level results are obtained by:  
\[Y_{v}=\sum_{i}w_{i}\mathbf{f}(c^{(i)}), \tag{7}\]  
where \(w_{i}\) is the weight for each clip-level result. We attempt to assign higher weights for more confident clip-level results and enlarge the margin between the weight of lower-ranked and higher-ranked clip results. Therefore, a quadratic nonlinear function is applied to arrange the weight distribution:  
\[w_{i}=\Bigg{(}\frac{s_{i}-\text{min}(\{s_{i}\}_{i=1}^{N})}{\text{max}(\{s_{i }\}_{i=1}^{N})-\text{min}(\{s_{i}\}_{i=1}^{N})}\Bigg{)}^{2}, \tag{8}\]  
Figure 4: Comparison of the output when the classifier makes (**a**) right and (**b**) wrong classifications. We set the distillation layer at fc layer in both the Swin-B-based network and Swin-S-based network. It can be found that when the Swin-S based network is training, the dimensions between distillation layer \(f_{WKS}\in\mathbb{R}^{1024}\) and \(f_{distill}\in\mathbb{R}^{768}\) are not matching. Therefore, a liner layer is designed concatenated after the fc layer to connect the 768-dim layer to 1024-dim distillation layer, as shown in Table 1.  
To obtain a well-trained WKS model, we first generated a WKS dataset by marking the UCF-101 and HMDB-51 datasets using MMPose [17]. The marking process followed the official instructions. It is noted that not all samples in the UCF-101 and HMDB-51 datasets can be well marked, and we filtered the noisy samples in which keypoints' mean confidence scores were below 0.4. At the training stage, only qualified samples were used. For clip-level training and inference, to satisfy the requirement of memory consumption, we divided each video to a series of clips with 16 frames, and spatially resized to 112. At the training stage, to minimize overfitting, several data augmentation techniques were applied. From input frames, we first decided a base position and crop scale, then cropped the input frame in terms of the base position and the crop scale. The scale randomly sampled from set \(\left\{1,\frac{1}{2^{1/4}},\frac{1}{\sqrt{2}},\frac{1}{2^{3/4}},\frac{1}{2} \right\}\). Next, we randomly flipped half of the clips. These tricks are all operated in a spatial domain, and we also applied a temporal technique.