# Whole-Body Keypoint and Skeleton Augmented RGB Networks for Video Action Recognition
## 3 Proposed Method
### Training Strategy  
The distillation strategy was first proposed by [PERSON] et al. [18], and aims to transfer knowledge from a cumbersome pretrained network to a simple lightweight network. We explore how to distill the prior knowledge of a teacher model to a student model on the target layer. As shown in Figure 2, \(f_{distill}\) is denoted as the distillation layer in the RGB classifier **f**; it transfers action recognition ability from \(\textbf{f}_{WKS}\) to **f** by performing a mean squared error (MSE) loss function between \(f_{WKS}\) and \(f_{distill}\):  
\[\mathcal{L}_{WKS}=\frac{1}{M}\sum_{i=1}^{M}\left\|f_{WKS}^{i}-f_{distill}^{i} \right\|^{2}, \tag{3}\]  
where \(f_{distill}^{i}\) is obtained from **f** fed with the \(i\)-th RGB clip, and the same clip that is marked with WKS is fed to \(\textbf{f}_{WKS}\), producing \(f_{WKS}^{i}\). As \(\mathcal{L}_{WKS}\) decreases gradually, \(f_{distill}\) will approximate to \(f_{WKS}\), and accordingly, the feature extraction method of f will be more similar to \(\textbf{f}_{WKS}\).  
Although the well-trained WKS model can provide knowledge that is learned from refined body information, they can also bring in noise interference, such as improper and low-discriminative features which can lead to confused recognition. Based on this consideration, we introduce a loss function to select the backpropagation path, which uses the following loss error to update the parameters:  
\[\mathcal{L}=\lambda_{1}\mathcal{L}_{WKS}+\lambda_{2}CrossEntropy(\textbf{f}( c^{(i)}),\hat{y}) \tag{4}\]  
In Equation (4), the RGB model's **f** is fed with clip \(c^{(i)},\hat{y}\) is ground truth label, and we use cross-entropy as our classification loss. Meanwhile, applying \(\lambda_{1}\) and \(\lambda_{2}\) to adjust the weights of two loss functions, \(\lambda_{1}\) can be computed as follows:  
\[\lambda_{1}=\begin{cases}\theta_{1}&if\ \textbf{f}_{WKS}\Big{(}c^{(i)} \Big{)}=\hat{y}.\\ 0&if\ \textbf{f}_{WKS}\Big{(}c^{(i)}\Big{)}\
eq\hat{y},\end{cases} \tag{5}\]  
We can infer from the Equation (5) that only the teacher model recognizes the actions successfully, namely \(\textbf{f}_{WKS}\Big{(}c^{(i)}\Big{)}=\hat{y}\), then the features of a teacher model are participated in the backpropagation course through the parameter \(\theta_{1}\).  
The updating of parameters connected to distillation \(f_{distill}\) can be described as follows:  
\[W_{d}^{+}=W_{d}-\eta(\lambda_{1}\frac{\partial\mathcal{L}_{WKS}}{\partial W_{ d}}+\lambda_{2}\frac{\partial\mathcal{L}_{CE}}{\partial\phi}\frac{\partial\phi}{ \partial W_{d}}), \tag{6}\]where \(W_{d}\) are the parameters before updating, and the function map from distillation to final output is represented as \(\phi:f_{distill}\rightarrow\mathbf{f}(c^{(i)})\). As observed from Equation (6), the parameters will be modified by \(\mathcal{L}_{WKS}\) only when the \(\lambda_{1}>0\), in this case, the \(\mathbf{f}_{WKS}\) recognize the action correctly and target features are considered as sufficiently discriminative. From this method, we can effectively prevent the model from learning noisy information.