# Whole-Body Keypoint and Skeleton Augmented RGB Networks for Video Action Recognition
## 3 Proposed Method
### Swin Transformer-Based RGB Network  
The Swin transformer is a powerful framework for the imaging task, but it has no ability to extract temporal features of videos. It first divides the input image with a size of \(\mathrm{H}\times\mathrm{W}\times 3\) into non-overlapping patches with a size of \(\frac{\mathrm{H}}{4}\times\frac{\mathrm{W}}{4}\times 48\); this part is referred to as \"patch partition\" in study [19]. Next, a liner projection function is applied on it, and the input matrix is shifted to a sequence that is calculated using a multi-head self-attention mechanism. To extract temporal information, we utilize a 3D convolutional module before the self-attention mechanism, yielding the module architecture shown in Figure 3. It is fed with fixed-length clips of F RGB frames with spatial resolution \(\mathrm{H}\times\mathrm{W}\). We set F as 16; thus, after two down-sampling operations, four feature maps are produced in the temporal dimension: \([f_{1},f_{2},f_{3},f_{4}]\), each \(f_{i}\in\mathbb{R}^{96\times\frac{\mathrm{H}}{4}\times\frac{\mathrm{W}}{4}}\). Then they are concatenated as one \(f_{W}\in\mathbb{R}^{96\times\frac{\mathrm{H}}{2}\times\frac{\mathrm{W}}{2}}\). Such operations map the temporal motion information into the spatial domain.  
During the inference, two 3D convolution operations are performed on the input clip, and the temporal dimension is finally reduced to \(1/4\) of its original size. The convolution kernels' size in the temporal dimension are 7 and 3, respectively. Thus, the receptive field region in the temporal dimension of the final output feature map is of size 11. This means that the concatenated feature map \(f_{w}\) can access all of the temporal information.  
It was demonstrated in study [19] that the computed objects of the self-attention mechanism in the Swin transformer are unlike those in ViT [46]; its calculations are performed in a window. The self-attention computations in the windows are defined as follows:  
\[Attention(Q,K,V)=SoftMax(QK^{T}/\sqrt{d}+B)V, \tag{2}\]  
where \(Q,K,V\in\mathbb{R}^{M^{2}\times d}\) are the _query_, _key_, and _value_ matrices; \(d\) is the _query_/_key_ dimension, and \(M^{2}\) is the number of patches in a window. \(Q,K,V\) is obtained from a liner function which operated on the pixels in a window, and we can infer from Equation (2) that the spatial features in the window are fully compared and without constraints by the size of the  
Figure 3: Architecture of a 3D convolutional module. Two 3D convolutional operations are included in the module. After layers of Bn and ReLU, feature map concatenation module concatenate four feature maps into one.  
receptive field, which occurred in convolutional operation. The Swin transformer utilizes a window architecture to reduce the computational complexity, but this architecture lacks the computation between windows. Therefore, a method that shifts the window at the next layer is proposed, and by continuously changing the position of the window, the features in different area can be connected and compared. Benefiting from this architecture, in our situation, the temporal differences among \(\{f_{1},f_{2},f_{3},f_{4}\}\) can be effectively compared and captured. In addition, because the size of the feature map is reduced in the top layers, the receptive field of each window is relatively expanded, so that the temporal features are more comprehensively extracted.  
The complete architecture of our proposed framework is illustrated in Figure 2, in which the produced features \(f_{W}\) are fed into the network that is formed by the stacked Swin transformer block and patch merging [19]. Considering that the features of the top layers represent high-level global information [31], we designed the distillation layer \(f_{distill}\) to be located after the last Swin transformer block. The logit that is obtained by a fully connected layer over \(f_{distill}\) is then converted by the SoftMax function, and the final category probabilities are obtained.