## 3 Proposed Framework
### The Skeleton Stream  
Readily available large amounts of RGB videos captured by RGB cameras and progressively improved pose estimation algorithms have led to many recent works using pose estimation algorithms on RGB videos to acquire skeleton data. The authors in [10] conducted a review on key aspects of pose extraction methods to find the best practice for the final recognition accuracy. In this paper, following [10], we obtain skeleton data by adapting HRnet, a 2D Top-Down pose estimator proposed by [27]. **Skeleton joint representation. ** By applying the pose estimator, we obtain the locations of \(K\) human joints from a frame \(F\) of size \(W\times H\times 3\), where \(K\) is the number of skeleton joints, and \(W\) and \(H\) are the width and height of the frame. For \(k\)-th skeleton joint, we obtain the representation \((x_{k},y_{k},c_{k})\), where \((x_{k},y_{k})\) is the coordination of the joint in the frame, and \(c_{k}\) is the confidence score of the joint. **Pseudo skeleton heatmaps. ** We describe a 2D pose estimated from a frame as a heatmap of size \(K\times W\times H\). We can directly use the heatmap produced by the HRNet and make it zero-padded to match the bounding box given by the origin frame. So the joint heatmap \(J\) composing \(K\) Gaussian maps centered as every point can be described as:  
\[J_{kij}=e^{\frac{(i-x_{k_{k}})^{2}+(j-y_{k_{k}})^{2}}{2\sigma^{2}}}\times c_{ a_{k}} \tag{1}\]  
where \(\sigma\) controls the variance of Gaussian maps, \((x_{a_{k}},y_{a_{k}})\) means the coordinate of the point \(a_{k}\), and \(c_{a_{k}}\) means the confidence score of joint \(a_{k}\). The heatmap \(L\) of the limb between two joints \(a_{k}\) and \(b_{k}\) can be described as:  
\[L_{kij}=e^{-\frac{D((i),j)\text{seg}(i_{k},b_{k})|^{2}}{2\sigma^{2}}}\times min (c_{a_{k}},c_{b_{k}}), \tag{2}\]  
where \(seg\) means the segment between the two joints, and the function \(D\) calculates the distance from the point \((i,j)\) to the segment \([(x_{a_{k}},y_{a_{k}}),(x_{b_{k}},y_{b_{k}})]\). From the above formulas, it can be seen that compared to the skeleton-based methods which treat joints and limbs in graph form, heatmaps are easy and low-cost to solve the multi-person cases by accumulating the k-th Gaussian map of every person without enlarging the heatmap. The example of the generated heatmaps are shown in Figure 3.  
**Reformulating to the input. ** To feed the 2D heatmaps to RGBSformer, we should stack all the heatmaps along the temporal dimension. In this process, the heatmap does not need to be as large as the original frame because most of the time people only occupy a small part of the video frame, and the zero-padded, blank background will only occupy more storage space and may affect the performance of the network. Hence, we crop all the frames to the size of the smallest detection bounding box which can envelop all 2 Dposes. The human-centered cropping is also helpful for the backbone network to extract the motion feature precisely. After cropping, we have acquired the 3D heatmap videos which are the input of Transformer. **Video Decomposition. ** Following [18], we denote a heatmap video as \(V\in R^{F\times H\times W\times 3}\), where \(H\) is the height of one heatmap, and \(W\) is the width, \(F\) means there are \(F\) heatmaps in \(V\), and then we separately decompose the heatmap video into \(N\) non-overlapping spatio-temporal \"tubes\" [18]\(x_{1},x_{2},...,x_{N}\in R^{f\times h\times w\times 3}\), where \(N=\lfloor\frac{F}{f}\rfloor\times\lfloor\frac{H}{h}\rfloor\times\lfloor\frac{ W}{w}\rfloor\). Next, we linearly map each tube \(x_{i}\) into a token \(z_{i},z_{i}=Ex_{i}\). Finally, we concatenate all tube tokens \(z_{i}\) into a vector \(z^{0}\). The same as BERT [28], we add a special learnable vector \(z_{cls}\in R^{d}\) in the first position of the token sequence to represent the embedding of the classification token, and a positional embedding \(p_{pos}\in R^{(N+1)\times d}\) is also added to this sequence. In fast pathway, there are \(L^{\prime}\) self-attention blocks.