## 3 Proposed Framework
### The Skeleton Stream
At block \(l\), we compute the query/key/value of each tubelet from the last layer's embedding vector \(z^{(l-1)}\):  
\[q^{(l,a)}_{(p,t)}=W^{(l,a)}_{Q}LayerNorm(z^{(l-1)}_{(p,t)})\in R^{D_{h}}, \tag{3}\]  
\[k^{(l,a)}_{(p,t)}=W^{(l,a)}_{K}LayerNorm(z^{(l-1)}_{(p,t)})\in R^{D_{h}}, \tag{4}\]  
\[v^{(l,a)}_{(p,t)}=W^{(l,a)}_{V}LayerNorm(z^{(l-1)}_{(p,t)})\in R^{D_{h}}, \tag{5}\]  
where \(a=1,...,A\) is an index over multiple attention heads, \(A\) denotes the total number of attention heads, \(p=1,...,N\) denotes the spatial location, and \(t=1,...,\lfloor\frac{F}{f}\rfloor\) depicts the temporal index. **Spatial Transformer Encoder. ** In each block, we compute the attention of tokens in the same temporal index firstly. The self-attention weights are as follows:  
\[\alpha^{(L,a)space}_{(p)}=softmax\left(\frac{q^{(L,a)^{\top}}_{(p)}}{\sqrt{D_ {h}}}\left[k^{(l,a)}_{(0)}\left\{k^{(l,a)}_{(p^{\prime})}\right\}_{p^{\prime} =1,...,N}\right]\right), \tag{6}\]  
where \(D_{h}=\frac{D}{A},D=(N+1)\times d\). The encoding \(z^{(l)}_{(p)}\) at block \(l\) is obtained as follows:  
\[s^{(l,a)}_{(p)}=\alpha^{(l,a)}_{(p),(0)}v^{(l,a)}_{(0)}+\sum_{p^{\prime}=1}^ {N}\alpha^{(l,a)}_{(p),(p^{\prime})}v^{(l,a)}_{(p^{\prime})}, \tag{7}\]  
\[z^{(l)space}_{(p)}=W_{O}\left[ \begin{array}{c}s^{(l,1)}_{(p)}\\ \vdots\\ s^{(l,A)}_{(p)}\end{array} \right]+z^{(l-1)}_{(p)}, \tag{8}\]  
where \(s\) is the concatenation of the vectors from all head. Figure 3: The examples of generated joint heatmap, limb heatmap, and joint–limb heatmap. **Temporal Transformer Encoder. ** We obtain the frame-level representation \(z_{(p)}^{(l)space}\), which can be seen as the classification token \(z_{cls}^{L^{\prime}space}\). Then, we represent the frame-level representation as \(h_{i}\in R^{d}\) and concatenate all the frame-level tokens:  
\[H=\left\{h_{1},...,h_{\lfloor\frac{F}{f}\rfloor}\right\}\in R^{\lfloor\frac{F} {f}\rfloor\times d}. \tag{9}\]  
The encoding \(H\) is then fed into the temporal Transformer encoder which consists of \(L_{t}\) layers to compute the attention from different temporal indexes, and new key/query/value vectors are obtained by using Equations (3)-(5). The temporal attention is computed by the following formula:  
\[\alpha_{(p,t)}^{(L_{t})time}=softmax\left(\frac{q_{(p,t)}^{(L_{t})^{\top}}}{ \sqrt{D_{h}}}\left[k_{(0,0)}^{(L_{t})}\left\{k_{(p,t^{\prime})}^{(L_{t})} \right\}_{t^{\prime}=1,...,\lfloor\frac{F}{f}\rfloor}\right]\right). \tag{10}\]  
The same as spatial attention, Equations (7) and (8) are adapted to obtain the encoding. Then, the encoding is passed to Equation (11). Then, we obtain the final classification token which will be send to an MLP head which consists of two linear projections separated by a GELU [29]:  
\[z_{(p)}^{(l)}=MLP(LayerNorm(z_{(p)}^{\prime}))+z_{(p)}^{\prime}. \tag{11}\]  
Finally, the classification results are obtained by a classification head consisting of a cross-entropy loss function and a linear projection, where the loss function can be expressed as the following formula:  
\[Loss=-\sum_{i=0}^{C-1}y_{i}log(p_{i}), \tag{12}\]  
where \(C\) means the number of categories, \(i\) means the \(i\)-th sample, \(y_{i}\) means the one-hot vector of the sample, and \(p_{i}\) means the probability distribution result of the sample. The detailed structure is shown in Figure 4: