# Pyramid Spatial-Temporal Graph Transformer for Skeleton-Based Action Recognition
## 3 Method
### Graph Embedding  
The skeleton data used for action recognition consisted of several joints of humans, which were the 3D coordinates specifically. Unlike image data, skeleton graphs are more lightweight and more structurally relevant. For one action, they have three-dimensional \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Stage** & **Block** & **Channel** & **Shape** \\ \hline \multirow{2}{*}{0} & Graph Embedding & 3 & \((3\times T/4\times V)\) \\ \cline{2-4}  & ST Transformer Block \(\times 2\) & \(64\times 2\) & \((64\times T/4\times V)\times 2\) \\ \hline \multirow{2}{*}{1} & Graph Embedding & 64 & \((64\times T/8\times V)\) \\ \cline{2-4}  & ST Transformer Block \(\times 2\) & \(64\times 2\) & \((64\times T/8\times V)\times 2\) \\ \hline \multirow{2}{*}{2} & Graph Embedding & 128 & \((128\times T/16\times V)\) \\ \cline{2-4}  & Joint Transformer Block \(\times 2\) & \(128\times 2\) & \((128\times T/16\times V)\times 2\) \\ \hline \multirow{2}{*}{3} & Graph Embedding & 256 & \((256\times T/32\times V)\) \\ \cline{2-4}  & Joint Transformer Block \(\times 2\) & \(256\times 2\) & \((256\times T/32\times V)\times 2\) \\ \hline \hline \end{tabular}
\end{table} 
Table 1: The architecture parameters of the pyramid spatial-temporal graph transformer on the NTU-RGBD 60 dataset. Figure 2: Implementation of the pyramid spatial-temporal graph transformer (PGT). A pyramid architecture is designed for multi-scale feature learning. The model consists of four stages with similar modules to capture the long-range correlations in the spatial and temporal dimensions. Each stage has a graph embedding module and a spatial-temporal transformer block. The temporal dimension decreases exponentially with each stage, while the spatial dimension stays the same. In this way, the spatial granularity is retained, and temporal redundancy is removed. parameters, such as the frame, joint number and coordinate. To transform the skeleton graph into a uniform pattern, we propose a graph embedding module for the pyramid architecture. For traditional vision transformer models, a graph embedding module is utilized to obtain a series of feature tokens sent to the multi-head self-attention (MHSA) in the transformer block to reweight the local features based on long-range attention. However, for skeleton graphs, we introduce a GCN-based embedding module to enhance the spatial-temporal correlations as shown in Figure 3. Meanwhile, the graph embedding module plays a vital role in the information aggregation in the temporal dimension of the skeleton graphs. As shown in Figure 2, the temporal dimension shrinks with the stage progression, which forms a deep pyramidal structure for high-level semantic learning. Unlike the image-based vision tasks, we denote the 3D skeleton coordinate sequences as \(\mathbf{F}_{in}\in\mathbb{R}^{T\times V\times C}\), where \(T\), \(V\) and \(C\) represent the frame, joint number and coordinates, respectively. At the start of each stage \(i\), a spatial graph convolution is first applied to aggregate local information from neighbors according to the natural connection of joints. According to the network in [10], the vertex-domain graph convolution operation is as follows:  
\[f_{\text{out}}(v_{ti})=\sum_{v_{ij}\in N(v_{ti})}\frac{1}{Z_{ti}(v_{tj})}f_{in} \big{(}v_{tj}\big{)}\cdot\mathbf{w}\big{(}d_{ti}(v_{tj})\big{)} \tag{1}\]  
where \(v_{ti}\) denotes the \(i_{th}\) skeleton joint at time \(t\) and \(\mathbf{w}(\cdot)\) calculates the distance partitioning \(d_{ti}(\cdot)\) to determine the neighborhood of the graph convolution kernel. Cardinality \(Z\) shows the contribution of the neighbor joints.