# Pyramid Spatial-Temporal Graph Transformer for Skeleton-Based Action Recognition
## 1 Introduction  
Skeleton-based action recognition has been a hot topic with vast application areas: surveillance systems, human-computer interaction, medical care assistance, etc. Due to the development of depth cameras and pose estimate algorithms, human skeletons are much easier to obtain as efficient data. Therefore, many skeleton-based action recognition algorithms have been proposed in recent years. Compared with the RGB images and videos, the skeleton data consisting of 3D joint coordinates show advantages in the following aspects: being robust to illumination, stable to the background and lightweight. The semantic information of skeleton data was extracted with handcrafted features in the early years [1]. [PERSON] et al. [1] adopted lie group theory in this task for the features of the joints and angles. However, with the development of deep learning, many works introduced convolutional neural networks (CNNs) and recurrent neural networks (RNNs) into their models to learn powerful features with high-level semantics. The CNN-based methods construct the skeleton joints into a 2D pseudo-image, and the popular CNN architectures are applied to classify the images into a special spectral form [2, 3, 4, 5]. RNN-based methods treat the skeletons as a series of feature vectors and construct a deepnetwork in the spatial domain with long short-term memory (LSTM) units [6; 7; 8; 9]. Therefore, the dynamic changes over a long period of time are captured. Then, a series of graph convolution networks (GCNs) was proposed and showed steps of improvement compared with previous works [10; 11; 12; 13; 14; 15]. [PERSON] et al. [10] were the first to propose the graph convolution operation and treat the skeletons as non-Euclidean graphs. The key to graph convolution lies in constructing the learnable graph convolution kernels. Based on this work, [PERSON] et al. [11] proposed a two-stream graph convolution network with an adaptive mechanism to learn the adjacency matrix. [PERSON] et al. [12] further presented a shift graph convolution operation with a similar backbone. Inspired by previous works, [PERSON] et al. [14] introduced the attention mechanism into their two-stream architecture to improve the performance with more learnable parameters. Previous GCN-based methods mainly explored a dynamic network based on a graph convolution backbone proposed by [10]. The graph convolution kernel is fixed, and the network is suitable for the static topology of graphs. However, the spatial-temporal correlations between all nodes are essential. At the same time, GCN-based methods adopt a cascading spatial graph convolution and temporal convolution layers to learn this information, as shown in Figure 1. As graph convolution operation is inferred from CNNs, it means GCNs will also come across the problem of the local receptive field. As far as we are concerned, these multi-layer GCN-based methods perform poorly in the dynamic fusions of the global joints. For example, the correlation between the joint of the hand in the first frame and the joint of the foot in the last frame cannot be measured in most cases, which may be crucial spatial-temporal information for the perception of actions. Therefore, we propose a new backbone based on transformers to learn the long-range dependencies in the spatial and temporal domains. The transformer architecture was proposed in recent years and has shown itself to be a strong backbone for natural language processing (NLP) tasks. For computer vision (CV) tasks, the vision transformer (ViT) has also demonstrated more robust effects compared with CNNs or RNNs. Inspired by this architecture, we propose a pyramid transformer backbone named PGT, which can provide multi-scale features with a four-stage architecture for skeleton graph data. The long-range, multi-scale spatial-temporal correlations are learned through an alternate connection structure of the GCN module and the transformer module.