# Topology-Embedded Temporal Attention for Fine-Grained Skeleton-Based Action Recognition
## 2 Related Work  
### Skeleton-Based Action Recognition  
Researchers have conducted a lot of work on skeleton-based action recognition based on the three main lines of RNNs, CNNs, and GCNs.  
Given that RNNs are able to model well the long-term context-dependent information of time series, they were first used by [PERSON] et al. [8] for this task. Based on this work, some innovative work was derived. For example, to pay different levels of attention to the outputs of different frames, [PERSON] et al. [13] proposed an end-to-end spatial and temporal attention model based on RNNS, which achieved attention learning with the distinguishing joints of the bones in each input frame; In the work by [PERSON] et al. [9], SR-TSL was proposed to capture the high-level spatial structure information within each frame and the detailed temporal dynamics of the modeled skeleton sequence. To process data with noise to enhance the effectiveness of learning sequences, the work [10] by [PERSON] et al. introduced a new gating mechanism for LSTMs. A novel attention-enhanced graph convolutional LSTM network (AGC-LSTM) was proposed in [7], which could effectively extract discriminative spatiotemporal features.  
CNNs were first applied to skeleton-based action recognition in the work by [PERSON] et al. [15]; then, [PERSON] et al. integrated Lie groups into a deep network architecture to complete action recognition [20].  
The spatiotemporal graph convolutional network was first proposed to model skeleton data by [PERSON] et al. [3]. Then, some creative improvements were derived based on this work. For example, to improve the flexibility of constructing graph structures and make full use of skeleton information, [PERSON] et al. made improvements to GCN and proposed new network architecture, 2s-AGCN [4]. [PERSON] et al. [12] used depth progressive reinforcement learning, extracting the most informative frames for GCN; to adaptively learn the intrinsic higher-order connections of skeleton joints, [PERSON] et al. [14] proposed the STGR scheme, which could be seamlessly integrated into graph convolution networks. Introducing self-attention to action recognition [15], three different self-attention networks (SAN V1, V2, V3) were proposed to describe long-range associations or capture long-term spatiotemporal relationships, improving the ability to extract high-level semantics. To enhance the performance of graph convolution, in ContextAware graph convolution [16],  
Figure 1: Topology-embedded temporal attention. Different-colored bars represent the responses of the topology relationships of the skeleton data on each frame in the time dimension. Thus, the final feature is more distinct for classification.  
the author proposed a context-aware graph convolution network (CA-GCN) to model long-range dependencies between joint points in spatial dimensions. In [17], a spatiotemporal transformer network was introduced to understand intra-frame interactions between different body parts and inter-frame correlations.  
In addition to the above three mainstream methods, there were some other works, such as semantic-guided [24], which combined GCN and CNN to build a simple and effective semantic-guided neural network and PoseC3D [25]; it relies on a 3D heat map stack as the basic representation of the human skeleton.  
Summarizing the above research process, it is not difficult to see that the improvements of related works around RNNs, GCNs, and other works, involved different methods to continuously strengthen the modeling abilities of the networks in the spatial and temporal dimensions, to better express the data samples. In our work, we also abided by such a principle and had a similar purpose to [12; 13] (in that the model pays more attention to the temporal dimension). However, the difference is that we used the method of topology-embedding to gain attention; our module can be plugged into GCN-based models and improve their performances.