# Topology-Embedded Temporal Attention for Fine-Grained Skeleton-Based Action Recognition
###### Abstract  
In recent years, graph convolutional networks (GCNs) have been extensively applied in numerous fields, demonstrating strong performances. Although existing GCN-based models have extraordinary feature representation capabilities in spatial modeling and perform exceptionally well in skeleton-based action recognition, they work poorly for fine-grained recognition. The key issue involves tiny distinctions between multiple classes. To address this issue, we propose a novel module named the topology-embedded temporal attention module (TE-TAM). Through embedding the temporal-different topology modeled with local area skeleton points in spatial and temporal dimensions, the TE-TAM achieves dynamical attention learning for the temporal dimensions of distinct data samples, to capture minor differences among intra-frames and inter-frames, making the characteristics more discriminating, and increasing the distances between various classes. To verify the validity of the proposed module, we inserted the module into the GCN-based models and tested them on FSD-30. Experimental results show that the GCN-based models with TE-TAMs outperformed the property of pred GCN-based models. topology embedding; temporal attention; GCNs; fine-grained recognition +
Footnote †: journal: Applied  
the accuracy of fine-grained action recognition. Although some new methods for modeling the temporal dimensions are proposed, for example, by capturing long-term context information [8] or modeling inter-frame relationships in the temporal dimensions [12; 17], they are still insufficient in fine-grained action recognition and are not for GCN-based models. Obtaining ideas from fine-grained image classification and the attention mechanism, in this paper, we propose a novel module named topology-embedded temporal attention, which can achieve dynamic and effective temporal attention learning of skeleton data samples end-to-end (as shown in Figure 1) by embedding the topology-temporal relationship modeled with local area skeleton points in spatial and temporal dimensions, and then refine the input features with learned attention. Our proposed TE-TAM first introduces two mutually independent functions to raise the dimensions of input features, and then establishes the dependency relationship of input features in local areas (both temporal and spatial dimensions). Following that, by performing a matrix multiplication operation on the exporting dependency relation, topology relationship data in temporal dimensions are then obtained, which contain more discriminative data than only modeling in spatial dimensions, as they include not only the spatial topology relationships among bone points in a specific frame, but also the topology relationships among bone points in adjacent frames in the same sampling space. The topology relationship information is then aggregated and embedded as the excitation features of attention. After that, multi-layer perceptron is utilized to transform and activate the excitation features, to obtain the attention of the sample in the temporal dimension and capture the small movement changes among similar movements. Finally, from the perspective of signal processing, we add a residual connection [19] branch to the TE-TAM, introducing a small number of parameters to perform a simple transformation on the input and then fuse with the output (thereby restricting the output signal). To some extent, our proposed module can make up for the deficiency of the existing GCN-based models: it can capture the difference off skeleton data in the temporal dimension, to improve the performance of the existing GCN-based model network in the fine-grained skeleton-based action recognition. Inserting TE-TAM into existing GCN-based models, we conducted a large number of experiments on FSD-30; the proposed TE-TAM achieved remarkable performances and the experimental results demonstrated the effectiveness of the TE-TAM.