# Topology-Embedded Temporal Attention for Fine-Grained Skeleton-Based Action Recognition
## 3 Methods
### Topology-Embedded Temporal Attention
To model the connections of the bone points, in the temporal and spatial dimensions, we set \(t_{s}=2\) and \(K_{M}=2\). Although the function was limited to modeling neighboring elements, the capacity to represent both the temporal and spatial dimensions is adequate for topological relational embedding. Specifically, in our experiment, the formula was achieved by the convolution operator. Moreover, the influence of the formula settings is shown in Section 4. Based on the topology relationship modeled by the above formulation: the output \((F_{topology})\) of functions \(\Phi(\cdot)\) and \(\Omega(\cdot)\) are all in the space of \(\mathbb{R}^{T^{1}\times T^{1}\times M^{1}}\). We first transposed one of them to \(\mathbb{R}^{T^{1}\times M^{1}\times T^{1}}\), and then multiplied them. Thus, the output of the matrix multiplication is in the space of \(\mathbb{R}^{T^{1}\times M^{1}\times M^{1}}\). Moreover, it was used as the representation of the topology relationship among the temporal dimensions. Finally, the topology relationship was embedded as excitation features of attention with function \(\theta\). The function is formulated as:  
\[F_{embedded}=\left\{ \begin{array}{c}AvgPool(F_{topology})\\ MaxPool(F_{topology})\end{array} \right. \tag{3}\]  
The primary goal of feature embedding is to gather and capture effective data in topological relations, as well as to locate the features that best characterize the topological relations. Only in that way, could they be used as effective excitation features of attention. Therefore, we employed average pooling to aggregate topological information or used max pooling to capture the distinction on each frame, and at the same time reduce the feature dimensions without introducing any parameters to the network structure. The experimental findings of the performances of the two distinct pooling approaches on the verification set are shown in Section 4. Figure 3: Schematic diagram with different \(K_{M}\); \(\left(\mathbf{A}\right)\) with \(K_{M}=1\), which only raises the dimensions of the bone points on each frame; \(\left(\mathbf{B-D}\right)\) with different \(K_{M}\) settings, which model the relationships of the bone points in local areas of spatial and temporal dimensions. Attention TransformThe embedded feature \(F_{embedded}\) in \(\mathbb{R}^{T^{1}\times 1\times 1}\) is first reshaped to \(\mathbb{R}^{T^{1}}\). Moreover, it is transformed to \(\mathbb{R}^{T}\) with functions \(\delta(\cdot)\) and \(\gamma(\cdot)\), formulated by:  
\[A_{attention}=\sigma(MLP(F_{embedded})) \tag{4}\]  
Functions \(\delta(\cdot)\) and \(\gamma(\cdot)\) are critical to the overall TE-TAM since they are intended to turn excitation features into attention. Therefore, we employed the multilayer perceptron (MLP), which contained powerful fitting abilities. Moreover, there were several choices for the activation functions \(\sigma(\cdot)\) and \(\sigma_{1}(\cdot)\) in MLP, such as sigmoid, ReLU, and tanh. We chose ReLU and sigmoid as the two activation functions according to our experiments in Section 4. Residual ConnectionAs shown in the cyan block in Figure 2, the third branch of the input feature is a very simple operation and works efficiently. Moreover, it is formulated as:  
\[f_{out}=\Psi(f_{in}) \tag{5}\]  
To improve the performance of the framework and constraint the output feature, we added the residual connection to our framework. In addition, as the output and input does not change the distribution of the data, we used the residual connection with function \(\Psi\), which provided a parameter that was trained end-to-end, to adjust the output dynamically.