# Topology-Embedded Temporal Attention for Fine-Grained Skeleton-Based Action Recognition
## 3 Methods
### Topology-Embedded Temporal Attention  
As shown in Figure 2, we provide a straightforward topology-embedded temporal attention framework. To obtain the topology relationships of skeleton data for each frame or multiple neighboring frames and bone points, we first modeled the input feature in both spatial and temporal dimensions. The derived topology relationship was then embedded as the attention excitation features. Following that, the excitation features were converted and transferred to the time dimension space. Finally, the learned attention was applied to modify the input feature, which was then fused with the transformed input feature to produce the final output. Specifically, the TE-TAM is composed of three parts: (1) functions \(\Phi(\cdot)\) and \(\Omega(\cdot)\) are used to finish the topology modelings of bone points while function \(\theta(\cdot)\) is used to embed the topology relationship. (2) Attention transformation is conducted by functions \(\delta(\cdot)\) and \(\gamma(\cdot)\), which transform the excitation features to the temporal space. (3) The residual connection feature is generated by function \(\Omega(\cdot)\) with an input feature. Therefore, the TE-TAM is formulated as  
\[Out=\gamma(\sigma(\theta(\Phi(X)\bigotimes\Omega(X))))\bigodot X+\Psi(X) \tag{1}\]  
Given an input of \(X\in\mathbb{R}^{C\times T\times M}\), the output is \(Out\in\mathbb{R}^{C\times T\times M}\), which is refined by learned attention and constrained by a residual connection. The topology-embedded temporal attention framework is introduced in detail as follows: _Topology Embedding_. The embedding process is shown in the blue block in Figure 2. We first employed two independent functions to expand the dimensions of the input features along channel C, while modeling the dependent relationships between the bone points in both spatial and temporal dimensions. The modeling method (functions \(\Phi(\cdot)\) and \(\Omega(\cdot)\)) are formulated as:  
\[f_{out}(M_{it_{s}})=\sum_{M_{j_{t}}\in N_{it_{s}}}^{K_{M}}\frac{1}{Z_{j_{t}j_ {t}}}f_{in}(M_{j_{t}})\cdot\omega\cdot(l_{it}(M_{j_{t}})) \tag{2}\]  
where \(M_{it_{s}}\), \(M_{j_{t}}\) denote the \(i_{th}\), \(j_{th}\) bone point data in the temporal dimension of frame \(t_{s}\) (\(t_{s}\) is a set, \(t_{s}=\{s=1,\ldots,T\}\), if \(s=1\), \(t_{1}=\{frame_{1}\}\), if \(s=2\), \(t_{2}=\{frame_{1},frame_{2}\}\); thus, the input bone point subset extends to the temporal dimension), while \(N_{it_{s}}\) is a bone points  
Figure 2: The structure of the proposed topology-embedded temporal attention module. Topology embedding is aimed at modeling the effective topology relationship, embedding as excitation features. Attention transformation completes the transformation of the excitation to the attention score. The goal of the residual connection is to restrict the output by the transformed input. subset, consisting of the neighbors of \(M_{i_{t}s}\). Moreover, \(K_{M}\), which is similar to the kernel size of the convolution operator, decides the size of the subset. The normalizing term \(Z_{i_{t}s_{t}s_{t}}\)\(=\)\(|(M_{k_{t}s_{t}}|_{i_{t}}(M_{k_{t}s_{t}})=\)\(l_{i_{t}s_{t}}(M_{i_{t}s_{t}})\)\(]\) involves balancing the contributions of different subsets to the output. Moreover, \(\omega\) is a weighting function to assign weights for bone points that are spatially and temporally adjacent. \(l_{i_{t}}\) is a mapping function, which aims to expand the dimensions of each bone point of the input features. The given formula is a generalized version. We can change the settings of the function to express different functionalities. For example, if we set the \(t_{s}=1\) and \(K_{M}=1\), as shown in Figure 3A, the function will only model a single bone point in each frame. If we set \(t_{s}=2\) and \(K_{M}=1\), demonstrated in Figure 3B, the dependencies of the bone points between the adjacent frames will be modeled. As we set \(t_{s}=1\) and \(K_{M}=2\), indicated in Figure 3C, the correlation of the neighboring bone joints will be modeled.