## 3 Proposed Method
### Spatial Self-Attention Augmented Graph Convolution
#### 3.3.2 Spatial Self-Attention (SAA) Module  
The transformer model employing self-attention originally designed to operate on words in NLP tasks. The self-attention mechanism reduces the dependence on external information and is better at capturing the internal correlation of data or features. The SAA applies a modified self-attention operator, as depicted in Figure 5, to capture the spatial features of different joints in the same frame and dynamically build spatial relationships within and between joints to strengthen the correlation of body joints that are not directly connected in human skeletons.  
The relations between joints are dynamically generated in SAA, thus the relevant structure of skeleton is adaptively generated, not fixed for all actions. The SAA is achieved by independently calculating the correlation between each pair of joints in each frame, as shown in Figure 6. When the source node that calculates the weight needs to calculate the weighted results, all the other nodes are required to participate in the calculation, which is a manifestation of the ability to capture the global characteristics.  
Figure 5: Self-attention of skeleton joints. (**1**) We calculate a query \(q\) a key \(k\) and a value vector \(v_{i}\). (**2**) The query of the joint and the key of all the other joints is performed by dot product (\(\odot\)), and a weighted value is obtained to represent the strength of the connection between each pair of joints; (**3**) Each joint is scaled to a new node due to its correlation; (**4**) The new features are added (\(\oplus\)) the weighted nodes together.  
For each joint \(v_{tn}\) of the skeleton at time \(t\), we first calculate the query vector \(q_{n}^{t}\in\mathbb{R}^{d_{q}}\), the key vector \(k_{n}^{t}\in\mathbb{R}^{d_{k}}\), and the value vector \(v_{n}^{t}\in\mathbb{R}^{d_{v}}\) by applying trainable linear transformations to the joint features \(j_{n}^{t}\in\mathbb{R}^{C_{in}}\) with parameters \(W_{q}\in\mathbb{R}^{C_{in}\times d_{q}}\), \(W_{k}\in\mathbb{R}^{C_{in}\times d_{k}}\), \(W_{v}\in\mathbb{R}^{C_{in}\times d_{v}}\), shared by all nodes. Where \(C_{in}\) is the number of input features and \(d_{k}\), \(d_{q}\), \(d_{v}\) are the channel dimensions of the key vectors, the query vector and the value vector, respectively. Then, for each pair of body joints \((V_{tn},V_{lm})\), the score \(\alpha_{mn}^{t}\), which represents the strength of the correlations between the two joints, is determined by \(\alpha_{nm}^{t}=a_{n}^{t}\cdot(k_{m}^{t})^{T}\in\mathbb{R},\forall t\in T\), then it is used to weight each joint value \(v_{m}^{t}\), and a weighted sum is calculated to get a new embedding \(z_{n}^{t}\in\mathbb{R}^{C_{out}}\) for joint \(v_{tn}\), as shown in Equation (3).  
\[z_{n}^{t}=\sum_{m}softmax_{m}(\frac{\alpha_{nm}^{t}}{\sqrt{d_{k}}})v_{m}^{t}. \tag{3}\]  
Multi-head attention [5] is multiple independent self-attention calculations applied by repeating the process many times, each time with a diverse set of learnable parameters as an integrated function to prevent overfitting.  
\[head_{N_{h}}(X_{N})=Softmax(\frac{(X_{N}W_{q})(X_{N}W_{k})^{T}}{\sqrt{d_{k}^{N _{h}}}})(X_{N}W_{V}), \tag{4}\]  
where \(X_{N}\) is the reshaped input, and \(W_{q}\in\mathbb{R}^{C_{in}\times N_{h}\times d_{q}^{h}}\), \(W_{k}\in\mathbb{R}^{C_{in}\times N_{h}\times d_{k}^{h}}\), and \(W_{v}\in\mathbb{R}^{C_{in}\times N_{h}\times d_{v}^{h}}\) are learned linear transformations. Then the outputs of all heads are concatenated as  
\[SA_{N}=Concat(head_{1},\dots,head_{N_{h}})W^{o}, \tag{5}\]  
where \(W^{o}\) is a learnable linear transformation combining outputs of all heads.