## 2 Materials and Methods
### Early Multi-Branch Fusion of Spatial-Temporal Joint Attention GCN
#### 2.2.2 Spatial-Temporal Graph Convolutional Attention Module  
According to the definition of ST-GCN, the convolution of the graph with any frame \(t\) can be written as  
\[f_{out}(v_{ti})=\sum_{v_{ij}\in N(v_{ti})}\frac{f_{in}\big{(}v_{tij}\big{)} \cdot w\big{(}l_{ti}(v_{tij})\big{)}}{Z_{ti}\big{(}v_{tj}\big{)}}, \tag{5}\]  
where \(v_{ti}\) is the \(i\)-th joint of the \(t\)-th frame; \(f_{in}\) and \(f_{out}\) are the input and output features of the corresponding joint, respectively; \(N(v_{ti})\) is the set of joints adjacent to joint \(v_{ti}\); \(Z_{ti}\) is the normalization term to balance the contributions of different adjacency sets; and \(w(\cdot)\) is a weight function that assigns weights indexed by the label function \(l_{ii}(\cdot)\), which constructs multiple adjacency sets \(N(v_{ti})\) by assigning different labels to different graph nodes. In the proposed approach, we adopt a distance-based division method to define the label assignment. Specifically, we define that \(l_{li}(v_{tij})=d\big{(}v_{ti},v_{ij}\big{)}\), where \(d\big{(}v_{ti},v_{tj}\big{)}\) denotes the graph distance between \(v_{ti}\) and \(v_{tj}\). The joints with the same distance comprise a subset and share a learnable weight function \(w(\cdot)\). By the adjacency matrix \(A\), Equation (5) above can be transformed into the form given below:  
\[f_{out}=\sum_{d=0}^{D}W_{d}f_{in}\bigg{(}\Lambda_{d}^{-\frac{1}{2}}A_{d}\Lambda _{d}^{-\frac{1}{2}}\odot M_{d}\bigg{)}, \tag{6}\]  
where \(D\) is the preset maximum graph distance; \(f_{in}\) and \(f_{out}\) are the input and output feature maps, respectively; \(\odot\) is a point-by-point convolution; \(A_{d}\) is a \(d\)-order adjacency matrix that labels joint pairs with graph distance \(d\); \(\Lambda_{d}\) is the normalization operator of \(A_{d}\), \(W_{d}\), and \(M_{d}\), which are learnable parameters for implementing the convolution operation and tuning the importance of each edge. In this study, we introduce a spatial-temporal joint attention mechanism. The proposed approach includes an added attention module after the spatial-temporal graph convolution to form a new spatial-temporal convolutional layer. In the field of skeleton-based action recognition, previous attention mechanisms were mainly realized by multilayer perceptron (MLP) models, in which each channel or spatial dimension was independently processed, while the other dimensions were equally distributed to individual units. Given that temporal and spatial information may be correlated with each other in a spatial-temporal skeleton series, in the proposed model, we included a spatial-temporal joint attention mechanism (ST-JointAtt) to distinguish between the most informative joints in a particular frame and the whole skeleton series together. The mechanism helps to identify the most critical gait regions and joints in the entire walking process. Figure 3 illustrates the ST-JointAtt module. First, the skeleton series is temporally and spatially pooled for frames and joints, respectively. Next, the two eigenvectors are stitched together, and the information is compressed by a fully connected layer. Subsequently, two independent fully connected layers are used to obtain the inter-frame and inter-joint attention scores. Finally, the frame score and joint score are multiplied by the outer product of channels. The result can be regarded as the attention score of the entire skeleton series. This process can be expressed as follows. \[ \begin{split}& f_{inner}=\theta((\mathit{pool}_{t}(f_{in})\oplus \mathit{pool}_{v}(f_{in}))\cdot W),\\ & f_{out}=f_{in}\odot(\sigma(f_{inner}\cdot W_{t})\otimes\sigma( f_{inner}\cdot W_{v})),\end{split}