## 4 The Proposed Method
### Adaptive Multi-Channel Spatio-Temporal Hypergraph Neural Network
#### 4.3.1 Hypergraph Convolution Module Based on Spatio-Temporal Information  
The hypergraph convolution module based on spatio-temporal information consists of a spatial convolution block, a temporal convolution block, and a global pooling layer, with a spatial convolution block and a temporal convolution block representing one layer in series, for a total of \(L\) layers, as shown in Figure 6. Among them, the spatial convolutional block is used to aggregate the semantic features of the hypergraph in the spatial dimension, and the temporal convolutional block is used to learn the temporal information of the hypergraph in the temporal dimension. Finally, a global pooling layer is connected to output the updated feature vector. In order to better learn the temporal information of the human skeleton in the process of movement, the incidence matrix of the constructed hypergraph is split according to the temporal dimension in this paper, and the incidence matrix of the spatio-temporal hypergraph is dimensionally transformed, and the incidence matrix of the spatio-temporal hypergraph of each channel after the transformation is expressed as  
\[\tilde{\mathbf{H}}_{*}=\textit{reshape}(\mathbf{H}_{*})=\{\mathbf{H}_{*}^{(t) }|t=1,2,\cdot\cdot\cdot,T\} \tag{15}\]  
where \(\tilde{\mathbf{H}}_{*}\) can be viewed as the subincidence matrix of \(\mathbf{H}_{*}\), and \(*\) denotes the different strategies for constructing the hypergraph (including PCJ, N-PCJ, and N-PCB). The Laplacian matrix \(\mathbf{G}_{*}^{(t)}\) of each subincidence matrix is calculated according to the transformation pro cess of Laplacian matrices in Section 4.2, and stitched according to the temporal dimension to obtain the 3D matrix \(\tilde{\mathbf{G}}_{*}\), denoted by  
\[\tilde{\mathbf{G}}_{*}=\{\mathbf{G}_{*}^{(t)}|t=1,2,\cdots,T\} \tag{16}\]  
In addition, the initial joint features and bone features are also represented according to the temporal dimension. Specifically, the initial feature matrix of the joints in frame \(t\) is denoted by \(\mathbf{V}_{t}=\{\mathbf{v}_{1}^{t},\mathbf{v}_{2}^{t},\cdots,\mathbf{v}_{t}^{ t}\}\), and the initial feature matrix of the joints in all frames is denoted by \(\mathbf{V}=\{\mathbf{V}_{t}|t=1,2,\cdots,T\}\); the initial feature matrix of the bones in frame \(t\) is denoted by \(\mathbf{B}_{l}=\{\mathbf{b}_{1}^{t},\mathbf{b}_{2}^{t},\cdots,\mathbf{b}_{l}^ {t}\}\), and the initial feature matrix of the bones in all frames is denoted by \(\mathbf{B}=\{\mathbf{B}_{l}|t=1,2,\cdots,T\}\). Taking the physically connected joint spatio-temporal hypergraph as an example, \(\tilde{\mathbf{G}}_{PCJ}\) and \(\mathbf{V}\) are first input to the spatial convolution block for spatial feature learning, and the vertex features after Laplacian matrix learning are weighted and fused with the original vertex features to form the input feature matrix for the \(l\)-th temporal convolution block:  
\[\mathbf{\hat{V}}^{(l)}=\sigma(\mathbf{V}^{(l)}+\tilde{\mathbf{G}}_{PCJ}\odot \mathbf{V}^{(l)}\mathbf{\Theta}^{(l)}) \tag{17}\]  
where \(\sigma\) is the activation function, \(\odot\) denotes the element multiplication, \(\mathbf{V}^{(l)}\) denotes the input features of the \(l\)-th spatial convolution block, and \(\mathbf{\Theta}^{(l)}\) denotes the learnable parameters. \(\tilde{\mathbf{V}}^{(l)}\) is then fed into the temporal convolution block for training, inspired by [27], to obtain the new vertex feature matrix as follows:  
\[\mathbf{V}^{(l+1)}=tanh(\Gamma_{1}^{(l)}\divide\tilde{\mathbf{V}}^{(l)}) \odot\Phi(\Gamma_{2}^{(l)}\divide\tilde{\mathbf{V}}^{(l)}) \tag{18}\]  
where \(\divide\) is the convolution operation, \(\Gamma_{1}^{(l)}\) and \(\Gamma_{2}^{(l)}\) are two different learnable parameters in the \(l\)-th temporal convolution block, and \(\Phi(\cdot)\) is the gating unit to control the utilization of historical information.