## 4 The Proposed Method
### Adaptive Multi-Channel Spatio-Temporal Hypergraph Neural Network
#### 4.3.2 Multi-Feature Fusion Module Based on Cross-Channel Attention Mechanism  
After obtaining the spatio-temporal features of each channel, the features \(\tilde{\mathbf{X}}_{PCJ}\), \(\tilde{\mathbf{X}}_{N-PCJ}\) and \(\tilde{\mathbf{X}}_{N-PCB}\) of the three channels are adaptively fused by a multi-feature fusion module  
Figure 6: Hypergraph convolution module based on spatio-temporal information: (**a**) incidence matrix; (**b**) subincidence matrix; (**c**) initial features of hypergraph vertices; (**d**) initial features of hypergraph vertices after splitting by time dimension; (**e**) spatial convolution block; (**f**) temporal convolution block; (**g**) hypergraph vertex features after splitting by temporal dimension; and (**h**) trained hypergraph vertex features. based on the cross-channel attention mechanism, which fully considers the complementarity among the data features of different channels. The structure of the module is shown in Figure 7.  
The module takes the spatio-temporal features of the respective channels, the incidence matrix of the spatio-temporal hypergraph, and the corresponding Laplacian matrix as input, and learns the spatio-temporal features of each channel once on the entire spatio-temporal hypergraph separately, using the Laplacian matrix while supplementing the original information in the incidence matrix to make full use of the incidence matrix and Laplacian matrix of the hypergraph to update the spatio-temporal features of each channel. The updated data features of each channel are represented as follows:  
\[\mathbf{F}_{PCJ} =(\mathbf{H}_{PCJ}\mathbf{H}_{PCJ}^{\top}+\beta_{1}\mathbf{G}_{ PCJ})\mathbf{\hat{X}}_{PCJ} \tag{19}\] \[\mathbf{F}_{N-PCJ} =(\mathbf{H}_{N-PCJ}\mathbf{H}_{N-PCJ}^{\top}+\beta_{2}\mathbf{G} _{N-PCJ})\mathbf{\hat{X}}_{N-PCJ}\] (20) \[\mathbf{F}_{N-PCB} =(\mathbf{H}_{N-PCB}\mathbf{H}_{N-PCB}^{\top}+\beta_{3}\mathbf{G} _{N-PCB})\mathbf{\hat{X}}_{N-PCB} \tag{21}\]  
where \(\mathbf{H}_{*}^{\top}\) denotes the transpose of the correlation matrix \(\mathbf{H}_{*}\) and \(\beta_{1}\), \(\beta_{2}\), \(\beta_{3}\) are learnable scalars. Considering that different channel features have different degrees of influence on action recognition, in order to make full use of the correlation and complementarity among different channel features, this paper uses a cross-channel attention mechanism to calculate the attention score of each channel feature in human action, uses this score as the weight of each channel feature, and performs a weighted fusion of the features of each channel to adaptively obtain the final fused features. Specifically, first, the features \(\mathbf{F}_{*}\) of each channel are compressed into a one-dimensional vector \(\mathbf{f}_{*}\) for attention weighting using a one-dimensional convolutional layer, and the process is represented as  
\[\mathbf{f}_{*}=Conv1d(\mathbf{F}_{*}) \tag{22}\]  
where \(Conv1d\) denotes a one-dimensional convolutional layer. Next, the attention matrix between each of the two channel features is calculated separately, as follows:  
\[\mathbf{A}_{PCJ,N-PCJ} =Softmax\Bigg{(}\frac{(\mathbf{W}_{q_{1}}\mathbf{f}_{PCJ})(\mathbf{W} _{k_{1}}\mathbf{f}_{N-PCJ})^{\top}}{\sqrt{d}}\Bigg{)} \tag{23}\] \[\mathbf{A}_{N-PCJ,N-PCB} =Softmax\Bigg{(}\frac{(\mathbf{W}_{q_{2}}\mathbf{f}_{N-PCJ})(\mathbf{ W}_{k_{2}}\mathbf{f}_{N-PCB})^{\top}}{\sqrt{d}}\Bigg{)}\] (24) \[\mathbf{A}_{PCJ,N-PCB} =Softmax\Bigg{(}\frac{(\mathbf{W}_{q_{3}}\mathbf{f}_{PCJ})(\mathbf{ W}_{k_{3}}\mathbf{f}_{N-PCB})^{\top}}{\sqrt{d}}\Bigg{)} \tag{25}\]  
where \(\mathbf{W}_{q}\) and \(\mathbf{W}_{k}\) are trainable parameter matrices, respectively, and \(d\) denotes the dimensionality of \(\mathbf{f}_{*}\).