## 4 The Proposed Method
### Spatio-Temporal Hypergraph Modeling of Human Skeleton Correlation
#### 4.2.3 Non-Physically Connected Bone Spatio-Temporal Hypergraph  
For the non-physically connected bone (N-PCB) spatio-temporal hypergraph, the construction process is similar to that for the non-physically connected joint spatio-temporal hypergraph, as shown in Figure 4c. In our study, the bones of the human body are used as the vertices of the hypergraph, and bones with similar values of change in successive frames are connected by a single hyperedge, which characterizes both the spatial and temporal information of the human bones.  
In our method, bones are represented using a vector representation, pointing from the source joint to the target joint, such as \((\mathbf{x}_{1},\mathbf{y}_{1})\rightarrow(\mathbf{x}_{2},\mathbf{y}_{2})\), and the magnitude of change of the bones between two consecutive frames is calculated as  
\[d(\mathbf{b}_{j}^{t},\mathbf{b}_{j}^{t+1})=\sqrt{(\mathbf{b}_{j}^{t+1}-\mathbf{b}_{j}^{t})^{ \top}(\mathbf{b}_{j}^{t+1}-\mathbf{b}_{j}^{t})} \tag{12}\]  
where \(\mathbf{b}_{j}^{t}\) denotes the coordinate representation of the \(j\)-th bone at moment \(t\) and \(\mathbf{b}_{j}^{t+1}\) denotes the coordinate representation of the \(j\)-th bone at moment \(t+1\). The value of the magnitude of change of the bone \(\mathbf{b}_{j}\) over the whole time series is  
\[d_{sum}^{\mathbf{b}_{j}}=\sum_{t=1}^{T-1}d(\mathbf{b}_{j}^{t},\mathbf{b}_{j}^{t+1}) \tag{13}\]  
The process of normalizing the magnitude of change values of bones over the entire time series is as follows:  
\[d(\mathbf{b}_{j})=\frac{d_{sum}^{\mathbf{b}_{j}}}{\sum_{j=1}^{J}d_{sum}^{\mathbf{b}_{j}}} \tag{14}\]  
where \(J\) denotes the number of human bones in a frame.  
The non-physically connected bone spatio-temporal hypergraph is denoted by \(\mathbf{\mathcal{G}}_{N-PCB}=(\mathbf{\mathcal{V}}_{N-PCB},\mathbf{\mathcal{E}}_{N-PCB}, \mathbf{W}_{N-PCB})\), where \(\mathbf{\mathcal{V}}_{N-PCB}\) denotes the set of vertices of this hypergraph, \(\mathbf{\mathcal{E}}_{N-PCB}\) denotes the set of hyperedges of this hypergraph, and \(\mathbf{W}_{N-PCB}\) denotes the diagonal matrix of hyperedge weights in this hypergraph. Similarly to the process of constructing a spatio-temporal hypergraph of non-physically connected joint,the incidence matrix \(\mathbf{H}_{N-PCB}\) of this hypergraph is constructed and the Laplacian matrix \(\mathbf{G}_{N-PCB}\) of this hypergraph is calculated.  
The above process of constructing a hypergraph correlates the temporal and spatial relationships of the target objects. For spatial information, the hyperedge connects joints or bones with similar values of change in the same frame, characterizing the dependency among the joints or bones in the action; for temporal information, the hyperedge connects the same joints or bones in different frames, characterizing the temporal information of the joints or bones in the change of the action. In addition, the joints and bones of the human action are also considered, with the joints as the basic information of the human action and the bones as the complementary information, the modeling process fully reflects the completeness of the data. For the action recognition in complex scenes, the use of both joint and bone can obtain more effective information from the human skeleton than a single joint, which is beneficial to improve the accuracy of human action recognition. In addition, the modeling process characterizes the association among physical connections and the dependency among non-physical connections in the human skeleton, taking into account both the fixed human topology and the dependency among different joint and bone in the changing human action, and more accurately solving the problem of the inaccurate recognition of human action in complex scenes.