## 4 The Proposed Method
### Spatio-Temporal Hypergraph Modeling of Human Skeleton Correlation
#### 4.2.2 Non-Physically Connected Joint Spatio-Temporal Hypergraph  
For example, in a traffic police gesture, the arm of the command 'go straight' changes more, while the torso and legs change less, as shown in Figure 5. Therefore, we propose to add an attention mechanism to obtain the key areas of the action, so as to highlight the influence of the key areas on human action recognition. This paper proposes an attention-based representation of non-physically connected skeletal information, aiming to capture the key regions that have a greater impact on action recognition, and thus increase the efficiency of action recognition. For non-physical connections, this paper introduces the concept of magnitude of change values to calculate the magnitude of change that occurs in the temporal sequence for each joint. Each joint has a magnitude of change value, which indicates the degree of influence that a particular joint has on human action recognition. Specifically, a joint is taken and its magnitude of change between the first and second frames is calculated using two-parametric numbers, where the joint is represented using \(xy\)-coordinates, such as \((x_{0},y_{0})\), representing the two-dimensional coordinates of the joint. The magnitude of change of the joint between two consecutive frames is calculated as  
\[d(\mathbf{v}_{i}^{t},\mathbf{v}_{i}^{t+1})=\sqrt{(\mathbf{v}_{i}^{t+1}-\mathbf{v}_{i}^{t})^{ \top}(\mathbf{v}_{i}^{t+1}-\mathbf{v}_{i}^{t})} \tag{8}\]  
where \(\mathbf{v}_{i}^{t}\) denotes the coordinates of the \(i\)-th joint at time \(t\) and \(\mathbf{v}_{i}^{t+1}\) denotes the coordinates of the \(i\)-th joint at time \(t+1\). The value of the magnitude of change of the joint \(v_{i}\) over the whole time series is  
\[d_{sim}^{v_{i}}=\sum_{t=1}^{T-1}d(\mathbf{v}_{i}^{t},\mathbf{v}_{i}^{t+1}) \tag{9}\]  
where \(T\) denotes the number of frames of the video data segmentation. After obtaining the change amplitude value of each joint over the whole time series, it is normalized and the normalization process of the change amplitude value of the joint over the whole time series is as follows:  
\[d(v_{i})=\frac{d_{sim}^{v_{i}}}{\sum_{i=1}^{I}d_{sim}^{v_{i}}} \tag{10}\]  
where \(I\) indicates the number of human joints in a frame. The association is modeled for joints that are more dependent but not directly physically connected in the human action, using a clustering algorithm that clusters joints according to the value of the magnitude of change of each joint. Joints of the same class in consecutive frames are connected by a hyperedge, and the joints in the same hyperedge have a similar degree of influence on the action. In general, for a given action, there is a dependency among the joints connected by the hyperedge, and this is used to construct a spatio-temporal hypergraph representing the association information among the non-physically connected joints in the different actions of the human body. Assuming that the clustering algorithm divides the joints into a total of \(Q\) classes, denoted by the set \(\{\mathbf{\mathcal{V}}_{q}\}_{q=1,2,\cdots,Q}\), the spatio-temporal hypergraph of the non-physically connected joint (N-PCJ) is denoted by \(\mathbf{\mathcal{G}}_{N-PCJ}=(\mathbf{\mathcal{V}}_{N-PCJ},\mathbf{\mathcal{E}}_{N-PCJ}, \mathbf{W}_{N-PCJ})\), where \(\mathbf{\mathcal{V}}_{N-PCJ}\) denotes the set of vertices of this  
Figure 5: Illustration of the skeletal changes in traffic police gesture of ‘go straight’. Red changes more, which indicates the key area, and blue changes less. hypergraph, \(\mathbf{\mathcal{E}}_{N-PCJ}\) denotes the set of hyperedges of this hypergraph, and \(\mathbf{W}_{N-PCJ}\) denotes the diagonal matrix of hyperedge weights in this hypergraph. Specifically, \(\mathbf{\mathcal{V}}_{N-PCJ}\) is the set of all joints \(\mathbf{v}_{t}^{(i)}\) in all frames; the elements in \(\mathbf{\mathcal{E}}_{N-PCJ}\) are denoted by \(e_{q}=\{\mathbf{v}_{t}^{(i)}|\mathbf{v}_{t}^{(i)}\in\mathbf{\mathcal{V}}_{q}\}\) and \(\mathbf{\mathcal{E}}_{N-PCJ}=\{e_{q}|q=1,2,\cdots,Q\}\).