## 3 The Proposed Method
### Graph Convolutional Neural Network  
After the correction of STN, the skeleton images are adjusted to the appropriate size and pose, and the input video is also converted into a series of images at different positions on the time axis. As shown in Figure 3, each blue dot represents a keypoint on the human body, the green lines represent the human limbs and the torso, and the orange line represents the connection of the same keypoint in consecutive frames. Human skeleton images at different moments form an interconnected network in 3D space, called a spatial temporal skeletal graph, the three dimensions of which are the width direction and height direction of the image and the temporal dimension, so any keypoint can be represented as a space coordinate \(\left(w,d,c,t\right)\). \(w\) and \(d\) correspond to the position of the keypoint in the skeleton image, \(c\) represents the confidence of this keypoint, \(t\) corresponds to the moment corresponding to the keypoint. Therefore, the remaining question is how to extract the spatial and temporal information from these keypoints and the connections between them to recognize actions. Here we use the spatial temporal graph convolution network [28] to form a hierarchical representation of the human skeleton. For a video with a total of \(T\) frames, the spatial temporal skeletal graph with \(N\) keypoints can be represented as \(G=(V,E)\), where \(V=\{v_{ti}|t=1,2,\ldots,T,i=1,2,\ldots,N\}\) denotes the keypoint set of the graph, \(E\) represents the line between the keypoints in the spatial temporal skeletal graph, composed of two subsets of \(E_{S}\) and \(E_{F}\). \(E_{S}=\left\{v_{ti}v_{tj}|(i,j)\in H\right\}\) is the connection of the keypoints in each frame, \(H\) represents a collection of keypoints of the human body, \(E_{F}=\left\{v_{ti}v_{(t+1)i}\right\}\) represents a link between keypoints in consecutive frames, and each element in \(E_{F}\) represents a trajectory of a particular joint over time. In terms of spatial information, according to [PERSON] et al. [28] and [PERSON] et al. [38], the graph convolution in single frame can be expressed as  
\[f_{\text{out}}\left(v_{ti}\right)=\sum_{v_{tj}\in B(v_{ti})}\frac{1}{Z_{ti} \left(v_{tj}\right)}f_{in}\left(v_{tj}\right)\cdot\mathbf{w}\left(v_{ti},v_{tj}\right) \tag{2}\]  
where \(f_{in}\) and \(f_{out}\) are the input and output of the graph convolution, \(B_{v_{ti}}\) is the collection of 1-distance neighbors of the target keypoint \(v_{ti}\), \(Z_{ti}\left(v_{tj}\right)\) is used to balance the contribution of different subsets to the output, which equals the cardinality of the corresponding subset, and \(w\) is the weight function, which is used to compute the inner product of the input skeleton image. For the fusion of spatial and temporal information, suppose that \(A\) is an adjacency matrix of \(N\times N\), which represents the connection relationship between keypoints between adjacent frames. However, unlike ordinary images, if an adjacent matrix is used to define a neighborhood, the number of keypoints in the neighborhood of each keypoint is not fixed. This makes it difficult for us to determine: (1) The parameter dimensions of the convolution kernel that need to be used. (2) If the weight matrix is aligned with the keypoints in the neighborhood for inner product operations. Here we have designed some strategies similar to [PERSON] et al. [28]: (1) Suppose \(v_{tg}\) is the barycenter of the human body in skeleton images. (2) For any keypoint \(v_{ti}\), the keypoints connected to it form a neighbor set, the keypoint \(v_{ti}\) is called the root node of this neighbor set. (3) In the neighbor set of any root node, all points with a distance greater than the barycenter from the root node are called  
Figure 3: Model structure of stacked hourglass model (SHM). centrifugal subsets, and all points with a distance less than the barycenter from the root node are called centripetal subsets. According to the research in [PERSON] et al.