## 1 Introduction and Related Works
### Deep Learning-Based Methods  
In recent years, with the progress of deep learning technology, the accuracy of human motion recognition technology using RGB video input directly has developed rapidly. One important direction is to use RGB video streams with human as input. The mainstream approach is to learn different video features through two or more backbones,which is called 2D dual flow networks [21; 22]. In [21], the two inputs are RGB frames and multiple optical flow frames; while in [22], low resolution RGB frames and high resolution RGB frames are the two inputs, in order to reduce computational complexity. The core idea of this type of accurate optical flow acquisition usually requires a high computational cost, so how to obtain approximation or substitution of optical flow at a lower computational cost is also a research focus of such methods. In [16], a framework based on knowledge extraction is proposed to achieve knowledge transfer from a teacher network trained using optical flow to a student network using motion vectors as input. A disadvantage of 2D dual flow networks is the insufficient modeling of long-term dependencies on timing. Networks using timing modeling, such as LSTM (Long Short-Term Memory), can compensate for this [23; 24; 25]. In recent years, more complicated models, such as 3D CNN based approaches [26; 27; 28; 29; 30] are developed to deal with original image sequences directly, instead of using features, they are more like an end-to-end job than previous methods. Compared to simply and directly throwing image sequences into a depth neural network for training, in the specific issue of human action recognition, another way is to to analyze and recognize the posture of the human body and limbs based on accurate segmentation and positioning [31; 32]. Skeleton sequence lists the trajectories of human joints, which can be used to characterize human action. Skeleton data provides body structure and posture information, which has two obvious advantages: (1) It has proportional invariance. (2) It is robust to clothing textures and backgrounds. Therefore, skeleton data mode is relatively suitable for HAR tasks. In terms of obtaining skeleton data: With the emergence of some inexpensive depth cameras such as Microsoft Kinect and Intel RealSense, as well as some excellent pose estimation algorithms (such as Openpose algorithm) [33; 34; 35; 36; 37], researchers can quickly and accurately obtain depth information and bone information of human motion. For using the skeleton data, the direct method is to convert the skeleton into images, and then use a mature image processing depth neural network for recognition. For example, [PERSON]. [38] divides the human skeleton into five parts, inputs them into multiple bidirectional RNNs, and then layers and fuses their outputs to generate high-level representations of actions; [PERSON]. [39] proposed a local sensing LSTM and simulated the relationship between different body parts in the LSTM unit; The idea behind [40; 41] is to encode bone sequence data into images and then send them to CNN for action recognition. They provide bone spectral maps and joint trajectory maps, respectively. In addition, there is some work focused on solving certain specific issues, such as viewpoint changes and high computing costs. Although the above methods has achieved considerable results, there are more ways to use the skeleton data more efficiently. Because the skeleton of the human body itself is a 'graph', it is theoretically more intuitive and appropriate to calculate it directly using the 'graph' method. In the recent years, methods based on graph neural network (GNN) and graph convolution neural network (GCN) has been proposed and has achieved good results in the field of HAR. In [42], the human skeleton is represented as a directed acyclic graph to effectively merge bone and joint information. In [43] and [44], Spatial temporal GCN were designed to learn spatial and temporal features from skeleton data, respectively.