# Skeleton Motion Recognition Based on Multi-Scale Deep Spatio-Temporal Features
## 3 Proposed Methods
### Selection of Feature Fusion Methods  
Because so many features are extracted from skeleton data, the question of how to deal with these features has also become a key part of our work. In the process of multi-scale feature fusion, the time of fusion is an important consideration. For different fusion periods, we tried three feature fusion methods: early fusion and late fusion, also known as feature fusion and decision-making level fusion. Early fusion, also known as feature fusion, refers to a fusion method immediately after multi-scale feature extraction. Late fusion is also called decision-making level fusion, which refers to making decisions (classification or  
Figure 5: Illustration of time multi-scale sampling module.  
ogy and other network parameters in an end-to-end manner. This graph has uniqueness for different layers and different samples and can adapt to the model very flexibly. At the same time, a residual connection is added to the model to ensure the stability of the original model. More specifically, in the spatial dimension, to make the graph adapt to different samples and graph structures, we use the iterative formula shown in Equation (13).  
\[f_{out}=\sum\limits_{K}^{K_{v}}W_{k}f_{in}(A_{k}+B_{k}+C_{k}) \tag{13}\]  
\(K_{v}\) represents the kernel size of spatial latitude. According to the division rules of subgraphs, we set \(K_{v}=3\), \(W_{k}\) is the weight matrix, and \(A_{k}\) is an \(N\times N\), which represents the physical structure of the human body. \(B_{k}\) is also an adjacency matrix of \(N\times N\), but \(B_{k}\) has no specific constraints on the inner value, which means that the graph is completely learned from the training data. In the data-driven task, we can completely learn the graph from the target task, considering that the value in \(B_{k}\) can be any value, which can represent not only the physical structure of the human body but also the connection strength between adjacent nodes. \(C_{k}\) can represent a data correlation graph, which can learn a unique graph for each sample. To determine whether there is a connection between two adjacent nodes and how strong the connection is, we use the normalized Gaussian function to calculate the similarity of the two nodes, as shown in Equation (14).  
\[f(v_{i},v_{j})=\frac{e^{\theta(v_{i})^{T}\phi(v_{j})}}{\sum\limits_{j=1}^{N}e^ {\theta(v_{i})^{T}\phi(v_{j})}} \tag{14}\]  
Here, \(N\) represents the sum of all nodes. We use the point product to calculate the similarity of two nodes in the embedded space. \(C_{k}\) is a similarity matrix of \(N\times N\); we normalize the value to be between 0 and 1. This is consistent with the time dimension in ST-GCN in the time dimension. Every BN layer and Relu layer is connected behind each spatial graph convolution and time graph convolution.  
For the multi-scale deep spatiotemporal features proposed in this paper, we tried three different feature fusion methods. In the experiment discussed in Section 4, we concluded that the fusion effect at the decision level is the best. Therefore, when introducing the model in this section, we only introduce the third method in detail. Figure 10 depicts the skeleton action recognition model based on multi-scale deep spatiotemporal features proposed in this paper. Firstly, an action sequence is given, and four groups of actions with different scales are obtained after multi-scale time sampling. Then, the deep spatiotemporal feature extraction module (STFEM) is used to extract the deep spatiotemporal features of actions of four scales, and then the spatiotemporal features of these four scales are sent to the AGCN, respectively. After the Softmax classifier is applied, four different classification scores are obtained, and then the four classification scores are added in equal proportion to obtain the final classification result.  
Figure 9: Illustration of the adaptive graph convolutional block (**left**) and adaptive spatiotemporal convolution model (**right**).