# Multi-Scale Adaptive Aggregate Graph Convolutional Network for Skeleton-Based Action Recognition
## 3 Multi-Scale Adaptive Aggregate Graph Convolutional Network
### Multi-Scale Aggregate Graph Convolution Module  
For better action recognition based on skeleton graphs, multi-scale spatial structural features need to be paid attention to, while traditional methods only consider the physical connections between joints, ignoring that there are also potential correlations between disconnected joints in the structure. For example, there are multi-hop connections between the left hand and the right hand, but it is very important to recognize the action of clapping. Therefore, we propose a multi-scale aggregate graph convolution module to explore the relevance of human skeleton data in the same frame. As shown in Figure 2, we build the multi-order adjacency matrices and adopt an adaptive method in each adjacency matrix to learn the topological structure of the graph to improve the flexibility of modeling semantic information, and finally we aggregate the obtained multi-order information to explore the spatial feature information of the skeleton sequence. In the graph convolutional network, we construct a graph \(G(V,E,A)\) to represent human skeleton data, where \(V\) is the set of nodes that symbolizes the joints of the human body; \(E\) is the set of edges of the skeleton connected by adjacent joints; and \(A\in R^{N\times N}\) represents the adjacency matrix of the edge, which encodes the connection of the edge, that is, if the node \(V_{i}\) and \(V_{j}\) are connected, then \(A_{i,j}=1\); otherwise, \(A_{i,j}=0\). We define \(\widetilde{A}=A+I_{n}\), where \(I_{n}\) represents the identity matrix, and \(\widetilde{A}\) is the adjacency matrix with self-connection. Since \(\widetilde{A}\) cannot be multiplied directly, we first need to standardize it. According to [39], the standardized Laplacian L can be formulated as \(L=D^{-\frac{1}{2}}\widetilde{A}D^{-\frac{1}{2}}\), where the diagonal matrix \(D\in R^{N\times N}\), \(D_{i,j}=\sum\limits_{j=1}^{n}A_{i,j}\). For the input \(X\), the graph convolution with filter \(f_{\theta}\) can be expressed as \(Y=f_{\theta}(L)X=Uf_{\theta}U^{T}X\); \(U\) represents the eigenvector matrix of \(L\); \(U^{T}X\) is the Fourier transform of the graph of \(X\); and \(f_{\theta}\) can be regarded as a function of the eigenvector of \(L\). [PERSON] et al. [40] have mentioned that \(f_{\theta}\) can be approximated by Chebyshev polynomials, which is formulated as  
\[Y\approx\sum\limits_{k=0}^{K}\theta_{k}T_{k}\left(\widetilde{L}\right)F_{in} \tag{1}\]  
where \(\widetilde{L}=\frac{2}{\lambda_{max}}-I_{n}\), and \(\theta_{k}\) represents the \(k\)-order Chebyshev coefficient. Moreover, a Chebyshev polynomial is defined as \(T_{k}\left(\widetilde{L}\right)=2\widetilde{L}T_{k-1}\left(\widetilde{L} \right)-T_{k-2}\left(\widetilde{L}\right)\), \(k\geq 2\), \(T_{0}\left(\widetilde{L}\right)=1\), \(T_{1}\left(\widetilde{L}\right)=\widetilde{L}\). According to [39], for the signal \(x\), it satisfies  
\[\left(\widetilde{L}x\right)(i)=\sum\limits_{j\in N_{i}}a_{i,j}\left(x_{i}-x_{ j}\right) \tag{2}\]  
\(\widetilde{L}X\) means to calculate the difference between each vertex and its one-hop neighbor nodes. When \(k>1\), \(\widetilde{L}^{k}X\) means the k-order connection, and \(\widetilde{L}^{k}X\) means the difference between each vertex and its k-hop neighbor nodes, so it is possible to perform convolution operations on more neighbor nodes in a larger receptive field. Moreover, the spatial graph convolution with a receptive field of k can also be regarded as a linear transformation of the  
Figure 2: Spatial graph of the skeleton (**left**) and multi-scale aggregate graph convolution module (**right**). In the graph, the solid nodes represent the body joints, and the edges represent the bones that connect the joints. The edges of different colors intuitively reflect the relationship between nodes with different hop connections. k-order Chebyshev polynomial.