## 3 The Proposed Method
### Graph Convolutional Neural Network  
There are two approaches to define convolutional filters in a convolutional neural network: spectral-based graph convolution and spatial-based graph convolution. The spectral-based graph convolution has a solid mathematical foundation and it produced a good result in some applications. However, the method is not flexible to apply to many structures. The filter defined is domain-dependent. Additionally, the eigen decomposition in spectral-domain costs higher computational complexity. Like the typical convolutional method applied on videos, images, and sounds, a spatial convolutional method is implemented based on spatial relationships of the entity-nodes. The skeleton of the body is represented as an undirected graph \(G=\{V,E\}\) on a skeleton sequence with \(N\) joints and \(T\) frames featuring both intra-body and inter-frame connection. In this graph, the node set \(V=\{v_{i}|i=0,\ldots,N-1\}\) includes all the joints in a skeleton sequence. Instead of taking the spatial and temporal features as a separate entity, we joined them into a single dimension. Therefore, the set of joints V consists of all joints from intra-body and inter-frame connection. Thus, the total number of nodes, \(N=n_{j}\times T\), where \(n_{j}\) is number of joints per frame and \(T\) is number of frames. Figure 1 shows the connectivity of joints in a frame and the connection of the same joint in consecutive frames. In spatial and temporal graph convolution where the edge is defined on both spatial and temporal dimensions the edge, set \(E\) is composed of two subsets, the first subset depicts the intra-skeleton connection at each frame, and the second subset contains the inter-frame edges, which connect the same joints in consecutive frames. The intra-skeleton denoted as \(E_{S}=\{v_{ti},v_{tj}|(i,j)\in n_{j}\}\) and inter-frame edges denoted as \(E_{T}=\{v_{ti},v_{(t+1)i}\}\), where \(n_{j}\) is the set of joints. For two consecutive joints in skeleton body, if the joints are connected with a single bone, we set the value of the edge between them 1. For intra-body connectivity, we set the edge between the same joint in consecutive frames to 1. We set a small value \(\delta=0.01\) for joints that are not connected in intra-body or inter-frame connectivity. The connectivity of all joints in a given video is represented in adjacency matrix, **A** as it is shown in Equation (2). \[\mathbf{A}= \begin{pmatrix}d_{0,0}&d_{0,1}&d_{0,2}&...&d_{0,N-1}\\ d_{1,0}&d_{1,1}&d_{1,2}&...&d_{1,N-1}\\ d_{2,0}&d_{2,1}&d_{2,2}&...&d_{2,N-1}\\.&.&.&.&.\\.&.&.&.&.\\ d_{N-1,0}&d_{N-1,1}&d_{N-1,2}&...&d_{N-1,N-1}\end{pmatrix}  \tag{2}\]  
where \(d_{i,j}\) is the edge between two joints. As it is stated earlier, if there is no edge between two joints then \(d_{i,j}\) set to \(\delta=0.01\). In Equation (2), the row of the adjacency matrix shows the connection of joint 0 with all joints in intra-skeleton and inter-frame. Let the number of frames is \(M\) and each frame has 4 joints. The total number of joints would be, \(N=MX4\). Thus, in Equation (2), \(d_{i,j}\) means the connection between the first joints in the first frame and the second joint which is located in the same frame. If there is an edge between the joints the value of \(d_{i,j}\) set to 1, otherwise it is set to 0. The toy example in Figure 2 describe for node, \(N=16\). Each rows shows how node i is related to all of the nodes in the input. Since we organized the spatial, temporal data and the adjacency matrix as a 2D dimension, we can directly use the graph convolutional definition from [34] into our problem.