## 3 The Proposed Method
### Graph Convolutional Neural Network
\[\mathbf{f}_{out}=\mathbf{D}^{-1/2}(\mathbf{A}+\mathbf{I})\mathbf{D}^{-1/2} \mathbf{f}_{in}\mathbf{W} \tag{3}\]  
where, \(\mathbf{I}\) is the identity matrix representing self connections and \(\mathbf{D}\) is the diagonal node degree matrix used for normalization given as the summation of adjacency matrix across the column, i.e., \(\mathbf{D}=\sum_{i}\mathbf{A}_{i,j}\times\mathbf{f}_{in}\) is the feature with \(\mathbb{R}^{N\times d}\) dimension and before applying the first graph convolution \(\mathbf{f}_{in}\) considered as the spatial location of each node with \(\mathbb{R}^{N\times 2}\) dimension. Figure 1: The keypoints denoted as blue dots in first frame and green dots in the following frames are used as input to the proposed graph convolutional neural network. The dots denote the body joints of the subject. The solid line shows the natural connection in intra-body. The dot line connects the inter-frame edges connects the same joints in consecutive frames. For readers' understanding, Figure 2 depicts a toy example of the proposed graph convolutional method. As it is explained earlier, among 25 keypoints, 8 keypoints which are directly related to gait measurements are selected. The selected key points are divided into left and right key points to process further. The \((x,y)\) location of left key points (left-hip (LHIP)), left-knee (LKNE), left-ankle (LANK), and left-big-toe (LBTO) for t frames used as input feature for the first graph convolutional layer. In our model, we have used two graph convolutional layer; the first layer takes the location of each keypoints in a video segment as input and produce 8 features. The output of the first layer fed into second layer and produces features with 16 channel which is fed in to conventional convolutional layer for down sampling and finally the output would be predicted. Figure 3 further illustrates the pipeline of our method from input to output. This method can be applied to the classification problem. The only modification expected would be adding softmax after the fully connected layer. It is worth noting that, although we achieve a good result, using only key points extracted using OpenPose, we attained a better result when we comprise a hand-engineered time series data crafted from the relationship of the joints. Figure 2: Illustration of (**a**) proposed method using an example of \(N=16\), where \(N\) is the total number of keypoints in 4 frames. Each point at \(t\) shows the joints of left skeleton. The left hip (LHIP) at \(t\) is connected to the left knee (LKNE) and LHIP at \(t+1\). (**b**) For the first graph convolutional layer the \((x,y)\) position of each joints is taken as a feature. The adjacency matrix **A** describes the relation of each joints with each other. The LHIP joint in the frame \(t\) is connected with the LANK in the same frame \(t\) and with LHIP in frame \(t+1\). Hence, for LHIP in frame \(t\), \(1\) is assigned for joints that have direct connection. Basically, The ouput feature is computed from matrix multiplication of the input feature \(X\) and The adjacency matrix **A**. To avoid the impact of zero value in matrix multiplication we set a very small value \(\delta=0.01\) for the joints that are not connected. In addition to skeleton body joints, using derived time series data as input improves the overall performance of the network model. The first derived time series is computed as the angle between the vector from the knee to the hip and the vector from the knee to the ankle. The second time series was the difference between the x-coordinates of the left and right ankles. Our second model takes both types of input, i.e., skeleton joints and derived time series data, and predicts the gait parameters. As it is shown in Figure 4 a 1D convolutional neural network is applied on the derived time series data, and the output from this layer concatenated with the output from graph convolutional neural network.