# 3D Skeletal Joints-Based Hand Gesture Spotting and Classification
## 3 Proposed Algorithm
### Gesture Classification  
The isolated gestures segmented by the present gesture spotting module are classified into individual gesture classes in the gesture recognition module. The proposed gesture recognition module is a multi-model network called the M-3D network. This model is based on a multi-channel network with three different data modalities, as shown on the right of Figure 1.  
In our approach, from each frame of a video, we extract optical flow, 3D pose (hand joint, thumb tip, and index fingertip joint) information of multi-channel features input to the model. Optical flow is determined by two adjacent frames. There are some existing methods of optical flow extraction such as Farneback [28], MPEG flow [29], and Brox flow [30]. The quality motion information of optical flow clearly affects the performance of the gesture recognition model. Therefore, the Brox flow technique is applied to our approach as it has better quality performance compared to other optical flow extraction techniques. While the key hand and finger joints positions are extracted by the 3D human pose and 3D hand pose extraction network presented in Section 3.1, we only focus on the two most important joints of thumb tip and index fingertip which can describe all gesture types. Our gesture classification algorithm is based on the combination of three 3D_ResNet stream networks of RGB, Optical Flow, Depth channels with an LSTM network of 3D key joint features. * **Three stream RGB, Optical Flow, and Depth 3D_ResNet networks:** The 3D_CNN framework is regarded as one of the best frameworks for spatiotemporal feature learning. The 3D_ResNet network is an improved version of the residual 3D_CNN framework based on ResNet [31] architecture. The effectiveness of 3D_ResNet has been proved by remarkable performance in action video classification. The single 3D_ResNet is described in Figure 3. The 3D_ResNet consists of a 3D convolutional layer and is followed by a batch normalization layer and rectified-linear unit layer. Each RGB and Optical Flow stream model is pre-trained on the largest action video classification dataset of the Sports-1M dataset [32]. Input videos are resampled into 16 frames-clips before being fed into the network. Let a resampled sequence of 16 frames RGB frames be \(V_{c}=\{x_{c1},x_{c2},\ldots,x_{c16}\}\), Optical Flow frames be \(V_{of}=\{x_{of1},x_{of2},\ldots,x_{gt16}\}\) and Depth frames be \(V_{d}=\{x_{d1},x_{d2},\ldots,x_{d16}\}\) and operation function 3D_ResNet network of RGB, Optical Flow and Depth modalities be \(\Theta_{c}(. )\), \(\Theta_{oi}(. )\) and \(\Theta_{d}(. )\), respectively. Hence, the prediction probability of two single networks for i classes is  
\[P_{c}\{p_{1},\ p_{2},\ \ldots,\ p_{16}|V_{c}\}=\Theta_{c}(V_{c}) \tag{6}\]  
\[P_{of}\Big{\{}p_{1},\ p_{2},\ \ldots,\ p_{16}\Big{|}V_{of}\Big{\}}=\Theta_{of} \Big{(}V_{of}\Big{)} \tag{7}\]  
\[P_{D}\{p_{1},\ p_{2},\ \ldots,\ p_{16}|V_{D}\}=\Theta_{D}(V_{D}) \tag{8}\]  
where \(p_{i}\) is the prediction probability of video belonging to the \(i\)th class. * **LSTM network with 3D pose information:** In dynamic gesture recognition, temporal information learning plays a critical role in the performance of the model. In our approach, we utilize the temporal features by tracking the trajectory of the hand palm together with the specific thumb tip and index fingertip joint. LSTM framework is suitably proposed to learn the features for the gesture classification task. The parameters of our LSTM refer to the approach [33]. Input vectors from a sequence of the LSTM network frames is defined as: \(V_{j}=\{v_{j1},\ v_{j2},\ \ldots,\ v_{16}\}\) where: \(v_{jk}=\{h_{k}(x_{ik},\ y_{ik},\ z_{ik})\), \(v_{j2},\ldots,v_{j16}\}\) where: \(v_{jk}=\{h_{k}(x_{ik},\ y_{ik},\ z_{ik})\), \(h_{k}(x_{ik}  
\(f_{\eta}(x_{\text{\emph{IR}}},y_{\text{\emph{IR}}},z_{\text{\emph{IR}}})\), \(f_{\eta}(x_{\text{\emph{ink}}},y_{\text{\emph{ink}}},z_{\text{\emph{ink}}})\)) is a \(9\times 1\) vector which contains 3D position information of key joints at kth frame.