# 3D Skeletal Joints-Based Hand Gesture Spotting and Classification
## 3 Proposed Algorithm
### Gesture Spotting  
The gesture spotting module is shown on the left of Figure 1. All frames of continuous gesture sequence are utilized to extract 3D human pose using the algorithm proposed in [23]. Through RGB hand ROI localized from 3D hand palm position \(J_{h}(x,y,z)\) when the hand palm stands still and over spine base joint, we use a 3D hand pose estimation algorithm to effectively extract the 3D position of the finger joints. From the extracted 3D human pose, the hand speed \(v_{hand}\) is estimated using the movement distance of the hand joint between two consecutive frames. * **3D human pose extraction:** From each RGB frame, we obtain a 3D human pose by using one of the state-of-the-art methods for 2D/3D human pose estimation in the wild-pose-hgreg-3d network. This network has been proposed by [PERSON] et al. [23] which provides the pre-trained model on the Human3.6M dataset [24]. This is the largest dataset providing both 2D, 3D annotations of human poses in 3.6 million RGB images. This network is a fast, simple, and accurate neural network based on 3D geometric constraints for weakly-supervised learning of 3D pose with 2D joint annotations extracted through the state-of-the-art of 2D pose estimation method, i.e., stacked hourglass network of [PERSON] et al. [25]. In our proposed approach, we use this 3D human pose estimation network to extract the exact 3D hand joint information, which is effectively utilized for both gesture spotting and gesture recognition task. Let \(J_{h}(x_{hk},y_{hk},z_{hk})\), \(J_{h}(x_{hk-1},y_{hk-1},z_{hk-1})\) be the 3D position of the hand joint at the \(k\)th frame, and \((k-1)\)th frame, respectively. The hand speed is estimated as  
\[v_{hand}=\alpha\cdot\sqrt{\left(x_{hk}-x_{hk-1}\right)^{2}+\left(y_{hk}-y_{hk-1 }\right)^{2}+\left(z_{hk}-z_{hk-1}\right)^{2}} \tag{1}\]  
where \(\alpha\) is the frame rate. The finger speed is estimated by the change in distance between the 3D position of fingertips of the thumb and the index finger in sequence frames. Let denote \(J_{fh}(x_{hk},y_{hk},z_{hk})\), \(J_{fh}(x_{ink},y_{ink},z_{ink})\) the 3D position fingertips of the thumb and the index finger at the \(k\)th frame, respectively. The distance between the two fingertips at the \(k\)th frame is given as  
\[d_{fk}=\sqrt{\left(x_{ftk}-x_{ink}\right)^{2}+\left(y_{ftk}-y_{ink}\right)^{2}+ \left(z_{ftk}-z_{ink}\right)^{2}} \tag{2}\]  
where \(d_{fh}\) and \(d_{fh-1}\) represent the distances of the \(k\)th frame and previous frame, respectively, the finger speed \(v_{finger}\) is estimated as  
\[v_{finger}=\alpha\cdot\left(d_{fk}-d_{fk-1}\right) \tag{3}\]  
The function utilizes \(v_{hand}\) and \(v_{finger}\) extracted from each frame:  
\[v_{k}=v_{hand}+v_{finger} \tag{4}\]  
and is used as the input of the Bi-LSTM network to spot gestures from video streams, as shown in Figure 2. In our network, the Connectionist temporal classification [12] CTC loss is used to identify whether the sequence frames are in gesture frames or transition frames. * **3D hand pose extraction:** Using the hand palm location detected by the 3D human pose estimation network, we extract hand ROI and use it for 3D hand pose extraction when the hand palm stands still over the spine base joint. We also estimate the 3D hand pose by using the real-time 3D hand joints tracking network of OccludedHands proposed by [PERSON] et al. [26] and further additionally fine-tuned it with the hand pose dataset of Stereo Hand Pose Tracking Benchmark [27]. In this method, they utilized both RGB and Depth information to robustly and accurately localize the hand center position and regress the 3D joint from the 2D hand position heat-map. Firstly, they used a CNN network called HALNet to estimate the heat-map of the hand center and then crop the hand region. Secondly, they applied another CNN network called JORNet for a hand cropped frame to generate a heat-map of 2D hand joints and regress 3D hand joint positions from it.