# Robust Hand Shape Features for Dynamic Hand Gesture Recognition Using Multi-Level Feature LSTM
## 6 Experiments and Discussion
### Setup Environments, Metrics, and Training Parameters  
Environments: Our program was developed with Python 3.5 using the TensorFlow Keras framework to build the deep learning models. Our program ran on a desktop PC with Intel Corei7 8700k with 32 GB of RAM and one graphic card GeForce GTX 1080 Ti.  
Metrics: For the depth shape model in the hand component segmentation, we used the metric mean IoU to evaluate our segmentation results. This is the intersection over union between the ground-truth and prediction on every hand region. We used eight hand regions: background, palm, forearm, thumb, index, middle, ring, and pinky.  
\[MeanIoU=\sum_{i=1}^{C}\frac{TP_{i}}{(TP_{i}+FP_{i}+FN_{i})}100 \tag{46}\]  
where \(C\) is the number of hand regions and \(TP_{i}/FP_{i}/FN_{i}\) the true positive/false positive/false negative for region \(i\).  
\begin{table}
\begin{tabular}{c c c c} \hline
**14 Classes** & **28 Classes** & **Gesture** & **Label** \\ \hline
1 & 1, 2 & Grab & Fine \\ \hline
2 & 3, 4 & Expand & Fine \\ \hline
3 & 5, 6 & Pinch & Fine \\ \hline
4 & 7, 8 & Rotation CW & Fine \\ \hline
5 & 9, 10 & Rotation CCW & Fine \\ \hline
6 & 11, 12 & Tap & Coarse \\ \hline
7 & 13, 14 & Swipe Right & Coarse \\ \hline
8 & 15, 16 & Swipe Left & Coarse \\ \hline
9 & 17, 18 & Swipe Up & Coarse \\ \hline
10 & 19, 20 & Swipe Down & Coarse \\ \hline
11 & 21, 22 & Swipe X & Coarse \\ \hline
12 & 23, 24 & Swipe V & Coarse \\ \hline
13 & 25, 26 & Swipe + & Coarse \\ \hline
14 & 27, 28 & Shake & Coarse \\ \hline \end{tabular}
\end{table}
Table 2: List of gestures in the Dynamic Hand Gesture (DHG) dataset.  
Figure 11: A sample hand posture in the DHG dataset: (**a**) hand depth data with drawing the hand bounding box and the 22 hand joints; (**b**) hand regions in zoom mode; (**c**) 22 hand joints with 0 (wrist), 1 (palm), 2–5 (thumb), 6–9 (index), 10–13 (middle), 14–17 (ring), and 18–21 (pinky).  
For the dynamic hand gesture recognition, we quantified them based on the accuracy between the prediction and ground-truth and the time cost for predicting a gesture.  
Parameters in training: We performed a temporal augmentation on the sequence length of a hand gesture by randomizing the position of the first frame and getting 32 frames with equal step sizes. For the spatial augmentation of the depth data, we used basic transforms, such as random rotation by 45 degrees, translation by 5%, and scaling by 10%, based on the frame size.  
For training the model, we used Adam [45] with a learning rate of 0.001 and for the first time training reducing the learning rate on the plateau. For the fine-tuning step in the previous training, we used SGD [46] to train with a learning rate ranging from 0.004 to 0.0001 using the cosine annealing learning rate schedule.  
Experimenting with the features and models: We conducted the experiments based on the list of features shown in Table 3.  
There was a total of five hand features from the two input types, skeleton and depth. We divided the groups of features as the motion group (learning the global motion of hand gestures) with feature motion, hand shape (capturing the changes of hand components) with feature skeleton shape, joint point-cloud, and depth shape, and the others with the feature input by the normalizing points.  
For the experiments on our proposed models, we divided our proposed model into with/without Con1D-2D pyramid blocks, as in Table 4.