# Robust Hand Shape Features for Dynamic Hand Gesture Recognition Using Multi-Level Feature LSTM
## 4 Hand Posture Feature Extraction
### Depth-Shape Feature Model  
The depth shape feature model \(f_{depth\_shape}\) plays the role of the feature extraction block to obtain the feature vector that presents the hand shape in the depth data. The depth shape feature model in our study is based on the U-Net architecture [39] to learn the hand components from segmenting hand regions from depth data. It can be expressed as:  
\[\mathbf{D}^{t}_{depth\_shape} =f^{e}_{depth\_shape}\left(\mathbf{D}^{t}\right) \tag{38}\] \[\mathbf{H}^{t}_{mask} =f^{d}_{depth\_shape}\left(\mathbf{D}^{t}_{depth\_shape}\right)\] (39) \[\mathbf{D}_{depth\_shape} =\left\{\mathbf{D}^{t}_{depth\_shape}\right\} \tag{40}\]  
or:  
\[\mathbf{H}^{t}_{mask}=f^{d}_{depth\_shape}\left(f^{e}_{depth\_shape}\left(\mathbf{D }^{t}\right)\right) \tag{41}\]  
where \(f^{e}_{depth\_shape}\) is the encoder function to encode the depth data as the depth shape feature \(\mathbf{D}^{t}_{depth\_shape}\) at time \(t\) while \(f^{d}_{depth\_shape}\) is the decoder function to map the encoder feature to the hand component masks \(\mathbf{H}^{t}_{mask}\) by one-hot encoding.  
U-Net, as shown in Figure 6, consists of the encoder and decoder blocks and the skip connections between them. The structure of the encoder can be based on the common visual image classification models such as VGG16 [40], Resnet50 [41], etc. The backbone of the encoder using VGG16, as shown in Figure 6, has five blocks of two convolution layers, batch-normalization, and max-pooling. The encoder block converts the depth data input into the encoded features, then the decoder blocks convert the encoded features into the semantic representation of the input data with the hand component masks in the background, palm, thumb, index, middle, ring, and finger regions. The role of the skip connections helps our model to be trained stably and achieve a better result by passing the features from the encoder to the decoder at the same levels concurrently with the features from the decoders below. Through the skip connection, the model can keep learning even when the deeper encoder and layer cannot learn by the dying ReLU problem or the vanishing problem.  
Figure 6: Hand component segmentation model to extract depth shape features.  
The middle block between the encoder and decoder blocks is the feature block to return the feature vector. To enhance the feature vector, the hierarchical depth shape feature combines Blocks 3 and 4 of the encoder, as shown in Figure 6.  
For the dataset for training the depth shape, it should focus on the hand components instead of only the hand pose due to the complex structure of the hand such as the small size and self-occlusions. In this study, Finger-Paint [42] is a suitable dataset with all the requirements.  
For the loss function, the soft Dice loss [43] is a good choice in the unbalanced cases among the segmentation regions. Since the palm region often has a larger size compared to finger regions in the hand components, the loss function measures the overlap between the two regions the object region and the non-object region, in the binary classification. In the multi-class classification problem, the final score will be averaged with the Dice loss of each class expressed as:  
\[L\left(y_{true},y_{pred}\right)=1-\frac{1}{N_{c}}\sum_{c=1}^{N_{c}}\frac{2 \sum_{pixels}y^{c}_{true}y^{c}_{pred}+\epsilon}{\sum_{pixels}\left(y^{c}_{ true}\right)^{2}+\sum_{pixels}\left(y^{c}_{pred}\right)^{2}+\epsilon} \tag{42}\]  
where \(c\) is the region of the hand components including the \(N_{c}\) regions (background, palm, thumb, index, middle, ring, pinky); \(y^{c}_{true}\) and \(y^{c}_{pred}\) are the ground-truth and the prediction of the hand component masks, respectively, in region \(c\).