# A Comprehensive Study on Deep Learning-Based 3D Hand Pose Estimation Methods
## 3 Deep Learning Methods
### Depth-Based Approaches
#### 3.1.1 2D Depth Map Utilization  
Historically, the estimation of 3D hand joint locations was performed by leveraging depth images. Early deep learning methods did not deviate in that matter and employed depth data, as well, in the form of 2D depth images. In one of the earliest works, [PERSON] et al. [42] proposed the creation of a labeled ground-truth dataset of hand gestures by automatically annotating depth maps with the use of a Linear-Blend-Skinning (LBS) hand model, thus forming the NYU dataset. This dataset was then employed to train a 2-stage CNN network that outputs probability heatmaps for each joint. [PERSON] et al. [35] investigated different CNN network structures in order to find the most efficient one and introduced a pose prior module denoted as DeepPrior, which is effectively a bottleneck linear layer with significantly less neurons than the total number of joints (\(\ll\)3 J). The weights of the bottleneck layer were initialized by a PCA applied to the 3D hand pose data. The same authors, in DeepPrior++ [83], extended DeepPrior in three ways: (a) they incorporated the ResNet architecture into the network, (b) they introduced additional data augmentations during training, and (c) they improved the hand detection sub-task by introducing a CNN detection refinement network. Both DeepPrior and DeepPrior++ tried to physically constrain the predicted hand pose by the use of the bottleneck layer. [PERSON] et al. [84] proposed to additionally take into consideration the functionality and importance of each finger. They designed a deep CNN network, named as Hand Branch Ensemble (HBE), that had three branches: one for the thumb, one for the index, and the last one to express the rest of the fingers. The features of each branch were fused together and projected to a lower dimensional representation by a similar bottleneck layer and further processed in order to directly regress the 3D joint positions. [PERSON] et al. [85] proposed an approach that applied domain transfer between synthetic and real depth images. More specifically, the authors rendered the 3D pose over the real images in order to create the corresponding synthetic ones and used a network to map the extracted features of the real images to the feature space of the synthetic ones. Accordingly, they trained a network to minimize the distance between the mapped and the synthetic features and a decoder trained on the synthetic images to infer the 3D pose. Another way to constrain the final predicted hand pose is the use of intermediate heatmaps to guide the regression of the 3D hand joints. [PERSON] et al. [PERSON] et al. (2018) decomposed the hand pose estimation task into two sub-tasks, palm pose estimation and finger pose estimation, as well as the integrated heatmaps as constraints during the feature extraction phase from the input 2D depth map. Differently, [PERSON] et al. [PERSON] et al. (2018) proposed a re-parameterization layer to estimate intermediate representations, such as 3D heatmaps and 3D unit vector fields from joint positions. Their network design consisted of stacked joint regression modules with re-parameterization layers, except from the final regression module that directly outputted the final hand pose prediction. Some works chose to employ a hand model in order to geometrically constrain and ensure the validity of the final hand pose predictions. [PERSON] et al. [PERSON] et al. (2018) used a method similar to Reference [PERSON] et al. (2018) CNN architecture to estimate the pose parameters of a hand model in joint-angle format. The hand model was from libhand [PERSON] et al. (2018) and has 23 joints with 26 DOFs. A differentiable forward kinematics function was employed to map pose parameters to 3D joint positions. [PERSON] et al. [PERSON] et al. (2018) extended this work in order to generalize over different hand shapes, by jointly estimating 3D hand pose and bone-lengths of the hand skeleton.