# A Comprehensive Study on Deep Learning-Based 3D Hand Pose Estimation Methods
## 3 Deep Learning Methods
### Depth-Based Approaches
#### 3.1.2 3D Representation Utilization
[107], in order to utilize depth images without 3D annotation, proposed a hand pose encoder-decoder network, where the encoder hierarchically extracted a vector representation from the point cloud and the decoder reconstructed the point cloud from the extracted vector. The encoder architecture was based on the SO-Net [108], which built a self-organizing map (SOM) [109] and performed a hierarchical feature extraction of a point cloud and SOM nodes. Additionally, a hand pose regression module made use of the encoder's output by concatenating the extracted vector with the latent point cloud features, to directly estimate the output hand pose. The initial input were \(N\) sampled points from the point cloud, with \(N=1024\). Different from the above, [PERSON] et al. [64] cast the objective of 3D hand pose estimation to a voxel-to-voxel likelihood for each joint. The 2D depth map was projected into a 3D voxel representation and then fed into a 3D CNN encoder-decoder network that was based on the hourglass network [98] and directly predicted the per-voxel likelihood for each keypoint. The voxel with the highest likelihood was considered to be the joint location and as such, it was wrapped to the real world coordinates. The authors suggested that, by working with voxels instead of a 2D depth map,they eliminated the perspective distortion that may occur when there is a direct regression from 2D to 3D. Similarly, [PERSON] et al. [110] used a 3D hourglass model to directly regress 3D joint heatmaps. In order to add extra hand skeleton constraints, heatmaps for each bone were also estimated by their network, which worked as intermediate supervision. The number of bones varied depending on the dataset, for example, in the MSRA dataset, where there are 21 joints, the number of bones was set to 20. [PERSON] et al. [111] proposed a hand pose estimation network where intermediate supervision in the form of dense feature maps constrained the feature space of the joint predictions. The dense feature maps were explicitly designed to incorporate the geometric and spatial information of hand joints. Even through they also experimented with using 3D voxels as input, the best results were acquired by employing the projection technique of Reference [67] in combination with the outputs of Reference [64] as supervision feature maps. [PERSON] et al. [112] proposed a 3D CNN-based hand pose estimation method that could directly regress 3D joint locations from 3D voxels. More specifically, they represented the 3D hand as a 3D volume of projective Directional Truncated Signed Distance Function (D-TSDF) [113] values, which were fed into three 3D CNN layers and three fully connected layers. The authors suggest that a 3D CNN with 3D volumes as input, is easy to train end-to-end. In Reference [50], the same authors expended their previous work. The output of their 3D CNN network was a low dimensional representation of 3D hand joint relative locations in the 3D volume. By performing PCA reconstruction and coordinate transformations, they obtained the 3D hand joint locations in the camera's coordinate system. Additionally, they took into consideration the complete hand surface as intermediate supervision. A 3D U-Net [114] was applied on the input voxels in order to generate a truncated distance function (TDF) volume of complete hand surface points which was later concatenated with the original input voxel volume. A 3D DenseNet further processed the concatenated volumes and predicted the desired 3D hand pose. Both 3D U-Net and 3D DenseNet are pretrained separately. More recently, [PERSON] et al. [115] introduced a novel 3D CNN architecture to estimate 3D shape and 3D hand pose from voxelized depth maps. Initially, they employed a voxel-to-voxel pose regression network to estimate 3D joint heatmaps. The predicted 3D heatmaps coupled with the voxelized depth map were then processed by two sub-networks in order to obtain the voxelized shape and the hand surface.