# A Comprehensive Study on Deep Learning-Based 3D Hand Pose Estimation Methods
## 3 Deep Learning Methods
### Depth-Based Approaches
#### 3.1.1 2D Depth Map Utilization
The VAE part of Crossing Nets was responsible for the generation of the hand poses latent space, creating the distribution \(Z_{y}\). The GAN part, sampled from the distribution of the latent space of depth images \(Z_{x}\), in order to synthesize a depth map close to the original one. A fully connected layer was used to map the two spaces together. By mapping the two latent spaces, the network combined the generalization capabilities of the GAN part and the implicit pose constraints learned by the VAE and improved on the discriminative pose estimation. [PERSON] et al. [51] proposed a method to make use of the abundance of synthetic samples datasets in order to accurately estimate the hand pose of real samples. Therefore, they created a shared latent space between three different domains: synthetic depth images, real depth images, and poses. The pose latent space was created by a VAE, whereas the real and synthetic images domain by a VAE-GAN hybrid network. The cycle consistency and weight sharing constraints [96] were put to use in order to learn the shared latent space of real and synthetic depth images. [PERSON] et al. [52], in contrast to the above mentioned generative works, proposed to augment the latent space of training examples by adding variations in viewpoints and shapes in the 3D pose domain. Afterwards, a GAN was trained to synthesize depth maps for the newly created 3D poses, as there were no corresponding depth map entries. The hand pose estimator and the hand pose generator were trained in a unified manner, enforcing the cycle consistency constraint. A method that did not employ any of the constraints that were used in the aforementioned works is the one proposed by [PERSON] et al. [97]. The authors proposed a dense pixel-wise estimation method, where a 2D CNN stacked hourglass network [98] estimated 2D heatmaps, 3D heatmaps, and dense 3D unit vectors, from a given 2D depth image. The 3D heatmaps coupled with the 3D unit vectors effectively compose a 3D offset vector between the 2D depth image pixels and the hand joints. The authors argued that working with offset vectors made the estimations translation-invariant and boosted the method's generalization capabilities to different combinations of finger poses. Pixel-wise estimations, in general, are time-consuming, and estimations for background pixels may distract the neural network from learning effective features in the hand region [99]. [PERSON] et al. [100] proposed a method that did not densely regressed pixel-wise predictions, instead predicted 3D joint positions by aggregating the estimations of multiple anchor points. The main idea was to densely position anchors on the depth image and estimate the 3D joint positions by aggregating the estimations of multiple anchor points. Thus, the anchor points can be viewed as the local regressors with different viewpoints to the regressed joint. The network consisted of 3 branches, one for depth estimation, one for weighting the anchors of a joint (Anchor proposal branch), and an in-plain offset branch which computed the offset from an anchor point towards a joint. Afterwards, this method used the anchor proposals to locally regress the 3D location of a joint.