# A Comprehensive Study on Deep Learning-Based 3D Hand Pose Estimation Methods
## 3 Deep Learning Methods
### RGB-Based Approaches
#### 3.2.1 Model-Free Approaches  
One of the first deep learning methods that directly estimated 3D hand pose from RGB images is the work of [PERSON] et al. [43]. In their paper, the authors designed a network consisting of three sub-networks: a hand segmentation net that detected the hand and according to the predicted hand mask segmented the image, a 2D keypoint detection network that produced score maps for each joint and a 3D pose regression network that took the score maps and predicted the canonical 3 Djoint locations and a transformation matrix to the world coordinates. Additionally, the authors also developed a large synthetic dataset that facilitated the training process. [PERSON] et al. [116] proposed a 2.5D pose representation from single RGB images that was then reconstructed to 3D. For the purpose of obtaining 2.5D pose representations, both 2D joint locations, as well as depth values, are needed, so the authors designed a new heatmap representation, referred as 2.5D heatmap that consisted of a 2D heatmap and a depth map for each joint. However, no depth data were required during training, since the network learned to produce depth maps intrinsically. The network architecture consisted of an hourglass network which produced a latent 2.5D representation of the input RGB image, containing the latent 2D heatmaps and depth maps. [PERSON] et al. [117] proposed a variant of the above mentioned network, since they considered 2.5D representations to be better suited with the set of loss terms that they developed. Those terms implicitly enforced biomechanical constraints on the 3D joint predictions, even for networks trained only with 2D supervision. Such biomechanical constraints supervise the range of valid bone lengths, the range of valid palm structure, and the range of valid joint angles of the thumb and fingers, and, as a result, the final predictions were constrained to anatomically plausible poses. To differentiate from their previous work that relied on carefully designed loss terms to constrain the predicted poses to plausible ones, [PERSON] et al. [49] proposed a method that involved latent space alignment between the 3D pose space and the input data space. The authors considered as input data 2D keypoints, RGB images and to evaluate the generality of their method, 2D depth images. However, their main focus was on RGB images. The network consisted of two VAE networks, one for building the pose latent space and one for creating the input data space. Each VAE encoded and decoded the data to their respective modality, i.e., \(RGB\to RGB\) and \(3D\to 3D\). The latent space alignment was achieved by using the trained decoder of the \(3D\to 3D\) VAE and the trained encoder of the \(RGB\to RGB\) VAE. In order to align the latent space of 3D hand poses and RGB images, [PERSON] et al. [118] proposed a novel variational framework that was trained in two distinct phases. During the first training stage, two VAEs were independently trained to create the cross-modal latent space of \(RGB\to 3D\) and the single-modal latent space of \(3D\to 3D\). Subsequently, in the next training phase, the above branches remained frozen and a variational module (mapper) was introduced with the purpose to map the cross-modal latent space to the well-structured and more informative single-modal one. Despite the fact that this framework was generic and not explicitly designed for the 3D hand pose estimation task, since they presented experiments on food image analysis datasets too, it yielded state-of-the-art results on the task of RGB-based 3D hand pose estimation among model-free approaches. [PERSON] et al. [119] proposed a method that constructed a refined latent space by disentangling the representations of 3D hand poses and hand images, called disentangled VAE (dVAE). Each latent space was decomposed to its meaningful components; the 3D pose space was decomposed to the canonical coordinates and the viewpoint factor spaces and the RGB space to the world coordinates 3D hand pose space and background.