# A Comprehensive Study on Deep Learning-Based 3D Hand Pose Estimation Methods
## 3 Deep Learning Methods
### RGB-Based Approaches
#### 3.2.2 Model-Based Approaches  
One of the first deep learning RGB-based methods that made use of a hand model was the work of [PERSON] et al. [121]. The authors presented a method that employed the detection network YOLO v2 [122] for hand detection, a pretrained OpenPose [28] network for 2D joint positions prediction and an assembled geometric primitives hand model with 26 DOFs and 21 joints. The hand model was fitted by minimising the distances of the projections of its joints to the 2D joints predictions acquired from OpenPose, using non-linear least-squares minimization. In Reference [123], [PERSON] et al. proposed the creation of a synthetic dataset, with an additional GAN for the translation of synthetic hand images to realistic hand images. The translation GAN was trained using the cycle consistency with a geometric consistency loss that preserved the hand pose during translation. A regression CNN was trained on the synthetic data, in order to predict 2D joint heatmaps, as well as 3D positions relative to the root joint. The 2D/3D predictions were utilized during the kinematic hand model fitting to ensure the anatomically correctness of the resulting hand poses. In Reference [124], [PERSON] et al. incorporated the MANO hand model in their 3D hand mesh recovery method, called Hand Mesh Recovery (HAMR) framework. The mesh representations were extracted by regressing the mesh and camera parameters of MANO. The network was trained with five loss terms: a 2D heatmap loss, a camera perspective loss, a 2D keypoint projection loss, a 3D joint position loss, and a geometric constraints loss. The 3D joint locations were estimated by linear interpolations between the vertices of the mesh. The MANO model was also employed by [PERSON] et al. [125] to predict hand pose and shape from images in the wild. A ResNet-50 was used as the encoder network and generated the pose, shape and view parameters that the decoder, consisting of the MANO model and a reprojection module, processed in order to regress the final hand pose and shape estimation. By estimating the view parameters, the authors were able to project the estimated hand pose to 2D image coordinates and leverage 2D weak annotations as supervision. The encoder network was pretrained on a synthetic dataset created by the authors, in which hand images were paired with ground-truth camera and hand parameters. The training procedure was supervised by a combination of four loss terms: a 2D joint projection loss, a 3D joint loss, a hand mask loss, and a regularization loss that was applied on the encoder's hand model parameters outputs to reduce their magnitude and allow only physically plausible hand reconstructions. [PERSON] et al. [126] proposed a method that leverages the MANO hand model with 3D supervision, as well as hand segmentation masks and 2D supervision, to train a hand pose estimation CNN. Their network took an RGB image as input, estimated the 2D joint positions, and used them along with the image feature vector to predict the camera and the hand model parameters. At testing time, the initial 3D mesh estimation was further refined using gradient descent. The gradients were computed by projecting the predicted 3D mesh to 2D and comparing them with the predicted 2D joint locations. Differently from previous approaches, the 2D joints and segmentation masks were not kept fixed during the refinement step; instead, they recursively improved the 2D predictions. While previous model-based methods employed a hand model to constrain the predicted poses to be humanly feasible, the structure of hands has not been exploited thoroughly. [PERSON] et al. [127] presented an adversarial method where a generator predicted an initial prior pose using the MANO hand model and then a Graph Convolutional Network (GCN) refinement module was employed to further refine the prior pose. In particular, the GCN refinement module used the prior pose and the image features as inputs and predicted a deformation that was added to the features of the initial prior pose.