# A Comprehensive Study on Deep Learning-Based 3D Hand Pose Estimation Methods
## 5 Conclusions and Future Directions

Table 2: Results of top 3D hand pose estimation methods in terms of accuracy, categorized by dataset and our proposed taxonomy. Results in mPEE are in mm and for Area Under the Curve (AUC) within 20–50 mm error threshold. Despite that, there are still some unresolved challenges that create a gap between research and practical 3D hand pose estimation applications. Most of the existing works assume that: (a) the hand is positioned in front of a clean background making it easy to segment, (b) only one hand is considered to be present in a scene, and (c) hand-to-hand or hand-to-object interactions are not well explored. Three-dimensional hand pose detection in the wild is far from perfect, unlike the results achieved in existing datasets. Additionally, even though most of the methods presented in this survey are capable of real-time performance on top-end PCs, applications for mobile devices still fall short, due to their limited processing capabilities. Moreover, selecting an optimal method for a task at hand depends highly on the requirements and constraints imposed by the task (e.g., cost, accuracy, speed, etc.). For instance, highly accurate 3D hand representation is currently achieved by gloves, although deep networks show promising results. On the other hand, real-time applications or low-specification systems require deep networks with minimum trainable parameters, usually sacrificing accuracy for speed. However, elaborate network architectures with high memory and processing footprint achieve very robust results. Furthermore, the state-of-the-art in the hand pose estimation field changes so rapidly that novel deep networks achieving state-of-the-art results are continuously emerging, rendering older networks obsolete. Furthermore, existing works have validated that training strategies, such as intermediate supervision or physiological constraints and network designs, such as residual feature aggregation, stacked network architectures, and hand model-based priors, improve the recognition performance. Figure 1(a) draws a comparison between various DNN architectures in terms of network's performance (Accuracy), number of research papers (Volume of works), scope for improvement based on the performance gains and the volume of works (Future potential), computational complexity during training (Training complexity), and time complexity of inference (Inference speed). As can be observed, the research community has extensively utilized 2D CNN architectures in the past, and now the interest has shifted towards other architectures, such as 3D CNN and generative models. Since the frameworks based on 2D CNNs contain a limited number of parameters compared to the rest of the methods, their inference speed is low, coming at a performance cost. Additionally, they do not introduce, in general, any extra component during training, e.g., discriminator, maintaining a relatively low training complexity. On the other hand, it is deduced that there is much more room for improvement on works that employ generative networks, such as VAEs and GANs, since they have shown promising results, even though they are not exhaustively explored. The construction of latent spaces, where similar input poses are embedded close to each other, proves that these networks can learn a manifold of feasible hand poses, further boosting 3D hand pose estimation results. Moreover, in order to create humanly interpretable and tunable latent representations, methods producing disentangled latent sub-spaces should be investigated thoroughly. However, the fact that they require more delicate training strategies, especially applied to GANs due to the adversarial training, consists a minor drawback leading to a rather high training complexity. To this end, we believe that there should be an extensive research focus on fully exploiting the representation capabilities of such methods and thus fulfilling their potentials. Figure 1(b) demonstrates some attributes for each of the three major proposed categories.