# A Comprehensive Study on Deep Learning-Based 3D Hand Pose Estimation Methods
## 1 Introduction  
Markerless hand pose estimation can be defined as the task of predicting the position and orientation of the hand and fingers relative to some coordinate system, given an RGB image and/or volumetric data captured from a depth camera. It is an important research topic and even a fundamental one for the creation of immersive virtual and augmented reality (VR/AR) systems, efficient gesture-based human-computer interaction (HCI) systems and 3D humanoid reconstruction systems [1; 2; 3; 4]. Accurate hand pose estimation can enhance user experience in VR systems by enabling the performance of plausible and realistic virtual hand movements, as well as contribute towards a better understanding of human actions in smart HCI systems [5; 6; 7; 8], thus enabling a more intelligent interaction between users and smart systems. Apart from the aforementioned applications, hand pose estimation is crucial in a number of other tasks, such as gesture recognition [9; 10], action recognition [11; 12], support systems for motor impairments patients [13], sign language recognition [14; 15; 16; 17; 18], and representation [19]. In sign language recognition especially, accurate hand pose estimation is beneficial for promoting social inclusion and enhancing accessibility in the Deaf community. Although there has been a rapid progress in hand pose estimation, it still remains a challenging task with unique characteristics and difficulties, due to hardware limitations and constraints manifesting from the physiology of hands. The most important challenges that are present when implementing a markerless hand pose estimation method are the following:* High articulation: A human hand has more than 20 degrees of freedom (DOF) [20], therefore, a lot of parameters are needed in order to properly model the complexity of hand, as well as finger movements. * Occlusions: During the performance of a gesture, fingers of the same hand may be strongly occluded by each other, other body parts, or objects. This could potentially result in hidden hand parts or different fingers mistakenly inferred in the same location. * Low resolution: The size of a hand, let alone a finger, occupy a small area in an RGB image or a depth map, if it is not the focus of attention or there is some distance between the camera and the hand. Additionally, limitations in camera technologies (e.g., lens resolution, depth sensing capabilities) may lead to inaccurate data, further hindering the results of hand pose estimation methods. * Annotated data: The creation of annotated 3D data is a costly and time-consuming task. In order to capture accurate 3D data, an expensive marker-based motion capture system or a massive multi-view camera setting is required [21]. * Rapid hand and finger movements: Usually, hand motion is highly non-monotonic and consists of very fast movements, sudden stops, and joint rotations that are not met in other body parts. Currently, most conventional depth and RGB cameras can support 30 to 60 frames per second (fps) but still fall short to capture the speed of hand motions, resulting in blurry frames or uncorrelated consecutive frames. Recently, the inclusion of neural networks in computer vision tasks, such as image classification [22; 23], object detection [24; 25], semantic segmentation [26; 27], human pose estimation [28; 29], camera recalibration for mobile robotics, navigation and augmented reality [30], etc., has yielded state-of-the-art results in such tasks. Moreover, recent end-to-end Deep Neural Networks (DNNs), e.g., convolutional neural networks (CNNs) [31], recurrent neural networks (RNNs) [32], auto-encoders [33], and generative adversarial networks (GANs) [34], have proven to be capable of extracting much more meaningful features from the available data than previous handcrafted techniques. The employment of such networks has further raised the performance of methods that engage with complex data, such as RGB images. Consequently, hand pose estimation has also progressed rapidly by the use of neural networks [35; 36; 37].