# A Comprehensive Study on Deep Learning-Based 3D Hand Pose Estimation Methods
## 3 Deep Learning Methods
### Depth-Based Approaches
#### 3.1.2 3D Representation Utilization  
There are two 3D representations that a 2D depth map can be converted to: (a) point clouds and (b) 3D voxels. Most works that leveraged 3D data utilized point clouds, while 3D voxels were not explored as much. [PERSON] et al. [67] converted the 2D depth images to point clouds using the camera's intrinsic parameters (i.e., principal point location and focal length) and projected it to \(x\), \(y\), and \(z\) axis in order to get multiple views of the same hand pose. Three identical CNNs were employed to extract heatmaps from each projected view, which were later fused together in order to estimate the 3D hand pose. Even though [PERSON] et al. utilized the resulting point cloud of a depth image, they did not work directly with 3D points, since the designed network still required as input a 2D depth map. The development of PointNet [101], a network that works directly on point clouds, motivated works that utilized point clouds at input level. Later on, [PERSON] et al. [36] proposed a hierarchical PointNet, the Hand PointNet, that takes \(N\) points as input and extracts the hand features in a hierarchical way, which enabled better generalization. Additionally, a fingertip refinement network processed the neighboring points of the estimated fingertips and further refined the hand pose. The global hand orientation was estimated by an OBB (oriented bounding box) by performing PCA on the 3D coordinates of the points that the OBB surrounded. The final 3D hand joint locations were estimated by direct regression. In a later work, [PERSON] et al. [99] used a stacked PointNet architecture to directly take \(N\) sampled and normalized points from the point cloud as input and produce a set of heatmaps, as well as unit vector fields. These outputs were then put to use, in order to infer the offset of each sampled point from the hand joints and estimate the 3D coordinates of the latter. On the other hand, [PERSON] et al. [102] proposed a new network structure and used a spatial transformer to further improve the performance of the network. The designed framework consisted of a semantic segmentation network and a pose regression network. Two transformation matrices prediction networks were also implemented. The semantic segmentation network predicted for each point its corresponding semantic label. The pose regression network, using the semantic segmentation network's output together with the original point cloud, estimated the hand joint locations. The transformation matrices prediction networks estimated the necessary transformation matrices to normalize the input points and transform the 3D joints to the original space. Both the semantic segmentation and the regression network utilized the PointNet architecture. [PERSON] et al. [103] utilized a modified version of the PointNet++ [104] architecture, as well as an ensemble strategy, where they proposed \(n\) possible poses for a hand point cloud and weighted them, in order to get the final predicted 3D joint locations. They also presented a data augmentation method that divided the hand point cloud into 16 parts based on Euler distance and then bent the fingers according to kinematics constraints, thus creating a new gesture. In order to overcome PointNet's need for a meticulous preprocessing of the input point cloud, [PERSON] et al. [105] proposed a residual Permutation Equivariant Layer (PEL) [106] deep network to compute point-wise features and feed them to a point-to-pose voting module. The point-to-pose module consisted of 2 fully connected layers, one responsible for the computation of the importance of each point to the final pose estimation and the other for the production of 1-dimensional heat vectors or direct 3D location values, depending on the selection between detection and regression. In their work, the regression-based variation slightly outperformed the detection-based variation. [PERSON] et al.