DOR3D-Net: Dense Ordinal Regression Network for 3D Hand Pose Estimation

Authors: yamin  mao
zhihua  liu
weiming  li
soonyong  cho
qiang  wang
xiaoshuai  hao
Venue: thecvf_com_CVPR_2024_Workshop_POETS
Published: 2024-04-11 09:36:27

Abstract: Depth-based 3D hand pose estimation is an important but challenging research task in robotics and autonomous driving community. Recently,  dense regression methods have attracted increasing attention in 3D hand pose estimation task, which provide a low computational burden and high accuracy regression way by densely regressing hand joint offset maps.  However, large-scale regression offset values are often affected by noise and outliers, leading to a significant drop in accuracy. To tackle this, we re-formulate 3D hand pose estimation as a dense ordinal regression problem and propose a novel  Dense Ordinal Regression 3D Pose Network (DOR3D-Net).  Specifically, we first decompose offset value regression into sub-tasks of binary classifications with ordinal constraints. Then, each binary classifier can predict the probability of a binary spatial relationship relative to joint, which is easier to train and yield much lower level of noise.  The estimated hand joint positions are inferred by aggregating the ordinal regression results at local positions with a weighted sum.  Furthermore, both joint regression loss and ordinal regression loss are used to train our DOR3D-Net in an end-to-end manner. Extensive experiments on public datasets (ICVL, MSRA, NYU and HANDS2017) show that our design provides significant improvements over SOTA methods. The source code will be released.

Main paper: 
 introduction: 
 As intelligent machines like robots and autonomous vehicles strive to be socially aware in human-centric environments, accurately perceiving human hands and their intentions becomes vital for robotics [8, 19] and autonomous driving [32, 34] . High-quality hand pose estimation is an important prerequisite for accurately perceiving and interpreting human hand movements. With the development of deep learning, hand pose estimation from RGB images [1, 14, 24, 35] and depth images [3, 21, 23, 30] have attracted much attention. This paper focuses on improving show predictions from A2J [30] and JGR-P2O [6] respectively and row 3 shows our predictions. For each row, column 1 shows the final result of prediction for an exemplar hand joint (superimposed on input depth image) and column 2 and 3 show the x-offset and y-offset maps respectively. For better comparison, bright and light yellow dots surrounded by red circles show the predict joint and ground truth respectively. Notice that several error areas present in the offset maps from A2J and JGR-P2O (highlighted with red and yellow boxes). In contrast, our probability map is clean. This is brought by our dense ordinal regression design and empowers our DOR3D-Net to surpass SOTA methods in public benchmarks.depth-based 3D hand pose estimation, which aims to output joint coordinates in 3D space from an input depth image.Existing hand pose estimation methods with dominant performance employ deep learning models of different structures. Some models [4, 33] extract deep representations and then directly regress the joint coordinates or other forms of hand model parameters. Differently, more recent models explore dense predictions. They usually use a dense grid and for each grid position predict an offset vector that points to a joint. The densely predicted offset vectors form an offset map and then are used to infer joint coordinates. For example, A2J [30] proposes dense anchors and aggregates the offsets of these anchors to estimate each joint in the image plane. JGR-P2O [6] regresses joints by weighted averaging over all pixels' offsets in both image plane and depth space. For these methods, depending on the distance between a grid position and its target joint, the offset value varies in a large interval, especially for high-resolution images. However, large-scale regression offset values are often affected by noise and outliers. These flaws are difficult to completely remove and will propagate to subsequent steps resulting in degradation in the estimated joint accuracy. In this paper, we explore ordinal constraints to improve dense prediction methods for hand pose estimation. Specifically, as a point traverses in space along the scanline, the spatial relationship between the point's position to a target joint should vary smoothly with strict ordinal constraints. A closely related previous work [17] is ordinal regression which converts a regression task into a series of binary classifies with ordinal constraints. The ordinal regression has been proven to be useful for several tasks such as age estimation [17] and depth estimation [7] .To our best knowledge, we are the first re-formulate 3D hand pose estimation as a dense ordinal regression problem and propose a novel Dense Ordinal Regression 3D Pose Network (DOR3D-Net). Specifically, the design of DOR3D-Net includes: (1) The problem of hand joint regression in 3D is decomposed into sub-tasks of binary classifications. Each binary classifier is associated to a grid in 3D with different interval distributions in image and depth dimensions. Each binary classifier predicts probability of a binary spatial relationship between the position and a joint point. (2) The ordinal regression results at different local positions are aggregated to infer joint positions with weighted sum. This allows us using a joint position loss together with the ordinal regression loss to supervise our DOR3D-Net. (3) Three branches of networks are used for each of the three dimensions respectively.The experiments show that the binary classifiers in our design are easy to train and yield much lower level of noise. Fig. 1 visualizes the offset maps predicted by A2J [30] , JGR-P2O [6] and our probability map in image plane respectively. The first row shows the offset maps of A2J. Anchor offset values are wrong in several local areas (highlighted by red boxes). The boundary of zero offset value appears as a curve, which severely deviates from its ideal form as a straight line. For JGR-P2O, the learned offset maps also include apparent errors (highlighted by yellow boxes). In contrast, the probability maps generated by DOR3D-Net are much cleaner and well approximate those of their ideal forms. The main contributions are summarized as follows:• We are the first to re-formulate the 3D hand pose estima-tion as a dense ordinal regression problem and propose a novel Dense Ordinal Regression 3D Pose Network.• Specifically, we propose Ordinal Regression (OR) module to decompose offset regression into sub-tasks of binary classifications with less noises and outliers. Furthermore, both joint regression loss and ordinal regression loss are used to train DOR3D-Net in an end-to-end manner. • DOR3D-Net is remarkably superior to SOTAs on existing methods, revealing the effectiveness of our approach. DenseReg [29] decompose 3D hand pose as 3D heatmaps and 3D joint offsets and estimates these parameters by dense pixel-wise regression. Compared with heatmapbased method with relatively high computational burden, offset-based methods achieve a better trade-off between accuracy and efficiency and can be adapted in resourceconstrained platforms. A2J [30] predicts per-joint pixelwise offset through a dense set of anchor points on the input image. JGR-P2O [6] proposes a pixel-to-offset prediction network to address the trade-off between accuracy and efficiency for hand pose estimation. HandFoldingNet [3] inputs 3D hand point cloud and acquires the hand joint locations based on point-wise regression. SRN [20] regresses the joint position through multiple stacked network modules to capture spatial information. TriHorn-Net [23] computes two complementary attention maps of each joint and uses appearance-based data augmentation to improve the accuracy of hand pose estimation. 
ordinal regression: 
 The ordinal regression method maps direct regression into multiple binary classifiers and learns to predict ordinal labels. By preserving the natural order and supervising with multiple rank labels, the ordinal regression methods [7, 12, 13] have been proven to achieve much higher accuracy and faster convergence than the direct regression. DORN [7] converts the depth prediction into an ordinal regression problem, which discretizes depth value into several intervals and obtains ordinal labels to improve depth estimation accuracy. Furthermore, [18] also proposes the definitions of ordinal depth, which are based on comparing the relative depth between different joints. However, [18] is different from our DOR3D-Net. In this paper, we reformulate the pose estimation problem into an ordinal regression problem and compare binary spatial relationships between a sampling position with respect to a joint point. Our proposed dense ordinal supervision guarantees the probability maps are ordinal, which reduces the depth noise effect and improves 3D joint pose estimation accuracy. 
methods: 
 In this section, we first introduce the feature extractor module, which use a transformer-based feature extractor to learn dense local feature representations for capturing long-range relationships. Then, we elaborate on the details of dense ordinal regression module, which design to output pixel-wise probability maps and regress hand joints. Finally, we introduce the overall training procedure.In the transformer-based feature extractor module (Fig. 2 ), the input image plane is split into multiple 4 × 4 patches with the patch partition module. Each patch is treated as a 'token'. Four Swin Transformer stages are used to learn attention among patch tokens for capturing long-range contextual information. These stages consist of linear embedding layers, patch merging layers, and Swin Transformer blocks with their structure details specified in [15] . Since the Swin Transformer structure contains only relative positional embedding, we modify the input by adding U , V coordination maps (UVMap) and concat them together with the depth map to provide global absolute spatial information. U and V maps are generated by linear scaling in Eq. 1, which corresponds to the in-plain coordinate map of each pixel.EQUATION)Considering the in-plane xy regression and depth-plane z regression are quite different, following the design of A2J [30] , two feature maps F xy and F z are output from 'Stage-4' with the same dimensions H 32 × W 32 × 8C. Then, both decoded feature maps are regressed with the dense ordinal regression module for 3D joint prediction.Fig. 3 illustrates the pipeline for the dense ordinal regression module. With input as the learned feature maps, we utilize separate branches to estimate the three-dimensional coordinates of hand joints independently and output the predicted hand joint pose.Normal Discretization. Since the hand joints reside in three dimensional space, we decouple the 3D solution space and quantize it by representative discrete values along each of the three dimensions. In image plane x-axis and y-axis directions, the intervals are [0, W ) and [0, H) respectively. Uniform discretization (UD) is adopted to divide the image plane. Assuming that the intervals are discretized into K x , K y sub-intervals along x-axis and y-axis directions respectively, the UD can be formulated as:EQUATIONwhere x i , y j are sampling points and then form wellordered sets S x = x 0 , ..., x Kx-1 , S y = y 0 , ..., y Ky-1 .Here, we setK x = W/2, K y = H/2.In the z-axis direction, we analyze statistics of the joints z coordinate distribution and notice that it is close to normal distribution. Following this, the sampling interval of normal discretization (ND) becomes smaller as the sampling position becomes closer to the distribution center. In our specific -4- x-axis uses uniform discretization and z-axis applies normal discretization. For the x-probability map, each column represents the probability that the keypoint is larger than the corresponding discretization threshold. For the z-probability map, each map represents the probability that the keypoint is larger than the corresponding discretization threshold.Figure4 𝐻 32 × k x ⋯ ⋯ 𝑥 0 𝑥 1 𝑥 k x -1 𝑥 k x -2 (b) x-probability map 𝐻 32 × 𝑊 32 × k 𝑧 ⋯ ⋯ z 0 𝑧 1 𝑧 k 𝑧 -1 𝑧 𝑘 𝑧 -2implementation, we first divide the z-axis [0, D) into several sub-intervals evenly, and then increase the frequency of sampling points by the exponential power of 2 for the consecutive sub-intervals to the midpoint D/2. Sampling points for the other half of the z-interval can be obtained by symmetry. These sampling points form a well-ordered set S z = z 0 , ..., z Kz-1 . Here, we set K z = D/4, D refers to the depth range of the cropped image. Fig. 4 (a) and (c) visualize the distributions along x and z respectively.Ordinal Regression. After obtaining the discrete ordered classification sets S x , S y , S z , we cast the hand pose estimation problem into an ordinal regress problem to learn the network. These well-ordered sampling points along x-axis, y-axis, and z-axis directions construct multiple bi-nary classification sub-problems. For each predicted coordinate of joint a, these binary classifiers are used to predict whether the hand joint location is larger than these discretization thresholds respectively and then form probability map P rob x , P rob y , and P rob z . Here, coordination is represented as J a = (J a x , J a y , J a z ). P rob x (i, j, a) = P (J ax > x j ), ∀ i P rob y (i, j, a) = P (J a y > y i ), ∀ j P rob z (i, j, k, a) = P (J a z > z k ). ∀ i, j A is the number of joints. To guarantee the accuracy of classification, all these probability maps are densely supervised with the ground truth (GT) probability maps and introduced in section 3.3. Obviously, the ordinal regression solution only compares the binary spatial relationship between the keypoint and every discretization threshold. In comparison with dense offset regression methods, the solution space is reduced from a large interval to binary values which is easy to get the optimal solution and insensitive to the noise and outliers.After getting the reliable probability maps P rob x , we fuse the probability with its corresponding classification interval length and get the hand joint prediction vector in Eq. 4. The final predicted coordinate value x is the mean value of x (i). In the same way, we obtain the coordination ŷ and ẑ from ȳ (j), z (i, j) in Eq. 4, respectively.x (i, a) = j P rob x (i, j, a)EQUATIONHere, x j , y i and z k are sampling points. We concatenate the coordinations x, ŷ and ẑ as the final results.The network is jointly supervised by two loss: joint regression loss and dense ordinal regression loss. The binary probability maps along x-axis, y-axis and z-axis are supervised to guarantee that the learned features are robust to the low-quality depth image and have a reliable representation.Joint Regression Loss. The GT hand pose supervises the neural network to generate an accurate hand pose localization at the final stage. The loss function is formulated as below Eq. 3.3. The endpoint error with smooth L 1 is used to compute the joint regression loss.Here, L joint loss = a∈A L smooth (J a -J a * ) ,, where J a and J a * are the predicted coordinate and the GT of joint a, respectively.Dense Ordinal Regression (DOR) Loss. To supervise the binary classifiers, we generate the GT of middle results (GT binary probability maps P rob gt x , P rob gt y and P rob gt z ):EQUATIONwhere J a * x , J a * y and J a * z are the GT coordinates of joint a. Take the P rob x ∈ R H 32 ×W ×A for an example, the DOR loss is defined as the cross-entropy loss to densely supervise all binary classification probability maps.EQUATIONThe DOR losses in three dimensions are defined in Eq.7.L ord loss = L ord P rob x , P rob gt x + L ord P rob y , P rob gt y + L ord P rob z , P rob gt z .(7) Total Loss Function. The loss is defined as: L = λ 1 L joint loss + λ 2 L ord loss , where λ 1 and λ 2 are hyperparameters for balancing these terms. Here, λ 1 = 3 and λ 2 = 2. 
experimental setting: 
 Datasets. There are four widely adopted datasets for 3D hand pose estimation task, including HANDS2017 [31] , MSRA [26] , ICVL [27] , and NYU [28] . HANDS2017 dataset is composed of Big Hand 2.2M dataset [31] and the First-person Hand Action Dataset (FHAD) [9] . It contains 957K training and 295K testing depth images. MSRA dataset contains 9 subjects with each subject performs 17 hand gestures, and each hand gesture contains about 500 frames. ICVL dataset contains 22K training and 1.5k testing depth images with 3D annotations for 16 joints. The raw images with annotations are augmented to 330K samples by in-plane rotations. NYU dataset contains 72K training and 8.2K testing depth images labeled with 3D annotations for 36 joints. Following A2J [30] , we only use 14 of the 36 joints from frontal view for both training and testing. Since the number of raw images in HANDS2017 dataset is over ten times bigger than any of the rest three datasets, we conduct ablation study on the HANDS2017 dataset.Implementation Details. DOR3D-Net is trained with 2 NVIDIA V100 GPUs. We adopt the AdamW optimizer for all our experiments during training. In all experiments, the learning rate is 3.5 × 10 -4 with a weight decay of 10 -4 . The batch size for MSRA, ICVL and NYU datasets is 32, and the batch size for HANDS2017 is 64. For all datasets, the learning rate decays by 0.2 every 7 epoches. Similar to A2J [30] and V2V-PoseNet [16] , we use hand center point to crop the hand region from an depth image and resize the image to 224 × 224. For MSRA, ICVL and Hands2017, the hand center points are released by V2V-PoseNet [16] . For NYU, we follow DenseRecurrent [4] and use the average location of the ground truth joints. Evaluation Metric. We use two standard metrics to evaluate 3D hand pose estimation performance. The first one is the mean 3D Euclidean distance error (Mean Error) [30] . The second one is the percentage of success frames in which the worst joint 3D distance error is below a threshold [30] . Note that, the results of our method in this paper are all predicted by a single model.We compare our method with the state-of-the-art depth image-based hand pose estimation methods, i.e., Hand-PointNet [10] , DenseReg [29] , V2V-PoseNet [16] , Pointto-Point [11] , CrossInfoNet [5] , A2J [30] , JGR-P2O [6] , HandFoldingNet [3] , SRN [20] , DenseRecurrent [4] , TriHorn-net [23] , IPNet [22] , and HandR 2 N 2 [2] . Fig. 5 shows the result of 3D per-joint mean error and the percentages of success frames over different error thresholds. Meanwhile, Tab. 1 shows the overall performance of DOR3D-Net and all the methods on four datasets, respectively. It can be seen that our method outperforms all the other methods on the NYU and HANDS17 datasets. On the ICVL dataset, our method is ranked third, with a slightly lower accuracy than the TriHorn-net [23] and HandR 2 N 2 [2] . TriHorn-net [23] uses an innovative data agumentation approach and HandR 2 N 2 [2] uses five iterative corrections. As noted in DenseRecurrent [4] and IP-Net [22] , some of the 3D joint annotations in MSRA dataset contained significant errors. Therefore, the evaluation on the MSRA dataset may be less meaningful. Moreover, to further valid the effectiveness of the dense ordinal regression module, we compare it with the heatmapbased method [25] with the same backbone Resnet50 on NYU dataset [28] . Tab. 2 shows that our ordinal regression module surpasses the heatmap-based method. In a nutshell, DOR3D-Net shows significant superiority over other existing methods, indicating the benefit of the dense ordinal regression module.To demonstrate the effectiveness of each module in our method, we conduct extensive ablation studies on the HANDS2017 dataset. Tab. 3, Tab. 4, Tab. 5, Tab. 6 and Fig. 6 show the experimental results in detail.To verify the effectiveness of the dense ordinal regression module, we replace the regression module with an offset-based regression module and show the results in the Tab. DOR loss, the error increased by 0.32mm. This also validates that dense probability supervision plays an important role in our DOR3D-Net to learn representative features and improve hand pose accuracy. Qualitative Results. Fig. 7 visualizes the successful and failure cases of our full model performance on MSRA dataset [26] , NYU dataset [28] and ICVL dataset [27] . It can be seen that our method predicts well in most cases, while fails in cases of severe occlusion, large areas of missing pixels, and challenging viewpoint.Moreover, in order to intuitively verify our method, we make some changes to the full model: (1) Replace the transformer with Resnet50; (2) Replace the ordinal regression with offset-based regression. Fig. 8 visualizes and compares the hand pose estimation results. In each image, the color in yellow, red, green, blue, and pink represents five fingers respectively. The bright colors denote estimated joints and the dark are the ground truth. It can be seen that our full model has better performance on edge blur, background noise and occlusion cases, while the other two variants predict worse joints' position which may suffer from outliers and low-level feature representation.  
conclusion: 
 In this paper, we propose a DOR3D network that reformulates the 3D hand pose estimation as a dense ordinal regression problem. In comparison with offset-based regression methods, this formulation simplifies the solution space from a large interval to binary values which enables the network to learn easily and find out the optimal solution. Furthermore, a transformer-based feature extractor is utilized to enhance dense feature presentation and additional UV coordination maps are generated to provide absolute spatial information. Our DOR3D-net has achieved the SOTA performance on the HANDS2017, MSRA, NYU and ICVL datasets. This method provides robots and autonomous vehicles with the ability to accurately perceive human hand movements. Therefore, this technology is crucial for social-aware intelligent machines to function effectively in human-populated environments.In the future, we will improve the depth-based 3D hand pose estimation by improving the samples with the challenging scenes, such as extreme occlusions or very fast hand movements. Moreover, we will to improve the model efficiency for porting to robots or autonomous driving platforms for low-latency user interaction. 
 
 Appendix:  


Decision: This paper introduces the DOR3D-Net for 3D hand pose estimation using dense ordinal regression, simplifying the solution space and enhancing learning efficiency. While achieving state-of-the-art results, the paper faces criticism for minor improvements and relevance to the workshop theme. Despite its shortcomings, the solid technical framework and extensive experimental validation support its acceptance, with recommendations for addressing noted gaps in real-world application and novelty. However, the discussion of virtual humans in autonomous driving and robotics should be covered in the final version to match the workshop topic in the final version.

Reviews: review: title: title: Acceptable paper - suitability for POETS workshop uncertain
main_review: main_review: This paper addresses the task of hand pose estimation from depth imagery. The authors reformulate the 3D pose estimation problem as a dense ordinal regression problem and introduce Dense Ordinal Regression 3D Pose Network (DOR3D-Net) for 3D hand pose estimation. The DOR3D-Net simplifies the solution space to binary values, reducing noise and improving learning efficiency. It uses a transformer-based feature extractor and UV coordination maps for enhanced spatial information. The network has achieved state-of-the-art performance on the HANDS2017, MSRA, NYU, and ICVL datasets.

Pros:
- Improve upon previous state-of-the-art methods (Sec. 4.2)
- Thorough ablation (Sec 4.3), motivating the design choices
- Visualization of both successful and unsuccessful predictions

Cons: 
- The improvements are minor and could be considered within noise. Considering this, it would've been more appropriate to report an average of multiple seeded runs to make sure the improvements are not within error. This is exacerbated by the fact that the annotations of NYU/ICVL are noisy.
- The suitability of this paper in the POETS workshop is debatable. This paper address 3D hand pose estimation from depth imagery using standard benchmarking datasets. The POETS workshop webpage states that the workshop is about virtual humans for robotics and autonomous driving. None of these topics are addressed. Therefore I recommend resubmitting this paper to a pose workshop.
title: title: Review of <DOR3D-Net: Dense Ordinal Regression Network for 3D Hand Pose Estimation>
main_review: main_review: **Abstract and Introduction**

The paper introduces DOR3D-Net, a novel architecture focusing on 3D hand pose estimation using a combination of dense ordinal regression and joint regression loss. The premise is based on overcoming the high computational costs and inaccuracies associated with earlier dense regression methods by utilizing binary classifiers for depth estimation.

**Technical Soundness and Innovation**

The technical framework is solid, leveraging the strengths of transformers for feature extraction and introducing a novel DOR module for 3D localization. This dual approach is innovative in its handling of depth estimation, addressing the significant challenge of precision in 3D pose estimation tasks. However, as noted below, the idea of using ordinal regression is not entirely novel, echoing techniques used in full-body pose estimation tasks as indicated by references to similar works. 

**Critical Observations**

- Although the results are promising, the paper would benefit from a deeper exploration into the failures or limitations of the proposed method, especially under challenging conditions such as extreme occlusions or very rapid hand movements.
- The computational efficiency, although addressed, might still be a concern for real-time applications, as transformers are generally more computationally intensive. Exact comparisons in terms of inference time against other methods under similar hardware conditions would add more depth.

**Pros and Cons**
- **Pros:**
  - The innovative combination of joint and ordinal regression losses enhances model performance and robustness.
  - The method achieves state-of-the-art results, which are well-supported by extensive experiments and comparisons.
  
- **Cons:**
  - The novelty is somewhat diluted by the similarity in approach to existing methods in related fields, which could be a point of contention regarding the claimed contributions.
  - The paper does not address the limitations related to occlusions thoroughly, which are critical in real-world applications of hand pose estimation.

**Discussion and Questions**

The discussion around the pipeline's close resemblance to the original Swin Transformer and the modifications in stage-3 could have been more detailed, offering deeper insights into the choices that led to these design decisions. The absence of a follow-up on handling occlusions, as noted in the review, also leaves a gap in the discussion that could be crucial for practical applications.

It would be beneficial to test the model in real-world scenarios, possibly integrating it with real-time video input to evaluate performance dynamically. Expanding the discussion on the model's limitations by analyzing more thoroughly the conditions under which it fails could guide future improvements.

**Conclusion**

Overall, the paper presents a well-thought-out approach to tackling the problem of 3D hand pose estimation, introducing a robust method to handle the complexities associated with depth data. Despite the challenges in novelty and handling occlusions, the results are compelling, though some areas may require further exploration and validation in real-world conditions.
title: title: This paper proposed a novel dense ordinary regression network DOR3D-Net to solve the 3D hand pose estimation. Extensive experiment results are provided to demonstrate the effectiveness and superiority.
main_review: main_review: Pros:
1. The paper is well-written and easy to follow.

2. The paper proposes an approach that solves the hand posed estimation task by a novel dense ordinary regression. Moreover, the new discretization method and ordinary regression loss are introduced.

3. Extensive experiments were conducted and demonstrate improvements.

Cons:
1. Missing recent SOTA literature. Table 1 should include the following recent published results for comparison.

[R1] Cheng W, Ko J H. HandR2N2: Iterative 3D Hand Pose Estimation Using a Residual Recurrent Neural Network[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 20904-20913.

[R2] Ren P, Chen Y, Hao J, et al. Two heads are better than one: image-point cloud network for depth-based 3D hand pose estimation[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2023, 37(2): 2163-2171.

2. The best values in Table 5 should be highlighted as bold.

3. Line-322, how to obtain the "hand center point" for each dataset should be clarified. Calculating average location of GT joints? Or using the estimated center point provided by V2V [16]?

Minor editorial issue:

--Line-081, traing -> training