SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathcal {L} = \lambda_ {h} \mathcal {L} _ {h} + \lambda_ {o _ {1}} \mathcal {L} _ {o _ {1}} + \lambda_ {o _ {2}} \mathcal {L} _ {o _ {2}} + \lambda_ {d} \mathcal {L} _ {d} + \lambda_ {s} \hat {\mathcal {L}} _ {s}, \tag {9}
$$
where  $\lambda_h, \lambda_{o_1}, \lambda_{o_2}, \lambda_d, \lambda_s$  are hyper-parameters.
We evaluate our method and state-of-the-art methods on two datasets: DexYCB [6] and HO3D [18].
DexYCB. The DexYCB dataset [6] consists of 582K frames involving 20 different YCB objects [4]. We use the official "S0" train-test split for the training and evaluation. Following [52], we evaluate on right hand poses and filter out samples in which the hand or object is not within the field of view of the camera. To ensure consistent comparison in physics metrics, we remove test samples where the hand does not interact with the object and only select those that remain stable after simulation (see the GT results in Table 1), resulting in a total of 6348 samples. For training, we do not perform this selection of stability in order to include more data. However, we mask out the replicated stability loss on unstable training samples, e.g. no hand-object interaction, to avoid misleading supervision.
HO3D. The HO3D dataset [18] consists of 66K frames featuring 10 different objects. We select the "v2" version that is mostly evaluated by previous works [56, 20, 52, 34]. Since its ground truth hand poses for the test set are not released, we follow [56] to evaluate on a subset named "v2-", whose physics plausibility is manually verified by [56]. For training data, we use the official HO3D v2 training split and follow the same practice as the DexYCB dataset to perform sample selection and loss masking. The total HO3Dv2- test set consists of 6076 samples.
4.2 Metrics
Hand & Object Metrics. We evaluate the accuracy of pose estimation by comparing the hand and object results with the ground truth labels. For the hand metric, we calculate the mean joint error
(MJE) [60], which measures the Euclidean distance between the predicted and ground truth hand joint positions after root joint alignment. For the object metric, we compute the mean corner error (MCE) on the HO3D test set following [34], which calculates the average distance between the predicted and ground truth positions of the 8 tightest object corners. Since on the DexYCB dataset, many objects are rotationally symmetric, we thus borrow from [19] to compute the symmetry-aware mean corner error (SMCE) as defined in Eq.(8), for a fair comparison with [34].
Table 1: Quantitative comparison on the DexYCB test set. Best results are highlighted in bold and and inapplicable results are marked with "-". Our method achieves superior physics plausibility and stability with comparable accuracy compared to state-of-the-art methods [20, 52, 34].
Methods	Hand MJE (cm)↓	Object SMCE (cm)↓	Physics
CP (%)↑	PD (cm)↓	SD (cm)↓	SR (%)↑
GT	-	-	100	0.91	0.64	100
Hasson et al. [20]	1.25	-	84.35	1.80	4.83	16.80
ArtiBoost [34]	1.07	1.60	94.23	1.50	2.78	30.07
Wang et al. [52]	1.15	-	89.16	1.57	3.53	19.54
Ours	1.12	1.73	95.90	1.48	2.42	32.89
Table 2: Quantitative comparison on the HO3Dv2 test set. Best results are highlighted in bold and and inapplicable results are marked with "-". Our method consistently outperforms baseline methods [20, 52, 34, 56] in all physics metrics while maintaining comparable accuracy.
Methods	Hand MJE (cm) ↓	Object MCE (cm)↓	Physics
CP (%)↑	PD (cm)↓	SD (cm)↓	SR (%)↑
Hasson et al. [20]	-	5.35	78.52	2.02	6.40	10.37
Yang et al. [56]	-	5.74	96.47	1.65	3.16	17.00
ArtiBoost [34]	-	4.86	94.47	1.27	2.83	18.02
Wang et al. [52]	-	4.79	93.07	1.88	3.47	11.42
Ours	-	5.28	96.64	1.17	2.42	19.24
Physics Metrics. We evaluate the physical plausibility of the estimation following the criterions in [7, 56, 20]. Specifically, we evaluate the simulation displacement (SD) [56] to compute the average object center displacement after 200ms simulation time. We then evaluate the success rate (SR) [7] to classify the prediction whose SD is below a threshold of  $1\mathrm{cm}$  as success. These two metrics are used to evaluate the pose stability. In addition, we evaluate the contact percentage (CP) [28] as the ratio of predictions with hand-object contact and the penetration depth (PD) [3] as the maximum penetration distance of contacting hand-object predictions. It is worth noting that a stable estimation, i.e. with a lower SD, generally indicates a higher CP and a lower PD, but the converse is not necessarily true.
4.3 Implementation Details
We implement the model in PyTorch [42] and train it using the Adam [29] optimizer on a single NVIDIA RTX 3090 GPU. We adopt the base network pretrained in [34], and independently train the DeepSim for 40 epochs as a warm-up. We then jointly train both networks enabling the stability loss for 25 epochs. We set the learning rate to  $5 \times 10^{-5}$ . When training on both datasets, we follow [34, 52] to crop input images into  $224 \times 224$  pixels using the provided hand-object bounding boxes. In addition, we follow [34] and perform data augmentation with random translation and rescaling by a factor of 0.1. However, we exclude rotation augmentation as it can affect the ground truth stability. Finally, we set  $\lambda_h = 0.5$ ,  $\lambda_d = 0.1$ ,  $\lambda_s = 0.1$ , and follow [34] to set  $\lambda_{o_1} = 0$ ,  $\lambda_{o_2} = 0.2$  on the DexYCB dataset and  $\lambda_{o_1} = 0.2$ ,  $\lambda_{o_2} = 0$  on the HO3D dataset for a fair comparison.
For physics simulation, we use the MuJoCo [48] simulator. We initialize the hand and object inertia matrix  $\mathbf{M}$  using the Composite Rigid Body algorithm [11]. We further convexly decompose hand meshes by the MANO blending weights [44] and object meshes by the CoACD [53] algorithm to facilitate continuous collision detection [15] in MuJoCo. We set the gravity acceleration as  $9.8\mathrm{m / s}^2$  in the  $y$  direction of the camera frame. For the adhesion force, we empirically set the gain as 100 and the maximum control range as 10, so that a simple touch is not sufficient to stably grasp the object. Finally, we set the simulation step to be  $T = 100$  and time duration in each step as  $\Delta t = 0.02$ . More implementation and training details can be found in the supplementary materials.
Quantitative results. We compare our method with three categories of state-of-the-art methods: data-driven [34, 52], distance-based [20] and physics-based [56] approaches. In Table 1, we report the comparison on the DexYCB test set and re-evaluate all baseline methods using the officially provided weights [34, 52] or optimization scripts [20, 56]. To avoid unfair comparison, we exclude the SMCE result for [52, 20] as they do not consider object symmetry in implementation. From the table, we observe that our method achieves superior performance on all physics metrics, while maintaining comparable accuracy using significantly less data than [34, 52] when trained with the stability loss. Our method also noticeably improves the stability compared to distance-based approach [20].
Figure 3: Qualitative results on the HO3D and DexYCB test sets. State-of-the-art methods [34, 56] produce results that are either unstable, or stable but do not align with the input image, e.g. last row in the HO3D. In contrast, our method produces poses that are both stable and comparably accurate.
In Table 2, we compare all methods on the HO3Dv2 $^{-}$  test set. Since the selected split is not suitable for submitting to the official HO3D online evaluation server, we follow the same evaluation protocol as [56] to not evaluate for the hand metric. In addition, although [56] also report the SD in their paper, they use a different simulator [8] and the code for this part is not publicly available. Hence we re-evaluate the same metric using our physics modeling for a fair comparison. Consistently, we observe our method noticeably improves physics plausibility and stability with comparable accuracy. Finally, for the testing, [56] and [20] requires 12 and 45 seconds to optimize for one input image on a single NVIDIA RTX 3090 GPU, in comparison, our method only requires 0.02 seconds on the same device for a forward call of the refined base network, which is significantly more efficient.
Qualitative results. We present qualitative results on the DexYCB and HO3D test sets in Figure 3. We render the refined hand and object meshes in the camera view, illustrating that the estimated poses align well with the input image. More importantly, we compare in rotated views with [34, 56] to show that while [34] produce visually close results, the established contact is actually physically unstable, with some fingers floating in the air or penetrating the object. In addition, [56] do not consider pose accuracy when optimize for physics plausibility, therefore can generate results that are stable but do not align with the input image. In comparison, our results are both accurate and physically stable.
Effects of the DeepSim Network. To verify the effects of the DeepSim for back-propagation, we compare with two other approaches for calculating the state gradient: (i) numerical gradient using
the finite difference in the official MuJoCo implementation, and  $(ii)$  analytical gradient through the differentiable simulator NimblePhysics [54]. We use a similar physics model in the NimblePhysics based on its available functions (included in the supplementary materials). To avoid training failure, we include a gradient clip of 1.0 for all methods. In Figure 4 (b), we observe  $(i)$  produces incorrectly large gradient due to the sudden velocity change during penetration resolving. For  $(ii)$ , we observe it
Table 3: Comparison on design choices of DeepSim and stability loss. Best performing method (MLP+S) directly regresses a final scalar stability loss, which tackles a easier task and therefore outperforms variants that regress more state components (T/RT) or more simulation steps (LSTM).
Methods	DeepSim AE (mm)↓	Physics
CP (%)↑	PD (cm)↓	SD (cm)↓	SR (%)↑
w/o DeepSim	-	94.19	1.51	2.77	28.55
MLP + T	13.80	94.16	1.53	2.76	28.51
MLP + RT	17.22	94.57	1.52	2.81	26.60
LSTM + T	10.79	94.73	1.50	2.60	30.66
MLP + S	2.70	95.90	1.48	2.42	32.89
(a) Learning Curve by Gradient Methods
(b) Calculated Gradient Norm
Figure 4: Effects of the DeepSim for back-propagation. Figure (a) shows the trend of the stability loss (evaluated from the MuJoCo simulator) as the training progresses. Figure (b) shows the log of the raw gradient norm at each iteration  $(||\partial \mathcal{L}_s / \partial \mathbf{q_0}||$  for analytical gradient (NimblePhysics) and numerical gradient (Finite Difference), and  $||\partial \hat{\mathcal{L}}_s / \partial \mathbf{q}_0||$  for the DeepSim). Gradients directly obtained from the simulators are not suitable for training. In contrast, DeepSim provides numerically stable gradient and benefits the refinement over Base method trained with accuracy losses only. generates numerically unstable gradient, and even causes numerical overflow at singular points due to the discontinuity in contact geometry. The noisy gradient prevents the base network from being effectively refined as shown in Figure 4 (a). In contrast, the gradient through DeepSim is numerically stable and can better benefit the refinement over training with accuracy losses only.
Design Choices. In Table 3, we compare different variants of design in DeepSim and stability loss. We observe that regressing more state components, e.g. the 3 DoF object center  $(\mathrm{MLP} + \mathrm{T})$  or 6 DoF object pose  $(\mathrm{MLP} + \mathrm{RT})$  in the final simulation step, can be more challenging compared to regressing a scalar stability loss  $(\mathrm{MLP} + \mathrm{S})$ . As a result the DeepSim performs worse with a higher approximation error (AE), which measures the discrepancy between  $\hat{\mathcal{L}}_s$  and  $\mathcal{L}_s$ . Underfitting in DeepSim can cause severe misleading, thus can not help to improve the stability compared to the baseline method that only trains with accuracy losses (w/o DeepSim). In addition, we replace the MLP with a long short-term memory network (LSTM) [24] (implementation details in the supplementary materials), which sequentially estimates residual center displacements in each step  $(\mathrm{LSTM} + \mathrm{T})$ , so that the overall stability loss is the norm of the total displacement. However, we observe that dividing the total displacement into multiple partitions can cause the DeepSim to overfit by predicting only small residuals, therefore does not outperform against directly regressing for the final simulation step.
Limitations & Societal Impacts. Although DeepSim addresses the issues of simulator gradient in the back-propagation, our physics modeling and simulation in the forward step are still dependent on the simulator implementation, which can be imperfect when compared to real-world physics. In addition, we made several restrained assumptions about the physics properties and contact dynamics,
for instance we assumed rigid objects with known gravity direction, as well as simplified modeling of joint torques via the adhesion force. As a result, the potential sim-to-real gap may lead to unsafety when interacting with objects that have complex and unknown physics properties.
Future Works. The proposed pipeline can be benefited with improved designs of physics metrics, simulator and the DeepSim network. Future works are thus encouraged to explore on  $(i)$  stability losses with a more comprehensive consideration of the object state, e.g. rotation, velocity,  $(ii)$  more realistic simulation, especially for static hand,  $(iii)$  variant DeepSim networks that explicitly utilize physics priors for the regression, and  $(iv)$  extension to video inputs with known object initial states.
Conclusion. In this paper, we propose a novel pipeline DeepSimHO for estimating accurate and stable hand-object pose. We explicitly model the dynamical nature in hand-object interaction via forward physics simulation and stability evaluation. In the backward step, we introduce DeepSim, a neural network that learns the stability evaluation from the simulation and performs smooth gradient approximation to facilitate refinement of the base network. Thanks to them, we our method effectively and efficiently produces physically stable hand-object pose that conforms better to reality.
The research is funded in part by an ARC Discovery Grant (grant ID: DP220100800) to HL.
[1] Sheldon Andrews, Kenny Erleben, and Zachary Ferguson. Contact and friction simulation for computer graphics. In ACM SIGGRAPH 2022 Courses, pages 1-172. 2022. 4
[2] Mark Billinghurst, Hirokazu Kato, and Ivan Poupyrev. The magicbook: a transitional ar interface. Computers & Graphics, 25(5):745-753, 2001. 1
[3] Samarth Brahmbhatt, Chengcheng Tang, Christopher D Twigg, Charles C Kemp, and James Hays. Contact pose: A dataset of grasps with object contact and hand pose. In European Conference on Computer Vision, pages 361-378. Springer, 2020. 3, 7
[4] Berk Calli, Arjun Singh, Aaron Walsman, Siddhartha Srinivasa, Pieter Abbeel, and Aaron M Dollar. The ycb object and model set: Towards common benchmarks for manipulation research. In 2015 international conference on advanced robotics (ICAR), pages 510-517. IEEE, 2015. 6
[5] Zhe Cao, Ilija Radosavovic, Angjoo Kanazawa, and Jitendra Malik. Reconstructing hand-object interactions in the wild. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12417-12426, 2021. 1, 2, 3
[6] Yu-Wei Chao, Wei Yang, Yu Xiang, Pavlo Molchanov, Ankur Handa, Jonathan Tremblay, Yashraj S Narang, Karl Van Wyk, Umar Iqbal, Stan Birchfield, et al. Dexycb: A benchmark for capturing hand grasping of objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9044–9053, 2021. 2, 6
[7] Sammy Christen, Muhammed Kocabas, Emre Aksan, Jemin Hwangbo, Jie Song, and Otmar Hilliges. D-grasp: Physically plausible dynamic grasp synthesis for hand-object interactions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 1, 3, 5, 7
[8] Erwin Coumans and Yunfei Bai. Pybullet, a python module for physics simulation for games, robotics and machine learning. http://pybullet.org, 2016-2021. 1, 8
[9] Jonas Degrave, Michiel Hermans, Joni Dambre, et al. A differentiable physics engine for deep learning in robotics. Frontiers in neurorobotics, page 6, 2019. 3
[10] Bardia Doosti, Shujon Naha, Majid Mirbagheri, and David J Crandall. Hope-net: A graph-based model for hand-object pose estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6608-6617, 2020. 2, 3
[11] Roy Featherstone. Robot dynamics algorithms. Annexe Thesis Digitisation Project 2016 Block 5, 1984. 7
[12] Guillermo Garcia-Hernando, Edward Johns, and Tae-Kyun Kim. Physics-based dexterous manipulations with estimated hand poses and residual reinforcement learning. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 9561-9568. IEEE, 2020. 1, 3
[13] Erik Gartner, Mykhaylo Andriluka, Erwin Coumans, and Cristian Sminchisescu. Differentiable dynamics for articulated 3d human motion reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13190-13200, 2022. 3
[14] Moritz Geilinger, David Hahn, Jonas Zehnder, Moritz Bächer, Bernhard Thomaszewski, and Stelian Coros. Add: Analytically differentiable dynamics for multi-body systems with frictional contact. ACM Transactions on Graphics (TOG), 39(6):1-15, 2020. 3
[59] Yuxiao Zhou, Marc Habermann, Weipeng Xu, Ikhsanul Habibie, Christian Theobalt, and Feng Xu. Monocular real-time hand shape and motion capture using multi-modal data. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020. 3
[60] Christian Zimmermann and Thomas Brox. Learning to estimate 3d hand pose from single rgb images. In Proceedings of the IEEE international conference on computer vision, pages 4903-4911, 2017. 7