SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Datasets. A synthetic dataset ObMan [29] and two real-world datasets HO3D(v2) [28], MOW [6] are utilized to evaluate DDF-HO in various scenarios. ObMan consists of 2772 objects of 8 categories from ShapeNet [8], with 21K grasps generated by GraspIt [43]. The grasped objects are rendered over random backgrounds through Blender<sup>1</sup>. We follow [29, 67] to split the training and testing sets. HO3D(v2) [28] contains 77,558 images from 68 sequences with 10 different persons manipulating 10 different YCB objects [5]. The pose annotations are yielded by multi-camera optimization pipelines. We follow [28] to split training and testing sets. MOW [6] comprises a total of 442 images and 121 object templates, collected from in-the-wild hand-object interaction datasets [14, 56]. The approximated ground truths are generated via a single-frame optimization method [6]. The training and testing splits remain the same as the released code of [67].
<sup>1</sup>https://www.blender.org/
Figure 5: Reconstruction quality comparisons on Obman [29] by category. From 0 to 7, we demonstrate the results of IHOI [67] vs DDF-HO on the categories: Bottle, Bowl, Camera, Can, Cellphone, Jar, Knife, Remote Control sequentially. DDF-HO consistently outperforms IHOI.
Method	F-5 ↑	F-10 ↑	CD ↓	F-5 ↑	F-10 ↑	CD ↓
HO [29]	0.08	0.19	4.60	0.05	0.14	6.03
GF [36]	0.09	0.21	5.23	0.07	0.16	6.25
IHOI [67]	0.21	0.38	1.99	0.17	0.31	4.17
Ours	0.28	0.42	0.55	0.24	0.36	0.73
Table 1: Results on HO3D(v2) [28] dataset with finetuning (left) and zero-shot generalization from ObMan [29] dataset (right). Overall best results are in bold.
Evaluation Metrics. We first utilize [2] to convert the predicted DDF into point cloud representation and then compare against the sampled point cloud from the corresponding ground truth mesh. Following [67], we report Chamfer Distance (CD, mm), F-score at 5mm threshold (F-5) and 10mm threshold (F-10) of the converted object point cloud, which reflects the quality of hand-held object reconstruction.
Baselines. Three baseline methods are chosen to ensure fair comparisons with our DDF-HO method, taking into account that they aim to predict the shape of hand-held objects directly from a single RGB input and do not rely on known object templates, category information, or depth priors. The baselines include Atlas-Net [27] based HO [29], implicit field based GF [36] and SDF based IHOI [67].
Implementation Details. We conduct the training, evaluation and visualization of DDF-HO on a single A100 40GB GPU. We use the same off-line systems [29, 55] as [67] to estimate the hand and camera poses. We also adopt a ResNet34 [30] image encoder to extract a 5-resolution visual feature pyramid, the same with [67]. In all the three datasets, we sample 20K 3D rays for each object and generate the ground truth using the ray marching algorithm in Trimesh  $^{2}$ . The number of sampled points  $K_{l}$  along the projected 2D ray is set to 8 and number of multi-head attention is 2 for 2D Ray-Based Feature Aggregation technique.  $K_{3D}$  for  $\mathcal{F}_{3D}^{\mathcal{L}}$  introduced in Sec. 3.4 is set as 8. DDF-HO is trained end-to-end using Adam with a learning rate of 1e-4 on ObMan for 100 epochs. Following [67], we use the network weights learned on synthetic ObMan to initialize the training on HO3D(v2) and MOW. Training on HO3D(v2) and MOW also use Adam optimizer with a learning rate 1e-5 for another 100 and 10 epochs, respectively following [67]. The weighting factors of the loss for DDF-HO  $\lambda_{1},\lambda_{2}$  are set to 5.0 and 0.5, respectively. Note that we do not include the symmetry loss term  $\mathcal{L}_s$  in the training except in the Ablation Study Tab. 4 for a fair comparison, since other baselines do not leverage this additional information. Note that during evaluation, we convert the DDF representation to point cloud [2], while during visualization, we convert DDF to mesh [2, 31]. Details of ground truth generation and network architecture are provided in the Supplementary Material.
4.2 Evaluation on Synthetic Scenarios
As shown in Tab. 3, the evaluation on the synthetic large-scale dataset ObMan demonstrates that DDF-HO achieves high-quality hand-held object reconstruction with the assistance of suitable DDF representation. Our proposed method exhibits state-of-the-art performance under all three evaluation metrics of F-5, F-10, and CD. Specifically, we achieve a significant improvement over the current state-of-the-art SDF-based IHOI, with a gain of  $13\%$  and  $4\%$  on F-5 and F-10, respectively. This improvement can be attributed to DDF-HO's more suitable ray-based feature aggregation technique,
$^{2}$ https://trimsh.org/
Method	F-5 ↑	F-10 ↑	CD ↓	F-5 ↑	F-10 ↑	CD ↓
IHOI [67]	0.10	0.19	7.83	0.09	0.17	8.43
Ours	0.16	0.22	1.59	0.14	0.19	1.89
Figure 6: Visualization results on ObMan [29]. Our method consistently outperforms IHOI [67].
which allows for more exquisite hand-held object reconstruction. Notably, DDF-HO's CD metric is reduced by  $85\%$  compared to IHOI and  $77\%$  compared to HO, indicating that our predicted object surface contains much fewer outliers.
Fig. 6 presents improved visualizations of DDF-HO, showcasing enhanced and more accurate surface reconstruction of hand-held objects. While IHOI can achieve decent object surface recovery within the camera view, the reconstructed surface appears rough with numerous outliers when observed from novel angles. This suggests that IHOI lacks the ability to perceive 3D hand-held objects due to its limited modeling of hand-object interaction in 3D space. In contrast, DDF-HO utilizes a more suitable DDF representation, resulting in smooth and precise reconstructions from any viewpoint of the object.
4.3 Evaluation on Real-world Scenarios
In addition to synthetic scenarios, we conduct experiments on two real-world datasets, HO3D(v2) and MOW, to evaluate DDF-HO's performance in handling real-world human-object interactions. Table 1 constitutes the evaluation results on HO3D(v2) after finetuning (with related settings described in Section 4.1), as well as the zero-shot generalization results, where we directly conducted inference on HO3D(v2) using the weights from training on ObMan. Furthermore, Table 2 presents results on MOW under the same setting.
Table 2: Results on MOW [6] dataset, with the setting of finetuning (left) and zero-shot generalization from ObMan [29] dataset (right). Overall best results are in bold.
Method	F-5 ↑	F-10 ↑	CD ↓
HO [29]	0.23	0.56	0.64
GF [36]	0.30	0.51	1.39
IHOI [67]	0.42	0.63	1.02
Ours	0.55	0.67	0.14
Table 3: Results on ObMan [29] dataset. Overall best results are in bold.
Our method, after finetuning on the real-world data,
achieves state-of-the-art performance on both datasets. Specifically, on HO3D(v2), we observe a considerable improvement compared to IHOI and other methods, with an increase in F-5 by  $7\%$ , F-10 by  $4\%$ , and a significant decrease in CD by  $73\%$ . On MOW, our approach also outperforms the previous state-of-the-art methods, achieving a remarkable performance gain in terms of increased F-5  $(6\%)$ , F-10  $(3\%)$ , and reduced CD  $(80\%)$  compared with IHOI [67]. Fig. 7 shows the visualization comparison results on MOW. DDF-HO performs well under the real-world scenario, yielding more accurate reconstruction results.
Furthermore, the zero-shot experiments demonstrate that DDF-HO has a stronger ability for synthetic-to-real generalization. Specifically, on HO3D(v2), DDF-HO yields superior performance in terms of F-5 by  $7\%$ , F-10 by  $5\%$ , and a decreased CD by  $82\%$ . Moreover, the results on MOW also indicate
Figure 7: Visualization results on MOW [6]. Our method consistently surpasses IHOI [67].
that our method, trained only on synthetic datasets, can still achieve decent performance in real-world scenarios, thanks to the generic representation ability of DDF for hand-object interaction modeling.
4.4 Efficiency
To demonstrate the efficiency of DDF-HO, we compare the network size and the running speed with IHOI, which is a typical SDF-based hand-held object reconstruction method. All experiments are conducted on a single NVIDIA A100 GPU.
DDF-HO runs at 44FPS, which is slower than IHOI with 172 FPS, but still achieves real-time performance. The slower inference comes from the attention calculation for 2D ray-based feature aggregation and the more elaborated 3D feature generation. For model size, the two methods share a similar scale (24.5M of DDF-HO and 24.0M of IHOI). Generally, the increased parameters mainly come from the cross attention mechanism in 2D ray-based feature aggregation. Other modules are only adopted to collect features to model hand-object interactions and do not significantly increase the network size.
We conduct ablation studies on the ObMan and HO3D(v2) datasets to evaluate the impact of three key assets of DDF representation: Ray-Based Feature Aggregation (RFA), Intersection-aware Hand Feature (IHF), and Symmetry Loss (SYM). The results of the ablation studies are presented in Tab. 4.
On ObMan, we first replace the SDF representation with DDF without any modifications to feature aggregation, resulting in a slight improvement over the SDF-based IHOI with a  $3\%$  increase in F-5 metric. This indicates that although DDF is more suitable for representing hand-object interaction (with almost an  $80\%$  decrease in CD metric), more sophisticated feature aggregation designs are required. Next, we add RFA considering the characteristics of sampled rays, leading to a  $5\%$  increase in F-5 and a  $4\%$  increase in F-10. Subsequently, adding IHF, which models hand-object interaction locally by considering the intersection information of the hand, resulted in a  $5\%$  increase in F-5 and a  $3\%$  increase in F-10. This indicates that considering the intersection information of the hand can improve the accuracy of hand-held object reconstruction. Finally, adding the SYM loss, which captures the symmetry nature of everyday objects and handles self-occluded scenarios caused by hands, results in another  $2\%$  increase in F-5 and a  $1\%$  increase in F-10. On the HO3D(v2) dataset, RFA, IHF, and SYM modules play similar roles as on ObMan.
Additionally, we evaluate the influence of input hand pose on DDF-HO by adding Gaussian noise to the estimated hand poses (Pred) or ground truth hand poses (GT) in the input. The results of the ablation studies (Tab. 5) demonstrate the robustness of DDF-HO in handling noisy input hand poses.
In this paper, we present DDF-HO, a novel pipeline that utilize DDF as the shape representation to reconstruct hand-held objects, and demonstrate its superiority in modeling hand-object interactions
RFA	IHF	SYM	F-5 ↑	F-10 ↑	CD ↓	F-5 ↑	F-10 ↑	CD ↓
SDF baseline [67]	0.42	0.63	1.02	0.21	0.38	1.99
×	×	×	0.45	0.60	0.22	0.24	0.38	0.62
✓	×	×	0.50	0.64	0.17	0.26	0.39	0.58
✓	✓	×	0.55	0.67	0.14	0.28	0.42	0.55
✓	✓	✓	0.57	0.68	0.13	0.29	0.43	0.52
Table 4: Ablation studies for key assets of DDF-HO on ObMan [29] (left) and HO3D(v2) [28] datasets (right).
Noise	F-5 ↑	F-10 ↑	CD ↓	F-5 ↑	F-10 ↑	CD ↓
Pred	0.55	0.67	0.14	0.28	0.42	0.55
Pred+σ = 0.1	0.54	0.66	0.15	0.24	0.35	0.73
Pred+σ = 0.5	0.47	0.60	0.18	0.20	0.30	0.83
Pred+σ = 1.0	0.42	0.55	0.25	0.17	0.27	0.98
Pred+σ = 1.5	0.38	0.52	0.31	0.14	0.25	1.24
GT	0.59	0.70	0.10	0.30	0.45	0.50
GT+σ = 0.1	0.58	0.69	0.11	0.27	0.43	0.58
GT+σ = 0.5	0.51	0.63	0.16	0.23	0.34	0.76
Table 5: Ablation studies for input hand pose on ObMan [29] (left) and HO3D(v2) [28] datasets (right). The Pred row remains the same setting with the Tab. 1 and 3.
over competitors. Specifically, for each sampled ray in 3D space, we collect its features capturing local-to-global hand-object relationships by introducing a novel 2D ray-based feature aggregation and 3D intersection-aware hand pose embedding. Extensive experiments on synthetic dataset Obman and real-world datasets HO3D(v2) and MOW verify the effectiveness of DDF-HO on reconstructing high-quality hand-held objects.
Limitations. DDF-HO naturally inherits the shortcomings of DDF. First, the higher dimensional input of DDF makes it harder to train than SDF, resulting in more complex data, algorithm and network structure requirements. This may hinder the performance of DDF-based methods when scaling up to large-scale scenes, like traffic scenes. Second, to enable more photorealistic object reconstruction, there are other characteristics like translucency, material and appearance need to be properly represented. This requires further research to fill the gap.
Acknowledgements. This work was supported by the National Key R&D Program of China under Grant 2018AAA0102801, National Natural Science Foundation of China under Grant 61827804. We also appreciate Yufei Ye, Tristan Aumentado-Armstrong, Zerui Chen and Haowen Sun for their insightful discussions.
[1] Titas Anciukevicius, Zexiang Xu, Matthew Fisher, Paul Henderson, Hakan Bilen, Niloy J Mitra, and Paul Guerrero. Renderdiffusion: Image diffusion for 3d reconstruction, inpainting and generation. arXiv preprint arXiv:2211.09869, 2022.
[2] Tristan Aumentado-Armstrong, Stavros Tsogkas, Sven Dickinson, and Allan D Jepson. Representing 3d shapes with probabilistic directed distance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19343-19354, 2022.
[3] Adnane Boukhayma, Rodrigo de Bem, and Philip HS Torr. 3d hand shape and pose from images in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10843-10852, 2019.
[4] Samarth Brahmbhatt, Chengcheng Tang, Christopher D Twigg, Charles C Kemp, and James Hays. Contactpose: A dataset of grasps with object contact and hand pose. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIII 16, pages 361-378. Springer, 2020.
[5] Berk Calli, Arjun Singh, Aaron Walsman, Siddhartha Srinivasa, Pieter Abbeel, and Aaron M Dollar. The ycb object and model set: Towards common benchmarks for manipulation research. In 2015 international conference on advanced robotics (ICAR), pages 510-517. IEEE, 2015.
[6] Zhe Cao, Ilija Radosavovic, Angjoo Kanazawa, and Jitendra Malik. Reconstructing hand-object interactions in the wild. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12417-12426, 2021.
[76] Ruida Zhang, Yan Di, Fabian Manhardt, Federico Tombari, and Xiangyang Ji. Ssp-posed: Symmetry-aware shape prior deformation for direct category-level object pose estimation. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7452-7459. IEEE, 2022.
[77] Xiong Zhang, Qiang Li, Hong Mo, Wenbo Zhang, and Wen Zheng. End-to-end hand mesh recovery from a monocular rgb image. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2354-2364, 2019.
[78] Juntian Zheng, Qingyuan Zheng, Lixing Fang, Yun Liu, and Li Yi. Cams: Canonicalized manipulation spaces for category-level functional hand-object manipulation synthesis. arXiv preprint arXiv:2303.15469, 2023.
[79] Yuxiao Zhou, Marc Habermann, Weipeng Xu, Ikhsanul Habibie, Christian Theobalt, and Feng Xu. Monocular real-time hand shape and motion capture using multi-modal data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5346-5355, 2020.
[80] Christian Zimmermann and Thomas Brox. Learning to estimate 3d hand pose from single rgb images. In Proceedings of the IEEE international conference on computer vision, pages 4903-4911, 2017.