DDF-HO: Hand-Held Object Reconstruction via Conditional Directed Distance Field
Authors: Chenyangguang Zhang
Yan Di
Ruida Zhang
Guangyao Zhai
Fabian Manhardt
Federico Tombari
Xiangyang Ji
Venue: NeurIPS 2023 poster (2023)
Keywords: hand-held object reconstruction
directed distance field
human-object interaction

Abstract: Reconstructing hand-held objects from a single RGB image is an important and challenging problem. Existing works utilizing Signed Distance Fields (SDF) reveal limitations in comprehensively capturing the complex hand-object interactions, since SDF is  only reliable within the proximity of the target, and hence, infeasible to simultaneously encode local hand and object cues. To address this issue, we propose DDF-HO, a novel approach leveraging Directed Distance Field (DDF) as the shape representation. Unlike SDF, DDF maps a ray in 3D space, consisting of an origin and a direction, to corresponding DDF values, including a binary visibility signal determining whether the ray intersects the objects and a distance value measuring the distance from origin to target in the given direction. We randomly sample multiple rays and collect local to global geometric features for them by introducing a novel 2D ray-based feature aggregation scheme and a 3D intersection-aware hand pose embedding, combining 2D-3D features to model hand-object interactions. Extensive experiments on synthetic and real-world datasets demonstrate that DDF-HO consistently outperforms all baseline methods by a large margin, especially under Chamfer Distance, with about 80% leap forward. Codes are available at https://github.com/ZhangCYG/DDFHO.

DDF-HO: Hand-Held Object Reconstruction via Conditional Directed Distance Field
Chenyangguang Zhang' Yan Di Ruida Zhang" Guangyao Zhai', Fabian Manhardt Federico Tombari2 Xiangyang Ji 'Tsinghua University, TTechnical University of Munich, Google, (zcyg22emails. 2hangrd21@mails. tyiighss cn, (yan kyan.di@)tum..u
Abstract
Reconstructing hand-held objects from a single RGB image an important and challenging problem. Existing works utilizing Signed Distance Fields (SDF) reveal limitations in comprehensively capturing the complex hand-object interactions, since SDF is only reliable within the proximity of the target, and hence, infeasible to simultaneously encode local hand and object cues. To address this issue, we propose DDF-HÓ, novel approach leveraging Directed Distance Field (DDF) the shape representation. Unlike SDF, DDF maps a ray in 3D space, consisting of origin and direction, corresponding DDF values, including binary visibility signal determining whether the ray intersects the objects and a distance value measuring the distance from origin to target in the given direction. We randomly sample multiple rays and collect local to global geometric features for them by introducing novel 2D ray-based feature aggregation scheme and 3D intersection-avee -waree hand pose embedding, combining 2D)-3 features to model hand- object interactions. Extensive experiments on synthetic and real-world datasets demonstrate that DDF-HO consistently outperforms all baseline methods by a large margin, especially under Chamfer Distance, with about 80% leap forward. Codes are available at htttt:://// bestps://githng tht..ttttt...... com/ wtt.loco//mmhiioe hha//cC/nnCCC/DFO
Introduction
Hand-held object reconstruction refers to creating 3D model for the object grasped by the hand. It is
an essential and versatile technique with many practical applications, e.g. robotics [78 (66 39 73 72), augmented and virtual reality (37), medical imaging (50). Hence, in recent years, significant research efforts have been directed towards the domain of reconstructing high-quality shapes of hand-held objects, without relying object templates depth information. Despite the progress made. most existing methods rely on the use of Signed Distance Fields (SDF) as the primary shape representation, which brings about two core challenges in hand-held object reconstruction due to the inherent characteristics of SDF.
First, SDF is an undirected function in 3D space. Consequently, roughly determining the nearest point on the target object to sampled point in the absence of object shape knowledge infeasible. This limitation poses significant challenge for single image hand-held object reconstruction it is difficult to extract the necessary features to represent both the sampled point and its nearest neighbor the object surface. Previous methods (67] have attempted to address this challenge by aggregating features within a local patch centered around the projection of the point, as shown Fig. (S-2). However, this approach is unreliable when the sampled point is far from the object surface since the local patch may not include the information of its nearest point. Therefore, for hand-held object
Authors with equal contributions.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Initialization
SDF-Based Reconstruction
(a) (5-1)
(5-2)
(S-3)
(5-4)
RDF-Based Reconstruction
(b)
(D-1)
(D-2)
(D-3)
(D-4)
Figure 1: SDF vs DDF based hand-held object reconstruction. Given input RGB image (a) and estimated hand and camera pose (b), SDF-based and DDF-based reconstruction pipelines vary from sampling spaces (S-1, D-1) and feature aggregation techniques (S-2, D-2 and S-3, D-3). SDF points sampling space must stay close to the object surface (S-1) or would lead to degraded network prediction results [47], while DDF ray sampling space (D-1) can be large enough to encapsulate the hand and object meshes. SDF methods typically aggregate features for the sampled point P in its local patch, which is not reliable when P is far from the object surface (S-2). DDF, however, aggregates features along the projection line r' for ray R, which naturally captures both the information of the point and its intersection with the object surface (D-2). SDF methods cannot directly yield the contact points on the hand surface, so that only global relative hand joints encoding is used (S-3). On the contrast, DDF can get the intersection region of sampled rays and hand surface, leading to more representative local intersection-aware hand encoding (D-3). Due to these characteristics, we demonstrate that DDF more suitable model hand-object interactions. Consequently, DDF-based method achieves more complete and accurate hand-held object reconstruction results (S-4, D-4).
reconstruction, SDF-based methods either directly encode hand pose global cue [67 or propagate information between hand and object in 2D feature space [IO], which fails model hand-object interactions in 3D space. Second, SDF is compact and can not naturally encapsulate the inherent characteristics of an object such as symmetry. However, many man-made objects everyday scenes exhibit some degree of failure to recover high-quality shapes, especially when the object heavily occluded by the hand.
and the inability of SDF to capture this information results
To overcome aforementioned challenges, we present DDF-HO, novel Directed Distance Field (DDF) based Hand-held Object reconstruction framework, which takes single RGB-image input and outputs 3D model of the target object. In contrast to SDF, DDF maps a ray, comprising an origin and a direction, in 3D space corresponding DDF values, including a binary visibility signal determining whether the intersection exists and a scalar distance value measuring the distance from origin to target along the sampled direction.
As shown in Fig. we demonstrate the superiority of DDF over SDF in modeling hand-object interactions. For each sampled ray, collect its features capture hand-object relationship by combining 2D-3D geometric features via our 2D ray-based feature aggregation and 3D intersection- aware hand pose embedding. We first project the ray onto the image, yielding 2D ray or dot (degeneration case), and then aggregate features of all the pixels along the 2D ray as the 2D features, which encapsulate 2D local hand-object cues. Then we collect 3D geometric features, including direct hand pose embedding local geometric prior. In this manner, hand pose and shape serve as strong priors enhance the object reconstruction, especially when there is heavy occlusion. Additionally, we also introduce geometric loss term to exploit the symmetry of everyday objects. In particular, we randomly sample two bijection sets of 3D rays, where corresponding rays have identical origin on the reflective plane with opposite directions. Thus the DDF predictions corresponding rays in the two sets should be the same, enabling direct supervision loss for shape.
global information [77] and ray-hand intersection embedding as

---

In summary, our main contributions are as follows. First, we present DDF-HO, a novel hand-held object reconstruction pipeline that utilizes DDF as the shape representation, demonstrating superiority in modeling hand-ohject intersections over SDF-based competitors. Second, we extract local to global features capturing hand-object relationship by introducing novel 21) ray-based feature aggregation scheme and 3D intersection-aware hand pose embedding. Third, extensive experiments on synthetic and real-world datasets demonstrate that our method consistently outperforms competitors by a large margin, enabling real-world applications requiring high-quality hand-held object reconstruction.
2 Related Works
Hand Pose Estimation. Hand pose estimation methods from RGB(-D) input can be broadly
categorized into two streams: model-free and model-based methods. Model-free methods typically involve lifting detected 2D keypoints to 3D joint positions and hand skeletons [2 44 45
46 [33 [52]
[0]. Alterntivvly. they directly predict 3D hand meshes [1 21 48
On the other hand, model-basee
methods 3 55 58 77 [79 utilize regression or optimization techniques to estimate statistical models
with low-dimensional parameters, such as MANO (44) Our approach aligns with the model-based stream of methods, as they tend to be more robust to occlusion (67.
Single-view Object Reconstruction. The problem of single-view object reconstruction using neural networks has long been recognized as an ill-posed problem. Initially, researchers focus on designing category-specific networks for 3D prediction, either with direct 3D supervision 7 35 T6]
or without it [25] 34
38 68 17]. Some approaches aim to learn a shared model across multiple
categories using 3D voxel representations [12 22 64 65 57 meshes (24 27 63 51 or point
clouds 19, 40 Recently, neural implicit representations have emerged as powerful technique in These approaches have demonstrated impressive performance.
the field 42 2
Hand-held Object Reconstruction. Accurately reconstructing hand-held objects presents a signif- icant challenge, yet it plays a crucial role in understanding human-object interaction. Prior works 20 28 59 [1] aim to simplify this task by assuming access to known object templates and jointly 76
regressing hand poses and 6DoF object poses [15] [8 75
60
71
Joint reasoning approaches
encompass various techniques, including implicit feature fusion 9 23
56] leveraging geometric
constraints 4 6 26 74. and encouraging physical realism
62 49
Recent researches focus on
directly reconstructing hand-held object meshes without relying on any prior assumptions. These methods aim to recover 3D shapes from single monocular RGB inputs. For instance, (2 designs joint network that predicts object mesh vertices and MANO parameters of the hand, while (36 predicts them in the latent space. Additionally, 100 and (67 utilize Signed Distance Field (SDF) as the representation of hand and object shapes. In contrast, our method introduces novel representation
called Directed Distance Field (DDF) and demonstrates its superiority in reconstructing hand-held objects, surpassing the performance of previous SDF-based methods.
3 Method
3.1
Preliminaries
SDF. Consider a 3D object shape O C B, where B 3 denotes the bounding volume that will act as the domain of the field, SDF maps randomly sampled point P E B to a scalar value d representing the shortest distance from P to the surface of the 3D object shape O. This scalar value can be positive, negative, or zero, depending on whether the point lies outside, inside, or on the surface of the object, respectively.
From SDF to DDF, SDF is widely used in the object reconstruction, however, due to its inherent undirected and compact nature, it is hard effectively represent the complex hand-object interactions, as explained in Fig. [ Hence, in this paper, we propose to utilize DDF, recently proposed and applied by 22 31 69.. as an extension of SDF for high-quality hand-held object reconstruction.
DDF, Given a 3D ray Lp,e(t) ())= = P + t0, consisting of an origin P E B and a view direction 0 E S?, where 8² denotes the set of 3D direction vectors having degree-of-freedon If this ray intersects with the target object O C B at some 1 2 0, it is considered as visible, and DDF maps it to non-negative scalar field : B $ - TR,,, measuring the distance from the origin P towards the first intersection with the object along the direction 0. To conveniently model the visibility of a ray,
3

---

Inibaaiizaiinn
Ray Sampling
Ray-Based Aggregation
(--A)
(R-A)
(1-B)
(R-B)
Encoder
(2-A)
[2-B] Feature Maps
(2-C) Interpolation
Ray Feature Extraction
(2-D)
(2-E)
(2-F)
F2D
3D Intersection-Aware Hand Embedding
Reconstruction
ECP B)
Camera Pose
(R-C)
(I-C)
Local J eint Coordinate
(3-8)
(3-D)
Hand Pose
Global Embedding
y-
Intersection
3p
Feature
MLP Layers
Ray
Features
Concatenation
Positional Embedding
Figure 2: Overview DDF-HO. Given RGB-image (I-A), we first employ an off- shellf pose detector to predict camera pose Og and hand pose Hμ (4SD parameters defined in MANO model [54]). as shown in (I-B) and (I-C) respectively. For the input of DDF, we sample multiple rays (R-A) in 3D space, and project them onto the 2D image (R-B). The corresponding intersections with the hand skeleton are also calculated (R-C). Then for each ray Cp.0, we collect 2D ray-based feature F2D from (2-A) to (2-F), and intersection-awant hand embedding F and F5D from (3-A) to (3-D). Finally, we concatenate all features and ray representation as F (P, 0, F2D, F 3D FāD to predict corresponding DDF values.
binary visibility field is introduced as E(P,0) 1 visible], i.e. for visible ray, =. Moreover, (2] 33 provide several convenient ways to convert DDF to other 3D representations including point cloud, mesh vanilla SDF.
3.2 DDF-HO: Overview
Objective. Given a single RGB image I containing a human hand grasping an arbitrary object, DDF-HO aims at reconstructing the 3D shape O of the target object, circumventing the need of object template, category or depth priors.
Initialization. As shown in Fig- (I-A-(I-C,, first adopt off-the-shelf framework 29 55
to estimate the hand articulation Or and the corresponding camera pose Oc for the input image I, where 0gr is defined in the parametric MANO model with 45D articulation parameters (54) and fg
denotes the 6D pose, rotation R E 83(3) and translation E R3, of the perspective camera with respect the world frame.
Image Feature Encoding. Hierarchical feature maps of image I are extracted via ResNet 30 to encode 2D cues, shown in Fig. (2-A) and (2-B).
Ray Sampling. We sample 3D rays {Cp.e}, with origins transform the rays into the normalized wrist frame with the predicted hand pose 0r, as in IHOI [77 The specific ray sampling algorithm adopted in the training stage is introduced in detail in Supplementary Material.
B and directions 0 S and
Ray Feature Aggregation. To predict corresponding DDF values of {Cp,)), collect and concatenate three sources of informatiom: basic ray representations (P, 0}, 2D projected ray features FED. 3D intersection-awarr hand features F&D. For FzD. we project each 3D ray onto the feature maps extracted from I, yielding 2D ray (lp.0. or a dot (degeneration case). Note that in the degeneration case, the sampled 3D ray passes through the camera center, we only need collect FeD inside the patch centered at as in the SDF-based based methods [67]. For other non-trivial cases, we aggregate features along (p..-) using the Ray-Based Feature Aggregation technique, introduced in detail in Sec. 3.3 For Fan. besides global hand pose embedding as in [67]. we also encapsulate the intersection the ray with the hand joints as local geometric cues to depict the relationship of hand-object interaction, which is further introduced in detail in Sec. 3.4

---

DDF Reconstruction. Concatenating F2D. as input, we employ an 8-layer MLP
network [1 to predict corresponding DDF values.
3.3 Ray-Based Feature Aggregation
Previous SDF-based methods [67] typically ag- gregates feature for each sampled point within local patch centered at its projection, posing a significant challenge for hand-held object re- construction the aggregated feature may not contain necessary information for predicting the intersection, illustrated in Fig- (S-2). When the sampled point is far from the object sur- face, its local feature may even be completely extracted from the background, making it in- feasible to predict corresponding SDF values. As consequence, SDF-based methods either leverage hand pose as a global cue [67] or only propagate hand-object features in 2D space [[O], failing to capture hand-object interactions in 3D space. In DDF-HO, besides the ray representa- tion {P.0) 0), we combine two additional sources features F2D and F3D for each sampled 3D ray to effectively aggregate all necessary information for predicting the DDF value.
Object
Ray-Hand Intersection
Ray-Object Intersection
Hand oints (b)
(a)
Figure 3D intersection-aware local geometric
feature 3D= We collect it by resolving the nearest
neighboring hand joints of ray-hand intersection.
We first collect F2n from the input image I. by employing our 2D Ray-Based Feature Aggregation technique. Given a 3D ray Lp.0 as shown in Fig. 2 (R-A) and (R-B), the origin is projected via p KRR + t)/ where Ps denotes 2 component
P. and the direction 8* is determined as the normal vector from p towards the projection of another point P on the 3D ray, yielding the projected 2D ray p, Then we sample K₁ points {p/. i I,..,,K,, Ki} on the 2D ray, and extract local patch features F2, {F'} for all K₁ points as well the feature FPD of origin projection p via bilinear interpolation on the hierarchical feature maps of I. Finally, we leverage the cross aross-attention mechanism to aggregate 2D ray feature F2D for Lp.e as,
F2n 2D + MulliH (F2
20 F F.D.
(1)
where MultiH refers to the multi-head attention and Q F V ==
Comparing with single point based patch fea- tures used in SDF methods, Fan naturally cap- tures more information, leading to superior re- construction quality. First, 2D features from the origin of the 3D ray towards its intersection on the object surface are aggregated, enabling re- liable DDF prediction for the ray whose origin far from the object surface. In this manner, we can sample 3D rays in the whole domain, as shown in Fig. D (D-1), encapsulating and recon- structing hand and object simultaneously with a single set of samples. Second, features related to the hand along the projected 2D ray are also considered, providing strong priors for object reconstruction.
B1
B2
82
Figure 4: Construction of the symmetry loss.
3.4 Hand-Object Interaction Modelling
We model hand-object interactions in two aspects. First, in 2D features maps, hand information along the projected 2D ray is encoded into Fan. introduced in Sec. 3.3 Second, we collect FaD, which encodes both global hand pose embedding FC in [67] (details in Supplementary Material) and local geometric feature Ff indicating the intersection of each ray with the hand. As shown in Fig. 2 (3-C) and (3-D), for each 3D ray Lp 8. Fh is collected in three steps. First, calculate the
5

---

shortest path from Lp, towards the hand skeleton, constructed by the MANO model and predicted hand articulation parameter 0H, yielding starting point ps on Lp 9 and endpoint pD on the hand skeleton. Then we detect Kap nearest neighboring hand joints of p on the hand skeleton, using geodesic distance. Finally, pe is transformed to the local coordinates of detected hand joints (Fig. 3-C), indicated by the MANO model, and thereby obtaining FāD by concatenating all these local coordinates of ipš In summary, F3D is represented as F3D JTD)
Our hand-object interaction modeling technique has two primary advantages over IHOI [67. in which sampled points are only encoded with all articulation points of the hand skeleton, as shown in Fig. (S-3). First, our technique extracts and utilizes 2D features F2n that reflect the interaction between the hand and object, providing more useful cues to reconstruct hand-held objects. Second, we incorporate 3D intersection-baser hand embeddings F3D that offer more effective global to local hand cues as geometric priors to guide the learning of object shape, especially when a 3D ray passing through the contact region between the hand and object. In such case, the intersection with the hand skeleton is closer to the intersection with the object than the origin of the ray, thereby encoding the local hand information around the intersection provides useful object shape priors, as shown in Fig. B In other cases, our method works similarly to IHOI, where the embeddings serve as a global locator to incorporate hand pose.
3.5 Conditional DDF for Hand-held Object Reconstruction
for each 3D ray Lp.0. we leverage an 8-layer ML.P to map F the corresponding DDF value: distance 1) and binary visible signal Ç. The input {P, 0} is positional encoded by y function as [42]. & is output after the 3rd layer to leave the network capacity for the harder distance estimation task. We also introduce skip connection of the input 3D ray {P, 0} to the 4th layer to preserve low-level local geometry.
Given concatenated feature F = (P,
The loss functions of our conditional directed distance field network consist of depth term LD- <) Dj. visibility term L3 BCF {) and symmetry term L. = 1ú D22 with ń) being the predictions of D, E respectively. For symmetric objects, we randomly sample two bijection sets of 3D rays, where corresponding rays have identical origin on the reflective plane but with opposite directions. Specifically, before our experiments, all symmetric objects are preprocessed to be symmetric with respect the XY plane {X 0, Y 0). To determine whether the object is symmetric, we first flipping the sampled points P on the object surface w.r.t the XY plane, yielding Pr. Then we compare the Chamfer Distance between the object surface and p'. If the distance lies below a threshold (Ie-3), the object is considered symmetric. As for building two bijection sets B, : {P,.6) 6,} } and B, : {P2. )2), we first randomly sample origins P1- {(x 3/. (1)2))) }1) and directions 0, {(1,, (1)3 to construct B1. Then, we flip B, with respect to the reflective plane to generate Bg by P2 {(11, (001,81,-30 =21))]. 0₂ ((01, 31; -))))- Since the object is symmetrie, the DDF values D1 D2 of corresponding rays in B₁ and B, should be the same, which establishes our symmetry loss term. The process of constructing the symmetry loss is shown in Fig,
The final loss defined as Lg + >iLn + d2ha, where \ A2 are weighting factors.
4 Experiments
4.1 Experimental Setup
Datasets. A synthetic dataset ObMan 29 and two real-world datasets HO3D(v2)
MOW 6 are 28
utilized to evaluate ODF-HO in various scenarios. ObMan consists of 2772 objects of 8 categories from ShapeNet (8) with 21K grasps generated by GraspIt 43) The grasped objects are rendered
over random backgrounds through Blender We follow 29
67 to split the training and testing sets.
HO3D(v2) 28 contains 77.558 images from 68 sequences with 10 different persons manipulating 10 different YCB objects [5). The pose annotations are yielded by multi-camera optimization pipelines. We follow (28) to split training and testing sets. MOW [6] comprises a total of 442 images and
121 object templates, collected from in-the-wild hand-object interaction datasets [14 [66.. The approximated ground truths are generated via a single- frame optimization method 6 The training and testing splits remain the same as the released code of [67].
hatpss//www.beedeer.og

---

Category ID < F-5t
Category ID F-1Q1
Category ID y CD4
0.6
0.8
IHOI [67
DDF-HO
0.4
0.6
0
2 3 5
01
4
7
2
3
5 6
Figure 5: Reconstruction quality comparisons on Obman 29 by category. From to we demon- strate the results of IHOI [67 Jar, Knife, Remote Control sequentially. DDF-HO consistently outperforms IHOI.
DDF-HO on the categories: Bottle, Bowl, Camera, Can, Cellphone,
Method
F-5 f
F-10 1
CD 4
F-5 f
F-10 T
CD
1
HO [29]
0.08
0.19
4.60
0.05
0.14
6.03
GF [36]
0.09
0.21
5.23
0.07
0.16
6.25
IHOI 66
0.21
0.38
1.99
0.17
0.31
4.17
Ours
0.28
0.42
0.55
0.24
0.36
0.73
Table 1: Results on HO3D(v2) [28]
dataset with finetuning (left) and zero-shot generalization from ObMan [29 dataset (right). Overall best results are in bold.
Evaluation Metries. We first utilize 2 convert the predicted DDF into point cloud representation and then compare against the sampled point cloud from the corresponding ground truth mesh. Following (67] report Chamfer Distance (CD, mm), F-score at 5mm threshold (F-5) and 10mm threshold (F-10) of the converted object point cloud, which reflects the quality of hand-held object reconstruction.
Baselines. Three baseline methods are chosen to ensure fair comparisons with our DDF-HO method, taking into account that they aim to predict the shape of hand-held objects directly from single RGB input and do not rely on known object templates, category information, or depth priors. The baseline
include Atlas-Net (27 based HO [29]. implicit field based GF (36 and SDF based IHOI 677.
Implementation Details. We conduct the training, evaluation and visualization DDF-HO on a single A100 40GB GPU. We use the same off-line systems [29 55 as 667 to estimate the hand and camera poses. We also adopt a ResNet34 [30] image encoder to extract a 5-resolution visual feature pyramid, the same with [67]. In all the three datasets, we sample 20K 3D rays for each object and generate the ground truth using the ray marching algorithm in Trimesh P The number of sampled points K₁ along the projected 2D ray is set to 8 and number of multi- head attention is 2 for 2D Ray-Based Feature Aggregation technique. KAD for F&D introduced in Sec. 3.4 is set as 8. DDF-HO is trained end-to-end using Adam with learning rate of le-4 on ObMan for 100 epochs. Following [7], we use the network weights learned on synthetic ObMan to initialize the training on HO3D(v2) and MOW. Training HO3D(v2) and MOW also use Adam optimizer with learning rate le-5 for another 100 and 10 epochs, respectively following [67]. The weighting factors of the loss for DDF-HO \1, A2 are set to 5.0 and 0.5, respectively. Note that we do not include the symmetry loss term L, in the training except in the Ablation Study Tab. 4 for a fair comparison, since other baselines do leverage this additional information. Note that during evaluation, we convert the DDF representation to point cloud (2). while during visualization, we convert DDF to mesh (1] Details of ground truth generation and network architecture are provided in the Supplementary Material.
4.2 Evaluation on Synthetic Scenarios
As shown in Tab. 3] the evaluation on the synthetic large-scale dataset ObMan demonstrates that DDF-HO achieves high-quality hand-held object reconstruction with the assistance of suitable DDF representation. Our proposed method exhibits sfate-of-the-a of-thherrt performance under all three evaluation metrics of F-5, F-10, and CD. Specifically, we achieve significant improvement over the current state-of-the-art baseb IHOI, with a gain of 13% and 4% on F-5 and F-10, respectively. This improvement can be attributed to DDF-HO's more suitable ray-based feature aggregation technique,
hhttss//trimsh.or/
7

---

Method
F-5
F-10
CD 4
F-5
F-10 f
CD
IHOI [67]
0.10
0.19
7.83
0.09
0.17
8.43
Ours
0.16
0.22
1.59
0.14
0.19
1.89
Table 2: Results on MOW [6] dataset, with the setting of finetuning (left) and zero-shot generalization
from ObMan
29
dataset (right). Overall best results are in bold.
Input image
Ground truth Camera view
Ground truth Novel view
IHOI Camera view
IHOI Novel view
Ours Camera view
Ours Novel view
Figure 6: Visualization results ObMan [29]. Our method consistently outperforms IHOI [67]
which allows for more exquisite hand-held object reconstruction. Notably, DDF-HO's CD) metric is reduced by 85% compared IHOI and 77% compared to HO, indicating that our predicted object surface contains much fewer outliers.
Fig. 6 presents improved visualizations of DDF-HO, showcasing enhanced and more accurate surface reconstruction of hand-held objects. While IHOI can achieve decent object surface recovery within the camera view, the reconstructed surface appears rough with numerous outliers when observed from novel angles. This suggests that IHOI lacks the ability to perceive 3D hand-held objects due to its limited modeling of hand-object interaction in 3D space. In contrast, DDF-HO utilizes a more suitable DDF representation, resulting in smooth and precise reconstructions from any viewpoint of the object.
4.3 Evaluation on Real-world Scenarios
In addition to synthetic scenarios, we conduct exper- iments on two real-world datasets, HO3D(v2) and MOW, to evaluate DDF-HO's performance in handling real-world human-object interactions Table [ consti- tutes the evaluation results on HO3D(v2) after finetun- ing (with related settings described in Section ) as well as the zero-shot generalization results, where directly conducted inference on HO3D(v2) using the weights from training on ObMan. Furthermore, Table 2 presents results on MOW under the same setting.
Method
F-5
F-10 1
CD 1
HO 29
0.23
0.56
0.64
GF (36
0.30
0.51
1.39
IHOI 667
0.42
0.63
1.02
Ours
0.55
0.67
0.14
Table 3: Results on ObMan 29 dataset. Overall best results are in bold.
Our method, after finetuning on the real-world data, achieves oftte-of-the-att performance on both datasets. Specifically, on HO3D(v2), we observe a considerable improvement compared to IHOI and other methods, with an increase in F-5 by 7%, F-10 by 4%, and a significant decrease in CD by 73%. On MOW, our approach also outperforms the previous state-of-the-art methods, achieving remarkable performance gain in terms of increased F-5 (6%), F-10 (3%). and reduced CD (80%) compared with IHOI [67] Fig. 7 shows the visualization comparison results on MOW. DDF-HO performs well under the real-world scenario, yielding more accurate reconstruction results.
Furthermore, the zero-shot experiments demonstrate that DDF-HO has a stronger ability for synthetic- to-real generalization. Specifically, on HO3D(v2), DDF-HO yields superior performance in terms of F-5 by 7%, F-10 by 5%, and a decreased CD by 82%. Moreover, the results on MOW also indicate
8

---

Input Image
Ground truth Camera view
Ground truth Novel view
IHOI Camera view
IHO1 Novel view
Ours Camera view
Ours Novel view
Figure 7: Visualization results on MOW [6]. Our method consistently surpasses IHOI [67
that our method, trained only synthetic datasets, can still achieve decent performance in real-world scenarios, thanks to the generic representation ability of DDF for hand-object interaction modeling.
4.4 Efficiency
To demonstrate the efficiency of DDF-HO, we compare the network size and the running speed with IHOI, which is typical SDF-based hand-held object reconstruction method. All experiments are conducted on single NVIDIA A100 GPU.
DDF-HO runs at 44FPS, which is slower than IHOI with 172 FPS, but still achieves real-time performance. The slower inference comes from the attention caleulation for 2D ray-based feature aggregation and the more elaborated 3D feature generation. For model size, the two methods share a similar scale (24.5M of DDF-HO and 24.0M of IHOI). Generally, the increased parameters mainly come from the cross attention mechanism in 2D ray-based feature aggregation. Other modules are only adopted collect features model hand-object interactions and do not significantly increase the network size.
4.5 Ablation Studies
We conduct ablation studies the ObMan and HO3D(v2) datasets evaluate the impact of three key assets DDF representation: Ray-Based Feature Aggregation (RFA), Intersection-awan Hand Feature (IHF), and Symmetry Loss (SYM). The results of the ablation studies are presented in Tab.
On ObMan, we first replace the SDF representation with DDF without any modifications to feature aggregation, resulting in slight improvement over the SDF-based IHOI with 3% increase in F-5 metric. This indicates that although DDF is more suitable for representing hand-object interaction (with almost 80% decrease in CD metrie), more sophisticated feature aggregation designs required. Next, we add REA considering the characteristics of sampled rays, leading to 5% increase in F-5 and 4% increase in F-10. Subsequently, adding IHF, which models hand-object interaction locally by considering the intersection information of the hand, resulted 5% increase F-5 and 3% increase in F-10. This indicates that considering the intersection information of the hand can improve the accuracy of hand-held object reconstruction. Finally, adding the SYM loss, which captures the symmetry nature of everyday objects and handles self-occluded scenarios caused by hands, results in another 2% increase in F-5 and 1% increase in F-10. On the HO3D(v2) dataset, RFA, IHF, and SYM modules play similar roles as on ObMan.
Additionally, we evaluate the influence of input hand pose on DDF-HO by adding Gaussian noise to the estimated hand poses (Pred) ground truth hand poses (GT) in the input. The results of the ablation studies (Tab. demonstrate the robustness of DDF-HO in handling noisy input hand poses.
Conclusion
In this paper, we present DDF-HO, novel pipeline that utilize DDF the shape representation to reconstruct hand-held objects, and demonstrate its superiority in modeling hand-object interactions

---

RFA IHF SYM
F-ST
F-10
CD 4
F-5
F-10
CD
SDF baseline [67
0.42
0.63
1.02
0.21
0.38
1.99
0,45
0.60
0.22
0.24
0.38
0.62
0.50
0.64
0.17
0.26
0.39
0.58
7
0,55
0.67
0.14
0.28
0.42
0.55
0.57
0.68
0.13
0.29
0.43
0.52
Table 4: Ablation studies for key assets of DDF-HO on ObMan 22 datasets (right).
(left) and HO3D(v2)
Noise
F-5
F-10
CD
F-5
F-10
CD
Pred
0.55
0.67
0.14
0.28
0.42
0.55
Pred+o 0.1
0.54
0.66
0.15
0.24
0.35
0.73
Pred+a 0.5
0.47
0.60
0.18
0.20
0.30
0.83
Pred+o 1.0
0,42
0.55
0.25
0.17
0.27
0.98
Pred+o 1.5
0.38
0.52
0.31
0.14
0.25
1.24
GT
0.59
0.70
0.10
0.30
0.45
0.50
GT++ 0.1
0.58
0.69
0.11
0.27
0.43
0.58
GT+o 0.5
0.51
0.63
0.16
0.23
0.34
0.76
Table 5: Ablation studies for input hand pose on ObMan
29
(left)
and HO3D(v2) datasets
(right). The Pred row remains the same setting with the Tab. and
over competitors. Specifically, for each sampled ray in 3D space, we collect its features capturing seal-tt-global hand-object relationships by introducing novel 2D ray-based feature aggregation and 3D intersection-aware hand pose embedding. Extensive experiments on synthetic dataset Obman and real-world datasets HO3D(v2) and MOW verify the effectiveness of DDF-HO on reconstructing high-quality hand-held objects.
Limitations. DDF-HO naturally inherits the shorteomings of DDF. First, the higher dimensional input of DDF makes it harder train than SDF, resulting in more complex data, algorithm and network structure requirements. This may hinder the performance of DDF-based methods when scaling up to large-scale scenes, like traffic scenes. Second, to enable more photorealistic object reconstruction, there are other characteristics like translucency, material and appearance need to be properly represented. This requires further research to fill the gap.
Acknowledgements. This work was supported by the National Key R&D Program of China under Grant 201 BAAA0102801, National Natural Science Foundation of China under Grant 6 61827804. We also appreciate Yufei Ye, Tristan Aumentado-Armstinn Zerui Chen and Haowen Sun for their insightful discussions.
References
[1] Titas Anciukevičius, Zexiang Xu, Matthew Fisher, Paul Henderson, Hakan Bilen, Niloy J Mitra, and Paul Guerrero. Renderdiffusion: Image diffusion for 3d reconstruction, inpainting and generation. arXiv preprint arXiv:22/ 1,09669, 2022.
[2] Tristan Aumentado-Armstt Armstrong, Stavros Tsogkas, Sven Dickinson, and Allan D Jepson. Representing
3d shapes with probabilistic directed distance fields. In Proceedings of the IEEF/CV Conference on Computer Vision and Pattern Recognition, pages 19343-19354, 2022.
[3] Adnane Boukhayma, Rodrigo de Bem. and Philip HIS Torr. 3d hand shape and pose from images in wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10843- 10852, 2019.
[4] Samarth Brahmbhatt, Chengeheng Tang, Christopher D Twigg, Charles C Kemp, and James Hays. Con- tactpose: A dataset of grasps with object contact and hand pose. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020. Proceedings, Part XIII 16. pages 361-378.
Springer, 2020.
Berk Calli, Arjun Singh, Aaron Walsman, Siddhartha Srinivasa, Picter Abbecl, and Aaron M Dollar. The yeb object and model set: Towards common benchmarks for manipulation rescarch. In 2015 international conference on advanced robotics (ICAR), pages 510-517. IEEE, 2015.
[6] Zhe Cao, 1lija Radosavovic, Angjoo Kanazawa, and Jitendra Malik. Reconstructing hand-object interactions in the wild. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12417-
12426, 2021.
10

---

[7] Thomas J Cashman and Andrew W Fitzgibbon. What shape are dolphins? building 3d morphable models from 2d images. IEEE transactions on pattern analysis and machine inteliigenee,35 (5()):232-244 2012. [8] Angel X Chang. Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang. Zimo Li, Silvio Savarese, Mamolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint TrXXiv:/512.00000
2015.
[9] Yujin Chen. Zhigang Tu. Di Kang, Ruizhi Chen, Linchao Bao, Zhengyou Zhang, and Junsong Yuan. Joint hand-object 3d reconstruction from a single image with cross-brande feature fusion. JEEE Transactions on Image Processing, 30:4008-4022 2021.
[10] Zerui Chen, Yana Hasson, Cordelia Schmid, and Ivan Laptev. Alignsdf: Pose-aligned signed distance fields for band-object reconstruction. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Ivrael. October 23-27, 2022, Proceedings, Purt pages 231-245 Springer, 2022.
[11] Hongsuk Choi, Gyeongsik Moon, and Kyoung Mu Lee. Pose2mesh: Graph convolutional network for 3d human pose and mesh recovery from a 2d human pose. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 233-28, 2020, Proceedings, Part VII 16, pages 769-787. Springer, 2020. [12] Christopher Choy, Danfei Xu. Jun Young Gwak. Kevin Chen, and Silvio Savarese. 3d-r2n2: unified approach for single and multi-view 3d object reconstruction. In Computer Vision ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VIII 14, pages 628 644, Springer, 2016.
[13] Enric Corona, Albert Pumarola, Guillem Alenya, Francesc Moreno-Noguer, and Grégory Rogez. Ganhand: Predicting human grasp affordances in multi-object scenes. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5031-5041. 2020.
[14] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Scaling egocentric vision: The epic-kitchens dataset. In Proceedings of the European Conference on Computer Vision (ECCV), pages 720-736, 2018.
[15] Yan Di, Fabian Manhardt, Gu Wang, Xiangyang Ji, Nassir Navab, and Federico Tombari. So-pose: Exploiting self-occlusion for direct 6d pose estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12396-12405, 2021.
[16] Yan Di. Chenyangguang Zhang. Pengyuan Wang. Guangyao Zhai, Ruida Zhang, Fabian Manhardt, Benjamin Busam, Xiangyang Ji, and Federico Tombari. Ced-3dr: Consistent conditioning in diffusion for single-image 3d reconstruction. arXiv preprint arXiv:2308.
2023.
[17] Yan Di, Chenyangguang Zhang. Ruida Zhang, Fabian Manhardt, Yongzhi Su, Jason Rambach, Didier Stricker, Xiangyang Ji, and Federico Tombari. U-red: Unsupervised 3d shape retrieval and deformation for partial point clouds. arXiv preprint arXiv:23308 23088 06383, 2023.
[18] Yan Di. Ruida Zhang, Zhiqiang Lou, Fabian Manhardt, Xiangyang Ji. Nassir Navab, and Federico Tombari. Gpv-pose: Category-level object pose estimation via geometry guided point-wise voting. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 6781-6791, 2022.
[19] Haoqiang Fan, Hao Su, and Leonidas Guibas. point set generation network for 3d object reconstruction from a single image. In Proceedings of the IEEE conference on computer vision and puttern recognition. pages 605-613, 2017.
[20] Guillermo Aunria-Hernaddo, Shanxin Yuan, Seungryul Back, and Tae-Kyun Kim. First-person hand action benchmark with rgb-d videos and 3d hand pose annotations. In Proceedings of the IEEE conference on computer vision and pattemn recognition, pages 409-419, 2018
[21] Liuhao Gc, Zhou Ren, Yuncheng Li, Zehao Xue, Yingying Wang. Jianfci Cai, and Junsong Yuan. 3d hand shape and pose estimation from a single rgb image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10833-10842, 2019.
[22] Rohit Girdhar, David Fouhey, Mikel Rodriguez, and Abhinav Gupta. Learning predictable and generative vector representation for objects. In Computer Vision ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VI 14, pages 484-499. Springer,
2016.
[23] Georgia Gkioxari, Ross Girshick, Piotr Dollár, and Kaiming He. Detecting and recognizing human-ohject interactions. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 8359-8367, 2018.
[24] Georgia Gkioxari, Jitendra Malik, and Justin Johnson. Mesh r-cnn. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9785-979999 2019.
[25]
Shubham Goel, Angjoo Kanazawa, and Jitendra Malik. Shape and viewpoint without keypoints. In Wsson-EECCV 2020: 16th European Conference, Glasgow, UK. August 23-28, 2020, Proceedings. Part XV16, 16. pages 104. Springer, 2020.
Computer Vision ECCV
[26] Patrick Grady, Chengeheng Tang, Christopher D Twigg, Minh Vo, Samarth Brahmbhatt, and Charles Kemp. Contactopt: Optimizing contact to improve grasps. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 1471-1481, 2021.
[27] Thibault Groucix, Matthew Fisher, Vladimir G Kim, Bryan C Russell, and Mathieu Aubry. A papier- måché approach learning 3d surface generation. In Proceedings of the IEEE conference on computer vision
and pattern recognition, pages 216-224, 2018.
[28] Shreyas Hampali. Mahdi Rad, Markus Oberweger, and Vincent Lepetit. Honnotate: method for 3d annotation of hand and object poses. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3196-3206, 2020.
11

---

[29] Yana Hasson, Gul Varol, Dimitrios Tzionas, Igor Kalevatykh, Michael J Black, Ivan Laptev, and Cordelia Schmid. Learning joint reconstruction of hands and manipulated objects. In Proceedings of the IEEE/CVF conference on computer vision und pattern recognition. pages 11807-11816, 2019.
Shaoqing [30] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition.
In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[31] Trevor Houchens, Cheng- You Lu. Shivam Duggal, Rao Fu, and Srinath Sridhar. Neuralodf: Learning omnidirectional distance fields for 3d shape representation. arXiv preprint arXiv:2206. 2052323200008377, 2022.
[32] Umar lqbal, Pavlo Molchanov, Thomas Breuel Juergen Gall, and Jan Kautz. Hand pose estimation via latent 2.5 d heatmap regression. In Proceedings of the European Conference on Computer Vision (ECCV), pages 118-134, 2018.
[33] Wonbong Jang and Lourdes Agapito. Codenerf: Disentangled neural radiance fields for object categories. In Proceedings of the IEEE/CVF International Conference Computer Vision. pages 12949-12958, 2021. [34] Angjoo Kanazawa, Shubham Tulsiani, Alexei A Efros, and Jitendra Malik. Learning category-specilic mesh reconstruction from image collections. In Proceedings of the European Conference on Computer Vision (ECCV). pages 371-386,22 2018.
[35] Abhishek Kar, Shubham Tulsiani, Joao Carreira, and Jitendra Malik. Category-speciffc object recon- struction from a single image. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1966-1974, 2015.
[36] Korrawe Karunratanakul. Jinlong Yang. Yan Zhang, Michael Black, Krikamol Muandet, and Siyu Tang. Grasping field: Learning implicit representations for human grasps. In 2020 International Conference on 3D Vision (3DV), pages 333-344. IEFF, 2020.
[37] Zhiying Leng, Jiaying Chen, Hubert PH Shum, Frederick WB Li, and Xiaohui Liang, Stable hand pose estimation under tremor via graph neural network. In 202/ IEEE Virtual Reality and 3D User Interfaces
(VR), pages pages, 226-234. IEEE, 2021.
[38] Xueting Li, Sifei Liu, Kihwan Kim, Shalini De Mello, Varun Jampani, Ming-Hsuan Yang. and Jan Kautz. Self-xupervised single-view 3d reconstruction via semantic consisteney. In Computer VisionEECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIV 16, pages 677-693, Springer. 2020.
[39] Yong-Lu Li, Xinpeng Liu, Xiaoqian Wu, Yizhuo Li, and Cewu Lu. Hoi analysis: Integrating and decomposing human-object interaction. Advances in Neural Information Processing Systems, 33:5011- 1-
5022, 2020.
[40] Chen-Hsuan Lin, Chen Kong, and Simon Lucey. Learning efficient point cloud generation for dense 3d object reconstruction. In proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018. [41] Shaowei Liu, Hanwen Jiang, Jiarui Xu, Sifei Liu, and Xiaolong Wang. semi-supervised 3d hand-object poses estimation with interactions in time. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 14687-14697, 2021.
[42] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106 2021.
[43] Andrew T Miller and Peter K Allen. Graspit! versatile simulator for robotic grasping. IEEE Robotics & Automation Magazine, 11-22:10--22, 2004.
[44] Franziska Mueller, Florian Bernard, Oleksandr Sotnychenko, Dushyant Mchta, Srinath Sridhar, Dan Casas, and Christian Thcobalt. Ganerated hands for real-time 3d hand tracking from monocular rgb. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 49-59, 2018.
[45] Franziska Mueller, Micah Davis, Florian Bernard, Oleksandr Sotnychenko, Mickeal Verschoor, Miguel Otaduy, Dan Casas, and Christian Theobalt. Real-time pose and shape reconstruction of two interacting hands with a single depth camera. ACM Transactions on Graphies (ToG), 38(4):1-13, 2019.
[46] Paschalis Panteleris, lason Oikonomidis, and Antonis Argyros. Using a single rgb frame for real time 3d hand pose estimation in the wild. In 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 436-445. IEEE, 2018
[47] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 165-174, 2019.
[48] Akila Pemasiri,
Nguyen Thanh. Sridha Sridharan, and Clinton Fookes. Im2mesh gan: Accurate 3d
hand mesh recovery from a single rgb image. arXiv preprint arXiv:2101. 12239, 2021.
[49] Tu- Ho-Hoa Pham, Nikolaos Kyriazis, Antonis A Argyros, and Abderrahmane Kheddar. Hand-object contact foree estimation from markerless visual tracking. JEEE transactions on pattern analysis and machine intelligence, (0(12):88832889 2017.
[50] Riikaa Rantamaa, Jari Kangas, Sriram Kishore Kumar, Helena Mehtonen, Jorma Järnstedt, and Roope Raisamo. Comparison of a vr stylus with controller, hand tracking, and mouse for object manipulation and medical marking tasks in virtual reality. Applied Sciences, 13(4):2251, 2023.
[51] Edoardo Remelli, Artem Lukoianov, Stephan Richter, Benoit Guillard, Timur Bagautdinov. Pierre Baque and Pascal Fua. Meshsdf: Differentiable iso so-surface extraction. Advances in Neural Information Processing Systems. 33:22468-22477 [52] Grégory Rogez, James S Supancic, and Deva Ramanan. Understanding everyday hands in action from rgb-d images. In Proceedings of the IEEE international conference on computer vision, pages 3889-3897, 2015.
2020.
12

---

[53] Grégory Rogez, James S Supancic III. Maryam Khademi. Jose Maria Martinez Montiel, and Deva Ramanan.
3d hand pose detection in egocentric rgb-d images. arXiv preprint arXiv:1412.006 2014.
[54]
Javier Romero, Dimitrios Tzionas, and Michacl J Black. Embodied hands: Modeling and capturing hands
and bodies together. arXiv preprint S.Eiv:220.0026//1 2022.
[55] Yu Rong. Takaaki Shiratori, and Hanbyul Joo. Frankmocap: Fast monocular 3d hand and body motion
capture by regression and integration. arXiv preprint arXiv:2008.088: 2020.
[56] Dandan Shan, Jiaqi Geng, Michelle Shu, and David Fouhey. Understanding human hands in contact internet scale. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9869-9878, 2020.
[57] Zai Shi, Zhao Meng. Yiran Xing. Yunpu Ma, and Roger Wattenhofer. 3d-retr: End-to-end single and -iew 3d reconstruction with transformers, arXiv preprint 10.0012100.0886.. 2021.
[58] Srinath Sridhar, Franziska Mueller, Antti Oulasvirta, and Christian Theobalt. Fast and robust hand tracking using detection-guidde optimization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3213-3221, 2015.
[59] Srinath Sridhar, Franziska Mueller, Michael Zollhöfer, Dan Casas, Antti Oulasvirta, and Christian Theobalt. Real-time joint tracking of hand manipulating an object from rgb-d input. In Computer Vision ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, Octoher 11-14, 2016, Proceedings, Part 11 14, pages 294-310. Springer, 2016.
[60] Yongzhi Su. Yan Di, Guangyao Zhai, Fabian Manhardt, Jason Rambach, Benjamin Busam, Didier Stricker, and Federico Tombari. Opa-3d: Occlusion-
Dixel-wise aggregation for monocular 3d object detection. IEEE Robotics and Automation Letters, 8(3):1327-133 1134,, 2023.
[61] Bugra Tekin, Federica Bogo, and Mare Pollefeys. II+ o: Unified egocentric recognition of 3d hand-object poses and interactions. In Proceedingy of the IEEE/CVF conference on computer vision and puttern recogmition, pages 451144555555 1-45200 2019.
[62]
Dimitrios Tzionas, Luca Ballan, Abhilash Srikantha, Pablo Aponte, Mare Pollefeys, and Juergen Gall. Capturing hands in action using discriminative salient points and physics simulation. International Journal of Computer Vision, 1 18:1172-193, 2016.
[63] Nanyang Wang, Yinda Zhang. Zhuwen Li, Yanwei Fu, Wei Liu, and Yu-Gang Jiang. Pixel2mesh: Generating 3d mesh models from single rgb images. In Proceedings of the European conference on computer vision (ECCV), pages 52-67, 2018.
[64] Jiajun Wu, Chengkai Zhang, Tianfan Xue, Bill Freeman, and Josh Tenenbaum. Learning probabilistic latent space of object shapes via 3d generative-adversarial modeling. Advances in neural information processing systems, 29, 2016.
[65] Haozhe Xte, Hongxun Yao, Xiaoshuai Sun, Shangehen Zhou, and Shengping Zhang. Pix2vox: Context- aware 3d reconstruction from single and multi-view images. In Proceedings of the IEFF/CVF international
conference on computer vision, pages 2690-2698, 2019.
Yinzhen Xu, Weikang Wan, Jialiang Zhang. Haoran Liu, Zikang Shan, Hao Shen, Ruicheng Wang. Haoran Geng, Yijia Weng, Jiayi Chen, et al. Unidexgrasp: Universal robotic dexterous grasping via learning diverse proposal generation and goal-conditioned policy, arXiv preprint arXiv:2303.009 [67] Yufei Ye, Abhinav Gupta, and Shubham Tulsiani. What's 's in your hands? 3d reconstruction of generic objects in hands. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3895-3905, 2022.
[66]
2023.
[68]
Yufci Yc, Shubham Tulsiani, and Abhinay Gupta. Shelf-supervised mesh prediction in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8843-8852,
2021.
[69] Yuta Yoshitake, Mai Nishimura, Shohei Nobuhara, and Ko Nishino. Transposer: Transformer as an optimizer for joint object shape and pose estimation. arXiv preprint arXiv:2303. 13477, 2023.
[70] Alex Yu, Vickie Yc, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one or few images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 4578-4587, 2021.
[71] Michela Zaccaria, Fabian Manhardt, Yan Di, Federico Tombari, Jacopo Alcotti, and Mikhail Giorgini. Self-supervised category-level 6d object pose estimation with optical flow consistency, IEEE Robotics and Automation Letters, 8(5):2510-2511 2023.
[72] Guangyao Zhai, Xiaoni Cai, Dianye Huang. Yan Di, Fabian Manhardt, Federico Tombari, Nassir Navab, and Benjamin Busam. bot: Object rearrangement via coarse-to-fing robotic imagination on scene graphs. arXiv preprint 22Xiv:2309.111111
2023.
[73] Guangyao Zhai, Dianye Huang, Shun-Cheng Wu, HyunJun Jung, Yan Di, Fabian Manhardt, Federico Tombari, Nassir Navab, and Benjanin Busam. Monograspnet: 6-dof grasping with single rgb image. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 1708-1714. IECE, 2023. [74] Jason Y Zhang, Sam Pepose. Hanbyul Joo, Deva Ramanan, Jitendra Malik, and Angjoo Kanazawa. Perceiving 3d noman--bbjcct spatial arrangements from a single image in the wild. Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XII 16, pages 34-51. Springer, 2020.
[75] Ruida Zhang, Yan Di, Zhiqiang Lou, Fabian Manhardt, Federico Tombari, and Xiangyang Ji. Rbp-pose: Residual bounding box projection for category-level pose estimation. In European Conference on Computer Vision, pages 655-672. Springer, 2022.
13

---

[76] Ruida Zhang, Yan Di, Fabian Manhardt. Federico Tombari, and Xiangyang Ji. Ssp-pose: Symmetry-awant
shape prior deformation for direct category-level object pose estimation. 2022 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7452-7459, IEEE, 2022.
[77] Xiong Zhang, Qiang Li, Hong Mo, Wenbo Zhang, and Wen Zheng. End-to-end hand mesh recovery from monocular rgh image. Proceedings of the IEEF/CVF International Conference on Computer Vision,
pages 2354-2364, 2019.
[78] Juntian Zheng, Qingyuan Zheng. Lixing Fang, Yun Liu, and Li Yi. Cams: Canonicalized manipulation spaces for category-level functional hand-object manipulation synthesis. arXiv preprint Xiv:2333. 15469, 2023.
[79] Yuxiao Zhou, Mare Habermann, Weipeng Xu, Ikhsanul Habibie, Christian Theobalt, and Feng Xu. Monocular real-time hand shape and motion capture using multi-modal data. In Proceedings of the Computer Vision and Puttern Recognition, pages 5346 5355, 2020.
JEEE/CVF Conference
[80] Christian Zimmermann and Thomas Brox. learning estimate 3d hand pose from single rgh images. In Proceedings of the IEFE international conference on computer vision, pages 4903-4911, 2017.
14


Reviews:
[{"review_id": "tgdi4i0H5L", "reviewer": "Reviewer_9guR", "summary": "This paper presents a system for joint hand and hand-held object 3D reconstruction. The authors propose a pipeline to (1) predict the hand (MANO hand model) and camera poses with an off-the-shelf pose estimator; (2) extract image features with an off-the-shelf ResNet; (3) sample \"3D ray representations\", project them to the 2D image plane, and extract the corresponding image features and aggregating them with cross-attention along the ray directions; (4) for the same 3D points, extract 3D features from the global hand embedding from the geodesic nearest joint; (5) predict a directed distance function (DDF) with an MLP, using the above features as input. Experiments show that the proposed system predicts more accurate shapes.", "questions": "It would be great if the authors could address the questions raised in the weakness section.", "limitations": "Yes", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "- The proposed method is evaluated on synthetic and real hand-object interaction datasets, showing improvements over baselines on nearly all metrics. Ablative analyses are also provided to better understand the behaviors of the proposed DDF-HO.", "weaknesses": "- The presentation is poor. Specifically, it is unclear what the use of ray sampling is, and how the random ray directions transform to the directions pointing to the target hand (R-A to R-B in Fig 2). Also, it is unclear how one would know a sampled ray would (not) point to the direction closest to the hand shape, without the DDF even being optimized. In addition, it seems that the major contribution of the paper is a system proposal as a whole, rather than the choice of representation (DDF vs. SDF). I think the message of the paper is somewhat misleading in this sense.\n- Why is the final output in the form of DDF, if the end goal is to extract the surface of the hand shape? Predicting an SDF or occupancy field would serve the same purpose as well. Also, it is not clear how DDF is being taken advantage of in the proposed system. The evaluation metric is based on surface point cloud representations as well.\n- It is not clear to me how the proposed method is better in the real-world datasets. Visually, they seem to be very far from good predictions of the object shapes.\n- It would be good to analyze failure cases to better understand the limitations of the proposed system.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your valuable and constructive comments. We are delighted that you found our method “predicts more accurate shape” than competitors. Our detailed responses are listed below and we revise the manuscript accordingly.\n\n## Q1. About Ray Sampling from (R-A) to (R-B)\nWe define DDF in L.106 — L.113 in Sec. 3.1. It maps a 3D ray (consisting of an origin and a view direction) to a non-negative scalar field (L.109) and a binary visibility field (L.111). The binary visibility field plays a crucial role in identifying which sampled rays do not intersect with the hand-held object. This identification is significant as these non-intersecting rays do not contribute to the overall reconstruction process. Therefore, in the pipeline Fig. 2, we only showcase the rays that are necessary in reconstruction. The ray sampling algorithm is introduced in detail in Sec. 3 in the Supplementary Material. To avoid misleading the readers, we have incorporated a distinct color to denote rays that do not intersect with the object in Fig. 2 in the final version. This visual distinction serves to enhance the accuracy of the demonstration.\n\n## Q2. Our Contributions\nWe first explain why SDF is not a suitable choice for hand-held objection reconstruction in our paper (L.27 – L.40 in Introduction, L.102 – L.105 in Sec. 3.1, L.142 – L.159 in Sec. 3.3). SDF is undirected and too compact that fails to naturally encapsulate local interactions between hand and the object (L.27 – L.40). Then we point out that DDF is a better choice and design DDF-HO which utilizes DDF as the shape representation for hand-held object reconstruction. We list our contributions in L.60—L.66 in the main paper. We briefly summarize them here for clarity. We present DDF-HO, a novel pipeline that utilizes DDF for hand-held reconstruction, outperforming SDF-based competitors. Based on the DDF representation, we propose a novel ray-based feature aggregation scheme to model hand-object relationship, which boosts the overall reconstruction quality. Extensive experiments and ablation studies demonstrate the effectiveness of DDF-HO.\n\n## Q3. Occupancy as the Shape Representation\nSince no off-the-shelf hand-held object reconstruction pipelines leverage occupancy as the shape representation, we design a baseline method ourselves following the widely-used 2D-3D lifting scheme in single-view reconstruction [Pix2Vox]. We first use the same backbone as our method (ResNet34) to extract per-pixel features. The extracted features are then back-projected to the volume (32x32x32, the same as [Pix2Vox]). For each voxel inside the volume, the predicted hand pose (the same as used in DDF-HO, parameterized as the MANO model parameters) is also concatenated to its feature vector. Finally, we predict the occupancy as in Pix2Vox [Pix2Vox]. The results on ObMan (the upper block) and HO3D(V2) (the bottom block) are shown as follows.\n\n|Method|F5|F10|CD|\n|------|------|------|------|\n|IHOI| 0.42 | 0.63 | 1.02  |      \n|DDF-HO| 0.55 |0.67 |0.14  |    \n|Pix2Vox |0.24 |0.45 |1.81 |    \n|------|------|------|------|\n|IHOI|0.21 |0.38 |1.99|\n|DDF-HO|0.28 |0.42 |0.55|\n|Pix2Vox |0.06 |0.17 |6.12|\n\n\nThereby we explain why occupancy is also not a suitable choice in hand-held object reconstruction. First, like SDF, occupancy representation is also undirected. It can only model the local shape of either object or hand. For voxels that are not near the hand, the occupancy representation also cannot naturally capture the relationship between the hand and the object. Second, the reconstruction quality is limited by the resolution of the volume. Since the entire volume is typically uniformly divided into voxels, for object regions with particularly complex geometric structures, large voxels might limit the network's expressive capacity, resulting in decreased accuracy of the results.\n\n## Q4. DDF in DDF-HO\nWe use DDF to represent the 3D shape in our pipeline DDF-HO. As introduced in Sec. 3.5 in the paper, after extracting corresponding features of each ray (Sec. 3.3 and Sec.3.4), we leverage an 8-layer MLP to map the features to the corresponding DDF values: distance and binary visibility signal. (L.139 – L.140, L.200 – L.201). DDF can then be converted into other commonly used 3D representations including point cloud, mesh and vanilla SDF (L.112 – L.113), using the algorithms introduced in [2, 27].The superiority of DDF to SDF is accentuated in our main paper (L.27 - L.46, and also in Fig. 1). We demonstrate the DDF is a more suitable representation for hand-held object reconstruction.\n\n## Q5. Unsatisfactory Visualization Results\nOur goal is to reconstruct the hand-held object from a single RGB image without any prior knowledge of the object. This is a recently emerging and very challenging research topic. Although our method still does not completely solve the problem, we make a significant leap forward in accuracy compared to previous methods (Tab. 1, 2, 3). In the future, combining other techniques, like diffusion models, with our DDF based reconstruction framework may further boost the performance.\n\n## Q6. Failure Cases\nMOW pencil is a typical failure case (Row 3 in Fig. 3 of the Supplementary Material). Currently, reconstructing very thin objects is still a big challenge for all methods. In the final version and the submitted PDF attachment, we have added more visual results of the failure cases to comprehensively showcase our method.\n\n[Pix2Vox] Xie, H., Yao, H., Sun, X., Zhou, S., \\& Zhang, S. (2019). Pix2vox: Context-aware 3d reconstruction from single and multi-view images. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 2690-2698)."}, {"review_id": "zhiDeuPYJY", "reviewer": "Reviewer_7sso", "summary": "The paper proposes an algorithm for reconstructing hand-held object from a single RGB image. Instead of using the traditional Signed Distance Fields (SDF), this paper proposes to leverage Directed Distance Field (DDF) as the shape representation. Experiment shows that the proposed algorithm outperforms SOTA.", "questions": "see weaknesses", "limitations": "did not talk about potential negative societal impact", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed pipeline utilizes DDF as the shape representation to reconstruct hand-held objects, which has a stronger modelling capability for this specific task, e.g., reconstruction for hand-held objects from a single RGB image. It introduce a 2D ray-base feature aggregation and 3D intersection-aware hand pose embedding.\nThe experiments are conducted on both real and synthetic dataset,", "weaknesses": "The paper did not mention the running speed, model complexity, I wonder if it's comparable with SDF based models.\nThe paper mentioned that the DDF is harder to train, and requires more complex data, algorithm, and network structure, I wonder if the paper can give more quantitative measurement? e.g. 1 or 2 magnitude harder/longer/more parameters?\n\nAnother alternative to SDF will be the Occupancy, the paper did not mention Occupancy at all. Won't Occupancy be a strong baseline model? or replacing SDF with Occupancy will make the algorithm (proposed and SOTA) perform better?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive review. We revised the manuscript accordingly. Below we try our best to address your concerns and questions.\n\n## Q1. Running Speed and Model Complexity\nIHOI is a typical SDF-based hand-held object reconstruction method. We list detailed comparisons with IHOI in the following table.\n\n||DDF|IHOI\n|------|------|------|\nTraining time| 2d 8h |IHOI: 2d|\nParameters|24.5M| 24.0M\nRunning Speed|23ms/img (44FPS) |5.8ms/img (172FPS)|\n\nFrom the above table, it can be seen that our method runs slower than IHOI but still achieves real-time performance.All experiments in the above table are conducted on a single NVIDIA A100 GPU. For model size, the two methods share a similar scale. Generally, the increased parameters mainly come from the cross attention mechanism in 2D ray-based feature aggregation (Sec. 3.3). Other modules are only adopted to collect features to model hand-object interactions and do not significantly increase the network size. Moreover, since the network needs to predict both the distance and visibility signal, the convergence speed of DDF-HO during training is also slower than IHOI.\n\n## Q2. Occupancy as the Shape Representation\nSince no off-the-shelf hand-held object reconstruction pipelines leverage occupancy as the shape representation, we design a baseline method ourselves following the widely-used 2D-3D lifting schemed single-view reconstruction method [Pix2Vox]. We first use the same backbone as our method (ResNet34) to extract per-pixel features. The extracted features are then back-projected to the volume (32x32x32, the same as [Pix2Vox]). For each voxel inside the volume, hand pose is also concatenated to its feature vector. Finally, we predict the occupancy as in Pix2Vox [Pix2Vox]. The results on ObMan (the upper block) and HO3D(V2) (the bottom block) are shown as follows.\n\n|Method|F5|F10|CD|\n|------|------|------|------|\n|IHOI| 0.42 | 0.63 | 1.02  |      \n|DDF-HO| 0.55 |0.67 |0.14  |    \n|Pix2Vox |0.24 |0.45 |1.81 |    \n|------|------|------|------|\n|IHOI|0.21 |0.38 |1.99|\n|DDF-HO|0.28 |0.42 |0.55|\n|Pix2Vox |0.06 |0.17 |6.12|\n\nThereby we explain why occupancy is also not a suitable choice in hand-held object reconstruction. First, like SDF, occupancy representation is also undirected. It can only model the local shape of either object or hand. For voxels that are not near the hand, the occupancy representation also cannot naturally capture the relationship between the hand and the object. Second, the reconstruction quality is limited by the resolution of the volume.  Since the entire volume is typically uniformly divided into voxels, for object regions with particularly complex geometric structures, large voxels might limit the network's expressive capacity, resulting in decreased accuracy of the results.\n\n[Pix2Vox] Xie, H., Yao, H., Sun, X., Zhou, S., \\& Zhang, S. (2019). Pix2vox: Context-aware 3d reconstruction from single and multi-view images. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 2690-2698)."}, {"review_id": "ZfxzXJ7qQ1", "reviewer": "Reviewer_e2Ui", "summary": "The authors introduce a novel approach called DDF-HO, which uses Directed Distance Field (DDF) for 3D hand-held object reconstruction. Unlike SDF, DDF includes origins and directions of the views in the 3D space. They show that the ray-based feature aggregation scheme and 3D intersection-aware hand pose embeddings are more suitable for 3D hand-held object reconstruction. The DDF-HO method outperforms prior work.", "questions": "1. How long does it take to reconstruct one image?", "limitations": "1, As the authors mentioned, it is hard to train DDF on higher dimensional inputs.\n2. As the authors mentioned, the reconstructions do not reflect the translucency, material, and appearance of the objects\n3. The reconstruction quality is not good, especially on real-world images", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Proposed a new data structure DDF for hand-held object reconstruction.\n2. Ray-Based Feature Aggregation technique has better representations of local geometric features.\n3. The interaction modeling reflects the interaction between the hand and object rather than just hand as in the prior work.\n4. Significant improvements over prior work.\n5. Comprehensive evaluation and ablation studies.", "weaknesses": "1. Lack of efficiency comparison between SDF-based methods and proposed DDF. Such as the amount of resources required for reconstructing the same input.\n2. No appearance on the reconstruction results", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate you for the precious review time and valuable comments. We revised the manuscript according to the review. Below we try our best to address your concerns and questions.\n\n## Q1. Efficiency\nIHOI is a typical SDF-based hand-held object reconstruction method. We list detailed comparisons with IHOI in the following table.\n\n||DDF|IHOI\n|------|------|------|\nTraining time| 2d 8h |IHOI: 2d|\nParameters|24.5M| 24.0M\nRunning Speed|23ms/img (44FPS) |5.8ms/img (172FPS)|\n\nFrom the above table, it can be seen that our method runs slower than IHOI but still achieves real-time performance.All experiments in the above table are conducted on a single NVIDIA A100 GPU. For model size, the two methods share a similar scale. Generally, the increased parameters mainly come from the cross attention mechanism in 2D ray-based feature aggregation (Sec. 3.3). Other modules are only adopted to collect features to model hand-object interactions and do not significantly increase the network size. Moreover, since the network needs to predict both the distance and visibility signal, the convergence speed of DDF-HO during training is also slower than IHOI.\n\n## Q2. No Appearance in the Reconstruction Results\nIn this paper, we focus on recovering the geometric shape of the hand-held object. As other reviewers point out, currently reconstructing the shape of hand-held object without prior knowledge of the object is still a challenging and open problem, since no existing methods demonstrate satisfactory performance. Experiments showcase that our method achieves state-of-the-art performance in real-world and synthetic datasets. As for the appearance, in the future work, a coloring network like in PC2 [PC2] can be adopted to predict the color for each pixel. Moreover, since DDF shares similar input as NeRF, neural view synthesis may be also possible by slightly adjusting the DDF representation, which will make the problem very interesting. \n\n[PC2] Melas-Kyriazi, L., Rupprecht, C., \\& Vedaldi, A. (2023). PC2: Projection-Conditioned Point Cloud Diffusion for Single-Image 3D Reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 12923-12932)."}, {"review_id": "SQA7LPo717", "reviewer": "Reviewer_1CC4", "summary": "This paper proposes a directed distance field-based method for hand-held object reconstruction from a single RGB image. The paper aggregates 2D ray-based features to capture ray-object intersection and 3D geometric features of ray-hand intersection. In particular, it extracts local to global cues via the above features and introduces a symmetry loss term to handle symmetric objects. Experiments on three datasets and ablations on the introduced modules show the effectiveness of the proposed method.", "questions": "1. The baselines mentioned in the paper (HO, GF, AlignSDF, and IHOI) all consider the intersection between hand and object and report the corresponding metric Intersection Volume [1], which are not mentioned in this paper. The authors should provide experimental results of Intersection Volume and state why the proposed method is able to outperform other methods on this metric (if applicable) without considering the contact/intersection between hands and objects.\n\n2. In this paper, the canonical hand shape of MANO is used and only the articulation parameters of the hand are taken for modeling. However, hand shape is also a geometric cue, important for hand-held object reconstruction, but difficult to learn, which is considered in the baselines (HO and GF). The authors should make sure that they are comparing these baselines in a fair setting.\n\n[1] Hasson, Y., Varol, G., Tzionas, D., Kalevatykh, I., Black, M.J., Laptev, I., Schmid, C.: Learning joint reconstruction of hands and manipulated objects. In: CVPR (2019)", "limitations": "The paper has well clarified the limitations of the proposed methods, which are inherited from the shortcomings of DDF.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The proposed idea of extracting local 2D image features along the ray and local 3D features from the ray-hand relationship to provide geometric cues is novel in hand-held object reconstruction.\n\n2. Supporting experiments show that DDF is a more suitable representation than SDF for handheld object reconstruction.", "weaknesses": "1. Unclear descriptions. It's better to provide a clearer description of how to sample the bijection sets on the reflective plane for symmetry loss, preferably with an illustration.\n\n2. Insufficient qualitative results. It's suggested to provide qualitative results of the ablation studies, especially the ablation on symmetry loss term. Besides, it should be declared that whether adding such a symmetry loss term leads to a wrong distance field for asymmetric objects.\n\n3. Insufficient ablations on input hand poses. It would be better to add Gaussian noises at a larger scale (e.g., \\sigma=1.0,1.5) to the input hand poses.\n\n4. Insufficient ablations on K_l and K_{3D}. Why K_l=8 is set to sample points along the projected 2D ray. And why K_{3D}=8 (nearly half of hand joints) is set to select the neighboring hand joints, since considering fewer hand joints seems to be more efficient for capturing local ray-hand features.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your valuable and constructive comments. Our detailed responses are listed below and we revised the manuscript accordingly.\n\n## Q1. Unclear Descriptions of Symmetry Loss\nFollowing other baseline methods, we exclude the symmetry loss except for ablation studies (L. 243 in the manuscript). Hence, our results across different datasets and visualizations are symmetry loss-free. Tab. 4 shows a slight improvement due to the symmetry loss. Our updated paper and the uploaded PDF attachment now include qualitative object comparisons with and without the symmetry loss.\n\nBefore our experiments, all symmetric objects in the datasets are preprocessed to be symmetric with respect to the XY plane $\\{x=0, y=0\\}$ . To determine whether the object is symmetric, we first flipping the sampled points P on the object surface w.r.t the XY plane, yielding P'. Then we compare the Chamfer Distance between the object surface and P'. If the distance lies below a threshold (1e-3), the object is considered symmetric. As for building two bijection sets B1 $\\{P_1, \\theta_1\\}$ and B2 $\\{P_2, \\theta_2\\}$, we first randomly sample origins $\\{P_1: (x_1, y_1, z_1)\\}$ and directions $\\{\\theta_1: (\\alpha_1, \\beta_1, \\gamma_1)\\}$ to construct B1. Then, we flip B1 with respect to the reflective plane to generate B2 by $\\{P_2: (x_1, y_1, -z_1)\\}$, $\\{\\theta_2: (\\alpha_1, \\beta_1, -\\gamma_1)\\}$. Since the object is symmetric, the DDF values of corresponding rays in B1 and B2 should be the same, which establishes our symmetry loss term. (L. 206 in the manuscript). In the final version, we refine our explanation in L. 207—L.209 by adding mathematical symbols and also a figure (shown in the uploaded attachment) to illustrate the construction of the symmetry loss in the Supplementary Material.\n\n\n## Q2. Insufficient Ablations on Symmetry Loss\nPlease note that adding this symmetry loss is a natural choice of using DDF representation. Therefore, symmetry loss is not our main contribution. For fair comparisons, we do not use the symmetry loss except in the ablation studies in Tab. 4, as explained in L.243 in the manuscript. Moreover, as explained in L. 207, we apply this loss only on symmetric objects during training. Thus, it should make little influence on the asymmetric objects. Such observation is made on our ablations shown in the table below.\n||F5 |  F10 |   CD\n|-|-|-|-|\nSymmetric objects (w/o symmetry loss) |0.57 | 0.69  |0.12|\nAsymmetric objects (w/o symmetry loss)|0.50 | 0.62|0.17|\nSymmetric objects (w/ symmetry loss) | 0.59 | 0.70 | 0.11|\nAsymmetric objects (w/ symmetry loss)| 0.51 | 0.62|0.16|\n\n## Q3. Additional Experiments on the Noise of Hand Pose\nWe also conduct hand pose ablations with larger Gaussian noise on ObMan (the upper block) and HO3D (the bottom block). The results are exhibited below. Adding too large noise on hand poses will lead to drop on the reconstruction precision (F5). However, the CD metric remains to be 0.31 on ObMan even with sigma=1.5 Gaussian noise, which is still lower than SDF based baselines. This demonstrates the robustness of DDF.\n\n||F5 |  F10 |   CD\n|-|-|-|-|\nPred (no noise)        |            0.55 | 0.67 | 0.14|\nsigma=1.0 |              0.42  |0.55 | 0.25|\nsigma=1.5  |             0.38 | 0.52 | 0.31|\nPred  (no noise)  |     0.28  |0.42  |0.55|\nsigma=1.0|0.17 | 0.27  |0.98|\nsigma=1.5   |            0.14|0.25 | 1.24|\n\n## Q4. Additional Ablations on $K_l$ and $K_{3D}$\nWe show the relevant ablations of $K_l$ and $K_{3D}$ on ObMan in the table below. When setting $K_l=12$, the performance will be enhanced a little, but with the cost of larger network parameters and slower inference speed. Increasing $K_{3D}$ to $12$ does not provide any improvements towards the reconstruction since (1) DDF-HO has already considered the global hand feature $F_{3D}^G$; (2) collecting local intersection feature $\\mathcal{F}_{3D}^L$ in a larger field will not provide more local information.\n\n|keep $K_{3D}=8$   |     F5|F10 |   CD|\n|-|-|-|-|\n$K_l=4$        |       0.53 | 0.65 | 0.16|\n$K_l=8$        |       0.55 | 0.67 | 0.14|\n$K_l=12$        |      0.56 | 0.68 | 0.13|\n|keep $K_{l}=8$ |||\n$K_{3D}=4$       |     0.54 | 0.66 | 0.15|\n$K_{3D}=8$|0.55|0.67  |0.14|\n$K_{3D}=12$     |      0.55 | 0.67|  0.14|\n\n## Q5. Results w.r.t Intersection Volume\nResults are shown in the following table. It can be seen clearly that DDF-HO outperforms IHOI by 0.07 on ObMan and 0.57 on HO3D. Our method explicitly models the hand-object interactions, or in other words, 'considers the contact/intersection between hands and objects', in two aspects. First, on the 2D image, hand information along the projected 2D ray is encoded into the corresponding 2D features $F_{2D}$ (Sec. 3.3). Second, Our 3D embeddings $F_{3D}$ encapsulate both global hand shape feature and local hand-object geometric features (Sec.3.4).\n\n||   ObMan|HO3D |\n|-|-|-|\n|HO |8.64|                     10.03 |\n|GF |1.84 |                    7.16|\n|IHOI| 1.74|                  4.92|\n|Ours| 1.67 |                 4.35 |\n\n## Q6. Fair Comparison with Baselines\nWe strictly follow the setting of the SDF-based baseline IHOI. Given an RGB image and predicted hand pose as input, we reconstruct the hand-held object. Therefore, our comparisons with IHOI are strictly fair. For other methods, since they also predict the hand shape while we directly get the hand mesh with the predicted hand pose and MANO models, we mainly compare the object reconstruction quality with them, like IHOI in its paper. Sharing the exactly same setting towards IHOI, we can ensure the fairness when comparing with such methods. On an overall level, our method and all other methods can start from an RGB image and ultimately obtain the reconstructed result of the hand-held object. Therefore the comparisons are supposed to be fair."}, {"review_id": "tSmcjDEhc0", "reviewer": "Reviewer_dmTA", "summary": "This work proposes a novel pipeline that uses DDF as the shape representation for hand-held object reconstruction from a single image. DDFs provide benefits over SDFs, eg. they are directed, provide intersection with object information, and can capture symmetry. Extensive experiments on ObMan, HO3D, and MOW datasets show the effectiveness of the proposed approach over existing methods.", "questions": "Clarifications required are mentioned in the weaknesses above.", "limitations": "Limitations are discussed.\n\n---\nI appreciate the additional ablations and clarifications provided by the authors. After reading the rebuttal, other reviews, and discussion, I think that the authors have addressed the main concerns pointed out in the reviews. So, I am retaining my rating of `Weak Accept`.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- This work proposes to use DDF, which is more expressive than SDF, for hand-held object reconstruction. While SDFs are undirected and cannot capture symmetry, DDFs provide a directed distance to the surface along with intersection with object information (visibility) and can capture symmetry.\n- The proposed approach uses ray-based feature aggregation and intersection-aware hand features to better capture hand-object interaction compared to existing SDF-based methods.\n- Extensive experiments on ObMan (Table 3), HO3D (Table 1) and MOW (Table 2) datasets show the effectiveness of the proposed approach over existing methods.\n- Ablation studies on different components (Table 4) and robustness to noise in hand pose (Table 5) are helpful in understanding the capabilities of the proposed approach. Also, error bars over 5 training seeds are provided in the supplementary.", "weaknesses": "- The HO3D splits used are different than IHOI[62]. Is there any reason for this?\n- The IHOI[62] scores on MOW (Table 2) are different than those reported in IHOI paper, even though the same splits are used (L221-222). Why is this the case?\n- It'd be useful to have ablations on the different ray sampling strategies used during training, as stated in Sec. 3 in the supplementary. Specifically, how well does uniform sampling perform by itself? This could be a limitation when scaling DDF-HO to cases where 3D ground truth object models are not available (eg. in-the-wild settings).\n- How does the training & inference time for DDF-HO compare to IHOI? Since several rays need to be sampled, it seems that DDF-HO would be slower than IHOI.\n- It'd be helpful to include more details about ray sampling during testing in the main paper, eg. how many rays, how are the origin & directions sampled.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for taking the time to thoroughly review our paper. Your insights are highly valuable and provide us with guidance to enhance the paper. Below we try our best to address your concerns and questions. We have revised the paper according to your suggestions. \n\n## Q1. Different Splits with IHOI\nThe released code of IHOI is incomplete. We found that the preprocessed SDF data provided by IHOI doesn’t match its split files on HO3D and MOW. Thus we contacted the first author of IHOI for this issue in early April, 2023 during development of DDF-HO. She replied that her released data is not the same as used in the experiments in the paper. Under her help, we re-trained and re-evaluated all baseline methods on the two datasets using the split files provided in the released code of IHOI. Therefore, the results on HO3D and MOW are a little different from the results reported in IHOI.\n\n## Q2. Different Results on MOW\nAs explained in L. 221–L. 222, we follow the splits of the released code of IHOI. The split files in the released code are a little different from the ones used in the original paper. We use the new split files to evaluate the performance, leading to slightly different results.\n\n## Q3. Ablations on Different Ray Sampling Strategies\nThanks for the suggestion! We have conducted the ablations of different ray sampling methods on ObMan in the following table (Sampling methods 1 -- 5 are introduced in Sec. 3 of the Supplementary Material). It can be seen clearly that applying multiple kinds of sampling strategies contributes to improvements in performance, which is also verified in [DDF]. Please note that in testing stage, DDF-HO only uses uniform sampling since no prior shape information is accessible (introduced in the first paragraph in Sec. 3 of the Supplementary Material). Therefore, it can handle in-the-wild hand-held objects.\n\n| |     F5|F10 |   CD|\n|------|------|------|------|\n1                 |     0.53 | 0.66 | 0.17\n1,2               |     0.53 | 0.66 | 0.16\n1,2,3            |      0.54 | 0.66 | 0.15\n1,2,3,4           |     0.55 | 0.66 | 0.15\n1,2,3,4,5        |      0.55 | 0.67 | 0.14\n\n\n## Q4. Efficiency of DDF-HO\nIHOI is a typical SDF-based hand-held object reconstruction method. We list detailed comparisons with IHOI in the following table.\n\n||DDF|IHOI\n|------|------|------|\nTraining time| 2d 8h |IHOI: 2d|\nParameters|24.5M| 24.0M\nRunning Speed|23ms/img (44FPS) |5.8ms/img (172FPS)|\n\nFrom the above table, it can be seen that our method runs slower than IHOI but still achieves real-time performance. All experiments in the above table are conducted on a single NVIDIA A100 GPU. The slower inference comes from the attention calculation for 2D ray-based feature aggregation and the more elabored 3D feature generation. Ray sampling in the testing time only takes less than 1ms since only randomly sampling on unit sphere for ray starting points and directions is conducted. For model size, the two methods share a similar scale. Generally, the increased parameters mainly come from the cross attention mechanism in 2D ray-based feature aggregation (Sec. 3.3). Other modules are only adopted to collect features to model hand-object interactions and do not significantly increase the network size. Moreover, since the network needs to predict both the distance and visibility signal, the convergence speed of DDF-HO during training is also slower than IHOI.\n\n## Q5. Ray Sampling During Inference\nAs introduced in L. 24—L. 25 in the supplementary material, during inference, the ground truth object in the datasets is already canonicalized and resized to fit in a unit sphere. Thus, we simply use the uniform sampling to generate 20K 3D rays. The origins are randomly sampled inside the sphere and the directions are also randomly sampled from the uniform distribution. Since the sampled 3D rays may not intersect with the object, the yielded point cloud from DDF prediction contains less than 20K points. Thus, we randomly sample 12K points for evaluation. For fair comparisons, all other methods are trained with 20K points and tested using 12K points (This is also in accordance with the released code of IHOI). We clarified this information in the final version.\n\n[DDF] Aumentado-Armstrong, T., Tsogkas, S., Dickinson, S., & Jepson, A. D. (2022). Representing 3D shapes with probabilistic directed distance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 19343-19354)."}, {"review_id": "3R2LufMINY", "reviewer": "Reviewer_hGDA", "summary": "The paper presents DDF-HO, a method for handheld object reconstruction based on Directed Distance Fields. Given a single RGB image containing a hand grasping an arbitrary object, DDF-HO reconstructs the object without requiring a template or depth priors. Previous methods addressing this problem have relied on the Signed Distance Field (SDF) for the same purpose. The paper includes experiments comparing DDF-HO with such methods on multiple datasets demonstrating the advantages of the proposed method in terms of accuracy.\n\n\n**Final Rating**\n\nAfter reading the numerous reviews and the responses of the authors, I believe that this is a paper that can/should be accepted. As I wrote in my previous post, thorough rewriting is needed in some parts to improve clarity.", "questions": "Answers to the first two weaknesses listed above could go a long way toward improving my rating. (I went with a conservative rating at this stage.)", "limitations": "More information on complexity and run times would have been useful.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1. DDF-HO obtains 3D reconstructions of much higher quality than recent methods [25, 32, 62]. The paper provides sufficient evidence that this is due to the use of DDF instead of SDF as the representation, the use of a ray-based feature aggregation scheme, and a 3D intersection-aware hand pose embedding. These contributions lead to significant increases in reconstruction accuracy.\n\nS2. The method is described clearly and with sufficient detail to enable reasonable reproduction. The code is also included, but I did not try to work with it, or identify the key pieces. Handling of symmetry is the exception to this comment.\n\nS3. Three large-scale, widely used datasets, one synthetic and two real, are used to generate the experimental results. The protocol follows that of IHOI [62], which is the most closely related prior work. Two additional recent baseline methods have been chosen for the experiments.\n\nDDF-HO outperforms the baselines on all three datasets. It also shows good zero-shot generalization properties, suggesting that it does not overfit. The experimental results section includes some analysis of potential reasons for the differences in performance across algorithms. The ablation studies are also informative, especially the comparisons to IHOI.", "weaknesses": "W1. My primary concern about the paper is the number and complexity of steps. DDF-HO requires sampling rays, computing and aggregating 2D and 3D features, measuring geodesic distances etc. The authors acknowledge that their method has higher complexity than the baselines in the limitations section, but I would like to see more analysis and data on the tradeoffs between speed and accuracy. How long do training and inference take for IHOI and DDF-HO on similar hardware and data? Is interactive deployment feasible?\n\nW2. Handling of symmetric objects is not presented clearly overall. How is the reflective plane of symmetric objects discovered?\n\n\nMinor Comments (not affecting my recommendation)\n\nI find the use of “undirected” as a property of SDF confusing. The sign and the distance value of SDF direct us to the nearest surface. I do not have a good suggestion of an alternative.\n\nSmall language errors can be found throughout the paper. Examples include missing articles and minor inconsistencies.\n\n98: “Arbitrarily” is more appropriate than “randomly.”\n\nFigure 4 and Tables 1 and 2 can be placed closer to the text that refers to them.\n\nRefereed, rather than arXiv, versions of papers should be cited whenever possible.\n\nThe first paragraph of Section 3 of the supplement is very important for understanding the algorithm, in my opinion. I suggest finding some space for it in the main paper. One of the weaknesses I had noted was lack of clarity on the ray sampling process.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive review. Below we try our best to address your concerns and questions. We have already revised the paper according to the reviews.\n\n## Q1. Efficiency and Model Complexity of DDF-HO\nIHOI is a typical SDF-based hand-held object reconstruction method. We list detailed comparisons with IHOI in the following table.\n\n||DDF|IHOI\n|------|------|------|\nTraining time| 2d 8h |IHOI: 2d|\nParameters|24.5M| 24.0M\nRunning Speed|23ms/img (44FPS) |5.8ms/img (172FPS)|\n\nFrom the above table, it can be seen that our method runs slower than IHOI but still achieves real-time performance. All experiments in the above table are conducted on a single NVIDIA A100 GPU. The slower inference comes from the attention calculation for 2D ray-based feature aggregation and the more elabored 3D feature generation. Ray sampling in the testing time only takes less than 1ms since only randomly sampling on unit sphere for ray starting points and directions is conducted. For model size, the two methods share a similar scale. Generally, the increased parameters mainly come from the cross attention mechanism in 2D ray-based feature aggregation (Sec. 3.3). Other modules are only adopted to collect features to model hand-object interactions and do not significantly increase the network size. Moreover, since the network needs to predict both the distance and visibility signal, the convergence speed of DDF-HO during training is also slower than IHOI.\n\n## Q2. Symmetry Loss\nSince other baseline methods don’t leverage this prior knowledge, we do not use the symmetry loss except in the ablation studies in Tab. 4, as explained in L.243 in the manuscript. Therefore, our results on different datasets and visualization results are all obtained without using the symmetry loss. From Tab. 4, we demonstrate that this symmetry loss can improve the results slightly.\n\nBefore our experiments, all symmetric objects in the datasets are preprocessed to be symmetric with respect to the XY plane $\\{x=0, y=0\\}$ . To determine whether the object is symmetric, we first flipping the sampled points P on the object surface w.r.t the XY plane, yielding P'. Then we compare the Chamfer Distance between the object surface and P'. If the distance lies below a threshold (1e-3), then the object is considered symmetric. As for building the two bijection sets B1 $\\{P_1, \\theta_1\\}$ and B2 $\\{P_2, \\theta_2\\}$, we first randomly sample origins $\\{P_1: (x_1, y_1, z_1)\\}$ and directions $\\{\\theta_1: (\\alpha_1, \\beta_1, \\gamma_1)\\}$ to construct B1. Then, we flip B1 with respect to the reflective plane to generate B2 by $\\{P_2: (x_1, y_1, -z_1)\\}$, $\\{\\theta_2: (\\alpha_1, \\beta_1, -\\gamma_1)\\}$. Since the object is symmetric, the DDF values of corresponding rays in B1 and B2 should be the same, which establishes our symmetry loss term. (L. 206 in the manuscript). In the final version, we refine our explanation in L. 207—L.209 by adding mathematical symbols and also a figure to illustrate the construction of the symmetry loss in the Supplementary Material.\n\n## Q3. Minor Comments\nThank you very much for the detailed suggestions.\n\n(1) We'll consult with native speakers to identify a more fitting term to characterize the SDF's nature. We acknowledge that 'undirected' primarily pertains to graph descriptions, and while we are yet to discover a superior alternative, we remain open to suggestions.\n\n(2) We checked the typos again and corrected them.\n\n(3) Arbitrarily —-> Randomly\n\n(4) The positions of the figures are adjusted in the final version.\n\n(5) All citations are checked again.\n\n(6) Thank you again for the suggestion. We also find the information introduced in Sec. 3 is very important to understand our method. We put it in the main paper in the final verison.\n"}]

Author rebuttal:
We sincerely appreciate the valuable work by all ACs and reviewers. We are delighted that DDF-HO is considered to show "improvements over baselines on nearly all metrics" [9guR, e2Ui, dmTA, hGDA], "stronger modelling capability" [7sso], "novelty in hand-held object reconstruction" [1CC4]. We revised the manuscript according to the suggestions. Below we try our best to address the common concerns and questions.


## Q1. Efficiency [All Reviewers]
IHOI is a typical SDF-based hand-held object reconstruction method. We list detailed comparisons with IHOI in the following table.

||DDF|IHOI
|------|------|------|
Training time| 2d 8h |IHOI: 2d|
Parameters|24.5M| 24.0M
Running Speed|23ms/img (44FPS) |5.8ms/img (172FPS)|

From the above table, it can be seen that our method runs slower than IHOI but still achieves real-time performance.All experiments in the above table are conducted on a single NVIDIA A100 GPU. For model size, the two methods share a similar scale. Generally, the increased parameters mainly come from the cross attention mechanism in 2D ray-based feature aggregation (Sec. 3.3). Other modules are only adopted to collect features to model hand-object interactions and do not significantly increase the network size. Moreover, since the network needs to predict both the distance and visibility signal, the convergence speed of DDF-HO during training is also slower than IHOI.

## Q2. Occupancy as the Shape Representation [9guR, 7sso]
Since no off-the-shelf hand-held object reconstruction pipelines leverage occupancy as the shape representation, we design a baseline method ourselves following the widely-used 2D-3D lifting scheme in single-view reconstruction [Pix2Vox]. We first use the same backbone as our method (ResNet34) to extract per-pixel features. The extracted features are then back-projected to the volume (32x32x32, the same as [Pix2Vox]). For each voxel inside the volume, the predicted hand pose (the same as used in DDF-HO, parameterized as the MANO model parameters) is also concatenated to its feature vector. Finally, we predict the occupancy as in Pix2Vox [Pix2Vox]. The results on ObMan (the upper block) and HO3D(V2) (the bottom block) are shown as follows.

|Method|F5|F10|CD|
|------|------|------|------|
|IHOI| 0.42 | 0.63 | 1.02  |      
|DDF-HO| 0.55 |0.67 |0.14  |    
|Pix2Vox |0.24 |0.45 |1.81 |    
|------|------|------|------|
|IHOI|0.21 |0.38 |1.99|
|DDF-HO|0.28 |0.42 |0.55|
|Pix2Vox |0.06 |0.17 |6.12|


Thereby we explain why occupancy is also not a suitable choice in hand-held object reconstruction. First, like SDF, occupancy representation is also undirected. It can only model the local shape of either object or hand. For voxels that are not near the hand, the occupancy representation also cannot naturally capture the relationship between the hand and the object. Second, the reconstruction quality is limited by the resolution of the volume. Since the entire volume is typically uniformly divided into voxels, for object regions with particularly complex geometric structures, large voxels might limit the network's expressive capacity, resulting in decreased accuracy of the results.

## Q3. Symmetry [hGDA, 1CC4]
We provide a figure in the attachment to demonstrate the sampling method of building the bijection sets for the symmetry loss. We also provide qualitative comparisons of results with or without symmetry loss.

Meta-review: This submission received the following scores: R, BR, SA, BR, WA, WA

The authors complained about the "Reject" reviewer, and I agreed with them it seems the reviewer has a very superficial read of the paper.

There was some ethic concerns at some point (about skin color) but I think this aspect is solved - the datasets have various skin tones.

Overall, the reviewers are positive about the contribution and experiments. The authors gave additional, better explanations about the contribution and this new content should be included in the final version.
Decision: Accept (poster)