SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Chenyangguang Zhang $^{1*}$ , Yan Di $^{2*}$ , Ruida Zhang $^{1*}$ , Guangyao Zhai $^{2}$ , Fabian Manhardt $^{3}$ , Federico Tombari $^{2,3}$ , Xiangyang Ji $^{1}$
$^{1}$ Tsinghua University,  $^{2}$ Technical University of Munich,  $^{3}$ Google, {zcyg22@mails., zhangrd21@mails., xyji@}tsinghua.edu.cn, {yan.di@}tum.de
Reconstructing hand-held objects from a single RGB image is an important and challenging problem. Existing works utilizing Signed Distance Fields (SDF) reveal limitations in comprehensively capturing the complex hand-object interactions, since SDF is only reliable within the proximity of the target, and hence, infeasible to simultaneously encode local hand and object cues. To address this issue, we propose DDF-HO, a novel approach leveraging Directed Distance Field (DDF) as the shape representation. Unlike SDF, DDF maps a ray in 3D space, consisting of an origin and a direction, to corresponding DDF values, including a binary visibility signal determining whether the ray intersects the objects and a distance value measuring the distance from origin to target in the given direction. We randomly sample multiple rays and collect local to global geometric features for them by introducing a novel 2D ray-based feature aggregation scheme and a 3D intersection-aware hand pose embedding, combining 2D-3D features to model hand-object interactions. Extensive experiments on synthetic and real-world datasets demonstrate that DDF-HO consistently outperforms all baseline methods by a large margin, especially under Chamfer Distance, with about  $80\%$  leap forward. Codes are available at https://github.com/ZhangCYG/DDFHO.
Hand-held object reconstruction refers to creating a 3D model for the object grasped by the hand. It is an essential and versatile technique with many practical applications, e.g. robotics [78, 66, 39, 73, 72], augmented and virtual reality [37], medical imaging [50]. Hence, in recent years, significant research efforts have been directed towards the domain of reconstructing high-quality shapes of hand-held objects, without relying on object templates or depth information. Despite the progress made, most existing methods rely on the use of Signed Distance Fields (SDF) as the primary shape representation, which brings about two core challenges in hand-held object reconstruction due to the inherent characteristics of SDF.
First, SDF is an undirected function in 3D space. Consequently, roughly determining the nearest point on the target object to a sampled point in the absence of object shape knowledge is infeasible. This limitation poses a significant challenge for single image hand-held object reconstruction as it is difficult to extract the necessary features to represent both the sampled point and its nearest neighbor on the object surface. Previous methods [67] have attempted to address this challenge by aggregating features within a local patch centered around the projection of the point, as shown in Fig. 1 (S-2). However, this approach is unreliable when the sampled point is far from the object surface since the local patch may not include the information of its nearest point. Therefore, for hand-held object
*Authors with equal contributions.
Figure 1: SDF vs DDF based hand-held object reconstruction. Given an input RGB image (a) and estimated hand and camera pose (b), SDF-based and DDF-based reconstruction pipelines vary from sampling spaces (S-1, D-1) and feature aggregation techniques (S-2, D-2 and S-3, D-3). SDF points sampling space must stay close to the object surface (S-1) or would lead to degraded network prediction results [47], while DDF ray sampling space (D-1) can be large enough to encapsulate the hand and object meshes. SDF methods typically aggregate features for the sampled point  $P$  in its local patch, which is not reliable when  $P$  is far from the object surface (S-2). DDF, however, aggregates features along the projection line  $r'$  for ray  $R$ , which naturally captures both the information of the point and its intersection with the object surface (D-2). SDF methods cannot directly yield the contact points on the hand surface, so that only global relative hand joints encoding is used (S-3). On the contrast, DDF can get the intersection region of sampled rays and hand surface, leading to more representative local intersection-aware hand encoding (D-3). Due to these characteristics, we demonstrate that DDF is more suitable to model hand-object interactions. Consequently, DDF-based method achieves more complete and accurate hand-held object reconstruction results (S-4, D-4).
reconstruction, SDF-based methods either directly encode hand pose as a global cue [67] or propagate information between hand and object in 2D feature space [10], which fails to model hand-object interactions in 3D space. Second, SDF is compact and can not naturally encapsulate the inherent characteristics of an object such as symmetry. However, many man-made objects in everyday scenes exhibit some degree of (partial-)symmetry, and the inability of SDF to capture this information results in a failure to recover high-quality shapes, especially when the object is heavily occluded by the hand.
To overcome aforementioned challenges, we present DDF-HO, a novel Directed Distance Field (DDF) based Hand-held Object reconstruction framework, which takes a single RGB-image as input and outputs 3D model of the target object. In contrast to SDF, DDF maps a ray, comprising an origin and a direction, in 3D space to corresponding DDF values, including a binary visibility signal determining whether the intersection exists and a scalar distance value measuring the distance from origin to target along the sampled direction.
As shown in Fig. 1, we demonstrate the superiority of DDF over SDF in modeling hand-object interactions. For each sampled ray, we collect its features to capture hand-object relationship by combining 2D-3D geometric features via our 2D ray-based feature aggregation and 3D intersection-aware hand pose embedding. We first project the ray onto the image, yielding a 2D ray or a dot (degeneration case), and then aggregate features of all the pixels along the 2D ray as the 2D features, which encapsulate 2D local hand-object cues. Then we collect 3D geometric features, including direct hand pose embedding as global information [67] and ray-hand intersection embedding as local geometric prior. In this manner, hand pose and shape serve as strong priors to enhance the object reconstruction, especially when there is heavy occlusion. Additionally, we also introduce a geometric loss term to exploit the symmetry of everyday objects. In particular, we randomly sample two bijection sets of 3D rays, where corresponding rays have identical origin on the reflective plane but with opposite directions. Thus the DDF predictions of corresponding rays in the two sets should be the same, enabling a direct supervision loss for shape.
In summary, our main contributions are as follows. First, we present DDF-HO, a novel hand-held object reconstruction pipeline that utilizes DDF as the shape representation, demonstrating superiority in modeling hand-object intersections over SDF-based competitors. Second, we extract local to global features capturing hand-object relationship by introducing a novel 2D ray-based feature aggregation scheme and a 3D intersection-aware hand pose embedding. Third, extensive experiments on synthetic and real-world datasets demonstrate that our method consistently outperforms competitors by a large margin, enabling real-world applications requiring high-quality hand-held object reconstruction.
Hand Pose Estimation. Hand pose estimation methods from RGB(-D) input can be broadly categorized into two streams: model-free and model-based methods. Model-free methods typically involve lifting detected 2D keypoints to 3D joint positions and hand skeletons [32, 44, 45, 46, 53, 52, 80]. Alternatively, they directly predict 3D hand meshes [11, 21, 48]. On the other hand, model-based methods [3, 55, 58, 77, 79] utilize regression or optimization techniques to estimate statistical models with low-dimensional parameters, such as MANO [54]. Our approach aligns with the model-based stream of methods, as they tend to be more robust to occlusion [67].
Single-view Object Reconstruction. The problem of single-view object reconstruction using neural networks has long been recognized as an ill-posed problem. Initially, researchers focus on designing category-specific networks for 3D prediction, either with direct 3D supervision [7, 35, 16] or without it [25, 34, 38, 68, 17]. Some approaches aim to learn a shared model across multiple categories using 3D voxel representations [12, 22, 64, 65, 57], meshes [24, 27, 63, 51], or point clouds [19, 40]. Recently, neural implicit representations have emerged as a powerful technique in the field [42, 31, 2, 1, 47, 33, 70]. These approaches have demonstrated impressive performance.
Hand-held Object Reconstruction. Accurately reconstructing hand-held objects presents a significant challenge, yet it plays a crucial role in understanding human-object interaction. Prior works [20, 28, 59, 61] aim to simplify this task by assuming access to known object templates and jointly regressing hand poses and 6DoF object poses [15, 18, 75, 76, 60, 71]. Joint reasoning approaches encompass various techniques, including implicit feature fusion [9, 23, 41, 56], leveraging geometric constraints [4, 6, 13, 26, 74], and encouraging physical realism [62, 49]. Recent researches focus on directly reconstructing hand-held object meshes without relying on any prior assumptions. These methods aim to recover 3D shapes from single monocular RGB inputs. For instance, [29] designs a joint network that predicts object mesh vertices and MANO parameters of the hand, while [36] predicts them in the latent space. Additionally, [10] and [67] utilize Signed Distance Field (SDF) as the representation of hand and object shapes. In contrast, our method introduces a novel representation called Directed Distance Field (DDF) and demonstrates its superiority in reconstructing hand-held objects, surpassing the performance of previous SDF-based methods.
SDF. Consider a 3D object shape  $\mathcal{O} \subset \mathcal{B}$ , where  $\mathcal{B} \subset \mathbb{R}^3$  denotes the bounding volume that will act as the domain of the field, SDF maps a randomly sampled point  $\mathcal{P} \in \mathcal{B}$  to a scalar value  $d$  representing the shortest distance from  $\mathcal{P}$  to the surface of the 3D object shape  $\mathcal{O}$ . This scalar value can be positive, negative, or zero, depending on whether the point lies outside, inside, or on the surface of the object, respectively.
From SDF to DDF. SDF is widely used in the object reconstruction, however, due to its inherent undirected and compact nature, it is hard to effectively represent the complex hand-object interactions, as explained in Fig. 1. Hence, in this paper, we propose to utilize DDF, recently proposed and applied by [2, 31, 69], as an extension of SDF for high-quality hand-held object reconstruction.
DDF. Given a 3D ray  $L_{\mathcal{P},\theta}(t) = \mathcal{P} + t\theta$ , consisting of an origin  $\mathcal{P}\in \mathcal{B}$  and a view direction  $\theta \in \mathbb{S}^2$  where  $\mathbb{S}^2$  denotes the set of 3D direction vectors having 2 degree-of-freedom. If this ray intersects with the target object  $\mathcal{O}\subset \mathcal{B}$  at some  $t\geq 0$ , it is considered as visible, and DDF maps it to a non-negative scalar field  $\mathcal{D}:\mathcal{B}\times \mathbb{S}^2\to \mathbb{R}_+$ , measuring the distance from the origin  $\mathcal{P}$  towards the first intersection with the object along the direction  $\theta$ . To conveniently model the visibility of a ray, a
Figure 2: Overview of DDF-HO. Given an RGB-image (I-A), we first employ an off-the-shelf pose detector to predict camera pose  $\theta_{C}$  and hand pose  $\theta_{H}$  (45D parameters defined in MANO model [54]), as shown in (I-B) and (I-C) respectively. For the input of DDF, we sample multiple rays (R-A) in 3D space, and project them onto the 2D image (R-B). The corresponding intersections with the hand skeleton are also calculated (R-C). Then for each ray  $\mathcal{L}_{\mathcal{P},\theta}$ , we collect 2D ray-based feature  $\mathcal{F}_{2D}$  from (2-A) to (2-F), and 3D intersection-aware hand embedding  $\mathcal{F}_{3D}^{\mathcal{G}}$  and  $\mathcal{F}_{3D}^{\mathcal{L}}$  from (3-A) to (3-D). Finally, we concatenate all features and ray representation as  $\mathcal{F} = \{\mathcal{P},\theta ,\mathcal{F}_{2D},\mathcal{F}_{3D}^{\mathcal{G}},\mathcal{F}_{3D}^{\mathcal{L}}\}$  to predict corresponding DDF values.
binary visibility field is introduced as  $\xi(\mathcal{P},\theta) = \mathbb{1}[L_{\mathcal{P},\theta}$  is visible], i.e. for a visible ray,  $\xi(\mathcal{P},\theta) = 1$ . Moreover, [2, 31] provide several convenient ways to convert DDF to other 3D representations including point cloud, mesh and vanilla SDF.
3.2 DDF-HO: Overview
Objective. Given a single RGB image  $\mathcal{I}$  containing a human hand grasping an arbitrary object, DDF-HO aims at reconstructing the 3D shape  $\mathcal{O}$  of the target object, circumventing the need of object template, category or depth priors.
Initialization. As shown in Fig. 2 (I-A)-(I-C), we first adopt an off-the-shelf framework [29, 55] to estimate the hand articulation  $\theta_{H}$  and the corresponding camera pose  $\theta_{C}$  for the input image  $\mathcal{I}$ , where  $\theta_{H}$  is defined in the parametric MANO model with 45D articulation parameters [54] and  $\theta_{C}$  denotes the 6D pose, rotation  $\mathcal{R} \in \mathbb{SO}(3)$  and translation  $t \in \mathbb{R}^3$ , of the perspective camera with respect to the world frame.
Image Feature Encoding. Hierarchical feature maps of image  $\mathcal{I}$  are extracted via ResNet [30] to encode 2D cues, as shown in Fig. 2 (2-A) and (2-B).
Ray Sampling. We sample 3D rays  $\{\mathcal{L}_{\mathcal{P},\theta}\}$ , with origins  $\mathcal{P} \in \mathcal{B}$  and directions  $\theta \in \mathbb{S}^2$ , and transform the rays into the normalized wrist frame with the predicted hand pose  $\theta_H$ , as in IHOI [67]. The specific ray sampling algorithm adopted in the training stage is introduced in detail in the Supplementary Material.
Ray Feature Aggregation. To predict corresponding DDF values of  $\{\mathcal{L}_{\mathcal{P},\theta}\}$ , we collect and concatenate three sources of information: basic ray representations  $\{\mathcal{P},\theta\}$ , 2D projected ray features  $\mathcal{F}_{2D}$ , 3D intersection-aware hand features  $\mathcal{F}_{3D}$ . For  $\mathcal{F}_{2D}$ , we project each 3D ray onto the feature maps extracted from  $\mathcal{I}$ , yielding a 2D ray  $\{l_{p,\theta^*}\}$  or a dot (degeneration case). Note that in the degeneration case, the sampled 3D ray passes through the camera center, we only need to collect  $\mathcal{F}_{2D}$  inside the patch centered at  $p$ , as in the SDF-based methods [67]. For other non-trivial cases, we aggregate features along  $\{l_{p,\theta^*}\}$  using the 2D Ray-Based Feature Aggregation technique, introduced in detail in Sec. 3.3. For  $\mathcal{F}_{3D}$ , besides global hand pose embedding as in [67], we also encapsulate the intersection of the ray with the hand joints as local geometric cues to depict the relationship of hand-object interaction, which is further introduced in detail in Sec. 3.4.
DDF Reconstruction. Concatenating  $\{\mathcal{P},\theta ,\mathcal{F}_{2D},\mathcal{F}_{3D}\}$  as input, we employ an 8-layer MLP network [31] to predict corresponding DDF values.
3.3 Ray-Based Feature Aggregation
Previous SDF-based methods [67] typically aggregates feature for each sampled point within a local patch centered at its projection, posing a significant challenge for hand-held object reconstruction as the aggregated feature may not contain necessary information for predicting the intersection, as illustrated in Fig. 1 (S-2). When the sampled point is far from the object surface, its local feature may even be completely extracted from the background, making it infeasible to predict corresponding SDF values. As a consequence, SDF-based methods either leverage hand pose as a global cue [67] or only propagate hand-object features in 2D space [10], failing to capture hand-object interactions in 3D space. In DDF-HO, besides the ray representa
Figure 3: 3D intersection-aware local geometric feature  $\mathcal{F}_{3D}^{\mathcal{L}}$ . We collect it by resolving the nearest neighboring hand joints of ray-hand intersection.
tion  $\{\mathcal{P},\theta \}$ , we combine two additional sources of features  $\mathcal{F}_{2D}$  and  $\mathcal{F}_{3D}$  for each sampled 3D ray to effectively aggregate all necessary information for predicting the DDF value.
We first collect  $\mathcal{F}_{2D}$  from the input image  $\mathcal{I}$ , by employing our 2D Ray-Based Feature Aggregation technique. Given a 3D ray  $\mathcal{L}_{\mathcal{P},\theta}$ , as shown in Fig. 2 (R-A) and (R-B), the origin is projected via  $p = K(\mathcal{RP} + t) / \mathcal{P}_z$ , where  $\mathcal{P}_z$  denotes  $z$  component of  $\mathcal{P}$ , and the direction  $\theta^*$  is determined as the normal vector from  $p$  towards the projection of another point  $\mathcal{P}^*$  on the 3D ray, yielding the projected 2D ray  $l_{p,\theta^*}$ . Then we sample  $K_l$  points  $\{p_l^i,i = 1,\dots,K_l\}$  on the 2D ray, and extract local patch features  $\mathcal{F}_{2D}^{l} = \{\mathcal{F}^{i}\}$  for all  $K_l$  points as well as the feature  $\mathcal{F}_{2D}^{p}$  of origin projection  $p$  via bilinear interpolation on the hierarchical feature maps of  $\mathcal{I}$ . Finally, we leverage the cross-attention mechanism to aggregate 2D ray feature  $\mathcal{F}_{2D}$  for  $L_{\mathcal{P},\theta}$  as,
$$
\mathcal {F} _ {2 D} = \mathcal {F} _ {2 D} ^ {p} + M u l t i H \left(\mathcal {F} _ {2 D} ^ {p}, \mathcal {F} _ {2 D} ^ {l}, \mathcal {F} _ {2 D} ^ {l}\right) \tag {1}
$$
where MultiH refers to the multi-head attention and  $Q = \mathcal{F}_{2D}^{p}$ ,  $K = V = \mathcal{F}_{2D}^{l}$ .
Comparing with single point based patch features used in SDF methods,  $\mathcal{F}_{2D}$  naturally captures more information, leading to superior reconstruction quality. First, 2D features from the origin of the 3D ray towards its intersection on the object surface are aggregated, enabling reliable DDF prediction for the ray whose origin is far from the object surface. In this manner, we can sample 3D rays in the whole domain, as shown in Fig. 1 (D-1), encapsulating and reconstructing hand and object simultaneously with a single set of samples. Second, features related to the hand along the projected 2D ray are also considered, providing strong priors for object reconstruction.
Figure 4: Construction of the symmetry loss.
3.4 Hand-Object Interaction Modelling
We model hand-object interactions in two aspects. First, in 2D features maps, hand information along the projected 2D ray is encoded into  $\mathcal{F}_{2D}$ , as introduced in Sec. 3.3. Second, we collect  $\mathcal{F}_{3D}$  which encodes both global hand pose embedding  $\mathcal{F}_{3D}^{\mathcal{G}}$  as in [67] (details in Supplementary Material) and local geometric feature  $\mathcal{F}_{3D}^{\mathcal{L}}$  indicating the intersection of each ray with the hand. As shown in Fig. 2 (3-C) and (3-D), for each 3D ray  $\mathcal{L}_{\mathcal{P},\theta}$ ,  $\mathcal{F}_{3D}^{\mathcal{L}}$  is collected in three steps. First, we calculate the
shortest path from  $\mathcal{L}_{\mathcal{P},\theta}$  towards the hand skeleton, constructed by the MANO model and predicted hand articulation parameter  $\theta_{H}$ , yielding starting point  $\mathcal{P}^S$  on  $\mathcal{L}_{\mathcal{P},\theta}$  and endpoint  $\mathcal{P}^D$  on the hand skeleton. Then we detect  $K_{3D}$  nearest neighboring hand joints of  $\mathcal{P}^D$  on the hand skeleton, using geodesic distance. Finally,  $\mathcal{P}^S$  is transformed to the local coordinates of detected hand joints (Fig. 2 3-C), indicated by the MANO model, and thereby obtaining  $\mathcal{F}_{3D}^{\mathcal{L}}$  by concatenating all these local coordinates of  $\mathcal{P}^S$ . In summary,  $\mathcal{F}_{3D}$  is represented as  $\mathcal{F}_{3D} = \{\mathcal{F}_{3D}^{\mathcal{G}},\mathcal{F}_{3D}^{\mathcal{L}}\}$ .
Our hand-object interaction modeling technique has two primary advantages over IHOI [67], in which sampled points are only encoded with all articulation points of the hand skeleton, as shown in Fig. 1 (S-3). First, our technique extracts and utilizes 2D features  $\mathcal{F}_{2D}$  that reflect the interaction between the hand and object, providing more useful cues to reconstruct hand-held objects. Second, we incorporate 3D intersection-based hand embeddings  $\mathcal{F}_{3D}$  that offer more effective global to local hand cues as geometric priors to guide the learning of object shape, especially when a 3D ray passing through the contact region between the hand and object. In such case, the intersection with the hand skeleton is closer to the intersection with the object than the origin of the ray, thereby encoding the local hand information around the intersection provides useful object shape priors, as shown in Fig. 3. In other cases, our method works similarly to IHOI, where the embeddings serve as a global locator to incorporate hand pose.
3.5 Conditional DDF for Hand-held Object Reconstruction
Given concatenated feature  $\mathcal{F} = \{\mathcal{P},\theta ,\mathcal{F}_{2D},\mathcal{F}_{3D}\}$  for each 3D ray  $\mathcal{L}_{\mathcal{P},\theta}$ , we leverage an 8-layer MLP to map  $\mathcal{F}$  to the corresponding DDF value: distance  $D$  and binary visible signal  $\xi$ . The input  $\{\mathcal{P},\theta \}$  is positional encoded by  $\gamma$  function as [42].  $\xi$  is output after the 3rd layer to leave the network capacity for the harder distance estimation task. We also introduce a skip connection of the input 3D ray  $\{\mathcal{P},\theta \}$  to the 4th layer to preserve low-level local geometry.
The loss functions of our conditional directed distance field network consist of depth term  $\mathcal{L}_D = \xi |\hat{D} - D|$ , visibility term  $\mathcal{L}_{\xi} = BCE(\hat{\xi}, \xi)$  and symmetry term  $\mathcal{L}_s = |\hat{D}_1 - \hat{D}_2|$ , with  $\hat{D}, \hat{\xi}$  being the predictions of  $D, \xi$  respectively. For symmetric objects, we randomly sample two bijection sets of 3D rays, where corresponding rays have identical origin on the reflective plane but with opposite directions. Specifically, before our experiments, all symmetric objects are preprocessed to be symmetric with respect to the XY plane  $\{X = 0, Y = 0\}$ . To determine whether the object is symmetric, we first flipping the sampled points  $P$  on the object surface w.r.t the XY plane, yielding  $P'$ . Then we compare the Chamfer Distance between the object surface and  $P'$ . If the distance lies below a threshold (1e-3), the object is considered symmetric. As for building two bijection sets  $B_1: \{P_1, \theta_1\}$  and  $B_2: \{P_2, \theta_2\}$ , we first randomly sample origins  $P_1: \{(x_1, y_1, z_1)\}$  and directions  $\theta_1: \{(\alpha_1, \beta_1, \gamma_1)\}$  to construct  $B_1$ . Then, we flip  $B_1$  with respect to the reflective plane to generate  $B_2$  by  $P_2: \{(x_1, y_1, -z_1)\}$ ,  $\theta_2: \{(\alpha_1, \beta_1, -\gamma_1)\}$ . Since the object is symmetric, the DDF values  $\hat{D}_1, \hat{D}_2$  of corresponding rays in  $B_1$  and  $B_2$  should be the same, which establishes our symmetry loss term. The process of constructing the symmetry loss is shown in Fig. 4.
The final loss is defined as  $\mathcal{L} = \mathcal{L}_{\xi} + \lambda_1\mathcal{L}_D + \lambda_2\mathcal{L}_s$ , where  $\lambda_{1},\lambda_{2}$  are weighting factors.