SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Rong Wang Wei Mao Hongdong Li
{rong.wang,wei.mao,hongdong.li}@anu.edu.au
This paper addresses the task of 3D pose estimation for a hand interacting with an object from a single image observation. When modeling hand-object interaction, previous works mainly exploit proximity cues, while overlooking the dynamical nature that the hand must stably grasp the object to counteract gravity and thus preventing the object from slipping or falling. These works fail to leverage dynamical constraints in the estimation and consequently often produce unstable results. Meanwhile, refining unstable configurations with physics-based reasoning remains challenging, both by the complexity of contact dynamics and by the lack of effective and efficient physics inference in the data-driven learning framework. To address both issues, we present DeepSimHO: a novel deep-learning pipeline that combines forward physics simulation and backward gradient approximation with a neural network. Specifically, for an initial hand-object pose estimated by a base network, we forward it to a physics simulator to evaluate its stability. However, due to non-smooth contact geometry and penetration, existing differentiable simulators can not provide reliable state gradient. To remedy this, we further introduce a deep network to learn the stability evaluation process from the simulator, while smoothly approximating its gradient and thus enabling effective back-propagation. Extensive experiments show that our method noticeably improves the stability of the estimation and achieves superior efficiency over test-time optimization. The code is available at https://github.com/rongakowang/DeepSimHO.
3D hand-object pose estimation from a single image facilities a wide range of applications, including extended reality (XR) [2], human-robot interaction [32], and animation [17]. To this end, it has drawn increasing attention in recent years [34, 52, 56, 20, 19, 5]. A key challenge in this task is to ensure the estimated pose conforms to real-world physics, i.e. the hand should make contact with the object without penetrating it, and more importantly, the contact can form a stable grasp to counteract gravity. In this work, we aim to estimate hand-object pose that is both accurate and stable.
To model hand-object interaction, related works [21, 5, 57, 18, 20] often impose distance-based attraction and repulsion constraints that encourage contact and penalize penetration. However, this approach only enforces the proximity and do not explicitly reason the dynamical effects of the contact. In consequence, it allows the hand to simply touch the object, which can not form a stable grasp. As illustrated in Figure 1, such contact can cause the object to slip or fall under gravity, therefore is physically unstable. Meanwhile, in other related tasks such as grasping synthesis [7, 50] and dexterous manipulation [43, 12], physics simulators [48, 8, 54] are commonly used to ensure stable grasping. However, as discussed in [50, 55], existing differentiable simulators supporting mesh-to-mesh contact often produce numerically unreliable state gradient due to the discontinuity in contact geometry [54] and penetration [48], therefore making them difficult to integrate into the
learning framework. Alternatively, [51] perform a brute-force search over predefined poses, however, the approach is time-consuming and only constrained to a limited configuration space.
(b) GT
(c) [34]
(d) Ours
(e) [34] Rotated
(f) Ours Rotated
(g) [34] Simulated
Figure 1: Illustration of our method. Given an input image (a), recent work [34] can estimate accurate hand-object pose that is close to the ground truth (b) in the camera view (c). However, due to occlusion ambiguity and lack of explicit dynamics constraints modeling in hand-objection interaction, they can produce physically unstable results, where the hand only touches the object at one side and thus can not form a stable grasp (e). Such contact can cause the object to fall under gravity (g). In contrast, our result is both accurate (d) and physically stable, as in (f) and (h).
(h) Ours Simulated
In this paper, we propose an effective and efficient method DeepSimHO for estimating accurate and stable hand-object pose, by combining physics simulation in the forward step and gradient approximation with a deep network in the backward step. Specifically, for an initial hand-object pose estimated from a base network, we leverage a physics simulator to evaluate its stability, which is measured by the object center displacement due to gravity and hand contact. As shown in Figure 1, optimally stable hand-object pose results in minimal displacement after simulation. However, due to the aforementioned noisy gradient issues, the evaluated stability can not be directly used as a loss to refine the base network via back-propagation. To address this issue, we further introduce a neural network named DeepSim to reproduce the stability evaluation and back-propagate through it. On one hand, DeepSim acts as a smooth approximation of the state gradient from the simulator. On the other hand, it learns from the simulator to determine the stability of the initial pose, akin to the discriminator in the Generative Adversarial Network (GAN) [16]. However, as there is no adversarial relationship between the base network and DeepSim, the training process is much simpler. In training, we jointly refine the base network with DeepSim in an alternating order similar to [16]. In testing, we can estimate physically more stable poses by a simple forward call of the refined base network, contrasting with [56, 51, 20] that require a computationally expensive test-time optimization.
Our contributions can be summarized as follows. (i) We propose a novel pipeline that connects a learning framework with a physics simulator to estimate accurate and stable hand-object pose. (ii) We introduce DeepSim, a deep network that learns from simulator for stability evaluation and smooth gradient approximation. Extensive experiments on multiple benchmarks show that our method noticeably improves the stability of the estimation while maintaining comparable accuracy.
Hand-Object Pose Estimation. The problem of estimating hand and object poses from monocular images is ill-conditioned due to depth and occlusion ambiguity. To enforce physically plausible results, related works have employed three categories of approaches to model constraints in contact: data-driven [34, 19, 10, 52], distance-based [21, 20, 5, 18] and physics-based refinement [56, 26]. The data-driven approach leverages real [18, 6] or synthetic [34, 21] data to generate natural contact as a result of accurate estimation. Contact correlations can also be implicitly learned from the data using
the attention [19, 49, 52] or graph convolution [10]. However, this approach is sensitive to noise in the annotation, and perfect estimation is difficult to achieve in practice. In contrast, [21, 20, 5] explicitly employ attraction and repulsion constraints using the signed distance field (SDF) to encourage contact and penalize inter-penetration. However, such constraint only enforces the hand and object vertices to be in proximity, regardless of whether the hand can stably grasp the object or not. While additional affinity labels like [3, 27] can mitigate this issue, they require significantly more efforts to annotate.
Recently, physics-based refinement have been widely adopted in many tasks, including 3D shape modeling [36, 37], human motion estimation [13], and particular those for modeling hand-object interaction, such as task-oriented dexterous manipulation [43, 12] and grasp synthesis [7, 50]. In these works, a sequence of control forces is optimized to ensure a stable grasp on the target object, which is typically achieved via deep reinforcement learning. Motivated by their success, several works [51, 56, 26] have applied physics-based refinement for hand-object pose estimation. [51] query in a physics simulator and search for local configurations that are stable in a brute-force fashion. [56] model the contact as a virtual spring-mass system and minimize the elastic energy. [26] compute the acceleration from the initial kinematic estimation over multiple frames, then optimize for the contact force that aligns best with the regressed acceleration. However, all above approaches rely on computationally expensive post-fitting processes and can only handle a limited configuration space, defined by candidate finger parts [51, 26] or anchors [56].
Physics Simulation. Physics simulators for modeling contact dynamics between rigid bodies have been widely used in robotics [43, 12, 31, 30]. Recently, differentiable simulators [38, 40, 39] that can compute the state gradient, i.e. the derivative of the simulated state with respect to the input state, have drawn increasing attention. However, they suffer intrinsic issues of unstable numerical gradient caused by the non-smoothness in the linear complementary problem (LCP) [55, 50]. In practice, some also have restrictions in applications, such as being limited to primitive body shapes like spheres or cubes [25, 14, 9], which prevent them from being generally applied. Meanwhile, several works [23, 33, 46, 41] have introduced deep neural networks in physics simulations, primarily aimed to reduce the sim-to-real gap [23]. Specifically, [46, 41] adopt a graph convolution network to learn to predict physics states as a replacement or refinement of the simulator, while [23, 33] leverage a hybrid framework that regresses underlying physics quantities for fine-grained control. However, these works require training on a large amount of real-world physics data and thus can not handle noisy configurations, e.g. when the hand penetrates the object, since they do not exist in real-world data. In contrast, we introduce the neural network to learn from the simulator instead of real-world data, therefore can utilize rich training data obtained during simulation and are capable of analyzing noisy configurations. More importantly, all above works attempt to regress high-dimension physics quantities or states at each simulation step, while we only regress a scalar stability loss, which effectively reduces the difficulty of regression and enjoys superior generalizability.
In this section, we present the proposed pipeline as shown in Figure 2. Given an input image  $\mathbf{I} \in \mathbb{R}^{H \times W \times 3}$ , we first adopt a base hand-object pose estimation network (base network)  $f_{b}(\cdot)$  to estimate the initial hand mesh  $\hat{\mathbf{M}}^{h} \in \mathbb{R}^{778 \times 3}$  and rigid object pose  $\hat{\mathbf{p}}^{o} = [\hat{\mathbf{R}}^{o}, \hat{\mathbf{t}}^{o}] \in SE(3)$  (Section 3.1). Next, we collect the initial configuration  $\mathbf{q}_{0} = [\hat{\mathbf{M}}^{h}, \hat{\mathbf{p}}^{o}]$  and query its stability from a simulator  $f_{s}(\cdot)$  (Section 3.2). To effectively refine the base network, we further introduce the DeepSim network  $f_{d}(\cdot)$  to learn from the simulator and smoothly approximate its gradient (Section 3.3). Finally, we describe the overall training objectives (Section 3.4) to train the proposed model.
3.1 Base Hand-Object Pose Estimation Network
Hand Pose Estimation. Motivated by previous works [34, 59], we adopt a two-stage approach to estimate the hand mesh  $\hat{\mathbf{M}}^h$ . In particular, at the first stage, we follow the architecture in [47] to estimate the hand joint locations using heatmaps. Given the input image, we first adopt a ResNet-34 [22] encoder pre-trained on the ImageNet [45] to extract image features. Next, we decode the image feature with deconvolution layers and obtain  $K$  hand heatmaps  $\hat{\mathbf{H}}^h \in \mathbb{R}^{K \times D \times D \times D}$  after the softmax normalization. Each heatmap  $\hat{\mathbf{H}}_k^h \in \mathbb{R}^{D \times D \times D}$  represents the 2.5D location distribution of the  $k$ -th joint in the discretized pixel and depth spaces with  $D$  partitions. The expected value  $[u_k, v_k, z_k] \in [0, H] \times [0, W] \times [-R, R]$  of such distribution determines the position of this joint.
Figure 2: Overview of our method. Given an input image  $\mathbf{I}$ , we aim to enhance the stability of estimated poses from a base network  $f_{b}(\cdot)$  by learning from physics simulation. Specifically, we forward the initial configuration  $\mathbf{q}_0$  consisting of the estimated hand mesh  $\hat{\mathbf{M}}^h$  and object pose  $\hat{\mathbf{p}}^o$  to a physics simulator  $f_{s}(\cdot)$ , and evaluate the stability loss as the discrepancy between the simulated configuration  $\mathbf{q}_T$ . We further use another network named DeepSim  $f_{d}(\cdot)$  to replicate the stability loss value from the same initial configuration, so that the gradient can be effectively back-propagated through it. In testing, only the refined  $f_{b}(\cdot)$  is needed to generate more stable poses  $\tilde{\mathbf{M}}^h$  and  $\tilde{\mathbf{p}}^o$ .
$R > 0$  represents the depth radius relative to the hand root joint and is pre-calculated from the training data following [52]. From the  $uvz$  joint positions, we then recover their Euclidean coordinates  $\hat{\mathbf{J}}^h\in \mathbb{R}^{K\times 3}$  in the camera frame using the given camera intrinsic  $\mathbf{K}$ . At the second stage, we employ a pretrained inverse kinematics (IK) network [35] to regress the MANO [44] parameters  $\pmb{\theta}$  and  $\beta$  from the estimated joint positions. The pose parameter  $\pmb{\theta}\in \mathbb{R}^{16\times 3}$  represents the joint rotations along the hand kinematic tree, including the global rotation. The shape parameter  $\beta \in \mathbb{R}^{10}$  represents the hand shape in PCA components. The final hand mesh vertices positions  $\hat{\mathbf{M}}^h$  can be differentiably reconstructed from the MANO layer [21].
Object Pose Estimation. We follow previous works [52, 34] to assume a known object mesh template  $\mathbf{M}^o\in \mathbb{R}^{1000\times 3}$  in the canonical frame, and predict the rigid object pose  $\hat{\mathbf{p}}^o$  that transforms the template into the camera frame. Using the same image feature shared with the hand branch, we adopt a multi-layer perceptron (MLP) to regress the 6D representation [58] of the object rotation  $\hat{\mathbf{R}}^o$ , and estimate the object center translation  $\hat{\mathbf{t}}^o$  relative to the hand root joint in a separate heatmap head. It is worth noting that the simulation-based refinement described in following sections is not limited to the above base network and makes no assumption on its architecture.
3.2 Physics Simulation and Stability Loss
Physics Simulation. Given the initial configuration  $\mathbf{q}_0 = [\hat{\mathbf{M}}^h, \hat{\mathbf{p}}^o]$ , we wish to analyze whether it can form a stable grasp or not. To this end, we leverage a physics simulator to evaluate the pose stability considering the dynamical effects of gravity, penetration and contact. Specifically, the physics simulator  $f_s(\cdot)$  can be considered as a function that advances the current state, including the configuration  $\mathbf{q}_t$  and velocity  $\dot{\mathbf{q}}_t$  at time step  $t$  as,
$$
f _ {s} \left(\mathbf {q} _ {t}, \dot {\mathbf {q}} _ {t}\right) = \left[ \mathbf {q} _ {t + 1}, \dot {\mathbf {q}} _ {t + 1} \right], \quad \dot {\mathbf {q}} _ {0} = \mathbf {0}, \quad t = 0, 1, \dots T - 1. \tag {1}
$$
The state transition is governed by the Lagrangian dynamics equation [1, 48] with a semi-implicit Euler integration scheme [48], which we formulate for our task as
$$
\begin{array}{l} \mathbf {M} (\mathbf {q} _ {t}) \dot {\mathbf {q}} _ {t + 1} = \mathbf {M} (\mathbf {q} _ {t}) \dot {\mathbf {q}} _ {t} + \mathbf {c} (\mathbf {q} _ {t}, \dot {\mathbf {q}} _ {t}) \Delta t + \mathbf {J} _ {C} (\mathbf {q} _ {t}) ^ {T} \left(\boldsymbol {f} _ {C} (\mathbf {q} _ {t}, \dot {\mathbf {q}} _ {t}) + \boldsymbol {f} _ {A} (\mathbf {q} _ {t})\right) \Delta t \\ \mathbf {q} _ {t + 1} = \mathbf {q} _ {t} + \Delta t \dot {\mathbf {q}} _ {t + 1}, \tag {2} \\ \end{array}
$$
where  $\mathbf{M}$  represents the hand-object inertia matrix,  $\mathbf{c}$  represents the bias forces including the gravitational and Coriolis force, and  $\Delta t$  represents the time duration of each step. The hand-object interaction produces the contact force  $f_{C}$  expressed in the contact frame, which consists of the normal force to separate inter-penetrating bodies and the frictional force to resist relative siding. In
addition, human imposes torques on finger joints to counteract gravity when grasping the object. However, as our goal is to refine the stability for the initial hand pose, we require the hand remains static during simulation and thus joint torques are not applicable as it can change the hand pose. To model a similar effect we alternatively introduce an adhesion force  $f_{A}$  in the contact normal direction to maintain attraction to the object. In this way, stable contact can be established without biasing towards deeper penetration as in [56]. We empirically adjust the quantity of  $f_{A}$  to ensure a simple touch is not sufficient to maintain the stability, as shown in Figure 1. Finally, both  $f_{C}$  and  $f_{A}$  are mapped to the generalized coordinate by the contact Jacobian  $\mathbf{J}_C(\mathbf{q}_t)$ . The overall dynamics equation can be solved numerically via the steps shown in Figure 2. In the following section we describe the inference of stability based on the simulation results.
Stability Loss. Motivated by [7], we impose a stability loss as the object center displacement after a period of simulation time  $T$  as
$$
\mathcal {L} _ {s} = \left\| \mathbf {t} _ {T} - \mathbf {t} _ {0} \right\| _ {2}, \quad \mathbf {t} _ {T} \in \mathbf {q} _ {T} \quad \mathbf {t} _ {0} = \hat {\mathbf {t}} ^ {o} \in \mathbf {q} _ {0}. \tag {3}
$$
In this formulation, we assume a perfectly stable estimation enables the hand to stably grasp the object, avoiding (i) free fall due to being out-of-contact, (ii) slipping due to unrealistic interaction, e.g. with fingers only touching to the object at one side, and (iii) recoil due to penetration resolving by the normal force. All above implausible configurations result in a large displacement and thus are unstable. Our goal is therefore to minimize the stability loss for the initial configuration  $\mathbf{q}_0$ , which can be achieved by refining the base network via back-propagation.
However, existing differentiable simulators can not provide reliable gradient for computing  $\partial \mathcal{L}_s / \partial \mathbf{q}_0$ , making the back-propagation intractable through the simulator. First, the contact normal of object meshes are discontinuous at corners of triangle faces, thus the analytical gradient of the normal force can be undefined at these singular points [50]. Second, simulators like [48] often apply a large normal force to quickly resolve the penetration, which results in a large gradient for small pose perturbation in numerical gradient methods. Consequently, direct gradient calculation can be noisy and does not benefit the refinement, as will be shown in Figure 4. To address this issue, we propose the DeepSim network which acts as a smooth approximation of the gradient as described in the following section.
3.3 DeepSim Network
To smoothly approximate the stability evaluation process, we introduce a differentiable function that learns to produce the same stability loss as the simulator will given any initial configuration. To this end, we parameterize such function with a MLP named DeepSim. Specifically, since the simulator solves for the contact force based on the contact points, we first compute two feature vectors  $\hat{\mathbf{c}}^h\in \mathbb{R}^{778}$  and  $\hat{\mathbf{c}}^o\in \mathbb{R}^{1000}$  from the initial configuration, which represent the signed distances of hand vertices to the object mesh and object vertices to the hand mesh respectively. The two vectors jointly provide rich local contact information to facilitate the stability regression. We then concatenate the two vectors with the flattened initial configuration as the input for the DeepSim  $f_{d}(\cdot)$  and regress the replicated stability loss  $\hat{\mathcal{L}}_s$  as
$$
\hat {\mathcal {L}} _ {s} = f _ {d} \left(\hat {\mathbf {M}} ^ {h} \oplus \left(\hat {\mathbf {R}} ^ {o} \mathbf {M} ^ {o} + \hat {\mathbf {t}} ^ {o}\right) \oplus \hat {\mathbf {c}} ^ {h} \oplus \hat {\mathbf {c}} ^ {o}\right), \tag {4}
$$
where  $\oplus$  represents concatenation. The objective of the DeepSim network is therefore to minimize the approximation loss as
$$
\mathcal {L} _ {a} = \left\| \hat {\mathcal {L}} _ {s} - \mathcal {L} _ {s} \right\| _ {2}. \tag {5}
$$
In contrast to [46, 41] that regress all components in the simulated state at each step, we directly predict the final stability loss as a scalar, which is empirically an easier objective and better generalizes to noisy unstable configurations, as will be compared in Section 4.5.
Joint Training and Refinement. Similar to the GAN [16], we jointly train the base network and DeepSim in an alternating order for an effective refinement. When updating the DeepSim, we use the approximation loss as in Eq.(5) while fixing the base network weights. As the training progresses, the base network produces various configurations with corresponding stability losses automatically labelled by the simulator, which serve as rich training pairs for the DeepSim. In addition, to avoid the DeepSim overfitting to the base network outputs, we perform random perturbations in the estimated hand and object poses as data augmentation. We further use ground truth poses to train the DeepSim in order to include more data other than the base network output. Meanwhile, when updating the base
network, DeepSim facilities to back-propagate the smoothed gradient with respect to the replicated stability loss, with its own weights fixed. Note that we freeze weights in the shared image encoder when refining with the stability loss to avoid corruption of image features. Finally, to avoid being misled by an incorrect regression in the DeepSim, we mask out the replicated stability loss on training samples where  $\hat{\mathcal{L}}_s$  and  $\mathcal{L}_s$  do not fall in the same range. More details are discussed in the supplementary materials.
3.4 Training Objectives
To ensure the estimated poses are both stable and accurate, we impose a multi-task training objective for the base network, which contains two components: the replicated stability loss  $\hat{\mathcal{L}}_s$  defined in Eq.(4) and accuracy losses as described in the following. Specifically, we first adopt an  $L_{2}$  loss to supervise the hand joint and mesh estimation as
$$
\mathcal {L} _ {h} = \left| \left| \hat {\mathbf {J}} ^ {h} - \mathbf {J} ^ {h} \right| \right| + \left| \left| \hat {\mathbf {M}} ^ {h} - \mathbf {M} ^ {h} \right| \right|, \tag {6}
$$
where  $\mathbf{J}^h$  and  $\mathbf{M}^h$  represent ground truth hand joints and mesh vertices positions respectively. Next, we impose an object loss to supervise the object pose estimation as
$$
\mathcal {L} _ {o _ {1}} = \left\| \left(\hat {\mathbf {R}} ^ {o} \boldsymbol {c} ^ {o} + \hat {\mathbf {t}} ^ {o}\right) - \left(\mathbf {R} ^ {o} \boldsymbol {c} ^ {o} + \mathbf {t} ^ {o}\right) \right\|, \tag {7}
$$
where  $c^o$  represents the 8 tightest object corners positions in the canonical frame, and  $[\mathbf{R}^o, \mathbf{t}^o]$  represents the ground truth object pose. To account for the object rotation symmetry, we follow [19] to impose a symmetry-aware object loss as
$$
\mathcal {L} _ {o _ {2}} = \min  _ {\mathbf {R} ^ {o} \in \mathcal {S}} | | (\hat {\mathbf {R}} ^ {o} \boldsymbol {c} ^ {o} + \hat {\boldsymbol {t}} ^ {o}) - (\mathbf {R} ^ {o} \boldsymbol {c} ^ {o} + \boldsymbol {t} ^ {o}) | |, \tag {8}
$$
where  $\mathcal{S}$  represents the set of all equivalent rotation matrices defined by the objects' symmetry axes [4]. Furthermore, we borrow from [34] to impose an ordinal loss  $\mathcal{L}_d$  that supervises for the coarse hand-object relative position. The overall loss is a weighted sum of all individual loss functions as