Title: DeepSimHO: Stable Pose Estimation for Hand-Object Interaction via Physics Simulation
Venue: NeurIPS 2023
Accepted: yes
Keywords: hand-object pose estimation

physics simulation
Abstract: This paper addresses the task of 3D pose estimation for a hand interacting with an object from a single image observation. When modeling hand-object interaction, previous works mainly exploit proximity cues, while overlooking the dynamical nature that the hand must stably grasp the object to counteract gravity and thus preventing the object from slipping or falling. These works fail to leverage dynamical constraints in the estimation and consequently often produce unstable results. Meanwhile, refining unstable configurations with physics-based reasoning remains challenging, both by the complexity of contact dynamics and by the lack of effective and efficient physics inference in the data-driven learning framework. To address both issues, we present DeepSimHO: a novel deep-learning pipeline that combines forward physics simulation and backward gradient approximation with a neural network. Specifically, for an initial hand-object pose estimated by a base network, we forward it to a physics simulator to evaluate its stability. However, due to non-smooth contact geometry and penetration, existing differentiable simulators can not provide reliable state gradient. To remedy this, we further introduce a deep network to learn the stability evaluation process from the simulator, while smoothly approximating its gradient and thus enabling effective back-propagation. Extensive experiments show that our method noticeably improves the stability of the estimation and achieves superior efficiency over test-time optimization. The code is available at https://github.com/rongakowang/DeepSimHO.

DeepSimHO: Stable Pose Estimation for Hand-Object Interaction via Physics Simulation

Abstract

This paper addresses the task of 3D pose estimation for a hand interacting with an object from a single image observation. When modeling hand-object interaction, previous works mainly exploit proximity cues, while overlooking the dynamical nature that the hand must stably grasp the object to counteract gravity and thus preventing the object from slipping or falling. These works fail to leverage dynamical constraints in the estimation and consequently often produce unstable results. Meanwhile, refining unstable configurations with physics-based reasoning remains challenging, both by the complexity of contact dynamics and by the lack of effective and efficient physics inference in the data-driven learning framework. To address both issues, we present DeepSimHO : a novel deep-learning pipeline that combines forward physics simulation and backward gradient approximation with a neural network. Specifically, for an initial hand-object pose estimated by a base network, we forward it to a physics simulator to evaluate its stability. However, due to non-smooth contact geometry and penetration, existing differentiable simulators can not provide reliable state gradient. To remedy this, we further introduce a deep network to learn the stability evaluation process from the simulator, while smoothly approximating its gradient and thus enabling effective back-propagation. Extensive experiments show that our method noticeably improves the stability of the estimation and achieves superior efficiency over test-time optimization. The code is available at https://github.com/rongakowang/DeepSimHO .

1 Introduction

3D hand-object pose estimation from a single image facilities a wide range of applications, including extended reality (XR) [2], human-robot interaction [32], and animation [17]. To this end, it has drawn increasing attention in recent years [34, 52, 56, 20, 19, 5]. A key challenge in this task is to ensure the estimated pose conforms to real-world physics, i.e. the hand should make contact with the object without penetrating it, and more importantly, the contact can form a stable grasp to counteract gravity. In this work, we aim to estimate hand-object pose that is both accurate and stable.

To model hand-object interaction, related works [21, 5, 57, 18, 20] often impose distance-based attraction and repulsion constraints that encourage contact and penalize penetration. However, this approach only enforces the proximity and do not explicitly reason the dynamical effects of the contact. In consequence, it allows the hand to simply touch the object, which can not form a stable grasp. As illustrated in Figure 1, such contact can cause the object to slip or fall under gravity, therefore is physically unstable. Meanwhile, in other related tasks such as grasping synthesis [7, 50] and dexterous manipulation [43, 12], physics simulators [48, 8, 54] are commonly used to ensure stable grasping. However, as discussed in [50, 55], existing differentiable simulators supporting mesh-to-mesh contact often produce numerically unreliable state gradient due to the discontinuity in contact geometry [54] and penetration [48], therefore making them difficult to integrate into the

learning framework. Alternatively, [51] perform a brute-force search over predefined poses, however, the approach is time-consuming and only constrained to a limited configuration space.

(a) Input

(d) Ours

(c) [34]

(b) GT

(e) [34] Rotated

(f) Ours Rotated

(g) [34] Simulated

(h) Ours Simulated

Domino

Domino

In this paper, we propose an effective and efficient method DeepSimHO for estimating accurate and stable hand-object pose, by combining physics simulation in the forward step and gradient approximation with a deep network in the backward step. Specifically, for an initial hand-object pose estimated from a base network, we leverage a physics simulator to evaluate its stability , which is measured by the object center displacement due to gravity and hand contact. As shown in Figure 1, optimally stable hand-object pose results in minimal displacement after simulation. However, due to the aforementioned noisy gradient issues, the evaluated stability can not be directly used as a loss to refine the base network via back-propagation. To address this issue, we further introduce a neural network named DeepSim to reproduce the stability evaluation and back-propagate through it. On one hand, DeepSim acts as a smooth approximation of the state gradient from the simulator. On the other hand, it learns from the simulator to determine the stability of the initial pose, akin to the discriminator in the Generative Adversarial Network (GAN) [16]. However, as there is no adversarial relationship between the base network and DeepSim, the training process is much simpler. In training, we jointly refine the base network with DeepSim in an alternating order similar to [16]. In testing, we can estimate physically more stable poses by a simple forward call of the refined base network, contrasting with [56, 51, 20] that require a computationally expensive test-time optimization.

Our contributions can be summarized as follows. ( i ) We propose a novel pipeline that connects a learning framework with a physics simulator to estimate accurate and stable hand-object pose. ( ii ) We introduce DeepSim, a deep network that learns from simulator for stability evaluation and smooth gradient approximation. Extensive experiments on multiple benchmarks show that our method noticeably improves the stability of the estimation while maintaining comparable accuracy.

2 Related Works

Hand-Object Pose Estimation. The problem of estimating hand and object poses from monocular images is ill-conditioned due to depth and occlusion ambiguity. To enforce physically plausible results, related works have employed three categories of approaches to model constraints in contact: data-driven [34, 19, 10, 52], distance-based [21, 20, 5, 18] and physics-based refinement [56, 26]. The data-driven approach leverages real [18, 6] or synthetic [34, 21] data to generate natural contact as a result of accurate estimation. Contact correlations can also be implicitly learned from the data using

the attention [19, 49, 52] or graph convolution [10]. However, this approach is sensitive to noise in the annotation, and perfect estimation is difficult to achieve in practice. In contrast, [21, 20, 5] explicitly employ attraction and repulsion constraints using the signed distance field (SDF) to encourage contact and penalize inter-penetration. However, such constraint only enforces the hand and object vertices to be in proximity, regardless of whether the hand can stably grasp the object or not. While additional affinity labels like [3, 27] can mitigate this issue, they require significantly more efforts to annotate.

Recently, physics-based refinement have been widely adopted in many tasks, including 3D shape modeling [36, 37], human motion estimation [13], and particular those for modeling hand-object interaction, such as task-oriented dexterous manipulation [43, 12] and grasp synthesis [7, 50]. In these works, a sequence of control forces is optimized to ensure a stable grasp on the target object, which is typically achieved via deep reinforcement learning. Motivated by their success, several works [51, 56, 26] have applied physics-based refinement for hand-object pose estimation. [51] query in a physics simulator and search for local configurations that are stable in a brute-force fashion. [56] model the contact as a virtual spring-mass system and minimize the elastic energy. [26] compute the acceleration from the initial kinematic estimation over multiple frames, then optimize for the contact force that aligns best with the regressed acceleration. However, all above approaches rely on computationally expensive post-fitting processes and can only handle a limited configuration space, defined by candidate finger parts [51, 26] or anchors [56].

Physics Simulation. Physics simulators for modeling contact dynamics between rigid bodies have been widely used in robotics [43, 12, 31, 30]. Recently, differentiable simulators [38, 40, 39] that can compute the state gradient, i.e. the derivative of the simulated state with respect to the input state, have drawn increasing attention. However, they suffer intrinsic issues of unstable numerical gradient caused by the non-smoothness in the linear complementary problem (LCP) [55, 50]. In practice, some also have restrictions in applications, such as being limited to primitive body shapes like spheres or cubes [25, 14, 9], which prevent them from being generally applied. Meanwhile, several works [23, 33, 46, 41] have introduced deep neural networks in physics simulations, primarily aimed to reduce the sim-to-real gap [23]. Specifically, [46, 41] adopt a graph convolution network to learn to predict physics states as a replacement or refinement of the simulator, while [23, 33] leverage a hybrid framework that regresses underlying physics quantities for fine-grained control. However, these works require training on a large amount of real-world physics data and thus can not handle noisy configurations, e.g. when the hand penetrates the object, since they do not exist in real-world data. In contrast, we introduce the neural network to learn from the simulator instead of real-world data, therefore can utilize rich training data obtained during simulation and are capable of analyzing noisy configurations. More importantly, all above works attempt to regress high-dimension physics quantities or states at each simulation step, while we only regress a scalar stability loss, which effectively reduces the difficulty of regression and enjoys superior generalizability.

3 Approach

In this section, we present the proposed pipeline as shown in Figure 2. Given an input image I ∈ R H × W × 3 , we first adopt a base hand-object pose estimation network (base network) f b ( · ) to estimate the initial hand mesh ˆ M h ∈ R 778 × 3 and rigid object pose ˆ p o = [ ˆ R o , ˆ t o ] ∈ SE (3) (Section 3.1). Next, we collect the initial configuration q 0 = [ ˆ M h , ˆ p o ] and query its stability from a simulator f s ( · ) (Section 3.2). To effectively refine the base network, we further introduce the DeepSim network f d ( · ) to learn from the simulator and smoothly approximate its gradient (Section 3.3). Finally, we describe the overall training objectives (Section 3.4) to train the proposed model.

3.1 Base Hand-Object Pose Estimation Network

Hand Pose Estimation. Motivated by previous works [34, 59], we adopt a two-stage approach to estimate the hand mesh ˆ M h . In particular, at the first stage, we follow the architecture in [47] to estimate the hand joint locations using heatmaps. Given the input image, we first adopt a ResNet-34 [22] encoder pre-trained on the ImageNet [45] to extract image features. Next, we decode the image feature with deconvolution layers and obtain K hand heatmaps ˆ H h ∈ R K × D × D × D after the softmax normalization. Each heatmap ˆ H h k ∈ R D × D × D represents the 2.5D location distribution of the k -th joint in the discretized pixel and depth spaces with D partitions. The expected value [ u k , v k , z k ] ∈ [0 , H ] × [0 , W ] × [ -R,R ] of such distribution determines the position of this joint.

<latexi sh

1_b

64="7VzmyoZkcFCU

w

KD

W

AY

>

B9H

LS

N

2pr

f

ug

R

G

n

8

IO

Q

/

j

3P

v

J

5

q

+

M

E

X

Collision

Detection

and

Calculate

Jacobian

f

b

I

ˆ

L

s

q

T

L

s

|| · ||

f

d

ˆ

M

h

ˆ

p

o

q

0

Testing

⊕

ˆ

c

h

ˆ

c

o

˜

M

h

˜

p

o

Training

f

b

I

Compute

Velocity

Integration

J

C

Solve LCP

and Apply

Adhesion

Force

q

t

+1

f

C

f

A

˙

q

t

+1

Current

State and

Forward

Kinematic

˙

q

t

q

t

f

s

×

T

|| · ||

φ

L

a

L2 Distance

Back-Propagation

⊕

Concatenation

SDF Calculation

|| · ||

φ

Forward

R > 0 represents the depth radius relative to the hand root joint and is pre-calculated from the training data following [52]. From the uvz joint positions, we then recover their Euclidean coordinates ˆ J h ∈ R K × 3 in the camera frame using the given camera intrinsic K . At the second stage, we employ a pretrained inverse kinematics (IK) network [35] to regress the MANO [44] parameters θ and β from the estimated joint positions. The pose parameter θ ∈ R 16 × 3 represents the joint rotations along the hand kinematic tree, including the global rotation. The shape parameter β ∈ R 10 represents the hand shape in PCA components. The final hand mesh vertices positions ˆ M h can be differentiablely reconstructed from the MANO layer [21].

Object Pose Estimation. We follow previous works [52, 34] to assume a known object mesh template M o ∈ R 1000 × 3 in the canonical frame, and predict the rigid object pose ˆ p o that transforms the template into the camera frame. Using the same image feature shared with the hand branch, we adopt a multi-layer perceptron (MLP) to regress the 6D representation [58] of the object rotation ˆ R o , and estimate the object center translation ˆ t o relative to the hand root joint in a separate heatmap head. It is worth noting that the simulation-based refinement described in following sections is not limited to the above base network and makes no assumption on its architecture.

3.2 Physics Simulation and Stability Loss

Physics Simulation. Given the initial configuration q 0 = [ ˆ M h , ˆ p o ] , we wish to analyze whether it can form a stable grasp or not. To this end, we leverage a physics simulator to evaluate the pose stability considering the dynamical effects of gravity, penetration and contact. Specifically, the physics simulator f s ( · ) can be considered as a function that advances the current state, including the configuration q t and velocity ˙ q t at time step t as,

The state transition is governed by the Lagrangian dynamics equation [1, 48] with a semi-implicit Euler integration scheme [48], which we formulate for our task as

where M represents the hand-object inertia matrix, c represents the bias forces including the gravitational and Coriolis force, and ∆ t represents the time duration of each step. The hand-object interaction produces the contact force f C expressed in the contact frame, which consists of the normal force to separate inter-penetrating bodies and the frictional force to resist relative siding. In

addition, human imposes torques on finger joints to counteract gravity when grasping the object. However, as our goal is to refine the stability for the initial hand pose, we require the hand remains static during simulation and thus joint torques are not applicable as it can change the hand pose. To model a similar effect we alternatively introduce an adhesion force f A in the contact normal direction to maintain attraction to the object. In this way, stable contact can be established without biasing towards deeper penetration as in [56]. We empirically adjust the quantity of f A to ensure a simple touch is not sufficient to maintain the stability, as shown in Figure 1. Finally, both f C and f A are mapped to the generalized coordinate by the contact Jacobian J C ( q t ) . The overall dynamics equation can be solved numerically via the steps shown in Figure 2. In the following section we describe the inference of stability based on the simulation results.

Stability Loss. Motivated by [7], we impose a stability loss as the object center displacement after a period of simulation time T as

In this formulation, we assume a perfectly stable estimation enables the hand to stably grasp the object, avoiding ( i ) free fall due to being out-of-contact, ( ii ) slipping due to unrealistic interaction, e.g. with fingers only touching to the object at one side, and ( iii ) recoil due to penetration resolving by the normal force. All above implausible configurations result in a large displacement and thus are unstable. Our goal is therefore to minimize the stability loss for the initial configuration q 0 , which can be achieved by refining the base network via back-propagation.

However, existing differentiable simulators can not provide reliable gradient for computing ∂ L s /∂ q 0 , making the back-propagation intractable through the simulator. First, the contact normal of object meshes are discontinuous at corners of triangle faces, thus the analytical gradient of the normal force can be undefined at these singular points [50]. Second, simulators like [48] often apply a large normal force to quickly resolve the penetration, which results in a large gradient for small pose perturbation in numerical gradient methods. Consequently, direct gradient calculation can be noisy and does not benefit the refinement, as will be shown in Figure 4. To address this issue, we propose the DeepSim network which acts as a smooth approximation of the gradient as described in the following section.

3.3 DeepSim Network

To smoothly approximate the stability evaluation process, we introduce a differentiable function that learns to produce the same stability loss as the simulator will given any initial configuration. To this end, we parameterize such function with a MLP named DeepSim . Specifically, since the simulator solves for the contact force based on the contact points, we first compute two feature vectors ˆ c h ∈ R 778 and ˆ c o ∈ R 1000 from the initial configuration, which represent the signed distances of hand vertices to the object mesh and object vertices to the hand mesh respectively. The two vectors jointly provide rich local contact information to facilitate the stability regression. We then concatenate the two vectors with the flattened initial configuration as the input for the DeepSim f d ( · ) and regress the replicated stability loss ˆ L s as

where ⊕ represents concatenation. The objective of the DeepSim network is therefore to minimize the approximation loss as

In contrast to [46, 41] that regress all components in the simulated state at each step, we directly predict the final stability loss as a scalar, which is empirically an easier objective and better generalizes to noisy unstable configurations, as will be compared in Section 4.5.

Joint Training and Refinement. Similar to the GAN [16], we jointly train the base network and DeepSim in an alternating order for an effective refinement. When updating the DeepSim, we use the approximation loss as in Eq.(5) while fixing the base network weights. As the training progresses, the base network produces various configurations with corresponding stability losses automatically labelled by the simulator, which serve as rich training pairs for the DeepSim. In addition, to avoid the DeepSim overfitting to the base network outputs, we perform random perturbations in the estimated hand and object poses as data augmentation. We further use ground truth poses to train the DeepSim in order to include more data other than the base network output. Meanwhile, when updating the base

network, DeepSim facilities to back-propagate the smoothed gradient with respect to the replicated stability loss, with its own weights fixed. Note that we freeze weights in the shared image encoder when refining with the stability loss to avoid corruption of image features. Finally, to avoid being misled by an incorrect regression in the DeepSim, we mask out the replicated stability loss on training samples where ˆ L s and L s do not fall in the same range. More details are discussed in the supplementary materials.

3.4 Training Objectives

To ensure the estimated poses are both stable and accurate, we impose a multi-task training objective for the base network, which contains two components: the replicated stability loss ˆ L s defined in Eq.(4) and accuracy losses as described in the following. Specifically, we first adopt an L 2 loss to supervise the hand joint and mesh estimation as

where J h and M h represent ground truth hand joints and mesh vertices positions respectively. Next, we impose an object loss to supervise the object pose estimation as

where c o represents the 8 tightest object corners positions in the canonical frame, and [ R o , t o ] represents the ground truth object pose. To account for the object rotation symmetry, we follow [19] to impose a symmetry-aware object loss as

where S represents the set of all equivalent rotation matrices defined by the objects' symmetry axes [4]. Furthermore, we borrow from [34] to impose an ordinal loss L d that supervises for the coarse hand-object relative position. The overall loss is a weighted sum of all individual loss functions as

where λ h , λ o 1 , λ o 2 , λ d , λ s are hyper-parameters.

4 Experiments

4.1 Datasets

We evaluate our method and state-of-the-art methods on two datasets: DexYCB [6] and HO3D [18].

DexYCB. The DexYCB dataset [6] consists of 582K frames involving 20 different YCB objects [4]. We use the official "S0" train-test split for the training and evaluation. Following [52], we evaluate on right hand poses and filter out samples in which the hand or object is not within the field of view of the camera. To ensure consistent comparison in physics metrics, we remove test samples where the hand does not interact with the object and only select those that remain stable after simulation (see the GT results in Table 1), resulting in a total of 6348 samples. For training, we do not perform this selection of stability in order to include more data. However, we mask out the replicated stability loss on unstable training samples, e.g. no hand-object interaction, to avoid misleading supervision.

HO3D. The HO3D dataset [18] consists of 66K frames featuring 10 different objects. We select the "v2" version that is mostly evaluated by previous works [56, 20, 52, 34]. Since its ground truth hand poses for the test set are not released, we follow [56] to evaluate on a subset named "v2 -", whose physics plausibility is manually verified by [56]. For training data, we use the official HO3D v2 training split and follow the same practice as the DexYCB dataset to perform sample selection and loss masking. The total HO3Dv2 -test set consists of 6076 samples.

4.2 Metrics

Hand & Object Metrics. We evaluate the accuracy of pose estimation by comparing the hand and object results with the ground truth labels. For the hand metric, we calculate the mean joint error

(MJE) [60], which measures the Euclidean distance between the predicted and ground truth hand joint positions after root joint alignment. For the object metric,we compute the mean corner error (MCE) on the HO3D test set following [34], which calculates the average distance between the predicted and ground truth positions of the 8 tightest object corners. Since on the DexYCB dataset, many objects are rotationally symmetric, we thus borrow from [19] to compute the symmetry-aware mean corner error (SMCE) as defined in Eq.(8), for a fair comparison with [34].

Physics Metrics. We evaluate the physical plausibility of the estimation following the criterions in [7, 56, 20]. Specifically, we evaluate the simulation displacement (SD) [56] to compute the average object center displacement after 200ms simulation time. We then evaluate the success rate (SR) [7] to classify the prediction whose SD is below a threshold of 1cm as success. These two metrics are used to evaluate the pose stability. In addition, we evaluate the contact percentage (CP) [28] as the ratio of predictions with hand-object contact and the penetration depth (PD) [3] as the maximum penetration distance of contacting hand-object predictions. It is worth noting that a stable estimation, i.e. with a lower SD, generally indicates a higher CP and a lower PD, but the converse is not necessarily true.

4.3 Implementation Details

We implement the model in PyTorch [42] and train it using the Adam [29] optimizer on a single NVIDIA RTX 3090 GPU. We adopt the base network pretrained in [34], and independently train the DeepSim for 40 epochs as a warm-up. We then jointly train both networks enabling the stability loss for 25 epochs. We set the learning rate to 5 × 10 -5 . When training on both datasets, we follow [34, 52] to crop input images into 224 × 224 pixels using the provided hand-object bounding boxes. In addition, we follow [34] and perform data augmentation with random translation and rescaling by a factor of 0.1. However, we exclude rotation augmentation as it can affect the ground truth stability. Finally, we set λ h = 0 . 5 , λ d = 0 . 1 , λ s = 0 . 1 , and follow [34] to set λ o 1 = 0 , λ o 2 = 0 . 2 on the DexYCB dataset and λ o 1 = 0 . 2 , λ o 2 = 0 on the HO3D dataset for a fair comparison.

For physics simulation, we use the MuJoCo [48] simulator. We initialize the hand and object inertia matrix M using the Composite Rigid Body algorithm [11]. We further convexly decompose hand meshes by the MANO blending weights [44] and object meshes by the CoACD [53] algorithm to facilitate continuous collision detection [15] in MuJoCo. We set the gravity acceleration as 9.8 m/s 2 in the y direction of the camera frame. For the adhesion force, we empirically set the gain as 100 and the maximum control range as 10, so that a simple touch is not sufficient to stably grasp the object. Finally, we set the simulation step to be T = 100 and time duration in each step as ∆ t = 0 . 02 . More implementation and training details can be found in the supplementary materials.

4.4 Results

Quantitative results. We compare our method with three categories of state-of-the-art methods: data-driven [34, 52], distance-based [20] and physics-based [56] approaches. In Table 1, we report the comparison on the DexYCB test set and re-evaluate all baseline methods using the officially provided weights [34, 52] or optimization scripts [20, 56]. To avoid unfair comparison, we exclude the SMCE result for [52, 20] as they do not consider object symmetry in implementation. From the table, we observe that our method achieves superior performance on all physics metrics, while maintaining comparable accuracy using significantly less data than [34, 52] when trained with the stability loss. Our method also noticeably improves the stability compared to distance-based approach [20].

DexYCB

Ours

[34]

Camera View

Input

Ours

[56]

Camera View

Input

HO3D

In Table 2, we compare all methods on the HO3Dv2 -test set. Since the selected split is not suitable for submitting to the official HO3D online evaluation server, we follow the same evaluation protocol as [56] to not evaluate for the hand metric. In addition, although [56] also report the SD in their paper, they use a different simulator [8] and the code for this part is not publicly available. Hence we re-evaluate the same metric using our physics modeling for a fair comparison. Consistently, we observe our method noticeably improves physics plausibility and stability with comparable accuracy. Finally, for the testing, [56] and [20] requires 12 and 45 seconds to optimize for one input image on a single NVIDIA RTX 3090 GPU, in comparison, our method only requires 0.02 seconds on the same device for a forward call of the refined base network, which is significantly more efficient.

Qualitative results. We present qualitative results on the DexYCB and HO3D test sets in Figure 3. We render the refined hand and object meshes in the camera view, illustrating that the estimated poses align well with the input image. More importantly, we compare in rotated views with [34, 56] to show that while [34] produce visually close results, the established contact is actually physically unstable, with some fingers floating in the air or penetrating the object. In addition, [56] do not consider pose accuracy when optimize for physics plausibility, therefore can generate results that are stable but do not align with the input image. In comparison, our results are both accurate and physically stable.

4.5 Ablation Study

Effects of the DeepSim Network. To verify the effects of the DeepSim for back-propagation, we compare with two other approaches for calculating the state gradient: ( i ) numerical gradient using

the finite difference in the official MuJoCo implementation, and ( ii ) analytical gradient through the differentiable simulator NimblePhysics [54]. We use a similar physics model in the NimblePhysics based on its available functions (included in the supplementary materials). To avoid training failure, we include a gradient clip of 1.0 for all methods. In Figure 4 (b), we observe ( i ) produces incorrectly large gradient due to the sudden velocity change during penetration resolving. For ( ii ), we observe it

Stability Loss

(b) Calculated Gradient Norm

0

100

200

300

400

500

0

.

0400

0

.

0425

0

.

0450

0

.

0475

0

.

0500

0

.

0525

0

.

0550

0

.

0575

0

.

0600

Base

NimblePhysics

Finite Difference

DeepSim

0

100

200

300

400

500

-

10

0

10

20

30

40

NimblePhysics

Finite Difference

DeepSim

Log of Gradient Norm

Iterations

Iterations

Design Choices. In Table 3, we compare different variants of design in DeepSim and stability loss. We observe that regressing more state components, e.g. the 3 DoF. object center (MLP + T) or 6 DoF. object pose (MLP + RT) in the final simulation step, can be more challenging compared to regressing a scalar stability loss (MLP + S). As a result the DeepSim performs worse with a higher approximation error (AE), which measures the discrepancy between ˆ L s and L s . Underfitting in DeepSim can cause severe misleading, thus can not help to improve the stability compared to the baseline method that only trains with accuracy losses (w/o DeepSim). In addition, we replace the MLP with a long short-term memory network (LSTM) [24] (implementation details in the supplementary materials), which sequentially estimates residual center displacements in each step (LSTM + T), so that the overall stability loss is the norm of the total displacement. However, we observe that dividing the total displacement into multiple partitions can cause the DeepSim to overfit by predicting only small residuals, therefore does not outperform against directly regressing for the final simulation step.

5 Discussion

Limitations & Societal Impacts. Although DeepSim addresses the issues of simulator gradient in the back-propagation, our physics modeling and simulation in the forward step are still dependent on the simulator implementation, which can be imperfect when compared to real-world physics. In addition, we made several restrained assumptions about the physics properties and contact dynamics,

for instance we assumed rigid objects with known gravity direction, as well as simplified modeling of joint torques via the adhesion force. As a result, the potential sim-to-real gap may lead to unsafety when interacting with objects that have complex and unknown physics properties.

Future Works. The proposed pipeline can be benefited with improved designs of physics metrics, simulator and the DeepSim network. Future works are thus encouraged to explore on ( i ) stability losses with a more comprehensive consideration of the object state, e.g. rotation, velocity, ( ii ) more realistic simulation, especially for static hand, ( iii ) variant DeepSim networks that explicitly utilize physics priors for the regression, and ( iv ) extension to video inputs with known object initial states.

Conclusion. In this paper, we propose a novel pipeline DeepSimHO for estimating accurate and stable hand-object pose. We explicitly model the dynamical nature in hand-object interaction via forward physics simulation and stability evaluation. In the backward step, we introduce DeepSim, a neural network that learns the stability evaluation from the simulation and performs smooth gradient approximation to facilitate refinement of the base network. Thanks to them, we our method effectively and efficiently produces physically stable hand-object pose that conforms better to reality.

References

Sheldon Andrews, Kenny Erleben, and Zachary Ferguson. Contact and friction simulation for computer graphics. In ACM SIGGRAPH 2022 Courses , pages 1-172. 2022. 4

Mark Billinghurst, Hirokazu Kato, and Ivan Poupyrev. The magicbook: a transitional ar interface. Computers & Graphics , 25(5):745-753, 2001. 1

Samarth Brahmbhatt, Chengcheng Tang, Christopher D Twigg, Charles C Kemp, and James Hays. Contactpose: A dataset of grasps with object contact and hand pose. In European Conference on Computer Vision , pages 361-378. Springer, 2020. 3, 7

Berk Calli, Arjun Singh, Aaron Walsman, Siddhartha Srinivasa, Pieter Abbeel, and Aaron M Dollar. The ycb object and model set: Towards common benchmarks for manipulation research. In 2015 international conference on advanced robotics (ICAR) , pages 510-517. IEEE, 2015. 6

Zhe Cao, Ilija Radosavovic, Angjoo Kanazawa, and Jitendra Malik. Reconstructing hand-object interactions in the wild. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1241712426, 2021. 1, 2, 3

Yu-Wei Chao, Wei Yang, Yu Xiang, Pavlo Molchanov, Ankur Handa, Jonathan Tremblay, Yashraj S Narang, Karl Van Wyk, Umar Iqbal, Stan Birchfield, et al. Dexycb: A benchmark for capturing hand grasping of objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9044-9053, 2021. 2, 6

Sammy Christen, Muhammed Kocabas, Emre Aksan, Jemin Hwangbo, Jie Song, and Otmar Hilliges. D-grasp: Physically plausible dynamic grasp synthesis for hand-object interactions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2022. 1, 3, 5, 7

Erwin Coumans and Yunfei Bai. Pybullet, a python module for physics simulation for games, robotics and machine learning. http://pybullet.org , 2016-2021. 1, 8

Jonas Degrave, Michiel Hermans, Joni Dambre, et al. A differentiable physics engine for deep learning in robotics. Frontiers in neurorobotics , page 6, 2019. 3

Bardia Doosti, Shujon Naha, Majid Mirbagheri, and David J Crandall. Hope-net: A graph-based model for hand-object pose estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 6608-6617, 2020. 2, 3

Roy Featherstone. Robot dynamics algorithms. Annexe Thesis Digitisation Project 2016 Block 5 , 1984. 7

Guillermo Garcia-Hernando, Edward Johns, and Tae-Kyun Kim. Physics-based dexterous manipulations with estimated hand poses and residual reinforcement learning. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 9561-9568. IEEE, 2020. 1, 3

Erik Gärtner, Mykhaylo Andriluka, Erwin Coumans, and Cristian Sminchisescu. Differentiable dynamics for articulated 3d human motion reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13190-13200, 2022. 3

Moritz Geilinger, David Hahn, Jonas Zehnder, Moritz Bächer, Bernhard Thomaszewski, and Stelian Coros. Add: Analytically differentiable dynamics for multi-body systems with frictional contact. ACM Transactions on Graphics (TOG) , 39(6):1-15, 2020. 3

Elmer G Gilbert, Daniel W Johnson, and S Sathiya Keerthi. A fast procedure for computing the distance between complex objects in three-dimensional space. IEEE Journal on Robotics and Automation , 4(2):193203, 1988. 7

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM , 63(11):139144, 2020. 2, 5

Henning Hamer, Juergen Gall, Raquel Urtasun, and Luc Van Gool. Data-driven animation of hand-object interactions. In 2011 IEEE International Conference on Automatic Face & Gesture Recognition (FG) , pages 360-367. IEEE, 2011. 1

Shreyas Hampali, Mahdi Rad, Markus Oberweger, and Vincent Lepetit. Honnotate: A method for 3d annotation of hand and object poses. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 3196-3206, 2020. 1, 2, 6

Shreyas Hampali, Sayan Deb Sarkar, Mahdi Rad, and Vincent Lepetit. Keypoint transformer: Solving joint identification in challenging hands and object interactions for accurate 3d pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11090-11100, 2022. 1, 2, 3, 6, 7

Yana Hasson, Gül Varol, Cordelia Schmid, and Ivan Laptev. Towards unconstrained joint hand-object reconstruction from rgb videos. In 2021 International Conference on 3D Vision (3DV) , pages 659-668. IEEE, 2021. 1, 2, 3, 6, 7, 8

Yana Hasson, Gul Varol, Dimitrios Tzionas, Igor Kalevatykh, Michael J Black, Ivan Laptev, and Cordelia Schmid. Learning joint reconstruction of hands and manipulated objects. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 11807-11816, 2019. 1, 2, 3, 4

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016. 3

Eric Heiden, David Millard, Erwin Coumans, Yizhou Sheng, and Gaurav S Sukhatme. Neuralsim: Augmenting differentiable simulators with neural networks. In 2021 IEEE International Conference on Robotics and Automation (ICRA) , pages 9474-9481. IEEE, 2021. 3

Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation , 9(8):1735-1780, 1997. 9

Taylor A Howell, Simon Le Cleac'h, Jan Brüdigam, J Zico Kolter, Mac Schwager, and Zachary Manchester. Dojo: A differentiable physics engine for robotics. arXiv preprint arXiv:2203.00806 , 2022. 3

Haoyu Hu, Xinyu Yi, Hao Zhang, Jun-Hai Yong, and Feng Xu. Physical interaction: Reconstructing hand-object interactions with physics. In SIGGRAPH Asia 2022 Conference Papers , pages 1-9, 2022. 2, 3

Hanwen Jiang, Shaowei Liu, Jiashun Wang, and Xiaolong Wang. Hand-object contact consistency reasoning for human grasps generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 11107-11116, 2021. 3

Korrawe Karunratanakul, Jinlong Yang, Yan Zhang, Michael J Black, Krikamol Muandet, and Siyu Tang. Grasping field: Learning implicit representations for human grasps. In 2020 International Conference on 3D Vision (3DV) , pages 333-344. IEEE, 2020. 7

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014. 7

Mia Kokic, Danica Kragic, and Jeannette Bohg. Learning to estimate pose and shape of hand-held objects from rgb images. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 3980-3987. IEEE, 2019. 3

Mia Kokic, Danica Kragic, and Jeannette Bohg. Learning task-oriented grasping from human activity datasets. IEEE Robotics and Automation Letters , 5(2):3352-3359, 2020. 3

Hema Swetha Koppula and Ashutosh Saxena. Anticipating human activities for reactive robotic response. In IROS , page 2071. Tokyo, 2013. 1

Simon Le Cleac'h, Hong-Xing Yu, Michelle Guo, Taylor Howell, Ruohan Gao, Jiajun Wu, Zachary Manchester, and Mac Schwager. Differentiable physics simulation of dynamics-augmented neural objects. IEEE Robotics and Automation Letters , 8(5):2780-2787, 2023. 3

Kailin Li, Lixin Yang, Xinyu Zhan, Jun Lv, Wenqiang Xu, Jiefeng Li, and Cewu Lu. Artiboost: Boosting articulated 3d hand-object pose estimation via online exploration and synthesis. arXiv preprint arXiv:2109.05488 , 2021. 1, 2, 3, 4, 6, 7, 8

Jun Lv, Wenqiang Xu, Lixin Yang, Sucheng Qian, Chongzhao Mao, and Cewu Lu. Handtailor: Towards high-precision monocular 3d hand recovery. arXiv preprint arXiv:2102.09244 , 2021. 4

Mariem Mezghanni, Théo Bodrito, Malika Boulkenafed, and Maks Ovsjanikov. Physical simulation layer for accurate 3d modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13514-13523, 2022. 3

Mariem Mezghanni, Malika Boulkenafed, Andre Lieutier, and Maks Ovsjanikov. Physically-aware generative network for 3d shape modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9330-9341, 2021. 3

Mattia Montanari, Nik Petrinic, and Ettore Barbieri. Improving the gjk algorithm for faster and more reliable distance queries between convex objects. ACM Transactions on Graphics (TOG) , 36(3):1-17, 2017. 3

Louis Montaut, Quentin Le Lidec, Antoine Bambade, Vladimir Petrik, Josef Sivic, and Justin Carpentier. Differentiable collision detection: a randomized smoothing approach. arXiv preprint arXiv:2209.09012 , 2022. 3

Louis Montaut, Quentin Le Lidec, Josef Sivic, and Justin Carpentier. Collision detection accelerated: An optimization perspective. arXiv preprint arXiv:2205.09663 , 2022. 3

Damian Mrowca, Chengxu Zhuang, Elias Wang, Nick Haber, Li F Fei-Fei, Josh Tenenbaum, and Daniel L Yamins. Flexible neural representation for physics prediction. Advances in neural information processing systems , 31, 2018. 3, 5

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32 , pages 8024-8035. Curran Associates, Inc., 2019. 7

Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint arXiv:1709.10087 , 2017. 1, 3

Javier Romero, Dimitrios Tzionas, and Michael J Black. Embodied hands: Modeling and capturing hands and bodies together. arXiv preprint arXiv:2201.02610 , 2022. 4, 7

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision , 115(3):211-252, 2015. 3

Alvaro Sanchez-Gonzalez, Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec, and Peter Battaglia. Learning to simulate complex physics with graph networks. In International conference on machine learning , pages 8459-8468. PMLR, 2020. 3, 5

Xiao Sun, Bin Xiao, Fangyin Wei, Shuang Liang, and Yichen Wei. Integral human pose regression. In Proceedings of the European conference on computer vision (ECCV) , pages 529-545, 2018. 3

Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems , pages 5026-5033. IEEE, 2012. 1, 4, 5, 7

Tze Ho Elden Tse, Kwang In Kim, Ales Leonardis, and Hyung Jin Chang. Collaborative learning for hand and object reconstruction with attention-guided graph convolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1664-1674, 2022. 3

Dylan Turpin, Liquan Wang, Eric Heiden, Yun-Chun Chen, Miles Macklin, Stavros Tsogkas, Sven Dickinson, and Animesh Garg. Grasp'd: Differentiable contact-rich grasp synthesis for multi-fingered hands. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part VI , pages 201-221. Springer, 2022. 1, 3, 5

Dimitrios Tzionas, Luca Ballan, Abhilash Srikantha, Pablo Aponte, Marc Pollefeys, and Juergen Gall. Capturing hands in action using discriminative salient points and physics simulation. International Journal of Computer Vision , 118:172-193, 2016. 2, 3

Rong Wang, Wei Mao, and Hongdong Li. Interacting hand-object pose estimation via dense mutual attention. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) , pages 5735-5745, January 2023. 1, 2, 3, 4, 6, 7, 8

Xinyue Wei, Minghua Liu, Zhan Ling, and Hao Su. Approximate convex decomposition for 3d meshes with collision-aware concavity and tree search. ACM Transactions on Graphics (TOG) , 41(4):1-18, 2022. 7

Keenon Werling, Dalton Omens, Jeongseok Lee, Ioannis Exarchos, and C Karen Liu. Fast and featurecomplete differentiable physics for articulated rigid bodies with contact. arXiv preprint arXiv:2103.16021 , 2021. 1, 9

Jie Xu, Viktor Makoviychuk, Yashraj Narang, Fabio Ramos, Wojciech Matusik, Animesh Garg, and Miles Macklin. Accelerated policy learning with parallel differentiable simulation. arXiv preprint arXiv:2204.07137 , 2022. 1, 3

Lixin Yang, Xinyu Zhan, Kailin Li, Wenqiang Xu, Jiefeng Li, and Cewu Lu. Cpf: Learning a contact potential field to model the hand-object interaction. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 11097-11106, 2021. 1, 2, 3, 5, 6, 7, 8

Yufei Ye, Abhinav Gupta, and Shubham Tulsiani. What's in your hands? 3d reconstruction of generic objects in hands. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3895-3905, 2022. 1

Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li. On the continuity of rotation representations in neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5745-5753, 2019. 4

Yuxiao Zhou, Marc Habermann, Weipeng Xu, Ikhsanul Habibie, Christian Theobalt, and Feng Xu. Monocular real-time hand shape and motion capture using multi-modal data. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2020. 3

Christian Zimmermann and Thomas Brox. Learning to estimate 3d hand pose from single rgb images. In Proceedings of the IEEE international conference on computer vision , pages 4903-4911, 2017. 7

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a stable hand pose estimation method leveraging a neural network for stability estimation trained from simulation to improve the physical stability of estimated hand poses. The main idea is training a neural network via simulated results. The network can further provide smooth gradients to refine estimated hand poses. Experiments on two datasets demonstrate the effectiveness of the method and the ability of DeepSim network to provide better gradients where the overall learning framework can benefit from.\n\nSTRENGTHS:\n- The idea of training a neural network as a simulator that can both provide accurate simulation results regrading stability and provide smooth gradients friendly for network training is sound. Gradients provided by the network are also demonstrated to be of higher quality compared to analytical gradients or those from finite differences. \n- Experiments are valid and reasonable that are able to demonstrate the superiority of the proposed method. Implementation details are also provided.\n\nWEAKNESSES:\n- The idea of designing neural networks as a differentiable simulator is not a new thing [1,2]. The network designed in the paper does not has explicit physics priors and only approximates the stability value, which make its generalization ability towards unseen and out-of-distribution data ambiguous. \n- Using networks to learn the stability prediction process is not fully explored. For instance, could the current strategy generalize well towards out-of-distribution test data? Is it possible to improve the generalization ability and train a very powerful stability prediction network by creating and leveraging a large scale synthetic dataset via simulator? Is there any opportunity to fuse physical priors into the design of DeepSim for generalization enhancement? \n\n\n\n[1] Mezghanni, M., Bodrito, T., Boulkenafed, M., & Ovsjanikov, M. (2022). Physical simulation layer for accurate 3d modeling. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition* (pp. 13514-13523).\n\n[2] Mezghanni, M., Boulkenafed, M., Lieutier, A., & Ovsjanikov, M. (2021). Physically-aware generative network for 3d shape modeling. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition* (pp. 9330-9341).\n\nQUESTIONS:\n- Is there any in-depth analysis of the DeepSim network (please see weakness for details)? \n\n- Some works in robotics propose to relax and improve contact models so that smooth gradients can be provided for optimization [1,2]. It is hard to compare with them directly since they are not open-sourced. However, is it possible to conduct toy analysis to compare the effectiveness of soften contact models and the neural simulator proposed in the paper? \n\n\n\n[1] Pang, T., & Tedrake, R. (2021, May). A convex quasistatic time-stepping scheme for rigid multibody systems with contact and friction. In *2021 IEEE International Conference on Robotics and Automation (ICRA)* (pp. 6614-6620). IEEE.\n\n[2] Jain, S., & Liu, C. K. (2011, December). Controlling physics-based characters using soft contacts. In *Proceedings of the 2011 SIGGRAPH Asia Conference* (pp. 1-10).\n\nLIMITATIONS:\nLimitations are stated in Section 5.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes using an external physics simulation to aid in monocular joint 3D hand and object pose estimation. By analyzing the stability of a perceived grasp inside the simulation, the proposed models learn to factor in grasp dynamics when estimating hand and object pose, producing stable and physically plausible grasps. To circumvent the problem of non-differentiable simulation, a DeepSim model is used as a proxy for learning the dynamics of the physics simulator and enables gradients to be propagated through. Quantitative and qualitative results show that the proposed method produces state-of-the-art results in pose estimation while improving physical realism.\n\nSTRENGTHS:\n1. Leveraging the laws of physics effectively can benefit vision-based systems by providing physical prior. However, the effective use of physical laws and/or simulation is difficult as it creates extra overhead which may lead to intractable systems. This paper provides an effective hand/object pose estimation method that intelligently leverages a (simplified) world model (DeepSim) to learn physical prior from simulation. I find the formulation intuitive, and since the ending estimation pipeline is no longer reliant on the simulator, the pipeline is efficient. Essentially, the network is trained in a physics-aware fashion through the use of the world model.\n    1. I think the simplified world model (DeepSim) formulation is interesting and effective. By directly estimation the stability loss the model is easier to learn and can be directly used to optimize the objective.\n2. The motivation of the paper is clear, and the proposed solution and components solve the raised issues. The experiments on the analytical gradient and numerical gradient show the necessity of learning the DeepSim model and provide a clear view of the limitation of the current differentiable physics simulator (namely, unstable gradient when dealing with complex contact geometries).\n3. Results show that the learned pose and hand estimator outperform SOTA methods in terms of physical plausibility and are comparable in terms of pose estimation accuracy. Qualitative results and simulation videos also show that the proposed method is effective in estimating physically stable grasps.\n4. The proposed stability analysis is general and could be applied to other base pose estimation networks as well as other domains such as stable human pose estimation.\n\nWEAKNESSES:\n1. I find the applied adhesion force a little questionable. In the real world, humans grasp objects by applying forces, which is akin to small penetration. Applying an adhesion force is similar to having extra suction cups on the fingertips, which is not realistic. How is the force modeled? Is it a constant or is it a function of the contact forces like static friction?\n2. The current formulation essentially biases the model towards firm grasps and static holdings of objects. All of the examples shown in the results are grasps that require almost all fingers. How about when the object is resting on the palm or not requiring the support of the hand? Since no ground or tabletop is modeled, would the model also provide grasping when the object is resting on the table? In that case, the object is supported by other forces and fingers do not need to apply to grasp. Would the model still bias toward a solution where all fingers are closed in on the object?\n3. As dexterous manipulation is a study of motion, the current setup can be quite limited in modeling faster motion and movement of the objects. Similar to 2, the method is biased toward static and firm grasp, which is not always true when handling an object. The object's own momentum and movement can have a large effect on its stability, which a single-frame model would not factor in.\n\nQUESTIONS:\nI would like to see some discussion on how he adhesion force is applied and modeled, and how well does the model handle non-grasp poses.\n\n--- \nAfter rebuttal, my question about adhesion force and some other details are addressed. I would like to maintain a positive rating of this work. \n\n---\n\nLIMITATIONS:\nSee weakness. I think the method is biased toward stable grasp and would not be able to model hand and object motion (as opposed to pose).", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis manuscript presents a new approach for generating physically realistic estimates of hand and object pose during hand-object interaction. They argue that a weakness of existing approaches is  that while they utilize approaches to avoid hand-object penetration or enforce contact, they do not enforce physically realistic contact that would obey e.g. gravitational forces, which they term dynamical constraints. Their approach is based on the assumption that the hand-object should be stable, that is, if physical forces were applied to an observed frame for a period of a few hundred milliseconds, the hand and object should show minimal displacement. First, following estimation of hand pose and mesh parameters and object rotation using existing approaches, they use a physics simulator to forecast displacements of the hand-object configuration forward 200 ms, and estimate a stability loss equal to the L2 norm of the displacement. They argue that the gradient of this loss relative to state in the physics simulator is unstable, and therefore they use a separate MLP they call DeepSim to approximate the stability loss, taking as input a concatenation of the hand and object signed distance field and hand-object configuration. They alternately train DeepSim and the base network.\n\nThey compare their DeepSim approach on two benchmark datasets with state of the art techniques and achieve competitive, albeit weaker performance on hand and object estimation metrics, but improved performance on a set of more physical criteria – the penetration depth, distance of object displacement and fraction of frames with contact. They motivate the chosen DeepSim architecture through ablations, and show decreases in the magnitude and variance of the gradient using the appproximation approach. \n\nOverall I found the manuscript fairly complete, although of a fairly narrow scope that may mean it is more suited to a specialized computer vision conference. I haven concerns about some of the assumptions made about the physical environment and number of evaluations performed. If some of these questions are addressed I would consider moving up the score.\n\nSTRENGTHS:\n•\tThe manuscript is well motivated, clearly explained, and well contextualized within the field. There are evaluations across multiple datasets, set of ablations. The conclusions for the most part match the results. This should be published somewhere.\n•\tThe architecture and approach are novel to me, and the approach of having a network learn to smooth the state gradient of a physics simulator could be more broadly useful. \n•\tThe qualitative examples are quite compelling and there are seemingly significant increases in the physics scores for the hand-object interactions.\n\nWEAKNESSES:\n•\tThe method does not achieve state of the art in hand and object pose performance on the chosen datasets.\n•\tWhile the approach of smoothing the state gradient could be quite general, the application is narrow in scope. Not only is it limited to hand-object interactions, but the subset of interactions where the grasp is stable. It would be a bit awkward to use this method in practice since it is restricted to the subset of stably grasped frames and those would have to be pre-detected. \n•\tThere are assumptions about contact force strength and the orientation of the gravity axis that are made and it is not clear to what degree they impact the results.\n\nQUESTIONS:\n•\tHow were hyperparameters for the losses chosen?\n•\tCan you provide accuracy comparison values (eg MJE) for the chosen examples? It would be nice to see if they are representative. Alternatively, it would help to see examples of what the failure mode of the techniques were. \n•\tWhy are results from only a single random seed presented?\n•\tL258 “We set the gravity acceleration as 9.8 m/s2  in the y direction of the camera frame. For the adhesion force, we empirically set the gain as 100 and the force as 10N, so that a simple touch is not sufficient to stably grasp the object. “ To what degree is it accurate to assume the gravity axis in these datasets is aligned with the images? To what degree do slight variations in the gravity axis produce large changes in the object pose? Does this affect the comparison between techniques in Tables 1 and 2.\n•\tIt is disappointing that the hand ground truth is not available in Table 2 and it makes it difficult to evaluate the technique as there is really only a single dataset with joint errors. Are there other splits or datasets that could be used? \n•\tCan you report the performance of networks trained with NimblePhysics and FiniteDifference gradients? It is unclear, especially with the Finite Difference network, if the networks simply don’t converge well or if there is a more catastrophic reason why they cannot be used for training. This was a key motivation for the DeepSim approach and it would be nice to verify that they are problematic. It would be interesting if they produced e.g. more physically plausible results.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work presents a novel pipeline for 3D hand-object pose estimation, focusing on improving arbitrary base hand/object pose estimators with applying physical simulation and its induced physical loss (fitted as a neural network). The performance is tested on DexYCB and HO3D data sets and the method is compared against many SotA baselines. Results show improved performance mostly on the physics metrics.\n\nSTRENGTHS:\n- The proposed idea and method are reasonable, interesting, and valid.\n- The performance on the introduced physics metrics gets improved over baseline methods.\n- The paper writing is good and the paper is easy to read and follow.\n- Qualitatively, the proposed method does generate more physically realistic hand/object interaction poses.\n\nWEAKNESSES:\n- My major concern is that the performance regarding hand/object pose estimations is worse than the baselines [32, 52] in Table 1 & 2. Since the proposed method is particularly emphasized on optimizing physical realism, I think it's under the expectation that the method performs better in terms of the proposed physical metrics. But the whole point of doing this should be to improve the hand/object pose estimation results, which is not achieved as shown in the tables.\n- I'm confused by Fig. 1. The gt object pose looks quite different from the input image. Is this the dataset annotation issue or any visualization issue? Why is the gt object pose annotated so off? It's hard to say it's the problem of the baseline [32] if the gt is so off.\n- Why the related work [47, 24] are not compared in Table 1 / 2? I think it's important to compare against them as they also considered physical realism for the same task and the authors have discussed them in related work that they have disadvantages. It's better to show them in numbers.\n- In additional, I feel the idea of using physical simulation as loss to supervise interaction-oriented perception tasks is not new, as the many papers cited by the authors in the related work section and more others.\n\nQUESTIONS:\nsee weakness\n\nLIMITATIONS:\nno issue found", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAuthors propose an approach for 3D pose estimation for hand-object interaction from a single image. Unlike prior work which is purely data driven (except for some works in robotics literature) and focus on visual quality, this work aims to also grasp stability. To this end, authors use physics based simulation to model grasp stability. Since PBS is typically non-differentiable, authors propose to train a neural network (with full supervision from the simulation) to emulate the simulation. This allows them to differentiably approximate the simulation.\nAuthors outperform baselines on DexYCB and HO3D datasets.\n\nSTRENGTHS:\n+ Most existing works focus on visual plausibility. It is interesting to see this work also reason about physical stability.\n+ The idea of approximating non-differentiable simulation has previously been used () but it is still under explored. It would be better if authors provide more context about prior work in this space.\n+ Authors outperform the baselines.\n\nWEAKNESSES:\n[Technical]\n1. Eq. 2: Can the authors provide a bit more motivation/intuition around the equation. What does it do? Why is it correct? Is it a general formulation or does it have specific applications. Current reference ([14]) is a 650+ page book on Classical Mechanics, which does not help the reader much. Maybe point to specific portion of the text that elaborates on the equation.\nIt is not clear how did the authors arrive at this equation from Euler Lagrange equation. \nThis derivations should be included in supp. mat. at least.\n\n2. Sec 3.3: Approximating non-differentiable functions with a neural network is a well known problem. The most straightforward solution is to train a MLP with supervised data. Isn’t this exactly what DeepSim is doing?\nThis can be traced back to “Approximation of functions and their derivatives: A neural network implementation with applications”, Nguyen-Thien et. al. Applied Mathematical Modelling, 1999\nOther domains such as cloth simulation have also used similar techniques to train a neural network to predict the outcome of a physical simulation. (Holden et al. Eurographics’19) \nCan the authors elaborate how is their proposed DeepSim different? This is an under explored area so it is okay to have some similarities with prior work but authors should provide more context and flesh out what are the new insights here.\n\n3. L142: How are the physical properties of the hand and object eg. mass, coefficient of friction etc. obtained from the input image for simulating forces? Does this generalise?\n\n4. Eq. 7: Do we need to mark 8 corners on all template meshes? Isn't this restrictive? Why not put the loss on object mesh vertices directly?\n\n5. Eq. 3: What about rotation? A grasp is still not stable if the object rotates due to a static grasp? Why consider only translation?\n\n6. L140: If M is a matrix, what does M(\\cdot) mean?\n\n[Minor]\n- Missing related work:\nTOCH. Zhou et al. ECCV’22. They learn to predict stable grasps from data.\nJiang et al. ICCV’21: They learn to predict stable contacts on object and synthesise hand to match the contacts.\n\nQUESTIONS:\nSome key formulations are unclear to me (see pt.1) this is important to clarify as it is one of the main contributions of the work.\nMore clarity around design choices would also help the manuscript (see pts. 2-6).\n\nLIMITATIONS:\nAuthors discuss potential limitations and broader impact in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}