DeepSimHO: Stable Pose Estimation for Hand-Object Interaction via Physics Simulation
Authors: Rong Wang
Wei Mao
Hongdong Li
Venue: NeurIPS 2023 poster (2023)
Keywords: hand-object pose estimation
physics simulation

Abstract: This paper addresses the task of 3D pose estimation for a hand interacting with an object from a single image observation. When modeling hand-object interaction, previous works mainly exploit proximity cues, while overlooking the dynamical nature that the hand must stably grasp the object to counteract gravity and thus preventing the object from slipping or falling. These works fail to leverage dynamical constraints in the estimation and consequently often produce unstable results. Meanwhile, refining unstable configurations with physics-based reasoning remains challenging, both by the complexity of contact dynamics and by the lack of effective and efficient physics inference in the data-driven learning framework. To address both issues, we present DeepSimHO: a novel deep-learning pipeline that combines forward physics simulation and backward gradient approximation with a neural network. Specifically, for an initial hand-object pose estimated by a base network, we forward it to a physics simulator to evaluate its stability. However, due to non-smooth contact geometry and penetration, existing differentiable simulators can not provide reliable state gradient. To remedy this, we further introduce a deep network to learn the stability evaluation process from the simulator, while smoothly approximating its gradient and thus enabling effective back-propagation. Extensive experiments show that our method noticeably improves the stability of the estimation and achieves superior efficiency over test-time optimization. The code is available at https://github.com/rongakowang/DeepSimHO.

DeepSimHO: Stable Pose Estimation for Hand-Object Interaction via Physics Simulation
Rong Wang Wei Mao
Hongdong
The Australian National University
{rong. wang, wei mao, hongdong 11] edu.
Abstract
This paper addresses the task 3D pose estimation for a hand interacting with an object from single image observation. When modeling hand-object interaction, previous works mainly exploit proximity cues, while overlooking the dynamical nature that the hand must stably grasp the object to counteract gravity and thus preventing the object from slipping or falling. These works fail to leverage dy- namical constraints in the estimation and consequently often produce unstable results. Meanwhile, refining unstable configurations with physics-based reasoning remains challenging, both by the complexity of contact dynamics and by the lack of effective and efficient physics inference in the data-driven learning framework. To address both issues, present DeepSimHO: a novel deep-learning pipeline that combines forward physics simulation and backward gradient approximation with a neural network. Specifically, for an initial hand-object pose estimated by base network, we forward it to physics simulator to evaluate its stability. However, due to non-smooth contact geometry and penetration, existing differentiable simulators can not provide reliable state gradient. To remedy this, we further introduce a deep network to learn the stability evaluation process from the simulator, while smoothly approximating its gradient and thus enabling effective back-propagation Extensive experiments show that our method noticeably improves the stability of the estimation and achieves superior efficiency over test-time optimization. The code is available at https: //github...... com SaangakkooannDDepSSiiie
Introduction
3D hand-object pose estimation from a single image facilities a wide range of applications, including extended reality (XR) [2], human-robot interaction [32], and animation [17]. To this end, it has drawn increasing attention in recent years 52, 56, 20, 19, 5J. key challenge in this task to ensure the estimated pose conforms to real-world physics, i.e. the hand should make contact with the object without penetrating it, and more importantly, the contact can form a stable grasp to counteract gravity. In this work, aim to estimate hand-object pose that is both accurate and stable.
To model hand-object interaction, related works [21, 5, 57, 18, 20] often impose distance-based attraction and repulsion constraints that encourage contact and penalize penetration. However, this approach only enforces the proximity and do not explicitly reason the dynamical effects of the contact. In consequence, it allows the hand to simply touch the object, which can not form stable grasp. As illustrated Figure I, such contact can cause the object to slip or fall under gravity, therefore is physically unstable. Meanwhile, in other related tasks such as grasping synthesis [7, 50] and dexterous manipulation [43, 12], physics simulators [48, 8, 54] are commonly used to ensure stable grasping. However, discussed in [50, 55), existing differentiable simulators supporting mesh-to-me toehshh contact often produce numerically unreliable state gradient due to the discontinuity in contact geometry [54] and penetration [48], therefore making them difficult to integrate into the
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

learning framework. Alternatiell, [51] perform a brute-force search over predefined poses, however, the approach is time-consuming and only constrained limited configuration space.
(a) Input
(b) GT
[34]
(d) Ours
Duskko
(e) [34] Rotated
(1) Ours Rotated
(g) [34] Simulated
(h) Ours Simulated
Figure 1: Illustration of our method. Given an input image (a), recent work [34] can estimate accurate hand hand-object pose that is close to the ground truth (b) in the camera view (c). However, due to occlusion ambiguity and lack explicit dynamics constraints modeling in hand-objection interaction, they can produce physically unstable results, where the hand only touches the object thus can not form a stable grasp (e). Such contact can cause the object to fall under gravity (g). In contrast, our result is both accurate (d) and physically stable, in (f) and (h).
side and
In this paper, we propose an effective and efficient method DeepSimHH for estimating accurate and stable hand-object pose, by combining physics simulation in the forward step and gradient approximation with deep network in the backward step. Specifically, for initial hand-object pose estimated from a base network, we leverage a physics simulator to evaluate its stability, which is measured by the object center displacement due gravity and hand contact. As shown in Figure optimally stable hand-object pose results in minimal displacement after simulation. However, due to the aforementioned noisy gradient issues, the evaluated stability can be directly used loss to refine the base network via back preeppoppagaaion. To address this issue, we further introduce neural network named DeepSim to reproduce the stability evaluation and back pack-propagate through it. one hand, DeepSim acts as a smooth approximation of the state gradient from the simulator. On the other hand, learns from the simulator to determine the stability of the initial pose, akin to the discriminator in the Generative Adversarial Network (GAN) [16]. However, there is no adversarial relationship between the base network and DeepSim, the training process is much simpler. In training, jointly refine the base network with DeepSim in alternating order similar [16]. In testing, we can estimate physically more stable poses by a simple forward call of the refined base network, contrasting with [56, 51, 20] that require computationally expensive test-time optimization.
Our contributions can summarized as follows. (i) We propose novel pipeline that connects learning framework with physics simulator to estimate accurate and stable hand-object pose. (ii) We introduce DeepSim, a deep network that learns from simulator for stability evaluation and smooth gradient approximation. Extensive experiments on multiple benchmarks show that our method noticeably improves the stability the estimation while maintaining comparable accuracy.
Related Works
Hand-Object Pose Estimation. The problem of estimating hand and object poses from monocular images is ill-conditioned due to depth and occlusion ambiguity. To enforce physically plausible results, related works have employed three categories of approaches to model constraints contact: data-driven [34, 19, 52), distance-based [21, 20, 5, 18] and physics- refinement [56, 26). The data-driven approach leverages real [18, 6] or synthetic [34, 21] data to generate natural contact as result of accurate estimation. Contact correlations also be implicitly learned from the data using

---

the attention [19, 52] or graph convolution [10]. However, this approach is sensitive to noise in the annotation, and perfect estimation is difficult achieve in practice. contrast, [21, 20, 5] explicitly employ attraction and repulsion constraints using the signed distance field (SDF) to encourage contact and penalize inter-penetration. However, such constraint only enforces the hand and object vertices to be in proximity, regardless of whether the hand can stably grasp the object not. While additional affinity labels like [3, 27] can mitigate this issue, they require significantly more efforts to annotate.
Recently, physics-based refinement have been widely adopted in many tasks, including 3D shape modeling [36, 37], human motion estimation [13], and particular those for modeling hand-object interaction, such as task-oriented dexterous manipulation [43, 12] and grasp synthesis [7, 50]. In these works, a sequence of control forces is optimized to ensure a stable grasp on the target object, which is typically achieved via deep reinforcement learning. Motivated by their success, several works [51, 56, 26] have applied physics- bassec refinement for hand-ohject pose estimation. [51] query in physics simulator and search for local configurations that are stable in a brute-force fashion. [56] model the contact as a virtual spring-mast system and minimize the elastic energy. [26] compute the acceleration from the initial kinematic estimation over multiple frames, then optimize for the contact force that aligns best with the regressed acceleration. However, all above approaches rely on computationally expensive post-fitting processes and can only handle limited configuration space, defined by candidate finger parts [51, 26] or anchors [56].
Physics Simulation. Physics simulators for modeling contact dynamics between rigid bodies have been widely used in robotics [43, 12, 31, 30J. Recently, differentiable simulators [38, 40, 39] that can compute the state gradient, i.e. the derivative of the simulated state with respect to the input state, have drawn increasing attention. However, they suffer intrinsic issues of unstable numerical gradient caused by the non- monssmothhness in the linear complementary problem (LCP) [55, 50J. In practice, some also have restrictions in applications, such as being limited to primitive body shapes like spheres cubes [25, 14, 9], which prevent them from being generally applied. Meanwhile, several works [23, 33, 46, 41] have introduced deep neural networks in physics simulations, primarily aimed to reduce the sim-to-real gap [23]. Specifically, [46, 41] adopt graph convolution network to learn to predict physics states as a replacement or refinement of the simulator, while [23, 33] leverage hybrid framework that regresses underlying physics quantities for fine- fineggrined control. However, these works require training on a large amount of real-world physics data and thus can not handle noisy configurations, e-g. when the hand penetrates the object, since they do not exist in real-world data. In contrast, we introduce the neural network to learn fearnfrom the simulator instead of real-world data, therefore can utilize rich training data obtained during simulation and are capable of analyzing noisy configurations. More importantly, all above works attempt to regress high-dimension physics quantities states at each simulation step, while we only regress scalar stability loss, which effectively reduces the difficulty of regression and enjoys superior generalizability.
3 Approach
In this section, we present the proposed pipeline as shown in Figure 2. Given an input image I E #xxx x3 we first adopt a base hand-object pose estimation network (base network) fo(-) to estimate the initial hand mesh M E 77788x3 and rigid object pose po = [Ro to] E (3) (Section 3.1). Next, we collect the initial configuration qo [M², pa] and query its stability from a simulator f.(-) (Section 3.2). To effectively refine the base network, we further introduce the DeepSim network fa(-) to learn from the simulator and smoothly approximate its gradient (Section 3.3). Finally, we describe the overall training objectives (Section 3.4) to train the proposed model.
3.1 Base Hand-Object Pose Estimation Network
Hand Pose Estimation. Motivated by previous works [34, 59], we adopt a two-stage approach to estimate the hand mesh M In particular, at the first stage, we follow the architecture in [47] to estimate the hand joint locations using heatmaps. Given the input image, we first adopt ResNet-34 [22] encoder pre-trained on the ImageNet [45] to extract image features. Next, we decode the image feature with deconvolution layers and obtain K hand heatmaps H E RRxDxDxD after the softmax normalization. Each heatmap H E DDDDD represents the 2.5D location distribution of the k-th joint in the discretized pixel and depth spaces with D partitions. The expected value [ug, 2k zx] E [0, H] x [0, W] x [-R, RR of such distribution determines the position of this joint.
3

---

qo
Gunent Farward Kinematio
Collsion and Jc
LGP Apply Adhesion Farce
Compute Valoclly Integration
fc
9(+1
1
9++1
XT
M
M M
Training
Testing
Distance 4 Concatenation
Calculation
Forward
Back Propugation
Figure 2: Overview of our method. Given an input image I, we aim to enhance the stability of estimated poses from base network fo(-) by learning from physies simulation. Specifically, forward the initial configuration qo consisting of the estimated hand mesh M and object pose jo to a physics simulator (-) and evaluate the stability loss as the discrepancy between the simulated configuration qr. We further use another network named DeepSim fa(-) replicate the stability
loss value from the same initial configuration, so that the gradient can be effectively back pack-propagated through it. In testing, only the refined fo(-) is needed to generate more stable poses M* and pa.
R > O represents the depth radius relative to the hand root joint and is pre-calculated from the training data following [52]. From the wvz joint positions.
then recover their Euclidean coordinates j E RR in the camera frame using the given camera intrinsic K. At the second stage, employ a pretrained inverse kinematics (IK) network [35] to regress the MANO [44] parameters 0 and B from the estimated joint positions. The pose parameter 0 E IR 16 represents the joint rotations along the hand kinematic tree, including the global rotation. The shape parameter 3 E R10 represents the hand shape in PCA components. The final hand mesh vertices positions M can be differentiablely reconstructed from the MANO layer [21].
Object Pose Estimation. We follow previous works [52, 34] to assume a known object mesh template M E R 1000x3 in the canonical frame, and predict the rigid object pose pe that transforms the template into the camera frame. Using the same image feature shared with the hand branch, we adopt a multi-layer perceptron (MLP) to regress the 6D representation [58] of the object rotation R", and estimate the object center translation to relative to the hand root joint in a separate heatmap head. It is worth noting that the simulation-barl to the above base network and makes no assumption on its architecture.
refinement described in following sections limited
3.2 Physies Simulation and Stability Loss
Physics Simulation. Given the initial configuration qo [MA, ,p). we wish to analyze whether it can form stable grasp not. To this end, we leverage physics simulator to evaluate the pose stability considering the dynamical effects of gravity, penetration and contact. Specifically, the physics simulator (-) can be considered as function that advances the current state, including the configuration q and velocity q at time step t as,
(gt, Qr) =
d 0, 1ottt 0, ... .. T
(1)
The state transition governed by the Lagrangian dynamies equation [1, 48] with semi-implicit Euler integration scheme [48], which we formulate for our task as
M(q |r++1 M(q:)q: + cc(q (t) + Jc(q.) (fc(q. fa(q)))At
qi+1 q: Atq
(2)
where M represents the hand-object inertia matrix, e represents bias forces including the grav- itational and Coriolis force, and A/ represents the time duration of each step. The hand-object interaction produces the contact force fc expressed in the contact frame, which consists of the normal force to separate inter-penttrating bodies and the frictional force to resist relative siding. In

---

addition, human imposes torques on finger joints to counteract gravity when grasping the object. However, as our goal is to refine the stability for the initial hand pose, we require the hand remains static during simulation and thus joint torques are not applicable as it can change the hand pose. To model a similar effect we alternatively introduce an adhesion force fa in the contact normal direction to maintain attraction to the object. In this way, stable contact can be established without biasing towards deeper penetration as in [56]. We empirically adjust the quantity of fA to ensure a simple touch is not sufficient to maintain the stability, as shown in Figure 1. Finally, both fc and fA are mapped to the generalized coordinate by the contact Jacobian J Je (q:). The overall dynamics equation can be solved numerically via the steps shown in Figure 2. In the following section we describe the inference of stability based on the simulation results.
Stability Loss. Motivated by [7], we impose a stability loss as the object center displacement after a period of simulation time 7 as
L, = |t - to br ET to = i E qo
In this formulation, we assume a perfectly stable estimation enables the hand to stably grasp the object, avoiding (f) free fall due to being out-of-contact, (ii) slipping due unrealistic interaction, e.g. with fingers only touching to the object at one side, and (iii) recoil due to penetration resolving by the normal force. All above implausible configurations result in a large displacement and thus are unstable. Our goal is therefore to minimize the stability loss for the initial configuration qo. which can be achieved by refining the base network via back- poo-ppropaattio.
However, existing differentiable simulators can not provide reliable gradient for computing / making the propppppaantiinn intractable through the simulator. First, the contact normal of object meshes are discontinuous at corners of triangle faces, thus the analytical gradient of the normal force can be undefined these singular points [50]. Second, simulators like [48] often apply a large normal force to quickly resolve the penetration, which results in a large gradient for small pose perturbation in numerical gradient methods. Consequently, direct gradient calculation can be noisy and does not benefit the refinement, as will be shown in Figure 4. To address this issue, we propose the DeepSim network which acts smooth approximation of the gradient described in the following section.
3.3 DeepSim Network
To smoothly approximate the stability evaluation process, we introduce a differentiable function that learns to produce the same stability loss as the simulator will given any initial configuration. To this end, we parameterize such function with a MLP named DeepSim. Specifically, since the simulator solves for the contact force based on the contact points, we first compute two feature vectors Th E 78778 and ? E R 10.000 from the initial configuration, which represent the signed distances of hand vertices to the object mesh and object vertices to the hand mesh respectively. The two vectors jointly provide rich local contact information to facilitate the stability regression. We then concatenate the two vectors with the flattened initial configuration as the input for the DeepSim fa(-) and regress the replicated stability loss L. as
c. [a (R + +09) ① ch a 00)
where represents concatenation. The objective of the DeepSim network is therefore to minimize the approximation loss as
L. I1.. - La
(5)
In contrast to [46, 41] that regress all components in the simulated state at each step, we directly predict the final stability loss as a scalar, which is empirically an easier objective and better generalizes will be compared in Section 4.5.
to noisy unstable configurations,
Joint Training and Refinement. Similar the GAN [16], we jointly train the base network and DeepSim in an alternating order for an effective refinement. When updating the DeepSim, we use the approximation loss as in (5) while fixing the base network weights. As the training progresses, the base network produces various configurations with corresponding stability losses automatically labelled by the simulator, which serve as rich training pairs for the DeepSim. addition, to avoid the DeepSim overfitting to the base network outputs, we perform random perturbations in the estimated hand and object poses as data augmentation. We further use ground truth poses to train the DeepSim in order to include more data other than the base network output. Mcanwhile, when updating the base
5

---

network, DeepSim facilities to back propaatee the smoothed gradient with respect to the replicated stability loss, with its own weights fixed. Note that we freeze weights in the shared image encoder when refining with the stability loss to avoid corruption of image features. Finally, to avoid being misled by an incorrect regression in the DeepSim, we mask out the replicated stability loss on training samples where É, and L. do not fall the same range. More details are discussed in the supplementary materials.
3.4 Training Objectives
To ensure the estimated poses are both stable and accurate, we impose a multi-task training objective for the base network, which contains two components: the replicated stability loss L. defined in (4) and accuracy losses described in the following. Specifically, we first adopt an L2 loss to supervise the hand joint and mesh estimation as
LA ||Jh - J-
M - M
(6)
where Jh and M represent ground truth hand joints and mesh vertices positions respectively. Next, impose object loss supervise the object pose estimation
Lo ' e + + eR + t.]l
where cº represents the 8 tightest object corners positions in the canonical frame, and RR, t represents the ground truth object pose. To account for the object rotation symmetry, we follow [19] to impose symmetry-awand object loss as
Lor min ||(R'c + eR' +t)l
(8)
where S represents the of all equivalent rotation matrices defined by the objects' symmetry axes [4]. Furthermore, we borrow from [34] to impose an ordinal loss La that supervises for the coarse hand-object relative position. The overall loss is a weighted sum of all individual loss functions as
= An + Aor Co: + too + daLa + A.hs,
where Xn, Aoy Aog Aa d are hyper-parameters.
4 Experiments
4.1 Datasets
We evaluate our method and state-of-the-art methods datasets: Dex [6] and HO3D [18].
DexYCB. The DexYCB dataset [6] consists of 582K frames involving 20 different YCB objects [4]. We use the official "SO" train-test split for the training and evaluation. Following [52], evaluate right hand poses and filter out samples in which the hand or object is not within the field view of the camera. To ensure consistent comparison in physics metries, remove test samples where the hand does not interact with the object and only select those that remain stable after simulation (see the GT results in Table 1), resulting in total of 6348 samples. For training, we do not perform this selection of stability in order to include more data. However, we mask out the replicated stability loss on unstable training samples, e.g. no hand-object interaction, to avoid misleading supervision.
HO3D. The HO3D dataset [18] consists of 66K frames featuring different objects. We select the "v2" version that is mostly evaluated by previous works [56, 20, 52, 34]. Since its ground truth hand poses for the test set are not released, we follow [56] to evaluate on a subset named "v2 "" whose physics plausibility is manually verified by [56]. For training data, we use the official HO3D v2 training split and follow the same practice as the DexYCE dataset to perform sample selection and loss masking. The total HO3Dv2 test set consists of 6076 samples.
4.2 Metrics
Hand Object Metrics. We evaluate the accuracy of pose estimation by comparing the hand and object results with the ground truth labels. For the hand metric, we calculate the mean joint error

---

(MJE) [60], which measures the Euclidean distance between the predicted and ground truth hand joint positions after root joint alignment. For the object metric, compute the mean corner error (MCE) on the HO3I) test set following [34], which calculates the average distance between the predicted and ground truth positions of the 8 tightest object corners. Since on the Dex dataset, many objects are rotationally symmetric, we thus borrow from [19] to compute the symmetry- mean corner error (SMCE) as defined in (8), for a fair comparison with [34].
Table Quantitative comparison on the DexYCB test set. Best results are highlighted in bold and and inapplicable results are marked with "-". Our method achieves superior physics plausibility and stability with comparable accuracy compared to -fate-of-the-- methods [20, 52, 34).
Hand
Object SMCE (cm)
Physics
Methods
MJR (cm)
CP (%)t
PD (em)
SD (em)
SR (%)
GT
100
0.91
0.64
100
Hasson al. [20]
1.25
84.35
1.80
4.83
16.80
ArtiBoost [34]
1.07
1.60
94.23
1.50
2.78
30.07
Wang et al. [52]
1.15
89.16
1.57
3.53
19.54
Ours
1.12
1.73
95.90
1.48
2.42
32.89
Table Quantitative comparison on the HO3Dv2 test set. Best results are highlighted in bold
and and inapplicable results are marked with "-". Our method consistently outperforms baseline methods [20, 52, 34, 56] in all physics metrics while maintaining comparable accuracy.
Hand
Object
Physics
Methods
MJE (em) 4
MCE (cm)
CP (%)f
PD (cm)4
SD (cm))
SR (%)
Hasson ef al. [20]
5.35
78.52
2,02
6.40
10.37
Yang et al. [56]
5.74
96.47
1.65
3.16
17.00
ArtiBoost [34]
4.86
94.47
1.27
2.83
18.02
Wang et al. [52]
4.79
93.07
1.88
3.47
11.42
Ours
5.28
96.64
1.17
2.42
19.24
Physics Metrics. We evaluate the physical plausibility of the estimation following the criterions in 56, 20]. Specifically, we evaluate the simulation displacement (SD) [56] to compute the average object center displacement after 200ms simulation time. We then evaluate the success rate (SR) [7] to classify the prediction whose SD is below threshold of lcm success. These two metrics to evaluate the pose stability. In addition, we evaluate the contact percentage (CP) [28] as the ratio of predictions with hand-object contact and the penetration depth (PD) [3] the maximum penetration distance of contacting hand-object predictions. is worth noting that a stable estimation, i.e. with a lower SD, generally indicates a higher CP and lower PD, but the converse necessarily true.
used
4.3 Implementation Details
We implement the model in PyTorch [42] and train it using the Adam [29] optimizer on a single NVIDIA RTX 3090 GPU. We adopt the base network pretrained in [34], and independently train warm-up. We then jointly train both networks enabling the stability
the DeepSim for 40 epochs loss for 25 epochs. We set the learning rate to 5 10-5 When training on both datasets, we follow [34, 52] to crop input images into 224 x 224 pixels using the provided hand-object bounding boxes. In addition, we follow [34] and perform data augmentation with random translation and rescaling by factor of 0.1. However, we exclude rotation augmentation as it can affect the ground truth stability.
Finally, we set Ao, = 0.5, A = 0, = 0.1, and follow [34] to set A = 0, Aoz = 0.2 on the DexYCB dataset and Ao = 0.2, λoz on the HO3D dataset for fair comparison.
For physics simulation, we use the MuJoCo [48] simulator. We initialize the hand and object inertia matrix M using the Composite Rigid Body algorithm [11]. We further convexly decompose hand meshes by the MANO blending weights [44] and object meshes by the CoACD [53] algorithm to facilitate continuous collision detection [15] in MuJoCo. We set the gravity acceleration as 9.8 m/s² the y direction of the camera frame. For the adhesion force, we empirically set the gain as 100 and the maximum control range as 10, so that simple touch is not sufficient to stably grasp the object. Finally, we set the simulation step to be = 100 and time duration in each step as Al = 0.02. More implementation and training details can be found in the supplementary materials.
7

---

4.4 Results
Quantitative results. We compare our method with three categories of state-of-the-art methods: data-driven [34, 52], distance-based [20] and physics-based [56] approaches. In Table 1, we report the comparison on the Dex YCB test set and re-evaluate all baseline methods using the officially provided weights [34, 52] optimization scripts [20, 56]. To avoid unfair comparison,
exclude the SMCE result for [52, 20] as they do not consider object symmetry in implementation. From the table, we all physics metrics, while maintaining comparable accuracy using significantly less data than [34, 52] when trained with the stability loss. Our method also noticeably improves the stability compared to distance- based approach [20].
observe that method achieves superior performance
DexYCB
HO3D
Input
Camera View
[34]
Input
Camera View
[56]
Ours
Figure 3: Qualitative results on the HO3D and DexYCB test sets. State-of-the-art methods [34, 56] produce results that are either unstable, or stable but do align with the input image, e.g. last row in the HO3D. In contrast, our method produces poses that are both stable and comparably accurate.
In Table 2, compare all methods the H03Dv2 test set. Since the selected split is not suitable for submitting to the official HO3D online evaluation server, we follow the same evaluation protocol as [56] to not evaluate for the hand metric. In addition, although [56] also report the SD in their paper, they use a different simulator [8] and the code for this part is not publicly available. Hence re-evaluate the same metric using our physics modeling for fair comparison. Consistently, observe our method noticeably improves physics plausibility and stability with comparable accuracy. Finally, for the testing, [56] and [20] requires 12 and 45 seconds optimize for one input image single NVIDIA RTX 3090 GPU, in comparison, our method only requires 0.02 seconds on the same device for a forward call of the refined base network, which is significantly more efficient.
Qualitative results. We present qualitative results on the Dex YCB and HO3D test sets in Figure 3. We render the refined hand and object meshes in the camera view. illustrating that the estimated poses align well with the input image. More importantly, we compare in rotated views with [34, (6) to show that while [34] produce visually close results, the established contact actually physically unstable, with some fingers floating in the air or penetrating the object. In addition, [56] do not consider pose accuracy when optimize for physics plausibility, therefore can generate results that stable but do not align with the input image. In comparison, our results are both accurate and physically stable.
4.5 Ablation Study
Effects of the DeepSim Network. To verify the effects the DeepSim for paok-propagation, we compare with two other approaches for calculating the state gradient: (i) numerical gradient using

---

the finite difference in the official MuJoCo implementation, and (ii) analytical gradient through the differentiable simulator NimblePhysics [54]. We use a similar physics model in the NimblePhysics based on its available functions (included in the supplementary materials). To avoid training failure, we include a gradient clip of 1.0 for all methods. Figure (b), we observe (i) produces incorrectly large gradient due to the sudden velocity change during penetration resolving. For (ii), we observe it
Table 3: Comparison on design choices of DeepSim and stability loss. Best performing method (MLP+S) directly regresses final scalar stability loss, which tackles a easier task and therefore outperforms variants that regress more state components (T/RT) or more simulation steps (LSTM).
DeepSim AE (mmm)]
Physics
Methods w/o DeepSim
CP (%yt
PD (cm)]
SD (cm))
SR (%+)
94.19
1.51
2.77
28.55
MLP + T
13.80
94.16
1.53
2.76
28.51
MLP RT
17.22
94.57
1.52
2.81
26.60
LSTM + T
10.79
94.73
1.50
2.60
30.66
MLP S
2.70
95.90
1.48
2.42
32,89
0.000
0.0673
NinssePlassiss Fluits Differnoe DespSin
Defirence
DospSim
0.0000
0.0025
Gradinnt
Soaliit
0.0000
0.0478
0.0030
0.0425
0.0000
100
-10 1000
100
500
Deration
(a) Learning Curve by Gradient Methods
(b) Calculated Gradient Norm
Figure 4: Effects of the DeepSim for back prok-propagation. Figure (a) shows the trend of the stability loss (evaluated from the MuJoCo simulator) as the training progresses. Figure (b) shows the log of the raw gradient norm at each iteration (l@āc,/ag /dqoll for analytical gradient (NimblePhysies) and numerical gradient (Finite Difference), and | for the DeepSim). Gradients directly obtained from the simulators are not suitable for training. In contrast, DeepSim provides numerically stable gradient and benefits the refinement over Base method trained with accuracy losses only.
generates numerically unstable gradient, and even causes numerical overflow at singular points due to the discontinuity in contact geometry. The noisy gradient prevents the base network from being effectively refined as shown in Figure (a). In contrast, the gradient through DeepSim is numerically stable and can better benefit the refinement over training with accuracy losses only.
Design Choices. Table 3, we compare different variants of design in DeepSim and stability loss. We observe that regressing more state components, e.g. the 3 DoF. object center (MLP + T) or 6 DoF. object pose (MLP RT) in the final simulation step, can be more challenging compared to regressing scalar stability loss (MLP + S). As a result the DeepSim performs worse with higher approximation error (AE), which measures the discrepancy between 4, and L.g. Underfitting in DeepSim can cause severe mislcading, thus can not help to improve the stability compared to the baseline method that only trains with accuracy losses (w/o DeepSim). In addition, we replace the MLP with long short-term memory network (LSTM) [24] (implementation details in the supplementary materials), which sequentially estimates residual center displacements in each step (LSTM + T), so that the overall stability loss is the norm of the total displacement. However, we observe that dividing the total displacement into multiple partitions can cause the DeepSim to overfit by predicting only small residuals, therefore does not outperform against directly regressing for the final simulation step.
5 Discussion
Limitations & Societal Impacts. Although DeepSim addresses the issues of simulator gradient in the back-propagation, our physics modeling and simulation in the forward step are still dependent on the simulator implementation, which can be imperfect when compared to real-world physics. In addition, made several restrained assumptions about the physics properties and contact dynamics,

---

for instance we assumed rigid objects with known gravity direction, as well as simplified modeling of joint torques via the adhesion force. As result, the potential sim-to-real gap may lead to unsafety when interacting with objects that have complex and unknown physics properties.
Future Works. The proposed pipeline can be benefited with improved designs of physies metrics, simulator and the DeepSim network. Future works are thus encouraged to explore on (i) stability losses with a more comprehensive consideration of the object state, e.g. rotation, velocity, (ii) more realistic simulation, especially for static hand, (iii) variant DeepSim networks that explicitly utilize physies priors for the regression, and (iv) extension to video imputs with known object initial states.
Conclusion. In this paper, we propose a novel pipeline DeepSinHO for estimating accurate and stable hand-object pose. We explicitly model the dynamical nature in hand-object interaction via forward physics simulation and stability evaluation. In the backward step, we introduce DeepSim, a neural network that learns the stability evaluation from the simulation and performs smooth gradient approximation to facilitate refinement of the hase network. Thanks to them, we our method effectively and efficiently produces physically stable hand-object pose that conforms better to reality.
Acknowledgement
The research is funded in part by an ARC Discovery Grant (grant ID: DP220100800) to HL.
References
[1] Sheldon Andrews, Kenny Erleben, and Zachary Ferguson, Contact and friction simulation for computer graphics. In ACM SIGGRAPH 2022 Courses. pages 1-172. 2022. 4
[2] Mark Billinghurst, Hirokazu Kato, and Ivan Poupyrev. The magicbook: a transitional ar interface. Computers & Graphics, 25(5):745 753, 2001.
[3] Samarth Brahmbhatt, Chengcheng Tang, Christopher D Twigg, Charles C Kemp, and James Hays. Con- tactpose: A dataset of grasps with object contact and hand pose. In European Conference Computer Vision, pages -37. Springer, 2020. 7.7
[4] Berk Calli, Arjun Singh, Aaron Walsman, Siddhartha Srinivasa, Pieter Abbeel, and Aaron M Dollar. The ycb object and model set: Towards common benchmarks for manipulation research. In 2015 international conference on advanced robotics (ICAR), pages 510-517. IEEE, 2015.
[5] Zhe Cao. Ilija Radosavovic, Angjoo Kanazawa, and Jitendra Malik. Reconstructing hand-object interactions in the wild. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12417-
12426, 2021. 1, 2,
[6] Yu- Wei Chao. Wei Yang. Xiang, Pavlo Molchanov, Ankur Handa, Jonathan Tremblay, Yashraj Narang. Karl Van Wyk. Umar lqbal. Stan Birchfield, et al. Dexycb: A benchmark for capturing hand grasping of objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9044-9053, 2021. 2,
[7] Sammy Christen. Muhammed Kocabas, Emre Aksan, Jemin Hwangbo, Jie Song, and Otmar Hilliges. D-grasp: Physically plausible dynamic grasp synthesis for hand-object interactions. In Proceedings of the IEFF/CV Conference Computer Vision and Pattern Recognition (CVPR), 2022. 3, 5,7
[8] Erwin Coumans and Yunfei Bai. Pybullet, python module for physies simulation for games, robotics and
machine learning. http:/ /ttp:///pbuulet.... .rr.. 2016-2021. 1, 8
[9] Jonas Degrave, Michiel Hermans, Joni Dambre, al. A differentiable physics engine for deep learning in roboties. Frontiers neurorobotics, page 6, 2019.
[10] Bardia Doosti, Shujon Naha, Majid Mirbagheri, and David J Crandall. Hope-net: A graph-basee model for hand-object pose estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6608-6601 6617, 2020. 2, [11] Roy Featherstone. Robot dynamics algorithms. Annexe Thesis Digitisation Project 2016 Block S, 1984. 7 Edward Johns, and Kyun Kim. Physics-based dexterous manipulations
[12] Guillermo with estimated hand poses and residual reinforcement learning. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 9561-9566 IEEE, 2020. 1,
[13] Erik Gärtner, Mykhaylo Andriluka. Erwin Coumans, and Cristian Sminchisescu. Differentiable dynamics for articulated 3d human motion reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3199013200, 2022. 3
[14] Moritz Geilinger, David Hahn, Jonas Zehnder, Moritz Bücher, Bernhard Thomaszewski, and Stelian Coros. Add: Analytically differentiable dynamics for multi-body systems with frictional contact. ACM Transactions on Graphics (TOG), 39(6):1 2020. 3
10

---

[15] Elmer G Gilbert, Daniel W Johnson, and S Sathiya Keerthi. A fast procedure for computing the distance between complex objects in three-dimensional space. IEEE Journal Robotics and Automation, 4(2):193-
203, 1988. 7
[16] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11):139
144, 2020. 5,5
[17] Henning Hamer, Juergen Gall, Raquel Urtasun, and Lue Van Gool. Data-driven animation of hand-object interactions. In 2011 IEEE International Conference on Automatic Face & Gesture Recognition (FG),
pages 360-367. IEEE, 2011. 1
[18] Shreyas Hampali, Mahdi Rad, Markus Oberweger, and Vincent Lepetit. Honnotate: A method for 3d annotation of hand and object poses. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3196-3206, 2020. 2,
[19] Shreyas Hampali, Sayan Deb Sarkar, Mahdi Rad, and Vincent Lepetit. Keypoint transformer: Solving joint identification in challenging hands and object interactions for accurate 3d pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11090-11100, 2022. 1,
2, 3, 6,7
[20] Yana Hasson, Gül Varol, Cordelia Schmid, and Ivan Laptev. Towards unconstrained joint hand-object reconstruction from rgb videos. In 2021 International Conference on 3D Vision (3DV), pages 659- 668.
IEEE, 2021. 1, 2, 3, 7,8
[21] Yana Hasson, Gul Varol, Dimitrios Tzionas, Igor Kalevatykh, Michael J Black, Ivan Laptev, and Cordelia Schmid. Learning joint reconstruction of hands and manipulated objects. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11807- 11816, 2019. 1, 2, 3,
[22] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770- 70-7778 2016.
[23] Eric Heiden, David Millard, Erwin Coumans, Yizhou Sheng, and Gaurav S Sukhatme. Neuralsim: Augmenting differentiable simulators with neural networks. In 2021 IEEE International Conference on Robotics and Automation (ICRA). pages 9474-9481. IEEE, 2021.
[24] Sepp Hochreiter and Jürgen Schmidhuher. Long short-term memory. Neural computation, (8):1735-1788,
1997. 9
[25] Taylor A Howell, Simon Le Cleac'h, h, Jan Brüdigam, Zico Kolter, Mac Sehwager, and Zachary Manchester.
Dajo: A differentiable physics engine for robotics. arXiv preprint arXiv:2203. 000066 2022. 3
[26] Haoyu Hu, Xinyu Yi, Hao Zhang, Jun-Hai Yong, and Feng Xu. Physical interaction: Reconstructing
hand-object interactions with physics. In SIGGRAPH Asia 2022 Conference Papers, pages 1-9, 2022. 2,
[27] Hanwen Jiang, Shaowei Liu, Jiashun Wang, and Xiaolong Wang. Hand-object contact consistency reasoning for human grasps generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11107-11 1110-1111,, 2021.
[28] Korrawe Karunratanakul, Jinlong Yang, Yan Zhang, Michael Black, Krikamol Muandet, and Siyu Tang. Grasping tield: Learning implicit representations for human grasps. In 2020 International Conference on 3D Vision (3DV), pages 333-344. IEEE, 2020.
[29] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint
arXiv: 1412.69188 2014.7
[30] Mia Kokic, Danica Kragic, and Jeannette Bohg. Learning to estimate pose and shape of hand-held objects from rgb images. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3980-3987. IEFF, 2019. 3
[31] Mia Kokic, Danica Kragic, and Jeannette Bohg. Learning task oriented grasping from human activity datasets. IEEE Robotics and Automation Letters, 5(2):3352-3355, 2020. 3
[32] Hema Swetha Koppula and Ashutosh Saxena. Anticipating human activities for reactive robotic response.
In IROS, page 2071. Tokyo, 2013. 1
[33] Simon Le Cleac'h, h, Hong-Xing Yu, Michelle Guo, Taylor Howell, Ruohan Gao, Jiajun Wu, Zachary Manchester, and Mac Schwager. Differentiable physics simulation of dynamics-augmented neural ohjects.
IEFE Robotics and Automation Letters, 8(5):2780-2787, 2023.
[34] Kailin Li, Lixin Yang, Xinyu Zhan, Jun Lv, Wenqiang Xu, Jiefeng Li, and Cewu Lu. Artiboost: Boost- ing articulated 3d hand-object pose estimation via online exploration and synthesis. arXiv preprint
arXiv:2109.0m 008.00999.98..... 2021. 1. 2, 3, 6. 7,
[35] Jun Lv, Wenqiang Xu, Lixin Yang. Sucheng Qian, Chongzhao Mao, and Cewu Lu. Handtailor: Towards high-precision monocular 3d hand recovery. arXiv preprint arXiv:2102.0922 2021. 4
[36] Mariem Mezghanni, Théo Bodrito, Malika Boulkenafed, and Maks Ovsjanikov. Physical simulation layer for accurate 3d modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern
Recognition, pages 14-12333,,, 2022. 3
[37] Mariem Mezghanni, Malika Boulkenafed, Andre Lieutier, and Maks Ovsjanikov. Physically-awar generative network for 3d shape modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9330-9341, 2021. 3

---

[38] Mattia Montanari, Nik Petrinic, and Ettore Barbieri. Improving the gik algorithm for faster and more reliable distance queries between convex objects. ACM Transactions on Gruphics (TOG), 36(3):1-17.
2017. 3
[39] Louis Montaut, Quentin Le Lidec, Antoine Bambade, Vladimir Petrik, Josef Sivic, and Justin Carpentier. Differentiable collision detection: a randomized smoothing approach. arXiv preprint 2rX1v:2209.0011.
2022. 3
[40] Louis Montaut, Quentin Le Lidec, Josef Sivic, and Justin Carpentier. Collision detection accelcrated: An optimization perspective. arXiv preprint arXiv:22050096. 2022. 3
[41]
Damian Mrowca, Chengxu Zhuang, Elias Wang, Nick Haber, Li F Fei-Fei, Josh Tenenbaum, and Daniel L Yamins. Flexible neural representation for physics prediction. Advances in neural information processing systems, 31, 2018. 3.5
[42] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary De Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pages 8024 8035, Curran Associates, Inc., 2019.
[43] Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint arXiv: 10087, 2017. 1,
[44] Javier Romero, Dimitrios Tzionas, and Michacl J Black. Emhodicd hands: Modeling and capturing hands and bodics together. arXiv preprint arXiv:2201. 2200002200.0 2022, 4,7
[45] Olga Russakovsky. Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211225
2015.
[46] Alvaro Sanchez-Gonzalez Gonzalez, Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec, and Peter Battaglia. Learning to simulate complex physics with graph networks. In International conference on machine learning, pages 8459-8468. PMLR, 2020. 3,5
[47] Xiao Sun, Bin Xiao, Fangyin Wei, Shuang Liang, and Yichen Wei. Integral human pose regression. In Proceedings of the European conference on computer vision (ECCV), pages 529-545, 2018. 3
[48] Emanuel Todorov, Tom Crez, and Yuval Tassa. Mujoco: physics engine for model-based control. In
2012 IEEE/RSJ international conference on intelligent robots and systems, pages 5026-5033. IEEE, 2012. 1. 4, 5.
[49] Tze Ho Elden Tse, Kwang In Kim, Ales Leonardis, and Hyung Jin Chang. Collaborative learning for hand and object reconstruction with attention-guided graph convolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1664-1674, 2022.
[50] Dylan Turpin. Liquan Wang. Eric Heiden.
Chen, Miles Macklin, Stavros Tsogkas, Sven Dickinson, and Animesh Garg. Grasp'tt d: Differentiable contact-rich grasp synthesis for multi-figgered hands. In Computer Vision-FECC 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part VI, pages 201-221, Springer, 2022. 1, 3,3,5
[51] Dimitrios Tzionas, Luca Ballan, Abbilash Srikantha, Pablo Aponte, Mare Pollefeys, and Juergen Gall. Capturing hands in action using discriminative salient points and physics simulation. International Journal of Computer Vision, 11::172-193, 2016. 2,3
[52] Rong Wang, Wei Mao, and Hongdong Li. Interacting hand-object pose estimation via dense mutual attention. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 5735-5745, January 2023. 3. 7.8
[53] Xinyue Wei, Minghua Liu, Zhan Ling, and Hao Su, Approximate convex decomposition for 3d meshes with collision aware concavity and tree search. ACM Transactions on Graphics (TOG), -1 2022.
[54] Keenon Werling, Dalton Omens, Jeongscok Lee, Ioannis Exarchos, and Karen Liu. Fast and feature- complete differentiable physics for articulated rigid bodies with contact. arXiv preprint arXiv:2103. 16021,
2021. 1.9 9
[55] Jie Xu, Viktor Makoviychuk, Yashraj Narang, Fabio Ramos. Wojeiech Matusik, Animesh Garg, and Miles Macklin. Accelerated policy learning with parallel differentiable simulation. arXiv preprint arXiv:220..07137 2022. 3.3
[56] Lixin Yang, Xinyu Zhan, Kailin Li, Wenqiang Xu, Jiefeng Li, and Cewu Lu. Cpf: Learning contact potential field to model the hand-object interaction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1097-11106, 2021. 1, 2, 3, 6, 7,8
[57] Yufci Yc, Abhinay Gupta, and Shubham Tulsiani. What's 's in your hands? 3d reconstruction of generic objects in hands. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 3895-3905, 2022.
[58] Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li. On the continuity of rotation representa- tions in neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9745-5753 2019. 4
12

---

[59] Yuxiao Zhou, Mare Habermann, Weipeng Xu, Ikhsanul Habibie, Christian Theobalt, and Feng Xu. Monocular real-time hand shape and motion capture using multi-modal data. In IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), June 2020.
[60] Christian Zimmermann and Thomas Brox. Learning to estimate 3d hand pose from single rgb images. In Proceedings of the IEEE international conference computer vision, pages 4903-4911, 2017. 7
13


Reviews:
[{"review_id": "7NflBVsGfv", "reviewer": "Reviewer_HRq6", "summary": "Authors propose an approach for 3D pose estimation for hand-object interaction from a single image. Unlike prior work which is purely data driven (except for some works in robotics literature) and focus on visual quality, this work aims to also grasp stability. To this end, authors use physics based simulation to model grasp stability. Since PBS is typically non-differentiable, authors propose to train a neural network (with full supervision from the simulation) to emulate the simulation. This allows them to differentiably approximate the simulation.\nAuthors outperform baselines on DexYCB and HO3D datasets.", "questions": "Some key formulations are unclear to me (see pt.1) this is important to clarify as it is one of the main contributions of the work.\nMore clarity around design choices would also help the manuscript (see pts. 2-6).", "limitations": "Authors discuss potential limitations and broader impact in the paper.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ Most existing works focus on visual plausibility. It is interesting to see this work also reason about physical stability.\n+ The idea of approximating non-differentiable simulation has previously been used () but it is still under explored. It would be better if authors provide more context about prior work in this space.\n+ Authors outperform the baselines.", "weaknesses": "[Technical]\n1. Eq. 2: Can the authors provide a bit more motivation/intuition around the equation. What does it do? Why is it correct? Is it a general formulation or does it have specific applications. Current reference ([14]) is a 650+ page book on Classical Mechanics, which does not help the reader much. Maybe point to specific portion of the text that elaborates on the equation.\nIt is not clear how did the authors arrive at this equation from Euler Lagrange equation. \nThis derivations should be included in supp. mat. at least.\n\n2. Sec 3.3: Approximating non-differentiable functions with a neural network is a well known problem. The most straightforward solution is to train a MLP with supervised data. Isn’t this exactly what DeepSim is doing?\nThis can be traced back to “Approximation of functions and their derivatives: A neural network implementation with applications”, Nguyen-Thien et. al. Applied Mathematical Modelling, 1999\nOther domains such as cloth simulation have also used similar techniques to train a neural network to predict the outcome of a physical simulation. (Holden et al. Eurographics’19) \nCan the authors elaborate how is their proposed DeepSim different? This is an under explored area so it is okay to have some similarities with prior work but authors should provide more context and flesh out what are the new insights here.\n\n3. L142: How are the physical properties of the hand and object eg. mass, coefficient of friction etc. obtained from the input image for simulating forces? Does this generalise?\n\n4. Eq. 7: Do we need to mark 8 corners on all template meshes? Isn't this restrictive? Why not put the loss on object mesh vertices directly?\n\n5. Eq. 3: What about rotation? A grasp is still not stable if the object rotates due to a static grasp? Why consider only translation?\n\n6. L140: If M is a matrix, what does M(\\cdot) mean?\n\n[Minor]\n- Missing related work:\nTOCH. Zhou et al. ECCV’22. They learn to predict stable grasps from data.\nJiang et al. ICCV’21: They learn to predict stable contacts on object and synthesise hand to match the contacts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer HRq6:\n\nWe thank you for providing valuable feedback and acknowledging the strength of our work. We hope the below responses can address your concerns.\n\n### 1. & 6. Clarification of Eq.(2)\nFor the notation, $\\mathbf{M}(\\mathbf{q}_t)$ denotes that the object inertia matrix $\\mathbf{M}$ is determined by the object configuration $\\mathbf{q}_t$, where the parenthesis $(\\cdot)$ indicates the dependency. Other quantities in Eq.(2) follow the same convention. The notation is commonly used in related literature [44, 50]. In addition, we wish to correct a typo that the left-hand side of the first equation in Eq.(2) should be $\\mathbf{M}(\\mathbf{q}_t)$ instead of $\\mathbf{M}(\\mathbf{q}\\_{t+1})$. We apologize for the confusion caused in understanding the equation and will revise it in the final version.\n\nTo elaborate further, Eq.(2) defines how the object state, including the configuration $\\mathbf{q}_t$ and velocity $\\dot{\\mathbf{q}}_t$, is calculated and updated at each simulation time $t$. Specifically, we formulate the first equation of Eq.(2) to state that the system momentum changes due to the corresponding impulses, *i.e.* $\\mathbf{M}\\dot{\\mathbf{q}}\\_{t+1} = \\mathbf{M}\\dot{\\mathbf{q}}_t + \\mathbf{f}\\Delta t$, where $\\mathbf{f}$ consists of the gravitational and Coriolis force $\\mathbf{c}$ as well as contact-induced forces $\\mathbf{f}_C, \\mathbf{f}_A$. We use the above Lagrangian dynamics equation in order to work in generalized coordinates. Once we have solved the updated velocity $\\dot{\\mathbf{q}}\\_{t+1}$, we can then compute the updated configuration $\\mathbf{q}\\_{t+1}$ using the discrete time Euler integration scheme, as defined in the second equation. The current citation [14], also co-cited in [B1], refers to the definition of the general Lagrangian dynamics equation for rigid bodies (Chapter 5), while the first equation of Eq.(2) is a specific instantiation of it in order to introduce relevant forces in $\\mathbf{f}$ for our task. This equation is implemented by the MuJoCo simulator [44] used in our work. We will include more references [B1,44] to provide a better clarification on this equation.\n\n### 2. Difference to Network-based Simulation Approximation\nPlease refer to the general responses #2. for discussion of our contribution and the difference to other works that use neural networks to approximate physics simulation.\n\n### 3. Obtaining Physics Properties\nWe set the object physics properties based on the previous annotations (See footnote in page 2, supplementary materials). Since the hand remains static during simulation, the hand mass and inertia is irrelevant and we set them as constants in a similar scale to the object. For all physics coefficients, *e.g.* coefficient of friction, we use default values in MuJoCo, which are optimized to be particularly suitable and generalizable for simulating common objects. We manually verified that all simulation parameters reasonably align with real world physics, where the effects are demonstrated in the supplementary video. However, we mentioned in our limitation that the simulation settings may not be perfect for objects with complex and rare physics properties, which is best to be modified for task-specific requirements. Note that we do not estimate physics properties from input images.\n \n### 4. Using Corner Losses\nWe follow the same training pipeline as the baseline method [32] to use object corner losses for a fair comparison. Specifically, we compute the 8 object corners from the ground truth mesh by taking the max and min values along each of the 3 axes, resulting in 8 vertices of the tightest object bounding box. No manual annotation of these corners is required. Since we are refining the 6 DoF object pose as the base network output instead of individual vertex positions, training with corner losses should be equally effective in principle and computationally more efficient than vertex loss. It also better supports batched training when objects have different numbers of vertices.\n\n### 5. Using Rotation in Stability Loss\nWe design the stability loss using only object center displacement after simulation as it is empirically sufficient to determine the stability of the estimated pose. In the implementation, since we simulate for a relatively long time (T = 200ms) and set a small threshold (displacement less than 1cm) to classify as being stable, we empirically observe that samples with a large object rotation change after simulation also tend to exceeds the displacement threshold, which can be correctly classified as unstable.\n\nIn addition, We show in Table 3 of the main paper that the model MLP + RT (using both rotation and translation), performs worse than MLP + T (translation only) due to the increasing difficulty of regression with the DeepSim. We therefore choose to consider object translation only.\n\n### [Minor] Other Related Works\nWe thank the reviewer for providing other related works. However, both suggested papers focus on generating or refining contact given *3D* inputs, which is not directly comparable to our methods as we estimate poses from *image* inputs. For completeness, we will include the discussion of these papers in Section 2 in the final version.\n\n### Bibliography\n[B1] Andrews S, Erleben K, Ferguson Z. Contact and friction simulation for computer graphics[M]//ACM SIGGRAPH 2022 Courses. 2022: 1-172."}, {"review_id": "HnHK8BP4bd", "reviewer": "Reviewer_oW7V", "summary": "This work presents a novel pipeline for 3D hand-object pose estimation, focusing on improving arbitrary base hand/object pose estimators with applying physical simulation and its induced physical loss (fitted as a neural network). The performance is tested on DexYCB and HO3D data sets and the method is compared against many SotA baselines. Results show improved performance mostly on the physics metrics.", "questions": "see weakness", "limitations": "no issue found", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The proposed idea and method are reasonable, interesting, and valid.\n- The performance on the introduced physics metrics gets improved over baseline methods.\n- The paper writing is good and the paper is easy to read and follow.\n- Qualitatively, the proposed method does generate more physically realistic hand/object interaction poses.", "weaknesses": "- My major concern is that the performance regarding hand/object pose estimations is worse than the baselines [32, 52] in Table 1 & 2. Since the proposed method is particularly emphasized on optimizing physical realism, I think it's under the expectation that the method performs better in terms of the proposed physical metrics. But the whole point of doing this should be to improve the hand/object pose estimation results, which is not achieved as shown in the tables.\n- I'm confused by Fig. 1. The gt object pose looks quite different from the input image. Is this the dataset annotation issue or any visualization issue? Why is the gt object pose annotated so off? It's hard to say it's the problem of the baseline [32] if the gt is so off.\n- Why the related work [47, 24] are not compared in Table 1 / 2? I think it's important to compare against them as they also considered physical realism for the same task and the authors have discussed them in related work that they have disadvantages. It's better to show them in numbers.\n- In additional, I feel the idea of using physical simulation as loss to supervise interaction-oriented perception tasks is not new, as the many papers cited by the authors in the related work section and more others.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer oW7V:\n\nWe thank you for providing valuable feedback and acknowledging the strength of our work. We hope the below responses can address your concerns.\n\n### 1. Concerns about Accuracy Performance\nWe hypothesize that the comment intends to mean that our method performs worse than the baseline method [32, 48] instead of [32, 52]. In Table 2 of the main paper we show that our method achieves superior performance in both accuracy and stability compared to [52], which also explicitly enforces physics realism in hand-object pose estimation. Our accuracy is comparable to [32, 48] even though they use more augmented data during training, and our method achieves superior results when the baseline methods are trained with the same amount of data. **Please refer to the general responses #1. for detailed discussion related to accuracy performance.**\n\n### 2. Clarification for Fig.1\nWe believe that there are some misinterpretations for Fig.1. In Fig.1, the visualization for the ground truth annotation refers to the figure in the *first row*, second column, instead of the second row, second column, *i.e.* caption at the *bottom* of the figure. The figure in the second row, second column instead visualizes the hand-object pose in a rotated view angle in order to better demonstrate that our method generates stable contact in the occluded area. **In the correct figure for visualizing the ground truth annotation, we project the ground truth hand and object mesh in the image space and prove that they align well with the input image, hence there should be no significant errors in the annotation.**\n\n### 3. Comparison to [47, 24]\nBoth related works [47, 24] do not release codes for evaluation. [47] also does not release the physics models and exact settings used in simulation. We are therefore unable to reproduce their works for evaluation. Besides, both [47, 24] are evaluated on self-collected small-scale datasets, which are also not publicly available. Hence we are unable to evaluate our method on their datasets and compare with their performance. Nevertheless, in quantitative comparison, we emphasize the comparison with [52], which is a recent work that also enforces physical realism in hand-object pose estimation. Table 2 of the main paper shows that our method consistently outperforms [52] in terms of both accuracy and stability.\n\n### 4. Clarification for Contribution and Novelty\n**Our main contribution is not about being the first to integrate physics simulator or simulator-induced losses in the refinement pipeline, but proposing a more *effective* and *efficient* method that learns from physics simulation for estimating stable hand-object pose with complex contact geometry**. Due to the intrinsic discontinuity in simulation process and resulting noisy state gradient, directly imposing losses from differentiable physics simulators and refining the base network via gradient descent is challenging. To this end, previous works perform brute-force search [47] over limited configuration space, or rely on additional global optimization [B2]. **Such test-time optimization strategy is computationally expensive and therefore has limited application**. Alternatively, other works [39, 11] propose to adopt a deep reinforcement learning framework to work with non-differentiable simulators. However, **these methods do not generalize to unseen data and are also difficult to converge in training**. In contrast, we propose to adopt a neural network  DeepSim that can smoothly approximate state gradient from the simulator and effectively refine the base network via back-propagation, **which is not addressed by related works**. Besides, since our method does not require test-time optimization, more stable results can be produced via a simple forward call of the refined base network, which is more efficient and practically applicable compared to previous works. Ablation study in **Section 4.5 of the main paper also demonstrates the effectiveness and superiority of our method compared to directly using the physics simulator as supervision**.\n\nFurthermore, We refer the reviewer to the general responses #2. for the novelty of our method compared to other works that adopt neural networks for simulation approximation.\n\n\n### Bibliography\n[B2] Gärtner E, Andriluka M, Coumans E, et al. Differentiable dynamics for articulated 3d human motion reconstruction[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 13190-13200."}, {"review_id": "Smw865K9Pb", "reviewer": "Reviewer_gag8", "summary": "This manuscript presents a new approach for generating physically realistic estimates of hand and object pose during hand-object interaction. They argue that a weakness of existing approaches is  that while they utilize approaches to avoid hand-object penetration or enforce contact, they do not enforce physically realistic contact that would obey e.g. gravitational forces, which they term dynamical constraints. Their approach is based on the assumption that the hand-object should be stable, that is, if physical forces were applied to an observed frame for a period of a few hundred milliseconds, the hand and object should show minimal displacement. First, following estimation of hand pose and mesh parameters and object rotation using existing approaches, they use a physics simulator to forecast displacements of the hand-object configuration forward 200 ms, and estimate a stability loss equal to the L2 norm of the displacement. They argue that the gradient of this loss relative to state in the physics simulator is unstable, and therefore they use a separate MLP they call DeepSim to approximate the stability loss, taking as input a concatenation of the hand and object signed distance field and hand-object configuration. They alternately train DeepSim and the base network.\n\nThey compare their DeepSim approach on two benchmark datasets with state of the art techniques and achieve competitive, albeit weaker performance on hand and object estimation metrics, but improved performance on a set of more physical criteria – the penetration depth, distance of object displacement and fraction of frames with contact. They motivate the chosen DeepSim architecture through ablations, and show decreases in the magnitude and variance of the gradient using the appproximation approach. \n\nOverall I found the manuscript fairly complete, although of a fairly narrow scope that may mean it is more suited to a specialized computer vision conference. I haven concerns about some of the assumptions made about the physical environment and number of evaluations performed. If some of these questions are addressed I would consider moving up the score. \n", "questions": "•\tHow were hyperparameters for the losses chosen?\n•\tCan you provide accuracy comparison values (eg MJE) for the chosen examples? It would be nice to see if they are representative. Alternatively, it would help to see examples of what the failure mode of the techniques were. \n•\tWhy are results from only a single random seed presented?\n•\tL258 “We set the gravity acceleration as 9.8 m/s2  in the y direction of the camera frame. For the adhesion force, we empirically set the gain as 100 and the force as 10N, so that a simple touch is not sufficient to stably grasp the object. “ To what degree is it accurate to assume the gravity axis in these datasets is aligned with the images? To what degree do slight variations in the gravity axis produce large changes in the object pose? Does this affect the comparison between techniques in Tables 1 and 2.\n•\tIt is disappointing that the hand ground truth is not available in Table 2 and it makes it difficult to evaluate the technique as there is really only a single dataset with joint errors. Are there other splits or datasets that could be used? \n•\tCan you report the performance of networks trained with NimblePhysics and FiniteDifference gradients? It is unclear, especially with the Finite Difference network, if the networks simply don’t converge well or if there is a more catastrophic reason why they cannot be used for training. This was a key motivation for the DeepSim approach and it would be nice to verify that they are problematic. It would be interesting if they produced e.g. more physically plausible results. \n", "limitations": "Yes. ", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "•\tThe manuscript is well motivated, clearly explained, and well contextualized within the field. There are evaluations across multiple datasets, set of ablations. The conclusions for the most part match the results. This should be published somewhere.\n•\tThe architecture and approach are novel to me, and the approach of having a network learn to smooth the state gradient of a physics simulator could be more broadly useful. \n•\tThe qualitative examples are quite compelling and there are seemingly significant increases in the physics scores for the hand-object interactions. \n", "weaknesses": "•\tThe method does not achieve state of the art in hand and object pose performance on the chosen datasets.\n•\tWhile the approach of smoothing the state gradient could be quite general, the application is narrow in scope. Not only is it limited to hand-object interactions, but the subset of interactions where the grasp is stable. It would be a bit awkward to use this method in practice since it is restricted to the subset of stably grasped frames and those would have to be pre-detected. \n•\tThere are assumptions about contact force strength and the orientation of the gravity axis that are made and it is not clear to what degree they impact the results. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer gag8:\n\nWe thank you for providing valuable feedback and acknowledging the strength of our work. We hope the below response can address your concerns.\n\n### 1. Concerns about Performance in Accuracy\nPlease refer to the general responses #1. for the discussion about the accuracy performance.\n\n### 2. Scope of Application\n**Our proposed method is a general framework and works with any task-specific base networks and physics models, therefore is not restricted in hand-object interaction only**. For instance, other tasks including human pose [B3] and motion estimation [B2] also require the results to be physically stable under the interaction to the ground and scenes. Our methods can be easily extended to these tasks with modified base networks and physics simulators, using the DeepSim network to smoothly connect the two parts. This is also acknowledged by the reviewer `bTxd` in strength 4.\n\n**Besides, our method is also not restricted to stable grasping frames only since we train the model jointly using accuracy and stability losses**. We mentioned in L219 of the main paper that during training, we include all samples whose ground truth are both stable and unstable, *e.g.* having no hand-object contact. To avoid biasing towards grasping, we mask out the stability loss for samples whose ground truth is not stable, and train on these samples using only accuracy losses. Therefore, our method can generalize to both types of test samples. We include more results on non-grasping samples in the Fig.1 of the PDF in the general responses for illustration. **In summary, since our model is conditioned on the input image, it will not blindly predict all samples as having a grasping hand if the image does not indicate so**.\n\n### 3. Impact of Design Choices\nWe make assumptions about the contact force strength and the direction of gravity based on the observation of the training data. We empirically find these design choices reasonably align with real world physics. Please refer to the supplementary video for qualitative evaluation.\n\nSince in practice, it is difficult to perfectly simulate real world physics, we take special care in training to avoid the model being misled by imperfect simulation. Specifically, we examine all training data and *do not* impose stability loss on samples whose ground truth pose are *unstable under our assumption*. For firm grasping cases and non-contacting cases, they remain stable/unstable regardless of the gravity direction. For other samples, only those that align with our assumption will be included for training with stability loss. **Consequently, the impact is restricted mainly to the reduction of training data**.\n\nIn testing, we also ensure that the selected samples follow our assumption (see the GT in Table 1 of the main paper) or manually verified by the previous work [52]. Hence the physics metrics shown in Table 1 & 2 are meaningful and comparable.\n\n### 4. Hyper-parameters in Losses\nWe follow the same training pipeline as [32] to set the same weights for accuracy losses. For the stability loss, we set the weight as $\\lambda_s = 0.1$ so that all losses are roughly in the same scale.\n\n### 5. Accuracy Comparison for Selected Examples\nPlease refer to Table 4 of the PDF in general responses for the accuracy of selected examples in the main paper.\n\n### 6. Analysis of Failure Cases\nPlease refer to Fig. 2 of the PDF in general responses for the analysis of the failure case.\n\n### 7. Random Seed\nWe show it in Table 3 of the PDF in general responses for the results of multiple runs on the two datasets. The result shows a low variance in our method and indicates a stable performance of our model. \n\n### 8. More Quantitative Evaluation\nWe further follow [52] to compare on the HO3Dv1 split for more quantitative evaluation. We adopt the same setting as [52] for a fair comparison. Please refer to Table 1 of the PDF in general responses for the result. Note that both [19, 52] are state-of-the-art methods that explicitly adopt physics priors when modeling hand-object interaction. Compared to them, our method achieves consistently better performance in both accuracy and stability.\n\n### 9. Results for NimblePhysics and FiniteDifference\nAs shown in the Fig. 4(a) of the main paper, both methods fail to converge as the training losses can not be decreased, **leading to significantly worse performance and out-of-range scores**. For FiniteDifference (green curve), we can clearly observe an increase of the loss as training progresses, indicating a failure of training. This is because it produces incorrectly large gradient due to the sudden velocity change during penetration resolving. For NimblePhysics, the noisy gradient, as demonstrated in Fig.4 (b), also prevents the loss from decreasing, which illustrates the necessity of our method in smoothing the gradient. In addition, we mentioned in L37 of the supplementary material that the NimblePhysics takes around 120 hours to train a single epoch, as computing the state gradient for complex contact geometry is computationally expensive. **In consequence, it is also intractable to apply it when training on large-scale datasets**.\n\n### Bibliography\n[B2] Gärtner E, Andriluka M, Coumans E, et al. Differentiable dynamics for articulated 3d human motion reconstruction[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 13190-13200.\n\n[B3] Tripathi S, Müller L, Huang C H P, et al. 3D human pose estimation via intuitive physics[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 4713-4725."}, {"review_id": "xY2rhbhPWJ", "reviewer": "Reviewer_bTxd", "summary": "This work proposes using an external physics simulation to aid in monocular joint 3D hand and object pose estimation. By analyzing the stability of a perceived grasp inside the simulation, the proposed models learn to factor in grasp dynamics when estimating hand and object pose, producing stable and physically plausible grasps. To circumvent the problem of non-differentiable simulation, a DeepSim model is used as a proxy for learning the dynamics of the physics simulator and enables gradients to be propagated through. Quantitative and qualitative results show that the proposed method produces state-of-the-art results in pose estimation while improving physical realism.", "questions": "I would like to see some discussion on how he adhesion force is applied and modeled, and how well does the model handle non-grasp poses.\n\n--- \nAfter rebuttal, my question about adhesion force and some other details are addressed. I would like to maintain a positive rating of this work. \n\n---", "limitations": "See weakness. I think the method is biased toward stable grasp and would not be able to model hand and object motion (as opposed to pose).", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Leveraging the laws of physics effectively can benefit vision-based systems by providing physical prior. However, the effective use of physical laws and/or simulation is difficult as it creates extra overhead which may lead to intractable systems. This paper provides an effective hand/object pose estimation method that intelligently leverages a (simplified) world model (DeepSim) to learn physical prior from simulation. I find the formulation intuitive, and since the ending estimation pipeline is no longer reliant on the simulator, the pipeline is efficient. Essentially, the network is trained in a physics-aware fashion through the use of the world model.\n    1. I think the simplified world model (DeepSim) formulation is interesting and effective. By directly estimation the stability loss the model is easier to learn and can be directly used to optimize the objective.\n2. The motivation of the paper is clear, and the proposed solution and components solve the raised issues. The experiments on the analytical gradient and numerical gradient show the necessity of learning the DeepSim model and provide a clear view of the limitation of the current differentiable physics simulator (namely, unstable gradient when dealing with complex contact geometries).\n3. Results show that the learned pose and hand estimator outperform SOTA methods in terms of physical plausibility and are comparable in terms of pose estimation accuracy. Qualitative results and simulation videos also show that the proposed method is effective in estimating physically stable grasps.\n4. The proposed stability analysis is general and could be applied to other base pose estimation networks as well as other domains such as stable human pose estimation.", "weaknesses": "1. I find the applied adhesion force a little questionable. In the real world, humans grasp objects by applying forces, which is akin to small penetration. Applying an adhesion force is similar to having extra suction cups on the fingertips, which is not realistic. How is the force modeled? Is it a constant or is it a function of the contact forces like static friction?\n2. The current formulation essentially biases the model towards firm grasps and static holdings of objects. All of the examples shown in the results are grasps that require almost all fingers. How about when the object is resting on the palm or not requiring the support of the hand? Since no ground or tabletop is modeled, would the model also provide grasping when the object is resting on the table? In that case, the object is supported by other forces and fingers do not need to apply to grasp. Would the model still bias toward a solution where all fingers are closed in on the object?\n3. As dexterous manipulation is a study of motion, the current setup can be quite limited in modeling faster motion and movement of the objects. Similar to 2, the method is biased toward static and firm grasp, which is not always true when handling an object. The object's own momentum and movement can have a large effect on its stability, which a single-frame model would not factor in.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer bTxd:\n\nWe thank you for providing valuable feedback and acknowledging the strength of our work. We hope the below responses can address your concerns.\n\n### 1. Modeling Adhesion Force\nIn practice, we find applying hand-object penetration to emulate the effects of joint torques can degrade the simulation realism as it often encourages the model biasing towards deeper penetration, also observed by [52]. In this way, stability of contact can be achieved by having deep but balanced penetration over multiple sides of the object, or even penetrating through the object, considering the limitation of the penetration resolving in modern physics simulators. **Such artifacts are undesired in the applications and violate physics realism, therefore we choose not to model the interaction via hand-object penetration**.\n\nFor the details of adhesion force modeling, we investigate the training data and **carefully adjust the strength of the adhesion force so that a simple touch can not form a stable grasp**. The overall adhesion force strength depends on the number of hand-object contact, where for each contact point, the adhesion force is applied in the direction of contact normal with a fixed strength. The effect of the adhesion force can be observed in the supplementary video, which demonstrates that it reasonably simulates with real world physics **without the need of undesired hand-object penetration**. **Overall, we find it as a better model to emulate the effects of joint torques when the hand is required to remain static during simulation**.\n\n### 2. Bias Towards Grasping\nOur method does not bias towards grasping or specific forms of grasping as it is additionally conditioned on the input image and supervised with accuracy losses. We mentioned in L219 of the main paper that we include training samples whose ground truth poses are both stable and unstable, *e.g.* with no hand-object interaction, and only impose the stability loss on samples whose ground truth poses are stable. For samples whose ground truth poses are unstable, we train on them using only the accuracy losses. Therefore, our method avoids blindingly producing firm grasping when no hand-object interaction is indicated from the input image. **Please also refer to the Fig.1 of the PDF in the general responses for qualitative results on such cases**.\n\nIn addition, we make no assumptions on the contacting fingers and forms of grasping, but rely on the simulator to evaluate the effect of contact on *all detected contacting vertices*. Specifically, unlike [52] that only studies contact on predefined anchors, we consider contact forces on the entire hand mesh for all contact points detected by the simulator in collision detection. This allows us to explore various forms of contact other than grasping with all fingers. **We include more qualitative results in Fig.1 in the PDF of general responses to show that our method can generalize to various forms of grasping, *e.g.* resting on the palm or interacting with a few fingers**.\n\n### 3. Generalize to Fast Object Motion\nIn this work, we address the task of estimating hand and object poses from only a *single image input*, **which is already a challenging task**. Since we do not have information about the object's previous states, from the training data we make a reasonable assumption that the movement is slow and the internal object acceleration is negligible compared to gravity. However, **our method can be easily extended to estimating the hand-object motion from a sequence of input frames with various object initial states**. First, the object velocity $\\dot{\\mathbf{q}}_0$ can be adjusted to appropriate values if the object does not start to be static. Besides, if the object has its own internal driven force, *e.g.* equipped with a motor, the Eq.(2) can be correspondingly modified to take into the account of additional sources of forces. We believe these extensions are beyond the scope of our work and are suitable for future research."}, {"review_id": "Gd2CA9ygMN", "reviewer": "Reviewer_fSiU", "summary": "The paper presents a stable hand pose estimation method leveraging a neural network for stability estimation trained from simulation to improve the physical stability of estimated hand poses. The main idea is training a neural network via simulated results. The network can further provide smooth gradients to refine estimated hand poses. Experiments on two datasets demonstrate the effectiveness of the method and the ability of DeepSim network to provide better gradients where the overall learning framework can benefit from. ", "questions": "- Is there any in-depth analysis of the DeepSim network (please see weakness for details)? \n\n- Some works in robotics propose to relax and improve contact models so that smooth gradients can be provided for optimization [1,2]. It is hard to compare with them directly since they are not open-sourced. However, is it possible to conduct toy analysis to compare the effectiveness of soften contact models and the neural simulator proposed in the paper? \n\n\n\n[1] Pang, T., & Tedrake, R. (2021, May). A convex quasistatic time-stepping scheme for rigid multibody systems with contact and friction. In *2021 IEEE International Conference on Robotics and Automation (ICRA)* (pp. 6614-6620). IEEE.\n\n[2] Jain, S., & Liu, C. K. (2011, December). Controlling physics-based characters using soft contacts. In *Proceedings of the 2011 SIGGRAPH Asia Conference* (pp. 1-10).", "limitations": "Limitations are stated in Section 5.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The idea of training a neural network as a simulator that can both provide accurate simulation results regrading stability and provide smooth gradients friendly for network training is sound. Gradients provided by the network are also demonstrated to be of higher quality compared to analytical gradients or those from finite differences. \n- Experiments are valid and reasonable that are able to demonstrate the superiority of the proposed method. Implementation details are also provided. ", "weaknesses": "- The idea of designing neural networks as a differentiable simulator is not a new thing [1,2]. The network designed in the paper does not has explicit physics priors and only approximates the stability value, which make its generalization ability towards unseen and out-of-distribution data ambiguous. \n- Using networks to learn the stability prediction process is not fully explored. For instance, could the current strategy generalize well towards out-of-distribution test data? Is it possible to improve the generalization ability and train a very powerful stability prediction network by creating and leveraging a large scale synthetic dataset via simulator? Is there any opportunity to fuse physical priors into the design of DeepSim for generalization enhancement? \n\n\n\n[1] Mezghanni, M., Bodrito, T., Boulkenafed, M., & Ovsjanikov, M. (2022). Physical simulation layer for accurate 3d modeling. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition* (pp. 13514-13523).\n\n[2] Mezghanni, M., Boulkenafed, M., Lieutier, A., & Ovsjanikov, M. (2021). Physically-aware generative network for 3d shape modeling. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition* (pp. 9330-9341).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer fSiU:\n\nWe thank you for providing valuable feedback and acknowledging the strength of our work. We hope the below responses can address your concerns.\n\n### 1. Difference to Previous Physics-based Works.\nWe thank the reviewer for providing other related works. However, the suggested works [B4, B5], similar to other works in grasp synthesis [6, 46], all address the task of *synthesizing* physically valid results, which are not directly comparable to our method **as we address a more challenging task of jointly estimating *stable* and *accurate* hand-object poses *conditioned on a monocular image observation***. In addition, **we tackle the problem of effective joint training of the DeepSim and base network to avoid overfitting and enforce the generalizability, which is not addressed by previous works.** For completeness, we will include [B4, B5] in the discussion of related works in the revised paper.\n\nWe also refer the reviewer to the general responses #2. for more discussion and insights about our method compared previous works.\n\n### 2. Generalizability of the DeepSim Network\nThe trained DeepSim network can generalize to unseen test data thanks to our training strategy using large-scale perturbed hand-object data. Specifically, We mentioned in L189 of the main paper that to avoid overfitting, we include randomly perturbed initial hand and object poses before forwarding to the simulator when training the DeepSim network (note the augmentation is for training the DeepSim only). **This is essentially the same as generating large-scale synthetic pose-stability data pairs as the training progresses**. In addition, ablation study in Table 3 of the main paper further quantitatively proves the generalizability of the DeepSim network. In Table 3, we observe that the Approximation Error (AE, measuring the distance between the predicted stability and ground truth stability evaluated by the simulator on unseen test data), is significantly smaller than the stability threshold used in the physics metric, **indicating that the DeepSim is sufficiently accurate to distinguish if the estimated hand-object pose is stable or not during testing**, which is the key to the success of our method.\n\n### 3. Integrating Physics Priors to the DeepSim Network\nIn addition to initial hand and object poses, we included two feature vectors $\\hat{\\mathbf{c}}^h$ and $\\hat{\\mathbf{c}}^o$ as the contact priors for the input of the DeepSim network to improve its effectiveness, as mentioned in L174 of the main paper. Since the stability of the estimated hand-object pose depends primarily on the contact force, which is further dependent on the contact configuration, *i.e.* contact points, penetration volume etc., **these two vectors embed rich physics priors and help to improve the precision of stability prediction**. \n\nThere are various ways to fuse additional physics priors to better faciliate the DeepSim network. First, the DeepSim network can condition on other physics properties, *e.g.* mass, gravity etc. for more accurate regression. In addition, since collision detection and penetration resolving often cause numerically unstable gradient, the DeepSim network can instead aproximate these subroutines by regressing components of contact forces *e.g.* contact positions, penetration direction and volume etc., leaving other rountines like object acceleration and velocity calculation handled by the simulator. We believe these practices are suitable for future research works. \n\n### 4. Analysis on Soften Contact Model\nWe thank the reviewer for providing additional works on the improved soften contact model. However, since we need to compute the overall gradient $\\frac{\\partial \\mathcal{L}_s}{\\partial \\mathbf{q}_0}$ in order to refine the base network in an end-to-end fashion, **having only improved soften contact model is not sufficient to guarantee the robustness of the overall gradient**. First, as the base network can potentially generate initial poses where the hand penetrates the object, the penetration must be first resolved by the simulator by applying a large normal force to separate the hand and object mesh. We observe numerically unstable gradient often arises at this stage due to the sudden velocity change in penetration resolving. Furthermore, existing methods that successfully leverage differentiable physics simulators mostly consider primitive shapes only, where the collision detection procedure is straightforward. However, in our task the contacting object meshes often have complex and discontinuous mesh geometry, causing the state gradient associated to the collision detection to be numerically unstable as well. Therefore, we believe adopting only a soften contact model can not fully address the gradient issue and emphasize the necessity of the proposed DeepSim network.\n\n### Bibliography\n[B4] Mezghanni, M., Bodrito, T., Boulkenafed, M., & Ovsjanikov, M. (2022). Physical simulation layer for accurate 3d modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 13514-13523).\n\n[B5] Mezghanni, M., Boulkenafed, M., Lieutier, A., & Ovsjanikov, M. (2021). Physically-aware generative network for 3d shape modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 9330-9341)."}]

Author rebuttal:
We thank all the reviewers for their time and valuable comments. Below, we first clarify common concerns raised by multiple reviewers.

### 1. Concerns about Accuracy Performance (Reviewer oW7V, gag8)
In this paper, we aim to address the task of estimating stable hand and object poses from single-image inputs, which is **important for applications that demand *robust* hand-object interaction**. For instance, in dexterous manipulation, ensuring successful grasp and manipulation of the target object often takes precedence over precisely replicating the exact contact points. However, previous learning-based methods such as [32, 48] often produce suboptimal results where although the hand fingers are close to the ground truth contact positions, a stable grasp is not actually formed. In consequence, while they exhibit higher estimation accuracy, i.e. with reduced hand and object pose errors, they are unsuitable for these applications. To this end, we place a stronger emphasis on physics metrics to better cater to the requirements of such applications.

**Compared to previous methods [52, 19] that also explicitly optimize for physics realism, our method achieves state-of-the-art performance in both accuracy and stability**. In particular, [52, 19] impose over-simplified assumptions on contact dynamics, in contrast, we learn complete dynamics priors from the physics simulator and thus avoid biasing towards a restricted set of stable poses. We also highlight that our methods are more efficient compared to [52, 19] since no test-time optimization is needed.

We note that baseline methods [32, 48] have a higher accuracy performance, this is because we evaluate them using their officially released model weights, which are **trained with significantly more augmented data**. In particular, they synthesize additional hand and object images and corresponding annotations from various rotated views to mitigate occlusion ambiguity. However, since rotating the hand and object poses may alter the stability status of the original configuration, we did not apply the same augmentation strategy and only used a reduced amount of data for the training with the stability loss. Nevertheless, our method still achieves comparable accuracy thanks to the dynamics priors learned from the physics simulation. Qualitative results also demonstrate that estimated hand and object poses visually align with input images. **In Table 4 of the attached PDF, we show that our method achieves better accuracy than [32] (which uses the same base network with us) when training with the same amount of data, proving that the accuracy of our method is comparable**. Furthermore, the evaluation also proves that having a higher accuracy performance does not necessarily result in better stability, which jusitfies the motivation of our work.

### 2. Difference to Methods using Neural Networks to Approximate Simulator (Reviewer HRq6, oW7V, fSiU)
Unlike previous methods [42, 37] that attempt to directly approximate the entire physics simulator by regressing *complete simulated states*, **our key insight in designing the DeepSim is to regress the *scalar* stability loss supervised by the simulator, which is a simplified task and practically more feasible to achieve**. This improved design allows the DeepSim network to accurately infer the pose stability and better generalize to unseen test data, as proved in the ablation study, *i.e.* Table 3 of the main paper.

To elaborate the motivation, our goal during physics refinement is to quantitatively evaluate the stability of the hand and object pose estimated from a base network and refine the estimation stability. While modern physics simulators can serve as such an evaluator, due to the *complex contact geometry* and resulting noisy state gradient, the stability loss evaluated from the simulator can not be directly back-propagated to refine the base network. We therefore propose the DeepSim network to learn from the simulator, *i.e.* asked to replicate the same evaluated stability supervised by the simulator, *instead of being itself as a simulator*, while preserving smooth gradient that are suitable for back-propagation. We believe the design of the DeepSim shares more similarity to the *discriminator* in the GAN [15], however, it has no adversarial relationship to the base network (akin to the generator) and thus is much easier to train.

Meta-review: The paper presents a physics-aware solution for 3D hand-object pose estimation and originally obtains borderline to positive scores. The rebuttal has successfully addressed some of the reviewers' concerns. All reviewers gave positive scores in the end. The ACs concurred with the reviewers' ratings and chose to accept the paper. The authors are strongly encouraged to fulfill their promise of making necessary content edits when preparing the final version.
Decision: Accept (poster)