D-REX: Differentiable Real-to-Sim-to-Real Engine for Learning Dexterous Grasping
Authors: Haozhe Lou
Mingtong Zhang
Haoran Geng
Hanyang Zhou
Sicheng He
Zhiyuan Gao
Siheng Zhao
Jiageng Mao
Pieter Abbeel
Jitendra Malik
Daniel Seita
Yue Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Real-to-Sim-to-Real; Differentiable Simulation; Learning Robotic Policies from Videos; System Identification;
Primary area: applications to robotics, autonomy, planning
TL;DR: Differentiable Real-to-Sim-to-Real Engine for Learning Robotic Grasping

Abstract: Simulation provides a cost-effective and flexible platform for data generation and policy learning to develop robotic systems. However, bridging the gap between simulation and real-world dynamics remains a significant challenge, especially in physical parameter identification. In this work, we introduce a real-to-sim-to-real engine that leverages the Gaussian Splat representations to build a differentiable engine, enabling object mass identification from real-world visual observations and robot control signals, while enabling grasping policy learning simultaneously. Through optimizing the mass of the manipulated object, our method automatically builds high-fidelity and physically plausible digital twins. Additionally, we propose a novel approach to train force-aware grasping policies from limited data by transferring feasible human demonstrations into simulated robot demonstrations. Through comprehensive experiments, we demonstrate that our engine achieves accurate and robust performance in mass identification across various object geometries and mass values. Those optimized mass values facilitate force-aware policy learning, achieving superior and high performance in object grasping, effectively reducing the sim-to-real gap.

Published as a conference paper at ICLR 2026
D-REX: DIFFERENTIABLE REAL-TO-SIM-TO-REAL EN- GINE FOR LEARNING DEXTEROUS GRASPING
Haozhe Lou 1.2 Mingtong Zhang!.; Haoran Geng?, Hanyang Zhou², Sicheng He! Zhiyuan Gao 1.2 Siheng 7.2 Jjageng Mao Pieter Abbeel Jitendra Malik Daniel Seita?, Yue Wang' 1.2physical Superintelligence (PST) Lab, University of Southern California Viterbi School of Engineering, University of Southern California á DDeparmeent of EECS, University of California, Berkeley
ABSTRACT
Simulation provides cost-effcctive and flexible platform for data generation and policy learning to develop robotic systems. However, bridging the gap between simulation and real-world dynamics remains a significant challenge, especially physical parameter identification. In this work, introduce real-to-sim-to- engine that leverages the Gaussian Splat representations to build differentiable engine, enabling object mass identification from real-world visual observations and robot control signals, while enabling grasping policy learning simultancously. Through optimizing the mass the manipulated object, our method automatically builds high-fidelit and physically plausible digital twins. Additionally, we pro- pose novel approach to train force-award aorce- grasping policies from limited data by transferring feasible human demonstrations into simulated robot demonstrations. Through comprehensive experiments, we demonstrate that our engine achieves accurate and robust performance in mass identification across various object ge- ometries and mass values. Those optimized mass values facilitate force- aware policy learning, achieving superior and high performance in object grasping, ef- fectively reducing the sim-to-real gap. Our code and project page is available at drex.github..
Real-to-Sim
Differentiable Engine
Maso identiicationn
Learning Object Mass from Robot Videos
4D Gaussian Splats Rendering
Learning Robot Policy from Human Videos
Figure 1: We present D-REX, differentiable call-to-sim-to-ee engine that enables 4D photorealistic rendering and physical simulation by identifying object mass from real-world visual observations and robot interaction data. D-REX reconstructs object geometry using Gaussian Splat representations and leverages a differentiable physics engine for end-to-end mass identification. The identified mass then used to enable force-aware policy learning from human demonstrations, supporting robust grasping and sim-to-real transfer in dexterous grasping tasks.
INTRODUCTION
Simulation has become an essential platform for robotics, providing cost-effective and scalable plat- form that reduces the reliance on extensive robotics expertise. Through reusable and controlled data generation, simulation has driven significant advancements in accelerating policy learning Akkaya
"Equal contribution.

---

Published as a conference paper at ICLR 2026
et al. (2019); Hafner al. (2023); Chen et al. (2021); Agarwal et al. (2023); He et al. (2024); Ma et al. (2023). However, despite these benefits, replicating the visual realism and complex physical dynamics of the real world remains significant challenge. High-fidelity physical simulations often demand specialized knowledge and complex modeling, which limits the scalability and robustness of simulation-basse approaches for real-world deployment.
long line of research has focused on bridging the sim-to-real gap, which arises when transferring models trained in simulation to real-world configurations. This gap remains a fundamental challenge in robotics. fimulation-basee policies typically assume accurate knowledge and modeling of real- world configurations, including underlying physical parameters. However, differences between the estimated geometry and mass from visual observations and their real-world values increase the sim-to-real gap. Existing strategies to mitigate this gap include domain randomization Tobin et al. (2017); Sadeghi and Levine (2016); Peng et al. (2018a) 8a), which enhances robustness by varying simulation parameters, and system identification Hwangbo et al. (2019); Tan al. (2018); Khalil et al. (2007), which refines simulation dynamics by calibrating with real-world observations. Advances in simulation fidelity Ho et al. (2020); Mittal et al. (2023) and domain adaptation Bousmalis et al. (2018); Ren et al. (2023); Chen et al. (2023) have further facilitated the transfer of models from simulation to reality in robotics applications. Complementary to these efforts, real-to-sim frameworks attempt to construct digital twins that replicate real-world geometry and dynamics with high precision Chen et al. (2024); Torne et al. (2024); Jiang et al. (2022). Nonetheless, building accurate digital twins typically requires integrating multiple approaches, such as geometric reconstruction and parameter identification. Despite these advances, achieving precise modeling from visual observations remains challenging for current real-to-sim methods.
This challenge is fundamentally tied to the problem of system identifiattoon-inerring physical parameters from visual observations to ensure simulated environments faithfully reflect real-world dynamics. Estimating object attributes and system dynamics from images is difficult, even with full system state access. While robust forward simulators Macklin et al. (2014) exist, their non- differentiability limits applicability to inverse problems. Surrogate gradient methods such as finite differences are commonly used Cranmer et al. (2020); Ramos et al. (2019); Wu et al. (2017), but scale poorly in high-dimensional settings. Recent progress in differentiable simulation improves learning efficiency. In particular, GradSim Jatavallabhula et al. (2021a) enables end-to-end differentiation from visual observations to object-level physical parameters. Inspired by this, our work optimizes object mass directly from video, enabling force- force-aware grasping policy learning conditioned on mass and substantially improving performance.
To address these challenges, we introduce D-REX in this paper, a differentiable real-to-sim--o-real framework that builds our simulation engine upon differentiable simulation Jatavallabhula al. (2021a); Freeman et al. (2021); Müller et al. (2007); Macklin et al. (2016) and Gaussian Splat representations Kerbl et al. (2023). This differentiable engine enables object mass identification through visual observations and robot control signals in robot-object interactions. Additionally, we propose a novel learning-basse based method for dexterous manipulation, where we transfer human demonstrations into iimulation-execttabe robot demonstrations, then utilize the proposed method to optimize the grasp position and force simultaneously.
Our main contributions include:
A seal-to-sim-to-res framework that enables end-to-end object mass identification through differ- entiable simulation from visual observations and robotic control signals.
A novel approach to learn grasping policies from human demonstrations, conditioned on the identified object mass, that integrates position and force control to reduce the sim-to-real gap and achieve robust, nigh-performance grasping.
Empirically, we show that identifying accurate mass with our differentiable framework and condi- tioning the policy on it improve dexterous grasping challenging object.
RELATED WORKS
2.1 DIFFERENTIABLE PHYSICAL SIMULATION FOR ROBOTICS
The development of physical simulation enables efficient data generation and policy training for robotics Liu and Negrut (2021); Xu et al. (2021; 2022). Specifically, differentiable physical simula- tions have had great advancements recently, as they provide efficient gradients for policy learning. A popular approach is to develop physical simulation with automatic differentiable program-
2

---

Published as a conference paper at ICLR 2026
ming de Avila Sebbute-Peres al. (2018); Hu et al. (2019); Xu et al. (2022); Li et al. (2025a). Another line of work focuses on learning neural networks to approximate the real-world dynamics Li et al. (2019); Pfaff et al. (2020); Xian et al. (2021), which are inherently differentiable and suitable for applications in planning and control optimization. On the robotic application side, a variety of down- stream tasks have been studied: fluid manipulation Xian al. (2023), soft-hody manipulation Huang et al. (2021), cloth manipulation Peng et al. (2024); Yu et al. (2023), and the co-optimization of soft robot morphology and control policies Bhatia al. (2021): : Li al. (2023a); Li et al. (2025b). Notably, Jatavallabhula et al. (2021a) proposes to leverage differentiable multiphysics simulation for system identification from pixels. Our framework proposes a novel perspective to backpropagate the gradients obtained from visual observations for system identification through the differentiable approach, and enables dexterous manipulation policy learning from our real-to-sim results.
2.2 TEAM--OO-SM--O-RREE TRANSFER
Real-to-sim enables the replication of real-world assets and dynamics in simulation, enhancing data-driven insights, optimization, and robotic capabilities. By capturing natural statistics, dynamic behaviors, and kinematic structures, it supports robust decision-making, efficient model training, and evaluation of complex scenarios. Several recent works exemplify these trends. Jiang et al. (2022) creates interactive digital twins of articulated objects for simulation. Chen al. (2024); Mandi et al. (2024) generate articulated simulations from images, while Sundaresan et al. (2022) adapts parameters for deformable objects using point clouds. Neural Radiance Fields have also been applied to robotic tasks like manipulation and locomotion Kerr et al. (2022); Rashid et al. (2023); Zhou et al. (2023); Byravan et al. (2023); Wang et al. (2023), though often without accurate physical realism. More recent work Abou-Chakra et al. (2024); Zhang et al. (2024a); Kerr et al. (2024); Jiang et al. (2025); Zhobro et al. (2025); Abou-Chakra et al. (2025); Yang et al. (2025); Xie et al. (2023) leverages Gaussian Splats construct digital twins from real-world visual input. Pfaff et al. (2025); Khalil et al. (2007) identify robot and payload parameters via joint-torque sensing. In contrast to prior work Zhu al. (2025); Turpin al. (2022), which often lacks integration with differentiable real-to-sim-to-rean frameworks due to limitations in representation or simulation engines, our approach enables accurate object mass identification and policy learning in the simulation for direct real- world deployment.
2.3 LEARNING FROM HUMAN VIDEOS FOR ROBOTIC MANIPULATION
Human demonstration videos provide a scalable and semantically rich source for robotic manipula- tion. However, mapping human actions to robot control remains challenging due differences in embodiment and sensing. Prior work addresses this gap by leveraging pre-trained visual representa- tions Nair al. (2022); Radosavovic et al. (2023); Ma et al. (2022), or by extracting intermediate cues such as affordances Bahl et al. (2023) and object-centric flow Xu et al. (2024), which are hard to perform fine-grained and dexterous manipulation. Others focus on 3D human motion estimation for skill transfer Shaw et al. (2023a); Patel et al. (2022); Peng et al. (201 8b); Lum et al. (2025), which are often limited by the human and robot embodiment gap. Recent methods attempt to relax this constraint: Guzey et al. (2024) constructs reward functions from object tracking, and Singh et al. (2024) generates 3D hand-object trajectories from in-the-wild videos for retargeting. While promising, these approaches still struggle with generalizable policy learning from human videos. In contrast, our framework transfers human demonstrations into simulation -ex.utttoneeecuaalleeee.... robotic demonstrations, enabling scalable policy learning with improved adaptability conditioned on object mass to improve the performance.
PROBLEM STATEMENT
We focus on the real-to-sime caa--to-sim-t---e -io---tt---al task, which aims to construct simulation environment that closely mirrors real-world geometry, physics, and appearance. We assume access to several types of RGB videos: scene-centric video sequences, denoted as I,, which capture the static environment; object-centric videos, denoted as I,, which provide multiple views of the manipulated object to support accurate visual and geometric reconstruction; and human demonstration videos (I, )==1 which illustrate task exccution. We also extract object trajectories from real-world robot rollouts, 1/=1, and simulate corresponding trajectories [stim)7 ) within our framework.
denoted as (sq=)))
The scene and object videos (I,, Io) are used to initialize the real-to-sim process via visual and geometric reconstruction. Trajectories from both real and simulated rollouts enable mass identification through a differentiable engine, producing an optimized object mass m that is incorporated as a physical parameter in the simulator. Meanwhile, human demonstration videos {I, translated
3

---

Published as a conference paper at ICLR 2026
Reconstructio
Colliioon Mush
Goussian
Simulation Environment
Real-to-Sim Initialization
Human Demonstrations to Robot Demonstrations
rool
Gradient Flow
Force-Aware Oplimization
Learning
π
Differentiable Engino
Grasping Position
Objective
Aim (m)
Mass
Mass Identification through Differentiable Engine
Pollcy
Transforred Demonstrations
Figure 2: Overview of our method. Our approach consists of four components: (1) Real-to-Sim, (2) Mass Identification, (3) Learning from Human Demonstrations, and (4) Policy Learning. We begin by capturing videos of the scene and human demonstrations, Robotic actions are then executed in both simulation and the real world to identify object mass via our differentiable physics engine. Lastly, manipulation policy is trained using the demonstrations and identified mass.
into robot-executable trajectories {Ar] using our proposed method and are used to train force- aware manipulation policy π.
4 METHODS
We propose real-to- to--sim-to-ree framework that constructs accurate simulation environments and identifies object mass via system identification using differentiable engine from visual observations and robot control signals, enabling robust policy learning and sim-to-real transfer. The framework built on MuloCo Todorov et (2012), general-purposs, physies simulator, the differentiable engine Brax Freeman et al. (2021), and Gradsim Jatavallabhula et al. (2021a). It operates in four steps shown in Figure 2: Real-to-Sie Sim (Section 4) Mass Identification (Section 4.2), Learning from Human Demonstrations (Section 4.3), and Policy Learning (Section 4.4). First, the scene and object are reconstructed from RGB videos I, and Ig, capturing static environments and target objects. The output simulation is formalized as S = {AC, Q} in MJCF format, where K. denotes collision meshes and the physical parameters. Next, the framework executes consistent robotic actions in simulation )
and the real world, collecting (s(cal] Jt= and {s to identify object mass m. Third, human demonstrations {I, )Z))²111 are translated into robot-executable trajectories (A)) Finally, these trajectories used to train policy in simulation, which is then deployed in the real world directly.
4.1 VISUAL AND GEOMETRIC RECONSTRUCTION
Our framework starts with reconstructing high-fidelity visual and geometric models of key elements in the manipulation environment with I., including objects, robotic arms, dexterous hands and workspaces. This reconstruction ensures accurate representations of both collision geometry and visual appearance. To integrate these models into differentiable simulation, Splat representation Kerbl et al. (2023); Huang et al. (2021), which enables photorealistic rendering and high-qualit mesh generation for collision detection following Lou et al. (2024). Specifically, process videos collected from mobile devices to train two ensembles of Gaussian primitives: one for collision geometry and another for visual appearance. Specifically, 2D Gaussian Splats with surface normal estimation Ye et al. (2024a) provide accurate geometry for simulation, while 3D Gaussian Splats ensure high-fidelity rendering. This process yields two complementary outputs: collision mesh K. and Gaussian particles P. Additional details are provided in the Appendix.
adopt the Gaussian
4.2 PHYSICAL PARAMETER IDENTIFICATION FROM ROBOT-OBJES INTERACTIONS
Accurate identification of physical parameters 0 is essential for constructing physically plausible simulations. We begin by using a Vision-Language Model Hurst et al. (2024) to generate an initial MJCF representation S from environment images and prompts Zhang al. (2024b). While this approach provides a reasonable structural prior, parameters inferred solely from visual inputs often deviate from real-world values due the lack of observable physical cues Asenov al. (2020).
4

---

Published as a conference paper at ICLR 2026
To address this, we focus identifying the object mass, which is key parameter in dynamics that can be reliably measured. Accurate mass identification improves simulation fidelity and enables robust policy learning. We choose planar pushing task with virtual fulerum assumption to reduce frictional effects, and optimize mass v to minimize the discrepancy between simulated and real-world trajectories Jatavallabhula et al. (2021a):
min (m) m>0
8 (m) Σ (m))mm))))
is
(I)
where 8 [p. TR? denotes the object's 6-DoF pose, consisting of position p E RR3 and orientation represented unit quaternion q RR sTaa is obtained by FoundationPose Wen et al. (2024) in the real world while si (m) is obtained by executing the same actions in the simulation. Dynamics Modeling. To simulate object motion, we adopt standard rigid-body formulation of the Newton-Eules mechanism. Let u, Vv, denote the object's ' velocity at timestep t, where Vi and w/ are the linear and angular velocity components, respectively, We express the governing equation as the second order differential equation(ODE) Chen et al. (2019):
M(s:, Ur, m, 0) úi [(St, Ur 0),
where M is the mass-inertia matrix Baraff (1992) and f collects external and contact forces equation 3, gravity and torques. We adopt a compliant penalty- baaed contact model, parameterized by stiffness and damping (ke: ka) E 0, which applies normal forces proportional to penetration depth and contact velocity Todorov et al. (2012); Erez et al. (2015):
f, (s. U, 0) = - (ke ((s) + kia (())),
where f, is the contact force, is the contact normal, C(ss) the penetration depth, and C(u) is the derivative of C(s). This contact model is differentiable and readily integrated into our simulation framework. In practice, implement the dynamics using a discrete-time update Erez al. (2015):
S(1 G(s, ur: m. ft); t= 0.....,
f, are external forces at timestep t, including actuator impulses, gravity, and abject-ground contacts. Differentiable Physics. To optimize equation 1, we compute gradients of the simulated trajectory with respect to the object mass m. Following the discrete adjoint method from Jatavallabhula et al. (2021a), we adopt a semi- cemi-implicit Euler integration scheme for stability under contact dynamies. We couple kinematics from MjX/Brax Freeman et al. (2021) with rigid-body dynamics equation 2 and the contact model equation 3, forming a differentiable computation graph Hu al. (2020).
Semi-Implicit Euler Modeling. The update function G(-) in equation is implemented using semi-implicit Fuler integration scheme:
= 81 + A A141
ss Al ArM u, m. 0) (81, mi)) A m, 0) f(s u.)
G [ss, Uq m, 8)
=
U++1
(5)
where At is the integration timestep. and f()) encapsulates both external and contact forces. Differentiable Real-to-Sim-to-Rer Optimization. condition sim. and iteratively update the state via equation 4. To quantify the discrepancy between simulated and real-world trajectories, we define the trajectory loss between the simulated state spin and the corresponding real-world state steal
simulate the system starting from the initial
Curjj (m) = Σ |
real
(6)
This objective encourages the simulated trajectory, parameterized by mass m, to closely match the observed real-world dynamics time. The gradient V (m) is computed via automatic differentiation, using hackpropagation as implemented in PyToorch Paszke et al. (2019), as follows:
dCuai 0m.
T HLuij Jspim
as
iM,
dm
5

---

Published as a conference paper at ICLR 2026
Unlike system identification methods such GradSim Jatavallabhula et al. (2021a), which rely on manually specified external forces, our approach supports end-to-end optimization by directly leveraging consistent robotic control signals in both simulation and the real world to model the external forces applied to the object. This creates a tight coupling between real-world and simulated trajectories, enabling to capture contact dynamics through robot-object interactions.
Importantly, our method does not require ground-truth object mass or contact points. Object geometry and poses are obtained via Section 4.1, while actuator signals and robot-object interactions derived from the MJCF kinematic model. These serve as inputs to our differentiable framework for accurate mass optimization. Additional modeling details and experiments are provided in the Appendix.
4.3 TRANSFERRING HUMAN DEMONSTRATIONS TO ROBOT DEMONSTRATIONS
After accurately modeling the scene and object, the next step is to collect real-world human demon- strations and transfer them into robot demonstrations for policy learning. Although learning directly from human demonstrations is intuitive, substantial differences between human and robotic hands complicate grasp interaction transfer, particularly due to varied object geometries and masses.
Our approach aims transform human demonstrations captured from RGB video sequences {Ie]T= into executable robotic demonstrations within the simulation. Each video frame I, is processed using HaMeR Pavlakos al. (2024) and MCC-HO Wu al. (2024a) to reconstruct detailed articulated models of the human hand and the manipulated object. At each timestep f, these methods output:
h, E SE(3) x " O, E SE(3),
where h, encodes the 6-DoF wrist pose and finger joint angles (Jh), and o, describes the object's 6-DoF pose. Subsequently, we employ Dex-Retargetiin Retargeting Qin et al. (2023) to map these human hand-object poses h, O, the robotic hand with J, degrees of freedom. This produces robot actions:
A: = R(,,, 0.) E R
where A, represents target joint angles for robotic actuators. Given our assumption that the object geometry remains consistent between human demonstration and robotic manipulation, the resulting action set A, directly serves as a data source for our policy learning.
4.4 POLICY LEARNING WITH TRANSFERRED ROBOT DEMONSTRATIONS
We initialize policy learning using robot demonstrations {A,).- described in Section 4.3. Each demonstration maps the reconstructed object's 'ss collision mesh vertices as inputs to the corresponding robotic grasp pose. These observation-action pairs directly supervise training of the manipulation policy T.. which maps object-centric observations to dexterous grasp configurations.
Grasping Position Learning. To capture an object's geometry and pose, we encode the vertices of its collision mesh K. using positional encoding Tancik et al. (2020), forming the input to our policy, This policy conditions the observation o on the reconstructed collision mesh and the identified mass Concretely, Tg is a multi-head neural network that predicts dexterous hand joint positions A, contact-related rewards i, and a mass-related control force f.
A
(o)
f E 19
= Mlctive
m
(10)
where A E 16 denotes the predicted joint positions, f E R² represents the contact constraint, and f E R denotes the grasping force constraint. The variable nautive indicates the number of active contacts between the robotic hand and the object. The network uses fully connected layers with ReLU activations, followed by pooling layer. More details are in the Appendix.
Force-Aware Optimization Design. At training onset, we define a two-dimensional contact con- straint f: one term encourages sustained contact during the rollout, the other ensures object retention at the end. The policy dynamically influences this constraint based on hand-object interactions
through our simulation engine and the simulation asset S from Section 4.1 and 4.2. It remains high when active contact points exceed a threshold Nui over the time horizon H:
[1, if rlactive (1) 2 1,
Vt [to, to + H]: Resetive 2 Nuin- (and
(11)
0, otherwise.

---

Published as a conference paper at ICLR 2026
We subsequently retrain the manipulation policy with the force-based constraint in 10, enabling adaptive force control that responds to object mass variations Vassiliadis et al. (2021); Zhang et al. (2025). This enhances the robustness of grasp poses learned from demonstrations, ensuring stability under diverse dynamics.
Traditional position-based policies replicate grasp poses from human demonstrations Patel al. (2022); Lum et al. (2025); Chen et al. (2025a); Wan et al. (2023); Wei et al. (2025); Shaw et al. (2022) but overlook unobserved forces, particularly those countering gravity. Applying uniform forces across varying object masses often leads to instability. To address this, we propose hybrid control framework that combines position and force control, using a prediction module conditioned on the optimized mass m. This jointly optimizes the policy parameters 6 and grasping force, enabling more robust and physically grounded manipulation.
EXPERIMENTS
The objective our experiments to evaluate the performance of our system across following key aspects: (1) Evaluate the effectiveness and robustness of mass identification using the differentiable engine across varying object geometries, densities, and categories. (2) Analyze how incorporating object mass affects policy learning performance, assessing the fcasibility of force-based control. (3) Assess the effectiveness of learning grasping policies from transferred robot demonstrations and their direct sim-to-real deployment.
5.1 MASS IDENTIFICATION VIA OBJECT PUSHING
We evaluate mass identification method through object pushing experiments by applying identical actions in both the real world and simulation. The resulting trajectories are used to optimize object mass via our differentiable engine, detailed in Section 4.2. We assess the performance in two settings: (1) across objects with varying geometries, and (2) across replicas with identical geometry but different internal densities. Our method accurately recovers mass in both cases, demonstrating generalization to diverse shapes and sensitivity to subtle physical differences. The objects we used for mass identification are shown in Figure
728g
2105
134g
106g
59g
829
1259
218g Bg
Figure 3: Objects for Mass Identification. We conduct experiments mass identification across diverse object geometries and identical geometries with varying densities. Our method accurately estimates mass in both settings, demonstrating robustness shape and density variations.
To evaluate robustness across shapes, sizes, and mass scales, we select diverse set of objects for mass identification. This tests the pipeline's 's ability to generalize across varying contact geometries. As shown in Table 1, percentile eтrors range from 4.8% to 12.0%, demonstrating accurate mass optimization without object-specific tuning.
To isolate the effect of mass, fabricate three replicas with identical geometry but varying internal densities p using different 3D printing infill ratios. By keeping shape constant, any identification error reflects mass sensitivity. As shown in Table 2, mass is accurately identified with deviations under 13 grams, confirming the effectiveness of estimating physical parameters independent of geometry.
As shown in Figure simulations using the optimized mass closely match real-world object dynamics, while those using an incorrect lighter mass deviate significantly. This demonstrates that accurate mass identification improves both the physical realism and visual quality of simulated rollouts.
Object
Letter
Letter A
Lego
Domino
Cookie
Ketchup
Inforee Mass (g)
500
500
300
500
500
1000
Density
P1
Pa
Pa
Idenilied Muss
110
145
53
117
200
667
Identified Mass (g)
95
129
207
Ground Trush
Ground Truth Mass (g)
82
125
218
Perventile Error (%)
9.0
86
Table 2: Mass identification across ob-
Table 1: Mass identification across diverse objects with varying shapes, sizes, and mass seales. The inferred mass is obtained from VLM described in Section 4.2.
jects with identical geometry but vary- ing densities.

---

Published as a conference paper at ICLR 2026
Figure Quantitative Results of Mass Identification. We show the real world object pushing (top) and render object trajectories using Gaussian Splats: simulated with optimized mass (middle), and simulated with a lighter mass (bottom), using the same robot actions. The optimized mass closely reproduces real-world dynamics, reducing the sim-and-real gap with high visual fidelity.
Train Eval
P1
P2
Ps
Success Rate Object Mass
75%
30%
15%
Success Rate (16)
P2
40%
80 %
30%
75
Ps
15%
40%
95%
Dane
Table 3: Cross-evaluation of grasping poli- cies trained on different object densities and evaluated across varying masses. Each cell shows the grasp success rates. Policies perform well only when the training and evaluation masses match.
Crene of
Mass (g)
Figure 5: Grasping success rates across three ob- jects with different mass values.
These experiments demonstrate that our differentiable Real-to-Sim--t-Reaa framework achieves accurate mass identification across both inter-object diversity and intra-object density variations.
5.2 EFFECTIVENESS OF FORCE-BASED CONTROL THROOGHI GRASPING
In our grasping experiments, we evaluate how incorporating force-based constraints conditioned on object mass influences sim-to-real performance. This setup highlights the need for mass-awar force control and demonstrates the impact of accurate mass identification policy success.
We first evaluate our grasping policy on three objects that share identical geometry and demonstrations but differ in mass. Each policy is trained with a specific object mass to assess the impact of mass- aware force control. As shown in Figure 6, policies perform well only when training and evaluation masses matched: the medium-mass policy succeeds on the medium object but fails the heavier and lighter ones due to under- and over- apr-aep force, respectively. Mass mismatches likewise lead to unstable grasps for the other two policies. Table 3 confirms this trend, with the highest success rate (80%) on the training mass, while performance drops to 40% and 30% on mismatched cases. These results highlight the importance of accurate conditioning for robust, reliable grasping.
8

---

Published as a conference paper at ICLR 2026
Figure 6: Qualitative Results. Left to right: policies trained on medium, light, and heavy objects. Only the mass-matched policy achieves stable grasps, while mismatched ones fail due to excessive or insufficient force, causing bounce-off for lighter objects slippage for heavier ones.
Second, we evaluate whether policies conditioned on automatically identified mass can match the performance of those trained with object mass. As shown in Figure success rates consistently peak at either the ground-truth or identified mass. Notably, policies using identified mass often match or even exceed those using ground-truth values, while substantially outperforming policies conditioned on arbitrary masses. These results underscore the effectiveness of our mass identification approach in enabling robust, force-aware grasping without requiring access to true mass values.
These experiments demonstrate that accurate mass is essential for effective force-aware grasping. Policies trained with object mass consistently outperform those trained on mismatched masses, and policies conditioned on automatically identified mass achieve comparable performance to those using ground- truth values. Together, these results validate our mass identification framework as a practical and reliable solution for enabling robust grasping without prior knowledge of object mass.
5.3 TABLETOP OBJECT GRASPING EXPERIMENTS
We compare our grasping policy against two baselines across various objects: (1) Dex GraspNet 2.0 Zhang et al. (2024c), trained on large-scale simulation datasets, and (2) Human2Sim2Robot Lum et al. (2025), a recent real-to-sim-to-real method that learns dexterous manipulation policies from RGBD vidcos of human demonstrations. All use the collision mesh K. generated by real-to-sim framework as input. As shown in Figure 7, our method consistently outperforms the baselines across eight objects with diverse geometries and masses, achieving high success rates with substantially lower variance. While baseline performance degrades as object mass increases, our force-aware policy maintains stable, reliable grasps across the full range of object characteristics.
DexGraspNet 2.0
Human2Sim2Robot
DREAM
Success (%)
100
80
80
75
80
85
70
66 60
60
45
40
10
115
10
20
Lightbulb 37g
Cube 120g
A 134g
Cookie 210g
Spam 365g
Nutella 414g
Spray 548g
Ketchup 726g
Average
Figure 7: Quantitative Results of Grasping Policies. Grasp success rates across eight objects with varying geometries and mass values, with the average and standard deviation each method.
Figure 8 presents qualitative results of our force-aware grasping policy across a range of objects. The top row captures the motion leading to the gre-grasp pose, while the bottom row displays the resulting post- grasp configurations. These examples demonstrate the policy's 's ability to consistently achieve stable and secure grasps under varying object geometries and mass values.

---

Published as a conference paper at ICLR 2026
Object
Images
COLMAP43DGSS+2DDDS runtime
Mesh vertices
Lego
306
30 min
47,266
Ketchup
337
35 min
13,487
Domino
320
30 min
35,445
Cookie
309
30 min
25,682
306
30 min
67,895
U
327
33 min
15,322
Table 4: Offline reconstruction statistics for representative objects, including the number of input images, total runtime, and the number of vertices the extracted mesh.
Figure 8: Qualitative Results of Our Policy. We evaluate our force-aware grasping policy across various objects. The first row illustrates the approach the pre-grasp pose, while second row shows two post-grasp positions, demonstrating that the policy achieves stable, secure grasps.
5.4 RUNTIME ANALYSIS
We next analyze the computational cost our Real-to- to-Sim reconstruction and mass identification pipeline. For each object, we capture approximately 300-340 RGB images. The end-to-end offline Strrrtoonnnstuuuuu---rrrooo
via COLMAP et (2024); Schönberger and Frahm (2016) followed by 3DGS and 2DGS -apiiallyyyy requires 30-35 minutes per object Representative statistics are reported in Table 4. Despite mesh complexity varying from 1.3 x 104 to ~ 6.8 x 10t vertices, reconstruction time remains within narrow range across objects.
device.
Mass identification via differentiable physics scales with the number mesh vertices. On our hardware, each training iteration takes approximately 1.43-1.68 s, and convergence is typically achieved within about 200 epochs. In practice, this corresponds to 5-20 minutes for most of the objects listed in Table 4. All reconstruction and mass-identification steps performed offline and thus do not affect the real-time execution of the learned grasping policy.
CONCLUSION
D-REX is real-to-sim-to-ret framework that leverages differentiable simulation to create visually realistic and physically accurate digital twins from visual observations and robot control signals, enabling robust dexterous grasping policies. Through identifying object mass through robot-object interactions, achieves generalization across diverse object shapes and densities. Furthermore, integrating force-award control conditioned on mass into imitation learning enhances policy robustness and adaptability, thus offering promising potential for scalable data generation and the development of generalizable policies, representing a significant step toward robust real-worls robotic systems.
7 REPRODUCIBILITY STATEMENT
We rely on several open- open-source foundation models. The implementation details necessary to repro- duce experiments provided in the Appendix and Supplementary Material. For our primary contributions leaabbiioon--laainng mass from video and the force force-position hybrid policy-we we include the codebase and dataset in the Supplementary Material and our website.
10

---

Published as a conference paper at ICLR 2026
8 ETHICS STATEMENT
This work adheres to the ICLR Code of Ethics; all authors have read and acknowledge the Code as part of the submission process. Where our research involved human annotators or user data, we obtained approval or documented exemption from an institutional ethics review board and informed consent, used only data collected with permission, applied de-identification, and restricted access; otherwise we relied on publicly available datasets under their licenses. We assessed potential risks fisksiincluding privacy, security, nias/dssrimination, dual-use or misuse, and environmental limitations, failure modes, and mitigations to minimize harm. We also disclose any use of large language models (LLMs) in ideation, coding, or writing and accept full responsibility for all content; LLMs not authors.
ACKNOWLEDGEMENT
The USC Physical Superintelligence Lab acknowledges generous supports from Toyota Research Institute, Dolby, Google DeepMind, Capital One, Nvidia, Bosch, NSF, and Qualcomm. Yue Wang is also supported by Powell Research Award.
This work benefited from discussions with Professor Stephen Tu, Dr. Quankai Gao, and Dr. Yuming Gu.
REFERENCES
Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, et al. Solving rubik's cube with robot hand. arXiv preprint arXiv:1910.071I. 13, 2019.
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.0411 2023.
Tao Chen, Jie Xu, and Pulkit Agrawal. system for general in-hand object re-orientation, 2021. 03043.
URL http:/awwwvv..a...aaaaaaaa/oo/ae
Ananye Agarwal, Ashish Kumar, Jitendra Malik, and Deepak Pathak. Legged locomotion in chal- lenging terrains using egocentric vision. In Conference on robot learning, pages 403-415. PMLR, 2023.
Tairan He, Chong Zhang, Wenli Xiao, Guanqi He, Changliu Liu, and Guanya Shi. Agile but safe: Learning collision-free high-speed legged locomotion. In Robotics: Science and Systems (RSS), 2024.
Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Eureka: Human-level reward design via coding large language models. arXiv preprint arXiv: Arxiv-2310. /2931, 2023.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 JEEE/RSJ international conference on intelligent robots and systems (IROS), pages 23-30. IEEE, 2017.
Fereshteh Sadeghi and Sergey Levine. Cad2rl: Real single-imag flight without a single real image. arXiv preprini arXiv: wee.::.11....2., 2016.
Xue Bin Peng. Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE international conference robotics and automation (ICRA), pages 3803-3810. IEEE, 2018a.
Jemin Hwangbo, Joonho Lee, Alexey Dosovitskiy, Dario Bellicoso, Vassilios Tsounis, Vladlen Koltun, and Marco Hutter. Learning agile and dynamic motor skills for legged robots. Science Robotics, 4(26)eaau5872, 2019.
11

---

Published as a conference paper at ICLR 2026
Jie Tan, Tingnan Zhang, Erwin Coumans, Atil Iscen, Yunfei Bai, Danijar Hafner, Steven Bohez, and Vincent Vanhoucke. Sim-to-real: Learning agile locomotion for quadruped robots. arXiv preprint arXiv:1804. 10332, 2018.
Wisama Khalil, Maxime Gautier, and Philippe Lemoine. Identification of the payload inertial parameters of industrial manipulators. In Proceedings 2007 IEEE International Conference on Rebolics and Automation, pages 4943-4948, 2007. doi: 10. 12110900BT02200.3442222
Daniel Ho, Kanishka Rao, Zhuo Xu, Eric Jang, Mohi Khansari, and Yunfei Bai. Retinagan: An object-aware approach to sim-to-real transfer, 2020. URL https: www.a/wwwww..///////////// 03148.
Mayank Mittal, Calvin Yu, Qinxi Yu, Jingzhou Liu, Nikita Rudin, David Hoeller, Jia Lin Yuan, Ritvik Singh, Yunrong Guo, Hammad Mazhar, Ajay Mandlekar, Buck Babich, Gavriel State, Marco Hutter, and Animesh Garg. Orbit: A unified simulation framework for interactive robot learning environments. IEEE Robotics and Automation Letters, 86))3740-33777, 2023, doi: 10.100//
Konstantinos Bousmalis, Alex Irpan, Paul Wohlhart, Yunfei Bai, Matthew Kelcey, Mrinal Kalakrish- nan, Laura Downs, Julian Ibarz, Peter Pastor, Kurt Konolige, et al. Using simulation and domain adaptation to improve efficiency of deep robotic grasping. In 2018 JEEE international conference on robotics and automation (ICRA), pages 4243-4250. IEEE, 2018.
Allen Z Ren, Hongkai Dai, Benjamin Burchfiel, and Anirudha Majumdar. Adaptsim: Task-driven simulation adaptation for sim-to-real transfer. arXiv preprint arXiv:2302.04903 2023.
Tao Chen, Megha Tippur, Siyang Wu, Vikash Kumar, Edward Adelson, and Pulkit Agrawal. Visual dexterity: In-hand dexterous manipulation from depth. In Ieml workshop on new frontiers in learning, control, and dynamical systems, 2023.
Zoey Chen, Aaron Walsman, Marius Memmel, Kaichun Mo, Alex Fang, Karthikeya Vemuri, Alan Wu, Dieter Fox, and Abhishek Gupta. Urdformer: A pipeline for constructing articulated simulation environments from real-world images. arXiv preprint arXiv:2405.11666, 2024.
Marcel Torne, Anthony Simeonov, Zechu Li, April Chan, Tao Chen, Abhishek Gupta, and Pulkit Agrawal. Reconciling reality through simulation: A real-to-sim-to-rea approach for robust manipulation. arXiv preprint arXiv:2403.03944 2024.
Zhenyu Jiang, Cheng-Chun Hsu, and Yuke Zhu. Ditto: Building digital twins of articulated objects from interaction. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Miles Macklin, Matthias Müller, Nuttapong Chentanez, and Tae- Yong Kim. Unified particle physies for real-time applications. ACM Trans. Graph., 33(4), July 2014. ISSN 0730-0301. doi: 10. 2601097.2601 152. URL https: / /ow. org/ /10 12 2004////2609990 2601152.
Kyle Cranmer, Johann Brehmer, and Gilles Louppe. The frontier of simulation-baal inference. Proceedings of the National Academy of Sciences, 1 117(49)30055-33066 2020.
Fabio Ramos, Rafael Carvalhaes Possas, and Dieter Fox. Bayessim: adaptive domain randomization via probabilistic inference for robotics simulators. arXiv preprint arXiv:1906.01 01728, 2019.
Jiajun Wu, Joshua B Tenenbaum, and Pushmeet Kohli. Neural Scene De-rendering. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Krishna Murthy Jatavallabhula, Miles Macklin, Florian Golemo, Vikram Voleti, Linda Petrini, Martin Weiss, Breandan Considine, Jerome Parent-Levesque, Kevin Xie, Kenny Erleben, Liam Paull, Florian Shkurti, Derek Nowrouzezahrai, and Sanja Fidler. gradsim: Differentiable simulation for system identification and visuomotor control, 2021a. URL ptetps://ariiv... http///xxvv..o/ obs/ 21 04 6.
C. Daniel Freeman, Erik Frey, Anton Raichuk. Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax a differenniable physics engine for large scale rigid body simulation. 2021. URL http: //github com googleoooleeeooooooooooooooooooo e/brax.
12

---

Published as a conference paper at ICLR 2026
Matthias Müller, Bruno Heidelberger, Marcus Hennix, and John Ratcliff. Position based dynamics. Journal of Visual Communicattion and Image Representation, 18(2):09--118, 2007. ISSN 1047- 3203. doi: http://dbo.ag////..000diivvviooo
.05. URL httpp::: /www. sciencedirett..
com/ com/scieneee http://cannn/aaaa.aaaaa/////////
Miles Macklin, Matthias Müller, and Nuttapong Chentanez. Xpbd: position-based simulation of compliant constrained dynamies. In Proceedings of the 9th International Conference Motion in Games, MIG 16. page 49-54, New York. NY. USA, 2016. Association for Computing Machinery. ISBN 9781450345927. doi: 10. N0.1145/2994253 21144//9999228229922222 Uttpstps: //tttp:://doo. wrt./0..111111..... 22....999999 2994272.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph., 42(4):139-1, 2023.
C. Karen Liu and Dan Negrut. The role of physics-based simulators in robotics. An- mual Review of Control, Robotics, and Autonomous Systems, 4(Volume 4, 2021):35-58, 2021. ISSN 2573-5144. doi: URL https: wwtp:/wwwwwwwwwwwwwwwaeaaa. orgg//ooteent//ouraal///// amnueer-contt22-022220--99--
Jie Xu, Tao Chen, Lara Zlokapa, Michael Foshey, Wojeiech Matusik, Shinjiro Sueda, and Pulkit Agrawal. An end-to-end differentiable framework for contact-aware robot design. arXiv preprint arXiv:2107.07501 2021.
Jie Xu, Viktor Makoviychuk, Yashraj Narang, Fabio Ramos, Wojeiech Matusik, Animesh Garg, and Miles Macklin. Accelerated policy learning with parallel diffcrentiable simulation. arXiv preprint vrXiv:2204.07137 2022.
Filipe de Avila Belbute-Peres, Kevin Smith, Kelsey Allen, Josh Tenenbaum, and J. Zico Kolter. End-to-end differentiable physics for learning and control. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Ad- vances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL https: https::/ http://poooeeeenn.e.eee/ eweeuips.cc/paate file/ 82228442224000
sallee/paper/22111
Yuanming Hu, Luke Anderson, Tzu-Mao Li, Qi Sun, Nathan Carr, Jonathan Ragan-Kelley, and Frédo Durand. Difftaichi: Differentiable programming for physical simulation. arXiv preprint arXiv:1910.00935 2019.
Wenxuan Li, Hang Zhao, Zhiyuan Yu, Du, Qin Zou, Ruizhen Hu, and Kai Xu. Pin-wm: Learning physics-informed world models for non-prehensile manipulation, 2025a. http::::
//arxiv. www.ivi.orraaaoooo 2254 166933
Yunzhu Li, Jiajun Wu, Russ Tedrake, Joshua B Tenenbaum, and Antonio Torralba. Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids. In ICLR, 2019.
Tobias Pfaff, Meire Fortunato, Alvaro Sanchez-Gonzalez, and Peter Battaglia. Learning mesh-based simulation with graph networks. In International conference on learning representations, 2020.
Zhou Xian, Shamit Lal, Hsiao-Y Yu Tung, Emmanouil Antonios Platanios, and Katerina Fragkiadaki. Hyperdynamics: Meta-learning object and agent dynamics with hypernetworks. arXiv preprint arXiv:2103.09439, 2021.
Zhou Xian, Bo Zhu, Zhenjia Xu, Hsiao- Yu Tung, Antonio Torralba, Katerina Fragkiadaki, and Chuang Gan. Fluidlab: A differentiable environment for benchmarking complex fluid manipulation. In International Conference on Learning Representations, 2023.
Zhiao Huang, Yuanming Hu, Tao Du, Siyuan Zhou, Hao Su, Joshua B Tenenbaum, and Chuang Gan. Plasticinelab: A soft-body manipulation benchmark with differentiable physics. arXiv preprint TrXiv:2104.03311 2021.
Weikun Peng, Jun Lv. Yuwei Zeng. Haonan Chen, Siheng Zhao, Jichen Sun, Cewu Lu, and Lin Shao. Tiebot: Learning to knot a tie from visual demonstration through a real-to-sim- -i----i-t---eltt approach. arXiv preprint -Xix:2407.03444 2024.
13

---

Published as a conference paper at ICLR 2026
Xinyuan Yu, Siheng Zhao, Siyuan Luo, Gang Yang, and Lin Shao. Diffclothai: Differentiable cloth simulation with intersection free frictional contact and differentiable two-way coupling with articulated rigid bodies. In 2023 JEEE/RSJ International Conference Intelligent Robots and doi: 10.1 10110000555222223.10055 10341557.
Systems (IROS), pages 400-407, 2023.
Jagdeep Bhatia, Holly Jackson, Yunsheng Tian, Jie Xu, and Wojciech Matusik. Evolution gym: large-scale benchmark for evolving soft robots. Advances in Neural Information Processing Systems, 34:2201-2214 2021.
Sizhe Li, Zhiao Huang, Tao Chen, Tao Du, Hao Su, Joshua B. Tenenbaum, and Chuang Gan. Dexde- form: Dexterous deformable object manipulation with human demonstrations and differentiable physics, 2023a. URL https: arxiv. org/ abs 3044. 03223.
Sizhe Lester Li, Annan Zhang, Boyuan Chen, Hanna Matusik, Chao Liu, Daniela Rus, and Vincent Sitzmann. Controlling diverse robots by inferring jacobian fields with deep networks. Nature, 643(8070):89 95, June 2025b. ISSN 1476-4687. http://dx: doi wttp//...//.................o1./ /1000///41150 w11///11155600 0001118860255--0 000005--255911111 09100-00
URL doi: 10. (1888//4158-025--000
Zhao Mandi, Yijia Weng, Dominik Bauer, and Shuran Song. RealZcode: Reconstruct articulated objects via code generation. arXiv preprint wrXiv:2406.08474, 2024.
Priya Sundaresan, Rika Antonova, and Jeannette Bohgl. Diffcloud: Real-to-sim from point clouds with differentiable simulation and rendering of deformable objects. In 2022 TEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 10828-10835. IEEE, 2022.
Justin Kerr, Letian Fu, Huang Huang, Yahav Avigal. Matthew Tancik, Jeffrey Ichnowski, Angjoo Kanazawa, and Ken Goldberg. Evo-neRF: Evolving neRF for sequential robot grasping of transparent objects. In 6th Annual Conference on Robot Learning, 2022. URL https:// openreview wtt.llewwwweeeeeeeee forum Bxr SkeYrf.
Adam Rashid, Satvik Sharma, Chung Min Kim, Justin Kerr, Lawrence Yunliang Chen, Angjoo Kanazawa, and Ken Goldberg. Language embedded radiance fields for zero-shot task-oriented grasping. In 7th Annual Conference on Robot Learning, 2023. URL https: /ttpa/////ppeeeeeeeeee net/forum? ?ink FFgBmm--
Allan Zhou, Moo Jin Kim, Lirui Wang, Pete Florence, and Chelsea Finn.
Nerf in the palm of
your hand: Corrective augmentation for robotics via novel-view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17907-17917, 2023.
Arunkumar Byravan, Jan Humplik, Leonard Hasenclever, Arthur Brussee, Francesco Nori, Tuomas Haarnoja, Ben Moran, Steven Bohez, Fereshtch Sadeghi, Bojan Vujatovic, et al. Nerf2real: Sim2real transfer of vision-guided bipedal motion skills using neural radiance fields. In 2023 IEEE International Conference Robotics and Automation (ICRA), pages 9362 2369. IEEE, 2023.
Luobin Wang, Runlin Guo, Quan Vuong, Yuzhe Qin, Hao Su, and Henrik Christensen. real2sim2real method for robust object grasping with neural surface reconstruction, 2023. URL https::/aarktt org/ als 22100202. 02685.
Jad Abou-Chakra, Krishan Rana, Feras Dayoub, and Niko Suenderhauf. Physically embodied gaussian splatting: A realtime correctable world model for roboties. In 8th Annual Conference Robot Learning, 2024. URL http::/ http///pppeeeeeeeeeeee/ee/ee net / forum? ? -eee:idd-Eeosee AEn0OGGG.2
Mingtong Zhang, Kaifeng Zhang, and Yunzhu Li. Dynamic 3d gaussian tracking for graph- neural dynamics modeling, arXiv preprint arXiv:2410. 189/2, 2024a.
Justin Kerr, Chung Min Kim, Mingxuan Wu, Brent Yi, Qianqian Wang, Ken Goldberg, and Angjoo Kanazawa. Robot see robot do: Imitating articulated object manipulation with monoc- ular 4d reconstruction. In Sth Annual Conference on Robot Learning, 2024. URL https: // //openrevveww.. net forum? iineddd=211 d=2LLuugavF1.
Hanxiao Jiang, Hao- Yu Hsu, Kaifeng Zhang, Hsin-Ni Yu, Shenlong Wang, and Yunzhu Li. Phystwin: reconstruction and simulation of deformable objects from videos. arXiy preprint arXiv:2503. 17973, 2025.
14

---

Published as a conference paper at ICLR 2026
Mikel Zhobro, Andreas René Geist, and Georg Martius. Learning 3d gagssiasii simulators from rgb videos. arXiv preprint arXiv:2503. 2025.
Jad Abou-Chakra, Lingfeng Sun, Krishan Rana, Brandon May, Karl Schmeckpeper, Maria Vittoria Minniti, and Laura Herlant. Real-is-sim: Bridging the sim-to--rel gap with dynamic digital twin for real-world robot policy evaluation. arXiv preprint arXiv:2504.03597, 2025.
Sizhe Yang, Wenye Yu, Jia Zeng, Jun Lv, Kerui Ren, Cewu Lu, Dahua Lin, and Jiangmiao Pang. Novel demonstration generation with gaussian splatting enables robust one-shot manipulation, 2025. URL https::////.. /awxivv.aiiiv.a//aa/iaa orq/ 2abs/2504 13175.
Tinyi Xie, Zeshun Zong, Yuxing Qiu, Xuan Li, Yutao Feng, Yin Yang, and Chenfanfu Jiang. Physgaussian: Physics-integrated 3d gaussians for generative dynamics. arXiv preprini arXiv:2311.12198, 2023.
Nicholas Pfaff, Evelyn Fu, Jeremy Binagia, Phillip Isola, and Russ Tedrake. Scalable real2sim: Physics-aware asset generation via robotic pick-and-place setups, 2025. URL http::t:///aaa org/abs/220 00370.
Yifan Zhu, Tianyi Xiang. Aaron Dollar, and Zherong Pan. One-shot real-to-sim via end-to-end differ- entiable simulation and rendering, 2025. URL https wwwpeiiiiiorraaaaaaoaeeeaeeee ht////bs/22441 02:42.00225
Dylan Turpin, Liquan Wang, Eric Heiden, Yun-Chun Chen, Miles Macklin, Stavros Tsogkas, Sven Dickinson, and Animesh Garg. Grasp'd: d: Differentiable contact-rich grasp synthesis for multi- fingered hands, 2022. URL https: /tt////axivvvv/aa wwtp:oo////////// 12250.
Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A universal visual representation for robot manipulation. arXiv preprint 2tt::2203.126011 2022.
Ilija Radosavovie, Tete Xiao, Stephen James, Pieter Abbeel, Jitendra Malik, and Trevor Darrell. Real-world robot learning with masked visual pre-training. In Conference Robot Learning, pages 416-426. PMLR, 2023.
Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang. Vip: Towards universal visual reward and representation via value- iilueiimplicit pre-training. arXiv preprint 22.0v:220000003000 2022.
Shikhar Bahl, Russell Mendonca, Lili Chen, Unnat Jain, and Deepak Pathak. Affordances from human videos as versatile representation for robotics. CVPR, 2023.
Mengda Xu, Zhenjia Xu, Yinghao Xu, Cheng Chi, Gordon Wetzstein, Manuela Veloso, and Shuran Song. Flow as the cross-domain manipulation interface. arXiv preprint arXiv:2407.15208 2024.
Kenneth Shaw, Shikhar Bahl, and Deepak Pathak. Videodex: Learning dexterity from internet videos. In Conference on Robot Learning, pages 654-665, PMLR, 2023a.
Austin Patel, Andrew Wang, Ilija Radosavovic, and Jitendra Malik. Learning to imitate object interactions from internet videos. arXiv:2211.1322. 2022.
Xue Bin Peng, Angjoo Kanazawa, Jitendra Malik, Pieter Abbeel, and Sergey Levine. Sfv: Reinforce- ment learning of physical skills from videos. ACM Transactions On Graphics (TOG), 3716):1-4, 2018b.
Tyler Ga Wei Lum, Olivia Y Lee, C Karen Liu, and Jeannette Bohg. Crossing the human-robot embod- iment gap with sim-to-real rl using one human demonstration. arXiv preprint urXiv:2504.126699 2025.
Irmak Guzey, Yinlong Dai, Georgy Savva, Raunaq Bhirangi, and Lerrel Pinto. Bridging the human to robot dexterity gap through object-oriented rewards. arXiv preprint arXiv:2410.23229, 2024.
Himanshu Gaurav Singh. Antonio Loquercio, Carmelo Sferrazza, Jane Wu, Haozhi Qi, Pieter Abbeel, and Jitendra Malik. Hand-object interaction pretraining from videos. arXiv preprint arXiv:2409.08277 08273, 2024.
15

---

Published as a conference paper at ICLR 2026
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-basee control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026-5033 IEEE, 2012. doi: 10.1 00.100//ROS.2212.688.111
Haozhe Lou, Yurong Liu, Yike Pan, Yiran Geng, Jianteng Chen, Wenlong Ma, Chenglong Li, Lin Wang, Hengzhen Feng, Lu Shi, Liyi Luo, and Yongliang Shi. Robo-gs: A physics consistent spatial-temporal model for robotic arm with hybrid representation, 2024. URL httpss://axxiv. org/ 2408 14873.
Chongjie Ye, Lingteng Qiu, Xiaodong Gu, Qi Zuo, Yushuang Wu, Zilong Dong. Liefeng Bo, Yuliang Xiu, and Xiaoguang Han. Stablenormal: Reducing diffusion variance for stable and sharp normal. ACM Transactions on Graphics (TOG), 2024a.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, al. Gpt-4o system card. arXiv preprint arXiv:2410.21276 2024.
Tianyuan Zhang, Hong-Xing Xing Yu, Rundi Wu, Brandon Y. Feng, Changxi Zheng, Noah Snavely, Jiajun Wu, and William Freeman. PhysDreamer: Physics-based interaction with 3d objects via video generation. In European Conference on Computer Vision. Springer, 2024b.
Martin Asenov, Michael Burke, Daniel Angelov, Todor Davchev, Kartic Subr, and Subramanian
Ramamoorthy. Vid2param: Modelling of dynamics parameters from video, 2020. URL https:
//aaxiiv. ///axxiv.org/abb/ abtp//bb//i// 19...00000000
Bowen Wen, Wei Yang, Jan Kautz, and Stan Birchfield. FoundationPose: Unified 6d pose estimation and tracking of novel objects. In CVPR, 2024.
Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary org. abs 180..88088000 07366.
differential equations, 2019. URL https: www.aa
David Baraff. Rigid body simulation. SIGGRAPH Course Notes 1992, 19, 1992.
Tom Erez. Yuval Tassa, and Emanuel Todorov. Simulation tools for model-basee robotics: Compari- son of bullet, havok, mujoco, ode and physx. In 2015 IEFE International Conference on Robotics and Automation (ICRA), pages 4397 2015. doi: 10.1 20.1109/AAA2221...773
Yuanming Hu, Luke Anderson, Tzu-Mao Li, Qi Sun, Nathan Carr, Jonathan Ragan-Kelley, and Frédo Durand. Difftaichi: Differentiable programming for physical simulation, 2020. URL 00935.
https:: /ttp:///aavvv.//ia http//aaii.....aaaaaaaoa/a///aaa
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach Vetito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, nigh-performance deep learning library, 2019. URL https::tp/// / arxiv. wwrg/aass.............. 21 01 703.
Georgios Pavlakos, Dandan Shan, Ilija Radosavovic, Angjoo Kanazawa, David Fouhey, and Jitendra Malik. Reconstructing hands in 3D with transformers. In CVPR, 2024.
Jane Wu, Georgios Pavlakos, Georgia Gkioxari, and Jitendra Malik. Reconstructing hand-held objects in 3d. arXiv preprint (4XX::2404406500 2024a.
Yuzhe Qin, Wei Yang, Binghao Huang. Karl Van Wyk, Hao Su, Xiaolong Wang, Yu-Wei Chao, and Dieter Fox. Anyteleop: general vision-based dexterous robot arm-hand teleoperation system. In Robotics: Science and Systems, 2023.
Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Keidovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. NewrIPS, 2020.
Pierre Vassiliadis, Gerard Derosiere, Cecile Dubue, Aegryan Lete, Frederic Crevecoeur, Friedhelm C. Hummel, and Julie Duque. Reward boosts reinforcement based motor learning. iScience, 24(7): 102821, Jul 2021. doi: 10.1016/j.isei..ooo 20lseii.222.11100 102222...
16

---

Published as a conference paper at ICLR 2026
Hui Zhang, Zijian Wu, Linyi Huang, Sammy Christen, and Jie Song. Robustdexgrasp: Robust dexter- ous grasping of general objects, 2025. URL
org/ abs/ 2504. 05287.
Hongyi Chen, Yunchao Yao, Yufei Ye, Zhixuan Xu, Homanga Bharadhwaj. Jiashun Wang, Shubham Tulsiani, Zackory Erickson, and Jeffrey lchnowski. Web2grasp: Learning functional grasps from web images of hand-object interactions, 2025a. URL https: arriiv. org/ abs /2005. 05517.
Weikang Wan, Haoran Geng, Yun Liu, Zikang Shan, Yaodong Yang, Li Yi, and He Wang. Unidex- grasp++: Improving dexterous grasping policy learning via geometry-aware curriculum and itera- tive generalist-specialis learning, 2023. URL https://aaxatit org/ /abs///////// 23004. 404
Zhenyu Wei, Zhixuan Xu, Jingxiang Guo, Yiwen Hou, Chongkai Gao, Zhehao Cai, Jiayu Luo, and Lin Shao. D(R, O) grasp: A unified representation of robot and object interaction for cross- embodiment dexterous grasping, 2025. URL aaaaariiiiive/eee org/ abs 2410. 01
Kenneth Shaw, Shikhar Bahl, and Deepak Pathak. Videodex: Learning dexterity from internet videos, 2022. URL https: arxiv. org/ aaa/bb//// 2212. 04498.
Jialiang Zhang. Haoran Liu, Danshi Li. Xinqiang Yu, Haoran Geng, Yufei Ding, Jiayi Chen, and He Wang. Dexgraspnet 2.0: Learning generative dexterous grasping in large-scale synthetic cluttered scenes, 2024c. URL httpt:t/t////ettm httpaaaivvvrrro
http:/:://axxivvorr/bb/s.s/ /ttpg///ab///111// 2410. 23004.
Linfei Pan, Dániel Baráth, Marc Pollefeys, and Johannes L. Schönberger. Global structure-from- motion revisited, 2024. URL
Johannes L. Schönberger and Jan-Michael Frahm. Structure-from--motion revisited. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4104-4113, 2016. doi: 10.1109/CVPR.2216.445
Krishna Murthy Jatavallabhula, Miles Macklin, Florian Golemo, Vikram Voleti, Linda Petrini, Martin Weiss, Breandan Considine, Jerome Parent-Levesquee Kevin Xie, Kenny Erleben, Liam Paull, Florian Shkurti, Derek Nowrouzezahraa, and Sanja Fidler. gradsim: Differentiable simulation for system identification and visuomotor control. International Conference on Learning Representa- tions (ICLR), 2021b. URL https: openrevieweww net
Clement Fuji-Tssng,, Masha Shugrina, Jean-Francois Lafleche, Charles Loop, Towaki Takikawa, Jiehan Wang. Wenzheng Chen, Sanja Fidler, Jason Gorski, Rev Lebaredian, Jianing Li, Michael Li, Krishna Murthy Jatavallabhula, Artem Rozantsev, Frank Shen, Edward Smith, Gavriel State, and Tommy Xiang. Kaolin: A pytorch library for accelerating 3d deep learning research. https: //github. Bww.://VIIIAGGaaeee..c. waatekkk//kkkk...o.oa in, 2019.
Xuan Li, Yi-Ling Qiao, Peter Yichen Chen, Krishna Murthy Jatavallabhula, Ming Lin, Chenfanfu Jiang, and Chuang Gan. Pac-nerf: Physies augmented continuum neural radiance fields for soometry-agnostee system identification, 2023b. URL https: //tpx/ar.vvvv//////////e///awa/ org/ owg/abs///a//eooooooooooee ht///bb//3333 05512.
Peter Corke. A simple and systematic approach to assigning denavit-hartenberg parameters. IEEE transactions on robotics, 23(3):590-594, 2007.
Matthias Müller, Bruno Heidelberger, Marcus Hennix, and John Ratcliff. Position based dynamics.
Journal of Visual Communication and Image Representation, 18(2):109-118, 2007.
Chenfanfu Jiang, Craig Schroeder, Joseph Teran, Alexey Stomakhin, and Andrew Selle. The material point method for simulating continuum materials. In ACM SIGGRAPH 2016 Courses, SIGGRAPH 16, New York, NY, USA, 2016. Association for Computing Machinery. ISBN 9781450342896. doi: 10. 011.2897886.2222734 URL htt.h:t//h/ett. //ts:////ooiio org. 10l..11452222 281222898888 .12.22222292272 2927348.
Jan Bender, Matthias Müller, and Miles Macklin. survey on position based dynamics, 2017. In Proceedings of the European Association for Computer Graphics: Tutorials, EG '17, Goslar, DEU, 2017. Eurographics Association. doi: 10.231 10.2312/egt201110044 URL https: / /////dd. doi. org/10.
2312 2312/egtt 201 01111134..
17

---

Published as a conference paper at ICLR 2026
Trevor Standley, Ozan Sener, Dawn Chen, and Silvio Savarese. image2mass: Estimating the mass of an object from its image. In Sergey Levine, Vincent Vanhoucke, and Ken Goldberg. editors, Proceedings of the 1st Annual Conference Robot Learning, volume 78 of Proceedings of Machine Learning Research, pages 324-333. PMLR, 13-15 Nov 2017. URL https::/ proceedings ngs .mlr
www.gnwwww...............ep wwwps/esssvv.e stasslanddeee ey1 7a html.
Chongjie Ye, Yinyu Nie, Jiahao Chang. Yuantao Chen, Yihao Zhi, and Xiaoguang Han. Gaustudio: A modular framework for 3d gaussian splatting and beyond. arXiv preprint arXiv:2403.19632, 2024b.
Binbin Huang, Zehao Yu, Anpei Chen, Andreas Geiger, and Shenghua Gao. 2d gaussian splatting for geometrically accurate radiance fields. In ACM SIGGRAPH 2024 conference papers, pages 1-11, 2024.
Andy Zeng, Shuran Song, Matthias Nießner, Matthew Fisher, Jianxiong Xiao, and Thomas Funkhouser. 3dmatch: Learning local geometric descriptors from rgb-d reconstructions. In CVPR, 2017.
Peter Yichen Chen, Chao Liu, Pingchuan Ma, John Eastman, Daniela Rus, Dylan Randle, Yuri Ivanov, and Wojeiech Matusik. Learning object properties using robot proprioception via differentiable robot-object interaction, 2025b. URL https: / /arxiviiii. org/ /t.b////11.. 03920.
Yiren Lu, Justin Fu, George Tucker, Xinlei Pan, Eli Bronstein, Rebecca Roelofs, Benjamin Sapp, Brandyn White, Aleksandra Faust, Shimon Whiteson, Dragomir Anguelov, and Sergey Levine. Imitation is not enough: Robustifying imitation with reinforcement learning for challenging driving scenarios, 2023. URL httpt:tt///// //axxii org/ abs/ / 2212. 11419.
Eli Bronstein, Mark Palatucci, Dominik Notz, Brandyn White, Alex Kuefler, Yiren Lu, Supratik Paul, Payam Nikdel, Paul Mougin, Hongge Chen, Justin Fu, Austin Abrams, Punit Shah, Evan Racah, Benjamin Frenkel, Shimon Whiteson, and Dragomir Anguelov. Hierarchical model-based imitation learning for planning in autonomous driving, 2022. URL https: http://aaxvvvvv/.o org/ abs/2210 09539.
Balakumar Sundaralingam, Siva Kumar Sastry Hari, Adam Fishman, Caelan Garrett, Karl Van Wyk, Valts Blukis, Alexander Millane, Helen Oleynikova, Ankur Handa, Fabio Ramos, et al. curobo: Par- allelized collision-free minimum-jerk robot motion generation. arXiv preprint arXiv:2310. 17274, 2023.
Kenneth Shaw, Ananye Agarwal, and Deepak Pathak. Leap hand: Low-cost, efficient, and anthropo- morphic hand for robot learning. Robotics: Science and Systems (RSS), 2023b.
Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-time dynamic scene rendering. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 20310- 20320, June 2024b.
Haoqi Yuan, Bohan Zhou, Yuhui Fu, and Zongqing Lu. Cross-embodiment dexterous grasping with 2024.
reinforcement learning, arXiv preprint
Zhenjia Xu, Jiajun Wu, Andy Zeng, Joshua B. Tenenbaum, and Shuran Song. Densephysnet: Learning dense physical object representations via multi-step dynamic interactions, 2019. URL petps://arze org/ 1906. 03853.
Vickie Ye, Ruilong Li. Justin Kerr, Matias Turkulainen, Brent Yi, Zhuoyang Pan, Otto Seiskari, Jianbo Ye, Jeffrey Hu, Matthew Tancik, and Angjoo Kanazawa. gsplat: An open-source library for Gaussian splatting. arXiv preprint arXiv:2409.06765, 2024c. URL /ttptpss///xxl..orgggg abs/2409 06765.
OpenAl: Marcin Andrychowicz, Bowen Baker, Macick Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et Learning dexterous in- hand manipulation. The International Journal of Robotics Research, 39(1):3-20, 2020.
18

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 PRELIMINARIES
This paper aims to accurately reconstruct the physical process of human hand grasping using only visual observations and robot control signals, without requiring access to ground ground-truth physical parameters. Our approach is grounded in two key components. The first is a differentiable, particle- based physies simulation engine Freeman et al. (2021), which enables gradient- based optimization of physical properties such as object mass. The second is a Real-to-Sing reconstruction pipeline based on Gaussian Splatting Lou et al. (2024), which allows us to build photorealistic and spatially consistent 3D scenes from video input. By combining these two components, pipeline that bridges real-world perception and physical simulation, supporting accurate modeling of dynamic hand-object interactions and enabling robust policy learning in simulation.
construct fully differentiable
Robotic simulation engines such as MuJoCo Todorov et al. (2012), Isaac Sim Mittal et al. (2023), and GradSim Jatavallabhula al. (2021b); Fuji-T'sang al. (2019) are fundamentally built upon the Lagrangian formulation of mechanics Li et al. (2023b), which models the evolution of physical systems by tracking a fixed set of particles or reference points through space and time. This approach assumes a consistent and predefined structure in the simulation environment, typically described using formats such as MICF or URDF. These configurations specify the number and arrangement of system components, such as joints, links, and actuated elements, which remain constant throughout the simulation. At each discrete timestep, the state of every object is updated based on dynamic and kinematic equations that reflect the physical principles embedded in the simulation engine. As result, the evolution of object poses, velocities, and contact interactions is governed by the engine's internal numerical solvers and integration schemes. This structured and physics- physics-ifformed representation is crucial for accurately modeling force transmission, contact behavior, and motion in robotic grasping scenarios.
Differentiable Physies. foundational assumption of our engine is that once the static scene reconstruction is completed, the physical configuration of the environment remains unchanged throughout the system identification and policy training stages. That is, no additional objects or robots are introduced to, nor are existing components removed from, either the simulation environment or the real-world scene. Consequently, the states of all entities captured during the observation phase remain consistent and are used directly for deploying control signals in both simulation and real-world execution. This guarantees the fidelity of simulation rollouts and the alignment of dynamics between domains.
Our system architecture is governed by two fundamental categories of equations. The first involves kinematic equations Corke (2007), which model the articulated motion of the robotic arm and hand, accounting for joint angles, velocities, and end-effector trajectories. These equations underpin the robot's ability to reach and manipulate objects in a controlled fashion. The second set comprises dynamic equations Jatavallabhula et al. (2021b), which govern the interactions between the object, the robotic hand, and the supporting surface (e.g., table). These dynamics describe the forces and torques that arise during contact, enabling accurate simulation of object responses.
To simulate and optimize object behavior, we employ a dual-engine architecture consisting of MJX (the JAX-based backend of Brax) and GradSim. For spatial representation within the differentiable physics engine, we use the object's mesh vertices as the fundamental particles. These vertices serve geometric and physical descriptors that enable fine- ine-ggrined modeling of object-hand and object-environment interactions.
MJX is used to model robot kinematics and extract detailed contact information during simulation rollouts. It provides precise contact points, surface normals, and force vectors arising from interactions with the robotic hand. This information is crucial for establishing accurate boundary conditions for system identification and subsequent policy learning.
In parallel, GradSim Jatavallabhula et al. (2021b) offers a PyTorch-based engine for gradient- based simulation of object dynamics. models the effects of gravity, inertial forces, and external perturbations (such as pushes from the robot or collisions with the ground), enabling smooth gradient flow through time. This setup facilitates efficient mass parameter optimization and supports end-to- end training pipelines involving both perception and control.
19

---

Published as a conference paper at ICLR 2026
A key assumption in our setup is that the relative poses between the object, the ground, and the robotic hand within the simulation closely approximate those in the real world. This alignment is critical to ensure that simulated contact events reflect real-world conditions, enabling high-fidelity modeling of physical interactions. To this end, we align object placement using estimated poses obtained from visual tracking pipelines such FoundationPose, ensuring consistent coordinate frames.
Although our engine incorporates "esition-Based Dynamics (PBD) Müller et al. (2007) for stabil- ity and efficiency, we introduce tailored modifications to enhance collision detection and contact resolution. Specifically, we refine the broad-phase collision detection algorithm to better handle nigh-resolution meshes and non-convex geometries. This is essential for accurately modeling complex objects with fine surface detail and for ensuring robust gradient propagation during contact-rich interactions.
By combining MJX's strengths in kinematic modeling and contact extraction with GradSim's 's gradient- based physical simulation, our engine enables end-to-end mass identification and force-aware policy training. These capabilities lay the foundation for accurate and generalizable robotic grasping in real-world settings, bridging the sim-to-real gap through physically grounded learning.
A.1.1 PARTICLE-BASEE PHYSICS SIMULATION
Particle-basee physics simulation is extensively used in computational physies and graphies for modeling dynamic behaviors of objects Jiang et al. (2016). Unlike traditional methods that rely on continuous volumes polygonal meshes, particle-based methods discretize objects into numerous discrete particles, each endowed with physical attributes such as mass m,, position x,. and velocity v,, as well material properties including elasticity, friction, and damping. This discrete representation allows the efficient and realistic simulation of complex behaviors, especially beneficial in scenarios involving deformable or fragmented objects, fluids, and granular materials.
The center of mass COM for carticle-based system can be computed by:
COM = Σ, m,x; Σ, m4
(12)
The inertia tensor which describes an object's resistance rotational accelcration, computed relative to the center of mass as:
I
Σ m4 [|rs| - rar.]
where r; = x; -
(13)
with E denoting the identity matrix.
Position-Bard Based Dynamics (PBD). Position- Dosition-Based Dynamics (PBD) is a widely adopted paradigm in real-time and interactive physics simulation due to its stability, simplicity, and efficiency in handling constraint-driven dynamics Bender et al. (2017). Unlike traditional force-based methods that compute motion by integrating forces and torques explicitly, PBD enforces physical consistency by iteratively projecting particle positions to satisfy a set of predefined geometric and physical constraints. This projection-based formulation naturally accommodates large simulation time steps, making it particularly suitable for high-speed applications such as robotic grasping and interactive environments.
Prediction Step (Implicit Integration). The simulation begins by predicting particle states using semi-implieit Fuler integration, which offers numerical stability and reduces oscillations during stiff interactions. For each particle i, the translational motion is computed as:
v ++t v
(14)
++ = x + Atv
(15)
where f is the external force (e.g., gravity or contact impulses), m; is the particle mass, and At is the simulation timestep. For rigid-body components, angular motion is predicted using:
w + w/ + A/I (T! wt x (Iww)),
(16)
At w alot
HAi q. +
(17)
2
20

---

Published as a conference paper at ICLR 2026
where I, is the inertia tensor, 7: is the external torque, and of is the orientation represented as unit quaternion. Here, W; [0, [ embeds angular velocity into the quaternion algebra.
Constraint Projection Step. Once predicted states are available, positional constraints are enforced through iterative corrections. Each constraint C(x,, q:) 2 0 represents a physical requirement (e.g.. no interpenetration, fixed distances, volume preservation) and resolved using a gradient-based position correction scheme. For constraint satisfaction, the positional update is computed as:
Vx Ax; - V C(xi),
((x,)
with
(18)
m4
Σj Vx, C(x;)]l2
where the Lagrange multiplier À ensures physically consistent constraint enforcement. Iterative Gauss-Seidel Jacobi solvers are used to converge the system to a valid constrain--satisfing configuration.
Velocity Update Step. After the constraints enforced, partiele velocities are updated to reflect the corrected positions:
(19)
Al
This ensures consistency between position corrections and subsequent dynamics, maintaining mo- mentum while preserving the stability advantages of PBD.
Discussion. The particle-based formulation enables fine-grained spatial resolution and direct grasping of geometric attributes, which is particularly beneficial for simulating high-DOF robotic hands interacting with rigid, deformable complex-shaped objects. Furthermore, the implicit treatment of constraints circumvents many of the numerical instabilities associated with stiff force-based models, making PBD highly suitable for differentiable simulation settings where robustness and gradient flow are important Standley et al. (2017).
A.1.2 GAUSSIAN SPLATTING
Gaussian Splatting has emerged as powerful technique in robotic real-to-sim pipelines for capturing scenes, objects, and backgrounds with high geometric fidelity and photorealistic detail. enables flexible and efficient modeling of complex environments from monocular video input, facilitating accurate spatial reconstruction and rendering. In our engine, we adopt the real-to-sim pipeline proposed in Lou et al. (2024), which transforms real- world scanned videos into simulation-rena assets. By leveraging Gaussian Splatting, we efficiently align the reconstructed object meshes with the simulation environment, enabling seamless integration.
To further enhance geometric consistency, we incorporate the stable normal constraint introduced in Ye al. (2024b;a), which enforces consistent surface normals across reconstructed points. This constraint is particularly important for preserving fine surface details and mitigating noise, especially in scenes with complex geometry intricate textures.
Together, this process allows us to recover two critical components for our differentiable physics modeling: (I) the object's 3D geometry and (2) its relative pose with respect the robotic arm, both of which are essential for accurate system identification and simulation alignment.
A.2 IMPLEMENTATION DETAILS
A.2.1 IMPLEMENTATION OF REAL-TO-SIM RECONSTRUCTION
We begin by constructing visually and geometrically precise digital twin of the target environment, leveraging a particle-baat based Gaussian splatting approach Kerbl et al. (2023); Huang et al. (2024). From environment-cente (I.) video streams captured by a mobile device, we obtain calibrated camera trajectories via structure-fromt friteefrroooooonnn (SfM) Pan et al. (2024); Schönberger and Frahm (2016). The pipeline then trains two disjoint ensembles of Gaussian primitives, each pursuing a separate objective.
1) Volumetric rendering set. We maintain set of 3D Gaussians
prend
{(11, yis Fi, Fi, 9i, bi, O2, 81, Ex) Naad
i=1
21

---

Published as a conference paper at ICLR 2026
where (2., Mk. 24) E R& is the center of the i-th Gaussian, (r. E [0,1]3 its RGB color, or E [0, 1] the opacity coefficient for alpha blending, Σ, E R3x3 a symmetric positive-definite covariance specifying anisotropic extent, 8; represent the semantic and instance id of the gaussian, and Need the total count of such primitives. These particles are optimized exclusively for photometric fidelity, enabling differentiable volume splatring and achieving real-time novel-view synthesis.
2) Surface reconstruction set. Geometry is approximated with separate set of 2D surface-aligned Gaussians
Pour = {TT M/j Rjo bu begi Sujj Su. Nari )
where (zn 2j) ) RR3 represents the disk center, fu,j to. E R$ are orthonormal tangent vectors, and su So.j set the standard deviations along those directions. The outward surface normal is
n tu.j X t
This ensemble is trained with depth distortion and normal consistency terms for geometric accuracy, remaining untouched by photometric loss.
After training. the surface Gaussians in Paarf are rasterized into multi view depth maps, fused into truncated ggnedddistance field, and converted via marching cubes into triangle mesh. Surface normals are estimated Ye et al. (2024a), giving the final collision mesh M. Since prend and paorf do share parameters and employ disjoint loss functions, improvements in appearance do not degrade geometric fidelity.
A.2.2 CONSTRUCTING MJCF MODELS USING RECONSTRUCTED GAUSSIAN AND MESH REPRESENTATIONS
The MuJoCo XML Control Format (MJCF) encodes key simulation components, including object's s kinematic structure, PID control gains, stiffness parameters, collision geometries IC along with the surface point cloud poarf Zeng et al. (2017), and specifications of actuated joints. To construct complete MJCF model from our reconstructed Gaussian splats and mesh representations, we first embed the static environment free joint body within the simulation environment.
unmovable background and define the reconstructed object
We then align the reconstructed Gaussian coordinate frame and chirality with MuJoCo's convention, following the transformation procedure described in Lou et al. (2024). To ensure simulation realism, we extract the relative pose between the object and the robotic arm in the real-world scene and apply this transformation as the initial configuration of the free joint object in simulation. After integrating all relevant positional and control information, we use Vision-Languaae Models (VLMs) to infer initial estimates of physical parameters, including object mass, which are critical for downstream simulation fidelity.
The resulting MJCF model, with accurately aligned coordinates, initial pose, and geometry, provides a strong foundation for subsequent system identification and physics-basee based policy learning. also enables high-fidelity rendering and precise real-to-sim transitions.
A.2.3 IMPLEMENTATION OF MASS IDENTIFCATIION
This section addresses two key aspects of our mass identification engine: (1) the strategy for mass- inertia modeling, and (2) the of adaptive parameters necessary support mass learning across objects with diverse physical properties and geometric variations.
Mass-Inertia Modeling. In conventional settings, an object's ground- truth mass is typically dis- tributed uniformly across its constituent particles, defined in Fquation 12. However, this strategy often leads to numerical instability and gradient explosion within real-to-sim-to-reee engines, particularly when dealing with high-resolution objects that contain over 50,000 vertices but possess relatively low mass Chen et al. (2025b). Under such conditions, the resulting average particle mass can fall below 10 kg, introducing significant numerical errors.
optimization
To mitigate this issue, we assign the full object mass to each particle. Gravitational forces are uniformly applied all particles, and external forees are scaled proportionally to the number of sampled vertices. This formulation preserves numerical stability by avoiding exceedingly small per-particle mass values.
22

---

Published as a conference paper at ICLR 2026
Additionally, because the number of vertices varies across reconstructed objects, adaptively select a subset of active vertices that lie on contact surfaces between the object and the robotic fingers. This further improves simulation fidelity and ensures relevant physical interactions are emphasized.
To guarantee consistency between the real-world observations and simulation environment, we explicitly synchronize frame rates, temporal bounds (start and end times), and spatial centering between the FoundationPose tracking system and the MuloCo simulation defined in MJCF format.
Contact Modeling, Explicit Gradient Representation, Adaptive Learning Parameters. We extract contact points and corresponding forces from robotic action rollouts conducted in both simulated and real-world environments. In the simulation, following the real-to-sim reconstruction, objects are placed in relative positions consistent with their real-world configurations. To ensure stable contact modeling within a Position-Basee Dynamics (PBD) engine, objects are initialized slightly above the ground (e.g., [0.05, 0.05, maintaining simulation stability.
Height + 0.01)), preventing premature ground contact and 2
Precise temporal synchronization across real-world object trajectories, robot control signals, and simulation rollouts is essential for reliable mass identification. We leverage FoundationPose Wen et al. (2024) to obtain accurate object pose estimates, and align simulation timelines accordingly to ensure consistency between observed and simulated motion.
For explicit gradient computation, we implement a semi-implicit integration scheme following the formulation introduced in Jatavallabhula al. (2021a), enabling differentiable backpropagation through contact events and object dynamics.
Adaptive Learning Strategy accommodate objects with varying mass scales, we employ an adaptive learning strategy. Initially, particle masses uniformly set approximately 0.002 kg per vertex, but this baseline must be adjusted according to the object's ' overall mass to ensure stable convergence. For heavier objects, such a ketchup bottle (0.8 kg), training requires higher learning rates and longer schedules, often up to 2000 epochs, to achieve convergence. In contrast, medium- mass objects (0.1 kg) typically converge efficiently within 100 epochs using a moderate learning rate. Lightweight objects (0.05 kg) benefit from learning rate decay and similarly converge within 100 epochs.
Successful mass learning also depends on several key factors. The duration of the applied impulse, determined by the active contact interval between the robotic fingers and the object, directly influences the estimated dynamics. We select the active tracking frame from Foundation Pose to mark the critical transition from motion onset to rest. Additionally, we apply a canonical re-centering vector to align object positions in simulation space, reducing variation introduced by camera viewpoint differences. Finally, the estimated contact area adjusted proportionally to the object's 's vertex count and active contact regions, allowing accurate modeling of the hand-object interaction Lu et al. (2023); Bronstein et al. (2022).
Discussion of the z-axis error from FoundationPose. In practice, the z-axis error produced by FoundationPose is non- negligiieee... We observe that, in some cases, this error must be manually post-processed better align the estimated pose with the real-to-sim digital asset.
A.2.4 ABLATION STUDY SEMI-IMPLICIT EULER AND EXPLICIT EULER
In this supplementary section, we expand the parameter- idnttficiidenniiicaaiinnn formulation and provide more formal ODE-based description of our objective, including boundary and initial conditions, sampling, and convergence properties.
The object dynamics in our setup are modeled by the Newton-Euu equations
mi( = (x()) u(() + Sgav:
(20)
where xt)) denotes the object pose (or the position its center of mass), u(t) encodes the robot hand actions, and m is the unknown object mass.
The initial conditions and geometric constraints are determined by: (i) the object geometry and pose reconstructed via Gaussian splatting (initial pose/shape), (ii) the robotic hand configuration and
23

---

Published as a conference paper at ICLR 2026
known actuation commands (control input u(t)), and (iii) the ground plane and environment, also provided by the reconstruction.
The dynamics residual is evaluated discrete time steps {tx] along the observed trajectory, with contact forces computed from the mesh vertices of the Gaussian-sllatinng reconstruction.
The ground-truth trajectories { at times (tx {tx) are provided by FoundationPose. Our differentiable simulator enforces the governing Newton- Euler equations at these sampled points, so that convergence of the learned dynamics and mass parameter driven by minimizing the residual of a well-posed ODE system.
In the specific "push-down" interaction used for mass identification, let =(1) denote the object position along the push axis, and u(f) the known contact force applied by the robot:
mz(t) uut) mg.
(21)
Integrating twice in time, we obtain
a(1;m) = + uot + 1
1 /'' [ u(s) u(s) dr 996
(22)
m 3o
which can be written
=(1;m) = a(l) + 3(1):
(23)
where (t) and (t) depend only on known quantities (initial conditions and measured robot forces). Thus, under this simplified contact model, the trajectory is affine in the inverse mass 1/m. Discretizing time gives
z (m) = Bk,
h == 1.....,,? .T
(24)
The trajectory loss we use,
T
(mai(m) Σ [|zx(m)) - 2x
(25)
then becomes a standard least-squares problem in the parameter 1/m. In this parametrization, Lugj (f) is quadratic function of 0 and therefore convex. In practice, our learning procedure operates through vector-Jacobian products in the differentiable simulator, but the underlying structure is equivalent to solving well-conditioeed least-squares identification problem for the mass.
We now clarify the effect of using a semi-implicit Buler scheme versus an explicit Euler scheme in differentiable simulator.
For the discrete-time dynamics, we adopt a semi-implicit (symplectic) Euler scheme. For a time step At, the updates are
Uk = 2kk At m m--ff(a ux),
(26)
Tk-1 = I + At Ukt1
(27)
where fire, up) includes gravity and contact forces. In contrast, the explicit Euler scheme would use
Uk = 2/ + At m - f(x, ux),
(28)
Tt-1 IA Atek-
(29)
This semi-implicit update rule is closely related the integrators commonly used in position-based dynamics (PBD) methods, and is known to have a larger stability region for stiff systems.
In contact-rich dynamics, especially for small masses or stiff contact forces, explicit Euler has a much smaller stability region and tends to produce large, oscillatory updates in zg. This is particularly problematic in our setting because we initialize the mass at a relatively small value (a conservative and can lead to very large position
lower bound), which amplifies the effective acceleration m f m- updates and exploding gradients when using explicit Euler. Semi-implicit Euler, on the other hand, updates the velocity first and then uses the updated velocity advance the position, which empirically leads to more stable and accurate trajectories.
We empirically compare the two schemes our push- based mass- ieestiiiiiiiiiitonnn tasks. Table 5 reports both the identified mass (in grams) and the average optimization time per iteration (in seconds) for each object:
24

---

Published as a conference paper at ICLR 2026
Object
Ground truth (g)
Semi-implicit (g)
Explicit (g)
Semi-implicit (s)
Explicit (s)
Lego
59
51
34
1.38
1.21
Ketchup
726
667
685
1.43
1.19
Cookie
210
200
189
1,41
1.18
Domino
106
117
135
1.36
1.17
125
110
98
1.39
1.20
A
134
145
120
1.42
1.22
Table Ablation study comparing semi-implicit and explicit Euler schemes for mass identification. We report the ground-truth mass, the identified mass under each integrator, and the average optimiza- tion time per iteration (in seconds). Semi-implicit Buler yields more accurate mass estimates at modest runtime cost.
Algorithm Force-Aware Policy Training
Input:
Set object meshes and masses: {(K;, M,)
Output: Learned actions and forces: {[Action; Force:))
for each demonstration M,) do
2:
Extract human hand poses and object poses using HaMeR Pavlakos et al. (2024) and MCC-
HO Wu et al. (2024a).
3: Retarget human hand poses and corresponding end-effector poses onto the robotic hand.
4: Positional Encoding: Encode vertices using positional encoding to obtain feature represen-
tations.
5:
Prepare training batches comprising encoded vertices, object mass M. and ground-truth actions. Load corresponding MJCF files generated by Real2Sim.
Dataset Construction:
6:
Stage One Training (Supervised): Train the policy network by setting force and contact head ground-truth labels optimizing initial grasp prediction.
7: Stage Two Training "Simulation-basee Refinement): Roll out predicted actions within
the MuloCo simulator using the Real2Sim- RealZSim--enerated MJCF files. Compute force and contact rewards from simulation outcomes and perform backpropagation to refine the model.
8:
Real-world Deployment: Deploy the grasping policy onto the real robotic system using the reconstructed object mesh, executing predicted actions with force control.
9: end for
Overall, explicit Euler is slightly faster per iteration but consistently less accurate, with larger deviations from the ground-truth mass and more frequent numerical instabilities during optimization. Semi-implicit Fuler provides a better trade-off between stability and accuracy, which is crucial for reliable differentiable mass identification in our Real2Sim pipeline.
A.2.5 IMPLEMENTATION OF D-REX'S ' GRASPING POLICY
Table details the neural network architecture used in our GraspMLP, while Algorithm and Algorithm 2 describe the training pipeline. For standard objects, the grasping policy is trained with approximately 200 demonstrations per object. For objects with higher geometric or dynamic complexity, we scale the dataset to include up 5000 demonstrations, ensuring sufficient coverage of the variance necessary for robust policy learning. Empirically, we find that the integration of lightweight policy network, accurate modeling human hand-object interactions, and precise physics-informed constraints enables reliable and high-performance grasping behavior tailored to each object.
A.2.6 IMPLEMENTATION DETAILS OF THE FORCE-BASED GRASPING POLICY
For the grasp-position policy, we use two-stage training procedure that combines human demonstra- tions with iimulation--bsed refinement (see Appendix A.2.4 for additional details). Our grasping policy, denoted GraspMLP, is first trained on human data and subsequently refined in simulation.
Input representation and supervision from human demonstrations. We begin by collect- ing human grasp demonstrations and reconstructing the human hand and object poses using
25

---

Published as a conference paper at ICLR 2026
Algorithm 2 Two-phase Training Procedure
1: Initialize: model parameters 0, optimizer, dataloader D, environment &, loss functions: MSELoss
(LMSE), BCELoss (LRCF).
2: Phase 1: Supervised Pre-training
3: for epoch 1...., _1 do D do
4: for batch (zz 0,
5:
Compute predictions: (a, m
model(a;;))
6:
Compute losses:
7:
C. + LMSE (ā, a)
8:
L, +- LBC.
9:
Cj f- LMSE(f,
10:
Backpropagate total loss: L La + Lr + Ls Update parameters
12:
end for
end for
14:
Phase 2: Environment Interaction
for epoch 1ooch=1,.... 1...., F2 do
16:
for batch x ~ D do
17:
Predict actions and rewards: (ā, F, f) model(2;0)
18:
Execute in environment & and observe rewards Tew and contact-based forces fen
19:
Compute scaled ground-trum force: feav = clip( m-g num contacts 0, 1)
fus
20:
Compute losses:
21:
L, +- LBCE (f, Teme
+ LMSE faw
23:
Backpropagate weighted loss: L = 0.8C, + 0.3L,
Update parameters
25:
end for
26:
end for
HaMeR Pavlakos et al. (2024) and MCCHO Wu et al. (2024a). These human hand trajectories are retargeted to the target robotic hand, yielding joint-space reference actions. For each object, we use the reconstructed mesh from the Real2Sim pipeline together with its identified mass. We uniformly sample mesh vertices, apply positional encodings their 3D coordinates, and concatenate these features with the object mass to form the per-vertex input to GraspMLP.
Network architecture. GraspMLP consists of shared MLP backbone that processes the mass- conditioned per-vertex features, followed by three task-specific heads: (i) a 16D head for joint-space grasp actions, (ii) a 2D head that predicts contact-based reward/feasibility score, and (iii) 1D head that outputs a normalized grasp force in [0, 1]. The full architecture (layer widths and activations) is summarized in Table 4 of the main paper.
Phase 1: Supervised pre-training. In the first phase, train GraspMLP human demonstrations to regress joint actions and predict contact/force labels. We use an E (MSE) loss for the joint actions and the normalized force output, and binary cross-entropy (BCE) loss for the contact/feasibility head (Alg (Algorithm 2, lines 3-13). All demonstrated grasps are treated as successful in this stage, and the contact/force labels are derived directly from the human data.
Phase Simulation- based refinement. the second phase, we refine the policy in simulation. We roll out GraspMLP in MuloCo using the Real2Sim-generaate MJCF models and the identified object masses (Algorithm 1, lines 6-7; Algorithm 2, lines 15-26 the Appendix). From these rollouts, we compute: (i) contact-based reward that reflects grasp success and stability, and (ii) scaled force target based on object mass, gravity, and the number of active contact points. We then update GraspMLP with weighted combination of BCE loss (for the reward/fcasibility signal) and MSE loss (for the force targets), effectively performing RL-style fine-tuning while remaining in supervised learning framework.
26

---

Published as a conference paper at ICLR 2026
Deployment and force control. At deployment time, the policy takes as input the observed object point cloud (in the same format as the training vertices) and the identified mass. On real hardware, implement& force-based controller by mapping the predicted normalized grasp force to actuator currents. For Allegro and LEAP hands, which employ direct-drive brushless motors, the mapping from motor current fingertip force is approximately linear, current control serves reliable proxy for controlling grasp force.
Applicability to different object scales. Although our experiments focus objects of standard tabletop scale, our method is not intrinsically limited to these sizes. Both the mass-identification and grasping modules operate on pose trajectories and reconstructed geometry, and thus in principle be applied to smaller objects as well. In practice, however, very small objects are constrained by two factors: (i) the perception pipeline must still provide accurate 6-DoF poses and sufficiently detailed geometry, and (ii) the robot hardware (including end-effector design, finger size, and sensing resolution) must be capable of reliably grasping and manipulating such objects. In our current setup, these perception and hardware constraints are the main bottlenecks for tiny objects, rather than limitations of the Real2Sim or caas-identifiiation formulation itself. These challenges are shared by most pose-based manipulation methods. Consequently, in this work focus on object sizes that are standard in existing 6-DoF pose estimation benchmarks and robotic grasping setups, where both perception and hardware are reliable.
A.2.7 COMPUTATIONAL DETAILS AND TIMINGS
Our grasping policy is trained on datasets containing 200 to 300 demonstration poses per object by default, which results in a training duration of approximately 2 minutes per object on one NVIDIA RTX 4090 GPU. For more complex or high- high-variance objects that require additional data coverage, we scale the training dataset to include up to 5000 demonstrations. In such cases, the training time increases to approximately 20 minutes per object, due to the additional dataset batch size.
Inference is highly efficient. Once the policy is trained and deployed. it requires only a reconstructed URDF or MICF representation as input, capturing the object's 's geometry, pose, and physical properties. Given such input, the policy predicts stable grasp configuration in approximately 0.5 seconds per object pose. This low-latency inference time makes the system practical for real-time and on-robot applications, particularly in scenarios that demand quick adaptation to dynamic object placements or orientations.
Overall, our engine demonstrates a favorable trade-off between training cost and deployment effi- ciency, with scalable training capabilities and low runtime overhead for inference.
Component
Operation
Output Dim.
Details
Input
Positional Encoding
N x
N object vertices (XYZ)
Linear Layer 1
Fully Connected
256
Input: 3 + 256 Non-linearity
Activation 1
ReLU
256
Linear Layer 2
Fully Connected
256
256 - 256
Activation
ReLU
Non-linearity 256 - 256
Linear Layer 3
Fully Connected
256
Activation 3
ReLU
256
Non-linearity
Action Head
Linear
16
Joint action output Contact constraint prediction Grasping force prediction
Reward Head
Linear + Sigmoid
2
Force Head
Linear Sigmoid
Table 6: Architecture the proposed GraspMLP network. The input consists of per-vertex 3D coordinates. The shared backbone maps the input into latent feature space, which is subsequently decoded into separate heads for predicting joint actions, contact- contact-based reward signals, and grasping force.
A.2.8 IMPLEMENTATION OF BASELINES ON OBIECT GRASPING
Human2Sim2Robot Baseline. In the Human2Sim2Robot engine Lum et al. (2025), we operate under the assumption that the grasping end-effector pose extracted from human demonstration videos
27

---

Published as a conference paper at ICLR 2026
is both accurate and physically feasible for robot execution. These grasp poses- typically obtained from hand-object interaction sequences are directly retargeted to the Leap Hand using the official retargeting implementation provided by the Leap Hand repository, preserving the spatial fidelity of the original grasp intent.
For fair and consistent baseline comparison, we replace the original demonstration assets and object meshes used in Human2Sim2Robot with our own Real-to-Sine reconstructed meshes, which incorporate photogeometrie fidelity and physical realism as described in Section Real2sim. Using these assets, grasping policies are trained until convergence, which generally requires approximately 20,000 training epochs to stabilize reward signals and behavior.
At deployment, we assume that the relative end-effector pose remains feasible under the Franka arm and CuRobo motion planning stack. That is, we expect the grasp pose transferred from human demonstrations to be executable without requiring additional replanning or corrections during real- world trials. While this assumption aligns with the original baseline setting, it introduces potential limitations in robustness, particularly under challenging object configurations.
important to note that the original controller, fabric, used in Rumman5Sim2oobtt-nnhhu closed-loop visual servoing and grasp adjustment mechanisms-is not publicly available. Conse- quently, our reimplementation focuses solely on static inference: given a fixed RGBD frame and known object pose, the system predicts single-step grasp action without online feedback or corrective replanning. This constraint is taken into account in our evaluations to ensure fair comparison.
DexGraspNet 2.0 Baseline. We adopt the two-stage grasping pipeline proposed in DexGraspNet 2.0 Zhang et al. (2024c), which separates grasp pose generation from execution via motion planning. However, rather than directly regressing relative translations and rotations from synthetic training data, our method infers these grasp parameters through MCC-HO, a pretrained model that extracts meaningful grasp features from real human hand-object interactions captured in video. These interactions are grounded in geometry reconstructed through our Real-to-Sim pipeline, where object vertices derived from point clouds are directly used estimate feasible grasp poses in 3D space.
Once grasp poses are generated, we utilize CuRobo for trajectory planning and execution. The planned trajectories are constrained by the robotic arm's kinematic and dynamic limits, ensuring safe and feasible real-world deployment of the inferred grasp poses.
To ensure fair comparison with DexGraspNet 2.0, which assumes fixed object mass of approximately 0.1 kg across all test scenarios. We limit our evaluation to objects of similar mass to match the conditions under which their policy was trained. However, in contrast to this fixed-mass assumption, approach explicitly optimizes grasp strategies using the mass identified through our differentiable real-to-sim-to- -ttrrll pipeline. This enables force-aware grasping, as the identified mass is used to refine force predictions and enhance grasp stability.
By leveraging human demonstrations and accurate physical modeling, our approach generalizes more robustly across varying object shapes and dynamic properties, offering improved realism and adaptability compared to methods relying solely on simulated training data and heuristic mass assumptions.
Object Tracking and Motion Planning. We employ FoundationPose Wen et al. (2024) for real- time 6-DoF object pose estimation during grasping. This robust visual tracking system provides temporally consistent pose predictions that enable dynamic, collision- aware trajectory planning for the robotic end-effector. These object pose estimates serve as a foundation for constructing grasping trajectories in cluttered or dynamic environments.
Once the object pose reliably tracked, we incorporate wrist pose predictions generated by MCC- HO Wu et al. (2024a), pretrained model designed to reconstruct hand-object interaction trajectories from human demonstration videos. The wrist poses extracted from these interactions represent feasible, human-derived grasping configurations. Together with the Real-to-Sim object pose, they define the target end- effeecoor pose required for grasp execution.
To generate collision free motion plans, we formulate constrained inverse kinematics (IK) opti- mization problem using CuRobo Sundaralingam et al. (2023). Specifically, we seek the robot joint configuration A that minimizes the distance between the robot's forward kinematics (FK) output and
28

---

Published as a conference paper at ICLR 2026
the desired end-effector pose X while remaining within the robot's collision-fii free configuration space Qiree:
A arg min AE
[F((A)
X
(30)
Here, X is derived from aligning the object pose (reconstructed via Gaussian Splatting and photogrammetry) with the wrist pose from human demonstration, forming a grounded and physically meaningful grasp target. The norm || ll (typically L2) measures the spatial error in SE(3) between the planned and desired poses.
This enables physically plausible and task-relevant grasp execution that leverages real-world percep- tion, human demonstration priors, and differentiable simulation to close the sim-to-real loop.
A.2.9 REAL.-WORLH EXPERIMENTS
In our experimental setup, the scene is composed of five primary components: static table, fixed background, a target object, a robotic arm, and a robotic hand. Both the table and background remain stationary and unchanging throughout the duration of each experiment, providing consistent spatial context. The robotic arm and hand are fully actuated and precisely controlled, with all joint movements accurately tracked ensure reproducibility and reliable system behavior.
The target object is entirely passive, which is not actuated or directly controlled. Its motion arises solely from physical interactions with the robotic hand, such contact-induced forces during grasping or pushing. This object-centric dynamic behavior forms the basis for our system identification and policy learning tasks.
For visual tracking, we employ third-person Intel RealSense D435i RGB-D camera positioned capture the entire grasping workspace. To estimate the 6-DoF object pose over time, we use FoundationPose Wen et al. (2024), real-time object pose estimation engine that ensures robust, Trame-consistent predictions even under occlusion or clutter.
To reconstruct the geometric details of the experimental scene-including the table, object, and robot-we we supplement the depth camera data with smartphone-based photogrammetry. Capturing short monocular video using mobile phone, we apply multi-view stereo techniques to generate dense 3D reconstructions of the environment. This process enables us to build high-resolution object meshes and spatially aligned scene representations, which are later used for initializing simulation environments and real-to-sim transfers.
Together, this combination of accurate tracking and high-fidelity geometric reconstruction provides the foundation for grounded simulation, physical parameter identification, and robust real-world policy deployment.
1. HARDWARE SETUP
We employ two distinct robotic hands in our experimental engine accommodate the varying requirements of system identification and dexterous grasping: the Allegro Hand and the LEAP Hand, each equipped with 16 independently actuated degrees of freedom (DoF). These platforms are selected to balance mechanical precision and torque capabilities across the experimental tasks.
Both Allegro and Leap hand are direct drive hand, which we have test our method's 's success rate increase. For the linkage driven hand like Inspire, we do not know the effectiveness of our force based control.
The Allegro Hand is widely used 16-DoF anthropomorphic robotic hand developed specifically for research in dexterous grasping. It features internalized wiring and a compact mechanical structure, minimizing external interference during physical interactions. Its low-profile design and clean joint layout simplify kinematic and dynamic modeling, making well-suited for physical parameter identification tasks such object mass estimation. The reduced presence of external cabling allows for more stable contact modeling and cleaner gradient flow during differentiable physics-based based optimization.
29

---

Published as a conference paper at ICLR 2026
The LEAP Hand Shaw et al. (2023b) is high-torque, cost-efficient robotic hand designed with modularity and real-world upplicability in mind. is constructed from a combination of 3D-printed components and off- fff-the-shelf actuators, enabling easy customization, repair, and experimentation. Critical mechanical attributes atrributes-including finger length, joint stiffness, and inter-finger spacing be modified to suit specific grasping scenarios object geometries. The LEAP Hand features a novel lendon-driven kinematic structure that enables highly dexterous and human-like articulation. Each joint is capable of exerting torques that exceed those of the human hand, while maintaining realistic velocities up to approximately 8 radians per second.
core design principle of the LEAP Hand is maximize the proportion of mass allocated to actuators relative to the hand's total weight, thereby enhancing grip strength while preserving compact form factor. This focus enables to handle heavy or irregularly shaped objects that require strong and adaptive force control. Importantly, the LEAP Hand includes integrated current- and torque-limiting mechanisms, allowing for both powerful and delicate grasping. These features make it especially suitable for executing real-world grasping tasks, where force control must be both robust and compliant.
In our experiments, we regulate the grasping force exerted by the LEAP Hand by tuning its actuator current limits, which are linearly correlated with the applied joint torques. This control scheme enables precise modulation of contact force based on object mass and surface properties, critical requirement for sim-to-real generalization in force-aware policy learning.
By leveraging the complementary strengths of the Allegro and LEAP Hands, our engine supports both accurate physical modeling and sesh-pefformnnce real-world grasping, facilitating end-to-end real-to-sim-tt-real learning and deployment.
Rationale for Using Different Hands We employ the Allegro Hand for mass identification exper- iments due its compact, self-contained mechanical design, which minimizes external interference. Its internalized wiring and low-torque actuation contribute to stable and noise-free contact dynamics, making it ideal for tasks that require accurate gradient propagation and precise system identification. These attributes are particularly advantageous when using differentiable physics to estimate object mass from robot-object interactions, where mechanical noise or inconsistent contact can significantly degrade optimization performance. The consistent kinematics and fow-inertia structure of the Allegro Hand further improve the fidelity of object dynamics modeling during the real-to-sim identification stage.
We utilize the LEAP Hand Shaw et al. (2023b) for grasping and grasping tasks due to its high-torque capabilities and modular, human-like kinematic structure. The LEAP Hand features tendon-driven actuation with robust motors that can generate significantly higher forces than the Allegro Hand, enabling it to perform reliable grasps on objects with varying shapes, weights, and compliance. This is particularly important when evaluating real-world policy deployment, where robustness and grasp stability are critical. Its design prioritizes strength and dexterity, making it suitable for executing force-aware policies under physically realistic conditions. The hand's current-controlled actuation also enables precise regulation of grasping force, which we leverage in our policy to adapt different object masses.
However, the LEAP Hand includes exposed wiring and tendon routing, which introduce mechanical noise and modeling complexity, especially during sensitive parameter estimation stages such as mass identification. These structural factors can interfere with accurate contact modeling and introduce inconsistencies in force feedback during differentiable simulation.
Through decoupling the roles of the two hands-using the Allegro Hand for precise physical parameter estimation and the LEAP Hand for robust grasping we are able to optimize each stage of our real-to- sim-to-real engine. This separation of concerns allows our engine to balance accuracy and practicality, supporting both high-fidelity modeling and real-world deployment across diverse set of grasping scenarios.
A.2.11 DATASET COLLECTION AND EXPERIMENT DEPLOYMENT
To support accurate real-to-sim modeling, collect approximately 300 RGB images per scene using a third sard--ersonn RGB-D camera (Intel RealSense D435i) or scanning device like Iphone. These images used for high-fidelity 3D reconstruction, which captures both the object geometry and
30

---

Published as a conference paper at ICLR 2026
environmental context. The full reconstruction process typically takes around 30 minutes per scene and produces a Gaussian Splats representation.
Then we convert the reconstructed visual assets into simulation-ready MJCF. This conversion en- codes the object geometry as collision meshes, specifies object kinematics, and initializes physical parameters for in simulation environments such MuJoCo. We also extract the relative pose between the object and the robotic base, which is crucial for alignment during simulation deployment.
Our experimental environment comprises a 7-DoF robotic arm (Franka Emika Panda), dexterous robotic hand (Allegro or LEAP, depending on the task), and a static table on which the object is placed. During data collection and evaluation, the robotic system executes predefined control trajectories or learned policies while interacting with the object. Simultancously. FoundationPose Wen et al. (2024) provides real-time 6-DoF object pose tracking using third-view RGB-D video input. This ensures precise alignment between real-world motion and the corresponding simulated trajectories.
All collected sensor data including RGB frames, depth maps, robot joint states, and object poses are synchronized and logged for later use in simulation, policy training, and evaluation. This structured dataset serves as the basis for mass identification and grasping policy learning, enabling consistent real-to-sim-to-real transfer across experiments.
A.3 ABLATION STUDY
A.3.1 SCALING PERFORMANCE
We investigate how the number of human demonstrations influences grasping performance challenging object: a compact, high-density screwdriver. This object is particularly difficult to manipulate due its small contact area and high moment of inertia, making for evaluating the scalability of our learning engine. As illustrated in Figure 9a, we assess grasping success over 20 real-world trials using policies trained on varying numbers of demonstrations. These demonstrations are automatically filtered and converted into robot-executable grasp poses using our Real-to-Sim pipeline.
ideal benchmark
The training dataset size ranges from 255 to 6,386 grasp poses, extracted from to 40 unique human video demonstrations. Our results show a clear positive correlation between demonstration count and grasping success rate: with just a handful of examples, the policy struggles to generalize and frequently fails to stabilize the object. However, as the number of demonstrations increases, the policy gains sufficient exposure to diverse object configurations and interaction patterns, enabling more robust and consistent grasps. Figure 9b provides qualitative visualizations of the grasp poses learned different data scales. With minimal data, the policy produces suboptimal unstable grasps, often misaligned with the object's geometry or balance point. As the dataset grows, the learned poses become progressively more aligned with physically stable and human- like strategies. These results underscore the importance of dataset scale in training force-aware grasping policies and highlight the effectiveness of our system in leveraging human video demonstrations improve dexterous grasping performance.
A.3.2 ROBUST MASS IDENTIFICATION ACROSS DIVERSE OBJECTS USING DITFERENTIABLE OPTIMIZATION
We present three representative examples of our differentiable mass optimization process in Figure 11, illustrating its convergence behavior across a diverse set of objects: Cookie, Lego, and Ketchup. In all cases, the optimization begins from a deliberately underestimated initial mass of 2 gg approximately 100x, 350x, and 30x smaller than the ground-truth masses for the Cookie, Lego, and Ketchup, respectively.
For the Cookie object, which has moderate mass and contact dynamics, the optimization converges smoothly to the correct value despite the large initial gap. This demonstrates the robustness of our engine under mild mass discrepancies. In the case of the Lego object, which features small contact surface and lower inertia, the large initial error induces an early overshoot. Nonetheless, the gradient-based optimizer is able to recover and guide the system toward the correct mass value within a stable number of iterations. The Ketchup bottle presents the most challenging case due to its high mass and complex geometry. The significant mismatch between the initial and true mass results in
31

---

Published as a conference paper at ICLR 2026
Method 4DGS Wu et al. (2024b)
Opt. runtime
PSNR 1
SSIM 1
LPIPS
30 min
17.14
0.632
0.411
D-REX (3DGS + opt. traj.)
20 min
23.54
0.783
0.237
Table 7: Real-to seal-to-sim reconstruction quality on the real mush-ketchup sequence. Our D-REX pipeline improves photometric metrics over vanilla 4DGS while reducing optimization time.
high initial loss. However, by applying adaptive learning rate and increasing the number of training epochs, the system successfully converges to an accurate mass estimate.
These examples collectively highlight the flexibility and effectiveness of our differentiable engine. Regardless of the object's ss scale or dynamic properties, our method reliably refines mass estimates from poor initializations, enabling physically grounded simulation essential for force-aware policy learning.
A.3.3 REAL--TO-SHT RECONSTRUCTION QUALITY
We first evaluate the quality of the reconstructed 4D scenes used for mass identification and policy learning. For the real executed robot trajectory and compare it against the recorded camera frames. video of this sequence, as well as an interactive 3DGS viewer, are provided on our anonymous project website (sections "Mass Identification via Object Pushing" and "Interactive 3D Workspace"), The reconstructions closely reproduce the observed object motion and contact interactions, indicating that the Real-to-Sim mapping is visually well aligned with the real-world data.
sequence in Sec. 5.1, we render the reconstructed scene along the
To quantitatively assess Real-to-Sim fidelity, compare D-REX Real2Sim pipeline (3DGS + optimized trajectory) with 4D Gaussian Splatting (4DGS) Wu et al. (2024b) on the same push- ketchup sequence. Following the standard D-NeRF bouncing ball protocol, evaluate rendering quality along the robot trajectory using PSNR, SSIM, and LPIPS, and we also report optimization runtime. Results summarized in Table 7.
D-REX clearly outperforms vanilla 4DGS on all three photometric metrics while requiring less optimization time. We attribute these gains supervising directly on the 6-DoF object pose and modeling the manipulated object as a single rigid body under differentiable physics, rather than optimizing each Gaussian independently.
A.3.4 SCOPE, LIMITATIONS, AND PATH TO GENERAL POLICIES.
These results highlight the flexibility of our engine while clarifying its scope. Generalization currently depends on (i) accurate mesh reconstruction and mass identification and (ii) task setups whose contact conditions are well approximated by rigid-body simulator. The present policies remain object- specific; however, conditioning on an estimated mass offers a plug-in signal that can be combined with architectures designed for category levee or multi-object training (e.g.. Zhang et al. (2024c)) to obtain more general policies when suitable demonstrations are available. Grounding learning in human demonstrations and targeted parameter identification reduces reliance on hand-engineered rewards and large-scale robot-collected datasets, enabling data-efficient transfer across tasks.
Table 8: Cross-object generalization from a larger to a smaller electric screwdriver. The policy trained on five human demonstrations and conditioned on object mass and reconstructed mesh.
Training Object
Test Object
Success Rate
10 x 3 cm, 600 g
7 x 2 cm, 500 g 90% - 70%
A.3.5 MASS-AWARE LEARNING vS. NOMAIN-RANDONEN
RL
We compare D-REX- trained from human demonstrations and conditioned on accurately inferred object mass against CrossDex Yuan et al. (2024), reinforcement-lermmig baseline using domain randomization (DR). CrossDex randomizes mass in the range 0.5-1.5 kg during training and reports an 89% success rate in simulation.
32

---

Published as a conference paper at ICLR 2026
Scaling Performance
95%
85%
Success Rate
1055
255
1001
2004
6899
Demonstration Poses
(a) Scaling Performance. Success rate improves the number of demonstra- tions increases. Grasping success in- creases with dataset size, demonstrat- ing the policy's ability to leverage more demonstrations for robust performance.
(b) Visualization of Grasping Pose with respect to number of demonstrations. As shown the figures, the policy produces unstable grasps with to demonstrations, but generates a stable grasp when trained with 20 demonstrations.
Figure 9: Scaling performance of our force-aware grasping policy with increasing number of demonstrations.
To isolate the effect of mass, we evaluate on family of Symbol Y objects that share identical geometry but differ in mass: 117g g, 206 and 324 (i.e., 0.117, 0.206, and 0.324 kg). Notably, all three test masses lie below the CrossDex training range. As summarized Table 9, CrossDex performs well on the heaviest variant and moderately on the medium one, but struggles the lightest object, illustrating DR's sensitivity when deployed on out-of-distribution (OOD) mass values, especially far from the training support.
In contrast, D-REX leverages low-cost human demonstrations to infer object mass and conditions the policy accordingly, enabling targeted adaptation without additional randomized training. The result is consistently high success across all three masses, despite the OOD shift relative to the DR baseline's training range.
Table 9: Real-world grasp success across mass variants of a single object geometry (Symbol Y); 10 trials per condition. CrossDex was trained with mass randomization in [0.5, 1.5) kg; all test masses are below this range. Higher is better.
Method CrossDex
117 g (Light) 206g (Medium) 324g (Heavy)
4/10
7/10
9/10
Ours
9/10
10/10
9/10
These results suggest two complementary points: (i) Domain Randomization can yield strong performance within or near its training support but degrades for larger OOD mass shifts (e.g.. very light objects), and (ii) explicit, mass-aware conditioning provides a simple and data-efficient mechanism for robust transfer across mass variation without requiring broad randomization. While Domain Randomization and mass-aware learning are not mutually exclusive, our findings indicate that accurate parameter identification is powerful lever for real-world generalization, particularly when deployment conditions fall outside the range covered by domain randomization.
A.4 LEARNING PHYSICAL PARAMETERS BEYOND MASS AND HANDLING FRAGILE OBJECTS
We focus on estimating object mass because admits a clear ground truth, straightforward to validate experimentally, and exerts an immediate and observable influence on grasping performance (e.g., grasp failures due to underactuation). While is in principle feasible to learn additional physical parameters- suche as friction, stiffnees, damping-wy we prioritize mass owing its measurability, stability across settings, and direct relevance to grasp dynamics. Moreover, mass variation can be applied systematically across diverse objects, which enables controlled assessment of generalization across geometries and densities.
Prior work has explored learning richer sets of physical properties in simulation (e.g., gradSim Jataval- labhula et al. (2021b)) and dense object attributes from visual observations (c.g., Xu et al. (2019)). However, reliable real-world validation of such parameters remains substantially more challenging
33

---

Published as a conference paper at ICLR 2026
Figure 10: Quantitative Results of Object Grasping trained on the heavy one. Only the mass- matched policy achieves stable grasps, while mismatched ones fail due to excessive or insufficient force, causing bounce-off for lighter objects or slippage for heavier ones.
Mass
Loss
GR 210g
Cookie
GT 726g
Ketchup
Lego
200
+0.
-50
50-
-0.51
1850
-40
Mass
-0.
40
100
20
-049
200
60
-047
Epochs
Figure 11: Mass-Loss curves. We present three examples of our system applied to mass identification. The blue curves represent the estimated masses, all of which converge reliably to the ground-truth values, demonstrating the accuracy of our approach.
due contact dependence, spatial and temporal variability, and sensitivity surface conditions. Consequently, extending parameter learning beyond mass is outside the scope of the present study.
In our current D-REX engine, rigid-body dynamics are assumed and object mass the sole learnable physical parameter. This design choice serves two purposes: (i) it isolates the causal role of mass in dexterous grasping and (ii) it demonstrates that D-REX can recover this key quantity directly from data. The policy is conditioned on demonstrations for the same object, making it object-specific rather than fully general. For broader generalization, the mass-estimation module can be used as plug-in: estimate object's mass and then fine-tune (or condition) task policy the inferred value.
For objects that are too fragile to tolerate pushing, do not attempt to learn additional contact parameters. Instead, we employ lower-force grasping strategies to reduce contact uncertainty- for example, rolling reorientation while following predefined orientation trajectory (via quater- nion s slerp) Xu et al. (2019)-thereby thereby limiting impulsive interactions without requiring explicit estimation of frictional properties.
34

---

Published as a conference paper at ICLR 2026
Finally, after the real-to-sim alignment step, execution proceeds without external human intervention in the physical scene. This assumption preserves consistency with the Lagrangian rigid-body dynam- ics underlying our simulator and cleanly attributes observed performance differences to the learned mass parameter rather than to uncontrolled external corrections.
A.5 ON THE GENERALIZATION OF D-REX
As noted in our limitations, the current policy is object-specific: training and evaluation assume that the object's mass is consistently identified and transferred between simulation and the real system. Nevertheless, we examine the extent to which D-REX exhibits cross-object and cross-task generalization under this assumption.
Within-category, cross-object transfer. To assess transfer within a category, we collected twenty human demonstration episodes for larger electric screwdriver and trained policy conditioned on its reconstructed mesh and mass. At test time, without any fine-tuning, we replaced the mesh and mass with those of smaller screwdriver from the same category and executed the policy for 10 trials. As summarized in Table 8, the policy maintained stable performance with only a minor decrease in success rate, indicating that D-REX generalizes across moderate variations in geometry and mass within a category. Qualitative rollouts are provided on the anonymous project website.
Beyond grasping: articulated and fine-grained tasks. To probe broader applicability, we eval- uated D-REX on more complex grasping scenarios, including articulated-objee interactions (c.g., operating stapler) and fine-grained tasks (e.g., manipulating computer mouse). For each task, we used 5-10 human demonstrations processed through the same real-to-sim pipeline, but for the articulated object digital asset creation, we manually perform the segmentation and re-assembly, with mass identification integrated into training. We find that, provided the reconstructed simulation captures the salient articulated structure and task-relevant geometry, the policy transfers reliably to these settings, suggesting that D-REX is not limited to grasping but can support wider class of object interactions.
A.6 PERFORMANCE DEGRADATION OF THE BASELINE
The primary cause of the baseline's degraded performance is the absence of explicit force (or impedance) control combined with a narrow training mass distribution. Both Human2Sim2Robot and DexGruspNet 2.0 Zhang al. (2024c) were trained entirely in simulation with object masses concentrated around = 0.1 kg. When deployed on substantially heavier items (c.g., Spam, Ketchup, Nutella), which lie outside this training distribution, the controller applies essentially fixed or weakly adaptive grasp forces that are insufficient to prevent slip-i.e. grasp failures due to underactuation.
For
frictional grasp, the required normal force per contact grows with the object weight and inversely with the effective friction coefficient. In simplified parallel-jaw setting with two symmetric contacts,
mg
Fn
7
2p
where m the object mass, g gravitational acceleration, i the effective desk-deppndentt friction coefficient, and y 2 1 absorbs wrench distribution, contact geometry, and safety margins. If m outside the range encountered during training- or underestimated at deployment- a fixed position-control policy lacks the ability to scale F,, accordingly, violating the inequality and inducing
slip.
Low-friction surfaces (e.g., plastic wrap or smooth metal) further increase the force requirement by reducing μs, exacerbating failures when the applied force is already marginal. Conversely, lighter objects are more tolerant to small positioning or force errors and may remain secured despite suboptimal control. We also observe exceptions where heavier objects succeed due to fortuitous geometry: for instance, spray-bottle nozzle heads can incidentally create partial form-closure (or caging) between fingers, partially compensating for insufficient frictional support.
Our method augments the policy with mass-aware force modulation: we estimate object mass from robotic action and videos and adjust the grasp force (or impedance setpoints) as a function of the inferred mass at test time. This targeted adaptation restores adequate contact forces on heavier
35

---

Published as a conference paper at ICLR 2026
otherwise challenging objects, reducing slip and improving success rates. More broadly, these findings underscore the necessity of mass-conditioned control for robust, generalizable dexterous grasping across diverse real-worke objects and surface conditions.
A.7 REASON TO BUILD UP ACCURATE DIGITAL TWIN
Building accurate digital twins and applying domain randomization are two complementary strategies for bridging the sim and real gap. each offering distinct advantages depending on the task and deployment context. Accurate digital twins aim faithfully reproduce real-world physical and visual fidelity, etc. enabling: 1) Precise policy evaluation and benchmarking under realistic dynamies, 2) System identification, particularly for contact-rich tasks sensitive physical parameters such as mass and friction, 3) Gradient- based optimization of physical properties or control strategies, which requires differentiable and realistic simulation feedback.
Our approach extends beyond visual or geometric digital twins by incorporating differentiable system identification to capture underlying physics long-standing challenge in roboties and graphics. This enables more accurate and efficient parameter adaptation, improving both realism and policy transfer. We view digital twins and domain randomization as complementary tools, with high-fidelity modeling serving to support informed adaptation in contact-rich or dynamic scenarios where randomization alone may overlook critical constraints.
A.8 RELATIONSHIP BETWEEN MASS IDENTIFICTIOO AND CORCE--ASSER POLICY LEARNING
We deliberately decouple mass identification from policy learning to isolate the causal role of mass in sim-to-real transfer and to enable clean evaluation. Concretely, from a small of human demonstrations and robot grasping 'D we estimate a scalar mass
m = arg min (m; D),
and then train control policy that is explicitly conditioned on this estimate,
u = Te ( (I, m).
where x denotes the robot/object state and u the control command. This two-stage design avoids confounding between parameter estimation and control optimization, making possible attribute downstream performance changes specifically to the accuracy of m.
Conditioning the policy on in enables explicit modulation of force or impedance setpoints and of feedforward gravity terms (e.g., u. C g(g: m)). In practice, we scale grasp-force targets and compliance parameters as functions ri, which restores adequate contact forces on heavier objects while avoiding unnecessarily high forces on lighter ones. The result is improved robustness across broad mass range without requiring extensive re-training.
A.9 SYSTEM SUBMODULES AND LIMITATIONS
Prior differentiable real-to-sim approaches (e.g., Chen et al. (2025b)) typically rely on rich robot proprioception (e.g., motor torque sensing) and tight hardware calibration to perform system identifi- cation. Such requirements limit deployability outside well-instrumented labs and differ substantially from our setting. By contrast, we pursue dision-driven identification that uses only externally ob- served signals. which we find more accessible and scalable practice. Accordingly, we evaluate against ground-truth physical measurements (e.g., mass) rather than sim-only metrics, providing direct assessment of real-world fidelity. our knowledge, few real-to-sim engines offer end-to-end differentiability that remains practical at deployment time; those that do often require assumptions that are difficult to satisfy in unstructured environments.
Our pipeline leverages FoundationPose Wen al. (2024) robust 6-DoF pose estimator. These poses serve as the primary observation signal for identification and control, replacing the need for onboard torque sensing. We combine these estimates with differentiable physics engine (MJX) operating on wwwllsim--eneraua MJCF assets, which supply geometry, inertial properties, and contact models.
36

---

Published as a conference paper at ICLR 2026
Gradsim Jatavallabhula al. (2021b) is designed for System Identification using rendered image observations from simulation, combining state-based and photometric losses. Our setting differs in two fundamental ways:
Real-world, partial observations. We operate directly on real videos and 6-DoF object poses estimated by FoundationPose Wen et al. (2024). Rather than assuming access to full simulator state and gradients as in Jatavallabhula et al. (2021b), we estimate physical parameters (e.g.. mass) from partial, noisy observations by minimizing state-space trajectory error over time in differentiable simulator.
Photometric supervision is impractical in our setup. Applying Jatavallabhula et al. (2021b) would require carefully controlled lighting, calibrated cameras, and often even 3D-printed objects with known properties obtain reliable photometric losses. We explored using 4D Gaussian Splatting to synthesize renders for photometric alignment, but optimization was unstable and inaccurate in our scenes, reinforeing the limitations of purely image-based losses for physical deployment.
Physics-constrained identification. In our engine, the differentiable simulator acts as a numerical solver obeying physical laws. Given known robot inputs ge. commanded joint trajectories) and accurate initial/boundary conditions from Real2Sim, we pose mass estimation as constrained optimization problem: find the parameter values that best reproduce observed FoundationPose trajectories.
For these reasons, we do not treat Jatavallabhula et al. (2021b) as a competing baseline our evaluation. Instead, we reuse its differentiable rendering mechanism internally while MJX provides the physical kinematic and dynamic of robotic hand.
Modularity, robustness, and extensibility. Our system intentionally modular: (i) pose estimation (FoundationPooe Wen et al. (2024)); (ii) asset generation and scene reconstruction (Real2Sim) Lou al. (2024); Ye al. (2024b;c); (iii) differentiable physics Gradsimm Jatavallabhula al. (2021b); Todorov et al. (2012); and (iv) policy learning. Similar multi-component designs are common in robotics and vision system work Pfaff al. (2025); Andrychowiez et (2020) because they enable targeted improvements, swapping of submodules as better tools emerge, and reusability of well-validated components. We do not treat these modules as black boxes; rather, we select them based empirical reliability and integrate them with sanity checks and data filtering.
Data strategy and practicality, rely on human demonstration videos as the primary supervision signal for policy learning. Such videos are inexpensive and widely accessible (e.g.. public internet platforms and existing datasets), dramatically reducing the collection burden compared to robot- executed demonstrations or reinforcement learning. which often require eegieeeneeeered rewards and long training cycles. Occasional failures of individual submodules (c.g., transient pose estimation errors) typically result only filtering out small fraction of low-quality demonstrations; overall effectiveness is maintained through scale. The combination of scalable supervision with differentiable real-to-sim identification yields a practical and extensible pathway toward robust sim- to-real transfer.
Limitations. The D-REX framework currently only supports rigid-body dynamics and relies solely on mass as the primary learnable parameter. Once the real-to-sim stage concludes, our simulation engine requires the absence of human interaction with the real-world operation scene to maintain consistency under the assumed Lagrangian dynamics framework.
A. LLM USAGE
We employed large language models solely for grammatical refinement and stylistic polishing of the manuscript. No part of the conceptualization, experimental design, implementation, or analysis relied on these models.
37

---

Published as a conference paper at ICLR 2026
A.11
LIST OF NOTATIONS
Symbol
Description
Scene-centric RGB video sequences
Io
Object-centric RGB video sequences
(L
Human demonstration RGB video sequences
(st
Real-world object trajectories
(st
Simulated object trajectories
Optimized object mass
Force-aware grasping policy
Simulation environment representation (MJCF)
K
Collision mesh for object geometry
0
Physical simulation parameters
P
Gaussian splatting particles for visual appearance
8t
Object's state at timestep (position and orientation)
Object's velocity at timestep t Linear velocity component timestep Angular velocity component at timestep { Mass-inertia matrix
Ut
Wi
M
f
External and contact forces
Jn
Contact force vector Contact stiffness and damping parameters Discrete-time update function
key ka
G())
At
Simulation timestep
Lwaj
Trajectory loss function Human hand pose at timestep
hy
Object pose at timestep Robot action at timestep
A
n
Learned grasping policy (parameterized by ()
Á
Predicted robot joint positions
A
Predicted contact constraint
Predicted grasping force constraint
Tactive
Number of active contacts between robot and object
Object density parameter
38


Reviews:
[{"review_id": "3nSMUZEWeR", "reviewer": "Reviewer_snrT", "summary": "This paper presents D-REX, a differentiable real-to-sim-to-real pipeline that couples Gaussian Splat Representations for photorealistic 3D reconstruction with a differentiable physics engine for object-mass identification and force-aware grasp policy learning. The system jointly optimizes physical parameters (mass) from robot interaction videos and learns manipulation policies conditioned on the inferred mass, closing the sim-to-real loop for dexterous grasping tasks.", "questions": "Please see my weaknesses section, thanks!", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper demonstrates a technically competent system that merges 3DGS and differentiable physics for vision-based grasping. Authors have performed real-world experiments validating some of their claims.", "weaknesses": "**1. Pipeline composition rather than a learning contribution.**\nThe full system is essentially a sequential pipeline: (1) Gaussian Splatting for 3D reconstruction with VLMs, (2) System identification to calibrate physical parameters, and (3) a procedural grasping policy that uses hand-designed grasp position and orientation heuristics. There is no novel algorithmic contribution or learning formulation that connects these modules beyond standard differentiable chaining. \n\n**2. Hand-designed grasp prediction.**\nThe grasping procedure relies on manually defined rules. This is not significantly different from prior grasp pipelines that use geometry-based scoring or analytical quality metrics.\n\n**3. No clear advantage over existing methods.**\nThe paper does not demonstrate how D-REX materially improves over existing differentiable grasping frameworks that already combine differentiable rendering and physics. The quantitative differences appear modest and could stem from tuning rather than a new principle. Additionally, just identifying mass, without taking materials into consideration, seems incomplete for robotics purposes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vlduxfB2vS", "reviewer": "Reviewer_6NV8", "summary": "The paper presents D-REX, a differentiable real-to-sim-to-real framework for dexterous grasping. It combines 4D Gaussian Splatting (4DGS) for scene reconstruction and differentiable physics for identifying physical parameters (notably object mass) from real-world robot trajectories. The identified mass is then used to train a force-aware grasping policy, which achieves higher grasp success rates in both simulation and limited real-world settings.\n\n\nThe paper explores an interesting and relevant direction by combining differentiable physics and 4DGS for Real2Sim2Real learning. While the current experiments do not fully validate the motivation of building a high-fidelity differentiable Real2Sim pipeline, the proposed idea is novel and potentially impactful. Overall, the paper presents a promising step toward differentiable Real2Sim learning and merits a weak accept pending more thorough validation.", "questions": "1. The paper introduces two separate sets of Gaussians to represent the visual appearance and geometry of the scene. How might potential optimization misalignment between these two representations affect the accuracy of mass identification and the subsequent policy learning?\n2. The Appendix mentions that the optimization of 4D Gaussian Splatting for photometric alignment is unstable and inaccurate. Could you provide detailed explanations or illustrative failure examples to clarify this issue?\n\n## Suggestions for Improvements\n\n1. Provide more qualitative and quantitative results/comparisons after Real2Sim stage, evaluating both the 4DGS scenes and the generated mesh.\n2. Provide more visual or quantitative results/comparisons of force values during robot excution to better substantiate the force-aware policy.\n3. Provide a systematic runtime analysis for different modules of D-REX to better assess the overall computational efficiency of the pipeline.\n4. Minor formatting issues: Line 97: “Empirically, We” — the word “We” should be lowercase; Line 1737: “Physics-constrained identification” — a line break is recommended for proper formatting.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- The paper is clearly written and presents a coherent overall system.\n- The proposed framework is conceptually appealing, and incorporating mass identification for robot policy learning is a novel and promising idea.  \n- The experimental results demonstrate accurate mass estimation and consistent grasping improvement with mass-aware policies.", "weaknesses": "- **The evaluation of Real2Sim quality is lacking.**  \n  The paper presents no analysis or evaluation of either appearance or geometry (mesh) of the generated digital scenes—offering neither quantitative metrics nor qualitative discussion. As a result, the claimed Real2Sim objective remains unsupported and unvalidated, which weakens the overall completeness and credibility of the contribution.\n\n- **The validation of the “force-aware policy” is insufficient.**\n  To validate the effectiveness of the force-based control, the authors report grasping success rates in Table 3 and Figure 5 undering different settings. A visual or quantitative comparison of force values across different settings during robot execution would better substantiate the effectiveness of the proposed force-aware policy learning.\n\n- **The efficiency and scalability should be discussed.**  \n  The authors present mass-loss curves in Figure 11 but seem to have omitted the actual optimization or training time, which is also an important factor for evaluating system efficiency. In addition, the offline 4DGS reconstruction step can be computationally expensive; however, no quantitative analysis (e.g., runtime or memory usage) is provided. Therefore, the practicality of the proposed Real2Sim pipeline for large-scale or online applications remains somewhat unclear.\n\n- **The generalization ability of D-REX is limited.**\n  The proposed pipeline requires real-world object trajectories (obtained via FoundationPose) and matched real/sim robot interactions to optimize object mass. Consequently, the generalization ability of the proposed method to novel scenes or objects, where such real trajectory data and robot interactions are unavailable, appears limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "S2fGt5q2Sc", "reviewer": "Reviewer_M426", "summary": "The paper presents a differentiable real-to-sim-to-real engine for object mass identification and grasping policy learning. Specifically, the Gaussian Splat representation is leveraged to facilitate estimation object mass through visual observations and robot control signals during interaction. Besides, a learning-based method is also proposed to train force-aware grasping policies from limited human demonstration videos. Comprehensive experiments have been conducted to validate the performances on mass identification and object grasping.", "questions": "Please see the weaknesses.", "limitations": "", "rating": 6, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe topic of end-to-end object mass identification is valuable and the solution using differentiable simulation is novel. \n2.\tBased on the mass estimation, the proposed system achieves satisfactory performances on the object grasping, reducing the gap between simulator and real-world environment.\n3.\tThe proposed approach outperforms strong baselines in object grasping, especially for the challenging object grasping. \n4.\tThe paper is well-written and the ablation studies are comprehensive.", "weaknesses": "1.\tThere should more details in the section of parameter identification from robot-object interactions. For instance, the rationale behind the trajectory discrepancy minimization for the object mass identification should be included. What’s the effects of the semi-implicit Euler modeling. Is there any ablation study for this learning objective?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EsMnLSuAIq", "reviewer": "Reviewer_KAq7", "summary": "The paper proposes D-REX, a real-to-sim-to-real method to improve the performance of grasping policies. The method first leverages off-the-shelf tools to convert human videos to robot kinematic trajectories. After that, the method identifies object masses based on differentiable physics engines. The identified object masses are further utilized to develop a force-based control policy for grasping. Extensive experiments prove the effectiveness of the method. Overall, the paper makes a step forward in developing physics-aware policies utilizing real-to-sim-to-real methods and demonstrates the effectiveness of incorporating physical properties into robot policies. Main limitations lie in the inherent restrictions of the mass identification method and the restricted upper limits of only identifying and utilizing masses in the robot policy learning.", "questions": "- How do you train the grasping position policy? \n- Could the method be extended to small objects, including both the mass identification and the grasping process?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Good motivations. The problem is well motivated. Real-to-sim system identification is an important way to bridge the sim-to-real gap. Beyond properties reflected from the visual appearances, such as object meshes, the paper makes a step forward and proposes to identify physical properties, i.e., masses, from a dynamic interaction sequence. After identifying masses, a force-adaptive method is developed to improve the grasping policy. \n- Reasonable methodology. Estimating masses from hand-object interaction sequences and the force-based policy are reasonable approaches. \n- Solid experiments. The authors carefully design experiments to validate the effectiveness of the object mass identification method design and the superiority of the force-based grasping policy.", "weaknesses": "- The paper utilizes the foundation pose to estimate the object pose sequences from real videos. However, the foundation pose cannot deal with axis-symmetric objects and tiny objects. Moreover, the estimated object poses always suffer from noise. Therefore, the quality of the mass identification step would be restricted by the quality of the identified masses. The applicability of the method would also be restricted to objects that the foundation pose can handle. Besides, utilizing foundation models to generate scene configurations, such as the robot mjcf, may also introduce errors. \n- When identifying masses from videos, the sim-to-real gap in other properties, such as frictions, is neglected, which would further make the estimation prone to errors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)