DexNDM: Closing the Reality Gap for Dexterous In-Hand Rotation via Joint-Wise Neural Dynamics Model
Authors: Xueyi Liu
He Wang
Li Yi
Venue: ICLR 2026 Poster (2026)
Keywords: In-Hand Object Rotation; Sim-to-Real; Neural Dynamics Model
Primary area: applications to robotics, autonomy, planning
TL;DR: A sim-to-real approach that enables unprecedented in-hand rotation (long, small, complex shapes; down-facing hand) in the real world.

Abstract: Achieving generalized in-hand object rotation remains a significant challenge in robotics, largely due to the difficulty of transferring policies from simulation to the real world. The complex, contact-rich dynamics of dexterous manipulation create a "reality gap" that has limited prior work to constrained scenarios involving simple geometries, limited object sizes and aspect ratios, constrained wrist poses, or customized hands. We address this sim-to-real challenge with a novel framework that enables a single policy, trained in simulation, to generalize to a wide variety of objects and conditions in the real world. The core of our method is a joint-wise dynamics model that learns to bridge the reality gap by effectively fitting limited amount of real-world collected data and then adapting the sim policy’s actions accordingly.  The model is highly data‑efficient and generalizable across different whole‑hand interaction distributions by factorizing dynamics across joints, compressing system-wide influences into low‑dimensional variables, and learning each joint’s evolution from its own dynamic profile, implicitly capturing these net effects. We pair this with a fully autonomous data collection strategy that gathers diverse, real-world interaction data with minimal human intervention. Our complete pipeline demonstrates unprecedented generality: a single policy successfully rotates challenging objects with complex shapes (*e.g.*, animals), high aspect ratios (up to 5.33), and small sizes, all while handling diverse wrist orientations and rotation axes. Comprehensive real-world evaluations and a teleoperation application for complex tasks validate the effectiveness and robustness of our approach. Website: [DexNDM](https://meowuu7.github.io/DexNDM/).

Published as a conference paper at ICLR 2026
DEXNDM: CLOSING THE REALITY GAP FOR DEXTEROUS IN-HAND ROTATION VIA JOINT-WISE
NEURAL DYNAMICS MODEL
Xueyi Liu He Wang Y
TTsinghua University "Peking University Shanghai Qi Zhi Institute 'Galbot
P Project website: meowuu7, github. Tealb..ooDeeeeie DexNDM
(A) Challenging Geometries
Any (Allegro)
Desierity (DICla))
Object Asgeet Ratio Maax. Min Esuest)
High Aspect Ratio
High
Jm
Loem
23cm
Object-ro-Hand Ratio
N
Mae' Deen Mowc
4cm
200m 4cm
Low
20bject-to-Hi
fócm
8.0
1,0
1.2
Object
(Max
Sizee)
(B) Complex Shapes
ii
2xis (0.82, 0.03, (.57)
EXIs - 00.71,0,071)
2xis (0.58, 0,58,
(C) Diverse Wrist Orientations
(D) Application: Teleoperating Complex Dexterous Tasks
Up
ii
Palm
Palm Up
Base
Thank Up
Thumh Down
Genual Wise
We introduce DexNDM, sim-to-real approach that enables unprecedented in-hand rotation in the real world. We master a wide object distribution, including (A) challenging geometries (B) complex shapes, across (C) rich wrist orientations. (D) A teleoperation application. Videos in website.
Figure 1:
ABSTRACT
Achieving generalized in-hand object rotation remains significant challenge in robotics, largely due to the difficulty of transferring policies from simulation to the real world. The complex, contact-rich dynamics of dexterous manipulation create a "reality gap" that has limited prior work to constrained scenarios involving sim- ple geometries, limited object sizes and aspect ratios, constrained wrist poses, customized hands. We address this sim-to-real challenge with a novel framework that enables a single policy, trained in simulation, generalize a wide variety of objects and conditions in the real world. The core of our method is a joint- wise dynamics model that learns to bridge the reality gap by effectively fitting limited amount of real-world collected data and then adapting the sim policy's 's ac- tions accordingly. The model highly data-efficient and generalizable across dif- ferent whole-hand interaction distributions by factorizing dynamics across joints, compressing system-wind wide influences into low-dimensional variables, and learning each joint's 's evolution from its own dynamic profile, implicitly capturing these net effects. We pair this with a fully autonomous data collection strategy that gathers diverse, real-worlh interaction data with minimal human intervention. Our

---

Published as a conference paper at ICLR 2026
plete pipeline demonstrates unprecedented generality: single policy successfully rotates challenging objects with complex shapes (e.g., animals), high aspect ratios (up to 5.33), and small sizes, all while handling diverse wrist orientations and ro- tation axes. Comprehensive real-world evaluations and a teleoperation application for complex tasks validate the effectiveness and robustness of our approach.
INTRODUCTION
Advancing dexterous manipulation is essential to achieving highly capable embodied intelligence. A fundamental yet challenging skill in this domain is in-hand object rotation. The long-standing goal, which we also pursue in this work, is to develop a general-purpon policy that can rotate a broad distribution of objects across diverse wrist orientations and rotation axes in the real world.
Despite recent progress, the community has yet to achieve this level of generality. Existing meth- ods (Chen al., 2022; Yang al., 2024; Qi et al., 2023; Wang et al., 2024; Zhao et al., 2025; Yuan et al., 2023) are often constrained to specific scenarios: some assume a consistently up-facing hand, others handle only a limited set of simple, regular-sized objects, and many rely on expen- sive, customized hardware with sophisticated tactile sensing. While some approaches (Yang et al., 2024) show generality in one dimension, such as rotation axes, they are limited in others, like object complexity. To our knowledge, no prior work demonstrates robust, in-the-air rotation for a wide spectrum of bibjects-including complex shapes, high aspect ratios, and varied sizes under diverse wrist orientations and rotation axes.
The primary barrier to this goal is the formidable "sim-to-reas gap". due to the difficulty in modeling the complex interaction dynamics marked by rich, rapidly varying, and load-dependent contacts. This undermines both model-based (Pang & Tedrake, 2021; Pang et al., 2023; Suh et al., 2025) and model-free (Qi et al., 2023; Chen et al., 2022; Yang al., 2024) approaches. A promising idea for sim-to-real transfer is learning a neural dynamics model from real-world data (He et al., 2025; bin Shi et al., 2024). This approach has proven effective in locomotion, where relatively easier fail- ure recovery and readily observable states permit efficient collection of distributionally relevant task data. This success, however, does not easily translate to general-purposs manipulation, where the quirements for data volume and distributional relevance create an inescapable conflict. The need for generality demands massive data to cover diverse objects. Yet, ensuring this data is distributionally relevant is sometimes impossible and operationally far more complex: suboptimal deployable policy cannot manipulate hard objects (e.g.. long); catastrophic failures (i.e., dropping the object) necessi- tates frequent human intervention for resets; severe hand-induced occlusions complicate accurately tracking states of diverse objects. This conflict creates critical bottleneck for the field.
To overcome these challenges, we introduce framework that breaks this inescapable conflict by fundamentally rethinking both the model and the data. Our central insight to factorize the learn- ing problem through a more generalizable dynamics model, which in turn enables a more scalable data collection strategy. First, instead of modeling the high-dimensional hand-object system as a whole (bin Shi et al., 2024), we learn joint- joint-wise neural dynamics model. This model factorizes the system and predicts the evolution of each joint using only its own proprioceptive history, generaliz- ing the idea of RMA (Kumar et al., 2021). This design directly confronts the challenges: it is inher- ently immune to object state estimation difficulty, and by distilling system-wide influences- Self- actuation, inter-joint couplings and object loads-into low-dimensional and task-sufficient net ef- fects with reduced nuisance variability, the model becomes highly sample-efficient and generaliz- able without sacrificing expressivity as evidenced by experiments. This enhanced generalizability is the key that unlocks our second innovation: a fully autonomous data collection strategy. By ap- plying randomized loads to the hand in a task-agnostic manner, we gather data while eliminating catastrophic failures and the need for human resets. This allows us to learn a dynamics model gener- alizing well to our task of interest from cheap and scalable data, which we then use to train a residual iomulatiin-trainee
policy that adapts attain the base policy via a specialist-to-gen to-generalisst pipeline: train category-speciffe experts on data spanning aspect ratios and geometric complexities, then distill them into unified policy.
base policy to the real world, achieving broad generality. We
We validate our method in both the simulation and the real world. In simulation, our base policy gen- eralizes to novel, complex shapes, outperforming strong baselines by 91%.. In real world, our sim-to-real method significantly and consistently improves rotation performance, enabling versatile rotation across diverse wrist orientations and rotation axes on broad object distributionnincludee
2

---

Published as a conference paper at ICLR 2026
Real Data
Whole-Bedly
Dynamiis
Data
Residial Action
Aa.
Real Data
Joint- Wise
Dynamics
Se+1
Reward
Action
Fescbook:
Simulator
Dased Controller
51
Policy
a2
Dynamies
8[+1
Residual Policy
Offiiee Duraeet
(A) Whule-Body Neural Dynamics
[Neural Londer, MB-Max]
(B) Delta Action Model [ASAP, UAN]
(C) Joint-Wike Neural Dynamies [DeNNDM]
Figure 2: Learning from Real-Worke Data for Control. (A) Learn a whole-body dynamics model from
real-world data for policy tuning model-based control. (B) Learn a residual action model to finetune a base policy. (C) Learn joint-wise dynamics and residual policy to adapt the base policy.
complex geometries (e.g., animal models), aspect ratios up to 5.33, and object-to-hant hand ratios of 0.31-1.68 (Fig. 1; videos on our website). Notably, in a challenging downward- figuration, we are, to our knowledge, the first to rotate long objects (10-16 em) around their long axis for about one full circle in the air. Compared to Visual Dexterity (Chen et al., 2022) on a large, customized D' D'Claw, our smaller LEAP hand matches or surpasses performance and succeeds on shapes it struggles with (e.g., elephant, bunny, teapot). We also generalize to a much broader, more challenging object distribution than the prior multi-wrist SOTA (Yang al., 2024). Moreover, we showcase an application enabled by our general rotation policy: building a telcoperation system to perform complex dexterous tasks, such as tool-using (e.g.. screwdriver, knife) and assembly (Heo et al., 2023). A systematic ablation study validates the crucial role of our key design choices in both the dynamics model and the data collection strategy. Our main contributions are four-fold:
hand con-
A novel sim-to-real framework for dexterous in-hand rotation, built on joint-wise neural dy- namics model and autonomous data collection to tackle the core challenges of learning complex interaction dynamics and acquiring real-world interaction data.
An in-hand object rotation policy that achieves unprecedented generality in rotating challenging objects (high-aspect-ratio. complex shapes, small sizes) under difficult wrist orientations. An in-depth analysis of the rationale, advantages, and scope of effectiveness of the joint-wise neural dynamics model from both theoretical and empirical perspectives.
A demonstration of a practical application in teleoperation for complex dexterous tasks.
2 RELATED WORK
Our work is broadly related to two research topics: in-hand object rotation and sim-to-real strate- gies. In-hand rotation is important yet challenging robite task. Despite advances, prior methods still (i) assume an up-facing hand (Qi et al., 2022; Wang et al., 2024; Yuan et al., 2023; Zhao et al., 2025), (ii) handle only normal-sizen objects with limited geometric diversity (Qi et al., 2023; Röstel et al., 2025; Pitz et al., 2024a;b; Yang et al., 2024), or (iii) rely on expensive hardware and sophisti- cated tactile sensing (Yang et al., 2024; Wang al., 2024; Qi et al., 2023). AnyRotate (Yang al., 2024) achieves axis and wrist generality, but only on normal-sized regular objects in the real world. Visual Dexterity (Chen et al., 2022) rotates complex shapes in the air, yet performance on small or high-aspect-ratio objects is unverified. DexterityGen (Yin et al., 2025) showcases a meaningful application of in-hand reorientation policies in teleoperation, but its rotation capability does not ex- ceed the upper bound of prior work. We aim to achieve generality in rotating challenging (e.g., long, small) and complex objects across diverse wrist orientations and rotation axes. A central obstacle to realizing this is the sim-to-real gap: mismatched parameters, model discrepancies, and unmodeled effects derail transfer of simulation-traine policies. Existing approaches include: (1) Domain Ran- domization (DR), which broadens training distributions (Loquercio et al., 2019; Peng et al., 2017; Tan et al., 2018; Yu et al., 2019; Mozifian et al., 2019; Siekmann et al., 2020); (2) System Identifi- cation (SysID), which fits simulator parameters from real data (An al., 1985; Mayeda al., 1988; Lee et al., 2023; Sobanbabu et al., 2025); (3) online adaptive policies (Kumar et al., 2021; Qi et al., 2022); and (4) neural modeling of real dynamics to guide transfer (He al., 2025; Fey al., 2025; Hwangbo et al., 2019). DR relies on heuristic ranges; SysID is bounded by its parameterization; and online adaptation typically depends on dynamics coverage in training. Learning real dynam- ics offers the highest ceiling: A classical line in neural control learns residual or full models for the whole system for model-based control (Fig. 2 (A), egg. Neural Lander (Shi et al., 2018). MB- Max (bin Shi et al., 2024)). As the task complexity increases, learning globally accurate, physically plausible dynamics that is super robust to support policy tuning or controller development is difli- cult (Shi, 2025). Therefore, another trend of methods proposed in sim-to-real RL (e.g., UAN (Fey et al., 2025) and ASAP (He et al., 2025)) learn sim-real delta actions and fine-tune policies based on that to bridge the dynamics gap (Fig. 2 (B)). Success hinges on collecting enough real-world data that is distributionally relevant to the task or can offer a comprehensive coverage- a minor issue in locomotion and static-contact tasks, but major bottleneck in dexterous manipulation. We address
3

---

Published as a conference paper at ICLR 2026
this with generalizable joint-wise neural dynamics model that relaxes the training data distribution requirement, followed by a residual policy to bridge the reality gap (Fig. 2 (C)).
3
METHODOLOGY
Oreck Policy Training
Cakgeia
Generalist
BC
(R) Residual Policy Training
Siin
4p 1a.77 Mt-7
[- B
F-ll Catogary- Specific Specialist
Rellan in
Triectory Datnse
d:-1
Geseralist Policy
9r )ا_
Residani
writ: pose, ror 2815
[-1
wriel painc ssis
(C) Auionmmous Real Data Cullection
(D) Resl-World Denumics Model Training
4t-W+1 Ac-w--
4r-w++ P--w+)
Joint Wise Neumi
Juint-Wise
c++1
Real Raplay
4r-1
L Nounal
4++1
q a,
Dytamics
4r. ae
Dynanses
4
Figure 3: Method Overview. (A) RI-train object category-specifie rotation specialists. (B) Distill them into single generalist via BC. (C-E) Neural sim-to-real: autonomously collect real-world transitions with stochastic contact conditions (C), learn a joint-wise neural dynamics model (D), and train a residual bridge reality gap (E). Deploy the base generalist (B) augmented with the residual (E).
Our goal is a generalist policy that can rotate a wide variety of objects under various conditions in the real world. We adopt a model-free RI. approach. Key challenges are the pronounced sim-to-real dy- namics gap in contact-rich dexterous manipulation and the need for broad object generalization. To achieve broad generalization, we adopt a specialisttoo-generiiistt approach: we first train category- specific oracle policies across curated object categories (Sec. 3.1), and then distill them into a single generalist policy (Sec. 3.2). We address the sim-to-real challenge via neural sim-to-real strategy centered on an expressive, data-efficient, and generalizable joint-wise dynamics model (Sec. 3.3). This model can generalize to the target rotation task by training on imperfectly aligned data, en- abling fully autonomous data-collection system in which real-world interaction data is gathered by applying stochastic contacts to the robotic hand. With the neural dynamics model learned from this autonomously collected data, we train a residual policy that compensates for the base generalist policy's s actions to close the sim-to-real dynamics gap. Workflow illustrated in Figure 3.
3.1 MULTI-WRIST- ORIENTAINNN IN- HAND OBJECT ROTATION ACROSS MULTI-AXIS
We formulate in-hand rotation as a finite-horizon Partially Observable Markov Decision Process (POMDP), M = A,, O, P, R), with state, action, and observation spaces (8, O), transition dynamics P. and reward R. We train a neural policy π O + A with RL to maximize expected
cumulative return over horizon N: x- = arg max, E- (r) EE Et=1 r(st, at)].
Observations and Actions. At timestep t. the policy receives Og: a short history of proprioception, fingertip and object states, per-joint/per-fingee orientation, and the target rotation axis (Sec. A.I). The policy outputs a distribution over relative target position. We sample Aa; T(,)) and update the joint target a, a a Aa, with = 1/24. a, is converted to torques via a PD controller and executed on the robot.
force measurements, binary contact signals, wrist
Reward Function. The reward consists of three weighted components r = Qza + Qgoal goal + openalty"penally with Foo and " Penalty following Rotatelt (Qi et al., 2023). The rotation term Fror en- courages rotation about the target axis. The penalty Upenstty discourages off-axis angular velocity, deviation from canonical hand pose, object linear velocity, and joint work/torque. Since these rewards alone struggle on hard cases (e.g., rotating long objects), we add an intermediate goal-pose pose reward, goal, that guides the object a waypoint on the target rotation axis. Details in Sec. A.1
3.2 GENERALIST POLICY TRAINING VIA BEHAVIOUR CLONING
Having obtained the oracle policy with rich privileged observations for each object category, we use Behavior Cloning (BC) to train the unified, real-world deployable, multi-geometry generalist policy. Although DAgger-style distillation has been effective in prior work, our early experiments on deploying single-object-cate,
downward-facing hand rotation policies, we observe that they either fail to optimize in simulation or collapse in the real world. We attribute this to high task diffi- culty. We therefore switched to BC and discovered that the resulting proprioception--nly specialist policy behaves more stably than the DAgger policy and can rotate simple cylinder objects. We continue to use BC to train the generalist, real-world deployable policy: roll out all oracle policies, aggregate only successful trajectories, and train a generalist via supervised learning. This approach works well hardware. We hypothesize that its success stems from imitating only high-quality
4

---

Published as a conference paper at ICLR 2026
oracle behavior. The observation of of the generalist policy contains history of proprioception ((qk: 1)-1))kk=r__ wrist orientation and rotation axis. We use T = 10 and implement the policy as residual MLP (He et al., 2015).
3.3 CLOSING THE REALITY GAP VIA JOINT- WISE NEURAL DYNAMICS
While the generalist policy is already real-world deployable, persistent sim-to-real cause by mismatched physical dynamies and unmodeled effects preveets it from mastering challenging object interactions. We bridge this gap with novel neural sim-to-real strategy that effectively learns complex, real- world dynamics model.
The central challenge is to acquire useful and sufficient volume of real data so that the learned dynamics model can help sim-to-real transfer. For dexterous manipulation, prior data acquisition methods (Hwangbo et al., 2019; He et al., 2025; Fey et al., 2025; bin Shi et al., 2024) are often impractical. Rolling out a base policy (He et al., 2025; bin Shi et al., 2024) or executing wave actions (Fey et al., 2025) frequently fails on diverse and complex objects, requiring constant human intervention, while imperfect state estimators introduce heavy noise. This leads to real datasets that are small, biased, and insufficient in coverage and quality. We address these challenges by rethinking both model and data. We propose a joint-wise neural dynamics model that dramatically improves sample efficiency and generalizability while preserving expressivity by learning from a task-sufficient representation of the system dynamics.
low-dimensional, information-contractis This allows for an autonomous data collection strategy that gathers diverse, large-scale real-world data by applying stochastic contact, eliminating the need for task sppciifi rollouts and human resets.
Joset-Wise Neural Dynamics. To model the system's 's dynamics without relying on noisy and lim- ited object-state estimation, one way is to learn a "whole-hand" neural model. This model pre- dicts the hand's next state from its length-W W state-action history, q"+1 fo with H, thereby implicitly capturing the whole system dynamics, including external forces from the object (Qi et al., 2022). However, this approach remains data-hungry, inheriting the other data acquisition challenges described above.
(q.aa) a
Our solution is to factorize the problem. We introduce joint- wise neural dynamics where the dynam- ies of each joint i are modeled as H. + G = T/. where H.". G R are low-dimensional effective terms that distill high-dimensional, system-wide influences such as inter-joint coupling, actuation, and object-induced effects. The neural model then predicts the next state of each joint i from its own W-step state-action history: q fe, (ht) with hy = {q}. a 1 torization is effective as it acts as an information bottleneck, forcing the model to discard spurious correlations and learn only the essential dynamics of each joint. This projected history is sufficiently informative with enough information to accurately predict the joint's 's next state (Sec. 4.2, A.3). At the same time, it is also robustly simple as it is too -imeennsiional to permit the reconstruction of the original high- sigh--imensional system-widd influences, thus avoiding the need to model irrelevant complexity (Sec. A.4). The direct consequence is model that highly sample-efficient and gener- alizes broadly across interactions, yet retains expressivity (Sec. 4.2). We now provide a theoretical analysis to formalize why this simplification leads to better generalization.
+- - This fac-
Theoretical Rationale: Generalization via Information Contraction. We write the whole-hand model as fo = ( with q = fH(H,), and the joint-wise model as 4 = f3. (1()- Let P be the target distribution for (Il, 4i+1) (e.g., formed by task of our interest); consider a different distribution Q and the projection g (H,,gi+1) + (hi, of+1). ). i.e., : R2wdd x R + Raw x R. We compare the prediction error of joint on the target distribution P achieved by these two types of model, i.e., S/ and f to support the generalization benefit:
Claim 3.1 Under assumptions typical of our setting, V1 Si S d, the joint-wise model fú trained on g(Q) generalizes to g(P) better than the whole-hand model f trained on Q generalizes P.
We first show that, under mild assumptions typically satisfied in our setting, the projection 9 con- tracts distribution shift: KL(g(P)Ilo(Q) < <KL(PI]Q) (Theorem 3.1, proof deferred to Sec. A.2).
Theorem 3.1 (Data Processing Inequality for KL (strict form)) Let P and Q be probability dis- tributions on Rn x R with densities P and Q with respect to a common base measure. Let E RD R be measurable, g( 9())). Moreover, the inequality is strict if is ((P) and 9 Then KL(P Q) Z KL(g(P) ||
g X E R x R
< n, and denote the pushforwards by
non-injective in way that merges points where P and Q have a different relative structure. More
5

---

Published as a conference paper at ICLR 2026
concretely, it indicates that if there Byo R P(Y yo) 0, P(X yu) * Q( = yo),,
then (T (1 > (T)( ll (()))
The contraction of divergence implies tighter generalization guarantees (Theorem 3.2, proof A.2):
Theorem 3.2 (Generalization Gap Contraction) Let (X.Y) E R" x R and g(X.V Y) (gx (X) with 9x B. R.m m < n. Let P, be distributions on (X.Y) satisfy- ing covariate shift, i.., P(Y x) Q(Y X). Let L be a loss bounded by B, and de- fine (h) = (xxx)))L((((()))))))
If (L(()))) < KL((P])), then for function fi x + Y and f2: f2: + Y: sup/ R2 o 9x) Ro( o 9xxl < suup Rp - -
Assuming f2 a 9x is sufficiently expressive and relatively large domain shift from Q to P (typical of our setting), f2 ogx has lower prediction error than fi on target domain P, establishing Claim 3.1. See Sec. A.2 for details. In practice, we pretrain the model on simulation data for initialization.
Autonomous Data Collection. Our model's ability to generalize from distributionally dif- ferent data motivates our second innovation: a low-cost, autonomous data collection strat- egy. This approach, which we call the "Chaos Box" (Fig. 3(C)), embodies four principles: (i) policy-awareness (to roughly align the distribu- tion), (ii) object-loaded interaction, (iii) broad coverage, and (iv) scalability. The implementation is simple: the robotic hand is placed in container of soft balls. We then open-loop replay actions from the simulated base policy. which provides a coarse distributional prior (i). The hand's interac- tion with the balls imposes rich, stochastic contacts (ii-iii). With probability 0.5, we add Gaussian noise (a=0.01) to each action to broaden coverage (iii). This entire process is fully autonomous, hardware-safe, and requires no human resets (iv). Fig. 4 supports our model and data designs: I/O histories of a joint cover the task-relevant distribution, whereas histories of the whole hand do not.
ISNE Visualization @ laak Relevent Trajettnics
Astonenonally C Collater Ingecsario
(A) Single
Whole
Figure 4: State-Action History Distribution.
Bridging the Dynamies Gap via Residual Policy. Using the learned dynamics sar we train a residual policy res that compensates the base policy's actions to bridge the dynam- ies gap (Fig. 3(E)). By adding corrective term to the base action, it wishes that the result- ing state transition of the interaction system in the real world closely matches what would oc- cur in simulation. Concretely, given the base policy's observation of and base action a,, res outputs correction at and to match the simulator's 's next state qt+1, we solve n res*
arg min E,pp, (r) - - - f (fq; aj + π. T (of a;));== -=:-:++]/ w+1)ll We solve it by training res in a supervised manner on the trajectory dataset used to train the base policy. At de- ployment, we execute a, + a.th We opt not to use fu for policy training or finetuning the policy directly, which would require globally accurate, penetration-free contact-rich dynamics (including the object) and super robustness to -uttt--distribution exploration (Shi, 2025). See See. B.4 for a discussion on residual policy vs. direct finetuning.
4 EXPERIMENTS
We extensively evaluate our method in simulation and real world against strong baselines (Sec. 4.1). In simulation, our generalist policy generalizes to unseen geometries for multi-wrist poses, multi- axis rotation. On hardware, it achieves unprecedented in-air rotation with LEAP hand (Shaw al., 2023) under challenging wrist poses on difficult objects, including long (13,5-20cm), small (2-3cm) objects, and complex animal shapes (Sec. 4.2). We also show a teleoperation setup that pairs the policy with VR to perform complex dexterous tasks (Sec. 4.2), such as tool-using and assembly.
4.1 EXPERIMENTAL SETTINGS
Training and Evaluation Protocols. We create an object dataset spanning aspect ratios, sizes, and complexity with randomized physical properties for training. We split objects into five categories and train an oracle policy for each with PPO (Schulman et al., 2017) in Isaac Gym (Makoviychuk et al., 2021). We use objects from ContactDB (Brahmbhatt et al., 2019) as the test set in simulation to evaluate the generalization ability to shape variations. We evaluate rotation across randomized wrist orientations and four rotation-ax -xxis groups: Iz, Hy, #2, and a general axis set with 26 axes. We evaluate on three object sets in the real world (Fig. 5): (1) regular objects (including a high-aspect- ratio cuboid); (2) small objects; and (3) normal-sized irregular objects. Objects shown in purple

---

Published as a conference paper at ICLR 2026
and all small objects are unseen. We evaluate on three principle axis sets and cubic-diagonal set: (1,1,1, (1,0,1), (1,1,0), (0.1.1). Results are averaged over objects and reported as mean + standard deviation across three independent evaluations. Details in Sec. C. Baselines. We compare against in-hand rota- tion/reorientation 2024) and Visual Dexterity (VD) (Chen et al., 2022) and sim-to-real methods UAN (Fey et al., 2025) and ASAP (He et al., 2025). AnyRotate's 's code is unavailable and relies on specialized tactile sensing, so we use our re-implementation in simulation; on hardware, we evaluate on their replicable objects and compare to their reported performance. A direct comparison to VD is impractical: adapting their D' D'Claw code to LEAP failed to behave well in simulation, so we compare their qualitatively results (link). UAN and ASAP, designed for arms/legged robots and not modeling objects, are adapted by training compensators on object-free transitions; making them object-aware is nontrivial (see Sec. D).
(Yang et al.,
(As Regular Ohjects
(B) Sunil Chojets
Inrepsiar Chjjects
Figure 5: Objects for Real Experiment.
Metries. We evaluate using Rotatelt metries (Qi et al., 2023), plus goal-oriented success: Time- to- oo-Fall ((FF)duration until termination; in simulation, episodes are capped at 400 steps (20s) and TTF is normalized by 20s, while in the real world we report raw time; Rotation Reward (RotR)episte episode sum of w k (simulation only); Rotation Penalty (RotP"-per-step average w x k (simulation only); Radians Rotated (Rot)-total radians rotated in the real world; Goal-Oriented Success (GO Succ.) following Visual Dexterity: sample a goal pose; set the target axis to the rela- tive rotation axis; count success if the orientation is within 0.1 of the goal (simulation only).
4.2 IN-HAND ROTATION RESULTS AND ANALYSIS
Simulation Results. Our policy generalizes to unseen objects and outperforms our re-implemented baseline (Table 1). Among all settings, rotating along the gravity direction (# axis) is the easiest task, similar to the observations made in prior works (Qi et al., 2023; Yang et al., 2024).
Mateed
Budk 1
Ruatt
TIF1
TIF
Baak
Gened Roesion Ave
TEP
Sacc
Go
AngRoeate" (ee (ne-brgheramttttin Ouer Geserdol Sim))
DI 20
BA7
161 ao
163 10
0.7
854
234 234
354.28
341.30 00
Table I: Generalization Test in Simulation. Comparisons of the rotation performance on the unseen test object set along each axis with hand wrist orientation randomized over rotation metrics.
Doeaten Aolk
Toa Orforadion
Saraiie Ade
Beratin Ase
Drleradinn
Boarde Arth
Crowssee
27
29 1M
1946 2M0 pei
Ne
48 680
207 PM
40.00
13
47 222
1:0 20
Table 2: Comparisons to AnyRotate. Comparison of rotation degrees (Rot (radian)) and time-to-fall (TTF (s)) under two test settings introduced in AnyRotate (Table 12, 13) on replicable objects.
Melioo
Coe
Borr
Tink db
GRAZ Siyher
Sunay
Co
Toqua
Dogge
Teae 88
Haskeporr
Mar
Destonts BesNDR
Table 3: Comparisons to Visual Dexterity of Survival Angles ((radian/O.3) /0.5T roughly measuring (from vidcos) how many 90 degrees the object can be rotated before falling. The subseript denotes the performance achieved by rotating the object with a supporting table.
THE Rock Laid Docacial
Figure 6: Comparisons to Whole-Hand Neural Dynamics w.nf. Model Expressivity, Sample Efficiency and Transferrability. (A,A-0) In-domain and out-of-distribution performance in high (M) and low (7.5k) data regimes. (B) Sample efficiency. (C) Transferrability from different training distributions.
Real World Results. Our sim-to-real method consistently improves real-world performance, and the policy exhibits unprecedented dexterity, rotating aigh-aspect-ratio geometries, small objects, and complex shapes under challenging hand wrist orientations in the air (Tables 4 (multi-axis with palm- down), 5 (multi wrist-pose, 2-rot); Fig. 1; Fig. 22, object gallery (Fig. 21) (in Appendix); videos). Contrary to AnyRotate, which finds "Thumb Up/Down" most difficult, we observe "Base Up/Down" are harder, likely due to different actuator performance between Allegro and LEAP.
Comparisons to AnyRotate. We evaluate on four replicable items from AnyRotate's suit-"Tin Cylinder", Cube, "Gum Box", and "Container" (Sec. whih are their most difficult cases (ac- cording to Table 12-13), and compare with their reported real-world results. Table 2 shows our
7

---

Published as a conference paper at ICLR 2026
Objject Set
Mathod
Rot (rad)
dx-axis
TTF (s)
Rot (rad)
dy-axis
TTF (S)
Rot (rad)
Ez-axis
TTF (s)
Rot (rad)
Cubic Diagonal Axes
TTF (6)
Direct Transeer
9.84_aas
26.80
10.3740.55
30.73_1.07
11.09 so
21.872224
g.00
22.0
Regular
Whole Hand NDM
g5.04g:.m
8.50
7.38 1..8
16.33 79
3-30
8:87 10.0
DexNDM
11.36
32 32.40g11
14.24g1.9
44 1..6020 44
23 23.8
87 37.502a00
16.93 16.93
30, 30.44:3.0
Direct Transfer
4.7
25.17
6.11
26. 22:1.0
6.9
20.17
5.402asa
+3.2
Small
Whole Hand NDM
0,30
0.44
0.87.a.t
1.38
0.00ga.co
0:260-2.14
DexNDM
24
28. g0
6.81
29 25
9. 9.29
26.75
6.03gas1
27. 27.34
Direct Tramsfer
4.41a0s
10.95 20.96
6.13
2162-2204
5. 5,20,
21.19
6.53gn37
26 25
Ireeularr
Whole Hand NDM
1.53
2.91
10.32.6.72
0. 0.73aus
1.08.12 12
1.08
2.33
12.0
DeXNDM
24.21g227
11.32
39.04g72s
8.61
29.3
29.33
9.19
33.14g1.0
Table 4: Multi-Axis Rotation in Real. Comparison of rotation degrees (Rot (radian)) and to- (TTF (s)) along each axis under the palm down wrist orientation. The metric was first averaged over all objects within each trial. We then report avg. std of these results across three independent trials.
Melsed
Puico Up
Pulto Down
Base Up
Baie Down
Thank Up
Thunth Dwan
Ror (evi)
Rix (mit
TTP ())
Rex (ovi)
TTP 05
Ror (ruh
TTP 03
Ror inv)
Ror (nut)
TTF 60
Direct Thansfer
10:00_xus
5 5.
18 87 2oas
6.40
20.6
as.2s
DesNDM
14.82.5.1
32
13.200.mm
29
9.62
566
7.50
44
11.90..1.9
26
8.80..r
Table 5: Multi-Wrist Orientation Rotation in Real. Comparison of rotation degrees (Rot (radian)) and time-
to-fall (TTF (s)) under six representative hand orientations across direction z.
method substantially outperforms AnyRotate and is more versatile: whereas AnyRotate targets mod- erately sized, simple shapes (min 5cm, max aspect ratio 1.67) with conservative motions, our policy handles smaller objects (3cm) and high aspect ratios (up to 5.3) with sophisticated finger gaiting, Comparisons to Visual Dexterity. A direct comparison with Visual Dexterity (VD) infeasible due to differing task definitions (axis-oriented continuous rotation vs. goal-oriented reorientation). To enable comparison, we introduce the survival rotation angle metric: the angle an object rotated before being dropped. We estimate VD's best performance by analyzing their videos. Despite this metric favoring VD (their setup sometimes includes a supporting table), we achieve comparable or superior results on their showcased and replicable objects (Table 3). Besides, we can uniquely manipulate small objects and high aspect ratios as well as handle diverse wrist orientations (Fig. 22). Comparisons to Whole-Hand Nueral Dynamics. We compare against the whole-hand dynamics model to answer: (Q1) Does predicting each joint's transition from its own history (without global information) reduce expressivity?
(Q2) Is our model more sample-efficient? (Q3) Does it generalize better? (A1) Trained on 3. IM simulated trajectories and evaluated in-domain,
model is nearly as expressive as the whole-hand model (Fig. 6(A, column ()AA0)). (A2) With limited data Mata-using 7.5k autonomously collected trajectories in the real world (Fig.6(A, column 3)) and across varying real- world dataset sizes (Fig.6(B))oo) our model achieves better in-domain performance, indicating higher sample efficiency. The advantage is more obvious under insufficient data settings. (A3) On an OOD real-world test set (task-relevant transitions under "Thumb Up" wrist), our model generalizes much better in both high- and low-data regimes; see Fig.6 column 2,4) and F(g. Fig. 6(C) sys- tematically studies the cross-domale transferability in various settings. Summary: For data-driven neural dynamics, joint-wise model significantly outperform whole-hand models in insufficient-data or train-tes distribution-shift settings; with ample data and in-domain evaluation, performance is similar, with only a slight loss in expressivity for joint-wist models.
Comparisons to ASAP and UAN. We implement UAN and ASAP, but their resulting policies fail en- tirely in real-world tests-unable to rotate even a simple cylinder (Fig. 36; videos). We attribute this to an OOD issue: compensators trained solely on free-hand data do not generalize to the interaction dynamics introduced by manipulated objects. Please note that their methods can only use either free- hand data or task-relevant data with object diftss-d and noisy to obtain, and unusable even for compensator training-aan andding-anddnn cannot leverage our autonomously collected data with randomized object loads; see Sec. D. Our strategy is more tolerant of real-data imperfections (Figs. 8, 9, 96).) "Sim-to-Sim" Comparisons. We conduct a cross-
Sieclatte
Granc
MageC
simulator transfer evaluation (Isaac Gym to Gene- sis and MuJoCo). We collect object-loaded rotation data in the target simulator for training. Table 4.2
Mithed
Res
RedR
RaP
Diver Trinder
15:0 072baar 82.03.5..
ASAP DrENOM
75 72 1811g
1.48gno 25.25
15
111.2 -mm 19.20...: Mallaas 334.9) 18.9....
Table 6: "Sim-to-Sim" Transfer.
shows our method consistently surpasses prior work, owing to designs on dynamics modeling, higher data efficiency, and practical choices (e.g., pre-train in source sim). We find UAN outper- forms ASAP, likely because its history-based design better captures object effects. Details in Sec. C. Applications. A meaningful application of in-hand rotation/reorientation policies is their integration into teleoperation systems to enhance manipulation capabilities (Yin et al., 2025). In our work, demonstrate a similar application using our in-hand rotation policy within a teleoperation setup (with a Meta Quest 3). The robotic hand is controlled by our rotation policy, conditioned on the
8

---

Published as a conference paper at ICLR 2026
(A) Tool-Using (hammer. beodh, pee. grrigg, aut)
(R) Furniture Assembly (feundez calós, lightelh)
Figure 7: Application. Our rotation policy enables a teleoperation system to perform complex, long-horizon manipulation tasks. See videos and more results on our project website.
rotation-axis commands sent by the human operator, while the robot arm is teleoperated through the operator's arm motion. Details are provided in Sec. C. We demonstrate its strong ability in performing ang-horizon and complex dexterous manipulation tasks (Fig. 7, videos).
ABLATION STUDIES
(K) Dynomies Medel Graeritatiie
(8) Red World Perferarase
Oiject
Aut Super
Figure 8: Ablation Study of the Dynamies Model. (A) Generalization error of different model ablations (lower better). (B) Corresponding real-world task performance.
Reppariud Ookject
Scating Quanity Cellecue Duratlms
Tak Maat-
Loeld
Thilo fari-aam
Tapeinu TherCatt
Figure 9: Analysis of Data Collection Strategies. (A) Time efficiency of different collection methods. (B) Resulting model performance on datasets of equal size. (C) Performance scaling with dataset size and data collection itcrations, including a power-law for extrapolation.
We conduct ablations to validate key design choices of our method. Real-world experiments are performed with the hand fixed palm-down, evaluating z-axis rotation; data are collected under the same wrist pose. Dynamics model are evaluated an OOD test setting, See Sec. for details.
Designs on the Joint-Wiss Neural Dynamics Model. We ablate five design choices: (i) joint- wise vs. finger-wise (per finger prediction from its own history) and whole-hand modeling: (ii) simulation pretraining: (iii) injecting noise into replayed actions during real-world data collection; (iv) collecting with object loads rather than free-hand w/o load; and (v) replaying policy rollouts instead base waves (Fey et al., 2025). As summarized Fig. 8, these choices consistently improve learned dynamics generalization and real-world performance.
Real-World Data Collection Strategies. We compare our autonomous data collection against three baselines task-aware with vision-based object states, task-aware without object states, and free- hand
limitations, efficiency, and model performance (Figure 9). Task-aware pipelines are slow and intervention-heavys estimating object poses prohibitively slow (~200ss on average), requires continuous human supervision, yields noisy poses and complex setup, and fails on small, occluded, or akis-symmetri objects; without vision they still need intervention, remain slow (42.86 s), and produce low-diversity, low-coverage data (data restricted policy's 's ability). In contrast, our method fully automated and, by continuously varying hand loads, collects diverse data spanning wide range of external influences. Figure 9(B) shows the resulting performance

---

Published as a conference paper at ICLR 2026
gains: broader coverage improves prediction, and the joint-wise model is most robust to training- distribution shifts, whereas other variants tend to overfit to the source data.
Scaling with Real-World Data Quantity and Collection Iterations. As shown Fig. our performance improves with more real-world data. However, iterative data align real- and simulated transition distributions for better policy updates-yields only modest gains. We hypothesize this is because the dynamics model already generalizes well, and adding noise to replay actions provides broad coverage, reducing sensitivity to this distribution shift. In contrast, the whole-hand model benefits little from additional data, especially under autonomous collection, likely due to its higher dimensionality and a distributional mismatch between autonomous data and rotation task transitions. A simple extrapolation suggests matching 4.000-trajectory result would require 7.5M task-aware trajectories (417k hours; 52k 8-hour workdays), which is impractical. While approximate, this highlights the superiority of our approach.
to
CONCLUSIONS AND LIMITATIONS
We propose neural sim-to-real framework centered on joint-wise neural dynamics model and autonomous data collection. This enables unprecedented dexterity in rotating challenging objects. The main limitation is that the model's performance ceiling is constrained by its reliance on partial, proprioception-onl observations, as well as task- and hurdware-specife evaluation. Jointly model- ing hand-ofe object transitions using richer sensory signals, incorporating vision and tactile feedback, and extending the framework to a broader set of tasks and hardware platforms represent valuable directions for future work.
REFERENCES
Chae H. An, Christopher G. Atkeson, and John M. Hollerbach. Estimation of inertial parameters of rigid body links of manipulators. 1985 24th IEEE Conference Decision and Control, pp- 990-995, 1985. 3,
Hao bin Shi, Tingguang Li, Qing Zhu, Jiapeng Sheng, Lei Han, and Max Q.-H. Meng. An effi- cient model-based approach on learning agile motor skills without reinforcement. 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 5730-5730, 2024. URL hitpss:/aage www.ntiisschheeeo wwwhholar.rgg/oooooo ID 226228331. 2,
Samarth Brahmbhatt, Cusuh Ham, Charles C. Kemp, and James Hays. Contactdb: Analyz- ing and predicting grasp contact via thermal imaging. 2019 IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pp. 8701-8711, 2019. URL https://api.
semant. www.cccllcooooo watholar.co org. wwwa.orgg/ooouuseeee 1D:
Tao Chen, Megha H. Tippur, Siyang Wu, Vikash Kumar, Edward H. Adelson, and Pulkit Agrawal. Visual dexterity: In-hand reorientation of novel and complex object shapes. Science Robotics, 8, 2022. URL https: http:/pp...aaiiiiieeeee semanticscholas wttp/oopgsaooooooeooooooooeeeee/ : 253734517. 2, 36, 38
Ho Kei Cheng and Alexander G. Schwing. Xmem: Long-term video object segmentation with atkinson-shiffrin memory model. In European Conference Computer Vision, 2022. URL https://agp semant lascholarr ar. Argg/CrrpusDD/ : 250526250. 30
Xuxin Cheng, Jialong Li, Shiqi Yang. Ge Yang, and Xiaolong Wang. Open-television: 'eeleoperation
with immersive active visual feedback. In Conference on Robot Learning, 2024. URL https:
/ //api. semanticschoo
org/ Corpus 2708 99033. 43
John J Craig. Introduction robotics: mechanics and control, 3/E. Pearson Education India, 2009.
19
Mare Peter Deisenroth and Carl Edward Rasmussen. Pilco: A model-based and data-efficient ap- proach to policy search. In International Conference on Machine Learning, 201 44
Runyu Ding, Yuzhe Qin, Jiyue Zhu, Chengzhe Jia, Shiqi Yang, Ruihan Yang, Xiaojuan Qi, and Xiao- long Wang. Bumny- visionpo: Real-time bimanual dexterous teleoperation for imitation learning. 2024. https:::: //arxii .ow.iioooroooooooooooooo abb /2407 031 62. 43
10

---

Published as a conference paper at ICLR 2026
Nolan Fey, G. Margolis, Martin Peticco, and Pulkit Agrawal. Bridging the sim-to-real gap for athletic loco- cooo-manpullatin. ArXiv, abs/2502.10894, 2025. URL https://api. semanticschon org/ :tt.://ccc......aoooooooooaaaaaa
7, 30, 44
Victor Guillemin and Alan Pollack. Differential topology, volume 370. American Mathematical Soc., 2010. 19
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2015. URL https: /app.. semant escholar. org/ http://o:p:oitSs/////.
Tairan He, Jiawei Gao, Wenli Xiao, Yuanhang Zhang, Zi Wang. Jiashun Wang. Zhengyi Luo, Guanqi
He, Nikhil Sobanbab, Chaoyi Pan, Zeji Yi, Guannan Qu, Kris Kitani, Jessica Hodgins, "Jim" Fan, Yuke Zhu, Changliu Liu, and Guanya Shi. Asap: Aligning simulation and real-world physics for learning agile humanoid whole-body skills. ArXiv, abs/2502.01 143, 2025. URL https:
//api. semant www.mmmiiccooaa.ooooooooooo wtg/ Satgg/CorpsII:222. 27000I:::769500000 2, 3, 4.
Minho Heo, Youngwoon Lee, Doohyun Lee, and Joseph J. Lim. Furniturebench: Reproducible real-world benchmark for long-horizon complex manipulation. In Robotics: Science and Systems, 2023. 3
Jemin Hwangbo, Joonho Lee, Alexey Dosovitskiy, Dario Bellicoso, Vassilios Tsounis, Vladlen Koltun, and Marco Hutter. Learning agile and dynamic motor skills for legged robots. Sci- ence Robotics, 4, 2019. URL https: httt::://ap..mmaaiissssseeeeeeee
semanticscholl
org/ CorpusID:
58C31572. 3,5, 5, 44
Ashish Kumar, Zipeng Fu, Deepak Pathak, and Jitendra Malik. Rma: Rapid motor adaptation for legged robots. ArXiv, has/2107.04034, 2021. URL https: sttp:/ppsssttitccchhooooooooooea
org/CorpustD: 235650911 6. 2,
Taeyoon Lee, Jaewoon Kwon, Patrick M. Wensing, and Frank C. Park. Robot model identification and learning: A modern perspective. Annu. Rev. Control. Robotics Auton. Syst., 7, 2023. 3, 44
Antonio Loquercio, Elia Kaufmann, René Ranftl, Alexey Dosovitskiy, Vladlen Koltun, and Davide Scaramuzza. Deep drone racing: From simulation reality with domain randomization. IEEE Transactions on Robotics, 36:1-14, 2019. URL https: api. semanticscholar.c.
CorpusID:16218339
4.
Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, et al. Isaac gym: High performance gpu-based physics simulation for robot learning. arXiv preprint arXiv:2 10470, 2021.
Hirokazu Mayeda, Koji Yoshida, and Koichi Osuka. Base parameters manipulator dynamic mod- cls. Proceedings. 1988 IEEE International Conference on Robotics and Automation, pp. 1367- 1372 vol.3, 1988. 3, 44
Melissa Mozifian, Juan Camilo Gamboa Higuera, David Meger, and Gregory Dudek. Learning domain randomization distributions for training robust locomotion policies. 2020 IEEE/RS/ In- ternational Conference on Intelligent Robots and Systems (IROS), pp. 6112-6117, 2019. URL https::/aag semant cccchochlloooo ar. wrog http/CrrpussD. 2041 85733. 3,
James R Munkres. Analysis manifolds. CRC Press, 2018. 19
Richard M Murray, Zexiang Li, and S Shankar Sastry. A mathematical introduction to robotic manipulation. CRC press, 2017. 19
Michael O'Connell Guanya Shi, Xichen Shi, Kamyar Azizzadenesheli, Anima Anandkumar, Yisong Yue, and Soon-Jo Chung. Neural-fly enables rapid learning for agile flight in strong winds. Science Robotics, 7, 2022. URL https: / api. semant iescholar org/ CorpusID. ID: 248527107. 44
Tao Pang and Russ Tedrake. A convex quasistatic time-stepping scheme for rigid multibody systems with contact and friction. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 6624- IEEE, 2021. 19

---

Published as a conference paper at ICLR 2026
Tao Pang, HJ Terry Suh, Lujie Yang, and Russ Tedrake. Global planning for contact-rich manipula- tion via local smoothing of quasi-dynamm contact models. IEEE Transactions on Robotics, 2023.
2
Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, and P. Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 1-8, 2017. URL https: /ttp://appp... semant www.steetacnnntnoetttt www.ttiiichhhoooooooooooaoeeeeee orrg/
CorpusID: 3707448. 3, 44
Johannes Pitz, Lennart Röstel, Leon Sievers, and Berthold Bauml. Learning time-optimal and speed-adjustable tactile in-hand manipulation. 2024 IEEE-RAS 23rd International Con- ference on Humanoid Robots (Humanoids), pp. 973-979, 2024a. URL https://ap..
semanticscholars org/ htt.ho/aa..ot//oorruuaa:::222200 /ttp///aa:::I11S0///
Johannes Pitz, Lennart Röstel, Leon Sievers, Darius Burschka, and Berthold Bauml. Learning a shape-conditioned agent for purely tactile in-hand manipulation of various objects. 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 13112-13119, 2024b. URL https::/appt
semanticcen wis.ttcascchca.c. .tr.iaar..rrrr.ooeeeee..e Cotpusrrruuuucceecceoceooee ID: 2715161119.. 3
Haozhi Qi, Ashish Kumar, Roberto Calandra, Yinsong Ma, and Jitendra Malik. In-hand object rotation via rapid motor adaptation. In Conference Robot Learning. 2022. URL https: 3, 5, 19, 25, 26, 36, 37
//api. www.stematnittne wwnnnticcchlaaa org.
Haozhi Qi, Brent Yi, Sudharshan Suresh, Mike Lambeta, Y. Ma, Roberto Calandra, and Jitendra Malik. General in-hand object rotation with vision and touch. ArXiv, abs/2309.09979, 2023. 3, 4. 7, 14,
URL https::///// //ap://apii... www.eeeeatccaiieeeeeeee wtt.:aaccsccccoaoooooooooooooeee 25, 26, 37
Lennart Röstel, Dominik Winkelbauer, Johannes Pitz, Leon Sievers, and Berthold Bauml. Com- posing dextrous grasping and in-hand manipulation via scoring with a reinforcement learn- ing critic. ArXiv, abs/2505.13255, 2025. URL https:://appp semartisoeme
:S4puSID:27878865
Fereshteh Sadeghi and Sergey Levine. Real single-image flight without single real image. ArXiv,
abs/1611.04201, 2016. 44
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proxi- mal policy optimization algorithms. ArXiv, abs/1707 004,0777776377, 2017. URL https://apt.
wewee.org/CCoe wrg/CorpussiiCie ID: 28:2286999 28695052. 6
Kenneth Shaw, Ananye Agarwal. and Deepak Pathak. Leap hand: Low-cost, efficient, and an- thropomorphic hand for robot learning. ArXiv, abs/2309.06444 2023. URL https://aagt semant ccccolllcllloooooocooooco aaro org/ Corpus ID:
6, 41
Guanya Shi. From sim2real 1.0 to 4.0 for humanoid whole-body control and loco-manipulation. 2025. URL sttps://opendze: From Sim2Real_ to 1e__o__400o____aooooaiaee_e_ Bttaaaaaaaaaaaaaaoaaaaaaaaao___e Bod_le--oody_Coorro-e .pdf. 3,
Shi
Guanya Shi, Xichen Shi, Michael O'Connell, Rose Yu, Kamyar Azizzadeneshelli Anima Anandku- mar, Yisong Yue, and Soon-Jo Chung. Neural lander: Stable drone landing control using learned dynamics. 2019 International Conference on Robotics and Automation (ICRA), pp. 9784-9790, 2018. URL https: api wwwanteeeiiie crrmanticscholar.. org/ Io: :tt.aaarrrorr/oooppa:::::::0
3, 44
Jonah Siekmann, Yesh Godse, Alan Fern, and Jonathan W. Hurst. Sim-to-real learning of all common bipedal gaits via periodic reward composition. 2021 IEEE International Confer- ence on Robotics and Automation (ICRA), pp. 7309-7315, 2020. URL https://api.
semaaticscholar.oom btt.eeeeeee........ooooooooooooa
Nikhil Sohanbabu, Guanqi He, Tairan He, Yuxiang Yang, and Guanya Shi. Sampling- system identification with active exploration for legged robot sim2real learning. ArXiv, abs/2505. 2025. URL https: api. semant icscholar. org/Corpus IIp 2787 77777766644 68643. 3, 44
Mark W. Spong, Seth A. Hutchinson, and Mathukumalli Vidyasagar. Robot modeling and control. 2005. URL https: //tpt//ppp.. semant aascholar.ore wwwporr///ooooooooooooo 10677777:066677335 19
12

---

Published as a conference paper at ICLR 2026
Mark W Spong, Seth Hutchinson, and M Vidyasagar. Robot modeling and control. John Wiley &amp, 2020. 19
H. J. Terry Suh, Tao Pang, Tong Zhao, and Russ Tedrake. Dexterous contact-rich manipu- lation via the contact trust region. ArXiv, abs/2505.0229) 2025. URL https://api.
semanti www.nniccchoaa.....m www.olaarrrrrrooooooooooooooeoeo wwtgCorppssrrCCaaoooooo ID : 278327864. 2
Omid Taheri, Nima Ghorbani, Michael Black, and Dimitrios Tzionas. Grab: A dataset of whole- body human grasping of objects. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IV 16, pp. 581-600. Springer, 2020. 36
Jie Tan, Tingnan Zhang, Erwin Coumans, Atil Iscen, Yunfei Bai, Danijar Hafner, Steven Bohez, and Vincent Vanhoucke. Sim-to-real: Learning agile locomotion for quadruped robots. ArXiv, htss1804.11332.. 2018. URL https:// www.ssiccchoooool....... http:s//app..emmntcccchhoo.... .org/
CorpusID: :1: 3, 4.
Russ Tedrake and the Drake Development Team. Drake: Model-based design and verification for robotics, 2019. URL https: mit. edu.
Jun Wang, Ying Yuan, Haichuan Che, Haozhi Qi, Ma, Jitendra Malik, and Xiaolong Wang- Lessons from learning spin" pens". arXiv preprint arXiv:2407.18902, 2024. 2,
Bowen Wen, Wei Yang, Jan Kautz, and Stanley T. Birchfield. Foundationpose: Unified 6d pose estimation and tracking of novel objects. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 17868-17879, 2023. URL https://api.
semanticscholest
:etpg//ooousI::::0. 266191252. 44
Max Yang, Chenghua Lu, Alex Church, Yijiong Lin, Christopher J. Ford, Haoran Li, Efi Pso- mopoulou, David A. Barton, and Nathan Lepora. Anyrotate: Gravity-invariant in-hand object rotation with sim-to-real touch. In Conference on Robot Learning, 2024. URL https: //api. swmantisschthtl
wrg/Corpusutio 269757396. 66977// 2, 25, 38
Zhao-Heng Yin, Changhao Wang, Luis Pineda, Francois Hogan, Chaithanya Krishna Bodduluri, Akash Sharma, Patrick Lancaster, Ishita Prasad, Mrinal Kalakrishnan, Jitendra Malik, Mike Lambeta, Tingfan Wu, Pieter Abbeel, and Mustafa Mukadam. Dexteritygen: Foundation con- troller for unprecedented dexterity. ArXiv, abs/2502.04307, 2025. URL https://api. org
semanticscholse www.oolaa.rr/oooooooooeee :I: 3,8
Wenhao Yu, Visak C. V. Kumar, Greg Turk, and C. Karen Liu. Sim-to-real transfer for biped locomotion. 2019 IEFE/RSJ International Conference on Intelligent Robots and Sys- tems (IROS), pp. 3503-3510, 2019. URL https:/ http:/pp..saaaaicccccc.ooooooooe org/ CorpusID: 67856268..886888 4.
Ying Yuan, Haichuan Che, Yuzhe Qin, Binghao Huang, Zhao-Heng Yin, Kang-Won Lee, Yi Wu, Soo-Chul Lim, and Xiaolong Wang. Robot synesthesia: In-hand manipulation with visuotactile sensing. 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 6558- 6565, 2023. URL https: / //pp..p...//// semantics cwolar.c.....oo.oooooooocoo.a... www.hhr.rrgg.ooooeeooooooooooooe :ttp:/oopuSDD::660440.. 2,
Shuqi Zhao, Ke Yang, Yuxin Chen, Chenran Li, Yichen Xie, Xiang Zhang, Changhao Wang, and Masayoshi Tomizuka. Dexctrl: Towards sim-to-real dexterity with adaptive controller learning. ArXiv, abs/2505.0099 2025. URL https: sttps://apil semanticscholar. orgg/
CorpusID:278310700 2.3
13

---

Published as a conference paper at ICLR 2026
APPENDIX
Additional Explanations of the Method
14
A.1 Policy Design...
14
A.2 Proof of Main Theorems.
15
A.3 Rationality of Joint-Wise Dynamics Modeling (part I)
19
A.4 Rationality of Joint-Wise Dynamics Modeling (part II)
22
A.5 Comparisons of Data Distributions between Collected Trajectories and Rotation Tra- jectories
24
B Additional Experiments and Analysis
25
B.1
Training Performance
25
B.2 Additional Real World Results
26
B.3 Case Study on the Effectiveness of Our Sim-to-Rea Method
B.4 Further Discussions, Analysis, and Ablation Studies
28
B.5 Runtime Performance Analysis..
C Additional Experimental Details
36
D Discussions on Related Sim-to-Real Works
43
LLMs Usage Disclosure
45
We include video and website to introduce our work. The website and the video contain robot videos. We highly recommend exploring these resources for an intuitive understanding of the chal- lenges, the effectiveness of our method, and its superiority over prior approaches.
A ADDITIONAL EXPLANATIONS OF THE METHOD
A.1 POLICY DESIGN
Observations. The observation of the oracle policy contains: 3-length joint position history (48- dim), 3-1ength joint positional target history (48-dim), joint velocity (16-dim), fingertip state and velocity (52-dim), object state and velocity (13-dim), object guiding goal pose (4-dim), joint and rigid body forces (40-dim), contact force and binary contact (92-dim), wrist orientation (quaterion, 4-dim), and rotation axis (3-dim).
Rewards. The reward function consists of three parts r = OraaTra + O'gaal goal + Oppenaltypeenatty with and Fpenalyy following Rotatelt (Qi et al., 2023). The rotation term Fao = clip(w: k, -c, c) encourages rotation about the unit target axis k RR, | = 1, where is the object an- gular velocity and c 0.5 caps excessive speed. The penalty Fpenstty discourages off-axis angu- lar velocity, deviation from canonical hand pose, object linear velocity, and joint work/torque: "pemalty = Gropp ||, k//1 Onin |vel - Oposee - qinin 1| O - Quanque || l1g, where v.. qinits and denote the object pose, initial hand joint position, and joint commanded torques the current timestep f,, Qlin 0.3, Olpose = 0.3, atorque = 0.1, Ctwerk = 2.0. We schedule the coefficient Oruip linearly: set it to zero at the beginning of the training: training process; at the 10 resets, we keep Oroap to zero; from 10 to 100, linearly increase it to 0.1; after 100, keep it at 0.1. Opeenalty 1.0
the number of resets to count the
We find that solely relying on these rewards cannot solve challenging problems like rotating a long object. Therefore, add an intermediate goal: episode start set peea 90" ahead along the
14

---

Published as a conference paper at ICLR 2026
desired rotation and update it whenever ang-diff( (pt, p < 15:; the guidance term is Tgoal clip
ang dig. pppddl) O, Cgoal) + Doons diif(p..ppel)ee. ccrl)ctt where ang diggddff(;;))) is the quaternion
Opsl
angular distance, c > O ensures numerical stability, and Citresbold is the proximity threshold. We set
Ygoal 1.0.
Control Strategy. We use torque control with 20Hz, where each control step is realized by running the torque control for 6 times. Each time the joint torque is calculated 7; (q -) Kaq where the and represent the current joint position and joint velocity, Kp and K are preset constant positional gain and damping parameters.
Generalist Policy Architecture. We use a residual MLP with five residual blocks. The input layer is single linear network with hidden dimension of 1024, After that, we stack five residual blocks each with the hidden dimension of 1024. Each residual block processes input via = ReLU((N (x) + NN₃ (ReLU(NN, (x)))) The output layer a single linear network that maps the latent to the output dimension.
Further Discussions on Design Choices. The BC-style training allows us to achieve real-world deployable multi-geometry policy in a simple way by combining datasets resulting from different multiple oracle policies, each trained for a specific object category, to train a unified policy, We use BC to achieve both real-world deployment ability and generality across diverse objects. An alter- native is achieving the generality in the teacher level, e.g., training RL for an any-wrist orientation any-axis on all object categories. However, this can hardly work. This may require to add an automatic or multi-stage curriculum to make sure the final policy can perform at least as good as each individual poliey. This is a valuable research direction. In this work, we choose leave the oracle policy training a neat pipeline, adopt to train a collection of teacher policies, and achieve the unified real-world deployable policy at once in the student policy training stage.
A.2 PROOF OF MAIN THEOREMS
Theorem A.1 (Data Processing Inequality for KL (strict form)) Let P and Q be two probability distributions on Ra x R with respective probability density functions (PDFs) P(x) and (z)). Let g: ::R - R x F. be measurable function, where m S n. This function transforms a random variable X P (or X Q) into a new random variable = g(X). Let g(P) and gi () denote the resulting pushforward distributions on I x R.
(KL) divergence between the distributions reduced or remains the same after the transformation, a property known as the Data Processing Inequality:
The
(L(P]I Z (L))())))(()))
The inequality is strict, QL(PIIQ > (l))())) if g is non-injective in a way that merges points where P and Q have a different relative structure. More concretely, it indicates that there Byo E RTth x R. P(Y vo) > 0, P(X vo) + Q(XYY yo))
Proof A.1 We start with prove that ( Z KL(g(P)
always holds for any function g. Let X be random variable drawn from one of two distributions, P or Q. Denote their PDFs as Px and (x).
Let Y be new random variable created by applying a function to X: Y =g()). The distributions Y are the pushforward distributions S(P) and f(2), with PDFs Py(y) and Qy Consider the joint distribution of (X, Y). since Y is deterministic function of X, the joint probability is simple: Px,r(x, = Px Px(x), yf g(x)
(2)
(x, 0, if y g(x)
(3)
Using "chain rule" of KL divergence, we can expand the joint distributions in two ways:
(A) (|xx) KL(Px 1xxx + (L(Prix (QQ)x)
(4)
(B) (Tx (xxx) KT.(Pr + KT.(PxxX
(5)
Since Y is completely determined by X (Y- = f(X)). we have
P(y[z) = 1, if = f(x). P(y]=) O, if + f(x)
(6)
15

---

Published as a conference paper at ICLR 2026
And the same property for Qtyaz):
Q(ylz) 1, f()). Q(v(z) 0, if + f(z)
(8)
(9)
Therefore PY|x- Qyix, and the KL divergence between them is zero:
(L((xxxI(X11x)= Be NPx P(ulz)) log (
(((lat) dy Ex ~PX px = 0.
(10)
Q(vlx))
Thus, the expansion simplifies to
KL(Pxy -
(11)
We have:
XL(PxI(x)= = KL(P) + (L(PXX/1XXX)
(12)
Since KL divergence is always non-negative, which implies x 2 0, we have
KL(PxI1(x) 2 KL(Pv\Qv).
(13)
The inequality is strict if and only if the second term of the RHS in Eq. 12 is strictly positive, i.e. xXx > O. This term is the expected KL divergence between the conditional distributions P(xly) and Qxlyy), averaged over the distribution Py(y), It will be strictly positive if and only if Byu E R" x R. Py (yo) > 0, P(X = yo) + Q yo).
This is direct. We provide the proof below.
Sufficiency. Since (L lIQxxr) = EymPy [L [x if the condition satis- fied, we have (I(XI)IIXXII) 2 (L 1Qx(y==)) > 0. Thus, it is a sufficient condition.
Necessity. We can prove it by disproof. Suppose that we can find a case with O but for every yo with non-zero Py (yo), we have (L(PxXy= [1Qx O, then we have X [[Qx])) = Ey-py [KIPPx/==/1Q0xx= = 0, which contradicts the assumptions. Thus, it is necessary condition.
(Q >
In our setting, as strictly reduces the dimensionality and is continuous function (because it extracts the history of a joint from the whole hand history), g a non-injective function, which we will show later in Theorem A.3. Since P and Q lie in different data domains (a visualization is shown in Figs. 16 17), and since as we' ve demonstrated ((P) and g( () share similarities (a visualization is R x R. P(Y vo) O, P(X vo) + Q( yo) shown in Fig. 15), the condition Eyo is then typically satisfied.
Theorem A.2 (Generalization Gap Contraction) Given data point (X, R" R. a measur- able function g (X,Y) Y) E R" + (Xx(X)))) RT m < and two different distributions P, Q in the manifold R" whose pushforward distribution by g satisfy KL(g(Pllg Q)) < KL(P]] Q). Un- der the covariant shift condition, i.e., PYY|X) = X),,)), any function fi X E Rd + Y E R
and J2 : gx (X) E R. Y R. we have
Rup o gx) - Ro(f2 o gx)l < sup) Rf(i) - Ro(Si)l.
(14)
where (p(h) = (xx.))=p[L(((X)) Y)] is the risk for the predictor h, L measures prediction error and is bounded by B.
Proof A.2 Using the law of total expectation and the covariate shift assumption:
(h) = Ex~Px (BX=P Y)]
R2(h) = Ex-Qx 1X) YiIl = Ex-ex [EywPry x)(((h())
Define the "inner risk" function for a fixed x:
ru(x) := Ey-P(YIX==) ((((x)
The risk difference could be converted to expectation over the marginals Px and Qx:
Rp(h) Ro(h) Exxpxxrx(X)] EE--Exxx- EEx (xx))x -aann)))mmmmlmxeeeaeea rx(x)(px(x)
qx dx)))
16

---

Published as a conference paper at ICLR 2026
An IPM between two distributions Px and Qx over function class F is defined as:
dy Qx) = sup ΦET
PE [x(X) - Exx~xx [x(X)]
Define two classes of "inner risk" functions:
Fi = {ra 1 R" +R is in the function space for Ji} F2 {T
1R RR in the function space for f23
The inequality we want to prove becomes:
dr (Px Qx) < (Px, Qx)
T- f20gx for some function f2- Define new function fi(x) = (f2 9x (z). Assuming the Fi is rich enough to contain this composition, we have rn = f2ogx Φ. This means & E F1. Therefore, F2 S F1.
Consider any function & F2. By definition,
We immediately have the non-strict inequality, since we are taking the supremum over smaller set: set:
P [o] 6EF
sup
Bax
sup ΦEFi
|E [o] Exx
Consider the given KL condition (ll((()))) condition, we have: KL(gx 1ax (xx) < KL(Px1Ixx). This implies that (x ) is not a sufficient statistic for distinguishing Px from Qx. This means the likelihood ratio w(x) = Px (r))/xx (z) cannot be written as function of gx (z). This further implies there exist Ta such that 9x (Ea) = 9x (26) but w(xx) & w(x))-
[L(PxIIQx) and the covariant shift
Now, consider the function classes:
Any function & F2 must be constant on the level sets of gx- If gx (x.) gx gx(xx))))) (1t), then ó(I,) = ((z)). These functions blind to the information that 9x discards.
The function o' F1 that maximizes the IPM difference, dy, (Px. must be maxi- mally sensitive to the difference between Px and Qx- Since this difference (captured by the likelihood ratio w(x)) depends information discarded by gx, the optimal discrimi- nating function o² cannot be a function of gx (x) alone.
This means that the function o² that achieves the supremum for the larger set F1 not contained in the smaller set F2 (i.e., 6² & Fz).
Because the supremum for Fs achieved by function that is not available in the strictly smaller set F2, the inequality is strict.
sup $eF.
[E
Eex
[o]]
sup
|Epx [o]
Eqx [o]]
deF.
This completes the proof.
Define the optimal predictors trained on the source distribution Q as:
/ = arg min Ro({1) /
(15)
f arg min Ro(fa gxx
(16)
We move on to show that under specific conditions, the predictor trained the simpler representa- tion generalizes better to the target distribution P.
Proposition Let 12 and o the optimal predictors on source distribution Q in the full and deituced-dinensione spaces, respectively. Let the following assumptions hold:
Assumption (Small Approximation Error) The function class { a gx 12 R + R is
sufficiently expressive model the relationship on the source distribution Q. The increase in source risk due to the reduced representation is bounded by small constant eAi
Rolse gx)
Ro( EA.
(17)
17

---

Published as a conference paper at ICLR 2026
Assumption (Generalization Gap Reduction) Building on Theorem A.2. we further assume rel- atively large distribution shift from P to Q such that f2 exhibits a strong generalization advantage, and the difference in generalization gap achieved by the f and f2 satisfies:
(Rp(12 gx) - Rol12 o gxx) - (Rp - -6B
(18)
where is a positive constant.
If EB > A then the risk of the predictor trained in the reduced-dimensiiondl space is strictly lower on the target distribution:
Rp(52 gx) < R-
(19)
Proof A.3 Decompose the target risk:
Rp(h) Re( + (Rp(h) Re(h))
(20)
We further have:
Rp(12 gxx) RP(sP) = RRo o oxx) (Rp(12 o gx) Re( gx))]
[ + - Re(fi))]
(21)
Rearranging the terms, we have:
Rp(f2 o 9x) Rp( [Re(f2 gxx - Re(f9)
Term A: Approximation Error
+ [Rp gx) Rols2 o gx))
Term B: Difference Generalization Gaps
(22)
From Assumption 1, Term A is equal cA:
Ro(S2 o gx) Ro(S)) = -AA
(23)
From Assumption 2, Term is equal to -B:
(Rp(f2 gx) - Rolf2 o gxx) - (Rp(fl) Re(s1)) -BB
(24)
We have:
Rp(f2 - EA CB.
(25)
Given the condition ep > CA, we have:
Rplf2 oxx) < Rp(
(26)
This completes the proof.
When are these assumptions valid? Assumption 1 characterizes the in-domain performance gap between the joint-wise neural dynamics model and the whole-hand model. As shown Sec. 4.2 and Fig. 6, it holds even when data are sufficient. In low-data regimes, the joint-wise model not only avoids increasing source- domain risk but actually reduces it, thanks better sample efficiency.
Assumption 2 characterizes the generalization behavior of these two models. Under train-tes, dis- tribution shift, it satisfied in all our experiments (Sec. 4.2; Fig. 6); the joint-wise model exhibits much better transferability than the whole-hand dynamics model.
In our dexterous manipulation setting, data scarcity and train-test shift are pervasive, because ob- taining perfectly distributionally aligned data is often infeasible or difficult to scale (Sec. 3.3), with empirical evidence in Secs. and B.4. Even with autonomous data collection, the volume of real- world data is far smaller than in simulation, keeping us in the low-data regime. Consequently, joint-wise modeling is the preferable choice for our task and a key to our success. By contrast, using whole-hand dynamics model degrades sim-to-real transfer (Tables 4 and 5). We attribute the success of the whole-body dynamics model employed in bin Shi et al. (2024) to its in-distribution setting and dynamics that are less complex than in scenario.
18

---

Published as a conference paper at ICLR 2026
Theorem A.3 Y C function f IR" -> RR, < n. that projects n-dim data point in IR" to that in a lower dimensional space R.,, then f is a non-injective function.
Proof A.4 For any point x E R", its derivative is the Jacobian matrix Dxx, which represents linear map from the tangent space at x (i.e. TR²)) to the tangent space at f(x) (ie., IE D is an m x matrix. The rank of this matrix is at most min(m,n) Theorem to this linear map Dxx : I2" + R", we find that its null space has dimension 2 n-m > O. According the Inverse Function Theorem (Munkres, 2018; Guillemin & Pollack, 2010), which states that a function is locally injective around a point x only if its derivative Dfx is injective. As we 've shown, ffx is never injective when > m. Since f is not locally injective at any point, it cannot possibly be globally injective.
m. Applying the Rank-Nullity
A.3 RATIONALITY OF JOINT- WISE DYNAMICS MODELING (PART I)
We model the hand with the standard manipulator equation (Murray et al., 2017; Spong et al., 2020), treating the object effect as external force:
M(q) + C(q. q)q + G(q) = Tex::
(27)
where M(q). C(q. q), and ((q) are the inertia, Coriolis, and gravity matrices, respectively. is the applied joint torque, and Tea represents the external force from the object. Given low-speed operation, we neglect the Coriolis term (Craig, 2009; Spong al., 2005), C(q. q:) 0.
Assuming we are modeling the i-th joint, we use (q"), q'") to represent the state of "modeled joints", e.g. q =qq"] R', while treating the joints "slave" joints and denote their state as (q" 4"), ie., q = [ V1 < J 16, & 1)" E R11, Rearranging other full dynamic equations (Eq. 27), we write it
M M
M M
q q
+
[G. G
=
Tt
m total
totaal
(28)
Tt
Derive the equation of the modeled joints:
(M T
q + M (M -1 (7
(.otta GO) + G = -" = [7' + -
(29)
Introducing an "effective" " torque as of = [T E RR, and write the equation as follows:
(Mmm M (M"") "" M (M"*) -1 (1 (ttt G*) + G
- [Ta]".
(30)
Let Hetf denote the effective inertia matrix, H.th A Mm M (M M mm and let G
denote the effective external term, G e (M ( (7 G + G m ell Given H
G and the modeled joint torque rÝ. the acceleration q is uniquely determined. H. and Geff are related to joint state and torques of other joints.
It indicates that in the highly coupled interaction system, the dynamics of each single joint is related to other joints' states, torque, and the external influence of the objects. Employing neural-based approach to solve the dynamics evolution with the aim to account for all of those high-DoF influ- ences would inevitably require a large amount of data with correct distribution, cannot resolve the challenges in the data aspect.
Focusing on each single joint dynamics system, joint-wise neural dynamics predicts each single joint transition from its own state-action history. Predicting from history generalizes the idea of the RMA approach in rotation (Qi et al., 2022) to implicitly account for time-varying influences at a high level. We will show that, in a short time window (e.g., 10 frames, corresponding to 0.5s) and under certain assumptions, this approach is reasonable.
Specifically, we assume that in any short time window during the action trajectory execution, the state trajectory of each slave joint, i.e., q", the active torque applied to each slave joint, i.e., 7", and the effective external torque applied to each joint, -ex1, can be approximated by infinitely dif- ferentiable continuous function to within an acceptable error threshold. Intuitively, this assumption holds true for joint states and active torques (related to input positional targets) in a continuously evolved dynamical system where the actions are the policy network's 's output. If we further assume soft contact model (Tedrake & the Drake Development Team, 2019; Pang & Tedrake, 2021), the
19

---

Published as a conference paper at ICLR 2026
assumption of the effective external torques, which is caused by contact forces with the object, is thus reasonable.
We give statistical evidence for these two assumptions. Specifically, demonstrate that they could be fitted to an acceptable error using polynomial functions, special group of infinitely differentiable continuous functions.
State-Action Trajectory Each Juint (Free Hand)
State (spoe)
Action (qurr)
Acar
Aar
Jebo
2407
M W
3.40
12
Jeacc 12
Jukan 43
3.0
10
Figure 10: Per-Joint State-Action Sequences (Free Hand, w/o Load).
Patterns of Per-Joint State Trajectory. Figure 10, 11, and 12 show the real-world state-action trajectories collected using a free robot hand without object load, via our autonomous data collection system with load, and the task-aware data collection with human interventions. Both action and state trajectories of the hand under such three types of external influences are visually smooth.
We further analyze their polynomial fitting results. Figure 41 shows the 3-ordered polynomial fitting results of per-joint state sequence over 10-length time window. Figure 43 shows the per-joint fitting eтror averaged over all tested 10-length sequences. We can observe good fitting results where the original curve can be roughly approximated by the fitted curve. If we increase the polynomial order to 5, we could observe excellent fitting results (Figure 42 44). These statistical results show rationality of the continuous function assumption on joint state sequences.
Patterns Per-Joint Active Torque Trajectory. Since we cannot sense the torque directly, for each joint i, we analyze the difference between the positional target and the joint state at each timestep i.e., q. qf reflect the corresponding statistics of actuation torques. Figure 45 and 46 illustrate the fitting results using 3-ordered polynomial functions and 5-ordered polynomial functions, respectively. Figure 47 and 48 further show the per-joint average fitting error. The action
20

---

Published as a conference paper at ICLR 2026
State-Action Trajectory for Each Teint (w/ Payloodd, Noised Action Trajectory)
State (qpoo)
Action (quar)
1e03
208
1.5
Jeare
1.20
-1.05
2010
Jolan
Timestops
Figure 11: Per-Joint State-Action Sequences (Autonomous Data Collection, w/ Load).
force's evolution is more complex than joint states. But we could still see satisfactory fitting results. As the polynomial order increases, the fitting results become better.
Patterns of Per-Joint External Torques Trajectory. Since we cannot measure per-joint effective external torques from the real world directly, which is related to the contact force between the object and the hand, we introduce "virtual object force" (also denoted as "virtual force" or "virtual torque") proxy of the actual external torque. Specifically,
first train per-joint inverse dynamics models that predicts the applied action from the state-action history and the next actual state, i.e., finedyn, : E 222W + â++1 E R from the freee hand replay trajectories. Thus, predicts what action should be applied so that the next joint state can reach the desired value, without the influence of the object (without the external torques). Then, for a collected task-awar trajectory, we first use the inverse dynamics model to predict the desired action å.t1- We then calculate the "virtual force" using its difference from the actual action, i.e., ac+1 â Since this discrepancy reflects what amount of additional action is required to resist the object so that the joint can reach the desired state. We then analyze the statistics of this quantity.
{($ a
As shown in Figure 49, 50, 51, 52, of this quantity is more complex than both that of the active torque and the joint state.
still get satisfactory fitting results, although the evolution
Based on this, we can assume the evolution of Helr and Gell are good continuous functions over the considered time window. We can then approximate their evolution by a low-order function, e.g., using its Taylor expansions, to acceptable error. Assuming hn order for Hff while k for Geff the underlying number of unknown variables becomes k; + k2. Solving for all unknown variables enough to solve the next step transition. The state-action history of each joint could be viewed as
21

---

Published as a conference paper at ICLR 2026
State-Action Trajectory for Each Joint (Task-Aware Data)
State (qpoo)
Action (quar)
2003
2003
115
Jaar7
2010
Jolan
4.274
4.20
Timestope
Figure 12: Per-Joint State-Action Sequences (Task-Aware Data).
the input and output of the function 30 with k1 12 unknown parameters, which contain enough information to solve for them if the history is long enough. It then indicates the reasonability of using neural network to predict the next transition from the state-action history, considering the sufficient information contained in the input and the universal approximation ability of neural networks.
A.4 RATIONALITY OF WOSE DYNAMICS MODELING (PART II)
Dynamics Model Generalization
39
33
Arei
Prs.
Pay.
Figure 13: Predicting via Single Joint State-Action History (Generalization Error).
In the previous section, we demonstrated that the state-action history of a single joint is sufficient predict its next transition. This indicates that the information contained in the single joint
22

---

Published as a conference paper at ICLR 2026
Dynamics Model Generalization
19"
14"
Figure 14: Predicting via Single Joint State-Action History (In-Distribution Validation Error).
m
state action history is at least sufficient to account for the evolution of low-dimensional effective variables over a short time window, i.e. H and G. However, this is not enough to demonstrate that model that learns to predict from the history would not implicitly learn to predict the original nigh-dimensional complex forces like inter-join coupling to predict the transition. Demonstrating this point is important since if the single joint sate-action history contains sufficient information to predict a higher-ordered system's 's states, learning from the single joint history is thus not an effective dimensionality reduction and would hamper the generalization ability the model would still overfit to the system's s high- vegganvreecce influences.
We demonstrate via experiments aiming to say that the state action history of specific joint does not contain sufficient information to predict other joints' information.
We train the joint-wise dynamics model predict the following information 1) its next joint's s current state, 2) the previous joint's current state, 3) the next joint's 's action (positional target), and 4) the previous joint's action (positional target). We then compare their prediction and generalization error with that achieved by the joint-wise dynamics model (predicting itself's s next state) for analysis.
We train all models from scratch using weal-world transition data without pretraining using simula- tion data. Real-world transition data is the same as that we use in the ablation study. As shown in Figure 14 and 13, utilizing single joint state aate-nction history to predict statisties of other joints cannot even achieve reasonable performance in the original distribution. The generalization error is three order larger than that achieved by using a single joint state-action history to predict its own next tran- sition. As for the in-distribution validation error (which is achieved on the in-distribution validation set and close to the training error), predicting neighboring joints' states achieves a slightly better performance than predicting their actions. However, this is still far from a reasonable prediction, with the error two-ordered larger than that achieved in predicting the joint's own transition.
These experiments demonstrate that even predicting the easiest information that results in the com- plex coupling (i.e,, neighboring joints' state and action) via a single joint's state-action history is not feasible. This further indicates that single joint's state-action history does not contain enough information to account for the complex influence factors in the original high-dimensions Since such information is sufficient predict the joint's own transition, a reasonable assumption is that the network tends to leverage such net effects implicitly from the history for predicting the dynamics evolution.
space.
What does the joint-wise neural dynamics model implicitly capture? Analyses and experiments in Secs.A.3 and A.4 clarify what and is not predictable from a single joint's state- aattenaction his- tory. Our comprehensive experiments (Sec.4.2) show that joie neural dynamics are expressive, sample-efficient, and generalize well. The analysis Sec.A.3 indicates that single joint's history contains sufficient information to approximate its next transition. whereas Sec.A.4 shows it cannot recover each underlying coupling effect. Thus, the per-joint history captures low-dimensional net effects while avoiding overfitting system- wide variations. This factorized, per-joint modeling transfers across changes whole-hand interaction because the distribution of net effects is compar- atively more stable than that of full-system interactions.
Limitations of joint-wise neural dynamics mode. As shown in Fig. 6, the joint-wise dynamics model performs slightly worse than the whole hand dynamics model in the in-domain test setting
23

---

Published as a conference paper at ICLR 2026
under the multi-task high data regime. The optimization speed is also limitation, as iterating all joints takes time, resulting a longer training time.
A.5 COMPARISONS OF DATA DISTRIBUTIONS BETWEEN COLLECTED TRAJECTORIES AND ROTATION TRAJECTORIES
Scanirr Distrbuttonn of Trojettciess for Each Joint Receizntaionn Trijoctorics
Tejjettrics
MID
Figure 15: Per-Joint Distribution
Discilatioo
Figure 16: Per-Finger Distribution
Figure 15, 16, and 17 summarize the per-joint, per-finger, and whole hand data distribution. It compares trajectories collected by our autonomous data collection strategy and task-relevant rotation trajectories. The task relevant trajectories are 20 cube-rotation trajectories (~~8,000 data points in total) collected using under the "Thumb Up" wrist orientation. Per-Joint state-action trajectories can well cover the distribution of task-aware rotation trajectories. However, per-finger and whole hand distributions exhibit huge discrepancy.
We additionally compare the data distribution between trajectories collected via rolling out the policy on a small object and that collected autonomously in "Chaos Box" to further demonstrate the broad coverage of per-joint transition data collected in "Chaos Box". We use T-SNE to visualize the per- joint I/O history space achieved by the autonomously collected data and the real-world rollouts on the small corn object. Results show that the joint I/O history space formed by the autonomously collected data can still offer good coverage for that of policy rollouts on the small object (Fig. 18). In comparison, their whole-hand I/O history distributions differ lot from each other (Fig. 19).
24

---

Published as a conference paper at ICLR 2026
Scatter Distribution of Trajectories for Whole Hand
Reorientation Trajectories
Collected Trajectories
Whole Iland
100
50-
V-cordinate
-50-
-100
-100
100
X-coardinate
Figure 17: Whole Hand Distribution
Scarkrr Distribution Trjeccries for Each Juin
Renrizmation Trajestories
Collectad Trajectories
MIDI
2M11
Figure 18: Per-Joint Distribution. Real-world data collected by rolling out the policy small object, i.c. cor, for axis rotation.
B ADDITIONAL EXPERIMENTS AND ANALYSIS
B. TRAINING PERFORMANCE
AnyRotate (Yang et al., 2024) improves over prior works regarding the generality to diverse writing orientations and various rotation axes. However, they only considered regular objects. Achieving such general rotation ability for complex objects poses additional challenges, even the policy train- ing aspect. In experiments, we find that prior RL designs for rotation policies (Qi et al., 2022; 2023; Yang et al., 2024), where only proprioceptions and object and system parameters- related priv- ileged information, such masses, considered in the observation, may let the training get stuck in a local optimum. Thus, we include more privileged information into the observation, followed by observation space distillation for sim-to-real (Sec. 3.1). We compare with our re-implemented
25

---

Published as a conference paper at ICLR 2026
Scatter Distribution of Trajectories for Whole Hand
Reorientation Trajectories
Collected Trajectories
Whole Iland
Y-cordinate
100
X-coordinate
Figure 19: Whole Hand Distribution. Real-world data collected by rolling out the policy small object, corn, for rotation.
Uedtto Objott
Dodit Opjoct
Dextar Objeri
Cyinlee
Cytiotes
Cytndes
Figure 20: Training Performance. Comparison of the final training performance (total reward) achieved by method and the re-implemented AnyRotate different training sets. "DexEnv Objects" denote irregular training object category.
AayRotate demonstrate this design's superiority. Our method shows noticeably better training performance over AnyRotate (Fig. 20), especially on challenging object sets, i.e., "DexEnv OB- jects" with irregular and complex geometries and "Small Cylinders" featured by small sizes, where stable finger gaiting cannot emerge in AntRot We also re-implement Rotatlt (Qi et al., 2023) in the Hora (Qi al., 2022) codebase, but find that hardly achieve satisfactory results in the most basic cylinder object set. We also adapt Hora to the down-facing hand scenario but find it cannot work.
B.2 ADDITIONAL REAL WORLD RESULTS
Fig. 22 and 23 provide more real-world qualitative results. See more results and videos in our website.
Recovery Behaviour. Fig. 24 illustrates two examples of the recovery behavior exhibited by our final policy.
Effectiveness in In-Hand Translation. We conduct preliminary validations on the in-hand trans- lation task confirm our method's effectiveness beyond the specific in-hand rotation task. For instance, in the downward-facing configuration, the NDM enables the leap hand to translate a 3cm 3cm 10cm cuboid from being initially grasped by the thumb, middle, and pinky fingers be grasped by the thumb, middle, and index finger (Fig. 25). Without NDM, for such thin object, directly transferring the corresponding
policy fails to do this.
This study aims to demonstrate the cross-task effectiveness of our sim-to-real methodology. Con- ducting a more comprehensive evaluation and developing multi-task benchmark are high-priority directions for our future work.
B.3 CASE STUDY ON THE EFFECTIVENESS OF OUR SIM-TO-REAL METHOD
As shown in Table 4 and 5, our design on learning neural dynamics and residual policy for sim- to-real can achieve notably superior results than the policy without sim-to-real design. Below,
26

---

Published as a conference paper at ICLR 2026
UNCONDITIONAL
Figure 21: Evaluated Objects in the Real World.
(A) Challenging Geometries long, small sizes)
81
3emp 13.5em
3cm
Complex Shapes
MOS 1U.55 0. 0.593
Figure 22: Real World Results. Rotating challenging objects in the air. See and videos in website.
introduce several empirical observations and case studies on our sim-to-real method. Notably, the residual policy can effectively improve the performance on challenging shapes, helping us solve previously unsolvable rotation tasks, and also enhancing the stability of the rotation (Table 7).
Rotating Challenging Objects. One of the important features of the residual policy is enabling us to rotate challenging objects with high aspect ratios or difficult object-to-hand ratios. For in- stance, without the sim-to-real strategy, the policy can only rotate the long "Lego" leg (width=3cm, lenght=13.5omm for at most 180 degrees. However, introducing the residual policy can help us ro- tate it for (almost) complete circle (demonstrated in Figure 22 and videos in website). For longer (14cm, 16cm) objects, the base policy can only rotate the object for at most 90 degrees, while incorporating sim-to-real method can let it rotate the object for almost full circle. Similar ob-
27

---

Published as a conference paper at ICLR 2026
W
Figure 23: Diverse Wrist Orientations.
EXX
DRI
Figure 24: Recovery Behaviour.
servations for the "book" object, which is 16cm long. For complex shapes (e.g., Cow, Truck, Bear, Dragon), the base policy rotate for 1/4, 1/2, at most 3/4 circles. While sim-to-real strategy can improve it to a full circle rotation. Similar observations for small objects (e.g., "Zongzi", Broc- coli). For in-hand rotation, there is typically no standard definition of "success". If one is required, a reasonable criterion could be completing at least one full rotation of the object. Failing to achieve a full 360 degree rotation indicates that the policy cannot successfully navigate the fingers around the object's surface and would likely fail on challenging features (e.g., the four legs of the cow). For such difficult shapes, the role of our sim-to-real strategy can be seen as elevating performance from failure to success.
Improving the Stability. Apart from rotating, equipping us with the ability to rotate challenging objects, the residual policy can effectively make the rotation more stable and thus help us achieve long-term rotation. representative example is rotating the 3cm xem33emx 10cm cuboid this vertical pose. When dealing with such thin objects, the policy would use three fingers the thumb, middle, and pinky fingers - to rotate the object. Compared to using four fingers, this rotation gait is unstable. If we do not include the residual policy, we can rotate the object for at most 5 circles. However, including the residual policy can let us rotate the object continuously for more than 5 minutes, which corresponds to about 30 circles. Similar observations for rotating the "cube" object along the y-axis.
B.4 FURTHER DISCUSSIONS, ANALYSIS, AND ABLATION STUDIES
Residual Policy v.s. Direct Finetuning. A natural alternative for adapting the base policy is direct fine-tuning. We evaluated this by fine-tuning the base policy on the learned dynamics model. We empirically find that the direct fine-tuning fails to produce good behaviour and is even unable to execute basic rotations. We attribute this to two main reasons: 1) Fine-tuming requires careful hy- perparameter selection, such using small initial learning rate prevent overly large updates to the base policy. 2) The learned neural dynamics do not cover the full state-action distribution (i.e., not "globally accurate"),, making them unsuitable for direct policy tuning. For instance, when the
28

---

Published as a conference paper at ICLR 2026
(iA)
Thunle
Figure 25: Example of In-hand Translation
Meiled
Heney of
Ove
Yrak
Bearoy
Cavel (o
Howe Elvelli
(SvenCiBoalo)
Thrgwooo
"Weege
Divet Traedir
7:33
6-26
1:0
31:03
3.57
10:17
5.55
16-07
1-5
311
DaNDH
5:35
7,47
6.28
0.81
0.26
29.68
6:26
16.76
10:47
190.00
0:29
6.28
6.26
T.6
Table 7: Effectiveness of the Sim-to-Real Method on Challenging Shapes. Comparison on Rot (in radian) achieved by the base policy w/ and w/o DexNDM on challenging shapes (i.e. high aspect ratios, small sizes, and complex geometry). Performance tested down-facing hand. Symbols in parentheses indicate the rotation axis. Values are the average over three independent trials.
fine-tuned policy outputs actions outside the distribution supported by the learned dynamics, i.e., (s, π finetune a)), the predicted state transitions are unreliable.
One potential solution is to regularize the base policy update, e.g., penalizing deviations from the original base policy's outputs. This is similar in spirit to our practical residual-policy design, which uses relatively small action scale. Compared to direct fine-tuning, this approach is easier to imple- ment and yields stable training without complex hyperparameter tuning.
In terms of performance, adding a residual policy on top of the base policy and directly fine-tuning the base policy could produce roughly comparable results. Practically, the residual-polle -policy approach more flexible, easier implement, and more stable, which is why we adopted it.
Evaluated Objects in the Real World. Our policy demonstrates effectiveness in rotating wide variety of objects in the real world. Photo of real-world object gallery: Figure 21.
Jount Indes
10
13
14
15
Action Magtitale
0,0755
0,0144
0.0074
0.0343
60116
0.0089
0,0051
0.0113
0.0066
0.0054
0.0085
0.0113
0.0053
0.0007
Table 8: Per-Joint Delta Action Magnitude. Running average of per-joint delta action scale when rotating a cylinder (radius 5.5em, length 5.5cm) along the axis in the real world. Joints arranged according the joint order in Isaac Gym.
Per-Joint Delta Action Value. Table 8 summarizes the per-joint delta-action magnitudes observed when rotating a cylinder (radius 5.5 cm, length 5.5 cm) about the z-axis real-world experiments. These values quantify the amount of compensation applied to each joint.
Inherent Limitations of Task-Relevan Data Collection. Collecting task-relevant transitions with estimating object poses suffer from the following inherent limitations: 1) Inability to be ap- plied to small objects due to heavy occlusions; 2) Inability to estimate an accurate full pose for axis-symmetric objects like cylinders. 3) Noisy poses caused by fast movements, tracking inaceu- racy, and heavy occlusions; 4) Huge time cost for the first time setup, iee. several days, and large time cost for launching the pipeline before each data collection, i.e., about one minute. Besides, only successful trajectories can be kept, as the hand would then experience no load, and the object falling off would lead to fast movement and an estimation failure. We can only roll out the policy and use clean actions without the flexibility to add noise, which may lead to task failure. As such, the diversity of the data would be restricted to objects that can be estimated and is biased towards easy geometries. Moreover, the object shape and scales used should match those used in the training. The dynamics model learning, even though we can collect a large amount of data, relatively ill- posed if learning only from object states without the shape information, as for different objects, the same states and actions may lead to different transitions. Including the object shape in the dynamics modeling would inevitably further increase the modeling dimensionality and require even larger amount of data to learn.
Collecting task-relevant data, even without estimating object poses, is also inherently limited to low efficiency, limited coverage, and restricted diversity since 1) data would be biased to easy objects that can be rotated well, 2) cannot add noise as it leads to the rotation failure, and 3) requires human
29

---

Published as a conference paper at ICLR 2026
interventions to reset the object to the hand. According experiments, the average time cost is 42.86s.
Object
Object
Deviated pose
Object
Insert the object into the rubotic hand
Human band retructs from
(iii) Pose tracking
deviates significunlly
Figure 26: Pose Tracking During Manipulation for A Small Object.
Timestep
Cylinder
Pose Estimation "Rotates"
Figure 27: Pose Tracking for Axis-Symmetric Objects.
Case Study on Estimating Object Poses via Foundation Pose. Collecting real-world transitions by leveraging vision bisson-based estimator track object poses is difficult, requires frequent and tedious human interventions, and is prone to yielding noisy results. For each object, we need its CAD model with exactly the same scale. Initialization steps involve capturing images via the camera and utilizing XMem (Cheng & Schwing, 2022) to get the object mask. At the beginning of each trail, need to put the object near to the pose where we get the mask. After that, we need to move the object from the table the robotic hand and launch the policy.
The difficulty of the data collection varies across the object geometry. For normal-sized objects, limitations primarily lie in noisy estimations, time-consuming, and human labor extensive. On average, we need 200s to collect a usable transition trajectory.
However, for small objects, it struggles to yield successful or even usable data. If we put the object initially on a table, then as we move the object up to the robotic hand, the pose tracking would fail, pose near the
even if move it very slowly. To resolve this, hold the object by hand robotic hand for initialization. After that, we need to insert it to the robotic hand for rotation. As the human hand retracts from the object, the estimated pose deviates from the object (Fig. 26).
Besides, for axis-symmetric objects, Foundation Pose cannot give stable estimations, where the pose continuously "rotates" while the object is kept still (Fig. 27). prevents us from getting high-quality and clean pose estimations.
Superiority of Our Autonomous Data Collection. Compared to task-relevant data, our au- tonomous data collection is object-agnostic. The hand would be continuously affected by time- varying object influences during the task execution. Joint effects all loads each joint simulate various external influences coming from coupling effects and the object. One can also use any other objects he data collection to expand the diversity. Besides, we can add noise the replay actions to expand the diversity and coverage. Moreover, it is efficient and requires human intervention.
Inherent Limitations of Playing Base Waves to Collect Data. To get real-world transitions, different approach from open-loop replaying policy action rollouts and rolling out the policy is play- ing parameterized waves such as sine waves, square waves, and Gaussian noise (Fey et al., 2025). This strategy suffers from the following drawbacks compared using policy data: 1) For dexterous hands, sending signals to a single joint while keeping others still would cause celf-collision, which may harm the hardware. 2) The model, either the dynamics model in our work or the compensator
30

---

Published as a conference paper at ICLR 2026
Sealing Law w.rtt Real World Data Quantity
14-
12-
Rot (radiann) (X-Value)
Per-Joinx (Autenomous)
Task-Aware Obj. (Fitted)
Extrapolated Trend (y 0.30x
2-
Prodiction (X 52483440)
0.00
0.50
5.00
6,00
0.50
8.61
6.10
1.50
ctroics (X-Value, Logarithmic Seale)
Figure 28: Performance scaling with dataset size. We fit the curve of TTas--Are w/ Obj. Pose" via power- law and extrapolate estimate the number data required achiove the desired result.
in UAN and ASAP, learned based on transition data obtained via playing such signals, would poten- tially suffer from distribution shift when applied in the following policy finetuning or compensator training scenarios, especially when the model input contains history. 3) Designing the frequency and magnitude of such waves is labor-intensive and time-consuming. Thus, we adopt to use of policy rollout to obtain real-world transitions.
Task-Relevant Data w/ Obj. Pose. We use a 5 cm x 5 cm 5 cm cube to collect real-world transition trajectories with object-state annotations. During data collection, we roll out the policy while rotating the object about the z-axis, and estimate its pose with FoundationPose. Because the cube is symmetric, we resolve the pose-frame ambiguity at the start of tracking by flipping the model to align with our frame convention. Each data-collection episode lasts about 200 average. We evaluated datasets containing 17 and 54 trajectories. Under the same real-world evaluation protocol in our ablations, the average rotation is 0.55 and 0.70, respectively. Fitting learning curve to these points, we estimate how many trajectories would be required to match the performance of method with 4,000 autonomous trajectories. As shown in Figure 28, the estimate is 52,483,444 vaaeetories-clean impractical. Although this extrapolation is based on a small number of data points, it highlights the data efficiency and generalization of our approach.
We attempted to train the sim-to-real baselines (ASAP and UAN) using these task-relevant, object-state--anntttee data, but even the first conge-compensttor training- converge, and rewards showed no meaningful improvement, likely due to poor data quality.
Figure 29: Out-of- Distr---itiiittion Behaviour.
BC v.s. DAgger. The BC vs. DAgger issue arose in our early experiments when we tried to dis- till specialist single-objer--categrry rotation oracle policies, trained with downward-facing hand, into real-world deployable proprioception-only policies. We ultimately chose BC because, in these downward-facinn rotation experiments, DAgger failed, whereas BC could reliably work well on hardware. We explain below:
We first trained basic cylinder-rotation specialist policies with the robotic hand kept in the Nownward-facing direction for X-, y-, z-axis rotation, and attempted to distill them using both DAgger and BC, followed by real-world testing.
Initially, we tried DAgger, following prior works. For cylinder-rotation policies with an up-facing hand, the distillation process could be successfully optimized (achieving total
31

---

Published as a conference paper at ICLR 2026
rewards around 80-100), and the distilled policy could be successfully transferred to the real robot, rotating the cylinders with the hand facing up.
However, when the oracle policy is trained with the hand kept in the downward facing orientation, DAgger fails to optimize the distillation process (i.e., rewards around 10-30; rotations along the z-axis may reach 50). The transferred policy immediately fails in the real world. For visualizations of typical out-o--distribution behavior, see the section (Out- of of-Distribution Behaviour) on the website and Fig. 29 for a visualization.
We then switched to BC and observed that it works well in this setting. Here. "works well on hardware" means that the transferred cylinder-rotation BC policy with the hand facing down avoids the -aitt--diisributionn failures seen with DAgger and can perform basic rotations reliably.
Opgal regris
Figure 30: Performance w.r.f. Object Physical Weight.
Appl Olde (N)
Fighte Fo
Rew
Rows
Payhart
Mare
"lagi"
Oollegr Cyll Seal
Oen Reaoek
Legs
Swoe Parly Topee Meg
Wighip)
10
150
157
582
100
578
17
51
33
23
32
Table 9: Object Weight measured in the real world.
Performance w.r.t. Object Weight. The plot of performance w.r.t. object weight is shown in Fig. 30. We measure the object weight in the real world (Table 9).
Because the objects vary significantly in shape, size, and geometric complexity, the plot does not exhibit clear trend showing how performance changes weight increases.
Weight (g)
15.7
33.2
39.2
49.4
62.0
Rot (radian)
12.57
12.57
12.57
9.42
7.85
Table 10: Rot (radian) v.s. Weight for cylinder (5.5cm x 5.5cm x 5.5cm) with z-axis rotation.
To conduct more controlled study and isolate the effect object weight, we designed an exper- iment using cylinders of identical geometry but different masses as test objects, and evaluated the performance on z-axis rotation. Results are summarized in Table 10 and Fig. 31. During training, the upper limit of the object weight randomization range is 50.0g. When the test object weight is be- low 40 g, we do not observe clear difference in rotation performance. This may seem unintuitive, as lighter objects might appear easier to rotate. However, in practice, for relatively light objects, the robotic hand often tends to continuously translate the object upward during the rotation, causing it become "stuck" between several fingers, which can actually hinder rotation. For heavier ob- jects-for for example, the 62 g case a common cause of failure that the robot hand cannot maintain stable grasp during the rotation, leading to the object being dropped.
32

---

Published as a conference paper at ICLR 2026
Weghtse
Morght 1g0
Figure 31: Performance w.r.t. Object Physical Weight. Rotate the cylinder object (5.5cm x 5.5cm x 5.5cm) along the z-axis with the hand facing down.
Figure 32: "Half Load" Chaos Box. Instead totally sinking into balls, the hand only touches surface of the balls in the box using fingertips.
Sensitivity to Noise and Load Distribution. To explore the influence of the magnitude of the injected noise, we design additional settings and replay ation sequences with noise magnitude 0.005 and 0.02, ie., a = 0.005, 0.02. and compare the results with those acheived using default value, i.e., 0.01.
To explore the influence of the distribution of random loads, we add an experiment where the hand only touches the surface of balls in the box using fingertips (denoted as "half loads" (Fig. 32), while the original setting where the hand is sunk into the balls is represented as "full loads")
Other settings, including the replayed trajectories and the distribution of balls in the box, are kept the same as the settings in our ablation study.
Geverlization Aliliy (Predictos Essc)) Real Woold Pefoxmace RRx indiani
121 x 10
130
19-3
12.33 1.34
7:5
10.56 0.00
6.38 =2
Table 11: Sensitivity to Noise and Load Distribution. Generalization performance and the final real world effectiveness w.r.t. magnitude of the noise injected to the replayed actions and the load distribution.
Same as evaluate in the ablation study, evaluate the generalization ability of the resulting neural dynamics model and the final real world performance achieved by the residual policy with the base policy. Results summarized in Table 11.
Regarding the noise magnitude, increasing the noise has only a minor impact on both the general- ization ability of the neural dynamics model and the final policy performance. It does slightly affect the model's 's behavior, likely because expanding the randomization range broadens the data distri- bution while reducing the amount of data available at noise levels, which make learning
33

---

Published as a conference paper at ICLR 2026
more difficult. Reducing the noise magnitude, however, has more noticeable negative effect the learning of the neural dynamics model, its generalization ability, and the final policy performance. When the noise magnitude is too small, the data distribution becomes narrower, making the model less robust and reducing the degree of extrapolation required for training the residual policy. This in turn degrades overall performance.
Regarding the load distribution, narrowing the load distribution example, when only the fin- gertips are able to contact the small sphere-reduces the effective disturbances experienced by other joints. This weakens the robustness of the neural dynamics model and the extrapolation ability needed for training the residual policy, ultimately harming generalization and sim-to-real perfor- mance.
Cross Joint Finetuning
Transfering Leap to Allegro
Cross-lim Fine-Turing
Tomafir Allayo
Prediction beror (Log Scato)
Training
Prodiction (Log Scats)
Allegro, Truining from Scrtth (kin)
12.0
148t
201
1200
Training Stepa (8)
Teinning Stops (K)
(A) Unoeediiitt Plomtinng on Toaniig from Seraach (Riad Dats)
(8)) Ehoo-troodmer Fireturing Tramiig from Serwch (Sim Dana
Figure 33: Cross-Joint and Cross-Embodiment Transfer. (A) Comparisons of the training loss in the cross-joint transfer setting (fine-tuning v.s. training from scratch); (B) Comparisons of the training loss in the cross-embodiment transfer setting (fine-tuning v.s. training from scratch).
Cross-Joint and Cross-Embodiment Transfer. We test the cross-joint and cross-embodiment transfer ability of our joint-wise neural dynamics model. We conduct the following experiments:
* Cross-joint transfer on the same embodiment (Leap Hand, real-world collected data), where we transfer the dynamics model of joint + (mod Ny) to joint i.
Cross-embodiment transfer (Leap Hand Allegro Hand, simulator rollout data).
In both scenarios, we observe that:
the neural dynamics model initialized from different joint or from a different embodiment provides good starting point: and
achieving the same prediction loss requires significantly less fine-tuning time compared to training from scratch.
More specifically, with such initialization, for cross-joint transfer:
The initial training loss over five times lower than that obtained when training from scratch;
Fine-tuning a pre-trained network requires approximately 51x less time to reach the final prediction loss of the training from scratch.
For cross-embodiment transfer in the simulator:
The initial training loss is over four times lower than that obtained when training from scratch;
Fine-tuning a pre-trained trained network requires approximately 11x less time to reach the final prediction loss of the training from scratch.
Fig. 33 plots the training loss curves in such two settings.
34

---

Published as a conference paper at ICLR 2026
B.5 RUNTIME PERFORMANCE ANALYSIS
While the joint-wise neural dynamics model enjoys generalization ability and sample efficiency, would require longer training time. Compared whole-hand neural dynamics model, both the neural dynamics training process and the residual policy learning would take more time. We include the detailed training process of the joint-wise neural dynamics model and the residual policy in the following text.
Joint-wise dynamics model trained in per-joint manner. All 16 joint-wise models are trained in parallel. To be more specific:
Given a batch of input history and output state of the whole hand, Le., {h,s}, we split for each joint, i.e., {(h,, s:))
For each joint i, the history passed to the model get the corresponding prediction 8/4-
After that the full prediction is obtained by concatenating them together, i.e., s' { And the loss is computed between s' and s.
Compared to the whole hand dynamics model, the extra time cost come from forward- ing through 16 joint networks.
Comparisons of the time consumption. In practice, training the neural dynamics in joint-wise manner roughly takes 4 6 times longer than training the whole hand dynamics model.
The residual policy is full-hand model, i.e., accepting the I/O history and the base action of the hand and outputting the residual action for all joints in the hand. Since we need to query the learned neural dynamics model, compared to whole hand dynamics model, the extra time cost of the joint-wise dynamics model come from forwarding through 16 joint networks.
Comparisons of the time consumption. In practice, training the residual policy using a joint-wise neural dynamics model is roughly 6.2 times slower than training it using the whole hand dynamics model.
Trajectories
4,000
6,000
8,000
24,000
Joint- Wise
3.44 hrs
5.30 hrs
6.83 hrs
18.72 hrs
Whole Hand
0.61 hrs
0.88 hrs
1.47 hrs
4.67 hrs
Training Time Comparison between joint- wise neural dynamics model and the whole hand dy- namies model using training datasets with different sizes.
Table 12:
# Trajectories
937,275
1,792,431
2,073,973
Joint Wise
6.2 hrs
13.68 hrs
15.25 hrs
Whole Hand
3.6 hrs
5.5 hrs
5.8 hrs
Table 13: Training Time Comparison between residual policies using the joint-wise neural dynamics and that using whole hand dynamics.
We train both the dynamics model and the residual policy parallel on eight 23GB A10 GPUs in Ubuntu 20.04 server. For the neural dynamics model, vary the dataset size (#Trajectories, where each trajectory contains 400 step transitions) and train each model for 100 epochs. The training time for the josnt-wise dynamics model and the whole hand dynamics model is summarized in Table 12.
Training the neural dynamics in wise manner roughly takes 4 6 times longer than training the whole hand dynamics model. However, the time cost is still acceptable. For our final model where 24,000 trajectories are leveraged to train the neural dynamics model, it takes less than 19 hours to complete.
35

---

Published as a conference paper at ICLR 2026
For the residual policy, train the simulation rollout dataset for epoch. We vary the dataset size (#Trajectories, where each trajectory contains 400 step transitions) and train the residual policy using the joint-wise neural dynamics model and the whole hand dynamics model, respectively. The training time for them is summarized in Table 13.
C ADDITIONAL EXPERIMENTAL DETAILS
18
12
Figure 34: General Rotation Axes.
Objectt
Cylinder
Normal-Sized Cubcids
Loug Cuboids
Cylinder
DexElvv Obijets
ContactD88 Objects
SeO)
4SSspes
Object Misimum
0.05]
0.115)
[0.017,
Object Aspect Ratios
[1.6, 2.4]
1.5]
[2.5.
2.56)
[1,06, 000
[1.0, 11
Object Scale
[0.70. 0,86]
[0.70, 0.86]
0.3
[0.5, 0.6]
[0.6, 0.71
[0.5, 0.5]
Mass
[0.01. 0.05) kg
[0.01. 0.06 kg
[0.01, 0.05) kg
[0:01, 0.05] kg
[0.01. 0.06] kg
(0.01, 0.05) kg
Coeflicien of Friction
(0.3.
[0.3,
[0.3, 3.0]
[0.3, 5.00
[0.3, 3.0]
[0.3, 3.0] (2.0.25)
External Distrisanrr
(2,0.25)
(2. 0.25)
(2,0.25)
(2.0.25)
(2. 0.25)
Table 14: Information and Physical Parameter Randomization Ranges of Training Object Sets and the Test Object Set.
(2,58,275,3 (5,33,85) 0.3.3) (3S,3)
(2..1.3)
(4.5444))
(,2,3)
(cm)
Figure 35: Dimensions of Small Objects Used in Real World Experiments.
Datasets. Our training objects comprise the following subsets: 1) Normal-sized cylinders from Hora (Qi et al., 2022); 2) Normal-sized cuboids from Hora (Qi et al., 2022); 3) Long cuboids; 4) Small-sized cylinders; and 5) Normal-sized complex shapes from Visual Dexterity (Chen et al., 2022) (denoted as "DexEnv Objects"). Details with scale randomization ranges are summarized in Table 14. To test the generalization performance in unseen shapes, we filter objects with an aspect ratio no larger than 2:1 from the ContactDB dataset (Taheri et al., 2020) (obtained from GRAB dataset) as our test set, resulting 26 objects total. The filter rule follows Rotatelt (Qi et al., 2023). As we aim to test the generalization performance on shape variation in this evaluation, do not consider high aspect ratio ones or scale them to small sizes. the real world, we test the performance on three subsets (Fig. 5, purple objects and small objects are unseen):
Regular objects: cube (5 em x 5 em x 5 cm), cylinder (radius 5.5 cm, length 5.5 cm), apple (GRAB/ContactDB apple, scaled to 0.5x), cuboid (3 cm 10 em x cm), and light bulb ("lamp_bulb" from FurnitueBBench).
Small objects: Purchased online; vendor links are withheld to preserve anonymity during review and will be provided upon acceptance. Fig. 35 shows dimensions of those objects used in the real-world experiment.
36

---

Published as a conference paper at ICLR 2026
Normal-sized irregular objects: bear, truck, and cow from Visual Dexterity (each scaled to 0.7x); and bunny, elephant, duck, mug, teapot, and mouse from GRABIContactDB (each scaled to 0.5x).
Policy Optimization. We use PPO for policy optimization. Training environments are 30,000 for cylinders and cuboids, while 50,000 for long cuboids, small cylinders, and "DexEnv Objects". We randomly sample wrist pose and a target rotation axis at each environment reset.
General Rotation Axes. To construct the general rotation axis set, we generate 32 axes evenly distributed in SO(3). Removing six principal axes, =x, Hy, and =2, get the general rotation axis set. Figure 34 provides a visualization of all 32 evenly distributed rotation axes.
Generalist Training via Behaviour Cloning. To obtain the dataset to train the generalist policy, roll out each oracle policy in the simulation to construct the dataset. Only transition trajectories that would not terminate in the full 400 steps would be saved in the dataset. We set the maximum number of tested environments to 1,500,000. In each step, the hand joint states, positional targets, object states, rotation axis, and the hand wrist orientation would be saved. Numbers of trajectories collected by each object category are summarized in Table 15. The number of successful rollouts could reflect the difficulty of different training object sets. Among all five object sets, regular cylinders and cuboids construct the casiest rotation tasks. Small cylinders introduces additional challenges due to its small scales. Complexity the geometry further increases the difficulty. Rotating long objects with large aspect ratios is the most difficult task, which yields the smallest transition dataset.
Object Set
Cylinders
Cuboids
Long Cuboids
Small Cylinders
DexEnv Objects
Transitions
1,333,282
1,282,973
235,413
743,543
681,199
Table 15: The Number of Collected Transition Trajectories in Simulation.
Metries (detailed version). We evaluate using RotateIt metrics (Qi et al., 2023) in simulation and the real world, plus goal-oriented success metric: Time-to-Fall (TTF)-
until the object drops; in simulation, episodes are capped at 400 steps (20s) and TTF is normalized by 20s, while in the real world we report raw time; Rotation Reward (RotR)-episode sum of w k (simulation only); Rotation Penalty (Rot))per-stee (Rot)-tttal radians rotated in the real world, measured from videos. We also report Goal-Oriented Success (GO Succ.) following Visual Dexterity (simulation only): we sample a random goal pose, set the target axis the relative rotation axis, and count success if the final orientation within 0.1 of the goal.
average w x k (simulation only); and Radians Rotated
Automatic System Identification. In addition training neural dynamics models and the delta ac- tion model to bridge the sim-to-real gap, we would align the dynamics between the simulator and the real world by performing an automatic system identification process at the beginning. The process involves the following steps: 1) Training probing rotation skills in the simulator using the default PD gains and link configurations in the URDF. 2) Rollout probing skills in the simulator for multiple state-action trajectories (denoted as "probing trajectories"). Replay probing trajectories on the real robot. 3) Collect the resulting state and action trajectories. 4) Launch multiple parallel environments in the simulator, each with different system parameters; 5) Replay probing action trajectories to get resulting state trajectories. 6) Select parameters of the environment whose resulting state trajectories the most similar to those in the real world as the identified system parameters. We identify PD gains and the mass of each link. Identified values are summarized in Table 16 and 17.
Joint Index
0
4
5
9
10
12
13
14
15
3.52
1.78
1.94
2.18
2.30
3.76
3.44
4.52
1.53
D Gain
0.194
0.106
0.091
0.195
0.199
0.192
0.149
0.050
0.088
0.135
0.027
0.081
0.123
D042
0.082
0.068
Table 16: Identified PD Gains. Per-Joint PD Gains identified by the automatic system identification process. Joints are arranged according to the joint order in Isaac Gym.
Domain Randomization. We apply domain randomization during training. We also randomize the physical parameters during the test in the simulator. The randomization ranges of each object set summarized in Table 14. Following previous works (Qi et al., 2022; 2023), apply a random
37

---

Published as a conference paper at ICLR 2026
Link Index
W
1o
Mins (kg)
1,00 10
2.57 x 10
2.41 10
1.00 10
2.70 io
1.05 10
1.00 10
4.68 10
3.00 x 10
3.65 10
5,38 x 10
Lank Inles
16
20
Misa (eg)
1.00 10-1
3.32% 10-2
245 10-1
2.81 1012
165 x 10-2
1.0x 10-7
5,0 10-4 3.03 10-4
4.70 x 10-2
2.23
10-1
1.00 10-1
Table 17: Identified Link Mass. Per-Link mass identified by the automatic system identification process. Links are arranged according to IsaacGym''s 's link order.
disturbance force to the object. The force scale is 2m, where mn is the object mass. We also resample the force at each timestep with the probability 0.25. We add noise sampled from the distribution (0, 0.005) to the joint positions to increase the robustness.
Baselines (detailed version). We compare our method against both previous in-hand rota- tion/reorientation works and prior neural-based sim-to-real works. We compare with two strong in-hand rotation/reorientationn works, Visual Dexterity (Chen et al., 2022) and AnyRotate (Yang et al., 2024). The experimental setup of AnyRotate is the most similar to ours. It demonstrates multi-axis object rotation under various wrist orientations. However, its code is not publicly avail- able, and the method requires tactile information. We re-implemented their environment setup and training pipeline in IsaacGym based the paper's description. We've tried our best to set up fair comparison with it in the real world. Unfortunately, faithfully replicating their tactile sensor model and sim-to-real methodology from the paper alone is difficult. We find that discarding the tactile in- formation in its second stage training can hardly yield policy with even basic rotation capabiliiiess in the real world. Thus, a direct real-world comparison was not possible. Instead, we demonstrate our method's superior performance by evaluating it on the same challenging object shapes used in their experiments. For Visual Dexterity, the open-sourced code is designed for the D'Claw hand, which is much large than and quite morphologically different from anthropomorphic hands like the Allegro or LEAP. Despite our extensive efforts to adapt their code to the LEAP hand, the policy failed to achieve reasonable performance in simulation of training. Thus, a direct comparison was infeasible. We therefore compare our method's 's perfor- mance with the quantitative results reported in their paper and the qualitative results shown in their website.
basic cylinder shape, even after 1.5 days
We also compare with prior sim-to-real methods designed for robotic arms and legged robots, namely UAN (Unsupervised Actuator Net) and ASAP. The core of both UAN and ASAP is sim- ilar, which lies in collecting real-world transition data for actuators, training neural compensators to bridge the dynamics gap between the simulator and the real world, followed by tuning/training the task policy based on the learned neural compensator. The main differences lie in two aspects, includ- ing data collection and model design. ASAP rollouts tracking policies and locomotion policies in the real world for collecting real-world transitions, while UAN avoids using policy data by playing sine waves, square waves, and Gaussian noises to prevent overfitting. UAN uses a shared network for every actuator while ASAP trains a full-body compensator (four ankle joints for sim-to-real). As discussed before (Sec. 3.3), neither including the object into the system modeling nor replicating object influence in the simulator is possible. Thus, we collect 24,000 real-world free-hand replay tra- jectories train their corresponding compensators. To compare UAN, we employ their real-world collection strategy and train a shared compensator for each joint in the hand. To compare ASAP, we replay the policy rollouts and train a compensator for each finger in the sim-to-real comparison, mirroring their four ankle joints sim-to-real setting, In sim-to-sim, we train a compensator for the whole hand and the object.
Comparisons to AnyRotate (detailed version). We compare our real-world performance against reported values in AnyRotate. As they did not provide links to obtain their real-world test objects, we test our model on four of its tested objects that are easy to replicate, including "Tin Cylinder", Cube, "Gum Box" and "Container" (see details below). While the remaining plastic vegetable models and the "Rubber Toy" are not reproducible according to the object size information provided in their Table 10. According to its experiments, objects with sharp edges are more difficult to rotate compared to plastic vegetable models (their performance on "Tin Cylinder", "Gum Box", and "Container" is the worst regarding the number of rotations and survival time among all of its tested objects as shown in its Table 12 and 13). We test the performance on three test rotation axes from AnyRotate in the rotation axis test setting. We also employ the same rotation axis setting and the hand orientation setting to AnyRotate in the hand orientation test setting. We conduct three
38

---

Published as a conference paper at ICLR 2026
independent experiments and present the average and deviations across the three trials in the Table 2. As shown, we can outperform AnyRotate by a large margin.
Besides, as demonstrated, our policy can rotate wide range of objects with diverse aspect ratios and various abject-to-hane ratios. Rotating some of them, such as the long Lego leg and animal shapes, requires quite sophisticated finger gaiting. However, AnyRotate only demonstrates the ability of rotating normal sized objects with relatively flat surfaces using conservative behaviours. As stated in their paper, they would encounter difficulties when rotating objects with sharp edges. Besides, the smallest objects that they have demonstrated the effectiveness are the "Rubber Toy" (8em x 5.3cm x 4.8cm ), "Tin Cylinder" (4.5 x 4.5cm x 6.3cm), and "Cube" (5. (5.lem x 5.1 cm x 5.1cm). However we can deal with much smaller objects like vegetable models with sizes 3cm x 3cm x2.5cm, 3em x 2.75cm x 2.75cm, and 3em x 2cm x 2. c.m. Moreover, the most challenging aspect ratios of their objects is 1.67 (Rubber Toy), while we can handle objects with challenging aspect ratios such as Lego leg (4.5), Book (5.3), and long cuboid (3.33). Such comparisons further demonstrate the superiority of our method in solving difficult in-hand rotation problems.
Details w.r.t Our Replicated Objects from AnyRotate. We replicated their four test objects as follows:
Cube:
We 3D-printed a cube to the specified dimensions 5.1cm x 5.1cm x 5.1cm.
Container: We buy commercially available product that precisely matches the container used in their experiment. We removed the labels from the container to maintain regional anonymity.
Tin Cylinder: We 3D-printed a cylinder with the specified 4.5cm radius and 6.3cm length.
Gum Box: We identified a discrepancy in the documented dimensions (9cm x 8cm X 7. 7.6cm), which were identical to those of the "Container". However, figures in the original paper indicate the "Gum Box" is substantially smaller. Therefore, we estimated its dimen- sions from the figures to be approximately 5em x 4cm 8em and 3D-printed an object of this size to serve as proxy.
Comparisons to Visual Dexterity (detailed version). Compared to prior works, visual dexterity shows improved results in rotating more complex objects with uneven surfaces and better general- ization ability unseen geometries. Conducting direct and completely fair comparison between our method and Visual Dexterity, however, is infeasible due to the different task settings (i.e. ours axis-oriented continuous rotation v.s. Visual Dexterity's goal pose-driven reorientation). Therefore, we introduce a new metric, survival rotation angles, that could be computed from qualitative results in both settings to facilitate a comparison. Specifically. it evaluates the angles the object could be rotated before it falls from the hand. This metric is friendly for Visual Dexterity since, in some settings, it has a supporting table. The object can touch the table during the rotation process. We obtain Visual Dexterity's 's results by carefully examining all of its demos present in all videos from its website. Its best performance and the comparisons to our results are summarized in Table 3. Though the metric is more friendly to Visual Dexterity, we can still achieve on par performance bypass its results for all irregular objects included in its demos (see videos in our website). Specifically, make the following observations: 1) For objects which Visual Dexterity has demonstrated strong results, including cow, bear, and truck, where they have shown the ability to rotate the object to achieve several goals continuously without falling, we can at least achieve on-par performance with it. 2) For objects that it struggles with, including elephant, bunny, duck, teapot, and dragon, we can outperform it and achieve much better performance regarding the survival angles. 3) We have shown superiorities in rotating objects with challenging aspect ratios (up to 5.33) and difficult object-to-hame hand ratios (i.e., long objects like the Lego leg and small plastic vegetable models, Fig. l). However, Visual Dexterity does not demonstrate such ability.
Comparisons to ASAP and UAN (detailed version). We evaluated our method against two promi- nent sim-to-real transfer approaches in both sim-to-sim and sim-to-real settings. Considering the difficulty in collecting real-world data with object states and the fact that their original data col- lection strategy does not account for the object influence, we collect 24,000 freehand trajectories in the real world by replaying policy action rollouts using the same hand wrist configurations as in our data collection strategy for data with load. After that, train dynamics compensator in the corresponding rrad simulation setup. This compensator is subsequently used to fine- tune the original policy. We reward the compensator training using the hand-only training penalty:
39

---

Published as a conference paper at ICLR 2026
rouseS
Jolns-Wie Wine (win Lund)
Pubak
Whele Hand
(w) Load)
UAN
Figure 36: Case Study on Failure Cases of Baselines (UAN and ASAP) and Ablated Versions (Joint-Wise (w/o Load) and Whole Hand (w/ Load)).
Genesis
Mujoco
Dloee Trande
Objict exppoiddt
Iopped
Ohe
Figure 37: Qualitative "Sim-to-Sim" Evaluation. Left:: Results in Genesis. Right: Results in MuJoco.
compensatorr = --||qq- 4/l2, where q). and qu are the reference joint state and the current joint state respectively. While we originally intended conduct a comprehensive comparison in all settings covered in Table and 5, we found that the policies produced by these baseline methods failed function in the real world. They were unable to rotate the easiest cylinder object. The typical failure modes involved the robot either grasping the object firmly without movement or failing af- ter strange perturbation (Fig. 36 (A)). (Videos demonstrating these failures are available on our website.) Notably, the policy fine-tuning process did achieve satisfactory results. We therefore hy- pothesize that an OOD issue causes this: the compensator, trained only on the dynamics of a free hand, fails when the policy must handle the novel dynamics introduced by an object during the rotation. This finding underscores the critical importance of modeling object dynamics in the de- sign of sim-to-real strategies for manipulation, which also aligns with discoveries in ablation studies (Sec. 5).
We also attempted train the baseline sim-to-real methods (ASAP and UAN) using our collected task-relevant, object-state-annotaeed dataset (54 trajectories). However, the first stage-compensatoo training-failed converge: the reward showed little to improvement. We attribute this to the dataset's 's limited size and object state noise.
Sim-to-sim comparisons are summarized in Table 4.2.
Our compensation strategy also shows better resistance to the quality of real-world transitions. As shown in Figure 36, ablated version "Joint-Wiss (w/o Load)" trains the dynamics model via free hand replay data, whose data amount is even smaller than that used to train UAN and ASAP, can rotate the basic cylinder object for least one circle, though its final performance cannot even surpass the base policy. However, the above two strategies totally fail in this task. Since they would use the compensator fine-tune the base policy, their final policy's performance is quite sensitive to the quality of the learned compensator. Thus, only if the learned compensator is of very high quality and can generalize quite well can its fine-tuning achieve satisfactory results. Otherwise, the final policy may totally fail since they are learned with "wrong" dynamics. However, compensate the base policy by using it with the learned residual policy together. With good base, the final performance would not least totally fail.
"Sim-to-Sim". We collect the data in Genesis by running the evaluation for the unified policy using 30.000 environments. We use cylinders to collect the data. We run the evaluation each cylinder
40

---

Published as a conference paper at ICLR 2026
instance with the maximum number of evaluation trails set to 1,500,000. We use all rollout data to train the joint-wise neural dynamics model (pre-trained using transitions in Isaac Gym). The training conducted on eight A10 GPUs for 2 epochs with a batch size of 64, which takes approximately two days. We collect the data in MuloCo using one environment. For each training cylinder instance, we collect 4000 trajectories, resulting in 36,000 trajectories in total. We all data to train a joint-wise dynamics model (pre-trained using transitions in Isaac Gym).
After that, train the residual policy for epochs, which takes about 13 hours. We then deploy the residual policy with the original base policy to the target simulator. The policy is tested on the ContactDB test object set. We roll out the policy using 10 different initial grasps. Reported values are the mean and standard deviation values of per-object average results over 10 trials.
Figure 37 shows a qualitative comparison of the policy's performance w/ and w/o our method to bridge the dynamics gap.
"Sim-to-Sim" Comparison Settings. We use the same data collection strategies to collect tran- sitions in each simulator. The difference is that only successful rollouts are kept, resulting in 3280673 trajectories
Genesis, while 23650 trajectories leveraged to train their corresponding action compensators for ASAP and UAN. For ASAP, we use the whole hand formulation, different from the per-finger compensator that we leveraged in ASAP's sim-to-real setting. We reward the policy to track both the object state and the hand state: compensaltor = = - gll- - ang diff(oo"" or), where qid of and o, are the hand reference joint state, object reference orientation and object current orientation respectively. k;, and ko coefficients to balance hand and object tracking. ky is set to 1.0. While we add curriculum to k It is set small value, i.e., 0.001, first. And we use the reset number of the first environment to count the reset step. During the first 10 reset steps, hi is kept at the initial value. While starting from that and until the 200-th reset step, ko is linearly increased to 2.0. After the compensator has been trained, we tune the policy based on it. The tuned policy is then deployed to the target simulator. We adopt the same evaluation strategy as for our method.
MuloCo. These trajectories are
Bandage
Ball
Plastic Object
Plastic Objer.
Bandage
(A) The Chaos Box with Balls
(B) The Bandaged Rall
(C) Bandaged Objects
(D) Object on Table
Figure 38: Autonomous Real Data Collection Setup with Load. (A) A large box with many soft balls. (B) Bind the object to three fingertips to avoid the object falling off and add external object influence to the hand. (C) Bind objects to two fingertip,s which adds external influence to the hand via collisions between these objects. (D) Adding a supporting table to avoid the object falling off.
Grasping Pose Generation. We generate grasping poses with the "Palm Down" orientation, which are used for the omni wrist orientation rotation training. For details, please refer to the cdoe in the supp IDexNDMM-odd/RLA. wwW.DRRRRRDDEE.....m
The canonical qpos of LEAP hand, from which we sample random noise to generate the grasping poses, is set to [1.244, 0.082, 0.265, 0.298, 1.163, 1.104, 0.953, -0.138, 1.096. 0.005, 0.080, O. 150, 1.337, 0.029, 0.285, 0.317].
Real-Worke Hardware Setup. We LEAP hand (Shaw et al., 2023) and Franka Arm for conducting real-world experiments (Fig. 39). We use positional control with control rate of 20 Hz. The positional gain and damping coefficient are set to 800 and 200, respectively.
Real-World Data Collection Setup. To collect real-world transition data with varying loads while minimizing human intervention, we developed several strategics, as illustrated in Figure 38.
Among these, the "Chaos Box" with balls proved most effective, Its setup is straightforward: place the box on a table, open it, and position the robot's hand inside with a desired orientation. Crucially, this method operates autonomously, requiring no human intervention during data collection. This setup ensures continuous interaction with a load, as the robot's hand is always in contact with the balls. The constantly shifting positions of the lightweight balls provide diverse and continuous
41

---

Published as a conference paper at ICLR 2026
FOARMA 98
Franka Arm
LEAP Hand
Figure 39: Real World Experiment Hardware Setup.
range of loads. Furthermore, the balls' deformable surfaces ensure that these interactions do not damage the robot's hardware. The autonomy of this system allows us to initiate data collection in the evening and let it run overnight unattended.
key limitation of the Chaos Box is its inability collect data in palm-up orientation due the robot arm's 's kinematic constraints. To address this, we developed a second setup where ball is secured three of the robot's fingers with bandage (Fig. 38 (B)). Similar the Chaos Box, this method runs autonomously once initiated. However, binding the ball takes time. A drawback is that the ball's fixed position results less diverse set of perturbation patterns.
Two other approaches were explored but ultimately not adopted (Fig. 38 (C,D)). One involved at- taching an object to the finger (C), but this was unreliable as the object could fall and require manual reattachment. The other used supporting table (D), but the object often moved outside the robot hand's workspace, necessitating human intervention to reposition
Robotic Hand Sizes. We define hand size as the fingertip span: for the D'Claw hand, the distance between diagonally opposite fingertips (19.10 cm); for the Allegro and Leap hands, the distance between the index and pinky fingertips (10.05 cm and 9.50 em, respectively).
Real-World Transition Data Collection. We collect real-world transition data by replaying action trajectories rolled out in the simulation. Each episode contains 400 steps. Actions executed in the hardware at 20Hz. Collecting one trajectory with a full episode takes approximately 20s. We collect transitions with all six tested hand wrist orientations, that is, palm up, palm down, thumb up, thumb down, base up. and base down. In each orientation, we collect 4,000 transition trajectories. In more detail, randomly uniform select 4,000 trajectories from rollouts of all oracle polices with the corresponding wrist orientation. We collect transitions using the "Chaos Box" system.
Experimental Settings of A Ablation Studies. When comparing real-worke performance of different models in ablation studies, we keep the hand in the palm down orientation and test the z-rot per- formance on three representative objects, including a regular cylinder, a cylinder with higher aspect ratios, and an irregular object. We roll out the policy for rotating the regular cylinders in this specific hand orientation and the rotation direction to construct the simulation dataset, which is composed of 937,275 trajectories, each of which has 400 transition steps.
World Data Collection. We collect transition data via the Chaos Box setup (Fig. (A)). We replay action trajectories rolled out in the simulation in the real world collect the data. We collect 4,000 trajectories, resulting in 1,600,000 transitions in total. In addition, we collect 20 successful rotation trajectories (i.e., object does not fall during the whole episode) with the thumb up orientation on a 5cm size cube by deploying policies in the real environment as the out-of-domain test data.
Data Collection. We collected I hour of data per object using three objects: a 5 em 5 cm x 5 em cube, the Stanford Bunny, and a cylinder (radius 5.5 em, length 5.5 cm). In total, we obtained 111, 87, and 54 trajectories with the cube, cylinder, and Stanford Bunny, respectively.
42

---

Published as a conference paper at ICLR 2026
Buttons
rotatinn BEeS
Left Controller
VR Headset
Right Controller
Figure 40: Quest 3. teleoperate arm using the right controller's pose, while the controller's pose specifics the desired rotation axis. We also provide a contonconttooledd mode that restricts rotation to three fixed axes, selected via the X, Y, and LG buttons on the left controller.
Collecting via Base Waves. We collect 2,000 trajectories using sine waves, 1,000 trajectories using square waves, while 1,000 using Gaussian noise. When collecting the trajectory using the sine wave, randomly select joint to send signals while leaving the other joints fixed. Specifically, we fix other joints to the midpoint of their angle range. For LEAP hand, actuating the joint between mep link to pip link when fixing other joints would lead to self-collision. So would not select such joints when replaying trajectories. We use the sine wave with the form f(t)
sin (2wt). At the beginning of each data collection, we sample and w from uniform distribution, i.e., (0055, 1.0), ~ (0.22. 0.5). When using the square waves, we use g(t) = A sign where A ((0.5,, 1.0). w U(0.2, 0.5). We add Gaussian noise the square wave collect remaining 1,000 trajectories, i..., (() = A sign(sin(2 w ()) + e, where
(sin (2 w t)),
0.000.
Dynamics Model Training. The pretrained dynamics model is obtained by leveraging the same model architecture to fit the roll-out simulation trajectories. We then directly tune the model weights on the real-world data for fine-tuning. An evaluation dataset is split out from the 4000 training tra- jectories with train: eval ratio of 9:1. The Model with the best evaluation loss is then leveraged to train the residual policy model. We report the final result on the OOD test dataset as the generaliza- tion performance. We train the residual policy the simulation data for one epoch, which would typically cost for about 10 hours using eight A10 GPUs
Teleoperation System for Complex Dexterous Manipulation Data Collection. We demonstrate an important application of our rotation policy: teleoperation system for complext dexterous ma- nipulation tasks with in-hand rotation. We implement it by pairing the policy with Quest 3 headset (Fig. 40). Leveraging in-hand rotation, the system completes complex tasks requiring fine-grained finger coordination-scenanee et al., 2024) often struggle.
where traditional teleoperation systems (Ding et al., 2024; Cheng
We adapt Bunny VisionPro (Ding al., 2024) for Franka arm teleoperation. The arm is controlled with the Quest 3 right-hand controller, and we obtain controller states via sculus_reades We use the left controller's orientation define the rotation axis and down-weight the component around its short axis to reduce errors when inferring the axis from pose. In practice, this orientation- specification is not very intuitive, so we introduce a button-controlled mode in which the rotation the left controller. Although this restricts
axis is selected by pressing the X, Y, or LG buttons the available axes to three, we find sufficient for single tasks; for example, lightbulb assembly and disassembly can be completed using z, -z, and -y rotation modes.
All hand motions, including grasping. are controlled by the policy. We initialize the robotic hand in a default pose. To grasp an object, approach it and activate the rotation policy. Conditioned on an initial open-hand observation, the policy outputs an action sequence that closes the fingers around the object to achieve secure grasp.
D DISCUSSIONS ON RELATED SIM-TO-REAL WORKS
Misaligned physical parameters, discrepancies in their physical models, and numerous unmodeled effects in the actuator and contact dynamics hinder successfully transferring the policy trained in simulation to the real world. Efforts to close this gap mainly fall into four types of approaches: 1) Domain Randomization (DR) expands the distribution of training environment train robust poli-
43

---

Published as a conference paper at ICLR 2026
cies that are expected to function well in different environments (Loquercio et al., 2019; Peng al., 2017: Tan et al., 2018; Yu et al., 2019; Mozifian et al., 2019; Siekmann et al., 2020; Sadeghi & Levine, 2016). 2) System Identification (SysID) aligns The simulator dynamics to the real-world in a principled and interpretable way by estimating critical physical parameters from real data (An al., 1985; Mayeda et al., 1988; Lee et al., 2023; Sobanbabu et al., 2025). 3) Adaptive Policy adapts the policy online according to the real-world dynamics that are implicitly identified from real-world feedback. 4) Neural-based Real World Modeling learns real dynamics help with policy's 's trans- fer (He et al., 2025; Fey et al., 2025; Deisenroth & Rasmussen, 2011; Shi et al., 2018; Hwangbo et al., 2019). As a popular and standard strategy, DR requires heuristic designs (Sobanbabu et al., 2025) to find proper randomization ranges. While generalizable and interpretable, the upper bound of SysID restricted by the coverage of parameters to be identified. For a successful adaption, the training environment should cover a wide distribution, which is typically achieved by DR. This limits their effectiveness when the real-world dynamics cannot be covered by randomizing the sim- ulated environment. With the potential of aligning all kinds of discrepancies, guiding the policy's s transfer via modeling real-world dynamics has the highest upper capabilities, making it the focus of our work. One approach is leveraging neural networks to perform system identification, learning residual dynamics or representations (Shi et al., 2018; O' O'Connell et al., 2022), followed by develop- ing a model-based controller (Fig. 2 (A)). For systems involving higher degrees of freedom (DoFs) and more complex dynamics, learning a comprehensive dynamics model that supports controller optimization is difficult. An alternative strategy is bridging the gap between an existing simulator and the real world by learning delta function (He et al., 2025; Fey et al., 2025), followed by policy finetuning to bridge the gap (Fig. (B)).
However, directly extending those approaches to dexterous manipulation, with rich, rapidly varying contacts on moving objects, cannot work. The primary challenge lies in collecting high-quality real world transition data that can cover the vast task distribution, thereby reflecting dynamics during the task execution. This is achieved by replaying waveforms (e.g., sine) or rolling out policies-none can work in our setting.
Wave-based collection is untenable: manipulated objects enlarge the transition space and impose time-varying, loads, yielding dynamics unlike the no-object regime (see Appendix A.3). Because parameterized waves cannot reliably manipulate an object in air, they must be run without it, of- fering poor coverage of in-hand dynamics. On-policy rollouts across diverse objects are costly and unscalable muscalable-requiriing frequent human resets (placing the object back in hand), biasing data toward easy objects, confining coverage the policy rollout distribution, and suffering from low quality (imperfect policy).
Extending their methods manipulation also necessitates modeling the interaction dynamics, which inevitably involves modeling the object. There are two approaches to model the object: 1) Explicitly including the object in the dynamics system. Achieving this requires collecting real-world transition trajectories with object state annotations. However, obtaining object states (e.g.. using rision-based pose trackers like FoundationPose (Wen et al., 2023))) is difficult and impossible for some cases. For instance, Foundation (Wen et al., 2023)) are unreliable for axis-symmetric, tiny, and oc- cluded objects (see Sec. B.4). Besides, the object pose tracking results are noisy. It is also very time-consuming, requiring extra time to launch and frequent human interventions. Using the small, noisy dataset cannot even make the first stage, compensator training, successful. Another strategy is modeling the object as a time-varying disturbance. This requires us to a) collect transition data with the object loads; b) manage to simulate the object's influence to the hand in the simulator; and c) train the compensator to track the hand state only. However, it is almost impossible, as repro- ducing its influence would require near-perfect alignment of geometry, initialization, and contact unreeiion-unrellissic under mismatched dynamics.
What data can we use to train ASAP and UAN in dexterous manipulation? We discuss three options: (1) transitions with object-state cinnotations-ossiibel in principle but impractical, as object states are hard and noisy to obtain and, in our tests, such small and noisy data fail to train their compensator; (2) our autonomously collected trajectories with randomized object lususlus because replicating the influence of such object loads to the hand in the simulator is infeasible; (3) free-hand data- the only practical choice, on which we train their compensator to close the dynamics gap in the free hand scenario. Hence, we use free hand transitions when comparing with their methods.
44

---

Published as a conference paper at ICLR 2026
E LLMS USAGE DISCLOSURE
We use LLMs to aid and polish writing. Details are described as follows:
Polish sentence or paragraph to correct grammatical errors, improve the writing, and make statements more concise. Typically, we will give the LLM sentence with the instruction like "Help me polish this and make it more concise: [sentence to polish]".
Refine theorems and proofs to improve rigor and professionalism. After writing each the- orem and its proof, we iterate with an LLM to identify potential logical gaps and revise accordingly-for example, by adding missing assumptions or clarifying intermediate steps. We also ask the model to assess the proof's s soundness, offer suggestions, based on which we will revise the proof. Finally, we use to make the statements' wording more precise and professional.
45

---

Published as a conference paper at ICLR 2026
Per-Joint State (qpos) Analysis
Cus?
Cak2
C Cas Fiiss
Cus iPaid
Cls 207
Card Chocd
Jaaco
Juac
Mee Spee
Johar
Jabed
Jola1
A2
Jeard
Johat 4
Juhr
Pho Nww.errccooeee
Phot Nac.hercchoo00000
Jann to
Jda1 38
Jaac 11
Squeec
174
11.
Figure 41: Polynomial Fitting (order 3) and Error Distribution of Per-Joint State Sequences (window length = 10). each group with two subfigures, the left one draws the original data sequence and the fitted sequence using 3-ordered polynomial function while the right one shows the fitting error distribution.
46

---

Published as a conference paper at ICLR 2026
Per-Joint State (qpos) Analysis
Cus?
Car
C Cas Fiiss
Cus iPaid
Cas 2
Cls 20700
Carck Chocd
Juiat
Jolat1
Stan Spued
Juhrt
Jeta15
Jelat5
Juant
Juhat
Jeint
Joiat
V.A.
col
Shon Mww.bpancaaaa...
Join
Jake
Johei
Squend
Jolne 12
Juine 12
Jaln
Joinet LA
+25 4:60
Jabe
Juine
Figure 42: Polynomial Fitting (order 5) and Error Distribution of Per-Joint State Sequences (window length = 10). each group with two subfigures, the left one draws the original data sequence and the fitted sequence using 5-ordered polynomial function while the right one shows the fitting error distribution.
47

---

Published as a conference paper at ICLR 2026
Average Per-Joint Firting Error (State (qpos)
23-
Most Sppee fare- (M60)
6.5
0.0
Jnher
Figure 43: Per-Joint Average Polynomial Fitting (order = 3) Error.
Average Per-Joint Fitting Error (State (qpos))
XII
Seanne Sare
1001
Na
R
Adbrt
Figure 44: Per-Joint Average Polynomial Fitting (order = 5) Error.
Polyiater fadox
48

---

Published as a conference paper at ICLR 2026
Per-Joint Active Torque Analysis
CMu?
C
Cus iPaid
Cas 207a0
Card Chocd
Caa Fiiss
Jobst
3aa11
cal
Mon Powert S0Bp
12
Teer (0C)
Jubst
Johat
Joler
Jdar7
Julnn
Jrhar
Jelar
Jear9
Juta 10
Joint 1D
Joias
Joiai It
Jo
Figure 45: Polynomial Fitting (order = 3) and Error Distribution of Per-Joint Active Force Sequences (window length = 10). In each group with two subfigures, the one draws the original data sequence and fitted sequence using a 3-ordered polynomial function while the right one shows the fitting error distribution.
49

---

Published as a conference paper at ICLR 2026
Per-Joint Active Torque Analysis
Car2
Cuo 17
Cls 207000
Carck Chocd
Cas Firsd
Juhnt
114
164
Squend Dter
Spuend
Meane (010)
Jutax
Juhar
Jubrr
Juinn
2012
Auil
Jucn
Mas Sound
Wesiiquantiin 2012
Scuedd
Jntas 12
Jolnn
Juked
Jniat 14
Jann 14
Jean
Jokes
Figure 46: Polynomial Fitting (order 5) and Error Distribution of Per-Joint Active Force Sequences (window length = 10). In each group with two subfigures, the one draws the original data sequence and fitted sequence using a 5-ordered polynomial function while the right one shows the fitting error distribution.
50

---

Published as a conference paper at ICLR 2026
Average Per-Jeintt Fitting (Active Torque)
2.5
Meet Neer- Eare (430)
5.5
Figure 47: Per-Joint Average Polynomial Fitting (order = 3) Error.
Average Firting (Active Turque)
Nooo Seaeeaaann Sare (M00)
1001
R
Ad
Figure 48: Per-Joint Average Polynomial Fitting (order = 5) Error.
Iadox
51

---

Published as a conference paper at ICLR 2026
Per-Joint Virtual Force Analysis
C
Cuu 17 aod
Car 2.ri
Cas 207a00
Carc Clocd
Cau Firsd
Juin
042
Booe 2012
1261
Johart
Juhn
2star
Jabss
00.4
Desttgunthe 2010
Swwedd
Jain
2:dar
Jokn
Eese 501
Jamst to
Jult 10
Joiar
Jain LI
John
Jaar
Figure 49: Polynomial Fitting (order= and Error Distribution of Per-Joint Virtual Force Sequences (window length = 10). In each group with two subfigures, the one draws the original data sequence and the fitted sequence using a three-order polynomial function while the right one shows the fitting emor distribution.
52

---

Published as a conference paper at ICLR 2026
Per-Joint Active Torque Analysis
Car2
Cus 17
Cls 207a00
Carck Chocd
Cas Firsd
Juhnt
114
164
Squend Dter
Spuend
Meane (010)
Julax
Juhar
Jubrr
Juinn
2012
Jutex
Auil
Jucn
Mas Sound
Wesiiquantiin 2012
Scuedd
Jntas 12
Jolnn
Juked
Jniat 14
Jann 14
Jean
Jokes
Figure 50: Polynomial Fitting (order and Error Distribution of Per-Joint Virtual Force Sequences (window length = 10). In each group with two subfigures, the one draws the original data sequence and fitted sequence using a five-order polynomial function, while the right one shows the fitting error distribution.
53

---

Published as a conference paper at ICLR 2026
Average Per-Jolat Fitring (Virtual Force)
1.5
1.0
Mest Sepee fare (M65
22-
0.5
Jaher
Feler
Figure 51: Per-Joint Average Polynomial Fitting (order = 3) Error.
Average Per-Joint Fitting (Virtual Force)
1.1
1.2
Mon Senee (ee
1.2
100
/
A
Labe
Figure 52: Per-Joint Average Polynomial Fitting (order = 5) Error.
54


Reviews:
[{"review_id": "fnX4Gm9TwA", "reviewer": "Reviewer_kQoB", "summary": "This paper proposes DexNDM, a novel sim-to-real framework for dexterous in-hand object rotation. The core component is a joint-wise neural dynamics model (NDM) that factorizes the system dynamics across individual joints instead of modeling the entire system, using the data collected in the real world. To collect these data without human supervision, the authors design an autonomous data collection where the robot hand interacts with random soft loads to generate diverse, object-loaded contact experiences. The learned dynamics model is then used to train a residual policy that adapts a simulation-trained base policy to real-world physics, closing the sim-to-real gap. The authors show extensive experiments on the LEAP hand demonstrate the ability to rotate a wide variety of objects under diverse wrist orientations and rotation axes. DexNDM significantly outperforms strong baselines such as AnyRotate, Visual Dexterity, ASAP, and UAN. The system also enables teleoperation applications for tool use and assembly, showcasing versatility beyond single-object rotation.", "questions": "How sensitive is the performance to the number of joints modeled jointly (e.g., finger-wise vs. joint-wise factorization)?\n\nHow long does the autonomous data collection (“Chaos Box”) take to gather sufficient data, and how does this compare to task-aware data collection in wall-clock time?\n\nFor the teleoperation tasks, are the actions purely executed by the learned policy after human high-level input, or is there continuous human feedback control?\n\nHow transferable is the joint-wise model to other hands with different kinematics or actuation? Would retraining be required, or could it generalize zero-shot with minimal fine-tuning?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(+) The joint-wise neural dynamics model is conceptually elegant and practically effective. It addresses dynamics mismatch by factorizing high-dimensional motion into independent low-dimensional predictions, with a solid theoretical rationale for generalization through information contraction.\n\n(+) The “Chaos Box” design for collecting diverse contact data is simple yet effective. It removes human intervention and provides a realistic, high-coverage dataset for learning dynamics models in dexterous manipulation.\n\n(+) The system achieves in-air rotation of challenging objects across multiple wrist orientations, as well as detailed ablation studies that validate each key design choice.\n\n(+) Applications beyond benchmark tasks: The demonstration of teleoperated long-horizon tasks (e.g., using tools and assembling furniture) highlights the real-world relevance and generality of the learned policy.", "weaknesses": "(-) Complexity of presentation: The paper packs many contributions (specialist-to-generalist policy, joint-wise model, residual policy, and autonomous data collection). While each is justified, the narrative can be dense, and the hierarchy between contributions could be clarified further.\n\n(-) The approach relies mainly on proprioception and does not yet leverage tactile or vision-based feedback for joint-wise modeling. This may limit performance in highly ambiguous contact configurations.\n\n(-) The project website should more clearly indicate which motions in the teleoperation demos are executed by the learned policy versus human guidance, as this distinction affects perceived autonomy.\n\n(-) While the joint-wise model improves data efficiency, training multiple per-joint networks and the residual policy could be resource-intensive, and runtime performance scaling is not discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "U9VtDYrjrP", "reviewer": "Reviewer_Kiqx", "summary": "The paper introduces DexNDM, a sim-to-real pipeline for dexterous in-hand rotation. A specialist-to-generalist policy is trained in sim, then adapted to real via a joint-wise neural dynamics model learned from fully autonomous “Chaos Box” data that does not require human resets. A residual policy uses the learned dynamics to correct the base policy’s actions, yielding strong real-world rotation across diverse objects, high aspect ratios, and varied wrist orientations—substantially outperforming strong baselines including AnyRotate and matching/surpassing Visual Dexterity on shared items.", "questions": "See Weaknesses.\n\nI would like to increase my score if they are well explained / addressed.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper presents a novel sim-to-real design with a distinctive joint-wise dynamics perspective and extensive hardware evaluation. Additional strengths include:\n\ni) Clear core idea with theoretical support. Factorizing into joint-wise dynamics (per-joint prediction from its own history) is argued to contract domain shift and improve generalization, with formal statements (e.g., Theorem 3.2) backing the information-contraction view.\n\nii) Practical, scalable data collection. The autonomous “Chaos Box” setup (soft-ball container, open-loop action replays, small Gaussian action noise) yields diverse, reset-free real data that are cheap and safe to collect.\n\niii) Consistent real-world gains. Across multi-axis and multi-wrist settings, DexNDM improves radians rotated and time-to-fall over direct transfer and whole-hand dynamics baselines.", "weaknesses": "Overall, the writing is clear, the presentation is strong, and the proposed methods are both promising and interesting. I have a few suggestions:\n\ni) Tighten the narrative. The paper tries to cover a lot of useful details; consider trimming or re-structuring to keep the main storyline crisp.\n\nii) Clarify L207 (“works well on hardware”). Do you mean the distilled base policy from the specialized oracles already works on hardware, or only after adding the learned residual policy? If it’s the former, what is the algorithmic role of the neural dynamics/sim-to-real transfer (i.e., the main novelty of this work)? If it’s the latter, please state that explicitly and explain how the transfer enables generalization to more complex objects (shape, rotation axes).\n\niii) Data coverage from success-only rollouts. If oracle rollouts are all successful, the dataset likely underrepresents near-failure states and local corrections. Did you observe failures of this kind, and how were they handled? If not, did you apply simple augmentations (e.g., noise injection, reset-to-perturbed states, small pose/force perturbations) to broaden the distribution? \n\niv) Terminology on “randomized loads.” In autonomous data collection the hand is placed in a container of soft balls; “randomized loads” may be misleading. Use a more precise term (e.g., “randomized contact interactions” or “stochastic contact conditions”) to reflect the actual setup.\n\nv) Residual policy section needs intuition. Could the trained base policy, combined with the learned joint-wise dynamics model, be used directly—i.e., update the base policy to produce slightly adjusted actions—rather than training a separate residual policy? More broadly, the procedure resembles system identification: the policy operates through a dynamics model trained on real data. Please expand the intuition in “Bridging the Dynamics Gap via a Residual Policy”: clarify the role of the neural dynamics model—especially since it is not used at real-world execution time—and explain why training the residual policy through this model helps overcome the sim-to-real gap.\n\nvi) Object-agnostic design and task specificity. The base policy, dynamics model, and residual policy are all object-agnostic. While I accept the claim that complex hand–object interactions can be inferred from observation history, this effectively makes the connection task-specific: without explicit object conditioning, contact dynamics are largely memorized via state transitions. From an application angle, integrating teleop is promising; algorithmically, however, this limits generalization to other type of hand-object interactions. This is common in recent in-hand reorientation work (i.e., the Leap hand paper), so it’s not a primary weakness here—but I’d like to hear the authors’ perspective and any plans to incorporate object-aware conditioning or descriptors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "T51fhWVLMu", "reviewer": "Reviewer_5oZW", "summary": "The paper DexNDM proposes a neural sim2real framework for dexterous in-hand rotation. Unlike prior methods limited to simple geometries or constrained wrist poses, DexNDM enables a single policy to generalize across diverse objects, wrist orientations, and rotation axes. The method introduces two key innovations: (i) joint level dynamics to bridge sim2real gap and (ii) an reset-free, scalable data collection strategy (“Chaos Box”) that applies randomized loads. A residual policy is trained atop this learned model to adapt the simulation-trained base policy to real hardware. Experiments on the LEAP hand demonstrate impressive real-world in-air rotation of complex, small, and high-aspect-ratio objects across multiple wrist poses, outperforming strong baselines such as AnyRotate, Visual Dexterity, ASAP, and UAN, with additional applications in teleoperation tasks. This paper is solid in methodology, theory, and real-world experiment, shown a possible paradigm for dextrous manipulation.", "questions": "1. Sensitivity to Noise and Load Distribution. How robust is the dynamics model learning to variations in noise magnitude and randomized load distributions within the Chaos Box setup?\n\n2. Performance–object weight Relationship. In the real-world experiments, could the authors report the physical weight of each test object and plot a performance curve (x-axis: object weight, y-axis: rotation performance) to examine whether and how DexNDM’s effectiveness degrades as object weight increases?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. Novel Sim-to-Real Framework. The introduced joint-wise neural dynamics model is novel in sim-to-real transfer for dexterous manipulation. By decomposing system dynamics into per-joint models, the method avoids reliance on noisy object-state estimation and achieves data efficiency through information contraction formalized via KL-divergence reduction\n\n2. Autonomous and Scalable Real-World Data Collection. This idea is really creative, the “Chaos Box” procedure autonomously generates rich, diverse contact data using randomized loads without manual resets, which significantly mitigating the challenge of efficiently obtaining distributionally relevant real-world trajectory for dynamics model learning.\n\n3. Comprehensive Empirical Validation.\nExtensive experiments cover simulation, real-world, and cross-simulator settings, and the provided demo videos show impressive performance of rotating objects from small to large aspect ratio.\n\n4. Strong Theoretical Underpinning.\nTheoretical analysis  rigorously argues that the joint-wise projection reduces domain divergence (KL contraction) and narrows the generalization gap, supporting the empirical findings.", "weaknesses": "1. Limited Hardware Validation. While the method demonstrates strong performance on the LEAP hand, its hardware generality remains untested. The paper would be strengthened by deploying DexNDM on additional robotic hands (e.g., Allegro, XHand) to verify whether the joint-wise dynamics model and residual adaptation strategy transfer effectively across different actuation mechanisms and kinematic structures.\n\n2. Scope of Generalization Remains Task-Specific. Although the model generalizes across object shapes and orientations, it is unclear whether the learned dynamics transfer to different manipulation tasks (e.g., regrasping, non-rotational dexterity, tool using).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4BI073aep6", "reviewer": "Reviewer_zo2q", "summary": "This paper proposes a neural joint dynamics alignment approach to address the sim-to-real challenge in dexterous manipulation. The authors first train a neural joint dynamics model on real-world data, then use this learned model to train a residual policy on top of a simulation-pretrained policy. The goal is to ensure that the predicted future hand state under the learned real-world dynamics matches that observed in simulation. The approach is evaluated on several rotation-style manipulation tasks, showing some empirical improvement.", "questions": "See above.\n\nI believe this paper would be significantly stronger if the authors removed the unnecessary (and wrong) theoretical and adaptation components, and instead focused on genuinely extending the capabilities of the considered skills while clearly presenting what works and what does not. These \"novelties\" feel very forced.\n\nI appreciate the engineering effort and the demo. However, these results have been demonstrated by several previous works, and the authors did not show any significantly new capability (e.g., precisely rotate the object to a specific pose and stop). Therefore, given that the science part of this paper is also flawed, this paper can not be accepted. **A demonstration is only meaningful if it showcases substantially new capabilities or is supported by sound, novel technical contributions.** Personally speaking, I believe a strong paper in this field should emphasize capability breakthroughs. Since reproducing an existing robotic system is inherently challenging and involves many compounding factors (what if you remove the tape from your fingertip and object? what if you tune SysID better? what if your ROS interface is slightly less laggy?), it becomes difficult to clearly quantify the effectiveness of an additional component integrated into an already functional setup. Novel capabilities are definitely more measurable and visible. I encourage the authors to continue their solid engineering efforts and submit an improved version to a future robotics conference.", "limitations": "", "rating": 2, "confidence": 5, "soundness": 1, "presentation": 4, "contribution": 2, "strengths": "The engineering effort to reproduce rotation manipulation tasks on a low-cost robotic hand is commendable.\n\nThe experimental section is relatively thorough, demonstrating multiple tasks and ablations.", "weaknesses": "**1. Soundness**. The main weakness of this paper lies in the soundness and validity of the proposed method. Conceptually, the approach overlooks several key aspects of manipulation dynamics: \n\n1A. Object-dependent dynamics. Unlike locomotion on a rigid terrain, where the dynamics primarily depend on the agent itself, manipulation dynamics are determined by both the hand and the object. The proposed neural joint dynamics model cannot predict finger–object contact events when the past frames correspond to free-hand motion. Because the model relies solely on proprioceptive input, it is impossible to infer or reconstruct the object’s geometry and contact state from proprioception (think about parallel gripper gripping a small marble-- you can only know there is something between the fingers. You can not tell the exact object location without touch).\n\n1B. Based on 1A, we can further show that this method can be wrong in simplest scenarios. See comments in 4.\n\n1C. Limited need for learned joint dynamics. For commercial robot hands driven by high-quality servo motors, the joint behavior is typically very repeatable (this is unlike humanoid robots, whose joint dynamics is more complex due to whole-body coupling). In such cases, traditional system identification (SysID) methods are often sufficient to close the sim-to-real gap in actuation dynamics. The authors should include SysID baseline results to determine whether their learned model actually provides any improvement.\n\n1D. Unconvincing data transfer across object scales.\nIt is unclear why training on large spherical objects would generalize to manipulating much smaller objects. This data choice is questionable, since the resulting joint states for small-object manipulation are likely out of distribution. The claimed transferability is therefore unconvincing.\n\n**2. Novelty.**\n\n2A. The conceptual novelty of this work appears limited. Learning a neural joint dynamics model has already been explored in previous research, including for humanoid and manipulation systems.\n\n2B. The idea of distilling multitask experience into a single model has been investigated by prior work such as DexterityGen, which also demonstrated teleoperation capabilities under a very similar setup. The authors also refer to their performance as \"unprecedented\" while this is not true. These related works are not cited or discussed adequately.\n\n**3. Theoretical section**\n\nThe theoretical discussion in this paper is weak and disconnected from the proposed method. Theorem 3.1 is a well-known result in information theory. Theorem 3.2 is trivial because if $B\\subset A$, then for any function or functional $h$, $\\sup_{x\\in B} h(x)\\leq \\sup_{x\\in A} h(x)$. In the theorem, $A=[f],B=[f\\circ g_X]$ and the authors did not make any assumptions on f. If all the $f, g_X$ are continuous (which is natural as the authors use neural networks), then $B\\subset A$ and we are done with $\\leq$ part.  The value of  showing $<$ is marginal, a nontrivial result typically requires $\\gamma$ contraction ($\\gamma < 1$).\n\nBesides, these results appear to have little relevance to the main technical contribution and seem to have been added without meaningful connection. A more appropriate analysis would examine error propagation or model mismatch under dynamics error, similar to what has been done in model-based reinforcement learning literature. For instance, the authors should explicitly present the return bound relating $J_\\pi$ and $J_\\pi^{res}$ ($J_\\pi$ is sum of reward when rolling out $\\pi$), we should expect some form of $ J_\\pi^{res} \\geq J_\\pi - C(\\epsilon)$, where $\\epsilon$ is the error of the model.\n\n**4. Conceptual counterexample**\n\nThe proposed approach can fail even in simple cases. Consider a minimal manipulation world model with two parts of the state $s=(s_o,s_h)$: the object state $s_o$ (which can take values 0, -1, or 1, representing neutral, fail, and success) and the hand state $s_h$ (which can be 0 or 1). Together there are six possible world states. We have 2 actions, a=0 and a=1.\n\n**In simulation, action a=1 always succeeds**. Specifically,\n\n(0, 0):  a=0 -> (0, 1), receive reward 0.\n\n(0, 0):  a=1 ->  (1, 1), receive reward 1, terminate.\n\n(0, 1):  a=0 -> (0, 1), receive reward 0.\n\n(0, 1):  a=1 -> (1, 1), receive reward 1, terminate.\n\nClearly, the best policy is to execute A=1 at any state. So, $\\pi^*(s)=\\pi_{sim}(s) = 1$.\n\n**In the real world, action a=1 may immediately cause failure**:\n\n(0, 0): a=0  -> (0, 1), receive reward 0.\n\n(0, 0): a=1   ->  (-1, 1). **Fail**, receive reward -100, terminate. (e.g. an abrupt action in real world at initialization can lead to failure)\n\n(0, 1): a=0   ->  (0, 1), receive reward 0.\n\n(0, 1): a=1   -> (1, 1).   receive reward 1, terminate.\n\nThe best policy in real world takes a=0 at (0, 0), and then a=1 at (0, 1). The optimal simulation policy $\\pi^*$ will run into (-1, 1) and receive -100 reward at the first step in this real world environment.\n\nNow, we examine the proposed approach. We first train a neural joint dynamics model $F(s_h, a)=s_h’$ which only considers the dynamics of $s_h$. According to the real world data above, we have $F(0, a) = 1, F(1, a) = 1$ for $a= 0,1$. Then according to L291, we further train a residual policy $\\pi_r$ so that $F(s_h, \\pi_r(s)+\\pi^* (s))=s’_h$ given any $(s, s’)$ drawn from the optimal simulation policy $\\pi^*$'s trajectory distribution. Since the simulation policy’s trajectory is (0, 0): a=1 -> (1, 1), the requirement is $F(0, \\pi_r(0)+1)=1$. Since $F(0, 1)=1$, $\\pi_r(0)= 0$ can work and no correction is required. As a result, the corrected policy still outputs 1 and runs into failure state (-1, 1) at the first step.\n\nThis simple example shows that without explicitly modeling object-dependent dynamics, the proposed residual adaptation cannot reliably bridge the sim-to-real gap.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)