UniHM: Unified Dexterous Hand Manipulation with Vision Language Model
Authors: Zhenhao Zhang
Jiaxin Liu
Ye Shi
Jingya Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Hand Manipulation Synthesis;Multimodal Large Language Model;
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: we introduce UniHM, the first framework for synthesizing unified dexterous hand manipulation sequences guided by free-form language commands.

Abstract: Planning physically feasible dexterous hand manipulation is a central challenge in robotic manipulation and Embodied AI. Prior work typically relies on object-centric cues or precise hand-object interaction sequences, foregoing the rich, compositional guidance of open-vocabulary instruction. We introduce UniHM, the first framework for unified dexterous hand manipulation guided by free-form language commands. 
We propose a Unified Hand-Dexterous Tokenizer that maps heterogeneous dexterous-hand morphologies into a single shared codebook, improving cross-dexterous hand generalization and scalability to new morphologies. Our vision language action model is trained solely on human-object interaction data, eliminating the need for massive real-world teleoperation datasets, and demonstrates strong generalizability in producing human-like manipulation sequences from open-ended language instructions. To ensure physical realism, we introduce a physics-guided dynamic refinement module that performs segment-wise joint optimization under generative and temporal priors, yielding smooth and physically feasible manipulation sequences. Across multiple datasets and real-world evaluations, UniHM attains state-of-the-art results on both seen and unseen objects and trajectories, demonstrating strong generalization and high physical feasibility.

Published as a conference paper at ICLR 2026
UNIHM: UNIFIED DEXTEROUS HAND MANIPULATION WITH VISION LANGUAGE MODEL
Zhenhao Zhang Jiaxin Liu'; Ye Shi Jingya Wang
02aaggzhh004, liujx2024, shiye, wanggaaaoyyaeehhaaaaaieeee ShanghaiTech University InstAdapt
ABSTRACT
Planning physically feasible dexterous hand manipulation is central challenge in robotic manipulation and Embodied AI. Prior work typically relies on object- centric cues or precise hand-object interaction sequences, foregoing the rich, com- positional guidance of open-vocabulary instruction. We introduce UniHM, the first framework for unified dexterous hand manipulation guided by free-form lan- guage commands. We propose a Unified heterogeneous dexterous-hand morphologies into a single shared codebook, im- proving cross-dexterous hand generalization and scalability to new morpholo- gies. Our vision language action model trained solely on human-object interac- tion data, eliminating the need for massive real-world teleoperation datasets, and demonstrates strong generalizability in producing human-like manipulation se- quences from open-ended language instructions. To ensure physical realism, we introduce a physics-guided dynamic refinement module that performs segment- wise joint optimization under generative and temporal priors. yielding smooth and physically feasible manipulation sequences. Across multiple datasets and real- world evaluations, UniHM attains state-of-the-ar results on both seen and unseen objects and trajectories, demonstrating strong generalization and high physical feasibility. Our project page at https://unihm, github.i...
Tokenizer that maps
INTRODUCTION
Dexterous hand manipulation involves perceiving, grasping, and reconfiguring objects in com- plex environments. Generating and understanding diverse, long-horizon, and physically feasible dexterous-hand manipulation sequences are critical for advancing robotic capabilities in humanoid- centrie applications. Such human-like interactions are ubiquitous in the real world and underpin fine-grained, complex tasks for embodied agents.
Real-world manipulation tasks require sequential, contact-rich control that couples semantic intent (what to do) with precise geometry and physics (how to do it). Conventional methods (Wang et al., 2022; Xu al., 2023; Li et al., 2025) either take object-centric inputs and optimize static grasp pose or transfer ooject interaction (HOI) video to fixed sequences. Lacking open-vocabulary instruction. these pipelines cannot guide diverse and complex dexterous hand manipulation.
Recent vision-language approaches have begun to guide static grasping and manipulation by map- ping free-form language to grasp representations. SemGrasp (Li al., 2024b) uses a language- aligned discretization of the grasp space and fine-tunes multimodal LLM so that text, object cues, and discrete grasp tokens lie in a shared semantic space, yielding language-conditioned static hu- man grasp poses. AffordDexGrasp (Wei et al., 2025a) targets open-set generalization by predicting affordances from language and then generating dexterous grasps, with the primary output still being static pose rather than multi-step interaction. Despite advances in semantic controllability and open- vocabulary coverage, most ignore temporal structure, and therefore fail to produce smooth and rich manipulation sequences.
approaches focus on generating static grasp poses,
Building on these observations, we propose UniHM, a unified framework for generating dexterous hand-manipulation sequences for both seen and unseen objects under open-vocabulary language in-
Equull contribution;
Thee corresponding author.

---

Published as a conference paper at ICLR 2026
Generalize to Open-world Tusks!
CHEZZI
GGrb oitt."
Pick the crecker box."
muster bottol
UniHM
Training Closed-set HOI Datasets.
Figure 1: Overview. We introduce UniHM, the first unified maniplaaiillaiion framework condi- tioned on free-form language. UniHM is trained solely closed-set HOI datasets to follow target trajectories and execute physically feasible interactions, and generalizes to open-world tasks real- world interactions.
structions. Our approach couples a vision-language model with a Unified and physies-guided dynamic refinement, which together enable two key capabilities: 1) Dynamic language-guidet manipulation. Prior methods typically either generate only static grasp poses or lack open-vocabulary language understanding: in contrast, we progressively train both the trajectory planner and the DexHand VLM, allowing the system to perform dynamic dexterous-her ulation along arbitrary planned trajectories with substantially improved temporal consistency and scalability. 2) Learning from video. We leverage diverse, scene-rich human video data that tains abundant information about both environments and interactions(Zeng et al., 2025), enabling the model synthesize human-like grasping poses while achieving strong generalization across wide range of manipulation scenarios. Extensive real-world cross-embodiment experiments further demonstrate the effectiveness and generalization ability of this training paradigm. By integrating these components, UniHMM achieves state-of-the- performance and exhibits strong generalization across hand morphologies, object categories, task horizons, and linguistic complexity.
Tokenizer
manip-
Our contributions summaries are as follows:
Unified Dexterous Hand Manipulation. We introduce UniHM, the first unified, language-conditioned framework for dynamic dexterous hand manipulation beyond static grasps directly from images and open-vocabulary instructions.
Morphology-Agnostie Codebook. We introduce a unified VQ token codebook with cross- hand consistency that maps heterogeneous hand kinematics into one discrete action lattice and decodes tokens into hand-specife joint trajectories, which enables direct token reuse and transfer across robotic and anthropomorphic hands.
Physics-Guided Dynamic Trajectory Optimization. We employ tailored energy- based refinement that fuses a generative prior shaping feasible pose manifolds, a temporal prior enforcing smooth velocity- aeccltt-aacolleration profiles and time-aware consistency, and contact- aware dynamic trajectory optimization to optimize the generation result.
Generalization without Teleoperation. Our framework eliminates the dependency on ex- pensive teleoperation data by learning dexterous manipulation skills from human videos. This paradigm achieves robust generalization to unseen scenes and instructions, signifi- cantly lowering the barrier to developing dexterous manipulation systems.
2

---

Published as a conference paper at ICLR 2026
2
RELATED WORK
2.1 DEXTEROUS GRASP GENERATION.
Research on dexterous grasping largely follows two lines. Language-free pipelines advance dex- terous grasping without text by exploiting geometry, simulation, or video demonstrations. UniDex- Grasp (Xu et al., 2023) learns from point clouds using a two-stage scheme that first proposes diverse grasp poses and then executes a goal-conditioned policy for stable lift, thereby generalizing across many categories. DexGraspNet (Wang et al., 2022) synthesizes and validates at scale via differ- entiable force-closure and collision checks to supervise grasp policies. DexMV (Qin et al., 2022) builds an imitation pipeline that converts human videos into robot demonstrations through pose es- timation, retargeting, and demonstration translation for dexterous manipulation. Language-guided dexterous grasping aligns natural language with grasp spaces using discretization or affordance cues; for example, SemGrasp (Li al., 2024b) learns language-aligned discrete representation and uses a VIM to produce semantically consistent static dexterous grasp poses. DexGYS (Wei et al., 2024) introduces tanguage-guides dataset and model that maps commands to dexterous grasp poses. Af- fordDexGrasp (Wei et al., 2025a) leverages a generalizable, instructive affordance representation to guide open-set dexterous grasp generation. While these methods improve semantic controllability and open-vocabulary coverage, they remain pose-centric and typically generate static poses rather than sequence-level hand manipulation.
2.2 VISION LANGUAGE MODEL FOR MANIPULATION
Generative vision-language model approaches increasingly cast manipulation as sequence predic- tion conditioned vision and language. MotionGPT (Zhu et al., 2025) treats human motion language via VQ tokenization and unified text-motion modeling for sequence generation. HOIGPT (Huang al., 2025) extends token-based generation long 3D hand-object interaction, learning a bidirectional mapping between text and HOI sequences. OWG (Tziafas & Kasaei, 2024) com- poses VLMMguided referring segmentation, grounded grasp planning, and contact-aware ranking for zero-shot grasping rather than full sequence synthesis. ReKep (Huang et al., 2024) converts language/vision into relational keypoint constraints and solves actions via hierarchical optimiza- tion instead of an autoregressive policy. DexGrasp Anything (Zhong et al., 2025b) complements VLM pipelines with a physies-aware diffusion generator and the largest dexterous- grass dataset to date, improving feasibility through explicit physical constraints. Multi Guati-GraspLLM (Li et al., 2024a) aligns point-cloud and text features to generate language-guided grasp poses across mul- tiple robotic hands within single framework. Taken together, these generative models advance instruction following but predominantly target Digital Hand, low-DoF grippers, or static grasp poses; explicit sequential dexterous- hend manipulation with feasibility remains limited, motivating our hand-agnostic tokenizer and physics-guider refinement.
3 METHOD
We propose UniHM, unified framework for dexterous hand manipulation. UniHM first annotates large-scale hand-object interaction data, then employs vision language model coupled with a Uni- fied Dexterouss Tokenizer to synthesize manipulation sequences, and finally applies physics- based optimization to ensure physical feasibility.
3.1 AUTO DATA ANNOTATION
Open-vocabulary Language Annotation. We annotated dexterous hand manipulation sequences with GPT-4o (Hurst et al., 2024). For each sequence, we provide keyframes as visual context, specifically the first and last frames and the three frames preceding first contact, and the model returns five distinct -pca-vocabulary natural language instructions.
Dexterous Hand Retargeting. For HOI sequences to dexterous hand manipulation sequences trans- fer, we first apply Dex- tex-Retargeting (Qin et al., 2023) to map MANO poses onto five dexterous robot hands (Shadow hand, Allegro hand, SVH hand, Leap hand and Panda hand). We then perform
3

---

Published as a conference paper at ICLR 2026
Unitied Dexterous Hand Tokenizer
Dexterous Hand Manipulation
Inferenc Only
RGBD Instruction
Dief
CLIPort
Prene
Tageecury
Retargeting
Distillation
Tranihgl Shrnch
Text Encoder
Pabi Encuder
Jux Encoder
Unified
Fp
Dx
Unified Codcbook
(3) Pbysical-Guided Dynamic Refinement
Optimization Result
Egen(4p "
DexHand VLM
Generaaion
201 PRUURAR
Econtact (cpy Tix Pats)
Temporal Price
D2
Dr
Erime( 9c-2.
Course Result
Coarse Resalt
Figure 2: Pipeline. UniHM converts open- open- vecabuucyy instructions and RGB-D inputs into exe- cutable dexterous-ham trajectories via three stages: (1) morphology-agnone motion tokenization; (2) language-gun guieeeguidd generation that fuses text, perception, and token history to produce manipula- tion token sequences; and (3) physics-aware decoding with smoothness/contact priors for feasible, stable execution.
energy-based physical optimization to enforce feasibility, yielding physically consistent dexterous hand manipulation sequences.
3.2 UNIFIED HAND-DEXTERON TOKENIZER
Morphology-Agnoste Codebook. We discretize dexterous hand poses and short motion segments across heterogeneous hand morphologies with shared VQ-VAE (Van Den Oord al., 2017) code- book. Let En and Dj, denote the encoder and decoder for hand type h (e.g., MANO robot hand), and let the global codebook be = {e*]== with e₁ E Rd. Given a sequence chunk
specific
(*h,, define the encoder output (h) (x()x)) (h)) and the vector-quantizzttio operator Q(-) that
returns an index c E {l,..., K.....,K,, KJ:
c Q En (x())) = arg min kE(K
(h) ze
ek
and the quantized representation is Zig (N)
Cc. The reconstructed sequence chunk (M) is then
( Dr. (h))))
Dr (eo)
So that every encoder E,, maps to the same discrete index space and every decoder D,, realizes the token within its own morphology.
Scalable Cross-Morphology Training. To align representations across diverse hand types, we train our model in a staged, scalable manner. We first establish reference encoder-decoder pair (Erss Diat) for a specific hand morphology, along with the shared codebook Z.
When integrating new hand morphology, instead of direct non-differentiable token alignment, first align the new encoder's latent space with the reference encoder via knowledge distillation. This bypasses the gradient discontinuity of the quantization step. The distillation objective is:
Laistill = Enew (Xnew) - Eve 112
(3)
where Enew is the new encoder and Exs is the pre-trained reference encoder. Xaow and Xyet corresponding hand sequences for the new and reference hands, respectively, obtained through re- targeting.
4

---

Published as a conference paper at ICLR 2026
After the new encoder is aligned, it is integrated into the full VQ-VAE training pipeline. The new encoder and its corresponding decoder are then fine-tuned with a combined objective that includes reconstruction and bodebook-related terms. For new hand type h, the training objectives are:
Leo
(x())
Dr. 2 )12
Lxg
(6)
(h)
|| + 8 2 sg (h)
(5)
where sg[-] denotes the stop-gradion operator and B > 0 is commitment weight. The total training objective for new morphology is a combination of these terms, as well as the distillation loss for encoder alignment.
Unified Hand Pose Translation. Because the encoders are aligned shared latent space and the decoders are mapped to the unified codebook, translating pose from hand i to hand j is straight- forward. We simply encode the source pose and decode using the target hand's decoder:
x =D; (EQ(EE((())))
(6)
where x(t) is the source hand pose and x(1) is the retargeted pose for the target hand.
3.3 DEXTEROUS-HAND MANIPULATION WITH VLM
Network structure of VLM. Given the scarcity of HOI and dynamic dexterous hand data and the high cost of collecting it, training large multimodal language models (e.g.. 7B or 1 13B parameters) is data-inefficieet and yields limited performance in this regime(Zeng et al., 2026; Wang et al., 2026; He al., 2025; Zhong al., 2025a). We therefore adopt Qwen3-0.6B scale enables stable convergence on HOI and dexterous-hand datasets. To compensate for the lack of dynamic manipulation data while retaining strong visual grounding. We adopt a decoupled architec- ture that separates scene perception from HOI sequence generation. A CLIPort-style vision module trajectory
the base model, whose
consumes RGB-D images and language infer target trajectories, and an encoder serializes these targets for the VLM. The main VLM focuses on instruction-condiioons token generation with progressive masking curriculum, and the unified tokenizer maps tokens to hand-specific poses across different robotic hands. At inference time, only the CLIPort perception head adapted to distribution shifts, which improves data efficiency and robustness while keeping the HOI generator unchanged.
Unified Dexterous Manipulation. We first use a CLIPort model to take an RGB-D image and its corresponding instruction (e.g.. "grasp the bottle into the box") as input. The model then decodes target execution trajectory Tur This process can be formally described as
T = CLIPort(I dd T Tenstruction):
The target trajectory is denoted by Tar = (Ta the total sequence.
Tel
Thi E SE(3), where K is the length of
We leverage RGB-D data to reconstruct the scene point cloud and then use Point-SAM to segment the object point cloud Paby with the corresponding semantics.
Pobj Point PointSAM((Lgg-dd T Tinetruction).
Pabi denotes the point cloud of the target object. Specifically, Pabi 1xxx is set of 3D coordinates (I, y, 2). where is the number of points in the point cloud.
Next, the initial hand pose is encoded by E-)-) and concatenated with Tur Payj. and text token T. This combined input then fed into a Vision Language Model (VLM) to obtain a code sequence in the VQ-VAE latent space. Finally, we use the decoder of the target hand, Dr.().. to decode the corresponding q-positions:
Qpos
Taary Pabi T)).
Training Stage. When fine-tuning the main model, we input the real object sequence trajectory as target trajectory into the large model. To enable the model understand the spatial continuity of
5

---

Published as a conference paper at ICLR 2026
hand poses, initially input the ground truth hand poses and the VQ- VAE an encoder into the model. As training progresses, we use masking to randomly occlude portion of the hand poses, replacing the masked poses with unified, learnable token, until all hand poses are masked (Liu et al., 2019; Oquab et al., 2023).
The masking process be expressed
mmskked M @ E(Q) (1 M) @
Tmask
(10)
where Q is the ground truth hand pose sequence, E is the VQ-VAE encoder, M is a binary mask, and mask is the learnable token.
Inference Stage. Our training and inference pipelines differ by design. During training, we con- dition the model on ground-truth target trajectories and object point clouds, enabling it to reliably generate physically feasible dexterous-hame manipulation sequences that follow the specified tar- gets. At inference, a separate CLIPort module estimates these quantities from RGB-D observations, decoupling spatial perception from hand-object interaction. This modularization allows the main model to focus solely on HOI sequence generation. A practical advantage is robustness to environ- mental shift: when the scene distribution changes, we fine-tune only CLIPort, which smaller and less data-hungry, rather than the entire model.
3.4 PIYSIIALL-UIDDE DYNAMIC REFINEMENT
To bridge the gap between the generated grasping trajectory Qgen and the object point cloud Parj under the target pose trajectory (t), we formulate posterior optimization. We solve spatio- temporally regularized, frame-by-frame Gauss-Newton problem that enforces physical plausibility while preserving the semantic intent of the generated actions. We proceed frame-by-framyy time optimize de while treating 9:-1 and 4:-2 as fixed from previous steps.
Let (x))) be the i-th fingertip position from forward kinematics in the world frame. We transform it into the object frame using (Ia(())= and query the nearest neighbor (p;, n;) on Parj. where n, is the local surface normal. We define a signed point-to-plane distance (Grant al., 2012):
Contact Energy.
d<().) n..' (Tmr(()''ss())) s. PP:) -
(11)
and an asymmetric, smooth penalty that is continuous and mlope-matched at d = 0, which is essential during the optimization:
o d2
d M (outside)
f(d)
kd
kd kd-1). 1). d (inside)
(12)
c
k2
where a > O and k > O are scale parameters. Stacking per- pir-fingertip residuals yields:
Foomtact, (gr)) = 2 Aa (d)( Econtncr (4:) = reomact (a:)mee(())))) [a)l12- do Σ (d((a)),
(13)
&Elingers
with > 0 controlling the contact strength.
Generative HOI Prior. To preserve the intent of the generative model, we penalize deviations from the generated configuration c.
(g. (g. Tt Rcm
(g:
o:n)) W gen (qi o
(14)
where W gen > O is a symmetric positive-definite weighting matrix. We use the weighted norm shorthand ||a|| 2 A T.WW. below.
Temporal Prior. We regularize first- and second-order temporal differences to ensure smooth, co- herent motion:
Eime (gt, 9- 1.
cgt
ape 9:-1
w... + 11 9--1) (qiTh
apt ---2) 2212))1² W._.
(15)
where W... Wxl 0 and W.s. velocity/acceleration the generated states boundary priors, meaning that q-2 q E
Mecc > 0 (often W = Alald W Amce I). For < 2, we use zero-

---

Published as a conference paper at ICLR 2026
Table 1: Main Result DexYCB. The arrow pointing the right means closer to the GT.
Method
MPJPFL
FO14
FPI.
F
Diversity -
GT
125.53
TM2 et al., 2022)
85,33
36,57
24. 24.10
54.83 axiV
37.12
Seen
MDM( Tevet et al., 2023)
88.06*1s
33.40
23.06
52.33=t
33.95
Flow (Brguaa al., 2024)
82.75 1:31
31.24
21.55
48.05
61.25
MotionGPT3(Zhu et al., 2025)
74.80
28.76*1u
19.32
43.35 -.75
72.51 22000
Ours
61.40 tt
23.14-tmss
12.15
31.24
39.62
TM2 al., 2022)
94.22 EM
37.25
27.03
27
55,94
31.25
Unseen
MDMeee et al., 2023)
93.05 40
39.04
25.89
55.13
29.0
FHowMDMM(Barqug et al., 2024)
86.13
32.67=131
24.09
51.33
58.21 #1.9
MotionGPT3(Zhu et al., 2025)
77.93
30.555-t22
21.48-tus
46. 46.14
75.84 100
Ours
63.56 2:
27.29 d.ness
13.06 4.
41.03 15
42.70 1.M
The per-frame objective is the sum all terms:
El(q,, li 11-11-9- Ecuntucl(:t) Egen (qt, di + Eime (gt, 91 9t-22)
(16)
and the sequence energy is (qna( Et= Eg(9t, of - -4-1. 91-2)
opt
We linearize Feomtuc (4:) and apply
with Levenberg-Marquard damping 2 0 (Yu
& Wilamowski, 2018). Let Jj = drccnnac (n: (qt)/Oq. The normal equations for the update Aqi are:
(J+ Jr + W gen + W. Wel + W sec + AT) JT- reaniac (9:) W
(17)
W A W gen (1e
li
W
(
qt + Wacc acc (1 ). opt
q aTm)))
(18)
Here the prior terms enter both quadratic regularizers the left and linear gradients the right, stabilizing the optimization while respecting the generated intent and temporal smoothness under the object's target pose trajectory (t).
EXPERIMENTS
Our framework comprises three components: (1) we annotate hand-object interaction (HOI) quences with a vision language model (VLM) and, via retargeting plus physics-based refinement, obtain manipulation sequences for multiple dexterous hands; (2) train Unified Hand-Dexterous Tokenizer together with an VLM for text-to-hand manipulation sequence generation; and (3) we ap- ply an energy-basee physical refinement and validate feasibility in simulation. Comparative exper- iments and ablations demonstrate the effectiveness of our approach. All experiments are conducted on NVIDIA A100 GPUs
4.1 DATASET
In experiments, we evaluate on two of the most widely used datasets. DexYCB (Chao al., 2021) is a multi-view RGB-D dataset of human grasps on YCB objects with precise 3D labels: 582K frames, 1,000 sequences, 10 subjects, 20 objects, and views; benchmarks include 2D/6D object pose, 3D hand pose, and handover. OakInk (Yang et al., 2022) A large-scale hand-object interaction repository integrating OakInk-Image (230K multi-view frames from 12 subjects manipulating 100 objects across 32 categories) and OakInk-Shape (3D grasp-pose meshes with affordance labels), plus 50K affordance-awee aware interactions transferred via Tink. For both DexYCB and OakInk, we adopt an 80/20 split: 80% for training/validatt validation (seen) and 20% held out as an unseen test set. This protocol enables a rigorous assessment of generalization across seen and unseen objects, trajectories, and interaction patterns for UniHM.
4.2 EVALUATION METRIC
To assess the quality, diversity, realism, and physical plausibility our generated dexterous-hard manipulation sequences, adopt multi-pronged evaluation protocol that follows prior work (Wei
7

---

Published as a conference paper at ICLR 2026
Table 2: Main Result OakInk.
The arrow pointing the right means closer to the GT.
Method
MPJPEL
FOL.
FPL.
FID
Diversity -
GT
147.40
TM2T et al., 2022)
71.08
91.25
34.51
311.90
277.38=1n.
Seen
MDM( et al., 2023)
67.55
93,8
30.06
285.22
275.42
(laqueDM(Barquero et al., 2024)
60.74
85,434s42
26.47
249.08
douss
189.54
MotionGPT3(Zhu al., 2025)
56.29 2225
79.24
23.98
221.10
247.10
Ours
52.73 +:20s
72.32 cbuss
19.86 1.35
204.91 1:764
165.47 10.00
TM2 ct al., 2022)
75.34
125.33
45,51
337.08
362,08
Unseen
MDM(Tevet et al,, 2023)
72.90
112.94#*.
42.93
325. 325.58
354.93
FlawMMDMMBarquero al., 2024)
63,39
101.25:
36. 14
298.04
224.67
MotionGPT3(Zhu et al., 2025)
61.95-224
93.65
28.25 1115
272.69
316.58 12:26
Ours
58.62
83,27
22.87
253.41
153.28 1o.4
Table 3: Real-World Experiments
Success Rate Pull&Push
Split
Method
Grab
Pick&Place
Open&Closs
MDM+Dex Rectaggginnggg
20%
10%
0%
5%
Seen
MotionGPT3+Dex-Reeeele Retaargetnng
30%
15%
25%
25%
Ours
65%
50%
60 %
55 %
Unseen
MDM+Dex- -eett-rDt-ttttggtinngg
5%
0%
0%
5%
MotionGPPT3+Dex-Reeann Retargeting
45%
25%
15%
20%
Ours
60 %
35%
55 %
45%
al., 2025a; Zhang al., 2025; Deng al., 2025; Ji et al., 2025; Liu al., 2025; Wang et al., 2025b; Jian et al., 2025; Wei et al., 2025b) and combines quantitative and qualitative measures across five metries.
Physically Feasible. Mean Per-Joint Position Error (MPJPE) for hand joints, Final Posi- tion Location Error (FPL) and Final Orientation Location Error (FOL) for dexterous hand placement and orientation. Success Rate means the Real-world grasping success rate.
Generation Realism. Fréchet Inception Distance (FID) between real and synthesized. Diversity, which measures the variability across different prompts and within outputs from the same prompt
Lower MPJPE, FPL., FOL, and FID indicate higher accuracy and fidelity. Diversity closer to the ground truth indicates a more reasonable generation.
4.3 MAIN RESULT
Comparison with SOTA Methods. We conduct extensive comparisons against prior state-of-the methods(i.e.. TM2T TM2T (Guo al., 2022), MDM (Tevet et al., 2023), FlowMDM (Barquero al., 2024), MotionGPT3 (Zhu et al., 2025)) on both DexYCB (Chao et al., 2021) and OakInk (Yang guarantees,
al., 2022). Because prior action- generraionn baselines lack explicit thysical-feasibile we post-pre pooteproces their outputs with our physics- physics-guided refinement to ensure fair comparison. As shown in Table and Table 2, our method consistently outperforms baselines across both seen and unseen objects.
These results unequivocally affirm our method as cutting-edge, showcasing the robust generalization capability of our VLM framework and physical refinement, thereby substantiating the efficacy of our UniHM synthesis framework in producing sequential hand manipulation sequences for both seen and unseen objects and trajectories guided by open-vocabulary instructions.
Visualization in Real-world Results. We conduct real-world evaluations on a dexterous hand across both and unseen objects and trajectories. Results show improvements in grasp cess rate and grasp quality. As summarized Table 3, UniHM achieves a higher success rate than prior methods, and Dig. presents qualitative visualizations of our real-world grasps.
8

---

Published as a conference paper at ICLR 2026
Grab the apple!
Grab the bow!!
Pick& Place the fruit!
Pick&PPacce the bottle!
Open the lid!
eabinet!
Pull drawer!
Push the drawer!
World Results. UniHM achieves higher success rates than prior methods on both seen and unseen objects, producing physically consistent and executable real-world manipulations.
Figure 3:
Table 4: Ablation Result on DexYCh The arrow pointing to the right means closer to the GT.
Method
MPJPE|
FOL
FPL
FID
Diversity -
GT
125.53
w/o Depth Input
85.47 4M
33.41 E.34
20,9740x
56.36 1:25
66.40 4335
Seen
w/o Masked Training
73.41 2200
28.22 ti.3
14.42
=toss
44.87
73.09 1:292
w/o Physical Refinement
65.78
t100
15.35
=bow:
33.57
11.0
38.06
Ours
61.40
23.14 to0ss
2
12.15
=b0.24
31.24 L
39.62 toos
w/o Depth Input
90. 90.12 1
39.77
21.70
10030
77.38 1:
67.53 1:70
Unseen
w/o Masked Training
74.63
28.08
17.25
43.09
74.88 #100
w/o Physical Refinement
65.39
28.55
16.05
=b0:
45.06 1
41.03 1M
Ours
63.56-2200
27.29
13.06
-00.03
41.03
42.70
4.4 ABLATION STUDY
We conduct controlled ablation studies on DexYCB (Chao et al., 2021) to verify the necessity of each module, Each component is essential for UniHM to synthesize realistic and physically feasible dexterous hand manipulation sequences; the details be found in Table 4.
Masked Training (w/o Masked Training). We adopt a progressive masking curriculum for language-conditioned sequence generation. Training starts with the teacher forcing the use both language and ground-truth sequences, then gradually replaces a fraction of ground truth with [MASK]. Pt increases from 0 1, the model relies on language and its autoregressive history; the final stage uses language only, matching inference. This reduces exposure bias and improves sequential stability while retaining strong supervision early on.
RGB-D or RGB Input (w/o Depth Input). To better estimate object poses and reconstruct scene point clouds, we adopt RGB-D inputs and infer scene trajectories with language-condttoneed vi- suomotor module. When RGB-D is replaced by RGB-only, pose estimation and 3D reconstruction degrade substantially.
Physical Refinement (w/o Physical Refinement). Physical refinement is postprocessing step that makes a plausible plan physically valid for dexterous hands. We run lightweight simulation-base optimization that adjusts poses, contacts, and timing reduce collisions and slips, enforce joint and torque limits, and improve stability. The objective penalizes penetration, excessive contact forces, and abrupt accelerations while staying close to the original.

---

Published as a conference paper at ICLR 2026
5 CONCLUSION
We present UniHM, a unified framework for synthesizing sequential dexterous hand-manipulation sequences guided by open-vocabulary instructions. UniHM couples Unified Hand-Dexterous Tok- enizer, which utilizes a shared codebook with cross-hand code distillation to form a common discrete action space, with vision language model for instruction grounddd token generation. It applies physical-gune guided dynamic refinement, yielding semantically aligned and physically consistent trajec- tories across heterogeneous robotic hands. Extensive evaluations of DexYCB and OakInk, along with simulation checks and real-world trials, demonstrate strong generalization to unseen objects and trajectories, and validate contribution of each component through ablations. While UniHM advances conditee-od dexterous manipulation beyond pose-centric grasp generation, sev- cral challenges remain: reliance on RGB-D perception without tactile or force sensing, simplified energy terms for contact and friction, and limited coverage of bimanual tool-use scenarios. Future work will incorporate richer contact priors and feedback, scale the unified codebook to more hand morphologies, and close the loop with online adaptation better handle sequential, contact-rich tasks.
ACKNOWLEDGEMENT
This work was supported by National Natural Science Foundation China (62406195, 62303319) Shanghai'Tech A14S Initiative SHTAI4S202404, HPC Platform of ShanghaiTech University, and MoE Key Laboratory of Intelligent Perception and Human-Machine Collaboration (ShanghaiTecch University), and Shanghai Engineering Research Center of Intelligent Vision and Imaging. This work was also supported in part by computational resources provided by Feloud CO., LTD.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
German Barquero, Sergio Escalera, and Cristina Palmero. Seamless human motion composition with blended positional encodings. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 457-469 2024.
Yu-Wei Chao, Wei Yang. Yu Xiang, Pavlo Molchanov, Ankur Handa, Jonathan Tremblay, Yashraj S. Narang, Karl Van Wyk. Umar lqbal, Stan Birchfield, Jan Kautz, and Dieter Fox. DexYCB: A benchmark for capturing hand grasping of objects. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Zekai Deng, Ye Shi, Kaiyang Ji, Lan Xu, Shaoli Huang, and Jingya Wang. Human-object interaction via automatically designed vlm-guided motion policy, 2025. URL htttps://axivv.or
abs/2503. 18349.
Darion Grant, James Bethel, and Melba Crawford. Point-to-plane registration of terrestrial laser scans, ISPRS Journal of Photogrammetry and Remote Sensing, 72:16-26, 2012.
Chuan Guo, Xinxin Zuo, Sen Wang, and Li Cheng. Tm2t: Stochastic and tokenized modeling for the reciprocal generation of 3d human motions and texts. In European Conference on Computer Vision, pp. 580-597. Springer, 2022.
Peiqi He, Zhenhao Zhang, Yixiang Zhang, Xiongjun Zhao, and Shaoliang Peng. Spatial-ormllm: Improve spatial relation understanding in the operating room with multimodal large language model, 2025. URL https: //l:///araa waaaiiiviv.o.. org/ aaasbb//// /208 08199.
Mingzhen Huang, Fu-Jen Chu, Bugra Tekin, Kevin J Liang, Haoyu Ma. Weiyao Wang, Xingyu Chen, Pierre Gleize, Hongfei Xue, Siwei Lyu, Kris Kitani, Matt Feiszli, and Hao Tang. Hoigpt: Learning long sequence hand-object interaction with language models, 2025.
Wenlong Huang, Chen Wang, Yunzhu Li, Ruohan Zhang, and Li Fei-Fei. Rekep: Spatio- temporal reasoning of relational keypoint constraints for robotic manipulation. arXiv preprint arXiv:2409.01652, 2024.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
Kaiyang Ji, Ye Shi, Zichen Jin, Kangyi Chen, Lan Xu, Yuexin Ma, Jingyi Yu, and Jingya Wang. Towards immersive human-x interaction: A real-time framework for physically plausible motion synthesis. 2025. URL https: arxiv. org//bgg/sss.o aass/b/ssa 02106.
Juntao Jian, Yi-Lin Wei, Chengjie Mou, Yuhao Lin, Xing Zhu, Yujun Shen, Wei-Shi Zheng, and Ruizhen Hu. Zerodexgrasp: Zero-shot task-oriented dexterous grasp synthesis with prompt-based multi-stage semantic reasoning. 2025. URL https: /arxiv..ivv. org/ abs 2511. 13327.
Haosheng Li, Weixin Mao, Weipeng Deng, Chenyu Meng. Haoqiang Fan, Tiancai Wang, Yoshie Osamu, Ping Tan, Hongan Wang, and Xiaoming Deng. Multi-gras Multi-graspllm: A multimodal 1lm for multi-hand semantic guided grasp generation. arXiv preprint arXiv:2412.08-66. 2024a.
Kailin Li, Jingbo Wang, Lixin Yang, Cewu Lu, and Bo Dai. Semgrasp: Semantic grasp generation via language aligned discretization. In European Conference on Computer Vision, pp. 109-127. Springer, 2024b.
Kailin Li, Puhao Li, Tengyu Liu, Yuyang Li, and Siyuan Huang. Maniptrans: Efficient dexterous bimanual manipulation transfer via residual learning. In Proceedings of the Computer Vision and Pattern Recognition Conference, 2025.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: arXiv:1907.11 190.. 11692, 2019.
Zhirui Liu, Kaiyang Ji, Ke Yang. Jingyi Yu, Ye Shi, and Jingya Wang. Commanding humanoid by free-form language: A large language action model with unified motion vocabulary, 2025, URL https://arxins http//xivvvrrr.. www.vorrg/abae abs 2211 229633
11

---

Published as a conference paper at ICLR 2026
Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael Black. Smpl: A skinned multi-person linear model. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2, pp. 151-866.220 2023.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint urXiv:2304.07799. 2023.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 652-660, 2017 7a.
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical fea- ture learning on point sets in a metric space. Advances in neural information processing systems, 30, 2017b.
Yuzhe Qin, Yueh-Hua Wu, Shaowei Liu, Hanwen Jiang, Ruihan Yang, Yang Fu, and Xiaolong Wang. Dexmv: Imitation learning for dexterous manipulation from human videos. In European Conference on Computer Vision, pp. 570-587. Springer, 2022.
Yuzhe Qin, Wei Yang, Binghao Huang. Karl Van Wyk, Hao Su, Xiaolong Wang, Wei Chao, and Dieter Fox. Anyteleop: A general vision-based dexterous robot arm-hand teleoperation system. In Robotics: Science and Systems, 2023.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. arXiv preprint (00.0099,.,7999 2015.
Guy Tevet, Sigal Raab, Brian Gordon, Yoni Shafir, Daniel Cohen-or, and Amit Haim Bermano. Human motion diffusion model. In The Eleventh International Conference on Learning Repre- sentations, 2023.
Georgios Tziafas and Hamidreza Kasaei. Towards open-world grasping with large vision-language models. arXiv preprint arXiv:2406. 18722, 2024.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Hanqing Wang, Shaoyang Wang, Yiming Zhong, Zemin Yang. Jiamin Wang, Zhiqing Cui, Ji- ahao Yuan, Yifan Han, Mingyu Liu, and Yuexin Ma. Affordance-rl: Reinforcement learn- ing for generalizable affordance reasoning in multimodal large language model, 2025a. URL https: / arxiv. org/ abs 06206.
Hanqing Wang. Zhenhao Zhang, Kaiyang Ji, Mingyu Liu, Wenti Yin, Yuchao Chen, Zhirui Liu, Xiangyu Zeng, Tianxiang Gui, and Hangxing Zhang. Dag: Unleash the potential of diffusion model for open-vocabulary 3d affordance grounding, 2025b. URL https: /ttps:////xxvoorr/// abs/2508 01651.
Hanqing Wang, Yuan Tian, Mingyu Liu, Zhenhao Zhang, and Xiangyang Zhu. Sdeval: Safety dynamic evaluation for multimodal large language models, 2026. URL https://arxtt org/abs 2508.00... 06142.
Ruicheng Wang. Jialiang Zhang, Jiayi Chen, Yinzhen Xu, Puhao Li, Tengyu Liu, and He Wang- Dexgraspnet: A large-scale robotic dexterous grasp dataset for general objects based on simula- tion. arXiv preprint arXiv:2210.02697 2022.
Zifan Wang, Junyu Chen, Ziqing Chen, Pengwei Xie, Rui Chen, and Li Yi. Genh2r: learning gen- eralizable human-to-robel handover via scalable simulation demonstration and imitation. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16362- 16372, 2024.
Yi-Lin Wei, Jian-Jian Jiang. Chengyi Xing, Xian- Tan, Xiao-Ming Wu, Hao Li, Mark Cutkosky, and Wei-Shi Zheng. Grasp as you say: Language-guus -uiddgduddddd dexterous grasp generation. Advances in Neural Information Processing Systems, 37:46881-4668 2024.
12

---

Published as a conference paper at ICLR 2026
Yi-Lin Wei, Mu Lin, Yuhao Lin, Jian-Jian Jiang, Xiao-Ming Wu, -in Zeng. and Wei-Shi Zheng. Afforddexgrasp: Open-ser set language-guin dexterous grasp with generalizable-instue instructive affor- dance. arXiv preprint arXiv:2503. 2025a.
Yi-Lin Wei, Zhexi Luo,, Yuhao Lin, Mu Lin, Zhizhao Liang, Shuoyu Chen, and Wei-Shi Zheng. Om- nidexgrasp: Generalizable dexterous grasping via foundation model and force feedback, 2025b.
URL https: aawiiv.vvvv..///.i//.aaaa.aaaivv org/ abs/ 2510. 23119.
Shijie Wu, Yihang Zhu, Yunao Huang, Kaizhen Zhu, Jiayuan Gu, Jingyi Yu, Ye Shi, and Jingya Wang. Afforddp: Generalizable diffusion policy with transferable affordance, 2025. URL
https: arxii.... http///wwv.rr////ee...///.. alss//b//// 2412 2 03142.
Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, Fangchen Liu, Minghua Liu, Hanxiao Jiang. Yifu Yuan, He Wang, et al. Sapien: A simulated part-based interactive environment. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11097- 11107, 2020.
Peng Xiang, Xin Wen, Yu-Shen Liu, Yan-Pei Cao, Pengfei Wan, Wen Zheng, and Zhizhong Han. SnowflakeNet: Point cloud completion by snowflake point deconvolution with krip-rranfformer. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2021.
Peng Xiang, Xin Wen, Yu-Shen Liu, Yan-Pei Cao, Pengfei Wan, Wen Zheng, and Zhizhong Han. Snowflake point deconvolution for point cloud completion and generation with skip-transformer. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(5):6320-633 2023. doi:
10.1109/TPAMI.000 202..3212002.33271111
Yinzhen Xu, Weikang Wan, Jialiang Zhang, Haoran Liu, Zikang Shan, Hao Shen, Ruicheng Wang, Haoran Geng, Yijia Weng, Jiayi Chen, et al. Unidexgrasp: Universal robotic dexterous grasp- ing via learning diverse proposal generation and goal-conditioned policy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4737-4746, 2023.
Lixin Yang, Kailin Li, Xinyu Zhan, Fei Wu, Anran Xu, Liu Liu, and Cewu Lu. Oakink: A large-scale knowledge repository for understanding hand-object interaction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 20953-20962, 2022.
Zuyao You and Zuxuan Wu. Seg-rl: Segmentation can be surprisingly simple with reinforcement learning, 2025. httpst // wwwvvvvvvvvv//////////////////ia org/ abs///////// 2500 22662,
Hao Yu and Bogdan M Wilamowski. Levenberg -maqqurrdt training. In Intelligent systems, pp- 12-1. CRC Press, 2018
Xiangyu Zeng, Kefan Qiu, Qingyu Zhang, Xinhao Li, Jing Wang. Jiaxin Li. Ziang Yan, Kun Tian, Meng Tian, Xinhai Zhao, Yi Wang, and Limin Wang. Streamforest: Efficient online video un- derstanding with persistent event memory, 2025. URL https: wwwpsiiioorrrooooooooooooo abs/2509/ 24871.
Xiangyu Zeng, Zhiqiu Zhang, Yuhan Zhu, Xinhao Li, Zikang Wang. Changlian Ma, Qingyu Zhang, Zizheng Huang, Kun Ouyang, Tianxiang Jiang, Ziang Yan, Yi Wang, Hongjie Zhang, Yali Wang, and Limin Wang. Video-o3: Native interleaved clue seeking for long video multi-hop reasoning, 2026. URL https::////... /ttp://aaxvv..v/.v org/ abs 22201.2224.
Zhenhao Zhang, Ye Shi, Lingxiao Yang, Suting Ni, Qi Ye, and Jingya Wang. Openhoi: Open- world hand-object interaction synthesis with multimodal large language model. arXiv preprint arXiv:2505. 18947, 2025.
Ming Zhong, Yuanlei Wang, Liuzhou Zhang, Aretanx An, Renrui Zhang, Hao Liang, Ming Lu, Ying Shen, and Wentao Zhang. Veu-bridge: Hierarchical visual connotation understanding via semantic bridging, 2025a. URL http::::: /aaxivv.vvvv//////a//aaa//aaaav.
18121.
Yiming Zhong, Qi Jiang, Jingyi Yu, and Yuexin Ma. Dexgrasp anything: Towards universal robotic dexterous grasping with physics awareness. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 22584-22594, 2025b.
Bingfan Zhu, Biao Jiang, Sunyi Wang, Shixiang Tang, Tao Chen, Linjie Luo, Youyi Zheng, and Xin Chen. Motiongpt3: Human motion as a second modality, arXiv preprint arXiv:2506.2400 2025.
13

---

Published as a conference paper at ICLR 2026
UniHM: Unified Dexterous Hand Manipulation with Vision Language Model Appendix
A USE OF LLMS
We used an GPT to polish the experiments and introduction sections. We also use the Nano to generate the icon in Figure 1.
B IMPLEMENTATION DETAILS
B.1 RETARGETING
Hand Pose Retargeting, The DexYCB (Chao et al., 2021) and OakInk (Yang et al., 2022) datasets provide MANO pose sequences (Loper et al., 2023), object point clouds, and associated object trajectories. We employ Dex rex-Rettrggting (Qin et al., 2023) to map MANO joint configurations to multiple robotic hands via differentiable inverse- kinematics objective with joint limits and device- specific kinematic calibration. Given a MANO pose (t),o the device configuration is:
Qdev (1) arg min 96 [ymmn Amaxel
Σ wk (g)
(1)) > l|g - Grev 1
(B1)
where fA denote forward-kinematice constraints on fingertip/phalange keypoints, ddvv aligns MANO and device frames and scales link lengths, and qpev is the previous solution for temporal smoothness. We generate retargeted sequences for Allegro, Shadow, Schunk, LEAP, and Ability hands, and for the Panda gripper (mapping the thumb-index aperture to a scalar opening width via linear scaling with clamping).
Target Trajectory Generation. Let Mext E SEE(3) denote the camera extrinsics that map world coordinates the camera frame, and let Toyy cam (l) E SE(() be the object pose in the camera frame time f. The object target trajectory in the world frame used during training is
(t) M Tabi (t).
(B2)
We time-align Tea(t) with the retargeted hand poses using the dataset timestamps.
B.2 UNIFIED HAND TOKENIZER
Unified Codebook Training. We adopt shared VQ- VAE codebook z Shadow Hand due to its data availability and wildly used. For any hand type h, denote its encoder and decoder by E, and D,. Given an input sequence chunk (()), the encoder output, quantization, and reconstruction follow the main text:
{ initialized with
(1) = (x())))))
(B3)
C = ((z(h)) ) = arg min
(M) ek
(B4)
KE(K]
(1) B.s
(B5)
(h) = Dr. (z! (cc),
(B6)
where Q(-) performs nearest neighbor lookup in the codebook. finding the code vector with the minimum squared Euclidean distance. We train with the standard reconstruction and VQ losses:
Lree
| Dn
(b)
113
(B7)
z(Ai)) 1| + 3 (())))
sg (M) 2
Evg
sg
2
(B8)
where sg[:] is the stop-gradient operator and / > 0 is the commitment weight.

---

Published as a conference paper at ICLR 2026
To avoid codebook collapse with purely statistical procedure, maintain "cold-code" table based on per-epoch usage counts. Let mi be the usage of code during epoch computed from token indices {cg Jb=1
(6)
(f) B()
B
10
-b
(B9)
b=1
where 1(-) denotes the indicator function that evaluates to 1 if the condition inside its brackets is true, and otherwise. Define the cold set at epoch by count threshold To:
S A [K]
< Te
(B10)
At scheduled refresh epochs 1 Erestt, we selectively update only the codes belonging to the cold set. Collect buffer (t()) of encoder outputs since the previous refresh and run K-Means with R | obtain centroids {μr)/= We directly replace the cold codes with the newly computed centroids. With an injective assignment Stt) + {I....,,R? RR:
ek + (m(k) for ls E s())
e₁ e for ls & Stt
(B11)
Thus, only the codes in the cold-code table are refreshed those epochs, while frequently used codes remain unchanged. The objective for this stage is:
Lioss = Lree + fey.
(B12)
and Decoder Training. To integrate new hand morphology, we first align the new encoder to the reference encoder via retargeted pairs. Let Eret the reference (Shadow Hand) encoder and (Xnew; Xtef) be retargeted pair:
Casstill
[[Ee (Xnew) - Eret (xx
(B13)
After encoder alignment, we train the new encoder-decoder pair within the shared VQ-VAE using the same reconstruction and VQ losses:
(b) rotal Lree + Liq + CaistiCCiistt
(B14)
where Adistill 2 0 controls the contribution of the distillation term during the encoder alignment stage.
Training Details. We train a unified codebook with capacity K 8192 on YCB sequences. Since the inputs are ID pose and short-mothe signals, we instantiate encoders/decoders with either MLP or ID convolutional backbones and ablate both choices, as shown in Table B1. When training the reference encoder, we set the B = 0.25 and train the network with learning rate 1e 4. We also update the codebook at epochs 50, 100, 150, For the multi-encoder and multi-decoder training, the cold-code threshold To is set 1, and the distillation weight Adistin set to 0.1.
Table B1: comparison of the validation set performance for the MI.P and ID-Conv models.
Allegro
Shadow
Schunk
LEAP
Ability
Panda Gripper
Overall
MLP
0.0268
0.0450
0.0293
0.0348
0.0432
0.0221
0.0350
MAE
ID-Conv
0.0216
0.0297
0.0221
0.0257
0.0327
0.0182
0.0256
MLP
0.0545
0.0886
0.0656
0.0656
0.0913
0.0630
0.0736
RMSE
ID-Conv
0.0465
0.0654
0.0551
0.0531
0.0705
0.0519
0.0581
B.3 PHYSICAL OPTIMIZATION
Physical Model. We expand the contact term used the per-frame objective. Let 8, sg(q)) denote the i-th fingertip position (forward kinematics) in the world frame and (t) -t- be the object-frame transform at time f. Query the nearest neighbor (ps, n:) on Potj in the object frame, and define the signed point-to-laane distance:
d((g)) =n, (Tarr))) -1 -s)(())) -
(B15)
2

---

Published as a conference paper at ICLR 2026
We adopt an asymmetric, smooth penalty f(d) that is continuous and slope-matched 1 (outside)
matched at d 0:
d?
f(d)
kd kd -dd
0 (inside)
(B16)
At d 0, continuous and has matched first derivatives:
lim f(d) 0, lim (d) 0, lim f'(d) a - 0. d-02
(B17)
d-00
d-00-
Moreover, ("'(-) (0(0)) ae k-0 2 O. so ʃ is twice-differentiable and convex. Stacking
per-fingertip residuals gives:
T' ((t)) / 22 (d.:))) Evontacr (g:)
211 comttel (4:) ((d(a))
(B18)
with d > 0. For Gauss-Newton we linearize the residuals at the current iterate. Let x. x.( (t) (4:) and (g,((a)) = 0r,/0q; (the kinematic Jacobian in the object frame). Treating (p;, n,) as fixed within inner iteration:
(d.(ge) Jz,()))) Oq.
@ecottatt
22x (d)(())) nf Ja,(
(B19)
0q:
2 fVd (q)) +
which yields the contact Jacobian rows that form Jr = ooontcr (9:) with small > 0.
Generative Prior and Temporal Prior. We keep the generative and temporal priors from the main text and unify their roles as dynamic smoothers from two information Lources-semante intent (generator) and history (velocity/acceleratton)) The generative prior anchors qi to g. ".
Egaa(et f (en) = 1/4 (yc- of "T W gen - " W gen >
(B20)
The temporal prior penalizes first- and second-order differences:
Eiime (gts 42- (6-2) = 1 Il1 1pt w. + E1((t= - 4i-1) (di-1 41-2) W.
(B21)
where W vel objective:
0 and W acc > 0 (often Wvs Avell, W ncc )ooe)). Define the per-frame
Eg L. gon ope opt
ope opt -9-11 4:-2) Evontace (4:) Egen gteh) ) + Etiee (gt, 96- 96-2)
(B22)
Let Peomtaci(e) denote the stacked contact residuals. Linearizing Feontae (9t + Aqi) = Feontact (4:)
+
J, Jg and expanding the two quadratic priors around qi give:
El(g + Aqe) (contacc(()) J.Dqellē + A (Wges + W W Waee Dcc DqF TW (B23)
where:
W W gen (4: - =) + W We opt + W Waco(lg: 9
(4-11
ope
opt 41-22)
))
(B24)
Setting the gradient w.r.t. Aq to zero and adding Levenberg- Marquurrd damping À 2 0 yields the normal equations:
JJ W gen + W vel
Wass
acc + +l)Aq - reootace (g.)
W.
(B25)
Fort t < 2, we use boundary priors consistent with the main text, e.g., opt
opt 9-1
40
gen
Optimization Procedure. Given the decoded grasping trajectory 2gens the object point cloud Pabj- and the target pose trajectory T.n we refine each frame by Gauss with LM damping while freezing nearest- neightt correspondences inside each inner iteration. In our implementation, we set = k- = and use small positive constant e to handle cases where f(d) approaches zero.
In practical environments, obtaining precise point clouds is often challenging. significant advan- tage of employing the kernel function (d) is the inherent robustness it provides to the segmentation process.
As shown in Fig. B1, the kernel function is relatively flat in neighborhood of d = 0, which implies that, even when the segmented point cloud contaminated with noise, the resulting cost term does not deviate substantially from that constructed from the ground-truth point cloud, and thus the optimization still converges to the correct solution in the presence of noisy inputs. We further demonstrate the optimization results under noisy conditions (for clearer illustration of the grasp point, we have omitted the remaining structure the dexterous hand, plotting only the object point cloud and the positions of the hand's fingers), shown in Fig. B2.

---

Published as a conference paper at ICLR 2026
Algorithm Physical-guided dynamic refinement
Require: Generated sequence Qgen = {qr 2 T point cloud Pabj. target poses (Tm(
weights A W gen- W vel, W acc. damping À 2 0
Ensure: Refined sequence Qupt {9: 00 1 ==
1: Set boundary priors opt 4-1 T0
2: f = 0 to T do
3: Initialize q: t- gi
4: repeat
5:
Transform fingertips 24 (4)4)) =
6:
For each fingertip i, find (pi, n.) NN((I.(:); Poni) and determine inside/outside sign Compute d, (4t). (oottac((:).. and Jr òr Orcontact/ /3ge with correspondences fixed opi opt Form W using 4- 9:-2
7:
8:
9:
Solve (J1² Jr + W gen + W vel + W uce + Ag -f comtact (q:) W
10:
Update: Q: f- + Aq; adjust À by decrease/increase of &
until converged or max
9: + 9t
13:
end for
Parametric Piecewise Function: a, k
40
1.0
0.5, R=03
0.5
g= k005
1.0
2.3.5
1.0, 1=0.0
0.0
y= Xx]
Figure Bl: The function plots under varying values of a and & are displayed. Note that a and control the curve behavior for 0 and < 0, respectively. For comparison, the curve of |a| plotted using a dashed line.
4

---

Published as a conference paper at ICLR 2026
wos Trannss
Woing raperips
(A)
(B)
(C)
(D)
Figure B2: Optimization results when the point cloud includes noise. Here, the black points rep- resent the object point cloud, and the red/green points denote the positions the dexterous hand's fingertips. (A) Optimization using the (dd kernel, visualized the noisy input. (B) Optimization using the Euclidean distance, visualized on the noisy input. (C) The f(d) kernel optimization result projected onto the clean noise-free point cloud. (D) The Euclidean distance optimization result pro- jected onto the noise-free point cloud.
C MODEL INSTRUCTION
C.I TEXT TOKENIZER
We employ a pretrained language model from the Qwen series, specifically Qwen3, as the core component for our language input pipeline. This choice is motivated by its strong performance across a wide range of natural language tasks and its robust, efficient tokenizer.
The Qwen3 tokenizer operates on a byte-level Byte-Pair Encoding algorithm (Sennrich al., 2015). This method ensures that all possible input characters are representable, and it is particularly effec- tive handling diverse linguistic inputs, including code and technical terms, without resorting to unknown tokens. The tokenizer's s vocabulary size is Viok. and each input sequence S is tokenized .....,,,,,, IN), where is the sequence length. The tokenization
into a sequence of tokens = (41, .,,..., process can be formally expressed as:
tokenize(S) T.
(C1)
This token sequence T then converted into sequence of token embeddings, which serve as the input the subsequent layers of our model. We utilize the tokenizer's built-in padding and truncation functionalities to handle variable-length sequences, ensuring a consistent input shape for the model.
C.2 POINT CLOUD ENCODER
We use PointNet-based architecture (Qi al., 2017a;b) our point cloud feature extractor, which a key component of our model. The network directly consumes raw point cloud data, which is an unordered set of n points, with each point represented by its (æ, 2) coordinates. The PointNet architecture addresses the permutation invariance point clouds by using a symmetric function to aggregate information from each point.
5

---

Published as a conference paper at ICLR 2026
Specifically, the network first applies shared multi-layer perceptron to each point individually to transform the input features. This is followed by a max-pooling layer, which acts as a symmet- ric function to aggregate the point-wise features into global feature vector. This process can be concisely described as:
f(X) = max {h(z:)),
(C2)
i=
where X { (21.....: Fn is the input point cloud, h represents the shared MLP, and f(X) is the resulting global feature vector. The network is configured to output several feature vectors, which then concatenated and fed into the subsequent main model for further processing.
C.3 TRAJECTORY ENCODER
Our model takes the target trajectory Turr as input. We encode this trajectory using two-layer MLP obtain feature representation zt). The MLP processes the pose Pur (t) each timestep and is defined as:
(() = 02 (W20) (W ( + bi) + b2).
where and 02 are non-linear activation functions. The hidden layer dimension is set to 512 during training.
C.4 MASKED TRAINING
During training, we progressively mask the ground truth hand poses encourage the model to learn the temporal relationships within a sequence. This approach, which has been shown to be highly effective in fields like language modeling (e.g., BERT (Liu al., 2019)) and computer vision (e.g.. DINOv2 (Oquab et al., 2023)), allows the model to generate hand poses from given context.
Specifically, we feed the ground truth trajectory into the VQ-VAE encoder and then use mask to conceal the corresponding hand poses. The masked hand pose tokens are replaced with a single, learnalle token. This masking process can formalized as:
Qmasked = M o E(2) + (1 - MM) @ Tmask
(C4)
where Q is the ground truth hand pose sequence, E is the VQ-VAE encoder, M is binary mask, and Taask is the learnable token.
We implement this masking strategy with a progressive curriculum across training epochs. For the first 20% epochs, we not mask any hand poses. From 20% to 80% of the epochs, we linearly increase the masking ratio until it reaches 100%. Finally, during the last 20% of epochs, all hand poses are consistently masked, forcing the network to perform full generation. We train the model for 100 epochs using the AdamW optimizer with a learning rate of le-4.
C.5 TARGET TRAJECTORY PLANER
We use CLiPort to plan the target trajectory Tear Specifically, we take RGB-D and instructions as input data. Then, based on GenH2R (Wang et al., 2024), we generate a smooth trajectory in the output space, which is then encoded using the aforementioned trajectory encoder and fed into the model.
D ADDITIONAL VISUALIZATION
We employ Sapien (Xiang al., 2020) visualization engine for observing and validating the generated results. We also present some additional visualizations here.
E REAL-WORLD SETUP
The experimental setup utilizes a Franka manipulator arm equipped with Panda Hand, an XHand dexterous hand, and Inspire Hand in EIg. The t-degree-offreeddon (DoF) Franka provides

---

Published as a conference paper at ICLR 2026
Figure D1: The visualization results of our grasping sequence in Sapien. More real world examples could be found in the video.
Figure D2: The visualization results of our grasping sequence in Sapien. More real world examples could be found in the video.
7

---

Published as a conference paper at ICLR 2026
workspace comparable to that of human amm. The 2-DOF Panda Hand, 12-DOF XHand, and 6-DOF Inspire Hand, with overall dimensions similar to a human hand, feature a proportionally elongated pinky finger. This specific design requires adaptive scaling heuristic to ensure natural and fluid motion.
We employ Zed camera acquire RGB-D observations, apply PointSAM( You & Wu, 2025; Wang et al., 2025a; Wu et al., 2025) to segment the target point cloud, and subsequently leverage Snowflak- eNet (Xiang et al., 2023; 2021) reconstruct a complete point cloud that serves as input our model.
Panda Gripper
X-Hand
Inspire Hand
Figure El: Cross-embodiment Set up
F EVALUATION METRIC
MPJPE (Mean Per-Joint Position Error). For each sequence, compute the Euclidean error of every hand joint in every frame, Then average over joints and time:
MPJPE
P1
(
(4 9t. 2
(F1)
1=
The final MPJPE reported
1 N Σ MPJPE,
MPJPE
(F2)
where N is the number of test sequences. FPL (Final Position Location Error) FPL. measures how close the final hand placement is to the ground truth. Let cz and ale be the 3D positions the hand root / palm center at the last frame T, of sequence
FPL:
(8) C ll
FPL
Σ FPL. N
1
(F3)
FOL (Final Orientation Location Error). FOL measures the orientation error of the hand at the final frame. Let R2(!) and fe(i) be the ground-truth and predicted rotation matrices of the hand root; the orientation error is
ttrace (RZ)) R(()) -1
(
0, = arccos
(F4)
8

---

Published as a conference paper at ICLR 2026
converted to degrees. We report
FOL Σ 8,.
1 N
(F5)
Feature extractor. We first embed every hand-object interaction sequence (either real or gener- ated) using a pretrained motion encoder s(-) that consumes the full temporal joint trajectory (and object pose) and outputs -ied-dimensional feature vector z f(x) R We use the same encoder for all methods and compute all distributional metrics in this learned feature space.
FID (Fréchet Inception Distance). Let {
Mr be the features of real test sequences and Mg the features of generated sequences. We estimate the empirical means and covariances:
(*k Jk=1
1
Mr
real K
E, = Cov((=# });
(F6)
Mr
1
M4
gen
Eg Cov 1)
(F7)
Mg
and compute
Tr FID ||,, +
-(E,,,,)))
(F8)
Diversity. Diversity measures how varied the generated sequences are in the same feature space. Given the features {= M for method, we compute
Div M(M -1) 1)
2
Σ ISacbEM
Za 2b 20l12-
(F9)


Reviews:
[{"review_id": "NWyUXS2ToV", "reviewer": "Reviewer_s9Mv", "summary": "The paper presents a VLM approach for unified dexterous manipulation. Their approach consists of multiple components. First, a unified tokenizer is learned for multiple hand morphologies using a VQ-VAE codebook. The tokenizer is learned in such a way that all morphologies share the same codebook but different encoder and decoders. For the VLM training, qwen 0.6B is used as base modell. The VLM is trained to ouptut the hand configurations given the object target trajectory and the point cloud of the object. For inference, the point cloud is inferred using PointSAM and the target trajectory using CLIPort. Finally, the hand poses are refined via optimization using a cost function that aligns contact normals with surface normals of the point cloud as well as produces smooth motions. The approach is tested on several benchmark datasets as well as real robot tasks.", "questions": "- I got confused in section 3.3 as its unclear how the mathematical objects in Eq. 7 and 8 look like. Could you please add more information here and clarify what T_tar and P_obj is (also formally, what is the dimensionality)?\n- Is the target trajectory the 6D pose trajectory of an object or the 3D positions of a keypoint?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- First VLM for unified (multi-embodiment) dexterous manipulation\n- The VLM can be trained purely from human data\n- Performance seems to be competitive\n- Real robot experiments are convincing", "weaknesses": "- Some sections are unclear and missing detail (see questions)\n- The paper would benefit from further ablations. E.g. the different parts of the cost function. Or the benefits of using a unified latent space. Here, the (maybe naive) alternative would be to learn everything in a single space (e.g. MANO) and retarget the output of the VLM afterwards. Such a comparison would be insightful. \n- There is no further information on the costs of the optimization of the grasp. How long does it take? Is it real-time capable?\n\nHowever, I think the approach is interesting and shows a promising performance. The strengths do outweight the weaknesses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aorhmbgi1b", "reviewer": "Reviewer_7Tpz", "summary": "In this work, the authors propose a novel framework for manipulation that generalizes across different dexterous-hand morphologies. Furthermore, the proposed method can generalize over unseen objects. The method consists of three main stages: first, the motion is tokenized in a morphology-independent way. Second, a token sequence is generated based on combined text, perception, and token history. Finally, the motion is generated by using physics-aware decoding. The authors report results that show that their method is superior to baselines on existing datasets, as well as in real-world evaluations.", "questions": "Have you tried different VLMs?  \n\nHave you tried others pretrained models instead of PointSam and CLIPort? \n\nHave you tried different capacities of the Codebook?\n\nWhich simulation engine have you used? In the Appendix D it is mentioned that Sapien is used for visualization.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is considering a relevant problem of generalization to different hand morphologies. The authors proposed a comprehensive framework that shows good results across multiple metrics and in real-world robot setups. They successfully manage to combine and benefit from existing models, such as PointSam and CLIPort, and integrate them into their framework. Overall, the approach is well-defined, and evaluation supports the claims.", "weaknesses": "The main weakness of the approach is the complexity of the method. Currently, the approach consists of multiple parts and many different pre-trained models that need to be finetuned.  \n\nThe authors do not provide code. It is unclear which simulation environment they have used.\n\nThe authors mention some terms without providing sufficient explanation in their context or a reference to related work. Such terms are MANO poses (line 153), vector-quantization operator (line 187), knowledge distillation (line 203).\n\nSmaller writing comments and typos:\n- In the introduction, the authors listed 4 core capabilities, and 4 contributions. However, they almost identically, and their repetition is not well motivated. Therefore, the authors should either differentiate them better, or combine them in order to improve legibility.\n- Line 045: No space before AffordDexGrasp\n- Line 272: No space between sentences (generation.A practical)\n- Table 1, 2, and 4: The arrow pointing to right next to Diversity is unintuitive, and its meaning might confuse readers, who did not read the evaluations section in detail.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mB3pctwoGs", "reviewer": "Reviewer_ZSGv", "summary": "The paper proposes a unified pipeline for language-conditioned, sequential dexterous-hand manipulation. The method comprises: (i) a Unified Dexterous-Hand Tokenizer (VQ-VAE) that maps heterogeneous hand morphologies into a shared discrete codebook and decodes back to hand-specific joint trajectories; (ii) a VLM-based generator that, given RGB-D input, an instruction, object point clouds, and tokenized history, autoregressively produces manipulation tokens; and (iii) a physics-guided dynamic refinement that enforces contact and temporal smoothness while adhering to the generator’s intent, yielding physically feasible, executable trajectories. Experiments on DexYCB and OakInk show consistent improvements over strong motion-generation baselines (TM2T, MDM, FlowMDM, MotionGPT3) and higher real-robot success rates; ablations demonstrate each module’s contribution.", "questions": "See weaknesses.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper proposes a strong framework that incorporates a cross-dexterous-hand representation, language-conditioned sequence generation, and a physics-guided dynamic trajectory refinement module.\n\nThe authors perform extensive experiments across multiple datasets, covering both seen and unseen settings, with comprehensive evaluations and ablations.", "weaknesses": "Tokenizer evaluation limited to a single hand. Although a unified dexterous-hand tokenizer is proposed, both the HOI and real-world experiments appear to use only one hand type. A broader evaluation across multiple robot hands would more convincingly validate the tokenizer’s generality and cross-hand transfer.\n\nUnderspecified sequence-generation metrics. The paper introduces a manipulation sequence generator, but the evaluation protocol for sequences is insufficiently detailed. Please clearly define the quantitative metrics and how they are computed.\n\nMissing text–motion alignment evaluation. Given the language-conditioned setup, include explicit Text–Motion Alignment assessments is necessary. Please provide both quantitative measures and qualitative analyses (e.g., human judgments of instruction adherence).\n\nMinor issues.\n\nLine 075: “Language -guided” → “Language-guided”.\n\nLine 115: missing space before “introduces”.\n\nLine 377: only five metrics are listed", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fCXAkz8aWQ", "reviewer": "Reviewer_uKvV", "summary": "This paper explores the use of multimodal large models for cross-embodiment dexterous manipulation. The main contributions are as follows:\n1. Constructed a dexterous manipulation dataset encompassing multiple heterogeneous dexterous hands, based on human manipulation datasets.\n2. Proposed a language-guided, multimodal large model-based framework for cross-embodiment dexterous manipulation.", "questions": "1.\tIn real-world deployment in open-world scenarios, high-fidelity 3D assets of the objects being manipulated are unavailable. Could this affect the performance of the physics-guided dynamic refinement?\n2.\tThe experiments just include comparisons with several human motion generation models. What about the performance of language-guided Hand-Object Interaction models [1] or language-guided grasp models [2, 3] (which can extend to the setting of this paper) on the current dataset.\n\n[1] Cha, Junuk, et al. \"Text2hoi: Text-guided 3d motion generation for hand-object interaction.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[2] Wei, Yi-Lin, et al. \"Grasp as you say: Language-guided dexterous grasp generation.\" Advances in Neural Information Processing Systems 37 (2024): 46881-46907.\n\n[3] Zhong Y, Huang X, Li R, et al. Dexgraspvla: A vision-language-action framework towards general dexterous grasping[J]. arXiv preprint arXiv:2502.20900, 2025.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1.\tThe proposed dataset features highly aligned grasp poses with semantic annotations for cross-embodiment manipulation, which is quite a contribution to the community.\n2.\tProposes and validates a viable methodology for applying Multimodal Large Language Models (MLLMs) to the challenging problem of high-DoF dexterous manipulation generation.\n3.\tAchieving promising results in real-world deployment on high-DoF dexterous embodiments using only human demonstration videos, providing a positive signal for addressing the issue of expensive data collection.", "weaknesses": "1.\tThe selected tasks are relatively simple. On one hand, many demonstrated actions (e.g., opening a drawer) can be performed by simpler, lower-cost two-fingered grippers, failing to highlight the advantage of dexterous hands. On the other hand, the tasks designed for dexterous hands (e.g., picking and placing objects) involve overly simplistic semantics, lacking part-level manipulation or interactions with clear subsequent intent (e.g., \"handing the scissor handles to the user\" or \"grasping the teapot lid\", which are feasible in OakInk). We expect to see complex manipulation sequences that are hard to do for non-dexterous hands and feature richer semantic hierarchies.\n2.\tThe evaluations in simulation are primarily analytical metrics. I believe success rates for the manipulations are essential.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)