Scaffolding Dexterous Manipulation with Vision-Language Models
Authors: Vincent de Bakker
Joey Hejna
Tyler Ga Wei Lum
Onur Celik
Aleksandar Taranovic
Denis Blessing
Gerhard Neumann
Jeannette Bohg
Dorsa Sadigh
Venue: NeurIPS 2025 poster ()
Keywords: Dexterous Manipulation
Vision-Language Models
Reinforcement Learning
Primary area: reinforcement_learning
TL;DR: We use VLMs to generate high-level hand-object plans for dexterous manipulation.

Abstract: Dexterous robotic hands are essential for performing complex manipulation tasks, yet remain difficult to train due to the challenges of demonstration collection and high-dimensional control. While reinforcement learning (RL) can alleviate the data bottleneck by generating experience in simulation, it typically relies on carefully designed, task-specific reward functions, which hinder scalability and generalization. Thus, contemporary works in dexterous manipulation have often bootstrapped from reference trajectories. These trajectories specify target hand poses that guide the exploration of RL policies and object poses that enable dense, task-agnostic rewards.
However, sourcing suitable trajectories---particularly for dexterous hands---remains a significant challenge. Yet, the precise details in explicit reference trajectories are often unnecessary, as RL ultimately refines the motion. Our key insight is that modern vision-language models (VLMs) already encode the commonsense spatial and semantic knowledge needed to specify tasks and guide exploration effectively. Given a task description (e.g., “open the cabinet”) and a visual scene, our method uses an off-the-shelf VLM to first identify task-relevant keypoints (e.g., handles, buttons) and then synthesize 3D trajectories for hand motion and object motion. Subsequently, we train a low-level residual RL policy in simulation to track these coarse trajectories or ``scaffolds'' with high fidelity. Across a number of simulated tasks involving articulated objects and semantic understanding, we demonstrate that our method is able to learn robust dexterous manipulation policies. Moreover, we showcase that our method transfers to real-world robotic hands without any human demonstrations or handcrafted rewards.

Scaffolding Dexterous Manipulation with Vision-Language Models
Vincent de Bakker Joey Hejna'
Tyler Ga Wei Lum
Onur Celik²
Aleksandar Taranovic*
Denis Blessing
Gerhard Neumann"
Jcannette Bohgb
Dorsa Sadigh'
Stanford University
Karlsruhe Institute of Technology
Abstract
Dexterous robotic hands are essential for performing complex manipulation tasks, yet remain difficult to train due to the challenges of demonstration collection and high-dimensional control. While reinforcement learning (RL) can alleviate the data bottleneck by generating experience in simulation, typically relies on carefully designed, task-specific reward functions, which hinder scalability and generalization. Thus, contemporary works in dexterous manipulation have often bootstrapped from reference trajectories. These trajectories specify target hand poses that guide the exploration of RL policies and object poses that enable dense, for
task-agnostic rewards. However, sourcing suitable trajectories-partiiue dexterous remains significant challenge. Yet, the precise details in explicit reference trajectories are often unnecessary. as RL ultimately refines the motion. Our key insight that modern vision-languaae models (VLMs) already encode the commonsense spatial and semantic knowledge needed to speciffy tasks and guide exploration effectively. Given a task description (e.g.. "open the cabinet") and a visual scene, our method uses an off-the-shelf VLM to first identify task-relevant keypoints (e.g., handles, buttons) and then synthesize 3D trajectories for hand motion and object motion. Subsequently, we train low-level residual RL policy in simulation to track these coarse trajectories or "scaffolds" with high fidelity. Across number of simulated tasks involving articulated objects and semantic understanding, we demonstrate that our method able to learn robust dexterous manipulation policies. Moreover, we showcase that our method transfers to real- world robotic hands without any humaan demonstrations or handcrafted rewards.
1 Introduction
Dexterous manipulation is essential for a range of real-world tasks such as using a power-drill or twisting door knob which require the fine-grained control offered by human-like hands [2]. Enabling dexterous capabilities has therefore been a long-standing goal in robotics. Despite the intrinsic advantages of dexterous hands over simpler end-effectors, existing learning paradigms have struggled to cope with their complexity [56]. The prevailing approach for training generalist policies imitation learning from demonstrations [5, 49] has achieved limited success with robot hands, primarily due to the challenges of collecting accurate data with dexterous hardware, resulting in scarcity of high-quality demonstrations [54, 66]. While alternative approaches attempt to re-target demonstrations from easier interfaces [19, 25, 32, 53, 57, 71, 73, 78], e.g., human hands, such approaches often induce irrecoverable errors for fine- fine-grained tasks.
To avoid both data scarcity and the embodiment gap, a combination of reinforcement learning (RL) and sim-to-real transfer has emerged as a promising approach by enabling large-scale experience generation [3]. However, using RI. simply shifts the burden from data collection to reward design.
Videos htttss:/ssssteesss http://ssiits.gggleeeeeee www..///ise//eeooaaaaaaoooaaaaaa
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

"Hammer once"
VLM Planner
Sampled Plans
Residual RLL
Semantic Keypoinks
Adjust dW
Finger
hanck
control
Image
Motion Reasocing
Go Ri Go.yhup
Rand omiration
Figure 1:
Overview of our method: VLM generates hand and object keypoint trajectories from language command and scene image. low-level residual RL policy is trained track these trajectories in simulation.
Standard RL approaches for dexterous manipulation necessitate hand hand-crafting complex, task-specific reward functions. A large amount of this complexity arises from the need to guide exploration; with large action spaces, dexterous hands need to be coaxed towards the correct part of the observation space to make progress on a task. Thus, various approaches have used demonstrations to bootstrap the RL process [15, 20, 21, 46, 55, 56]. In dexterous manipulation, this is often done through trajectory tracking, where instead of designing a complex reward function, a policy is rewarded for tracking the exact wrist and object positions in demonstration, leaving RL to only make adjustments [6, 79]. By re-framing dexterous manipulation as a trajectory tracking problem, such approaches can leverage dense, task-agnostic rewards and guide exploration by using residual policies [16, 26].
Though demonstration tracking overcomes the design challenges associated with RL, paradoxically re-introduces the same dependence on demonstrations we sought to avoid in the first place. For example, prior works [6. 79] that use tracking- bbaen RL. for dexterous manipulation often require large prior datasets with thousands of teleoperated demonstrations [13, 76], restricting the method to tasks for which data has already been collected. Furthermore, the dependence on accurate reference trajectories extends to test time, meaning that either new demonstrations are needed each time the environment changes, or bigh-level policy must accurately predict them. In the context of prior work, either solution requires more demonstrations. However, for the majority of practical tasks it is unclear why demonstrations were chosen as reference trajectories when RL only requires that motions are accurate enough to provide a suitable reward and guide exploration.
Our key insight that coarse motion plans ("scaffolds") can be sufficient for both of these criteria. Generating such plans only requires high-level spatial and semantic reasoning, the exact abilities afforded by new advancements in vision-language models (VLMs). Due to their training on vast and diverse datasets encompassing
knowledge and embodied reasoning [64], VLMs have been able to synthesize [14, 44] and discriminate [24, 48] desired robot motions. Consequently, VLMs have the potential to supply the high-level reward signals and exploratory guidance needed for RL through coarse motions. So long as these motions generally encapsulate the desired behavior, RL can optimize per- per-timestep offsets and finger motions to maximize the tracking reward, ultimately surpassing human telcoperation in both performance and precision, climinating the reliance on demonstrations.
Building upon this insight, we introduce framework for learning manipulation policies for dexterous robot hands with VLM -LM-generated motion plans and residual RL. Given a natural language instruction (e.g., "hammer once" Fig. 1) and image, an off the-shelf VLM first identifies relevant object keypoints. Then, provided the initial keypoints and hand pose, the VLM generates the associated 3D trajectories for both object and hand motions to define the supervision targets for "low-level" residual RL policy trained in simulation. By controlling the robot's hands and fingers, the low-level policy learns to effectively track the trajectory and complete the desired task. Using VLMS for high-level guidance has a number of distinct benefits. Through repeated querying, we can randomize the initial keypoints and high-level trajectories, enabling generalization to new unseen initial conditions and new trajectories at test-time. Moreover, in situations where a VLM's high-level planning is error-prone, performance can often be substantially improved by simply providing additional in-context examples.
We evaluate our method across a suite of challenging dexterous manipulation tasks in simulation requiring semantic understanding, human knowledge about concepts like "hammering". and precise manipulation for difficult or articulated objects. Across tasks, our method achieves close perfor- mance in both success rate and generalization to handcrafted, oracle plans despite requiring no manual reward engineering. We demonstrate successful sim-to-real transfer on a physical dexterous robot, achieving robust performance without any human demonstrations or manually designed rewards.

---

2 Related Work
Planning with Vision-Language Models. Recent advances have demonstrated the potential of VLMs to guide robotic planning through their powerful semantic and spatial reasoning capabilities. One family of approaches directly synthesizes policies by translating natural language instructions into executable code using low-level perception and control APIS [23, 34, 59, 72]. To extend this to dexterous manipulation, [36] integrates predefined skill libraries, at the expense of limiting generalization and behavioral diversity. Other efforts propose using VLMs to plan actions by generating spatial keypoint constraints [22] or directly producing waypoints [14, 50J. However, such methods operate in an open-loop fashion and lack the closed-loop feedback necessary for fine-grained, adaptive control in dexterous tasks. Aside from directly learning policies, several works use VLMs to code dense reward functions [41, 65, 80], but these approaches often require privileged access to environment internals and result in opaque and sometimes hard-to-interpret reward structures. Other approaches more directly leverage the vision capabilities of VLMs to act as success detectors [12, 30, 75], reward functions [42], or value functions [40, 77] for RL. Oftentimes, these quantities can be learned from VLM generated preferences [28, 31, 69]. However, all of these approaches are often too imprecise to produce the dense optimization signals required for dexterous manipulation and are less efficient than using the VLM to simply produce a plan.
Learned Dexterous Manipulation. Though early works demonstrated the feasibility of deploying in- hand manipulation policies trained in simulation on real robots [3, 7, 18, 35], they relied on carefully crafted reward functions for each task. Such approaches have proven most successful in locomotion [1, 4, 29], where rewards are more easily designed and terrain can be replicated, unlike object dynamics in manipulation. More recent efforts scale to full-arm dexterity and multi-object grasping [39, 60J, while others incorporate human priors to improve sample efficiency [43]. Despite these advances, most approaches are still limited to only a set task, e.g. object grasping or rotation [52, 67]. where manually, task specific rewards can be designed. However, this approach remains inherently unscalable to more complex and non-cyclic tasks. Prior works address this bottleneck by tracking motions from demon- strations [6]. Our work aligns with this shift, but instead sources coarse motions from VLM feedback.
Dexterous Manipulation by Tracking Motions. When framing dexterous manipulation as a tracking problem, dense rewards are easy to obtain via tracking error [4, 15, 51]. Some systems leverage motion capture data to extract object and wrist trajectories from human demonstrations, which are then used to train tracking policies in simulation via residual RL [6, 33] or through behavior cloning [17]. Other approaches improve robustness by iteratively adding successful rollouts to the training dataset [37]. Recent work also shows that a single demonstration can bootstrap effective policy learning [15, 38]. However, all of these methods depend on human demonstrations, which are expensive to collect and difficult to scale. Moreover, policies trained on such data often fail to generalize to novel initial states. Our method retains the advantages of the trajectory tracking framework - dense supervision and residual policy learning - while eliminating the dependency on demonstrations by using VLMs to infer target trajectories directly.
3 Dexterous Manipulation via VLM Feedback
We focus on dexterous manipulation using robotic hands with visual observations and natural language instructions, with the aim of developing general approach transferable across diverse applications and settings. Following prior work [4, 6). we adopt a hierarchical approach that naturally delineates planning and control. However, instead of centering plans around demonstrations, we leverage VLM to produce coarse plans sufficient to "scaffold" low-level RL. We interface between these two components using 3D keypoints, as they provide sufficient precision for effective manipulation [68, 74], yet are abstract enough for VLM reasoning [14, 44] and often used during pre-training [27, 64].
3.1 Problem Formulation
Our goal is to learn a hierarchical policy for dexterous manipulation, where the high- and low-level policies interface via 3D keypoint-based plans or trajectory "scaffolds". While several prior works assume access to ground-truth states (often in simulation), such information is typically only partially observable practice. For example, it is unrealistic to assume that one is able to precisely measure object poses and velocities in the real world. Only the dexterous hand's proprioceptive state (w, q. q)
3

---

comprised of the current wrist pose w E SE(3), finger joint positions q and velocities q is exactly known. Instead of ground-truth states we assume access to RGB images I, depth D, and language instruction I. which communicates the task. Following standard practice in dexterous manipulation, we use an absolute action space comprised of desired wrist w and finger joint positions qute, i.e.,
(w q ( E A.
The high-level policy x² produces a coarse, 3D keypoint-based plan 7 from the language instruction L and an initial high-level observation of at time t = I containing the initial image I1 and wrist position W1. As we instantiate T² using VLM, we assume the ability to project 2D keypoints u R in image space to 3D keypoints x(i) E R in world coordinates, which is easily accomplished in practice using depth information D and camera parameters (intrinsic and extrinsic). The number of 3D keypoints k in the final plan is specified through the instruction L. We enumerate these keypoints as (1), x (8) and abbreviate sequences of length T through time via the short-hand T. The final keypoint plan includes 3D keypoint sequences x Xi.T..... (k) and sequence of predicted wrist poses WI.T. This coarse plan encapsulates both information about the task via the keypoint sequences which can capture object movements, and information to guide the agent's exploration via the wrist position w. The high-level policy can be written as:
T WI:TT
x
(As 1:T
I1. W1, L)
The high-level policy only provides a coarse plan for the wrist w - not the finger joint positions which will be learned by the low-level policy with RL.
The low-level policy n² produces wrist and finger actions a to execute the keypoint plan т. We assume access keypoint tracking model, which given an initial 3D keypoint x)" at time t = 1 is able to track its position over time to produce estimates x The low-level policy x' is then optimized via RL using reward function that encourages consistency between the estimated 3D keypoints X and those produced by the plan T, x; (8) To accomplish this task, it takes as input both a low-level observation of E O', consisting of the proprioceptive state (w, q. q) and estimated keypoints x(1)
(1) and all future steps of the plan TE. Succinetly,
Larg W
q:
qr Ge- W.,
WET,
propeio
keypoiat estimates
plan TAT
Provided the high- and low-level decomposition of our approach, we now describe each component.
3.2 Trajectory Generation for High-Level Policies via VLMs
We implement the high-level policy x² using a VLM, which must be able to effectively translate the task description L and initial image 1 into a coarse motion plan T for x' to complete. This necessitates a high-degree of semantic and spatial reasoning capabilities: the paths of different
keypoints ( must obey the desired relationships between objects (e.g- put the apple on the cutting board) and physical constraints (e.g. the head of the hammer must remain attached to the handle).
At the same time, the predicted wrist trajectory w W.T must remain close to the objects of interest to facilitate manipulation. We generate coarse keypoint plans using VLMs in three phases: (I) semantic keypoint detection, (2) coarse trajectory generation, and (3) interpolation. The left of Fig. provides a visual overview. Optionally, generated plans 7 can be improved using few-shot prompting.
Keypoint Detection. First, we elicit task-relevant keypoint names from the VLM using a standardized preamble. For example, for the "hammering" task, these keypoints (Fig. 1) could include both the handle and head of a hammer, or the position of an object and its desired location for semantic pick-place (Fig. 2). The VLM then identifies 2D keypoints u1)), ..... (k)) in the image / that are relevant to completing the task described via text L. The VLM is prompted with useful keypoints for the task. The full prompts used are included in Appendix H. Since the VLM operates in the 2D image plane, we lift 2D keypoints u to 3D world coordinates x using depth information.
Trajectory Generation. Second, provided the text description 1, the VLM generates waypoint sequences of length < T for each of the initial 3D keypoints x x)),....()))) and the wrist position

---

Training
High-level Policy
Low-level Policy Te
1.20 Keypoirts
2. Plan Generation
Environment (wth leyp trackigg
PPO
boand
cuulo with karpelins
Residual Policy
2i apple 3D Proj.
State
Au
Image
Tracking
2,'4" -
(Optional Few-Shot Feedbback)
b) Inference
Krypoines
Metion Plan
Closed-Loop Loop Execution
200l bourd
Keypoini Model
240 apple
x²
Unseen conditions
Figure 2:
Training: high-level VLM predicts 3D keypoint plans, which a low-level policy learns to track via plans generated by the VLM. which executed by the frozen low-level policy.
RL. b) Inference:
W1. In total, this results in (k 1) 3D waypoints which will serve the basis of the plan т. We include the full prompts used Appendix H. While the first keypoint detection stage depends on the VLM's image understanding, this phase depends more on spatial understanding and reasoning the VLM must translate semantic descriptions into motions, e.g., what "hammering" implies or how door opens, while respecting the physical constraints between keypoints and proximity between the hand and manipulated objects. Note that we do not have the VLM produce keypoint trajectories of the full horizon T, as doing so might be more difficult and inaccurate. Instead, we posit the quality of each waypoint matters more than the number, as low-level RL can compensate for small mistakes in position but not large errors in reasoning.
Interpolation. Finally, though we have coarse waypoint trajectories of length for all keypoints and the agent's wrist pose, directly using these waypoints as motion targets may result in excessively fast or jittery motion. Thus, we additionally apply linear interpolation to convert the n waypoints into sequences of length T. e.g. x x to form the final plan used for training the low-level policy x'.
Few-Shot Improvement. Though VL.M-generated keypoint plans are often correct, they are not infallible. For example, sometimes the high-level policy T" will flip the world-coordinate axes, resulting in unintelligible keypoint plans. Such errors from the VLM plan are irrecoverable if they fail to complete the task. However, the accuracy of VLMs in-context examples [11, 40J. After deploying the final system as described later in Section 3.4, we can use examples of plans successfully executed by the low-level policy as in-context examples for future generations. Provided m successful plans (1), ..... - (mm), we can prompt the high-level policy as n² (T[ss,, experiments, iteratively repeating this procedure can further increase performance as the in-context plans improve.
often be improved by providing
7 ((m)) to produce better plans for the low-level policy. As shown in our
3.3 Low-Level Control with Reinforcement Learning
The low-level policy x' ensures that the keypoint plan 5 provided by n² is effectively tracked. We learn z' using residual reinforcement learning [16, 26], which formalize through "plan" conditioned MDP on top of the low-level observation space O! and action space A with horizon T, We assume the dynamics to be stochastic p(o,_1lor, a,) account for noise in keypoint estimation and that the initial state of pl is always consistent with the high-level plan r to ensure its validity. Naïvely, X' is optimized to maximize the expected cumulative reward provided plans sampled from Th, max, (E--r" (-1o)) BB, (-)( (Et) Fr (of)] where <-177) represents the distribution of full trajectories of length 7' under n' and pimit In this section, we describe how we use the plan t to further guide the learning and exploration n through the reward function, policy parameterization, and environment termination conditions (right half of Fig. 2).
Dense Keypoint Rewards. Standard RL based approaches for dexterous manipulation often require complex, hand- cand-crafted reward functions. However, provided high-level keypoint plan dictating
5

---

how all objects should move and interact, we can simply reward the agent for following the plan via keypoint distances. Though similar ideas have been used for tracking reference demonstrations [6] with ground-truth object poses, we instead track keypoints, which do not require full observability. Our final reward function is given by
r- (oL) exp
3 EI 112 exp (-1/(Neomtaac)) contuee (ot) + c))
(3)
Maintaining Contact
Keypoint Tracking
where the first term a function of the mean Euclidean distance between the planned and observed keypoint positions, and the second term Neomaz (0,) represents the number of finger tips in contact with the environment. Rewarding contact (second term) incentivizes stable grasping, while the first term encourages accurate tracking of the plan. This reward formulation is significantly simpler than those used in previous RL approaches that lack trajectory supervision [39, 60] and can be applied to any task sufficiently captured by keypoint trajectories. We do not reward the policy for tracking the wrist WI.T to allow it to adjust as needed. Instead, we use W1.T in the policy parameterization itself.
Residual Policy. To guide the agent towards the objective specified by the high-level plan T, we employ "residual" RL [16, 26] in the absolute pose action space A. Specifically, the learned low-level policy Tá predicts offsets Aw to the wrist plan w, instead of absolute actions wrrgg Mathematically, this can be written as follows:
= (w: Aw, ). where (Aw, (aTE) (-(o0)
This guarantees that the learned policy follows the plan's wrist trajectory WI.T by default and clipping of Aw ensures never deviates too far. This residual approach uses the world knowledge encoded by the VLM plan to guide exploration of the low-level policy to relevant parts of the state space for completing the objective. Practically, x' is implemented as a multi-layer perceptron where keypoints provided in fixed order and future planning steps TE.T are down-sampled to fixed length.
Termination Conditions. To improve learning efficiency, we terminate episodes early if the tracking error, & [ x(*) ll2, exceeds a threshold 6. This early stopping criterion serves as a strong supervisory signal, encouraging the policy to remain close to the intended trajectory. To further guide learning, we introduce a curriculum: the initial threshold dinit is linearly annealed to 8mni/ 2 over the course of training. This facilitates broad exploration in the early stages while promoting precise trajectory tracking later on. We select task-specific values for oini. provided in Appendix B.3.
3.4 The Full Pipeline
The aforementioned components define the process of generating single plan ~ Tt (-1s1) and using it to learn a low-level residual policy n¹. Here we describe our full training and inference pipeline.
Training. The final low-level policy must be able to perform well across all plans generated by n², which can differ in initial conditions, the selected keypoint locations, and the generated motions. Thus, following the objective from Section 3.3, we train the full system variations in the initial poses of objects the dexterous hand. For each of N initial conditions from the environment, we sample cor- responding high-level plans from n² We then train the low-level policy using PPO [58] by randomly sampling from the set of N initial conditions and plans across massively parallelized simulation environments. In simulation, we track keypoints using ground-truth object information to generate low-level observations o.. Training across randomized plans is crucial for T' to be robust to both the keypoints and plans generated by "h. Further training details and hyperparameters are in Appendix A.
Evaluation. At test time (Fig. 2 b)), we randomize the initial conditions of the environment. Afterwards, we generate new plan using the VLM R² and supply it to the frozen learned policy Th for closed loop control. The high-level policy inherits the robustness of the underlying VLM to visual perturbations, allowing it easily transfer to the real world. We generate 7 using captured RGB-D images, and deploy the low-level policy zero-shot in the real world. We estimate the keypoint positions for low-level observations of using pose estimators [70].
4 Experiments
We conduct a comprehensive suite of experiments to assess the effectiveness, generality, and ro- bustness of our method across diverse range of dexterous manipulation tasks. Our evaluation is

---

structured around four core questions: 1) Are VLM scaffolds effective for broad range of dexterous tasks? How much can iterative refinement improve performance? 3) What causes VLM scaffolds to fail? 4) Can our method successfully learn policies that transfer to the real world?
4.1 Experimental Setup
Task Suite We construct an evaluation suite using the ManiSkill simulator [45, 62] and Allegro Hand model designed to evaluate four core dexterous manipulation capabilities for which motion planning difficult: semantic understanding, ii) unstructured motion, iii) articulated object manipulation, and iv) precise manipulation. Each of the eight tasks, two per category, is depicted in Fig. 3. Instead of reward functions, each task is specified by a language instruction L. For example, the instruction for the "Move Apple" task is "Move the apple to the cutting board". The high-level VLM x² additionally guided by a prompt to detect specified keypoints. Further details can be found in Appendix F. Crucially, the capabilities evaluated by our task set are difficult to design reward functions for (articulated object manipulation or requiring complex and unstructured motion) or are challenging to specify using classical motion planning (requiring semantic knowledge or precision). Methods Given the novelty of our problem setting, there are few applicable baselines which are also aanguag--oonditoonel demonstration-fret and do not require ground-truth state esti- mation. Thus, mainly focus experiments comparison with a variety of oracles and ablations:
Semantic Understanding Apple
Move Bottle
Iterative Keypoint Rewards (IKER): We implement Iter- ative Keypoint Rewards [50], in which VLM generates code specifying reward-fure fnwrtofuncciion parameters. We use the same prompting procedure and setup as in our method for a fair comparison. Since IKER assumes fixed set of keypoints, we adopt the same VLM-identified keypoints used by our system to ensure parity.
Unstructured Motion
Hammer
Wipe with Sponge
Articulated Object Manipulation
Pre-recorded Trajectories: This method reuses pre-recorded trajectories from the training set at test-timer eliminating adapt- ability to new scenarios.
Open Drower
Open Fridge
Oracle Keypoints and Trajectories: This baseline uses fixed, manually defined keypoints and hard-coded trajectories for
Precise Manipulation
each task, representing an upper bound on performance with
Close Selesoes
Close Pliers
perfect semantic understanding and keypoint detection.
Reduced Waypoints: We artificially constrain the VLM to produce shorter waypoint sequences, e.g., length three instead of 20, reducing the complexity of motion that can be expressed via the keypoints and wrist.
Figure depiction the eight tasks used for evaluation. Each task belongs to one of four overarching categorics.
We compare against additional reinforcement learning and imi- tation learning baselines and additionally ablate adding system- atic noise into VLM predictions in Appendix E
We evaluate two versions of our system: zero-shot variant, where the languag model (VLM) receives no example plans, and few-shot variant, where it is provided with three examples of successful plans t in-context (Section 3.2).
Architectures. We use Gemini 2.5 Flash Thinking [63] as the high-level policy with a thinking budget of 1000 tokens for plan generation. A discussion on the use of open-source VLMs is provided in Appendix D.
The low-level policy N! is implemented as 3-layer MLP with hidden dimensions of size 512 and ELU activations [9]. We sample 100 initial states and corresponding plans 7 for training n' with PPO [58].
Evaluation. For evaluation, we construct task-specific binary success metrics (e.g., object reaches target position, door opens minimum angle) to measure performance. All policy evaluations are conducted across 100 initial states with novel object configurations and hand poses. We run 20 trials for each configuration for total of 2000 evaluation episodes and average results across three seeds.

---

Pre recorded
IKER (Zero-Shot)
Ours (Zeoo Shot)
Ours (Few Shot)
Oracle
Succes Rate (%6
100
190
100
Apple
Drawer
Fridge
Sponge
Hammer
Scissors
Pliers
Figure Results on the simulation task suite. Success rate (in %) averaged across seeds and uncertainty given by the standard error. Our method performs nearly as well as the oracle with perfectly scripted plans.
Few shot Scaling
Trojectory Distribution Across Iterations
Sucoess Rato (56
Pridge
Llammee
Seassors
Jocraii
Berstonn
Heroooo
Itoratinn
Figure (I.eft) The performance of our method we iteratively refine the high-level policy s² by providing successful plans T in-context. (Right) The projected 3D plans on the evaluation set for each iteration.
Simulation Results. Fig. 4 shows the success rates for the different simulation tasks. Our method with few-shot adaptation achieves consistently high success rates, with an average success rate of 72%, often approaching the performance of the oracle with perfect scripted plans. This suggests that modern VLMs are capable planners for scaffolding dexterous policies. Moreover, we observe that adding a few successful examples can further improve performance beyond that of zero-shot in six out of eight tasks. The most notable improvement is from the Fridge task, where the VI.M commonly flips the world axes, resulting in implausible plans. Thus, providing examples of successful plans drastically improves performance. Performance of the pre-recorded baseline remains poor for all tasks, except the drawer task, where the novel plans are likely less important at test time due to the sizable width of the drawer handle.
Iterative Refinement. We provide the VLM with successful trajectories from the training set as in-context examples to improve the proposed waypoints. We iterate this process up to three times in Fig. All tasks show an increase the success rate after the first iteration, with diminishing returns after the second iteration. This is likely because plan-generation performance plateaus and errors in other parts of the system dominate. As before, the most significant improvement is in the Fridge task. While VLMs already have solid semantic understanding, few-shot iteration can improve performance and correct common mistakes during inference. After iterative refinement, the overall success rate improves to 81%.
4.2 What Causes VLM Scaffolds To Fail?
Failure Modes. To comprehensively evaluate the failure modes of our pipeline across all tasks, present Sankey diagram in Fig. 6, categoriz- ing errors into three primary sources: (i) incor- rect keypoint detection, where keypoints do not lie on target objects, indicating deficiencies in VLM keypoint detection; (ii) incomplete trajec- tory tracking by the low-level policy, suggesting either inaccuracies in the low-level policy suitable trajectories; and (iii) tracked trajectories that nonetheless fail achieve success, reveal- ing shortcomings in VLM trajectory generation. Notably, (iii) can occur because success criteria are independent of tracking, allowing fully tracked trajectories to fail. Our analysis reveals that the most significant failure mode is incomplete trajectory tracking, occurring 26% of the rollouts. This suggests a critical bottleneck in the low-level policy's 'ss ability to follow the planned path, though it remains unclear whether this stems from unrealistic trajectory proposals or intrinsic policy limitations. In contrast, keypoint detection errors account for 5% of failures, while unsuccessful but fully tracked trajectories contribute 3% (not shown in diagram). Interestingly, some partially tracked trajectories still succeed, implying that perfect tracking is not always required. These findings highlight the importance of improving both trajectory generation and
Coeplec Trcking
60%
Spccan 72%
Comua Respviaa
Incoeplle Touckcn
24%
28%
Incener Kopekie
Figure 6: Error decomposition across failure cases. Most errors stem from incomplete trajectory tracking. followed by keypoint detection issues.
8

---

Waypoint S Sealing
Ours (Zeco-Shol)
Traj. Oracle
Keypoini Orscle
Succusa Rato (N)
8 3. 1.
Apple
Sponge
Piless
Succesn (90
500
Num Waypoints
Num Waypoints
Num Waypolats
Num Waypelats
Apple
Drawer
Sponge
Phers
Figure 7: (Left) Task success vs. number of waypoints in VLM plans. Most tasks saturate by 10 waypoints; only the hammer task benefits from denser trajectories. (Right) Ablation of VLM components. Replacing keypoints or trajectories with oracles highlights their relative impact across tasks.
tracking robustness. However, further automated decomposition of these errors remains challenging due to the complex interplay between high-level planning and low-level execution.
Number of Waypoints. Fig. (Left), we evaluate the performance of our method using 3, 5, 10, 20, and 40 waypoints for plan generation. The results show that planning fidelity is typically not a large source of error, unless very few (3 or 5) waypoints are used. saturates with 10 waypoints. Only the hammer task requires 20, as it needs repeated motion.
of 4 tasks, performance
VLM Components. To ablate the impact of using a VLM for keypoint detection and plan generation, we replace each component with an oracle in Fig. (Right). For the Keypoint oracle, we use hand- specified keypoints for generating т. For the Traj. oracle, we use VLM keypoints but script plans for The resulting improvements vary across tasks: in the drawer task, the Traj. oracle achieves near perfect performance indicating planning was the bottleneck, however, the sponge task the keypoint oracle improves performance the most, indicating that x most commonly mis-specifies keypoints.
4.3 Real-World Results
To evaluate sim-to-real transfer, we deploy our sys- tem on real robot using the same inference pipeline as in simulation. From single real-world RGB-D image and natural-language command, the vision- language planner generates wrist and keypoint tra- jectories. The low-level policy is trained entirely simulation using digital twin of the real-world environment, and then executed in the real-world, conditioned on the generated trajectories. Robustness to visual distractors is achieved by training the policy on state-based observations, decoupling it from raw visual input. Robustness to discrepancies in physical parameters is achieved through domain randomiza- tion in simulation. Our real-world experiments performed using a 16-DoF Allegro hand mounted 7-DoF KUKA LBR iiwa 14 arm in tabletop setting, and a ZED stereo camera rigidly mounted to the table.
Bottle Plate (90%)
Slide Box to Bottle (85%)
Hammer Three Times (65%)
Figure Real-world rollouts of Place Bottle onto Plate, Slide Box to Bottle and Hammer Three Times.
We evaluate three real-world manipulation tasks: Place Bottle onto Plate, Slide Box to Bottle, and Hammer Three Times, covering semantic placement, non- non- prehensile pushing, and complex motion planning (Fig. 8). Each task is executed for 20 rollouts. Our system achieves 90% success rate Place Bottle onto Plate, 85% on Slide Box to Bottle, and 65% on Hammer Three Times. These results indicate that the proposed modular, trajectory- based framework generalizes effectively from simulation to the real world, maintaining robust performance across a range of manipulation settings. Additional implementation and hardware details provided in Appendix C.
5 Conclusion
We presented new framework for dexterous robotic manipulation that combines VLMs with reinforcement learning to generate and execute semantically meaningful hand-object trajectories. By casting manipulation trajecng problem using VLM-generated keypoint plans,

---

our method eliminates the need for human demonstrations or handcrafted reward functions, while enabling generalization across diverse objects, goals, and scene configurations.
Our experiments in both simulation and the real world show that this approach reliably solves a variety of complex manipulation tasks, including articulated objects, semantic reasoning, and fine finger control. The system exhibits strong generalization to novel keypoints and configurations, and transfers effectively to physical hardware without additional tuning or data collection.
Limitations and Future Work. While our method demonstrates strong performance across a range of tasks, several limitations remain along with exciting directions for future work. First, our current approach assumes rigid-body interactions, which simplifies keypoint tracking but limits applicability to deformable objects. Extending to more complex, deformable objects would require the ability to track keypoints on non-rigid surfaces (e.g., point tracking models [27]). Additionally, while our approach can implicit infer orientation from three non-colinear keypoints, this is less effective for smaller objects with fewer keypoints. We see potential in exploring richer scaffold representations and more advanced VLMs to better capture orientation in these cases. The high-level trajectory generation in our system currently does not account for the low-level controller's 's capabilities, which limits adaptability. Integrating feedback from the low-level controller into the VLM planner to form a more closed-loop system offers an exciting opportunity for improvement. Additionally, the current reasoning time of 1-2 minutes for VLMs restricts real-time responsiveness. motivating a shift towards faster, more efficient models. While our scaffold- based reward design is lightweight and scalable, we see opportunities to combine it with additional task-specific signals, such force or torque constraints, to enhance performance in more complex tasks. Although the system shows strong zero-shot generalization across various object configurations, generalizing to new tasks or object categories remains a challenge. We believe combining scaffold- reasoning with object- agnostic representations could unlock broader adaptability. Lastly, sim-to-real transfer remains an ongoing challenge, especially for tasks requiring high-precision control. We see exciting directions for improving simulation realism, multi-view perception, and sensor fusion to narrow the sim-to-real gap, particularly for fineggrained tasks such manipulation.
Acknowledgments
We would like to thank Marcel Torne Villasevil, Michelle Pan, Shuang Li, Max Du, Amber Xie, Suvir Mirchandani, and Priya Sundaresan for their important feedback the paper draft. This work is supported by the National Science Foundation under Grant Numbers 2342246, 2006388, 2132847 and 2218760, and by the Natural Sciences and Engineering Research Council of Canada (NSERC) under Award Number 526541680. This work is additionally funded by the German Research Foundation (DFG, German Research Foundation) SSFB-1574 471687386, from the pilot program Core Informatics of the Helmholtz Association (HGF), and supported by the European Research Council (ERC) under the European Union's 's Horizon Europe programme through the project
SMARTI (Grant Agreement No. 101171393)
References
[1] Ananye Agarwal, Ashish Kumar, Jitendra Malik, and Deepak Pathak. Legged locomotion in challenging terrains using egocentric vision. Karen Liu, Dana Kulic, and Jeff Ichnowski, editors, Proceedings The 6th Conference on Robot Learning, volume 205 of Proceedings of Machine Learning Research, pages 403-415. PMIR, 14-18 Dec 2023.
[2] Shan An, Ziyu Meng, Chao Tang, Yuning Zhou, Tengyu Liu, Fangqiang Ding, Shufang Zhang, Yao Mu, Ran Song, Wei Zhang, Zeng-G Guang Hou, and Hong Zhang. Dexterous manipulation through imitation learning: A survey, 2025.
[3] OpenAI: Marcin Andrychowicz, Bowen Baker, Maciek Chociej. Rafal Jozefowicz, Bob Mc- Grew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3-20 2020.
[4] Xue Bin Peng, Erwin Coumans, Tingnan Zhang, Tsang-Wei Lee, Jie Tan, and Sergey Levine. Learning agile robotic locomotion skills by imitating animals. Robotics: Science and Systems (RSS), Virtual Event/Corvalis, July, pages 12-16, 2020.
10

---

[5] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan. Karol Hausman, Alex Herzog, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Kuang-Huei Lee, Sergey Levine, Yao Lu, Utsav Malla, Deeksha Manjunath, Igor Mordatch, Ofrr Nachum, Carolina Parada, Jodilyn Peralta, Emily Perez, Karl Pertsch, Jornell Quiambao, Kanishka Rao, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Kevin Sayed, Jaspiar Singh, Sumedh Sontakke, Austin Stone, Clayton Tan, Huong Tran, Vincent Vanhoucke, Steve Vega, Quan Vuong. Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. Rt-1: Roboties transformer for real-world control at scale. In arXiv preprint arXiv:2212.068// 2022.
[6] Yuanpei Chen, Chen Wang, Yaodong Yang, and Karen Liu. Object-centric dexterous manipula- tion from human motion data. In 8th Annual Conference on Robot Learning, 2024.
[7] Yuanpei Chen, Tianhao Wu, Shengjie Wang. Xidong Feng, Jiechuan Jiang, Zongqing Lu, Stephen McAleer, Hao Dong, Song-Chun Zhu, and Yaodong Yang. Towards human-level bimanual dexterous manipulation with reinforcement learning. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Process- ing Systems, volume 35, pages 5150-5161 Curran Associates, Inc., 2022.
[8] Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, page 02783649241273666, 2023.
[9] Djork-Amé Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). arxiv 2015. arXiv preprint arXiv:1511.07200 10, 2020.
[10] Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salchi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. Molmo and pixmo: Open weights and open data for state-of-the-ae vision-language models. In Proceedings of the Computer Vision and Puttern Recognition Conference, pages 91-104, 2025.
[11] Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, et al. A survey on in-context learning. arXiv preprint arXiv:2301.00234 2022.
[12] Yuqing Du, Ksenia Konyushkova, Misha Denil, Akhil Raju, Jessica Landon, Felix Hill, Nando de Freitas, and Serkan Cabi. Vision-language models success detectors. arXiv preprint arXiv:2303.07280, 2023.
[13] Zicong Fan, Omid Taheri, Dimitrios Tzionas, Muhammed Kocabas, Manuel Kaufmann, Michael J. Black, and Otmar Hilliges. ARCTIC: dataset for dexterous bimanual hand-object manipulation. In Proceedings IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[14] Kuan Fang, Fangchen Liu, Pieter Abbeel, and Sergey Levine. Moka: manipulation through mark-based visual prompting. Robotics: Science and Systems (RSS), 2024.
Open-world robotic
[15] Irmak Guzey, Yinlong Dai, Georgy Savva, Raunaq Bhirangi, and Lerrel Pinto. Bridging the human to robot dexterity gap through object-oriented rewards. arXiv preprint arXiv:2410.23289, 2024.
[16] Siddhant Haldar, Jyothish Pari, Anant Rai, and Lerrel Pinto. Teach a robot to fish: Versatile imitation from one minute of demonstrations. Robotics: Science and Systems, 2023.
[17] Siddhant Haldar and Lerrel Pinto. Point policy: Unifying observations and actions with key points for robot manipulation. arXiv preprint arXiv:2502.203311 2025.
[18] Ankur Handa, Arthur Allshire, Viktor Makoviychuk, Aleksei Petrenko, Ritvik Singh, Jingzhou Liu, Denys Makoviichuk, Karl Van Wyk, Alexander Zhurkevich, Balakumar Sundaralingam, et al. Dextreme: Transfer of agile in-hand manipulation from simulation to reality. In 2023
11

---

IEEE International Conference on Robotics and Automation (ICRA), pages 5977-5984, IEEE, 2023.
[19] Ankur Handa, Karl Van Wyk. Wei Yang, Jacky Liang, Yu-Wei Chao, Qian Wan, Stan Birchfield, Nathan Ratliff, and Dieter Fox. Dexpilot: Vision-bases teleoperation of dexterous robotic hand-arm system. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 9164 9170. IEEE, 2020.
[20] Todd Hester, Matej Vecerik, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, al. Deep q-learning from demonstrations. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
[21] Hengyuan Hu, Suvir Mirchandani, and Dorsa Sadigh. Imitation Bootstrapped Reinforcement Learning. In Proceedings of Robotics: Science and Systems, Delft, Netherlands, July 2024.
[22] Wenlong Huang, Chen Wang, Yunzhu Li, Ruohan Zhang, and Li Fei-Fei. Rekep: Spatio- temporal reasoning of relational keypoint constraints for robotic manipulation. arXiv preprint arXiv:2409.01652 2024.
[23] Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models. In Conference on Robot Learning, pages 540-562. PMLR, 2023.
[24] Minyoung Hwang, Jocy Hejna, Dorsa Sadigh, and Yonatan Bisk. Motif: Motion instruction fine-tuning. 2024.
[25] Aadhithya Iyer, Zhuoran Peng, Yinlong Dai, Irmak Guzey, Siddhant Haldar, Soumith Chintala, and Lerrel Pinto. OPEN TEACH: A versatile teleoperation system for robotic manipulation. In 8th Annual Conference on Robot Learning, 2024.
[26] Tobias Johannink, Shikhar Bahl, Ashvin Nair, Jianlan Luo, Avinash Kumar, Matthias Loskyll, Juan Aparicio Ojea, Eugen Solowjow, and Sergey Levine. Residual reinforcement learning for robot control. In 2019 international conference on robotics and automation (ICRA), pages 6023 6029. IEEE, 2019.
[27] Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Cotracker: It is better to track together. In European Conference on Computer Vision, pages 18-35. Springer, 2024.
[28] Martin Klissarov, Pierluca D'Oro, Shagun Sodhani, Roberta Raileanu, Pierre-Luc Bacon, Pascal Vincent, Amy Zhang, and Mikael Henaff. Motif: Intrinsic motivation from artificial intelligence feedback. arXiv preprint arXiv:2310.00/66
2023.
[29] Ashish Kumar, Zipeng Fu, Deepak Pathak, and Jitendra Malik. RMA: rapid motor adaptation for legged robots. CoRR, 0ttss/107.040333 2021.
[30] Minae Kwon, Sang Michael Xie, Kalesha Bullard, and Dorsa Sadigh. Reward design with language models. In The Eleventh International Conference on Learning Representations, 2023.
[31] Teyun Kwon, Norman Di Palo, and Edward Johns. Language models as zero-shot trajectory generators. IEEE Robotics and Automation Letters, 9())6728-6735, 2024.
[32] Marion Lepert, Jiaying Fang, and Jeannette Bohg. Phantom: Training robots without robots using only human videos, 2025.
[33] Kailin Li, Puhao Li, Tengyu Liu, Yuyang Li, and Siyuan Huang. Maniptrans: Efficient dexterous bimanual manipulation transfer via residual learning. arXiv preprint arXiv:2503.21888 2025.
[34] Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 9493-9500. IEEE, 2023.
[35] Toru Lin, Zhao-Heng Yin, Haozhi Qi, Pieter Abheel, and Jitendra Malik. Twisting lids off with two hands. arXiv preprint arXiv:2403.02338 2024.
12

---

[36] Haichao Liu, Sikai Guo, Pengfei Mai, Jiahang Cao, Haoang Li, and Jun Ma. Robodexvim: Vi- sual language model-enabled task planning and motion control for dexterous robot manipulation. arXiv preprini urXiv:2503.01616, 2025.
[37] Xueyi Liu, Jianibieke Adalibieke, Qianwei Han, Yuzhe Qin, and Li Yi. Dextrack: Towards generalizable neural tracking control for dexterous manipulation from human references. arXiv preprint rrXvv:2502.09614 2025.
[38] Tyler Ga Wei Lum, Olivia Y. Lee, C. Karen Liu, and Jeannette Bohg. Crossing the human-robot embodiment gap with sim-to-real rl using one human demonstration, 2025.
[39] Tyler Ga Wei Lum, Martin Matak, Viktor Makoviychuk, Ankur Handa, Arthur Allshire, Tucker Hermans, Nathan D. Ratliff, and Karl Van Wyk. DextrAH-g: Pixels-to-action dexterous arm-hand uum-hand grasping with geometric fabrics. In 8th Annual Conference on Robot Learning, 2024.
[40] Yecheng Jason Ma, Joey Hejna, Chuyuan Fu, Dhruv Shah, Jacky Liang, Zhuo Xu, Sean Kirmani, Peng Xu, Danny Driess, Ted Xiao, et al. Vision language models are in-context value learners. In The Thirteenth International Conference on Learning Representations, 2024.
[41] Yecheng Jason Ma. William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Eureka: Human-level reward design via coding large language models. arXiv preprint arXiv: 10. /2931, 2023.
[42] Parsa Mahmoudieh, Deepak Pathak, and Trevor Darrell. Zero-shot reward specification via grounded natural language. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepes- vari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 14743-14752. PMLR, 17-23 Jul 2022.
[43] Priyanka Mandikal and Kristen Grauman. Dexvip: Learning dexterous grasping with human hand pose priors from video. In Conference on Robot Learning (CoRI.), 2021.
[44] Suvir Mirchandani, Fei Xia, Pete Florence, brian ichter, Danny Driess, Montserrat Gonzalez Arenas, Kanishka Rao, Dorsa Sadigh, and Andy Zeng. Large language models as general pattern machines. In 7th Anmual Conference on Robot Learning, 2023.
[45] Tongzhou Mu, Zhan Ling, Fanbo Xiang, Derek Cathera Yang, Xuanlin Li, Stone Tao, Zhiao Huang, Zhiwei Jia, and Hao Su. Maniskill: Generalizable manipulation skill benchmark with large-scale demonstrations. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
[46] Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojeiech Zaremba, and Pieter Abbeel. Over- coming exploration in reinforcement learning with demonstrations. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 6292-6299, 2018.
[47] Soroush Nasiriany, Abhiram Maddukuri, Lance Zhang, Adect Parikh, Aaron Lo. Abhishek Joshi, Ajay Mandlekar, and Yuke Zhu. Robocasa: Large-scale simulation of everyday tasks for generalist robots. arXiv preprint arXiv:2406.0253: 2024.
[48] Soroush Nasiriany, Fei Xia, Wenhao Yu, Ted Xiao, Jacky Liang, Ishita Dasgupta, Annie Xie, Danny Driess, Ayzaan Wahid, Zhuo Xu, et al. Pivot: Iterative visual prompting elicits actionable knowledge for vlms. In International Conference on Machine Learning, pages PMLR, 2024.
[49] Open X- E-Embodiment Collaboration, Abhishek Padalkar, Acorn Pooley, Ajinkya Jain, Alex Bewley, Alex Herzog, Alex Irpan, Alexander Khazatsky, Anant Rai, Anikait Singh, Anthony Brohan, Antonin Raffin, Ayzaan Wahid, Ben Burgess- Limerrik, Beomjoon Kim, Bernhard Schölkopf. Brian Ichter, Cewu Lu, Charles Xu, Chelsea Finn, Chenfeng Xu, Cheng Chi, Chenguang Huang, Christine Chan, Chuer Pan, Chuyuan Fu, Coline Devin, Danny Driess, Deepak Pathak, Dhruv Shah, Dieter Büchler, Dmitry Kalashnikov, Dorsa Sadigh, Edward Johns, Federico Ceola, Fei Xia, Freek Stulp, Gaoyue Zhou, Gaurav S. Sukhatme, Gautam Salhotra, Ge Yan, Giulio Schiavi, Hao Su, Hao-Shu Fang, Haochen Shi, Heni Ben Amor, Henrik I Christensen, Hiroki Furuta, Homer Walke, Hongjie Fang. Igor Mordatch, Iljja Radosavovie,
13

---

Isabel Leal, Jacky Liang, Jachyung Kim, Jan Schneider, Jasmine Hsu, Jeannette Bohg, Jeffrey Bingham, Jiajun Wu, Jialin Wu, Jianlan Luo, Jiayuan Gu, Jie Tan, Jihoon Oh, Jitendra Malik, Jonathan Tompson, Jonathan Yang, Joseph J. Lim, João Silvério, Junhyek Han, Kanishka Rao, Karl Pertsch, Karol Hausman, Keegan Go, Keerthana Gopalakrishnan, Ken Goldberg, Kendra Byrne, Kenneth Oslund, Kento Kawaharazuka, Kevin Zhang, Keyvan Majd, Krishan Rana, Krishnan Srinivasan, Lawrence Yunliang Chen, Lerrel Pinto, Liam Tan, Lionel Out, Lisa Lee, Masayoshi Tomizuka, Maximilian Du, Michael Ahn, Mingtong Zhang, Mingyu Ding, Mohan Kumar Srirama, Mohit Sharma, Moo Jin Kim, Naoaki Kanazawa, Nicklas Hansen, Nicolas Heess, Nikhil J Joshi, Niko Suenderhauf, Norman Di Palo, Nur Muhammad Mahi Shafiullah, Oier Mees, Oliver Kroemer, Pannag R Sanketi, Paul Wohlhart, Peng Xu, Pierre Sermanet, Priya Sundaresan, Quan Vuong. Rafael Rafailov, Ran Tian, Ria Doshi, Roberto Martin-Martín, Russell Mendonca, Rutav Shah, Ryan Hoque, Ryan Julian, Samuel Bustamante, Sean Kirmani, Sergey Levine, Sherry Moore, Shikhar Bahl, Shivin Dass, Shuran Song, Sichun Xu, Siddhant Haldar, Simeon Adebola, Simon Guist, Soroush Nasiriany, Stefan Schaal, Stefan Welker, Stephen Tian, Sudeep Dasari, Suneel Belkhale, Takayuki Osa, Tatsuya Harada, Tatsuya Matsushima, Ted Xiao, Tianhe Yu, Tianli Ding, Todor Davchev, Tony Z. Zhao, Travis Armstrong, Trevor Darrell, Vidhi Jain, Vincent Vanhoucke, Wei Zhan, Wenxuan Zhou, Wolfram Burgard, Xi Chen, Xiaolong Wang, Xinghao Zhu, Xuanlin Li, Yao Lu, Yevgen Chebotar, Yifan Zhou, Yifeng Zhu, Ying Xu, Yixuan Wang, Yonatan Bisk, Yoonyoung Cho, Youngwoon Lee, Yuchen Cui, Yueh hua Wu, Yujin Tang. Yuke Zhu, Yunzhu Li, Yusuke Iwasawa, Yutaka Matsuo, Zhuo Xu, and Zichen Jeff Cui. Open X- X-Embodiment: Robotic learning datasets and RT-X models, 2023.
[50] Shivansh Patel, Xinchen Yin, Wenlong Huang, Shubham Garg, Hooshang Nayyeri, Li Fei-Fei, Svetlana Lazebnik, and Yunzhu Li. A real-to-sim-to-rean approach to robotic manipulation with ulm-generated iterative keypoint rewards. arXiv preprint arXiv:2502.
2025.
[51] Xue Bin Peng. Glen Berseth, KangKang Yin, and Michiel Van De Panne. Deeploco: Dynamic locomotion skills using hierarchical deep reinforcement learning. Acm transactions on graphics (tog), 3664))1-113, 2017.
[52] Haozhi Qi, Ashish Kumar, Roberto Calandra, Yi Ma, and Jitendra Malik. In-Hand Object Rotation via Rapid Motor Adaptation. In Conference on Robot Learning (CoRL), 2022.
[53] Yuzhe Qin, Hao Su, and Xiaolong Wang. From one hand to multiple hands: Imitation learning for dexterous manipulation from single-camera teleoperation. IEEE Robotics and Automation Letters, 1(4): 10881, 2022.
[54] Yuzhe Qin, Wei Yang, Binghao Huang, Karl Van Wyk, Hao Su, Xiaolong Wang, Yu-Wei Chao, and Dieter Fox. Anyteleop: A general vision-based dexterous robot arm-hand teleoperation system. In Robotics: Science and Systems, 2023.
[55] Ilija Radosavovic, Xiaolong Wang, Lerrel Pinto, and Jitendra Malik. State-only imitation learning for dexterous manipulation. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7865-7871, 2021.
[56] Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. In Proceedings of Robotics: Science and Systems, Pittsburgh. Pennsylvania, June 2018
[57] Juntao Ren, Priya Sundaresan, Dorsa Sadigh, Sanjiban Choudhury, and Jeannette Bohg. Motion tracks: A unified representation for human-robot transfer in few-shot imitation learning, 2025.
[58] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal 2017.
policy optimization algorithms. arXiv preprint /
[59] Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. Progprompt: Generating situated robot task plans using large language models. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 11523-115533 IEEE, 2023.
14

---

[60] Ritvik Singh, Arthur Allshire, Ankur Handa, Nathan Ratlif,, and Karl Van Wyk. Dextrah-rgb: Visuomotor policies to grasp anything with dexterous hands, arXiv preprint arXiv:2412.01991 2024.
[61] Andrew Szot, Alex Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Tumnr,, Noah Maestre, Mustafa Mukadam. Devendra Chaplot, Oleksandr Maksymets. Aaron Gokaslan, Vladimir Vondrus, Sameer Dharur, Franziska Meier, Wojeiech Galuba, Angel Chang, 7solt Kira, Vladlen Koltun, Jitendra Malik, Manolis Savva, and Dhruv Batra. Habitat 2.0: Training home assistants to rearrange their habitat. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
[62] Stone Tao, Fanbo Xiang, Arth Shukla, Yuzhe Qin, Xander Hinrichsen, Xiaodi Yuan, Chen Bao, Xinsong Lin, Yulin Liu, Tse-kai Chan, et al. Maniskill3: Gpu parallelized robotics simulation and rendering for generalizable embodied ai. arXiv preprint arXiv:2440.008. 00.450 2024.
[63] Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- -aan-Baptsste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312. 1805, 2023.
[64] Gemini Robotics Team, Saminda Abeyruwan, Joshua Ainslie, Jean- nan-Baptiste Alayrac, Montser- rat Gonzalez Arenas, Travis Armstrong, Ashwin Balakrishna, Robert Baruch, Maria Bauza, Michiel Blokzijl, et al. Gemini robotics: Bringing ai into the physical world. arXiv preprint arXivv2503..20020 2025.
[65] David Venuto, Mohammad Sami Nur Islam, Martin Klissarov, Doina Precup, Sherry Yang, and Ankit Anand. Code as reward: Empowering reinforcement learning with VLMs. In Forty-first International Conference on Machine Learning. 2024.
[66] Chen Wang, Haochen Shi, Weizhuo Wang, Ruohan Zhang, Li Fei-Fei, and C Karen Liu, Dexcap: Scalable and portable mocap data collection system for dexterous manipulation. arXiv preprint arXivv2403.07788 2024.
[67] Jun Wang, Ying Yuan, Haichuan Che, Haozhi Qi. Yi Ma, Jitendra Malik, and Xiaolong Wang. Lessons from learning to spin "pens". ". In CoRL, 2024.
[68] Qianxu Wang, Congyue Deng, Tyler Ga Wei Lum, Yuanpei Chen, Yaodong Yang, Jeannette Bohg, Yixin Zhu, and Leonidas Guibas. Neural attention field: Emerging point relevance in 3d scenes for ohe-shot dexterous grasping. arXiv preprini arXiv:2410.23039, 2024.
[69] Yufei Wang, Zhanyi Sun. Jesse Zhang, Zhou Xian, Erdem Biyik, David Held, and Zackory Erickson. RL-VLM-f: Reinforcement learning from vision language foundation model feedback. In Forty-first International Conference on Machine Learning, 2024.
[70] Bowen Wen, Wei Yang, Jan Kautz, and Stan Birchfield. Foundationpose: Unified 6d pose estimation and tracking of novel objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 77868-17877 2024.
[71] Chuan Wen, Xingyu Lin, John Ian Reyes So, Kai Chen, Qi Dou, Yang Gao, and Pieter Abbeel. Any-poine Trajectory Modeling for Policy Learning. In Proceedings of Robotics: Science and Systems, Delft, Netherlands, July 2024.
[72] Tianbao Xie, Siheng Zhao, Chen Henry Wu, Yitao Liu, Qian Luo, Victor Zhong, Yanchao Yang. and Tao Yu. Text2reward: Reward shaping with language models for reinforcement learning. arXiv preprint 2rXiv:2309.11489 2023.
[73] Mengda Xu, Zhenjia Xu, Yinghao Xu, Cheng Chi, Gordon Wetzstein, Manuela Veloso, and Shuran Song. Flow as the cross-domain manipulation interface. In 8th Annual Conference on Robot Learning. 2024.
[74] Fan Yang, Dongsheng Luo, Wenrui Chen, Jiacheng Lin, Junjie Cai, Kailun Yang, Zhiyong Li, and Yaonan Wang. Multi- Multi-keypoint affordance representation for functional dexterous grasping, 2025.
15

---

[75] Jingyun Yang, Max Sobol Mark, Brandon Vu, Archit Sharma, Jeannette Bohg, and Chelsea Finn. Robot fine-tuning made easy: Pre-training rewards and policies for autonomous real-world reinforcement learning, 2023.
[76] Lixin Yang. Kailin Li, Xinyu Zhan, Fei Wu, Anran Xu, Liu Liu, and Cewu Lu, OakInk: A large-scale knowledge repository for understanding hand-object interaction. In IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2022.
[77] Weirui Ye, Yunsheng Zhang, Haoyang Weng, Xianfan Gu, Shengjie Wang, Tong Zhang, Mengchen Wang, Pieter Abbeel, and Yang Gao. Reinforcement learning with foundation priors: Let embodied agent efficiently learn its own. In 8th Annual Conference Robot Learning.
[78] Zhao-Heng Yin, Changhao Wang. Luis Pineda, Krishna Bodduluri, Tingfan Wu, Pieter Abbeel, and Mustafa Mukadam. Geometric retargeting: A principled, ultrafast neural hand retargeting algorithm. XrXiv preprint arXiv:2503.07541, 2025.
[79] Zhao-Heng Yin, Changhao Wang, Luis Pineda, Francois Hogan, Krishna Bodduluri, Akash Sharma, Patrick Lancaster, Ishita Prasad, Mrinal Kalakrishnan, Jitendra Malik, et al. Dexteri- tygen: Foundation controller for unprecedented dexterity, arXiv preprint arXiv:2502.0433 2025.
[80] Wenhao Yu, Nimrod Gileadi, Chuyuan Fu, Sean Kirmani, Kuang-Huei Lee, Montserrat Gonza- lez Arenas, Hao-Tien Lewis Chiang, Tom Erez, Leonard Hasenclever, Jan Humplik, brian ichter, Ted Xiao, Peng Xu, Andy Zeng, Tingnan Zhang, Nicolas Heess, Dorsa Sadigh, Jie Tan, Yuval Tassa, and Fei Xia. Language to rewards for robotic skill synthesis. In 7th Annual Conference on Robot Learning, 2023.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our method performs better than baselines and closer to oracle performance. We do not claim any theoretical results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We provide thorough failure analysis in the results section and limitations in the conclusion.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
17

---

Answer: [NA]
Justification:
The paper does not include any theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
The method is fully explained with details in the appendix. The prompts used are also provided. We believe this provides sufficient detail reproduce the results, up to but excluding the real world experiments which are impossible to exactly recreate in different lab.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
18

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification: We will provide code for simulated experiments in the the supplementary materials, though instructions may be lacking due to time limitations. We working improving this.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips) cc/
http://uddeee/eeeuuubiiieeeeeeee cisssoiiccyyl for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to rum to reproduce the results. See the NeurIPS code and data submission guidelines (https:
//n /nips. /ttcc///pbbc. cec/public//ae weeiic/gggee//eeeeeeeeeeeeeeeeee /ot///eooomisssseeeeeee wwmiisssoonpii aessonPooiice for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We include hyperparameters in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We run three seeds for all experiments and visually depict error bars. If they do not appear in the charts it is because they are too small to appear.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
19

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, we provide compute resources in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: : /neuriiss....// cc
Answer: [Yes]
Justification: Yes, we have reviewed the code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Dexterous manipulation too broadly in its infancy to have a direct socictal impact at present.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
20

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper does not include release large models or datasets.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Yes, only open-source and permissible license environments which cite.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
21

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not introduce new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do not conduct experiments with human subjets.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
We do not use crowdsoruce or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
22

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: Yes, we include descriptions of how we use VLMs in our work, including the full prompts.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https: http://eeenucceeeeeeeee / for what should or should not be described.
23

---

A Hyperparameters
A.1
PPO
The hyperparameters of our PPO training are detailed in Table 1.
Table 1: PPO Hyperparameters
Hyperparameter
Value
Normalize Advantage per
True
Value Loss Coefficient
1.0
Clip Parameter
0.2
Use Clipped Value Loss
True
Desired KL.
0.01
Entropy Coefficient
0.01
Discount Factor (Gamma)
0.99
GAE Lambda (Lam)
0.95
Max Gradient Norm
1.0
Learning Rate
0.0003
Number of Learning Epochs
5
Number Mini-Batches
16
Schedule
Adaptive
Policy Class Name
ActorCritic
Activation Function
ELU [512, 512, 512] [512, 512, 512]
Actor Hidden Dimensions
Critic Hidden Dimensions
Initial Noise Std
1.0
Noise Std Type
Scalar
Number of Steps per Environment
24 2000
Max Iterations
Empirical Normalization
True
Number Environments
2048
A.2 Simulation
We Maniskil13 [62] for simulations. Our hyperparameters listed in Table We situate our tasks in simulated scenes from the ReplicaCAD dataset [61]. Some of the objects in the tasks are from the RoboCasa project [47].
A.3 VLM
We detail our query settings in Table
B
Environment
B.1
Observation Space
Table 4 details the components of our observation space. Importantly, our policy does not rely on privileged information such as contact forces during training, making the observation space more amenable to real-world deployment.
B.2
Action Space
We use residual action space the wrist pose, and directly control the fingers. We normalized the action space the range [-1, 1]. "zero" action corresponds following the reference trajectory precisely with an entirely open hand.
24

---

Table 2: Simulation and Control Settings
Setting
Value
Action Exponential Average Gamma
0.9
Simulation Frequency
120 Hz
Control Frequency
60 Hz
Max Rigid Contact Count
2048 x 2048 x 8
Max Rigid Patch Count
2048 x 2048 2
Found Lost Pairs Capacity
2
Gravity
[0. 0, -9.81]
Bounce Threshold
2.0
Solver Position Iterations
Solver Velocity Iterations
0
Default Dynamic Friction
1.0
Default Static Friction
1.0
Restitution
Finger Static Friction
2.0
Dummy Joint Stiffness
2000
Dummy Joint Damping
100
Dummy Joint Force Limits
1000
Finger Joint Stiffness
10
Finger Joint Damping
0.3
Finger Joint Force Limit
10
Controller Type
PD Joint Targets
Table 3: VLM Configuration
Hyperparameter
Value
Image Size
800 800
Trajectory Query Thinking Budget
1000
Keypoint Query Temperature
Trajectory Query Code Execution Enabled
We only control the fingers individually for the scissors and pliers tasks since we did not see any benefit for the other tasks. For the scissors and pliers task we have one action for every finger (instead of every joint). This makes hand action space four dimensional for the allegro hand. For all other tasks we control all fingers with one action, only opening or closing entire hand.
B.3 Termination Thresholds
We detail our initial termination thresholds per task Table 5. The initial thresholds are linearly reduced half of their initial value over the course training.
C Hardware Experiment Details
C.1 Inference-Tine Time Pipeline Details
At inference-time,
policy in the real-world as follows:
1. Capture RGB-D D image of the scene.
2. Query the VLM for 2D keypoints, given the RGB image and natural language command.
3. Backproject the 2D keypoints into 3D keypoints using the depth camera and camera intrin- sics.
4. Transform the 3D keypoints from camera frame to world frame using camera extrinsics.
5. Query the VLM for a wrist pose trajectory and keypoint trajectories, given the initial wrist pose and 3D keypoints.
25

---

Table 4: Observation Space Configuration
Observation Type
Dimension
Joint Position (Dummy Joints Fingers)
22
Joint Velocity (Dummy Joints + Fingers)
22
Exponential Average Action
22
Finger Poses
4 x = 28
Initial Keypoint Positions
3 X k
Current Keypoint Positions
3 x k
Planned Future Keypoint Positions
3 x 15 x k
Current Wrist Pose
6
Planned Future Wrist pose
x 15 30
Total
130 + x
Table 5: Initial Termination Thresholds for Manipulation Tasks
Object
Threshold (cm)
Apple 10
Bottle 10
Hammer
Drawer
15
Sponge
Plier 5
Scissors
Fridge 20
6. Run FoundationPose [70] to track the objects, which allows us to track their associated keypoints (we assume the keypoint does not move relative to the object frame)
7. Run the low-level policy, given the base wrist pose trajectory and tracked keypoints.
C.2 Mapping Wrist Actions to Arm Joints
In simulation-only experiments, control a floating (non-physical) hand whose wrist pose be commanded directly. To train policy in simulation that can be executed on a real robot, the wrist attached 7-DoF KUKA LBR iiwa 14 arm, so the residual wrist-pose action produced by the policy must be converted into incremental arm joint commands. We perform this conversion with damped-Ieasttsquare inverse kinematics (DLS-IK).
Let J E R6 Nom the analytical Jacobian of the arm (Nam = 7 in our setup), evaluated at the current joint configuration 0 E IR Nam, and let R6 be the 6-D spatial error twist (concatenated position and orientation error) between the current wrist pose and the target wrist pose. The arm joint update A0 R-Nan is computed
AO - + e,
(5)
where 0.5 is constant damping factor. Equation (5) implements the damped pseudoinverse
JÍ = (JJT + x216)=1, yielding the minimum morimum-normm norm solution to JAo = e while regularising the update near kinematic singularities. Lastly, we compute joint position target Glrryet + A0, clamp this to stay within the joint limits, and then send this as the target to a low-level joint-position PD controller running at 200 Hz.
By default, the target wrist pose is specified by the wrist pose trajectory generated by the VLM. The policy outputs a residual wrist pose action that modifies this target, allowing fine-grained corrections. The resulting target is then used to compute the spatial error By construction, if the residual wrist pose action O, the error corresponds exactly the difference between the current wrist pose and the original VLM- -LM-ggeerated trajectory, the arm will simply follow the given wrist pose trajectory.
26

---

C.3
Tasks
We evaluate our system on three real-world manipulation tasks:
Slide Box to Bottle: The goal to push the box the bottle. The box starts from face-down orientation approximately 35cm away from the bottle. A trial is considered successful if the box makes contact with the bottle.
Place Bottle onto Plate: The goal is to grasp the bottle and place it onto plate. The bottle starts from an upright orientation approximately 42cm away from the plate. The task is considered successful if the bottle is lifted and makes contact with the top surface of the plate.
Hammer Three Times:
The goal to grasp a hammer by its handle and strike a fixed surface three times in succession. The hammer starts in a resting position on the table within the rohot's reachable workspace. A trial is considered successful if three distinct strikes are executed with observable contact between the hammer head and the table surface. This task requires precise control of contact dynamies and pose stabilization, making substantially more challenging than pushing or pick-and-place tasks.
For each task, we run 20 trials across 4 VLM -LM-generated trajectories. The procedure is as follows: We first initialize the objects in random positions with the same range used in simulation training. Next, we capture an RGB-D image of the scene, and then query the VLM to generate a wrist trajectory and keypoint trajectories based on the image and natural language instruction. Each generated trajectory is tested in 5 repeated trials, resetting the objects to similar initial poses before each attempt. This process is repeated 4 times with new randomized object positions and new trajectory queries, resulting in 20 total trials per task.
C.4 Domain Randomization
We improve the policy's s ability to transfer to the real world a zero-shot setting through domain randomization. This enables robustness to physical parameters that are unknown, noisy, or inaccu- rately modeled in the real environment. Specifically, we apply the following randomizations during training:
Joint stiffness and damping are multiplied from their default values by a factor sampled from uniform distribution: 20.0., 3.0). These parameters sampled once at the start of training (independently for each parallel environment) and remain fixed throughout training.
Observation noise is added to each of the robot proprioception observations, sampled independently from a normal distribution: (0, 0.05²). This is uncorrelated noise that is sampled at every control timestep.
Action noise is added to exponential average action, sampled from: (O, 0.05²). This is uncorrelated noise that is sampled at every control timestep.
C.5
Aditioonal Adjustments
The observation space is nearly identical to that described in Table 4, except that the dummy joints used to control the floating-hand wrist pose are replaced with the arm's actual joints (for joint positions, velocities, and exponentially averaged actions).
We increase the exponential smoothing factor for the action average to 7 = 0.98 to produce smoother motions and reduce jitter in the executed actions.
We adjusted the trajectories to be twice as long for real-world experiments to effectively slow the robot motion down. We found that higher-speed motions typically resulted in less reliable policies, as this likely worsened the sim-to-real gap.
To prevent significant collisions between the hand and the table, we clamp the z-coordinate of the target wrist pose remain above the table height.
27

---

C.6 Digital Twin Construction
Our digital twin simulation environment consisted of robot, table, and two objects per task. The robot URDF and physics parameters were acquired by standard open-source repositories. We measured the dimensions of the table and its position relative to the robot with a measuring tape, which took about 5 minutes. The objects were scanned using off-the-shelf 3D LiDAR scanning app called Kiri Engine, which took about 3 minutes per object.
C.7 Kinematic Reachability
Object positions are initialized to ensure kinematic feasibility within the robot's workspace. Although VLMs are not explicitly aware of robot kinematics, few-shot refinement process biases the model toward feasible trajectories by conditioning on successful prior samples. Furthermore, we introduce substantial noise during plan sampling, encouraging diversity of action proposals such that a significant number of the sampled plans during RL training are kinematically viable.
C.8 Additional Qualitative Analysis
We found that VLM keypoint detection worked significantly better on real world images, as they are more likely to in-distribution than simulation images.
As the low-level policy operates a closed-loop fashion, we find it to be robust to dynamics differences between simulation and reality.
The low-level RL policy appeared optimize the task objective (moving the object keypoint along the generated keypoint trajectory) very well. For example, on the Slide Box to Bottle task, when the predicted box keypoint was on the bottom side of the box, the policy would not only push the box to the bottle, but rotate the box so that the bottom side of the box would be as close as possible to the bottle. On the Place Bottle onto Plate task, when the predicted bottle keypoint was on the upper half of the bottle, the policy would often place the bottle on its side so that the keypoint would be as close to the plate as possible.
The most common failure mode came from the keypoint tracking errors. While the initial predicted keypoints
accurate, our pose tracker was only reliable when the object was completely unoccluded. The pose predictions got worse when the object was occluded and occasionally got very bad when highly occluded, which degraded policy performance.
We performed preliminary experiments testing our policy on unseen objects with different but similar geometry (e.g., replacing the bottle with mustard bottle or tall cup, replacing the plate with a different sized plate). The policy still worked reasonably well on these unseen objects due to the VLM's common-sense understanding to select good keypoints and the RL policy's state-based observations.
D Open Vision-Language Models
While our main experiments use Gemini 2.5 Flash Thinking, we additionally evaluate whether open-source VLMs can serve as effective alternatives within our framework. Specifically, we test Molmo [10], recent open VLM designed for general multimodal understanding.
We evaluate Molmo on three representative tasks from simulated benchmark: drawer opening, hammer, and pliers. For each task, both Gemini and Molmo are queried using identical prompts to generate 2D keypoints and trajectories. A keypoint prediction counted as correct if it corresponds to semantically meaningful and geometrically relevant location on the object, and a trajectory is considered feasible if it produces physically executable motion in simulation.
Molmo successfully identified accurate and semantically relevant keypoints across all evaluated tasks but consistently failed to produce physically plausible coherent trajectories. This suggests that while open VLMs like Molmo demonstrate strong visual grounding and scene understanding, they currently lack the higher-level spatial reasoning needed for multi-step action sequencing.
In contrast, Gemini 2.5 Flash Thinking generated both correct keypoints and feasible trajectories in the majority of trials. This can be attributed to its broader training complex multimodal reasoning
28

---

Table 6: Comparison between open VLM (Molmo) and proprietary general-purpose VLM (Gemini 2.5
Flash Thinking) on three simulated tasks.
Model
Drawer Keypoints (%)
Hammer
Pliers
Traj. (%)
Keypoints (%)
Traj. (%)
Keypoints (%)
Traj. (%)
Molmo
100 (10/10)
0 (0/10)
100 (10/10)
(0/10)
100 (10/10)
(0/10)
Gemini
100
87
87
67
90
80
tasks, which supports stronger high-level planning capabilities, critical for generating structured motion scaffolds in our setting.
Further benchmarking across open VLMMs could provide additional insights into model-specific strengths, but such exploration considered orthogonal to the core contribution of this work, which focuses the framework design rather than VLM selection.
E
Additional Baselines and Ablations
E.1
Reinforcement Learning (RL)
(Simple Reward)
(Complex Reward)
Keypoint
Success Rata (8)
16
Apple
Bottle
Drawer
Pridge
Sponge
Hammer
Scissors
Pliers
Mean
Figure 9: Results the simulation task suite. Success rate (in %) is averaged across three seeds; uncertainty reflects the standard error. Our method performs comparably to the oracle with perfectly scripted plans.
introduce two additional RL-based baselines: one with a simple reward function and another with complex, task-specific reward.
RL with Simple Reward. this baseline, a policy is trained from scratch using reinforcement learning (RL). The reward function mirrors that of our main method, combining contact and keypoint- distance-tor to-treettt---tetttteeee terms, but uses oracle keypoints rather than those detected by the VLM. Importantly, this baseline does not use any trajectory guidance or demonstrations.
RL with Complex Reward. This baseline employs more detailed, hand-crafted, task-specific reward function that includes contact, an additional success signal (assuming access to success detector). This setup is intended to approximate the upper bound of performance achievable through extensive manual reward engineering. As with the previous baseline, no trajectory supervision or demonstrations are used.
and
To ensure fair comparison, evaluate it alongside method under identical oracle keypoint conditions (i.e., Ours + keypoint oracle), while still relying on VLM-gennrated trajectories. Policies trained using PPO. The results are summarized in Fig.
E.2 Imitation Learning (IL.
Diffusion
Diflusion Policy
Diffusion Policy (50 demos)
(Zero-Shot)
(Few-Shot)
Sucoess
Apple
Bottle
Drawer
Fridge
Sponge
Hammer
Scissors
Pliers
Mean
Figure 10: Results of imitation learning baselines the simulation task suite. Success rate (in %) is averaged across three seeds, with uncertainty represented the standard error, Our method performs comparahly Diffusion policy with perfect demonstrations.
29

---

9.06 For imitation learning evaluation, we train a Diffusion Policy [8] model using successful rollouts from the oracle RL policy as demonstrations. Experiments are conducted with 10, 20, and 50 demonstrations. This represents a best-case scenario for imitation learning, as the demonstrations are perfect, originating from scripted trajectories and the oracle RI. policy, rather than noisy human demonstrations. Success rates are reported with the mean standard error computed over three random seeds per task in Fig. 10.
E.3
Noisy VLM Predictions
Waypoint Noise Scaling Drawer
Success Rate (%)
Apple
Hammer
Scissors
-00
8 8 58
0.030 030000
0.25 0.03) 0220-030.02 Waypoint std Waypoint Waypoint Waypoint std
0,55 0.000030.222
0.55 0.000.020.02
Figure 11: Effect of Gaussian noise VLM predictions in the simulation task suite. Success rate (in %) is averaged across three seeds; uncertainty indicates the standard error.
The performance of our method depends on the accuracy of the VLM's outputs. Errors in keypoint detection trajectory prediction can adversely affect downstream RL performance.
To quantify this sensitivity, we report results using oracle keypoint and trajectory predictions, repre- senting upper bound corresponding to perfect VLM outputs, in the main paper (see 7). Oracle predictions yield varying levels of performance improvement across tasks, and in some cases enable near-perfect success rates.
We additionally evaluate the opposite case by introducing artificial noise into the VLM outputs. Specifically, we add Gaussian noise (O00 a²) to the waypoints proposed by the VLM, thereby simulating planning errors. Noise injected during both training and inference. Success rates are reported as the mean standard error computed over three random seeds per task in Fig. 11.
F
Tasks
We provide brief descriptions of the eight simulated tasks we evaluated:
An apple and a cutting board are placed on a kitchen counter. The keypoints are the apple and the cutting board. The agent's objective is pick up the apple and place it on top of the cutting board.
Move Apple:
Move Bottle: A bottle is positioned on a kitchen counter next to a sink, either lying down standing upright. The keypoints the bottle and target point the counter across the sink. The goal is for the agent to pick up the bottle and place it upright on the opposite side of the sink.
Open Drawer: A closed cupboard with multiple drawers is located in living room. The handle of the top drawer serves as the sole keypoint. The objective is to open this drawer by at least 20 cm.
Open Fridge: closed refrigerator is situated in kitchen. The handle of the fridge is the only keypoint. The agent's 's task is to open the fridge door by at least 60 degrees.
Hammer: hammer rests on a kitchen counter, with the head and handle defined as keypoints. The goal is for the agent to pick up the hammer and perform a hammering motion with at least three swings. A swing is defined as an upward and downward movement of at least cm.
Wipe with Sponge: A sponge is located on kitchen counter, acting as the sole keypoint. The task is to perform a wiping motion on the counter, with success defined as moving the sponge at least 30 em on the counter.
Close Scissors: An open pair of scissors is situated on a kitchen counter, with the handles serving as keypoints. The goal is to close the scissors until the blades form an angle of less than 5 degrees.
30

---

An open pair of pliers is positioned on a kitchen counter, with the handles defined keypoints. The objective close the pliers until the handles form angle of less than 5 degrees.
Close Pliers:
G
Compute Resources
Our training is performed on NVIDIA GPUs, ranging from A5000s to L40s, Depending on the specific task and hardware configuration, training durations vary between 1.5 and 6 hours. For real-world inference, we utilize two RTX 4090 GPUs.
H Prompt Examples
H.1 Keypoint Elicitation Prompt
You expert in robot manipulation. Above is an image of the environment. Your task
deccippicc)) with robot hand.
Identify a *eminimal set of keypoints** needed to plan the motion. Avoid small, sharp, or occluded areas like corners or edges. Choose large, stable regions visible from many angles.
Use **spatially descriptive, short names** (e.g., "left handle", "top surface") and avoid ambiguous labels like "part Only include points that essential for completing the task.
Reply with a JSON list of keypoint namea.
H.2
Move Apple
H.2.1
Keypoint Prompt
Point apple and the cutting board image.
The answer should follow the json format: [["name:: "apple", "point": [...],, { ("name
": "cutting board", "point": [...]]]
The points are in [y, x] format normalized to 0-1000.
H.2.2 Trajectory Prompt
Your are controlling a robot hand to pick up an apple and put it on a cutting board. The coordinates are measured in meters.
The axis is forward, the y axis is left and the axis is up.
First the robot hand towards the apple.
Then grasp the apple and lift up. Finally move the apple on the cutting board and put it down.
Describe a very realistic trajectory of exactly 20 waypoints. Use code to generate the output.
The initial position of the apple [0.00, 0.00, 0.00].
The initial position of the cutting board is [-0.01, -0.38, -0.05). The initial position of the hand is [-0.07, -0.09, 0.26]. Use the following json format for the trajectory:
[[
"waypoint_num" : 0,
"apple": ("x": float, "y": float, "z": float), "cutting board": ("x"; float, "y": float, "z": float), "hand": ["x": float, "y": float, "z": float)
} ....]
**Only** print the jaon output. **not** print anything else with the code.
H.3
Move Bottle
H.3.1 Keypoint Prompt
31

---

Point water bottle the kitchen counter, and pinpoint a point on the kitchen counter to the right of kitchen sink the image. The answer should follow the json format: [["name:: "bottle", "point"; [...),, {" name"; "point", "point"; [...]}]
The points are in [y, x] format normalized to 0-1000.
H.3.2 Trajectory Prompt
Your are controlling robot hand to move bottle to the target position called point" the kitchen counter.
The coordinates are measured in meters.
The x axis is forvard, the y axis is left and the z axis is up. First move the robot hand tovards the bottle.
Then grasp the bottle and lift it up.
Finally
the bottle the target position called "point" and put down. Deacribe a very realistic trajectory of exactly 20 waypoints. Use code to generate the output.
The initial position of the bottle is [0.00, 0.00, 0.00].
The initial position of the point is [-0.22, 0.80, -0.13]. The initial position of the hand is [0.25, -0.08, 0.20].
Use the following json format for the trajectory:
[X
"waypoint_num": 0,
"bottle": ("x": float, "y": float, "z": float), "point": {"x": float, "y": float, "z": float),
"hand": ("x": float, "y": float, "z": float) ....
**Only** print the json output. Do **not** print anything else with the code.
H.4
Open Drawer
H.4.1 Keypoint Prompt
Point handle the top cabinet drawer in the image. The ansser should follow the json format: [["name": "handle", "point"; [...])] The points are in [y, x] format normalized to 0-1000.
H.4.2 Trajectory Prompt
Your are controlling a robot hand to pull open a cabinet drawer. The coordinates are measured in meters.
The axis is forward, the y axis is left and the axis is up. handle the draver.
First move robot hand towards Then grasp the handle.
Finally pull the drawer open by at least 30cm. Describe very realistic trajectory of exactly 20 waypoints. Use code generate the output.
The initial position of the handle is [0.00, 0.00, 0.00]. The initial position of the hand is [0.32, -0.05, 0.12]. Use the following json format for the trajectory:
[[
"waypoint_num": 0,
"handle": [[xx: float, "y": float, "z": float), "hand": ("x": float, "y": float, "z": float)
} ...]
**Onl+** print the json output. Do **not** print anything else with the code.
32

---

H.5 Open Fridge
H.5.1 Keypoint Prompt
Point to the top handle of the refrigerator door in the image. The answer should follow the json format: [{"name": "handle", "point"; [...])] The points are in [y, x] format normalized 0-1000.
H.5.2 Trajectory Prompt
Your are controlling robot hand open a refrigerator door. The coordinates are measured in meters.
The x axis is forward, the axis is left and the z axis is up.
The refrigerator faces in direction.
The y axis points the right, and the axis pointa up. First figure out how large the door is.
Then describe how the x and coordinates of the handle change as the door is opened
Now the robot hand towards the handle.
Then grasp handle.
Finally fully open the door.
Describe a very realistic trajectory of exactly 20 waypoints. Use code to generate the output.
The initial position of the handle is [0.00, 0.00, 0.00] The initial position of the hand is [0.50, 0.00, -0.22] Use the following json format for the trajectory:
[(
"waypoint_nume 0, "handle": {"x": float, "y": float, "z": float), "hand": {"x": float, "y": float, "z": float) } ....]
**Only** print the json output. Do **no*** print anything else with the code.
H.6 Hammer
H.6.1 Keypoint Prompt
Point to the brown handle and the metal head of the hammer in the image. The answer should follow the json format: [["name": "handle", "point"; [...]), {" name": "head", "point": [..]]]]
The points are in [y, x] format normalized to 0-1000.
H.6.2 Trajectory Prompt
Your are controlling a robot hand to make a hammering motion. The coordinates are measured in meters.
The axis is forward, the y axis is left and the axis is up. First the robot hand towards the handle. Then grasp the handle.
Finally hit on the kitchen counter times. Describe a very realistic trajectory of exactly 20 waypoints. Use code to generate the output.
The initial position of the handle [0.00, 0.00, 0.00].
The initial position of the head is [-0.02, 15, 0.03]. The initial position of the hand is 01, 0.06, 0.27]. Use the following json format for the trajectory:
[[
"waypoint_num"; 0,
"handle": ("x": float, "y": float, "z": float), "head": ("x": float, "y": float, "z": float),
"hand"; ("x": float, "y": float, "z": float)
33

---

} ....
**Only** print the jaon output. **not** print anything else with the code.
H.7 Wipe with Sponge
H.7.1 Keypoint Prompt
Point to the green yellow sponge on the kitchen counter in the image. The answer should follow the json format: [["name:: "sponge", "point"; [...])] The points in [y, x] format normalized 0-1000.
H.7.2 Trajectory Prompt
Your controlling robot hand wipe kitchen counter with sponge. The coordinates are measured in meters. The x axis is forward, the y axis left the z axis up. First move the robot hand towards the aponge. Then grasp the sponge.
Finally wipe the kitchen counter with the sponge.
Describe a very realistic trajectory of exactly 20 waypoints. Use code to generate the output.
The initial position of the sponge is [0.00, 0.00, 0.00]. The initial position of the hand is [0.26, 0.03, 0.29]. Use the following json format for the trajectory:
[[
"vaypoint_uum", 0, "sponge": {"x": float, "y": float, "z": floath, "hand": {"x": float, "y": float, "z": float)
} ....
+*Only** print the json output. Do **not** print anything else with the code.
H.8 Close Scissors
H.8.1 Keypoint Prompt
Point the loops of the scissors the image. The answer should follow the jaon format: [["name": "loop 1", "point"; [...]), {" name": "loop 2", "point": [...]]]
The points are in [y, x] format normalized to 0-1000.
H.8.2 Trajectory Prompt
Your are controlling a robot hand to close pair of acissors. The coordinates are measured in meters.
The axis is forward, the y axis is left and the axis is up. First move the robot hand towards Then grasp the two loops and entirely close the scissors. Describe a very realistic trajectory of exactly 20 waypoints. Use code generate the output.
scissors,
[0.00, 0.00, 0.00]. The initial position of the loop is [-0.07, 0.07, 0.01]. The initial position of the hand is [0.03, -0.06, 0.33]. Use the following json format for the trajectory:
The initial position of the loop
[(
"waypoint_nun": 0,
"loop 1": ("x": float, "y": float, "z": float), "loop 2": {"x": float, "y": float, "2": float),
"hand": ["x": float, "y": float, "z": float) ....]
**Only** print the json output. **not** print anything else with the code.
34

---

H.9 Close Pliers
H.9.1 Keypoint Prompt
Point to the left and right handles of the plier in the image. The answer should follow the jeon format: : "handle left", "point": [.. .,.,, ("name": "handle right", "point": [...]}] The points are in [y, x] format normalized to 0-1000.
H.9.2 Trajectory Prompt
Your controlling robot hand close plier. The coordinates are measured in meters.
The x axis is forward, the y axis is left and the z axis is up. First move the robot hand towards the plier.
Then grasp the left and right handles and entirely close the plier. Describe very realistic trajectory of exactly 20 waypoints. Use code to generate the output.
The initial position of the handle left is [0.00, 0.00, 0.00]
The initial position of the handle right is [-0.05, 0.16, 0.00] The initial position of the hand is [0.01, -0.08, 0.31]. Use the following json format for the trajectory:
[{
"waypoint_num". 0,
"handle left": ("x"; float, "y": float, "2": float), "handle right": {"x": float, "y": float, "z": float),
"hand": {"x": float, "y": float, "z": float)
} ....]
**Only** print the json output. Do **not** print anything else with the code.
H.10 Place Bottle onto Plate
H.10.1 Keypoint Prompt
Point to the middle of the bottle and the plate on the table in the image. The answer should follow the json format: [["name": "bottle", "point": [...]), {" name"; "plate", "point": [...]]]
The points in [y, x] format normalized 0-1000.
H.10.2 Trajectory Prompt
Your controlling robot hand The coordinates
bottle onto plate.
meaaured The x axis is foruard, the y axis left and the z axis up. First move the robot hand towards the bottle.
Then grasp the bottle and lift it up. Then place the bottle to the plate.
Describe a very realistic trajectory of exactly 20 waypoints. Use code to generate the output.
The initial position of the bottle is [0.00, 0.00, 0.00].
The initial position of the plate is (0.30, 0.38, -01 The initial position of hand is [-0.15, -0.29, 0.09]. Use the following jeon format for the trajectory:
I[
"vaypoint_nun": 0,
"bottle"; ("x": float, "y": float, "z"; float). "plate": {"x": float, "y": float, "z": float),
"hand"; ("x": float, "y": float, "z": float) } ....
**Onl+* print the json output. Do **not** print anything else with the code.
35

---

H.11 Slide Box to Bottle
H.1111 Keypoint Prompt
Point to the box and the bottle on the table in the image. The answer should follow the jeon format: [ "box", "point": ..,,, {"name": "bottle", "point"; [...]}]
The points are in [y, x] format normalized 0-1000.
H.112 Trajectory Prompt
Your controlling robot hand slide the box the table the bottle. The coordinates The x axis is forward, the y axis left and the z axis up. First move the robot hand towards the box.
measured
Then slide the box over the table to the bottle.
Describe very realistic trajectory of exactly 20 waypoints. Use code to generate the output.
The initial position of the box is [0.00, 0.00, 0.00].
The initial position of the bottle is [0.17, 0.23, 0.08]. The initial position of the hand is [-0.22, -0.29, 0.19]. Use the following jaon format for the trajectory: [[
"waypoint_me paypoint_num". _umb::: 0, "box": ("x"; float, "y"; float, "z": float), "bottle"; ("x": float, "y": float, "z": float), "hand": {"x": float, "y": float, "z": float) } ....]
**Only** print the jaon output. Do **not** print anything else with the code.
H.12
Hammer Three Times
H.12.1 Keypoint Prompt
Point to handle and the head of hammer the image. The answer should follow the jeon format: [["name": "handle", "point"; [...]), {" name": "head", "point": [...]}]
The points in [y, x] format normalized to 0-1000.
H.12.2 Trajectory Prompt
Your are controlling a robot hand to make a hammering motion. The coordinates are measured in meters.
The axis is forward, the y axis is left and the axis is up. First move robot hand towards hammer handle. Then grasp the hammer handle.
Finally hit on the kitchen counter 3 times. Describe very realistic trajectory of exactly 20 waypoints. Use code generate the output.
The initial position of the handle is [0.00, 0.00, 0.00].
The initial position of the head is 19, 0.13, 04].
The initial position of the hand is [2.12, 0.01, 0.12]. Use the following json format for the trajectory:
[[
"waypoint_num" 0,
"handle": ("x": float, "y": float, "2": float), "head": {"x": float, "y": float, "z": float),
"hand": {"x": float, "y": float, "z": float) } ....]
*+Only** print the json output. Do **not** print anything else with the code.
36


Reviews:
[{"review_id": "dQtHK7wxgh", "reviewer": "Reviewer_a8sd", "summary": "This paper presents a two-stage method for dexterous manipulation. An off-the-shelf VLM serves as a high-level planner that generates keypoints or waypoints and synthesizes 3D trajectories, while a reinforcement-learning policy in simulation tracks the coarse trajectories. The experiments cover an eight-task suite in simulation, comparisons with several baselines, failure analysis, and ablation studies on the number of few-shot examples and waypoints. The Place Bottle and Slide Box tasks are also deployed on real robots.", "questions": "See weaknesses.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n\n1. Combining VLM and RL in simulation is reasonable. It shows that VLM-generated trajectories can solve articulation and precise manipulation tasks (although these are in simulation only, not real world).\n2. The simulation task suite contributes to the community.\n3. The failure analysis provides useful insights.\n4. The presentation quality is high; figures and writing are well polished.\n\n**Weaknesses:**\n\n1. The method is not compared with strong baselines. All baselines in Fig. 4 are self-crafted (not compared with SOTA methods mentioned in Section 2 \"Related Work\" such as RoboDexVLM and Iterative Keypoint Rewards). It is unclear how the proposed method differs from related works, and more importantly, whether it performs better.\n2. The RL is trained in simulation, which assumes access to a real-to-sim digital twin. This limits generalizability to new tasks and environments. Reproducing the experiment requires building a real-to-sim setup, which is not well discussed in the paper.\n3. Why are only simpler tasks deployed in the real world? What about articulation and precise manipulation, which are only shown in simulation?\n4. How is kinematic reachability handled when deploying in the real world? In simulation, a free-floating hand is used. Would there be many invalid trajectories generated by the VLM or RL due to this simplification?", "was_revised": true, "final_justification": "Considering the new baselines and tasks, I will increase my score to borderline accept", "rebuttal": "We thank the reviewers for their constructive feedback. Below, we address their stated concerns.\n# Baseline Comparisons\nTo strengthen the empirical evaluation, we have incorporated four additional baselines. We report success rates as the mean +- standard error computed over three random seeds for each task.\n## Iterative Keypoint Rewards (IKER)\nWe implement an IKER-style baseline in which a vision-language model (VLM) generates code for reward function parameters, using the exact prompts and methodology from the original paper. However, as IKER assumes a fixed set of keypoints, we use the same VLM-identified keypoints as in our method for parity. \n| method           | Apple       | Bottle      | Drawer      | Fridge      | Hammer      | Pliers      | Scissors    | Sponge      |\n|------------------|-------------|-------------|-------------|-------------|-------------|-------------|-------------|-------------|\n| IKER (zero-shot) | 0.45 ± 0.14 | 0.63 ± 0.08 | 0.23 ± 0.01 | 0.19 ± 0.04 | 0.53 ± 0.03 | 0.19 ± 0.11 | 0.27 ± 0.03 | 0.06 ± 0.02 |\n| Ours (zero-shot) | 0.84 ± 0.01 | 0.88 ± 0.00 | 0.87 ± 0.01 | 0.12 ± 0.00 | 0.67 ± 0.01 | 0.80 ± 0.03 | 0.67 ± 0.01 | 0.87 ± 0.02 |\n\n## RL From Scratch\nWe introduce an additional baseline in which a policy is trained from scratch using RL. The reward function mirrors that of our method – combining contact and keypoint-distance-to-target – but relies on oracle keypoints rather than those detected by a VLM. Importantly, this baseline does not use any trajectory guidance or demonstrations. To ensure a fair comparison, we evaluate it alongside our method using the same oracle keypoints (i.e., Ours + keypoint oracle), but retaining VLM-generated trajectories. Policies are trained using PPO.\n\n| method                      | Apple       | Bottle      | Drawer      | Fridge      | Hammer      | Pliers      | Scissors    | Sponge      |\n|-----------------------------|-------------|-------------|-------------|-------------|-------------|-------------|-------------|-------------|\n| Simple RL + keypoint oracle | 0.00 ± 0.00 | 0.99 ± 0.00 | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.65 ± 0.33 | 1.00 ± 0.00 | 0.00 ± 0.00 |\n| Ours + keypoint oracle      | 0.90 ± 0.02 | 0.85 ± 0.02 | 0.82 ± 0.00 | 0.26 ± 0.01 | 0.81 ± 0.01 | 0.85 ± 0.01 | 0.91 ± 0.02 | 0.96 ± 0.01 |\n\n## RL with Complex Reward Design\nWe also introduce a baseline that employs a more detailed, task-specific, hand-crafted reward function. This reward includes contact, keypoint-distance-to-target, hand-distance-to-object, and an additional success signal, assuming access to a success detector. This baseline is intended to reflect the upper bound of performance achievable through extensive manual reward engineering. As with the previous baseline, it does not utilize any trajectory supervision or demonstrations. To ensure a fair comparison, we evaluate it alongside our method using the same oracle keypoints (Ours + keypoint oracle). Policies are trained using PPO.\n| method                       | Apple       | Bottle      | Drawer      | Fridge      | Hammer      | Pliers      | Scissors    | Sponge      |\n|------------------------------|-------------|-------------|-------------|-------------|-------------|-------------|-------------|-------------|\n| Complex RL + keypoint oracle | 0.98 ± 0.01 | 0.99 ± 0.00 | 1.00 ± 0.00 | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.99 ± 0.00 | 1.00 ± 0.00 | 0.00 ± 0.00 |\n| Ours + keypoint oracle       | 0.90 ± 0.02 | 0.85 ± 0.02 | 0.82 ± 0.00 | 0.26 ± 0.01 | 0.81 ± 0.01 | 0.85 ± 0.01 | 0.91 ± 0.02 | 0.96 ± 0.01 |\n\n## Imitation Learning (IL)\nTo evaluate imitation learning, we train Diffusion Policy using successful rollouts of our oracle RL policy as demonstrations. We conduct experiments using 10, 20, and 50 demonstrations. This represents the best-case performance for imitation learning: the demonstrations are perfect, resulting from scripted trajectories and an RL policy, unlike standard human demonstrations. \n\n| method                      | Apple       | Bottle      | Drawer      | Fridge      | Hammer      | Pliers      | Scissors    | Sponge      |\n|-----------------------------|-------------|-------------|-------------|-------------|-------------|-------------|-------------|-------------|\n| Diffusion Policy (10 demos) | 0.26 ± 0.04 | 0.07 ± 0.01 | 0.47 ± 0.03 | 0.36 ± 0.29 | 0.45 ± 0.11 | 0.45 ± 0.10 | 0.52 ± 0.08 | 0.94 ± 0.03 |\n| Diffusion Policy (20 demos) | 0.52 ± 0.02 | 0.17 ± 0.01 | 0.74 ± 0.10 | 0.60 ± 0.20 | 0.45 ± 0.07 | 0.60 ± 0.15 | 0.70 ± 0.10 | 0.99 ± 0.00 |\n| Diffusion Policy (50 demos) | 0.84 ± 0.02 | 0.80 ± 0.05 | 0.85 ± 0.08 | 0.96 ± 0.02 | 0.65 ± 0.09 | 0.79 ± 0.08 | 0.89 ± 0.02 | 0.99 ± 0.00 |\n| Ours (few-shot)             | 0.87 ± 0.02 | 0.86 ± 0.04 | 0.92 ± 0.01 | 0.64 ± 0.04 | 0.75 ± 0.00 | 0.86 ± 0.02 | 0.76 ± 0.03 | 0.84 ± 0.05 |\n| Ours (zero-shot)            | 0.84 ± 0.01 | 0.88 ± 0.00 | 0.87 ± 0.01 | 0.12 ± 0.00 | 0.67 ± 0.01 | 0.80 ± 0.03 | 0.67 ± 0.01 | 0.87 ± 0.02 |\n\n\nWe do not compare against RoboDexVLM because their method relies on a library of manually crafted primitive skills, while our method tries to learn those skills implicitly with RL.  Those primitives are difficult to scale to unstructured motions such as hammering, or precise skills such as closing a pair of scissors. Because of this, RoboDexVLM sticks to simpler grasping tasks. Additionally, the authors of RoboDexVLM have not released their code, making replication difficult. \n# Real-to-Sim Transfer\nWe acknowledge that our approach assumes a real-to-sim transfer pipeline. However, we would like to note that this is a common assumption made by works in dexterous manipulation [ObjDex Chen et al. 24, RoboDexVLM Liu et al. 25, Solving a Rubrik’s Cube, OpenAI et al. 19]. To our knowledge there are only a handful of works that train dexterous RL policies entirely in the real world, which comes with its own challenges (resets etc.) [PDDM, Nagabandi et al. 19]. While we do assume access to a simulated model of the robot, such an assumption is commonplace in several renowned works in dexterous manipulation. We will add more details about our specific sim-to-real pipeline in the final version of the manuscript.\n# Real-World Task Complexity\nWe appreciate the suggestion to explore more challenging tasks in the real world. In response, we have added a **hammering** task, where the robot grasps a hammer by the handle and performs three strikes on a table. This task requires substantially more precision than previous ones (bottle-picking or pushing) and our method achieves a 60% success rate out of 20 trials. Due to the limitations of the rebuttal this year, we are unfortunately unable to upload links or images. To our knowledge, this is the first demonstration of such a task using a multi-fingered robot hand trained via RL.\n\nFor more precise tasks, we are currently limited by hardware constraints, including depth sensor resolution [Learning to estimate 3d states of deformable objects from single-frame occluded point clouds, Lv et al 23] and the mechanical tolerances of the Allegro Hand [LEAP Hand, Shaw et al RSS 23] which restrict performance on high-precision manipulation. \n# Kinematic Reachability\nFor all real-world experiments, we use a 7-DoF robotic arm in both simulation and the physical setup. This ensures that the policy is trained with kinematic constraints similar to those of the real world. The floating-hand configuration is used only for the simulation-only experiments. In both setups, the policy outputs 3D wrist poses, which are converted to joint angles via the pseudo-inverse method. We ensure that the initial hand pose is kinematically feasible and that all objects are placed within the robot’s reachable workspace.\n\nWhile VLMs are initially unaware of robot kinematics, our few-shot refinement phase conditions the model on successful prior samples, improving its ability to generate feasible trajectories. Additionally, we introduce substantial noise during plan sampling, encouraging diversity of action proposals such that a significant number of the sampled plans during RL training are kinematically viable."}, {"review_id": "EE6HlHHVof", "reviewer": "Reviewer_5ndj", "summary": "The paper tackles the long-standing challenge of training dexterous robotic hands without labor-intensive demonstrations or painstakingly engineered reward functions. The authors observe that the fine-grained details in traditional reference trajectories are largely superfluous, because reinforcement learning (RL) ultimately fine-tunes the motion anyway. Their key insight is to replace those trajectories with coarse spatial “scaffolds” automatically produced by a general-purpose vision-language model (VLM).", "questions": "Refer to the weakness section.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\nThe framework sidesteps both manual reward engineering and demonstration collection by using VLM-generated 3-D trajectories as supervisory “scaffolds,” greatly reducing human effort and boosting scalability.\n\nRe-querying the VLM to randomize keypoints and trajectories during training exposes the policy to diverse initial conditions, leading to strong generalization to unseen states and motion plans at test time.\n\nPolicies trained entirely in simulation transfer to a physical dexterous hand with robust performance, demonstrating practical feasibility without any human demonstrations or task-specific rewards.\n\nWeakness:\n-Have the author use robot specific VLM that are fine-tuned for spatial affordance prediction such as RoboPoint or even Molmo? Instead of Gemini.\n\n-The method assumes the VLM can reliably pick out task-relevant key-points and generate a plausible 3-D scaffold. If key-point detection or trajectory synthesis is off (e.g., due to clutter, novel objects, bad lighting), the residual RL layer starts from a poor initialization and may converge to sub-optimal or unsafe motions. However, there isn't ablations that inject systematic VLM errors, nor does it quantify how much deviation the low-level controller can tolerate before success drops. \n\n-The key-point/waypoint representation ignores force/torque constraints, finger–object contact states, and within-hand re-grasping. This abstraction works for the hammer, pliers, and bottle tasks shown, but tasks that hinge on precise in-hand posture changes (e.g., turning a screw two full revolutions) may require richer supervision than the scaffold can provide.", "was_revised": true, "final_justification": "I would like to maintain my rating based on overall novelty of the paper and rebuttal made.", "rebuttal": "We thank the reviewers for their constructive feedback. Below, we respond to the key points raised.\n\n# Use of Robot-Specific Vision-Language Models (VLMs)\n\nWe appreciate the suggestion to explore robot-specific VLMs. As part of our investigation, we evaluated Molmo as an alternative to general-purpose models. We have tried detecting keypoints and generating scaffolds for 3 tasks. The table below shows the percentage of correct keypoints and feasible trajectories.\n\n|        | Drawer            |                   | Hammer            |                   | Pliers            |                   |\n|--------|-------------------|-------------------|-------------------|-------------------|-------------------|-------------------|\n|        | Correct Keypoints | Feasible Trajectories | Correct Keypoints | Feasible Trajectories | Correct Keypoints | Feasible Trajectories |\n| Molmo  | 100% (10/10)      | 0% (0/10)          | 100% (10/10)      | 0% (0/10)          | 100% (10/10)      | 0% (0/10)          |\n| Gemini | 100%              | 87%                | 87%               | 67%                | 90%               | 80%                |\n\nWhile Molmo reliably identified keypoints, it consistently failed to generate coherent plans and exhibited similar failure modes across multiple tasks and prompts – indicating limited capacity for planning. In contrast, Gemini offers more advanced high-level reasoning, which is central to our framework. Additionally, recent versions of Gemini have also demonstrated strong performance in spatial grounding and pointing, making it particularly well-suited for scaffold generation in our setting. One could continue running experiments to determine the best VLM for each component of the pipeline, but we view doing so as further optimization which would only improve performance.\n\n# Impact of VLM Prediction Errors\n\nWe agree that the effectiveness of our method is influenced by the accuracy of outputs from the VLM. Errors in keypoint detection or trajectory prediction can indeed degrade downstream RL  performance.\n\nTo quantify this, we have already included results using oracle keypoint and trajectory predictions – providing an upper bound on performance in the presence of perfect VLM outputs. These are shown in Figure 7 (right) as “Keypoint Oracle” and “Trajectory Oracle.” As discussed in the paper, oracle predictions yield varying degrees of performance improvement across different tasks, and in some cases, enable near-perfect success rates.\n\nAs requested, we have added results in the reverse direction -- demonstrating how performance degrades with artificially injected gaussian noise. Specifically, we add Gaussian noise N(0, sigma^2) to the waypoints proposed by the VLM, thereby simulating planning errors. We add noise both during training and inference. We report success rates as the mean +- standard error computed over three random seeds for each task.\n| method              | Apple       | Drawer      | Hammer      | Scissors    |\n|---------------------|-------------|-------------|-------------|-------------|\n| No noise (sigma: 0) | 0.84 ± 0.01 | 0.87 ± 0.01 | 0.67 ± 0.01 | 0.67 ± 0.01 |\n| Noisy (sigma: 0.01) | 0.58 ± 0.29 | 0.88 ± 0.00 | 0.72 ± 0.01 | 0.57 ± 0.02 |\n| Noisy (sigma: 0.02) | 0.84 ± 0.01 | 0.87 ± 0.00 | 0.72 ± 0.02 | 0.13 ± 0.01 |\n| Noisy (sigma: 0.05) | 0.36 ± 0.16 | 0.86 ± 0.01 | 0.24 ± 0.03 | 0.34 ± 0.08 |\n\n# Limitations of Representation\n\nNote that motion scaffolds are designed to 1) provide dense rewards for the agent and 2) guide exploration with residual actions. This does not limit the information that the RL policy can learn from – it could use force information or contacts if desired to learn to do the task effectively though we did not as such information is more difficult in the real world. However, there is nothing about our framework that prevents the policy from doing so. While our reward function is based on spatial information, the low-level RL agent can explore different amounts of force, torque, and contact behaviors. For example, we found that the RL agent automatically used stable grasps with high contact forces for pick and place tasks with large objects, while choosing small forces for finer tasks such as closing the pair of scissors.\n\nOn the other hand, adding force and torque constraints to the reward function is possible, but doing so requires an extensive amount of manual reward design which may not be broadly applicable. The purpose of our work, however, was to show how much scaffolds reduce the burden of reward design across a broad set of tasks. To this end, while adding more specific information could increase performance in specific cases, it comes at the cost of generality. If a task requires additional terms, they could be added on top of the terms for the motion scaffold and used in combination. We leave this exploration to future work.\n\nFurthermore, the spatial information encoded by our reward design can go really far. To demonstrate this, we modified the original bottle task: the bottle may now start either lying down or standing upright on one side of a sink and must be placed upright on the opposite side. This setup requires the policy to infer the initial pose and adjust its behavior accordingly to rotate the bottle upright – all from using multiple keypoints to infer rotation. Unfortunately, due to the limitations of rebuttal this year we are unable to attach images or videos. We report results across three seeds below:\n\n| Method               | Success Rate (mean ± SE) |\n|----------------------|--------------------------|\n| Pre-recorded baseline | 0.09 ± 0.02              |\n| Ours (zero-shot)      | 0.64 ± 0.10              |\n\nWhile the discussion above focused primarily on reward design, we would like to note that there is a plethora of existing literature in constrained RL [Responsive Safety in RL by PID Lagrangian Methods, Stooke et al 2020, Constrained RL with Smoothed Log Barrier Function Zhang et al 24], which is orthogonal to our effort which could be used for actually enforcing constraints instead of “encouraging compliance” with reward penalties."}, {"review_id": "wyhL98N6S0", "reviewer": "Reviewer_5Gyk", "summary": "In this work,  the authors present a framework for dexterous manipulation which leverages: (i) a high-level VLM policy that infers (2D) keypoints in an image, and provides a (3D) trajectory for those keypoints and the hand's wrist that leads to the task solution, (ii) a low-level residual RL policy, which can correct the wrist trajectory to follow and control the hand's joints. By leveraging a detailed prompt structure and the pre-trained knowledge of a large VLM (Gemini 2.5 Flash Thinking), the framework achieves near-oracle performance in simulation and can transfer to real settings (after training on a digital twin in simulation).", "questions": "* What are the technical novelties introduced in the work, apart from the overall (novel) framework?\n* What happens if we reduce the amount of \"human supervision\" in the prompts?\n* What are some ways that the prompt structure could be made more general/modular? E.g. the user only inputs the task as \"move the bottle to the right\"\n* Given that most of the evaluation is performed in simulation, demonstrations should be easy to produce (e.g. using the oracle). How does the approach compare to imitation learning approaches? E.g. how many demos do we need to reach the framework's performance?\n* Given that most of the evaluation is performed in simulation, it should be possible to use SOTA RL techniques to solve the given tasks. How many timesteps/hours are needed to match the performance of the framework proposed?\n* The authors claim their simple reward function is beneficial. This could be a useful novelty of the work. Could the authors compare to other reward designs to assess this?", "limitations": "The authors should discuss how their framework lacks generality and discuss how to generalize their prompting or get rid of it.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strengths \n* **Design and engineering**: the framework presented is modular, e.g. the VLM can be upgraded with a newer version, and the components seem to fit well together, nicely allowing to leverage the advantages of each module, e.g. the pre-trained knowledge of the VLM or the residual RL policy for correcting coarse trajectories. The deployment on a real-world platform is also remarkable.\n* **Presentation**: the work is well presented, with high-quality and helpful figures and great clarity.\n## Weaknesses\n* **Limited novelty**: the overall framework is, to the best of my knowledge, novel. However, the novelties introduced in the single components seem to be quite limited.\n* **Limited generality**:  looking at the structure of the prompt provided in Appendix F, the framework seem to require task-dependent and environment-dependent prompts. Example keypoint prompt: \"Point to the water bottle on the kitchen counter, and pinpoint a point on the kitchen counter to the right of the kitchen sink in the image.\". This prompt contain specific information about the structure of the environment, the position of the bottle and detailed instructions on where to place the bottle. A similar limitation applies to the trajectory prompts. Thus, while the method requires no demonstration, it seems to require some form of human annotation to solve the task.\n* **Limited comparison**: it's hard to assess the system if no baselines are provided. I understand the problem settings is somehow unique. However, there are several ways the authors could expand their comparison of the method. (see Questions)", "was_revised": true, "final_justification": "I think the paper has plenty of value from an engineering perspective. The idea presented is novel and enables a framework that is flexible and performant. During the rebuttal, the authors have also improved the generalizability of the approach and have provided a more extensive empirical evaluation, generally improving the quality of the work. \n\nFrom a machine learning perspective, the novelties introduced in the work are minor and mostly adapt previous ideas to work together seamlessly. Nonetheless, I am leaning towards acceptance of the paper, as I believe part of the NeurIPS community (those working on robotics) may find the work valuable.", "rebuttal": "We thank the reviewer for their thoughtful and constructive feedback. Below, we address the key points regarding the novelty of our components, prompt design, and experimental baselines.\n# Novelty of Individual Components\n\nWe respectfully disagree with the assessment that the individual components of our framework lack novelty. While our primary contribution lies in the system as a whole – which successfully enables real-world task execution, a recognized contribution in itself – this system is built on several novel and impactful ideas.\n\nTo the best of our knowledge, no prior work leverages **VLMs to generate keypoint trajectories for guiding reinforcement learning**, nor uses **keypoint sequences as a scaffold for both reward shaping and exploration in RL**. Prior works have instead only used keypoints for rewards [IKER Patel et al 25] or used demonstrations for residual RL and tracking [ObjDex Chen et al 24]. Our work represents a significant departure from conventional approaches, which typically rely on these more expensive sources of supervision like motion capture, demonstrations, or scripted heuristics.\n\nWe instead use structured, semantically meaningful guidance derived directly from natural language prompts to a VLM. This eliminates the need for demonstrations or complex setup procedures, and substantially lowers the cost and complexity of deploying RL in real-world manipulation tasks.\n\n# Prompt Specificity\n\nWe appreciate the concern about the risk of prompt specificity introducing unintended task knowledge. In response, we revised our approach to use a **single, concise natural language instruction per task** (e.g., “grasp the apple and move it onto the cutting board” or “grasp the refrigerator handle and fully open the top refrigerator door”).\n\nTo do this, we use a standardized preamble to elicit task-relevant keypoints from the VLM:\n\n*\"You are an expert in robot manipulation. Above is an image of the environment. Your task is to {task_description} with a robot hand.\nIdentify a minimal set of keypoints needed to plan the motion. Avoid small, sharp, or occluded areas like corners or edges. Choose large, stable regions visible from many angles.\nUse spatially descriptive names (e.g., \"left handle\", \"top surface\") and avoid ambiguous labels like \"part 1\". Only include points that are essential for completing the task.\nReply with a JSON list of keypoint names.\"*\n\nThe keypoints generated by the VLM using this prompt are employed both for keypoint detection and for motion trajectory planning. We find that the chosen keypoints closely align with human intuitions about salient features necessary for task completion.\n\nTo assess the impact of the new prompting strategy, we evaluated performance across three manipulation tasks and report the mean success rate ± standard error over three seeds.\n\n| Prompt      | Apple       | Fridge      | Pliers      |\n|-------------|-------------|-------------|-------------|\n| Old Prompts | 0.84 ± 0.01 | 0.12 ± 0.00 | 0.80 ± 0.03 |\n| New Prompts | 0.88 ± 0.01 | 0.15 ± 0.01 | 0.81 ± 0.02 |\n\nThese results indicate that simplifying and standardizing the prompts preserves task performance while reducing the risk of embedding task-specific priors, thereby improving the generality of our approach.\n\n# Baselines\n\nTo strengthen the empirical evaluation, we have incorporated four additional baselines. We report success rates as the mean +- standard error computed over three seeds for each task.\n\n## Iterative Keypoint Rewards (IKER)\n\nWe implement an IKER-style baseline in which a vision-language model (VLM) generates code for reward function parameters, using the exact prompts and methodology from the original paper. However, as IKER assumes a fixed set of keypoints, we use the same VLM-identified keypoints as in our method for parity. \n\n| Method           | Apple       | Bottle      | Drawer      | Fridge      | Hammer      | Pliers      | Scissors    | Sponge      |\n|------------------|-------------|-------------|-------------|-------------|-------------|-------------|-------------|-------------|\n| Iker (zero-shot) | 0.45 ± 0.14 | 0.63 ± 0.08 | 0.23 ± 0.01 | 0.19 ± 0.04 | 0.53 ± 0.03 | 0.19 ± 0.11 | 0.27 ± 0.03 | 0.06 ± 0.02 |\n| Ours (zero-shot) | 0.84 ± 0.01 | 0.88 ± 0.00 | 0.87 ± 0.01 | 0.12 ± 0.00 | 0.67 ± 0.01 | 0.80 ± 0.03 | 0.67 ± 0.01 | 0.87 ± 0.02 |\n\n## RL From Scratch\n\nWe introduce an additional baseline in which a policy is trained from scratch using RL. The reward function mirrors that of our method – combining contacts and keypoint-distance-to-target – but relies on oracle keypoints rather than those detected by a VLM. Importantly, this baseline does not use any trajectory guidance or demonstrations. To ensure a fair comparison, we evaluate it alongside our method using the same oracle keypoints (i.e., Ours + keypoint oracle), but retaining VLM-generated trajectories. Policies are trained using PPO for the same number of timesteps as our method.\n\n| Method                      | Apple       | Bottle      | Drawer      | Fridge      | Hammer      | Pliers      | Scissors    | Sponge      |\n|-----------------------------|-------------|-------------|-------------|-------------|-------------|-------------|-------------|-------------|\n| Simple RL + keypoint oracle | 0.00 ± 0.00 | 0.99 ± 0.00 | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.65 ± 0.33 | 1.00 ± 0.00 | 0.00 ± 0.00 |\n| Ours + keypoint oracle      | 0.90 ± 0.02 | 0.85 ± 0.02 | 0.82 ± 0.00 | 0.26 ± 0.01 | 0.81 ± 0.01 | 0.85 ± 0.01 | 0.91 ± 0.02 | 0.96 ± 0.01 |\n\n## Imitation Learning (IL)\n\nTo evaluate imitation learning, we train Diffusion Policy using successful rollouts of our oracle RL policy as demonstrations. We conduct experiments using 10, 20, and 50 demonstrations. This represents the best-case performance for imitation learning: the demonstrations are perfect, resulting from scripted trajectories and an RL policy, unlike standard human demonstrations. \n\n| Method                      | Apple       | Bottle      | Drawer      | Fridge      | Hammer      | Pliers      | Scissors    | Sponge      |\n|-----------------------------|-------------|-------------|-------------|-------------|-------------|-------------|-------------|-------------|\n| Diffusion Policy (10 demos) | 0.26 ± 0.04 | 0.07 ± 0.01 | 0.47 ± 0.03 | 0.36 ± 0.29 | 0.45 ± 0.11 | 0.45 ± 0.10 | 0.52 ± 0.08 | 0.94 ± 0.03 |\n| Diffusion Policy (20 demos) | 0.52 ± 0.02 | 0.17 ± 0.01 | 0.74 ± 0.10 | 0.60 ± 0.20 | 0.45 ± 0.07 | 0.60 ± 0.15 | 0.70 ± 0.10 | 0.99 ± 0.00 |\n| Diffusion Policy (50 demos) | 0.84 ± 0.02 | 0.80 ± 0.05 | 0.85 ± 0.08 | 0.96 ± 0.02 | 0.65 ± 0.09 | 0.79 ± 0.08 | 0.89 ± 0.02 | 0.99 ± 0.00 |\n| Ours (few-shot)             | 0.87 ± 0.02 | 0.86 ± 0.04 | 0.92 ± 0.01 | 0.64 ± 0.04 | 0.75 ± 0.00 | 0.86 ± 0.02 | 0.76 ± 0.03 | 0.84 ± 0.05 |\n| Ours (zero-shot)            | 0.84 ± 0.01 | 0.88 ± 0.00 | 0.87 ± 0.01 | 0.12 ± 0.00 | 0.67 ± 0.01 | 0.80 ± 0.03 | 0.67 ± 0.01 | 0.87 ± 0.02 |\n\n# Comparison to Other Reward Designs\n\nThe reviewer asked about comparisons to other, more complicated reward designs. To shed light on the effectiveness of our reward design, we additionally compare RL from scratch using more detailed, task specific, hand-crafted reward functions. This reward includes contact, keypoint-distance-to-target, hand-distance-to-object, and an additional ground truth success signal, assuming access to a success detector. This baseline is intended to reflect the upper bound of performance achievable through extensive manual reward engineering. As with the previous baseline, it does not utilize any trajectory supervision or demonstrations. To ensure a fair comparison, we evaluate it alongside our method using the same oracle keypoints (Ours + keypoint oracle). Policies are trained using PPO.\n\n| Method                       | Apple       | Bottle      | Drawer      | Fridge      | Hammer      | Pliers      | Scissors    | Sponge      |\n|------------------------------|-------------|-------------|-------------|-------------|-------------|-------------|-------------|-------------|\n| Complex RL + keypoint oracle | 0.98 ± 0.01 | 0.99 ± 0.00 | 1.00 ± 0.00 | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.99 ± 0.00 | 1.00 ± 0.00 | 0.00 ± 0.00 |\n| Ours + keypoint oracle       | 0.90 ± 0.02 | 0.85 ± 0.02 | 0.82 ± 0.00 | 0.26 ± 0.01 | 0.81 ± 0.01 | 0.85 ± 0.01 | 0.91 ± 0.02 | 0.96 ± 0.01 |"}, {"review_id": "qDXfLYUKz1", "reviewer": "Reviewer_b94o", "summary": "This paper proposes a two-stage framework for general dexterous manipulation. The first stage uses a VLM to generate semantic keypoints and corresponding trajectories, while the second stage employs reinforcement learning to train a residual policy for low-level control. Both simulation and real-world experiments are conducted to validate the effectiveness of the method.", "questions": "The current method requires successful trajectories from the training set to enhance VLM planning. How can this method be extended to unseen scenes and objects?", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. This paper focuses on an important problem—current general dexterous manipulation methods mostly rely on reference trajectory imitation. However, sourcing such trajectories can be challenging. This paper addresses the issue by leveraging the capabilities of VLM.\n\nWeaknesses:\n1. The proposed keypoints and trajectories may not be fine-grained, yet the current method encourages the agent to imitate them. For example, in the hammer case, it actually grasps the head, causing clipping. Such a policy would likely to fail in real-world scenarios.\n2. The keypoints proposed by VLM contain only positional information without orientation. For instance, if the hammer lies flat on the table rather than standing upright, the policy would not learn to reorient it. This missing information may limit the generalization ability of VLM.\n3. From the simulation results, although the policy achieves a high success rate, the physical realism for complex manipulation seems questionable. For example, when closing the pliers, the dexterous hand does not actually grasp the object, yet it still completes the task. Additionally, the real-world experiments do not include more complex manipulation, which raises concerns about the actual capability of VLM in handling complex dexterous manipulation tasks.", "was_revised": true, "final_justification": "While the author provides additional justification and experiments, I think the unrealistic motion shown in the video still exposes the limitation of the current method, and the experiment provided by the author also validates the concern. Therefore, I think the current method is not a very ready paper to show the potential of the current method.", "rebuttal": "We thank the reviewer for the constructive feedback. Below, we address each of the key points raised.\n\n# Coarseness of Scaffolds\n\nNote that scaffolds are only intended to be used as a reward signal and to guide exploration – the policy is free to learn whatever it wants on top of the reward signal and thus is not strictly “imitating”.  We acknowledge that our scaffold-based reward signals are coarser than those derived from human demonstrations or hand-crafted dense rewards. However, this is a deliberate design choice, balancing scalability and supervision quality. Human demonstrations are often costly, time-intensive, and difficult to scale – especially as new tasks require collecting entirely new datasets. Similarly, crafting dense, high-quality reward functions demands significant manual engineering and domain expertise.\n\nIn contrast, our scaffolds provide lightweight, task-agnostic, semantically meaningful supervision with minimal human input. While inherently less precise, we show that these signals are sufficiently informative to guide learning across a wide variety of tasks.\n\nWe see scaffolds as a practical middle ground: they are significantly more scalable than demonstrations or hand-tuned rewards, while still effective across diverse domains. Future work may explore hybrid approaches – e.g., combining scaffold-based rewards with sparse success signals or contact-based feedback – to enable finer control when needed.\n\n# Lack of Explicit Orientation Information\n\nThough each keypoint does not encode orientation, multiple keypoints together do implicitly capture orientation when at least three non-colinear keypoints are annotated – an approach that is feasible for many real-world objects. In this way, motion scaffolds can effectively be used for tasks requiring rotation.\n\nTo demonstrate this capability, we modified the original bottle task: the bottle may now start either lying down or standing upright on one side of a sink and must be placed upright on the opposite side. This setup requires the policy to infer the initial pose and adjust its behavior accordingly. We report a success rate of 0.64 ± 0.1 across three seeds. By using multiple spatial keypoints together, the policy learns to rotate and upright a bottle. Unfortunately, due to the limitations of rebuttal this year we are unable to attach images or videos. \n\n| Method               | Success Rate (mean ± SE) |\n|----------------------|--------------------------|\n| Pre-recorded baseline | 0.09 ± 0.02              |\n| Ours (zero-shot)      | 0.64 ± 0.10              |\n\nWe acknowledge that for smaller objects, where one or two keypoints may be insufficient to disambiguate orientation, this becomes more challenging. In such cases, richer scaffold representations – potentially powered by advances in spatially grounded vision-language models (VLMs) – represent a promising direction to improve expressiveness and generality.\n\n# Physical Realism and Real-World Complexity\n\nWe appreciate the reviewer’s concern regarding the complexity and physical realism of our tasks. To address this, we have extended our real-world evaluation to include a significantly more demanding **hammering** task, in which the robot must grasp a hammer by the handle and strike a table three times. This task demands substantially more precision and physical realism than pick-and-place or pushing tasks. Due to NeurIPS guidelines, we are unable to provide additional visual materials (e.g., images or videos) at this stage.\n\nDespite the increased difficulty, our system achieves a 60% success rate over 20 rollouts. Importantly, most failure cases arise not from deficiencies in the learning framework or simulation fidelity, but from hardware-related constraints, such as the mechanical limitations of the Allegro Hand [LEAP Hand, Shaw et al RSS 23] and noise in depth sensing [Learning to estimate 3d states of deformable objects from single-frame occluded point clouds, Lv et al 23].\n\nWe acknowledge that sim-to-real transfer remains an open and well-recognized challenge, particularly for tasks requiring high-precision contact and control [ObjDex Chen et al. 24, RoboDexVLM Liu et al. 25, Solving a Rubrik’s Cube, OpenAI et al. 19]. As part of ongoing work, we plan to investigate improvements in simulation realism, multi-view visual perception, and sensor fusion – directions we believe are essential for narrowing the sim-to-real gap. \n\n# Generalization to Unseen Scenes and Objects\n\nWe thank the reviewer for highlighting the importance of generalization. Our approach is designed to promote generalization by leveraging VLM-generated scaffolds and a policy conditioned on keypoint observations and proprioception, rather than raw visual input. This decouples the learning process from pixel-level variability and allows for greater robustness.\n\nEmpirically, we observe strong zero-shot generalization in real-world settings. For example, in the bottle pick-and-place task, we successfully replaced the bottle with cups and mugs, maintaining high success rates by modifying only a single word in the prompt. Additionally, the system demonstrates robustness to changes in lighting conditions, background clutter, and instruction phrasing, with no significant degradation in performance across these variations.\n\nThat said, we acknowledge that generalization to entirely new task categories or structurally different objects remains a significant challenge. In practice, doing so has required training policies across an extremely broad set of objects [DextrAH, Singh et al 24], or learning object agnostic representations. We see this as an important and exciting direction for future work in combination with our proposed motion scaffolds."}]

Author rebuttal:

Meta-review: This paper proposes a two-stage framework for dexterous manipulation: a high-level VLM module that produces semantic keypoints and trajectories, and a low-level reinforcement learning policy that tracks and refines them. The system is evaluated on an eight-task suite in simulation and several real-world tasks, with ablations, baseline comparisons, and a new hammering task added in the rebuttal.

**Strengths**

- The idea of using VLM-generated motion scaffolds as lightweight supervision for RL is novel and impactful. It sidesteps the reliance on expensive demonstrations or hand-engineered rewards, thereby reducing human effort.
- The modular design is attractive: the VLM can be upgraded, and the residual RL policy naturally complements the coarse guidance.
- The work includes a thorough set of experiments, including new baselines (IKER, RL from scratch, imitation learning with diffusion policies, and complex reward engineering) that position the method more convincingly relative to prior art.
- The presentation quality is consistently high, with polished writing, clear figures, and informative failure analyses.
- Real-world deployment, especially the added hammering task, strengthens the contribution and shows feasibility beyond simulation.

**Weaknesses**

- Concerns remain about the realism and generality of the approach. Several reviewers noted that simulated motions (e.g., hammering, pliers) appear physically unrealistic, raising questions about whether coarse scaffolds are sufficient for complex dexterous control.
- The reliance on a real-to-sim pipeline limits applicability, and while the authors clarify that VLM planning is re-run in the real setting, the necessity of digital twins remains a barrier to broader deployment.
- Orientation and force/torque information are not explicitly represented in the scaffold, which constrains performance in precision tasks. The authors partially address this with multi-keypoint orientation inference, but success rates degrade in harder reorientation tasks.
- Prompting remains somewhat task- and environment-specific, and while the rebuttal shows improvements with standardized prompts, generalization to unseen tasks and scenes is still not fully resolved.
- From a technical standpoint, individual components (VLM planning, residual RL, scaffold-based rewards) are adaptations of existing ideas rather than fundamentally new algorithms. The contribution is primarily an engineering integration, though one that is creative and useful.

**Overall Assessment**

Reviewers converged on a borderline but leaning accept consensus. Some (e.g., R5ndj, R5Gyk) valued the system-level novelty and engineering contribution, while others (e.g., Rb94o) remained unconvinced about physical realism and generalization. The rebuttal added meaningful new evidence: stronger baselines, more challenging real-world demonstrations, and clarifications on kinematic reachability and scaffold orientation. These updates improve confidence in the technical soundness of the approach, even if limitations remain.

**Recommendation**

On balance, the paper presents an interesting and well-executed idea with clear relevance to the NeurIPS robotics community. Despite limited technical novelty and lingering concerns about sim-to-real fidelity, the integration of VLM planning with RL for dexterous control is interesting and worth to be presented to the community. As a result, the ACs recommend acceptance.
Decision: Accept (poster)