Title: Scaffolding Dexterous Manipulation with Vision-Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Dexterous Manipulation

Vision-Language Models

Reinforcement Learning
Abstract: Dexterous robotic hands are essential for performing complex manipulation tasks, yet remain difficult to train due to the challenges of demonstration collection and high-dimensional control. While reinforcement learning (RL) can alleviate the data bottleneck by generating experience in simulation, it typically relies on carefully designed, task-specific reward functions, which hinder scalability and generalization. Thus, contemporary works in dexterous manipulation have often bootstrapped from reference trajectories. These trajectories specify target hand poses that guide the exploration of RL policies and object poses that enable dense, task-agnostic rewards.
However, sourcing suitable trajectories---particularly for dexterous hands---remains a significant challenge. Yet, the precise details in explicit reference trajectories are often unnecessary, as RL ultimately refines the motion. Our key insight is that modern vision-language models (VLMs) already encode the commonsense spatial and semantic knowledge needed to specify tasks and guide exploration effectively. Given a task description (e.g., “open the cabinet”) and a visual scene, our method uses an off-the-shelf VLM to first identify task-relevant keypoints (e.g., handles, buttons) and then synthesize 3D trajectories for hand motion and object motion. Subsequently, we train a low-level residual RL policy in simulation to track these coarse trajectories or ``scaffolds'' with high fidelity. Across a number of simulated tasks involving articulated objects and semantic understanding, we demonstrate that our method is able to learn robust dexterous manipulation policies. Moreover, we showcase that our method transfers to real-world robotic hands without any human demonstrations or handcrafted rewards.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a two-stage framework for general dexterous manipulation. The first stage uses a VLM to generate semantic keypoints and corresponding trajectories, while the second stage employs reinforcement learning to train a residual policy for low-level control. Both simulation and real-world experiments are conducted to validate the effectiveness of the method.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. This paper focuses on an important problem—current general dexterous manipulation methods mostly rely on reference trajectory imitation. However, sourcing such trajectories can be challenging. This paper addresses the issue by leveraging the capabilities of VLM.\n\nWeaknesses:\n1. The proposed keypoints and trajectories may not be fine-grained, yet the current method encourages the agent to imitate them. For example, in the hammer case, it actually grasps the head, causing clipping. Such a policy would likely to fail in real-world scenarios.\n2. The keypoints proposed by VLM contain only positional information without orientation. For instance, if the hammer lies flat on the table rather than standing upright, the policy would not learn to reorient it. This missing information may limit the generalization ability of VLM.\n3. From the simulation results, although the policy achieves a high success rate, the physical realism for complex manipulation seems questionable. For example, when closing the pliers, the dexterous hand does not actually grasp the object, yet it still completes the task. Additionally, the real-world experiments do not include more complex manipulation, which raises concerns about the actual capability of VLM in handling complex dexterous manipulation tasks.\n\nQUESTIONS:\nThe current method requires successful trajectories from the training set to enhance VLM planning. How can this method be extended to unseen scenes and objects?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-18"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work,  the authors present a framework for dexterous manipulation which leverages: (i) a high-level VLM policy that infers (2D) keypoints in an image, and provides a (3D) trajectory for those keypoints and the hand's wrist that leads to the task solution, (ii) a low-level residual RL policy, which can correct the wrist trajectory to follow and control the hand's joints. By leveraging a detailed prompt structure and the pre-trained knowledge of a large VLM (Gemini 2.5 Flash Thinking), the framework achieves near-oracle performance in simulation and can transfer to real settings (after training on a digital twin in simulation).\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths \n* **Design and engineering**: the framework presented is modular, e.g. the VLM can be upgraded with a newer version, and the components seem to fit well together, nicely allowing to leverage the advantages of each module, e.g. the pre-trained knowledge of the VLM or the residual RL policy for correcting coarse trajectories. The deployment on a real-world platform is also remarkable.\n* **Presentation**: the work is well presented, with high-quality and helpful figures and great clarity.\n## Weaknesses\n* **Limited novelty**: the overall framework is, to the best of my knowledge, novel. However, the novelties introduced in the single components seem to be quite limited.\n* **Limited generality**:  looking at the structure of the prompt provided in Appendix F, the framework seem to require task-dependent and environment-dependent prompts. Example keypoint prompt: \"Point to the water bottle on the kitchen counter, and pinpoint a point on the kitchen counter to the right of the kitchen sink in the image.\". This prompt contain specific information about the structure of the environment, the position of the bottle and detailed instructions on where to place the bottle. A similar limitation applies to the trajectory prompts. Thus, while the method requires no demonstration, it seems to require some form of human annotation to solve the task.\n* **Limited comparison**: it's hard to assess the system if no baselines are provided. I understand the problem settings is somehow unique. However, there are several ways the authors could expand their comparison of the method. (see Questions)\n\nQUESTIONS:\n* What are the technical novelties introduced in the work, apart from the overall (novel) framework?\n* What happens if we reduce the amount of \"human supervision\" in the prompts?\n* What are some ways that the prompt structure could be made more general/modular? E.g. the user only inputs the task as \"move the bottle to the right\"\n* Given that most of the evaluation is performed in simulation, demonstrations should be easy to produce (e.g. using the oracle). How does the approach compare to imitation learning approaches? E.g. how many demos do we need to reach the framework's performance?\n* Given that most of the evaluation is performed in simulation, it should be possible to use SOTA RL techniques to solve the given tasks. How many timesteps/hours are needed to match the performance of the framework proposed?\n* The authors claim their simple reward function is beneficial. This could be a useful novelty of the work. Could the authors compare to other reward designs to assess this?\n\nLIMITATIONS:\nThe authors should discuss how their framework lacks generality and discuss how to generalize their prompting or get rid of it.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper tackles the long-standing challenge of training dexterous robotic hands without labor-intensive demonstrations or painstakingly engineered reward functions. The authors observe that the fine-grained details in traditional reference trajectories are largely superfluous, because reinforcement learning (RL) ultimately fine-tunes the motion anyway. Their key insight is to replace those trajectories with coarse spatial “scaffolds” automatically produced by a general-purpose vision-language model (VLM).\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\nThe framework sidesteps both manual reward engineering and demonstration collection by using VLM-generated 3-D trajectories as supervisory “scaffolds,” greatly reducing human effort and boosting scalability.\n\nRe-querying the VLM to randomize keypoints and trajectories during training exposes the policy to diverse initial conditions, leading to strong generalization to unseen states and motion plans at test time.\n\nPolicies trained entirely in simulation transfer to a physical dexterous hand with robust performance, demonstrating practical feasibility without any human demonstrations or task-specific rewards.\n\nWeakness:\n-Have the author use robot specific VLM that are fine-tuned for spatial affordance prediction such as RoboPoint or even Molmo? Instead of Gemini.\n\n-The method assumes the VLM can reliably pick out task-relevant key-points and generate a plausible 3-D scaffold. If key-point detection or trajectory synthesis is off (e.g., due to clutter, novel objects, bad lighting), the residual RL layer starts from a poor initialization and may converge to sub-optimal or unsafe motions. However, there isn't ablations that inject systematic VLM errors, nor does it quantify how much deviation the low-level controller can tolerate before success drops. \n\n-The key-point/waypoint representation ignores force/torque constraints, finger–object contact states, and within-hand re-grasping. This abstraction works for the hammer, pliers, and bottle tasks shown, but tasks that hinge on precise in-hand posture changes (e.g., turning a screw two full revolutions) may require richer supervision than the scaffold can provide.\n\nQUESTIONS:\nRefer to the weakness section.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a two-stage method for dexterous manipulation. An off-the-shelf VLM serves as a high-level planner that generates keypoints or waypoints and synthesizes 3D trajectories, while a reinforcement-learning policy in simulation tracks the coarse trajectories. The experiments cover an eight-task suite in simulation, comparisons with several baselines, failure analysis, and ablation studies on the number of few-shot examples and waypoints. The Place Bottle and Slide Box tasks are also deployed on real robots.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1. Combining VLM and RL in simulation is reasonable. It shows that VLM-generated trajectories can solve articulation and precise manipulation tasks (although these are in simulation only, not real world).\n2. The simulation task suite contributes to the community.\n3. The failure analysis provides useful insights.\n4. The presentation quality is high; figures and writing are well polished.\n\n**Weaknesses:**\n\n1. The method is not compared with strong baselines. All baselines in Fig. 4 are self-crafted (not compared with SOTA methods mentioned in Section 2 \"Related Work\" such as RoboDexVLM and Iterative Keypoint Rewards). It is unclear how the proposed method differs from related works, and more importantly, whether it performs better.\n2. The RL is trained in simulation, which assumes access to a real-to-sim digital twin. This limits generalizability to new tasks and environments. Reproducing the experiment requires building a real-to-sim setup, which is not well discussed in the paper.\n3. Why are only simpler tasks deployed in the real world? What about articulation and precise manipulation, which are only shown in simulation?\n4. How is kinematic reachability handled when deploying in the real world? In simulation, a free-floating hand is used. Would there be many invalid trajectories generated by the VLM or RL due to this simplification?\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}