Manipulate-Anything: Automating Real-World Robots using Vision-Language Models

Authors: jiafei  duan
wentao  yuan
wilbert  pumacay
yi ru wang
kiana  ehsani
dieter  fox
ranjay  krishna
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-05-23 00:48:39

Abstract: Large-scale endeavors like RT-1 and widespread community efforts such as Open-X-Embodiment have contributed to growing the scale of robot demonstration data. However, there is still an opportunity to improve the quality, quantity, and diversity of robot demonstration data. Although vision-language models have been shown to automatically generate demonstration data, their utility has been limited to environments with privileged state information, they require hand-designed skills, and are limited to interactions with few object instances. We propose Manipulate-Anything, a scalable automated generation method for real-world robotic manipulation.
Unlike prior work, our method can operate in real-world environments without any privileged state information, hand-designed skills, and can manipulate any static object. We evaluate our method using two setups. First, Manipulate-Anything successfully generates trajectories for all 5 real-world and 12 simulation tasks, significantly outperforming existing methods like VoxPoser. 
Second, Manipulate-Anything's demonstrations can train more robust behavior cloning policies than training with human demonstrations, or from data generated by VoxPoser and Code-As-Policies.
We believe Manipulate-Anything can be the scalable method for both generating data for robotics and solving novel tasks in a zero-shot setting.  Anonymous project page: manipulate-anything.github.io.

Main paper: 
 introduction: 
 The success of modern machine learning systems fundamentally relies on the quantity [6, 7, 8, 9, 10, 11] , quality [12, 13, 14, 15, 16] , and diversity [17, 18, 19, 20, 21] of the data they are trained on. The availability of large-scale internet data made possible significant advances in vision and language [22, 23, 24] . However, the dearth of data has prevented similar advancements in robotics. Human demonstration collection methods do not scale to sufficient quantity or diversity. Projects like RT-1 [1] demonstrated the utility of high-quality human data collected over 17 months. Others have developed low-cost hardware for data collection [25, 26, 27] . However, all these procedures require expensive human data collection.Automated data collection methods do not scale to sufficient diversity. With the advent of visionlanguage models (VLMS), the robotics community has been abuzz with new systems that leverage VLMS to guide robotic behavior [28, 5, 29, 30, 3, 4, 31] . In these systems, VLMS decompose tasks into language plans [28, 5] or generate code to execute predefined skills [32, 3] . Though successful in simulation, these methods underperform in the real world [32, 3] . Some methods rely on privileged Figure 1 : MANIPULATE-ANYTHING is an automated method for robot manipulation in real world environments. Unlike prior methods, it does not require privileged state information, hand-designed skills, or limited to manipulating a fixed number of object instances. It can guide a robot to accomplish a diverse set of unseen tasks, manipulating diverse objects. Furthermore, the generated data enables training behavior cloning policies that outperform training with human demonstrations. state information only available in simulation [29, 4] , require hand-designed skills [30] , or are also limited to manipulating a fixed set of object instances with known geometric shape [3, 32] . As VLMs improve in performance, and with the vast common-sense knowledge they have shown to possess, could we harvest their capabilities for diverse task completion and scalable data generation?The answer is yes -with careful system design and the right set of input and output formulations, we can not only use VLMs as a means to successfully perform diverse tasks in a zero-shot manner, but also generate quality data at a high quantity to train behaviour cloning policies.We propose MANIPULATE-ANYTHING a scalable automated demonstration generation method for real-world robotic manipulation. MANIPULATE-ANYTHING produces high quality data, at large-quantities (if needed), and can manipulate a diverse set of objects to perform a diverse set of tasks. When placed in a real world environment and given a task (e.g., "open the top drawer" in Figure 2 ), MANIPULATE-ANYTHING effectively leverages VLMS to guide a robotic arm to complete the task. Unlike prior methods, it doesn't need privileged state information, hand-designed skills, or limited to specific object instances. Not relying on privileged information makes MANIPULATE-ANYTHING environment-agnostic. MANIPULATE-ANYTHING plans a sequence of sub-goals and generates actions to execute the sub-goals. It can verify whether the robot succeeded in the sub-goal using a verifier and re-plan from the current state if needed. This error recovery enables mistake identification, re-planning, and recovering from failure. It also injects recovery behavior into the collected demonstrations. We further enhanced the VLMs' capabilities by incorporating reasoning from multi-viewpoints, significantly improving performance.We showcase the utility of MANIPULATE-ANYTHING through two evaluation setups. First, we show that it can be prompted with a novel, never-before-seen task and complete it in a zero-shot manner. We quantitatively evaluate across 7 real-world and 14 RLBench [33] simulation tasks and demonstrate capabilities across many real-world everyday tasks (refer to appendix). Our method significantly outperforms VoxPoser [3] in 10/14 simulation tasks for zero-shot evaluation. It also generalizes to tasks where VoxPoser completely fails because of its limitation to specific object instances. Furthermore, we demonstrated that our approach can solve real-world manipulation tasks in a zero-shot manner, achieving a task-averaged success rate of 38.57%. Second, we show that MANIPULATE-ANYTHING can generate useful training data for a behavior cloning policy. We compare MANIPULATE-ANYTHING generated data against ground truth hand-scripted demonstrations as well as against data from VoxPoser [3], Scaling-up [4] and Code-As-Policies [5] . Surprisingly, policies trained on our data outperforms even human hand-scripted data on 4 out of 12 tasks and performs on par for 3 more when trained with PerAct [34] . Meanwhile, the baselines are unable to generate the training data for some of tasks. MANIPULATE-ANYTHING demonstrates the broad possibility of large-scale deployment of robots across unstructured real-world environments. It also highlights its utility as a training data generator, aiding in the crucial goal of scaling up robot demonstration data.Figure 2 : Manipulate Anything Framework. The process begins by inputting a scene representation and a natural language task instruction into a VLM, which identifies objects and determines sub-tasks. For each sub-task, we provide multi-view images, verification conditions, and task goals to the action generation module, producing a task-specific grasp pose or action code. This leads to a temporary goal state, assessed by the sub-task verification module for error recovery. Once all sub-tasks are achieved, we filter the trajectories to obtain successful demonstrations for downstream policy training. 
related work: 
 MANIPULATE-ANYTHING enables scaling of robotic manipulation data using VLMS. As such, we review recent efforts in 1) scaling manipulation data, and 2) applications of VLMS in robotics. Scaling manipulation data. When deploying vision and language-based control policies for realworld applications, a significant challenge revolves around acquiring data. Traditionally, a convenient avenue to collect such trajectories is through human annotations for action (i.e. through teleoperation) and language labeling [34, 35, 36] , however, this approach is limited in scale. To address this limitation and achieve autonomous scalability, prior works employ vision-language models or procedurally generate language annotations in simulated environments [4, 37, 38] . For action labels, strategies range from random exploration to learned policies [39] . While human egocentric videos are relevant, they lack action labels and require cross-embodiment transfer [40] . Another strategy involves model-based policies, such as task and motion planning (TAMP) [41] . Our approach extends these methods by incorporating common-sense knowledge from LLMs and VLMs, providing a framework that combines the strengths of VLMs, object pose prediction, and dynamic retry to synthesize demonstrations in simulated and real environments.Language models for robotics. In the field of robotics, large language models have found diverse applications, including policy learning [42, 43, 44] , task and motion planning [45, 46] , log summarization [47] , policy program synthesis [5] , and optimization program generation [28] . Previous research has also explored the physical grounding capabilities of these models [3, 32, 48, 49] , while ongoing work [4] investigates their integration with task and motion planners to create expert demonstrations. [36] attempted to collect extensive real-world interaction data, with short-horizon trajectories. [50] proposed a key-point based visual prompting method for real-world manipulation, through predicting affordances and corresponding motions. Our work complements the existing line of works by leveraging the high-level planning capabilities of language models, scene understanding capabilities of vision language models, and action sampling to enable synthesis of robot trajectories, which include language, vision, and robot state, given arbitrary tasks and environments. 
manipulate-anything: 
 We propose MANIPULATE-ANYTHING, a framework that solves everyday manipulation tasks conditioned on language. Under the hood, MANIPULATE-ANYTHING leverages VLMS to decompose tasks into sub-tasks, generates code for new skills or task-specific grasp pose, and verifies the success of each sub-task (Figure 3 ). Note that due to the modularity aspect of our framework, MANIPULATE-ANYTHING will continue to improve as the underlying VLMs continue to improve.3.1 Task plan generation MANIPULATE-ANYTHING takes as input any task described by a free-form language instruction, T (e.g., 'open the top drawer'). Creating robot trajectories that adheres to T is challenging due to its potential complexity and ambiguity, requiring a nuanced understanding of the current environment state. Given T, and an image of the scene, we apply a VLM to first identify task-relevant objects in the scene, appending them to a list. Subsequently, we use a LLM along with those information to decompose the main task T into a series of discrete, smaller sub-tasks, represented as T i , along with the corresponding verification conditions v i , where i ranges from 1 to n. For instance, the above task could be decompose into sub-tasks include 'grasp the drawer handle' or 'pull open the drawer', and verification conditions are 'did the robot grasp the handle?' or 'is the drawer opened?'. This transforms the instruction T into a sequence of specific sub-tasks{(T 1 , v 1 ), (T 2 , v 2 ), . . . , (T n , v n )}.For each sub-task, MANIPULATE-ANYTHING generates desired actions ( § 3.3) and verifies them against the corresponding conditions to ensure successful completion of that sub-task( § 3.4). This verification step also allows MANIPULATE-ANYTHING to recover from mistakes and attempt again in the case of failure.Many prior works that investigated VLM's reasoning capability found that they do not work well given a single viewpoint [51, 52, 53] . In robotic manipulation, we leverage multiple viewpoints from either more than one camera or re-rendering to minimize object occlusion [54, 55] . Leveraging these insights, we proposed a multi-viewpoint selection phase via VLM before using the selected viewpoints for either action generation or sub-task verification for MANIPULATE-ANYTHING Ẇe concatenate all available viewpoints from the current observations, either from multiple cameras or re-rendered viewpoints from a single RGB-D, into a single frame. Numbers are annotated on the top left of each concatenated frame to correspond to a specific viewpoint. Using the concatenated multi-viewpoints, we then query VLMS to choose an ideal viewpoint conditioned on the sub-task. Therefore, for both agent-centric and object-centric action generation, we render multiple viewpoints of the scene and query VLMS to choose an ideal viewpoint for either generating actions given the sub-task or verifying if the sub-task verification condition has been met as shown in Figure 3 .1Given a sub-task, the desired output from the action generation module is a sequence of low-level actions represented as a 6 DoF end-effector pose. The actions can be categorized into two sets: agent-centric or object-centric. Based on the generated sub-task, the LLM planner will classify it as either agent-centric or object-centric actions. For agent-centric actions, it will modify the agent's state; e.g., it can move the robot's end-effector from the current state (e.g., "rotate 90 • "). We first employed our multi-viewpoint selection to sample out the most optimal viewpoints to provide the VLM with along with the in-context learning technique to generate the code for synthesizing the desired motion. Unlike prior methods that use language models solely for code generation [5] , we leverage VLMS to reason about object locations and the scene, grounding generation in the current state. This advantage is demonstrated in the ablation studies in the Appendix.For object-centric actions, it is often used to generate a task-specific grasp pose for grasping a certain object (e.g., "grasp a knife for cutting") or to synthesize a pre-action pose for non-prehensile tasks by allowing the VLM to assign translation offsets to task-specific grasp poses. To obtain an objectcentric action for a given sub-task, we first use an object-agnostic grasp prediction model [56] to generate all possible 6-DOF grasping poses in the scene. These poses are not conditioned on task specifications and may include invalid grasp poses for the given task. Next, we use the VLM We show zero-shot demonstrations for 14 tasks in simulation and 7 tasks in the real world.to obtain the bounding box of the target object parts conditioned on the task specification from all available or re-rendered viewpoints (e.g., if the task is "grasp a knife," the VLM will detect the handle of the knife and generate a bounding box for the handle). The VLM then performs multi-viewpoint selection to identify the most optimal viewpoint free from occlusion. Finally, using the bounding box detection of the task-specific part of the object from the optimal viewpoint selected by the VLM, we filter out a list of proposed candidate grasp poses and select the highest confidence grasp pose. This approach allows us to obtain the most optimal task-specific grasp pose, placement pose, and pre-action pose for non-prehensile tasks, all leveraging the capabilities of the VLM. After the action is generated, a motion planner can be used to move the robot to the desired pose as detailed in Fig. 2 .To ensure that each sub-task T i is executed correctly, we introduce a VLM-based verifier. After every action for each sub-task is executed, we use the VLM to check if the end state matches the verifier condition v i . Similar to the action generation module, we use multi-viewpoint VLM selection to find the optimal view, avoiding errors due to occlusion or ambiguity from a single viewpoint. If the verifier identifies failure, we re-attempt the action generation step for the previous sub-task from the current state. Otherwise, the next sub-task T i+1 is attempted. More details are in the appendix 6.1. 
experiments: 
 Our experiments are designed to address two questions: 1) Can MANIPULATE-ANYTHING accurately solve a diverse set of tasks in a zero-shot manner? 2) Can data generated from MANIPULATE-ANYTHING be used to train a robust policy? Implementation details. We use both GPT-4V and Qwen-VL [57] as our VLM. We use GPT-4V for task decomposition, action generation, and verification. We use Qwen-VL to detect and extract object information. To ensure zero-shot execution within a reasonable budget, we limit the number of action steps in each trajectory to 50 and the verification module allows a maximum of 30 tries to accomplish a sub-goal. For the task plan generation, we follow the prompting structure adapted from ProgPrompt [28] . All prompts input into the VLM are accompanied by few-shot demonstrations [58] . Additionally, we provide three manually curated primitive action code snippets as examples to prompt the VLM for new action code generation. Full prompts are included in the Appendix. We use four viewpoints M 4 = [f ront, wrist, lef t_shoulder, right_shoulder] for the simulation experiments, and re-render three viewpoints for the real-world experiments [54] . For better reasoning by the VLM, we use a resolution of 256 × 256.We empirically study the zero-shot capability of MANIPULATE-ANYTHING in solving 14 diverse tasks in simulation, covering a wide range of task configurations and action primitives for both prehensile Table 1 : Task-averaged success rate % for zero-shot evaluation. MANIPULATE-ANYTHING outperformed other baselines in 10 out of 14 simulation tasks from RLBench [33] . Each task was evaluated over 3 seeds to obtain the task-averaged success rate and standard deviations. and non-prehensile tasks. Our simulation experiments are reported to ensure reproducibility and provide a benchmark for future methods.Environment and tasks. The simulation setup involves a Franka Panda robot with a parallel gripper, using CoppeliaSim and PyRep for interfacing. Four RGB-D cameras capture input observations around a tabletop environment. RLBench [33] is used as the task benchmark, with 14 sampled tasks that cover various action primitives, task horizons, and object position variations. The robot's actions are represented as waypoints, with trajectories computed and executed via a motion planner [59] . Baselines. We compare against three state-of-the-art zero-shot data generation approaches: Code-as-Policies (CAP) [5] , Scaling-up-Distilling-Down (Scaling-up) [4] and VoxPoser [3] . CAP uses language models to generate programs that call hand-crafted primitive actions, while VoxPoser predicts waypoints via a 3D voxel map of value functions. Scaling-up leverages an LLM with 6 DoF exploration primitives to generate robotic data for policy distillation. We provided CAP and Scaling-up with ground truth simulation states and object models, and VoxPoser with segmented object point clouds, which inherently disadvantages MANIPULATE-ANYTHING in comparison.Results: MANIPULATE-ANYTHING can generate successful trajectories for all 14 tasks while Scaling-up, VoxPoser and CAP cover only 10, 9 and 7 tasks, respectively (Table 1 ). MANIPULATE-ANYTHING outperforms the baselines in 10 out of the 14 tasks. The three lowest-performing tasks by MANIPULATE-ANYTHING are non-prehensile or complex fine-grained manipulation tasks that require generating new primitive actions with precise parameters. VoxPoser fails in the tasks that require moving the arm beyond 4-DoF. MANIPULATE-ANYTHING outperforms the strongest baseline, VoxPoser, by an average task-averaged margin of up to 22%.Next, we analyze the quality of the generated data by comparing the success rates of behavior cloning models trained with the data. Zero-shot methods like MANIPULATE-ANYTHING are computationally expensive but hold the potential to generate useful training data. To evaluate the quality and effectiveness of the generated training data, we use the methods described in the previous section to generate data for each task. We also compare performance against a model trained on human-generated demonstrations across the 12 tasks. We use the data to train behavior cloning policies. Data generation details. We generate 10 successful demonstrations per task. We use the system's success condition to filter for successful demonstrations. Each of the demonstrations consist of a language instruction, RGB-D frames for the trajectory, and waypoints represented as 6 DoF gripper poses and states. For the tasks that the baselines were unable to generate any successful demonstrations, we patched the missing training data with RLBench system-generated demonstrations.Training and evaluation protocol. We train two models using the generated demonstrations: the Perceiver-Actor (PerAct) model [34] , a transformer-based robotic manipulation behavior cloning model that expects tokenized voxel grids and language instructions as inputs and predicts discretized voxel grid 6 DoF poses and gripper states, and RVT-2 model [60] , a multi-view transformer-based BC model. The RVT-2 model uses tokenized image patches and CLIP-encoded language instructions as input to predict keypoint actions as translation heatmaps, discretized rotation in Euler angles, and the gripper's binary state. Notably, RVT-2 is currently the highest-performing model on the RLBench benchmark. For all the generated training datasets, we train a multi-task PerAct policy with a batch size of 4 for 30k iterations on a single RTX A100, and RVT-2 with a batch size of 24 for 3.3k iterations on 4 A100s. To ensure consistent evaluation, we generate one set of testing environments with RLBench. We evaluate the last checkpoint from each of the trained policies. Each policy is evaluated for 25 episodes across each task using 3 different seeds. We measure the success rate based on the simulation-defined success condition. Details on RVT-2 results can be found in the Appendix.Results: Policies trained using MANIPULATE-ANYTHING data perform similarly to policies trained using hand-scripted demonstrations (p = 0.973) for PerAct (Table 2 ). Training on either MANIPULATE-ANYTHING or hand-scripted demonstrations results in a performance difference of just 0.27% average across all tasks. Furthermore, models trained on data from the baselines exhibit a statistically lower performance (p ≤ 0.01 for VoxPoser, Scaling-up and CAP). One of the main factors potentially contributing to the performance differences could be that MANIPULATE-ANYTHING generates diverse expert trajectories preferable to humans. This is shown in Fig. 11 , which illustrates the action distribution of generated data by different methods for the same given tasks. We also observed that the policy trained on MA data achieves a lower standard deviation of 7.19 average across all tasks, compared to the zero-shot performance standard deviation of 8.81. This suggests the benefits of training over generated data instead of relying solely on zero-shot deployment.Environment and tasks. We employ a Franka Panda manipulator equipped with a parallel gripper. We use a front-facing Kinect 2 RGB-D camera. To generate multi-view inputs for the MANIPULATE-ANYTHING framework, we re-render virtual viewpoints from the generated point cloud, similar to prior work [54] . We selected 7 representative real-world tasks, both prehensile and non-prehensile: open_jar, sort_objects, correct_dices, open_drawer, on_lamp, press_switch, and close_laptop, all conditioned on language instructions. Each task was evaluated over 10 episodes with varying object poses across 3 trials. Data generation details. We used MANIPULATE-ANYTHING to generate 6 demonstrations for each task and manually perform scene resets when failures occur. We train a similar multi-task PerAct for 120k iterations and evaluate the trained policies in a manner similar to the zero-shot experiments.Results: MANIPULATE-ANYTHING is able to generate successful demonstrations for each of the 7 real world tasks. Even for the worst-performing task, MANIPULATE-ANYTHING achieves a success rate of more than 25%. Our approach outperforms CAP by 38%. Consistent with the simulation results, training with the data generated by MANIPULATE-ANYTHING produces a more robust policy compared to performing zero-shot. Additionally, in 4 out of 5 tasks, the trained policies perform better than the zero-shot approach. The policy under-performs on the sort_object task, because it requires longer-horizon memory -a known limitation pointed out in PerAct [34] .For effective real-world deployment of MANIPULATE-ANYTHING, it's crucial that the collected data supports scaling of robotics transformers and offers diverse skills and interacted objects. We conducted an ablation study to evaluate the quality of MANIPULATE-ANYTHING-generated data for scaling and its generalization to language instruction changes. For scaling, we generated behavior cloning data, ranging from 1 to 100 training demonstrations from RLBench and MANIPULATE-ANYTHING for a single task (put_block), and trained a PerAct policy. For generalization, we varied the sort_mustard task with different language instructions and target objects. We compared our approach to VoxPoser to assess robustness to object and language instruction changes. Further implementation details are in the supplementary materials. Result: Our scaling experiments demonstrate that generating more training data via MANIPULATE-ANYTHING improves PerAct policy performance (Fig. 4 ). The data from our approach shows a better rate of change with a slope of 0.503 for a linear fit, compared to 0.197 for RLBench-generated data. Additionally, MANIPULATE-ANYTHING data is more generalizable and robust to language instruction changes, outperforming VoxPoser in task success across language and object variations. Detailed results in the appendix. 
discussion: 
 Limitations. Despite promising results, MANIPULATE-ANYTHING has limitations: reliance on large language models (VLMS), which may lessen with the development of open-source VLMS; inability to generate alternative grasp poses for dynamic manipulation and non-prehensile tasks; potential compounding errors from integrating multiple VLMs, possibly mitigated by emerging specialized VLMs; and the need for manual prompt engineering for in-context learning, with advancements in alignment and prompting techniques potentially reducing this effort. Conclusion. MANIPULATE-ANYTHING is a scalable, environment-agnostic approach that leverages VLMS for high-level planning, scene understanding, and error recovery to generate zero-shot demonstrations for robotic tasks without privileged environment information. This results in high-quality data for behavior cloning, outperforming human data. 
manipulate-anything implementations: 
 Action Generation Module. We generate each action using either an agent-centric or object-centric approach. For object-centric action generation, we utilize M2T2 [56] , NVIDIA's foundational grasp prediction model, for pick and place actions. For 6-DoF grasping, we input a single 3D point cloud from either a single RGB-D camera (in the real world) or multiple cameras (in simulation). The model outputs a set of grasp proposals on any graspable objects, providing 6-DoF grasp candidates (3-DoF rotation and 3-DoF translation) and default gripper close states. For placement actions, M2T2 outputs a set of 6-DoF placement poses, indicating where the end-effector should be before executing a drop primitive action based on a VLM plan. The network ensures the object is stably positioned without collisions. We also set default values for mask_threshold and object_threshold to control the number of proposed grasp candidates. After proposing a list of template grasp poses, we use QWen-VL [57] to detect the target object by prompting the current image frame with the target object's name, translated into Chinese using a machine translation model [61] . This detection is applied to all re-rendered viewpoints or viewpoints from different cameras. We then concatenate these frames into a single image, annotating each sub-image with a number at the top right corner. Next, we call the GPT-4V API with few-shot demonstrations and the task goal to prompt GPT-4V to output the selected number of viewpoints that provide the most unobstructed views for sampling the grasp pose to achieve the sub-goal. Using the selected viewpoint, we execute the grasp by moving the end-effector to the sampled grasp pose via a motion planner.For agent-centric action generation, we first perform the same steps of viewpoint selection. Using the selected viewpoint, a few demonstration examples, and the sub-goal, we prompt GPT-4V to generate an action function with code snippets that include the necessary code to perform a delta-action on the current robot pose. We then execute this by moving the end-effector based on these delta changes. This process is iterated until we obtain the most desirable code snippet function for the given sub-goals, which is then appended to a skill library for future use.The sub-goal verification module helps with error recovery by ensuring all potential attempts at resolving the current step action have been tried. With the temporary goal state obtained by the action generation module, we use multi-viewpoints to sample the optimal viewpoint for answering the verification condition generated for the given sub-goal during the task plan generation phase. Using the same viewpoint selection method as in the Action Generation Module, we obtain the optimal view and then perform a two-step sequence rollout of frames: one from the current frame at this viewpoint and another from the previous action step. We concatenate these two frames, annotate them with numbers to indicate their temporal relation, and use this image to prompt GPT-4V to check if the verification condition is fulfilled as shown in Fig. 5 . If the answer is Yes, we proceed to the next sub-goal. If the answer is No, we resample new viewpoints, generate new actions, and reattempt the entire sub-goal with a different seed.Simulation Setup. All the simulated experiments use a four-camera setup as illustrated in Fig. 7 . The cameras are positioned at the front, left shoulder, wrist, and right shoulder. All cameras are static, Figure 5 : Sub-goal Verification Module. We used viewpoint selection similar to action generation, to find the optimal viewpoints, and roll out the two step sequences of the previous and current frames for prompting the verification condition. except for the wrist camera, which is mounted on the end effector. We did not modify the default camera poses from the original RLBench [33] . These poses maximize coverage of the entire table, and we use a 256 x 256 resolution for better input to the VLMs.Task Details. We describe in detail each of the 12 tasks for simulation evaluation, both for trained policies and zero-shot methods, along with their RLBench variations and success conditions. We have made some modifications to the original tasks to enhance the detection rate by Code-As-Policies and VoxPoser. Table 4 : Behavior Cloning with different generated data. The behavior cloning policy trained on the data generated by MANIPULATE-ANYTHING provides the best performance on 10 out of 12 tasks compared to the other autonomous data generation baselines. We report the Success Rate % for behaviour cloning policies trained with data generated from VoxPoser [3] and Code as Policies [5] in comparison. Note that the RLBench [33] baseline uses human expert demonstrations and is considered an upper bound for behavior cloning.Figure 7 : Results for visual prompting techniques (Left).We reported the various results for different visual prompting technique decision, and reported that selected viewpoint sequence yield the best performance.Simulation scene setup (Right). We leverage 4 different camera for evaluation.Figure 8 : Real-world experiment setup (Left). We set up the real-world using this configuration. Robustness and generalization evaluation. We evaluated MANIPULATE-ANYTHING against VoxPoser for capability in generalizing to different language instructions and also object-specific manipulation.The real-robot experiments use a Franka Panda manipulator with a parallel gripper. For perception, we use a Kinect-2 RGB-D camera mounted on a tripod, at an angle, pointing towards the tabletop. Kinect-2 provides RGB-D images of resolution 512 × 424 at 30Hz. The extrinsic between the camera and robot base-frame are calibrated with the easy hand-eye package. We use an ARUCO AR marker mounted on the gripper to aid the calibration process, as shown in Figure 8 .Beyond the five real-world experiments used for systematically evaluating MANIPULATE-ANYTHING, we also have additional real-world demonstrations generated in a zero-shot manner via MANIPULATE-ANYTHING. These demonstrations cover a range of tasks, from reasoning tasks to more precise everyday tasks. All of the tasks can be seen in Fig. 10 . We also demonstrates that MANIPULATE- For most of the baselines we followed the original implementation with minor modifications. For the Code as Policies baseline we re-implemented most of the environment code using PyRep instead of PyBullet. This includes the implementation of various motion primitives that form thePrompts used for multi-viewpoint VLM selection, task plan generation, action generation and sub-task verification can be found below.• Multi-viewpoint VLM (Verification): Takes in 4 images concatenated into a single frame with number annotated on top and returns a selection number of the most optimal viewpoint for verfying the sub-task. Simulation: multi-viewpoint-VLM-selection.txt• Multi-viewpoint VLM (Object centric action): Takes in 4 images concatenated into a single frame with number annotated on top and returns a selection number of the most optimal viewpoint for filter the grasp candidate poses. Simulation: Multi-viewpoint VLMverification-object-centric.txt• Task plan generation: Takes in a natural language instruction, and outputs a task plan in json file with the correct format. Simulation: task-plan-generation-prompt.txt• Sub-task verification: Takes in the selected viewpoint rollout along with the verification condition, and outputs binary 'yes' or 'no'. Simulation: sub-task-verification-prompt.txt• Action-Generation: Takes the action primitive and generates codes for executing the action in simulation. Simulation: action-generation.txtWe ran MANIPULATE-ANYTHING with multi-processing during simulation to obtain the best set of task plans and action primitives for any given task. This set of information is then used to generate more data at scale for distilling a policy. We have compiled all the task plan JSON files and action primitive code that achieved the highest success rates in these tasks.Best Task Plans: best-task-plans.txt Skills Library: skills-library.txt 
 
 Appendix:  


Decision: The paper presents a framework for manipulating objects by combining VLMs and foundation models to help plan and execute tasks.

Strengths: The reviewers agree on the importance of the problem setup. They also acknowledge the value of the real-world evaluation.

Weaknesses: The main criticism lies in the technical novelty of the method and its relationship to prior works, while concerns about overclaiming with the title. Improving the clarity and reproducability of the paper is suggested by  3U2k, Ssao and 1Hnu suggest stronger baseline methods among others.

Post response period: The authors have addressed most of reviewer's concerns. In particular, they have added baselines, more experiment videos and improved the overall clarity of the presentation. However, being able to manipulate any object still seems to be overclaiming. Given the author response, the updated ratings and their justifications, the AC recommends the paper to be accepted with a poster presentation.

Reviews: review: title: title: Well motivated, but mainly hampered by writing issues and fewer experiments than the claims imply.
paper_summary: paper_summary: The authors propose Manipulate-Anything, a framework for generating data for robot manipulation which uses VLMs to help plan and execute tasks to generate data for policies to learn from.
main_review: main_review: ## Strengths

**Results:** Results are good, demonstrating significant gains over prior work in most of the tasks evaluated.

**Motivation:** This paper is well motivated, targeting *quantity, quality, and diversity* of training data. These are increasingly more important problems to tackle as models scale in size.

**Method:** The authors sensibly combined many existing frameworks together into one pipeline for data generation of robot manipulation data: grasping networks, VLMs, code generation, etc. 

**Real world experiments:** It’s always great to see real world experiments at CoRL. These experiments are not vacuous and do demonstrate the ability of Manipulate Anything to actually work in real world tasks.

## Weaknesses

**Clarity Issues:** 

- In the introduction, I don’t really get a sense of a “key idea” when reading. The 3rd paragraph contains many things that Manipulate-Anything enables but I am not sure what key insight drives the method to be better than the prior works mentioned in the 2nd paragraph. This isn’t a huge issue, but it would make the paper more clear.
- Figure 3: Everything is quite small yet there’s a lot of whitespace that could be filled.
- L124-131: Sentences in this paragraph seem a little disjoint, it could be re-written to flow more smoothly.
- L134: This multiple-viewpoint selection process seems like an important detail, perhaps a reference to a specific appendix section where this is discussed, or even better, a bit more detail here would be good (e.g., what does the prompt look like? how does the VLM choose?  etc.)
- Section 3.3: how does the multi-view VLM reasoning module work? What is the input/output? There is a lack of detail that doesn’t require that much extra space to fill and this seems too important to the method to be left to the appendix.
    - Following up on this, the appendix doesn’t actually contain that much detail for these method implementations. Example prompts/example responses are pretty standard in LLM for robotics papers, these should be included.
- Following up on the previous point, without the sense of a “key idea,” the method section is a little bit hard to understand as many low-level details are understandably left out due to space concerns. But, for example, if the key idea was to use VLMs to look at multiple viewpoints, then questions about minor implementation details that readers would have regarding other parts of the method would be less pressing. It would be good to rewrite just a little bit of the methods section overview and introduction to highlight a major idea which underpins the entire method and distinguishes it from all prior work, add extra details about this, and then according to space considerations other parts can be moved to the appendix as needed.

**Baselines:** VoxPoser and Code-as-Policies are both not really made for data generation. I think they’re valid comparisons, but likely not the best comparison for data generation for training policies. A better comparison would be https://arxiv.org/abs/2307.14535, which the authors already do cite in the related works but don’t really distinguish their method from too convincingly. I’m not specifically asking for this in the rebuttal period, but this does seem much better of a baseline.

**Experiments:** The claims made in the introduction about the *diversity* and *quantity* of possible data collected aren’t really supported by the somewhat limited set of experiments:

- There’s more than 12 tasks alone in RLBench, why just these 12 tasks?
- How about non-prehensile tasks?

**Overclaims:** 

- There’s a claim of manipulating “any” static object but this isn’t really backed up by the experiments. I would remove this specific statement (abstract).
- In comparing to prior work: L32: Open-X contains *much* more than 20 tasks…. What is the definition of task here? If it’s the same as Manipulate Anything uses, then it should be far more than 20.

**Minor Issues:** 

- L30 “human collection” → “human data collection”
- L34: “abuz” → “abuzz”
- L38: “.” → “,”
- L105: “We” → “we”
- Readability would be improved if appendix references also contained the actual sections they refer to.
- Some extra related work that seems quite relevant:
    - LLMs + offline RL for scaling manipulation data: https://clvrai.com/sprint
    - LLMs + motion planners for robotics to learn new tasks: https://mihdalal.github.io/planseqlearn/
    - Collecting demos with LLMs/VLMs: https://auto-rt.github.io/ — this paper is actually very relevant and should be contrasted against specifically in the related works.
    - LLMs +  RL for robotics to learn new tasks: https://clvrai.github.io/boss/
    - LLMs/VLMs for new task learning: https://rlvlmf2024.github.io/
    - Instruction augmentation with VLMs for data generation: https://instructionaugmentation.github.io/
questions: questions: Integrated in the review, please see the review.
limitations: limitations: 2
review_summary: review_summary: I recommend a weak reject because the paper, while well-motivated and with some good results, suffers from writing issues and lacks comprehensive experimental validation to fully support its claims. Addressing these concerns would significantly strengthen the work and change my score.
title: title: A framework utilizing VLMs and a grasp prediction model to address tasks within the limit of TAMP, while "an automated method for robot manipulation in real world environments" sounds like an overclaim
paper_summary: paper_summary: The paper proposes a framework, named "Manipulate-Anything", which leverages VLMs to plan a sequence of sub-goals (text) and an off-the-shelf grasp prediction model to generate grasp poses addressing sub-goals. A sub-goal verification module is introduced to support retry behaviors. It addresses some technical limitations in prior works like VoxPoser and Code as Policies.
main_review: main_review: ## Strengths
1. The paper introduces using an off-the-shelf grasp prediction model to improve "action generation", which is commonly needed in similar frameworks relying on LLM/VLM for high-level or mid-level planning. The authors also introduce a sub-goal verification module to enhance "action generation".

## Weaknesses
1. It sounds like an overclaim that "Manipulate-Anything" is "a scalable automated generation method for real-world robotic manipulation". However, it can not manipulate "anything", even potentially. The existing frameworks that use LLMs or VLMs to generate plans or actions are mostly limited to which actions can be done. The ability to understand (detect or grasp) the concept of objects ("anything") is usually what LLMs or VLMs can provide, rather than the ability to "manipulate" objects. The tasks tackled in the paper are in general limited to quasi-static manipulation.
2. In my own opinion, the framework seems to be an LLM/VLM-based version of PDDLStream [1]. The error recovery only happens for trying different grasp poses, which is similar to sampling grasp poses in PDDLStream. The authors also agree that their method extends prior methods like TAMP in L79-L80. Thus, it is expected to see more quantitative comparison with TAMP methods. However, such comparison in experiments is missing. Although TAMP methods relies on privileged states in general, it seems not to be hard to replace privileged states with predicted states in the tasks evaluated in this paper. And there may exist prior works (see [2] for reference) that support online-planning (retry) or uncertainty.
3. The authors emphasize "real-world robotic manipulation" and "automated". However, main experiments are done in simulation and some important pain points in the real world are not addressed. For example, resetting environments is notoriously known as a bottleneck for real-world data collection, and how to recover from exploration or planning failure is not discussed in this paper. Besides, (correct me if wrong) RLBench does not fully simulate the dynamics (the grasp behavior is implemented as a constraint when the object is close enough to the gripper based on PyRep's proximity sensors). It makes the simulated experiments in the paper less useful for real-world applications.
4. The paper addresses some technical limitations of prior works, while some of them are hard to considered as a contribution. For example, VLMs might not be powerful enough when CAP and VoxPoser were done.
5. More related works can be included, like [3] and [4], for quantitative comparison.

- [1] Caelan R. Garrett, Tomás Lozano-Pérez, Leslie P. Kaelbling. PDDLStream: Integrating Symbolic Planners and Blackbox Samplers via Optimistic Adaptive Planning, International Conference on Automated Planning and Scheduling (ICAPS), 2020.
- [2] Garrett, Caelan Reed, et al. "Integrated task and motion planning." Annual review of control, robotics, and autonomous systems 4.1 (2021): 265-293.
- [3] Arenas, Montserrat Gonzalez, et al. "How to prompt your robot: A promptbook for manipulation skills with code as policies." Towards Generalist Robots: Learning Paradigms for Scalable Skill Acquisition@ CoRL2023. 2023.
- [4] Di Palo, Norman, and Edward Johns. "Keypoint Action tokens enable in-context imitation learning in robotics." arXiv preprint arXiv:2403.19578 (2024).

## Typos

There exist some typos in the paper.
1. L47: "Thus it can easily generalized to the real world."
2. Fig 4 caption: "12 tasks in simulation 5 tasks in the real world"
3. Fig 5 caption: "thehuman-generated"
4. L216: "tozero-shot"
questions: questions: 1. It is unclear how the candidate grasp poses are filtered in L128-L129. Especially, how the grasp poses are represented for VLMs to understand (e.g., by text or visual)?
2. What few-shot demonstrations are used for in-context learning to select viewpoints (L23 in Appendix)? Are they the same for all tasks, or are task-specific demonstrations needed?
3. The authors mention several times "avoiding errors due to occlusion or ambiguity from a single viewpoint". However, if the image is re-rendered from the same point cloud (perhaps partial), it could not handle occlusion caused by the camera setup (e.g., 1 front camera in the real-world experiments). Choosing a different viewpoint may help VLMs, but can the authors use quantitative evidence to support the claim for avoiding errors due to occlusion?
4. For re-attempting the action generation step, what will happen if no actions lead to success? In L145-L146, it is said that "the next sub-goal is attempted", but it does not sound convincing if the skipped sub-goal is essential for the full task. And what will happen if an object is already grasped by the end-effector but with a "bad" grasp pose (e.g., the blade of a knife is held and thus the knife can not be inserted into the holder while VLMs happen to fail to find this issue when handling the previous sub-goal)?
5. Can the authors also use the ground truth simulation state information for zero-shot experiments in Sec 4.1? It can indicate how accurate the information of the target object’s asset names or positions is needed. 
6. What are "human-generated demonstrations" in Sec 4.2? Are they tele-operated, or generated by hand-crafted scripts (e.g., provided by RLBench)? According to Table 2, it seems to be the latter.
7. Only PerAct is used for behavior cloning experiments while its limitations are also noticed by authors (e.g., poor for long-horizon memory). It is better to add more methods like ACT or Diffusion Policies to provide more comprehensive and convincing comparison.
8. Why is VoxPoser not included in Table 3? For 6 demonstrations generated by Manipulate-Anything (MA), do the authors repeatedly run MA until 6 demos are collected?
9. Which single task (L247) is used for the ablation study on scaling? Or is the success rate in Figure 6 averaged over 12 simulated tasks?
10. Can the authors provide error breakdown (like Sec 4.4 in VoxPoser) to analyze which aspects the proposed method improve over baselines?
limitations: limitations: 1
review_summary: review_summary: The paper shows some technical improvements over prior works, while the framework and experiments fail to show that it can be scaled up.
title: title: Impressive results; writing needs improvement
paper_summary: paper_summary: This paper proposes a VLM-based method for manipulation and then uses the method to generate data for behavioral cloning.
main_review: main_review: ### Strengths
* The paper results are quite impressive. I’m especially impressed with the performance given that object models are not needed.
* The work is certainly relevant and is likely to be of considerable interest to the CoRL audience.
* The scope of the paper is ambitious. As discussed below, I think the writing needs to be improved, but the experiments are largely successful in achieving the ambitious scope.
* Among other clever choices, I like the idea to let the VLM decide which camera view to use. (I am not sure if this is an original idea, but I haven’t seen it before.)
* I also appreciated the ablation studies. There could be even more of them.

### Weaknesses
* My main concerns are related to clarity and reproducibility. Several important details are unclear; I will ask about them in the Questions section below. Also, no code was provided with the submission (but certain Python files are referenced in Appendix Section 2).
* A secondary concern is novelty. The paper ultimately joins together existing methods. But the experiments are sufficiently thorough and impressive to override this concern---it is a good systems paper.
* The name “Manipulate Anything” is likely to frustrate people in the manipulation community. The tasks considered in this paper are a small subset of manipulation. (I know that the name is an homage to Segment Anything and that other recent papers have followed this pattern.)
* One of the claimed advantages with respect to baselines is that MANIPULATE-ANYTHING doesn’t need “hand-designed skills”. I think this is overclaimed. The object-centric actions here are certainly hand-designed skills (that happen to be implemented with a grasp foundation model.) The agent-centric skills are arguably less hand-designed, but there is not enough information presented to assess whether the “program policies” here are meaningfully more sophisticated than the primitive actions. For example, does pulling out a shelf just involve one primitive action? Can that really be called a “skill”?
* The limitations section is extremely weak. The proposed approach is fairly involved; a number of design choices are made; and each choice has potential weaknesses. I know space is limited, but an extended limitations section could go in the appendix.
* The authors could have presented this work as just an open-world manipulation approach, but they instead describe it as that _and_ a data generation approach. I like this in theory, but I think the current execution is not entirely successful:
   * The justification is buried: “L188: Zero-shot methods like MANIPULATE-ANYTHING are computationally expensive but hold the potential to generate useful training data.” This should be said earlier.
   * Also, the computational expense of MANIPULATE-ANYTHING is not actually measured or discussed in detail.
   * As the paper says, one of the requirements for a data generation approach is that the data are diverse. But it is never really explained why MANIPULATE-ANYTHING would produce more diverse data than the baselines.
   * This more-involved narrative means that there is less room in the paper to clearly describe the main methods. It also means that some of the impressive aspects of the work are buried.
   * Readers of this paper may ultimately be confused about whether they should use MANIPULATE-ANYTHING as a general manipulation approach, or instead, as a data generator for some learning-based manipulation approach.
* Some of the videos on the supplemental website have issues:
   * One of the first videos is labeled “uncap the jar”, but the robot instead turns on a light. Is this a video labeling bug, or a failure example? It is presented as though it is meant to be a success.
   * The videos are not annotated with timing markers and appear to be highly edited.
   * The simulated version of “close box” seems to “jump” in such a way that avoids the robot needing to actually carefully release the box lid (which is an interesting aspect of the manipulation problem.)
   * The “pick up cup” version of PerAct (with MANIPULATE-ANYTHING) shows a physically unrealistic grasp. Other videos have similar issues.
   * Some of the videos appear to suddenly jitter to other scenes after the task is complete.
* The paper figures could be improved.
   * Figure 1 is not very informative and it takes up a lot of room. I didn’t get anything from it that wasn’t clear from the text. Also, when I first started looking at the paper, I did not realize that the “existing robot data generator” panel represents baselines; I thought it was part of the proposed approach.
   * Figure 2 is more informative but could be polished.
   * Figure 3 is largely redundant with the bottom half of Figure 2.
* I am glad that most of the experiments were run over multiple seeds, but 3 is insufficient.

### Minor
* L34: “abuz” → abuzz
* L38: extraneous period after [29]
* L47: “Thus it can easily generalized to the real world” grammar
* L48: “excute”
* Figure 1 caption: “priviledged”
* Check the rest of the paper carefully for typos and grammatical issues; there are many others.
* The paper says “zero-shot” so many times -- something like 17 times. L54 says “...  it can be prompted with a novel, never-before-seen task and complete it in a zero-shot manner.” Three different ways of saying the same thing!
questions: questions: 1. In-context learning is used in multiple places. For each, what are the examples used for the context? Are these examples always the same, or are they task-dependent? How different from the test tasks are they?
2. The paper occasionally refers to a “target object” as part of the method. Is there an assumption that each natural language task description explicitly mentions one target object? If not, where do the target objects come from?
3. The appendix says that code snippets are “appended to a skill library for future use.” Is this talking about within a single task? What is the “future use” here?
4. L131 refers to a “detected bounding box” -- where in the pipeline is this bounding box detected?
5. How does the approach decide whether to use an object-centric or agent-centric action?
6. How many actions per subgoal are generated?
7. The paper mentions “recovery behavior into the collected demonstrations.” Does that mean that the entire collected trajectory is used for behavioral cloning? So the learned policy will also imitate the parts of the trajectory that lead to subgoal verification failures?
8. Which aspects of MANIPULATE-ANYTHING are responsible for generating diverse data?
9. Can you discuss and/or analyze the common failure modes of MANIPULATE-ANYTHING?
limitations: limitations: 2
review_summary: review_summary: Post rebuttal: the paper still has room for improvement in clarity, but I think it is above the bar for publication.