Manipulate-Anything: Automating Real-World Robots using Vision-Language Models
Authors: Jiafei Duan
Wentao Yuan
Wilbert Pumacay
Yi Ru Wang
Kiana Ehsani
Dieter Fox
Ranjay Krishna
Venue: CoRL 2024 (2024)
Keywords: Robot Learning; Multimodal Large Language Model; Data Generation; Imitation Learning; Behavior Cloning
TL;DR: A scalable automated demonstration generation method leveraging on Vision-Language Models for real-world robotics manipulation

Abstract: Large-scale endeavors like RT-1 and widespread community efforts such as Open-X-Embodiment have contributed to growing the scale of robot demonstration data. However, there is still an opportunity to improve the quality, quantity, and diversity of robot demonstration data. Although vision-language models have been shown to automatically generate demonstration data, their utility has been limited to environments with privileged state information, they require hand-designed skills, and are limited to interactions with few object instances. We propose Manipulate-Anything, a scalable automated generation method for real-world robotic manipulation.
Unlike prior work, our method can operate in real-world environments without any privileged state information, hand-designed skills, and can manipulate any static object. We evaluate our method using two setups. First, Manipulate-Anything successfully generates trajectories for all 5 real-world and 12 simulation tasks, significantly outperforming existing methods like VoxPoser. 
Second, Manipulate-Anything's demonstrations can train more robust behavior cloning policies than training with human demonstrations, or from data generated by VoxPoser and Code-As-Policies.
We believe Manipulate-Anything can be the scalable method for both generating data for robotics and solving novel tasks in a zero-shot setting.  Anonymous project page: manipulate-anything.github.io.

Manipulate-Anythinn A
Automating Real-World Robots using Vision-Language Models
Jiafei Duan 1+ Wentao Yuan
Wilbert Pumacay 2 Yi Ru Wang
Kiana Ehsani 3 Dieter Fox 1.4 Ranjay Krishna 1,3
University of Washington Universidad Católica San Pablo 3 Allen Institute for Artificial Intelligence "NVIDIA
Abstract: Large-scale endeavors like RT-1(1] and widespread community efforts such as Open-X-Ennt Dpen-XXEmmoodmmnt [2] have contributed to growing the scale of robot demonstration data. However, there is still an opportunity to improve the quality, quantity, and diversity of robot demonstration data. Although lasion-language models have been shown to automatically generate demonstration data, their utility has been limited to environments with privileged state information, they require hand-designed skills, and are limited to interactions with few object instances. We propose CONPUULATE-ANYTIEE
scalable automated generation method for real-world robotic manipulation. Unlike prior work, our method can operate in real- world environments without any privileged state information, hand-designed skills, and can manipulate any static object. We evaluate our method using two setups. First,
successfully generates trajectories for all 7 real- world tasks and 14 simulation tasks, significantly outperforming state-of-the-art methods such as VoxPoser. Second, can train more robust behavior cloning policies than training with human demonstra- tions, or from data generated by VoxPoser [3], Scaling-up up [4] and Pollcies [5]. We believe
CAAPPULATE-ANTHIII
demonstrations
can be the scalable method for both
generating data for robotics and solving novel tasks in a zero-shot setting. Project page: robot-ma. mabot-ma.githubii.
Keywords:
Zero-shot data generation, vision-language models, robot skill genera- tion, behavior cloning, and robotic manipulation
Introduction
The success modern machine learning systems fundamentally relies on the quantity [6, 7, 8, 10, 11]. quality [12, 13, 14, 15, 16], and diversity [17, 18. 19, 20, 21] of the data they are trained The availability of large-scale internet data made possible significant advances in vision and language [22, 23, 24). However, the dearth of data has prevented similar advancements in robotics. Human demonstration collection methods do not scale to sufficient quantity or diversity. Projects like RT-1 [1] demonstrated the utility of high-quality human data collected over 17 months. Others have developed low-cost hardware for data collection [25, 26, 27]. However, all these procedures require expensive human data collection.
Automated data collection methods do not scale to sufficient diversity. With the advent of vision- language models (VLMs), the robotics community has been abuzz with new systems that leverage VLMS to guide robotic behavior [28, 5, 29, 30, 3. 4, 31]. In these systems, VLMs decompose tasks into language plans [28, 5] or generate code to execute predefined skills [32, 3]. Though successful simulation, these methods underperform in the real world [32, 3]. Some methods rely on privileged
'Euua contribution
2 Mainly suited for solving most quasistaic, prehensile, and some non-prehensile tasks.
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Vision
MA
Language
Models
Ro
Scaling up zero-shot robot data generation
hhthhhh
M
Error Recovery
Figure 1: SHIIPULATE-ANTHIIE is an automated method for robot manipulation in real world environments. Unlike prior methods, it does not require privileged state information, hand-designed skills, or limited to manipulating fixed number of object instances. It can guide robot to accomplish a diverse set of unseen tasks, manipulating diverse objects. Furthermore, the generated data enables training behavior cloning policies that outperform training with human demonstrations.
state information only available in simulation [29, 4], require hand-designed skills [30], or are also limited manipulating fixed set of object instances with known geometric shape [3, 32].
As VLMs improve performance, and with the vast common-senss knowledge they have shown to possess, could harvest their capabilities for diverse task completion and scalable data generation? The answer is yes with careful system design and the right set of input and output formulations, can not only use VLMs as means successfully perform diverse tasks in a zero-shot manner, but also generate quality data at a high quantity to train behaviour cloning policies.
We propose for real-world robotic manipulation. large-quantities (if needed), and can manipulate diverse set of objects perform diverse set of tasks. When placed in real world environment and given a task (e.g., "open the top drawer" in Figure 2), MANIPULATE- ANYTHING effectively leverages VLMs to guide a robotic arm to complete the task. Unlike prior methods, it doesn't need privileged state information, hand-designed skills, or limited to specific object instances. Not relying on privileged information makes MANIPULATE- plans sequence of sub- soab-goals and generates actions execute the sub-goals. can verify whether the robot succeeded in the sub-goal using verifier and re-plan from the current state if needed. This error recovery enables mistake identification, re- re-planning, and recovering from failure. It also injects recovery behavior into the collected demonstrations. We further enhanced the VLMs' capabilitiess by incorporating reasoning from multi-viewpoints, significantly improving performance.
a scalable automated demonstration generation method MANIPULLTE--ANTHUUE produces high quality data, at
ANYTHING environmennt-gnnosiic.
We showcase the utility of MANIPUL.ATE- show that it can be prompted with novel, never-before-seen task and complete in zero-shot manner. We quantitatively evaluate across real-world and 14 RLBench [33] simulation tasks and demonstrate capabilities across many real-world everyday tasks (refer to appendix). Our method significantly outperforms VoxPoser [3] in 10/14 simulation tasks for zero-shot evaluation. It also generalizes to tasks where VoxPoser completely fails because of its limitation to specific object instances. Furthermore, we demonstrated that our approach can solve real-world manipulation tasks in a zero-shot manner, achieving lask-ave success rate of 38.57%. Second, we show that MANIPULATE-ANYTIIGG
through two evaluation setups. First, we
can generate useful training data for a behavior cloning policy. We com- pare MANIPULATE- ANYTHING generated data against ground truth hand-scripted demonstrations well against data from VoxPoser [3], Scaling-up [4] and Code-As-Policies [5]. Surprisingly, policies trained on our data outperforms even human hand-scripted data on 4 out of 12 tasks and performs on par for 3 more when trained with PerAct [34]. Meanwhile, the baselines are unable to generate the training data for some of tasks. MANIPULATE- ANYTHING demonstrates the broad possibility of large-scale deployment of robots across unstructured real-woria environments. also highlights its utility as training data generator, aiding the crucial goal of scaling up robot demonstration data.

---

Plan Generation
Spec Berr Stwee
Sib-sal
Fall oudiiii dranci
Cast lar cacas
Data gunsraation
}
Cilicted darceddrsson
Sub
Sub-
Action
Sub-task Verifiattion
Action Generation Module
Sub task Venfication Module
Generation
Module
Module
Herr gron
your pnr
Action Generation Module
Sub-task Verification
Treadaics
Talogeclly
Tagat Olgect Reverbule
Groede Jutios
Figure Manipulate Anything Framework. The process begins by inputting scene representation and natural language task instruction into a VLM, which identifies objects and determines sub-tasks. For each sub-task, we provide multi-view images, verification conditions, and task goals to the action generation module, producing task-specific grasp pose action code. This leads to temporary goal state, assessed by the sub-task verification module for error recovery. Once all sub-tasks are achieved, we filter the trajectories to obtain successful demonstrations for downstream policy training.
2
Related work
MANIPUL.ATE: CEMIPULLTE--ANTHIIEEE ANYTHING enables scaling of robotic manipulation data using VLMS.. As such, we review recent efforts in I) scaling manipulation data, and 2) applications of VLMS in robotics. Scaling manipulation data. When deploying vision and language-based control policies for real- world applications, a significant challenge revolves around acquiring data. Traditionally, a convenient avenue to collect such trajectories is through human annotations for action (i.e. through teleoper- ation) and language labeling [34, 35, 36], however, this approach limited in scale. To address this limitation and achieve autonomous scalability, prior works employ vision-language models procedurally generate language annotations in simulated environments [4, 37, 38). For action labels, strategies range from random exploration to learned policies [39]. While human egocentric videos are relevant, they lack action labels and require cross- emboddment transfer [40]. Another strategy involves model-based policies, such as task and motion planning (TAMP) [41]. Our approach extends these methods by incorporating common-sense knowledge from LLMs and VLMs, providing framework that combines the strengths of VLMs, object pose prediction, and dynamic retry to synthesize demonstrations in simulated and real environments.
Language models for robotics. In the field of robotics, large language models have found diverse applications, including policy learning [42, 43, 44], task and motion planning [45, 46), log summa- rization [47], policy program synthesis [5], and optimization program generation [28]. Previous research has also explored the physical grounding capabilities these models [3, 32, 48, 49], while ongoing work [4] investigates their integration with task and motion planners to create expert demonstrations. [36] attempted to collect extensive real-world interaction data, with short-horizon trajectories. [50] proposed key-poin based visual prompting method for real-world manipulation, through predicting affordances and corresponding motions. Our work complements the existing line of works by leveraging the high-level planning capabilities of language models, scene understanding capabilities of vision language models, and action sampling to enable synthesis of robot trajectories, which include language, vision, and robot state, given arbitrary tasks and environments.
3

---

3 MANIULATE-ANYTHNNG
We propose MANIPULATE- ditioned language. Under the hood, MANPULATE-ANYTHING leverages VLMS to decompose tasks into sub-tasks, generates code for new skills or task-specific grasp pose, and verifies the success of each sub-task (Figure 3). Note that due to the modularity aspect of our framework, MANIPULATE-ANYTIING will continue to improve as the underlying VLMs continue to improve.
framework that solves everyday manipulation tasks con-
3.1 Task plan generation
MANIPULATE-ANYTIIGA takes as input any task described by free-form language instruction, T (e.g., "open the top drawer'). Creating robot trajectories that adheres to T is challenging due to its potential complexity and ambiguity, requiring a nuanced understanding of the current environment state. Given T, and an image of the scene, we apply a VLM to first identify task-relevant objects in the scene, appending them to a list. Subsequently, we use a LLM along with those information to decompose the main task T into series of discrete, smaller sub-tasks, represented as T, along with the corresponding verification conditions U; where ranges from n. For instance, the above task could be decompose into sub-tasks include 'grasp the drawer handle' or 'pull open the drawer', and verification conditions are 'did the robol grasp the hundle?' or "is the drawer opened?'. This transforms the instruction T into a sequence of specific sub-tasks (TT, generates desired actions ($ 3.3) and verifies them against the corresponding conditions to ensure successful completion of that sub-task($ 3.4). This verification step also allows MANIPULATE-ANYTHIIN to recover from mistakes and attempt again in the case of failure.
U2), (Ta, vx)}-
For each sub-task, SANIPULLTE-ANNTHAE
3.2 Multi-viewpoint VLM selection
Many prior works that investigated VLM's reasoning capability found that they do not work well given single viewpoint [51, 52, 53]. In robotic manipulation, we leverage multiple viewpoints from either more than one camera re-rendering to minimize object occlusion [54, 55]. Leveraging these insights, we proposed a muuli--iewpoon viewpoints for either action generation sub-task verification for MANIPULATE- ANYTIINNN concatenate all available viewpoints from the current observations, either from multiple cameras or re-rendered viewpoints from a single RGB-D, into a single frame. Numbers are annotated on the top left of each concatenated frame to correspond to a specific viewpoint. Using the concatenated multi-viewpoints, we then query VLMs to choose an ideal viewpoint conditioned on the sub-task. Therefore, for both agent-centric and object-centric action generation, we render multiple viewpoints of the scene and query VLMs to choose ideal viewpoint for either generating actions given the sub-task or verifying if the sub-task verification condition has been met shown in Figure 3.1
selection phase via VLM before using the selected
3.3 Action generation module
Given a sub-task, the desired output from the action generation module sequence of low-level actions represented 6 DoF end-effector pose. The actions can be categorized into two sets: agent-centric object-centric. Based on the generated sub-task, the LLM planner will classify it as either agent-centric or object-centric actions. For agent-centric actions, it will modify the agent's s state; e.g., it can move the robot's 's end-effector from the current state (e.g., "rotate 90%"). We first employed our multi-viewpoint selection to sample out the most optimal viewpoints to provide the VLM with along with the in-context learning technique to generate the code for synthesizing the desired motion. Unlike prior methods that use language models solely for code generation [5], we leverage VLMS reason about object locations and the scene, grounding generation in the current state. This advantage is demonstrated in the ablation studies in the Appendix.
For object-centric actions, it is often used to generate a task-specific grasp pose for grasping a certain object (c.g., "grasp a knife for cutting") or to synthesize a pre-action pose for non- non-prehensile tasks by allowing the VL.M to assign translation offsets to task-specific grasp poses. To obtain an object- centric action for given sub-task, we first use an object-agnostic grasp prediction model [56] to generate all possible 6-DOF grasping poses in the scene. These poses not conditioned on task specifications and may include invalid grasp poses for the given task. Next, we use the VLM

---

Simulation Benchmark
Joige"
assoee
'our blek grees
vie
"Unogp
up the cap"
"Puun lagge"
"Cuose the box"
Ther on laws
Vyen de Nox*
"Uwrage me jar"
"Pick up the cule"
"Meert the jenge bick"
clupping
Real-world Tasks
"Pla objects on thiir conresyonuling coler was. "Oren the top draeer*
loop"
dioe 1 furing fenond"
"Press the piice"
"Closr dee lepage"
Figure 3: MANIPULATE-ANTTINNG is an open-vocabulary robot demonstration generation system. We show zero-shot demonstrations for 14 tasks in simulation and 7 tasks in the real world.
obtain the bounding box of the target object parts conditioned on the task specification from all available re-rendered viewpoints (e.g., if the task is "grasp knife," the VLM will detect the handle of the knife and generate bounding box for the handle). The VLM then performs multi-viewpoint selection to identify the most optimal viewpoint free from occlusion. Finally, using the bounding box detection of the task task-specific part of the object from the optimal viewpoint selected by the VLM, we filter out a list of proposed candidate grasp poses and select the highest confidence grasp pose. This approach allows us to obtain the most optimal task-specific grasp pose, placement pose, and pre-action pose for non-prehensile tasks, all leveraging the capabilities of the VLM. After the action generated, a motion planner can be used move the robot the desired pose as detailed Fig.
3.4 Sub-task verification
To ensure that each sub-task T, executed correctly, we introduce VLM- verifier. After every action for each sub-task is executed, we use the VLM check if the end state matches the verifier condition t. Similar to the action generation module, find the optimal view, avoiding errors due to occlusion or ambiguity from single viewpoint. If the verifier identifies failure, we re-attempt the action generation step for the previous sub-task from the current state. Otherwise, the next sub-task T;- is attempted. More details are in the appendix 6.
multi-viewpoint VLM selection to
Experiments
Our experiments are designed to address two questions: 1) Can MANIPULATE-AN ANYTHING accurately solve diverse set of tasks in zero-shot manner? 2) Can data generated from MANIPULATE. ANYTHIING be used to train a robust policy?
Implementation details. We use both GPT-4V and Qwen- VL [57] as our VLM. We use GPT-4V for task decomposition, action generation, and verification. We Qwen VI. detect and extract object information. To ensure zero-shot execution within reasonable budget, we limit the number of action steps in each trajectory to 50 and the verification module allows maximum of 30 tries to accomplish sub-goal. For the task plan generation, we follow the prompting structure adapted from ProgPrompt [28]. All prompts input into the VLM are accompanied by few-shot demonstrations [58]. Additionally, we provide three manually curated primitive action code snippets as examples to prompt the VLM for new action code generation. Full prompts are included in the Appendix. We use four viewpoints M, [front wrist, fefftssoouder, right_stteee [rggt_shholder]] for the simulation experiments, and re- render three viewpoints for the real-worle experiments [54]. For better reasoning by the VLMM, we use resolution of 256 x 256.
4.1 Zero-shot Performance in Simulation
We empirically study the zero-shot capability of CIEHIPULATE-ANTTHIIE in solving 14 diverse tasks in simulation, covering wide range of task configurations and action primitives for both prehensile
5

---

Table 1: Task-averaged success rate for zero-shot evaluation. CEAIIPLATE--NNTHIEE outperformed other baselines in 10 out of 14 simulation tasks from RLBench [33]. Each task was evaluated over seeds to obtain the aask-averaged success rate and standard deviations.
Method
Pat_block
Play-jenga
Open_jar
Close_box
Open_box
Pickup_cup
Puck_block
VoxPoser [3] 70.7012.31
0.0010.00
0.00
0.00=0.00
0,00+0.00
26.70114.00
25.33:18.33
CAP [5]
84.00=16.00
0.00:10.00
0.00+10.00
0.00
0.00
14.67:44.62
8.00:1-1.00
Scaling-up [4] 77.33-16.11
0.00-10.00
78.671111.55
0.00110.00
0.00110.00
5.3311..11 9.33112.26
MA (Ours)
96.00+44.00
77.33-+6.11
80.00++4.00
13.83
29.00=11.007
82.67++14.04
20.00++4.00
Method
Take_umbrells
Sort_mastard
Open_vine
Lemp_oo
Put_kkiffe
Pick_a_1ift
Insert_block
VoxPoser[3]
33.33-+8.33
96.00=6..3
8.00++400
57.30=12.22
92.00=t4.00
96.00++0000
0.00++0000
CAP[S]
4.00=4.00
0.00:10.00
0.00:10000
64.00=6.93
14.67=8.33
100.00:0.000
0.00:10.00
Scaling-up [4]
6.67=2.31
41.33
33.33:1:20.13
60.00
24.00:10.00
100.00:10.00
0.00:10000
MA (Ours)
61.33 13.13 64.00.16.93
42.00.14.00
69.33
52.00-11.558
84.00116.93
33.33
and non-prehensile tasks. Our simulation experiments are reported to ensure reproducibility and provide benchmark for future methods.
Environment and tasks. The simulation setup involves a Franka Panda robot with a parallel gripper, using CoppeliaSim and PyRep for interfacing. Four RGB-D cameras capture input observations around a tabletop environment. RLBench [33] is used as the task benchmark, with 14 sampled tasks that cover various action primitives, task horizons, and object position variations. The robot's actions are represented as waypoints, with trajectories computed and executed via motion planner [59].
Baselines. We compare against three state-of-the-art zero-shot data generation approaches: Code-as-PPllccies (CAP) [5], Saatiuu--pDDsstilinggg and VoxPoser [3]. CAP uses language models to gen- erate programs that call hand-crafted primitive actions, while VoxPoser predicts waypoints via a 3D voxel map of value functions. Scaling-up leverages an LLM with 6 DoF exploration primitives to generate robotic data for policy distillation. We provided CAP and Scaling-up with ground truth simulation states and object models, and VoxPoser with segmented object point clouds, which inherently dis- advantages MANIULATE-ANYHHING in comparison. Results: MANIPULATTE-ANTHIIE
Scaling effect with increasing data
60
(Scaling-up) [4]
50
Success ratet%)
10-
generated diata Ground truth data
0-
0
20
40
60
80
100
tra ining demos
Figure 4: Scaling experiment. Scal- ing effect of model performance with increasing training demonstrations.
can generate suc- cessful trajectories for all 14 tasks while Scaling-up, VoxPoser and CAP cover only 10, 9 and 7 tasks, respec- tively (Table 1). the baselines in 10 out of the 14 tasks. The three lowest-performing tasks by MANIPULATE. ANYTHING are non-pre preheneeee or complex fine-grained manipulation tasks that require generating new primitive actions with precise parameters. VoxPoser fails the tasks that require moving the arm beyond 4-DoF. MANIPULATE-- - outperforms the strongest baseline, VoxPoser, by average task-averaged margin of up 22%.
ANYTHING outperforms
4.2 Behavior cloning with demonstrations from MANIPULATE-
Next, we analyze the quality of the generated data by comparing the success rates of behavior cloning models trained with the data. Zero-shot methods like MANIPULLTE-ANYTHIN are computationally expensive but hold the potential to generate useful training data. To evaluate the quality and effective- ness of the generated training data, we use the methods described in the previous section generate data for each task. We also compare performance against model trained on numan-generatee demonstrations across the 12 tasks. We use the data to train behavior cloning policies.

---

Table 2: Behavior Cloning with different generated data. The behavior cloning policy trained on the data generated by MANIPULATE- ANYYHHING provides the best performance on 10 out of 12 tasks compared the other autonomous data generation baselines (excluding RLBench). We report the Success Rate % for behaviour cloning policies trained with data generated from VoxPoser [3] and Code as Policies [5] in comparison. Note that the RI Bennch(33] baseline uses human expert demonstrations and is considered upper bound for behavior cloning.
Data
Models
Put_block
Play-jeng'
Open_jaar Close_box Open_box Pickup_cup
VoxPoser[3]
PerAct[34]
2.6712.31
4.00.14.00
CAP[S]
PcrAct[34]
6.6742.31
14.67 12.86
Scaling-up [4] PerAct[34]
22.67:+11.11
5.3316.11
14.67:42.31
MA (Ours)
PerAct[34]
85.33+10.00
81.33+2.31
21.33=10.07
42.67=8.33
30.67=11.55
54.00=12249
[LBench[333
PerAct[34]
20.00:1:1.33
81,3319224
58.67:445.49
68.00114.98
14.6716.11
54.67:1:3.00
Data
Models
Take, Take_umbrelle umbrella Sort_mustare Open_wine
Lamp_og
tai_e Pick Lick_&_p. &_lift
VoxPoser[3]
PerAct[34]
4.0014..00
0.0010.00
1.3312331
5.63
1.33.12.31
5.6711..64
CAP[5]
PerAct[34]
10.03
8.00+11.000
9.3346.11
46.67:42.31
Scaling-up [4] [3)
4.00:1:.00
0.00-10.00
11.33:11.80
76.00:14.00
5.3342..11
10.33110.06
MA (Ours)
P3rAct[34]
84,00 93
53.33#6.11 11
86.67 11
89.33=6.11 11
8.00+11.00
33.33+2.31
[LBench[333
PerAct[34]
58.67:1:00.80
53.33:133.022
86.67:12.86
84.001:13.86
30.67:11.00
62.67:19.24
Data generation details. We generate 10 successful demonstrations per task. We use the system's success condition to filter for successful demonstrations. Each of the demonstrations consist of lan- guage instruction, RGB-D frames for the trajectory, and waypoints represented as 6 DoF gripper poses and states. For the tasks that the baselines were unable generate any successful demonstrations, patched the missing training data with RLBench ristem-generated demonstrations.
Training and evaluation protocol. We train two models using the generated demonstrations: the Perceiver-Actor (PerAct) model [34], a transformer based robotic manipulation behavior cloning model that expects tokenized voxel grids and language instructions as inputs and predicts discretized voxel grid 6 DoF poses and gripper states, and RVT-2 model [60], multi-view transformer-bic BC model. The RVT-2 model uses tokenized image patches and CLIP-encoded language instructions as input to predict keypoint actions translation heatmaps, discretized rotation in Euler angles, and the gripper's 's binary state. Notably. RVT-2 is currently the teahhestperformingg model on the RLBench benchmark. For all the generated training datasets, we train a multi-task PerAct policy with a batch size of 4 for 30k iterations on a single RTX A100, and RVT-2 with a batch size of 24 for 3.3k iterations on 4 A 100s. To ensure consistent evaluation, we generate one set of testing environments with RLBench. We evaluate the last checkpoint from each of the trained policies. Each policy is evaluated for 25 episodes across each task using 3 different seeds. We measure the success rate based on the simulation-defined success condition. Details RVT-2 results can be found in the Appendix.
data perform similarly to policies trained using hand-scripted demonstrations (p = 0.973) for PerAct (Table 2). Training on either MANIPULATE-ANYTIIE or hand-scripted demonstrations results in a performance difference of just 0.27% average across all tasks. Furthermore, models trained on data from the baselines exhibit statistically lower performance (p < 0.01 for VoxPoser, Scaling-up up and CAP). One of the main factors potentially contributing to the performance differences could be that MANIPULATE- ANYTHING generates diverse expert trajectories preferable to humans. This is shown in Fig. 11, which illustrates the action distribution of generated data by different methods for the same given tasks. We also observed that the policy trained on MA data achieves a lower standard deviation of 7.19 average across all tasks, compared the zero-shot performance standard deviation of 8.81. This suggests the benefits of training over generated data instead of relying solely on zero-shot deployment.
Results: Policies trained using MANIPULATE-ANYTIIG
4.3 Real-world experiments
Environment and tasks. We employ Franka Panda manipulator equipped with a parallel grip- per. We use front-facing Kinect 2 RGB-D camera. To generate multi-view inputs for the
7

---

Table 3: Real-world Results. The model trained on the data generated by our model in the real world (no expert in the loop) demonstrates on par results with the model trained on human expert collected data. We present a comparison of success rates for task completion zero-shot manner (Code as Policies [5] and MANIPUL.AEE-AYTHIN ANYTHING data and human expert data.
and using trained policies from MANIPULATEE.
Open drawer
obje_t
On_lamp
Open_jar
Corroct_dic
Press_swie swiss
Close laptop
CAP (0-shot)
5.77 0.00 + 0.00 13.33 5.77 0.00 + 0.00 6.67 = 5.77 6.67 +: 5.77
0.00 + 0.00 0.00 + 0.00
MA (0-shor)
36.6755777
60.00=10.00
16.67=11.56
40.00=00.00
53.33 77
20.00=10:00
33.33+5.77
PerAct
(MA data)
50.00 + 0.00
33.33 + 5.77 50.00 + 0.00
56.67 + 5.77 60.00
0.00
56.67 + 5.77
33.33 4: 5.77
PerAct (Human data)
53.33+11.53
36.67+5.77
60.00=0.000
76.67=3.77
10.00=10.00
33.3326.77 53.3325.77
MANIPULATE- framework, we re-render virtual viewpoints from the generated point cloud, similar to prior work [54]. We selected 7 representative real-world tasks, both prehen- sile and non-prehensile: open_jar, sort_ cort_sbjects, correct_ _iccs, open_ drawer, on_lamp, press_switch, and close_laptop. all conditioned on language instructions. Each task was evalu- ated over 10 episodes with varying object poses across 3 trials.
Data generation details. We used MAMIPULATE-ANTHHIN to generate demonstrations for each task and manually perform scene resets when failures occur. We train similar multi-task PerAct for 120k iterations and evaluate the trained policies in manner similar to the zero-shot experiments.
is able to generate successful demonstrations for each of the real world tasks. Even for the worst porst-performing task, MANIPULATE-ANYTHING achieves a success rate of more than 25%. Our approach outperforms CAP by 38%. Consistent with the simulation results, training with the data generated by MANIPULATE-ANYY ANYTHING produces more robust policy compared to performing zero-shot. Additionally, in 4 out of 5 tasks, the trained policies perform better than the zero-shot approach. The policy under-performs on the sort_object task, because it requires longer-horize memory -a known limitation pointed out in PerAct [34].
Results:
4.4 Ablations
For effective real-world deployment of MANIPULATE AYYTHUG it's crucial that the collected data supports scaling of robotics transformers and offers diverse skills and interacted objects. We conducted an ablation study to evaluate the quality of scaling and its generalization to language instruction changes. For scaling, we generated behavior cloning data, ranging from 1 to 100 training demonstrations from RLBench and MANIPULATE- ANYTHING for single task (put_block) and trained a PerAct policy. For generalization, we varied the cort_uustard task with different language instructions and target objects. We compared our approach to VoxPoser to assess robustness to object and language instruction changes. Further imple- mentation details are in the supplementary materials. Result: Our scaling experiments demonstrate that generating more training data via mance (Fig. 4). The data from our approach shows a better rate of change with a slope of 0.503 for a linear fit, compared to 0.197 for RLBench- seennhhgeeeeeted data. Additionally, MANIPULATE-ANNV data is more generalizable and robust to language instruction changes, outperforming VoxPoser in task success across language and object variations. Detailed results in the appendix.
data for
SHHPPULATE-ANTHHEE
improves PerAct policy perfor-
Discussion
Limitations. Despite promising results, MANIPULATE-ANTHINGG has limitations: reliance on large language models (VLMs), which may lessen with the development of open-source VLMs; inability to generate alternative grasp poses for dynamic manipulation and non-prehensile tasks; potential compounding eтrоrs from integrating multiple VLMs, possibly mitigated by emerging specialized VLMs; and the need for manual prompt engineering for in-context learning, with ad- vancements in alignment and prompting techniques potentially reducing this effort. Conclusion. MANIPULATE--AL ANYTHING is scalable, environment-agnosticc approach that leverages VLMS for high-level planning, scene understanding, and error recovery generate zero-shot demonstrations for robotic tasks without privileged environment information. This results in high-quality data for behavior cloning, outperforming human data.
8

---

Acknowledgments
Jiafei Duan is supported by the Ageney for Science, Technology and Research (A*STAR) National Science Fellowship. Wilbert Pumacay is supported by grant 334-2015-FONDECYY from Cienciactiva of the National Council for Science, Technology and Technological Innoation(CONCCTECCPEER We would also like to thank Winson Han for the help in designing of the figures. This project was partially funded by an Amazon Science grant. This project is also sponsored by RACER2 Project: HR001 1230010..
References
[1] A. Brohan, N. Brown, Carbajal, Y. Chebotar, Dabis, C. Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817, 2022.
[2] O. X.-E. Collaboration. Open X-Embodiment: Robotic learning datasets and RT-X models. https://arxiz: httpr/ii..orggabb////// http://ab//311..000/eg 2023.
[3] W. Huang. C. Wang, R. Zhang, Y. Li, J. Wu, and L. Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models. arXiv preprint arXiv:2307.0597. 2023.
[4] H. Ha, P. Florence, and Song. Scaling up and distilling down: Language-gun guided robot skill acquisition. arXiv preprint arXiv:2307. 14535, 2023.
[5] J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng. Code policies: Language model programs for embodied control. In 2023 IEEE International Conference Robotics and Automation (ICRA), pages 9490- IEEE, 2023.
[6] J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Rad- ford, J. Wu, and D. Amodei. Scaling laws for neural language models. arXiv preprint drXiv:2001.08311,2222 2020.
[7] M. Cherti, R. Beaumont, R. Wightman, M. Wortsman, G. Ilharco, C. Gordon, C. Schuhmann, L. Schmidt, and J. Jitsev. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 28182,8,, 2023.
[8] J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203. 15556, 2022.
[9] C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, al. Laion-5b: An open large-scale scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35: 25278-25294, 2022.
[10] V. Ramanujan, T. Nguyen, S. Oh, A. Farhadi, and L. Schmidt. On the connection between pre- training data diversity and fine-tuning robustness. Advances in Neural Information Processing Systems, 36, 2024.
[11] V. Udandarao, A. Prabhu, A. Ghosh, Y. Sharma, H. Torr, A. Bibi, Albanie, and M. Bethge. No" zero-shot" without exponential data: Pretraining concept frequency determines multimodal model performance. arXiv preprint arXiv:2404.041 04 2024.
[12] S. Y. Gadre, G. Ilharco, A. Fang, Hayase, G. Smyrnis, T. Nguyen, R. Marten, M. Wortsman, D. Ghosh, J. Zhang, et al. Datacomp: In search of the next generation of multimodal datasets. Advances in Neural Information Processing Systems, 36, 2024.

---

[13] C. Zhou, P. Liu, P. Xu, S. Iyer, J. Sun, Y. Mao, X. Ma, A. Efrat, P. Yu, L. Yu, et al. Lima: Less more for alignment. Advances in Neural Information Processing Systems, 36, 2024.
[14] T. Nguyen, S. Y. Gadre, G. Ilharco, S. Oh, and L. Schmidt. Improving multimodal datasets with image captioning. Advances in Neural Information Processing Systems, 36, 2024.
[15] T. Nguyen, G. Ilbarco, M. Wortsman, Oh, and L. Schmidt. Quality not quantity: On the interaction between dataset design and robustness of clip. Advances Neural Information Processing Systems, 35:21455-21469, 2022.
[16] K. Lee, D. Ippolito, Nystrom, C. Zhang, Eck, Callison-Burch, and N. Carlini. Dedupli- cating training data makes language models better. arXiv preprini arXiv: 107. 107.06499, 2021.
[17] A. Fang, G. Ilharco, M. Wortsman, Y. Wan, V. Shankar, A. Dave, and L.. Schmidt. Data determines distributional robustness in contrastive language image pre-training (clip). In International Conference on Machine Learning, pages 623444 PMLR, 2022.
[18] S. Gururangan, A. Marasović, S. Swayamdipta, K. Lo, I. Beltagy, D. Downey, and N. A. Smith. Don't stop pretraining: Adapt language models to domains and tasks. arXiv preprint arXiv:2004. 10964, 2020.
[19] Y. Tian, L. Fan, K. Chen, D. Katabi, D. Krishnan, and P. Isola. Learning vision from models rivals learning vision from data, arXiv preprint urXiv:23/2.17742, 2023.
[20] H. Xu, S. Xie, X. E. Tan, P.-Y. Huang, R. Howes, Sharma, S.-W. Li, G. Ghosh, L. Zettlemoyer, and C. Feichtenhofer. Demystifying clip data. arXiv preprint arXiv:2309. /6671, 2023.
[21] Z. Chen, A. Cano, Romanou, A. Bonnet, Matoba, F. Salvi, M. Pagliardini, S. Fan, A. Köpf, A. Mohtashami, al. Meditron-70b: Scaling medical pretraining for large language models. arXiv preprint arXiv:2311.16073 2023.
[22] J. Deng, W. Dong, R. Socher, L.JJ. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pages 248-255, Icee, 2009.
[23] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
[24] C. Schuhmann, Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, Coombes, A. Katta, C. Mullis, M. Wortsman, Schramowski, S. R. Kundurthy, K. Crowson, L. Schmidt, R. Kaczmarczyk, and Jitsev. LAION-5b: An open large-scale dataset for training next generation image-text models. In Thit Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2022. URL id=M3Y74vmsMcY.
wwwppprrrvieeeeee//eoeeeeeeee net/forum?
[25] C. Chi, Z. Xu, C. Pan, E. Cousineau, B. Burchfiel, S. Feng, R. Tedrake, and S. Song. Universal manipulation interface: In-the-wild robot teaching without in-the-wild robots. arXiv preprint arXiv:2402. 10329, 2024.
[26] C. Wang, Shi, Wang, R. Zhang, Fei-Fei, and C. K. Liu. Dexcap: Scalable and portable mocap data collection system for dexterous manipulation. arXiv preprint arXiv:2403.07788. 2024.
[27] J. Duan, Y. R. Wang. M. Shridhar, D. Fox, and R. Krishna. Ar2-d2: Training robot without robot. arXiv preprint 23t.:23066138888 2023.
10

---

[28] I. Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg. Progprompt: Generating situated robot task plans using large language models. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 11523-11530. IEEE, 2023.
[29] L. Wang, Y. Ling, Z. Yuan, M. Shridhar, C. Bao, Y. Qin, B. Wang, H. Xu, and X. Wang. Gensim: Generating robotic simulation tasks via large language models. arXiv preprint arXt::2310.01361, 2023.
[30] Y. Wang. Z. Xian, F. Chen, T.-H. Wang, Z. Erickson, D. Held, and C. Gan. Genbot: Generative simulation empowers automated robotic skill learning at scale. arXiv, 2023.
[31] S. Nasiriany, F. Xia, W. Yu, T. Xiao, Liang, Dasgupta, A. Xie, D. Driess, A. Wahid, Z. Xu, Q. Vuong, T. Zhang, T.-W. E. Lee, K.-H. Lee, P. Xu, S. Kirmani, Y. Zhu, A. Zeng, K. Hausman, N. Heess, C. Finn, S. Levine, and B. Ichter. Pivot: Iterative visual prompting elicits actionable knowledge for vlms, 2024.
[32] H. Huang, F. Lin, Y. Hu, S. Wang, and Y. Gao. Copa:
General robotic manipulation through spatial constraints of parts with foundation models. arXiv preprint arXiv:2403.0822 08244, 2024.
[33] S. James, Z. Ma, D. R. Arrojo, and A. J. Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 5(2):3019-3026, 2020.
[34] M. Shridhar, L.. Manuelli, and D. Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. In Conference on Robot Learning, pages 785-799. PMLR, 2023.
[35] M. Shridhar, L. Manuelli, and D. Fox. Cliport: What and where pathways for robotic manipula- tion. In Conference on Robot Learning, pages 894-906 PMLR, 2022.
[36] A. Brohan, Y. Chebotar, C. Finn, K. Hausman, A. Herzog, D. Ho, J. Ibarz, A. Irpan, E. Jang, R. Julian, et al. Do as can, not as i say: Grounding language in robotic affordances. In Conference on Robot Learning, pages 287-318. PMLR, 2023.
[37] K. Ehsani, T. Gupta, R. Hendrix, J. Salvador, L. Weihs, K.-H. Zeng, Singh, Y. Kim, W. Han, A. Herrasti, et al. Imitating shortest paths in simulation enables effective navigation and manipulation in the real world. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2024.
[38] M. Ahn, D. Dwibedi, C. Finn, M. G. Arenas, K. Gopalakrishnan, K. Hausman, B. Ichter, A. Irpan, N. Joshi, Julian, et al. Autort: Embodied foundation models for large scale orchestration of robotic agents, arXiv preprint arXiv:2401.12963, 2024.
[39] Y. Wang, Z. Xian, F. Chen, T.-H. Wang, Y. Wang, K. Fragkiadaki, Z. Erickson, D. Held, and C. Gan. Robogen: Towards unleashing infinite data for automated robot learning via generative simulation. arXiv preprint arXiv:2311.0143 2023.
[40] K. Grauman, A. Westbury, E. Byrne, Z. Chavis, A. Furnari, R. Girdhar, Hamburger, H. Jiang, M. Liu, Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18995-19012. 2022.
[41] C. R. Garrett, R. Chitnis, R. Holladay, B. Kim, T. Silver, L. P. Kaelbling, and T. Pozano-Pérez Integrated task and motion planning. Annual review of control, robotics, and autonomous systems, 4:265-293, 2021.
[42] T. Xie, S. Zhao, C. H. Wu, Y. Liu, Q. Luo, V. Zhong, Y. Yang, and T. Yu. Text2reward: Automated dense reward function generation for reinforcement learning. arXiv preprint arXiv:2309.11489 2023.
11

---

[43] J. Zhang, K. Pertsch, J. Zhang. T. Nam, S. J. Hwang, X. Ren, and J. Lim. Sprint: Scalable semantic policy pre-training via language instruction relabeling. 2022.
[44] Y. Wang, Z. Sun, J. Zhang, Z. Xian, E. Biyik, D. Held, and Z. Erickson. RI-vlm-f: Reinforcement learning from vision language foundation model feedback. arXiv preprint arXiv:240..0368. 2024.
[45] K. Lin, C. Agia, T. Migimatsu, M. Pavone, and J. Bohg. Text2motion: From natural language instructions feasible plans, 2023.
[46] W. Huang, F. Xia, T. Xiao, Chan, Liang. Florence, A. Zeng. J. Tompson, Mordatch, Y. Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2007.0600s 2022.
[47] Z. Liu, A. Bahery, and S. Song. Reflect: Summarizing robot experiences for failure explanation and correction, 2023.
[48] J. Zhang, J. Zhang, K. Pertsch, Z. Liu, X. Ren, M. Chang, S.-H. Sun, and J. Lim. Bootstrap your own skills: Learning to solve new tasks with large language model guidance. arXiv preprint arXiv:2310.1003 2023.
[49] J. Duan, W. Pumacay, N. Kumar, Y. R. Wang, S. Tian, W. Yuan, R. Krishna, D. Fox, A. Man- dlekar, and Y. Guo. Aha: A vision-language-modee for detecting and reasoning over failures in robotic manipulation. arXiv preprint xrXiv:2411.00031,,2222
[50] F. Liu, K. Fang, P. Abbeel, and S. Levine. Moka: Open-vocabulary robotic manipulation through mark-based visual prompting. arXiv preprint arXiv:2403.03174, 2024.
[51] B. Liu, Y. Dong, Y. Wang, Y. Rao, Y. Tang, W.-. Ma, and R. Krishna. Coarse corre- spondence elicit 3d spacetime understanding in multimodal language model. arXiv preprint arXiv:2408.00754, 2024.
[52] W. Yuan, J. Duan, V. Blukis, W. Pumacay, R. Krishna, A. Murali, A. Mousavian, and D. Fox. Robopoint: A vision-language model for spatial affordance prediction for robotics. arXiv preprint arXiv:2406. 10721, 2024.
[53] Y. Hong, Zhen, Chen, Zheng, Y. Du, Z. Chen, and C. Gan. 3d-llm: Injecting the 3d world into large language models. Advances in Neural Information Processing Systems, 36: 20482-20494, 2023.
[54] A. Goyal, Xu, Y. Guo, V. Blukis, Y.W. Chao, and D. Fox. Rvt: Robotic view transformer for 3d object manipulation. In Conference on Robot Learning, pages 694-710. PMLR, 2023.
[55] Y. R. Wang, Y. Zhao, H. Xu, S. Eppel, A. Aspuru-Guzik, F. Shkurti, and A. Garg. Mytrans: Multi-view perception of transparent objects. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 3771 IEEE, 2023.
[56] W. Yuan, A. Murali, A. Mousavian, and D. Fox. M212: Multi-task masked transformer for object-centric pick and place, 2023.
[57] J. Bai, S. Bai, S. Yang, S. Wang. S. Tan, P. Wang, J. Lin, C. Zhou, and J. Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint 120::6600.11966. 2023.
[58] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[59] I. A. Sucan, M. Moll, and L. E. Kavraki. The open motion planning library. IEEE Robotics & Automation Magazine, 19(4):72-82, 2012. 2012.
12

---

[60] A. Goyal, V. Blukis, J. Xu, Y. Guo, Y.-W. Chao, and D. Fox. Rvt-2: Learning precise manipula- tion from few demonstrations. arXiv preprint arXiv:24006.08555, 2024.
[61] J. Tiedemann, M. Aulamo, D. Bakshandaeva, M. Boggia, S.-A. Grönroos, T. Nieminen, A. Ra- ganato, Y. Scherrer, R. Vazquez, and S. Virpioja. Democratizing neural machine translation with opus-ml. Language Resources and Evaluation, pages 1-43, 2023.
[62] J. Yang, H. Zhang, F. Li, X. Zou, C. Li, and J. Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v, arXiv preprint arXiv:2310./1441, 2023.
6 Appendix
6.1 MANIPULATE-ANYHHIN implementations
Action Generation Module. We generate each action using either an agent-centric or object-centric approach. For object-centric action generation, we utilize M2T2 [56], NVIDIA's foundational grasp prediction model, for pick and place actions. For 6-DoF grasping, we input a single 3D point cloud from either a single RGB-D camera (in the real world) multiple cameras (in simulation). The model outputs set of grasp proposals any graspable objects, providing 6-DoF grasp candidates (3-DoF rotation and 3-DoF translation) and default gripper close states. For placement actions, M2T2 outputs a set of 6-DoF placement poses, indicating where the end-effector should be before executing a drop primitive action based on a VLM plan. The network ensures the object is stably positioned without collisions. We also set default values for mask_thn thkesshhl and object_threed thresholld to control the number of proposed grasp candidates. After proposing list of template grasp poses, we use QWen-VL [57] detect the target object by prompting the current image frame with the target object's 's name, translated into Chinese using machine translation model [61]. This detection is applied to all re-rendered viewpoints or viewpoints from different cameras. We then concatenate these frames into single image, annotating each sub-image with number at the top right corner. Next, we call the GPT-4V API with few-shot demonstrations and the task goal to prompt GPT-4V to output the selected number of viewpoints that provide the most unobstructed views for sampling the grasp pose to achieve the sub-goal. Using the selected viewpoint, we execute the grasp by moving the end-effector to the sampled grasp pose via a motion planner.
For agent-centric action generation, we first perform the same steps of viewpoint selection. Using the selected viewpoint, few demonstration examples, and the sub-goal, we prompt GPT-4V to generate an action function with code snippets that include the necessary code to perform a delta-action on the current robot pose. We then execute this by moving the end-effector based on these delta changes. This process is iterated until we obtain the most desirable code snippet function for the given sub-goals, which is then appended skill library for future use.
Sub-goal Verification Module The sub-goal verification module helps with eтror recovery by ensuring all potential attempts at resolving the current step action have been tried. With the temporary goal state obtained by the action generation module, we use nulti-viewpoints to sample the optimal viewpoint for answering the verification condition generated for the given sub-goal during the task plan generation phase. Using the same viewpoint selection method as in the Action Generation Module, we obtain the optimal view and then perform two-step sequence rollout of frames: one from the current frame at this viewpoint and another from the previous action step. We concatenate these two frames, annotate them with numbers to indicate their temporal relation, and use this image to prompt GPT-4V to check if the verification condition is fulfilled as shown in Fig. 5. If the answer is Yes, we proceed to the next sub-goal. If the answer is No. we resample new viewpoints, generate new actions, and reattempt the entire sub-goal with a different seed.
6.2 Simulation experiments
setup illustrated in Fig. 7. The cameras are positioned at the front, left shoulder, wrist, and right shoulder. All cameras are static,
Simulation Setup. All the simulated experiments use a
13

---

Venifcatioon conditio: the robot gryper group the block?
Instraction
G
Temporary Goal State
Prompt for <Task>>
Sub-task: Succeed/Failed
Figure 5: Sub-goal Verification Module. We used viewpoint selection similar to action generation, to find the optimal viewpoints, and roll out the two step sequences previous and current frames for prompting the verification condition.
except for the wrist camera, which is mounted on the end effector. We did not modify the default camera poses from the original RLBench [33]. These poses maximize coverage of the entire table, and we use 256 x 256 resolution for better input to the VLMs.
Task Details. We describe in detail each the 12 tasks for simulation evaluation, both for trained policies and zero-shot methods, along with their RLBench variations and success conditions. We have made some modifications the original tasks enhance the detection rate by Code-As-Policies and VoxPoser.
Additional Simulation Results. addition to evaluating the generated data using PerAct [34], we also assessed with RVT-2 [60], the current state-of-the-art model on the RLBench benchmark. While the results of models trained with RVT-2 were significantly higher than those trained with PerAct, the overall trends and performance patterns remained consistent across the same training data depicted in Table 4.
6.2.1 put block
Filename: put nut_block.pp py
Task: Pick up the green block and place it on the mat.
Success Metric: The success condition the red mat detects the target green block.
6.2.2 close box
Filename: close_box, box.py
Task: Close the box.
Success Metric: The revolute joint of the specified handle is at least 60% off from the starting position.
6.2.3 open box
Filename: open_box.Pp .yy
Task: Open the box.
Success Metric: The revolute joint of the specified handle is at least 60% off from the starting position.
6.2.4 play jenga
Filename: play_jenga.pp
Task: Pull out the green jenga block.
Success Metric: The green jenga block is out of its pre-defined location.
14

---

Table 4: Behavior Cloning with different generated data. The behavior cloning policy trained on the data generated by - provides the best performance on 10 out of 12 tasks compared to the other autonomous data generation baselines. We report the Success Rate for behaviour cloning policies trained with data generated from VoxPoser [3] and Code as Policies [5] in comparison. Note that the RLBench [33] baseline uses human expert demonstrations and
considered upper bound for behavior cloning.
Data
Models
Puo_klock
Play-jenga
Open_.jar Close_box _pen_box box Pickup_cup
VoxPoser [3]
PerAct [34]
2.67.12.31
4.00144.00
CAP
PerAct [34]
6.6742.31
14.67 12.86
Scaling-up [4] PerAct [34]
22.67:15..14
6.33
14.67:22.31
MA (Ours)
PerAct [34]
85.33+10.077
81.33+2.31
11.33=00.07
42.67=8.33
30.67=11.55
54,00=112:4
RLBench [33]
PerAct [34]
20.00:18.33
81.33:9.24
58.67
68.00-1-4.98 14.67:1.11
54.67:123.09
VoxPoser [3]
RVT-2 [60]
73.33+2..1
2.67
CAP [5]
RVT-2 [60]
78.66.88.32
17.38.119.77
Scaling-up [4]
RVT-2 [60]
38.6712.31
33.33:12.31
92.00:14.000
MA (Ours)
RVT-2 [60]
85.3342.31
82.6742.31
18.67
82.6712.31
24.00:14.00
97.33-12.31
RLBench [33]
RVT-2 [60]
86.67+2.31
85.33+2.31
81.33+6.11
76.00=11.00
4.00+11000
97.33+22311
Data
Models
Take Take_ wwbrellaaaal..........a.wwaaaala Sort_mustare Spne_wine
Lamp_on
Put_kmife Pick lift
VoxPoser [3]
PerAct [34]
4.00::4.00
0.00$0.000
1.3322331
5.33+41.22
1.33:2231
5.67#11611
CAP [5]
PerAct [34]
13.33.110006
1.00
9.33.16.11
46.67.1
Scaling-up [4] PerAct [34]
4.00::4.00
0.00:00.00
12.33
76.00::4.00
5,33::2.31
53.33::10.06
MA (Ours)
PerAct [34]
84.00-16.93
53.3326.11
86.67.6.11
89.33
8:00:1:.00
33.33-12.31
RLBench [33]
PerAct [34]
38.67#50.80
53.33:311002
86.67=2.88
84.00+133.88
30.67=10.00
62.67=9.24
VoxPoser [3]
RVT-2 [60]
5.33#6.11
1.334:2311
1.342.31
2.67422.11
1.33:22.1
17.33:2.311
CAP [5]
RVT-2 [60]
89.33.6.11
85.33 8.32
52.00 10.58
82.661120.53
Scaling-up [4]
RVT-2 [60]
94.67+4.62
24.00 34.00+44.00
62.67 31
2..33
53.33
80.00=
MA (Ours)
RVT-2 [60]
94.7712.31
73.33 12.31
93.33 6.11
84.00:110.58
69.33
82.67 :: 12.22
RLBeench [33]
RVT-2 [60]
97.33-2.31
09.33:18.33
88.00 8.00
93.33-14.62
72.00 10.58
64.00:1 10.58
6.2.5
open jar
Filename: open_jar.. py
Task: Uncap the green jar.
Success Metric: The green jar is out of its pre-defined capped location.
6.2.6 pickup cup
Filename: Filename: pickup_cup.py
Task: Pick up the red cup.
Success Metric: Lift up the red cup above the pre-defined location.
6.2.7 take umbrella
Filename: take_ wtt__uubeellloorte out_ of__ofndd__ppp_teo .py
Task:
Pick up the umbrella out of the umbrella stand.
Success Metric: Lift up the umbrella of the umbrella stand.
6.2.8 sort mustard
Filename: port_mustard.py
Task: Pick up the yellow mustard bottle, and place it into the red container.
15

---

SoM
SoM Bounding Box
viewpoints
Selected viewpoint sequenic
Front view (Single)
Figure 6: Evaluation of visual prompting. We systematic evaluate 5 different visual prompting techniques, and found that selected viewpoint sequence yields the highest performance.
Success Metric: The yellow mustard bottle inside red container.
6.2.9 open wine
Filename: open_wine. py
Task: Uncap the wine bottle.
Success Metric: The wine bottle cap is out of its original position.
6.2.10 1amp on
Filename: lamp_on,
Task: Turn on the lamp.
Success Metric: The lamp light up.
6.2.11 put knife
Filename: put_knin
py
Task:
Pick up the knife and place onto the chopping board.
Success Metric: Knife chopping board.
6.2.12 push block
Filename:
Task: Push the red block down towards the green target.
Success Metric: The red block fails within the green target.
6.2.13 insert block
Filename: insert_block. py
Task: Push the green block into the jenga tower.
Success Metric: The green block inserted in.
6.2.14 pick & lift
Filename: pick_ pick_and_ir and_lift.pp
Task: Pick up the red cube.
Success Metric: The red eube is lifted up.
16

---

Differant vitual prompting techniques
Visuel hooepprgg
50000 931018)
10
5.00
All slappants
Sebected senepend Mrristnge
Technique
Figure 7: Results for visual prompting techniques (Left). We reported the various results for different visual prompting technique decision, and reported that selected viewpoint sequence yield the best performance-Simulationn
Simulation scene setup (Right). We leverage different camera for evaluation. "Pick soup and place the red comtainer"
Franka Emika Panda
Original
AR Marker for hand-eye
Kinect2 RGB-D Camera
"Pcck up the red container and place it onto the green mat"
Language Variation
'Pick up the red contniner by the black and while pattern Ede
Object-
specific
variation
Figure 8: Real-world experiment setup (Left). We up the real-world using this configuration. Robustness and generalization evaluation. We evaluated MANIPULATE-ANYTHINE against VoxPoser for capability in generalizing to different language instructions and also object-specific manipulation.
6.3
Real-world experiments
6.3.1 Robot hardware setup
The real-robot experiments use a Franka Panda manipulator with a parallel gripper. For perception, we use a Kinect-2 RGB-D camera mounted on a tripod, at an angle, pointing towards the tabletop. Kinect-2 provides RGB-D images of resolution 512 424 30Hz. The extrinsic between the camera and robot base-frame are calibrated with the easy hand-eye package. We use an ARUCO AR marker mounted on the gripper aid the calibration process, shown in Figure 8.
6.3.2 Additional real-world everyday manipulation tasks
Beyond the five real-world experiments used for systematically evaluating MANIPULATE--ANYHHIE we also have additional real-world demonstrations generated in zero-shot manner via MANIPULATE- ANYTHING. These demonstrations cover range of tasks, from reasoning tasks to more precise everyday tasks. All of the tasks can be seen in Fig. 10. We also demonstrates that MANIPULATE-
17

---

"Uncap the jors
"Open the top drower"
"Uncapp the jor
"Reosoning
SSieetbbects to then torsenndiddd color
"Pull out the knife*
"Unstock the green cup"
"Correct oil the dices to 6 front-facing"
"Pick up the odd color cube"
Figure 9: More real-world experiments.
Tiak Plax Geraaaiio
tak-plas-
gerersice.co
GPT-4V
Velll
Sad Mociit
Artice Geseraiion Medule
Sub takk Vasifcation
Owen-VL+
apperdted task plan
GPT-4V
theobeee te Brrnbal
GPT-4V
selection
GPT-VV
grandiont
Figure 10: In-depth Examination of VLMs for Manipulate-Anythen The system consists of five components that utilize VLM API calls. We have provided detailed breakdown of the inputs and outputs for each of these VLMSs for reference.
ANYTHING generate trajectories compares hand-crafted trajectories by humans depicted in Fig 11.
6.3.3 Baseline implementation details
For most of the baselines followed the original implementation with minor modifications, For the Code Policies baseline re-implemented most of the environment code using PyRep instead of PyBullet. This includes the implementation of various motion primitives that form the
18

---

CD=0.088
CD-0157
CD=0.056
CAP
VoxPoser
MA (Ours)
RLBench
Figure 11: Action Distribution for Generated Data: We compare the action distribution of data generated by various methods against human-generated demonstrations via RLBench the same set of tasks. We observed a high similarity between the distribution our generated data and the human-generated data. This is further supported by the computed CD between methods and the RLBench data, which yields the lowest (CD=0.056).
exposed API to the language model program. For example, one of such primitives is shown in the following Figure 12.
6.4 Error Breakdown
We examine the potential errors introduced by each respective VLM and explore ways to further improve the overall system. We conducted simulation experiments where tasks could be easily reset, focusing two major components where VLMs make decisions: "Perception error" and "Reasoning error." "Perception error" refers to mistakes made by the VLM in detecting target objects and selecting optimal viewpoints for generating task-specific grasp pose, primarily related to the selection of the Multi-viewpoint VLM. "Reasoning error" involves the Sub-task Verification Module, where the VLM responsible for deciding sub-tasks have been successfully completed. Due to resource constraints, conducted experiments the play-jenga task. We systematically replaced the VLMs in each component with a human, allowing the human make decisions. By comparing the system's performance with human decision-making to that with VLMs, we were able to quantify the errors caused by the VLMs.
6.5 Additional Ablation Studies
We conducted two main set of ablation studies, we first look at how different visual prompting works for sub-goal verification, and then we further evaluated MANIPULATE: ANYTHINGS robustness and generalization to language instructions another set of experiments.
Liner breakdown
Reeachin Eiter #wmaprton wit
13.5
Drference
For evaluating different visual prompts for sub-goal verifi- cation the put_block task, employed the following methods: 1) Set-of-Mark [62] on single view, 2) Set-of- Mark with bounding box annotation, 3) Concatenated all viewpoints, 4) Front view only, and 5) Selected viewpoint sequence as shown in Fig. 6. We observed that the selected viewpoint sequences were the most effective achieving correct sub-goal verification, obtaining the highest success rate shown in Fig. 7.
22.0
10.0
7.5
5.0
(MC 1lock
May Joag
Pvi Jule
Figure 13: Error breakdown. Error breakdown for three tasks from simula- tion.
We further evaluated the generalization capabilities of our model in terms of object-specific manipulation and robustness to changes in language instructions. For language instruction variations, we altered the instructions for the same scene and found that MANIPULAT--AYYTING outperforms VoxPoser by 60% over 25 episodes. For object-specifie vari- ations, where instructions targeted specific parts of objects, MANIPULATE - ANYTHIN outperformed VoxPoser by 16%, shown in Fig. 8.
19

---

6.6 Prompts
Prompts used for multi-viewpoint VLM selection, task plan generation, action generation and sub-task verification can be found below.
Multi-viewpoint VLM (Verification): Takes 4 images concatenated into a single frame with number annotated on top and returns a selection number of the most optimal viewpoint for verfying the sub-task. Simulation; wwt---iieeoooi-LLLie--
Multi-viewpoint VLM (Object centric action): Takes in 4 images concatenated into single frame with number annotated on top and returns selection number of the most optimal viewpoint for filter the grasp candidate poses. Simulation: Multi-viewpoint VLMM- verfiicttion-objct--cntiiii.
Task plan generation: Takes in natural language instruction, and outputs task plan in json file with the correct format.
Simulation: mosk-paan-geerrttonn prompt.txt
Takes in the selected viewpoint rollout along with the verification condition, and outputs binary 'yes' or 'no'. Simulation: sub-task--errfficttooo--ooo
Sub-task verification:
GenertnG:n Takes the action primitive and generates codes for executing the action in simulation.
Simulation: iitioo-generaatin.ttx
6.7 Best Task Plans and Action Primitives
We
with multi-processing during simulation obtain the best set of task plans and action primitives for any given task. This set of information is then used generate more data at scale for distilling policy. We have compiled all the task plan JSON files and action primitive code that achieved the highest success rates in these tasks. Best Task Plans: pest-task-plans.tr
Skills Library: skills-library.txt
References
[1] A. Brohan, N. Brown, Carbajal, Y. Chebotar, Dabis, C. Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817, 2022.
[2] O. X.-E. Collaboration. https://arxiv: http://arxx...o//as/////a/ 08864, 2023.
Open X-Embodiment: Robotic learning datasets and RT-X models.
[3] W. Huang, C. Wang, R. Zhang, Y. Li, J. Wu, and L. Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models. arXiv preprint arXiv:2307.05973, 2023.
[4] H. Ha, P. Florence, and S. Song. Scaling and distilling down: Language-guides robot skill acquisition. arXiv preprint arXiv:2307.144. 145355....,55, 2023.
[5] Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng. Code policies: Language model programs for embodied control. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 3493-9500 IEEE, 2023.
[6] Kaplan, S. McCandlish, Henighan, T. Brown, B. Chess, R. Child, S. Gray, A. Rad- ford, J. Wu, and D. Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08661. 2020.
20

---

[7] M. Cherti, R. Beaumont, R. Wightman, M. Wortsman, G. Ilharco, C. Gordon, C. Schuhmann, I.. Schmidt, and J. Jitsev. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2818-2829, 2023.
[8] J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark, et al. Training oppputll large language models. arXiv preprint arXiv:2203. 15556, 2022.
[9] C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35: 25278 22279445294, 2022.
[10] V. Ramanujan, T. Nguyen, S. Oh, A. Farhadi, and L. Schmidt. On the connection between pre- training data diversity and fine-tuning robustness. Advances Neural Information Processing Systems, 36, 2024.
[11] V. Udandarao, A. Prabhu, A. Ghosh, Y. Sharma, H. Torr, A. Bibi, Albanie, and M. Bethge. No" zero-shot" without exponential data: Pretraining concept frequency determines multimodal model performance. arXiv preprint 24t::24440042255 2024.
[12] S. Y. Gadre, G. Ilharco, A. Fang, J. Hayase, G. Smyrnis, T. Nguyen, R. Marten, M. Wortsman, D. Ghosh, Zhang, al. Datacomp: In search of the next generation of multimodal datasets. Advances Neural Information Processing Systems, 36, 2024.
[13] C. Zhou, P. Liu, P. Xu, S. Iyer, J. Sun, Y. Mao, X. Ma, A. Efrat, Yu, L. Yu, et al. Lima: Less more for alignment. Advances Neural Information Processing Systems, 36, 2024.
[14] T. Nguyen, S. Y. Gadre, G. Ilharco, S. Oh, and L. Schmidt. Improving multimodal datasets with image captioning. Advances in Neural Information Processing Systems, 36. 2024.
[15] T. Nguyen, G. Ilharco, M. Wortsman, S. Oh, and L. Schmidt. Quality not quantity: On the interaction between dataset design and robustness of clip. Advances in Neural Information Processing Systems, 15:21455-21466 2022.
[16] K. Lee, D. Ippolito, A. Nystrom, C. Zhang, D. Eck, C. Callison- and N. Carlini. Dedupli- cating training data makes language models better. arXiv preprint arXiv:2107.0666 2021.
[17] A. Fang, G. Ilharco, M. Wortsman, Y. Wan, V. Shankar, A. Dave, and L. Schmidt. Data determines distributional robustness in contrastive language image pre-training (clip). In International Conference on Machine Learning, pages 6216-6234. PMLR, 2022.
[18] S. Gururangan, A. Marasović, S. Swayamdipta, K. Lo, I. Beltagy, D. Downey, and N. A. Smith. Don't stop pretraining: Adapt language models to domains and tasks. arXiv preprint arXiv:2004. 10964, 2020.
[19] Y. Tian, L. Fan, K. Chen, D. Katabi, D. Krishnan, and P. Isola. Learning vision from models rivals learning vision from data, arXiv preprint 1xXiv:2312.17742 2023.
[20] H. Xu, S. Xie, X. E. Tan, P.-Y. Huang, R. Howes, Sharma, S.-W. Li, G. Ghosh, L. Zettlemoyer, and C. Feichtenhofer. Demystifying clip data. arXiv preprint arXiv:2309.16677 16671, 2023.
[21] Z. Chen, A. H. Cano, A. Romanou, A. Bonnet, Matoba, F. Salvi, M. Pagliardini, S. Fan, A. Köpf, A. Mohtashami, al. Meditron-70b: Scaling medical pretraining for large language models. arXiv preprint arXiv:231 1. 16079, 2023.
[22] Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pages 248-255. Icee, 2009.
21

---

[23] T.-Y. Lin, M. Maire, S. Belongie, J. Hays. P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In Computer Vision-FCCC 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
[24] C. Schuhmann, R. Beaumont, R. Vencu, C. W. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, P. Schramowski, S. R. Kundurthy, K. Crowson, L. Schmidt, R. Kaczmarezyk, and Jitsev. LAION-5b: An open large-scale dataset for training next generation image-text models. In Thirty-sixth Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2022. URL httb::://openrevieww id-M3Y74vmsMcY.
[25] C. Chi, Z. Xu, C. Pan, E. Cousineau, B. Burchfiel, S. Feng, R. Tedrake, and S. Song. Universal manipulation interface: In-the-wild robot teaching without in-the-wild robots. arXiv preprint 1rXiv:2402.10329 2024.
[26] C. Wang, H. Shi, W. Wang, R. Zhang, I.. Fei-Fei, and C. K. Liu, Dexcap: Scalable and portable mocap data collection system for dexterous manipulation. arXiv preprint arXiv:2403.07788 2024.
[27] J. Duan, Y. R. Wang, M. Shridhar, D. Fox, and R. Krishna. Ar2-d2: Training robot without a robot. arXiv preprint arXiv:2306. 13818, 2023.
[28] I. Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg. Progprompt: Generating situated robot task plans using large language models. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 1153-11530. IEEE, 2023.
[29] L. Wang, Y. Ling, Z. Yuan, M. Shridhar, C. Bao, Y. Qin, B. Wang, H. Xu, and X. Wang. Gensim: Generating robotic simulation tasks via large language models. arXiv preprint arX00.2310.01361, 2023.
[30] Y. Wang. Z. Xian, F. Chen, T.-H. Wang, Z. Erickson, D. Held, and C. Gan. Genbot: Generative simulation empowers automated robotic skill learning at scale. arXiv, 2023.
[31] S. Nasiriany, F. Xia, W. Yu, T. Xiao, J. Liang, I. Dasgupta, A. Xie, D. Driess, A. Wahid, Z. Xu, Q. Vuong, Zhang, T.-W. E. Lee, K.-H. Lee, P. Xu, S. Kirmani, Zhu, A. Zeng, K. Hausman, N. Heess, C. Finn, S. Levine, and Ichter. Pivot: Iterative visual prompting elicits actionable knowledge for vlms, 2024.
[32] H. Huang, F. Lin, Y. Hu, S. Wang, and Y. Gao. Copa: General robotic manipulation through spatial constraints of parts with foundation models. arXiv preprint 2tt.::20030084455 2024.
[33] James, Z. Ma, D. R. Arrojo, and A. Davison. Rlbench: The robot learning benchmark learning environment. IEEE Robotics and Automation Letters, 5(2):3019-3026, 2020.
[34] M. Shridhar, L.. Manuelli, and D. Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. In Conference on Robot Learning, pages 785-799, PMLR, 2023.
[35] M. Shridhar, Manuelli, and D. Fox. Cliport: What and where pathways for robotic manipula- tion. In Conference on Robot Learning, pages 894-906. PMLR, 2022.
[36] A. Brohan, Y. Chebotar, C. Finn, K. Hausman, A. Herzog, D. Ho, Ibarz, A. Irpan, E. Jang, R. Julian, et al. Do as can, not as i say: Grounding language in robotic affordances. In Conference on Robot Learning, pages 287-318. PMLR, 2023.
[37] K. Ehsani, T. Gupta, R. Hendrix, J. Salvador, L. Weins, K.-H. Zeng, K. Singh, Y. Kim, W. Han, A. Herrasti, et al. Imitating shortest paths in simulation enables effective navigation and manipulation in the real world. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2024.
22

---

[38] M. Ahn, D. Dwibedi, C. Finn, M. G. Arenas, K. Gopalakrishnan. K. Hausman, B. Ichter,
A. Irpan, N. Joshi, R. Julian, et al. Autort: Embodied foundation models for large scale orchestration of robotic agents. arXiv preprint arXiv:2401. 2024.
[39] Y. Wang, Z. Xian, F. Chen, T.-H. Wang, Y. Wang, K. Fragkiadaki, Z. Erickson, D. Held, and C. Gan. Robogen: Towards unleashing infinite data for automated robot learning via generative simulation. arXiv preprint arXiv:2311.0145. 2023.
[40] K. Grauman, A. Westbury, E. Byrne, Z. Chavis, A. Furnari, R. Girdhar, J. Hamburger, H. Jiang, M. Liu, X. Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18995-19012, 2022.
[41] C. R. Garrett, R. Chitnis, R. Holladay, B. Kim, Silver, L. P. Kaelbling, and T. Lozano- Lozano-Pérez. Integrated task and motion planning. Annual review of control, robotics, and autonomous systems, 4:265-293, 2021.
[42] T. Xie, S. Zhao, C. H. Wu, Y. Liu, Q. Luo, V. Zhong, Y. Yang, and T. Yu. Tex(2rewar:: Automated dense reward function generation for reinforcement learning. arXiv preprint arXiv:2309. / 1489, 2023.
[43] J. Zhang, K. Pertsch, J. Zhang, T. Nam, S. J. Hwang, X. Ren, and J. J. Lim. Sprint: Scalable semantic policy pre-training via language instruction relabeling. 2022.
[44] Y. Wang, Z. Sun, J. Zhang, Z. Xian, E. Biyik, D. Held, and Z. Erickson. Ri-vIm-f: Reinforcement learning from vision language foundation model feedback. arXiv preprint arXiv:2402.03681 2024.
[45] K. Lin, C. Agia, T. Migimatsu, M. Pavone, and J. Bohg. Text2motion: From natural language instructions to feasible plans, 2023.
[46] W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. Florence, A. Zeng. J. Tompson, I. Mordatch, Y. Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.056000 2022.
[47] Z. Liu, A. Bahety, and S. Song. Reflect: Summarizing robot experiences for failure explanation and correction, 2023.
[48] J. Zhang, J. Zhang, K. Pertsch, Z. Liu, X. Ren, M. Chang, S.-H. Sun, and J. Lim. Bootstrap your own skills: Learning to solve new tasks with large language model guidance. arXiv preprint 1xXi:2230011002 2023.
[49] J. Duan, W. Pumacay, N. Kumar, Y. R. Wang, S. Tian, W. Yuan, R. Krishna, D. Fox, A. Man- dlekar, and Y. Guo. Aha: A vision-language-model for detecting and reasoning over failures in robotic manipulation. arXiv preprint arXiv:2410.00371, 2024.
[50] F. Liu, K. Fang, P. Abbeel, and S. Levine. Moka: Open-vocabulary robotic manipulation through mark-based visual prompting. arXiv preprint arXiv:2403.031774 2024.
[51] B. Liu, Y. Dong, Y. Wang, Rao, Y. Tang. W.-C Ma, and R. Krishna. Coarse corre- spondence elicit 3d spacetime understanding in multimodal language model. arXiv preprint a0Xiv:2408.00754, 2024.
[52] W. Yuan, J. Duan, V. Blukis, W. Pumacay, R. Krishna, A. Murali, A. Mousavian, and D. Fox. Robopoint: A vision-language model for spatial affordance prediction for robotics. arXiv preprint arXiv:2406. 20721) 2024.
[53] Y. Hong, H. Zhen, P. Chen, S. Zheng, Y. Du, Z. Chen, and C. Gan. 3d-llm: Injecting the 3d world into large language models. Advances in Neural Information Processing Systems, 36: 20482-20494, 2023.
23

---

[54] A. Goyal, J. Xu, Y. Guo, V. Blukis, Y.-W. Chao, and D. Fox. Rvt: Robotic view transformer for 3d object manipulation. In Conference on Robot Learning, pages 694-710. PMLR, 2023.
[55] Y. R. Wang, Y. Zhao, H. Xu, S. Eppel, A. Aspuru-Guzik, F. Shkurti, and A. Garg. Mvtrans: Multi-view perception of transparent objects. In 2023 IEEE International Conference Robotics and Automation (ICRA), pages 3771-3778. IEEE, 2023.
[56] W. Yuan, A. Murali, A. Mousavian, and D. Fox. M212: Multi-task masked transformer for object-centric pick and place, 2023.
[57] J. Bai, S. Bai, S. Yang, Wang, S. Tan, P. Wang. J. Lin, C. Zhou, and Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308. 12966, 2023.
[58] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[59] I. A. Sucan, M. Moll, and L. E. Kavraki. The open motion planning library. IEEE Robotics & Automation Magazine, 19(4):72-82, 02.7722-,22 2012.
[60] A. Goyal, V. Blukis, J. Xu, Guo, Y-W. Chao, and D. Fox. Rvt-2: Learning precise manipula- tion from few demonstrations. arXiv preprint arXiv:2406.08545. 2024.
[61] Tiedemann, M. Aulamo, D. Bakshandaeva, M. Boggia, S.-A. Grönroos, T. Nieminen, A. Ra- ganato, Y. Scherrer, R. Vazquez, and S. Virpioja. Democratizing neural machine translation with opus-mt. Language Resources and Evaluation, pages 1-43, 2023.
[62] Yang, H. Zhang, F. Li, X. Zou, C. Li, and J. Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v. arXiv preprint arXiv:2310.11441, 2023.
24

---

332
def prinitive pick place(self, obj1 nase: str, ob12 nene: neme:-stry str) None: ioneerooot -ot-in--riinttivee picking- -ade--andd-aaaeeeeee pllapipcnee an-obiject
333
334
335
-Paranttrrs
336
337
+...+.0+11011..... ob11 str
338
The nane-of- the -ojject too pick
339
obj2
340
..... The nare-ot thethe objectbjetttt which to -lappaace the first
341
342
obj1 obji replace( "*"""""_ ""
343
-bbj2_naree obj2_nane rep replaace( lace(" """""""))
344
365
pick nhj sel aelf.gge F-get
prists(nhj11 nane)
346
asseet pick, nbj ns Nonn
347
place_obj exself.get aef.geeeeoi__ttttte obd__f _oo__ffeexistssseeeeeeeeesss__s_ exxists (obs2_nane)
place, no-
349
350
piikkpos, is settf welffget__oneeeeeeeeeeeeeeeeeeee get_grasp__eetee cast (Shape, biok_obb),, filf.geteee ee, orientation()
352
353
place_pos, pos, place_rat seteerrttssseee setf. get_placeee__ppeppppeeopeetoeeee place_pose( (Shape, place_ob)), self, get_ee, orientatiinn)
355
356
pre_pick_poc post_graep greep_pes--- pickkppse ng..areyy( 0.21)
358
place placa, pos in aamny ([s (,,,,,,,,, B.0, (822]
359
pre-piede position sel. movep (se, xyz pick_pos, wotter-ick_rett
361
362
-white-na not op.attclese (are_pickk_appp pos, self. pocition)), atal-le 2):
364
365
--CCose in-to pick pe--piik---siiee pick-pasition
366
self .novep wyzzpick_ppo
pick_ rot, ignore citiisions while not no.allelose (pick_pos, get..get.ee. get_ggtt___e__eeeeee se_positionn) etol 2)
367
368
370
# CLose the gripper
371
wwite--ot natte-nttset...... robott, www.ot.grrpprr..ooo. aaipper.actuato(: -0.04):
self (step()
373
self._robu wrwpger..e....................ee grweer.grasppeeee (sick_obb)
374
past mest-ggab--oe prab-positiion
376
self (pust_ _ere___rrr__sss Lgmore cuw.issonsssTrre)
377
wwhlte-nn not no. allclose
past. cerr__gas____e__ pos, setf.get se positian(), atot-1l-2
379
360
sew.. step((
301
382
hove pre-place position (ta- avold wtw.dd-oolllsaaai Listars) setf [
363
384
-0 xyz pro_ j__paace_pee p eeterter re_eoter-ppace place_rot, ignore_ caaree_ootiiio Lisions True
585
386
while not no. ao..oee (pre_ place_pos, self. get_ee__t____eeeeeeeeeeeeeee_eee pos_oootooon atoi=le-2): -ee -self.step()
307
588
389
nove to place position
390
self movee (se xuz--y--ppppp pacc-paae_pppee se_euter-pt paaprpppppeetttteeeeeeeeee ignore_ cotoreeottto Lisions True)
391
while ng.allelesee pas, selff.get _ec_t_____p___eeeeeeeeeeeeeee pospitiiio), stol
392
393
Open-the gripper-te release ohjett self rabot. helpper.releesee while not self._ robot. gripoer. aitpoer.cttute(.... (1.8, -0.04): step()
395
396
397
Figure 12: Example of one of the primitives implemented for Code as Policies
25


Reviews:
[{"review_id": "zhNDxzdkAY", "reviewer": "Reviewer_1Hnu", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors propose Manipulate-Anything, a framework for generating data for robot manipulation which uses VLMs to help plan and execute tasks to generate data for policies to learn from.", "summary_of_recommendation": "I recommend a weak reject because the paper, while well-motivated and with some good results, suffers from writing issues and lacks comprehensive experimental validation to fully support its claims. Addressing these concerns would significantly strengthen the work and change my score.", "technical_quality": 2, "clarity_of_presentation": 2, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "Integrated in the review, please see the review.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VFjPmU9Ped", "reviewer": "Reviewer_Ssao", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes a framework, named \"Manipulate-Anything\", which leverages VLMs to plan a sequence of sub-goals (text) and an off-the-shelf grasp prediction model to generate grasp poses addressing sub-goals. A sub-goal verification module is introduced to support retry behaviors. It addresses some technical limitations in prior works like VoxPoser and Code as Policies.", "summary_of_recommendation": "The paper shows some technical improvements over prior works, while the framework and experiments fail to show that it can be scaled up.", "technical_quality": 2, "clarity_of_presentation": 3, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 1, "questions_for_rebuttal": "1. It is unclear how the candidate grasp poses are filtered in L128-L129. Especially, how the grasp poses are represented for VLMs to understand (e.g., by text or visual)?\n2. What few-shot demonstrations are used for in-context learning to select viewpoints (L23 in Appendix)? Are they the same for all tasks, or are task-specific demonstrations needed?\n3. The authors mention several times \"avoiding errors due to occlusion or ambiguity from a single viewpoint\". However, if the image is re-rendered from the same point cloud (perhaps partial), it could not handle occlusion caused by the camera setup (e.g., 1 front camera in the real-world experiments). Choosing a different viewpoint may help VLMs, but can the authors use quantitative evidence to support the claim for avoiding errors due to occlusion?\n4. For re-attempting the action generation step, what will happen if no actions lead to success? In L145-L146, it is said that \"the next sub-goal is attempted\", but it does not sound convincing if the skipped sub-goal is essential for the full task. And what will happen if an object is already grasped by the end-effector but with a \"bad\" grasp pose (e.g., the blade of a knife is held and thus the knife can not be inserted into the holder while VLMs happen to fail to find this issue when handling the previous sub-goal)?\n5. Can the authors also use the ground truth simulation state information for zero-shot experiments in Sec 4.1? It can indicate how accurate the information of the target object’s asset names or positions is needed. \n6. What are \"human-generated demonstrations\" in Sec 4.2? Are they tele-operated, or generated by hand-crafted scripts (e.g., provided by RLBench)? According to Table 2, it seems to be the latter.\n7. Only PerAct is used for behavior cloning experiments while its limitations are also noticed by authors (e.g., poor for long-horizon memory). It is better to add more methods like ACT or Diffusion Policies to provide more comprehensive and convincing comparison.\n8. Why is VoxPoser not included in Table 3? For 6 demonstrations generated by Manipulate-Anything (MA), do the authors repeatedly run MA until 6 demos are collected?\n9. Which single task (L247) is used for the ablation study on scaling? Or is the success rate in Figure 6 averaged over 12 simulated tasks?\n10. Can the authors provide error breakdown (like Sec 4.4 in VoxPoser) to analyze which aspects the proposed method improve over baselines?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3aUPNSrBck", "reviewer": "Reviewer_3U2k", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a VLM-based method for manipulation and then uses the method to generate data for behavioral cloning.", "summary_of_recommendation": "Post rebuttal: the paper still has room for improvement in clarity, but I think it is above the bar for publication.", "technical_quality": 4, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "1. In-context learning is used in multiple places. For each, what are the examples used for the context? Are these examples always the same, or are they task-dependent? How different from the test tasks are they?\n2. The paper occasionally refers to a “target object” as part of the method. Is there an assumption that each natural language task description explicitly mentions one target object? If not, where do the target objects come from?\n3. The appendix says that code snippets are “appended to a skill library for future use.” Is this talking about within a single task? What is the “future use” here?\n4. L131 refers to a “detected bounding box” -- where in the pipeline is this bounding box detected?\n5. How does the approach decide whether to use an object-centric or agent-centric action?\n6. How many actions per subgoal are generated?\n7. The paper mentions “recovery behavior into the collected demonstrations.” Does that mean that the entire collected trajectory is used for behavioral cloning? So the learned policy will also imitate the parts of the trajectory that lead to subgoal verification failures?\n8. Which aspects of MANIPULATE-ANYTHING are responsible for generating diverse data?\n9. Can you discuss and/or analyze the common failure modes of MANIPULATE-ANYTHING?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents a framework for manipulating objects by combining VLMs and foundation models to help plan and execute tasks.

Strengths: The reviewers agree on the importance of the problem setup. They also acknowledge the value of the real-world evaluation.

Weaknesses: The main criticism lies in the technical novelty of the method and its relationship to prior works, while concerns about overclaiming with the title. Improving the clarity and reproducability of the paper is suggested by  3U2k, Ssao and 1Hnu suggest stronger baseline methods among others.

Post response period: The authors have addressed most of reviewer's concerns. In particular, they have added baselines, more experiment videos and improved the overall clarity of the presentation. However, being able to manipulate any object still seems to be overclaiming. Given the author response, the updated ratings and their justifications, the AC recommends the paper to be accepted with a poster presentation.
Decision: Accept