VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models
Authors: Wenlong Huang
Chen Wang
Ruohan Zhang
Yunzhu Li
Jiajun Wu
Li Fei-Fei
Venue: CoRL 2023 Oral (2023)
Keywords: Manipulation
Large Language Models
Model-based Planning

Abstract: Large language models (LLMs) are shown to possess a wealth of actionable knowledge that can be extracted for robot manipulation in the form of reasoning and planning. Despite the progress, most still rely on pre-defined motion primitives to carry out the physical interactions with the environment, which remains a major bottleneck. In this work, we aim to synthesize robot trajectories, i.e., a dense sequence of 6-DoF end-effector waypoints, for a large variety of manipulation tasks given an open-set of instructions and an open-set of objects. We achieve this by first observing that LLMs excel at inferring affordances and constraints given a free-form language instruction. More importantly, by leveraging their code-writing capabilities, they can interact with a vision-language model (VLM) to compose 3D value maps to ground the knowledge into the observation space of the agent. The composed value maps are then used in a model-based planning framework to zero-shot synthesize closed-loop robot trajectories with robustness to dynamic perturbations. We further demonstrate how the proposed framework can benefit from online experiences by efficiently learning a dynamics model for scenes that involve contact-rich interactions. We present a large-scale study of the proposed method in both simulated and real-robot environments, showcasing the ability to perform a large variety of everyday manipulation tasks specified in free-form natural language.

VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models
Wenlong Huang', Chen Wang', Ruohan Zhang', Yunzhu Li Jiajun Wu', Li Fei-Fei!
Stannoord University University of Illinois Chaamaaha
Large language models (LLMs) are shown to possess a wealth of ac- tionable knowledge that can be extracted for robot manipulation in the form of reasoning and planning. Despite the progress, most still rely on pre-defined mo- tion primitives to carry the physical interactions with the environment, which remains major bottleneck. In this work, we aim to synthesize robot trajecto- ries, i.e., dense sequence of 6-DoF end-effector waypoints, for large variety of manipulation tasks given an open-set of instructions and an open-sel of objects. We achieve this by first observing that LLMs excel at inferring affordances and constraints given free-form language instruction. More importantly, by leverag- ing their code- writing capabilities, they can interact with a vision- laiguaee model (VLM) compose 3D value maps ground the knowledge into the observation space of the agent. The composed value maps then used in model-basee planning framework to zero-shot synthesize closed-loop robot trajectories with robustness to dynamic perturbations. We further demonstrate how the proposed framework can benefit from online experiences by efficiently learning a dynam- ics model for scenes that involve contact-rich interactions. We present large- scale study of the proposed method both simulated and real-robot environments, showcasing the ability to perform large variety of everyday manipulation tasks specified in free-form natural language. Project website: githube..
Abstract:
Keywords: Manipulation, Large Language Models, Model-based Planning
Open the top drawer, and watch for that vase!
Large
Vision Language
Language
Model
Model
"Sort
vitanin botte?
Code </>
'Meesure weight
"Cloee
"Sweep dustpan"
phone"
Madion
SD Value Map
high
reaved
"Hing tnwel rack"
"Press doan moisturiver
"Sel Sabw pesti?
"Turn on lamp"
Figure VOxPOSEE extracts language-conditionel affordances and constraints from I.I.Ms and grounds them to the perceptual space using VL.Ms, using code interface and without additional training to cither com- ponent. The composed map is referred to as 3D value map, which enables zero-shot synthesis of trajectories for large varieties of everyday manipulation tasks with open-set of instructions and open-set of objects.
Correspondence to Wenlong Huang < <wenlongh @ stanford.edu.>.
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

Introduction
Language is compressed medium through which humans distill and communicate their knowledge and experience of the world. Large language models (LLMs) have emerged as promising approach to capture this abstraction, learning to represent the world through projection into language space [1- 4]. While these models are believed to internalize generalizable knowledge as text, remains a question about how use it enable embodied agents to physically act in the real world.
We look at the problem of grounding abstract language instructions (c.g. "set up the table") in robot actions [5]. Prior works have leveraged lexical analysis to parse the instructions [6-8], while more recently language models have been used to decompose the instructions into a textual sequence of steps [9-11]. However, to enable physical interactions with the environment, existing approaches typically rely on repertoire of pre-defined motion primitives (i.e., skills) that may be invoked by an LLM or a planner, and this reliance individual skill acquisition is often considered major bottleneck of the system due to the lack of sarge-scale robotic data. The question then arises: how can we leverage the wealth of internalized knowledge of L.MSs at the even fine-graine tne-grained action level for robots, without requiring laborious data collection or manual designs for each individual primitive?
In addressing this challenge, we first note that it is impractical for LLMs to directly output control actions in text, which are typically driven by high-frequency control signals high-dimensional space. However, we find that LLMs excel at inferring Iigguae--condtionns affordances and con- straints, and by leveraging their code-writing capabilities, they can compose dense 3D voxel maps that ground them in the visual space by orchestrating perception calls (e.g., via CLIP [12] or open- vocabulary detectors [13-15]) and array operations (e.g., via NumPy [16]). For example, given an instruction "open the top drawer and watch out for the vase", LLMs can be prompted to infer: 1) the top drawer handle should be grasped, 2) the handle needs to be translated outwards, and 3) the robot should stay away from the vase. By generating Python code invoke perception APIs, LLMs can obtain opatial-geomette information of relevant objects parts and then manipulate the 3D voxels to prescribe reward cost at relevant locations in observation space (e.g., the handle region is assigned high values while the surrounding of the vase is assigned low values). Finally, the com- posed value maps can serve as objective functions for motion planners to directly synthesize robot trajectories that achieve the given instruction without requiring additional training data for each task or for the LLM. An illustration diagram and subset of tasks we considered are shown in Fig. 1.
We term this approach VOXPOSER, formulation that extracts affordances and constraints from LLLMs to compose 3D value maps in observation space for guiding robotic interactions. Rather than relying on robotic data that are often of limited amount or variability, the method leverages LLMs for open-world reasoning and VLMs for generalizable visual grounding in a model- planning framework that directly enables physical robot actions. We demonstrate its zero-shot generalization for open-set instructions with open-ser objects for various everyday manipulation tasks. We fur- ther showcase how VoxPoser can also benefit from limited online interactions efficiently learn dynamics model that involves contact-rich interactions.
2 Related Works
Grounding Language Instructions. Language grounding has been studied extensively both in terms of intelligent agents [19-22] and of robotics [23, 6, 24, 25, 5, 7, 26), where language can be used as tool for compositional goal specification [5, 27-33], semantic anchor for training multi- modal representation [12, 34, 35], or as an intermediate substrate for planning and reasoning [36- 38, 9, 10, 39, 40J. Prior works have looked at using classical tools such as lexical analysis, formal logic, and graphical models to interpret language instructions [27, 7, 6, 26]. More recently, end- to-end approaches. popularized by successful applications to offline domains [41-43, 1]. have been applied to directly ground language instructions in robot interactions by learning from data with
'The approach also bears resemblance and connections to potential field methods in path planning [17] and constrained optimization methods in manipulation planning [18].

---

language annotations, spanning from model learning [44], imitation learning [45, 46, 30, 17754], to reinforcement learning [55-57]. Most closely related to our work is Sharma et al. [50], where an end-to-end cost predictor is optimized via supervised learning to map language instructions to 2D costmaps, which are used to steer motion planner to generate preferred trajectories in a collision- free manner. contrast, we rely on pre-trained language models for their open-world knowledge and tackle the more challenging robotic manipulation in 3D.
Language Models for Robotics. Leveraging pre-trained language models for embodied applica- tions is an active area of research, where a large body of works focus on planning and reasoning with language models [9-11, 58, 31, 39, 59-72, 36, 73, 74]. To allow language models to perceive the physical environments, textual descriptions of the scene [39, 11, 59] or perception APIs [75] can be given, vision can be used during decoding [67] can be directly taken as input by multi-modal language models [68, 2]. In addition perception, truly bridge the perception-action loop, an embodied language model must also know how to act, which typically is achieved by library of pre-defined primitives. Liang et al. [75] showed that LLMs exhibit behavioral commonsense that can be useful for low-level control. Despite the promising signs, hand-designed motion primitives are still required, and while LLMs are shown to be capable of composing sequential policy logic, remains unclear whether composition can happen at spatial level. A related line of works has also explored using LLMs for reward specification in the context of reward design [76] and exploration in reinforcement learning [77-80], and human preference learning [81]. In contrast, we focus exclu- sively grounding the reward generated by LLMs in the 3D observation space of the robot, which we identify as most useful for manipulation tasks.
Learning-based Trajectory Optimization. Many works have explored leveraging learning-based approaches for trajectory optimization. While the literature is vast, they can be broadly categorized into those that learn the models [82-90] and those that learn the cost/reward or constraints [91- 94, 50, 95], where data are typically collected from in-domain interactions. To enable generaliza- tion in the wild, parallel line of works has explored learning task specification from large-scale offline data [96-98, 35, 34, 44, 99, 100, 54], particularly egocentric videos [101, 102], or leverag- ing pre-trained foundation models [103-105, 33, 106, 107]. The learned cost functions are then used by reinforcement learning [103, 100, 108], imitation learning [98, 97], or trajectory optimiza- tion [96, 35] to generate robot actions. In this work, we leverage I.LMs for zeno-shot in-the-wild cost specification with superior generalization. Compared prior works that leverage foundation models, we ground the cost directly in 3D observation space with real-time visual feedback, which makes VoxPoser amenable to closed-loop MPC that's robust in execution.
3 Method
We first provide the formulation of VoxPoser as an optimization problem (Sec. 3.1). Then we de- scribe how VoxPoser can be used as a general zero-shot framework to map language instructions to 3D value maps (Sec. 3.2). We subsequently demonstrate how trajectories can be synthesized in closed-loop for robotic manipulation (Sec. 3.3). While zero-shot in nature, we demonstrate how VoxPoser can learn from online interactions to efficiently solve contact-rich tasks (Sec. 3.4).
3.1 Problem Formulation
Consider a manipulation problem given as a free-form language instruction L (c.g.. "open the top drawer"). Generating robot trajectories according to L can be very challenging because L may be ar- bitrarily long-horizon or under-specified (i.e., requires contextual understanding). Instead, we focus on individual phases (sub-tasks) of the problem & that distinctively specify a manipulation task (e.g.. "grasp the drawer handle", "pull open the drawer"), where the decomposition L - (L1, (2.,,,....4 ex) is given by high-level planner (e.g., an LLM or a search-base based planner) 2. The central problem investigated in this work is to generate motion trajectory of for robot r and each manipulation
Note that the decomposition and sequencing of these sub-tasks are also done by LLMs in this work, though do not investigate this aspect extensively is not the focus of our contributions.
3

---

A Cam
offorderce pan 100, 1000
Vision Language
zonsorssze fondles desectt handle"))
Larada poste) sommdeyu repuhandiee
Model
ext-13
a,y tonchend pos resum anooch(eos)
Minn 62
Affordance Mape
der constrennt (eac):
Open top Please also watch out for that vase!
Large
vases desecst vise
Language Model
vare ecoaperey-grid
reauns snoothOnos))
(a) 3D Value Map Composition
Constraint Maps
Motion Planning
Figure 2: Overview of VOXPOSER. Given the RGB-D observation of the environment and a language in- struction, LLMs generate code, which interacts with VLMs, produce sequence of 3D affordance maps and constraint maps (collectively referred value maps) grounded the observation space of the robot (a). The composed value maps then serve as objective functions for motion planners to synthesize trajectories for robot manipulation (b). The entire process does not involve any additional training.
phase described by instruction f;- We represent T. as sequence of dense end-effector waypoints to be executed by an Operational Space Controller [109], where each waypoint consists of desired 6-DoF end-effector pose, end-effector velocity, and gripper action. However, is worth noting that other representations trajectories, such as joint space trajectories, can also be used. Given the i-th sub-task described by l, we formulate an optimization problem defined as follows:
min Frask (T; Es) + (Fl)))Fooonaae coontrl((T))) subject to C(T,)
(1)
where T, the evolution of enviromment state, and r.r S T, the robot trajectory. Frask scores the extent of T, completes the instruction while Foontrot specifies the control costs, e.g., encourage of to minimize total control effort total time. C(T,) denotes the dynamics and kinematics con- straints, which are enforced by the known model of the robot and physics-based or tearning-based model of the environment. By solving this optimization for each sub-task l, we obtain a sequence of robot trajectories that collectively achieve the overall task specified by the instruction L.
3.2 Grounding Language Instruction via VoxPoser
Calculating Frask with respect to free-form language instructions is extremely challenging, not only because of the rich space of semanties language can convey but also because of the lack of robot data labeled with T and f. However, we provide critical observation that large number of tasks can be characterized by voxel value map V e ux in robot's observation space, which guides the motion of "entity of interest" in the scene, such as the robot end-effector, an object, an object part. For example, consider the task "open the top drawer" and its first sub-task "grasp the top drawer handle" (inferred by L.Ms)) Fig. 2. The "entity of interest" is the robot end-effector, and the voxel value map should reflect the attraction toward the drawer handle. By further commanding "watch out for the vase", the map can also be updated reflect the repulsion from the vase. We denote the "entity of interest" as and its trajectory as 7 Using this voxel value map for given instruction f,, Frask can be approximated by accumulating the values of traversing through V., formally calculated as Frask = - Σ Vpf)), where pf E N3 is the discretized (æ, y, 2) position of e step j-
Notably, we observe large language models, by being pre-trained on Internet-scale data, exhibit pabilities not only to identify the "entity of interest" but also to compose value maps that accurately reflect the task instruction by writing Python programs. Specifically, when an instruction is given as a comment in the code, LLMs can be prompted to 1) call perception APIs (which invoke vision- language models (VLM) such as an open- vocabulacy detector [13-15]) to obtain information of relevant objects, 2) generate NumPy operations to manipulate 3D arrays, and 3) pre- scribe precise values relevant locations. We term this approach as VOxPOSER Concretely, we aim to obtain voxel value map V VoxPoser(o' (.) by prompting LLM and executing the code via Python interpreter, where of is the RGB-D observation at time t and l is the current

---

"swisop the frash to the biue dusipan"
"push claso the top diawer
t-2
"tum lamp
"apen the vitamin bottle on the rightt
"lake oul the breed from the toaster and put nat on the wooden plate" Figure 3: Visualization of composed 3D value maps and rollouts in real-world environments. The top row demonstrates where "entity of interest" is an object or part, and the value maps guide them toward target positions. The hottom two rows showcase tasks where "entity of interest" is the robot end-effector. The bottom-most task involves two phases, which are also orchestrated by LLMS.
instruction. Additionally, because V is often sparse, we densify the voxel maps via smoothing operations, as they encourage smoother trajectories optimized by motion planners.
Additional Trajectory Parametrization. The above formulation of VoxPoser uses LLMs to com- pose V N + E to map from discretized coordinates in voxel space real-valued "cost", which we can use to optimize path consisting only of the positional terms. To extend to SE(3) poses, we can also use LLMs to compose rotation maps V, N3 - SO(3) at coordinates relevant to the task objectives (c.g., "end-effector should face the support normal of the handle"). Similarly, we further compose gripper maps V, N + {0, } to control gripper open/close and velocity maps V. No R specify target velocities. Note that while these additional trajectory parametriza- tions mapped real-valued "cost", they also be factored in the optimization procedure (Equation 1) to parametrize the trajectories.
3.3 Zero-Shot Trajectory Synthesis with VoxPoser
After obtaining the task cost Frasks to plan a motion trajectory. We use simple zeroth-order optimization by randomly sampling trajec- tories and scoring them with the proposed objective. The optimization is implemented in model predictive control framework that iteratively replans the trajectory at every step using the current observation to robustly execute the trajectories even under dynamic disturbances where either a learned or physics-based model can be used. However, because VoxPoser effectively provides "dense rewards" in the observation space and we are able replan at every step, we surprisingly find that the overall system already achieve large variety of manipulation tasks considered in this work even with simple heuristics-baad based models. Since some value maps defined over "entity of interest", which may not necessarily be the robot, we also use the dynamics model to find the needed robot trajectory to minimize the task cost (i.e., what interactions between the robot and the environment achieve the desired object motions).
approach the full problem defined in Equation
3.4 Efficient Dynamics Learning with Online Experiences
While Sec. 3.3 presents a zero-shot framework for synthesizing trajectories for robot manipula- tion, VoxPoser can also benefit from online experiences by efficiently learning dynamics model. Consider the standard setup where robot interleaves between 1) collecting environment transition data (o,, a, 0141), where o, is the environment observation time and a, MPC(o.), and 2) training a dynamics model go parametrized by 0 by minimizing the I.2 loss between predicted next
Although involving an LLM in the loop, closed- loop execution is possible because the generated code remains the same throughout task &,, which allows cache its output for the current task.
5

---

observation 6i+1 and 0++1- A critical component that determines the learning efficiency is the ac- tion sampling distribution P(a, (o,) in MPC, which typically is a random distribution over the full action space A. This is often inefficient when the goal is to solve a particular task, such as opening a door, because most actions do not interact with the relevant objects in the scene (i.e., the door handle) nor do they necessarily interact with the objects in a meaningful way (i.e., pressing down the door handle). Since VoxPoser synthesizes robot trajectories with LLMs, which have wealth of commonsense knowledge, the zero-shot synthesized trajectory Tá can serve as useful prior to bias the action sampling distribution P(a (0,; -T), which can significantly speed up the learning process. In practice, this can be implemented by only sampling actions in the vicinity of Tō by adding small noise E to encourage local exploration instead of exploring in the full action space A.
4 Experiments and Analysis
We first discuss our implementation details. Then we validate VoxPoser for real-world everyday ma- nipulation (Sec. 4.1). We also study its generalization in simulation (Sec. 4.2). We further demon- strate how VoxPoser enables efficient learning of more challenging tasks (Sec. 4.3). Finally, we analyze its source of errors and discuss how improvement can be made (Sec. 4.4).
LLMs and Prompting. We follow prompting structure by Liang et al. [75], which recursively calls LLMs using their own generated code, where each language model program (LMP) is responsible for unique functionality (e.g., processing perception calls). We use GPT-4 [2] from OpenAl API. For each LMP, we include 5-20 example queries and corresponding responses part of the prompt. An example can be found in Fig. 2 (simplified for clarity). Full prompts are in Appendix.
VLMs and Perception. Given an object/part query from LLMs, we first invoke open-vocab detector OWL-VIT [15] to obtain a bounding box, then feed it into Segment Anything [I 10] to obtain mask, and finally track the mask using video tracker XMEM [111]. The tracked mask used with RGB-D observation reconstruct the object/part point cloud.
Value Map Composition. We define the following types of value maps: affordance, avoidance, end- effector velocity, end-effector rotation, and gripper action. Each type uses a different LMP, which takes in an instruction and outputs a voxel map of shape (100, 100, 100, k), where ic differs for each value map (e.g.. k for affordance and avoidance as it specifies cost, and k it specifies SO))). We apply Euclidean distance transform to affordance maps and Gaussian filters for avoidance maps. On top of value map LMPs, define two high-level LMPs to orchestrate their behaviors: planner takes user instruction L as input (e.g., "open drawer") and outputs a sequence of sub-tasks 4iN and composer takes in sub-task & and invokes relevant value map LMPs with detailed language parameterization.
for rotation as
Motion Planner. We consider only affordance and avoidance maps in the planner optimization, which finds sequence of collision-fres free end-effector positions PI.v R using greedy search. Then enforce other parametrization at each p by the remaining value maps (c.g.. rotation map, velocity map). The cost map used by the motion planner is computed as the negative of the weighted sum of normalized affordance and avoidance maps with weights 2 and 1. After 6-DoF trajectory is synthesized, the first waypoint is executed, and then a new trajectory is re-planned at 5 Hz.
Dynamics Model. We use the known robot dynamics model in all tasks, where it used in motion planning for the end-effector to follow the waypoints. For the majority of our considered tasks where the "entity of interest" is the robot, no environment dynamics model is used (i.e., scene is assumed to be static), but we replan at every step to account for the latest observation. For tasks in which the "entity of interest" is an object, we study only a planar pushing model parametrized by contact point, push direction, and push distance. We use a heuristic- dynamics model that translates an input point cloud along the push direction by the push distance. We use MPC with random shooting to optimize for the action parameters. Then pre-defined pushing primitive is executed based the action parameters. However, note that primitive is not necessary when action parameters are defined over the end-effector or joint space of the robot, which would likely yield

---

U-Net
Language Models
LLM + Prim. [75]
VoxPoser
Train/Test
Category
MP [50]
Prim. [75] MP (Ours)
Task
Static
Dist.
Static
Dist.
SI SA
Object Int.
21.0%
41.0%
64.0%
Move & Avoid 0/10
0/10
9/10
8/10
SI SA
Composition
53.8%
43.8%
77.5%
Set Up Table
7/10
0/10
9/10
7/10
SI UA
Object Int. Composition
3.0%
46.0%
60.0%
Close Drawer
0/10
0/10
10/10
7/10
SI UA
3.8%
25.0%
58.8%
Open Bottle
5/10
0/10
7/10
5/10
Sweep Trash 0/10
0/10
9/10
8/10
UI UA
Object Int. 0.0%
17.5%
65.0%
UI UA
Composition 0.0%
25.0%
76.7%
Total
24.0%
0.0%
88.0%
70.0%
Table 2: Success rate in simulated domain. "SI" and "UI" are seen and unseen instructions. "SA" and "UA" are seen and unseen attributes. VoxPoser outperforms both base- lines across 13 tasks from two categories both seen and unseen tasks and maintains similar success rates.
Table 1: Success rate in real-world domain. Vox- Poser performs everyday manipulation tasks with high success and more robust to disturbances than the baseline using action primitives.
smoother trajectories but takes more time for optimization. We also explore the use of a learning- based dynamics model in Section 4.3, which enables VoxPoser to benefit from online experiences.
4.1 VoxPoser for Everyday Manipulation Tasks
We study whether VoxPoser can zero-shot synthesize robot trajectories to perform everyday manip- ulation tasks in the real world. Details of the environment setup can be found in Appendix A.3. While the proposed method can generalize to an open-set of instructions and an open-set of objects as shown in Fig. 1, we pick 5 representative tasks to provide quantitative evaluations in Table 1. Qualitative results including environment rollouts and value map visualizations are shown in Fig. 3. We find that VoxPoser can effectively synthesize robot trajectories for everyday manipulation tasks with high average success rate. Due to fast replanning capabilities, it is also robust external dis- turbances, such moving targets/obstacles and pulling the drawer open after it has been closed by the robot. We further compare to a variant of Code as Policies [75] that uses LLMs to parameterize pre-defined list of simple primitives (e.g., move_to_pont chaining sequential policy logic, the ability to compose spatially while considering other constraints under a joint optimization scheme is a more flexible formulation, unlocking the possibility for more manipulation tasks and leading to more robust execution.
open_gr www.gripper ipper). We find that compared to
4.2 Generalization to Unseen Instructions and Attributes
To provide rigorous quantitative evaluations on generalization, we set up a simulated environment that mirrors real-world setup [112] but features 13 highly-randomizzale tasks with 2766 unique instructions. Eash task comes with a templated instruction (e.g., "push [obj] to [pos]") that con- tains randomizable attributes chosen from pre-defined list. Details are in Appendix A.4. Seen instructionssattriiuess may appear in the prompt (or in the training data for supervised baselines). The tasks are grouped into 2 categories, where "Object Interactions" are tasks that require interac- tions with objects, and "Spatial Composition" are tasks involving spatial constraints (e-g., moving slower near particular object). For baselines, we ablate the two components of VoxPoser, LLM and motion planner, by comparing to variant of [50] that learns U-Net [113] to synthesize costmaps for motion planning. Table 2 shows the success rates averaged across 20 episodes per task. We find VoxPoser exhibits superior generalization in all scenarios. Compared to learned cost specification, LLMs generalize better by explicitly reasoning about affordances and constraints. On the other hand, grounding LLM knowl- edge in robot perception through value map composition rather than directly specifying primitive parameters offers more flexibility that generalizes beyond the prompt examples.
variant of [75] that combines an LLM with primitives and
4.3 Efficient Dynamics Learning with Online Experiences
As discussed in Sec. 3.4, we investigate how VoxPoser can optionally benefit from online experi- ences for tasks that involve more intricacies of contact, such opening doors, fridges, and windows, in simulated environment. Specifically, we first synthesize k zero-shot trajectories using VoxPoser,
7

---

Zero-Shot
No Prior
w/ Prior
UNit MP
Task
Success
Success
Time(s)
Success
Time(s)
Door
6.7% +4mm
58.3+44s
TLE 88.3% 142.3
Window
3.3% 36.7%
TLE 80.0%
137.0a75
Fridge 18.3% 70.0%+29%
TLE
91.7% 11 71.0-4
Table 3:
VoxPoser enables efficient dynamies learning by
25.00%
using zero-shot synthesized trajectories as prior. TLE (time limit exceeded) means exceeding 12 hours. Results are re- ported over 3 runs different seeds.
Dynamios Enirr
Pecoption Error
Speciicctton Enerr
Ertrr
Figure Error breakdown of components. Vox- Poser significantly reduces specification error.
each represented as a sequence of end-effector waypoints, that act as priors for exploration (e.g., "handle needs to be pressed down first in order to open a door"). Then an MLP dynamics model is learned through an iterative procedure where the agent alternates between data collection and model learning. During data collection, we add E [N o?) to each waypoint in z to encourage local exploration. As shown in Tab. 3, we find zero-shot synthesized trajectories are typically meaningful but insufficient. However,
can learn effective dynamics model with less than minutes of online interactions by using these trajectories as exploration prior, leading to high eventual success rates. In comparison, exploring without prior all exceed the maximum 12-hour limit.
4.4 Error Breakdown
In this section, analyze the emrors resulting from each component of VoxPoser and how the overall system can be further improved. We conduct experiments in simulation where we have access to ground-truth perception and dynamics model (i.e., the simulator). "Dynamics error" refers to errors made by the dynamics model'. "Perception error" refers to errors made by the perception module.. "Specification error" refers to errors made by the module specifying cost or parameters for the low- level motion planner or primitives. Examples for each method include 1) noisy prediction by the U-Net, incorrect parameters specified by the LLM, and 3) incorrect value maps specified by the LLM. As shown in Fig. VoxPoser achieves lowest "specification error" due to its generalization and flexibility. We also find that having access to a more robust perception pipeline and physically- realistic dynamics model can contribute to better overall performance. This observation aligns with our real-world experiment, where most errors are from perception. For example, we find that the detector is sensitive to initial poses objects and is less robust when detecting object parts.
5 Conclusion, Limitations, & Future Works
In this work, we present VOXPOSER, general framework for extracting affordances and straints, grounded in 3D perceptual space, from LLMs and VLMs for everyday manipulation tasks in the real world, offering significant generalization advantages for open-set instructions and ob- jects. Despite compelling results, VoxPoser has several limitations. First, it relies on external per- ception modules, which is limiting in tasks that require holistic visual reasoning or understand- ing of fine-grained object geometries, Second, while applicable to efficient dynamics learning, general-purpose purpose dynamics model is still required to achieve contact-rich tasks with the same level of generalization. Third, motion planner considers only end-effector trajectories while whole-arm planning is also feasible and likely a better design choice [115-117]. Finally, manual prompt engi- neering is required for LLMs. We also see several exciting venues for future work. For instance, recent success of multi-modaa LLMs [68, 2, 118] can be directly translated into VoxPoser for direct visual grounding. Methods developed for alignment [119, 120] and prompting [121-124] may be used to alleviate prompt engineering effort. Finally, more advanced trajectory optimization methods can be developed that best interface with value maps synthesized by VoxPoser.
1 'LLM + Primitives [75] does not use model-base based planning, thus not having dynamics module. U-Nett MP [50] maps RGB-D to costmaps using U-Net [113, 114], thus not having perception module. Errors by which attributed to "specification erro"".

---

Acknowledgments
We would like to thank Andy Zeng, Igor Mordatch, and the members of the Stanford Vision and Learning Lab for the fruitful discussions. This work in part supported by AFOSR YIP FA9550- 23-1-0127, ONR MURI 200014-22-1-2744 ONR MURI N00014-21-1-2801, ONR N00014-23-1- 2355, the Stanford Institute for Human-Centered AI (HAI), JPMC, and Analog Devices, Wenlong Huang is partially supported by Stanford School of Engineering Fellowship. Ruohan Zhang is par- tially supported by Wu Tsai Human Performance Alliance Fellowship.
References
[1] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[2] OpenAL. Gpt-4 technical report. arXiv, 2023.
[3] A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.023111 2022.
[4] R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07255 2021.
[5] S. Tellex. N. Gopalan, H. Kress-Gazis and C. Matuszek. Robots that use language. Anmual Review of Control, Robotics, and Autonomous Systems, 2020.
[6] S. Tellex, T. Kollar, S. Dickerson, M. Walter, A. Banerjee, S. Teller, and N. Roy. Under- standing natural language commands for robotic navigation and mobile manipulation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 25, pages 1507-1514, 2011.
[7] T. Kollar, S. Tellex, D. Roy, and N. Roy. Toward understanding natural language directions. In 2010 5th ACM/EEE International Conference on Sluman-Robar Interaction (HRI), pages 2669-26.. IEEE, 2010.
[8] M. Bollini, S. Tellex, T. Thompson, N. Roy, and D. Rus. Interpreting and executing recipes with a cooking robot. In Experimental Robotics, pages 481-495. Springer, 2013.
[9] W. Huang, P. Abbeel, D. Pathak, and I. Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International Conference on Ma- chine Learning. PMLR, 2022.
[10] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, D. Ho, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, E. Jang, R. J. Ruano, K. Jeffrey, S. Jesmonth, N. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, K.-H. Lee, Levine, Y. Lu, L. Luu, C. Parada, P. Pastor, Quiambao, K. Rao, Rettinghouse, D. Reyes, P. Ser- manet, N. Sievers, C. Tan, A. Toshev, V. Vanhoucke, F. Xia, T. Xiao, P. Xu, Xu, and M. Yan. Do i can and not as say: Grounding language in robotic affordances. In arXiv preprint arXiv:2204.0166 2022.
[11] Zeng, A. Wong. Welker, K. Choromanski, F. Tombari, A. Purohit, M. Ryoo, V. Sind- hwani, J. Lee, V. Vanhoucke, et al. Socratic models: Composing zero-shot multimodal rea- soning with language. arXiv preprint urXiv:2204.00555 2022.
[12] A. Radford, W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language super- vision. In International Conference Machine Learning, pages
PMLR, 2021.

---

[13] X. Gu, T.-Y. Lin, W. Kuo, and Y. Cui. Open-vocabulary object detection via vision and language knowledge distillation. arXiv preprint arXiv:2104. 13921, 2021.
[14] A. Kamath, M. Singh, Y. LeCun, G. Synnaeve, I. Misra, and N. Carion. Mdetr-modulated detection for end-to-end multi-modal understanding. In Proceedings of the IEEE/CVF Inter- national Conference Computer Vision, pages 1780-1790, 2021.
[15] M. Minderer, A. Gritsenko, A. Stone, M. Neumann, D. Weissenbomn A. Dosovitskiy, A. Ma- hendran, A. Arnab, M. Dehghani, Z. Shen, et al. Simple open- pen-vocabulary object detection with vision transformers. arXiv preprint arXiv:2205.062: 06230, 2022.
[16] C. R. Harris, K. J. Millman, S. J. van der Walt, R. Gommers, P. Virtanen, D. Cournapeau, E. Wieser, J. Taylor, S. Berg, N. J. Smith, R. Kern, M. Picus, S. Hoyer, M. H. van Kerkwijk, Río, M. Wiebe, P. Peterson, P. Gérard-Marchant, K. Sheppard, T. Reddy, W. Weckesser, H. Abbasi, Gohlke, and T. E. Oliphant. Array programming with NumPy. Nature, 585(7825):357-362, Sept. 2020. doi: 100110..0088/11556-000-6666/// URL https://doi.. org.lorr///// 10. 1038 00888/4158655 000-5588-0----00
M. Brett, A. Haldane,
[17] Y. K. Hwang, N. Ahuja, et al. A potential field approach to path planning. IEEE transactions on robotics and automation, 8(1):23-32, 1992.
[18] M. Toussaint, J. Harris, J.-S. Ha, D. Driess, and W. Hönig. Sequence-of-constraint mpc: Reactive timing-optimal control of sequential manipulation. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 13753-13760. IEEE, 2022.
[19] J. Andreas, D. Klein, and S. Levine. Learning with latent language. arXiv preprint arXiv:1711.00422 2017.
[20] R. Zellers, A. Holtzman, M. Peters, R. Mottaghi, A. Kembhavi, A. Farhadi, and Y. Choi. Piglet: Language grounding through neuro-symbolic interaction in 3d world. arXiv preprint urXiv:2106.8 001.000088.8 18,, 2021.
[21] R. Zellers, X. Lu, J. Hessel, Y. Yu, J. S. Park, J. Cao, A. Farhadi, and Y. Choi. Merlot: Multimodal neural script knowledge models. Advances in Neural Information Processing Systems, 2021.
[22] V. Shwartz, P. West, R. L. Bras, C. Bhagavatula, and Y. Choi. Unsupervised commonsense question answering with self-talk. arXiv preprint wrXiv:2004.05440 2020.
[23] T. Winograd. Procedures as a representation for data a computer program for understanding natural language. Technical report, MASSACHUSETTS INST OF TECH CAMBRIDGE PROJECT MAC, 1971.
[24] V. Blukis, R. A. Knepper, and Y. Artzi Few-shot object grounding and mapping for natural language robot instruction following. arXiv preprint Sttiv:2011.0739900 2020.
[25] S. Tellex, R. Knepper, A. Li, D. Rus, and N. Roy. Asking for help using inverse semantics. Robotics: Science and Systems Foundation, 2014.
[26] T. Kollar, S. Tellex D. Roy, and N. Roy. Grounding verbs of motion in natural language commands to robots. Experimental robotics, pages 31-47. Springer, 2014.
[27] J. Thomason, S. Zhang, R. J. Mooney, and P. Stone. Learning to interpret natural language commands through human-robot dialog. In TwentyFFourhh International Joint Conference on Artificial Intelligence, 2015.
[28] J. Thomason, A. Padmakumar, J. Sinapov, N. Walker, Y. Jiang, H. Yedidsion, Hart, Stone, and R. Mooney. Jointly improving parsing and perception for natural language commands through human- dialog. Journal of Artificial Intelligence Research, 67:327-374, 2020.
10

---

[29] E. Jang, A. Irpan. M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn. Bc-z: Zero-shot task generalization with robotic imitation learning. In Conference on Robot Learning, pages 991-1002. PMLR, 2021.
[30] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:22
2022.
[31] D. Shah, B. Osinski, B. Ichter, and S. Levine. Lm-nav: Robotic navigation with large pre- trained models of language, vision, and action. arXiv preprint arXiv:2207.04429 2022.
[32] Cui, S. Karamcheti, R. Palleti, N. Shivakumar, Liang, and Sadigh. " no, to the right"- online language corrections for robotic manipulation via shared autonomy. arXiv preprint arXiv:2301.02553 2023.
[33] A. Stone, T. Xiao, Y. Lu, K. Gopalakrishnan, K.-H. Lee, Q. Vuong, P. Wohlhart, B. Zitkovich, F. Xia, C. Finn, et al. Open object manipulation using pre-trained vision-lane lasoonlanguuage models. arXiv preprint arXiv:2303. 00905, 2023.
[34] S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta. R3m: A universal visual represen- tation for rohot manipulation. arXiv preprint arXiv:2203. 12601, 2022.
[35] Y. J. Ma, V. Kumar, A. Zhang, O. Bastani, and D. Jayaraman. Liv: Language-image repre- sentations and rewards for robotic control. arXiv e-prints, 2023.
[36] P. A. Jansen. Visually- planning without vision: Language models infer detailed plans from high-level instructions. arXiv preprint (iXiv:2009.14255 2020.
[37] V. Micheli and F. Fleuret. Language models are few-shot butlers. arXiv:2104.07972.. 2021.
arXiv preprint
[38] P. Sharma, A. Torralba, and J. Andreas. Skill induction and planning with latent language. arXiv preprint 1.Xiv:2/100015:// 2021.
[39] W. Huang, Xia, T. Xiao, H. Chan, Liang, P. Florence, A. Zeng, Tompson, Mor- datch, Y. Chebotar, P. Sermanet, N. Brown, T. Jackson, L.. Luu, S. Levine, K. Hausman, and B. Ichter. Inner monologue: Embodied reasoning through planning with language models. In arXiv preprint arXiv:2207.056/0 2022.
[40] Z. Li, W. Chen, Sharma, and J. Andreas. Lampp: Language models as probabilistic priors for perception and action. arXiv e-prints, pages arXiv-2300 2023.
[41] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPRO9, 2009.
[42] A. Krizhevsky, Sutskever, and G. E. Hinton. Imagenet classification with deep convolu- tional neural networks. Communications of the ACM, 60(6):84 -0, 2017.
[43] Radford, Wu, Child, Luan, D. Amodei, and Sutskever. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
[44] S. Nair, E. Mitchell, K. Chen, S. Savarese, C. Finn, et al. Learning language-conditioned robot behavior from offline data and crowd-sourced annotation. In Conference on Robot Learning, pages 1315. PMLR, 2022.
[45] M. Shridhar, L. Manuelli, and D. Fox. Cliport: What and where pathways for robotic manip- ulation. In Conference on Robot Learning, pages 894-906. PMLR, 2022.
[46] M. Shridhar, L. Manuelli, and D. Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. In Proceedings of the 6th Conference on Robot Learning (CoRL), 2022.
11

---

[47] S. Li, X. Puig, Y. Du, C. Wang, E. Akyurek, A. Torralba, J. Andreas, and I. Mordatch. Pre- trained language models for interactive decision-making. arXiv preprint arXiv:2202.01771, 2022.
[48] O. Mees, L. Hermann, and W. Burgard. What matters in language conditioned robotic imita- tion learning. arXiv preprint
2022.
[49] O. Mees, J. Borja-Diaz, and W. Burgard. Grounding language with visual affordances over unstructured data. arXiv preprint arXiv:2210.01911, 2022.
[50] P. Sharma, B. Sundaralingam, V. Blukis, C. Paxton, T. Hermans, A. Torralba, J. An- dreas, and D. Fox. Correcting robot plans with natural language feedback. arXiv preprint arXiv:2204.05I 0510222600.0886, 2022.
[51] W. Liu, C. Paxton, Hermans, and D. Fox. Structformer: Learning spatial structure for language-guidee guudeed semantic rearrangement of novel objects. 2022 International Conference Robotics and Automation (ICRA), pages 5322-6329, IEEE, 2022.
[52] C. Lynch and P. Sermanet. Language conditioned imitation learning over unstructured data. Robotics: Science and Systems, 2021. URL
07648.
[53] C. Lynch, Wahid, Tompson, T. Ding, Betker, R. Baruch, T. Armstrong, and Florence. Interactive language: Talking to robots in real time. arXiv preprint arX1v:2210.06400, 2022.
[54] L. Shao, T. Migimatsu, Q. Zhang, K. Yang, and J. Bohg. Concept2robot: Learning manip- ulation concepts from instructions and human demonstrations. The International Journal of Robotics Research,
2021.
[55] J. Luketina, N. Nardelli, G. Farguhar, J. N. Foerster, J. Andreas, E. Grefenstette, S. Whiteson, and T. Rocktäschel. A survey of reinforcement learning informed by natural language. In IJCAI, 2019.
[56] J. Andreas, D. Klein, and Levine. Modular multitask reinforcement learning with policy sketches. ArXiv, abs/1611.01196, 2017.
[57] Y. Jiang, S. S. Gu, K. P. Murphy, and C. Finn. Language as an abstraction for hicrarchical deep reinforcement learning. Advances in Neural Information Processing Systems, 32, 2019.
[58] Chen, F. Xia, Ichter, K. Rao, K. Gopalakrishnan, M. S. Ryoo, A. Stone, and D. Kappler. Open-vocabulary queryable scene representations for real world planning. arXiv preprint 2XXiv:2299008874, 2022.
[59] I. Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg. Progprompt: Generating situated robot task plans using large language models. arXiv preprint arXiv:2209. /302, 2022.
[60] C. Huang, O. Mees, A. Zeng, and W. Burgard. Visual language maps for robot navigation. arXiv preprint arXiv:2210.057/44 2022.
[61] S. S. Raman, Cohen, Rosen, Idrees, D. Paulius, and S. Tellex. Planning with large language models via corrective re-prompting. arXiv preprint arXiv:2211.09935, 2022.
[62] C. H. Song. shot grounded planning for embodied agents with large language models. arXiv preprint arXiv:2212.04088, 2022.
J. Wu, C. Washington, B. M. Sadler, W.-L. Chao, and Y. Su. LIm-planner: Few-
[63] Liu, Y. Jiang, Zhang, Liu, S. Zhang, Biswas, and P. Stone. Llm+ p: Empowering large language models with optimal planning proficiency. arXiv preprint arXiv:2304.11477 2023.
12

---

[64] S. Vemprala, R. Bonatti, A. Bucker, and A. Kapoor. Chatgpt for robotics; Design principles and model abilities. 2023, 2023.
[65] K. Lin, C. Agia, T. Migimatsu, M. Pavone, and J. Bohg. Texi2motion: From natural language instructions to feasible plans. arXiv preprint arXiv:2303.1215. 2023.
[66] Y. Ding, X. Zhang, C. Paxton, and S. Zhang. Task and motion planning with large language models for object rearrangement. arXiv preprint arXiv:2303.06 06247,, 2023.
[67] W. Huang, F. Xia, D. Shah, D. Driess, A. Zeng, Y. Lu, P. Florence, I. Mordatch, S. Levine, Hausman, et al. Grounded decoding: Guiding text generation with grounded models for robot control, arXiv preprint arXiv:2303.00 000500085.,0 2023.
[68] Driess, F. Xia, M. Sajjadi, C. Lynch, Chowdhery, B. Ichter, A. Wahid, Tompson, Q. Vuong, T. Yu, al. Palm-e: An embodied multimodal language model. arXiv preprint arXhv:2303.03378, 2023.
[69] H. Yuan, C. Zhang, H. Wang, F. Xie, P. Cai, H. Dong, and 7. Lu. Plan4mc: Skill reinforce- ment learning and planning for open-world minecraft tasks. arXiv preprint arXiv:2303. 16563, 2023.
[70] Y. Xie, C. Yu, T. Zhu, J. Bai, Z. Gong, and H. Soh. Translating natural language to planning goals with large-language models. arXiv preprint arXiv:2302.05//28 2023.
[71] Y. Lu, P. Lu, Z. Chen, W. Zhu, X. E. Wang, and W. Y. Wang. Multimodal procedural planning via dual text-image prompting. arXiv preprint arXiv:2305.0173 2023.
[72] D. Patel, H. Eghbalzadeh, N. Kamra, M. L. Juzzolino, U. Jain, and R. Desai. Pretrained language models as visual planners for human assistance. arXiv preprint arXiv:2304.09179, 2023.
[73] Wang, Y. Xie, Y. Jiang, A. Mandlekar, C. Xiao, Y. Zhu, L. Fan, and A. Anandku- mar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291, 2023.
[74] J. Yang, W. Tan, C. Jin, B. Liu, Fu, R. Song, and L. Wang. Pave the way to grasp anything: Transferring foundation models for universal pick-place robots. arXiv preprint arXiv:2306.055 057166. 2023.
[75] J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng. Code as policies: Language model programs for embodied control. arXiv preprint arXiv:2209.07753, 2022.
[76] M. Kwon, S. M. Xie, K. Bullard, and D. Sadigh. Reward design with language models. arXiv preprint arXiv:2303.00001, 2023.
[77] A. Tam, N. Rabinowitz, A. Lampinen, N. A. Roy, S. Chan, D. Strouse, J. Wang, A. Banino, and F. Hill. Semantic exploration from language abstractions and pretrained representations. Advances Neural Information Processing Systems, 35:25377-25355 2022.
[78] J. Mu, V. Zhong, R. Raileanu, M. Jiang, N. Goodman, Rocktäschel, and Grefenstette. Improving intrinsic exploration with language abstractions. arXiv preprint arXiv:2202.08938, 2022.
[79] C. Colas, T. Karch, N. Lair, J.-M. Dussoux, C. Moulin-Frier, P. Dominey, and P.-Y. Oudeyer. Language as cognitive tool to imagine goals in curiosity driven exploration. Advances in Neural Information Processing Systems, 33:3761-3774, 2020.
13

---

[80] Y. Du, O. Watkins, Z. Wang. C. Colas, T. Darrell, P. Abbeel, A. Gupta, and J. Andreas. Guiding pretraining in reinforcement learning with large language models. arXiv preprint arXiv:230.06669, 2023.
[81] H. Hu and D. Sadigh. Language instructed reinforcement learning for human-ai coordination. arXiv preprint arXiv:2334.07229 072977 2023.
[82] I. Lenz, R. A. Knepper, and A. Saxena. Deepmpe: Learning deep latent features for model predictive control. In Roboties: Science and Systems, volume 10. Rome, Italy, 2015.
[83] L. Hewing, K. P. Wabersich, M. Menner, and M. N. Zeilinger. Learning based model pre- dictive control: Toward safe learning in control. Annual Review of Control, Robotics, and Autonomous Systems, 3:269-296, 2020.
[84] M. B. Chang, T. Ullman, A. Torralba, and J. B. Tenenbaum. compositional object-based approach to learning physical dynamics. arXiv preprint arXiv: 2016.
[85] P. Battaglia, R. Pascanu, M. Lai, D. Jimenez Rezende, et al. Interaction networks for learning about objects, relations and physics. Advances in neural information processing systems, 29, 2016.
[86] Z. Xu, Wu, A. Zeng, J. B. Tenenbaum, and S. Song. Densephysnet: Learning dense physical object representations via multi-step dynamic interactions. arXiv preprint arXivv1906.000 2019.
[87] A. Byravan and D. Fox. Se3-nets: Learning rigid body motion using deep neural networks In 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 173-180. IEEE, 2017.
[88] A. Nagabandi, K. Konolige, S. Levine, and V. Kumar. Deep dynamics models for learning dexterous manipulation. In Conference on Robot Learning, pages 1101-1112. PMLR, 2020.
[89] A. Gonzalz, N. Heess, T. Springenberg, J. Merel, M. Riedmiller, R. Hadsell, and P. Battaglia. Graph networks as learnable physics engines for inference and control. In International Conference on Machine Learning, pages 4470-4479. PMLR, 2018.
[90] Y. Li. J. Wu, R. Tedrake, J. B. Tenenbaum, and A. Torralba. Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids. arXiv preprint arXiv: 10/.:::8800.0.66 2018.
[91] C. Finn, S. Levine, and P. Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In International conference on machine learning, pages 49-58. PMLR, 2016.
[92] J. Fu, K. Luo, and S. Levine. Learning robust rewards with adversarial inverse reinforcement learning. arXiv preprint arXiv:1710.11244 2017.
[93] D. Driess, O. Oguz, J.-S. Ha, and M. Toussaint. Deep visual heuristics: Learning feasibility of programs for manipulation planning. In 2020 IEEE International Conference on Robotics and Automation (ICRA). pages 9563-9569. IEEE, 2020.
[94] B. Amos, I. Jimenez, J. Sacks, B. Boots, and J. Z. Kolter. Differentiable mpc for end-to-end planning and control. Advances in neural information processing systems, 31, 2018.
[95] M. Mittal, D. Hoeller, F. Farshidian, M. Hutter, and A. Garg. Articulated object interaction in unknown scenes with whole-body mobile manipulation. In 2022 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1647-1654, IEEE, 2022.
[96] S. Bahl, A. Gupta, and D. Pathak. Human-to robot imitation in the wild. arXiv preprint arXiv:2207.0910 094000 2022.
14

---

[97] S. Bahl, R. Mendonca, L. Chen, U. Jain, and D. Pathak. Affordances from human videos as a versatile representation for robotics. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pages 13778-13790, 2023.
[98] C. Wang, L. Fan, J. Sun, Zhang, Fei-Fei, D. Xu, Y. Zhu, and A. Anandkumar. Mimicplay: Long-horizon imitation learning by watching human play. arXiv preprini arXiv:2302.12422, 2023.
[99] H. Bharadhwaj. A. Gupta, S. Tulsiani, and V. Kumar. Zero-shot robot manipulation from passive human videos. arXiv preprint 2,0.:2330.0201,,.2.
[100] Y. Ma, Sodhani, D. Jayaraman, O. Bastani, V. Kumar, and Zhang. Vip: Towards universal visual reward and representation via value-implicit pre-training. arXiv preprint arXiv:2210.00030 2022.
[101] D. Damen, H. Doughty, G. M. Farinella. S. Fidler. A. Furnari, E. Kazakos, D. Moltisanti, J. Munro, Perrett, W. Price, et al. Scaling egocentric vision: The epic-kitchens dataset. In Proceedings of the European Conference on Computer Vision (ECCV), pages 720-736, 2018.
[102] K. Grauman, A. Westbury, E. Byrne, Z. Chavis, A. Furnari, R. Girdhar, J. Hamburger, H. Jiang, M. Liu, X. Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1 18995-19012, 2022.
[103] Y. Cui, S. Niekum, A. Gupta, V. Kumar, and A. Rajeswaran. Can foundation models perform zero-shot task specification for robot manipulation? In Learning for Dynamics and Control Conference, pages 893-905, PMLR, 2022.
[104] T. Yu, T. Xiao, A. Stone, J. Tompson, A. Brohan, S. Wang, J. Singh, C. Tan, J. Peralta, B. Ichter, et al. Scaling robot learning with semantically imagined experience. arXiv preprint arXiv:2302. 1150, 2023.
[105] 7. Mandi, H. Bharadhwaj. V. Moens, S. Song, Rajeswaran, and V. Kumar. Cacti: framework for scalable multi-task multi-scene visual imitation learning. arXiv preprint arXiv:2212.057111 2022.
[106] T. Xiao, H. Chan, P. Sermanet, A. Wahid, A. Brohan, K. Hausman, S. Levine, and J. Tompson. Robotic skill acquisition via instruction augmentation with vision-language models. arXiv preprint arXiv:22
2022.
[107] C. Wang, D. Xu, and L. Fei-Fei. Generalizable task planning through representation pretrain- ing, IEEE Robotics and Automation Letters, 7(3):229--8366 2022.
[108] C. Li, R. Zhang, J. Wong, C. Gokmen, S. Srivastava, R. Martín Martin, C. Wang, G. Levine, M. Lingelbach, J. Sun, et al. Behavior- 1k: A benchmark for embodied ai with 1,000 everyday activities and realistic simulation. In Conference on Robot Learning, pages 80-93. PMLR, 2023.
[109] O. Khatib. A unified approach for motion and force control of robot manipulators: The operational space formulation. IEEE Journal on Robotics and Automation, 3(1):43-55 1987.
[110) A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, et al. Segment anything. arXiv preprint arXiv:2304.006644 2023.
[111] H. K. Cheng and G. Schwing. Xmem: Long-term video object segmentation with an atkinson-shiffrin memory model. In Computer Vision-ECCV 2022: 17th European Con- ference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXVIII, pages 640-658. Springer, 2022.
15

---

[112] F. Xiang, Y. Qin. K. Mo, Y. Xia, H. Zhu, F. Liu, M. Liu, H. Jiang, Y. Yuan, H. Wang, et al. Sapien: A simulated part-based interactive environment. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 11097-11107, 2020.
[113] O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention-MICCAA 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234 241. Springer, 2015.
[114] Ó. Çięek, A. Abdulkadir, S. S. Lienkamp, T. Brox, and O. Ronneberger. 3d u-net: learn- ing dense volumetric segmentation from sparse annotation. In Medical Image Computing and Computer-Assister InterventionMMiCAA/ 2016: 19th International Conference, Athens, Greece, October 17-21, 2016, Proceedings. Part II 19, pages 424 432. Springer, 2016.
[115] L. E. Kavraki, P. Svestka, J.-C. Latombe, and M. H. Overmars. Probabilistic roadmaps for path planning in high-dimennional configuration spaces. IEEE transactions on Robotics and Automation, 1260):56--580 1996.
[116] N. D. Ratliff, Issac, D. Kappler, S. Birchfield, and D. Fox. Riemannian motion policies. arXiv preprint arXiv: /801.02854, 2018.
[117] T. Marcucci, M. Petersen, D. von Wrangel, and R. Tedrake. Motion planning around obstacles with convex optimization. arXiv preprint arXiv:2205.04422 2022.
[118] J. Li, D. Li, S. Savarese, and S. Hoi. 2lip-2: Bootstrapping language-iman image pre-training with frozen image encoders and large language models. arXiv preprint vrXiv:2301.12597 2023.
[119] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agar- wal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint 2.XX::2233002155: 2022.
[120] Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirho- seini, C. McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212008073, 2022.
[121] J. Wei, X. Wang, D. Schuurmans, M. Bosma, E. Chi, Q. Le, and D. Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201. 11903, 2022.
[122] Y. Wang, Y. Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212. 10560, 2022.
[123] T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa. Large language models are zero- shot reasoners. arXiv preprint 220.::2005/1916 2022.
[124] S. Yao, D. Yu, J. Zhao, I. Shafran, T. L. Griffiths, Y. Cao, and K. Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305. 10601, 2023.
[125] J. Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.0768) 2022.
[126] T. Gupta and A. Kembhavi. Visual programming: Compositional visual reasoning with- out training. In Proceedings of the IEEE/CVF Conference Computer Vision and Patlern Recognition, pages 14953-14962, 2023.
[127] D. Surís, S. Menon, and C. Vondrick. Vipergpt: Visual inference via python execution for reasoning, arXiv preprint arXiv:2303.08/28, 2023.
16

---

[128] Y. Zhu, A. Joshi, P. Stone, and Y. Zhu. Viola: Imitation learning for vision- bason manipulation with object proposal priors. 6th Annual Conference on Robot Learning, 2022.
17

---

A Appendix
A.1 Emergent Behavioral Capabilities
left-handed.
You're off by 1c to the left.
Open drawer precisely by
Which block heavier?
Figure Emergent hehavioral capabilities by VoxPoser inherited from the language model, including hehav- ioral commonsense reasoning (top left), fine-grained language correction (top right), multi-step visual program (bottom left), and estimating physical propertics of objects (bottom right).
Emergent capabilities refer to unpredictable phenomenons that are only present in large mod- els [125]. VoxPoser uses pre-trained LLMS as backbone, we observe similar embodied emergent capabilities driven by the rich world knowledge of LLMs. In particular, focus our study on the behavioral capabilities that unique to VoxPoser. We observe the following capabilities:
Behavioral Commonsense Reasoning: During task where robot is setting the table, the user can specify behavioral preferences such as "I am left-handed", which requires the robot to comprehend its meaning in the context of the task. VoxPoser decides that it should move the fork from the right side of the bowl to the left side.
Fine-grained Language Correction: For tasks that require high precision such as "cov- ering the teapot with the lid", the user can give precise instructions to the robot such as "you're re off by lcm". VoxPoser similarly adjusts its action based on the feedback.
Multi-step Visual Program [126, 127]: Given a task "open the drawer precisely by half" where there is insufficient information because object models are not available, VoxPoser can come up with multi-step manipulation strategies based on visual feedback that first opens the drawer fully while recording handle displacement, then close back the mid- point to satisfy the requirement.
Estimating Physical Properties: Given blocks of unknown mass, the robot is tasked conduct physics experiments using existing ramp determine which block is heav- ier. VoxPoser decides to push both blocks off the ramp and choose the block traveling the farthest as the heavier block. Interestingly, this mirrors common human oversight: in an ideal, frictionless world, both blocks would traverse the same distance under the influ- ence of gravity. This serves as a lighthearted example that language models can exhibit limitations similar human reasoning.
18

---

A.2
APIs for VoxPoser
Central to VoxPoser is an LLM generating Python code that executed by a Python interpreter. Besides exposing NumPy [16] and the Transforms3d library to the LLM, we provide the following environment APIs that LLMs can choose invoke:
(ettec(obbjaamma)) Takes in object name and returns list of dictionaries, where each dictionary corresponds to one instance of the matching object, containing center position, occupancy grid, and mean normal vector.
execute (movable, affordance map, avoidance map, rotation map, velocity map, gripper map): Takes in "entity of interest" as "movable" (a dictionary returned by detect) and (optionally) list of value maps and invokes the motion planner execute the trajectory. Note that in MPC settings, "movable" and the input value maps are functions that can be re-evaluated to reflect the latest environment observation.
laaeeeexmmmmmeeeeeeee direction): Takes in a desired offset distance centimeters along direction and returns 3-dim vector reflecting displacement voxel coordinates.
cm2index (cm,
index2cm index,, direction): Inverse of cm2index. Takes in an integer "index" and "direction" vector and returns the distance in centimeters in world coordinates displaced by the "integer" in voxel coordinates.
pointat2quat (vector): Takes in a desired pointing direction for the end-effector and returns satisfying target quaternion.
set_voxel set.voxel.by-rade bby-dddiuuuuuaaeaa (voxel map, voxel voxel_xyz, radirs.diusccmm .cm, value): Assigns "value" to voxels within "radious cm" from "voxel xyz" "voxel map".
map(): get.empty forddncc :)) Returns default affordance map initialized with 0, where a high value attracts the entity.
get_empty get.empty.avviddaceem.. avoidance_mappe value repulses the entity.
Returns a default avoidance map initialized with 0, where a high
get.empty.rotation map()): Returns a default rotation map initialized with current end-effector quaternion.
get_empe empty gr ::) Returns a default gripper map initialized with current gripper action, where indicates "closed" and 0 indicates "open".
get.empty.velciity Returns a default affordance map initialized with where the number represents scale factor (e.g., 0.5 for half of the default velocity).
Reset.to_default.poos Reset to robot rest pose.
19

---

A.3 Real-Woor World Environment Setup
We use a Franka Emika Panda robot with a tabletop setup. We use Operational Space Controller with impedance from Deoxys [128]. We mount two RGB-D cameras (Azure Kinect) at two opposite ends of the table: bottom right and top left from the top down view. At the start of each rollout, both cameras start recording and return the real-time RGB-D observations 20 Hz.
For each task, we evaluate each method on two settings: without and with disturbances. For tasks with disturbances, we apply three kinds of disturbances to the environment, which we pre-select sequence of them at the start of the evaluation: 1) random forces applied to the robot, 2) random displacement of task-relevant and distractor objects, and reverting task progress (e.g., pull drawer open while it's being closed by the robot). We only apply the third disturbances tasks where "entity of interest" is object or object part.
We compare to variant of Code as Policies [75] as baseline that uses an LLM with ac- tion primitives. The primitives include: move_to move_to_pos to_pos, rotate seeeebyyqquaae quat, set_vel, open-gripper, plose-grippe We do not provide primitives such as pick-and-place as they would be tailored for a particular suite of tasks that we not constrain to in our study (similar the control APIs for VoxPoser specified in Sec. A.2).
A.3.1 Tasks
Move & Avoid: "Move the top of [obji] while staying away from [obj2]", where [obj1] and [obj2] are randomized everyday objects selected from the list: apple, banana, yellow bowl, headphones, mug, wood block.
Set Up Table "Please set up the table by placing utensils for my pasta".
Close Drawer: "Close the [deixis] drawer", where [deixis] can be "top" "bottom".
Open Bottle: "Turn open the vitamin bottle".
Sweep Trash: "Please sweep the paper trash into the blue dustpan".
20

---

A.4 Simulated Environment Setup
We implement a tabletop manipulation environment with Franka Emika Panda robot in SAPIEN [112]. The controller takes as imput a desired end-effector 6-DoF pose, calculates a se- quence of interpolated waypoints using inverse kinematics, and finally follows the waypoints using PD controller. We use set of 10 colored blocks and 10 colored lines in addition cabinet with 3 drawers. They initialized differently depending the specific task. The lines used visual landmarks and are not interactable. For perception, total of 4 RGB-D cameras mounted at each end of the table pointing at the center of the workspace.
articulated
A.4.1 Tasks
We create a custom suite of 13 tasks shown Table 4. Each task comes with a templated instruction (shown in Table 4) where there may one or multiple attributes randomized from the pre-defined list below. reset time, number of objects are selected (depending on the specific task) and are randomized across the workspace while making that task completed reset and that task completion is feasible. A complete list of attributes can be found below, divided into "seen" and "unseen" categories:
Seen Attributes:
baakk left corner of the table", "front right corner of the table", "right side of the table", "back side of the table"]
[pos]:
['blue block", "green block", "yellow block", "pink block", "brown block"] [preposition]: ["left of", "front side of"" "top of""
[obj]:
[deixis]: ["topmost", "second to the bottom"]
[dist]: [3, 5, 7, 11]
[region]: ["right side of the table". "back side of the table")
[velocity]: ["faster speed", "a quarter of the speed"]
[line]: ['blue line", "green line", "yellow line", "pink linc", "brown line"]
Unseen Attributes:
[pos]: ["back right corner of the table", "front left corner of the table", "left side of the table", "front side of the table"]
[obj]: ['red block", "orange block", "purple block", "cyan block", "gray block"]
[preposition]: ["right of", "back side of"]
[deixis]: ["bottommost", "second to the top"]
[dist]: [4, 6, 8, 10]
[region]: ["left side of the table", "front side of the table"]
[velocity]: ["slower speed", "3x speed"]
[line]: ["red line", "orange line", "purple line", "cyan line", "gray line")
21

---

A.4.2
Full Results on Simulated Environments
U-Net + MP
LLM + Prim.
VoxPoser
Tasks
SA
UA
SA
UA
SA
UA
move to the ppeeposition the Joby]
95.0%
0.0%
85.0%
60.0%
90.0%
55.0%
move [pos] while staying [preposition] [obj]
100.0%
10.0%
80.0%
30.0%
95.0%
50.0%
move to the [pos] while moving at [velicity] when within [dist]em from the ohj
80.0%
0.0%
10.0%
0.0%
100.0%
95.0%
[deixis]
by pushing
0.0%
0.0%
60.0%
60.0%
80.0%
80.0%
push the [obj] aloog the [line]
0.0%
0.0%
0.0%
0.0%
65.0%
30.0%
grasp [obj]
[velacity]
35.0%
0.0%
75.0%
70.0%
65.0%
40.0%
drop the [obj] to the [pos]
70.0%
10.0%
60.0%
100.0%
60.0%
100.0%
push [obj] while letting it stay [region]
0.0%
5.0%
10.0%
0.0%
50.0%
50.0%
move to the [region]
5.0%
0.0%
100.0%
95.0%
100.0%
100.0%
move [pos] while staying [dist]em [ohj]
0.0%
0.0%
15.0%
20.0%
85.0%
90.0%
move to the [pos] while moving [velocity] in the [region]
0.0%
0.0%
90.0%
45.0%
85.0%
85.0%
push [obj] the [pos] while staying away from [obstaclee]
0.0%
0.0%
0.0%
10.0%
45.0%
55.0%
push the [ohi] to the [pos]
0.0%
0.0%
20.0%
25.0%
80.0%
7.50%
Table 4: Full experimental results in simulation on seen tasks and unseen tasks. "SA" indicates seen attributes
and "UA" indicates unseen attributes. Each entry represents success rate averaged across 20 episodes.
22

---

A.5 Prompts
Prompts used in Sec. 4.1 and Sec. 4.2 can be found below.
planner: Takes in user instruction and generates a sequence of sub-tasks & which is fed into "composer" (Note that planner is not used in simulation the evaluated tasks consist of single manipulation phase).
real-world: exaposergiihub.io//romppsss
planner prompt.xxt
composer: Takes in sub-task instruction and invokes necessary value map LMPs compose affordance maps and constraint maps.
simulation:
real-world: voxposer. httpoprrgichubbbooooopsss/eeee https//rrreeccccseeeeeeeeee composer composer.prompt.xx
narse_query.obj: Takes in text query of object/part name and returns list of dictionaries, where each dictionary corresponds to one instance of the matching object containing center position, oc- cupancy grid, and mean normal vector.
simulation: voxposer, wtwpeerrgihuuu..oooooopooeo
parse query obj prompt.txt.
real-world: voxposer. http://hiwibppooopp//ppppppeeeee wttpurrrooooppoop.....
get affordance map: Takes natural language parametrization from composer and returns a NumPy array for task affordance map.
simulation:
woxpoerriituubiopooopssiiii httpiinnsssssiitttttteiieeeeee afforrdncccceeeccc www.mampppppoa.pp waappprompt.xt.
real-world: oosposerrihuulloooooppsseeeeggee
get avoidance map: Takes in natural language parametrization from composer and returns NumPy array for task avoidance map.
simulation: voxposerggihuahioooooptsiiii get avoidance map prompt.txt.
real-world: voxposer. waapeerggthbbooooppppprrrr..eeue wwp.pppppp.ppt.t.......m.mmmmm
get_rotation map: Takes in natural language parametrization from composer and returns NumPy array for end-effector rotation map.
simulation: voxposer.githube httppprrgiituuboo/poopps////isii httpa///porooopsssiomeeeerrttttt get_rotation_nu www.ppmmmpp.ttt
real-world: voxposer. hatppssrgiihuu..ooooomp////////e get wwtpatioonmm--ppppp www.mmmmmppppppppmm prompt.t....xt.
get gripper map: Takes in natural language parametrization from composer and returns a NumPy
array for gripper action map.
simulation: voxposer voxpperrgittubbioproopsssii get get..griper.nnn www.maapppppp.p.. maapprromt.txt.
real-world: wwwppeeriiiubbbooromptttrreeeeee httpp//wwww..lllleeeepppcppppppm
get_velocis peelocity_mare _map: Takes in natural language parametrization from composer and returns a NumPy array for end-effector velocity map.
simulation: woxpssergihub..ooooonssssi... velovvvelottt...ooee www.:_mmppppppp..mm
real-world: voxposer. voxposer.gihubboo/rooptttssss get. map. prompt....tttt.....mmmmm
23


Reviews:
[{"review_id": "iWlZ4dQgoy", "reviewer": "Reviewer_7joN", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper focuses on exploiting LLM's code-writing capabilities to infer affordances and constraints from free-form language instructions. Additionally, for contact-rich tasks, the paper demonstrates how their framework can be utilized to learn a dynamics model using online experiences. The contributions of this work are threefold: \n\n1) Proposing a method for extracting affordances and constraints using the foundation model in a zero-shot manner.\n\n2) Presenting results from 30 different real-world tasks to showcase the effectiveness of their approach.\n\n3) Demonstrating the ability to learn a dynamics model from a limited amount of online interactions.", "summary_of_recommendation": "I found the results this paper presented were quite surprising. However, there are some limitations, particularly concerning contact-rich manipulation and the reliance on pushing motion primitives for tool use. Additionally, I believe that more details should be provided in the main text to enhance clarity and understanding.\n\nTherefore, based on these considerations, I would recommend a weak acceptance of the paper.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "1. More details and explanations are needed on the 'Environment dynamics model.' The current description in this section (lines 211-218) is not thorough, despite its apparent importance for task execution.\n* Are the \"action parameters\" or the \"pre-defined pushing primitive\" specific to an object? In other words, can the same parameters/functions be used with unseen tools?\n* What does \"MPC with random shooting\" refer to (line 215)? How expensive is this process in terms of computational resources?\n* Could you provide a clear formulation of how your environment dynamics model is used in your framework?\n\n2. When and how do you decide to learn online interactions? Table 3 indicates that your method heavily relies on online learning, especially for manipulating articulated objects, as the success rate without online learning ranges from 3.3% to 18.3% (Tab 3).\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes a highly promising framework for robotic manipulation that utilizes foundation models for zero-shot reasoning, planning, and motion primitives generation. I was pleasantly surprised by how the LLM can achieve motion primitives with 70-80% accuracy in the real world (Tab 1) without requiring further training or data collection. While this appears very promising and even somewhat magical, it is challenging to assess the novelty of this paper, as it seems to integrate existing studies (e.g. GPT-4, OWL-ViT, XMEM). \n\nFurthermore, the paper addresses the optimization problem for parameter identification (line 215), which is conditionally required, potentially raising doubts about whether the method can genuinely be labeled as zero-shot. To avoid any misinterpretations, it is crucial to highlight the heuristics and assumptions, such as the necessity of pre-defined pushing primitives for the \"entity of interest\" when introducing the paper (e.g., in the abstract or introduction).\n\nAdditionally, the paper attempts to handle contact-rich tasks through an environment dynamics model. However, as noted by the author (line 293), VoxPoser still lacks the capability to deal with the intricacies and complexities of contact between tools and the environment. Notably, the model does not reason over contact constraints (e.g., maintaining contact), and some examples of their contact-rich tasks, such as opening a door, do not actually involve significant contact.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "Q11d_wuY09", "reviewer": "Reviewer_NGjD", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper’s main contribution is VoxPoser, a technique that uses GPT-4 to convert high-level natural language commands into a sequential of subtasks. Each sub-task is then translated into several density maps of the workspace, where each voxel contains a value. Two “value maps”, an affordance map and avoidance map, are used for trajectory planning. A simple random shooting motion planner is then used to greedily maximize these two maps to produce a trajectory for a robotic end effector. Other maps directly control end effector rotation and the gripper state. The authors demonstrate the capability of the VoxPoser technique on 30 manipulator tasks and explores its performance on a zero-shot set-up. The proposed method appears to outperform existing baselines. The authors also demonstrate VoxPoser’s strengths for online training. LLM-generated value maps can guide RL exploration for faster convergence online training.", "summary_of_recommendation": "While the paper would benefit from some improvements in experiment design and clarity, I believe the core concept of the paper is sufficiently interesting and presents promising results. I would recommend accepting this paper.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Please address the questions in the weaknesses section. No additional questions.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Originality & Significance:** As far as I know, this is the first paper to use LLMs to generate value maps in real time for MPC. The results are promising and would be interesting to the community.\n\n**Strengths:**\n1. I found the authors’ approach to grounding abstract language commands as innovative and novel. The proper array of experiments was conducted for a thorough investigation of serial manipulator tasks, and the authors were able to identify a significant improvement over existing baselines.\n2. By leverage recent advancements in LLMs, the authors present new capabilities compared to past methods. The greatest advances are for zero-shot learning.\n\n**Weaknesses:**\n1. Although the authors conduct experiments on a variety of manipulator tasks, there are some limitations in the setup. First, the full occupancy of the robot arm is not considered for collisions and trajectory optimization. Second, the dynamic model for manipulated objects is simplistic. From what I understand, no consideration was given for the center of mass of the object to predict object rotation during contact. Lastly, there are better ways for trajectory optimization than random shooting.\n2. The paper does not dive deeper into the sources of VoxPoser perception error. Where in the perception pipeline does the error commonly occur (OWL-ViT, Segment Anything, XMEM, etc)?\n3. The paper’s scope is limited to robot manipulation, further work on mobile robot trajectory planning would validate the generalizability of this technique.\n4. The paper was generally concise. However, the writing is not always clear. The appendix was needed to understand the scope of prompt engineering for each LLM call, which should an important detail of the implementation.\n", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "22XVtOzazwH", "reviewer": "Reviewer_tgHR", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes a model that learns to create a 3D value map to achieve language-grounded manipulation tasks by leveraging large language and vision-language models. Instead of directly learning a policy, the authors formulate the task into optimization of trajectory generation for sub-tasks given by a high-level planner. LLM is used to write a program that uses VLM to identify objects and creates affordance and avoidance maps. The maps are used to create a trajectory and motion planning is done in closed loop. The approach outperformed baselines in experiments with both seen and unseen instructions and attributes. It also showcased the ability to learn dynamics model by leveraging the trajectory in a contact-rich task.", "summary_of_recommendation": "(Updated) I recommended a strong accept. It is well-written, the idea is easy to understand, and the result seems impactful. It cleverly leverages large models and decomposes scene understanding from policy learning which often combined to make the task harder. Maps and trajectories are interpretable and control is (partly) isolated which can make this approach adoptable. After the rebuttal, a question on extension to complex motion was resolved and the comparison was also promised.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "* What does the specification error mean in the proposed method?\n* Will it be possible to do re-try a task after a failure with some updated states? For instance, during opening a drawer the robot failed to grab the handle.\n* I wonder how difficult it would be to extend the control space for more complex motions; what would be the most challenging part when we want to try this with a robot hand?\n* Have you tried with other dataset such as RLBench? It would be helpful for readers to understand the performance of the system by comparing competitive baselines.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Strength\n* The paper is clearly written.\n* Easy to interpret the behavior as it is modular and produces value maps.\n* Showed strong generalization in unseen attributes.\n* High potential in practical use due to its modularity; free from primitives, applicable to different robots, close-loop control.\n\n# Weakness\n* No comparison on public dataset such as RLBench and no comparison with competitive language-grounded manipulation methods such as PerAct.\n* As the model relies on pre-trained vision model, it might not be able to apply to visually different/diverse datasets or adapt to a (visually) challenging task.\n* When the control dimension gets higher, the template-based code might not be sufficient to handle the robot constraints.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a method of creating a value map for manipulation task using an LLM model. The learned valued map can be used in model-based controller to create a trajectory. Thanks to the LLM, the method can ceate a value map conditioned on complex texts such as "measure weight of apple" and "turn on lamp".

All reviewers have positive opinions on this paper. One of the reviewer questioned about the novelty of this method because it is highly dependent on the LLM, but I agree with the authors' defense that code generation scheme is new and novel. Even though the contribution might look simple, this paper will make other researchers aware how powerful LLM's can be for robotics.

Other questions that the reviewers raised are minor and can be reflected on the revision easily. I also found that the authors' comments are clear and rational. The also clearly stated the limitations such as not considering the robot arm for collisions and trajectory optimization.

Another strength of this paper is an extensive robot experiments. Compared to other papers submited in this topic, this paper presents a rich set of experiments.
Decision: Accept (Oral)