Trust the PRoC3S: Solving Long-Horizon Robotics Problems with LLMs and Constraint Satisfaction
Authors: Aidan Curtis
Nishanth Kumar
Jing Cao
Tomás Lozano-Pérez
Leslie Pack Kaelbling
Venue: CoRL 2024 (2024)
Keywords: LLMs for planning
task and motion planning
constraint satisfaction
TL;DR: having LLM's produce code that corresponds to a CSP, then solving that CSP separately, enables solving long-horizon robotic manipulation tasks with complex and realistic constratins

Abstract: Recent developments in pretrained large language models (LLMs) applied to robotics have demonstrated their capacity for sequencing a set of discrete skills to achieve open-ended goals in simple robotic tasks. In this paper, we examine the topic of LLM planning for a set of *continuously parameterized* skills whose execution must avoid violations of a set of kinematic, geometric, and physical constraints. We prompt the LLM to output code for a function with open parameters, which, together with environmental constraints, can be viewed as a Continuous Constraint Satisfaction Problem (CCSP). This CCSP can be solved through sampling or optimization to find a skill sequence and continuous parameter settings that achieve the goal while avoiding constraint violations. Additionally, we consider cases where the LLM proposes unsatisfiable CCSPs, such as those that are kinematically infeasible, dynamically unstable, or lead to collisions, and re-prompt the LLM to form a new CCSP accordingly. Experiments across simulated and real-world domains demonstrate that our proposed strategy, \OursNoSpace, is capable of solving a wide range of complex manipulation tasks with realistic constraints much more efficiently and effectively than existing baselines.

Trust the PRoC3S: Solving Long-Horizon Robotics Problems with LLMs and Constraint Satisfaction
Aidan Curtis', Nishanth Kumar", Jing Cao, Tomás Lozano-Pérez, Leslie Pack Kaelbling
MIT Computer Science and Artificial Intelligence Laboratory fcurtisa, njk, jingeao, tlp, 1pk)0caail wwtpaaaiiiiit...mmmmmm.mm... edu.
Abstract:
Recent developments in pretrained large language models (LLMs) ap- plied roboties have demonstrated their capacity for sequencing set of discrete skills to achieve open-ended goals in simple robotic tasks. In this paper, we ex- amine the topic of LLM planning for a set of continuously parameterized skills whose execution must avoid violations of set of kinematic, geometric, and phys- ical constraints. We prompt the LLM to output code for a function with open parameters, which, together with environmental constraints, can be viewed as a Continuous Constraint Satisfaction Problem (CCSP). This CCSP can be solved through sampling or optimization find skill sequence and continuous param- eter settings that achieve the goal while avoiding constraint violations. Addition- ally, we consider cases where the L.M proposes unsatisfiable CCSPs, such those that are kinematically infeasible, dynamically unstable, or lead to colli- sions, and re-prompt the LLM to form a new CCSP accordingly. Experiments across simulated and real-world domains demonstrate that our proposed strategy, PRoC3S, is capable of solving wide range of complex manipulation tasks with realistic constraints much more efficiently and effectively than existing baselines. Website: https://proc3s.csat wwwp://wss..s............ mit.mm..
Keywords: LLMs for planning, task and motion planning, constraint satisfaction
1 Introduction
Recent progress large-scale foundation models, particularly large language models (LLMs) and vision-language models (VLMs), has enabled variety of flexible and general-purpose decision- making systems for robotic tasks [1, 2, 3, 4, 5, 6, 7]. These systems leverage few-shot prompting as well as the commonsense and sequence prediction abilities of LLMs and VLMs to output se- quences of robotic skills that achieve wide variety goals. Such systems are generally more capable at handling open-world environments than classical systems, like task and motion planners (TAMP) [8], since they often do not require hand-specified symbolic components (e.g. predicates and operators), and can perform tasks specified directly in natural language images.
While foundation models have been applied to range of robotic tasks, these tasks share many com- mon simplifying and limiting assumptions. In most cases, the system is provided with fixed set of discrete skills such as robot.. move_to .door or robot. wwcoot.pickkeoe pickk_ccannnn () and asked to perform tasks that simply require composing these skills in particular order. Discrete skills with minimal to no control over the skill outcome may be sufficient for simple tasks and settings, but insufficient for domains with complex constraints goals that depend continuous properties or relationships, which are common in robotics. For instance, consider the goal shown in the Arrange- YCB domain from Figure 4 in which the robot is tasked with packing set of YCB objects into a small region. Collision constraints on placing locations restrict the space possible picking grasps on each ob- ject. A single monolithic robot. pwikppiccaaaa can skill not sufficient in this context, regardless of
Equal contribution.
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Grasp Fallures
Obstacle Collisions
Kinematic Infeasibility
Lack Stability
Figure I: Illustration of some common constraints in robotic domains.
that skill's success rate isolation. Additionally, goals that specify continuous properties or rela- tionships between objects require actions with continuous parameters. For example, goal such as "place the object in the top right corner" "draw star" displayed in Figure 4 require the planning system to have fine-grain fine-grained access to specific skill outcomes to avoid the kinds of failures displayed in Figure 1. Such access requires parameterized skills [9, 10].
We seek to address these limitations and enable planning systems based on foundation model to address complex robotic tasks with realistic constraints using parameterized skills. Towards this goal, we take inspiration from TAMP, and separate planning into two distinct phases [11, 12, 8]. In the first phase, we ask an LLM to generate program that takes in some continuous parameters as input and produces sequence of skills with all their parameters specified. We also prompt the LLM to associate a sampling function with each continuous parameter. The result of this phase is thus kind continuous constraint satisfaction problem (CCSP) [13, 14, 8, 15]: the solver must now find values for the continuous parameters each skill such that the proposed sequence of skills can be executed achieve the goal without violating any constraints the world (Figure 1). The second phase attempts to solve this CCSP via simple generate-ance -ntrrate--nd-ess-- procedure. If it is unable to find setting for all the continuous parameters that leads to goal achievement, reports the failure modes encountered and asks the first stage for new, different sequence that resolves this issue. This planning process, similar to TAMP, is carried out entirely within a simulated world model, which can be constructed from visual inputs using pretrained perception models [16]. Once viable plan found, we execute the real environment and replan if necessary.
We evaluate approach, Planning for Robots via Code for Continuous Constraint Satisfaction (PRoC3S), on range of challenging robotic tasks in three different simulated domains, and one real-world domain. In particular, we measure the agent's 's success rate at completing the tasks in- volving drawing, rearranging, stacking, and packing objects into configurations specified by a natu- ral language goal. contrast to classical planning systems, our approach demonstrates the ability to satisfy diverse set of natural language goals without symbolic predicates or operators. Our system also exhibits greater robustness to real-world constraints than existing methods that apply foundation models to robotics problems.
2
Related Work
The traditional approach to solving long-horizon robotics problems with complex constraints is task and motion planning (TAMP), which combines both higher-level logical planner and a set of low level parameterized skills [8, 11, 16, 12, 17]. While this is powerful framework that enables zero-shot generalization to new problems, the set of capabilities of the system are limited to goals that can be expressed using some set of pre-specified predicates, and via a sequence of pre-specified symbolic operators. Some recent work has used LLMs to guide search and translate natural language goals into logical ones, but still require manual specification operator preconditions and effects or goal predicate classifiers [18, 19, 20]. By contrast, our approach is able accomplish tasks where specifying symbolic predicates and operators is either challenging or impossible (e.g. the tasks from our 'Drawing" domain in Section 5).

---

Scene Descziption
Scene Desexiption
Gemerator
Constraint Checken
Exaepee Tark
(Chipee:
des cTa Sata
Plan x N
"pes" (7.4.0)
DR
ca. 14
Goal:
Goal:
Skills s53)
(6ew, Linete, 10, 1000
Success
Plan Generatox
Paranetez: Samplez
Generated LMP
Exaepte LMP
Paranetez Donains
vale lorgh Flst
ext aeain jasstaat: Xaie)
Paraneter Donsins
Pretrained LLM
Yoweer [e. 198
donate cintesali: Geueag:
Senter [A. 191
Taa C5 +
"atde Lingers (3N, 20
Constraint Violacion Feedback
Figure 2: Overview of PRoC3S. An LLM prompted with an example initial state, goal, LMP, and associated LMP domain for drawing square. When prompted with a new state and goal for drawing a star, the language model outputs new LMP and associated domain. We then sample inputs to the function and test them against set of pre-specified constraints via simulator. If no satisfying assignment found after N samples, we feed back the primary failure modes the LLM to generate an updated LMP and domain.
Recent advances in LLMs have enabled LLM-based planning systems wherein the available skills are described in natural language and their effects on the world need not explicitly defined [21]. One of the first such approaches sequences discrete object-specific skills to satisfy rearrangement goals [4]. Many follow-up papers have extended this framework handle longer-horizon tasks with temporal dependencies [7, 22, 2]. Others have made these action- strategies more reactive by reprompting with feedback from the environment or running optimization over skill se- quences [23, 6, 24, 25]. Although more flexible than TAMP, these approaches can only make use of discrete skills with no continuous input parameters, which greatly restricts the class of problems they can solve. Some recent work has sought to remedy this by having the L.MM generate code that trans- forms environment parameters into action input parameters via a code interpreter [1]. A downside of this method is that does not properly handle kinematic, collision, or dynamic constraints of the robot's embodiment. Other approaches have the LLM directly output continuous action parameters and use environmental feedback to adjust those parameters to satisfy encountered constraints [3, 24]. These methods directly rely on an LLM or VLM to resolve constraints, which assumes these model are capable of complex geometric and physical reasoning they are generally not trained for.
Some existing work has taken environmental constraints into account when executing skills by build- ing more context-aware skill primitives [5, 26) used solvers to satisfy LLM-suggested constraints for non-robotic domains [27, 28). We instead focus on long horizon robotic manipulation problems with continuously parameterized skills and temporally dependent constraints.
3 Problem Setting
We consider a robot planning task with object-oriented states and parameterized skills defined by the tuple (Lg, S, 80, & f, C)) Here, lg is a natural language goal represented by a string corresponding to some unknown goal condition defined over the state space G S.
S is the robot's state-space. We assume the state is object-oriented and fully-observable: i.e., it is factored into a discrete set of objects, each with set of attributes that may be discrete, continuous, or string. We assume each object is an instance of a class in the object-oriented programming sense. Given finite set of objects, the state space S is defined by the possibly infinite set of assignments to these object's attributes. The initial state so is a collection of instantiated objects. For example, an initial state of our Arrange-Bloos Blocks environment with two objects might be:
("o1"; Object (cat="block' color="yellor", pose- [ose=[0.040 (0.04, -0.36, 0.02, 0.0, -0.0, -0.0]) "22": Object (cat="bowl", color="green", 10..=[-0.14 -0.35, 0303, 0.0, -0.0, 0.0})
3

---

We also assume that the robot has access to a set of lifted parameterized skills Q. Each lifted skill & E Φ (e.g. Pick([obj], [x, y, 2, r, P, y] )) has a name (i.e., Pick), a natural language deseription (e.g. "Move the robot's gripper to location x, y, z and close the gripper.)), and a tuple of discrete (i.e., [obj]) and continuous parameters θ (i.e., [x, y, 2, I, P, y)) that govern the behavior of the skill. Each À E A has discrete domain and each 0 E θ has continuous domain that are skill-specific. For example, the obj parameter would have domain consisting of the names of all the objects in the current world state (i.e., banana, spam, etc. in the Arrange-YCD domain), while the [x, y, z] parameters would have domains corresponding to the edges of the table surface, and the [r, P, y] parameters are constrained to be within [0, 2r] radians. A lifted skill & can be grounded á by selecting values for each of the parameters, resulting in a possibly infinite set of ground skills Φ. Ground skills can executed from any & E S and terminate upon reaching a skill-specific termination condition, which will result in new state s' E S, where it's possible that s s'. For instance, Pick (banana, [0.1, 0.2, 0.16, / 0, x/4]) attempts a grasp relative to the banana pose. It can be executed from any state in the Arrange-YCe YCB environment, and may or may not pick up the banana depending on these parameters. Lastly, as in any planning system, the robot is given access to a transition model f S x + 7 S. In our case, this is implemented with a physics simulator.
Lastly, we assume finite set of user-defined constraints C. Each constraint C₁ E C' has cor- responding natural language description of what violating the constraint entails (e.g. "Pose is not reachable by gripper"), and classifier ci S - {true, false) mapping a state E S to boolean value indicating whether or not the particular constraint is violated. Constraints may be induced by the kinematics of the robot, collisions with the environment, or dynamic properties like stability of the robot or objects the robot is interacting with, and are common across a wide range of robotic tasks. To check whether a constraint has been violated, we will set our simulator f to a particular state and call the constraint's classifier function?.
The robot's objective is to find plan L 2,..., 2x] defined by sequence of ground skills such that: (1) sequential execution of the plan from so yields a state sequence [$1, (s..,,.... sK+1] such that 8K+1 E G. and (2) no state E [$1, 82,...,,K 8k+1] violates a constraint function (i.e., As e (81, .2...., ",...., 8K+]] 3e, E C.: o (s) true)
4 Method
Following previous work [1, 3, 2], we solve planning tasks by querying an LLM to directly generate sequence of skills that achieve the goal from the initial state sg. However, generating plan with an LLM is challenging problem because it involves both correctly sequencing skills together, and also finding a specific setting of all the continuous parameters that enables the plan to achieve the goal. For instance, consider the "Draw a star" task depicted in Figure 4. Here, the robot is provided with a robot. . draw_line (ro, Nos 31, 1/1) skill, that draws straight line between the points (Zo: yo) and (I1, 91) respectively. To successfully accomplish the task, the robot must invoke this skill at least times in sequence. Moreover, it must specify at least 20 continuous parameters (10 pairs of (wo, yo) tuples) such that the shape can be drawn without violating collision reachability constraints.
To address these challenges, we take inspiration from TAMP in two significant ways: (1) we provide the LLM with access to code for a set of samplers [29, 8), Σ, to help it sample continuous param- eters, and (2) we separate planning into a two stage LLMMMooulo framework [30] with each stage designed to solve a different part of the overall planning problem. Samplers are named functions that take in one more arguments, as well as particular arguments, and output set of continuous values that may be useful for grounding skill(s). For instance, simple uniform random sampler (which we call Continuous in the code and examples below) might sample a number uniformly at random within some provided bounds. A grasp sampler might take in no arguments and simply
Note that our notion of constraint broader than the typical notion in the CSP literature. Some our constraints are implicir: they are checked via a simulator rather than expressed as simple symbolic expressions. 'Overall, our problem setting is analogous that of TAMP [8] without symbolic predicates operators.

---

output a valid grasp. Note importantly that these samplers are generally unaware of the constraints: a grasp sampler might output grasp that is kinematically infeasible or unstable.
We leverage these samplers within a two-stage planning process. In the first stage of planning, which call LMP generation, we prompt LLM generate a Language Model Program (LMP) [1]. This LMP is function that takes in the text representation of the object-oriented state and certain parameters and outputs plan that we assume achieves lg. We also ask the LLM to generate bounds for and invoke the provided samplers to yield sampling function that outputs values for the parameters of the LMP. In the next stage, which we call constraint satisfaction, we sample parameter choices for inputs to this LMP and execute them in our simulator to find parameters that ensure the plan does not violate any constraints from C. If the constraint satisfaction phase fails after fixed sampling budget, we pass information about the most common constraints violated back to the LMP generation phase and request new LMP and constraint bounds in light of the observed failure. An overview of this process is depicted in Figure 2.
We now discuss each phase in more detail. To ground this discussion, consider a simple running example in the Arrange-Blocks domain shown in Figure 4. Here, lg is: "Place the green block in the bowl". The state represented using the Object class mentioned in Section 3, and the initial state is such that orange block (o12) is atop the green block (o7). The robot is provided with single simple continuous sampler and two skills, pick (x, y ,2) and place (x, (x,y22) z), that move the gripper to particular (x,y,z) 2) location and then close/open the gripper respectively.
LMP Generation: The objective of this stage is to generate an LMP that consists of: (1) a plan- sketch function that takes in a state as well as some arbitrary input parameters and outputs a plan Lg 2, ..., @b dyJ when executed with an interpreter', and (2) sampling function that leverages sam- plers with with the user-defined environment constraints C' defines CCSP, and (2) helps define sampling procedure that can be leveraged to solve this CCSP. To achieve this, we prompt an LLM with (a) the classes and objects used to represent the state-space S, (b) the initial state 80. (c) the available parameterized skills 4, (d) the provided samplers Σ, and (e) an example of an expected output LMP from different task that shares the same state-space and many of the same skills (see Appendix for the prompts used in our three environments). Importantly, note that the LLM is not provided with any of the constraints (rather, these will be checked in the next phase).
bounds to output parameters that (1) takes input. Here, (1) together
Consider the following generated LMP on our running example task of placing a block into a bowl:
def gen_plan
dx, dy):
def gen_ domain (immainnini::Sttte)) return "dx": Continuous (4, .04), "dy": Continuous (-.iinuouus--00., .04),
plan []
block, bowl init["o7"], init ["""]
plan += action("picke block. point)]
x, y, bowl.
plan += (etiicm""plce"" [x+dx, ytdy, z])]
return plan
Here, the gen-plan function is the pkan-sketch. It takes in a particular state (named init) cor- responding to the initial state sg, as well as two parameters and generates a plan in terms of the provided pick and place skills. Importantly, note that the two input parameters to the function (namely dx, dy), are different from the parameters of the pick or place skills. The above gener- ated sequence of one pick and one place skill would ordinarily require six continuous parameters (x, y, z for each skill) for grounding. However, the generated LMP reduces the sampling space to a lower-dimensioonl, two-parameter space (namely dx, dy). Thus, the generated LMP makes downstream CCSP simpler by leveraging the common-sense and reasoning capabilities of LLMs.
Constraint Satisfaction and Feedback: In this work, we opt for very simple sample-and-test procedure for constraint satisfaction. Specifically, we sample fixed number of values for all the plan-sketch's 's input variables (namely dx and dy in the above example) using the sampling function
'This function represents a family of plans that only differ
continuous parameters.
5

---

Code Policy
LLM
Gaussian Noise
Ours
Figure 3: Top-down view of solutions produced by baselines the star drawing task in 'Drawing' domain.
that was generated by the previous stage. Given a particular sample, we can simply evaluate the LMP to output plan. We then check for constraint violations by executing each step of this plan using our simulator f, and running each constraint classifier cf from e₁ C.
If plan is found that does not violate any constraints, return this plan. If satisfying plan found after the fixed budget is exhausted, we enter a feedback stage. The objective of this stage is to provide information to the first stage such that it will return new LMP that avoids the same constraint violations. In our implementation, we return the following information during feedback: (I) descriptions of the top most common constraint violations, (2) the most common ground skill name that was run the simulator before the violation, and (3) the most common index in the plan at which each of the above constraint violations occurred. This loop between LMP generation, constraint satisfaction and feedback continues until legal plan is found.
In our rumning example, we exhaust our sampling budget due to collisions with the orange block and return the following feedback information: "Step 0, Action pick, Violation: Collision detected between object o12, gripper." ". Given this, as well as the context of its previ- ous LMP, the LLM generates a new LMP that moves object o12 out of the way before object o7 (the green block of interest) is manipulated:
def gen gen_plan (init: State, dx, dy,
def sen_domain (init:
x_place_12 12, y-place_12):
return
plan = [
"dx": Continuous (- 04, .0),
block 12 init [n12""
"dy": Continuous (- 04))
plan [Action ("pick", block block_12.pointt point)]
"xplace__12 Continuous
plan += [Action ("place"""ace",, [x_place_12]
TABLE_BOUNDS
BOUNDS [0] [o]
y-place_12, TABLE_BOUNDS [2] [2BLE_BOOUDDS]]][1]]] [1]])]
TTALL_BOUNDDS [0] [1]),
[block, bowl] init["o7"], init Anit["o8") [8" plan [Action ("pick", block.point))
"y "y_place__ 12": Continuous BOUNDS [1] [0],
TABLE_BOUNDS
x, y, 2 bowl.pose.poi plan += (Action""place,, [x+dx, y+dy, z])]
TABLE_ [1] (1])
}
return plan
Experiments
Our experiments are designed to test the ability of our method (PRoC3S) to sequence a set of simple continuously parameterized skills to generalize to satisfying unseen natural language goals while obeying environmental constraints, both in simulated domains and on real-world hardware.
Constraints. We make use of four general constraint types across environments. These are kine- matic constraints on the robot, collision constraints for robot motion, grasp constraints (i.e., check- ing for stable grasps), and placement constraints (i.e., checking for stable placements). Details on implementation are provided in Appendix A.2.
Environments. We now provide high-level environment and task descriptions with details in Ap- pendix A. All of our simulated domains consist of 6 DoF URS robot with Robotig 2F-85 gripper in front of table of objects. Experiments in these domains involve different initial states,

---

Drawing
Arrange Blocks Packing
Arrange YCB
Star
Arrow
Letters
Enclosed
Pyramid
Line
Unstack
Packing
Stacking
L.M
40%
40%
90%
50%
0%
50%
30%
20%
0%
0%
LLM'-NF
20%
40%
20%
10%
30%
60%
20%
0%
LLM--Gaussin Gaussian
20%
0%
0%
0%
30%
40%
30%
20%
0%
0%
CaP
10%
0%
50%
30%
20%
20%
20%
20%
40%
10%
CaP-Gaussian
10%
20%
0%
40%
10%
30%
30%
30%
20%
10%
PRoC3S-NF
100%
40%
50%
90%
30%
10%
70%
20%
10%
40%
PRoC3S
80%
80%
80%
90%
60%
70%
70%
70%
60%
70%
Table I: Percentage of correct final states over 10 evaluations across simulated domains. Values not signifi- cantly different from the top performer are bolded. Tests one-tailed Z-test with 0.1.
object sets, and goals based on the simulated Ravens tabletop environment first introduced by Zeng al. [31]. Our real-world domain consists of a tabletop setup with a Franka Emika Panda robot.
Drawing:
The robot is provided with parameterized skill to draw line, and variety of goals involving drawing different shapes. We attempt four different goals in this environment: "draw a star", "draw an arrow pointing at the biggest obstacle in the environment", "draw the letter M and "draw a shape that encloses two obstacles". The main challenge of these tasks is drawing a shape that avoids collisions with the obstacles in the cluttered environment.
Arrange-Blew Blocks: A tabletop front of the robot is strewn with a variety of colored blocks and bowls. We attempt 4 tasks in this environment: "stack an upright pyramid out of three blocks", "Put five blocks in line flat the table", "Place all blocks within 0.06 of the center of the table", and "Place the green block in a bowl". The first three are challenging due to low-tolerance stability and collision constraints. The final task is challenging because there are always other blocks atop the green one that prevent it from being picked directly.
Arrange-YCui YCB: The same as the above Arrange-Blocks Blocks environment, but with objects from the YCB dataset [32] instead of blocks and bowls. The goals are: "Place all objects within 0.06 of the center of the table", "stack any two objects". Both of these tasks require satisfying grasp constraints on arbitrary object meshes and kinematic/reachailiit constraints on the selected grasps. In the case of the packing problem, collision constraints between objects are the main constraint violation, whereas placement stability is more significant for the stacking task.
Real Robot Domain: We implement 4 tasks from the 'Arrange-Bloce Blocks' and 'Arrange-YCC YCB' domains on a real-world tabletop setup with a Franka Emika Panda robot. The goals are: "Put three/five blocks in line flat on the table" (referred to as '3-line' and '5-line" respectively). "Place all blue blocks in the blue bowl and red blocks in the red bowl" ('Sort'), and "Place See Appendix D for an
all objects within 0.06 of the center of the table" CYCB-packing illustration of our setup, as well implementation details necessary to apply our approach to this challenging domain. Note that we report results from directly executing computed plans 'open-loop" on the real robot and do not attempt to replan upon failure.
Approaches. We now briefly describe the approaches that we compare to PRoC3S.
PRoC3S without feedback (PRoC3S-NF): PRoC3S but with the feedback component ablated. Thus, if the first returned plan doesn't work, we consider the task failed.
Code as Policies (CaP) [1]: This approach attempts to write helper functions and leverage exist- ing Python libraries to output an LMP that produces skill sequences and corresponding continu- ous parameters give task, without any environment feedback.
In an approach loosely based on a previous paper [18], We add gaussian noise the action space of the skills output by the LMP generated by CaP, which can help avoid constraints violations but detracts from the intended goal. See Appendix B.5 for details.
LLM$33:: This approach performs LLM planning with parameterized skills and feedback, but uses the LLM to directly output continuous parameters. We also ablate the feedback component of this approach (.L.3-NFF. and include a version that places Gaussian noise on the output similar to CaP-Gaussian (LLM* -aaussian..
Experimental Setup. For each task and approach, run 10 random seeds where randomize the initial locations and sizes (where appropriate) of objects. For our real robot domain, we run 5 random
7

---

seeds of the approach that had the best performance across simulated domains with randomization of the initial state. For approaches that use feedback (i.e., PRoC3S and and LLM²), we limit the number of feedback iterations to 5. We specify three samplers (E) and provide all approaches with access to them (details in Appendix C and Appendix B). We a sampling budget of 10000 for tasks in the Drawing domain, and 1000 for all other domains. For all approaches, we use the OpenAl GPT- 4 LLM [33], specifically the gpt-4-0125-previts
checkpoint. We record success achieving the natural language task goal as judged by the authors and report additional metrics in Appendix A.1.
Results and Analysis. As can be seen from Table 1, PRoC3S consistently achieves the highest suc- cess rate across simulated domains. PRoC3S-NF's success rate is at least 30% lower on most tasks, illustrating the importance of feedback. This is further validated by the fact that the performance of LLM is significantly worse than LLMS. We see that LLM itself performs comparably to PRoC3S on tasks, and find that it falters because the LLM is unable directly output continu- ous parameters that satisfy the various task constraints, can be seen from the example in Figure 5. CaP is only comparable to our approach on one task and generally fails because it is unable to generate viable continuous parameters (e.g. Figure 5). While gaussian noise helps these baselines avoid constriant violations, it prevents them from achieving the overall task goal as shown in Figure Additionally, Table 2 indicates that our approach able to operate on challenging real-world variants of several simulated tasks and achieve a non-trivial success rate.
Qualitatively, we observe two common failures in simulation. Firstly, we
Seed
3-line
5-line
Sort
YCB BCB-packing
1
observe cases in the 'Ar- range Blocks' and 'Arrange YCB' domains where method cannot correctly re- fine its LMP given feed- back. Once an incorrect modification is made, we
2
V
Grasp fail
3
CSP timeout
Grasp fail
4
Grasp fail
5
Self-collision
Object collision
Table 2: Results on weal-world robot tasks. v/ indicates success.
find that the LLM rarely recovers in future feedback iterations. Secondly, as observed prior works [34, 30J, we find that the LLM is unable to yield a valid plan for the longer horizon problems. On the real-world tasks, we observe that most failures stem from the sim-to-real gap. Only one observed failure ('5-line" task seed 3) was due to our method failing to find plan: all other failures were due to failed grasps or unexpected collisions during real-world execution after planning.
6 Limitations and Future Work
There several limitations of our method. Firstly, our method requires a physics simulator, which introduces significant sim-to-real gap that can lead execution failures (as observed in Table I). Secondly, the open parameters in the generated LMPs from the first phase of our method depend heavily on the example task we choose to provide as part of the input prompt, making effective prompting crucial for the success of our approach. Thirdly, our method of solving CCSPs naive and can be very slow, especially in domains like Arrange-YCCe YCB. Additionally, our framework guar- antees neither soundness nor completeness nor optimality of the generated plans.
In future work, we aim to improve the sampling technique for continuous parameters, opting for a backtracking search or optimization algorithm to select parameter values as part of the sampling phase of our method. As part of this, we hope to explore expressing constraints terms of contin- wous cost functions instead of binary failure detectors. We also plan to instill the robot with visual reasoning by exploring the use of vision language models (VL.Ms), potentially enabling it to recog- nize wide range of constraint violations without us having explicitly define constraint classifier functions. Lastly, we hope extend our method to partially-observable environments and thereby tackle larger mobile-manipulation tasks.
8

---

Acknowledgements
We gratefully acknowledge support from NSF grant 2214177; from AFOSR grant FA9550-22-1- 0249: from ONR MURI grant N00014-22-1- 1-2740; from ARO grant W911NF-23-1-0035 from the MIT Quest for Intelligence; and from the Boston Dynamics Artificial Intelligence Institute. Aidan and Nishanth are supported by NSF Graduate Research Fellowships. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of our sponsors.
References
[1] J. Liang, W. Huang, Xia, P. Xu, K. Hausman, B. Ichter, Florence, and A. Zeng. Code as policies: Language model programs for embodied control. In arXiv preprint, 2022.
[2] I. Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg. Progprompt: Generating situated robot task plans using large language models, In IEEE International Conference Robotics and Automation (ICRA), 2023. URL https:
// //arxiv. owggvv.or///ie. pdf / 2209. 11302.
[3] S. Wang, M. Han, Z. Jiao, Z. Zhang, Y. N. Wu, S.-C. Zhu, and H. Liu. Llme:large language model-based task and motion planning with motion failure reasoning, In arXiv preprint, 2024. URL htttp:///axxiii org/pdf/ 2403. 11552.
[4] B. Ichter, A. Brohan, Y. Chebotar, C. Finn, K. Hausman, A. Herzog, D. Ho, J. Ibarz, A. Ir- pan, E. Jang, R. Julian, D. Kalashnikov, S. Levine, Y. Lu, C. Parada, K. Rao, P. Ser- manet, A. T. Toshev, V. Vanhoucke, Xia, T. Xiao, P. Xu, M. Yan, N. Brown, M. Ahn, O. Cortes, N. Sievers, C. Tan, S. Xu, Reyes, Rettinghouse, J. Quiambao, P. Pastor, L. Luu, K.-H. Lee, Y. Kuang, S. Jesmonth, N. Joshi, K. Jeffrey, R. J. Ruano, J. Hsu, K. Gopalakrishnan, B. David, A. Zeng, and C. K. Fu. Do i can, not as i say: Ground- ing language in robotic affordances. In Conference Robot Learning (CoRL), 2023. URL attps://proceett
mlr press/v205/e 2 html.
[5] H. Huang, F. Lin, Y. Hu, S. Wang, and Y. Gao. Copa: General robotic manipulation through spatial constraints of parts with foundation models. In arxiv preprint, 2024. URL https: // //arxiv.. www.ooggg......o /2033 /203.0000. 08248.
[6] Y. Guo, Y.-J Wang, L. Zha, Z. Jiang, and J. Chen. Doremi: Grounding language model by detecting and recovering from plan-execution misalignment. In arxiv preprint, 2023. URL nttps://arxx org/paf 2207. 00329.
[7] W. Huang, Xia, D. Shah, Driess, A. Zeng, Y. Lu, Florence, Mordatch, Levine, K. Hausman, and B. Ichter. Grounded decoding: Guiding text generation with grounded mod- els for embodied agents. In arxiv preprint, 2023.
[8] C. R. Garrett, R. Chitmis, R. M. Holladay, B. Kim, T. Silver, L. P. Kaelbling, and T. Lozano- Pérez. Integrated task and motion planning. CoRR, abs/2010.01083, 2020. URL https: // //arxiv.orz org/ abs/ 01083.
[9] B. Da Silva, G. Konidaris, and A. Barto. Learning parameterized skills. In International Conference on Machine Learning (ICML), 2012. URL https://arxiv wttpddv.orrp///////0/// 6398. 63..
[10] N. Kumar, T. Silver, W. McClinton, L. Zhao, Proulx, T. Lozano-Pérez, L. Kaelbling, and J. Barry. Practice makes perfect: Planning leam skill parameter policies. In Robotics: Science and Systems (RSS), 2024. URL https: // /ttp//.v.rggppppeaa//
org/pdt////// http//dff244222 /2002.4022222 15025.
[11] N. Kumar, W. McClinton, K. Le, and T. Silver. Bilevel planning for robots: An illustrated introduction. 2023.
URL https://lis csail. mit edu/

---

[12] S. Srivastava, E. Fang, L. Riano, R. Chimiss, S. Russell, and P. Abbeel. Combined task and motion planning through an extensible planter-independent interface layer. In IEEE interna- rional conference on robotics and automattion (ICRA), 2014. URL https: http:p::/pppp..../meee eecs.
[13] T. Miltzow and R. F. Schmiermann. On classifying continiuu constraint satisfaction problems. CoRR, abs/2106. 0209970 2021. URL http:////aaaaii.o//oaa////////// 023977.
[14] Y. Shang and M. Fromherz. Experimental complexity analysis of continuous constraint satisfaction problems. Information Sciences. URL https://www..cieeneediie
[15] Z. Yang, J. Mao, Y. Du, J. Wu, J. B. Tenenbaum, T. Lozano-Pérre Pérez, and L. P. Kaelbling. Com- positional Seffusion-Baset Continuous Constraint Solvers. In Conference on Robot Learning (CoRL), 2023.
[16] A. Curtis, X. Fang, I.. P. Kaelbling, T. Lozano-Pérez, and C. R. Garrett. Long-horizon ma- nipulation of unknown objects via task and motion planning with estimated affordances. In International Conference on Robotics and Automation (ICRA), 2022. URL https: //axivv. org/pdf 2200 04145.
[17] N. T. Dantam, Z. K. Kingston, S. Chaudhuri, and L. E. Kavraki. Incremental task and motion planning: A constraint-based approach. In Robotics: Science and Systems (RSS), 2016. URL attps://wwe www..rooocccpocceediieeeo http:://rr..rbootccpgcceddddd http://wwwbbbooo..oooooogggggg// pdf.
[18] Y. Ding, X. Zhang, C. Paxton, and S. Zhang. Task and motion planning with large language models for object rearrangement. In 2023 JEEE/RS/ International Conference on Intelligent Robots and Systems (IROS), 2023. URL https://arxiv; http://awxxvv....ppeeeee owg/pot/gg//oo/oooooo/goooo. pptpp////330 06247.
[19] K. Lin, C. Agia, T. Migimatsu, M. Pavone, and J. Bohg. Text2motion: from natural language instructions to feasible plans. Autonomous Robots (AuRo), 2023. htttts////axiii /2303 12153.
[20] Y. Chen, J. Arkin, C. Dawson, Y. Zhang, N. Roy, and C. Fan. Autotamp: Autoregressive task and motion planning with Imms as translators and checkers. In arXiv preprint, 2024.
[21] W. Huang, P. Abbeel, D. Pathak, and I. Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. CoRR, abs/2201.07207, 2022. URL
https: / /arxivv..... org/
07207.
[22] R. Hazra, P. Z. Dos Martires, and L. De Raedt. Saycanpay: Heuristic planning with large language models using learnable domain knowledge. In Proceedings of the AAAI Conference on Artificial Intelligence, 2024. URL https: / //t///axivvv////////////////aav. wwtrexiv.vogg/ee hrgg/dff/// /dtff///..0//// /2008 12682.
[23] W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. Florence, A. Zeng, J. Tompson, I. Mordatch, Y. Chebotar, P. Sermanet, N. Brown, T. Jackson, L. Luu, S. Levine, K. Hausman, and B. Ichter. Inner monologue: Embodied reasoning through planning with language models. Conference on Robot Learning (CoRL), 2023. URL https: http:://opppeeveeeeeee www.wwwwwwwwwwwe.eeeeeeeeeeeeeee hed/ppatidd-33322500r..
[24] M. Skreta, Z. Zhou, J. L. Yuan, Darvish, A. Aspuru-Guzik, and A. Garg. Replan: Robotic replanning with perception and language models. In arxiv preprint, 2024. URL https:// arxiv. wacii..org//eee otg/pdf///// 04117
[25] C. Agia, Migimatsu, J. Wu, and Bohg. Stap: Sequencing task-agnostic policies. In 2023 IEEE International Conference on Robotics and Automation (ICRA), 2023. URL https:
// arxai.vv.o wwwoorrgg.....o..o..ooo.oo...e //tfd//////// addf/2210 2210. 12250.
[26] W. Huang, C. Wang, R. Zhang, Y. Li, Wu, and L. Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models. In Conference Robot Learning (CoRL), 2023. URL https: //tpx///xx///e////// .ineerrggiie github. io htt//vopossrrr.o .dff.
10

---

[27] Y. Hao, Y. Chen, Y. Zhang, and C. Fan. Large language models can plan your travels rigorously with formal verification tools. In arxiv preprint, 2024. URL. https://arxiie wwwpiiv..rr///aa abs/ 2404. 11891.
[28] X. Ye, Q. Chen, Dillig, and G. Durrett. Satlm: Satisfiabiliy-aiieed language models using declarative prompting, Advances in Neural Information Processing Systems (NeurlPS), 2024.
URL. https: / //ar//iivvvo////// org/pdft /2005.2355.22 096566.
[29] R. Chitnis, T. Silver, J. B. Tenenbaum, T. Lozano-Perez, and I.. P. Kaelbling, Learning neuro- symbolic relational transition models for bilevel planning. 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2022. URL https://arxiv, org/ pdf 14074
[30] S. Kambhampati, K. Valmeekam, I.. Guan, K. Stechly, M. Verma, S. Bhambri, L. Saldyt, and A. Murthy. Llms can't plan, but can help planning in Ilm-modulo frameworks. In arXiv preprint, 2024. URL
/arxxvraivv/vv. org. ptfpp///2///// 22022 21807.1817.
[31] A. Zeng, P. Florence, J. Tompson, S. Welker, Chien, M. Attarian, T. Armstrong, I. Krasin, D. Duong, V. Sindhwani, and J. Lee. Transporter networks: Rearranging the visual world for robotic manipulation. Conference Robot Learning (CoRL), 2020. URL https:// proceedings mlr. press//115/2eng21/zzzzzg
pdf.
[32] B. Calli, A. Walsman, A. Singh, S. Srinivasa, P. Abbeel, and A. M. Dollar. Benchmarking in manipulation research: The ycb object and model set and benchmarking protocols. In arXiv preprint, 2015. URL https: aaxiv. org/pdf/ 150./1102..00 03143.
[33] O. J. Achiam, S. Adler, S. Agarwal, I.. Ahmad, I. Akkaya, F. L.. Aleman, D. Almeida, J. AI- tenschmidt, S. Altman, S. Anadkat, R. Avila, I. Babuschkin, S. Balaji, V. Balcom, P. Baltescu, H. Bao, M. Bavarian, J. Belgum, I. Bello, J. Berdine, G. Bernadett-Shapiro, C. Berner, L. Bog- donoff, O. Boiko, M. Boyd, A.-L. Brakman, G. Brockman, T. Brooks, M. Brundage, K. But- ton, Cai, R. Campbell, A. Cann, B. Carey, C. Carlson, R. Carmichael, B. Chan, C. Chang, Chantzis, D. Chen, Chen, R. Chen, J. Chen, M. Chen, B. Chess, C. Cho, C. Chu, H. W. Currier, Y. Dai, C. Decareaux, TT Degry, N. Deutsch, D. Deville, A. Dhar, D. Dohan, S. Dowling, S. Dunning, A. Ecoffet, A. Eleti, T. Eloundou, D. Farhi, L. Fedus, N. Felix, S. P. Fishman, J. Forte, I. Fulford, L. Gao, E. Georges, C. Gibson, V. Goel, T. Gogineni, G. Goh, R. Gontijo-Lopes, J. Gordon, M. Grafstein, S. Gray, R. Greene, J. Gross, S. S. Gu, Y. Guo, C. Hallacy, J. Han, J. Harris, Y. He, M. Heaton, J. Heidecke, C. Hesse, A. Hickey, W. Hickey, P. Hoeschele, B. Houghton, K. Hsu, S. Hu, X. Hu, Huizinga, S. Jain, S. Jain, Jang, A. Jiang, R. Jiang. H. Jin, D. Jin, S. Jomoto, B. Jonn, H. Jun, T. Kaftan, I. Kaiser, A. Kamali, I. Kanitscheider, N. S. Keskar, T. Khan, I.. Kilpatrick, J. W. Kim, C. Kim, Y. Kim, H. Kirchner, J. R. Kiros, M. Knight, D. Kokotajlo, L. Kondraciuk, A. Kondrich, A. Konstantinidis, K. Kosic, G. Krueger, V. Kuo, M. Lampe, I. Lan, T. Lee, J. Leike, J. Le- ung. D. Levy, C. M. Li, R. Lim, M. Lin, S. Lin, M. Litwin, T. Lopez, R. Lowe, P. Lue, A. A. Makanju, K. Malfacini, S. Manning, T. Markov, Y. Markovski, B. Martin, K. Mayer, A. Mayne, B. McGrew, M. McKinney, C. McLeavey, P. McMillan, J. McNeil, D. Medina, A. Mehta, Menick, L. Metz, A. Mishchenko, P. Mishkin, V. Monaco, E. Morikawa, D. P. Mossing, Mu, M. Murati, O. Murk, D. ely,y A. Nair, R. Nakano, R. Nayak, A. Neelakantan, R. Ngo, H. Noh, O. Long, C. Oeeefe, J. W. Pachocki, A. Paino, J. Palermo, A. Pantuliano, G. Paras- candolo, J. Parish, F. Parparita, A. Passos, M. Pavlov, A. Peng. A. Perelman, F. de Avila Bel- bute Peres, M. Petrov, H. P. de Oliveira Pinto, M. Pokorny, M. Pokrass, V. H. Pong. T. Powell, A. Power, B. Power, E. Proehl, R. Puri, A. Radford, J. Rae, A. Ramesh, C. Raymond, F. Real, K. Rimbach, C. Ross, B. Rotsted, H. Roussez, N. Ryder, M. D. Saltarelli, T. Sanders, S. San- turkar, G. Sastry, H. Schmidt, Schnurr, Schulman, D. Selsam, K. Sheppard, T. Sherbakov, Shich, S. Shoker, P. Shyam, Sidor, E. Sigler, M. Simens, J. Sitkin, K. Slama, Sohl, B. D. Sokolowsky, Y. Song, N. Staudacher, F. P. Such, N. Summers, I. Sutskever, J. Tang, N. A. Tezak, M. Thompson, P. Tillet, A. Tootoonchian, E. Tseng, P. Tuggle, N. Turley, J. Tworek,
Chung, D. Cummings,
11

---

J. F. C. Uribe, A. Vallone, A. Vijayvergiya, C. Voss, C. L. Wainwright, J. J. Wang, A. Wang, B. Wang, J. Ward, J. Wei, C. Weinmann, A. Welihinda, P. Welinder, J. Weng, L. Weng, M. Wi- ethoff, D. Willner, C. Winter, S. Wolrich, H. Wong, L. Workman, S. Wu, J. Wu, M. Wu, K. Xiao, T. Xu, S. Yoo, K. Yu, Q. Yuan, Zaremba, Zellers, Zhang, M. Zhang, S. Zhao, Zheng, J. Zhuang, W. Zhuk, and B. Zoph. Gpt-4 technical report. In arxiv preprint, 2023.
URL penttps://artte org/ooog/.......o..o /2003 087744
[34] K. Valmeekam, A. Olmo, S. Sreedharan, and S. Kambhampati. Large language models still can't plan (a benchmark for Ilms on planning and reasoning about change). NeurIPS 2022 Foundation Models for Decision Making Workshop. URL httpp::/oppnrrevie... net/pfff id=wUU-7XTL5X0.
[35] E. Coumans and Y. Bai. Pybullet, a python module for physics simulation for games, robotics and machine learning. 2016. URL https://docs. google. com/ document/d/10st 2ye70uns7io3.
[36] C. R. Garrett, T. Lozano-Perez, and L. P. Kaelbling. Pádistream: Integrating symbolic plan- ners and blackbox samplers via optimistic adaptive planning. In International Conference on Automated Planning and Scheduling. 2018. URL https://api. wew.ai.sseaan.cccoola..... org/
JorpusID:210861111 119.66.6666
[37] A. Curtis, X. Fang. L. P. Kaelbling, T. Lozano-Pérez, and C. R. Garrett. Long-horizon ma- nipulation of unknown objects via task and motion planning with estimated affordances. In 2022 International Conference on Robotics and Automation (ICRA), 2022. URL https:
//arxiv. wrg/pdft /2008. 04145.
[38] T. Ren, S. Liu, A. Zeng. J. Lin, K. Li, H. Cao, J. Chen, X. Huang, Y. Chen, F. Yan, Z. Zeng, H. Zhang, F. Li, J. Yang, H. Li, Q. Jiang, and L. Zhang. Grounded sam: Assembling open- world models for diverse visual tasks, 2024. URL http::ttp///aaaee wttpi/..rrg/bb///0/000/ 14159.
12

---

Pack dbjects into region
Drew letter
pyramud
region
painting largest
rechangle any
Form line out blocks
Pace block bowl
Stackany any objects
Drawing
Arrange-Blocks
Arrange-YCB
Figure 4: Illustration of tasks in our simulated environments, along with corresponding language goals.
A Simulated Environment Details and Setup
In this section, we describe details on the constraints implemented across environments, as well as the setup for each task each environment shown in Figure 4. All experiments simulated domains were conducted on machines with 8-core CPU's and 32GB of RAM.
A.1 Additional Experiment Statistics
Tables 3 and 4 show the number of samples and wall-clock time required by our various approaches to perform constraint satisfaction. As can be seen from these results, performing sampling is critical to solving each of our presented tasks. PRoC3S generally requires the smallest number of CSP samples compared to baselines, especially in the first two tasks of the 'Drawing' domain. For tasks where our approach requires more CSP samples (c.g. 'Pyramid' and 'Unstack' 'Arrange Blocks', and 'Packing' 'Arrange YCB'), our approach achieves much higher success rate, indicating the baselines were able to terminate easily because they yielded incorrect plan generators. We observe similar trend in Table 5, which shows the overall planning time required by each method. Table lists the number of feedback queries (i.e., number of times the LLM queried for LMP after the first time) across the various baselines that use feedback. Compared to LLM our approach uses significantly fewer feedback iterations on all tasks except 'Unstack', 'Packing'. and 'Stacking' in which our approach has significantly higher success rate. This is especially noteworthy in the 'Drawing' domain, where our approach requires less than feedback iteration on average. Finally, Table lists the wall-clock time spent querying the LLM for all approaches and tasks. We see that significant fraction of the total planning time taken by approach (over 50% in most tasks) is spent querying the LLM. There are only relatively few tasks (e.g. 'Packing' and 'Unstack' in the 'Arrange Blocks' domain) where sampling takes up the majority of the planning time.
A.2 Constraints
We make use of the PyBullet [35] inverse kinematic solver to reach desired end-effector positions. the solver returns joint positions that result in an incorrect end-effector position, we deem the target as kinematically infeasible.
Kinematic Constraints:
Collision Constraints: If the gripper collides with any unexpected objects during robot motion, consider this collision constraint violation. Expected collisions, such as those between the gripper and held object, not considered collisions.
13

---

Drawing
Arrange Blocks
Arrange YCB
Star
Arrow
Letters
Enclosed
Pyramid
Line
Packing
Unstack
Packing
Stacking
LLM'-NF
LLM'-Gausian
1983.1011
1371.80
5047.501
274.001
897.0011
0.001
3288.06
3028.34
724.10
4558.19
451.81
433.77
1422.02
192.37
0.00
0.00
CaP
1799.000
1145. 90=
2318 00-
4172. 10=
174.40=
199. 20=
126.00=
134.40=
$59.10=
249.00
CaP Gaussian
2534.45
2939.80 80
4766 29
113.95
115.76
PRoC3S-NF
5.70+
8.60=
2001 50+
1001.20 201
30.80
52.40+
513.78+
190.20 20
40.80+
8.56=
9.61
10.01
3998.
2999.22 27
74.34
88.91
162 69
116.52
11.31
3.501
8.70=
1000.70:1
5000.801
336.0021
54,501
312.221
523.701 702
1160.001
3,302
5.68
10.13
3000.10
14998.07
456.99
148.30
403.32
463.13
1332.93
2.24
Table 3: Average number of samples required to solve CSP. Standard deviation is indicated after +. Baselines that do not leverage sampling solve the CSP are listed with Note that these results do not enable direct comparison of methods (i.c., lower numbers are necessarily better) hecause they do account for success rate from Table 1.
Drawing
Arrunge Blocks
Arrange YCB
Star
Arrow
Letters
Enclosed
Pyramid
Line
Packing
Unstack
Packing
Stacking
LLM
LLM
LLM'-Gaussin
3,64
2,35
1,85
8.49 7.80
2,54 3.99
19:281 16.41
18.271 16.01
0.3011 1.16
0.001 0.00
0.004E
5.25
4.81
1.17
0.00
CaP
4.02= 6.26
1.33
3.581
6,632
2.674
3,424
2.751
0.684
473.53=
330.154
CaP-Gaussian
4.63
4.26
7.52
1.52
2.55
5.36
0.36
272.29
220.86
0.021
0.02.
2.691
1,261
6.3311
23.7411
193.7611
1.131
30.50.1
6.341
PRoC3S-NF
0.02
0.02
5.36
3.77
8.22
50.77
222.47
0.50
86.47
5.49
0.02
0.02+
1.26+
10.70+
79.69+
4.98+
165.51 +
29.13+
648.26
8.05++
PRoC3S
0.01
0.02
3.75
32.06
132.53
4.93
230.33
53.43
716.99
13.09
Table Average wall clock time spent on continuous constraint satisfaction. Standard deviation is indicated after -.. Baselines that do not use constraint satisfaction are listed with a -. Note that these results do not enable direct comparison of methods (i.c., lower numbers are not necessarily better) because they do not account for success rate from Table 1.
Grasp Constraints: We consider the following as constraint violations when selecting grasps: (1) the end-effector pose (before the gripper is closed) is in collision with the robot or any other objects in the scene, (2) part of the object is between the two gripper fingers, (3) the object falls out of the hand when tested in simulation.
Placement Constraints: If the object moves significantly if we step physics simulation for fixed number of time steps after the gripper opens to release it, we consider this placement constraint violation.
A.3 Drawing
For each of the four drawing tasks, we randomize the position five circles with random radii.
A.3.1 Skills
draw Fraw_line((e miinepppxx p1.y, p2,x, p2.y)
Moves the arm to a point (p1.x, p1-y, 0.1) (where the table surface is at height 0.0), draws a straight line from point (p1 x, p1 y) point (p2 x, p2 y) while keeping the pen at a pose 0.0, and then moves the arm back up to the point (pi.x, pi-y, 0. 1). The orientation of the gripper is fixed to be town throughout.
14

---

Drawing
Arrange
Blocks Packing
Arrange YCB
Star
Arrow
Letters
Enclosed
Pyramid
Line
Unstack
Packing
Stacking
29.11
14,62+
20.48+
28.28+
16,04
45.25+
24.41
4.33+
0.00+
0.00+
LLM
15.78
7.44
11.42
3.69
12.95
14.82
11.31
0.84
0.00
0.00
NLM
7.451
5.21
5.47
5.181
5.07
8.381
6.481
2.231
0.00.E
6.63
2.48
1.14
1.16
1.29
1.26
0.85
0.26
11.66+
9.09
7.40
16.19+
13.25+
86.09+
37.61+
7.15+
0.00+
0.00+
7.53
7.91
1.43
10.69
11.58
54.32
26.14
7.25
0.00
0.00
15,01
10.30+
16.82+
17.02#
15.77#
16.57=
15.17=
12.34=
15,88
15.57
CaP
1.66
1.43
2.16
2.39
2.64
4.66
4.07
14.50
3.09
2.67
19.714
11.952
20.661
28.531
15.682
22.87=
17.052
13:33=
489 632
349.08=
CaP-Gaussian
8.85
4.70
4.11
10.30
2.85
16.39
5.79
14.53
273.18
220.48
19.34
20.85+
26.10=
18.26+
42.01
42.47
213.86+
14.51=
53.31
25.51:
PRoC3S-NE
2.44
3.82
6.57
5.65
51.83
220.92
1.67
87.38
7.35
20.50=
20.35:
30.41=
37.931
140.48=
41.622
187.652
121.95
727.574
32.284
PRoC3S
3.55
2.68
10.15
58.66
172.30
38.85
236.75
130.72
751.57
15.75
Table Average wall-clock time taken return a plan (i.e., perform LMP generation and sampling to find non vio-vioag solution, including any feedback iterations). Standard deviation shown after +. Note that these results do not enable direct comparison of methods (i.e., lower numbers are not necessarily better) because they do not account for success rate from Table 1.
Drawing
Arrange Blocks
Arrange YCB
Star
Arrow
Letters
Enclosed
Pyramid
Line
Packing
Unstack
Packing
Stacking
3,101
2.104
2.601
4.00
1.901
4.00
2.44 1.71
0,70 0.46
0.00
0.00
LLM
1.97
1.64
1.80
0.89
2.21
1.49
0.00
0.00
-
0.102 0.30
0.104
0.001 0.00
0.40 0.49
1.102 1.81
3.704 1.79
1.564 1.42
0.701 0.78
0.00 0.00
0.00
LLM*-Gaussia
0.30
0.00
CaP
CaP-Gaussian
PRoC3S-NE
0.00
0.00
0.101
0.50
1.501
0.204
0.224
2.101
2.20
0.30
PRoC3S
0.00
0.00
0.30
1.50
2.11
0.60
0.42
1.81
2.04
0.64
Table 6: Average number of times feodback queried. Standard deviation indicated after +. Approaches that do not leverage feedback are listed with -. Note that these results do not enable direct comparison of methods (i.c., lower numbers are not necessarily better) because they do not account for success rate from Table 1.
A.4 Arrange-Blocks
For the pyramid-stackes task and the line-forming task, we randomize the position of two bowls and six blocks such that no objects are stacked on top of each other.
For the region-packing task, we randomize the position of five red blocks and create low square prism centered the middle of the table to represent the region for the red blocks to be packed. The red blocks arranged that blocks are stacked top of each other.
For the task of placing a green block into green bowl, we randomize the position green bowl and eight blocks with at least one green block. Blocks may stacked on top of each other.
A.4.1 Skills
(xck(x, y, z)
Move the gripper to location (x, y, 2) and close the gripper. The pick skill moves the robot's gripper to a specified location (osee), closes the grip- per to grasp an object, and then lifts the object back to a hover position that is simply (x, y, z + 0.1) (where the table height is 0.0). The orientation of the gripper is fixed to be top-down throughout.
place (x, y. 2)
Move the gripper to location (x, y, 2) and open the gripper.
15

---

Drawing
Arrange
Blocks Packing
Arrange YCB
Star
Arrow
Letters
Enclosed
Pyramid
Line
Unstack
Packing
Stacking
29.11=
14.62+
20.48+
28.28+
16.04
45.25+
24.41
4.33+
0.00+
0.00=
LLM
15.78
7.44
11.42
5.69
12.95
14.82
11.31
0.84
0.00
0.00
LLM
7,45#
5.21
5.47#
5.18
5.07=
8.384
6.484
2,234
0.004
6.632
2.48
1.14
1.16
1.29
0.86
1.26
0.85
0.26
0.00
0.94
LLMM-Gaussin
7.82+
6.74+
5.55
7.69
10.70=
66.81
19.34
6.26+
0.00+
0.00=
5.05
4.60
0.90
3..17
7.65
50.45
10.32
6.19
0.00
0.00
15.01
10.36:
16.821
17.024
15.77
16.57
16.174
12,341
15.88.
15.57
CaP
1.66
1.43
2.16
2.39
2.64
4.66
4.07
14.50
3.09
2.67
15.69-
10.02:
17.08+
16.89+
13.014
19.45:t
14.30=
12.65=
16.12::
18.93:
CaP-Gaussian
3.76
2.96
3.68
3,92
2.48
14.32
3.30
14.38
2.54
2.27
19:332
20.832
23.424
17.001
35.681
18.734
20.10:
13.382
22.824
19.16::
PRoC3S-NF
2.44
3.82
2.71
33.70
4.27
3.32
1.81
3.18
20.49=
20.33
29.16:
27.23+
60.79+
39.641+
22.14+
22.82
79.31
24.23+
PRoC3S
3.55
2.68
6.64
26.62
54.54
38.97
7.28
83.34
56.18
12.78
Table Average wall-clock time spent querying the language model in seconds. Standard deviation is included after +. Note that these results do not enable direct comparison of methods (i.c., lower numbers are not necessarily better) because they do not account for success rate from Table 1.
The place skill moves the robot's gripper to specified location (place .pose) to place the object and releases the object. It then moves the gripper back to hover position that is simply (x, y, z + 0.1) (where the table height 0.0). The orientation of the gripper is fixed to be top-down throughout.
A.5 Arrange-YCB
For the region- gegion-packkig task, we randomize the positions of three objects: a banana, strawberry, and a meat can. We also create low square prism centered on the table to represent the packing region for these objects.
For the stacking task, randomize the positions of six objects: a banana, power drill, meat can, strawberry, apple, and pear.
A.5.1 Skills
pock(o, g)
Pick up object at grasp g sampled from a grasp sampler. The pick skill moves the robot's gripper target object's position (gripper (gripper_target (tareet) using predefined poses (hover -pose and gripper gripper.targety .arget). The gripper closes around the object and lifts the object back to the hover position.
(lace 8. p)
holding an object at grasp g, place the object at pose p.
The place skill moves the robot's gripper to target location (gripper target) using hover position. then releases the object and the robot moves back the hover position.
B PRoC3S Prompting Details
Here we provide details on the prompting scheme used for each environment. As outlined in Section 4, the initial prompt to the LLM consists of (a) the classes and objects used to represent the state- space S, (b) the initial state Sp, (c) the available parameterized skills $ and (d) the provided samplers E, and (e) an example of an expected output from a different task. We now provide the prompts we use for each of our environments and task. We start by providing a common prompt 'template' that's shared by all tasks in environment. We then further specify elements that differ between tasks. Since there exists domain example for each domain/method combination, point readers to public code release for full example prompts.
16

---

The prompting template for each environment is structured as follows:
(rtsstemmprompt)))
(ffdmman__setup_ccdee))
(((skill_preffac)))
((fdomain_skills))
{(fmethoo_rooe))}
(<domain_example))
The system prompt, skill preface, and role are identical for all three environments and estab- lish context for the robot,
system.prompt
#define system
are robot operating an environment
with the following state
Skill.preface.
You have access to the following set of skills expressed pddl predicates followed + by descriptions.
have no other skills you can use, you must exactly follow the number of imputs described below.
The coordinate axes are y, where x is distance from the robot base, is left/right from the robot base, and is the height off the table.
We now provide prompts which are unique to each environment: domain setup code, domain.skills, example input task details, and example LMP output.
B.1 Drawing
drawing setup code
COLORS ["blue", "green", "pink", "purple"]
@dataclass
class Obstacle:
name: str
x_pos: float
y-pos: float
radius: float
color: str
@dataclass class DrawnLine:
pl_x: float
pl_y: float
p2_x: float
p2_y: float
@dataclass
class DrawingState:
obstacles:List [Obstacle] fiela((dfaalltfffctorrr drawn_ lime:::nns:sstt [DrawnLiee) fiell(ddefallt_factrryliidd
@dataclass class iontinouuSS:pe::: min: float =
17

---

max: float =
def sample
return random. mniformssll....mmmmm min, self max)
edataclass
class DiscreteSampler:
values: List [int]
def sample (self): return
values)
@dataclass
class Action:
name: str
params: List
drawing.skills
Action("draw_line" [pi_x, pl-y, p2_x, p2-yl)
Draws straight line from (pl_x, pl_y) to (p2_x, p2.y).
The pen is lifted up to get the the action.
B.2 Arrange
arrange setup code
CATEGORIES ["bow1", "block"]
TABLE_BOUNDE BOUNDS [[-0.3, 0.3], [-0.8, -0.2], [o, 0]] X
TABLE_ CENTER = [0, -0.5, 0]
BLOCK_SIZE 0.04
@dataclass class ArrangePose:
x: float = 0
y: float = 0
float 0
roll: float
pitch: float 0
yaw: float = 0
@property def (silf))
eproperty def euler culor
@dataclass
class ArrangeObject:
category: str
color: str
pose: ArrangePose fialddddefaull.ffacoorrraabbba ArrangePose )))
body: Optional [int] None
@dataclass
class ArrangeBelief:
objects: Dict [str, ArrangeObject] = fielddddefalll
observations: List[Any] field (defaultt factoooll__ctt
@dataclass class SontssuesSm:p:::: min: float =
18

---

max: float =
def sample return random. uniform(self min, self max)
@dataclass class Action:
name: str
params: List [float]
arrange.skills
Action("pick", [x, y, z])
Move to the gripper to location x, y, 2 and close the gripper
Action aetion("place,, [x, y, 2])
Move the gripper location x, y, and open the gripper
B.3 Arrange YCB
srrange_ycb.setupp
CATEGORIES ["bowl", "block"]
TABLE_BOUNDS BOUNDS [[-0.3, 0.3), [-0.8, -0.2], to, 0]]
TABLE, CENTER [0, -0.5, 0]
WEBLER
BLOCK_SIZE 0.04
odataclass
class ArrangePose:
x: float = 0
float = 0
float = 0
roll: float
pitch: float 0
yau: float = 0
eproperty def point(self))
@property def euler (sel)):
@dataclass
class ArrangeObject:
category: str
color: str
pose: ArrangePose fielddddefallllfaactoraaaaaad ArrangePose ())
body: Optional [int] = None
@dataclass
class ArrangeBelief:
objects: Dict [str, ArrangeObject] = fielddddefaulltffctoorrr
observations: List[Any] = field (tt.ffaallffcttoraaaaalaall
@dataclass
class ContinuossSampler:
min: float = 0 float 1
def sample (self): return random, uniform(self min, self max)
19

---

@dataclass class DiscreteSampler:
values: List [int]
def campl(see1)). return caadom.choice(seff.... values)
odataclass
class Action:
name: str
params: List [float]
@dataclass class GraepSanpler(Samplee))
def ssmple -> ArrangeGrasp:
verange.ycb.skill
Action("piche [o, gl)
Pick up object o graap g sampled from graap sampler. Graapa MUST from + graap samplers.
Actiom("place", [o, 8, p])
If holding object at grasp g, place the object at pose p.
B.4 Method Prompts
We now go through all of the method specific prompts, which involve a role specification and method-specific example.
proc3s role:
Your goal is generate two things:
First, generate python function named 'gen_plan" that can take any discrete continuous inputs. No list inputs allowed. and return the entire plan with all stepa included where the parameters the plan + depend on the inputs.
Second, generate a python function "geniddmmin" that returns a set of bounds for the continuous discrete input parameters. The number of bounds in the generated domain should exactly match the number inputs to the function excluding the state input
The function you give should always achieve the goal regardless of what parameters from the domain passed as input. The ggn_plann function therefore defines a family solutions to problem.
Explain why the function will always satisfy the goal regardless of the input parameters.
Make sure your function inputs allow for much variability in output plan + possible while still achieving the goal.
Your function should be as general as possible such that any correct answer
corresponda to some input parametera to the function.
All of these parameter samples may fail, in which what constraints caused the failure.
will return feedback about
In the event of constraint satisfaction fail, explain what went wrong and then return an updated gen_plan and gen_ gen_domain that fixes the issue.
This may involve adding actions the beginning the plan move obstructing
objects leading collisions adding new continuous input parameters that are used for thoge new actions.
20

---

Do not add complex logic or too much extra code to fix issues due to constraint violations.
The main function should be named EXACTLY gen_plan" and the domain of the main function should be named EXACTLY gen_domain". Do not change the names. Do not create any additional classes overwrite any existing ones.
Aside from the inital state all inputs the 'gen_plan' function MUST NOT be + type List or Dict. List Dict inputs to gen_plan" are not allowed.
cap.role:
Your goal is to generate a python function that returns plan that performs the provided task. This function can helper functions that must be defined within the scope of the function itself.
The main function should be named EXACTLY gen_plan", and it should take in only one parameter corresponding to the environment state as input. Do not change the names. Do not create any additional classes or overwrite any existing ones. You only allowed to create helper functions inside the 'gen_plan' function.
11m3 1lmeerole:
Your goal is to generate a sequence actions that are stored in variable named 4 "gen_plan"
The checker may fail, in which it will return feedback about what constraints caused the failure.
In the event failure, propose modified plan that avoids potential reasons for failure.
DO NOT placeholders, equations, mathematical operations. Always give ground plan that could be directly executed in the environment.
You must always return block of python code that assigns a list of actions to a variable named EXACTLY "gen_plan"
B.5 Gaussian Sampling
Gaussian sampling is applied to the parameter space the skills outputted by the action, which means can be used for any method. The sampling uses a similar feedback loop to constraint satisfaction and runs for the same number of samples as constraint satisfaction (domain dependent). and is increased linearly to until either
The initial Gaussian standard deviation is solution is found with no constraint violations or the maximum number of samples is reached.
C Sampler Details
We implement 3 different samplers (E) in python:
@dataclass
class ContimuoueSampler (Sampler)
min: float 0
max: float = 1
def sample (self): return random. uniform(self min, self max)
ddataclass class DiscreteSampler: values: List [int] def sample(self).
return random, cwoice values)
@dataclass
21

---

3-Line
5-Line
Sort
YCB-Packing
Detections
Reconstructions
Figure S: Examples from the perception system used part of implementing real-world domains. See Appendix for details.
class (aampae))) def sample(self) -> ArrangeGrasp:
return ArrangePose(
x=random umiform(-0.00) 02, 0.02),
y=random cerrandom.uiifortte unifor=(-00000 0.02), pitch-np.ph htt.//ddmmmiiffommmmommmm pi,
pi),
) ). www.mipplyyrrrrrt..m (uutttpplyykaraaggeeeee000000000
The GraspSampler is only provided in the Arrange- YCB environment.
D Real-world Robot Implementation Details
Similar to TAMP approaches [36, 37], deploy PRoC3S on real-world hardware by: (1) leveraging pre-trained vision models estimate the features of some known set of objects in real-scene, (2) instantiating a digital-twin simulation in which we perform planning, (3) executing the plan open- loop on the real-world robot
Our implementation assumes access to (I) a set of object models compatible with a physics sim- ulator, and (2) a of strings describing each object model (e.g. "apple"). Each of these models and its associated string corresponds object detection and segmentation (specifically, Ren al. [38]) segment and associate portions of point-cloud with each string. For each detection, extract the centroid of the segment, transform this into the robot's coordinate system, and use these as the position (i.e., 2) of the correspond- ing object model within our physics simulator. While full pose-estimation is possible within this setup, we leave this for future work. Additionally, we extract object colors from the masked point- cloud returned by object detection. This is done by taking the average RGB values of points in the pointeloud, and retrieving the closest color (as determined by L2 distance) to predefined color set. Each object is thus equipped with feature called "color' with string value referring to the name of its color. While assuming object models can be strong assumption, model-free versions of this setup have been demonstrated to work on TAMP systems [37], and could similarly be adopted this framework.
object class. Given these, leverage open-vocabulary
'If' the real-word weal-world execution fails, then the system can be simply made 'replan" restarting from step (1)
22


Reviews:
[{"review_id": "vUVCzl0HoX", "reviewer": "Reviewer_7svD", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes to satisfy environment constraints in TAMP by evaluating a task list against a set of constraint functions, and replanning based on language descriptions of how the constraint was violated.", "summary_of_recommendation": "The method is sound, and assumptions/limitations are outlined clearly. However, the assumptions are sufficiently restrictive such that it is not obvious to adapt such a method to hardware. Given such, I don't believe the proposed method would be very impactful in the robot learning community.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 2, "robotics_focus": 2, "quality_of_limitations": 3, "questions_for_rebuttal": "1. Ln 189-191: Can the authors please clarify what they mean by \"the generated LMP reduces the sampling space to two-parameter space\"? It is not clear whether the parameters are referring to dx, dy. As mentioned in W1 above, can the authors please clarify why this is related to the code generation capabilities of LLMs?\n2. While this paper mainly address constraints with respect to collisions, how would this method handle human preferences? For example, if the task description has multiple possible final goal configurations, or *no* possible goal configuration, both of which may require access to human preferences, how would this framework address the issue?\n3. Some of the \"constraints\" illustrated in Figure 1, such as \"Grasp Failure\" seem to be more attributed to test-time perturbations or error. In other words, while the robot may execute a suboptimal grasp some percentage of times and \"violate\" a constraint, this does not necessarily warrant a change in the plan. Can the authors please clarify the distinction on this point? \n4. Can the authors please clarify the definition of \"continuous\" parameters and why this is a traditionally hard problem? It seems as if reasoning and satisfying continuous constraints is a main contribution of the paper, but the inherent challenges in predicting/reasoning in such continuous spaces is a bit vague. What are examples of inputs/outputs that are not continuous? \n5. Can the authors provide more description as to the simulated world model in which they perform test rollouts? How is it different from the actual simulator in which evaluations is done?\n6. What were the prominent failure modes of the proposed algorithm in the experiments that were run?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JxkkAtKdYb", "reviewer": "Reviewer_tB3j", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes an LLM based planning approach to solve long horizon manipulation tasks. The planning part of the approach is somewhat similar to TAMP approaches, however, with certain differences. Overall, in this proposed approach, an LLM is used to generate a function that accepts some open parameters and outputs grounded skills. Together with this function, the approach outputs another function  from which continuous parameters can be sampled. Once both of these functions are returned by the LLM, a simple sample-and-test procedure is applied to check if the returned list of skills together with sampled parmaeters can reach goal states without violating any constraints. If not, some information (such as description of most common violations etc) are  returned to the LLM which is asked to provide a new plan for the task.  Experiments  are performed on 3 different table-top environment settings.", "summary_of_recommendation": "The paper is well written and aims to provide an interesting new approach. However, experimental details are very limited and its hard to say how well does the approach work compared to tranditional tamp approaches.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 2, "robotics_focus": 3, "quality_of_limitations": 3, "questions_for_rebuttal": "see above", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cK7CKhKo5p", "reviewer": "Reviewer_XQei", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This work tackles the problem of constraint satisfaction in the context of LLM planning by integrating in sampling and constraint testing into the LLM planning framework. The results demonstrates its capability on a set of simulated manipulation tasks.", "summary_of_recommendation": "Overall, I don't believe this work meets the bar for acceptance at CoRL. While it describes and aims to solve a concrete problem with LLM planning, it is not likely to operate on a robot in this form due to its significant assumptions and there is significant experimentation and analysis missing.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 2, "robotics_focus": 2, "quality_of_limitations": 3, "questions_for_rebuttal": "1. How could a method like this be deployed on robots? Currently it seems that the method requires significant knowledge of the environment or makes assumptions that are generally impractical in the real world including skills that execute perfectly, knowledge of perfect object poses, knowledge of exact constraint violations and access to a physics simulator to actually test if constraints are met or not.\n2. Lack of comparison to methods that combine LLMs and task planning, for example https://arxiv.org/pdf/2304.11477\n3. No ablations / analysis beyond the main results - Can you probe the method to understand why it works? What is it sensitive to? For example how sensitive is it to the number of samples taken?  \n4. Appendix lacking descriptions of the skills - how are they implemented, what is their individual success rate in simulation, what sort of information do they require to operate? \n5. Suggestion: For table 1, perhaps move your method to the bottom row and also include an average column that shows the average performance improvement\n6. What are the failure cases for this method, when it doesn't work is it due to incorrect plans, incorrect continuous parameters or something else? \n7. Lack of comparisons to VLM methods which could in principle take into account at least some of the constraints by simply processing the image of the scene.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Update: The authors provided sufficiently detailed responses in the rebuttals that the reviewers found satisfactory. The idea of using a simulator to act as a "testing ground" for LLMs to revise its outputs before deploying on a real robot is worth sharing more broadly with the CoRL community. I am recommending to accept this submission.

Original: The reviewers agree that the paper is well written and tackles an important problem. However, they are concerned about the strict assumptions the proposed method makes as well as a lack of convincing baselines. I ask the authors to carefully address all reviewers' feedback during the rebuttal period.
Decision: Accept