Trust the PRoC3S: Solving Long-Horizon Robotics Problems with LLMs and Constraint Satisfaction

Authors: aidan  curtis
nishanth  kumar
jing  cao
tomás  lozano-pérez
leslie  pack
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-05 21:40:38

Abstract: Recent developments in pretrained large language models (LLMs) applied to robotics have demonstrated their capacity for sequencing a set of discrete skills to achieve open-ended goals in simple robotic tasks. In this paper, we examine the topic of LLM planning for a set of *continuously parameterized* skills whose execution must avoid violations of a set of kinematic, geometric, and physical constraints. We prompt the LLM to output code for a function with open parameters, which, together with environmental constraints, can be viewed as a Continuous Constraint Satisfaction Problem (CCSP). This CCSP can be solved through sampling or optimization to find a skill sequence and continuous parameter settings that achieve the goal while avoiding constraint violations. Additionally, we consider cases where the LLM proposes unsatisfiable CCSPs, such as those that are kinematically infeasible, dynamically unstable, or lead to collisions, and re-prompt the LLM to form a new CCSP accordingly. Experiments across simulated and real-world domains demonstrate that our proposed strategy, \OursNoSpace, is capable of solving a wide range of complex manipulation tasks with realistic constraints much more efficiently and effectively than existing baselines.

Main paper: 
 introduction: 
 Recent progress on large-scale foundation models, particularly large language models (LLMs) and vision-language models (VLMs), has enabled a variety of flexible and general-purpose decisionmaking systems for robotic tasks [1, 2, 3, 4, 5, 6, 7] . These systems leverage few-shot prompting as well as the commonsense and sequence prediction abilities of LLMs and VLMs to output sequences of robotic skills that achieve a wide variety of goals. Such systems are generally more capable at handling open-world environments than classical systems, like task and motion planners (TAMP) [8] , since they often do not require hand-specified symbolic components (e.g. predicates and operators), and can perform tasks specified directly in natural language or images.While foundation models have been applied to a range of robotic tasks, these tasks share many common simplifying and limiting assumptions. In most cases, the system is provided with a fixed set of discrete skills such as robot.move to door() or robot.pick can() and asked to perform tasks that simply require composing these skills in a particular order. Discrete skills with minimal to no control over the skill outcome may be sufficient for simple tasks and settings, but are insufficient for domains with complex constraints or goals that depend on continuous properties or relationships, which are common in robotics. For instance, consider the goal shown in the Arrange-YCB domain from Figure 4 in which the robot is tasked with packing a set of YCB objects into a small region. Collision constraints on placing locations restrict the space of possible picking grasps on each object. A single monolithic robot.pick can() skill is not sufficient in this context, regardless of that skill's success rate in isolation. Additionally, goals that specify continuous properties or relationships between objects require actions with continuous parameters. For example, a goal such as "place the object in the top right corner" or "draw a star" displayed in Figure 4 require the planning system to have fine-grained access to specific skill outcomes to avoid the kinds of failures displayed in Figure 1 . Such access requires parameterized skills [9, 10] .We seek to address these limitations and enable planning systems based on foundation model to address complex robotic tasks with realistic constraints using parameterized skills. Towards this goal, we take inspiration from TAMP, and separate planning into two distinct phases [11, 12, 8] .In the first phase, we ask an LLM to generate a program that takes in some continuous parameters as input and produces a sequence of skills with all their parameters specified. We also prompt the LLM to associate a sampling function with each continuous parameter. The result of this phase is thus a kind of continuous constraint satisfaction problem (CCSP) [13, 14, 8, 15] : the solver must now find values for the continuous parameters of each skill such that the proposed sequence of skills can be executed to achieve the goal without violating any constraints in the world (Figure 1 ). The second phase attempts to solve this CCSP via a simple generate-and-test procedure. If it is unable to find a setting for all the continuous parameters that leads to goal achievement, it reports the failure modes encountered and asks the first stage for a new, different sequence that resolves this issue. This planning process, similar to TAMP, is carried out entirely within a simulated world model, which can be constructed from visual inputs using pretrained perception models [16] . Once a viable plan is found, we execute it in the real environment and replan if necessary.We evaluate our approach, Planning for Robots via Code for Continuous Constraint Satisfaction (PRoC3S), on a range of challenging robotic tasks in three different simulated domains, and one real-world domain. In particular, we measure the agent's success rate at completing the tasks involving drawing, rearranging, stacking, and packing objects into configurations specified by a natural language goal. In contrast to classical planning systems, our approach demonstrates the ability to satisfy a diverse set of natural language goals without symbolic predicates or operators. Our system also exhibits greater robustness to real-world constraints than existing methods that apply foundation models to robotics problems. 
related work: 
 The traditional approach to solving long-horizon robotics problems with complex constraints is task and motion planning (TAMP), which combines both a higher-level logical planner and a set of low level parameterized skills [8, 11, 16, 12, 17] . While this is a powerful framework that enables zero-shot generalization to new problems, the set of capabilities of the system are limited to goals that can be expressed using some set of pre-specified predicates, and via a sequence of pre-specified symbolic operators. Some recent work has used LLMs to guide search and translate natural language goals into logical ones, but still require manual specification of operator preconditions and effects or goal predicate classifiers [18, 19, 20] . By contrast, our approach is able to accomplish tasks where specifying symbolic predicates and operators is either challenging or impossible (e.g. the tasks from our 'Drawing' domain in Section 5). Recent advances in LLMs have enabled LLM-based planning systems wherein the available skills are described in natural language and their effects on the world need not be explicitly defined [21] .One of the first such approaches sequences discrete object-specific skills to satisfy rearrangement goals [4] . Many follow-up papers have extended this framework to handle longer-horizon tasks with temporal dependencies [7, 22, 2] . Others have made these action-selection strategies more reactive by reprompting with feedback from the environment or running optimization over skill sequences [23, 6, 24, 25] . Although more flexible than TAMP, these approaches can only make use of discrete skills with no continuous input parameters, which greatly restricts the class of problems they can solve. Some recent work has sought to remedy this by having the LLM generate code that transforms environment parameters into action input parameters via a code interpreter [1] . A downside of this method is that it does not properly handle kinematic, collision, or dynamic constraints of the robot's embodiment. Other approaches have the LLM directly output continuous action parameters and use environmental feedback to adjust those parameters to satisfy encountered constraints [3, 24] . These methods directly rely on an LLM or VLM to resolve constraints, which assumes these model are capable of complex geometric and physical reasoning they are generally not trained for.Some existing work has taken environmental constraints into account when executing skills by building more context-aware skill primitives [5, 26] or used solvers to satisfy LLM-suggested constraints for non-robotic domains [27, 28] . We instead focus on long horizon robotic manipulation problems with continuously parameterized skills and temporally dependent constraints. 
problem setting: 
 We consider a robot planning task with object-oriented states and parameterized skills defined by the tuple ⟨ℓ G , S, s 0 , Φ, f, C⟩. Here, ℓ G is a natural language goal represented by a string corresponding to some unknown goal condition defined over the state space G ⊆ S.S is the robot's state-space. We assume the state is object-oriented and fully-observable: i.e., it is factored into a discrete set of objects, each with a set of attributes that may be discrete, continuous, or a string. We assume each object is an instance of a class in the object-oriented programming sense. Given a finite set of objects, the state space S is defined by the possibly infinite set of assignments to these object's attributes. The initial state s 0 is a collection of instantiated objects. For example, an initial state of our Arrange-Blocks environment with two objects might be:{"o1": Object(cat="block", color="yellow", pose=[0.04, -0.36, 0.02, 0.0, -0.0, -0.0]) "o2": Object(cat="bowl", color="green", pose=[-0.14, -0.35, 0.03, 0.0, -0.0, 0.0])}We also assume that the robot has access to a set of lifted parameterized skills Φ. Lastly, we assume a finite set of user-defined constraints C. Each constraint c i ∈ C has a corresponding natural language description of what violating the constraint entails (e.g. "Pose is not reachable by gripper"), and a classifier c ψ i : S → {true, false} mapping a state s ∈ S to a boolean value indicating whether or not the particular constraint is violated. Constraints may be induced by the kinematics of the robot, collisions with the environment, or dynamic properties like stability of the robot or objects the robot is interacting with, and are common across a wide range of robotic tasks. To check whether a constraint has been violated, we will set our simulator f to a particular state and call the constraint's classifier function 2 . The robot's objective is to find a plan [ϕ 0 , ϕ 1 , ..., ϕ K ] defined by a sequence of ground skills such that: (1) ∃c i ∈ C : c ψ i (s) = true) 3 . 
method: 
 Following previous work [1, 3, 2] , we solve planning tasks by querying an LLM to directly generate a sequence of skills that achieve the goal from the initial state s 0 . However, generating a plan with an LLM is a challenging problem because it involves both correctly sequencing skills together, and also finding a specific setting of all the continuous parameters that enables the plan to achieve the goal. For instance, consider the "Draw a star" task depicted in Figure 4 . Here, the robot is provided with a robot.draw line(x 0 , y 0 , x 1 , y 1 ) skill, that draws a straight line between the points (x 0 , y 0 ) and (x 1 , y 1 ) respectively. To successfully accomplish the task, the robot must invoke this skill at least 5 times in sequence. Moreover, it must specify at least 20 continuous parameters (10 pairs of (x 0 , y 0 ) tuples) such that the shape can be drawn without violating collision or reachability constraints.To address these challenges, we take inspiration from TAMP in two significant ways: (1) we provide the LLM with access to code for a set of samplers [29, 8] , Σ, to help it sample continuous parameters, and (2) we separate planning into a two stage LLM-Modulo framework [30] with each stage designed to solve a different part of the overall planning problem. Samplers are named functions that take in one or more arguments, as well as particular arguments, and output a set of continuous values that may be useful for grounding skill(s). For instance, a simple uniform random sampler (which we call Continuous in the code and examples below) might sample a number uniformly at random within some provided bounds. A grasp sampler might take in no arguments and simply output a valid grasp. Note importantly that these samplers are generally unaware of the constraints: a grasp sampler might output a grasp that is kinematically infeasible or unstable.We leverage these samplers within a two-stage planning process. In the first stage of planning, which we call LMP generation, we prompt an LLM to generate a Language Model Program (LMP) [1] . This LMP is a function that takes in the text representation of the object-oriented state and certain parameters and outputs a plan that we assume achieves ℓ G . We also ask the LLM to generate bounds for and invoke the provided samplers to yield a sampling function that outputs values for the parameters of the LMP. In the next stage, which we call constraint satisfaction, we sample parameter choices for inputs to this LMP and execute them in our simulator to find parameters that ensure the plan does not violate any constraints from C. If the constraint satisfaction phase fails after a fixed sampling budget, we pass information about the most common constraints violated back to the LMP generation phase and request a new LMP and constraint bounds in light of the observed failure. An overview of this process is depicted in Figure 2 .We now discuss each phase in more detail. To ground this discussion, consider a simple running example in the Arrange-Blocks domain shown in Figure 4 . Here, ℓ G is: "Place the green block in the bowl". The state is represented using the Object class mentioned in Section 3, and the initial state is such that an orange block (o12) is atop the green block (o7). The robot is provided with a single simple continuous sampler and two skills, pick(x,y,z) and place(x,y,z), that move the gripper to a particular (x,y,z) location and then close/open the gripper respectively.LMP Generation: The objective of this stage is to generate an LMP that consists of: (1) a plansketch function that takes in a state as well as some arbitrary input parameters and outputs a plan [ϕ 0 , ϕ 1 , ..., ϕ K ] when executed with an interpreter 4 , and (2) a sampling function that leverages samplers with LLM-generated bounds to output parameters that (1) takes as input. Here, (1) together with the user-defined environment constraints C defines a CCSP, and (2) helps define a sampling procedure that can be leveraged to solve this CCSP. To achieve this, we prompt an LLM with (a) the classes and objects used to represent the state-space S, (b) the initial state s 0 , (c) the available parameterized skills Φ, (d) the provided samplers Σ, and (e) an example of an expected output LMP from a different task that shares the same state-space and many of the same skills (see Appendix B for the prompts used in our three environments). Importantly, note that the LLM is not provided with any of the constraints (rather, these will be checked in the next phase).Consider the following generated LMP on our running example task of placing a block into a bowl: Here, the gen plan function is the plan-sketch. It takes in a particular state (named init) corresponding to the initial state s 0 , as well as two parameters and generates a plan in terms of the provided pick and place skills. Importantly, note that the two input parameters to the function (namely dx, dy), are different from the parameters of the pick or place skills. The above generated sequence of one pick and one place skill would ordinarily require six continuous parameters (x, y, z for each skill) for grounding. However, the generated LMP reduces the sampling space to a lower-dimensional, two-parameter space (namely dx, dy). Thus, the generated LMP makes downstream CCSP simpler by leveraging the common-sense and reasoning capabilities of LLMs.Constraint Satisfaction and Feedback: In this work, we opt for a very simple sample-and-test procedure for constraint satisfaction. Specifically, we sample a fixed number of values for all the plan-sketch's input variables (namely dx and dy in the above example) using the sampling functionGaussian Noise Ours that was generated by the previous stage. Given a particular sample, we can simply evaluate the LMP to output a plan. We then check for constraint violations by executing each step of this plan using our simulator f , and running each constraint classifier c ψ i from c i ∈ C. If a plan is found that does not violate any constraints, we return this plan. If no satisfying plan is found after the fixed budget is exhausted, we enter a feedback stage. The objective of this stage is to provide information to the first stage such that it will return a new LMP that avoids the same constraint violations. In our implementation, we return the following information during feedback:(1) descriptions of the top 2 most common constraint violations, (2) the most common ground skill name that was run in the simulator before the violation, and (3) the most common index in the plan at which each of the above constraint violations occurred. This loop between LMP generation, constraint satisfaction and feedback continues until a legal plan is found.In our running example, we exhaust our sampling budget due to collisions with the orange block and return the following feedback information: "Step 0, Action pick, Violation:Collision detected between object o12, gripper.". Given this, as well as the context of its previous LMP, the LLM generates a new LMP that moves object o12 out of the way before object o7 (the green block of interest) is manipulated: ( TABLE_BOUNDS[0][0], TABLE_BOUNDS[0][1]), "y_place_12": Continuous( TABLE_BOUNDS[1][0], TABLE_BOUNDS[1][1]) } 
experiments: 
 Our experiments are designed to test the ability of our method (PRoC3S) to sequence a set of simple continuously parameterized skills to generalize to satisfying unseen natural language goals while obeying environmental constraints, both in simulated domains and on real-world hardware.Constraints. We make use of four general constraint types across environments. These are kinematic constraints on the robot, collision constraints for robot motion, grasp constraints (i.e., checking for stable grasps), and placement constraints (i.e., checking for stable placements). Details on implementation are provided in Appendix A.2.Environments. We now provide high-level environment and task descriptions with details in Appendix A. All of our simulated domains consist of a 6 DoF UR5 robot arm with a Robotiq 2F-85 gripper in front of a object sets, and goals based on the simulated Ravens tabletop environment first introduced by Zeng et al. [31] . Our real-world domain consists of a tabletop setup with a Franka Emika Panda robot.• Drawing: The robot is provided with a parameterized skill to draw a line, and a variety of goals involving drawing different shapes. We attempt four different goals in this environment: "draw a star", "draw an arrow pointing at the biggest obstacle in the environment", "draw the letter M", and "draw a shape that encloses two obstacles". The main challenge of these tasks is drawing a shape that avoids collisions with the obstacles in the cluttered environment. • Arrange-Blocks: A tabletop in front of the robot is strewn with a variety of colored blocks and bowls. We attempt 4 tasks in this environment: "stack an upright pyramid out of three blocks", "Put five blocks in a line flat on the table", "Place all blocks within 0.06 of the center of the table", and "Place the green block in a bowl". The first three are challenging due to low-tolerance stability and collision constraints. The final task is challenging because there are always other blocks atop the green one that prevent it from being picked directly. • Arrange-YCB: The same as the above Arrange-Blocks environment, but with objects from the YCB dataset [32] instead of blocks and bowls. The goals are: "Place all objects within 0.06 of the center of the table", "stack any two objects". Both of these tasks require satisfying grasp constraints on arbitrary object meshes and kinematic/reachability constraints on the selected grasps. In the case of the packing problem, collision constraints between objects are the main constraint violation, whereas placement stability is more significant for the stacking task. • Real Robot Domain: We implement 4 tasks from the 'Arrange-Blocks' and 'Arrange-YCB' domains on a real-world tabletop setup with a Franka Emika Panda robot. The goals are: "Put three/five blocks in a line flat on the table" (referred to as '3-line' and '5-line' respectively), "Place all blue blocks in the blue bowl and red blocks in the red bowl" ('Sort'), and "Place all objects within 0.06 of the center of the table" ('YCB-packing'). See Appendix D for an illustration of our setup, as well as implementation details necessary to apply our approach to this challenging domain. Note that we report results from directly executing computed plans 'open-loop' on the real robot and do not attempt to replan upon failure.Approaches. We now briefly describe the approaches that we compare to PRoC3S.• PRoC3S without feedback (PRoC3S-NF): PRoC3S but with the feedback component ablated. Thus, if the first returned plan doesn't work, we consider the task failed. • Code as Policies (CaP) [1] : This approach attempts to write helper functions and leverage existing Python libraries to output an LMP that produces skill sequences and corresponding continuous parameters give a task, without any environment feedback. • CaP-Gaussian: In an approach loosely based on a previous paper [18] , We add gaussian noise to the action space of the skills output by the LMP generated by CaP, which can help avoid constraints violations but detracts from the intended goal. See Appendix B.5 for details. • LLM 3 [3] :. This approach performs LLM planning with parameterized skills and feedback, but uses the LLM to directly output continuous parameters. We also ablate the feedback component of this approach (LLM 3 -NF), and include a version that places Gaussian noise on the output similar to CaP-Gaussian (LLM 3 -Gaussian).Experimental Setup. For each task and approach, we run 10 random seeds where we randomize the initial locations and sizes (where appropriate) of objects. For our real robot domain, we run 5 random seeds of the approach that had the best performance across simulated domains with randomization of the initial state. For approaches that use feedback (i.e., PRoC3S and LLM 3 ), we limit the number of feedback iterations to 5. We specify three samplers (Σ) and provide all approaches with access to them (details in Appendix C and Appendix B). We fix a sampling budget of 10000 for tasks in the Drawing domain, and 1000 for all other domains. For all approaches, we use the OpenAI GPT-4 LLM [33] , specifically the gpt-4-0125-preview checkpoint. We record success on achieving the natural language task goal as judged by the authors and report additional metrics in Appendix A.1.As can be seen from Table 1 , PRoC3S consistently achieves the highest success rate across simulated domains. PRoC3S-NF's success rate is at least 30% lower on most tasks, illustrating the importance of feedback. This is further validated by the fact that the performance of LLM 3 -NF is significantly worse than LLM 3 . We see that LLM 3 itself performs comparably to PRoC3S on 2 tasks, and find that it falters because the LLM is unable to directly output continuous parameters that satisfy the various task constraints, as can be seen from the example in Figure 5 . CaP is only comparable to our approach on one task and generally fails because it is unable to generate viable continuous parameters (e.g. Figure 5 ). While gaussian noise helps these baselines avoid constriant violations, it prevents them from achieving the overall task goal as shown in Figure 5 . Additionally, Table 2 indicates that our approach is able to operate on challenging real-world variants of several simulated tasks and achieve a non-trivial success rate. Qualitatively, we observe two common failures in simulation.Seed 3-line 5-line Sort YCB-packing 1 ✓ ✓ ✓ ✓ 2 ✓ Grasp fail ✓ ✓ 3 ✓ CSP timeout ✓ Grasp fail 4 ✓ ✓ ✓ Grasp fail 5 ✓ ✓ Self-collision Object collisionFirstly, we observe cases in the 'Arrange Blocks' and 'Arrange YCB' domains where our method cannot correctly refine its LMP given feedback. Once an incorrect modification is made, we find that the LLM rarely recovers in future feedback iterations. Secondly, as observed in prior works [34, 30] , we find that the LLM is unable to yield a valid plan for the longer horizon problems. On the real-world tasks, we observe that most failures stem from the sim-to-real gap. Only one observed failure ('5-line' task seed 3) was due to our method failing to find a plan: all other failures were due to failed grasps or unexpected collisions during real-world execution after planning. 
limitations and future work: 
 There are several limitations of our method. Firstly, our method requires a physics simulator, which introduces a significant sim-to-real gap that can lead to execution failures (as observed in Table 1 ).Secondly, the open parameters in the generated LMPs from the first phase of our method depend heavily on the example task we choose to provide as part of the input prompt, making effective prompting crucial for the success of our approach. Thirdly, our method of solving CCSPs is naive and can be very slow, especially in domains like Arrange-YCB. Additionally, our framework guarantees neither soundness nor completeness nor optimality of the generated plans.In future work, we aim to improve the sampling technique for continuous parameters, opting for a backtracking search or optimization algorithm to select parameter values as part of the sampling phase of our method. As part of this, we hope to explore expressing constraints in terms of continuous cost functions instead of binary failure detectors. We also plan to instill the robot with visual reasoning by exploring the use of vision language models (VLMs), potentially enabling it to recognize a wide range of constraint violations without us having to explicitly define constraint classifier functions. Lastly, we hope to extend our method to partially-observable environments and thereby tackle larger mobile-manipulation tasks. In this section, we describe details on the constraints implemented across environments, as well as the setup for each task in each environment shown in Figure 4 . All experiments in simulated domains were conducted on machines with 8-core CPU's and 32GB of RAM.Tables 3 and 4 show the number of samples and wall-clock time required by our various approaches to perform constraint satisfaction. As can be seen from these results, performing sampling is critical to solving each of our presented tasks. PRoC3S generally requires the smallest number of CSP samples compared to baselines, especially in the first two tasks of the 'Drawing' domain. For tasks where our approach requires more CSP samples (e.g. 'Pyramid' and 'Unstack' in 'Arrange Blocks', and 'Packing' in 'Arrange YCB'), our approach achieves a much higher success rate, indicating the baselines were able to terminate easily because they yielded incorrect plan generators. We observe a similar trend in Table 5 , which shows the overall planning time required by each method. Table 6 lists the number of feedback queries (i.e., number of times the LLM was queried for an LMP after the first time) across the various baselines that use feedback. Compared to LLM 3 , our approach uses significantly fewer feedback iterations on all tasks except 'Unstack', 'Packing', and 'Stacking' in which our approach has significantly higher success rate. This is especially noteworthy in the 'Drawing' domain, where our approach requires less than 1 feedback iteration on average. Finally, Table 7 lists the wall-clock time spent querying the LLM for all approaches and tasks. We see that a significant fraction of the total planning time taken by our approach (over 50% in most tasks) is spent querying the LLM. There are only relatively few tasks (e.g. 'Packing' and 'Unstack' in the 'Arrange Blocks' domain) where sampling takes up the majority of the planning time.• Kinematic Constraints: We make use of the PyBullet [35] inverse kinematic solver to reach desired end-effector positions. If the solver returns joint positions that result in an incorrect end-effector position, we deem the target as kinematically infeasible. • Collision Constraints: If the gripper collides with any unexpected objects during robot motion, we consider this a collision constraint violation. Expected collisions, such as those between the gripper and held object, are not considered collisions. • Grasp Constraints: We consider the following as constraint violations when selecting grasps:(1) the end-effector pose (before the gripper is closed) is in collision with the robot or any other objects in the scene, (2) no part of the object is between the two gripper fingers, (3) the object falls out of the hand when tested in simulation. • Placement Constraints: If the object moves significantly if we step physics simulation for a fixed number of time steps after the gripper opens to release it, we consider this a placement constraint violation.For each of the four drawing tasks, we randomize the position of five circles with random radii.draw line(p1 x, p1 y, p2 x, p2 y) Moves the arm to a point (p1 x, p1 y, 0.1) (where the table surface is at height 0.0), draws a straight line from point (p1 x, p1 y) to point (p2 x, p2 y) while keeping the pen at z pose 0.0, and then moves the arm back up to the point (p1 x, p1 y, 0.1). The orientation of the gripper is fixed to be top-down throughout. EQUATIONFor the pyramid-stacking task and the line-forming task, we randomize the position of two bowls and six blocks such that no objects are stacked on top of each other.For the region-packing task, we randomize the position of five red blocks and create a low square prism centered at the middle of the table to represent the region for the red blocks to be packed. The red blocks are arranged so that no blocks are stacked on top of each other.For the task of placing a green block into a green bowl, we randomize the position of a green bowl and eight blocks with at least one green block. Blocks may be stacked on top of each other.Move the gripper to location (x, y, z) and close the gripper. The pick skill moves the robot's gripper to a specified location (pick pose), closes the gripper to grasp an object, and then lifts the object back to a hover position that is simply (x, y, z + 0.1) (where the table height is 0.0). The orientation of the gripper is fixed to be top-down throughout.Move the gripper to location (x, y, z) and open the gripper. The place skill moves the robot's gripper to a specified location (place pose) to place the object and releases the object. It then moves the gripper back to a hover position that is simply (x, y, z + 0.1) (where the table height is 0.0). The orientation of the gripper is fixed to be top-down throughout.For the region-packing task, we randomize the positions of three objects: a banana, a strawberry, and a meat can. We also create a low square prism centered on the table to represent the packing region for these objects.For the stacking task, we randomize the positions of six objects: a banana, a power drill, a meat can, a strawberry, an apple, and a pear.A.5.1 Skills pick(o, g) Pick up object o at grasp g sampled from a grasp sampler. The pick skill moves the robot's gripper to a target object's position (gripper target) using predefined poses (hover pose and gripper target). The gripper closes around the object and lifts the object back to the hover position.If holding an object o at grasp g, place the object at pose p.The place skill moves the robot's gripper to a target location (gripper target) using a hover position. It then releases the object and the robot moves back to the hover position.Here we provide details on the prompting scheme used for each environment. As outlined in Section 4, the initial prompt to the LLM consists of (a) the classes and objects used to represent the statespace S, (b) the initial state s 0 , (c) the available parameterized skills Φ, and (d) the provided samplers Σ, and (e) an example of an expected output from a different task. We now provide the prompts we use for each of our environments and task. We start by providing a common prompt 'template' that's shared by all tasks in an environment. We then further specify elements that differ between tasks. Since there exists a domain example for each domain/method combination, we point readers to our public code release for full example prompts.The prompting template for each environment is structured as follows: The system prompt, skill preface, and role are identical for all three environments and establish context for the robot.system prompt:#define system You are a robot operating in an environment with the following state skill preface:You have access to the following set of skills expressed as pddl predicates followed by descriptions.You have no other skills you can use, and you must exactly follow the number of inputs described below.The coordinate axes are x, y, z where x is distance from the robot base, y is left/right from the robot base, and z is the height off the table.We now provide prompts which are unique to each environment: domain setup code, domain skills, example input task details, and example LMP output. We now go through all of the method specific prompts, which involve a role specification and a method-specific example.Your goal is to generate two things:First, generate a python function named `gen_plan`that can take any discrete or continuous inputs. No list inputs are allowed. The `gen_plan`function therefore defines a family of solutions to the problem. Explain why the function will always satisfy the goal regardless of the input parameters.→ → Make sure your function inputs allow for as much variability in output plan as possible while still achieving the goal.→ Your function should be as general as possible such that any correct answer corresponds to some input parameters to the function.All of these parameter samples may fail, in which case it will return feedback about what constraints caused the failure.In the event of a constraint satisfaction fail, explain what went wrong and then return an updated gen_plan and gen_domain that fixes the issue.This may involve adding actions to the beginning of the plan to move obstructing objects leading to collisions and adding new continuous input parameters that are used for those new actions. In the event of a failure, propose a modified plan that avoids all potential reasons for failure.→ DO NOT use placeholders, equations, mathematical operations. Always give a ground plan that could be directly executed in the environment.You must always return a block of python code that assigns a list of actions to a variable named EXACTLY `gen_plan` →Gaussian sampling is applied to the parameter space of the skills outputted by the action, which means it can be used for any method. The sampling uses a similar feedback loop to constraint satisfaction and runs for the same number of samples as constraint satisfaction (domain dependent). The initial Gaussian standard deviation is set to zero and is increased linearly to 1 until either a solution is found with no constraint violations or the maximum number of samples is reached.We implement 3 different samplers (Σ) in python: The GraspSampler is only provided in the Arrange-YCB environment.Similar to TAMP approaches [36, 37] , we deploy PRoC3S on real-world hardware by: (1) leveraging pre-trained vision models to estimate the features of some known set of objects in a real-scene, (2) instantiating a digital-twin simulation in which we perform planning, (3) executing the plan openloop on the real-world robot 5 .Our implementation assumes access to (1) a set of object models compatible with a physics simulator, and (2) a set of strings describing each object model (e.g. "apple"). Each of these models and its associated string corresponds to an object class. Given these, we leverage open-vocabulary object detection and segmentation (specifically, Ren et al. [38] ) to segment and associate portions of a point-cloud with each string. For each detection, we extract the centroid of the segment, transform this into the robot's coordinate system, and use these as the position (i.e., x, y, z) of the corresponding object model within our physics simulator. While full pose-estimation is possible within this setup, we leave this for future work. Additionally, we extract object colors from the masked pointcloud returned by object detection. This is done by taking the average RGB values of all points in the pointcloud, and retrieving the closest color (as determined by L2 distance) to a predefined color set. Each object is thus equipped with a feature called 'color' with a string value referring to the name of its color. While assuming object models can be a strong assumption, model-free versions of this setup have been demonstrated to work on TAMP systems [37] , and could similarly be adopted in this framework.Note that our notion of constraint is broader than the typical notion in the CSP literature. Some of our constraints are implicit: they are checked via a simulator rather than expressed as simple symbolic expressions.Overall, our problem setting is analogous to that of TAMP[8] without symbolic predicates or operators.This function represents a family of plans that only differ in one or more continuous parameters.If the real-world execution fails, then the system can be simply made to 'replan' by restarting from step(1) 
 
 Appendix:  


Decision: Update: The authors provided sufficiently detailed responses in the rebuttals that the reviewers found satisfactory. The idea of using a simulator to act as a "testing ground" for LLMs to revise its outputs before deploying on a real robot is worth sharing more broadly with the CoRL community. I am recommending to accept this submission.

Original: The reviewers agree that the paper is well written and tackles an important problem. However, they are concerned about the strict assumptions the proposed method makes as well as a lack of convincing baselines. I ask the authors to carefully address all reviewers' feedback during the rebuttal period.

Reviews: review: title: title: Well written paper, but assumptions seem a bit strong to render this method useful on actual robots.
paper_summary: paper_summary: This paper proposes to satisfy environment constraints in TAMP by evaluating a task list against a set of constraint functions, and replanning based on language descriptions of how the constraint was violated.
main_review: main_review: ## Strengths
1. The paper clearly defines the set of assumptions and input/output contract of the model. Overall, the paper is well written and generally free of typos/grammatical errors. The paper has clear figures that assist understanding the point that the authors are making.
2. The baselines chosen in the experiment are clearly described, and cover a good number of the design decisions of the main algorithm. 
3. The authors clearly highlight their method's differences from prior work. Specifically, incorporating feedback from constraint violations into future replans is an interesting approach to TAMP.

## Weaknesses
1. At a broad level, the paper seems to make a set of rather restrictive assumptions: full observability, access to constraint functions, and access to a model that *describes* the types of failure modes in the model. It is not immediately obvious how especially the latter would be available when applying this method on real robots. 
2. Ln 190-191: I do not follow the claim that the code-generation capabilities of LLMs helps reduce the reasoning to two-parameter space. It seems that this is true only for the running example, but cannot be stated as true in general. For example, inserting an arrow-shaped block into a mold requires both position and rotational deltas.
3. Assuming constraints to be a set of functions that return only binary values seems to be a rather strong assumption. While some constraints may fall under this paradigm, many real-world constraints (especially if dealing with human users) may be more appropriately expressed as a continuous scale for which tradeoffs can be made. 
4. The experiment section would benefit from additional metrics/runs. Currently, only success rate is reported as a metric, but there are components of the proposed algorithm (e.g. resampling budget, average number of times to resample, inference speed) that are left untested. Furthermore, experiments are done in only 3 simulated environments. It would be nice to see extensions to a couple more tasks and/or different robots, which would help emphasize the proposed method's ability to respect kinematic constraints. 
## Low-level technical:
* Ln 188-189: Authors mention "...pick and place skill requires six continuous parameters", but only 3 are specified.
questions: questions: 1. Ln 189-191: Can the authors please clarify what they mean by "the generated LMP reduces the sampling space to two-parameter space"? It is not clear whether the parameters are referring to dx, dy. As mentioned in W1 above, can the authors please clarify why this is related to the code generation capabilities of LLMs?
2. While this paper mainly address constraints with respect to collisions, how would this method handle human preferences? For example, if the task description has multiple possible final goal configurations, or *no* possible goal configuration, both of which may require access to human preferences, how would this framework address the issue?
3. Some of the "constraints" illustrated in Figure 1, such as "Grasp Failure" seem to be more attributed to test-time perturbations or error. In other words, while the robot may execute a suboptimal grasp some percentage of times and "violate" a constraint, this does not necessarily warrant a change in the plan. Can the authors please clarify the distinction on this point? 
4. Can the authors please clarify the definition of "continuous" parameters and why this is a traditionally hard problem? It seems as if reasoning and satisfying continuous constraints is a main contribution of the paper, but the inherent challenges in predicting/reasoning in such continuous spaces is a bit vague. What are examples of inputs/outputs that are not continuous? 
5. Can the authors provide more description as to the simulated world model in which they perform test rollouts? How is it different from the actual simulator in which evaluations is done?
6. What were the prominent failure modes of the proposed algorithm in the experiments that were run?
limitations: limitations: 3
review_summary: review_summary: The method is sound, and assumptions/limitations are outlined clearly. However, the assumptions are sufficiently restrictive such that it is not obvious to adapt such a method to hardware. Given such, I don't believe the proposed method would be very impactful in the robot learning community.
title: title: Review of Trust the PRoC3S
paper_summary: paper_summary: This paper proposes an LLM based planning approach to solve long horizon manipulation tasks. The planning part of the approach is somewhat similar to TAMP approaches, however, with certain differences. Overall, in this proposed approach, an LLM is used to generate a function that accepts some open parameters and outputs grounded skills. Together with this function, the approach outputs another function  from which continuous parameters can be sampled. Once both of these functions are returned by the LLM, a simple sample-and-test procedure is applied to check if the returned list of skills together with sampled parmaeters can reach goal states without violating any constraints. If not, some information (such as description of most common violations etc) are  returned to the LLM which is asked to provide a new plan for the task.  Experiments  are performed on 3 different table-top environment settings.
main_review: main_review: **Pros:**

The paper aims to address a challenging problem (long horizon manipulation) in robot manipulation. It assumes a primitive set of skills and focuses on how to chain them such that the task can be accomplished and all the constraints can be satisfied. Overall, the paper is well written and the approach makes sense. Experiments seem to show that the method works much better than baselines.

In doing the latter access to a simulator is assumed which can observe skill effects as well as constraint violation (using constraint violation functions). 

**Cons and Questions:**

Baselines: It seems that the proposed approach (with its model based reasoning from the simulator) works much better than model-free LLM approaches (such as CaP). Are these differences because the tasks require complex model based reasoning (i.e. the depth of the search tree to find the optimal plan can be large) or is it just that without any checks for constraint satisfaction, CaP based approaches often output plans that end up violating states. 

Optimality:  A nice thing about the TAMP framework, which is clearly a heavy inspiration for the proposed approach is that we can get optimal plans since a discrete planner can optimize the plan cost. But this seems to be much harder to achieve in the current framework. This is because the LLM outputs some plan and then constraint checking happens on this plan, and the output of this will be sent back to the LLM to refine its initial plan. It is unclear to me how long can this feedback loop last. Did the authors perform experiments on this? I would imagine that for some complex geometric tasks the LLM can completely fail beyond a point and it may not have back-tracking etc ability like a true planner does. Basically, with this approach we seem to be inheriting a lot of pitfalls of LLM as planners. Maybe the paper should discuss this in the limitation section. Similarly, the optimality of the final generated plans should be discussed. 

Similarly, using a normal TAMP pipeline seems like a very reasonable approach for the experiments considered here. Is there some particular reason the authors did not add that as a baseline? Interestingly, the no-feedback (PRoC3S-NF) baseline performs quite well on most tasks almost same as the proposed approach, failing on some other tasks. For these other tasks, where this approach fails how many rounds of feedback with the LLM is required. Quantitative analysis of this kind would be very useful to understand how complex the provided domains and problems are.
questions: questions: see above
limitations: limitations: 3
review_summary: review_summary: The paper is well written and aims to provide an interesting new approach. However, experimental details are very limited and its hard to say how well does the approach work compared to tranditional tamp approaches.
title: title: Review of Trust the PRoC3S
paper_summary: paper_summary: This work tackles the problem of constraint satisfaction in the context of LLM planning by integrating in sampling and constraint testing into the LLM planning framework. The results demonstrates its capability on a set of simulated manipulation tasks.
main_review: main_review: PRoC3S is a method for augmenting LLM planning with constraint satisfaction. The authors do so by integrating samplers for continuously parameterized skills into the planning process along with a simulator, enabling the LLM to plan, test and iterate till it solves the task with the given constraints. The results demonstrate clear improvement over baseline methods that do not leverage this information and are unable to take into account constraints. 

Strengths:
* well written paper that clearly communicates the ideas and method
* proposed method improves results over baselines across a range of tasks

Weaknesses:
* Largely unclear how this method can be used for real robots due to its significant assumptions about privileged information
* Missing comparisons to relevant LLM + task planning baselines such as LLM+P (see questions)
*  Analysis is a bit lacking, no experiments done beyond main results (see questions)
* Motivation is a bit shaky: TAMP requires manually specifying predicates and their effects while LLMs generally do not. Yet in this work a significant amount of additional information must be provided to the model, from the descriptions of the skills and examples of their operation to access to a simulator in order to test for constraint satisfaction. It is not entirely clear which is preferable.
questions: questions: 1. How could a method like this be deployed on robots? Currently it seems that the method requires significant knowledge of the environment or makes assumptions that are generally impractical in the real world including skills that execute perfectly, knowledge of perfect object poses, knowledge of exact constraint violations and access to a physics simulator to actually test if constraints are met or not.
2. Lack of comparison to methods that combine LLMs and task planning, for example https://arxiv.org/pdf/2304.11477
3. No ablations / analysis beyond the main results - Can you probe the method to understand why it works? What is it sensitive to? For example how sensitive is it to the number of samples taken?  
4. Appendix lacking descriptions of the skills - how are they implemented, what is their individual success rate in simulation, what sort of information do they require to operate? 
5. Suggestion: For table 1, perhaps move your method to the bottom row and also include an average column that shows the average performance improvement
6. What are the failure cases for this method, when it doesn't work is it due to incorrect plans, incorrect continuous parameters or something else? 
7. Lack of comparisons to VLM methods which could in principle take into account at least some of the constraints by simply processing the image of the scene.
limitations: limitations: 3
review_summary: review_summary: Overall, I don't believe this work meets the bar for acceptance at CoRL. While it describes and aims to solve a concrete problem with LLM planning, it is not likely to operate on a robot in this form due to its significant assumptions and there is significant experimentation and analysis missing.