Plan-Seq-Learn: Language Model Guided RL for Solving Long Horizon Robotics Tasks
Authors: Murtaza Dalal
Tarun Chiruvolu
Devendra Singh Chaplot
Ruslan Salakhutdinov
Venue: ICLR 2024 poster (2024)
Keywords: Long-horizon robot learning
reinforcement learning
LLMs
Primary area: applications to robotics, autonomy, planning
TL;DR: We propose a method that enables Language Model guided RL for long-horizon robotics tasks by appropriately integrating vision-based motion planning.

Abstract: Large Language Models (LLMs) are highly capable of performing planning for long-horizon robotics tasks, yet existing methods require access to a pre-defined skill library (*e.g.* picking, placing, pulling, pushing, navigating). However, LLM planning does not address how to design or learn those behaviors, which remains challenging particularly in long-horizon settings. Furthermore, for many tasks of interest, the robot needs to be able to adjust its behavior in a fine-grained manner, requiring the agent to be capable of modifying *low-level* control actions. Can we instead use the internet-scale knowledge from LLMs for high-level policies, guiding reinforcement learning (RL) policies to efficiently solve robotic control tasks online without requiring a pre-determined set of skills? In this paper, we propose **Plan-Seq-Learn** (PSL): a modular approach that uses motion planning to bridge the gap between abstract language and learned low-level control for solving long-horizon robotics tasks from scratch. We demonstrate that PSL is capable of solving 20+ challenging single and multi-stage robotics tasks on four benchmarks at success rates of over 80% from raw visual input, out-performing language-based, classical, and end-to-end approaches. Video results and code at https://planseqlearn.github.io/

PLAN-SEQ- LANGUAGE MODEL GUIDED RL FOR SOLVING LONG HORIZON ROBOTICS TASKS
Murtaza Dalal', Tarun Chiruvolu', Devendra Singh Chaplot', Ruslan Salakhutdinoy' Carnegie Mellon University, "Mistral Al
ABSTRACT
Large Language Models (LLMs) have been shown to be capable of performing high-level planning for long- aanghorizon robotics tasks, yet existing methods require access pre-defined skill library (e.g. picking, placing, pulling, pushing, navi- gating). However, LLM planning does not address how to design or learn those behaviors, which remains challenging particularly in long-horizon settings. Fur- thermore, for many tasks of interest, the robot needs to be able to adjust its behavior in a fine-grained manner, requiring the agent to be capable of modifying low-level control actions. Can we instead use the internet-scale knowledge from LLMs for high-level policies, guiding reinforcement learning (RL) policies to efficiently solve robotic control tasks online without requiring pre-determined set of skills? In this paper, we propose Plan-Seq-I.card (PSL): modular ap- proach that uses motion planning to bridge the gap between abstract language and learned low-level control for solving long-horizon robotics tasks from scratch. We demonstrate that PSL achieves state-of-the-an results on over 25 challenging robotics tasks with up to 10 stages, PSL solves long-horizon tasks from raw visual input spanning four benchmarks at success rates of over 85%, out-performing language-basedd classical, and end-to-end approaches. Video results and code at https: http///aaaaalllleeeeeeeeee/llela earn gi tuithhb..... io/
INTRODUCTION
In recent years, the field of robot learning has witnessed a significant transformation with the emergence of Large Language Models (LLMs) as mechanism for injecting internet-scall knowledge into robotics. One paradigm that has been particularly effective LLM planning over predefined set of skills (Ahn et al., 2022; Singh et al., 2023; Huang et al., 2022h; Wu et al., 2023), producing strong results across a wide range of robotics tasks. These works assume the availability of a pre-defined skill library that abstracts away the robotic control problem. They instead focus on designing methods to select the right sequence skills to solve a given task. However, for robotics tasks involving contact- rich robotic manipulation (Fig. 1), such skills are often not available, require significant engineering effort to design or train a-priori or are simply not expressive enough to address the task. How can move beyond pre-built skill libraries and enable the application of language models to general purpose robotics tasks with as few assumptions as possible? Robotic systems need to be capable of online improvement over low-level control policies while being able to plan over long horizons.
End-to-end reinforcement learning (RL) is one paradigm that can produce complex low-level control strategies on robots with minimal assumptions (Akkaya et al., 2019; Herzog* al., 2023; Handa et al., 2022; Kalashnikov et al., 2018; 2021: Chen et al., 2022; Agarwal et al., 2023). Unlike hierarchical approaches which impose a specific structure on the agent which may not be applicable to all tasks, end-to-end learning methods can, in principle, learn better representation directly from data. However, RL methods are traditionally limited the short horizon regime due the significant challenge of exploration in RI., especially in high high-dimensional continuous action spaces characteristic of robotics tasks. RI. methods struggle with longer-horizon tasks in which high-level reasoning and low-level control must be learned simultancously; effectively decomposing tasks into sub-sequences and accurately achieving them is challenging in general (Sutton et al., 1999; Parr & Russell, 1997).
Our key insight is that LLMs and RL have complementary strengths and weaknesses. Prior work (Ahn et al., 2022; Huang et al., 2022a; Wu et al., 2023; Singh et al., 2023; Song et al., 2023) has shown that when appropriately prompted, language models are capable of leveraging internet scale knowledge to break down horizon tasks into achievable sub- aob-goals, but lack a mechanism to produce low-level robot control strategies Wang al. (2023), while RL. can discover complex control behaviors

---

Plan: LLM Task Planning
LLM
Silver Peg Place
Gold Grasp -3
Gold Peg Place 4
Grasp
:2
Oresct
Seq: Pose Estimation Motion Planning
Learn: Low-level control via RL
Figure I:
Long horizon task visualization. We visualize PSL solving the NutAssembly task, in which the goal is to put both nuts on their respective pegs. After predicting the high-level plan using an LLM, PSL computes target robot pose, achieves it using motion planning and then learns interaction via RL (third row).
robots but struggles simultaneously perform long-term reasoning (Nachum et al., 2018). However, directly combining the two paradigms, for example, via training language conditioned policy to solve a new task, does not address the exploration problem. The RL agent must now simultaneously learn language semantics and low-level control. Ideally, the RI agent should be able to follow the guidance the LLM, enabling it to learn to efficiently solve each predicted sub-task online. How connect the abstract language space of an LLM with the low-level control space of the RL agent in order to address the long-horizon robot control problem?
In this work, we propose learning method to solve long- hong-horizon robotics tasks by tracking language model plans using motion planning and learned low-level control. Our approach, called Plan-Seq- Learn (PSL), is modular framework in which high-level language plan given by LLM (Plan) is interpreted and executed using motion planning (Seq), enabling the RL policy (L.earn) to rapidly learn short-horizon control strategies solve the overall task. This decomposition enables to effectively leverage the complementary strengths of each module: language models for abstract planning, vision-based motion planning for task plan tracking as well as achieving robot states and RL policies for learning low-level control. Furthermore, we improve learning speed and training stability by sharing the learned RL policy across all stages of the task, using local observations for efficient generalization, and introducing simple, yet scalable curriculum learning strategy for tracking the language model plan. To our knowledge, ours the first work enabling language guided RL agents to efficiently learn low-level control strategies for long-horizon robotics tasks.
Our contributions are: 1) A novel method for long-horizon robot learning that tightly integrates large language models for high-level planning, motion planning for skill sequencing and RL for learning low-level robot control strategies; 2) Strategies for efficient policy learning from high- level plans, which include policy observation space design for locality, shared policy network and reward function structures, and curricula for stage-wise policy training, 3) An extensive experimental evaluation demonstrating that PSL can solve over 25 long-horizon robotics tasks with up 10 stages, outperforming SOTA baselines across four benchmark suites at success rates of over 85 % purely from visual input. PSL produces agents that solve challenging long-horizon tasks such as NutAssembly at 96% success rate.
RELATED WORK
Classical Approaches to Long Horizon Roboties: Historically, robotics tasks have been approached via the Lense-Plan-Act (SPA) pipeline (Paul, 1981; Whitney, 1972; Vukobratović & Potkonjak, 1982; Kappler al., 2018; Murphy, 2019), which requires comprehensive understanding of the environment (sense), a model of the world (plan), and a low-level controller (act). Traditional approaches range from manipulation planning (Lozano-Perez et al., 1984; Taylor et al., 1987), grasp analysis Miller & Allen (2004), and Task and Motion Planning (TAMP) Garrett et al. (2021), 1), to modern variants incorporating learned vision (Mahler et al., 2016; Mousavian et al., 2019; Sundermeyer et al., 2021). Planning algorithms enable long horizon decision making over complex and high-dimensiioal action spaces. However, these approaches can struggle with contact-rich interactions (Mason, 2001; Whitney, 2004), experience cascading errors due imperfect state estimation (Kaelbling & Lozano-Péres
2

---

Planning Module
LLM
High-level
Region 1
Region Condition
Region N Condition NJ
Condition
Sequencing Module
Learning Module
Pose Estimation
Socol
O
T&
Opthobal
Motion Planner
Terminacion Condirios P-g grogg
Sequence component plan
Figure 2: Method overview. PSL decomposes tasks into a list of regions and stage termination conditions using LLM (top), sequences the plan using motion planning (left) and learns control policies using RL (right).
2013), and require significant manual engineering and systems effort setup (Garrett al.. 2020b). Our method leverages learning at each component of the pipeline to sidestep these issues: it handles contact-rich interactions using RI., avoids eascading failures by learning online, and sidesteps manual engineering effort by leveraging pre-trained models for vision and language.
Planning and Reinforcement Learning: Recent work has explored the integration of motion planning and RL to combine the advantages of both paradigms (Lee et al., 2020; Yamada al., 2021; Cheng Xu, 2022; Xia et al., 2020; James & Davison, 2022; James et al., 2022; Liu et al., 2022). GUAPO Lee et al. (2020) is similar to the Seq-Leare components of our method, yet their system considers the single-stage regime and is focused on keeping the RL agent in areas of low pose- estimator uncertainty. Our method instead considers long-horizon tasks by encouraging the RI agent to follow a high-level plan given by an LLM using vision-based motion planning. MoPA-RI. Yamada al. (2021) also bears resemblance via RL, requiring the RL agent to discover the right decomposition of planner vs. control actions on its own Furthermore, roll-outs trajectories using MoPA can result in the RL agent choosing to motion plan multiple times in sequence, which is inefficient one motion planner action is sufficient reach any position in space. our method, we instead explicitly decompose tasks into sequences of contact-free reaching (motion planner) and contact-rich environment interaction (RL).
method, yet it opts to learn when
the motion planner
Language Models for RI and Robotics L.MSs have been applied RI. and robotics in a wide variety of ways, from planning Ahn et al. (2022); Singh et al. (2023); Huang et al. (2022a;b); Wu et al. (2023); Liu et al. (2023a); Rana et al. (2023); Lin et al. (2023), reward definition Kwon al. (2023); Yu et al. (2023), generating quadrupedal contact-points Tang et al. (2023), producing tasks for policy learning Du et al. (2023); Colas et al. (2020) and controlling simulation-ban to produce diverse tasks Ha et al. (2023). Our work instead focuses the online learning setting and aims to leverage language model driven planning to guide RL agents to solve new robotics tasks in a sample efficient manner. BOSS Zhang et al. (2023) is closest to overall method; this concurrent work also leverages I.LM guidance to learn new skills via RI.. Crucially, their method depends on the existence of skill library and learns skills that are combination of high-level actions. Our method instead efficiently learns low-level robot control skills without depending on a pre-defined skill library, by taking advantage of motion planning to track L.LM plan.
trajectory generators
3 PLAN-SEQ-LEAT
In this section, we describe our method for solving horg-horizon robotics tasks, PSL., outlined in Fig. Given a text description of the task, our method breaks up the task into meaningful uub-seqeences (Plan), uses vision and motion planning to translate sub-sequences into initialization regions (Seq) from which we can efficiently train local control policies using RL (Learn).
3

---

3.1 PROBLEM SETUP
We consider Partially Observed Markov Decision Processes (POMDP) of the form (S.A,7 T, R. po: O. P.. S is the set of environment states, A is the set of actions. T s, 5. is the transition probability distribution, R(s, a, s') is the reward function, Po is the distribution over the initial state &o ~ po. O is the set of observations, Po is the distribution over observations conditioned on the state O po (O)ss) and 7 is the discount factor. In our case, the observation space is the set of all RGB-D (RGB and depth) images. The reward function is defined by the environment. The agent's s goal is to maximize the expected sum of rewards over the trajectory, IE (_,,2'R 2RR(s,, ag St+ i)] In our work, we consider POMDPs that describe an embodied robot agent interacting with a scene. We assume that a text description of the task, 9t, provided to the agent in natural language.
3.2 OVERVIEW
To solve long- moo-hhoizonn robotics tasks, we need module capable of bridging the gap between zero-shot language model planning and learned low-level control. Observe that many tasks of in- terest can be decomposed into alternating phases of contact-free motion and contact-rich interac- tion. One first approaches a target region and then performs interaction behavior, prior to mov- ing to the next sub-task. Contact-free motion generation is exactly the motion planning prob- lem. For estimating the position of the target region, we note that state-of-the-ar vision mod- els capable of accurate language-conditioned state estimation (Kirillov et al., 2023; Zhou et al., 2022; Liu et al., 2023b; Bahl et al., 2023; Ye et al., 2023; Labbé et al., 2022). As result, we propose Sequencing Module which uses off- the-shelf vision models to estimate target robot states from the language plan and then achieves these states using motion planner. From such states, we train interaction policies that optimize the task reward using RL. See Alg. 1 and Fig. 2 for an overview of our method.
Algorithm 1 Plan-Seq-LLemm Overview
Require: LLM, Pose Estimator P. task description 9l, Motion Planner MP, low-level horizon H!| Planning Module
High-level plan P + Prompt((LM, 91)
for p E P do
Sequencing Module target region (1), termination condition <- p
Compute pose Starget = P(Oglobat ,t)
Achieve pose MP(Atarget, Ohhobal Learning Module
for i 1,..., Hi do
Get action a ~ To To((Oooat)
Get next state Ofoerd ~ p((s ae).
Store (Olocal ae, Olocal .r) into R
update wy using RI.
if fetage (Ostabal) then
break
end if end for end for
3.3 PLANNING MODULE: ZERO-SHOT HIGH--EVEEL PLANNING
Long-horiee Long-horizon tasks can be broken into series of stages to execute. Rather than discovering these stages using interaction or using task planner Fikes & Nilsson (1971) that may require privileged information about the environment, we use language models to produce natural language plans zero shot without access to the environment. Specifically, given task description gi by a human, prompt an LLM to produce a plan. Designing the plan granularity and scope are crucial; we need plans that can be interpreted by the Sequencing Module, vision-based system that produces and achieves robot poses using motion planning. As result, the LLM predicts a target region (a natural language label of an object/receptacle in the scene, e.g. "silver peg") which can be translated into target pose to achieve at the beginning of each stage of the plan.
When the RL policy is executing a step of the plan, we propose add a stage termination condition (e.g. grasp, place, lurn, open, push) to know the stage is complete and to move onto the next stage. This condition defined as function (Oggge(Ollbball) that takes the current observation of the environment and evaluates a binary success criteria as well as a natural language descriptor of the condition for prompting the L.M (e.g. 'grasp" or "place"). These stage termination conditions are estimated using visual pose estimates. We describe the stage termination conditions in greater detail in Sec. 3.5 and Appendix D. The LLM prompt consists the task description gi, the list of supported stage termination conditions (which we hold constant across all environments) and additional prompting strings for output formatting. We format the language plans as follows: ("Region 1", "Termination Condition 1 1"), ... ("Region N "Termination Condition N"), assuming the LLM predicts N stages. Given the prompt, the LLM outputs a natural language plan in the format listed above. Below, include an example prompt and plan for the Nut Assembly task.
4

---

Stage termination conditions: (grasp, place). Task description: The silver goes the silver peg and the gold nut goes on the gold peg. Give me a simple plan solve the task using only the stage termination conditions. Make sure the plan follows the formatting specified below and make sure to take into account object geometry. Formatting of output: list in which each element looks like: (<object/reion>>, <stage termination condition>)). Don't Ion't output anything else. Plan: [("silver nut", "grasp"), ",, ("silver peg", "place"), " ("gold nut", "grasp"), ). ("gold peg". "place")]
Prompt:
While any language model can be used to perform this planning process, we found that of a variety of publicly available LLMs (via weights or APT), only GPT-4 OpenAl (2023) was capable producing correct plans across all the tasks we consider. We sample from the model with temperature 0 for determinism. We also delete components of the plan that contain LLM hallucinations (if present). We provide additional details in Appendix D and example prompts in Appendix G.
3.4 SEQUENCING MODULE: VISION-BASED PL.N TRACKING
Given a high-level language plan, we now wish to step through the plan and enable a learned RL policy to solve the task, using off-the-shelf vision to produce target poses for motion planning system to achieve. At stage X of the high-level plan, the Sequencing Module takes in the corresponding step high-level plan ("Region Y", "Termination Condition Z!") well as the current global observation of the scene Oalobal (RGB-D view(s) that cover the whole scene), predicts a target robot pose Starget and then reaches the robot pose using motion planning.
Vision and Estimation:
Using a text label of the target region of interest from the high-level plan and observation Oslobal, we need compute target robot state Qtarget for the motion planner to achieve. In principle, we can train RL policy solve this task (lear a policy T. map Ostobal to (torget) given the environment reward function. However, observe that the 3D position of the target region is a reasonable estimate of the optimal policy π. for this task: intuitively, we wish to initialize the robot nearby to the region of interest so it can efficiently learn interaction. Thus, we can bypass learning a policy for this step by leveraging a vision model to estimate the 3D coordinates of the target region. We opt to use Segment Anything Kirillov et al. (2023) to perform segmentation, as it capable of recognizing a wide array of objects, and use calibrated depth images to estimate the coordinates of the target region. We convert the estimated region pose into a target robot pose gtarget for motion planning using inverse kinematics.
Given robot start configuration @o and robot goal configuration Starget of robot, the motion planning module aims to find a trajectory of way-points 7 that form a collision- -free path between qu and Qborget. For manipulation tasks, for example, q represents the joint angles of a robot arm. We can use motion planning to solve this problem directly, such as search- based planning Cohen al. (2010), campiing-bbsed planning Kuffner Jr. & Valle (2000) or trajectory optimization Schulman et al. (2013). In our implementation, we use AIT* Strub & Gammell (2020), a sampling-baut based planner, due to its minimal setup requirements (only collision-checking and favorable performance on planning. For implementation details, please see Appendix D.
Motion Planning:
Overall, the Sequencing Module functions the connective tissue between language and control by moving the robot to regions of interest in the plan, enabling the RL agent to quickly learn short-horizon interaction behaviors to solve the task.
3.5 LEARNING MODULE: EFFICIENTLY LEARNING LOCAL CONTROL
Once the agent steps through the plan and achieves states near target regions of interest, it needs to train an RL policy Ty to learn lewelevel control for solving the task. We train Ty using DRQ-v2 Yarats et al. (2021), a SOTA visual model-free RI. algorithm, to produce low-level control actions (joint control or end-effector control) from images. Furthermore, we propose three modifications to the learning pipeline in order to further improve learning speed and stability.
First, we train a single RL policy across all stages, stepping through the language plan via the Sequencing Module, to optimize the task reward function. The alternative, training a separate policy per stage, would require designing stage specific reward functions per task. Instead, our design enables the agent to solve the task using single reward function by sharing the policy and value functions across stages. This simplifies the training setup and allowing the agent to account for future decisions as well as inaccuracies in the Sequencing Module. For example, if my is initialized at
5

---

sub-optimal position relative the target region, Te can adapt its behavior according to its value function, which is trained to model the full task return E IE, 7 Ri(ss at, $t+1]].
Second, instead of executing Te for fixed number of steps per stage H₁, predict stage termination condition Tstage(Oslobun using the language model and evaluate the condition at every time-step to test if a stage is complete, otherwise it times out after H1 steps. For most conditions, Satage is evaluated by computing the pose estimate of the relevant object and thresholding. This process functions as a form of curriculum learning: only once stage is completed is the agent allowed to progress to the next stage of the plan. As we ablate in Sec. 5, stage termination conditions enable the agent to learn more performant policies by preventing dithering behavior at each stage. As an example, in the nut assembly task shown in Fig. 1, once Te places the silver nut on the silver peg. the placement condition triggers (by compare the pose of the nut to the peg pose) and the Sequencing Module moves the arm to near the gold peg.
Finally, as opposed to training the policy using the global view of the scene (Onlobal), we train using local observations Olocal which can only observe the scene in small region around the robot (e.g. wrist camera views for robotic manipulation). This design choice affords several unique properties that we validate in Appendix C, namely: 1) improved learning efficiency and speed, 2) ease of chaining pre-trained policies. Our policies are capable of leveraging local views because of the decomposition in PSL: the RL policy simply has to learn interaction behaviors in small region, it has no need for global view of the scene, in contrast to an end-to-end RI agent that would need to a global view of the scene to know where go to solve task. For additional details in regarding the structure and training process of the Learning Module, see Appendix D.
4
EXPERIMENTAL SETUP
4.1 TASKS
We conduct experiments single and multi-stage robotics tasks across four simulated environment suites Meta-World, Obstructed Suite, Kitchen and Robosuite) which contain obstructed settings, contact-rich setups, and sparse rewards (Fig. F.1). See Appendix F for additional details.
Meta- Meta-World:
Yu et al. (2020) is an RL benchmark with a rich source of tasks. From Meta- World, select four tasks: MW-Di sa semble (removing nut from peg), MW-Bi ck (picking and placing a cube), MW- MW-Assembl ly y (putting nut on a peg), MW-Hammer (hammering a nail).
ObstructedSuite: Yamada et al. (2021) contains tasks that evaluate our agent's ability to plan, move and interact with the environment in the presence of obstacles. It consists of three tasks: OS-Lift (cube lifting in tall box), OS-Fush (push a block surrounded by walls), and OS-Assembly (avoiding obstacles to place table leg at target).
Kitchen: Gupta et al. (2019); Fu et al. (2020) tests two aspects of our agent: its ability to handle sparse terminal rewards and its long-horizon manipulation capabilities. The single-stage kitchen tasks include K-slide (open slide cabinet), K-Kettle (push kettle forward), K-Burner (turn burner), K- K-Light (flick light switch), and K-Microwar (open mierowave). The multi-stage Kitchen tasks denote the number of stages in the name and include combinations of the aforementioned single tasks.
Robosuite: Zhu et al. (2020) contains wide array of robotic manipulation tasks rang- ing from single stage (RS-Lift, cube lifting, RS-Door: door opening) to multi-stage SSS--uutRoundRRS-uutSSsuaa
RS AS-Nuttssemblly pick-place nut(s) onto target peg(s) and RS-Bread, RS-Cerea 1, RS-Mi 1k, RS-Can, RS-Cerea MMi 1 k, RS-CanBreat pick-place object(s) into appropriate bin(s)). Robosuite emphasizes realism and fidelity to real-world control, enabling us to highlight the potential of our method to be applied to real systems.
4.2 BASELINES
We compare against two types of baselines, methods that learn from data and methods that perform offline planning. We include additional details in Appendix D.
Learning Methods:
E2E::
Yarats et al. (2021) DRQ-v2 is SOTA model-free visual RL. algorithm.

---

Robosuite Lift
Kitchen Microwave
Robosuite Can
Robosuite NutAssembly Round
Success Rate
Success Rate
Success Rate
&
1003
1000
Number Trials
Number of Trials
Number Trials
Nomber Trials
OS-Assembly
Disassemble
Metaworld Assembly
Metaworld Bin Picking
Success Rate
Success Rate
Success Rate
Success
véc wio
Indes
3501 Number of Trials
odon
silo
Indon
1000
s000 1000
-50 560
1900
Number Trials
Number of Trials
Number Trials
Figure 3: Sample Efficiency Results. We plot task success rate as function of the number of trials. PSL improves the sample efficiency of the baselines Obstructed Suite. PSL able to do so because it initializes the RI. policy near the region of interest (as predicted by the Plan and Sequence Modules) and leverages local observations to efficiently learn interaction. Additional learning curves in Appendix C.
each task in Robosuite, Kitchen, Meta-World, and
RAPS:
Dalal et al. (2021) is a hierarchical RL method that modifies the action space of the agent with engineered subroutines (primitives). RAPS greatly accelerates learning speed, but is limited in expressivity due to its action space, unlike PSL.
MoPA-RL:
Yamada et al. (2021) is similar to PSL in its integration of motion planning and RL but differs in that it does not leverage an LLM planner; it uses the RL agent to decide when and where to call the motion planner.
Planning Methods:
TAMP: Garrett et al. (2020a) is classical baseline that uses privileged view of the world to perform joint high-level (task planning) and low-level planning (motion planning with primitives) for solving long-horizon roboties tasks.
SayCan: re-implementation of SayCan Ahn et al. (2022) using publicly available LLMs that performs LLM planning with fixed of pre-defined skills. Following the SayCan paper, we specify skill library consisting of object picking and placing behaviors using pose-estimation, motion- aittionnplaaning and heuristic action primitives. We do not learn the pick skill as done in SayCan because our setup does not contain a separate set of train and evaluation environments. this work, we evaluate the single-task RL regime which the agent is tested with held out poses, not held out environments.
4.3 EXPERIMENT DETAILS
We evaluate all methods aside from TAMP and MoPA-RL (which use privileged simulator infor- mation) using visual input. SayCan and PSL use Oslobal and Olocal For E2E and RAPS, provide the learner access to a single global fixed view observation from Oglobal for simplicity and speed of execution; performance in terms of task success rate with respect to the number of trials. We do so to provide a fair metric for evaluating variety of different control implementations across PSL, RAPS, and E2E. Each method is trained for 10K episodes total. We train on each task using the default environment reward function. For each method, run 7 seeds on every task and average across 10 evaluations.
did not find including Ooocal improves performance (Fig. C.5) We measure
RESULTS
We begin by evaluating PSL variety of single stage tasks across Robosuite, Horl,World, Kitchen and ObstructedSuite. Next, we scale our evaluation to the long-horizon regime in which we show that PSL can leverage LLM task planning to efficiently solve multi-stage tasks. Finally, we perform an analysis of PSL, evaluating its sensitivity to pose estimation error and stage termination conditions.
5.1 LEARNING RESULTS
PSL accelerate learning efficiency on a wide array of single-stage benchmark tasks. For single-stage manipulation, (in which the LLM predicts only single step in the plan), the Sequencing
7

---

RS Bzead
RB Can
RS MIlk
RS Cereal
RS ButRoand RS NutBquaze
E2F
.52 + .99
0.32 +: 44
02.4:
0.0 + 0.0
.06 =: .13
0.02 d: .045
RAP'
0.0 1: 0.0
0.0 0.0
0.0 1 0.0
0.0 1 0.0
0.0 =: 0.0
0.0 1 0.0
TAMP
1.0
4:
1.0 +:
d: 0.3
.35
SayCan
93 = 09
1.0
0.9 05
.63 = 09
.66 = 25
.27 + .11
PSL. 1.0 0.0
1.0 1. 0.0
1.0
.98 :: .04
.97 :: .02
Table 1: Robosuite Two Stage Results. Performance is measured in terms of success rate on two-stage (2 planner actions) tasks. SayCan is competitive with PSL on pick-place style tasks, but SayCan's performance drops considerably (86.5% to 41.5% on average) contact-rich tasks involving assembling nuts due to cascading failures. Online learning methods (E2E and RAPS) make little progress on long- -horionn tasks in Robosuite. On the other hand, PSI. is able to solve each task with at least 97% success rate.
Module motion plans to the specified region, then hands off control to the RL agent to complete the task. In this setting. solely evaluate the learning methods since the planning problem is trivial (only one step). We observe improvements in learning efficiency (with respect to number of trials) as well as final performance in comparison the learning baselines E2E, RAPS and MoPA-RL, across 11 tasks in Robosuite, Meta-World, Kitchen and ObstructedSuite (Fig. 3, left). For all learning curves, please see the Appendix C. PSL especially performs well on sparse reward tasks, such as in Kitchen, for which a key challenge is figuring out which object to manipulate and where it is. Additionally, we observe qualitatively meaningful behavior using PSL.: PSI. learns to use the gripper to grasp and turn the burner knob, unlike E2E or RAPS which end up using other joints to flick the burner.
PSL efficiently solves tasks with obstructions by leveraging motion planning. We now consider three tasks from the Obstructed Suite in order to highlight PSL's effectiveness at learning control in the presence of obstacles. As we observe in Fig. and Fig. C.2, PSL is able to do so efficiently, solving each task within 5K episodes, while E2E fails to make progress. PSL is able to do because the Sequencing Module handles the obstacle avoidance implicitly via motion planning and initializes the RI. policy in advantageous regions near the target object. In contrast, E2E spends significant amount of time attempting to reach the object in spite of the obstacles, failing to learn the task. While Mo is also able to solve many of the tasks, it requires more trials than PSL even though operates over privileged state input, as the agent must simultaneously learn when and where to motion plan as well as how to manipulate the object.
PSL enables visuomotor policies to learn long-horizon behaviors with up to 10 stages. Two-stage results across Robosuite and World are shown in Table and Table C.2, with learning curves in Fig. 3 (right) and Fig. C.3. On the Robosuite tasks, E2E and RAPS fail to make progress: while they learn to reach the object, they fail to consistently grasp it, let alone learn to place in the target location. On the Meta- World tasks, the learning baselines perform well on most tasks, achieving similar performance to PSL due to shaped rewards, simplified low-level control (no orientation changes) and small pose variations. However, PSL is significantly more sample-efficient than E2E and RAPS as shown in Fig. C.3. TAMP and SayCan are able to achieve high performance across each PickPlace variant of the Robosuite tasks (93.75%, 86.5% averaged across tasks), as the manipulation skills do not require significant contact-rich interaction, reducing failure skill failure rates. Cascading failures still occur due to the baselines' open-loop nature of execution, imperfect state estimation (SayCan), planner stochasticity (TAMP). Only PSL is able to achieve perfect performance across each task, avoiding cascading failures by learning from online interaction.
On multi-stage tasks (involving 3-10 stages), we find that TAMP and SayCan performance drops significantly in comparison to PSL (38% and 37% vs. 95% averaged across tasks). For multiple stages, the cascading failure problem becomes all the more problematic, causing all three baselines to fail at intermediate stages, while PSL is able to learn to adapt to imperfect Sequencing Module behavior via RI. See Table 2 for a detailed breakdown of the results.
PSL solves contact-rich, long-horizon control tasks such as NutAssembly. In these experi- ments, we show that PSL can learn to solve contact-rich tasks (RS-Nut RS-Nut.quai RS As-NutAs semblly that pose significant challenges for classical methods and LLMs with pre- trained skills due to the difficulty of designing manipulation behaviors under continuous contact. By learning an interaction policy whose purpose is to produce locally correct contact-rich behavior, we find that PSL is effective performing contact-rich manipulation over long horizons (Table 1, Table 2), outperforming SayCan by a wide margin (97% vs. 35% averaged across tasks). Our decomposition into contact-free motion generation and contact-rich interaction decouples the what
8

---

R3 Melkk_Milk RS CanBread RS NutAssembly
K MS
KS
5
K
MS
MB 10
Stages
4
7
10
E2E
0.0/0.0
0.0/0.0
0.0/0.0
0.0/0.0
0.0/0.0
0.0/0.0
0.0/0.0
RAPS
0.070.0
0.0/0.0
0.070.0
0.000.0
TAMP
7.15
.72/.25
0.2/0.3
1.0/0.0
0.0/0.0
0.0/0.0
0.0/0.0
SayCan
.737.05
.237.21
1.0/0.0
0.070.0
0.070.0
0.070.0
PSL
.85 = 21
0.9 - 0.2
.96 = .08
1.0 - 0.0 1.0 = 0.0 1.0 = 0.0 1.0 = 0.0
Table 2: Multistage (Loong-orizon) results. Performance is measublack in terms of mean task success convergence. PSL is the consistently solves each task, outperforming planning methods by over 70% on challenging contact-intensive tasks such as NutAssembly.
(target nut) and where (peg) from the how (precision grasp and contact-rich place), allowing the RL agent to simply focus on the aspect of the problem that challenging to estimate a-priori: how to interact with the objects in the appropriate manner.
5.2 ANALYSIS
We now turn to analyzing PSL, evaluating its robustness to pose estimates and the importance of our proposed stage termination conditions. We include additional analysis of PSL in Appendix C.
SayCan 1.0
.93
.27 + .12
0.0
0,0 0.0 .75 +
PST. 1.0
0.0
1.0
0.0
0.0
Table 3: Noisy Pose Ablation Results. We add noise sampled from (00 σ) to the pose estimates and evaluate
SayCan and PSL. PSL is able to handle noisy poses by training online with RL, only observing performance degradation beyond
0.1.
PSL leverages stage termination conditions to learn faster. While target object sequence is necessary for PSL to plan to the right location at the right time, in this experiment we evaluate if knowledge of the stage termination conditions is necessary. Specifically, on the RS- Can task, we evaluate the use of stage termination condition checks in PSL to trigger the next step in the plan versus simply using a timeout of 25 steps. We find that it is in fact critical to use stage termination condition checks enable the agent to effectively sequence the plan; use of timeout results in dithering behavior which slows down learning. After 10K episodes we observe a performance improvement of 31% (100% vs. 69%) by including plan stage termination conditions in our pipeline.
PSL produces policies that are robust to noisy pose estimates. In real world settings, there is often noise pose estimation due to noisy depth values, imperfect camera calibration or even network prediction errors. Ideally. the agent should be adapt to such potential failure modes: open-loop planning methods such TAMP and SayCan are not well-suited to do so because they do not improve online. In this experiment we evaluate the PSL's ability to handle noisy/inaccurate poses by leveraging online interaction via RL. On the RS-Can task, we add zero-mean Gaussian noise to the pose, with E 0.01, 0.025, .5 and report our results in Table 3. While SayCan struggles to handle a 0.01, PSL is able to learn with noisy poses at a = .I, at the cost of slower learning performance. Neither method performs well at 0.5, however at that point the poses not near the object and the effect similar to resetting to a random robot pose in the workspace every episode.
CONCLUSIONS
In this work, we propose PSL, a method that integrates the long-horizon reasoning capabilities of language models with the dexterity of learned RL policies via skill sequencing module. At the heart of our method lies the decomposition of robotics tasks into sequential phases of contact-free motion generation (using language model planning) and environment interaction. We solve these phases using motion planning (informed by visual pose-estimation) and model-free RL respectively, an approach which we validate via extensive experimental evaluation. We outperform state-of-the-arr methods for end-to-end RL, hierarchical RL, classical planning and LLM planning on over 20 challenging dision-based control tasks across four benchmark environment suites. In the future, this work could be extended to improving a pre-existing robot skill library over time using RL, enabling an agent to perform planning with an ever increasing repertoire of skills that can be refined at low-level. PSL can also be applied to sim2real transfer, since the policies we train in this work use local observations, they are more amenable to sim2real transfer Agarwal et al. (2023).

---

ACKNOWLEDGEMENTS
We thank Russell Mendonca, Ananye Agarwal, Mihir Prabhudesai and Ben Eyesenbach for their insightful discussions and feedback. We additionally thank Theophile Gervet, Rishi Veerapaneni, Ajay Mandlekar, Gaurav Parmar, Russell Mendonca and Ben Eyesenbach for feedback early drafts of this paper. Aditionally, Jared Mejia, Lili Chen, Ananye Agarwal, Kenny Shaw, Brandon Trabucco, Yue Wu, Jing Yu Koh, and Shagun Uppal provided valuable writing feedback. This work was supported in part by ONR N00014181288 12861, ONR 0000142312366 and DARPA/AFRL FA87502321015. Additionally, MD supported by the NSF Graduate Fellowship.
REFERENCES
Ananye Agarwal, Ashish Kumar, Jitendra Malik, and Deepak Pathak. Legged locomotion in chal- lenging terrains using egocentric vision. In Conference on Robot Learning, pp. 2023.
PMLR,
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al. Do as can, not as say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.0166
2022.
Ilge Akkaya, Marcin Andrychowicz. Macick Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, et al. Solving rubik's 's cube with a robot hand. arXiv preprint 191000910 2019.
Shikhar Bahl, Russell Mendonca, Lili Chen, Unnat Jain, and Deepak Pathak. Affordances from human videos versatile representation for robotics. 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D) Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. Advances in neural information processing systems, 33: 2020.
Tao Chen, Megha Tippur, Siyang Wu, Vikash Kumar, Edward Adelson, and Pulkit Agrawal. Visual dexterity: In-hand dexterous manipulation from depth. arXiv preprint arXiv:2211.1174 2022.
Shuo Cheng and Danfei Xu. Guided skill learning and abstraction for long-horizon manipulation. arXiv preprint arXiv:2210.1263: 2022.
Benjamin Cohen, Sachin Chitta, and Maxim Likhachev. Search-based planning for manipulation with motion primitives. In International Conference on Robotics and Automation, 2010.
Cédric Colas, Tristan Karch, Nicolas Lair, Jean-Michel Dussoux, Clément Moulin-Frice Peter Dominey, and Pierre-Yves Oudeyer. Language cognitive tool imagine goals in curiosity driven exploration. Advances in Neural Information Processing Systems, 33:3761-3774, 2020.
Murtaza Dalal, Deepak Pathak, and Russ R Salakhutdinov. Accelerating robotic reinforcement learning via parameterized action primitives. Advances in Neural Information Processing Systems, 34:21847-21859 2021.
Murtaza Dalal, Ajay Mandlekar, Caelan Garrett, Ankur Handa, Ruslan Salakhutdinov, and Dieter Fox. Imitating task and motion planning with visuomotor transformers. 2023.
Yuqing Du, Olivia Watkins, Zihan Wang, Cédric Colas, Trevor Darrell, Pieter Abbeel, Abhishek Gupta, and Jacob Andreas. Guiding pretraining in reinforcement learning with large language models. arXiv preprint arXiv:2302.0666 06692, 2023.
Richard E Fikes and Nils Nilsson. Strips: A new approach to the application of theorem proving to problem solving. Artificial intelligence, ((3-4):89-2208 1971.
Adam Fishman, Adithyavairan Murali, Clemens Eppner. Bryan Peele, Byron Boots, and Dieter Fox. Motion policy networks. arXiv preprint arXiv:2210. 12209, 2022.
10

---

Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
Caelan Reed Garrett, Tomás Lozano-Pérez, and Leslie Pack Kaelbling. PddIstream: Integrating symbolic planners and blackbox samplers via optimistic adaptive planning. In Proceedings of the International Conference on Automated Planning and Scheduling. volume 30, pp. 440-448, 2020a.
Caelan Reed Garrett, Chris Paxton, Tomás Jerzano-Pére., Leslie Pack Kaelbling, and Dieter Fox. Online replanning belief space for partially observable task and motion problems. 2020 IEEE International Conference Robotics and Automation (ICRA), pp. 5678-5684. IEEE, 2020b.
Caelan Reed Garrett, Rohan Chitnis, Rachel Holladay, Beomjoon Kim, Tom Silver, Leslie Pack Kaelbling, and Tom's as Lozano-P ereez. Integrated Task and Motion Planning. Annual review of control, robotics, and autonomous systems, 4, 2021.
erez.
Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman, Relay policy learning: Solving ong-horizon tasks via imitation and reinforcement learning. arXiv preprint arXiv:1910.1 1956, 2019.
Huy Ha, Pete Florence, and Shuran Song. Scaling up and distilling down: caanguuge-guudhee robot skill acquisition. Proceedings of the 2023 Conference on Robot Learning, 2023.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv: 2019.
Ankur Handa, Arthur Allshire, Viktor Makoviychuk, Aleksei Petrenko, Ritvik Singh, Jingzhou Liu, Denys Makoviichuk, Karl Van Wyk, Alexander Zhurkevich, Balakumar Sundaralingam, et al. Dextreme: Transfer of agile in-hand manipulation from simulation to reality. arXiv preprint arXiv:2210.13702, 2022.
Alexander Herzog*, Kanishka Rao*, Karol Hausman*, Yao Lu Paul Wohlhart Mengyuan Yan, Jessica Lin, Montserrat Gonzalez Arenas, Ted Xiao, Daniel Kappler, Daniel Ho, Jarek Retting- house, Yevgen Chebotar, Kuang-Huei Lee, Keerthana Gopalakrishnan, Ryan Julian, Adrian Li, Chuyuan Kelly Fu, Bob Wei, Sangeetha Ramesh, Khem Holden, Kim Kleiven, David Rendleman, Sean Kirmani, Jeff Bingham, Jon Weisz, Ying Xu, Wenlong Lu, Matthew Bennice, Cody Fong, David Do, Jessica Lam, Noah Brown, Mrinal Kalakrishnan, Julian Ibarz, Peter Pastor, and Sergey Levine. Deep rl at scale: Sorting waste in office buildings with a fleet of mobile manipulators. In 2023.
arXiv preprini arXiv:2305.032
Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models zero-shot planners: Extracting actionable knowledge for embodied agents. International Conference on Machine Learning, pp. 91 11-91447 PMLR, 2022a.
Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint 2tt.:22077.06008, 2022b.
Stephen James and Andrew Davison. Q-attention: Enabling efficient learning for vision-based robotic manipulation. IEEE Robotics and Automation Letters, 7(2):1612-1619 2022.
Stephen James, Kentaro Wada, Tristan Laidlow, and Andrew Davison. Coarse-to-fin fine q-attention: Ef- ficient learning for visual robotic manipulation via discretisation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13739-13748, 2022.
Leslie Pack Kaelbling and Tomás Lozano-Pérez. Integrated task and motion planning in belief space. The International Journal of Robotics Research, 12)9-10): 194-1227.. 2013
2013.
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Scalable deep reinforcement learning for dision-basee robotic manipulation. In Conference on Robot Learning, pp. 651-673. PMLR, 2018.
11

---

Dmitry Kalashnikov, Jacob Varley, Yevgen Chebotar, Benjamin Swanson, Rico Jonschkowski, Chelsea Finn, Sergey Levine, and Karol Hausman. Mt-opt: Continuous multi-tase robotic rein- forcement learning at scale. arXiv preprint arXhv:210..08212, 2021.
Daniel Kappler, Franziska Meier, Jan Issac, Jim Mainprice, Cristina Garcia Cifuentes, Manuel Wüthrich, Vincent Berenz, Stefan Schaal, Nathan Ratliff, and Jeannette Bohg. Real-time perception meets reactive motion generation. IEEE Robotics and Automation Letters, 100:1864--8711 2018.
Oussama Khatib. A unified approach for motion and force control of robot manipulators: The operational space formulation. IEEE Journal on Robotics and Automation, 3(1):43-53, 1987.
Alexander Kirillov. Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan- Yen Lo, et Segment anything. arXiv preprint 1rXiv:2304.026433 2023.
James Kuffner Jr. and Steven M LaValle. RRT-Connect: An efficient approach to single-query path planning. In IEEE International Conference on Robotics and Automation (ICRA), 2000.
Minac Kwon, Sang Michael Xie, Kalesha Bullard, and Dorsa Sadigh. Reward design with language models. arXiv preprint arXiv:2303 00001, 2023.
Yann Labbé, Lucas Manuelli, Arsalan Mousavian, Stephen Tyree, Stan Birchfield, Jonathan Tremblay, Justin Carpentier, Mathieu Aubry, Dieter Fox, and Josef Sivic. Megapose: 6d pose estimation of novel objects via render compare. arXiv preprint arXiv:2212.0663 06670000. 2022.
Michelle A Lee, Carlos Florensa, Jonathan Tremblay, Nathan Ratliff, Animesh Garg, Fabio Ramos, and Dieter Fox. Guided uncertainty-aware policy optimization: Combining learning and model- based strategies for sample-efficient policy learning. In 2020 IEEE International Conference on Robolics and Automation (ICRA), pp. 7505-7512. IEEE, 2020.
Kevin Lin, Christopher Agia, Toki Migimatsu, Marco Pavone, and Jeannette Bohg. Text2motion: From natural language instructions to feasible plans. arXiv preprint arXiv:2303. /2.533 2023.
Bo Liu, Yuqian Jiang, Xiaohan Zhang, Qiang Liu, Shiqi Zhang, Joydeep Biswas, and Peter Stone. Llm+ P: Empowering large language models with optimal planning proficiency. arXiv preprint arXiv:2304. 1477, 2023a.
I-Chun Arthur Liu, Shagun Uppal, Gaurav S Sukhatme, Joseph J Lim, Peter Englert, and Young- woon Lee. Distilling motion planner augmented policies into visual control policies for robot manipulation. In Conference on Robot Learning, pp. 641-650. PMLR, 2022.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang. Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint Xiv::2033.05499 2023b.
Tomas Lozano-Perez, Matthew T Mason, and Russell H Taylor. Automatic synthesis of fine-motion strategies for robots. The International Journal of Robotics Research, 3(1):3-24, 1984.
Jeffrey Mahler, Florian T Pokorny, Brian Hou, Melrose Roderick, Michael Laskey, Mathieu Aubry. Kai Kohlhoff, Torsten Kröger, James Kuffner, and Ken Goldberg. Dex-net 1.0: A cloud-base network of 3d objects for robust grasp planning using multi-armed bandit model with correlated rewards. In IEEE International Conference on Robotics and Automation (ICRA), pp. 1957-1964. IEEE, 2016.
Ajay Mandlekar, Caelan Garret, Danfei Xu, and Dieter Fox. Human-in-the-loop task and motion planning for imitation learning. Conference on Robot Learning, 2023.
Matthew T Mason. Mechanics of robotic manipulation. MIT press, 2001.
Andrew T Miller and Peter K Allen. Graspit! a versatile simulator for robotic grasping. IEEE 102 2004.
Robotics & Automation Magazine,
Arsalan Mousavian, Clemens Eppner, and Dieter Fox. 6-dof graspnet: Variational grasp generation for object manipulation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2901-2910, 2019.
12

---

Robin R Murphy, Introduction to Al robotics. MIT press, 2019.
Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning. Advances in neural information processing systems, 31, 2018.
R OpenAl. Gpt-4 technical report. arXiv, pp. 2303-08774, 2023.
Ronald Parr and Stuart Russell, Reinforcement learning with hierarchies of machines. Advances in neural information processing systems, 10, 1997.
Richard P Paul. Robol manipulators: mathematics, programming, and control: the computer control of robot manipulators. Richard Paul, 1981
Krishan Rana, Jesse Haviland, Sourav Garg, Jad Abou-Chaken Ian Reid, and Niko Suenderhauf. Sayplan: Grounding large language models using 3d scene graphs for scalable task planning. arXiv preprint FrXiv:2307.06135, 2023.
John Schulman, Jonathan Ho, Alex X Lee, Ibrahim Awwal, Henry Bradlow, and Pieter Abbeel. Finding locally optimal, collision-free free trajectories with sequential convex optimization. In Robotics: science and systems, volume 9, pp. 1-10. Berlin, Germany, 2013.
Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. Progprompt: Generating situated robot task plans using large language models. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 11523-1 1530. IEEE, 2023.
Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M Sadler, Wei-Lun Chao, and Yu Su. Llm-planner: Few-shot grounded planning for embodied agents with large language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2998-3009, 2023.
Marlin P Strub and Jonathan D Gammell. Adaptively informed trees (ait): Fast asymptotically optimal path planning through adaptive heuristics. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 3191-3198. IEEE, 2020.
Martin Sundermeyer, Arsalan Mousavian, Rudolph Tricbel, and Dieter Fox. Contact-graspnett Efficient 6-dof grasp generation cluttered scenes. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 3343--13444 IEEE, 2021.
Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: framework for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2):11--11, 1999.
Yujin Tang, Wenhao Yu, Jie Tan, Heiga Zen, Aleksandra Faust, and Tatsuya Harada. Saytap: Language to quadrupedal locomotion. arXiv preprint arXiv:2306.07580 2023.
Russ Taylor, Matthew T Mason, and Kenneth Y Goldberg. Sensor-based manipulation planning as game with nature. In Fourth International Symposium Robotics Research, pp. 421-429, 1987.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSI international conference on intelligent robots and systems, pp. 5026-5033 IEEE, 2012.
Hugo Touvron, Thibaut Lavril, Gautier Izacard. Xavier Martinet, Marie-Anne Lachaux, Timothéc Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv: 13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023b.
Miomir Vukobratović and Veljko Potkonjak. Dynamics of manipulation robots: theory and applica- tion. Springer, 1982.
Yen-Jen Wang. Bike Zhang, Jianyu Chen, and Koushil Sreenath. Prompt a robot to walk with large language models. arXiv preprint 2rXiv:2309.099699 2023.
13

---

Daniel E Whitney. The mathematics of coordinated control of prosthetic arms and manipulators. 1972.
Daniel E Whitney, Mechanical assemblies: their design, manufacture, and role in product develop- ment, volume 1. Oxford university press New York, 2004.
Jimmy Wu, Rika Antonova, Adam Kan, Marion Lepert, Andy Zeng. Shuran Song, Jeannette Bohg. Szymon Rusinkiewicz, and Thomas Funkhouser. Tidybot: Personalized robot assistance with large language models. arXiv preprint arXivv2305.05653 2023.
Fei Xia, Chengshu Li, Roberto Martin-Martin, Or Litany, Alexander Toshev, and Silvio Savarese. Relmogen: Leveraging motion generation in reinforcement learning for mobile manipulation. arXiv preprint arXiv:2008.07792, 2020.
Jun Yamada, Youngwoon Lee. Gautam Salhotra, Karl Pertsch, Max Pflueger, Gaurav Sukhatme, Joseph Lim, and Peter Englert. Motion planner augmented reinforcement learning for robot manipulation in obstructed environments. In Conference on Robot Learning, pp. 589-603, PMLR, 2021.
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. arXiv preprint arXiv:2107.09645, 2021.
Yufei Ye, Xueting Li, Abhinav Gupta, Shalini De Mello, Stan Birchfield, Jiaming Song. Shubham Tulsiani, and Sifei Liu. Affordance diffusion: Synthesizing hand-object interactions. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 22479-22489, 2023.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pp. 1094-1100 100. PMLR, 2020.
Wenhao Yu, Nimrod Gileadi, Chuyuan Fu, Sean Kirmani, Kuang Kuang-Huei Lee, Montse Gonzalez Arenas, Hao-Tien Lewis Chiang, Tom Erez, Leonard Hasenclever, Jan Humplik, Brian Ichter, Ted Xiao, Peng Xu, Andy Zeng, Tingnan Zhang, Nicolas Heess, Dorsa Sadigh, Jie Tan, Yuval Tassa, and Fei Xia. Language to rewards for robotic skill synthesis. Arxiv preprint arXiv:2306. 086477777 2023.
Jesse Zhang, Jiahui Zhang, Karl Pertsch, Ziyi Liu, Xiang Ren, Minsuk Chang, Shao- Etao-HHa Sun, and Joseph J Lim. Bootstrap your own skills: Learning to solve new tasks with large language model guidance. Conference on Robot Learning, 2023.
Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Krähenbüüh,, and Ishan Misra. Detecting twenty- thousand classes using image-level supervision. In Computer Vision-ECC ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part IX, pp. 350-368. Springer, 2022.
Yuke Zhu, Josiah Wong, Ajay Mandlekar, Roberto Martin-Martin, Abhishek Joshi, Soroush Nasiriany, and Yifeng Zhu. robosuite: modular simulation framework and benchmark for robot learning. arXiv preprint arXiv:2009. 2020.
14

---

Appendix
A
TABLE OF CONTENTS
Ethics, Impacts and Limitations Statement (Appendix B): Statement addressing potential ethics concerns and impacts as well as limitations of our method.
Additional Experiments (Appendix C): Additional ablations and analyses as well as learning curves for single-stage tasks and Meta-World
PSL Implementation Details (Appendix D): Full details how PSL is implemented, specifically the Sequencing Module.
Baseline Implementation Details (Appendix E): Full details regarding baseline implements (E2E, RAPS, MoPA-RL, TAMP, SayCan)
Tasks (Appendix F): Visualizations of each task as well descriptions of each environment suite.
LLM Prompts and Plans (Appendix G): Prompts that we use for our method as well as generated plans by the LLM.
15

---

B ETHICS, IMPACTS AND LIMITATIONS
B.1 ETIIICLL CONSIDERATIONS
There exist potential ethical concerns from the use of large-scale language models trained on internet- scale data. These models have been trained on vast corpi that may contain harmful content and implicit or even explicit biases expressed by internet users and may be capable of generating such content when queried. However, these issues are not specific to our work, rather they are inherent to LLMs trained at scale and other works that use LLMS face similar ethical concern. Furthermore, we note that our research only makes use of LLMs to guide the behavior of a robot at a coarse level specifyyng where a robot should go and how to leave the area. Our LLM prompting scheme ensures that this is all that is outputted from the LLM. Such outputs leave little scope for abuse, the LLM is not capable of performing the low-level control itself, which is learned through task reward independently.
B.2 BROADER IMPACTS
Our research guiding RL agents solve long-horizon tasks using LLMs has potential for both positive and negative impacts. PSL draws connections between work on language modeling, motion planning and reinforcement learning for low-level control, which could lead to advancements in learning for robotics. PSL reduces the engineering burden on the human, instead of manually seatffyinngpe-ttrinnigg a library behaviors, only reward function and task description need be specified. More broadly, enabling robots to autonomously solve challenging robotics tasks increase the likelihond of robots one day being able to complete labor intensive work in dangerous situations. However, with increased automation, there are risks of potential job loss. Furthermore, with increased robot capabilities, there is a risk of misuse by bad actors, for which appropriate safeguards should be designed.
B.3 LIMITATIONS
There are several limitations of PSL which leave scope for future work. 1) We impose a specific structure on the language plans and task solution (go to location X, interact there, so on). While this assumption covers a broad set of tasks as well illustrate our experimental evaluation, tasks that involve interacting with multiple objects simultaneously or continuous switching between interaction and movement in a fluid manner may not be directly applicable. Future work can explore integrating more expressive plan structure with the Sequencing Module. 2) Use of motion pootion-planning makes application to dynamic tasks challenging. To that end, research on motion- motion-planner distillation, such as Motion Policy Networks Fishman et al. (2022) could enable much faster, reactive behavior. 3) Although the RI. agent is capable of adapting pose estimation errors, in the current formulation, there not much the Learning Module can do if the high-level plan itself is entirely incorrect, or the Sequencing module misinterprets the language instruction and moves the robot to the wrong object. One extension to address this limitation would be to fine-tune the Plan and Seq Modules online using RL as well, to adapt the large models to the specific environment and reward function.
16

---

C
ADDITIONAL EXPERIMENTS
We perform additional analyses of PSL in this section.
RS-Can Camera Ablation
1.0
Success Rate
0.8
0.6
0.4
0.2-
0.0
1000
2000
3000
4000
5000
Number of Trials
PSL-Fixed
PSL-Witst (Ours)
PSL.-rrist: Fixed
Figure C.I: Camera View Learning Performance Ablation. wrist camera views clearly accelerate learning performance, converging to near 100% performance 4x faster than using fixed-view and 3x faster than using wrist+fixed-view observations.
Effect of camera view on policy learning performance: As discussed in Sec. 3, for PSL we use local observations to improve learning performance and generalization to new poses. We validate this claim the Robosuite Can task, in which we hypothesize that the local wrist camera view will accelerate policy learning performance. This is because the image of the can will be independent of the can's position in general since the Sequencing Module will initialize the RL agent close the can as possible. As observed in Fig. C.I, this is indeed the case - PSI. learns 4x faster than using a fixed view camera in terms of the number of trials. We additionally test if combining wrist and fixed view inputs (a common paradigm in robot learning) can alleviate the issue, however PSL with wrist cam still 3x faster at solving the task.
Effect of camera view on chaining pre-trained policies: In this ablation, we illustrate another important effect of using local views, such as wrist cameras: ease of chaining pre-trained policies. Since we leverage motion planning to sequence between policy executions, chaining pre-trained policies is relatively straightforward: simply execute the Sequencing Module to reach the first region of interest, execute the first pre-trained policy till its stage termination condition is triggered, then call the Sequencing Module on the next region, and so on. However, to do so, it is also crucial that the observations do not change significantly, so that the inputs to the pre-trained policies are not out of distribution (OOD). we use a fixed, global view of the scene, the overall scene will change multiple policies are executed, resulting in future policy executions failing due to OOD inputs. In Table C.I, we observe this exact phenomenon, in which any version of PSL that is provided a fixed-view input fails chain pre-trained policies effectively, while PSL. with local (wrist) views only is able to chain pre-trained policies on every task, up to 5 stages.
As noted in the Limitations Section (Sec. B), we acknowledge that as defined, if the Plan Module or Sequence Module fail catastrophically (incorrect plan or moving to the wrong region in space), there is currently no concrete mechanism for the Learning Module to adapt. However, we run an experiment in which we train the agent using PSL using an incorrect high-level plan on two stage tasks (MW-Assembly, MW-Bin Bin-Picking, MW-Hammer) and find that in some cases, the agent can still learn to solve the task, achieving performance close to F2E (Fig. C.4. Intuitively, this is possible because in PSI., the high-level plan is not expressed as a hard constraint, but rather series of regions for the agent to visit and set of exit conditions for those regions. In the end, however, only the task reward is used to train the RL policy so if the plan is wrong, the Learn Module must leam to solve the entire task end-to-end from sub-optimal initial states.
Effect of incorrect plans on training policies using PSL:
Ablating Camera View choice for Baselines: We evaluate if including Oloeal in addition to Ogoobal improves performance across four tasks (RS-Lift RS-Door, RS RS-Can, RS-NutRoundl and include the results in Fig. C.5. In general there is little to performance improvement for RAPS
17

---

E2E across the board. The additional local view marginally improves sample efficiency but it does not resolve the fundamental exploration problem for these tasks.
Single Task
K-MS-3
K-MS-4
K-MS-5
PSL-Wrist
1.0
1.0 + 0.0
10 0.0
1.
PSL--Fixed
1.0 100::000
0.0 1: 0.0
1
0.0
0.0.1.00
PSI.-Wrist-Fixcd Fixed
1 0.0
0.0 = 0.0
0.0 = 0.0
0.0 0.0
Table C.I: Chaining Pre-trained Policies Ablation. can leverage local views (wrist cameras) to chain together multiple pre-trained policies via motion-planning using the Sequencing Module. While PSL with each camera input is able to produce capable single-task task policy, chaining only works with wrist camera obscrvations the observations are kept in dissttibbuutin..
Robosuite Lift
Robosuite Door
Kitchen Slide Cabinet
Kitchen Kettle
18-
18-
1.8
Success Rate
Sucoesss Rate
Suceess Rate
Sucoess Rate
ne-- oo- n4-- 05
0.8-
0.8
08-
0s-
0.-
044
04
8.
63
83
00
60
100
08
200
400
8000
2000
1000
2000
4000
6020 2000
1000
280
4000
6600 si0e
1000
200
4000
6000 a6c
10600
Number of Trials
Number of Trials
Number of Trials
Number of Trials
Kitchen Light Switch
Kitchen Left Bumer
Kitchen Microwave
WW-Disnssemble
1.0
Supcess Rate
Success Rate
Success Rate
Success Rate
0.
044
04
84
84
220
3300
1000
2000
1000
330
4000
2000
10030
2001
4030
60.0
5000
10000
330 4001
M60
1960
Number of Trials
Number of Trials
Number of Trials
Number of Trials
Assembly
OifLif
OS Push
Success Rate
Success Rate
Success Rate
W
1900
0001
2000
1900
(00)
1000
10000
0003 Number Trials
6000
Number Trials
Number of Trials
McPA-RL
Figure C.2: Single Stage Results. We plot task success rate a function of the number of trials. PSL improves the efficiency of the baselines across single-stage tasks (plan length of 2) in Robosuite, Kitchen, Meta- World, Obstructed Suite, achieving an asymptotic success rate on 11 tasks.
Metaworld Hammer
Metaworld Assembly
Metaworld Bin Picking
Snccess Rate
Success Rate
Rx
Rate
E 8
Trials
Number
Number Trials
McPA-RL
(Oan)
Figure C.3: Meta-Worke Two Stage Learning Curves. We plot task function of the number
of trials. PSL learns faster than the baselines by employing high-level planning to accelerate RL performance.
18

---

MW BinPick MM Assenbly
MM flamer
E2F
1.0 0.0
0.4 0.5
0.0 1.0
RAPS
0.0
0.3
1.0
TAMP
0.0
1,0 0.0
0.0
SayCan
1.0 0.0
0.5 = 08
1.0 0.0
PSL.
1.0 4 0.0
1.0 10::: 0.0
1.0 d: 0.0
Table C.2: Metaworld Two Stage Results. While the baselines perform well on most of the tasks, only PSL able to consistently solve every task. This is because the LLM planning and Sequencing modules case the learning burden for the RL policy, enabling it to learn contact-rich, long-horizon behaviors.
AW-Hammer
AWW-ssseblly
MW-BinPick
Success Rate
Success Rate
Success Rate
a500
1000
Number Trials
Number of Trials
Number Trials
E2E
PSL (Ours)
PSL Bad Plan
Figure C.4: PSL Bad Plans. We plot task success rate function of the number of trials. Even when given the wrong high-level plan, PSL is able learn solve the task, albeit slower rate.
RS-Door
RS-1.an l.ft
RS. No--utRoudd
RS. Can
Success Rate
Success Rate
Success Rate
Sieceess
Number Trials
Trials
EZE-0
E2E-0 bval
O
PSL
RAPS-Qnaka
RAPS-O Otliel
Figure C.5: Ablating Camera Views for Baselines. We plot task success function of the number of
trials. As seen in the figure above, including Olocai docs not improve the performance of E2E or RAPS.
19

---

D PSL IMPLEMENTATION DETAILS
Algorithm 2 PSL Implementation
Require:
LLM, task description 9t, Motion Planner MP, low-level horizon Hi, segmentation model S, RGB-D global cameras, RGB wrist camera, Camera Matrix K Kaglobal
1:
initialize RL: ne. replay buffer R
Planning Module
High-level plan P <- Prompt(LI.M, gt)
3: for episode 1...N do
4: for ア do
Sequencing Module
target region (1), termination condition +-
6:
PCalobal Projectioon((flaab Oglobal K
7: Mrobot, Mary = Segmentation (Opion Ogtobal robot, object)
PCrobot oCojectt Mrooot PC Mars PC
9:
PC PCglnball - PCrobot
10:
eCtarget mean( PC
Qtarget = =I((eetarget)
12:
MotionPlan(MPP 9target. PCacene)
Learning Module
13:
for i L,..., h low-level steps do
Get action a, (T,(O (Ofoca))
15:
Get next state Ofgnn p(18s; az).
16:
Store (Olocal as. Ofter r) into R
17:
Sample (Olocat (eg Locai r) R update Ty using RL if post-condition then break end if
k = random index
18:
19:
20:
21:
22:
23:
end for
24:
end for
D.1 PLANNING MODULE
Given a task description 9t, we prompt LLM using the format described in Sec. 3.3 to produce a language plan. We experimented with variety of publicly available and closed-source LLMs including LLAMA Touvron et al. (2023a). LLAMA-2 Touvron et al. (2023b), GPT-3 Brown et al. (2020), Chat-GPT, and GPT-4 OpenAl (2023). initial experiments, we found that GPT-based models performed best, and GPT-4 in particularly most closely adhered to the prompt and produced the most accurate plans. As result, in our experiments, we use GPT-4 as the LLM planner for all tasks. We sample from the model with temperature 0 for determinism. Sometimes, the LLM hallucinates non-existen, stage termination conditions or objects. As a result, we add pre-processing step which we delete components of the plan that contain such hallucinations.
D.2 SEQUENCING MODULE
The input to the Sequencing Module is Oglobal In our experiments, we use two camera views, Oolobal and Oglobol which are RGB-D calibrated camera views of the scene, to obtain unoccluded views of the scene. We additionally provide the current robot configuration, which is joint angles for robot arms: Gjoint and the target region label around which the RL policy must perform environment interaction. From this information, the module must solve for a collision free path to a region near the target. This problem can be addressed by classical motion planning. We take advantage of sampling- based motion planning due to its minimal setup requirements (only collision -hirsiion-cheeiing) and favorable performance on planning. In order run the motion planner, we require collision checker, which we implement using point-clouds. To compute the target object position, we use predicted segmentation along with calibrated depth, opposed to dedicated pose estimation
20

---

Target: can
Segmentation (SAM)
Magge
Kinematics
M
Masy
rolnnt
PC
Motion Planner (AIT*)
Projection
Figure D.I: Sequencing Module. Inputs to the Sequencing Module are two calibrated RGB-D - fixed views, Oslobul the proprioception Gjoint and the target object. performs visual motion planning to the target object by computing seene point-cloud (PCalobal) ), segmenting the target object (Moty) to estimate its pose (gtarget), segmenting the robot (Mrohot) to remove from PC and motion planning using AIT*.
network, primarily because state of the art segmentation models (Kirillov et al., 2023; Zhou et al., 2022) have significant zero-shot capabilities across objects.
Projection: In this step, we project the depth map from each global view of the scene, Ogiobat and Ogiobal into a point-cloud PC using their associated camera matrices k-global and Kg Kyyobal We perform the following processing steps to clean up PCstobal, 1) cropping to remove all points outside the workspace voxel down-sampling with size of 0.005 m to reduce the overall size of PC 3) outlier removal, which prunes points that are farther from their 20 neighboring points than the average in the point-cloud as shown Fig. D.1.
Segmentation: We compute masks for the robot (Mroboz) and the target object (Mot,) by using segmentation model (SAM Kirillov et al. (2023)) S which segments the scene based on RGB input. We reduce noise in the masks by filling holes, computing contiguous mask clusters and selecting the largest mask. We use Mrohot to remove the robot from PCalobal, in order perform collision checking of the robot against the scene. Additionally, the object point-cloud PCobj, which we average to obtain an estimate of object position, which is the target position for the motion planner. For the manipulation tasks we consider in the paper, this the target end-effector pose of the robot, ECtarget-
use Mokj along with PC to compute
Visual Motion Planning: Given target end-effector pose eCtarget, we use inverse kinematics (IK) to compute Starget and pass Qjoint Starget; PColobal into a joint-space motion planner. To that end, use sampling-basee motion planner, AIT Strub & Gammell (2020), to perform motion planning. In order to implement collision checking from vision, for a sampled joint-configuration Qsample- we compute the corresponding position of the robot mesh and compute the occupancy of each point in the scene point-cloud against the robot mesh. If the object is detected as grasped, then we additionally remove the object from the scene pointcloud, compute its convex hull and the signed distance function the joint robot-object mesh for collision checking. As a result, the Sequencing Module operates entirely over visual input, and achieves pose near the region of interest before handing off control to the local RL policy. We emphasize that the Sequeneing Module does not need to be perfect, it merely needs produce reasonable initialization for the Learning Module.
D.3 LEARNING MODULE
D.3.1 STAGE TERMINATION DETAILS
As described in Section 3, use stage termination conditions to determine when the Learning Module should hand control back to the Sequencing Module to continue to the next stage in the plan. For most of the tasks consider, these stage termination conditions amount checking for a grasp or placement for the target object in the stage. For example, for RS As-NutRound the plan for the first stage is (grasp, nut) and the plan for the second stage is (place, peg). Placements straightforward
21

---

to check: simply evaluate if the object being manipulated is within small region near the target object. This can be computed using the estimated pose of the two objects (current and target). Grasps challenging to estimate and employ two stage pipeline to detecting grasp. First, estimate the object pose and then evaluate the z value has increased from when the stage began. Second, in order ensure the object is not simply tossed in the air, check if the robot's gripper is tightly caging the object. We do so by collision checking the object point-cloud against the gripper mesh. We use the same collision checking procedure outlined in Sec for checking collision between the scene point-cloud and robot mesh.
To estimate the turned condition, we compute the mask of the burner knob, evaluate its principal axis and measure its angle from vertical. If it is greater than X radians then the stage condition triggers. Checking the pushed condition straightforward, the object needs to have moved by distance forward from the start pose in xy. Finally, for checking the opened condition, we estimate the handle pose relative to the hinge and compute the angle of the door. If is greater than radians we consider the door opened. For the slide cabinet the handle pose itself can be used check opening. Closing is likewise estimated as the inverse opening. For all conditions, we take the threshold X from the environment success condition or reward function.
D.3.2 TRAINING DETAILS
For all tasks, we use the reward function defined by the environment, which may be dense or sparse depending on the task. We find that for PSL, it crucial use an action-repeat of 1, in general we found that increasing this harmed performance, in contrast to the E2E baseline which performs best with an action repeat from the paper, held constant across all tasks. We train policies using 84x84 images. We use the "medium" difficult exploration schedule defined in Yarats et al. (2021), which anneals the exploration from 1.0 to 0.1 over the course of 500K environment steps. Due to memory concerns, instead of using a replay buffer size of 1M as done in Yarats et al. (2021), ours is of size 750K across each task. Finally, for path length, we use the standard benchmark path length for E2E and MoPA-RL per stage for RAPS following Dalal et al. (2021), and 25 per stage for PSL.
For training policies using DRQ-v2, we use the default hyper-parameter
22

---

E BASELINE IMPLEMENTATION DETAILS
E.I RAPS
For this baseline, we simply take the results from the RAPS Dalal et al. (2021) paper as is, which use Dreamer Hafner et al. (2019) and sparse rewards. In initial experiments, we attempted to combine RAPS with DRQ-v2 Yarats et al. (2021) I) and found that Dreamer performed better, which is consistent with RAPS+Dreamer having the best results in Dalal et al. (2021). We additionally tried to run RAPS with dense rewards, but found that the method performed significantly worse. One potential reason for this that it not clear exactly how to aggregate the dense rewards across primitive executions tried simply taking the dense reward after executing a primitive well simply summing the rewards of intermediate primitive executions. Both performed worse than training RAPS with sparse rewards. Note that PSL outperforms RAPS even when both methods have only access to sparse rewards, c.g. the Kitchen environments. We observe clear benefits over RAPS on the single-stage (Fig. C.2) and multi-stage (Table 2) tasks.
E.2 MOPA-RL
As described in the main paper, we take the results from MoPA-RL Yamada et al. (2021) as is on the Obstructed Suite of tasks. Those results were run from state-based input and leveraged the simulator for collision checking. We do so as we were unable to successfully combine MoPA-RL with DRQ-v2 based the publicly released implementations of both methods.
E.3 TAMP
We use PDDLStream Garrett al. (2020a) as the TAMP algorithm of choice as it has been shown to have strong planning performance on long-horizon manipulation tasks in Robosuite (Dalal et al., 2023; Mandlekar et al., 2023). The PDDLStream planning framework models the TAMP domain and uses the adaptive algorithm, sampling based algorithm, to plan. This TAMP method uses samplers for grasp generation, placement sampling, inverse kinematics, and motion planning, making performance stochastic. Hence we average performance across 50 evaluations to reduce variance. We adapt the authors TAMP implementation (from (Dalal et al., 2023; Mandlekar al., 2023)) for our tasks. Note this method uses privileged access the simulator, leveraging knowledge about the task (which must be explicitly specified a problem file), the scene (from the domain file and access to collision checking) and 3D geometry of the environment objects.
E.4 SAYCAN
As described in the main paper, we re-implement SayCan Ahn al. (2022) using GPT-4 (the same LLM we use in our methdo) and manually engineered pick/place skills that use pose-estimation and motion-planning. Following our Sequencing module: 1) we build 3D scene point-cloud using camera calibration and depth images 2) we perform vision-based pose estimation using segmentation along with the scene point cloud and 3) we run motion planning using collision queries from the 3D point-cloud, which is used for collision queries. Finally, use heuristically engineered pick and place primitives to perform interaction behavior which we describe as follows. We note that for our tasks of interest, the pick motion can be represented as top-grasp Once we position the robot near the object; we then simply lower the robot ann till the end-effector (not the grippers) come in contact with the object. We then close the gripper execute the grasp. For place, we follow the implementation of Ahn et al. (2022) and lower the held object until contact with a surface, then release (open the gripper) and lift the robot arm. We set the affordance function for both skills following the design in Ahn et al. (2022) for motion planned skills.
For LLM planning, we specify the following prompt:
Given the following library of robot skills: Task description: Make sure to take into account object geometry. Formatting of output: a list of robot skills. Don't 't output anything else.
23

---

This prompt is the same prompt except specify the robot skill library in terms of object centric behaviors, instead of stage termination conditions.
Given the following library of robot skills: Task description: Give me a simple plan to solve the task using only the provided skill library. Make the plan follows the formatting specified below and make sure to take into account object geometry. Formatting of output: a list of robot skills. Don't 't output anything else.
Robosuite
Skill Library: pick can, pick milk, pick cereal, pick bread slice, pick silver nut, pick gold nut, put can on/in X, put milk on/in X, put cereal on/in X, put bread slide on/in X, put silver nut on/in X. put gold nut on/in X, grasp door handle, turn door handle, pick cube
Kitchen
Skill Library: grasp vertical door handle for slide cabinet, move left, move right, grasp hinge cabinet, grasp top left burner with red tip, rotate top left burner with red tip 90 degree clockwise, rotate top left burner with red tip 90 degrees counterclockwise, push light switch knob left, push light switch knob right, grasp kettle, lift kettle, place kettle on/in grasp microwave handle, pull microwave handle
Metaworld:
Skill Library: grasp cube, place cube on/in X, grasp hammer, place hammer, hit nail with hammer, grasp wrench, lift wrench
Obstrdcted Suite
Skill Library: grasp can, place can in bin, insert table leg in X, move table leg, grasp cube, place cube table, push cube
24

---

F TASKS
(a) KWW-Hameer
(b) MM Mssesbbybly
(c) MM-DLsasseebble
(d) WW-Bin-Picking Picking
OS-Lift
OS Assssmsmll
(g) OS-Push
(h) K-Slide
ii) X-Xettle
ii) K X-Microwave
(k) K-Burner
(I) K- K-Light
(m) RS ft.
(n) RS-Door
(o) RS- R8-NutRound
(p) RS-NutSquare
(q) RS N--uutsssembyy
(r) Can
(s) RR-rerl
(1) RS-Milk
(u) RS-Bread
(v) RS-CanBread
(w) RS RS-CerealMilk
Figure F.l: Task Visualizations, PSL is able to solve all tasks with at least 80% success rate from purely visual input.
25

---

We discuss each of the environment suites that we evaluate using PSL. All environments simulated using the MuJoCo simulator Todorov et al. (2012).
1. Meta-World (Row 1 of Fig. F.I). Meta- World, introduced by Yu et al. (2020), aims to offer standardized suite for multi-task and meta-leaming methods. The benchmark consists of 50 separate manipulation tasks with Sawyer robot, well-shaped reward functions, involve manipulating single object a randomized goal position, or multiple objects to deterministic goal position. We evaluate on the single-task, multi-goal, v2 variants of the Meta--Wrlb environments. All environments use ind-effector position control a 3DOF arm action space along with gripper control orientation is fixed. In our evaluation we use the default environment task rewards, a fixed camera view for the baselines and a wrist camera for our local policies. We refer the reader to the Meta-Worde World paper for additional details regarding the environment suite.
2. Obstructed Suite (Rows 1-2 of Fig. F.1). The Obstructed Suite of tasks introduced by Ya- mada et al. (2021) are challenging set of tasks requiring Sawyer arm to perform obstacle avoidance while solving the task. The OS-Li Et. task requires the agent to pick up a can that is inside a tall box, requiring it to reach over the walls to grab the object and then lift it without making contact with the edges of the bin. The OS -a-sh environment tasks the agent with push a block to the goal in the present of bin that forces the agent to adjust its motion in order to avoid being blocked by its upper joints. Finally, the OS-As s emb ly task involves moving the robot arm to precise placement location while avoiding obstacles, then performing the table leg placement. Note that we evaluate our method on these envi- ronments from visual input, a more challenging setting than the one considered by Yamada et al. (2021).
3. Kitchen (Rows 2-3 of Fig. F.I.. The Kitchen manipulation suite introduced in the Relay Policy Learning paper Gupta et al. (2019) and maintained in D4RL Fu et al. (2020) is set of challenging, sparse reward, joint-controlled manipulation tasks in a single kitchen. We modify the benchmark to use end-effector control as we find that this significantly improves learning performance. The tasks require the ability to explore efficiently whilst also being able to chain skills across long temporal horizons, the microwave, moving the kettle, flicking the light switch, turning the top left burner, and finally sliding the cabinet door (K-MS-5). Aside from the single-stage tasks described in Section 4, we evaluate on three multi-stage tasks which require chaining the single-stage tasks in a particular order. K-M5-3 involves moving the kettle, flicking the light switch and turning the top left burner, K-MS-7 involves doing the same tasks K-MS-5 then turning the bottom right burner and opening the hinge cabinet and K-MS- 10 involves performing the tasks in 7-MS-7 and then opening the top right burner, opening the bottom left burner and then closing the microwave.
achieve behaviors such as opening
4. Robosuite (Rows 3-6 of Fig. F.I). The Robosuite benchmark from Zhu et al. (2020) contains challenging, long-horizon manipulation tasks involving pick-place and nut assembly, as well as simpler tasks that involve lifting a cube and opening a door. The rewards are coarsely defined in terms of distances targets well as grasp/placement conditions, which, in fact, are straightforward to implement in the real world as well using pose estimation. This stands in contrast Meta- World which spends considerable engineering effort defining well-shaped dense rewards often by taking advantage of object geometry. As a result, learning-based methods struggle make any progress on Robosuite tasks that involve more than a single-stage optimizing the reward function tends to leave the agent local minima. The suite also contains well-tuned, realistic Operation Space Control Khatib (1987) implementation that we leverage to train policies in end- effeeetfooor space.
26

---

G LLM PROMPTS AND PLANS
In this section, we list the LLM prompts per task. Overall prompt structure:
Stage termination conditions: (grasp, place, push, open, close, turn). Task description: Give me a simple plan to solve the task using only the stage termination conditions. Make sure the plan follows the formatting specified below and make sure to take into account object geometry. Formatting of output: a list which each element looks like: (<object/region>, <operator>>. Don't output anything else.
G.1 ROBOSUITE
RS-F
Task Description can goes into bin 1. Plan: [("can", "grasp"), ("bin 1", "place")))
RS-PicXPlaceCeraal:
Task Description: cereal goes into bin 3. Plan: [("cereal", "grasp"), ("bin 3", "place")[)
RS-PickPlaceMi
Task Description: milk goes into bin 2. Plan: [("milk", "grasp"), ("bin 2", "place")))
RS-PickPlaceBread:
Task Description: bread slice goes into bin 4. Plan: [("bread slice", "grasp"), ("bin 4", "place")))
RS-PicAFPlacCaanSre.dt
Task Description: can goes into bin 1, bread slice bin Plan: [("can", "grasp"), ("bin 1", "place"), ("bread slice", "grasp"), ("bin 4", "place")])
RS-PicckPlaceCCrrea.Mii
Task Description: milk goes into in bin 2, cereal in bin 3. Plan: [("cereal", "grasp")) ("bin 3", "place"), ("milk", "grasp"), ("bin 2", "place")])
RS-MutAssembly:
Task Description: The silver nut goes on the silver peg and the gold nut goes on the gold peg. Plan: [["silver nut", "grasp"), ("silver peg", "place")) nut", "grasp"), ("gold peg", "place")]
27

---

RS-NuutAssemblyySquree:
Task Description: The gold nut goes on the gold peg. Plan: [("gold nut", nut""""rasp"" "grasp"), ("gold peg", "place")]
RS AS-NuttaseebbyyRomm
Task Description: The silver nut goes the silver peg. Plan: [("silver nut", "grasp"), ("silver peg", "place")]
RS-Lift:
Task Description: lift the red cube. Plan: [("red cube", "grasp")]
RS RS-Door:
Task Description: open the door. Plan: [("door handle", "grasp")]
G.2 META-WORLD
MW Assembembl
Task Description: put green wrench on maroon peg. Plan: [("green wrench", "grasp"), ("maroon peg", "place")]
Mi-Disassemble:
Task Description: remove the green wrench from the peg. Plan: [("green wrench", "grasp")]
Ma-Hameer:
Task Description: use the red hammer to push in the nail. Plan: [("red hammer", "grasp"), ("nail", "push")]
Ww-iinPPicking:
Task Description: move the cube in the red bin into the blue bin. Plan: [("cube red bin", "grasp"), ("blue bin", "place"))
G.3 KITCHEN
Task Description: pull the microwave door open. Plan: [("microwave door handle". "open"))
28

---

Kitchen-Slide
Task Description: use the rightmost vertical bar slide the door. Plan: [("rightmost vertical bar", "open")]
Kitchen-Iiight.
Task Description: use the round knob flick the light switch. Plan: [("knob", "turn")]
Metchen-Burnee
Task Description: rotate the top left burner with the red tip. Plan: [("top left burner with the red tip", "turn")]
Kitchen-Kettle
Task Description: move the kettle forward. Plan: [("kettle", ""uuh")]
G.4 OBSTRUCTED SUITE
OS-Lift:
Task Description: lift red can from wooden bin.
Plan: [("red can', "grasp")]
OS Assembly:
Task Description: move the table leg, which already in your hand, into the empty hole. Plan: [("empty hole', "place")]
OS OS-Push:
Task Description: push the red block onto the green circle. Plan: [["red block", "grasp")]
29


Reviews:
[{"review_id": "XH8omrvJnd", "reviewer": "Reviewer_ue89", "summary": "The paper proposes a new method/framework called Plan-Seq-Learn (PSL) for solving long-horizon robotics tasks. The key idea is a decomposition of long robotic manipulation tasks, and then tackle each part using a reasonable method. Specifically, they combine LLM for highly abstract task planning, off-the-shelf visual pose estimator and motion planner (AIT*) for sequencing each sub-tasks, and RL for the sub-tasks. This allows PSL to leverage the advantage of each module. Extensive experiments show PSL can efficiently solve 20+ long-horizon robotics tasks, outperforming prior methods.", "questions": "1. As the whole task is decomposed into stage 1, sequencing, stage 2, ..., stage n. Does it need to redesign the reward function of the RL process? Moreover, how does the sparse and dense reward influence the learning process?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The approach is sensible and reasonable to leverage current popular methods for robot learning - LLM for high-level planning, classical motion planner for efficient collision-free path planning, and RL for the contact-rich manipulation stage.\n2. The general framework is novel in combining these techniques although each part is not entirely new. And the paper clearly explains how to use their advantages in solving long-horizon tasks.\n3. Extensive experiments show reasonable/good results regarding their claims and methods.", "weaknesses": "1. The long-horizon task seems to be divided into only 'grasp' and 'place' (from the paper and appendix), it is unclear if there are more sub-tasks / skills that the LLM divided. From the webpage, I find other tasks besides the pick-and-place series so wonder how to implement these.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "80PQ7uL9UX", "reviewer": "Reviewer_KErm", "summary": "In past works, people utilized LLM's internet-scale of knowledge to give robots sufficient information when planning for long-horizon tasks. However, the author believes that it is important for a robotic system to be capable of online improvement over at least low-level control policies at the same time. Otherwise, with the lack of a library for pre-trained skills in every other scenario, robots aren't able to learn very well. To this end, the paper proposes a framework, PLAN-SEQ-LEARN, that utilizes both LLM's ability to guide agent's planning and RL's ability for online improvement. The experiments show that not only did PSL's performance surpass SOTA visual-based RL methods through the help of LLM, but it also performed better than SayCan for its ability to improve with online learning.", "questions": "As stated above.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "**Motivation and intuition**\n- Classical approaches to long-horizon robotics that can struggle with contact-rich interactions are convincing.\n- Use LLM for high-level planning guiding RL policy to solve robotic tasks online without pre-determined skills.\n\n​\n**Novelty**\n- The idea of utilizing RL to learn low-level skills under the framework of LLM planning is intuitive and convincing.\n​\n\n**Technical contribution**\n- Integrates LLM task planning, motion planning, and RL techniques.\n- Avoid cascading failures by learning online using RL algorithms.\n\n​\n**Clarity**\n- The overall writing is clear. The authors utilize figures well to illustrate the ideas. Figure 2 clearly shows the whole idea of PSL.\n- This paper provides a clear and detailed description of how to integrate the task planning module, motion planning module, and RL learning module.\n\n**Related work**\n- Give plenty of related works with short but clear descriptions.\n​\n**Experimental results**\n- The overall performance on single-stage and multistage benchmark tasks is good.", "weaknesses": "**Clarity**\n- Although details of how LLM was used were clearly written inside Appendix D, I feel like the author could illustrate the details in the main paper and also a better explanation of how stage termination and training details are implemented. Since how LLM was involved in this work seems to be one of the contributions of this paper, I do feel like making this part intuitive is a must.\n\n​\n**Method**\n- Trade-off: Planning without a library of pre-defined skills is mentioned as a strength in the paper, but this comes at the cost of relearning the whole process compared to other methods.\n- Also the paper seems to overlook the fact that the learning might fail. Did not see how the method handles this situation.\n- What would the PSL react to the situation when the agent failed to reach the termination condition?\n- What would happen if there are more than enough terms for the LLM to choose from, for example, unlearnable skill terms that may confuse the LLM in choosing?\n\n​\n**Related work**\n- Although Citing 'Inner Monologue' and 'Bootstrap Your Own Skills(BOSS)', they are not used for comparison or experiments, as these methods share many similarities. Therefore, it's a bit of a missed opportunity.\n\n​\n**Experimental conclusions**\n- In section 4.3, the author noted that \"For E2E and RAPS, we provide the learner access to a single global fixed view observation from O^global for simplicity and speed of execution, as we did not find meaningful performance improvement in these baselines by incorporating additional camera views.\". However, this results in an unfair comparison because PSL has taken O^local as an additional input, and may cause some questionable issues. If performances are similar, I believe that adding O^local for E2E and RAPS would result in a more convincing conclusion that PSL performs better.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kyfOLi0oSD", "reviewer": "Reviewer_oGjq", "summary": "The authors propose Plan-Seq-Learn (PSL) to address long-horizon robotics tasks from scratch with a modular approach using motion planning to bridge the gap between abstract language and low-level control learned by RL. The authors experiment with 20+ single and multi-stage robotics tasks from four benchmarks and report success rates of over 80% from raw visual input, out-performing previous approaches.", "questions": "1.\n\"Large Language Models (LLMs) are highly capable of performing planning for long-horizon robotics tasks\"\nThis is very arguable. There are evidences that it is not the case. And if it is so, there is no need to write the paper.\n\nSee e.g., \nOn the Planning Abilities of Large Language Models--A Critical Investigation, 2023.\nReasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks, 2023\n\n2.\n\"Language models can leverage internet scale knowledge to break down long-horizon tasks (Ahn et al., 2022; Huang et al., 2022a) into achievable sub-goals\"\nHow valid is such claim? Why is it so? What if the tasks are not available or not frequent in Internet texts?\n\nHow can we guarantee the decomposition of tasks always work?\nWhat if it does not work?\nHow can we guarantee the optimality of the decomposition of tasks?\nWhat if it is not optimal?\nThe current work uses simulations to validate the proposed method. There will be sim2real gap. How to bridge such gap? \n\nHow to improve the current work? If there is something wrong in the task decomposition stage, it is hard or impossible to make improvements, and a pre-trained or fine-tuned LM may be called for. It is beyond the current work. However, the point is, it is not clear how the proposed method deal with such issues. \n\n\n3.\nEnd-to-end vs hierarchical approaches, there are tradeoffs. The paper focuses on the advantages of hierarchical approaches and disadvantages of end-to-end approaches. Desirable to discuss from both sides.\n\n4.\n\"This simplifies the training setup and allowing the agent to account for future decisions as well as inaccuracies in the Sequencing Module.\" \nFor some mistakes at the higher level, a lower level RL can not deal with.\n\n5.\nTable 2, Multistage (Long-horizon) results. 5 stages are not quite long-horizon, and the success rate may be as low as .67 ± .22", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "originality\nThe authors propose PSL: 1) breaks up the task into sub-sequences (Plan), 2) uses vision and motion planning to translate sub-sequences into initialization regions (Seq), 3) train local control policies using RL (Learn).\n\nquality\nExperiments show that the proposed method outperforms previous methods in simulation.\n\n\nclarity\nThe paper is basically well-organized and clearly written.\n\nsignificance\nAs an LLM-based approach, the authors have made some progress.", "weaknesses": "It is a paper about robotics. However, experiments are based on simulations only.\n\nIt is about long horizon robotics tasks. However, the largest number of stages is 5 in the experiments. \n\nFrom the perspective of long horizon robotics tasks, it is not clear how the method may proceed forward. See detail below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)