Interactive Task Planning with Language Models
Authors: Boyi Li
Philipp Wu
Pieter Abbeel
Jitendra Malik
Venue: Accepted by TMLR ()

Abstract: An interactive robot framework accomplishes long-horizon task planning and can easily generalize to new goals or distinct tasks, even during execution. However, most traditional methods require predefined module design, which makes it hard to generalize to different goals. Recent large language model based approaches can allow for more open-ended planning but often require heavy prompt engineering or domain specific pretrained models. To tackle this, we propose a simple framework that achieves interactive task planning with language models by incorporating both high-level planning and low-level skill execution through function calling, leveraging pretrained vision models to ground the scene in language. We verify the robustness of our system on the real world task of making milk tea drinks. Our system is able to generate novel high-level instructions for unseen objectives and successfully accomplishes user tasks. Furthermore, when the user sends a new request, our system is able to replan accordingly with precision based on the new request, task guidelines and previously executed steps. Our approach is easy to adapt to different tasks by merely substituting the task guidelines, without the need for additional complex prompt engineering.

Published in Transactions on Machine Learning Research (01/2025)
Interactive Task Planning with Language Models
Boyi Li* UC Berkeley
boyili@eerkeeeeyccom edu
Philipp Wu* UC Berkeley
philippow..eerkeee..ccm edu
Picter Abbeel UC Berkeley
pubbecl@übrkeleyy.coo edu
Jitendra Malik
melkkeeeecs.brreeeee....o edu
UC Berkeley
Reviewed on OpenReview: https: openreview. net/ forum? id= GANGN tRIo
Abstract
An interactive robot. framework accomplishes long-horizon lask planning and can easily generalize new goals and distinct tasks, even during execution. However, most traditional methods require predefined module design, making it hard to generalize to different goals. Recent large language model based approaches can allow for more open-ended planning but often require heavy prompt engineering domain specific pretrained models. To tackle this, we propose simple framework that achieves interactive task planning with language models by incorporating both high-level planning and low-level skill exccution through function calling, leveraging pretrained vision models to ground the scene language. We verily the robustness of our system on the real world task of making milk tea drinks. Our system is able to generate novel high-level instructions for unseen objectives and successfully accomplishes user tasks. Furthermore, when the user sends new request, our system is able to replan accordingly with precision based on the new request, task guidelines and previously executed steps. Our approach is easy to adapt to different tasks by simply substituting the task guidelines, without the need for additional complex prompt engineering. Please check more details on our Project Page and Demo Video
1 INTRODUCTION
The rise of Large Language Models (LLMs) and proliferation of chatbots highlight the importance of human interaction in Al systems. Beyond merely executing user commands, an autonomous agent should fluidly receive and incorporate feedback at any step during the execution process. Consider the seemingly forward human task of preparing a flavorful milk tea drink, which we study in this work. Such a task, while simple humans, requires robot agent to decompose into numerous intermodiate steps. Not only does the robot need generate and execute the steps precisely, but the robot should also remain receptive to real-time modifications feedback to the initial request. For example, the user might request some boba to be added to their drink. A robot should be able to scamlessly incorporate such interaction during operation.
In light of these challenges, we propose a simple framework for Interactive Task Planning with language models, denoted as ITP. Our framework leverages LLMs to plan, execute, and adapt to user inputs throughout the task lifecyele. Figure 1 illlustrates an exemplary interaction with our system. Our primary objective is offer a blueprint for deploying real-wor wolldorld robotic systems that harness pretrain language models to coordinate the execution of lower-level skills of robot. in simple manner.

---

Published in Transactions on Machine Learning Research (01/2025)
May I have cup of with taro?
ITP LLM-based Planning Module
Plan
Replan
Sorry, boba milk instead
Task Guidelines (only contain few examples)
Perception feedback
Human in-the-loop feedback
High-level Plan
High-level Plan
Step get empty and put it on the table Step2: add taro Into empty cup Step3, 4:
Step-a-get-o--emppy and-put-iit sere--uu-i------eeee ah-h---b----e Step2: boba into the empty cup Step3, 4:
Execution feedback
grosp_ cup() place_ cup() scoop()
Low-level
Perception feedback
policy execution
An overview of ITP. ITP generates high-level plans and executes the low-level robot skills through LLMs. Our system generates high-level plan based
Figure 1:
requests and task guidelines. When the system is interrupted with new request, the system will replan, laking into consideration prior completed steps and task guidelines. Each step of the plan is executed by leveraging an LLM executor, equipped with additional visual grounded outputs, to call lower-level skills. the example shown, the user first requests 'May I have a cup of milk with taro?', a request for which the high-level plan is not provided in the task guidelines. After the robot has finished the first step, the user wants revise the order to a boba milk. Our system able replan and make new set of high-level steps based on the new request, history of completed steps, and task guidelines, which can then be completed by the lower level exccution module.
In this work, we utilize GPT-4 (OpenAl 2023) as the language model backbone. ITP consists of two primary modules. First, high level planner, which takes as input prompt and user request to specify the task and outputs step by step plan. Second, low level executor, which trics to achieve given step by converting robots skills into functional API, which enables GPT-4's function calling capabilities to directly interact with the robot, abstracting code level details from the system. ITP does not require the training of additional value functions such as SayCan (Zeng et al. prompts such as Code as Policies dynamically generates novel plans and re-adjusts
Huang et al., 2023). and does not require code level et al. 2023 or ProgPrompt (Liang et al. 2022). Furthermore, ITP
2022
plan based on user input. We hope our framework will be useful for accomplishing wide range of interactive robot tasks and will release our codebase foster advancements in this field. We outline the key features of ITP below:
ITP a training-fres free robotic system for interactive task planning with language models with a focus on simplicity. We showcase ITP the context of real-world boba drink-making robot that integrates planning, vision and skill exccution.
ITP leverages a simple prompt format, which we show effective across simulated and real settings. Additionally, our ITP system converts the lower-level skills into functional languago-basod API that can be leveraged by any function calling LLM (ic. GPT-4). This enables a user to prompt the system through natural language rather than code, removing the need for code-level prompt engineering.
2

---

Published in Transactions on Machine Learning Research (01/2025)
3.
Our system exhibits robustness in adapting to user requests during execution, allowing it to consider the updated goals, previously completed steps, and task guidelines in order to replan new steps.
2
RELATED WORK
2.1 Task planning
Task planning, the problem of developing plan to achiove desired goal, is integral component of our work. Traditionally, task planning in the robotics commonly leverages symbolic planners which reduces the planning problem into a search problem (Ghallab et al.
2004
Bonet & Geffner 2001). Practitioners often
define the problem in a declarative language (Jiang et al. 2019
Ghallab et al., 1998 Lifschitz 2008 Fikes &
Nilsson 1971). which can be restrictive as it requires meticulous definitions of the problem parameters, such as actions, their preconditions and their effects. Task and motion planning (TAMP), takes task planning a step further and also jointly considers the lower level exccution during higher level planning (Garrett et al., 2020a Mansouri et al. 2021 TAMP methods also consider symbolic representations and leverage search algorithms to extract the final sequence of lower-level primitives and has seen success in robotic manipulation
Siméon et al.
2004
Garrett et al. 2017 2020b). As the search space can often be prohibitively large, some
methods leverage hierarchy and/or or sampling (Bacchus & Yang 1991 Plaku & Hager 2010: Kaelbling &
Lozano-Pérez 2011
Kaelbling & Lozano-Péres 2013). Our approach replaces traditional planning pipelines
with LLMs, offering common-sense reasoning, enhanced interaction capabilities, and the ability to define the
problem's scope using natural language.
2.2 Language Models Planners in Robotics
Due to the popularity of LLMs, there has been rising interest in leveraging LLMs as policy in robot systems. One work in this direction leverages LLMs as zero-shot planners in simulated embodied settings (Huang et al. 2022 by converting the scene and task definitions into language, then letting the LLM directly predict actions. Works such as Socratic models, SayCan, and Grounded decoding (Zeng et al.] 2022 Ahn et al.] 2022 Huang et al. (023) follow in this line of work, coordinating many large pre-trained models with robot to solve various tasks. In contrast to approaches like SayCan (Zeng et al. 2022), which necessitate pretrained value function to ground actions, we rely on prompting the language model with task guidelines and robot skills. This implicitly encodes preconditions and effects, reminiscent of traditional declarative task planning approaches but can be done so with natural language, which is more expressive and easier for the average user to tunc. G-PlanET [Lin ct al. 2023) explores using language models for robot task planning by generating high-level subgoals in simulated envirouments. Tidy bot (Wu et al. 2023 shows that LLMs can help a robot follow a user's preferences based on a few examples. We also prompt the model with a small set. of examples but explore generalization new goals. Reflect (Liu al. (022bb) uses large models to make an agent recount their experiences and correct failures. LLMs have also been used to allow robots to seek help
when uncertain (Ren et al. 2023).
A related approach, used in Code as Policies (Liang et al. 2022) and ProgPrompt (Singh et al. 2023 ) leverages the code writing capabilities of LLMs to generate code that a robot agent can execute directly. This often requires heavy prompt engineering of example code to show the model how to properly use the provided functions to accomplish a directive. Language-guided Robot Skill Learning (Ha et al. (023), like us, takes a hierarchical approach to LLM planning, but assumes access to the simulator which provides ground truth state information. Voyager (Wang et al. 2023) uses LLMs to build lifelong learning agent for Minecraft by having the agent explore and solve new tasks through writing code that interacts with the APT.
Our work falls inlo this general category of leveraging LLMs to plan, and then execute actions in the environment. In contrast to prior work, we allow the LLM to generate a high-level plan based on contextual information. These low-level plans are then exccuted directly by an LLM with access to the functional API of the robot using pre-trained VLM to ground the visual scene into primitives. Our work focuses on how instantiate such a system in the real world.
3

---

Published in Transactions on Machine Learning Research (01/2025)
Options:
Pure milk, Strawberry milk,
Instructiona:
Pure milk
Material: milk
Steps:
0) get enpty cup bring
working area
1) pour the milk into the working cup
2) put the working cup
location
strasberry milk
Material: stremberry jan, nilk
Steps:
0) get an eupty cup and bring it to the working area
1) strawberry jan to the working cup
2) pour the milk into the working cup
3) put the working cup
finished
Boba milk
Naterial: bobs, nilk
Steps:
0) get an enpty cup and bring it to the working area working cup 2) pour the milk into the working cup
1)
3) put the working cup
rinished location
Available material have
boba, strawberry jam, nango jan, natcha powder, taro, bluberryy
Task Guidelines 1:
The task guidelines we use for our drink making experiments. Task guidelines only need contain simple; human interpretable few shot examples and description of relevant assets in the scene.
3 METHOD
ITP offers a blend of high-level planning and low-level execution, powered by LLMs. In contrast to prior work (Liang et al. 2022 Singh et al. 2023), our approach enables the LLM to create a high-level plan informed by contextual information in the form of list of steps. Each step of this plan subsequently realized by another L.M with access to the functional API of the robot. A pre-trained VLM grounds the visual scene into language. Our work focuses how to instantiate such a system in the real world. Our framework, shown in Figure 1 with more detailed breakdown in Figure 3 consists of a hierarchy of two levels, the high level and low level.
3.1 High-level Planning
language model, one of the most capable LLMs available at the time of this writing. The high-level planner takes as input given prompt, task guidelines, and & user request, and outputs a step-by-step plan to execute the request. It also retains past. user interactions for any necessary replanning.
LLMs for Planning. We utilize GPT-4 OOpenAl 2023
Simple Prompting Strategy. Task guidelines, deseribed using natural language, outline the scope of the robot's tasks and are provided to the high-level planner. The prompt contains the user's request and task guidelines which contain few-shot prompt examples of plans in the given domain of interest, and description of the available materials to the robot. In our milk tea system, the task guidelines consist of select set of menu items, their corresponding preparation steps, and list of relevant ingredients. This includes the procedures for a few drinks like 'pure milk' and 'boba milk'. See Task Guidelines I for the exact task guidelines we used in our experiments. Our system utilizes these guidelines to determine the fcasibility of making a new drink based on available materials. Leveraging LLMs' few-shot learning capabilities (Brown) let al.] 2020), ITP can generalize from the baseline guidelines to make detailed steps for other drinks such 'boba strawberry milk' or 'taro milk'
3.2 Low-level Execution
The low-level exccutor takes each generated step and does its best to complete successfully, conditioned on additional information about the scene and available robot skills. We use pretrained vision modules convert the scene into a language compatible format. Additionally, translate robot skills into functional
4

---

Published in Transactions on Machine Learning Research (01/2025)
API, automatically translating the python docstring of the robot skills into callable functions by the language model.
Visual Scene Grounding. The role of the vision module in our system is to process the camera inputs into concise language descriptions of the scene, which can further be processed for planning and task execution downstream. In our drink-making system, the visual grounding system accepts list. of menu items and generates corresponding bounding boxes. Using simple projective mapping, we then approximate the æ and y locations of each item in the robot frame. We employ the pretrained VLM: Grounde--DINO (Liu al. 2023a),, variant of the original DINO model (Caron et al. 2021) fine-tuned for extracting 2D bounding boxes given language descriptions. The final text description is represented as dictionary of object description to (x, y) location. The vision system gives a holistic "understanding" of the scene, despite the location assigmments being imprecise.
Robot Skill Grounding. The language model interfaces with a predefined skill in Python that controls the robot. These skills translated into a functional API by parsing of function definitions and related doc strings. This be directly used with GPT's function-calling layer (OpenAl 2023). In contrast to system does not require examples function internals
methods like ProgPrompt Code Policies, when prompting the LLM. Instead, more detailed prompting of the language model can be specified via natural language in the documentation of the functions.
We adopt Grounded-DINO for capturing the general location of each object
New user request: May change laro boba milk?
Initialization Stage
High-level step1: grasp the Low-level action:
High-level step1: grasp empty (then, we raplan for new request) Low-level action: place_ cup()
High-evee step2: add boba into the cup Low-level action: scoap_ boba to to_location()
request: sarjemmptyyppee harpyy_c___fooee from sssack())k)
grasp_emply_se
High-level slep2: add boba into the cup Low-leval action: scoop_ boba, to location() (during exacution)
High-level step cup Low-level action: grasp_cup(). pour(). place_cuph cup()
High-level step cup Low-level action: grasp_cup(). pourt). place_cup()
High-level place cup the final workapace of the table Low-level action: grasp_cup(). place_ cupp
In crasp_cup() we use Grounded-DINO to locate the objects to grasp accurately
An example of ITP make a cup of taro milk with boba. Our system first makes a high-level plan based on the users request using GPT-4: step 1) grasp the empty cup, step 2) add taro into the cup, step 3) add milk into the cup, step 4) place the cup in the final workspace. For each step in the high-level plan, we foed step into another instance of GPT-4 and obtain the corresponding low-level actions which is directly executed on the robot. As for the perception component, ITP uses Grounded-DINO to capture the general location of each object and locate the object accurately when taking the actions. However, after grasping the empty cup, the user sends new request 'May I change to taro boba milk?'. Considering the history of completed steps, the system replans and generates the following high-level steps and low-level executions. The following plan has been changed to: step 2) add boba into the cup, step 3) add taro into the cup, step 4) add milk into the cup, step 5) place the cup in the final workspace.
Figure 2:
5

---

Published in Transactions on Machine Learning Research (01/2023)
Plan
The uvrr han no.ered
Gompleted Staps
Groundes-DNg DING
dept
Lav propp
nut Mi
Scane Dencfppiio
Highiivel Pian
preer, 201 k2mm-1
de me
Dao Tie vaa au
comple
"auavi washila and 9033,
13.27 4005
Prn Zel in secord the
Mayid pui Baiing aill van Seeciion
eneg
"Wrsted ooalers 8-15
SatA.
"hat los tie stips skior anla ane
GPT4
GPT-4
Replan
Promsi
Low-lavel
oe orferee segty can fen sectbel in firm fort Tyglihook vit
Sep anaanna os
comp ded, piae))
Hightlass lefter eeplanmingi
Foet:
Soda
Hokurn Modli the gau
exar oor after feec aeach Than bather
ska the inte seited ocilos
otbotto yo conogeet
HISUT 55000)
F11 Cel fist
Pirn Le aidd whhre Sar ase
SelA.
Execcate nest steps.....
lot welers
sige eie any camp aldf aboudde anied from the man
ALSAAO (ELTA_))
Scatton arah impil cap.
Figure 3: Detailed diagram of ITP with specific examples of user requests and task guidelines and replanning. During Plan": we feed user requests and task guidelines to complete the prompt and input it into GPT-4 to obtain a high-level plan. We input the history of completed steps and next step to prompt into the lower level executor to call the corresponding low-level actions. Once the lower level executor completes a step, we will maintain the history by storing it into Completed Steps. GPT-4 directly makes function calls to a predefined robot skill library (which could be learned handerafted). During " "Replan:: we feed the completed steps and request to create a prompt, we append this new prompt to the previous conversation context and input the whole message into GPT-4 to obtain a new high-level plan. We refer this procedure as replanning, which previous language-basee task planning methods have not considered. The low level executor then completes the next steps based on the new high-level plan.
3.3 Replanning
Beyond the aforementioned components, ITP considers new requests from the user as human-in-the-lle feedback. We allow a human to interpret the robot execution at any stage with a new prompt. This then triggers our replanning pipeline. The system will consider completed steps, task guidelines, the new request, and the chat history to generate new plan. The details of replanning are in Figure 3) We also showcase
ITP's adeptness in planning and adaptive replanning of the same example in Figure 2
4 EXPERIMENTS
4.1 Robot Experiments
In our experiments, we focus on a drink-making system. Within the given scene, the robot is supplied with a set of ingredients that it must combine to produce a specific drink. Our setup also has includes an overhead camera that captures image of the scene. We leverage Grounded-DINO to process the overhead camera images for open vocabulary object detcction and scene understanding.
For the robot, provide predefined set of skills, which include actions like "grasp_eupp "pour', and "scoop_ boba. to location'. The "grasp_ccvv cup" skill is implemented with feedback policy that centers the gripper on the cup, given the approximate location from the scene description, enabling the robot to grasp it reliably. The "pour" skill is designed to accept a location and a descriplive cue of the ingredient being poured. This level of specification enables milk to be poured more than specific flavors. For example, when making a matcha latte, the pour function provided "matcha" or "milk" as input. When the input is " "matcha", the controllable tilt angle will be small, while when the input is " "milk", the controllable tilt angle will be much larger. This ensures that the robot can pour more milk and a bit of matcha liquid.
6

---

Published in Transactions on Machine Learning Research (01/2025)
User Request
Difficulty Level
Code as Policies
ITP
High-level Planning Success
High-level Planning Success
would like to order cup of milk.
Existed
3/3
3/3
I want to order bobe milk.
Existed
2/4
X
4/4
Can have cup of strowberry mälk?
Existed
4/4
4/4
wont a motcha latte.
Zero-shot easy
4/4
4/4
Mag have cup of milk with toro!
Zero-shot easy
3/3
3/3
I wont taro milk with boha.
Zero-shot moderate
3/5
X
5/5
V
Can I get struwberry boba milk?
Zero-shot moderate
3/5
X
5/5
mant to order strowberry matcha milk.
Zero-shot moderate
5/5
5/5
Dd order strowberry matcha milk with boba.
Zero-shot hard
3/6
x
6/6
I would like cup of possion fruit milk.
Unavailable material
X
Total
80%
5/10
100%
10/10
Table 1: Quantitative results with real robots for high-level planning rate and success rate with various user requests. For high-level planning, we extract planning accuracy by dividing the number of successful steps by the total number of steps, shown as 'Successful Steps / Total Steps'. We determine success by whether the robot succossfully accomplishes the task. To calculate the overall high-level planning score, we avorage the performance across all user requests.
4.2 Comparison on Task Planning
We consider Code Policies baseline. Code Policies provides a formulation for language model- generated programs executed real systems by prompting text completion model with code examples. For fair comparison, not only do we provide Code as Policies with the same information as given in ITP in the form of comments, but we also provide an additional 40 lines of code prompts providing example usage, as is done in Code as Policies, For both ITP and Code as Policies, we provide user requests and task guidelines as inputs. The task guidelines include instances, along with their associated high-level planning steps, current. available material, and other tpsk-specific conditions. Our task guidelines are shown in Task Guidelines 1
We evaluate the methods on two criteria: the number of high-level steps correctly generated and whether the real robot successfully finished the task. We send user requests of varying complexity levels, including 'existed', , 'zero-shot easy,' 'zero-shot moderate', 'zero-shot hard' and 'unavailable material. 'Zero-shot' means the instruction for making the corresponding drink is not provided in the task guidelines. 'Unavailable' indicates that we do not have the material for the requested beverage. We show the results in Table [ We could notice that ITP robust. in high-level plan generation and can easily be generalized to novel instructions of unseen drinks or unavailable drinks. For example, the user sends the request *I would like cup of passion fruit milk.' However, passion fruit jam is not available, the system will provide the response 'Passion fruit jam is not available' and stop the program. In comparison, Code as Policies failed to achieve this objective. To understand the failure case of Code as Policies, we provide some observations: 1) when making a cup of milk with boba, the system attempted to scoop boba from the working cup, improperly adhering to the correct usage of the lever-level skill. 2) When the prompt is more complex (9th row), the system adds milk first and then adds the boba, resulting in an incorrect execution order. 3) When the material is not available, it cannot justify that passion fruit doesn't exist. Additionally, since ITP is built based on task guidelines alone, it demands significantly less prompt engincering than Code as Policies, which makes our system very easy to use for various task planning purposes.
4.3 Replan with Human-in-the-loop Feedback
Our system is robust to diverse new requests during exccution. To verify this point, we assess the task replanning performance on real robots in response to a user's new request, referred to as human-in-the-loop feedback. We display the results in Table We notice that ITP demonstrates its capacity to effectively handle a range of new requests, even after progressing through various steps of the task. The last example is of particular note, where ITP adds one step more ('Stir the mixture until the matcha powder is well mixed') before putting the working cup in the finished location. Here the language model assumes the need to stir
7

---

Published in Transactions on Machine Learning Research (01/2025)
User Request
New Roquest
Step
When New Request is Made
1st
2nd
3rd
Can I have a cup of strowberry milk?
I want to add boba into the drink.
4/4
3/3
5/5
I want matcha lotte.
Sorry, wont bola bilk without matcha instend.
3/3
5/5
5/5
May I have a cup of milk with taro?
Can replace the taro with strawberry?
3/3
5/5
5/5
Con I get strawberry bobo milk..
Sorry, can reorder strowberry milk??
3/3
5/5
5/5
A strawberry matcha milk with boba. Can I fust get matcha boba milk and no strawberry?
4/4
5/4
7/6
Table 2: Replanning performance with real robots given human-in-the-loop feedback. After the user sends a request, we interrupt the procedure before different steps (1st, 2nd, and 3rd). Note that our replanning system is robust in handling these new requests. Interestingly, for the last example, after the 2nd and 3rd step, ITP adds one step more ('Stir the mixture until the matcha powder is well mixod") before putting the working cup in the finished location, leading to 5 and 7 steps instead of 4 and 6 steps respectively. We assume this is because GPT-4 assumes matcha powder is hard to mix, while we select water-soluble matcha powder. Including the instruction 'matcha powder is water-soluble" in the task guidelines could address this issue.
Task Description
[A] ProgPrompt
ITP
watch tv
3
0.42 + 0.13
0.83 = 0.06
turn off light
3
=.00
0.75 = 0.00
eat chips on the sofa 5
5
0.40 0.00
0.96 = 0.05
brush teeth
8
0.09
0.86 = 0.12
throw away apple
8
1.00
1.00 0.00
make toast
8
1.00 + 0.00
0.59 = 0.16
put salmon in the fridge
1.00
0.00
1.00 0.00
bring coffeepot and cupcake the coffee table microwaee sulmon wash the plate
8
1.00 + 0.00
1.00 = 0.00
11
0.76 + 0.13
0.89 = 0.09
18
0.97
0.04
0.95 - 0.01
S 5 Avg: 6 _ A
Avg: 0 < |A|
0.61 + 0.29
0.84 = 0.10
10
0.95 0.11
0.90 0.17
Avg: 10 S |A|
S 18
0.87 = 0.14
0.92 = 0.07
Table 3: Comparison of executability (Exee) on Simulation (Virtual Home) with ProgPrompt. Exec is the fraction of actions in the plan that are executable in the environment, even if they are not relevant for the task. ITP is not only user-friendly approach that allows users to provide high-level guidelines, but it also achieve superior results on varied tasks in the simulation.
the matcha due to the ambiguity of the correct procedure. Such superfluous steps can be reduced by adding restrictions in the task guidelines, which can easily be done by a general user of the system. This contrasts with methods like Code as Policies which require tuning prompts at the code level.
4.4 Comparison on Simulation Tasks
In this section, we aim to verify ITP's performance for simulation tasks. We compare our high-level planning module to that of ProgPrompt (Singh et al. 2023) by leveraging the simulated Virtual Home (VH) Environment (uug et all 2018). We would like to emphasize that ITP provides user-friendly approach for users to input their high-level guidelines, which requires little background knowledge, while other approaches such as ProgPrompt employ a code-like prompting strategy. To make a fair comparison, we obtain high-level planning from both ITP and ProgPrompt, and use ProgPromple low-level execution, strictly following the same evaluation protocol: each result is averaged over 5 runs in a single VH Environment across 10 different tasks. We display the results in Table 3. We notice that ITP not only a user-friendi approach that allows users to provide high-level guidelines in more straightforward natural language but is able to match or exceod ProgPrompt on executable functions on the Virtual Home benchmark.
8

---

Published in Transactions on Machine Learning Research (01/2025)
4.5 Adaptation to Other Tasks
ITP is simple adapt to new tasks. The system is principally reliant on task guidelines during high-level planning and predefined functions during low-level execution. This structure negates the need for intricate code implementation examples, making the system's generalization other tasks remarkably straightforward. Refer to Figure 3 for the necessary components that need to be adapted. For adapting to new task, only the Task Guidelines and documentation for the provided Low-level Skills need to be modified. Optionally, the Prompts for the high and low-level planner can also be tuned.
Task Guidelines 2: Task Guidelines for the Dishwashing Task. Again, we follow the simple format from before. Possible task options are provided as well as their high level steps. Additional options and tasks specific details are also provided to help guide the LLM during planning.
Options:
dishxasher cycle is complete and dichvasher has stopped, wait few minutes for the diahea cool down
11)
Wash one plate with rose flavor,
Waah the plates there two plates,
Nash plate and Tork
12) nake sure the plate is clean and dry, othervise go into step 8)
Instructions:
Wash plate with flavor
13) return all clean utensils the finished location
Material: rose datargent
Steps:
one plate
fork
0) grasp the dirty plate
Material: original detergent
1) renove large particle fron the plate
grasp the dirty plate
2) open the dishvssher
1) renove large particle from the plate
3) pull out the rack
2) open the dishvasher
4) put
plate the third rack
3) pull out
rack
5) add rose detergent into the detergent diapenser
4) put the plate on the third rack grasp
6)
the disheaster
7) select the cycle and atart diahvasher
6) ranava large particle from the fork
8) after the diehvasher cycle complete and the
7) put the fork on the first rack
dishvasher has stopped, wait fex minutes for the
die8) add original detergent into the detergent dispenser
to cool down
9) nake
the plata clean and dry, otherwise
close the dishraster
go into step 8)
10) select the cycle and start dishvasher
10) return
clean plate to finished location
dishxasher cycle is complate dishvasher has atopped, vait few minutes for the diches to cool down
11)
Wash all the plates and there two plates
Material: original detergent
0) grasp the first dirty plate
the plate and fork clean and dry, otherwise go into step 8)
12) make
1) ranove large particle from the plate dishvasher
2) open
13) return
utensils
finished
3) pull out the rack
location
4) put the plate
third rack 5) graap the second dirty plate
location se now: first rack for forks and sxall kitchen utensils
6) zenove large particle from the plate
second rack for boul/cup
7) put the plate the third rack
third rack plate/big kitchen utensils Available naterial wa have noz: detergent, original detergent
8) add original datargant into the datargent
dispenser
9) close the dishvaster
10) solect cycle start dishsasher
We provide an additional example of adapting to additional task with ITP. We adapt our system to study the high-level task planning capabilities of the task dishwashing. We simply replace the task guidelines for 'making a drink' with 'dishwashing' We show the dishwashing task guidelines below in Task Guidelines 4.5
We evaluate the high-level planning capability on how many steps are generated correctly. We show our results in Table We find out that ITP performs very well on the novel dishwashing task. It has the capability not only produce precise and novel instructions for new objectives but also to exhibit resilience when faced with entirely different tasks.
4.6 Analysis and Discussion
A Analysis. Although ITP demonstrates great generalization ability, relies solely on LLMs for replanning. As a result, errors from the recognition system or robot execution could lead to task failure. For instance, if the recognition system mistakenly identifies milk as 'taro jam', the robot might prepare the wrong drink by using the incorrect ingredient. Similarly, if line becomes entangled around the robot arm and knocks over a cup containing liquid, the task could fail.
9

---

Published in Transactions on Machine Learning Research (01/2025)
User Request
Task Type
High-level Planning
Wash one dirby plate with rose flavor.
Existed
11/11
Please wash 1 dirly bowl with rose flavor.
Zero-shot easy
11/11
Please clean the 2 dirty cups. Wash all forks, there are 3.
Zero-shol easy
14/14
Zero-shot easy
17/17
Can you wash 2 plates? (New request: Can you wash another?)
Zero-shot easy
17/17
Please wesh 2 forks and one bowl.
Zero-shot moderate
17/17
May you wash 2 cups and 2 plates?
Zero-shot moderate
20/20
Please wash 2 fork, 2 plate and 2 bowl.
Zero-shot hard
27/27
Wash plates, 1 bowrl fork and 1 knoje with rose flavor.
Zero-shot hard
23/23
Wash one dirty plate with lemon flavor
Unavailable material
Total
100%
Table 4: Generalization to dishwashing task. We only need to change the text guidelines make an accurate high-level plan. Since using the dishwasher to clean the dishes doesn't contain misleading material or content, the high-level planning rate is 100%. Please note that different utensils should be placed in different locations in the dishwasher, while ITP remains resilient in generating precise plans for each step, ensuring the correct order and appropriate location for different utensils. We envision the versatility of ITP's capabilitics being applicable to a wide range of lasks.
To address these challenges, we can design a replanning system using rision-language models (VLMs). VLMs can detect potential dangors in the system and generate actions to mitigate these issucs. Furthermore, due to the latency of several seconds when using GPT-4, tasks in our experiments sometimes take longer to complete. However, this issue can be alleviated by adopting open-source models, such as Llaaa (Dubey et al. 2024), as advancements in these techniques are progressing rapidly.
Discussion. Although many works (Singh et al. 2023 Liang al. 2022 Skreta al. 2023 Rana al.
2023) have explored LLMs for understanding feedback and planning, none of these works both consider user-friendly task guidelines and replan based on a user's new request. Progprompt (Singh et al. 2023) and Code as Policies focus more on making one plan and executing the task step by step, while also requiring complicated reference code. CLAIRIFY (Skreta et al. 2023 provides effective guidance to the language model by generating structured task plans and incorporating any errors as foodback, and SayPlan (Rana et al.) 2023 introduces an ilerative replanning pipeline that refines the initial plan using feedback from scene graph simulator. However, these two approaches pay attention more
task instead of users' experience. Our ITP aims to provide unified vision and language framework that can provide the best user experience, so the user with minimal specialized knowledge can still casily ask their robots to execute a task.
5 Conclusion
Conclusion. In this paper, we propose a simple yet effective system, ITP, which melds the capabilities of Large Language Models in an interactive robot system that constructs plans, and performs tasks centered around the users needs. Encouragingly, it precisely interprets user requests, generates pertinent step-by- by-step plans, and achieves the desired outcome - a testament to the potential of such systems for real-world applications. We embody our system in a robot designed make various drinks according to user preferences and adeptly demonstrate its ability to respond to feedback during execution. Our system is capable in the context of interactive task planning and replanning for robotics.
Limitations and Future Work. While ITP provides a working proof of concept of an interactive robot system, there is room for enhancing its capabilities with more powerful robot skills to tackle more intricate tasks. Similarly, the integration of more precise visual information that leverages 3D information would significantly elevate the robot's proficiency in understanding, planning, and interacting with its surroundings. We aim for our open-source system to inspire more research into using both established and emerging models to enhance real-world roboties.
10

---

Published in Transactions on Machine Learning Research (01/2025)
6 Acknowledgement
Philipp Wu was supported in part by the NSF Graduate Rescarch Fellowship Program. We thank the Machine Common Sense project and ONR MURI award number N00014-21-1-2801. We thank Valts Blukis for the insightful and valuable discussion. Pieter Abbeel holds concurrent appointments as a Professor at UC Berkeley and as an Amazon Scholar. This paper describes work performed at UC Berkeley and is not associated with Amazon.
11

---

Published in Transactions on Machine Learning Research (01/2025)
References
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang-Huei Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambaa, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan, and Andy Zeng. Do as I can, not as I say: Grounding language in robotic affordances. April 2022.
Fahiem Bacchus and Qiang Yang. The downward refinement property. In Proceedings of the 12th International Joint Conference on Artificial Intelligence Volume 1, IJCAI91, pp. 286-292, San Francisco, CA, USA, 1991. Morgan Kaufmann Publishers Inc. ISBN 1558601600.
Blai Bonet and Héctor Geffner. Planning as heuristic search. Artificial Intelligence, 129(1):5 33, 2001.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Nya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference on Compuler Vision (ICCV), 2021.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. orXiv preprint 2xXx:2207.2178. 2024.
Richard F. Fikes and Nils J. Nilsson. Strips: new approach to the application of theorem proving to problem solving. In Proceedings of the 2nd International Joint Conference on Artificial Intelligence, IJCA1'71, pp. 608 620, San Francisco, CA, USA, 1971. Morgan Kaufmann Publishers Inc.
Caclan Reed Garrett, Tomá s Lózano-Pérez, and Loslic Pack Kaelbling. FFRob:
Leveraging symbolic planning for efficient task and motion planning. The International Journal of Robolics Research, 37(1):104 13610066-1666611 nov 2017. doi: 10.1177/02783649977733
URL htttps://dol.org/ Ctt.://tg//.oo.oo///////0111/010 17X7777777777733973333
Caelan Reed Garrett, Rohan Chitnis, Rachel Holladay, Beomjoon Kim, Tom Silver, Leslie Pack Kaelbling, and Tomás Lozano-Pérez. Integrated task and motion planning, 2020a.
Caelan Reed Garrett, Tomás Lozano-Pérez, and Leslie Pack Kaelbling Pddlstream: Integrating symbolic planners and blackbox samplers via optimistic adaptive planning, 2020b.
M. Ghallab, A. Howe, C. Knoblock, D. Medermott, A. Ram, M. Veloso, D. Weld, and D. Wilkins. PDDL The Planning Domain Definition Language, 1998. URL http://citeseerx. ist ist.is...t........... wwtp:eeddvvvvveeeeee summaryy doi=10. 1.1.1.1.7.7. 1..7.222
Malik Ghallab, Dana Nau, and Paolo Traverso. Automated Planning: Theory and Practice. The Morgan
Kaufmann Series in Artificial Intelligence. Morgan Kanfmann, Amsterdam, 2004. ISBN 978-1-55800-856-6
URL http:://ww.tt. http:p//wwww..iieeedrrettttt www.sscciienceetecceceeeee
Huy Ha, Pete Florence, and Shuran Song. Scaling up and distilling down: Language-guided robot skill acquisition. July 2023.
Wenlong Huang, Pieter Abbecl, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. January 2022.
12

---

Published in Transactions on Machine Learning Research (01/2025)
Wenlong Huang, Fei Xia, Dhruv Shah, Danny Driess, Andy Zeng, Yao Lu, Pete Florence, Igor Mordatch, Sergey Levine, Karol Hausman, and Brian Tchter. Grounded decoding: Guiding text. generation with grounded models for robot control. March 2023.
Yuqian Jiang, Shiqi Zhang, Piyush Khandelwal, and Peter Stone. Task planning in robotics: an empirical comparison pddl-based and asp-based systems, 2019.
Leslie Pack Kaelbling and Tomás Lozano-Péree Integrated task and motion planning in belief space. Inl. J. Rob. Res., 32(9- 10):1194 -2219, aug 2013. ISSN 0278-3649. doi: 10.1777/027886499144550777 URL
attps://dol 1ttp1000.1777007734411448600
Leslie Pack Kaelbling and Tomás Lozano-Pérez. Hierarchical task and motion planning in the now. In 2011 IEEE International Conference on Robotics and Automation, pp. 1470-1477, 2011. doi: 10.1109/ICRA.
2011.5980391.
Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. September 2022.
Vladimir Lifschitz. What is answer set programming? In Proceedings of the 23rd National Conference on Artificial Intelligence- - Volume 3, AAAI008, pp. 1594 1597. AAAI Press, 2008. ISBN 9781577353683.
Bill Yuchen Lin, Chengsong Huang, Qian Liu, Wenda Gu, Sam Sommerer, and Xiang Ren. On grounded planning for embodied tasks with language models. Proceedings of the AAAI Conference on Artificial Intelligence, 37(11):13192 13200, June 2023. ISSN 2159-5399. doi: 10.1609 vat.aa..7:...265.... URL
http://áx.doo org/ 1609/119aa0 aaai v37111. 26549
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2903.54409 2023a.
Zeyi Liu, Arpit Bahety, and Shuran Song. Reflect: Summarizing robot experiences for failure explanation and correction. arXiv preprint arXiv:2806.1577 15724, 2023b.
Masoumeh Mansouri, Federico Pecora, and Peter Schüller. Combining task and motion planning: Challenges and guidelines. Frontiers in Robotics and AT, 8, 2021. ISSN 2296-9144. doi: 10.3389/frobt.22 URL httpp:://ww https::/www.frrotterrshhh
frontiersin, wwwp/rraaaeee/////ooooo. 3389/frobt 2021. 637888
OpenAL. Gpt-4 technical report, 2023.
Erion Plaku and Gregory D. Hager. Sampling-bass based motion and symbolic action planning with geometric and differential constraints. In 2010 IEEF Internalional Conference Robolics and Aulomation, pp- 5002-5008, 2010. doi: ww0p//OBBT.210000
Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba. Virtualhome: Simulating household activities via programs. In Proceedings of the IEEE Conference on Computer Vision and Paltern Recognition, pp. 4494-5502 2018.
Krishan Rana, Jesse Haviland, Sourav Garg, Jad Abou-Chakra, Tan Reid, and Niko Suenderhauf. Sayplan: Grounding large language models using 3d scene graphs for scalable task planning. arXiv preprint arXiu:2307.06193 2023.
Allen Z Ren, Anushri Dixit, Alexandra Bodrova, Sumeet Singh, Stephen Tu, Noah Brown, Peng Xu, Leila Takayama, Fei Xia, Jake Varley, Zhenjia Xu, Dorsa Sadigh, Andy Zeng, and Anirudha Majumdar. Robots that ask for help: Uncertainty alignment for large language model planners. July 2023.
Thierry Siméon, Jean-Paul Laumond, Juan Cortés, and Anis Sahbani. Manipulation planning with probabilistic roadmaps. The International Journal of Robotics Research, 23(7-8):729-746 2004. doi: 10.1177/022836440444467777 URL httts:///aaat wttp://..11777777777690066636650
13

---

Published in Transactions on Machine Learning Research (01/2025)
Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. Progprompt: Generating situated robot task plans using large language models. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 11523 11530. IEEE, 2023.
Marta Skreta, Naruki Yoshikawa, Scbastian Arellan-Rubach, Zhi Ji, Lasse Bjørn Kristensen, Kourosh Darvish, Alán Aspuru-Guzik, Florian Shkurti, and Animesh Garg, Errors are useful prompts: Instruction guided task programming with verifier-assisted iterative prompting, orXie preprint wrXiv:2303.14100 2023.
Guanzhi Wang, Yugi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXi:: Araiv- Araio-2305.16291, 2023.
Jimmy Wu, Rika Antonova, Adam Kan, Marion Lepert, Andy Zeng, Shuran Song, Jeannette Bohg, Szymon Rusinkiewicz, and Thomas Funkhouser. TidyBot: Personalized robot assistance with large language models. May 2023.
Andy Zeng, Maria Attarian, Brian Ichter, Krzysztof Choromanski, Adrian Wong, Stefan Welker, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, and Pete Florence. Socratic models: Composing zero-shot multimodal reasoning with language. April 2022.
14


Reviews:
[{"review_id": "EpNcc4qkFp", "reviewer": "Reviewer_VzSU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n-- Key difference from prior work: ITP allows users to provide instructions in a way that's easy to understand and also adjust its plan if the user makes a new request.\n\n-- Their experiments where ITP is implemented on a robot designed to make a variety of drinks based on user requests. They effectively showcased ITPs ability to adapt to feedback and make changes while preparing the drinks.\n\n\nWeakness / Questions:\n\n-- What happens when there is a discrepancy between the task guideline and the scene? How does ITP handles it, is it in the first stage or the second?\n\n-- How does error in DINO-bounding box, perception of location and error from GPT in planning compound i.e. are there scenarios where the errors add up on each other failing the task? A better error analysis of when failures occur and what to do for those cases will be helpful\n\n-- If I understand correctly the task guidelines are complete and the scene only reflects the scenarios mentioned in the task guidelines. I am unsure about the generalization capabilities of the method. In the paper, generalization to new goals doesn’t necessarily mean to goals that are not explained in the guidelines. Please correct me if I am wrong.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This study employs a large language model (LLM) as the foundation for its language processing. The proposed system, ITP, comprises two main components: a high-level planner and a low-level executor. The high-level planner takes as input a given prompt, task guidelines, and a user request, and outputs a step-by-step plan to execute the request. A low level executor converts the given step using LLM function calling capabilities to directly interact with the robot, abstracting code level details from the system. The paper also showcases ITP’s ability by embodying a robot to make drinks according to user preferences and requests.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The main request will be a better error analysis section where the authors can discern failure cases and why failure occurred. Which part of planning and executor mainly is responsible and possible path to improve it. This is not critical to the paper being accepted but it is also not just a nice to have. This is something that will improve the paper for me. It is something that brings the paper from a borderline/weak accept to accept category.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "O163voipA3", "reviewer": "Reviewer_LFrR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n1.    ITP offers a real world system that has the ability of adaptive planning by means of interaction with the user. A  first plug-and-play LLM lists each step of a proposed plan. Each step of this plan is subsequently realized by another LLM with access to the functional API of the robot. A pre-trained VLM grounds the visual scene into language. \n2.   The high level planner is technically the most straightforward approach which still lends itself to generalization - which is great to see functional in a real world system!\n3.   The vision system design offers “a holistic ‘understanding’ of the scene, despite the location assignments being imprecise”.  Which potentially allows room for generalization.\n4.   I believe the core strength and novelty in ITP is to consider new requests from the user as human-in-the-loop feedback wherein they allow humans to interpret the robot execution at any stage with a new prompt, which triggers the replanning pipeline which also considers the previous attempts etc.\n5.   The method is compared with two baselines Code as Policies and ProgPrompt where the baseline is provided with the task guidelines same as IPT that include multiple instances, along with their associated high-level planning steps, current available material, and other task-specific conditions.\n\n\n**Weaknesses:**\n1.    Positioning of the work in the literature:  In Related work, authors mention: “In contrast to prior work, we allow the LLM to generate a high-level plan based on contextual information.” This is confusing because prior work also asks LLM to generate high level plans and very much based on contextual information. Can the authors please clarify the positioning of the work further?\n2.   Assumption of a functional API of the robot: I believe this is a strong assumption and might be hard to generalize across different scenarios as that itself poses an open problem?\n3.   The high level planning is basically a few-shot prompting mechanism which is not very novel given the abundance of work present along these lines. \n4.   In this work, the  language model interfaces with a predefined skill set in Python that controls the robot.” What happens when the robot needs to update a skill when a new task is presented? What are the failure modes of this method since it's a training free method and adaptation is primarily by means of iterative prompting?\n5.   In Table 1: when you report “Quantitative results with real robots for high-level planning rate and success rate with various user requests.” How many seeds are these results over? Considering there is stochasticity in the LLM responses, how robust are the plans in meeting success criteria?\n6.   While it is true that,  “since ITP is built based on task guidelines alone, it demands significantly less prompt engineering than Code as Policies, which makes our system very easy to use for various task planning purposes.” isn't the success of this method based on knowing the task ahead of time in the form of task guidelines, which significantly limits scaling this approach?\n7. Lack of clarity on failure modes, and details on real time adaptation- see requested changes.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper addresses the problem of  interactive robot learning for long-horizon task planning that can generalize to new goals or distinct tasks. The paper proposes ITP, where a high level planner takes as input a prompt to specify and a user request to output a step by step plan, followed by  a low level executor which tries to achieve a given step by converting robots skills into a functional API. The paper claims that, when the task changes, the approach is able to be replan accordingly with precision based on the user input, the task guidelines and previously executed steps. Overall the biggest strength of ITP is its capacity to effectively handle a range of new requests, even after progressing through various steps of the task.\n\nOverall, I find the paper worthy of acceptance, and believe its merits outweighs its flaws. It is worthwhile to see the replanning abilities of large models with simply in-context learning. With this I recommend acceptance and encourage the authors to report the robustness in the results (i.e. how many seeds were used to test, what is the standard deviation across seeds?)  Please see weaknesses and requested changes.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Requested changes**\n\n**See from weaknesses:**\n\n5.   In Table 1: when you report “Quantitative results with real robots for high-level planning rate and success rate with various user requests.” How many seeds are these results over? Considering there is stochasticity in the LLM responses, how robust are the plans in meeting success criteria?\n6.   While it is true that,  “since ITP is built based on task guidelines alone, it demands significantly less prompt engineering than Code as Policies, which makes our system very easy to use for various task planning purposes.” isn't the success of this method based on knowing the task ahead of time in the form of task guidelines, which significantly limits scaling this approach?\n\n\n**Questions and clarifications**\n\nWhy do we not have a baseline performance reported in  Table 4: Generalization to dishwashing task. Could the authors please report those as well for baselines. \n\nTask Guidelines 2 are quite comprehensive. What is the performance when you test generalization without providing such detailed task guidelines during zero-shot generalization? \n\nWhat is missing from the manuscript is clarity and more transparency on how much time does it take for the system to replan? Considering there is no fine tuning, how are failure modes handled? Considering LLMs do hallucinate, it would be useful to add these modes in the manuscript.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Y2jerE6861", "reviewer": "Reviewer_aatG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* simple effective system that performs well\n* real world and sim evaluation\n\nWeaknesses:\n* how exactly low level skills are selected and conditioned seems a little unclear. \n* ideally there would be more trials in real to get a sense of the robust of the system\n* More analysis of exactly why this system outperforms baselines would be beneficial.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes an interactive task planning framework leveraging GPT4 and tests it on a real milk tea making tasks as well as a virtual home environment. \n\nGiven a user query, task guidelines giving few show examples and a scene representation coming from a vision language model, the system produces a step by step high level plan. For each step, the high level step is turned into a sequence of low level primitives by a LLM based on the scene representation. Replanning is supported after each step by simply appending the new user query to the interaction history and generating a new high level plan.\n\nThe method is evaluated on a real world milk tea making task and a virtual home environment and outperfoms a code-as-policies baseline both on tasks covered by the guidelines and unseen queries.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Questions:\n* I don't understand how exactly the low level skills are selected? Are these conditioned by the LLM based on the context and scene representation or are they completely fix? This could be clearer in the paper\n* What happens if a new user query requires starting from scratch? What happens if a low level skill fails?\n\nminor:\n* please change to citations in parenthesis: e.g. SayCan Zeng et al. (2022); -> SayCan (Zeng et al. (2022));", "broader_impact_concerns": "none.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The core strength of the paper is we summarized in this review: "to consider new requests from the user as human-in-the-loop feedback wherein they allow humans to interpret the robot execution at any stage with a new prompt, which triggers the replanning pipeline which also considers the previous attempts etc."  The proposed approach develops a practical system that has the ability of adaptive planning by means of interaction with the user.  The method is compared with two baselines: Code as Policies and ProgPrompt, and shows favorable comparisons.

The main shortcomings raised were lack of clarity on failure modes of the system, and somewhat weak novelty on the overall architecture. However, the reviews overall are positive for the contributions of this paper.
Decision: Accept (poster)