Open-Ended Instructable Embodied Agents with Memory-Augmented Large Language Models
Authors: Gabriel Herbert Sarch
Yue Wu
Michael J. Tarr
Katerina Fragkiadaki
Venue: EMNLP 2023 Findings (2023)
Keywords: Task Planning
Embodied AI
LLMs
Robotics
TL;DR: We introduce HELPER, an embodied agent equipped with an external memory of language-program pairs that parses free-form human-robot dialogue into action programs through retrieval-augmented LLM prompting.

Abstract: Pre-trained and frozen LLMs can effectively map simple scene re-arrangement instructions to programs over a robot's visuomotor functions through appropriate few-shot example prompting. To parse open-domain natural language and adapt to a user's idiosyncratic procedures, not known during prompt engineering time, fixed prompts fall short. In this paper, we introduce HELPER, an embodied agent equipped with an external memory of language-program pairs that parses free-form human-robot dialogue into action programs through retrieval-augmented LLM prompting: relevant memories are retrieved based on the current dialogue, instruction, correction or VLM description, and used as in-context prompt examples for LLM querying. The memory is expanded during deployment to include pairs of user's language and action plans, to assist future inferences and personalize them to the user's language and routines. HELPER sets a new state-of-the-art in the TEACh benchmark in both Execution from Dialog History (EDH) and Trajectory from Dialogue (TfD), with 1.7x improvement over the previous SOTA for TfD. Our models, code and video results can be found in our project's website: https://helper-agent-llm.github.io.

Open-Ended Instructable Embodied Agents with Memory-Augmented Large Language Models
Gabriel Sarch
Yue Wu
Katerina Fragkiadaki Carnegie Mellon University
Michael J. Tarr
ywsarch.wus,, mtmm Jeandrew emu, katefecs. cmu.edu
helper-agent-Ilm, Ill.gibhub.io
Abstract
to adapt in a closed-loop execution failures. Large Language Models (LLMs) trained on Internet-scale text parse language instructions to task plans with appropriate plan-like or code-like prompts, without any finetuning of the language model, as shown in recent works (Ahn et al., 2022; Liang et al., 2022; Zeng et al., 2022; Huang et al., 2022b; Singh et al., 2022a). The state the envi- romment is provided as list of objects and their spatial coordinates, or ree-form text descrip- tion from vision-language model (Liang et al., 2022; Liu et al., 2023b; Wu et al., 2023a; Ahn al., 2022). Using LLMs for task planning requires engi- neering a prompt that includes description of the task for the LLM perform, a robot API with func- tion documentation and expressive function names, environment and task instruction inputs, and set of in-context examples for inputs and outputs for the task (Liang et al., 2022). These methods are not trained in the domain of interest; rather they prompt-engineered having in mind the domain hand.
Pre-trained and frozen LLMs can effectively map simple scene re-arrangement instructions programs
robot's visuomotor func- tions through appropriate few-shot example prompting. parse open-domain natural lan- guage and adapt user's idiosyncratic pro- cedures, not known during prompt engineer- ing time, fixed prompts fall short. In this pa- per, we introduce HELPER, an embodied agent equipped with as external memory of language- program pairs that parses free-form human- robot dialogue into action programs through retrieval- aaumenel- LLM prompting: relevant memories retrieved based on the current dialogue, instruction, correction or VL.M de- scription, and used as in-context prompt ex- amples for LLM querying. The memory is expanded during deployment to include pairs of user's language and action plans, assist future inferences and personalize them to the user's language and routines. HELPER sets state-of-the-art in the TEACh benchmark in both Execution from Dialog History (EDH) and Trajectory from Dialogue (TID), with 1.7x improvement over the previous SOTA for TfD. Our models, code and video results can be found our project's website: helper-agent- Ilm. tit.uithub.io.
How can extend LLM-prompting for seman- tic parsing and task plamning to open-domain, free- form instructions, corrections, human-robot dia- logue, and users' idiosyncratic routines, not known at prompt engineering time? The prompts used for the domain of tabletop rearrangement are al- ready approaching the maximum context window of widely used LLMs (Singh et al., 2022a; Liang et al., 2022). Even context window size grows, more prompt examples result in larger attention operations and cause increase in both inference time and resource usage.
1 Introduction
Parsing free-form human instructions and human- robot dialogue into task plans that robot can ex- ecute is challenging due to the open-endedness of environments and procedures to accomplish, and to the diversity and complexity of language humans use to communicate their desires. Human language often contains long- long-term references, questions, er- rors, omissions, references and descriptions of routines specific
To this end, we introduce HELPER (Human- instructable Embodicd Language Parsing via Evolving Routines), model that uses retrieval- augmented situated prompting of LLMs parse free-form dialogue, instructions and corrections from humans and vision-language models to pro- grams over a set of parameterized visuomotor rou-
particular user (Tellex al., 2011; Liang, 2016; Klein and Manning, 2003). In- structions need to be interpreted in the environmen- tal context in which they are issued, and plans need

---

Language-Program Memory
Pathee landio bakey cortuul
Program
Human-Role Dialogue
Instruction Plans
Retrieved Examples & Input
Harger Feee in
How can help you?
burgh Iemter 100
LLM
Make my favorite
Dax sandwich
Poliee progrees
plate?
Yes please
Intrurtion Milie ran nor diyiit with ywu dececie
largrt Lamiee
Corrective Feedback
Corrective Plans
bilcd adkeen ryet lettuee Stock()
You forgot the cheese!
Tythen
Program Execution
Failure Description
Mapping Object Tracking
VLM Freebacc Sornetinn la bucking agent
The
Pythor peagray
Pre- Check
VLM
laptop is
obstructing the path.
elstiacicc hrika Car cindwits Thil.
Object States Quan Viue
Added
Expand Memory
LIM Object Search
API Skills
Figure 1: Open-ended instructable agents with retrieval- augmmeteddd L.Mss. We equip L.LSs with an external memory of language and program pairs to retrieve in-context examples for prompts during LLM querying for task plans. Our model takes input instructions, dialogue segments, corrections and VLM environment descriptions, retrieves relevant memories to use as in-context examples, and prompts LLMs to predict task plans and plan adjustments. Our agent executes the predicted plans from visual input using occupancy and semantic map building, 3D object detection and state tracking, and active exploration using guidance from LLMs' common sense locate objects not present in the maps. Successful programs are added to the memory paired with their language context, allowing for personalized subsequent interactions.
tines. HELPER is equipped with an external non- parametric key-value memory of language pro- gram pairs. HELPER uses its memory to retrieve relevant in-context language and action program examples, and generates prompts tailored the cur- rent language input. HELPER expands its memory with successful executions of user specific proce- dures; then recalls them and adapts them future interactions with the user. HELPER uses pre-trained vision-languaa models (VLMs) to diagnose plan failures in language format, and uses these to re- trieve similar failure cases with solutions from its memory to seed the prompt. To execute pro- gram predicted by the LLM, HELPER combines successful practices of previous home embodied agents, such as semantic and occupancy map build- ing (Chaplot et al., 2020a; Blukis et al., 2022; Min et al., 2021), LLM-based common sense object search (Inoue and Ohashi, 2022), object detection and tracking with off-the-shelf detectors (Chap- lot et al., 2020b), object attribute detection with VLMs (Zhang et al., 2022), and verification of ac- tion preconditions during execution.
in their ability to complete variety of long- horizon household tasks from RGB input given natural language dialogue between commander (the instruction-giving user) and follower (the instruction-sceking user). We achieve new state- of-the-art art in the TEACh Execution from Dialog History and miajectoryffrom-Diaae proving task success by 1.7x and goal-condition success by 2. 1x compared to prior work in TfD. By further soliciting and incorporating user feed- back, HELPER attains an additional 1 1.3x boost in task success. Our work is inspired by works in the language domain (Perez et al.. 2021: Schick and Schütze, 2020; Gao al., 2020: Liu et al., 2021) that retrieve in-context prompt examples based on the input language query for NLP tasks. HELPER extends this capability to the domain of instructable embodied agents, and demonstrates the potential of mneoory-aggment LLMs for semantic parsing of open-ended free-form instructive language into expandable library of programs.
settings, im-
Related Work
We test HELPER on the TEACh benchmark (Pad- makumar et al., 2021), which evaluates agents
Instructable Embodied Agents Significant strides have been made by training large neural

---

Executor
networks jointly map instructions and their sen- sory contexts to agent actions macro-actions using imitation learning (Anderson et al., 201 8b; Ku et al., 2020; Anderson et al., 20; Savva et al., 2019; Gervet et al., 2022; Shridhar et al., 2020; Cao
Memory
Mapping & Object Tracking
Dampka ka
Renlocd
Pre-
Candition
Mave
Check
Planner
API
Locator
Object States Object Apple Cleand
Ensirommere
et al.; Suglia et al., 2021; Fan et al., 2018; Yu et al.,
Language
LLM
2020; Brohan et al., 2022: Stone et al., 2023; Yu et al., 2023). Existing approaches differ-amont among others-in the way the state of the environment is communicated to the model. Many methods map RGB image tokens and language inputs di- rectly to actions or macro-actions (Pashevich et al., 2021; Wijmans et al., 2020; Suglia et al., 2021; Krantz et al., 2020). Other methods map language instructions and linguistic descriptions of the envi- ronment's s state in terms of object lists objects spatial coordinates to macro-actions, foregoing vi- sual feature description of the scene, in a attempt to generalize better (Liang et al., 2022; Singh al., 2022a: Chaplot et al., 2020a; Min al., 2021; Liu et al., 2022a; Murray and Cakmak, 2022; Liu et al., 2022b; Inoue and Ohashi, 2022; Song et al., 2022;
Object
Search
Object N:
Image
API Skills Stce
VLM
Figure 2: HELPER's architecture. The model uses memory-augmmented L.LM prompting for task planning from instructions, corrections and human-robor dialogue and for re-planning during failures given feedback from VLM model. The generated program is executed the EXECUTOR module. The EXECUTOR builds semantic, occupancy and 3D object maps, tracks object states, ver- ifies action preconditions, and queries L..Ms for search locations for objects missing from the maps, using the LOCATOR module.
evaluating home assistant frameworks, including Habitat (Savva et al., 2019), Gibson World (Shen et al., 2021), ThrleDWorld (Gan et al., 2022). and AI2THOR (Kolve et al., 2017). ALFRED (Shrid- har et al., 2020) and TEACh (Padmakumar et al.. 2021) are benchmarks in the AI2THOR environ- ment (Kolve et al., 2017), measuring agents' com- petence in household tasks through natural lan- guage. Our research focuses the "Trajectory from Dialogue' (TID) evaluation in TEACh, mir- roring ALFRED but with greater task and input complexity.
Zheng et al., 2022; Zhang et al., 2022; Huang et al., 2022b, 2023; Ahn et al., 2022; Zeng et al., 2022;
Huang et al., 2022a). Some of these methods fine- tune language models to map language input to macro-actions, while others prompt frozen LLMs to predict action programs, relying on the emergent in-context learning property LLMs to emulate novel tasks at test time. Some methods use natural language as the output format of the LLM (Wu et al., 2023a; Song et al., 2022; Blukis et al., 2022; Huang et al., 2022b), and others use code format (Singh al., 2022a; Liang et al., 2022; Huang et al., 2023). HELPER prompts frozen LLMs to predict Python programs over visuo-motor functions for parsing dialogue, instructions and corrective hu- man feedback.
Prompting LLMs for action prediction and vi- sual reasoning Since the introduction of few- shot prompting by (Brown et al., 2020), several approaches have improved the prompting ability of LLMs by automatically learning prompts (Lester et al., 2021), chain of thought prompting (Nye et al.; Gao et al., 2022; Wei al., 2022: Wang et al., 2022: Chen al., 2022: Yao al., 2023) and retrieval-augmented LLM prompting (Nakano et al., 2021; Shi et al., 2023; Jiang et al., 2023) for language modeling, question answering, and long-form, multi-hop text generation. HELPER uses memory-augmented prompting by retrieving and integrating similar task plans into the prompt to facilitate language parsing programs.
The work closest to HELPER is LLM Plan- ner (Song et al., 2022) which uses memory- augmented prompting of pretrained LLMs for in- struction following. However, it differs from HELPER in several areas such as plan memory expansion, VLM-guided correction, and usage of LLMSs for object search. Furthermore, while Singh et (2022b) frequently seeks human feed- back, HELPER requests feedback only post full task execution and employs LanguaLe Models (VLMs) for error feedback, reducing user interrup- tions.
L.MSs have been used as policies in Minecraft predict actions (Wang et al., 2023b,a), error cor- rection (Liu et al., 2023b), and for understand-
Numerous simulation environments exist for

---

ing instruction manuals for game play in some Atari games (Wu et al., 2023b). They have also significantly improved text-based agents in text- based simulated worlds (Yao et al., 2022; Shinn et al., 2023; Wu et al., 2023c; Richards, 2023). ViperGPT (Surís et al., 2023), and Code VoA (Sub- ramanian et al., 2023) use LLM prompting to de- compose referential expressions and questions to programs over simpler visual routines. Our work uses LLMs for planning from free-form dialogue and user corrective feedback for home task com- plction, a domain not addressed in previous works.
HELPER maintains key-value memory of lan- guage program pairs, shown in Figure 3A. Each language key is mapped to a ID vector using an LLM's frozen language encoder. Given current context 7, the model retrieves the top- keys, i.e., the keys with the smallest L2 distance with the embedding of the input context I, and adds the cor- responding language- program pairs to the LLM prompt as in-context examples for parsing the cur- rent input
Figure 3B illustrates the prompt format for the PLANNER. It includes the API specifying the primitives G parameterized as Python functions, the retrieved examples, and the language input I. The LLM is tasked to generate Python program parameterized primitives G. Examples of our prompts and LLM responses can be found in Sec- tion of the Appendix.
3 Method
HELPER an embodied agent designed to map human-robot dialogue, corrections and VLM de- scriptions to actions programs over a fixed API of parameterized navigation and manipulation prim- itives. Its architecture is outlined in Figure 2. At its heart, it generates plans and plan adjust- ments by querying LLMs using retrieval of relevant language-prograt pairs to include as in-context ex- amples in the LLM prompt. The generated pro- grams are then sent to the EXECUTOR module, which translates each program step into specific navigation and manipulation action. Before exe- cuting each step in the program, the EXECUTOR verifies if the necessary preconditions for an action, such as the robot already holding an object, met. If not, the plan is adjusted according to the current environmental and agent state. Should step involve an undetected object, the EXECUTOR calls on the LOCATOR module to efficiently search for the required object by utilizing previous user instructions and LLMs' common sense knowledge. If any action fails during execution, a VLM pre- dicts the reason for this failure from pixel input and feeds this into the PLANNER for generating plan adjustments.
3.1.1 Memory Expansion
The key-value memory of HELPER can contin- ually expanded with successful executions of in- structions to adapt to a user's specific routines, as shown in Figure 1. An additional key-value pair is added with the language instruction paired with the execution plan if the user indicates the task was successful. Then, HELPER can recall this plan and adapt it in subsequent interactions with the user. For example, a user instructs HELPER one day "Perform the Mary cleaning. This involves clean- ing two plates and two cups in the sink", the user need only say "Do the Mary cleaning" in future interactions, and HELPER will retrieve the previ- ous plan, include it in the examples section of the prompt, and query the LLM to adapt it accord- ingly. The personalization capabilities of HELPER are evaluated in Section 4.4.
3.1.2 Incorporating user feedback
A user's feedback can improve robot's perfor- mance, but requesting feedback frequently can de- teriorate the overall user experience. Thus, we enable HELPER to elicit user feedback only when it has completed execution of the program. Specifi- cally, it asks "Is the task completed to your satis- faction? ? Did miss anything?" ?" once it believes it has completed the task. The user responds either that the task has been completed (at which point HELPER stops acting) or points out problems and corrections in free-form natural language, such as, "You failed to cook a slice of potato. The potato slice needs to be cooked.". HELPER uses the language
3.1 PLANNER: Retrieval-Augmented LLM Planning
Given an input consisting of a dialogue segment, instruction, or correction, HELPER uses memory- augmented prompting of frozen LLMs to map the input into executable Python program over a parametrized set of manipulation and naviga- tion primitives G E |Gmenipulation Gravigation that the EXECUTOR can perform (e.g., gX), pickup(X), slice(X), ...). Our action API can be found in Section D of the Appendix.

---

A
Input Dialogue help you? I'd like a slice toast fridge On plate!
Prompt
Rol:
Python Program
Language Encoder
Querr
AP
Aupurree for desa Oiance
de coggeette fur Sir
Dremectianreeeet (mard) Cargin-e Srain-wene
please
ok?
camguuee 2e ne o9-1122
Falture Tascloock
Iglled subpoal
Conrectrcc Iff apsticatroo
Planner
langunge--rogram
memory
Self-refiection & Corrective Program
Key 3
Encoddr
Feecbookd
Ratrioord Examples
Ereal 1000 Viice otruur..
LLM
jeadic Spaet escul more cowartt the Frogge
Key
Lan guugge
Dillegve
I8 Foecbookd
Ker 3
Languugg Encoder
18 Feecbook)
Dillogue
Imput
large: Fridgy (0R0
Key
Language Entrsher
& Foachack)
Diaiogue
applicabhl) Falled subgoal
Feedback
Figure 3: HELPER parses dialogue segments, instructions, and corrections into visuomotor programs using retrieval- augmented LLM prompting, A. Illustration of the encoding and memory retrieval process. B. Prompt format and output of the PLANNER.
feedback to re-plan using the PLANNER. We eval-
List of failure reasons
Fallura Faadback Nu
Fallura
Dbjeet openitices sucomeetilly
uate HELPER in its ability to seek and utilize user feedback in Section 4.3.
objeci slidiig you inteaciing
That usged away intrrct
VLM
posdiora place usject found....
Figure 4: Inference of failure feedback description by matching potential failure language descriptions with the current image using vision-language model (VLM).
3.1.3 Visually-Grunnded Plan Correction using Vision-Language Models
Generated programs may fail for various reasons, such as when a step in the plan is missed or an object-of-interee is occluded. When the pro- gram fails, HELPER uses vision-language model (VLM) pre-trained web-scale data, specifically the ALIGN model (Jia et al., 2021), to match the current visual observation with a pre-defined list of textual failure cases, such as an object is blocking you from interacting with the selected object, as illustrated in Figure 4. The best match is taken to be the failure feedback F. The PLANNER module then retrieves the top-K most relevant error cor- rection examples, each containing input dialogue, failure feedback, and the corresponding corrective program, from memory based on encodings of in- put I and failure feedback from the VLM. The LLM is prompted with the the failed program step, the predicted failure description from the VLM, the in-context examples, and the original dialogue segment J. The LLM outputs a self-reflection as to why the failure occurred, and generates pro- gram over manipulation and navigation primitives G, and an additional of corrective primitives Georrective (e.g., step-back(), move-to-an-aae -nve-to-an-alterrate viewpoint(), ...). This program is sent to the Ex- ECUTOR for execution.
3.2 EXECUTOR: Scene Perception, Pre-Condition Checks, Object Search and Action Execution
The EXECUTOR module executes the predicted Python programs in the environment, converting the code into low-level manipulation and naviga- tion actions, as shown in Figure 2. At each time step, the EXECUTOR receives an RGB image and obtains estimated depth map via monocular depth estimation (Bhat et al., 2023) and object masks via an off-the-shelf object detector (Dong et al., 2021).
3.2.1 Scene and object state perception
Using the depth maps, object masks, and approxi- mate egomotion of the agent at each time step, the EXECUTOR maintains 3D occupancy map and ob- ject memory of the home environment to navigate around obstacles and keep track of previously seen objects, similar previous works (Sarch al., 2022). Objects are detected in every frame and are merged into object instances based on close- ness of the predicted 3D centroids. Each object instance is initialized with a set of object state at-

---

tributes (cooked, sliced, dirty, ...) by matching the object crop against each attribute with the pre- trained ALIGN model (Jia et al., 2021). Object attribute states are updated when an object acted upon via a manipulation action.
Experiments
We test HELPER in the TEACh benchmark (Pad- makumar et al., 2021). Our experiments aim to answer the following questions:
1. How does HELPER compare to the SOTA on task planning and execution from free-form dialogue?
3.2.2 Manipulation and navigation
pre-condition checks
The EXECUTOR module verifies the cre-conditions of an action before the action is taken to ensure the action is likely to succeed. In our case, these constraints are predefined for each action (for exam- ple, the agent must first be holding knife slice an object). If any pre-conditions are not satisfied, the EXECUTOR adjusts the plan accordingly. In more open-ended action interfaces, an LLM's s com- mon sense knowledge can be used to infer the pre- conditions for an action, rather than pre-defining them.
2. How much do different components of HELPER contribute performance?
3. How much does cliciting human feedback help task completion?
4. How effectively does HELPER adapt user's specific procedures?
4.1 Evaluation on the TEACh dataset
Dataset The dataset comprises over 3,000 hu- man-human, interactive dialogues, geared towards completing household tasks within the A12-THOR simulation environment (Kolve et al., 2017). We evaluate the Trajectory from Dialogue (TID) evaluation variant, where the agent is given di- alogue segment the start of the episode. The model is then tasked to infer the sequence of ac- tions to exccute based on the user's intents the dialogue segment, ranging from MAKE COFFEE to PREPARE BREAKFAST. We show examples of such dialogues in Figures 1 & 3. We also test on the Execution from Dialogue History (EDH) task in TEACh, where the TiD episodes are partitioned into "sessions". The agent is spawned at the start of one of the sessions and must predict the actions reach the next session given the dialogue and action history of the previous sessions. The dataset split into training and validation sets. The validation set is divided into 'seen' and 'unseen' rooms based on their presence in the training set. Validation 'seen' has the same room instances but different object locations and initial states than any episodes in the training set. At each time step, the agent obtains an egocentric RGB image and must choose an action from a specified set to transition to the next step, such as pic turn left() etc. Please see Ap- pendix Section G for more details on the simulation environment.
3.2.3
LOCATOR: LLM-based common sense object search
When HELPER needs to find an object that has not been detected before, calls the LOCATOR mod- ule. The LOCATOR prompts an LLM to suggest potential object search location for the EXECUTOR to search nearby, e.g. "search near the sink" or "search in the cupboard". The LOCATOR prompt takes in the language I (which may reference the object location, e.g., "take the mug from the cup- board") and queries the LLM to generate proposed locations by essentially parsing the instruction as well as using its common sense. Based on these predictions, HELPER will go to the suggested lo- cations if they exist in the semantic map (e.g., to the sink) and search for the ofject-of-interest. The LOCATOR's prompt can be found in Section D of the Appendix.
Implementation details. We use OpenAT's gpt-4-0613 (gpt, 2023) API, except when mentioned otherwise. We resort to the text-embedding-ada-e00 (ada. 2022) API to ob- tain text embeddings. Furthermore, we use the SOLQ object detector (Dong et al., 2021), which is
pretrained on MSCOCO (Lin et al., 2014) and fine- tuned on the training rooms of TEACh. For monoc- ular depth estimation,
the ZoeDepth net- work (Bhat al., 2023), pretrained on the NYU in- door dataset (Nathan Silberman and Fergus, 2012) and subsequently fine-tuned on the training rooms TEACC.. In the TEACh evaluations, we use K=3 for retrieval.
Evaluation metries Following evaluation prac- tises for the TEACh benchmark, we use the follow- ing two metrics: 1. Task success rate (SR), which refers to the fraction of task sessions in which the

---

No In Domain
Memory- Augmented
User Personal- ization
Accepts User Feedback
VLMM- Guided correction
L.M-- Guided Search
Pre-
Condition
L.M
L.IM
Check
E.T. (Falevch & 2011
X
x
X
X
x
X
X
JARVIS 2022)
X
FILM Uii dileeta 2021, 2022
2022
X
x
X
X
x
x
DANLI I jizhnng et ti,,
v
LLM-Planne SSon
Code as Policies (Linng at, 2022)
X
HELPER (ours)
Table Comparison of HELPER to previous work.
agent successfully fulfills all goal conditions. 2. Goal condition success rate (GC), which quan- tifies the proportion of achieved goal conditions across all sessions. Both of these metries have cor- responding path length weighted (PLW) variants. In these versions, the agent incurs penalties for ex- ecuting a sequence of actions that surpasses the length of the reference path annotated by human experts.
alogue History (EDH) validation split in Table 2. On the TfD validation unseen, HELPER achieves 13.73% task success rate and 14.17% goal- condition success rate, relative improvement of 1.7x and 2.1x, respectively, over DANLI, the prior SOTA in this setting, HELPER additionally sets a new SOTA in the EDH task, achieving 17.40% task success rate and 25.86 % goal- condition success rate on validation unseen.
4,2 Ablations
Baselines We consider the following baselines: 1. Episodic Transformer (E.T.) (Pashevich al.. 2021) an end-to-end multimodal transformer that encodes language inputs and a history of visual ob- servations to predict actions, trained with imitation learning from human demonstrations.
We ablate components of HELPER in order to quan- tify what matters for performance in Table 2 Ab- lations. We perform all ablations on the TEACh TfD validation unseen split. We draw the following conclusions:
2. Jarvis (Zheng et al., 2022) trains an LLM on the TEACh dialogue to generate high-level sub- goals that mimic those performed by the human demonstrator. Jarvis uses semantic map and the Episodic Transformer for object search.
1. Retrieval-augmmented prompting helps for planning, re- re-planning and failure recovery. Replac- ing the eeemory-augmente prompt (w/o Mem Aug; Table 2) led to relative 18% reduction in success rate.
prompts with fixed
3. FILM (Min et al., 2021, 2022) fine-tunes an LLM to produce parametrized plan templates. Sim- ilar to Jarvis, FILM uses a semantic map for carry- ing out subgoals and semantic policy for object search.
2. VLM error correction helps the agent recover from failures. Removal of the visually-groundee plan correction (w/o Correction; Table 2) led to a relative 6% reduction in success rate.
3. The pre-condition check and the LLM search help. Removal the action pre-condition checks (w/o Pre Check; Table 2) led to a relative 16% re- duction in success rate. Replacing the LOCATOR LLLM-based search with random search (w/o Lo- CATOR; Table 2) led to a relative 12% reduction in success rate.
4. DANLI (Zhang et al., 2022) fine-tunes an LLM predict high-level subgoals, and uses symbolic planning over an object state and spatial map to cre- ate an execution plan. DANLI uses an object search module and manually-defime error correction.
HELPER differs from the baselines in its use of memory-auggentee context-dependent prompting of pretrained LLMs and pretrained uisual-lagguage models for planning, failure diagnosis and recovery, and object search. We provide more in-depth comparison of HELPER to previous work in Table 1,
Larger LLMs perform better. Using GPT-3.5 5 (w GPT-3.5; Table 2) exhibits relative 31 % re- duction in success rate compared to using GPT-4, Our findings on GPT-4's superior planning abilities align with similar findings from recent studies of Wu et al. (2023d); Bubeck et al. (2023); Liu et al.
Evaluation We show quantitative results for HELPER and the baselines on the TEACh Trajec- tory from Dialogue (TfD) and Execution from Di-
(2023a); Wang et al. (2023a).
5. Perception is a bottleneck. Using GT depth

---

Table 2: Trajectory from Dialogue (TID) and Execution from Dialog History (EDH) evaluation on the TEACh validation set. Trajectory length weighted metrics are included in paarentheses ). SR success rate. GC gooal condition success rate.
TID
EDH
Unseen
Seen
Unseen
Seen
SR
GC
SR
GC
SR
GC
SR
GC
E.T.
0.48 (0.12)
0.35 (0.59)
1.02 (0.17)
1.42 (4.82)
7.8 (0.9)
9.1 (1.7)
(1.7)
(4.1)
JARVIS
1.80 (0.30)
(1.60)
1.70 (0.20)
5.40 (4.50)
15.80
16.60 (8.20)
22.60 (8.70) (3.30)
FILM
(1.0)
6.1 (2.5)
(2.6)
5,8 (11.6)
10.2 (1.0) 18.3 (2.7)
14.3
(5.6)
DANLI
7.98 (3.20) 6.79 (6.57)
4.97 (1.86)
10.50 (10.27)
16.98 (7.24) 23.44 (19.95)
24.93 (22.20) 18.59 (4.00) 32,09 (9.81)
17.76 (9.28)
HELPER (ours)
14.17 (1.61)
(4.56)
12.15 (1.79)
18.62 (9.28)
17.40 (2.91) 25,86 (7.90)
Ablations
HELPER, and HELPER will re-plan based on this feedback. As shown in in Table 2 User Feedback, asking for user's s feedback twice improves perfor- mance by 1.27x. Previous works do not explore this opportunity of cliciting human feedback partly due to the difficulty of interpreting it beinging free-form language-whin which our work addresses.
w/o Mem Aug
11.27 (1.39)
11.09 (4.00)
w/o Pre Check
(1.36)
11.32 (5.15)
w/o Correction
12.9 (1.53)
12.45 (4.91)
w/o LOCATOR
12.09 (1.29)
10.89 (3.83)
w/ GPT-3.5
9.48 (1.21)
10.05 (3.68)
w/ GT depth
15.85 (2.85)
14.49 (6.89)
w/ GT depth,seg22.55
(6.39)
30.00 (14.56)
w/ GT pereept
30.23 (9.12)
50.46 (20.24)
User Feedback
w/ Feedback
16,34 (1.67) 14.70 (4.69)
w/ Feedback 2
17.48 (1.97)
14.93 (4.74)
4.4 Personalization
w/ GT percept, 37.75 (10.7))56.77 (19.80)
We evaluate HELPER's ability to retrieve user- specific routines, as well as on their ability to mod- ify the retrieved routines, with one, two, or three modifications, as discussed in 3.1.1. For example. for three modifications might instruct HELPER: "Make a Dax sandwich with I slice of tomato, lettuce leaves, and add slice of bread".
Feedback 2
(w/ GT depth; Table 2) led to improvement of 1.15x compared using estimated depth from RGB. Notable is the 1.77x improvement in path- length weighted success when using GT depth. This change is due to lower accuracy for far depths in our depth estimation network lower, thereby causing the agent to spend more time mapping the environment and navigating noisy obstacle maps. Using lidar or better map estimation techniques could mitigate this issue.
Dataset The evaluation tests 10 user-specific plans for each modification category in five distinct tasks: MAKE A SANDWICH; PREPARE BREAK- FAST; MAKR SALAD; PLACCE: X ON Y; and CLEAN X. The evaluation contains 40 user re- quests. The complete list of user-specific plans and modification requests can be found in the Ap- pendix, Section C.
Using ground truth segmentation masks and depth (w/ GT depth, seg; Table 2) improves task success and coal-conditioned task success by 1.64x and 2.1 1x, respectively. This shows the limitations of frame-based object detection and late fusion of detection responses over time. 3D scene represen- tations that fuse features carlier across views may significantly improve 3D object detection. Using GT perception (w/ GT percept: Table 2), which in- cludes depth, segmentation, action success, oracle failure feedback, and increased API failure limit (50), led to 2.20x and 3.56x improvement.
Evaluation We report the success rate in Table 3. HELPER generates the correct personalized plan for all but three instances, out of 40 evaluation requests. This showcases the ability of HELPER to acquire, retrieve and adapt plans based on context and previous user interactions.
5 Limitations
Our model in its current form has the following limitations:
4.3 Eliciting Users' Feedback
We enable HELPER elicit sparse user feedback by asking "Is the task completed to your satisfac- tion? Did miss anything?"" ?" once it believes it has completed the task, as explained in Section 3. 1.2. The user will then respond with steps missed by
1. Simplified failure detection. The A12-THOR simulator much simplifies action failure detection which our work and previous works exploit (Min et al., 2021; Inoue and Ohashi, 2022). In a more general setting, continuous progress monitoring

---

Table 3: Evaluation of HELPER for user personaliza- tion. Reported success of generating the correct plan for 10 personalized plans for request of the original plan without modifications, and one, two, or three modi- fications to the original plan. These experiments use the text-davinci-000 model as the prompted L.I.M.
primitives, that it executes in household environ- ments from visual input. HELPER updates its mem- ory with user-instructed action programs after suc- cessful execution, allowing personalized interac- tions by recalling and adapting them. It sets a new state-of-the-an in the TEACh benchmark. Future research directions include extending the model to include visual modality by encoding visual context during memory retrieval or as direct in- put to the LLM. We believe our work contributes towards exciting new capabilities for instructable and conversable systems, for assisting users and personalizing numan-robon communication.
Success
Original Plan
100%
One Change
100%
Two Changes 80%
Three Changes 90%
from pixels would be required for failure detection, which model VLMs can deliver and will address in future work.
Acknowledgements
This material is based upon work supported by National Science Foundation grants GRF DGE1745016 & DGE2140739 (GS), a DARPA Young Investigator Award, a NSF CAREER award, an AFOSR Young Investigator Award, and DARPA Machine Common Sense, and an ONR award AWD00002287 Any opinions, findings and con- clusions recommendations expressed in this ma- terial are those of the authors and do not necessarily reflect the views of the United States Army, the Na- tional Science Foundation, or the United States Air Force.
2. 3D perception bottleneck. HELPER relies on 2D object detectors and depth 3D lifting for 3D ob- ject localization. We observe a 2X boost in TEACh success rate from using ground truth segmentation in HELPER. In future work plan to integrate early 2D features into persistent 3D scene feature representations for more accurate 3D object detec- tion.
4. Cost from LLM querying. GPT-4 API is the most accurate LLM used in HELPER and incurs significant cost. NLP research in model compres- sion may help decreasing these costs, or finetuning smaller models with enough input-output pairs.
This research project has benefitted from the Mi- crosoft Accelerate Foundation Models Research (AFMR) grant program through which Icading foundation models hosted by Microsoft Azure along with access Azure credits were provided to conduct the research.
3. Multimodal (vision and language) memory retrieval. Currently, environment state descriptions. Exciting fu- ture directions include exploration of visual state incorporation to the language model and partial adaptation of its parameters. multi-modal ap- proach to the memory and plan generation would help contextualize the planning more with the vi- sual state.
text bottleneck in
Ethics Statement
The objective of this research is to construct au- tonomous agents. Despite the absence of human experimentation, practitioners could potentially im- plement this technology in human -inlusiiie envi- ronments. Therefore, applications of our research should appropriately address privacy considera- tions.
Last, to follow human instructions outside of simulation environments our model would need to interface with robot closed-loop policies instead of abstract manipulation primitives, following previ- ous work (Liang et al., 2022).
All the models developed in this study were trained using Ai2Thor (Kolve et al., 2017). Con- sequently, there might be an inherent bias towards North American homes. Additionally, we only con- sider English language inputs in this study.
6 Conclusion
We presented HELPER, an instructable embodied agent that uses memory-augmente prompting of pre-trained LLMs to parse dialogue segments, in- structions and corrections to programs over action

---

References
Sébastien Bubeck, Varun Chandrasekaran, Ronen EI- dan, Johannes Gehrke, Eric Horvitz, Ece Kamar. Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lund- berg, et al. 2023. Sparks of artificial general intelli- gence: Early experiments with gpt-4. arXiv preprint arXiv:2303. 12712.
2022. New and improved embedding model.
2023. Openai. gpt-4 technical report. arXiv preprint arxiv:2303.087.
Michael Ahn, Anthony Brohan, Noah Brown, Yev- gen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman. Alex Herzog, Daniel Ho. Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jes- month, Nikhil Joshi, Ryan Julian, Dmitry Kalash- nikov, Yuheng Kuang. Kuang-Huei Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Pe- ter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nico- las Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan, and Andy Zeng. 2022. Do i can and not as say: Grounding language in robotic af- fordances. In arXiv preprint arXiv:220..0162)
Yuchen Cao, Nilay Pande, Ayush Jain, Shikhar Sharma, Gabriel Sarch, Nikolaos Gkanatsios, Xian Zhou, and Katerina Fragkiadaki. Embodied symbiotic assis- tants that see, act, infer and chat.
Devendra Singh Chaplot, Dhiraj Gandhi, Saurabh Gupta, Abhinav Gupta, and Ruslan Salakhutdinov. 2020a. Learning to explore using active neural slam. arXiv preprint arXiv:2004.05155
Devendra Singh Chaplot, Dhiraj Prakashchand Gandhi, Abhinav Gupta, and Russ R Salakhutdinov. 2020b. Object goal navigation using goal-oriented semantic exploration. Advances in Neural Information Pro- cessing Systems, 33.
Wenhu Chen, Xueguang Ma. Xinyi Wang, and William W Cohen. 2022. Program of thoughts prompting: Disentangling computation from reason- ing for numerical reasoning tasks. arXiv preprint arXiv:2211.12566
Peter Anderson, Angel Chang, Devendra Singh Chap- lot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mot- taghi, Manolis Savva, et al. 2018a. On evalua- tion of embodied navigation agents. arXiv preprint 107.07780..06577.
Bin Dong, Fangao Zeng. Tiancai Wang, Xiangyu Zhang, and Yichen Wei. 2021. Solq: Segmenting objects by learning queries. Advances Neural Information Processing Systems, 34:21898-21908
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Tan Reid, Stephen Gould, and Anton Van Den Hengel. 2018b. Vision- and-language navigation: Interpreting visually- grounded navigation instructions in real environ- ments. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3674- 3683.
Linxi Fan, Yuke Zhu, Jiren Zhu, Zihua Liu, Orien Zeng, Anchit Gupta, Joan Creus-Costar Silvio Savarese, and Li Fei-Fei. 2018. Surreal: Open-source rein- forcement learning framework and robot manipula- tion benchmark. In Conference on Robot Learning. pages 762-782. PMLR.
Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, and Matthias Müller. 2023. Zoedepth: Zero- shot transfer by combining relative and metric depth. arXiv preprint 2r02 12288.
Chuang Gan, Siyuan Zhou, Jeremy Schwartz, Seth Alter. Abhishek Bhandwaldar, Dan Gutfreund, Daniel L.K. Yamins, James J. DiCarlo, Josh McDermott, Antonio Torralba, and Joshua B. Tenenbaum. 2022. The three- dworld transport challenge: A visually guided task- and-motion planning benchmark towards physically realistic embodied ai. In 2022 International Con- ference on Robotics and Automation (ICRA), pages 8847-8854.
Valts Blukis, Chris Paxton, Dieter Fox, Animesh Garg, and Yoav Artzi 2022. A persistent spatial semantic representation for high-level natural language instruc- tion execution. In Conference on Robot Learning, pages 706-717. PMI.R
Anthony Brohan, Noah Brown, Justice Carbajal, Yev- gen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jas- mine Hsu, et al. 2022. Ri-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817.
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon. Pengfei Liu, Yiming Yang, Jamie Callan, and Gra- ham Neubig. 2022. Pal: Program-aided language models. arXiv preprint arXiv:221 10435.
Tianyu Gao, Adam Fisch, and Danqi Chen. 2020. Making pre-trained language models better few-shot learners. arXiv preprint arXiv:2012.15723.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 1877-1911
Theophile Gervet, Soumith Chintala, Dhruv Batra, Jiten- dra Malik, and Devendra Singh Chaplot. 2022. Nav- igating to objects the real world. arXiv preprint arXiv:2212.00922.

---

Chenguang Huang, Oier Mees, Andy Zeng, and Wol- fram Burgard. 2022a. Visual language maps for robot navigation. arXiv preprint arXiv: arXh::2210.05714
programs for embodied control. arXiv preprint arXiv:2209.07753.
Percy Liang. 2016. Learning executable semantic parsers for natural language understanding. Com- munications of the ACM, 59(9):68-76
Siyuan Huang, Zhengkai Jiang, Hao Dong, Yu Qiao, Peng Gao, and Hongsheng Li. 2023. Instruct2act: Mapping multi-modality instructions to robotic ac- tions with large language model. arXiv preprint arXiv:2305. 11176.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Computer Vision- ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer.
Wenlong Huang, Fei Xia. Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. 2022b. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint urXiv22077006000
Haoyu Liu, Yang Liu, Hongkai He, and Hangfang Yang. 2022a. Lebp language expectation & binding pol- icy: A two-stream framework for embodied vision- and-language interaction task learning agents. arXiv preprint arXiv:2203.046.
Yuki Inoue and Hiroki Ohashi. 2022. Prompter: Utiliz- ing large language model prompting for data effi- cient embodied instruction following. arXiv preprint
Jiachang Liu, Dinghan Shen, Yizhe Zhang. Bill Dolan,
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung. Zhen Li, and Tom Duerig. 2021. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pages 9904-9916. PMLR.
Lawrence Carin, and Weizhu Chen. 2021. What makes good in-context examples for gpt-3? arXiv preprint arXiv:2101.06800
Xiaotian Liu, Hector Palacios, and Christian Muise. 2022b. planning based neural-symholic approach for embodied instruction following. Interactions. 9(8):17.
Zhengbao Jiang. Frank F. Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi- Yu, Yiming Yang, Jamie Callan, and Graham Neubig, 2023, Active retrieval augmented generation.
Yiheng Liu, Tianle Han, Siyuan Ma, Jiayue Zhang, Yuanyuan Yang, Jiaming Tian, Hao He, Antong Li, Mengshen He, Zhengliang Liu, et al. 2023a. Sum- mary chatgpu/gpt-4 research and perspective to- wards the future of large language models. arXiv preprint arx.v:2304.0185.
Dan Klein and Christopher D Manning. 2003. Accurate unlexicalized parsing. In Proceedings of the s1 annual meeting of the association for computational linguistics, pages 420-430.
Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli Van- derBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. 2017. A12-THOR: An Interactive 3D Environment for Vi- sual AI. arviv.
Zeyi Liu, Arpit Bahety, and Shuran Song. 2023b. Reflect: Summarizing robot experiences for fail- explanation and correction. arXiv preprint arXiv:2306. 15724.
So Ycon Min, Devendra Singh Chaplot, Pradeep Raviku- mar, Yonatan Bisk, and Ruslan Salakhutdinov. 2021. Film: Following instructions in language with modu- lar methods.
Jacob Krantz, Erik Wijmans, Arjun Majumdar, Dhruv Batra, and Stefan Lee. 2020. Beyond the nav-graph: Vision-and-language navigation in continuous envi- ronments. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28. 2020, Proceedings, Part XXVIII 16, pages 104-120. Springer.
So Yeon Min, Hao Zhu, Ruslan Salakhutdinov, and Yonatan Bisk. 2022. Don't copy teacher: Data and model challenges in embodied dialogue. arXiv preprint 221.:221100444:
Alexander Ku, Peter Anderson, Roma Patel, Eugene Ie. and Jason Baldridge. 2020. Room-across coom-ccross-rom. room: Multilingual rision-and-languaal navigation with dense spatiotemporal grounding. arXiv preprint arXiv:2010.09944.
Michael Murray and Maya Cakmak. 2022. Follow- ing natural language instructions for household tasks with landmark guided search and reinforced pose ad- justment. IEEE Robotics and Automation Letters. 7(3):6870-6877.
Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.0888
Relichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu. Long Ouyang, Christina Kim, Christopher Hesse. Shantanu Jain, Vineet Kosaraju, William Saunders, et al. 2021. Webgpt: Browser-assisted question- answering with human feedback. arXiv preprint arXiv:2112.09332
Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian lchter, Pete Florence, and Andy Zeng. 2022. Code as policies: Language model

---

Pushmeet Kohli Nathan Silberman, Derek Hoiem and Rob Fergus. 2012. Indoor segmentation and support inference from rghd images. In ECCV.
Noah Shinn, Beck Labash, and Ashwin Gopinath. 2023. Reftexion: an autonomous agent with dy- namic memory and self-reflection. arXiv preprint arXiv:2303.I 1366.
Maxwell Nye, Anders Johan Andreassen, Guy Gur- Ari, Henryk Michalewski, Jacob Austin, David Bieher, David Dohan, Aitor Lewkowyez, Maarten Bosma, David Luan, et al. Show your work: Scratchpads for intermediate computation with lan- guage models, november 2021. URL http:/larxiv. 1220014/111..00114.
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. 2020. Alfred: A bench- mark for interpreting grounded instructions for ev- eryday tasks. In Proceedings of the IEEE/CVF con- ference on computer vision and patlern recognition. pages 10740-10749.
Aishwarya Padmakumar, Jesse Thomason, Ayush Shri- vastava, Patrick Lange, Anjali Narayan- Span- dana Gella, Robinson Piramuthu, Gokhan Tur, and Dilek Hakkani- Tur. 2021. Teach: Task-driven bodied agents that chat.
Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu. Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. 2022a. Progprompt: Generating situated robot task plans using large language models. arXiv preprint arXiv:2209.11300
Alexander Pashevich, Cordelia Schmid, and Chen Sun. 2021. Episodic transformer for lisionanddlanguuae navigation. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pages 15942- 15952.
Kunal Pratap Singh. Luca Weihs, Alvaro Herrasti, Jonghyun Choi, Aniruddha Kembhavi, and Roozbeh Mottaghi. 2022b. Ask4help: Learning to leverage an expert for embodied tasks. Advances in Neural Information Processing Systems, 35: 12221611612222
Ethan Perez, Douwe Kiela, and Kyunghyun Cho. 2021. True few-shot learning with language models. Ad- vances in neural information processing systems, 34:11054-11070.
Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M. Sadler, Wei-Lun Chao, and Yu Su. 2022. Lim-planner: Few-shot grounded planning for em- bodied agents with large language models. arXiv preprint arXiv:22 12t.22022.488
Toran Bruce Richards. 2023. Auto-gpt: An autonomous
gpt-4 experiment.
Gabriel Sarch, Zhaoyuan Fang, Adam W Harley, Paul Schydlo. Michael J Tarr, Saurabh Gupta, and Ka- terina Fragkiadaki. 2022. Tidee: Tidying up novel rooms using visuo-semantic commonsense priors. In Computer Vision-ECCV 2022: 17th European Con- ference, Tel Aviv, Israel, October 23-27, 2022, Pro- ceedings, Part XXXIX, pages 480-496. Springer.
Austin Stone, Ted Xiao, Yao Lu. Keerthana Gopalakrish- nan, Kuang- Huei Lee, Quan Vuong. Paul Wohlhart, Sean Kirmani, Brianna Zitkovich, Fei Xia, Chelsea Finn, and Karol Hausman. 2023. Open-world ob- ject manipulation using pre-trained vision-language model. In arXiv preprint.
Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub. Jia Liu, Vladlen Koltun, Jitendra Malik, et al. 2019. Habitat: A platform for embodied research. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9339-9347.
Sanjay Subramanian, Medhini Narasimhan, Kushal Khangaonkar, Kevin Yang, Arsha Nagrani, Cordelia Schmid, Andy Zeng, Trevor Darrell, and Dan Klein. 2023. Modular visual question answering via code generation. In Proceedings of the 61 Annual Meet- ing of the Association for Computational Linguisties. Toronto, Canada, Association for Computational Lin- guistics.
Timo Schick and Hinrich Schütze. 2020. It's not just size that matters: Small language models also few-shot learners. arXiv preprint 2rXiv:2009.07 (71 18.
Alessandro Suglia, Qiaozi Gao, Jesse Thomason, Govind Thattai, and Gaurav S. Sukhatme. 2021. Em- bodied bert: A transformer model for embodied, language-guided visual task completion. In EMNLP 2021 Workshop on Novel Ideas in Learning-to- Lo-m through Interaction.
Bokui Shen, Fei Xia, Chengshu Li, Roberto Martín- Martín, Linxi Fan, Guanzhi Wang, Claudia Pérez- D' Arpino, Shyamal Buch, Sanjana Srivastava, Lyne P. Tchapmi, Micael E. Tchapmi, Kent Vainio, Josiah Wong. Li Fei-Fei, and Silvio Savarese. 2021. igibson 1.0: a simulation environment for interactive tasks in large realistic scenes. In 2021 IEEE/RSJ Interna- tional Conference on Intelligent Robots and Systems, page accepted. IEEE
Didac Surís. Sachit Menon, and Carl Vondrick. 2023. Vipergpt: Visual inference via python execution for reasoning, arXiv preprint arXiv:2303.08/22
Stefanie Tellex Thomas Kollar, Steven Dickerson, Matthew Walter, Ashis Banerjee. Seth Teller, and Nicholas Roy. 2011. Understanding natural language commands for robotic navigation and mobile manip- ulation. Proceedings of the AAAI Conference on Artificial Intelligence, volume 25, pages 1507-1514.
Weijia Shi, Sewon Min, Michihiro Yasunaga, Min- joon Seo, Rich James, Mike Lewis, Luke Zettle- moyer, and Wen-tau Yih. 2023. Replug: Retrieval- augmented black-box language models. arXiy preprint 2rXiv:2301.126..

---

Guanzhi Wang, Yugi Xie, Yunfan Jiang, Ajay Man- diekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and An- ima Anandkumar. 2023a. Voyager: An open-ended embodied agent with large language models, arXiv preprint arXiv: Arxiv-2305.166
for multi-task and meta reinforcement learning. Conference on Robot Learning, pages 1094-1100. PMLR.
Tianhe Yu, Ted Xiao, Austin Stone, Jonathan Tomp- son, Anthony Brohan, Su Wang, Jaspiar Singh, Clay- ton Tan, Dee M. Jodilyn Peralta, Brian Ichter, Karol Hausman, and Fei Xia. 2023. Scaling robot learn- ing with semantically imagined experience. In arXiv preprint arXiv:2302.
Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoe Le, Fed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. Self-consisenncy improves chain of thought reasoning in language models. arXiv preprint arXiv:2003. 11177.1111....
Andy Zeng, Maria Attarian, Brian Ichter, Krzysztof Choromanski, Adrian Wong, Stefan Welker, Fed- erico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, et al. 2022. Socratic models: Compos- ing zero-shot multimodal reasoning with language. arXiv preprint orXiv:2204.00590
Zihao Wang. Shaofei Cai, Anji Liu, Xiaojian Ma, and Yitao Liang. 2023b. Describe, explain, plan and se- lect: Interactive planning with large language models enables sppe-world multi-task agents, arXiv preprint 0rXiv:2302.01566
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quae Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. arXiv preprint 2211:22211.190.
Yichi Zhang, Jianing Yang. Jiayi Pan. Shane Storks, Nikhil Devraj, Ziqiao Ma. Keunwoo Peter Yu. Yuwei Bao, and Joyce Chai. 2022. Danli: Deliberative agent for following natural language instructions. arXiv preprint arXiv:2210. 12485..
Erik Wijmans, Abhishek Kadian, Morcos, Stefan Lee, Irfan Essa, Devi Parikh, Manolis Savva, and Dhruv Batra. 2020. Decentralized distributed ppo: Solving pointgoal navigation. In International Con- ference on Lefvarning Representations (ICLR).
Kaizhi Zheng, Kaiwen Zhou, Jing Gu, Yue Fan, Jialu Wang, Zonglin Li, Xuehai He, and Xin Eric Wang. 2022. Jarvis: A neuro-symbolic commonsense reasoning framework for conversational embodied agents.
Jimmy Wu, Rika Antonova, Adam Kan, Marion Lep- ert, Andy Zeng, Shuran Song. Jeannette Bohg, Szy- mon Rusinkiewicz, and Thomas Funkhouser. 2023a. Tidybor: Personalized robot assistance with large language models. arXiv preprint arXin:235..0565.
Yue Wu, Yewen Fan, Paul Pu Liang, Amos Azaria, Yuanzhi Li, and Tom M Mitchell, 2023b. Read and reap the rewards: Learning to play atari with the help of instruction manuals. In NeurIPS.
Yue Wu, So Yeon Min, Yonatan Bisk, Ruslan Salakhut- dinov, Amos Azaria, Yuanzhi Tom Mitchell, and Shrimai Prabhumoye. 2023c. Plan, eliminate, and track-language models good teachers for embod- ied agents. arXiv preprint arXiv:2305.024 300.0221.......
Yue Wu, So Yeon Min. Shrimai Prabhumoye, Yonatan Bisk, Ruslan Salakhutdinov, Amos Azaria, Tom Mitchell, and Yuanzhi Li. 2023d. Spring: Gpt-4 out-performs algorithms by studying papers and reasoning. In NewrIPS.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint 2305:2300..060
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. arXiv preprint a:X1v:2210.03629,
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. 2020. Meta-world: A benchmark and evaluation

---

A Analysis of User Personalization Failures
(Listing 6).
E Pre-conditions
The three instances where the PLANNER made er- in the user personalization experiment (Sec- tion 4.4) involved logical mistakes or inappropriate alterations parts of the original plan that were not requested for modification. For instance, one case - involving a step to modify the plan from making one coffee to two coffees - the PLANNER includes placing two mugs in the coffee maker si- multaneously, which is not valid plan. In the other two instances, the PLANNER omits an object from the original plan that was mentioned in the modification.
An example of a pre-condition check for macro- action provided in Listing 7.
F Example LLM inputs & Outputs
We provide examples of dialogue input, retrieved examples, and LLM output for a TEACh sample in Listing 8, Listing and Listing 10.
G Simulation environment
The TEACh dataset builds on the Ai2thor sim- ulation environment (Kolve et al., 2017). At each time step the agent may choose from the following actions: Forward(,, Backward() Turn Left(, Turn Right(,, Look UpO, Look Down(, Strafe Left), Strafe Right(, Pickup(X), Place(X), Open(X), Close(X), ToggleOn(X), ToggleOff(X), Slice(X), and Pour(X), where X refers an object specified via relative coordinate (z, y) on the ego- centric RGB frame. Navigation actions move the agent in discrete steps. We rotate the yaw dirce- tion by 90 degrees, and rotate in the pitch direction by 30 degrees. The RGB and depth sensors are at a resolution of 480x480, field of view of 90 degrees, and lie at height of 0.9015 meters. The agent's 's coordinates parameterized by single (x, y, =) coordinate triplet with i and z corresponding movement in the horizontal plane and y reserved for the vertical direction. The TEACh benchmark allows a maximum of 1000 steps and 30 API fail- ures per episode.
B User Feedback Details
In the user feedback evaluation, once the agent has indicated completion the task from the original input dialogue, the agent will query feedback from the user. If the simulator indicates success of the task, the agent will end the episode. If the simu- lator indicates the task is not successful, feedback will be given to the agent for additional planning. This feedback is programatically generated from the TEACh simulator metadata, which gives us in- formation about if the task is successful, and what object state changes are missing in order to com- plete the task (e.g., bread slice is not toasted, etc.). For each object state that is incorrect, we form a sentence of the following form: "You failed to com- plete the subtask: subtask. For the object object: description of desired object state." We combine all subtask sentences to create the feedback. HELPER follows the same pipeline (including examples, re- trieval, planning, etc.) to process the feedback as with the input dialogue in the normal TiD eval- uation. We show experiments with one and two user feedback requests in Section 4.3 of the main paper (a second request is queried if the first user feedback fails to produce task success).
H EXECUTOR details
H.1 Semantic mapping and planning
Obstacle map HELPER maintains 2D overhead occupancy map of its environment #f that updates at each time step from the input RGB- D stream. The map used for exploration and navigation the environment.
User Personalization Inputs
At every time step I, we unproject the input depth maps using intrinsic and extrinsic information of the camera to obtain 3D occupancy map regis- tered to the coordinate frame of the agent, sim- ilar earlier navigation agents (Chaplot al., 2020a). The 2D overhead maps of obstacles and free space are computed by projecting the oceu- pancy along the height direction at multiple height levels and summing. For each input RGB image,
We provide full list of the user personalization requests in Listing for the user personalization experiments in Section 4.4.
D Prompts
We provide our full API (Listing 2), corrective API (Listing 3), PLANNER prompt (Listing 4), re- planning prompt (Listing 5), and LOCATOR prompt

---

we run SOLQ object segmentor (Dong et al., 2021) (pretrained on COCO (Lin al., 2014) then finetuned on TEACh rooms) to localize each of 116 semantic object categories. For failure detection, we use simple matching approach from Min et al. (2021) to compare RGB pixel values before and after taking an action.
pancy map built so far, and then navigates the sampled location, accumulating the new informa- tion into the maps at each time step. The number of observations collected at each point in the 2D occu- paney map is thresholded to determine whether a given map location is explored or not. Unexplored positions are sampled until the environment has been fully explored, meaning that the number of unexplored points is fewer than predefined thresh- old.
Object location and state tracking We main- tain an object memory as a list of object detection 3D centroids and their predicted semantic labels Y,K, E {1...N)). i I..K). where K is the number of objects detected thus far. The object centroids are expressed with respect to the coordinate system of the agent, and, similar to the semantic maps, updated over time using egomotion. We track previously detected objects by their 3D centroid C E R We estimate the centroid by tak- ing the 3D point corresponding to the median depth within the segmentation mask and bring it to com- mon coordinate frame. We do a simple form of non-maximum suppression on the object memory, by comparing the euclidean distance of centroids in the memory to new detected centroids of the same category, and keep the one with the highest score they fall within a distance threshold.
To navigate to goal location, we compute the geodesic distance to the goal from all map locations using graph search (Inoue and Ohashi, 2022) given the top-down occupancy map and the goal location in the map. We then simulate action sequences and greedily take the action sequence which results in the largest reduction in geodesic distance.
H.2 2D-to-3D unprojection
For the i-th view, 2D pixel coordinate (u, v) with depth is unprojected and transformed its coordinate (X the reference frame:
T
(x, = (s
Sx T2 Cx.
V Cy
2,
Lu
For each object in the object memory, main- tain an object state dictionary with a pre-defined list of attributes. These attributes include: category label, centroid location, holding, detection score, can use, sliced, toasted, clean, cooked. For the binary attributes, these are initialized by sending the object crop, defined by the detector mask, to the VLM model, and checking its match to each is (attribute)", T"The
where (fz, fy) and (Cor, Gy) are the focal lengths and center of the pinhole camera model and G2 e SE)3) is the camera pose for view relative to the reference view. This module unprojects each depth image 1, E R H xx into a pointcloud in the reference frame P. E MR333 with Mj being the number of pixels with an associated depth value.
I Additional Experiments
of [h" (object_ (abject_categor)) is not (attribute)"), We found that initializing these attributes with the VLM gave only a marginal difference to initializing them to default values in the TEACh benchmark, so we do not use it for the TEACh evaluations. However, we anticipate general method beyond dataset biases of TEACh would much benefit from such vision- based attribute classification.
L.1 Alternate EDH Evaluation Split
Currently, the leaderboard for the TEACh EDH benchmark is not active. Thus, to evaluate the true test set for TEACh. We
not able
used the original validation seen and unseen splits, which have been used in most previous works (Pa- shevich et al., 2021: Zheng et al., 2022: Min et al., 2022: Zhang et al., 2022). In Table 4 we report the alternative validation and test split as mentioned in the TEACh github README, and also reported by DANLI (Zhang et al., 2022).
Exploration and path planning HELPER ex- plores the scene using classical mapping method. We take the initial position of the agent to be the center coordinate in the map. We rotate the agent in-place and use the observations instantiate an initial map. Second, the agent incrementally com- pletes the maps by randomly sampling an unex- plored, traversible location based on the 2D occu-

---

Table Alternative TEACh Execution from Dialog History (EDH) evaluation split. Trajectory length weighted metrics are included in parentheses SR = success rate. GC goal condition success rate. Note that Test Seen and Unseen are not the true TEACh test sets, but an alternative split the validation set used until the true test evaluation is released, mentioned in the TEACh github README, and also reported by DANLI (Zhang al.. 2022).
Validation
Test
Unseen
Seen
Unseen
Seen
SR
GC
SR
GC
SR
GC
SR
GC
E.T.
8.35 (0.86) 6.34 (3.69)
8.28 (1.13) 8.72 (3.82)
7.38 (0.97) 6.06 (3.17)
8.82 (0.29) 9.46 (3.03)
DANLI
17.25 (7.16)
23.88 (19,38) 38)
16.89 (9.12)
25.10 (22.56)
16.71 (7.33)
23.00 (20.55)
18.63 (9.41)
24.77 (21.90)
HELPER
17.25 (3.22) 25.24 (8.12)
19.21 (4.72) 33.54 (10.95)
17.55 (2.59) 26.49 (7.67)
17.97 (3.44) 30.81 (8.93)

---

Listing 1: Full list of user personalization requests for the user personalization evaluation.
original input to LLM:
LL'Driver', 'What is my task?'], J, d'Commander', "Make me a sandwich. The name of this sandwich is called the Larry sandwich. The sandwich has two slices of toast, slices of tomato, and 3 slice of lettuce on a clean plate.
[[ (D'Driver', 'what is my task?'], ], ['Commander', "Make me a salad. The name of this salad is called the David salad. The salad has two slices of tomato and three slices of lettuce a clean plate. 'I]
[D'Driver', 'what is my task?'], ['Commander', "Make a salad. The name of this salad is called the Dax salad. The salad has two slices of cooked potato. You'll need to cook the potato on the stove. The salad also has slice of lettuce and slice of tomato. Put all components on clean plate." "]]
[['Driver', "what is my task?'J, l'Commander', 'Make breakfast. The name of this breakfast is called the Mary breakfast. The breakfast has mug of coffee, and two slices of toast clean plate. ']]
[['Driver', 'what is my task?'], ('Commander', 'Make breakfast. The of this breakfast is called the Lion breakfast. The breakfast has mug of coffee, and four slices of tomato clean plate.']]
[['Oriver', 'what is my task?'], L'Commander', "Rearrange rearrangement called the rearrangement. Place three pillows the sofa. ']] [['Oriver', 'What is my task?'], ['Commander', 'Rearrange rearrangement called the rearrangement. Place pencils and pens the desk. 'J1 [['Driver', 'What is my task?'], ['Commander', 'Clean plates two cups. ']]
objects. The of this
objects. The of this
objects. The name of this cleaning
called the Gax cleaning. Clean [['Driver', 'what is my task?'], J, ['Comnander', "Make called the Gabe sandwich. The sandwich has two slices of toast, 2 slices of tomato, slice lettuce a clean plate. "]]
sandwich. The name this sandwich
[['Driver', 'what is my task?'], [' l'Commander', 'Clean some objects. The name this cleaning called the Kax cleaning. Clean a mug pan. 'J]
change:
"Make me the Larry sandwich"
"Make me the David salad"
"Make me the Dax salad"
"Make me the Mary breakfast" "Make me the Lion breakfast"
"Complete the Lax rearrangement"
"Complete the Pax rearrangement" "Perform the Gax cleaning"
"Make me the Gabe sandwich"
"Perform the Kax cleaning"
One change:
"Make me the Larry sandwich with four slices of lettuce"
"Make the David salad with slice of potato"
"Make me the Dax salad without lettuce"
"Make the Mary breakfast with coffee" "Make the Lion breakfast with three slice of tomato"
"Complete the Lax rearrangement with two pillows"
"Complete the Pax rearrangement but use one pencil instead of the the two pencils" "Perform the Gax cleaning with three plates instead of two" "Make the Gabe sandwich with only 1 slice of tomato" "Perform the Kax cleaning with only mug"

---

Two changes:
"Make Larry sandwich with four slices lettuce two slices of tomato"
"Make David salad but a slice potato add one slice egg" "Make me Dax salad without lettuce and without potato"
"Make Mary breakfast with no coffee and add an egg"
"Make the Lion breakfast with three slice tomato and two mugs of coffee" "Complete the Lax rearrangement with two pillows and add remote"
"Complete the Pax rearrangement but use one pencil instead of the two pencils and add a book" "Perform the Gax cleaning with three plates instead of the two plates and include fork" "Make me the Gabe sandwich with only 1 slice of tomato and two slices of lettuce" "Perform the Kax cleaning without the pan and include spoon"
Three changes:
"Make me the Larry sandwich with four slices of lettuce, two slices of tomato, and place all components directly on the countertop"
"Make me the David salad and add slice of potato, add one slice of ege, and bring a fork with it" "Make me the Dax salad without lettuce, without potato, and add an extra slice of tomato" "Make me the Mary breakfast with no coffee, add an egg, and add a cup filled with water" "Make me the Lion breakfast with three slice of tomato, two mugs of coffee, and add a fork" "Complete the Lax rearrangement with two pillows, a remote, and place it on the arm chair instead" "Complete the Pax rearrangement but use one pencil instead of the two pencils and include book and a baseball bat"
"Perform the Gax cleaning with three plates instead of the two plates, include fork, and do not clean cups"
"Make the Gabe sandwich with only 1 slice of tomato, two slices of lettuce, and add slice of
egg"
"Perform the Kax cleaning without the pan, include spoen, and include pot"

---

Listing 2: Full API for the parametrized macro-actions G used in the prompts.
class InteractionObject:
This class represents an expression that uniquely identifies an object in the house.
def _sinn, object_class, str, landmark: str None, attributes: list []):
object_class: object category of the interaction object (e.g., "Mug", "Apple")
landmark: (optional if mentioned) landmark object category that the interaction object is in relation to (e.g., "CounterTop" for "apple is on the countertop")
attributes: (aptional) list of strings of desired attributes for the object. These are not necessarily attributes that currently exist, but ones that the object should eventually have. Attributes can only be from the following: "toasted", "clean", "cooked"
self self.object_class object_class
self landmark landmark self.attributes attributes
def pseku:
"picckup the object.
This function assumes the object is view.
Example:
dialogue: <Commander>> get lettuce on the kitchen counter. Python script:
target_lettuce Interactioonbbjecc((cttttt landmark "CounterTop") borget_lettuce, (ootee.gooo())) heapt__letucee..icee 6.e
pass
def (sece landmark_name :
"""uut the interaction object on the landmark_nane object.
landmark_name must be class InteractionObject instance
This function assumes the robot has picked up an object and the landmark object is in view.
Example:
dialogue: <Commander> Put the lettuce on the kitchen counter.
Python script:
target_lettuce = Interactiondbjeet("ltttut
landmark "CounterTop")
target_lettuce.go. to_tt())
target_lettuce. pickup()
target_countertop = Interactionnjjeec(CCootere
target_ccunteerto.gee.
larget_lettuce.p wttpliiee.paaeeeeeeeeeeeee (aaretttatgetttccouuetrooo
def slice(self):
"""slice the object into pieces.
This function assumes the agent is holding knife and the agent has navigated to the object using go_to().
Example:
dialogue: <Commander>> Cut apple the kitchen counter.
Python script:
target_knife Interrattionbjjeec(((nneet first need knife slice the apple with
karget.knife.go_too
target_knife.ple pickkuuuffeppckp()
target_apple InteractiinnObject(("))) landmark "CounterTop")
sargtt_ppple.llice unu
pass

---

def (sell))
"oogggees on the interaction object.
This function assumes the interaction object already off the agent has navigated to the object.
Only some landmark objects can toggled on. Lamps, stoves, and microwaves are some examples objects that can be toggled on.
Example:
dialogue: <Commander> Turn on the lamp.
Python script:
MMeerattooobjeet((ooooommm""" ctaaampp..ooooooooooooo go_to()
target_flcorlamp
farget_floorlamp toggle_on()
pass
def of
togggee off the interaction object.
This function assumes the interaction object is already and the agent has navigated to the object.
Only some objects can be toggled off. Lamps, stoves, and microwaves some examples of objects that can be toggled off.
Example:
dialogue: <Commander>> Turn off the lamp.
Python script:
target_floorramp Intrraciioobbet((Foooommmm)))) target.floorlamp. to()
floora_ toggle_off()
pass
def go ;
" "Navigate object
pass
def osen()
"** "pen the interaction object.
This function assumes the landmark object already closed and the agent has already navigated to the object.
Only some objects can be opened. Fridges, cabinets, and drawers are some example objects that can be closed.
Example:
dialogue: <Commander> Get the lettuce in the fridge.
Python script:
target_fridge "nterattionbjeec("Frdde""""
target_lettuce = interactionobeect(lettttt landmark "Fridge")
target_fridge.go go)
target_fridge.
tuce pickup()
pass
def close(self):
cloce the interaction object.
This function assumes the object is already and the agent has already navigated to the object.
Only objects be closed. Fridges, cabinets, and drawers example of objects that be closed.
pass
def clean(self):

---

"""wash the interaction object clean it in the sink.
This function assumes the object already picked up.
Example:
dialogue: <Commander> Clean the bowl
Python script:
target_ = InteractionObjeet(Meoo attributes ["clean"])
target_bowl.clean uwe
pass
def put_ dowt_doowssee):
puts the interaction object currently in the agent's 's hand on the nearest available receptacle
This function assumes the object is already picked up.
This function is most often used when the holding object is no longer needed, and the agent needs to pick another object
pass
def pour(self,, landmark_name):
"pours the contents of the interaction object into the landmark object specified by the landmark_name argument
landmark_name must be class InteractionObject instance
This function assumes the object is already picked up and the object is filled with liquid. pass
def fill_up(self):
ooffill up the interaction object with water
This function the object is already picked up. Note that only container objects can be filled with liquid.
pass
def sickup_and_placeeee landmark_name): "n "go_ """go_to to() and pickup() this interaction object, then go_to() place() the interaction object on the landmark_name object.
landmark_name must be class InteractionObject instance wno
pass
def empty(self):
"""Enpty the object of any other objects on/in it to clear it out.
Useful when the object is too full to place an object inside it.
Example:
dialogue: <Conmander> Clear out the sink.
Python script:
target_sink = InteractionobjecctSStnnn""
srrget_siinkeempt((
def cself):
"#*Cook the object
Example:
dialogue: <Commander>> Cook the potato.
Python script:
target_potato InteractionObjeet(PPoottt;; attributes ["cooked"]) carget_potato.ccok

---

pass
def (salt()elf):
Toast a bread slice a toaster
Toasting only supported with slices bread
Example:
dialogue: <Commander> Get me a toasted bread slice.
Python script:
target_breadslic = InteractionObjett ("Breaasiii"" attributes = ["toasted"])
target_breadslic. toast()
pass

---

Listing Full Corrective API for the parametrized corrective macro-actions Corrective used in the prompts.
class AgentCorrective:
This class represents agent corrective actions that can be taken to fix a subgoal error Example usage:
agent = AgentCorrective()
agent. movee
def :see_):
"Stp backwards away from the object
Useful when the object is too close for the agent to interact with it
def (sleer
"S"tt forward to towards the object to get closer to it
Useful when the object far for the agent interact with it pass
def move move_al seternatevieeooonnn(selll on "Move alternate viewpoint look the object
Useful when object occluded or an interaction failing due to collision occlusion.
pass

---

Listing 4: Full Prompt the PLANNER. () indicates areas that are replaced the prompt.
You are an adept at translating human dialogues into sequences of actions for household robots. Given a dialogue between a <Driver> and a <Commander>, you convert the conversation into a Python program to be executed by robot.
[API)
Write a script using Python and the InteractionOjject class and functions defined above that could be executed by household robot.
CEETRIEVED_EXAMLLES
Adhere to these stringent guidelines:
1. Use only the classes and functions defined previously. Do not create functions that not provided above.
2. Make sure that output consistent plan, For example, opening of the same object should not in successive steps.
3. Make sure the output is consistent with the proper affordances of objects. For example, couch cannot be opened, so your output should never include the open() function for this object, but fridge be opened.
The input is dialogue between <Driver> and <Commander>. Interpret the dialogue into robot actions. Do output any dialogue.
Object categories should only chosen the following classes: ShowerDoor, Cabinet, Top, Sink, Towel, HandTowel, TowelHolder, SoapBar, ToiletPaper, ToiletPaperHangger, Hweloowel,,, SoapBottle, GarbageCan, Candle, ScrubBrush, Plunger, SinkBasin, Cloth, SprayBottle, Toilet, Faucet, ShowerHead, Box, Bed, Book, DeskLamp, BasketBall, Pen, Pillow, Pencil, CellPhone, KeyChain, Painting, CreditCard, AlarsClock, CD, Laptop, Drawer, SideTable, Chair, Blinds, Desk, Curtains, Dresser, Watch, Television, WateringCan, Newspaper, FloorLamp, RemoteControl, HousePlant, Statue, Ottoman, ArmChair, Sofa, DogBed, BaseballBat, TennisRacket, BacuumCleaner, Mug, ShelvingUnit, Shelf, StoveBurner, Apple, Lettuce, Bottle, Egg, Microwave, CoffeeMachine, Fork, Fridge, WineBottle, Spatula, Bread, Tomato, Pan, Cup, Pot, SaltShaker, Potato, SepperShaker, ButterKnife, StaveKnob, Toaster, DishSponge, Spoon, Plate, Knife, DiningTable, Bowl, Laundryllamper, Vase, Stool, CoffeeTable, Poster, Bathtub, TissueBox, Footstool, BathtubBasin, ShowerCurtain, TVStand, Boots, RoomDecor, Paper TowelRol1, Ladle, Kettle, Safe, GarbageBag, TeddyBear, TableTopDecor, Dumbbell, Desktop, AluminumFoil, Window, LightSwitch, AppleSliced, BreadSliced, LettuceSliced, PotatoSliced, TomatoSliced
6. You can only pick up one object at time. If the agent is holding an object, the agent should place or put down the object before attempting to pick up a second object.
7. Each object instance should instantiate different InteractionObject class even if two object instances are the same object category.
Follow the autput format provided earlier. Think step by step to carry out the instruction.
Write a Python script that could be executed by a hausehold robot for the following: dialogue: (command) Python script:

---

Listing Full Prompt for the RECTIFIER, indicates areas that are replaced in the prompt.
You are an excellent interpreter of human instructions for household tasks. Given failed action subgoal by a household robot, dialogue instructions between robot <Driver> and user <Commander>, and information about the environment and failure, you provide sequence of robotic subgoal actions to overcome the failure.
[API)
CAPL_CORRECTIVE.
Information about the failure and environment are given follows:
Failed subgoal: The robotic subgoal for which the failure occured. Execution error: feedback as to why the failed subgoal occurred.
Input dialogue: full dialogue instructions between robot <Driver> and user <Commander> for the complete task. This or may not be useful.
will give you examples of the input and output you will generate. pretrieved_plans.
Fix the subgoal exectuion error using only the InteractionÜbject class and functions defined above that could be executed by household robot. Follow these rules very strictly:
Important! Use only the classes and functions defined previously. additional code that are provided the above API. Do include if-else statements. Important! Make
create functions
that you output consistent plan. example, opening the object should not occur successive steps.
3. Importane ! Make sure the output is consistent
the proper affordances objects. For example, couch cannot opened, so your output should never include the open() function for this object, but a fridge can opened.
4. Important! The dialogue is between <Driver> and <Commander>. The dialogue may or may not helpful. Do not output any dialogue.
5. Important! Object classes should only be chosen from the following classes: ShowerDoor, Cabinet, CounterTop: Sink, Towel, HandTowel, TowelHolder, SoapBar, ToiletPaper, ToiletPaperHanger, HandTowe oowloldeer,, SoapBottle, GarbageCan, Candle, ScrubBrush, Plunger, SinkBasin, Cloth, SprayBottle, Toilet, Faucet, Showertlead, Box, Bed, Book, DeskLamp, BasketBall, Pen, Pillow, Pencil, CellPhone, KeyChain, Painting, CreditCard, AlarsClock, CD, Laptop, Drawer, SideTable, Chair, Blinds, Desk, Curtains, Dresser, Watch, Television, WateringCan, Newspaper, FloorLamp, RemoteControl, HousePlant, Statue, Ottoman, ArmChair, Sofa, DogBed, BaseballBat, TennisRacket, VacuumCleaner, Mug, ShelvingUnit, Shelf, StoveBurner, Apple, Lettuce, Bottle, Egg. Microwave, CoffeeMachine, Fork, Fridge, WineBottle, Spatula, Bread, Tomato, Pan, Cup, Pot, SaltShaker, Potato, PepperShaker, ButterKnife, StoveKnob, Toaster, DishSponge, Spoon, Plate, Knife, DiningTable, Bowl, LaundryHamper, Vase, Stool, CoffeeTable, Poster, Bathtub, TissueBox, Footstool, BathtubBasin, ShowerCurtain, TVStand, Boots, RoomDecor, Paper Rowelll1,,, Ladle, Kettle, Safe, GarbageBag, TeddyBear, TableTopDecor, Dumbbell, Desktop, Aluminumfoil, Window, LightSwitch, AppleSliced, BreadSliced, LettuceSliced, PotatoSliced, TomatoSliced
6. Important| You
only pick up one object at time. If the agent is holding an object, the agent should place or put down the object before atterpting to pick up second object.
7. Important! Each object instance should instantiate different InteractionObject class even if two object instances are the same object category.
8. Important! Your plan should ONLY fix the failed subgoal. Do not include plans for other parts of the dialogue future plan that are irrelevant to the execution error and failed subgoal.
9. Important! output rot_nnothi if the agent should not take any corrective actions. Adhere to the output format I defined above. Think step by step to carry out the instruction.
Make of the following information help you fix the failed subgoal:
Failed subgoal:
Execution error: Input dialogue:
You should respond the following format:
Explain: there any steps missing to complete the subgoal? Why did the failed subgoal occur? what does the execution Plan (Python script): Python script only fix the execution error.
imply for how fix your future plan?
Explain:

---

Listing Full Prompt for the LOCATOR, indicates areas that are replaced in the prompt.
You are household robot trying to locate objects within a house.
You will be given target object category, your task is to output the top 3 most likely object categories that the target object category is likely to be found near: (CLASSESC For your answer, take into account commonsense co-occurances of objects within house and (if relevant) any hints given by the instruction dialogue between the robot <Driver> and user <Commander>.
For example, if given the target object category is "Knife" and the following dialogue: "<Commander>>
hi, make a slice of tomato. <Driver> where is the tomato? <Driver> where is the knife? <Commander> in the sink.", you might output the following top nost likely object categories:
"answer: Sink, CounterTop, Cabinet". Important: Your output should ONLY list (3 words seperated by commas) of three object categories from the list above.
What are the top 3 most likely object categories for where to find the target category SAPPT__ARGET_OBBE near? Here is the dialogue that may may not contain relevant information for your answer: (INPUT_DIALOGUE)

---

Listing 7: Example of a precondition check for the slice macro-action
if Subgosl=="Slice:
contraint check for "slice" macro-action if htwee_trrcker.ee_laageeee " first need to be holding a knife to slice progran_to_ado = [
"target_knife Interactionbbjeec((nnnee
Boarget_kiife.g_.uue vo_to()';
"target_knife fic.feepppcpp)

---

instantiate tomato slice object after Listing 8: Example of dialogue imput, retrieved examples, and LLM output for TEACh sample
Dialogue input:
<Driver> how can I help you today?<Commnder>> can you please make me a salad on a clean plate wi with tomato and cooked potato?<Driver? does the salad require chopped settuce?<Comandeen <Dpev<rriver> is that all7<Commander> can you place them on plate?<Driver> are they not already on plate?
Retrieved Examples:
Example 1:
dialogue: <Driver> What should I today? <Commander> hi, make a slice tomato. <Driver> where is the Tomato? <Driver? where is the knife? <Commander> in the sink, <Driver> Tomato next? <Comander> slice the potato. <Driver> Where is the potato? <Commander>, in the microwave. <Comnander>> place all salad components
Tomato sliced. What
plate. <Driver> How many slices of potato? <Commander> all salad components need to be place on a plate. <Driver> where is the plate? <Commander> plate. <Commander> try the tomato piece. <Driver> done. <Commander> we are finished. Python script:
target_tomato
target_knife Interactiondbject"" ("Knife") # need knife slice tomato
karget_knife,, www.kkiiffee.oo.ot.. g) first get knife
target_knife.piice. (kckp()
Larget__tmaato go_
target_tomato. slice()
sarget_tcmato_ssicae Interractonnbjjeec((Tooooto
slicing tomato
target_potato = Interactiondbject(Pttaa landmark = "Microwave")
target_potato.gee go_
target_potato.slice
Barget_potato_slicn
Intrractiooojje((ooocacsiieell benet_knife.put_ddoee
instantiate potato slice object after
slicing potato
put down the knife to free the agent's hand
target_plate intteractonobbeet((ell)
target_tomato_slice httpaattt_ssiellpcccup..ommmae wtwp:aacdpppcceeaeeeeee..c
pick up the tomato slice and place it the Carget_potato_slice htaptt/sssicee.pccuu.aadddd.eeo picku__an__pacccttrret_pp
plate
pick up the potato slice and place it the
plate
Example 2:
dialogue: <Driver> how help? <Commander> please
slices of tomato plate. <Driver' sure. <Driver> where can i find the tomato? <Diiver> an knife, <Commander> Tomato on counter top and knife in fridge. <Commander> there should be plate the right cupboard by the sink. <Driver> completed. <Comnander? great work thanks.
Python script:
target_tomato = Interactionbjeec(TTnnttt # need knife slice tomato
target_knife Interactionobjeet("nii)
Sarget_knife.gg go)to() first get knife
target_knife.picce
barget_tomato.slicc))
target_plate Interaatiimmbjeect((Paattt Landmark "Sink")
place slices of cut tomato the plate
target_tomate comato_slicel "ferraattooobeecctt"ootttttttll
instantiate first slice tomato instantiate second slice of tomato instantiate third slice tomato instantiate fourth slice of tonato Larget__omato_stt slice4 interaatttonbjeec(("ooooollllll pick up the first tomato slice and place it on
target_tomato__ _sicee
target_tomato_st slicee "reeraatiooobeectt("oontssss
setgee__tmatt__siiccss commatt_slliee..pccku.... the plate
target_tomato_sl semmatt_slice_.pppm hit.ie/ppiittppmmpaaeeeaaeeeeee
pick up the second tomato slice and place it
on the plate prrget_tomato_ssieee the plate
htttto_siiiee..cccp__pmpeeee sickup_.nn_ppaceetttseeepp..pp pick up the third tomato slice and place it on
target_tomato_stl slicto__sseee_... hwl.cce.pickup...ee pickup_and sickupp.nnpllaeeettggee...aa. pick up the fourth tomato slice and place
on the plate
Example 3:
<Driver> hi, what is task. <Commander>> make a plate of toast. <Commander>> bread in the sink. <Driver> knife. <Commander> on the chair beside table. <Commander> clean the plate. <Driver> there is a clean one, do need to clean the other one. <Commander> you can use the clean one. <Driver> done. <Commánder> good job. <Driver> Python script:
dialogue:
target_bread Interacttiondhjeet(Breeedd Landmark "Sink") sereeciiiooojee("eeee target_toasen toaster InteractionObject
target_knife
landmark "Chair")
("rooieeecTTTTtrrrr"" ("Toaster") need knife to slice bread
targe sete__knife.goooof first get knife

---

target_knife, p
target_bread. -
target_bread. slice() slice bread using knife
target_knife.. .efeeeepuu__ooo()) put down the knife to free up agent's hand
Larget_bread_slice InteractionObjet( "BBreadSlice"" (arrciionbjee("BBrrrllleeee""
instantiate bread slice object after
slicing
barget_bread_slicel..
# pick up the bread slice and place in the
toaster
target_toaster: wooaster.toggle_nee toggle_on() turn on toaster to toast bread
target_toaster toatterrooggl_oo((( # turn off toaster to complete toasting process # place toast on plate
target_plate (iteractioomdjjec(((aattt""
target_plate.cleem
target_plate. pttppppaaae.p__oon(__ Barget_bread_slicee www.ccc.pcccpp...p.llcctttrreeet # pick up the toasted bread slice and place it on the plate
LLM output:
target_ tomato = InteractionObject ("Tomatttt"")
target_potato = InteractionObect( (""oiii"atcc"""Pato""
target_plate
ineracciinnbjecti ("Platee)
target_stove Interactionobjet(( (wrrrooooooeettttSttuurrrre "Sttoeeuuner"")
# Clean the plate
target_plate.go_ target_plate.clen clean)()
# Cook the potato
target_potato.2 to)
target_potato pickup()
Larget_stove.go_cc target_potato. (aaaeotttppaeerteeessstttto target_stove. toggle_on() target_stave. toggleooff))))
Place the cooked potato the plate
target_potato Interacttonobbjec((Ptttat target_potato. pittttpppp target_plate, _o_t (twp://ttttoppaaeetaeee.e.e....
Place the tomato on plate target_tomato.g (oomato.go_t(()
target_tomato. picttoppp target_plate.go_cat pereectcoomoopllaccttaeeecccc

---

Listing Example of dialogue imput, retrieved examples, and LLM output for TEACh sample
Dialogue input:
<Driver> what do i do today<Commande>> Hi. Please place two tissue boxes on table. One is on the gray chair the corner. <Commander> other under tv<Commander> tv*<Driver> where is the other one
Retrieved Examples:
Example 1:
dialogue: <Driver> hi, what is task. <Commander> make plate of toast. <Commander>> bread in the sink. <Driver> knife. <Commander> on the chair beside the table. <Commander> clean the plate. <Driver> there <Driver> clean one, do need to clean the other one. <Commander> you can use the clean one. <Commander> good jab. <Driver> :).
Python script:
target_bread seerractioooojeet((eeee
landmark "Sink") target_knife Interactiondun "Knife", Landmark "Chair") toaste_ = InteractionObject (ecTooosse ("Toaster") need knife to slice bread
go_to() # first get knife wttpe//kiiffeiiiiiiiie
wwagge__beed..eee go
target_bread. bread.slice( slice bread using knife
target_knin knife. put_down()
down the knife to free the agent's hand sesee__beed_sleeee = InteractionObeet(( ("rraattte""raasiieed # instantiate bread slice object after
slicing bread
parget_bread_slicet
# pick up the bread slice and place it in the
toaster
target_toaster. toggle_on()
toaster to toast bread
target_toaster.. toggle_off() turn off toaster complete toasting process
horrroottteer.tggle_oooffoo place
plate
target_plate Interrattionbhjet(((Patt))"
target_plate.cll ()eann)
target_plate.put. put__dow(()
target_bread_ the plate
patppp.on___cceettrree...apa pick up the toasted bread slice and place it on
Example 2:
dialogue: <Driver> how can help? <Commander> please serve slices tomato on plate. <Driver> sure. <Driver> where i find the tomato? <Driver> knife. <Commander> Tomato countertop and knife in fridge. <Commander> there should be plate on the right cupboard by the sink. <Driver> completed. <Commander> great work thanks.
Python script:
target_tomato Interactionobjeet
Landmark "CounterTop")
target_knife Interactiondbjeet("Minnn ("Knifffe, landmark "Fridge") need knife to slice tomato
srrget_knifeego_coo first get knife
target_knife. pickup))
target_tomato.go..
target__omato.3 slice()
target_plate " ("Plate, Landmark = "Sink")
# place slices of the cut tomato on the plate
barget_tomato_ssice
interacttonObjectt"Too "oSiic"et""""""") instantiate first slice of tomato iiteraattoonbbeett((Toooott
Barget__toatto_lliee
instantiate second slice of tonato instantiate third slice of tomato
target tomatostttmmttssiccee wtt.ppp.aan..ppccectaaeeeee..... (lace
interractoonbjjec((Toooottott InteractionOb (tteraattiooeeeee(Tmmmtssiliii
instantiate fourth slice of tonato pick up the first tomato slice and place it on
target_tomato__l11
wtl.iicelppickp..... pickup_and
the plate
Larget_tomato_ss slicosssceee...... pick:ppcckppaapg hatpppand__llee(aaret_...... pick up the second tomato slice and place it
the plate
Barget_tomato_slll slicessllee...... htt.ssscee..iicpp.eeee
httpppammpplaceeaaaeet........ pick up the third tomato slice and place it on
the plate
pick up the fourth tomato slice and place it
the plate
Example 3:
dialogue: <Driver> How help? <Commander>> put all the scrub brushes on the counter <Driver> where can find then? <Comnander>> one is on top of the toilet. <Driver> there is only cloth. <Driver> I put it the counter <Dr are there more? <Commander> can try with the of the toilet please? <Driver> okay.
brush to
Python script:
target_scrub_bruse brush1 = Interactionbjjcct(Sruubbbbss
landmark "Toilet")
target_counter_tor interactionObjeet("" ("Contttrro"" target_scrub www.scrbbbuuuhh..r..eeeeu the CounterTop
pick the first ScrubBrush and place. it

---

barget_scrubbbruus brush2 aereciionnbeeet(ssstrrrrrrr landmark "Toilet")
target_sscrub brushhh pickup_.and cacckup.nd_llacettrrre...ee pll.ceetarget..oouteee.com pick up second ScrubBrush place CounterTop
LLM output:
terget_tissue_box Interactiooojjet((iissseoo landmark "Chair")
target_table Internctiondbjec(((Taaaee))
target_tissue__t aickpp..md_pllccctareet...dd it pick up the first TissueBox and place it on the
Table
target_tissue__btt box2 Interactionobjjec((Issssuxx landmark "TVStand")
trrget_tissue_btt picckp_.ar._placeeerttttbbbbbbl # pick up the second TissueBox and place it on the Table

---

Listing 10: Example of dialogue input, retrieved examples, and LLM output for a TEACh sample
Dialogue input:
<Driver> hi how can i help<Commander> Can you prepare me coffee in a clean mug<Comaande>> There
should be mug on the white cable<Cormander>, it might need cleaning firstsCommander>> that's
great<Driver> are they not already on plate?
Retrieved Examples:
Example 1:
dialogue: <Commander> Prepare coffee in clean mug.
Python script:
target_mug Interactiondhjeet(MMal
target_mug.go...
target_mug.picken
# clean mug in the sink
target_mug. clean()
target_coffer _aceeemaaaeeeeeee = Interactionobjec(((offeeee
target_coffee_machinn go_to()
target_mup meceeeinemaeeeeeeeeeeeeeeee
Barget_coofee_marle toggle_on() turn on coffee machine to make coffee target_coffee. wew_eeeaaaaeee.. toggle_off() turn off coffee machine after done
Example 2:
dialogue: <Driver> what can for you today? <Commander> could you boil a potato? <Driver> sure thing! <Dr Oprr> are there any pots? <Comnander>> pots for boiling potato? <Driver> yes. <Commander> just the one that were holding earlier and couple others of the same size. <Commander> does it <Commander>
the pot? <Driver> yes. mug <Commander> you re welcome! <Commander> pour the mugs water into the pot. <Driver> thanks. <Commander> excellent thank you!
fit the sink? <Driver>
Python script:
target_pot Interactiondhject("ott"
target_mug Interactiondhjett "Mug", landmark "Pot")
sorget_mugggo_too
target_mug.o (rrget_mug-piccuOOO
Larget_mug.pour pourppuraare (targee__pp))
pttp__muugpp___doo_[_ put down mug on any location to free up agent's hand
target_potato = pactionojject"Pooot
target_potato.p wwwp:oppcccpppcppeeeeeee
pick up the potato and place it in the pot target_stove__nod Interaction(bject, ("stovekooo"") # turn on the stove to boil potato by turning on the stove
target_stove_knob knob.go_to())
target_stove_knod toggle_on() boil water
target_stave_kmnb. toggle # complete boiling process by turning off
Example 3:
dialogue: <Driver> hi, what is task. <Commander>> make plate of toast. <Commander>> bread in the sink, <Driver> knife. <Commander> on the chair beside the table. <Commander> clean the plate. <Driver> there is clean one, do need to clean the other one. <Commander> you can use the clean one. <Driver> done. <Commänder> good job. <Driver>
Python script:
target_bread interactionbbjeec(BBeeddd landmark = "Sink")
sarget_knife Interacti inobjeect ("Knife", landmark = "Chair")
target_toaster = interactionObject, ("Toastrr"""" "Toaster") need knife to slice bread
target_knife.go.
first get knife
target_knife. pickup()
prrget_breadd bte.d..o_.ooo)) go)
slice bread using knife
ferrget__iife..ppe petdffnnut_doo( put down the knife to free the agent's 's hand garget_bread_slice = Interactionobjet( (Breadiiid))))) # instantiate bread slice object after
slicing bread
Barget_brea__slicee
htt.:/bra....iie...piip....adddd ciccupp.anpplccetaarrettttsssss # pick the bread slice and place it in the
toaster
target_toaster. toggle_on() turn toaster toast bread
target_toaster toggle_off() # turn off toaster complete toasting process
# place toast plate
target_plate Intersattimobbeet(((lattt")
target_plate.clles
target_plate.put put_down()
srrget_brea__slicer pickkpp.. prrckcpp.nn__lleeeaarree....t pick up the toasted bread slice and place it the plate

---

LLM output:
target_table Interacttionbbjec(((aalee""""
target_mug InteractionObjeec(("ee"""" landmark "Table")
target_mug.go__ (
serget_mug.piccupp
clean mug in the sink
target_sink = ""teracttoonjjjec((""sn""
carget_mug.cleang
serget_coofee_mmcchiee
interaccioobecc(fffffeehhiie place(target_ccomm stt.coffe_acchnee
go_tto(
target_mug.plaat
toggle_an() # turn on coffee machine to make coffee
target_coffee cooffee_machine toggle_off() turn off coffee machine after done


Reviews:
[{"review_id": "kx35NX4rYz", "reviewer": "Reviewer_xAvR", "summary": "", "questions": "- Question A: How often is a program from the examples copied or nearly copied to the output? Is the main load on the retrieval augmentation?\n- Question B: How do you initialize the memory with dialogue-program pairs?\n- Question C: How well performs HELPER without the Inspector? Why is it missing in the ablation? Is this method commonly used also in other work compared to?\n- Question D: The overall success rate is still quite low. What are the failures cases? Do you have an estimate for human or a perfect-planner performance?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough comments. We address your concerns below.\n\nQ: **How well does HELPER perform without the Inspector? Why is it missing in the ablation? Is this method commonly used also in other work compared to?**\n\nA: **We conducted the ablation study you suggested by removing the Inspector component and  observed only a 0.4% success rate decrease without the Inspector component.**  This shows the influence of our pre-conditions checks on HELPER's performance is minimal. We have since included this ablation in the revised manuscript.  We would like to clarify that, similar to HELPER, **the most competitive previous work, notably Danli (Zhang et. Al. 2022), incorporates pre-condition checks as part of their FormPDDLProblem component** and error correction module. This demonstrates that our approach is in line with the baselines, and thus directly comparable with them. Additionally, **our Inspector module does not use any  privileged information** that is not standard in the TfD TEACh evaluation. This ensures that our comparison with other techniques is fair. \n\nQ: **The length weighted success is only about the same or similar to its competitors.**\n\nA: The reason for this is the following:  Our depth estimation network is much more accurate for near depths than for far  depths, as discussed in Section 4.2. **This leads to an extended mapping time, affecting the length-weighted success.** **When employing ground truth depth, our approach not only maintains a significantly higher success rate but also exhibits a substantially improved path weighted metric compared to baselines.** Thus, when a precise depth sensor (e.g., LiDAR) is used or a more accurate depth estimation network, our method is expected to surpasses the baseline models in path-weighted metrics. We will add this explanation to the text. \n\nQ: **Missing results for openly available (huggingface) models (only GPT-3 and GPT-4 were tested)**\n\nA: The OpenAI Python API we use is publicly available and we use the GPT-4-0613 version. It is widely used on several paper publications. The reason is that models currently hosted on huggingface are weaker in code generation. \n\nQ: **The presentation lacks clarity on the overall interaction between the various components (Planner, Rectifier, VLM, Executor, Locator, Inspector). Would be hard pressed to reproduce the results.**\n\nA: Thank you for the constructive feedback regarding HELPER's module clarity.\nTo address your concerns, we have added an algorithm box, a visualization to demonstrate the interaction of the modules, and the following text:\n\n> The HELPER system is an embodied agent designed to convert natural language instructions and RGB sensor data into a sequence of robot actions. The system has a modular organization outlined in Algorithm S1 and visualized in Figure S4. The PLANNER module uses nearest neighbor retrieval to obtain the most relevant task plans from a memory bank by matching language encoding of accompanying dialogue. The retrieved language-plan pairs are used as in-context examples inside the prompt for an LLM to transform user instructions or dialogue into an executable program. This generated program is then sent to the EXECUTOR module, which translates each program step into specific navigation and manipulation actions. To support this, the system uses RGB sensor data to produce segmentation and depth images at each time step. These images are used for object identification and for building a semantic map, which is subsequently used for object tracking and navigation. Before executing each subgoal, the INSPECTOR module verifies if the necessary preconditions for an action, such as the robot already holding an object, are met. If not, the plan is adjusted according to the current environmental and agent state. Should a subgoal involve an undetected object, the EXECUTOR calls on the LOCATOR module. This module utilizes previous user instructions and LLMs' commonsense knowledge to efficiently search for the required object. If any action fails during execution, the RECTIFIER module is used. It predicts the reason for this failure from pixel input using vision language models and suggests a set of corrective subgoals.\n\nWe have added additional text descriptions of each of our modules in the paper. This text includes detailed descriptions, inputs/outputs, models used, and relations to other modules for the following aspects of HELPER: The `PLANNER`, `EXECUTOR`, `LOCATOR`, `RECTIFIER`, `INSPECTOR`, user feedback, user personalization. \n\nTo improve reproducibility and ease of understanding, we have also added an accessible repository link containing our code in the revised version of the paper, as we are not able to include code here. \n\nAll models and tools used are all publicly available, including OpenAI's API for LLM inference, and perception models like SOLQ, ZoeDepth, and ALIGN. It's worth noting that the perception modules or LLM are modular and can be swapped with alternative networks as desired.\n\nQ: **How often is a program from the examples copied or nearly copied to the output? Is the main load on the retrieval augmentation?**\n\nA: Our memory bank contains only 19 examples, which are derived from training samples. These examples do not span the variety of dialogue and tasks in the evaluation set. Consequently, the LLM cannot succeed by merely replicating  these examples, instead it uses them as in-context references. We did not train the retriever, but instead used  pre-trained language encoders.  Even without retrieval (Section 4.2), the LLM's performance is consistent with the previous SOTA. This showcases the LLM's capability to generate accurate outputs, even in the absence of the most relevant examples, showcasing its potential for more than simply copying. \n\nQ: **How do you initialize the memory with dialogue-program pairs?**\n\nA: We start with a memory bank of 19 examples (11 for the planner and 8 for the rectifier). These examples were selected from the first 100 training samples. Each dialogue from the chosen examples is paired with a program, manually written by a domain expert familiar with the task. \n\nOur approach required merely a few hours to initialize the memory bank. Despite its simplicity and speed, our model outperforms previous methods that necessitated months of data collection and training. For instance, the TEACh author's process of collecting 1482 training demonstrations and subsequent data cleaning is considerably more time-consuming. Additionally, the top-3 competitive baselines finetune an LLM to predict subgoals, which also demanded extended durations.  Despite being more efficient, our method displays an increased generalization to diverse input text, like user feedback and user customization requests (Sections 4.2 & 4.3), even when such feedback considerably differs from the dialogues in the training set.\n\nQ: **What are the remaining failure cases?**\n\nA: As you are pointing out, the task remains challenging, and there's room for improvement. The evaluation includes only episodes where human performers succeeded.\n\nThe main failure cases are as follows:\n\n- **Perception failures** \nWe observe a 2X increase in success rate using ground truth 3D object segmentation and labelling. Our system's modularity allows us to  replace our object segmentator and labeller with better ones  as they become available.\n\n\n- **Task Complexity** As tasks become more complex and long time horizon that require additional subgoals, we noticed a decrease in success rate. This decline is attributed to errors propagating across subgoals, sometimes arising from the LLM planner's inaccuracies, occasional error correction failure of a subgoal, or ineffective retrieval of task-relevant examples.\n\n- **Object Referential Grounding** Our current model has limitations when pinpointing objects described in referential noun phrases, like \"second cabinet to the left of the sink.\" If a general category-based search does not find the object, our search module might fail in locating the target object. \n\n\n\nQ: **Missing references for retrieval-augmentation and continual learning**\n\nA: We have added all the references you are pointing out to our related work with appropriate descriptions. \n\nQ: **References and typos**\n\nA: We sincerely appreciate your attention to detail. All the typos and references issues you pointed out have been corrected in the revised manuscript.\n\nQ: **How can the plan proceed when the state might have changed by the rectifier?**\n\nA: Thank you for bringing up this point of clarification. If the state changes, this does not affect the plan execution process. We do maintain an object memory map that keeps track of each object and its visual attributes (e.g., sliced, clean). The plan is adjusted on the fly based on these object attributes  (open/close, clean/dirty, open/close), which are critical during our *pre-condition checks*. The system evaluates the LLM's current plan based on the present object attributes and  dynamically adjusts it. For instance, if the bread is already sliced, the task of slicing is bypassed. At submission time, we used a mechanism that initializes object attributes to default values and updates them when actions are executed (like slicing). We will clarify this in the paper as part of the module and figure revisions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- a novel demonstration of connecting various pre-trained components into a working system\n- almost no training involved (only fine-tuning the object and depth detector)\n- new state-of-the-art in TEACh-TfD w.r.t. success rate and an ablation that shows the benefit of program retrieval (2.12%)", "reasons_to_reject": "- the presentation lacks clarity on the overall interaction between the various components (Planner, Rectifier, VLM, Executor, Locator, Inspector) and the general execution flow\n- the comparison with other techniques seems hard because of the bias that the Inspector component induces (\"the inspector verifies the preconditions [...] are all met [...] and adjusts the plan accordingly (L386-394)\")\n- missing results for openly available (huggingface) models (only GPT-3 and GPT-4 were tested)", "questions_to_authors": "- Question A: How often is a program from the examples copied or nearly copied to the output? Is the main load on the retrieval augmentation?\n- Question B: How do you initialize the memory with dialogue-program pairs?\n- Question C: How well performs HELPER without the Inspector? Why is it missing in the ablation? Is this method commonly used also in other work compared to?\n- Question D: The overall success rate is still quite low. What are the failures cases? Do you have an estimate for human or a perfect-planner performance?", "paper_topic_and_main_contributions": "This work presents a system of various pre-trained components (Planner, Rectifier, VLM, Executor, Locator, Inspector) that achieves a new state-of-the-art success rate on the TEACh-TfD benchmark. The length weighted success is only about the same or similar to its competitors (Episodic-Transformer, JARVIS, FILM and DANLI).\n\nThe authors built upon a technique that uses LLMs to produce executable code to navigate an agent in a virtual household environment and add a retrieval-augmentation memory to the mix. The memory mechanism is responsible to retrieve previously successfully executed programs given a natural language dialogue, so that these are placed into the the language model's prompt template as examples. This approach can also be used as a continual learning technique by to adding more programs after deployment. And the authors add a mechanism to the replan and recover plans.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "The \"context-dependant LLM prompting\" is highly related to retrieval-augmentation. For example [1] retrieve the top-k image caption candidates from an external datasource to augment the generation of a caption for an image (similarly context-dependant LLM prompting retrieves programs to produce a program).\n\n- [1] Sarto, S., Cornia, M., Baraldi, L., & Cucchiara, R. (2022, September). Retrieval-augmented transformer for image captioning. In Proceedings of the 19th International Conference on Content-based Multimedia Indexing (pp. 1-7).\n\nThe \"memory expansion\" is a variant of continual learning [2] that has been studied for example in [3] to map new natural language inputs to program execution and in [4] to improve an instruction following agent. \n\n- [2] Timothée Lesort, Vincenzo Lomonaco, Andrei Stoian, Davide Maltoni, David Filliat, Natalia Díaz-Rodríguez:\nContinual learning for robotics: Definition, framework, learning strategies, opportunities and challenges. Information Fusion\nVolume 58, June 2020, Pages 52-68.\n\n- [3] Sida I. Wang, Samuel Ginn, Percy Liang, and Christopher D. Manning. 2017. Naturalizing a Programming Language via Interactive Learning. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 929–938, Vancouver, Canada. Association for Computational Linguistics.\n\n- [4] Ta-Chung Chi, Minmin Shen, Mihail Eric, Seokhwan Kim, Dilek Hakkani-Tür: Just Ask: An Interactive Learning Framework for Vision and Language Navigation. AAAI 2020: 2459-2466\n", "Typos_Grammar_Style_and_Presentation_Improvements": "- L013: \"our model executes the program\" -- how can a model execute a program? maybe rephrase?\n- L072: semicolon should be replace by \"and\"\n- L085: \"PEsonalized\" -- is missing an \"r\"\n- L096: \"prompts helps the agent to parse language inputs\" -- I do not directly see the connection here: is the idea that text helps to parse text?\n- L109: \"HELPER navigates home environments\" -- virtual (!) home environments\n- L359: \"it uses data\" -- the data should be formally described or at least an example should be presented\n- L514: missing reference to Table 2\n\nGeneral Notes:\n\n- Figure 2: B is not showing what \"happens\" with the program; the connection to A should be made more clear\n- the general flow should be depicted: as far as I understand there is a loop involved, the VLM checks at each step for problems, then the Rectifier gets involved and then the plan is proceeded; but how can the plan proceed when the state might have changed?\n\nReferences: \n\n- lots of pre-prints (>25!), arxiv references should be checked again e.g. Progprompt is actually published in ICRA 2023\n- \"2022. New and improved embedding model.\" is missing authors and venue\n- \"Multilingual vision-and-language navigation with dense spatiotemporal grounding\" is missing authors and year"}}, {"review_id": "2mvWubGjbX", "reviewer": "Reviewer_hr3J", "summary": "", "questions": "1. How HELPER benefited from visual information? If the LLM Planner plans without vision input, how can it understand the current object status, such as if a bread is already sliced when the goal is to prepare a sandwich.\n\n2. Could the HELPER be applied to other two tasks in TEACh dataset?\n\n3. How many modules are there in total within the HELPER and how are they correlated? An independent figure that high-levelly shows the modules is highly encouraged to be added to the paper. \n\n4. How the user feedback is collected? If the HELPER fails to succeed after getting the user feedback once, will it receive another user feedback?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q: **The TEACh dataset also presents EDH task and TATC task, but HELPER is only evaluated in TfD task. Could the HELPER be applied to other two tasks in TEACh dataset?**\n\nA: **Per your suggestion, we have evaluated HELPER on the EDH task of TEACH.** For EDH, we provide the LLM with the subgoals completed thus far by the Driver in addition to the input dialogue and instruct the LLM to only output the remaining program to complete the task. We use the agent history provided by the EDH episode to initialize our semantic map and object memory.  **HELPER achieves 18.3% success and 31.5% goal condition success on validation seen**. Previous SOTA in this task is Danli (Zhang et. Al. 2022)  with  16.89% success and 25.10% goal condition success. \nHELPER not only outperforms the baselines, but it does so without any in domain training or finetuning on the TEACh dataset. Our focus was initially on the TfD task because of its increased complexity and similar task structure to a robot deployed in a real household. We will include these results in the camera ready.\n\nThe TATC task isn't yet available for comprehensive evaluation. Please refer to the TEACh github repository for more information. Nonetheless, our user feedback experiments did incorporate the \"Progress Check\" action from this task, showing HELPER's ability to handle task feedback. We will evaluate  HELPER on the TATC task upon its full release.\n\nQ: **The HELPER relies heavily on the text and it seems that the HELPER only uses image information when the execution fails and when searching for target objects.**\n\nA: Thank you for your comment. \n\nOur method in fact uses more visual information than the baselines! It uses visual information in the following ways: \nWe detect objects in 3D and track their location over time in our 3D object memory map, which we use for object search.   Visual image information is also used for implementing the manipulation and navigation actions predicted by the LLM, e.g., manipulating an object requires segmenting the object in the input 2D image.  Our error correction relies on visual data and uses off-the-shelf Vision-Language Models (VLMs) for discerning causes of failures. Section 4.2 demonstrates the importance of this feature in our pipeline. **On the contrary, previous methods, (e.g., Danli; Zhang et. Al. 2022) often use manually-defined subgoal error correction without visual context.**  \n\nYou are right that the LLM does not plan with visual inputs, and that visually-grounded planning is helpful for these tasks. Instead, our model adjusts the LLM's plan on the fly based on object attributes  (open/close, clean/dirty, open/close), which are used in pre-condition checks. For instance, if the bread is already sliced, the task of slicing is bypassed. At submission time, we used a mechanism that initializes object attributes to default values and updates them when actions are executed (like slicing). Although this method meets the requirements of the Teach benchmark, we acknowledge that real-world scenarios need visual attribute classification, as you are pointing out!\n\n**Following up on your suggestion, we have implemented a vision-based attribute classification** for each object added to our object memory.      We replace the default attribute initialization with CLIP-based attributed  classification. Specifically, we obtain the boolean value of each attribute by taking the best match of the image crop around each detected object with text prompts such as “The {object label} is {attribute}” and “The {object} is not {attribute}”. \n**Using the  CLIP attribute classifier for object attribute initialization instead of the default one, we achieve similar performance (-0.4% success difference; -0.1% goal condition difference; valid unseen)**. This shows that the default initialization of object attributes we used in the paper is sufficient for the TfD task. However, we anticipate  a general method beyond dataset biases of TEACh would much benefit from such vision-based attribute classification, and we thank you for raising this point.\n\nQ: **Ambiguity in HELPER's module descriptions. Would be hard pressed to reproduce the results.**\n\nA: Thank you for your feedback. To improve reproducibility and ease of understanding, we have  added an accessible repository link containing our code in the revised version of the paper, as we are not able to include code here. All models and tools used are all publicly available, including OpenAI's API for LLM inference, and perception models like SOLQ, ZoeDepth, and ALIGN. It's worth noting that the perception modules or LLM are modular and can be swapped with alternative networks as desired.\n\nTo improve clarity, we have added an algorithm box, a visualization to demonstrate the interaction of the modules, and the following text:\n\n> The HELPER system is an embodied agent designed to convert natural language instructions and RGB sensor data into a sequence of robot actions. The system has a modular organization outlined in Algorithm S1 and visualized in Figure S4. The PLANNER module uses nearest neighbor retrieval to obtain the most relevant task plans from a memory bank by matching language encoding of accompanying dialogue. The retrieved language-plan pairs are used as in-context examples inside the prompt for an LLM to transform user instructions or dialogue into an executable program. This generated program is then sent to the EXECUTOR module, which translates each program step into specific navigation and manipulation actions. To support this, the system uses RGB sensor data to produce segmentation and depth images at each time step. These images are used for object identification and for building a semantic map, which is subsequently used for object tracking and navigation. Before executing each subgoal, the INSPECTOR module verifies if the necessary preconditions for an action, such as the robot already holding an object, are met. If not, the plan is adjusted according to the current environmental and agent state. Should a subgoal involve an undetected object, the EXECUTOR calls on the LOCATOR module. This module utilizes previous user instructions and LLMs' commonsense knowledge to efficiently search for the required object. If any action fails during execution, the RECTIFIER module is used. It  predicts the reason for this failure from pixel input using vision language models and suggests a set of corrective subgoals.\n\n2. We have added additional textu descriptions of each of our modules in the paper. This text includes detailed descriptions, inputs/outputs, models used, initializations, and relations to other modules for the following aspects of HELPER: The `PLANNER`, `EXECUTOR`, `LOCATOR`, `RECTIFIER`, `INSPECTOR`, user feedback, user personalization. \n\nQ: **User feedback misses some details.**\n\nA: We generate user feedback as explained in the following text. We have added this to the paper.\n\n> In the user feedback evaluation, once the agent has indicated completion of the task from the original input dialogue, the agent will query feedback from the user. If the simulator indicates success of the task, the agent will end the episode. If the simulator indicates the task is not successful, feedback will be given to the agent for additional planning. This feedback is programatically generated from the TEACh simulator metadata, which gives us information about if the task is successful, and what object state changes are missing in order to complete the task (e.g., bread slice is not toasted, etc.). For each object state that is incorrect, we form a sentence of the following form: \"You failed to complete the subtask: {subtask}. For the object {object}: {description of desired object state}.\" We combine all subtask sentences to create the feedback. HELPER follows the same pipeline (including examples, retrieval, planning, etc.) to process the feedback as with the input dialogue in the normal TfD evaluation. We show experiments with one and two user feedback requests in Section 4.2 of the main paper (a second request is queried if the first user feedback fails to produce task success). \n\nQ: **Figure 2 contains too much information and blends too many key information together which makes reader lose focus easily. Break figure 2 into several figures would be beneficial.**\n\nA: Thank you for your feedback. Per your suggestion, we have broken down the figure into several figures, and included figures for each module in the revised supplemental.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed HELPER uses a novel modular design including modules such as the Planner module, Rectifier module and etc. Extensive experiments are conducted to study the HELPER's performance. The personalization study shows the potential of the HELPER to be applied to more general tasks.", "reasons_to_reject": "Here presents a few concerns based on the paper and more questions related to each concern are detailed in the next section.\n1. The HELPER relies heavily on the text and it seems that the HELPER only uses image information when the execution fails and when searching for target objects. \n2. The TEACh dataset also presents EDH task and TATC task, but HELPER is only evaluated in TfD task.\n3. The modules in HELPER are not described clearly enough. Especially the input and output of each module lack some details. \n4. User feedback misses some details.", "questions_to_authors": "1. How HELPER benefited from visual information? If the LLM Planner plans without vision input, how can it understand the current object status, such as if a bread is already sliced when the goal is to prepare a sandwich.\n\n2. Could the HELPER be applied to other two tasks in TEACh dataset?\n\n3. How many modules are there in total within the HELPER and how are they correlated? An independent figure that high-levelly shows the modules is highly encouraged to be added to the paper. \n\n4. How the user feedback is collected? If the HELPER fails to succeed after getting the user feedback once, will it receive another user feedback?", "paper_topic_and_main_contributions": "This paper presented an embodied agent, HELPER, that grounds free-form human-robot dialogue instructions and corrections to visuo-motor programs. With the help of the memory-augmented pretrained LLM prompting method, the HELPER achieves a new SOTA on TEACh TfD benchmark. ", "excitement": 2, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Figure 2 contains too much information and blends too many key information together which makes reader lose focus easily. Break figure 2 into several figures would be beneficial. "}, "extra_text": {}}, {"review_id": "1bCgLkHrUT", "reviewer": "Reviewer_31Xh", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q: **The evaluations are limited to TEACh. What about evaluation on CVDN (Thomason et al., 2019).**\n\nA: Thank you for your suggestion. You are correct that our model can be easily generalized to other tasks, and we agree this would be useful to show. Per your  (and hr3J's) suggestion, we have evaluated  HELPER to the EDH task of TEACH.** For EDH, we provide the LLM with the subgoals completed thus far by the Driver in addition to the input dialogue and instruct the LLM to only output the remaining program to complete the task. We use the agent history provided by the EDH episode to initialize our semantic map and object memory.  **HELPER achieves 18.3% success and 31.5% goal condition success on validation seen**. Previous SOTA in this task is Danli (Zhang et. Al. 2022)  with  16.89% success and 25.10% goal condition success. \nHELPER not only outperforms the baselines, but it does so without any in domain training or finetuning on the TEACh dataset. Our focus was initially on the TfD task because of its increased complexity and similar task structure to a robot deployed in a real household. We will include these results in the camera ready.\n\nThank you for bringing the CVDN task to our attention. We were not aware of this, and we think it would be a useful addition to the paper. The task involves a different simulation environment and thus we dis not have time to include these results in this rebuttal. We are working on completing this evaluation.  \n\n\n\nQ: **The overall organization of the paper could be greatly improved. A unifying problem statement would be helpful.**\n\nA: Thank you for your valuable feedback on the paper's organization. To improve the writing clarity we will do the following:\n\n1. We will include a unifying problem statement with an example in the introduction to improve the clarity of the problems and proposed solutions:\n\n> In a home setting, consider a robot that receives task instructions from a human user. These instructions, as depicted in Figure 1 (Left), may be in free form natural language, with omissions, as well as references to previous objects and events, as shown in Figure 1 (Right). To understand the user's intent and generate a correct action sequence, the robot can refer to past successful interactions that are contextually relevant to the current instruction. As it carries out the task, the robot may need to make real-time adjustments, such as when encountering challenges involving a cluttered placement area (Figure 1, Middle). To navigate these challenges, the robot should use visual feedback to understand why a particular sub-task failed and adjust its plan accordingly.\n\n2. To  address your concerns about organization, we have added an algorithm box, a visualization to demonstrate the interaction of the modules, and the following text:\n\n> The HELPER system is an embodied agent designed to convert natural language instructions and RGB sensor data into a sequence of robot actions. The system has a modular organization outlined in Algorithm S1 and visualized in Figure S4. The PLANNER module uses nearest neighbor retrieval to obtain the most relevant task plans from a memory bank by matching language encoding of accompanying dialogue. The retrieved language-plan pairs are used as in-context examples inside the prompt for an LLM to transform user instructions or dialogue into an executable program. This generated program is then sent to the EXECUTOR module, which translates each program step into specific navigation and manipulation actions. To support this, the system uses RGB sensor data to produce segmentation and depth images at each time step. These images are used for object identification and for building a semantic map, which is subsequently used for object tracking and navigation. Before executing each subgoal, the INSPECTOR module verifies if the necessary preconditions for an action, such as the robot already holding an object, are met. If not, the plan is adjusted according to the current environmental and agent state. Should a subgoal involve an undetected object, the EXECUTOR calls on the LOCATOR module. This module utilizes previous user instructions and LLMs' commonsense knowledge to efficiently search for the required object. If any action fails during execution, the RECTIFIER module is used. It  predicts the reason for this failure from pixel input using vision language models and suggests a set of corrective subgoals.\n\n3. We have added additional text descriptions of each of our modules in the paper. This text includes detailed descriptions, inputs/outputs, models used, and relations to other modules for the following aspects of HELPER: The `PLANNER`, `EXECUTOR`, `LOCATOR`, `RECTIFIER`, `INSPECTOR`, user feedback, and user personalization. \n\nQ: **The frequent use of “visuomotor” is a bit of an overclaim.**\n\nA: Thank you for pointing this out. We have removed all occurrences of this word in the paper, we agree that it can be misleading as we do not use any low-level skill visuomotor policies, given that the simulator API abstracts away from physics details.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "+ HELPER is highly applicable to robotics and embodied-AI applications. Error correction and user personalization are common issues in robotics. Most attempts either hardcode solutions or rely on fixed databases with extensive human engineering. HELPER uses LLM prompting as a unified approach to tackle these issues.\n+ HELPER achieves compelling quantitative results on the TEACh benchmark. HELPER improves over the previous best method by 1.4x on task success and 2.54x on goal-condition success (partial goals).\n+ The experiments include a good set of baselines and ablations. Several models, i.e ET, JARVIS, FILM, and DANLI, are investigated. Table 2 includes extensive ablation experiments that investigate HELPER’s individual components. \n", "reasons_to_reject": "- The overall organization of the paper could be greatly improved. There are at least 3 different problems tackled by HELPER: (1) dialogue grounding, (2) failure correction, and (3) user personalization. Perhaps before diving into methods, a unifying problem statement would be helpful, or at least a unifying example that motivates these problems, and why the paper chose to specifically tackle them. \n- The frequent use of “visuomotor” is a bit of an overclaim. “Visuomotor” commonly refers to connecting perception and low-level motor skills (at least in robotics). But HELPER only outputs high-level API actions like “pickUp(apple)”. This comment may seem like an issue with word choice, but it hints at a deeper limitation in the use of symbolic action spaces. Certain actions like “pulling noodles” or “folding clothes” might be difficult to abstract and parameterize with symbolic actions. \n- The evaluations are limited to TEACh. It seems like HELPER is generally applicable to other domains like navigation. Perhaps running some evaluations on CVDN (Thomason et al., 2019) and other vision-dialogue-tasks could show the generalizability of HELPER?  \n\nReferences:  \n\nThomason et al 2019 – https://arxiv.org/abs/1907.04957 ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents HELPER, an embodied agent that uses LLMs to plan, correct errors, and personalize behaviors for instruction-following tasks. HELPER is set in the TEACh environment, which consists of dialogue-based interactive tasks. HELPER takes language descriptions of the current and past observations, and outputs API-level symbolic programs. To incorporate feedback, the agent retrieves the closest language-program pair from a prior dataset. Experiments on the TEACh benchmark indicate that HELPER substantially outperforms prior SoTA approaches by 1.4x. \n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposed a new system containing mostly pre-trained components for the TEACH benchmark of virtual mobile manipulation from dialog history or instructions.

The strengths highlighted by reviewers are:
- Good relevance to robotics
- Strong results of the proposed method
- Convincing empirical evaluation on standard benchmarks and baselines (this was initially lacking, but got significantly improved during rebuttal with additional experiments)
- Method requires minimal training.

Outstanding weaknesses after rebuttal are:
- Organization of the paper and confusion over how the many modules interact. The authors have attempted to address it.
Decision: Accept-Findings