LoTa-Bench: Benchmarking Language-oriented Task Planners for Embodied Agents
Authors: Jae-Woo Choi
Youngwoo Yoon
Hyobin Ong
Jaehong Kim
Minsu Jang
Venue: ICLR 2024 poster (2024)
Keywords: task planning
language models
benchmarking
embodied agents
home robots
Primary area: datasets and benchmarks

Abstract: Large language models (LLMs) have recently received considerable attention as alternative solutions for task planning. However, comparing the performance of language-oriented task planners becomes difficult, and there exists a dearth of detailed exploration regarding the effects of various factors such as pre-trained model selection and prompt construction. To address this, we propose a benchmark system for automatically quantifying performance of task planning for home-service embodied agents. Task planners are tested on two pairs of datasets and simulators: 1) ALFRED and AI2-THOR, 2) an extension of Watch-And-Help and VirtualHome. Using the proposed benchmark system, we perform extensive experiments with LLMs and prompts, and explore several enhancements of the baseline planner. We expect that the proposed benchmark tool would accelerate the development of language-oriented task planners.

Published as a conference paper at ICLR 2024
LOTA-BENCH: BENCHMARKING LANGUAGE-ORIEN- TED TASK PLANNERS FOR EMBODIED AGENTS
Jac- Woo Choi Youngwoo Yoon' Hyobin Ong Jachong Kim', Minsu Jang*.?
1 Electronics and Telecommunications Research Institute
2 University of Science and Technology
[ [jwcho107]] 0717, h youngwoo, chnghb, jhk im504, minsu re.kkrrekkk.
ABSTRACT
Large language models (LLMMs) have recently received considerable attention alternative solutions for task planning. However, comparing the performance of language-oriented task planners becomes difficult, and there exists a dearth of de- tailed exploration regarding the effects of various factors such as pre-trained model selection and prompt construction. To address this, we propose a benchmark sys- tem for automatically quantifying performance of task planning for home-service embodied agents. Task planners are tested on two pairs of datasets and simula- tors: 1) ALFRED and A12-THOR, 2) an extension of Watch-And-Hell and Vir- tualHome. Using the proposed benchmark system, we perform extensive experi- ments with LLMs and prompts, and explore several enhancements of the baseline planner. We expect that the proposed benchmark tool would accelerate the devel- opment of language-orienteds task planners.
INTRODUCTION
The ability of emmbodidd agents to comprehend natural language instructions and perform the desired tasks has been Long-standing goal in the field of Al and robotics. When the agent has sufficiently diverse skill set, decomposing high-level tasks into sequences of executable skills becomes particu- larly important. Conventional approaches have addressed this challenge through symbolic planning in predefined domains (Fikes Nilsson 1971] Garrett al. 2020) or through learning- based task and motion planning (Silver et al. [023 Shah al. 2022, Li et al. 2022). Recently, large language models (LLMs) have emerged as promising alternative. These models, pre-trained on extensive
corpora, seem to have semantic knowledge about the world (Brown et al. 2020 Chowdhery et al.) Thoppilan et al. 2022: Zhang et al. 2022). This knowledge can effectively leveraged for
2023
high- level task planning through in-context learning without any additional training (Huang al.
2022
Singh et al. 2023 Liang et al. 2023a Ahn et al. 2023
Huang et al. 2023 Yao et al. 2023
However, the evaluation frameworks for LLM-based task planning remain underdeveloped. Most existing studies rely on human evaluation, which is not only time-consuming but also expensive. These evaluations often occur in custom environments, which also makes them difficult to repro- duce. Although some research (Huang et al. 2023 Liang et al. (022a)) has utilized simulators for automated evaluation, these efforts are typically confined to simple tabletop manipulation tasks. Fur- thermore, there is noticeable absence of in depth investigation into various influential factors, such as the type and size of pre-trained model, the number and select strategy of in-context examples, the capability for replanning based on natural language feedback, and the impact of fine-tuning.
To address the limitations, we introduce LoTa-Bench, a benchmark for language-oriented task plan- ning for embodied agents. Our system aims to automatically quantify planning performance, en- abling easier, fair, and reproducible comparison between systems. The framework consists a baseline task planner, a dataset, and simulator, illustrated in Figure [] The baseline task plan- ner capitalizes on the in-context learning ability of LLMs. constructs a prompt using a prefix, in-context examples (comprising pairs of natural language instructions and corresponding skill quences to accomplish the instruction), and a user-provided natural language instruction. With this
Equal contribution.

---

Published as a conference paper at ICLR 2024
Robott Hi there, Ihm robet operating home:
LLMM-aaeed Task Planner
Simulator
A12- AI2-THOR
licbot: can
serous
Pre-trained LLM
NL to API
you scquence actional
sccomp bih your tulk,
Step
API Call
spoen,
sptoon
Prompt
Skill
Simulator Environment
Instructoon
Human: Move book dest safi fibott find book. pick sook, Al (ind sofa, down the book. done
VirtualHome
Parning
rosulti
Input Query
Skill Sei Construction
Intiatization
Evaliaaioo
Skill Set
doose the buthroom cabinet, disse the dishweshee.... find bar soup, find coffie find cutlery fork... pick up the apple, pick up the book, pirk up (witch on the compuser switch the by.
Instruction
Environment
Initial
Goal Condition
Dataset
Figure I: Overall benchmarking configuration for LLM-based task planners. NL stands for Natural Language. We used two setups: 1) ALFRED dataset with A12-THOR simulator and 2) WAH-NL dataset with VirtualHome simulator. Exemplary prompt and skill set are presented on the left side.
prompt, the LLM calculates the probabilities all executable skills to complete task. The skill with the highest probability is selected and appended to the prompt for the next step in autore- gressive manner. In the proposed benchmark suite, we evaluate the planner on two
pairs: 1) ALFRED dataset (Shridhar al.] (2020) with A12-THOR simulator (Kolve et al.) 2017). and 2) our extension of Watch-And-Han Help (WAH) dataset (Puig et al.) (02T), WAH-NI, paired with VirtualHome simulator (Puig al.] 2018). Each dataset sample furmishes the planner with both natural language instruction and an environment context. The simulator executes the planned ac- tions, and the performance of task planning is automatically assessed by comparing the final state of the simulator with predefined goal conditions.
In addition to the introduction of the benchmark suite, we provide extensive experimental results to further understand I.LM-based task planning. Our baseline experiments explore the influence of various pre-trained models and their sizes. Given the critical impact of in-context examples on the performance of LLLM-based task planners, we investigate the effect of number of examples and selection strategies. Additionally, we probe into other influential factors such as replanning according to the failure of previous step and the effectiveness of model fine-tuning in the task planning domain.
Our contribution is fourfold: 1) first proposal of benchmark suite that enables automatic evaluation of LLM--based task planners for home-service agents, 2) extensive experiments of a bascline task planner, 3) exploring possible extensions of the baseline planner and its validation with the proposed benchmark, and 4) public release of benchmark code and extended dataset (WAH-NL): they are available at https:: petpps:/ggihubb com www.oo///aaa///11..e.ooo 2tt.aa22222LLMMaaaa www.Maaakllannnngggg
2 RELATED WORK
L.MSs have demonstrated remarkable generalization capabilities through zero-shot few-shot prompting (Brown et al.) 2020), leading to a transformative impact on task planning. Traditional
task planning methods predominantly focused on searching within predefined domains (Fikes &
Nilsson 1971 Garrett et al.) 2020 Hoffmann] 2001) or learning trajectories (Silver et al., 2023
Shah et al. [ [022 Li et al. 2022:10 lchter al. 2022, Nair & Finn 2020 Eysenbach et al. 2019
Xu et al. 2019). However, thanks to LLMs, new language-orr -rriutee- task planning methods have emerged. Huang al.] (2022) proposed a method where an LLM directly generates task plans prompt engineering, with each generated step translated into an executable action using another language model. SayCan (Ahn et al., 2023) employed an L.MM to score all predifined admissi- ble actions, concurrently considering skill affordance through leamed vision-based value functions.
LLMs have also been adopted to generate executable robot codes using program-style inputs such as function descriptions (Liang al.) 2023a Singh et al. 2023 Zelikman et al. 2023). Moreover, inte-
grating context into LLsed task planners has been shown to enhance planning efficacy (Huang
al. 2023
Yao al. 2023 Chen et al. 2023 Lin et al. 2023 Wu et al. 2023).
Although numerous LLM-based task planners have emerged, standardized automatic performance evaluation methods still scarce. Real robot experiments typically require time-intensive human
2

---

Published as a conference paper at ICLR 2024
evaluations. In these setups, human raters determine the success failure of planning (Ahn al., 2023 Huang et al. 2023 Chen et al. 2023). When using simulators and datasets for evaluations, each task requires the goal condition and the natural language instruction. If a dataset lacks goal con- ditions, such as ActivityPrograms (Puig et al. 2018), human evaluation remains necessary (Huang eet al. 2022 Zelikman et al.
2023). Similarly, datasets without natural language instruction, like Watch-And Help (Puig et al.
2021). or simulators not offering high-level APIs, such as Behavior- Ik et al. 2023), cannot support language-oriented task planning. Only a few studies, akin to our benchmark suite, have incorporated automated evaluations. For instance, ReAct (Yao et al. 2023) utilized the ALFWorld (Shridhar et al., 2021 text-based game and the ALFRED dataset (Shridhar al. 2020). ProgPrompt (Singh et al., 2023) engaged with the VirtualHome simulator and cus- tomized dataset. Nevertheless, these assessments were conducted in restricted settings, hindering insights into comprehensive potential of LL-M-based task planners.
3 BASELINE LLM-BASED TASK PLANNER
Problem Statement. In our proposed framework, a task planner receives natural language in- struction i from the user, e.g., "bring an apple and a cupcake and put them on the coffee table." The planner also has access to a predefined skill set S, where each skill sES represents atomic action the agent can perform, such as "pick up the apple." " "find a wine glass." or "open the fridge." We assume that these skills coupled with corresponding language-condttione low-level controllers (Jang et al. 2022] BBrohan et al. (023). The objective of the task planner is to select the skill st at time step by maximizing the likelihood of completing the given instruction as follows:
$2 arg max p(s/i, S0," 1t-1),
SES
where Sp, ".. 1 are previously executed skills and so = Exemplary step sequences for the instruction we mentioned above could be (I. find an apple, 2. pick up the apple, 3. find coffee table, put down the apple, 5. find a cupcake, 6. pick up the cupcake, 7, find a coffee table, 8. put down the apple, 9. done).
Baseline Task Planner. Our baseline task planner leverages the in-context learning capabilities of large language models (LLMs), resonating with recent research (Huang et al. 2022 Liang al. 2023a Ahn et al., 2023). To estimate the probability expressed in Equation|] we construct a prompt P. which consists of prefix, in-context examples, the instruction i, and history of previously
ecuted skills. For a skill s, described by n, subword tokens s = (wf, ... ",w,, ). the LLM computes the conditional probability as follows:
p(sli, 81, 8t-1) PP) = II PLIM ...... " -11
where PLLM is the pre-trained LLM and wi = 0. Instead of iterating every skill to find the best next skill to perform (Equation D. which requires extensive computation, we employ a greedy search strategy, but with constraints the next token selection match with one of the skills using Guid- ance tool (Microsofi) 2023); see Appendix B for details. Once a skill is selected, it is appended to the prompt P. and the planner continues use the updated prompt to select the next skill. This au- toregressive process continues until either the terminal skill ("donee") is selected or the skill sequence reaches a predefined maximum length.
BENCHMARK SETUP
To rigorously evaluate LLM--based task planners, we introduce a comprehensive evaluation frame- work, described in Figure [ The framework integrates three key components: task planner, dataset, and a simulator. The baseline task planner elaborated in Section 3 employed for compar- ative benchmarking. Then we offer two distinct dataset-simulator pairings: 1) the ALFRED dataset Shridhar et al., 2020) built on the A12-THOP TH-R simulator (Kolve et al., 2017). and 2) an extended version of the Watch-And-Hed (WAH) dataset (Puig et al., (021), named WAH-NL, incorporated with the VirtualHome simulator (Puig et al. 2018). Further details of the dataset and the simulator described in the following subsections.
3

---

Published as a conference paper at ICLR 2024
4.1 DATASET
Our benchmark employs two datasets, the ALFRED dataset (Shridhar et al. (020) and our extension of the WAH dataset (Puig et al.] 2021). Both datasets include sets of a natural language (NL) instruction, an initial environment state, and goal condition for home environments. The NL. instructions are user-provided and serve as inputs to the task planner of an autonomous agent. The initial environment states, containing object locations and states, used for initialization of the simulator and for skill set construction in the task planning. The goal condition specifies the criteria for task completion. Planning performance is assessed by comparing the final state of the simulator with this goal condition after the execution of the last skill generated by the task planner.
ALFRED. It is a benchmark dataset for embodied AI agents that plan and execute primitive actions to perform household tasks, such as heating mug cup. placing a salt shaker in a drawer, or putting vegetables in the fridge. This dataset was built on the A12-THOR simulation environment. There are 7 task types of Pick & Place, Stack & Place, Pick Two & Place, Clean & Place, Heat & Place, Cool & Place, and Examine in Light. Among them, we excluded Pick Two & Place type in the evaluation because of missing capability of object instance recognition, which is required to accomplish this task type, in the LLM-based task planner.
WAH-NL. The original WAH dataset focuses on the challenges of Al agents assisting humans in household tasks. It consists of a Watch stage where agents observe human demonstrations to infer goals, and a Help stage where agents assist human in achieving those goals with minimal time steps. The dataset includes 5 task types of Setup a dinner table, Put groceries. Prepare a meal, Wash dishes, and Read book. The goal condition of each task consists of multiple subgoals. For example, the goal condition of Pul groceries task can be "INSIDEccupcake, fridge): 2" and "INSIDE(appl,, fridge): 1", where the numeric values indicate the number of objects.
Our extended version, WAH-NL, introduced two significant modifications on the Help stage of the original WAH dataset. First, we adjust the goal conditions, originally designed for human-Al collab- orations, to suit autonomous agents, Additionally, we set the number of objects for all subgoals to 1, for similar reasons as with the ALFRED dataset (the lack of object instance recognition capability in our LLM-based task planner). Second, since the original dataset lacks NL instructions, which is must-needed element for language-orienten oriented task planners, we collected them via the Prolific crowd- sourcing platform. The final dataset includes 416 instructions for the train set and 195 for the test set. More details about WAH-NL are described in Appendix E
4.2 SIMULATOR
The simulator serves as an interactive environment that enacts the skills generated by the task plan- ner. We first define skill set for the task planner by combining available actions with optional parameters like target objects or receptacles. The skill set includes, for example, "find an apple," "turn on the faucet," "open the fridge". and "put down the pillow." Then, simulated language- conditioned low-level controllers by mapping the skills to executable agent action APIs of the sim- ulators.
Our approach primarily employs two types of skills: object navigation (ObjNav) and object inter- action. We opted for ObjNav over low-level move actions (such as move forward and rotate) as ObjNav is well studied and would be considered as unit capability of home-service agents (Duan) al. 2022). We assume that the agent fully aware of object locations in the scene (practically, can be accomplished by scene exploration with object map building). Object interaction skills are executable only when the interacting object is close to the agent. For example, "pick up the plate" is successful when the distance between the plate and agent is less than predefined distance. We had some other assumptions for the object interaction skill. For the "put down' skill, the agent always put a holding object receptacle last visited. If there are multiple objects with the same class, the object closest to the agent is selected. The agent can hold one object in A12-THOR and two objects in VirtualHome.
AL2-THOR. There are seven interaction actions- "pick up." "open," " "close," " "turn on," "turn off," "slice," and "put down""and one navigation action "find." Among all combinations of action and optional parameters such as target objects or receptacles, we used 214 skills that were used at least once in the ground-truth trajectories in the train set.
4

---

Published as a conference paper at ICLR 2024
20
OpenAl
Subgoal surcess rate 1%5
40
OpenAl
GPT-Neo
GPT-Neo
OPT
OPT
MPT
Success
10
LLaMA 1
LLaMA
5-
10
10
10
10a
10a
10'
10
20*
Model size (Millions
Model size (Millions)
(a) ALFRED
(h) WAH-NL
Figure 2: Baseline results on (a) ALFRED and (b) WAH-NL. We report task success rates (%) on the ALFRED dataset and average subgoal success rate (%) the WAH-NL datset for language models in different model classes and sizes (number of parameters). Base language models are represented solid lines. Fine-tuned models (by either instruction chat data) were shown in dashed line with a triangle maker.
VirtualHome. This simulator supports five interaction actions - "pick up," "open," "close," "cloee, "switch on," " and "put down" and one navigation action "find." VirtualHome, due to the extensive total number of skills, we constructed skill set by considering all possible combinations of actions and optional parameters for each environment. On average, we utilized 351.89 skills.
5 BASE EXPERIMENTS
We conducted experiments to measure the performance of the baseline LLM-based task planners by using the proposed benchmark. We tested various settings including LLMs in different model classes and sizes and the impact of the number of in-context examples.
5.1 EVALUATION PROTOCOL
Test Setup. We used the ALFRED and WAH-NL datasets, as introduced in Section 4.1 The ALFRED dataset consists of three sets: train, valid-seen, and valid-unseen. The valid-seen used to evaluate planning performance; the train set was only used to take examples to construct prompts. We used a small set of valid-seen, which has 208 samples (30% of the valid-seen set), accelerate the evaluation of various configurations (see Appendix C for the results with the full set). The WAH-NL dataset comprises train set and test set with 250 and 100 samples, respectively. The train set was used to construct in-context examples, and the test set was used to evaluate planning performance.
The prompt comprises prefix, describing the role of a home-service embodied agents, and number of in-context examples, which adhere to the template defined in SayCan (Ahn et al. 2023). Each example is a pair of an input query (task instructions) and the corresponding output step sequences to accomplish the task. See Appendix F for a complete prompt sample. The in-context examples were randomly selected from the train sets, and we tried to use the same number of examples per task type. The default setup is to include six examples in ALFRED and five examples WAH-NL (one example per task type).
Evaluation Metrics. We measured planning performance using the task success rates for ALFRED. Task success was determined based whether the final state after executing the step sequence generated by the task planner satisfies the expected goal condition. For WAH-NL, we measured the average subgoal success rate. Each WAH-NL task has multiple subgoals. We calculate the individual subgoal success rate as the ratio of successfully completed subgoals to the total number of subgoals for each task. The average subgoal success across fest set reported.
5.2 BENCHMARK RESULTS OF BASELINE TASK PLANNER
We evaluated the planning performance of the baseline planner described Section 3 Figure shows the results ALFRED and WAH-NI. for different pre-trained LLMMs: GPT (Brown
5

---

Published as a conference paper at ICLR 2024
et al. 2020).
GPT-Neo series (Black al.] 2021: Wang & Komatsuzaki 2021 Black et al.
2022). ). OPT Zhang et al. 2022). ) MPT (Team 2023b a). LLaMA (Touvron et al. 2023a). and
LLMM (Touvron et al. 2023b) (see Appendix Afoo the list). A few fine-tuned models on instruc-
tions or chat data were also tested.
Overall, task success rates increased with the size of the language model, but this was not always the case. For example, GPT-J 6B model performed better than GPT-NeoX 20B and OPT was better than OPT 6.7B in the WAH-NL experiment. Such results, that a smaller model performs better than a larger model, were also observed in HELM evaluation (Liang et al. 2023b), especially in reasoning tasks. GPT-3 (ext-davinei-0003 showed the best success rate of 31. on ALFRED and the best subgoal success rate of 40.8 40.82% on WAH-NL. LLaMA 2 and MPT performed well considering their model sizes. Instruction- and chat-tuned models (dashed lines in Figure 2 did not perform better than their base models. We also tested on GPT-4, the state-of-the-art L.M. As OpenAl provides only chat-style APIs for GPT-4 unlike other base models such as GPT-3, we were unable to directly compare GPT-4 in the same configuration. We modified experimental configurations and assessed GPT-4's performance. GPT-4 performed well in ALFRED, showing a 40.38% success rate, a 19% improvement over GPT-3. However, in WAH-NL, GPT-4 showed lower success rate of 34.17% compared to GPT-3. More details in Appendix D
We conducted a further analysis of the task types. For ALFRED, we found that the small model such as GPT-J 6B succeeded only for simple Pick & Place tasks and failed in complex tasks such as heating and cooling tasks, which require longer steps than simple tasks. The largest model, GPT-3 175B, succeeded similarly in both simple and complex tasks (20-30% success rates), except for the task type Stack & Place where the agent needs stack multiple objects in order. Additional results are shown in Table Appendix. For WAH-NL, all task types have a similar level of complexity, generally requiring the finding and placement of multiple objects. This resulted in a more balanced performance across task types when compared to ALFRED. Using our GPT-3 175B model, Put Fridge tasks yielded the highest average subgoal success rate at 54.50%, while Prepare Snack tasks registered the lowest average subgoal success rate of 25.00% See Figure 4 for the success samples (more results in Appendix[H]) H
We also examined the detailed reasons for the failure cases of the ALFRED results using GPT- 3 model, which showed the highest performance. Out of 162 failure cases, the reasons were catego- rized into six classes: 1) Action planning failures, such as performing 'Pick' instead of 'Slice' when tomato needs to be sliced. 2) Object selection failures, like grabbing pan instead of a pot. 3) Absence of visual grounding, for instance, trying to grab an object inside a closed drawer, 4) Lack of physical understanding, such as failing to put down an object on the already occupied table. 5) Misunderstanding user instructions, failing to distinguish between a desk lamp and a floor lamp when the user specified 'Lamp'. 6) Ambiguous or incorrect user instructions, like confusing 'Glass' for 'Cup' in an instruction. The results are presented in Table I Most failures (about 60%) stemmed from high-level planning (classes I and 2). Challenges in visual and physical grounding (classes 3 and 4) highlight the importance of integrating context in planning, as discussed in Section 7 Furthermore, the role of clarity in user instructions (classes and 6) opens up new research direction for interactive clarification of an ambiguous tasks.
Failure category
# Failures
Action planning failure Object selection failure Absence of visual grounding
46 (28.4%)
51 (31.5%)
21 (13.0%)
Lack of physical understanding
15 (9.3%)
Misunderstanding inst.
10 (6.2%)
Ambiguous/incorec inst.
19 (11.7%)
Table 1: The number of failure cases of the ALFRED results using GPT-3.
We have investigated the impact of the number of exam- ples in prompt with LLaMA 13B model that supports longer context length of 4096. The success rate mostly in- creased when there are 0 to 30 examples on ALFRED and
(Subgoal) success otte 196)
ALFRED
30
20
0 to 15 examples on WAH-NL (see Figure 3). It was not able to test more than 15 examples on WAH-NL because of the maximum token limitation. Note that the pool of examples was fixed, which means that, for example, 6 and 12 examples share the same 6 examples. An addi-
Number of examplcs the prompt
Figure 3: (Subgoal) success rates for the different number of examples for in- context learning.
tional experiment was performed to see whether different sets of examples matter for the same number of examples.

---

Published as a conference paper at ICLR 2024
(a) ALFRED: success example for Cool & Place task.
apple
1:find an apple
Zipick the spole
3:find fridge
4:open the frid
S:put apple
fridge
7:opan fridge
apple
9:pick
applo
10:close the fridge
11:Find alcrouive
12:open the nicrowave
13:put down the apple
14:elose the mite oaleve
(b) WAH-NL: success example for Wash dishes task.
Instruction: Put the fork and plate in the dishwasher and turn it
2:open
plate
diLswashar
5:ftnd dtshaasher
6:put down the plato
7:fLnd fridge
8:open the fridge
9:find cutlery fork
10spick up the cutlery fork
11:d dishwasher
12:put down the cutlory fork
1.r lose
14: f:switch on the
Figure 4: Planning results. Success of (a) the ALFRED task and (b) the WAH-NL task when GPT-3 175B model was used. The input instructions, inferred steps, and scene images after each step execution are presented each figure. The scene images show agent's point of view on AL- FRED and third person's 's point of view on WAH-NL. Additional results, including failure cases, are provided in Appendix

---

Published as a conference paper at ICLR 2024
We selected different sets of 12 examples that were randomly drawn from the example pool of the training set. The performance on ALFRED varied from minimum of 9.62 to a maximum of 17.79 (with average of 13,61 and standard deviation of 3.22) for the LLaMA 2 13B model.
Additionally., we tested on both full sets of valid-seen and valid-unseen splits of ALFRED using LLaMA 13B model. Valid-unser set contains scenes not present in the train set. The success rate on valid-unseen was similar (17.70%) to the one on valid-seen (17.82%) as two sets are different in visual scenes, not in task types.
VALIDATING EXTENSIONS OF THE TASK PLANNER
The primary merit of the proposed benchmark is that it allows faster and easier validation of new task planners. To demonstrate this, we explore some extensions (or improvements) of the base planner and validate them.
6.1 IN-CONTEXT EXAMPLE SELECTION
We explored three strategies for selecting in-context examples from the train set containing both input queries and their associated planning examples. The first strategy, named Random Sampling, is the same that used in the baseline task planner. It involves random sampling of N examples for each unique task type, leading to a collection of x M examples across different task types, where M is the number of distinet task types. In the second strategy, termed Task-Specific Sam- pling, we select examples from the subset of the train set that share the same task type as the input instruction (with the assumption that the task type is known). Lastly, Semantic Similarity Sampling employs Sentence BERT (Reimers & Gurevych 2019), to compute the similarity scores between the input instruction and all instructions in the frain set. The examples with the highest similarity scores are selected as in-context examples. This strategy not only aims to select the most relevant planning examples, but also offers potential utility in real-world scenarios where task types may not be explicitly provided.
To measure the effectiveness of these in-context example selection methods, we conducted experiments on WAH- NL using LLaMA models. For all strategies, we set the number of in-context examples to five. The results are summarized in Figure Across all model sizes, Seman- tic Similarity Sampling showed superior performance, followed by Task-Specific Sampling, and lastly Random Sampling. Importantly, Semantic Similarity Sampling led to significant performance gains; notably, the LLaMA 65B model achieved subgoal success rate of 43. 43.25%, surpassing the best performance of 40.82% achieved by our baseline using GPT-3 175B. These experiments con- firm that the in-context example selecting strategy has significant impact on the performance of L-beed task planners. We found similar results with other LLM mod- els like GPT-Neo and LLaMA 2.
Subgoal surcess (96)
45
random
semantic
40
30
25
65B
Model size
Figure 5: Subgoal success rate for different in-context example selection strategies. The dashed line represents the best performance of our baseline planner using GPT-3 175B.
6.2 FREDBACK AND REPLANNING
The baseline planner selects the skill for the next step independent of the success or failure of previ- ous action. However, adjusting the plan in response to the failure of the action is necessary for task planning in the wild. Using the ALFRED and A12-THOR configuration, we investigated whether our LLM-based task planner can reflect feedback from action failure and replan appropriately. We added natural- natural-language (NL) feedback at the end of each inferred step only when the step is failed. NL feedback is constructed based on error messages from the AI2THOR simulator and environment states. For instance, when the pick up action failed because the target object is inside a container, "(this action failed: [Object] is not visible because it is in [Container])" is appended after the step failed in the prompt, and the next step is inferred afterward (see more information of feedback message in Appendix F.. We assumed that the agent is fully aware of the location of objects. Exam-
8

---

Published as a conference paper at ICLR 2024
ples demonstrating feedback and replanning were added to the prompt for in-context learning. We manually crafted 3 replanning examples and added them after the baseline prompt of 18 examples. selected from the existing 18 examples to minimize the addition of task information (I (Listing 3 in Appendix shows the additional in-context examples for replanning). LLaMA 2 model used in this experiment.
The task instructions for the replanning examples
Table shows that replanning is helpful to improve the overall planning perofrmance when a large model, LLaMA 2 70B, is used. smller model, LLaMA 13B, did not show improvements; this might be due to limited capability of the smaller model to understand complex concepts of task planning and re- planning only with few examples. The qualitative results showing how the planner succeeded by replanning steps are shown Figure 12 in Appendix
Success Rate(%) (%)
LLaMA 13B
LLaMA 70B
Baseline Replanning
21.15 (44/208)
21.63 (45/208)
17.79 (37/208)
24.04 (50/208)
Table 2: Results with and without replanning. Suc- cess rates on ALFRED are reported.
6.3 FINE-TUNING ON TRAIN SET
We conducted experiments to investigate the potential to improve planner performance through fine- -unittnnng. We fine- tuned LLaMA 1 models using LoRA (Hu et al. (021) on
Faseine uxiderer
Success Rate
the ALFRED train set and evaluated their performance in
the same ALFRED domain both for valid-seen and valid-
unseen tasks. For the evaluation of the fine-tuned planners, we used zero in-context examples as the planner had been explicitly trained. As depicted in Figure [ fine-tuning sig- nificantly improved performance, especially for larger mod- els, e.g. LLaMA 30B, which showed performance jump from 13.66% to 60.08%. However, the planner, fine-tuned on ALFRED, performed significantly worse than the base- line planners in the WAH-NL domain (32.22% - 10.38%), suggesting that procedural knowledge trained a task domain does not transfer well to different task domains. See Appendix G for fine- uning details and results. Appendix G also presents WAH-NL fine-tuning results. Notably, the limited number of WAH-NL train set led to marginal performance improvements. Specifically, the LLaMA 13B and 30B models showed increases of 5.77 and 1.56 percent points, respectively.
20
130
300
Model Size
Figure Success rates of fine-tuned planners on ALFRED.
CONCLUSION AND LIMITATION
The impressive generalizability and performance of the large language model (LLM) has facilitated its versatile deployment across multiple domains. Task planning for embodied agents is such application, and after the pioneering work proposed to use LLM for this application by Ahn et al.) (2023) and HHang et al. (2022), tion. However, there was few way to automatically evaluate planners. In this paper, we proposed a quantitative evaluation benchmark for LLM-based task planning research to catalyze the rapid ad- vancement of this field. The results of the base experiments and extensions of in-context example selection, replanning, and fine-tuning would be helpful to future studies. We hope that our proposed benchmark framework will serve as starting point for the development of various extended models for language-orient oriented task planners.
believe that diverse research efforts will continue in this direc-
The present work has the following limitations. First, we decoupled high-level plans and low-level actions focus on high-level planning. An extension needed to support an end-to-end system that considers both high-level planning and low-level actions. Visual understanding (egocentric views) also necessary for low-level actions. Second, as is common in many simulator-base studies, there exists domain gap between simulation and the real world. For example, ALFRED poses an unrealistic assumption that an object is always cleaned when was once put into water in a sink basin. Furthermore, although AI2THOR and VirtualHome support multiple scenes and objects, they still lack diversity to reflect real-world environments. Future work will be to extend the proposed benchmark to support wider range of systems and to reduce domain gaps.

---

Published as a conference paper at ICLR 2024
REPRODUCIBILITY STATEMENT
We provide the source code of the benchmark suite and the configurations reproduce the exper- imental results. WAH-NL dataset that we extended from the original WAH dataset to have task
instructions also available. See htttps://github com/ lbaa2022 LLMTaskP / Setrraskllanni... lanning for the souree code and WAH-NL dataset.
ACKNOWLEDGMENTS
This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2022-0-00951, Development of Awwrr Agents Learning by Asking Questions).
REFERENCES
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang. Kuang-Huet Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan, and Andy Zeng. Do as can, not as say: Grounding language in robotic affordances. In Proceedings of The 6th Conference Robot Learning, pp. 287-318. PMLR, 2023.
Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Neo: Large Scale Au- toregressive Language Modeling with Mesh-Tensorflow, March 2021. URL https://doi.
org/10.5281/ 52t.1to
Sidney Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Ho- race He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, Usvsn Sai Prashanth, Shiv- anshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. Gpt-neox-20b: An open-source autoregressive language model. In Proceedings of BigScience Episode #5 Workshop on Challenges & Perspectives in Creating Large Language Models, pp. 95-136, 2022.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alexander Herzog, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil Joshi, Ryan Julian, Dmitry Kalash- nikov. Yuheng Kuang. Isabel Leal, Kuang-Huck Lee, Sergey Levine, Yao Lu, Utsav Malla, Deek- sha Manjunath, Igor Mordatch, Ofir Nachum, Carolina Parada, Jodilyn Peralta, Emily Perez, Karl Pertsch, Jornell Quiambao, Kanishka Rao, Michael S Ryoo, Grecia Salazar, Pannag R Sanketi, Kevin Sayed, Jaspiar Singh, Sumedh Sontakke, Austin Stone, Clayton Tan, Huong Tran, Vincent Vanhoucke, Steve Vega, Quan H Vuong, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. RT-1: Robotics Transformer for Real-World Control at Scale. In Proceedings of Robotics:
Science and Systems, Daegu, Republic of Korea, July 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess. Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901, 2020.
Boyuan Chen, Fei Xia, Brian Ichter, Kanishka Rao, Keerthana Gopalakrishnan. Michael S Ryoo, Austin Stone, and Daniel Kappler. Open-vocabulary queryable scene representations for real world planning. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp- 11509-11522. IEEE, 2023.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh,
10

---

Published as a conference paper at ICLR 2024
Kensen Shi, Sasha Tsyyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard. Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Lev- skaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Bren- nan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy
Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-133, 2023.
Jiafei Duan, Samson Yu, Hui Li Tan, Hongyuan Zhu, and Cheston Tan. A survey of embodied ai: From simulators to research tasks. IEEE Transactions Emerging Topics in Computational Intelligence, 6(2):230-244, 2022. doi: 10. 1..1099/TTCC12222311111
Ben Eysenbach, Russ Salakhutdinov, and Sergey Levine. Search on the replay buffer: Bridging planning and reinforcement learning. Advances in Neural Information Processing Systems, 32, 2019.
Richard E Fikes and Nils Nilsson. Strips: A new approach to the application of theorem proving to problem solving, Artificial intelligence, 2(3-4):189-208 1971.
Caelan Reed Garrett, Tomás Lozano-Pérez, and Leslie Pack Kaelbling. Pddistream: Integrating symbolic planners and blackbox samplers via optimistic adaptive plamning. In Proceedings of the International Conference Automated Planning and Scheduling. volume 30, pp. 440-448, 2020.
Jörg Hoffmann. Ff: The fortadorward planning system. Al magazine, 22(3):57-57,000 2001.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Con- ference on Learning Representations, 2021.
Wenlong Huang. Picter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero- shot planners: Extracting actionable knowledge for embodied agents. In Proceedings of the 39th International Conference on Machine Learning, pp. 111999111111 PMLR, 2022.
Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Tomas Jackson, Noah Brown, Linda Luu, Sergey Levine, Karol Hausman, and brian ichter. Inner monologue: Embodied reasoning through planning with language models. In Proceedings of The 6th Conference on Robot Learn- ing. pp. 1769-1782, PMLR, 2023.
Brian Ichter, Pierre Sermanet, and Corey Lynch. Broadly-exploring local-policy trees for long- horizon task planning. In Proceedings of the Sth Conference on Robot Learning, pp. 59-69. PMLR, 2022.
Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. Bc-z: Zero-shot task generalization with robotic imitation learning. In Pro- ceedings of the 5th Conference on Robot Learning, pp. 991-1002. PMLR, 2022.
Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. Ai2-thor: An interactive 3d environment for visual ai. arXiv preprint
2017.
Chengshu Li, Ruohan Zhang, Josiah Wong, Cem Gokmen, Sanjana Srivastava, Roberto Martín- Martín, Chen Wang, Gabrael Levine, Michael Lingelbach, Jiankai Sun, Mona Anvari, Minjune Hwang, Manasi Sharma, Arman Aydin, Dhruva Bansal, Samuel Hunter, Lou, Caleb R Matthews, Ivan Villa-Renters Jerry Huayang Tang, Claire Tang, Fei Xia, Silvio Savarese, Hyowon Gweon, Karen Liu, Jiajun Wu, and Li Fei-Fei. Behavior- lk: A benchmark for embodied with 1,000 everyday activities and realistic simulation. In Proceedings of The 6th Conference Robot Learning, pp. 80-93. PMLR, 2023.
Kim, Alan
11

---

Published as a conference paper at ICLR 2024
Shuang Li, Xavier Puig, Chris Paxton, Yilun Du, Clinton Wang, Linxi Fan, Tao Chen, De-An Huang, Ekin Akyürek, Anima Anandkumar, Jacob Andreas, Igor Mordatch, Antonio Torralba, and Yuke Zhu. Pre-trained language models for interactive maacii..nmmaiig.... In Advances in Neural Infor- mation Processing Systems, volume 35, pp. 199-31212, 2022.
Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 9493-9500. IEEE, 2023a.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga,
Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Alexander Cosgrove, Christopher Manning. Christopher Re, Diana Acosta-Navas, Drew Arad Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue WANG, Keshav Santhanam, Laurel Orr, Lucia 7.heng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri S. Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Andrew Chi, Sang Michael Xie. Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Ieard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. Holistic evaluation of language models. Transactions on Machine Learning Research, 2023b.
Bill Yuchen Lin, Chengsong Huang, Qian Liu, Wenda Gu, Sam Sommerer, and Xiang Ren. On grounded planning for embodied tasks with language models. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 37. pp. 13192-13200, 2023.
Microsoft. Guidance. http://///ggttt......oo/ggaaaaaa
dance 2023.
Suraj Nair and Chelsea Finn. Hierarchical foresight: Self-supervised learning of long-horizon horizon tasks via visual subgoal generation. In International Conference on Learning Representations, 2020.
OpenAI. GPT API. htttp::/pplatform.co http:ppaaa..ccooocoppppppeeeeeee
2023.
[Online; accessed 19-September-2223].
Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Tor- ralba. Virtualhome: Simulating household activities via programs. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 8494-8502, 2018.
Xavier Puig, Tianmin Shu, Shuang Li, Zilin Wang, Yuan-Hong Liao, Joshua B. Tenenbaum, Sanja Fidler, and Antonio Torralba. Watch-and-help: challenge for social perception and human-ai collaboration. In International Conference on Learning Representations, 2021.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019.
Dhruv Shah, Alexander Toshev, Sergey Levine, and brian ichter. Value function spaces: Skill- centric state abstractions for long-horizon reasoning. In International Conference on Learning Representations, 2022.
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10740-10749, 2020.
Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Cote, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht, ALFWorld: Aligning text and embodied environments for interactive learning. In International Conference Learning Representations, 2021.
Tom Silver, Ashay Athalye, Joshua B Tenenbaum. Tomás Lozano Pérez, and Leslie Pack Kaelbling. Learning neuro- symuoliymmolic skills for bilevel planning. In Conference on Robot Learning, pp. 701- 714. PMLR, 2023.
12

---

Published as a conference paper at ICLR 2024
Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. Progprompt: Generating situated robot task plans using large language models. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 15223-11330 IEEE, 2023.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An eestuuctio-foolloing llama model. https: //gi thub. wtt.:/t.m/ttttttee.c 1ab/ httbb/staffor_aappaaat
2023.
MosaicML NLP Team. Introducing mpt-30b: Raising the bar for open-source foundation models, 2023a. W mww.icml.mcccmmmmmm.mmm......... com wwt.:ccw....cc..cc http://tgggpppppppoo 30b Accessed: 2023-06-22.
MosaicMI NLP Team. Introducing mpt-7b: A new standard for open-source, commercially usable Ilms, 2023b. URL www. mosa cm .com/b/ b og hapg//mpt77 Accessed: 2023-05-05.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tzz Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang. Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Vin- cent Zhao, Yanqi Zhou, Chung-Ching Chang, Igor Krivokon, Will Rusch, Mare Pickett, Pranesh Srinivasan, Laichee Man, Kathleen Hellstem,,,,,mm, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Soraker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fen- ton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera-Areas, Claire Cui, Marian Croak, Ed Chi, and Quoe Le. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.088 082399 2022.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodrigue7, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13977 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao. Vedanuj Goswami. Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux. Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023b.
Ben Wang and Aran Komatsuzaki. GPT-J-6B: A Billion Parameter Autoregressive Language Model. https://github... http::///giituuuu.oooooot.occe wttp://ccwc.ioooooooooooooo htt.::////mmetttttoooooeeeeeeeee htt.:/hwww...cc.ooooooeeeeeeeeea
jax May 2021.
Thomas Wolf, Lysandre Debut, Vietor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the- natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38 45. Association for Computational Lin- guistics, October 2020.
Yue Wu, So Yeon Min, Yonatan Bisk, Ruslan Salakhutdinov, Amos Azaria, Yuanzhi Li, Tom Mitchell, and Shrimai Prabhumoye. Plan, eliminate, and track-language models are good teachers for embodied agents. arXiv preprint 2rXiv:2305.02412 2023.
13

---

Published as a conference paper at ICLR 2024
Danfei Xu, Roberto Martin-Martin, De-An Huang, Yuke Zhu, Silvio Savarese, and Li F Fei-Fei. Regression planning networks. Advances in Neural Information Processing Systems, 32, 2019.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2023.
Eric Zelikman, Qian Huang, Gabriel Poesia, Noah D Goodman, and Nick Haber. Parsel: A unified natural language framework for algorithmic reasoning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained transformer language models. arXiv preprini arXiv:2205.0m 01068, 2022.
14

---

Published as a conference paper at ICLR 2024
A LANGUAGE MODELS
Table 3 lists the language models used in the experiments.
Table 3: List of language models used in the experiments. Model names are either from OpenAl
API or HuggingFace model hub.
Class
Model name
Model size
Remark
ada babbage curie text-davinci-003
350M
1.3B
OpenAl GPT
6.7B
175B
EleutherrAU/ppt-nno-11555
125M
ElcutherAA/ggptneo--1333
1.3B
GPT Neo
BleutherrAlggpttnnc0--7777
2.7B
Eleuther/A//ptt--Gbb
6B
EleutherrAV/ottnneox-000
20B
facebookkopt-125mm
125M
facebook/opt-1.3b
1.3B
facebook/opt-2.77
2.7B
faceboookoop-6.77
6.7B
OPT
facebook/opt-
13B
facebook/opt-300
30B
facebook/opt-66b
66B
facebook/oop--mmmmax--13333
1.3B
Instruction-tune
ficebookk/optiiml-aax-33000
30B
Instruction-tun
mosaicml/mpt-7bb
7B
mosaicml/mmpt330b
30B
MPT
mosaicml/mmpt-7-innstuee
7B
Instruction-tunae
mosaicmn/mmtt--00-iissruuuttt
30B
Instruction-
huggyllama//lamm=7
7B
huggyllama/llam--33
13B
LLaMA
huggyllama/llama-33
30B
huggyllama/llam-653
65B
meta-llama//llam-22775
7B
meta-llama/LLlma-21111 3--
13B
meta-llaamalllama-277000
70B
LLaMA
7B
Chat-tuned
13B
Chat-tuned
meta- (alamm//llaaa--70----oa
70B
Chat-tuned
B IMPLEMENTATION DETAILS OF THE BASELINE PLANNER
The implementation of SayCan's 's scoring LM (Ahn al., 2023 calculates each skill probability by summing token log probabilities for each skill, so the calculation time increases proportionally to the number of skills. Unlike SayCan's 's implementation, we employed Guidance (Microsofi. (023) that supports the selection of one skill among candidates from single generation pass by penalizing tokens that did not appear in any skill candidates, which considerably reduces the total time needed for the experiments. Guidance also avoids tokenization artifacts, by so-called "token healing." We compared two implementation methods by their success rates and processing time. When we use GPT-Neo 6B WAH-NL, the SayCan's implementation yielded subgoal success rate of 1.665% and took 1649.2 minutes. Our implementation using Guidance achieved a success rate of 24.80% and took 87.4 minutes which is about 19x speedup.
15

---

Published as a conference paper at ICLR 2024
We used HuggingFace's Transformers library (Wolf al., 2020) and OpenAl's GPT API (OpeAAl 2023) for the experiments, along with Guidance library (Microsoft 2023). Most of the models were single NVIDIA A100 80GB GPU, while used two A100 GPUs and three RTX 6000 GPUs for inference larger models, such as OPT 66B and LLaMA 70B, with model parallelism.
C EVALUATION WITH THE FULL SET OF ALFRED
The 30% subset we used in the main experiment with ALFRED might have some biases in the selection of subsets, although we randomly selected the samples. Thus, we report the results with the full set of ALFRED for validation. Note that GPT models were not included in this experiment as the lagacy GPT base models were deprecated.
Figure shows the results with the full set and subset, and we found that the general trends are similar between two sets.
GPT-Neo
OPT
MPT
LLaMA 1
LLaMA 2
102
10a
10
Model size (Millions)
Figure 7: Baseline results on ALFRED with the full set and subset. We report task success rates (%) for language models in different model classes and sizes (number of parameters). The bold lines for the full set. The thin lines represent the subset results. Fine-tuned models (by either instruction chat data) shown in dashed line with triangle maker.
D GPT-4 EXPERIMENT DETAILS
GPT-4 API has three agent roles of "system," "user," and "assistant." We modified the original prompt (Listing 1) to fit these roles follows. It worked like prompt that gives the context of problem and the role of the robot agent. The number of examples provided for in-context learning was the same. Examples, robot skills, and task query in square brackets were replaced with actual texts according to the test sample and environment.
You are robot operating in home. A human user ask you to do various tasks and you are supposed to tell the sequence of actions you would do to accomplish your task. < <User Role> Examples of human instructions and possible your (robot) answers: [The same in-context-learning examples are provided] Now please answer the sequence of actions for the input instruction. You should of actions of this list: [List of robot skills provided] List the actions with comma separator. Input user instruction:
<System Role>>
[Task query]
An assistant (robot) agent generates action sequences for the given user instruction. Due to the unavailability of Guidance's token selection mechanism Microsoft (2023) for GPT-4, provided an admissible action list the prompt to circumvent the issuc.
GPT-4 generated the answer for the given user instruction, and extracted each step from the answer text. There was no additional post- prot-procsssing tailored to GPT-4 (steps not in the provided list of the skills would not be executed correctly).
We tested GPT-4 and GPT-33.-TTurbe (see the table below for the results), GPT-3.5- -urbbo showed lower success rates for both ALFRED and WAH-NL compared to GPT-3. GPT-4 performed well in
16

---

Published as a conference paper at ICLR 2024
ALFRED, showing a success rate of 40.38% and improvement of 19% over GPT-3. However, in WAH-NL, GPT-4 showed a lower success rate of 34.17% compared to GPT-3.
Table 4: Success rates of GPT-4 and GPT-3.5 for ALFRED and WAH-NL datasets.
Model
WAH-NL ALFRED
GPT-3 (text-davinci-003)
21.36%
40.82%
GPT-4
40.38%
34.17%
GPT-o.5-Turbo
10.58%
28.82%
Despite the limitations of the current setup, GPT-4's performance notably surpassed that of GPT- 3 in ALFRED. This observation leads us to anticipate that the incorporation of Guidance's token selection mechanism Microsoft (2023) for GPT-4 could yield more impressive planning capabilities.
E MORE DETAILS ON WAH-NL
WAH-NI. comprises task types: Setup a dinner table, Put groceries, Prepare meal, Wash dishes, and Prepare snacks. These tasks are adapted from the orignal WAH dataset. In WAH, the Wash dishes task was deemed successful when dishes were placed in the dishwasher. We enhanced this by adding subgoal, "SWITCHONddsswwen Read a book task in WAH, which implies a human reading a book while enjoying snacks, has been transformed into Prepare snacks task. This transformation involved the removal of two inappropriate subgoals, "HOLD(human, book): 1" and "SIT(human, sofa): 1." Moreover, the number of objects 1 since LLM-based task planner does not consider object instance
1," ensuring the dishwasher is activated. The
for all subgoals was recognition. Table 5 shows samples for these 5 task types.
We collected NI instructions for both the train and fest sets, comprising 250 and 100 samples respec- tively, using the Prolific crowdsourcing platform. Crowdworkers were provided with 10 different goal conditions and asked to generate the corresponding user instructions. Initially, we collected 1, 1,000 instructions, 712 for train set and 288 for test set. We eliminated erroneous samples that mis- interpreted the goal conditions, resulting the final dataset of 416 instructions for the train set and 195 for the test set. Every sample in both sets has at least one NL instruction. Examples of the instructions collected are presented in Table
Table 5:
Task types and samples for each type in the WAH-NL datset.
Task Type
Goal Condition
Instruction
Setup dinner ta- ble
ON(plate, kitchen table): 1,
put the following on the kitchen table cutlery fork, wine glass, water glass and one plate
ON(water glass, kitchen table): 1,
ON(wine glass, kitchen table): 1,
ON(cutlery fork, kitchen table):
Put groceries
INSIDE(cupeake, fridge):
Please get the apple, the pancake, the pound cake and the cupcake and put them all the fridge.
INSIDE(pancake, fridge): 1,
INSIDE(pound cake, fridge):
INsIDE fridge):
Prepare a meal
ON(pancake, kitchen table): I,
Robot, please put the pancake and pudding on the kitchen table.
ON(pudding, kitchen table): 1
Wash dishes
INSIDE(plate, dishwasher): I,
Place one wine glass and one plate the dishwasher and turn on.
INSIDE glass, dishwasher): 1,
SWITCHON (ishhashhisswwsshhee
Prepare snacks
ON(juice, coffee table): 1,
one cupcake and one apple on the
ON(apple, coffee table): 1
coffee table
17

---

Published as a conference paper at ICLR 2024
F PROMPT
We list the prompts used in the ALFRED and WAH-NL experiments. We also share the prompt of the replanning experiment, where replanning examples are added.
Listing 1: Prompt for ALFRED experiments when in-context examples are provided (3 examples
per task type). Subsets of the examples below are used when we use fewer examples, like 6 or 12 examples, in the base experiments.
Robot: Hi there, I'm rohot operating a home.
Robot: You can ask me to various tasks and I'll tell you the sequence of actions would do to accomplish + your task.
Human: Put spoon in the sink.
Robot: 1. find ladle. pick up the ladle, 3. find sink, 4. put down the ladle, 5. done.
Human: Move vase from the entertainment center to the coffce table.
Robot: 1. find a vase, 2. pick up the vase, 3. find coffee table, 4. put down the vase, 5, done.
Human: Move the newspaper in the living room to the couch.
Robot: 1. find newspaper, 2. pick up the newspaper, 3. find a sofa, 4. put down the newspaper, 5, done.
Human: Pick up tennis racket and turn on lamp.
Robot: 1. find a tennis racket, pick up the tennis racket, 3. find a desk lamp, turn on the desk lamp, 5.
+ done.
Human: Pick up pillow and turn on lamp,
Robot: 1. find pillow, 2. pick up the pillow. 3. find a desk lamp, 4. turn the desk lamp, 5. done.
Human: Examine credit card using the light from floor lamp.
Robot: 1. find a credit card, 2. pick up the credit card, 3. find a floor lamp, 4. turn on the floor lamp, 5. done.
Human: Put a pan with spoon in it on the table.
Robot: 1. find spoon, 2. pick up the spoon, 3. find pot, 4. put down the spoon, 5. pick up the pot, 6. find
dining table, 7. put down the pot, 8. done.
Human: a box with remote in on the green couch.
Robot: 1. find a remote control, 2. pick up the remote control, 3. find a box, put down the remote control,
+ pick the box, 6. find sofa, 7. put down the box, 8. done.
Human: Place box with a cell phone
chair.
Robot: 1. find cell phone, 2. pick up the cell phone, 3. find box, put down the cell phone, 5. pick up the
+ box, 6. find an arm chair, put down the box, 8. done.
Human: Place a cold potato on the table.
Robot: 1. find a potato, 2. pick up the potato, 3. find a fridge, 4. open the fridge, 5. put down the potato, 6, close the fridge. 7. open the fridge, 8. find potato, pick up the potato, 10. close the fridge, 11. find counter top. 12. put down the potato, 13. done.
Human: Put the chilled bowl the microwave.
Robot: 1. find a cabinet, 2. open the cabinet, 3. find a howl, 4. pick up the bowl, 5. close the cabinet, 6. find a fridge, 7. open the fridge, 8. put down the bowl, 9. close the fridge, 10. open the fridge, 11. find bowl, 12. pick up the bowl, 13. close the fridge. 14. find microwave, 15. open the microwave, 16.
put down the bowl, 17. close the microwave, 18. done.
Human: Put the chilled sliced cabbage in the trash bin.
Robot: 1. find a knife, 2. pick up the knife, 3. find a lettuce, 4. slice the lettuce, 5. find a fridge, 6. open the fridge. 7. put down the knife, 8. close the fridge. 9. find a lettuce, 10. pick up the lettuce, 11. find fridge, 12. open the fridge, 13. put down the lettuce, close the fridge, 15. open the fridge, 16. find a lettuce, 17. pick up the lettuce, 18. close the fridge, 19. find garbage can, 20. put down the lettuce, 21. done.
Human: Place heated plate the round table.
Robot: 1. find a plate. 2. pick up the plate, 3. find microwave, 4. open the microwave, 5. put down the plate, close the microwave, turn on the microwave, turn off the microwave, open microwave, 10. find a plate, 11. pick up the plate, 12. close the microwave, 13. find a dining table, 14. put down the plate, 15. done.
Human: Place microwaved potato in the sink.
Robot: 1. find a potato, 2. pick up the potato, 3. find a microwave, open the microwave, 5. put down the potato, 6. close the microwave, 7. turn on the microwave, 8. turn off the microwave, 9, open the microwave, 10. find a potato, 1. pick the potato, 12. close the microwave, 13. find a sink, 14, put down the potato, 15. done. Human: Moving bowl to the shelf.
Robot: 1. find a cabinet, 2. open the cabinet, find a cup, 4. pick up the cup. 5. close cabinet, 6. find a microwave, 7. open the microwave, 8. put down the cup. 9. close the microwave, 10. turn on the microwave, 11. turn off the microwave, 12. open the microwave, 13. find a cup, 14. pick up the cup. 15. close the microwave, 16. find a shelf, 17. put down the cup, 18. done. Human: Place cleaned spatula dining table.
18

---

Published as a conference paper at ICLR 2024
Robot: 1. find a spatula, 2. pick up the spatula, 3. find sink, 4. put down the spatula, 5. find faucet, turn
on the faucet, 7. turn off the faucet, 8. find a spatula, 9. pick up the spatula, 10, find a dining table,
11. put down the spatula, 12. done.
Human: Place the rinsed off apple on the brown kitchen table.
Robot: 1. find apple, 2. pick up the apple, 3. find sink, 4. put down the apple, 5. find a faucet, 6. turn on
the faucet, 7. turn off the faucet, find an apple, 9. pick up the apple, 10, find a dining table, 11. put
down the apple, 12. done.
Human: Put a clean spatula on the counter.
Robot: 1. find a spatula, 2. pick up the spatula, 3. find a sink, 4. put down the spatula, 5, find faucet, 6. turn on the faucet, 7. turn off the faucet, 8. find spatula, 9. pick up the spatula. 10. find counter top.
11. put down the spatula, 12. done.
Listing 2: Prompt for WAH-NL experiments when 10 in-context examples are provided (2 examples per task type). Subsets of the examples below are used when we use fewer examples, like 5, in the base experiments.
Robot: Hi there, I'm robot operating in a home. Robot: You can ask me to do various tasks and I'll tell you the sequence of actions I would do to accomplish
your task.
Human: Please put cutlery fork, I water glass and plate in the dishwasher and turn the dishwasher on
Robot: 1. walk to dishwasher, 2. open dishwasher, 3. walk to cutlery fork, 4. grab cutlery fork, 5. walk to dishwasher, 6. put cutlery fork in dishwasher, 7. walk to water glass, 8. grab water glass, 9. walk to dishwasher, 10. put water glass in dishwasher, 11. walk to plate, 12. grab plate, 13. walk to dishwasher, 14. put plate dishwasher, 15. close dishwasher, 16. switch on dishwasher, 17. done. Human: store the pancake, the glass of wine, the cupeake and the glass of juice in the fridge
+
Robot: 1. walk to fridge, 2. open fridge, 3. walk to pancake, 4. grab pancake, 5. walk to fridge, 6. put pancake
+ in fridge, 7. walk to wine, 8. grab wine, 9. walk to fridge, 10. put wine in fridge. 11. walk to cupcake 12. grab cupcake, 13. walk to fridge. 14, put cupcake in fridge, 15, walk to juice, 16. grab juice, 17. walk to fridge, 18. put juice in fridge, 19, close fridge, 20, done.
Human: Take pudding, an apple, a cupcake, and the juice and put them on the coffee table.
Robot: 1. walk to pudding. 2. grab pudding. 3. walk to coffee table, 4. put pudding on coffee table, 5. walk to apple, 6. grab apple, 7. walk to coffee table, 8. put apple on coffee table, 9, walk to cupcake, 10. grab cupcake, 11. walk to coffee table, 12. put cupcake on coffee table, 13. walk to juice, 14. grab juice, 15. walk to coffee table, 16. put juice on coffee table, 17, done.
Human: Find coffee pot. cupcake, pudding and put on the kitchen table.
Robot: 1. walk to coffee pot, 2. grab coffee pot, 3. walk to kitchen table, 4. put coffee pot on kitchen table, S.
walk to cupcake, 6. grab cupcake, 7. walk to kitchen table, 8. put cupcake on kitchen table, 9, walk to pudding, 10. grab pudding, 11. walk kitchen table, 12. put pudding on kitchen table, 13. done. Human: I need a wine glass, water glass and a plate on the kitchen table
Robot: 1. walk to wine glass, 2. grab wine glass, 3. walk to kitchen table, 4. put wine glass on kitchen table, 5.
walk to water glass, 6. grab water glass, 7. walk to kitchen table, 8. put water glass on kitchen table, 9. walk plate, 10. grab plate, 11. walk kitchen table, 12. put plate kitchen table, 13. done.
Human: Add a fork and plate to the dishwasher and switch it on.
Robot: 1. walk to dishwasher, 2. open dishwasher, 3. walk to cutlery fork, 4. grab cutlery fork, 5. walk to
+ dishwasher, 6. put cutlery fork in dishwasher, 7. walk to plate, 8, grab plate, 9. walk to dishwasher, + 10. put plate in dishwasher, 11. close dishwasher, 12. switch on dishwasher, 13. done.
Human: Could you put apple, pancake, cupcake and pudding in the fridge?
Robot: 1. walk to fridge, 2. open fridge, 3. walk to apple, 4. grab apple, 5. walk to fridge, 6. put apple
+ fridge, 7. walk to pancake, 8. grab pancake, 9. walk to fridge, 10. put pancake in fridge. 11. walk to cupcake. 12. grab cupcake. 13. walk to fridge, 14. put cupcake in fridge. 15. walk to pudding, 16. grab pudding, 17. walk to fridge, 18. put pudding in fridge. 19. close fridge, 20. done.
Human: Serve one wine on the coffe table
Robot: 1. walk to wine, 2. grab wine, 3. walk to coffee table, 4. put wine on coffee table, 5. done.
Human: Collect apple. 1 coffee pot, and I juice and place them on the kitchen table
Robot: 1. walk to apple. 2. grab apple. 3. walk to kitchen table, 4. put apple on kitchen table. 5. walk to coffee
pot, 6. grab coffee pot, 7. walk to kitchen table, 8. put coffee pot on kitchen table, 9. walk to juice, + 10. grab juice, 11. walk to kitchen table, 12. put juice on kitchen table, 13. done.
Human: Put a plate, a fork and a water glass onto the kitchen table.
Robot: 1. walk to plate, 2. grab plate, 3. walk to kitchen table, 4. put plate on kitchen table, 5. walk to cutlery
fork, 6. grab cutlery fork. 7. walk to kitchen table, 8. put cutlery fork on kitchen table, 9. walk to water glass, 10. grab water glass, 11. walk to kitchen table, 12. put water glass on kitchen table, 13.
done.
19

---

Published as a conference paper at ICLR 2024
Listing Part of the prompt used in the replanning experiment. We used three additional replanning in-context examples, starting on the line following the prompt for ALFRED. The three in-context examples for replanning take one example from the prompt for ALFRED and add feedback messages for three different actions.
Robot: can replan the action to perform successfully when the action fails with the message this action failed 4 If generate the wrong plan, I'll make sure the command succeeds by replanning.
Human: Place the rinsed off apple on the brown kitchen table.
Robot: 1. find apple, pick up the apple, find sink, 4. put down the apple, find faucet, 6. turn
the faucet, 7. turn off the faucet, find an apple, pick up the apple, 10. a dining table, put
down the apple (this action failed: put down failed), 12. find an apple, 13. pick up the apple, 14. find a dining table. 15. put down the apple. 16. done.
Human: Place the rinsed off apple on the brown kitchen table.
Robot: 1. find apple, pick up the apple, find sink. 4. put down the apple, find faucet, 6. turn on the faucet, 7. turn off the faucet, 8, find an apple, find dining table, 10, put down the apple (this action failed: Robot is not holding any object), 11. find an apple, 12. pick the apple, 13. find a dining table, 14. put down the apple 15, done.
Human: Place the rinsed off apple on the brown kitchen table.
Robot: 1. find apple, pick up the apple (this action failed: Apple not visible because is in fridge), 3. + find a fridge, 4. open the fridge, 5. find an apple, 6. pick up the apple, 7. find a sink, 8. put down the apple, 9. find a faucet, 10. turn on the faucet, 11. turn off the faucet, 12. find an apple, 13. pick up the apple. 14. find dining table, 15. put down the apple, 16. done.
In replanning experiment, all actions had failure feedback messages. We describes three of them used the prompt. The first feedback message is "(this action failed: put down failed)" when the put down action was failed because receptacle was not found properly or because of physics-based simulation error. The second the feedback message "(this action failed: Robot is not holding any object)" from the put down action, which failed to put down because the agent was not holding any object. Third, there is the feedback message "(this action failed: [Object] is not visible because in [Container])", which is the feedback message from the pick up action when the object is not visible because the object is inside container. The other feedback messages can be found in the supplementary code.
G DETAILS ON FINE-TUNED PLANNERS
We fine-tuned LLaMA 1 models using the ALFRED Irain set (Shridhar et al. 2020), which contains 17, 17,468 pairs of user instructions and corresponding ground-truth plans. The dataset was formatted in accordance with the Alpaca instruction instruction-following dataset (Taori et al. 2023) for fine-tuning, but utilizing a specialized prompt template, we aligned the final input for the fine-tnning process with the main prompt used in the baseline planner. A sample of this dataset is presented in Listing
Listing 4: An excerpt from the fine-tuning dataset the ALFRED domain
"instruction": Robott
Hi there, I'm a robot operating in a home. \nRobot: You can ask me to do + various tasks and I'll tell you the sequence of actions would do to accomplish your task. \ nHuman: Pick up the alarm clock and turn on the lamp. "nRobot:"
"input" "output"" "i. find an alarm clock. 2. pick up the alarm clock, find desk lamp. 4. turn on desk lamp. S. done."
"instruction": Roboot
Hi there, I'm robot operating a home. \nRobot: You can ask me do various tasks and I'll tell you the sequence of actions would do to accomplish your task. \ nHumam: Carry an alarm clock while turning on a "Robott".....""
"input":
"output" "i. find alarm clock, 2. pick up the alarm clock, 3. find desk lamp, 4. turn the desk lamp, 5. done."
In the fine-tuning process, we utilized LoRA Hu et al. 2021), with the parammttrsssers set as follows: the rank of LoRA modules was set 16 (reduced to 8 for the 30B model due GPU
20

---

Published as a conference paper at ICLR 2024
memory constraints), the dropout rate 0.1, and the number of epochs 5. The benchmark conditions were consistent with those applied in the baseline planner tests, with the exception that omitted in-context examples in the prompt since the fine-tuned planners had already been trained with 17,468 plan examples. Notably, in our experiment, the fine-tuned planners exhibited diminished performance when provided with in-context examples (for instance, LLaMA 30B showed the success rate of 44.59% in ALFRED valid-unseen set when there are six examples in the prompt, which is lower than the result without examples), phenomenon demanding further exploration for comprehensive understanding.
Table shows the results from the experiments of benchmarking fine-tuned planners. As shown in the table, our experiments indicate that the procedural knowledge does not transfer well to different task domains. Specifically, planners fine-tuned on the ALFRED domain underperformed compared to baseline few-shot planners in the WAH-NI domain.
Table 6: Planning success rates of planners fine-tuned on ALFRED. 8 shows changes in success rates with and without fine tuning (pp refers percentage points).
ALFRED - ALFRED
ALFRED WAH-NL
valid-seen tasks
valid unlid-unseen tasks
Success( )
8(pp)
Success(%)
5(pp)
Success(%)
ö(pp)
LLaMA 7B
32.21
+26.44
37.39
+31.62
3,70
224.95
LLaMA 13B
42.79
+30.29
45.05
+32.55
2.28
-21.42
LLaMA 30B
57.69
+44.03
60.08
+46.42
10.38
-21.84
The statisties of successful plans according to task types in ALFRED are shown in Table 7 Fine- tuned planners produced more uniform distribution of successful plans across different task types.
Table 7: Number of successful plans by ALFRED task type. LLaMA 1 models were used and we reports results of the baseline and fine-tuning experiments for ALFRED valid-seen tasks.
Task Type
Baseline
Fine-tuned
(number of samples)
Sample Instruction
7B
13B
30B
7B
13B
30B
Pick & Place (49)
Move book from a desk
6
4
19
28
28
a sofa.
Stack & Place (33)
Put a bowl with the watch it the shelf.
0
10
Cool & Place (39)
Place a cold potato slice the sink.
6
6
15
22
26
Clean & Place (35)
Put clean sponge on a 2
10
8
6
10
17
metal rack.
Heat & Place (34)
Put heated mug the
0
1
7
13
10
24
small black table, left of the
tomato.
Examine in Light (15)
Turn on the tall lamp in the corner while carrying a
3
3
3
3
9
9
white cushion.
We also fine-tuned ILaMa models on the WAH-NL train set. We utilized a single instruction task, resulting in 250 pairs of user instructions and corresponding ground-truth plans from total 416 instructions. Consistent with our fine-tuning experiments on the ALFRED dataset, the same papee--aarmmetes and conditions were employed. The only difference was the inclusion of five in-context examples in the test prompt, addressing the limited number of WAH-NL train set and their impact performance. Table 8 summarizes the results. While the LLaMA 1 7B model demonstrated a performance decrease of 7.34 percentage points, the larger LLaMA 13B and 30B models exhibited performance increases of 5.77 and 1.56 percentage points, respectively. These
21

---

Published as a conference paper at ICLR 2024
results suggest that fine-tuning can enhance the performance of task planners, particularly in larger- scale language models, even with a constrained number of training examples.
Table 8: Subgoal success rates of planners fine-tuned on WAH-NI.. shows changes in success rates with and without fine tuning (pp refers percentage points).
WAH-NL - WAH-NL
Success (%)
8(pp)
LLaMA 1 7B
21.36
-7.34
LLaMA I 13B
29.47
+5.77
LLaMA 30B
33.78
+1.56
H ADDITIONAL PLANNING RESULTS
Success and failure results on the ALFRED dataset (Figure and 9 and on the WAH-NL dataset
(Figure 10 and 11 Figure 12 shows a replanning example described in Section
22

---

Published as a conference paper at ICLR 2024
Instruction: book fron desk sofa.
1:find book
Ziplck up the book
3:find sefa
4:put down the book
Instruction: Put. the plunger into the bathr oon cabAnet
Lunger
2:pt.ck plunger
cablnut
sopen
cablaet
down the
6:close the cabinet
drop rog coffee
1:find 2:plck the nug
3:find stak
4:put down the
S:Flnd Faucet
6:turn on the faucet
Zsturn off the
Fnd meg
9:pick up the rag
18:find coffee nachine
11:put down
Figure 8: Success cases on ALFRED dataset. Results of the baseline planner with GPT-3 175B.
23

---

Published as a conference paper at ICLR 2024
Instruction: Put bar of the back the toilet.
1:FFind soap bar
Zzplck up the seap
3:find toilet paper hanger
4:put down the soap bar
5:find totter hanger
soap
bar
Instruction: Put the mug with Fork on the round teble.
1:ffnnd
Zsptck the
3:flnd Fork
4:plck the for
S:find neg
6:put down the York
7:find a dinteg table
8:put down the nug
Instruction: trash cooked potato slice
1:find potato
2:ptck up the potato
3:find
4:slfce the potato
5:ftnd plate
down the
7=fLnd
gar
B:plck up
9:put down the
Figure 9; Failure cases on ALFRED dataset. Results of the baseline planner with GPT-3 175B.
24

---

Published as a conference paper at ICLR 2024
Instruction: place the cupcoke the coffee table
1:flnd
cupcake
2:plck up the cupcoka
3nf coffee
4:put down the cupcoke
Instruction: Place pudding, pancakio, wine, and juice in the fridge
fridge
2:opan fridge
juice
4spick juico
S:Find
Frilge
6:put down the Juice
Fnd pudii
Rasplr pudd
9:1ind Fridge
18: down the puddingg
11:f FLnd
pancake
12:plck up the pancaka
13:find fridge
p4: down the pancale
15:f1nd
16:ptck up wise
17: fridge
18:put
Figure 10: Success cases WAH-NL dataset. Results of the baseline planner with GPT-3 175B.
25

---

Published as a conference paper at ICLR 2024
Instruction: On the kitchen table apple, wine, cupcake and pudding.
1: kitchee
2eflund eucle!
grick up the apple
4: kitchen
table
down the apple
6iffed
Tipkick the
B:find table
kitchan
9:put the
18:F1nd cupeake
11:p1ck up the cupcoke
12d kchhn table
13:put down the cupcoke
14:1tad
puddingg
up
17:put the pudding
pudding
table
Instruction:
Please serve water glass on the kitchen table
2sptck the
3:fini kitthhn table
the
glass
Figure 11: Failure cases on WAH-NL dataset. Results of the baseline planner with GPT-3 175B.
26

---

Published as a conference paper at ICLR 2024
Instruction: Place
heated tosato la stok.
1:ftnd
2xptck the
3:find fridge
4:open the
Tdge
Tonato vistble because
S:flnd
tonato
6:ptek up the tonato
7=flnd nlcrooeve
Biopen the nicronave
9:put down the torato
turn
open uwave
f4:filn
15sptek the tomoto
1stelase the atcrowave
17t sim
pup down
coffee table, put
Find plate
Zipick the plate
3:find dintng table
4sput down the plate
3:find
Gsput tissue
7:find
Sapick up
plate
10:put down the tissug
Figure 12: Success cases of replanning on the ALFRED dataset using LLaMA 2 70B. (Top) Success case when replanning in pick up action. (Bottom) Success case when replanning in put down action.
27


Reviews:
[{"review_id": "9fCL4WI3PO", "reviewer": "Reviewer_JjuX", "summary": "The paper proposes LoTa-Bench, a benchmark system to automatically evaluate the performance of task planning for home-service embodied agents using large language models (LLMs) across a variety of datasets and simulators. The authors conduct extensive experiments with a baseline planner and several of its extensions within this benchmark framework.", "questions": "- Why was there no fine-tuning conducted with the WAH-NL dataset?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "- The paper is well-written and structured.\n- The authors test multiple pre-trained LLMs (GPT, GPT-Neo series, LLaMA, OPT,MPT) on their benchmark.\n- The authors present results on not just a baseline planner but also other extensions (such as feedback incorporation etc) on their benchmark.", "weaknesses": "The primary contribution of the paper seems limited to the integration of several existing benchmarks into a single platform. There are existing works, as cited in the Related Work section, that already employ benchmarks like ALFRED and VirtualHome to test various planning and execution techniques. The paper does not make it clear how LoTa-Bench provides additional value over these existing benchmarks. It appears that the significant novel contribution is the extension of the WAH dataset and its adaptation for autonomous agents.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eBMpiGjZ9k", "reviewer": "Reviewer_JeGM", "summary": "The paper presents LoTa-Bench, a benchmark for evaluating language task planners in home-service robots, using two datasets/simulators: ALFRED with AI2-THOR and Watch-And-Help with VirtualHome. It contributes a benchmarking suite, extensive testing of baseline planners with different models and prompts, exploration of planner enhancements, and the release of the benchmark code and an extended dataset to the public. The findings highlight the importance of model selection, in-context example strategies, replanning mechanisms, and model fine-tuning in improving planner performance.", "questions": "1. How does the decoupling of high-level plans from low-level actions in your benchmark reflect on the practical deployment of task planners? Could this lead to overlooking important dynamics that only emerge when both levels of planning are considered together?\n\n2. Given that AI2THOR and VirtualHome lack diversity to reflect real-world environments, could you elaborate on your plans to enhance these simulators or to incorporate additional platforms that might offer more realistic and diverse scenarios?\n\n3. How do you plan to address the domain gap between simulation and real-world application, and do you have strategies for dealing with unrealistic assumptions like the one mentioned in ALFRED about objects being cleaned once put into water?\n\n4. Can you discuss the potential development directions for evaluation frameworks that can handle the complexity of LLM-based task planning beyond simple tabletop manipulation tasks, as mentioned in the current limitation of the field?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. **Innovative Benchmark Development**: The paper introduces LoTa-Bench, an innovative benchmark suite for language-oriented task planners. This benchmark is critical for the field as it allows for the automatic and reproducible evaluation of task planning, which is a significant step forward in embodied AI research.\n\n2. **Comprehensive Experimental Analysis**: The authors provide extensive experimental results, which is a comprehensive demonstration of the benchmark's capabilities. They conduct experiments across various pre-trained models, demonstrating the benchmark's utility in evaluating the impact of model selection, in-context examples, replanning, and fine-tuning.\n\n3. **Resource Contribution and Future Research Facilitation**: The paper contributes valuable resources to the research community, including the benchmark code and an extended dataset. Moreover, the findings and the benchmark itself are positioned to facilitate future research, potentially accelerating advancements in the field of language-oriented task planning for embodied agents.", "weaknesses": "1. **Decoupling of Planning Levels**: The research decouples high-level plans from low-level actions, focusing only on the former. This separation might not accurately reflect the complexities of end-to-end task planning where high-level decisions and low-level actions are interconnected.\n\n2. **Lack of Visual Understanding**: The benchmark does not incorporate visual understanding, which is critical for low-level actions, especially in embodied AI where egocentric views play a significant role in interpreting and interacting with the environment.\n\n3. **Domain Gap in Simulation**: There is a domain gap between the simulation environments used and the real world, leading to unrealistic assumptions. For example, the ALFRED simulator assumes an object is clean once it is placed in water, which may not always be true in real-world scenarios.\n\n4. Another limitation of the paper is the absence of an in-depth investigation into several influential factors that affect the performance of task planners. These factors include the type and size of pre-trained language models, the number and strategy for selecting in-context examples, the ability of planners to replan based on natural language feedback, and the impact of fine-tuning the models​.  Addressing these factors is crucial for a thorough understanding of how to optimize task planners for real-world applications.\n\nMinor one: \nMissing references:\n- On Grounded Planning for Embodied Tasks with Language Models \n- Plan, Eliminate, and Track - Language Models are Good Teachers for Embodied Agents", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KhfysXH4St", "reviewer": "Reviewer_DjjY", "summary": "The paper proposed the development of a benchmark to measure the effectiveness of LLM-based task planners in language-related settings (particularly where the system receives natural language instructions). The proposed benchmark consists of pairs of datasets and simulators, where the dataset includes example instructions and environment information, and the simulator allows the plan generated by the planner to be evaluated. They propose a baseline planner that selects the highest probability skill from a set of possible skills, given the relevant information. The base planner is evaluated on the two proposed dataset/simulator pairs, and they also perform additional experiments to evaluate factors like replanning, finetuning, and example selection.", "questions": "Please answer the questions raised under the weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The clear strength of the paper is the fact that this is a problem that is getting a lot of attention. As discussed by the authors, there aren’t quite too many benchmarks of this specific quality, particularly the presence of simulators ( though I am not completely sure if this aspect is being fully utilized here). In addition to the basic evaluation, I found the additional experiments quite insightful and useful, especially the ones related to the example selection.", "weaknesses": "NL to API Component: I am a little bit confused about the need for setting up an NL to API component for the current system. Why can’t the skills provided to the LLM consist of directly executable API calls or actions? Is the concern here that the API calls would be too abstract and arbitrary to be connected to the actual instruction? Can’t you get around it by using meaningful action names (as is the case in classical symbolic planning settings [1]) or provide a description of each action in natural language as part of the environment context?\n\nFailure Reasons: I wish there were a more detailed analysis of how the plans failed and the main results were not reported in terms of mere success rates. Some potential analyses the authors should consider here include things like checking if there were any phenomena akin to invalid preconditions, i.e., a selected low-level controller needed certain state constraints to be met before it was applicable. Similarly, how close were the failed plans to valid plans? Could you have turned them into valid plans by appending a few more actions? Were they moving the environment state closer to the goal state? Was the goal state achieved as an intermediate result and then changed by some future actions?\n\nLack of tests on GPT-4: One shortcoming of the current set of experiments is that it is missing any results that make use of GPT-4 as a planner, which is widely recognized as being one of the most powerful LLMs that are currently available. The reason cited is the unavailability of mechanisms to identify the probability associated with each possible next token. However, as I understand it, the basic planner uses a greedy approach to select the next most likely token. Each selected token is appended to the prompt, and the next token is selected. Isn’t this similar to the basic autoregressive mechanism used by all LLMs, including GPT-4, especially when using low temperatures? You could have forced GPT-4 to only select tokens from the pool of potential skills by careful prompt engineering and some filtering. Even if the mechanism weren’t exactly the same, this would have provided a useful point of comparison on how the state-of-the-art LLM models fair in these benchmark tasks.\n\n[1] Haslum, Patrik, et al. An introduction to the planning domain definition language. Vol. 13. San Rafael, California: Morgan & Claypool, 2019.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jSmYGJ1sH8", "reviewer": "Reviewer_mB86", "summary": "The paper proposes LoTa-Bench, an evaluation framework for the planning capability of LLMs for embodied agents.\nThe proposed benchmark utilizes two existing benchmarks, ALFRED and WAH, that require agents to plan a sequence of actions to satisfy desired goal conditions given natural language instructions.\nIt follows the evaluation protocol of ALFRED and WAH.\nThe paper compares various large language models for planning. It conducts some experiments to show possible extensions: 1) in-context example selection, 2) replanning based on feedback, and 3) fine-tuning LLMs on downstream tasks.", "questions": "See weaknesses above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is generally written well and easy to follow.\n- The paper provides an important framework for the evaluation of the planning ability of LLMs which have been widely used for the planning of embodied agents.\n- Focusing on planning by language by some assumptions (e.g., oracle information of objects for visual navigation) sounds reasonable.\n- The paper provides a useful extensive analysis of the planning capabilities of existing LLMs for the proposed benchmark.", "weaknesses": "- I appreciate the proposed benchmark for LLMs' planning but some of the proposed possible extensions on this benchmark are somewhat expected. For example, fine-tuning a pretrained model (here, LLMs) to downstream tasks is a widely used approach. And for replanning by feedback, while a larger LLM is improved with the replanning, a smaller one is not, implying that we need some large LLMs, but it is also quite expected that a larger model may yield better performance and a smaller one.\n- The feedback for replanning is based on metadata (i.e., oracle information about what is \"wrong\") of the simulated environments but this may not be always available beyond the benchmark for other downstream tasks (e.g., some tasks might not be able to provide why agents fail at actions). Can this direction be also useful for those cases without the oracle information?\n- The authors conduct experiments based on a subset (30%) of the original datasets but do not provide some significance tests. It might be better to include them in the results (e.g., Figure 2) with different subsets.\n\n\\* Minor\n - The used font looks a bit different from the one used in the original ICLR format. It might be better to revert to the original font.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)