Describe, Explain, Plan and Select: Interactive Planning with LLMs Enables Open-World Multi-Task Agents
Authors: Zihao Wang
Shaofei Cai
Guanzhou Chen
Anji Liu
Xiaojian Ma
Yitao Liang
Venue: NeurIPS 2023 poster (2023)
Keywords: open-ended learning
multi task
large language models
zero-shot planning
TL;DR: We propose a novel interactive planning approach using LLMs that brings a planning-based agent to the apex of Minecraft (an open-world environment) for the first time.

Abstract: In this paper, we study the problem of planning in Minecraft, a popular, democratized yet challenging open-ended environment for developing multi-task embodied agents. We've found two primary challenges of empowering such agents with planning: 1) planning in an open-ended world like Minecraft requires precise and multi-step reasoning due to the long-term nature of the tasks, and 2) as vanilla planners do not consider the achievability of the current agent when ordering parallel sub-goals within a complicated plan, the resulting plan could be inefficient. To this end, we propose ``$\underline{D}$escribe, $\underline{E}$xplain, $\underline{P}$lan and $\underline{S}$elect'' ($\textbf{DEPS}$), an interactive planning approach based on Large Language Models (LLMs). Our approach helps with better error correction from the feedback during the long-haul planning, while also bringing the sense of proximity via goal $\textbf{Selector}$, a learnable module that ranks parallel sub-goals based on the estimated steps of completion and improves the original plan accordingly. Our experiments mark the milestone of the first zero-shot multi-task agent that can robustly accomplish 70+ Minecraft tasks and nearly double the overall performances. Further testing reveals our method's general effectiveness in popularly adopted non-open-ended domains as well (i.e., ALFWorld and tabletop manipulation). The ablation and exploratory studies detail how our design beats the counterparts and provide a promising update on the $\texttt{ObtainDiamond}$ grand challenge with our approach.

Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agents
Zihao Wang Shaofei Cail?2, Guanzhou Chen', Anji Liu', Xiaojian Ma", Yitao Liang 1,5+
Team CraftJarvis
Institute for Artificial Intelligence, Peking University
School of Intelligence Science and Technology. Peking University
School of Computer Science, Beijing University of Posts and Telecommunications CComuteer Science Department, University of California, Los Angeles Beijing Institute for General Artificial Intelligence (BIGAI)
(zhwang, caishaofei @ssuu. pku. edu, en, waymenttuuuppp odu on liuanjiecs. ucla.edu, xiao siaojian. ma mww.cla......mmmmmm. edu, yit wtt.aaa..c..ytttalp..........
Abstract
We investigate the challenge of task planning for multi-task embodied agents in open-world environments." Two main difficulties are identified: 1) executing plans in an open-world environment (e.g.. Minecraft) necessitates accurate and multi-step reasoning due to the long-term nature of tasks, and 2) as vanilla planners do not consider how easy the current agent can achieve given sub-task when ordering parallel sub-goals within a complicated plan, the resulting plan could be inefficient even infeasible. To this end, we propose "Describe, Explain, Plan and Select" (DEPS), an interactive planning approach based on Large Language Models (LLMs). DEPS facilitates better eπоо correction initial LLM-generated plan by integrating description of the plan execution process and providing self- explanation of feedback when encountering failures during the extended planning phases. Furthermore, it includes a goal selector, which is a trainable module that ranks parallel candidate sub-goals based the estimated steps of completion, consequently refining the initial plan. Our experiments mark the milestone of the first zero-shot multi-task agent that robustly accomplish 70+ Minecraft tasks and nearly double the overall performances. Further testing reveals our method's s general effectiveness in popularly adopted non-open-ended domains as well (i.e., ALFWorld and tabletop manipulation). The ablation and exploratory studies detail how our design beats the counterparts and provide a promising update on the iitainDDamont grand challenge with our approach. The code is released at https:/ / http//gttttt/aa www.///ataaaavis//////aoaaaaasaa wtt.:aaaiviie/caaaaaaee
Introduction
Developing multi-task agents that can accomplish a vast and diverse suite of tasks in complex domains has been viewed one of the key milestones towards generally capable artificial intelligence [36, 1, 5, 10, 25). To enable such capabilities, earlier works have suggested employing a hierarchical goal exccution architecture [2, 4], where planner generates action plans that would then be executed by low-level goal-conditioned controllers. This architecture has been delivering promising progress in
"Corresponding Author.
"We borrow term "open world" from game community. highlights that agent can navigate inside a diverse environment and accomplish open-ended tasks freely.
37th Conference on Ncural Information Processing Systems (NcurIPS 2023).

---

Planning plumnet open worlds challengee
Challenge #2. State-dapendent Feosibility
Tablesp Endrormant
Challenge Mf; Complex Sub-task Dependency
Inttocaiia Maripoiation
Mie diarard Misscraft
Figure 1: Planning success rates plummet in open worlds due to new challenges.
many robotics domains, including table-top and mobile manipulation [46, 4], 2D shape drawing [20] and table rearrangement [17]. However, whether such success can be transferred to more open-ender world with unlimited exploration areas and internet-scale knowledge remains open [14, 10, 13, 12, 19].
To understand the gap, we run Inner Monologue [17], a general and competitive hierarchical goal execution model on a typical open- world domain Minecraft [18, 14, 10] and two classical robotic vironments ALFWorld [41] and Tabletop environments [40, 4]. The algorithm uses Large Language knowledge for all three environments. In
Model (LLM) based planner that contains all environments, we use either an Oracle goal-conditioned controller or a learned one. Results are shown in the bar plot in Figure 1. First, even when the Oracle controller is used, the success executing Minecraft tasks is much less than that of the other environments. Next, the task failure rate becomes even higher in Minecraft when the learned controller is substituted. Both failures originate from unique challenges brought by open-world environments, which we identify in the following.
of
First, compared canonical environments (e.g., Atari [29] and robotic control suite [40]), open worlds have highly abundant object types with complex dependency and relation. As a result, ground- truth plans typically involve long sequence of sub-goals with strict dependencies. As Figure 1 challenge #1 suggests, it requires at least 13 sub-goals executed in proper order to obtain a diamond in Minecraft, while in Tabletop a task typically no more than a few consecutive sub-goals.
Another challenge brought by the complicated tasks an open-ended world the feasibility of the produced plans. Consider the example shown in Figure (challenge #2). To craft bed in Minecraft, the fastest way is by either slaughtering a sheep to obtain wool, which can be used to craft beds, or collecting beds from village. However, since no sheep or village is reachable by the agent within 3 minutes of gameplay, to craft a bed efficiently, the agent should choose to slaughter a spider and use materials (e.g., string) it drops to craft wool, and then bed. That is, when dealing with task that can be completed by executing multiple possible sequences of sub-goals, the planner should be able to select the best route based on the current state of the agent. However, the complex and diverse state distribution of open-world environments makes state awareness hard to achieve.
To tackle these problems, we propose "Describe, Explain, Plan and Select" (DEPS), interactive planning approach based on Large Language Models (LLMs) to alleviate the aforementioned issues. The key to tackling the first challenge is to effectively adjust the generated plan upon failure. Specifically, whenever the controller fails to complete a sub-goal, descriptor will summarize the current situation text and send it back to the LLM-based planner. We then prompt the LLM as an explainer to locate the errors in the previous plan. Finally, a planner will refine the plan using information from the descriptor and explainer. To improve the feasibility of generated plans conditioned on the current state, which is the second identified challenge, we use a learned goal- selector to choose the most accessible sub-task based the proximity each candidate sub-goal.
Our experiments are conducted on 71 tasks in open-ended Minecraft without any demonstration. Given the goal-conditioned controller for atom sub-tasks (i.e., mine log and mine stone), our zero-

---

Instruction
Tusk instruction: Obtain diamond in Minecrafi survival mode step stspbpysteep?
Initial
Updated Plan Ps:
(Re-)Planner
Explainer
explatn
plan Pt
description de
Candidate goals Selected Goal ga: 7er sgont focms ee Alrid firett lick ouly has blick roud
Selected Goal
choices
Selector HPM
Descriptor VLN
Description d, succoad gool fuil on smelting d from My inventory
goat
feedback
Env
Description succeed goal 1-5, goal mining with Now my inventory has planks,
Explan Because smelting @ needs use and which I do not have. need craft first.
Controller
Goal-conditioned Policy
Task Finished
obs
action
Explamation: Because 7. Sn need to crdy 7 first
needs
Finishal By:
obs
Environment
Mine cak wood
Mine birch wood
Craft acacia planks
Craft crafting sable /Cfit (af sick
Mine iron ore
Minz cral
Cmfi furmace @
Mise daammnd
Mise acocia wood
Craft oak planks
Cruft binch pintk 7 Cruft woed pickase Cruft sose pickase
Mine comblosione iron ingor Aratl iron pickaxe
Figure Overview our proposed interactive planner architecture.
shot LLM-based planner can finish all tasks within a limited horizon (3000-12000 steps for different tasks). We find DEPS outperforms all language planner baselines by nearly doubling the overall success rate, with the same initial state and goal-conditioned controller. Our ablation and exploratory studies then explain how our approach beats the counterparts and becomes the first planning based agent that accomplishes the challenging Obt aataannii amond task. DEPS does not require any planning training for the environment. Additionally. DEPS achieves between on-par and more than 50% relative improvement over existing or concurrent L.-de planning methods on non-open-ended robotics domains such as ALFWorld [41] and Tabletop environments [40].
2 Background
We aim to develop an agent capable of solving long-horizon goal-reaching tasks using image observations and language goals. To accomplish this, we propose a combined approach involving goal-conditioned policies (termed controllers) and planner. The goal-conditioned policies are trained to complete sub-goals, while the planner decomposes long-horizon tasks into series of K short-horizon sub-goals, .1.....9K3 to be executed by the controller. At each time step I, the goal-conditioned policy T(a ( 8t, gk) generates an action a, based on the current state 8 and the specified sub-goal gk.
Planning with Large I Language Models Previous works have shown that LLMs such as Instruct- GPT [32] and Codex [8] can he used as zero-shot planners to generate sub-goal sequences for various tasks in embodied environments [16, 42]. Formally, given the task description T as prompt LLM acts as planner to decode T' into K sub-goals, ......, 9.. which are then executed one by one by the low-level controller π <a, sr, gk) accomplish the task.
However, the above pipeline suffers from both challenges identified in Section Regarding the first challenge, the probability of generating a flawless plan directly from the task description decreases significantly as the required number of sub-goal, increases. Moreover, even when the LLM generates correct plan, it is very likely that the plan is highly inefficient given the agent's current state (challenge #2). Prior works mostly focus on solving the first challenge by providing environmental feedback to the LIMM through affordance functions [4], success detector [20] scene descriptor [17]. However, although these approaches work well on many non-open-ended domains, they still suffer from high failure rates in open- world environments.
SSmiiarr to [5, 16), "zero-shot" here means gradient updates performed. However provide some related demonstrations as prompts during inference time.

---

3 Towards Reliable Planning in Embodied Open-World World Environments
In this section, we first give an overview of our proposed interactive planning framework "Descibe, Explain, Plan, and Select" (DEPS) for solving complex and long-horizon tasks in open-world environments (Sec. 3.1). Next, in Section 3.2, we elaborate how DEPS iteratively refines its plan to combat the first identified challenge. Section 3.3 introduces the selector module that is used to identify efficient plans response to the second identified challenge.
3.1 DEPS Overview
As demonstrated in Figure 2, our agent (DEPS) consists of an event-triggered Descriptor, Large Language Model (LLM) as Explainer and Planner, goal Selector based on horizon prediction and goal-conditioned controller. In the following, we use Minecraft elaborate our agent. Note that DEPS can be directly applied to other moon-oopen-endet tasks.
running example to better
We take a large language model (LLM) as a zero-shot planner of the agent to complete tasks. Given goal command (e.g., Obt aitii amond) as task T, the LLM-baseu planner decomposes this high-level task into a sequence of sub-goals instructions in natural language, such as mine oak wood (in Minecraft), find two cups (in 9kh 9.. J. as the initial plan Po- The goals are
ALFWorld), put block A on top of block B (in Tabletop Manipulation).
As described in Section 2, controller is then invoked to execute the provided sub-goals sequentially through a goal-conditioned policy π (a 8, 9, However, the initial plan provided by the planner often contains errors, which results in execution failures the controller. For example, the goal can not be finished only with a wooden pickaxe 7 shown in Figure 2.
When failure pops up, the descriptor will summarize the current state 8; and execution outcome of the most recent goal into text di and send it to the LLM. The LLM will first try to locate the errors in the previous plan Pt-1 by self- eeff-explanation, e.g., the goal need to be executed with a stone pickaxe 70. Then it will re-plan the current task and generate a revised plan P. according to the explanation. In this process, the LLM is also treated as an explainer in addition to the planner role. The Descriptor, Explainer, and Planner will be detailed in Section 3.2.
Description de = =D-ES
Explanation e, fEx(dd),
Prompt Pt CON de)
(1)
Plan P. = fLM(PE),
Goal 9: ~ fs((),)(-1),
Action a2 T(a=| 9t)
As shown in Equation (1), DEPS will iteratively update the plan P. until the task is finished, where fuesc is the descriptor model, fLM denotes the language model explainer and planner, fs is the selector model, π is goal-conditioned policies from the controller.
To filter out inefficient plans, the selector is trained to predict the number of time steps remaining to achieve every goal gk in set of parallel goals given current state st. When the generated plan contains alternative routes, the selector uses this information to choose suitable goal the current goal gt. For example, the horizon predicted by the selector of goal acacia tree is less than goal oak tree in Savanna biome, which leads to chop acacia tree as current goal gt-
3.2 Describe, Explain and Plan with LLM Generates Executable Plans
Current LLM-based planners usually query the LLM once at the beginning of every episode and use the output plan throughout the episode [16, 42]. However, as demonstrated by Figure 1, such one-shot planning methods often fail on long-horizar tasks that require many sub-goals. This caused by two major issues. First, since the correct plan for long- hong-horizon tasks needs to respect various complex preconditions, it is extremely hard for the LLM to generate flawless plan directly from the task instructions, resulting in failure when simply following the initial plan. Additionally, due to the unpredictable transition dynamics, some incidents may happen during the execution and make the initial plan non-executable. To remedy these problems, existing methods introduce feedback (c.g..

---

Prompt Planner prompt template, Python-like code
def craft__wodde_aae
step 1: mine 3 logs
"ob"" ("log":3), toel None)
step 2: craft 12 planka from logs
craft (obj = ("planka": materials (*log":3), tool None)
step 3: craft aticka from 2 planks
craft (ob1 ("stick":4), materials ("planka":2), tool None)
stop 4: graft crafting_table from planks
cref: [obj -
materiels ["planks": :4), toel Kone)
step 5; craft woaden_axe axe from planks and sticks on crafting table eraft (abj = [ ("wadden_xxx" :1,, ("planks": 3, *stick"; 2), svul "crafting_table"" return "xcodea_axe"
from success detector or scene descriptor) to reflect on the results of previous executions [17, 20, 4]. However, merely informing the LLM whether sub-goal is completed is often insufficient to correct the planning error.
To remedy this, we propose "describe, explain and plan", a new interactive planning method to generate more executable and explainable plans. We start with rewriting the prompt into an interactive dialogue format as in ChatGPT [32] so that subsequent feedhack can be passed to the L.LM effectively. The produced plan is also augmented with the preconditions and effects of each goal. The structured prompt improves the readability and interpretability of the plan and facilitates error-locating when the execution fails later, as demonstrated in Prompt
The descriptor will then collect the feedback generated by the agent during the execution of the task. The feedback can be practically obtained either by a person (human feedback [4]), by pre-trained lision-anguaage model CLIP [35]. While the previous type of feedback needs intensive human involvement, the latter from the pre-trained model needs to be fine-tuned for the specific domain, which decreases the automation and generalization of the agent. On the contrary, Minecraft returns the 'info' and other high-level observations (such as biome, GPS, and compass), translate the unstructured information into structured language. Therefore we take the symbolic information available in the game and translate it into feedback description de this work. To avoid carrying unrelated information in the prompt, we further distill plan-related messages (c.g., inventory information, biome) final event-level description d, as demonstrated Figure 2.
can easily
Notably, we also treat the LLM as an explainer to explain why the previous plans Pa-1 failed. Specifically, by analyzing the current state from description de and precondition of current goal gu- the explainer can identify the reason why the current goal cannot be executed successfully. As shown in Figure the reason may be the current goal requires the use of an iron pickaxe, but the tool is not prepared in advance, or the current goal requires the use of 3 planks, but the currently available planks are not enough. To implement this, provide few-shot demonstrations the LLM in tfain-of-thougee prompting [45], as shown in Prompt 1. Finally, the LLM goes back to its role as a planner and re-plans the task with the explicit explanation of existing bugs in the previous plan Pt-1. ultimately generating an updated plan P. according to the explanation.
3.3 Horizon-Predictive Selector Yields Efficient Plans
Due to the abundance of objects and the compositional nature of their functionalities, there often exist multiple feasible plans to complete a task, i.e., there usually multiple paths for the completion of particular goal. However, despite the feasibility of all such plans, most of them are highly inefficient to execute in the current episode. For example, shown in Figure 2, obtaining a wood can be done by chopping oak trees plains biome. So the planner needs to choose oak trees since it is more efficient, as the agent does not need to travel to another biome.
birch trees
acacia trees But only oak trees are available in the
On the other hand, there is no strict sequential requirement for some goals in the plan Pt, i.e.. Pe. Ii ~ Pt enjoy the same precondition, which means 9i and 9j can be executed in any order. As shown in Figure 1, the choice of different paths (sequences) may affect the execution efficiency of the plan P. as one goal might be closer to the agent. Always choosing the closer goal to execute first could yield more efficient plans and improve the final success rate under a limited episode length. Moreover, the dynamic nature of open-world environments further amplifies the impact of efficient
5

---

Goal: Meal"3
Goal: Log"2
Goal: Coal AND Iron, O
Goal: Survive in Night.
Cundidate Skill: Kill Sheep
Candidate Skill: Chop Oak
Candidate Skill: Mine Coul AND Tron
Candidate Skill: Sleep in bed OR Dig down.
OR Pig
Birch OR Acacia
Selection: Kill Sheep
Selection: Chop Acacia Tree
Selection: Mine Iron, Ore
Selection: Sleep in bed Explanation: Village has bods.
Explanation: Meet sheep first.
Explanation: Savanna biome
Explanation: Meet iron ore first.
only has Acacia tree.
Figure 3: Selection Demonstration from "Selector". Given parallel sub-goals, i.e. candidate skills, Selector will determine the sequence which carry out these sub-goals based on their current proximity to the agent and modify the original plan produced by the LM planner.
plans on the success rate. For example, in Minecraft, if the agent chooses to execute a further goal like collect wood first, the much closer target sheep may disappear and be hard to find again.
In order to improve the efficiency of our plans, we propose to use a selector that selects the most efficient path with the highest execution success rate as the final plan. Specifically, we design state-aware selector to choose the nearest goal under state 8, as the current goal Bt from the candidate goal sets in plan Pt. It predicts the goal distribution p(gel $t, Pt) under the current state 8; and plan Pt. where gt E G., G2 describes all current executable goals in Pt. A straight way to implement the selector is to leverage the semantic similarity between the current state and the goal text using a lision-language model (VLM) such as CLIP [35]. Nevertheless, this may not exactly reflect the difficulty of completing the goal since VLM lacks practical experience. For example, an "oak tree" in front of the agent could lead to high semantic similarity for the "chopping tree" goal, but it may be inefficient achieve this goal if canyon is in the middle between the agent and the oak tree.
To mitigate this, we implement horizon-predicttoo accurately rank the goals based on their efficiency and feasibility. Here, we define the horizon of goal he(g) := Tg f as the remaining time steps to complete the given goal, where T3 is the time of completing goal g. This metric accurately reflects how quickly the current state. To estimate the horizon, we learn a neural network js to fit the offline trajectories by minimizing the entropy loss log (g))(g) $t,, g). where h2 is the ground-truth horizon in trajectories of completing goal g. Therefore, the goal distribution can be formulated as follows:
selector that embeds practical task experience to
achieve the given goal from
f(g | St, Pt) = exp(-u(gr 8t))
(2)
Egec.
$t))
We set goal-sensitive Impala CNN [6] as the hackhone of the selector. In practice, the horizon predic- tive selector can be jointly trained with the controller policies and share the backhone parameters [6].
4 Experiments
This section analyzes and evaluates proposed "describe, explain, plan, and select" (DEPS) method. To minimize performance variation caused by the low-level controller, we standardize all experiments with one controller learned by behavior cloning. We refer to the details of this controller in Appendix C. In Section 4.1, we introduce our testing environments and our evaluation task set, consisting of the hardest 71 tasks from MCU SkillForgeChain [22]. In Section 4.2, we report our performance in the context of existing LLM-based planners. Ablation studies are conducted in Section 4.3. Finally, we pay close attention to the hardest task, ObtainD amond, which long-hailed as a major challenge in the community. The experiments on ALFWorld and Tabletop Manipulation environments are shown in Appendix A.
4.1 Experimental Setup
Environment and Task Setting We first evaluate our proposed method in Minecraft, a popular open-world environment with both challenges discussed Section For better reflecting the performance of DEPS, we choose three Minecraft environments with different versions for better

---

evaluation, including Minedojo [10] with Minecraft 1. 1.11.2, MineRL [3] with Minecraft 1.16.5, and MC-TextWorld [22] with Minecraft 1.19.2. Rules and items have something different in the above three Minecraft environments, which can better evaluate the dynamic and interactive planning abilities of DEPS.
Table I: Attributes of 8 meta tasks covering Task101: We evaluate the algorithm on Minecraft 10s We
group the consisted 71 task into 8 different meta groups, with each focusing on testing different aspect of our proposed method.
Meta
Name
Number
Example Task
Max. Steps
Initial Inventory
Given Tool
MTI
Basic
14
Make a wooden door.
3000
Empty
Axe
MT2
Tool (Simple)
12
Make stone pickaxe.
3000
Empty
Axe
MT3
Hunt and Food
7
Cook the beef.
6000
Empty
Axe
MT4
Dig-Down
6
Mine coal.
3000
Empty
Axe
MT5
Equipment
9
Equip the leather helmet.
6000
Emply
Axe
MT6
Tool (Complex)
7
Make shears and bucket.
6000
Empty
Axe
MT7
IronStage
13
Ohtain an iron sword.
6000
Empty
Axe
MT8
Challenge
I
Obtain diamond!
12000
Empty
Axe
We choose 71 tasks from the Minecraft Universe Benchmark SkillForgeChain [22] for evaluation. These tasks are related to items that can be obtained in the Minecraft overworld. To better present the results, we divide the 71 Minecraft tasks into 8 meta groups according to the ingredients and function of the tasks, i.e., MT1-MT8. The instruction for every task is written in natural language, e.g., ma me wooden door MT1 (Basic group) and obtain a diamond MT8 (Challenge group), as illustrated in Table 1. Considering how long it typically takes human players to complete each task as a ballpark [14], we set different maximum episode steps for different meta tasks from 3000 (for easiest Basic tasks) to 12000 (for the hardest Challenge tasks). The names, number of required skills, and functions of all tasks are listed in Appendix B. We give an empty inventory for every task in Survival mode and require the agent to obtain every item from the environment by itself. Note that our agent will be summoned in different environments randomly for each evaluation. Biomes and initial positions are also different each time. Following the previous work [18], we take the success rate as the evaluation metric.
Baselines We compare DEPS with other language-based planners, including GPT as Zero-shot (aanner(GPT) [16]. ProgPromp((PP) [42],
[45], Inner Monologue(IM) [17], and Code as Policies(CaP) [20]. For all baseline models, we use the same demonstration example in the prompt, the same LM model from OpenAI, and the same controller in all tasks for fair comparison. Since these methods were not originally experimented with Minecraft, we reproduce them to conform to the Minecraft specification based prompt and feedback template design. All planner methods access the LLM model through OpenAl API (text-di eext-davinncs -03 model [32] for GPT, CoT, and IM, and code- -aw-divviiii--o parameters of L.LM (including the temperature and best_of, etc.) are kept as default. We also list the full prompt of all different methods in Appendix G.
model [8] for PP, CaP, and Ours). All hyper-
4.2 Main Results
Every task executed 30 times and the average results in Minedojo [10] for every meta task are listed in Table 2. Our approach achieves the best performance with all meta tasks. As the complexity of the task increases from MTI-MT8, the planner usually needs give more accurate task steps (i.c., longer goal sequence) to achieve the final task. Therefore the success rate of all agents decreases with the reasoning steps increasing. Starting from MT6, almost all existing LLM-based planners fail (nearly 0 success rate). DEP (w/o Selector) already consistently beats existing LLM- planners in all meta tasks with significant margin. This validates that "describe, explain and plan" can estimate the reason for current plan failure and correct the original flawed plans. Due to the limited maximum episode length and restricted control success rate for hard goal (e.g., Mi ne d diamond wi th iron, pi ckaxe), the final success rate is still capped.
7

---

Table 2: Success rates of DEPS and existing LLM planners on Minecraft 101.
The full task-by-task list in
Appendix
Methods
MI1
MI2
MT4
M15
MI6
MI7
AVG
GPT
25.85-224.8
47.88+331.5
10.78+ 14.6
7.14+9.00
1,98+5.99
0.0+-0.0
0.0+0..
0.0+00.0
15.42
PP[42]
30.61+23.6
40.09+330.6
17.13+19.1
16.00+17,3
3,21+4,9
0,47=1.3
0.00+2.2
0.0+0.0
16.88
C6T[45]
40.24:130.8
25.211126.8
6.82-1166
4.76188.2
1.73-15.2
0.0.0.00
0.0.1..0
0.0.1..0
18.89
[M[17]
46.89+31.4
51.734-20.8
3.6446.9
18.4441744
4.574744
0.64+1.7
1.0242.5
0.0+00.0
21
CaP(20]
60.08
60.11
8.72.19.7
20.33.1
2.84
0.63.11.3
0.60.12.2
0.0.1..0
25.77
DEP
10.4
66
13-143433.4
45.69+16.2
43.354+20.2
15.93+13.9
5,71+3.7
4.60+7.1
0.50-0.5
39.36
DEPS
79.77+8.5
19.4
62.40:17.99
53.32429..
29.24=27.3
13.80+8.0
12.56=13.3
0.59=0.5
48.56
In addition, selector also greatly improves the final task success rate of the agent (from DEP w/o Selector to DEPS). Hard meta tasks usually require the completion of multiple sub-goals (up to dozens of goals), thus bringing more flexibility and providing more candidate goals for the Selector. At the same time, as the agent conducts experiments with limited episode length, it also places high demands the efficiency of the plan. Therefore, the Selector brings significant improvement on efficiency-sensitive tasks such as MT7 (up to +2.7 times success rate).
Robustness on different controller and different Minecraft versions We also evaluate DEPS on MineRL [3] and WCC-extworld [22]. Note that DEPS is planning method, which needs to equip the goal-conditioned controller for interacting with the Minecraft environments. We choose MC-Controller [6] and Steve- 1 [21] as controllers to interact with Minedojo and MineRL, respectively, These two methods are all control policies that perceive visual partial observations and produce mouse and keyboard actions. While MC-Textworld is text world, which only keeps the Minecraft crafting recipes and mining rules. So MCTTextworld does not require the controller. The DEPS results of the task set MTI-MT8 on different Minecraft environments results report that DEPS can generate effective plans in various Minecraft environments, The results on MC-Textworld [22] also show that the performance drops on more difficult task sets from MT6 to MT8 are mainly from the controller limitation.
shown in Table 3. The
Table 3: Success rates of DEPS under different Minecraft environments.
Environment
Version
Controller
MT1
MT2
MT3
MT4
MTS
MT6
MT7
MT8
MineDojo [10]
1.11.2
[6]
79.77
79.46
62.40
53.32
29.24
13.80
12.56
0.59
MineRL [3]
1.16.5
[2]]
84.05
80.32
24.25
36.21
9.16
17.22
16.79
1.84
MC-Textworld [22]
1.19.2
100.00
90.00
80.00
56.25
64.71
57.14
69.57
50.00
4.3
Ablation Study
We conduct ablation experiments to investigate the number of candidate executable goals for different Selector models and the specific impact of the rounds of DEPS.
4.3.1 Ablation on Selector
We verify the robustness of our proposed Selector under different parallel goals. The agent is asked to complete 2, 3, and 4 candidate goals (the precondition is consistent for all goals), respectively. The goals of the task correspond to different kinds of mobs or materials.
We report the final success rate of our method (DEP) with different selector implementations, including using a fixed sequence of goals, a random sequence of goals, and selecting a goal based on MineCLIP [10]. CLIP [35], and our horizon- predictive Selector (HPS). As Figure 4 shows, in one round of parallel candidate goals, an improvement our corizon--pedicttin Selector compared to not any selector (i.e., fixed plan), respectively.
3=+22.3% +29.2%, +32.6% is obtained using
At a limited episode length, e.g., 1000 steps, goal-model shows a greater advantage, which proves that goal-model can improve the execution efficiency of the plan in embodied environments. In addition, compared to using vision-language models such as CLIP [35] and MineCLIP [10] as a goal model, horizon-predictt has the best performance due to better estimation of the horizon information. The curve trend also demonstrates that agents with Selector scale up under large amounts of goals in an open-world environment.
8

---

Table 4: Success Rate of DEPS under different maxi- mum rounds of re-planning. Round 0 represents the vanilla Planner w/ the re-planning process. oo rep- resents the re- re-planning process will not end until task success reaching the maximum horizon, which is still limited the maximum tokens of LLMs. The maxi- mum number of rounds for Codex around 7-8 rounds.
1006
1568
250
3003
Puxffil
Geab PeaifeiGeaas-
Mairzes Ephode Lengh
Selasse Model
Rounds
3
5
0
(0 - oo)
Gark Seting
Biomee Platis
MT1
28.6
50.6
68.1
79.8
79.8
+51.2
MivC
37.1
71.2
71.4
79.2
79.5
+42.4
Raad
MT3
15.1
20.1
40.3
40.8
62.4
+47.3
MT4
15.9
17.4
48.3
50.7
53,3
+37.4
MT5
3.2
3.2
3.2
15.2
29.2
+26.0
MT6
0.5
0.5
1.1
1.9
13.8
+13.3
Figure The success rates DEPS with differ- ent selectors under varying numbers of parallel goals and maximum episode lengths.
MT7
0.6
2.3
2.9
2.9
12.6
+12.0
MT8
0.0
0.0
0.0
0.0
0.6
+0.6
4.3.2 Ablation Re-Planning Rounds
We evaluate our agent on all tasks with increasing maximum rounds of DEPS. The round is defined as a cycle of interactive LLM- based planning with description, explanation, and planning and selecting, ie., an updated plan. All tasks for every maximum round are executed 30 times and the average success rate is reported in Table 4. We take the vanilla LLM planner as the baseline, in which the model takes the initially generated plan as final execution plan, without involving any description, re-planning, or self-explanation processes during the task execution. Our results in the previous subsection utilize the maximum rounds possible under maximum tokens capped by OpenAI. We also report the success rate increment from vanilla planner to DEPS of every meta task in column in Table This set of experiments demonstrates that DEPS can iteratively improve its plan in open-world environments. More description, self-explanation, and re-planning rounds produce better results, especially for hard tasks.
4.4 ObtainDiamond Challenge
Mining diamonds in the open-world game Minecraft, i.c. MT8 in Table 2. has been a long-standing challenge for the community [14]. It is challenging because mining diamonds from scratch in Minecraft involves acquiring a sequence of difficult-to-obtain items that require complex planning on goals like mining, inventory management, crafting with and without crafting table, tool use, smelting iron ingot in furnace, and mining at the lowest depths. We take the Obtaini amond task as a bonus experiment show the capabilities of our zero-shot planner on complex tasks in embodied environments. Previous methods' success rates on this challenge further vouch for its difficulty. [43, 34] leverages domainn-seiite reward functions and RL fine-tuning to achieve -0 1% success rate in 15 minutes of game play. VPT further boosts the success rate to 20% within 20 minutes of play through pre-training on collects 700 hours human demonstrations and finetuning with numan-designee reward function [3]. DreamerV3 is trained from scratch to collect diamonds in a modified Minecraft environment (easier to break blocks) with world models to achieve a success rate of 2% [15].
Our DEPS manages to achieve on-par performance in this grand challenge; our agent achieves 0,59% success rate within 10 minutes of gameplay. Note our method does not specifically fine-tune for this challenge. is designed be multi-task its nature. Furthermore, considering our planner operates with demonstration prompts on a fixed Large Language Model, it can be straightforwardy adapted other open-ended environments with modifications.
5 Related Works
Task planning with LLMs There have been some methods leveraging the large language model
to generate action plans for high-level tasks embodied environments [46, 9, 11]. [16] decompose natural language commands into sequences of executable actions by text completion and semantic

---

translation, while SayCan generates feasible plans for robots by jointly decoding an LLM weighted by skill affordances from value functions [4]. For better executing the plan in embodied environments, some methods use an object detector describing the initial environment into the language prompt to produce environment-suitabl plans and adopt success detectors to check that each step is executed successfully [17, 20]. [42] and [20] use the pythonic-style prompt to produce more executable plans. However, all of the above methods assume that the initial plan from the LLM is correct. When there are bugs in the initial plan, it's difficult for the agent to finish the task successfully.
Interactive Planning with LLMs Inner Monologue [17] pilots the front of interactive planning with LLMs, which introduces the feedback (including success detection and scene description) to the planner. However, we found it could still suffer from accumulative planning error, especially in long-horizon open- world tasks. Rather, our "Describe, Explain, Plan and Select" (DEPS) method can produce more reliable plans by leveraging chain-of-though thinking and explanation to locate the errors in previous plans. Moreover, we also propose goal Selector further improve the efficiency of the plan, thereby yielding much better performances. Readers are encouraged to refer to the comparative results in Section 4.2 between DEPS and these prior arts. There are also some concurrent works on planning with LLMS [39, 26, 23, 33, 47].
Some previous works have employed the hierarchical architecture solve long-horizon tasks in Minecraft [30, 27, 24]. Recently, based on the internet-scale corpus, [10] pre-trains Ianguage-condtioone reward function and learns multi-task MineAgent. [3] collects vast amount of human demonstrations to train a behavior cloning agent. More recently, [15] utilized a learned world model to distill a policy that can efficiently explore in Minecraft. There are also some works focus on learning goal-conditioned policies for better instruction-folloiin these efforts all focus on improving the low-level controller. Rather, the planner in our architecture emphasizes applying domain knowledge to propose and arrange the sub-goals. It significantly influences the complexity and breadth of tasks that the agent can handle. Moreover, our planner is zero-shot, making it possible to generalize to other long- long-horizon open worlds.
Agents in Minecraft
[6, 7, 21]. While
6 Limitations
Albeit the impressive results of our approach, we believe there are at least two major limitations within our approach. First of all, our framework relies privately-held LLMs like GPT-3 and ChatGPT, which makes it less accessible to those who cannot afford or access the service. However, we're fully committed to ensuring more democratized method and will explore using open-sourced models including OPT [48] and BLOOM [38]. Another issue is the explicit step-by-step planning in our system. Although it brings us superior performances over the baselines, the planning bottleneck can also prevent our model from being further scaled up. A more appealing approach will be amortizing the planning within an end-to-end trainable goal-conditioned policy, which is worth exploring next. Furthermore, some previous fundamental challenges in planning (e.g., dead ends) may not prevalent in our adopted environments and hence could be inadvertently overlooked by our paper. We dedicated to addressing more fundamental challenges present in building multi-task generalist agent in our series of following work.
7 Conclusion
We investigate the problem of planning in open worlds. We identify two major challenges unique to these environments: 1) long-term planning requires precise and multi-step reasoning, and 2) planning efficiency could be compromised since canonical planners do not take the agent's proximity to parallel goals/subtasks into consideration. We propose "Describe, Explain, Plan and Select" (DEPS), an interactive approach based on Large Language Models (LLMs) to tackle them both. Our experiments in the challenging Minecraft domain verify the advantages of our approach over counterparts by marking the milestone of robustly accomplishing 70+ Minecraft tasks and nearly doubling the overall performances. DEPS also is the first planning- based agent that can reach the diamond in this game.
10

---

Acknowledgements
This work is funded in part by the National Key R&D Program of China #20222D0160 grant from CCF-Tencent Rhino-Bird Open Research Fund, NSF grants #1IS-1943411, 195--195441, #CCF-1837129) 129, an SRA from Meta and a research gift from Amazon Alexa AI, and a gift from RelationalAI. We thank Dai Zhixiang from NVIDIA and Xu Hongming from BIGAI on training LLMs and infrastructure supports, respectively.
References
[1] J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, et al. Flamingo: visual language model for few-shot learning. arXiv preprint arXiv:2204.1412 2022.
[2] P.-L. Bacon, J. Harb, and Precup. The option-critie architecture. In Proceedings of the AAAI conference on artificial intelligence, 2017. 1
[3] B. Baker, I. Akkaya, P. Zhokhov, J. Huizinga, J. Tang, A. Ecoffet, B. Houghton, R. Sampedro, and J. Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos. arXiv preprint arXiv:2206.11799 2022. 7, 8, 9,
[4] A. Brohan, Y. Chebotar, C. Finn, K. Hausman, A. Herzog, D. Ho, J. Ibarz, A. Irpan, E. Jang, R. Julian, et Do can, not i say: Grounding language in robotic affordances. In 6th Annual Conference on Robot Learning, 2022. 1, 2, 3, 5, 10,
[5] Brown, B. Mann, Ryder, M. Subbiah, D. Kaplan, Dhariwal, A. Neelakantan, Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020. 1, 15, 21,
[6] S. Cai, Z. Wang, X. Ma, A. Liu, and Y. Liang. Open- Open-world multi-task control through goal-aware representation learning and adaptive horizon prediction. arXiv preprint arXiv:2301. 2003, 2023. 6, 8, 10, 21, 24
[7] Cai, B. Zhang. Z. Wang, X. Ma. Liu, and Y. Liang. Groot: Learning follow instructions by watching gameplay videos. arXiv preprint arXiv:2310.08235, 2023. 10
[8] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. O. Pinto. J. Kaplan, H. Edwards, Burda, N. Joseph, G. Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.033744 2021. 21, 22
[9] I. Dasgupta, C. Kaeser-Chen, K. Marino, A. Ahuja, S. Babayan, F. Hill, and R. Fergus. Col- laborating with language models for embodied reasoning. NeurIPS Foundation Models for Decision Making Workshop, 2022. 9
[10] L. Fan, G. Wang, Y. Jiang, Mandlekar, Yang. H. Zhu, A. Tang, D.-A. Huang, Y. Zhu, and A. Anandkumar. Minedojo: Building open-ended embodied agents with internet-ccaee knowledge. Advances Neural Information Processing Systems Datasets and Benchmarks, 2022. 1, 7, 8, 10, 18, 21, 26
[11] R. Gong, Q. Huang. Ma, Vo, Z. Durante, Y. Noda, Z. Zheng, S.-C. Zhu, D. Terzopoulos, L. Fei-Fei, et al. Mindagent: Emergent gaming interaction. arXiv preprint arXiv:2309.0997 2023.1
[12] W. H. Guss, M. Y. Castro, S. Devlin, B. Houghton, N. S. Kuno, C. Loomis, S. Milani, S. P. Mohanty, K. Nakata, R. Salakhutdinov, J. Schulman, Shiroshita, N. Topin, Ummadisingu, and O. Vinyals. The minerl 2020 competition on sample efficient reinforcement learning using human priors. arXiv: Learning, 2021.
[13] W. H. Guss, C. Codel, K. Hofmann, B. Houghton, N. Kuno, S. Milani, S. Mohanty, D. P. Liebana, R. Salakhutdinov, N. Topin, al. Neurips 2019 competition: the minerl competition on sample efficient reinforcement learning using human priors. arXiv preprint 1rXiv:1904.10073 2019.
11

---

[14] W. H. Guss, B. Houghton, N. Topin, P. Wang, C. Codel, M. Veloso, and R. Salakhutdinov. Minerl: A large-scale dataset of minecraft demonstrations, arXiv preprint 1XXi::907.114400 2019. 2, 7,9, 9, 24
[15] D. Hafner, Pasukonis, J. Ba, and T. Lillicrap. Mastering diverse domains through world models, arXiv preprini arXiv:2301.04004 2023. 9, 10
[16] W. Huang, P. Abbeel, D. Pathak, and Mordatch. Language models zero shot planners: Extracting actionable knowledge for embodied agents. ICML, 2022. 3, 4, 7, 8, 9, 15, 16, 17, 21, 24, 25, 26, 28
[17] W. Huang. F. Xia, T. Xiao, H. Chan, Liang, P. Florence, A. Zeng, J. Tompson, Mordatch, Y. Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.0555 00500, 2022. 2, 3, 5, 7, 8, 10, 15, 16, 17, 24, 25, 26, 30, 34, 35
[18] M. Johnson, K. Hofmann, Hutton, and D. Bignell. The malmo platform for artificial intelligence experimentation. In Ijeai, pages 4246-4225 4247. Citeseer, 2016. 2,
[19] A. Kanervisto. S. Milani, K. Ramanauskas, N. Topin, Z. Lin, Li, J. Shi, D. Ye. Q. Fu, W. Yang, W. Hong, Z. Huang, H. Chen, G. Zeng, Y. Lin, V. Micheli, E. Alonso, F. Fleuret, A. Nikulin, Y. Belousov, O. Svidchenko, and A. Shpilman. Minerl diamond 2021 competition: Overview, results, and lessons learned. neural information processing systems, 2022. 2
[20] J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng. Code as policies: Language model programs for embodied control. arXiv preprint arXiv:220900755, 2022. 2. 5, 8, 10, 25, 26, 31
[21] Lifshitz, K. Paster, H. Chan, J. Ba, and S. Mcllraith. Steve-1: A generative model for text-to-behavior in minecraft. arXiv preprint (1Xi::2306.00937 2023. 8, 10
[22] H. Lin, Z. Wang, Ma, and Y. Liang. Mcu: A task-centric framework for open-ended agent evaluation in minecraft. arXiv preprint arXiv:2310.08367, 2023. 6, 7, 8, 18
[23] K. Lin, C. Agia, T. Migimatsu, M. Pavone, and J. Bohg. Text2motion: From natural language
instructions to feasible plans. arXiv preprint arXiv:2303. 2023. 10
[24] Z. Lin, J. Li, J. Shi, D. Ye, Q Fu, and W. Yang. Juewu-me: Playing minecraft with sample- efficient hierarchical reinforcement learning. arXiv preprint arXiv:2112.04900) 2021.
[25] X. Ma, S. Yong, Z. Zheng, Q. Li, Y. Liang, S.-C. Zhu, and S. Huang. Sqa3d: Situated question answering in 3d scenes. arXiv preprint 10X0422210007474, 2022.
[26] J. Mai, J. Chen, B. Li, G. Qian, M. Elhoseiny, and B. Ghanem. Llm as a robotic brain: Unifying egocentric memory and control. arXiv preprint wxXiv:2304009345 2023. 10
[27] H. Mao, C. Wang, X. Hao, Y. Mao, Y. Lu, C. Wu, J. Hao, D. Li, and P. Tang. Seihai: A sample- efficient hierarchical ai for the minerl competition. In Distributed Artificial Intelligence: Third International Conference, DAI 2021, Shanghai, China, December 17-18, 2021, Proceedings 3, pages 38 51. Springer, 2022. 10
[28] S. Min, Lyu, A. Holtzman, M. Artetxe, M. Lewis, H. Hajishirzi, and Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202. 12837, 2022. 23
[29] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 2013. 2
[30] J. Oh, S. Singh, H. Lee, and P. Kohli, Zero-shot task generalization with multi-task deep reinforcement learning. In International Conference Machine Learning, pages 2661-2670 PMLR, 2017. 10
[31] OpenAl. Gpt-4 technical report, 2023, 21, 22
12

---

[32] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, al. Training language models to follow instructions with human feedback. arXiv preprini arXiv:2203.02/55. 2022. 3, 7,8
[33] J. S. Park, J. C. O'Brien, C. J. Cai, M. R. Morris, P. Liang, and M. S. Bernstein. Generative agents: Interactive simulacra of human behavior. arXiv preprint arXiv:2304.03442, 20233 10
[34] V. P. Patil, M. Hofmarcher, M.-C. Dinu, M. Dorfer, P. M. Blies, J. Brandstetter, J. A. Arjona- Medina, and S. Hochreiter. Align-rudder: Learning from few demonstrations by reward redistribution. ICML, 2020. 9
[35] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark, al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8788 PMLR, 2021. 5, 6, 8
[36] S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikov, G. Barth-Maron, M. Gimenez, Sulsky, Kay, J. T. Springenberg, al. A generalist agent. arXiv preprint arXiv:2205.0617. 2022. 1
[37] N. Reimers and I. Gurevych. Centence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv: 1908. 10084, 2019. 23
[38] T. L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ilié, D. Hesslow, R. Castagné, A. S. Luccioni, F. Yvon, M. Gallé, al. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:22/1.0555
2022. 10
[39] N. Shinn, B. Labash, and A. Gopinath. Reflexion: an autonomous agent with dynamic memory and self-reflection. arXiv preprint arXiv:2303. 11 366, 2023. 10
[40] M. Shridhar, L. Manuelli, and D. Fox. Cliport: What and where pathways for robotic manipula- tion. In Conference on Robot Learning. PMLR, 2022. 2, 3, 15, 16, 17, 23
[41] M. Shridhar, X. Yuan, M.-A. Côté, Y. Bisk, A. Trischler, and M. Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. arXiv preprint arXiv:2010.03768, 2020. 2, 3, 15, 16
[42] I. Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg. Progprompt: Generating situated robot task plans using large language models. arXiv preprint arXiv:2209.11300 2022. 7, 8, 10, 25, 26, 28
[43] A. Skrynnik, A. Staroverov, E. Aitygulov, K. Aksenov, V. Davydov, and A. I. Panov, For- getful experience replay in hierarchical reinforcement learning from expert demonstrations. Knowledge-Bee Based Systems, 218:106844 2021.
[44] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint Xiv:2307.009288 2023. 22
[45] J. Wei, X. Wang, D. Schuurmans, M. Bosma, E. Chi, Q. Le, and D. Zhou. Chain of thought prompting elicits reasoning in large language models. 36th Conference on Neural Information Processing Systems (NeurIPS 2022), 2022. 5, 7, 8, 15, 25, 26, 29
[46] A. Zeng, A. Wong, S. Welker, K. Choromanski, F. Tombari, A. Purohit, M. Ryoo, V. Sindhwani, J. Lee, V. Vanhoucke, et al. Socratic models: Composing zero-shot multimodal reasoning with language. arXiv preprint arXiv:2204.00550
2022. 2,9
[47] C. Zhang, K. Yang. S. Hu, Z. Wang, G. Li. Y. Sun, C. Zhang, Z. Zhang, A. Liu, S.-C. Zhu, et al. Proagent: Building proactive cooperative ai with large language models. arXiv preprint ar0iv:2308.1 11339, 2023. 10
[48] S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li. X. V. Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint 200502205.000688 2022. 10
13

---

Appendix
Contents
A Additional Experiments
15
ALFWorld
15
A.1.1 Tasks
15
A.1.2 Results
15
A.2
Tabletop Manipulation
16
A.2.1 Tasks
16
A.2.2 Results
17
Minecraft Task Details
18
DEPS Implementation Details
20
C.1 Controller
20
C,2 LLM as Planner
C,3 LLM as Explainer
22
C.4 Other modules
23
Comparison with other LLM-based Planners
23
E Discussion ObtainDiamond Task
24
Success Rates of ALL Tasks in Minecraft
25
G Prompt for Different Tasks and Different Methods
27
G.1 Prompt for Minecraft Tasks
27
G.1.1 DEPS
27
G.1.2 Baselines
28
G.2
Prompt for ALFWorld Tasks
G.2.1 DEPS
G.2.2 Baselines
33
G.3 Prompt for Tabletop Manipulation Tasks
34
G.3.1 DEPS
34
G.3.2 Baselines
35
H Full Dialogue
35
14

---

A Additional Experiments
Additional experiments conducted on the ALFWorld [41] and Tabletop Manipulation environ- ments [40] to showcase the generalization capabilities of DEPS.
A.1 ALFWorld
ALFWorld [41] is interactive learning environment that aligns text and embodiment, allowing agents to acquire abstract, text-based policies in TextWorld, and subsequently execute goals from the ALFRED benchmark in visually rich environment.
A.1.1 Tasks
The ALFWorld framework contains six types (namely Pick & Place, Examine Light, Clean & Place, Heat & Place, Cool & Place, Pick Two & Place) of tasks with various difficulty levels. Tasks involve first finding a particular object, which often requires the agent to open and search receptacles like drawers or cabinets. Subsequently, all tasks other than Pick & Place require some interaction with the object such as heating (place the object a microwave and start it) or cleaning (wash the object in sink). To complete the task, the object must be placed in the designated location. We sample 10 tasks from ALFWorld randomly and list all the task names, types, and the number of receptacles in Table 5. We classify them into 6 groups based on their functionality. For all tasks, the maximum number of steps is set as 50.
Table 5: Task list in ALFWorld.
Group
No.
Task
Number of Receptacles
put some soapbottle on garbagecan
13
Pick & Place
put tissuebox in dresser
26
put some soapbar on drawer
15
put clean soapbar in bathtubbasin clean some tomato and put it in fridge
16
Clean & Place
35
put a cool tomato in countertop put a cool bread countertop
30
Cool & Place
27
Heat & Place
8
heat some cup and put it in cabinet
36
Pick Two Place
find two cup and put them cabinet
36
Examine in Light
10
look at mug under the desklamp
18
We select the GPT as Zero-Shor Planner (GPT) [16] and Inner Monologue (IM) [17] as baseline methods. For the Inner Monologue, the planning goal is the next goal among all candidate goals. For the GPT and DEP, which produce the full plan at once, the planning goal is the full plan (a goal sequence). Then the plan will executed step-by-step. i.e., the current goal will be given to the controller and select suitable action according to the current state. The goal termination module is also employed with the LLM., For better demonstrate the effectiveness of ixfeaaxaanntionn DEP, we also augment the zero-shot planner with re-planning ability (GPT+RP). All planner methods access the LLM model through OpenAl API (text (text-davince davddacvicc 03 model [5]). Since ALFWorld a text world, the environment will be given literal description and candidate language-conditione actions for each state, so the controller under ALFWorld is also LLM-based. Chain-of-Thought [45] is also employed in the controller for better decision-mail maaking All prompts for planner and controller in ALFWorld are listed in Section G.2.
A.1.2 Results
Each task is executed five times, and the average results for each task group are presented in Table 6. BUTLER is the a training-based method, the results are soureed from [41]. Re-planning a crucial capability in complex and exploratory environments. The short-horizon planning approach (IM) with
15

---

Task: put some soaphottie garbagacan
inball frame
Goal: pick soapootte cabinet Action: cabinet
Goal: pick scapbonte cabinet 4 Action: open
Goal: pick scapbotte cabinet 4
Goal: soapbottie
Goal: soapbottee
serbagecan begecan
serbagecan
Action: cabinet
begecan
Action go to gerbage- can
Action: put soephottle Inion gartagacan
Figure 5: Planning in the ALFWorld experiments.
re-planning capability outperforms the long-horizon planning approach (GPT) without re-planning capability with a large margin. Furthermore, the long-horizon planning method augmented with re-planning capability (GPT+RP) achieves superior performance ranging from 10% (GPT) to 52%. DEP further enhances the feasibility of planning with descriptions and self-explanation. Notably, all planning methods fail Place Two Place tasks, which is attributable LLM's lack of requisite knowledge for this task. worth investigating how to effectively incorporate the distinctive knowledge of environment into LLM.
Table Success rates of tasks in ALFWorld.
Group
BUTLER [41]
GPT [16]
GPT+RP
IM [17]
DEP
Pick Place
46.0%
33.3%
100.0%
33.3%
93.3%
Clean & Place
39.0%
0.0%
10.0%
50.0%
50.0%
Cool & Place
100.0%
0.0%
30.0%
50.0%
100.0%
Heat & Place
74.0%
0.0%
40.0%
0.0%
80.0%
Pick Two & Place
24.0%
0.0%
0.0%
0.0%
0.0%
Examine in Light
22.0%
0.0%
100.0%
0.0%
100.0%
Average
37.0%
10.0%
52.0%
30.0%
76.0%
Average
.0%
10.0%
52.0%
30.0%
76.0%
A.2
Tabletop Manipulation
The Tabletop Manipulation experiments are conducted Universal Robot UR5e with suction gripper in the simulated environments [40].
A.2.1 Tasks
The assessment of all methods is conducted in five tasks, illustrated in Table 7, wherein the seen tasks are employed for training the CLIPort [40] as the controller. The task involves a robotic arm equipped with gripper, which is tasked with rearranging number of blocks and bowls table to achieve desired configuration specified via natural language (e.g., "putting the blocks in the bowls with matching colors").
Table 7: Task list in CLIPort.
No
Task Name
Instruction
Assembling Kits
the objects in the corresponding holes. Move the rings to the darker brown side. Match the blocks and the bowls.
2 Towers of Hanoi
3 Put Block in Bowl
4 Packing Shapes
Pack the objects in the brown box. Stack the blocks into pyramid.
5 Stack Block Pyramid
16

---

assemeing-iis
Inttal trame
the eyan lotior shape in the lotter shape hole
2. put the green hexagon hesagon hole
3. put the blue letter M shape in the leder shape hole
the brown ring the ring hole
put red letarf shape in the leter shape hole
Task pub- ook-nnbow
put
bowl
Taski packing appes
hame
1. the broon
android the brown
figure brown box
the clock the
butler chocolatee the brown box
Figure 6: Planning in the Tabletop Manipulation experiments.
We utilized Inner Monologue (IM) [17] and Zero-shot Planner (GPT) [16] as planning baselines, in addition to comparing with a multi-task CLIPort policy directly trained on long-horizon task instructions (i.e., without utilizing LLM for planning). As CLIPort is single-step policy that does not spontaneously terminate during policy rollout, we report CLIPort evaluations with Oracle termination (i.e., repeat until the Oracle indicates task completion) and fixed-step termination (i.e., repeat for steps). For Inner Monologue, which directly produces the nexr-step goal and terminates when the LLM ceases generate new steps, similarly set the maximum number of steps be for practical considerations. For the zero-shot planner [16] and our DEP, which produce the full plan once, they are augmented with the LLM-based termination. DEP also involves the description, explanation, and re-planning process. The same k step is suitable for these two methods. In practice, is 15. The prompts for all methods are listed in Section G.3. We use the checkpoints provided by CLIPort as the controller and all planner methods access the ChatGPT (as LLM) through OpenAI API (gpt-3. 5-turbo model). Each task evaluated 5 times with different seeds.
A.2.2 Results
The results of each method are listed in Table 8. All LLM-based planning methods perform well on tabletop rearrangement tasks. Given the compact nature of the tabletop environment, the performance gap among the various L.MM-planiig methods not as pronounced as in the open-ended Minecraft. This observation underscores the robust generalization capabilities of LL--e planning methods across diverse environments.
Table 8: Success rates for various methods across different tasks in Tabletop Manipulation environment.
Task
CLIPort [40] +oracle
GPT [16]
IM [17]
DEP
Assembling Kits
60.0%
60.0%
60.0%
60.0%
Towers of Hanoi
100.0%
100.0%
40.0%
100.0%
Put Block in Bowl
100.0%
100.0%
82.0%
100.0%
Packing Shapes
40.0%
40.0%
60.0%
40.0%
Stack Block Pyramid
80.0%
100.0%
40.0%
100.0%
Average
76.0%
80.0%
56.4%
80.0%
17

---

B Minecraft Task Details
To fully validate the multitask planning and execution capability of our agent, we choose over 70 tasks from the Minecraft Universe Benchmark [22] as the set of evaluation tasks. These tasks are related items that can be obtained in the Minecraft overworld. These tasks are also a subset of MineDojo [10] programmatic tasks. Minedojo exists some programmatic tasks sharing the same object item given different conditions (c.g.. obtain wool given shear obtain wool given nothing). Minedojo expands the richness of the same tasks (sharing the same Minecraft item as an object) by giving different initial conditions (e.g., obtain wool given shears obt abtain wool given nothing). We keep only the 71 hardest conditions (i.e. given nothing) as tasks.
We list all task names, objects, and their required skills number for planning from Table 9 to Table 16. Object item is used as the basis for the successful completion of the task. These objects cannot be obtained directly from the environment, and usually require multiple goals (i.e., reasoning steps) to be constructed. Here we only consider the number of required goal types, and multiple identical goals are unified into I reasoning step. Note that the reasoning steps for each task are not fixed, and as the initial state of the agent and the biome is in change, more reasoning steps may be required to complete it, we only report the most basic case here.
As shown Figure 4, for each task, a relaxed (longer) maximum episode steps will increase the success rate of the task. To fully test the efficiency of our method, we set upper limit on the episode length for each task. Since different tasks have different difficulty levels, we double the average completion time of human players for different meta-tasks as the upper limit of the episode. The play time are computed as corresponding maximum steps (i.e., Max. Steps Table 1) of episode length at 20Hz.
Table 9: Task details on MTI Basic set.
Required Skills
Initial
Meta-Task Task
Task Name
Object
Instruction
Inventory
CrafLPlanks
planks
Obtain plank.
CraftSticka
stick
Obtain stick.
CraItWoodenSaab
4
wooden_sshe slab
Obtain wooden slab.
4
crafftoodeneresssrre
goodenn_pressure
Obtain wooden pressure plate.
CraftBowl
4
bowl
Obtain bowl.
CyaftWoodenButton
poodeen_bttton
Obtain wooden button.
MT1
CrafttChest
4
chest
Ohtain chest.
null
Basic
Crafttakktaiirs
4
oak_staing
Ohtain an oak stair.
CraftSign
sign
Obtain sign.
10
CraftFence
fence
Ohtain fence.
11
CraftForceeate
fence gate
Ohtain fence gate.
12
CraftBoat
4
boat
Obtain boat.
13
CrallTrapdoor
4
trapdoor
Obtain trap door.
14
WoodenDoor
door
Obtain door.
Table 10: Task details on MT2 Tool (Simple) set.
Required Skills
Initial Inventory
Mcta-Task
Task Name
Object
Instruction
Cra CrafftCraftinggabll.
crafting_taii table
Obtain crafting table.
16
CraftWoodene ickaxe
boooden_pickuee
Obtain wooden pickuxe.
17
CraftWoodenAxe
wooden_axe
Obtain wooden axe.
18
Cra Foe
wooden_hoe
Obtain wooden hoe.
CraftMoodenšwozd
wooden_ssoord
Obtain wooden sword.
MT2
CrafttWoodnnSheell
wooden_shover null
Obtain wooden shovel. Obtain a furnace.
Tool
21
CraftFurnace
furnace
(Simple)
22
SrafttstonePckkan
stone _inc___ccxxeeee
Obtain stone pickaxe.
23
CraftStoneAxe
stone_ axe
Obtain stone axc.
CraftStoneHoe
hoe
Obtain hoe.
Cra
stone_shovel
Obtain shovel.
Cra Srateesood
stone stone_sword
Obtain stone sword.
18

---

Table 11: Task details MT3 Hunt and Food set.
Required
Initial
Meta-Tash
ID
Task Name
Object
Instruction
Skills
Inventory
27
CraftBed
bed
Obtain bed.
28
CraftPainting
painting
Obtain painting.
MT3
29
CraftCarpet
carpet
Obtain carpet.
Hunt
30
CraftItemFrame
item frame
item_frame
null
Obtain item frame.
&
31
CookPorkchop
cooked porkkchpp
Cook the porkchop.
Food
32
CookBeef
cooked_beck beef
Cook the beef.
33
ton
cooked_mutton
Cook the mutton.
Table 12: Task details on MT4 Dig-Down set.
Required Skills
Initial
Task-Task
ID
Task Name
Object
Instruction
Inventory
34
CraftsttonBBtairs
stone_stail ssairs
Obtain stone stair.
35
CraftStoreSlatt
stone_sha
36
CraftArmozStand
armor stand
Obtain armor stand.
37
Crofftcabbbeettneallll
cobblestone wall
Obtain cobblestone wall.
38
CraftquartzBlock
10
quartz block
Obtain quartz block.
39
CraftsstoeRrrick
stone_bria
Obtain stone brick.
Dig-Down
40
SmeltStore
stone
null
Smelt stone. Obtain stone brick.
41
CractTorch
torch
42
ObtainCoal
cnal
Mine coal.
CraffestoneBricckttaiis
stonebrick stairs
Obtain stone brick.
44
CraftStoneP reassurcaae
stone stone_pressure plast
Obtain brick.
45
CraftStoneButton
stone_ button
Obtain brick.
46
Craetlaver
level
Obtain stone brick.
Table 13: Task details on MT5 Equipment set.
Required Skills
Initial
Meta-Task
Tusk Name
Object
Instruction
Inventory
47
SguugLeethee-eer
leather_ boots
Equip the leather boot.
48
pruuppeetherrheatppl.
Equip the leather chestplate.
49
Equi heetppettteeeeeeeeeee
leather helmet
Equip the leather helmet. Equip the leather leggings. Equip the shield. Equip the iron chestplate. Equip iron leggings. Equip helmet. Equip
Rqui pl.eatthe heegg ngs
leather leggings
MT5
EquipShield
shield
null
Equipment
iron chestplate
EquipIronLegging
iron Jeggings
ronHel
imn helmet
iquiporonBoots
imn_ boots
Table 14: Task details on MT6 Tool (Complex) set.
Required Skills
Initial Inventory
Meta-Task
ID
Task Name
Object
Instruction
56
CraftBucket
11
bucket
Obtain a bucket.
57
CraftShears
11
shears
Make shears.
MT6
58
CraftIronPickawe
11
iron_pickaxe
Obtain an iron pickaxe.
Tool
59
CraftIronAxe
11
iron_axe
null
Obtain iron axe.
(Complex)
60
CraftIronHoe
iron hoe
Obtain an iron hoe.
61
CraftIrenShovel
11
iron shovel
Obtain an iron shovel.
62
rraftrronsword
11
iron_sword
Obtain iron sword.
19

---

Table 15: Task details MT7 Iron-Stage
Required Skills
Initial Inventory
Meta-Task
ID
Task Name
Object
Instruction
63
craft
iron hars
Ohtain iron bar.
64
CraftIronNigget
11
tron_nugget
Obtain an iron nugget.
65
CraftMinccart
11
minecart
Obtain minecart.
66
CraftHopper
12
hopper
Ohtain hopper.
67
Crafttloppe:.Lecarr
hopper, hopper_minecart minecart
Obtain hopper minecart.
CrafttorraceeMirecart
furnace minecart
Obtain fumacee minecart. Obtain cauldron. Ohtain chest minecart. Obtain an iron door.
MT7
CraftCauldron
cauldron
null
Iron tron-Stage
70
CraftchhstMiieec
chest minecart
71
CraftIronDoor
iron_door 11
72
CraftIzonTrapdorr
11
iron trapdoor
Obtain an iron trapdoor.
73
PraftTripwiraHod
tripwire book
Ohtain tripwire hook.
74
CrafthNPressuzePee::
heavy weaay__weghhd__paee
Obtain heavy weighted plate.
CraftRail
ruil
Obtain
Table 16: Task details on MT8 Challenge set.
Required Skills
Initial
Meta-Task
ID
Task Name
Object
Instruction
Inventory
Challenge MT8
76
ObtainDiamond
12
diamond
null
Obtain diamond.
C DEPS Implementation Details
We study three different implementations of DEPS for each the experimental settings. While each version incorporates description and self-explanation to improve planning of LLM, there are differences in the internal components of each system, as seen Table 17.
Table 17: Comparison between different versions of DEPS implemented in three different environments.
Minecraft
ALFWorld
Tabletop Manipulation
LLM
code-davinci-e
text Text-davincil
3pt-3.5-turie
Controller
Behavior Cloning Learned
LLM-based
CLIPort
Descriptor
Inventory Description
Eny Support
heuristics
Explainer
LLM-based
LLM-based
LLM-based
Selector
Horizon Prediction Module
N/A
N/A
C.1 Controller
As the name implies, tasks in Minecraft are usually related to mine and cra ft goals. nine goals require the agent to collect raw materials from the environment using the appropriate tools. Craft goals ask the agent to synthesize using existing materials, Any raw material used requires the agent collect through suitable tools (e.g., diamonds can only be collected by an iron pickaxe pickaxe). So a task usually requires dozens of step-by-step mine and craft goals, as the required skills in Table 9. Note that the successful execution of task needs satisfy certain exact numerical constraints due to the presence of strict generation recipes in the environment (e.g., log can craft 4 planks, so harvesting 6 planks requires at least logs). When number materials collected is not enough, the goal cannot be completed successfully. When more materials are collected than actually needed, the execution success rate of the task could also be reduced because the plan can not be finished under the maximum action steps.
better
20

---

Table 18: The success rate of different skill/goal with imitation learning controller.
ID
Skill Description
Success Rate
Episode Length
0
Mine oak wood
0.39
600
1
Mine birch wood
0.29
600
2
Mine cobblestone with pickaxe
0.95
600
3
Mine 1 stone with pickaxe
0.70
600
4
Mine seed
0.18
600
5
Mine leaves with shears
0.68
600
6 Mine dirt
0.54
600
7 Mine iron ore with stone pickaxe
0.40
3000
8
Mine iron ore with stone pickaxe
0.16
3000
9
Mine diamond with iron pickaxe
0.35
12000
10
Mine diamond with stone pickaxe
0.00
12000
11
Kill 1 sheep with
0.44
600
12
Kill I cow with axe
0.60
600
13
Kill 1 chicken with axe
0.46
600
14
Kill 1 pig with axe
0.49
600
15
Kill 1 Ilama
0.50
600
16
Equip tool on mainhand
1.00
600
Craft w/o crafting table
1.00
600
17-261
Craft w/ crafting table
0.90
600
Smelt w/ furnace
0.80
600
We designed the agent's skill space based on these goals, as shown Table 18, with a total of 262 goals. Every goal is designed with objective item (e.g., mi necra ft cobbles lestone for skill "Mine 1 cobbl estone with pi ckaxe"), which used evaluate the achievement of the goal. The skill, as a goal-conditioned policy π (a]s, g) for decision-making. maps the current state and goal g to action a. The goal is specified as natural language instructions here, which similar to [4].
When training the controller, adopt the observation space provided by MineDoJo [10], which includes an RGB camera view, yaw/pitch angle, GPS location, and the type of x blocks surrounding the agent. We discretize the original actions. We use the proposed imitation learning method proposed by [6] in training. To be specific, modified goal-sensitive Impala CNN is used as the backbone network. The success rate under fixed episode length of every skill is listed in Table 18.
action space provided by MineDojo into 42 discrete
C.2 LLM as Planner
DEPS relies on Large Language Models (LLMs) to generate language-basee plans. In our Minecraft experiment, chose Codex [8] the LLM Planner because it accept longer input tokens and cost-effective. However, DEPS is compatible with various types of LLMs. Therefore, we used GPT3 [5] and ChatGPT as LLM Planners in the ALFWorld and Tabletop Manipulation experiments, respectively. Due to the effective planning and error correction performance of DEPS, the initial plan generated by the LLM has little impact on the final performance of the Agent. We also conduct ablation experiments on
even if the initial plan generated by the LLM has low accuracy, DEPS can generate final feasible plan through self-exxlanation and re- re-planning. Therefore, we conducted ablation experiments on LLM in Minecraft.
We choose Codex [8], ChatGPT, GPT3 [5], and recent GPT-4 [31] as Planners. We used Vanilla Planner [16] as baselines and excluded the re-planning process. Given the same prompt with DEPS, the performance baseline models reflects the planning ability of different L.LMs. The success rate of bascline and DEPS on different LLMs are reported in Table 19.
21

---

Table 19: Success rates for different LLMs Minecraft tasks.
Codex [8]
GPT-3 [5]
ChatGPT
GPT-4 [31]
Group
baseline
DEPS
baseline
DEPS
bascline
DEPS
baseline
DEPS
MTI
28.6
79.8
27.2
75.4
20.3
70.2
49.2
89.3
MT2
37.1
79.5
42.1
76.3
28.2
68.5
48.3
85.0
MT3
15.1
62.4
7.8
58.7
3.2
50.4
38.04
63.4
MT4
15.9
53.3
6.7
50.2
4.8
47.8
27.0
55.7
MT3
3.2
29.2
2.7
17.2
0.8
16.3
15.7
32.2
MT6
13.8
0.3
16.19
MI7
0.6
12.6
0.4
5.3
0.5
5.2
3.1
16.41
The success rate Vanilla Planner varies on the LLMs. The GPT-4 baseline achieved an initial plan accuracy twice as high as the baselines on other LLMs, demonstrating superior planning ability. After being augmented by Descriptor, Explainer, and Selector, DEPS based on different LLMs showed almost identical success rates. This indicates that DEPS-augmented LLMs can generate more feasible plans in open-worte wwoldd environments even if the initial plan is less successful.
is noteworthy that DEPS is constrained by the maximum token limits of various models, which dictate the maximum re-planning rounds that can be supported. Longer re-planning rounds tend to yield superior performance, particularly in long-horizon tasks requiring more skills (in MT6-MT7), detailed in the Section 4.3.
Since we use pretrained LLM as a planner, it indeed requires exposure to large amount Minecraft- related corpus during the pretraining phase. Considering that Minecraft is one of the most popular games worldwide, there is relatively abundant data about Minecraft available online. We conducted experiments using open-source pretrained LLaMA2-70B DEPS based on LLaMA2 also performs reliable planning under Minecraft conditions. Consider- ing limited training data used by LLaMA2, further finetuned an open-souree language model (LLaMA2-13N)) 13B) using Minecraft texts obtained from the internet which exhibited better planning performance. The results shown in Table 20.
several Minecraft tasks and found that
Table 20: Results of DEPS based on open-sourced LLaMA language models.
Language Model
CraftingTahhi
WondenPickaxe
Furnace
StonePickaxe
Pretrained LLaMA2-70B [44]
60.0
50.0
40.0
50.0
Finetuned LLaMA2-13B [44]
70.0
80.0
OpenAl Codex
80.0
66.7
73.3
C.3
LLM as Explainer
Given the description and previous plan, the explainer generate self-explanation of the failure of the current plan and give instructions to fix the bugs. The explainer is implemented with the OpenAI completion mode based text -davinci-ii models. The prompt for the explainer is listed in Listing! 1.
Here are some actions that the agent fails to perform in Minecraft. Please give the expl explanation action execution fai Pailure sccording to the current inventory information the agent.
#ff
Failed Action: mine (1' iron_ ooee null); step mine iren_ore witheut teal
Current Inventory: rull
Explanation: Because mining iron_ore needs to use the tool stone pickaxe, but my inventory does not have stone_ So need to craft stone_pickes first.
stene_ppccaxe,
If
Failed Action: craft (('stone pickaxe' :1), ['cobblestone' :3, 'stick' :2), ccafting_table'1z atep 1: craft stone_pickee from cobblestone and stick, crafting_table
Current Inventory: null
Explanation: Because crafting stone ccone_pickaxe needs to have 3 cobblestone and stick inventory, but my inventory does not have sobblestone and stick. So need mine cobblestone and craft stick first.
22

---

Iff
Failed Action: craft (l'stick' :4), ('planka' 12), mull); step 3: craft 4 stick from planks first
Current Inventory: rull
Explanation: Because crafting stick needs to have planks in inventory, but inventory does not have planks. So need to craft planks first.
##
Listing I: Prompt for Explainer in Minecraft tasks
C.4 Other modules
Goal Parser We need to map the plan expressed in free-form language the pre-defined controller skills set. We use the LLM as an automatic parser to parse the language plan first. For the goals not following pre-defined code expression, similarity with pre-trained Sentence- Bert model [37] and select the most similar skill as the corre- sponding goal. All executable goals are listed in Appendix C. The LLM-based parser is general and can be transferred to other domains easily by modifying the prompt. The prompt for Minecraft parser is listed in Listing
calculate its semantic distance to the skills by cosine
Extract the action name, action type, goal object, tool action input text.
the
input: mine (f' log' :3), null); step 1: mine 3 log without tool
name: mine_lag
action: mine
object: I l'log':3] :3:
tool: null
rank: 4tf
input: craft (1 planks' :12), ("log' :3), null); step 2: craft 12 planks from log
name: craft_ craft_ppaakk planks
action: craft
object: l'planks'
materials: ('log' :3)
tool: null
rank:
##f
Listing 2: Prompt for Goal Parser in Minecraft tasks
Success Detector The successful execution of plan is contingent upon the agent's perception of the current goal's completion status, which is assessed by the success detector. Minecraft, agents possess an inventory that contains all pertinent information regarding the agent's 's current state. Thus, the Success Detector can implemented by monitoring changes object information within the item inventory. In other scenarios, we can query the LLM to ascertain whether the agent has accomplished general goal by describing the agent's current state. Alternatively, in certain environments [40], the execution of goal is linked to the agent's current reward, signifying that these rewards can serve as automatic success detectors.
Prompt The generalization of the LLM to different tasks relies on well-designed prompts and related demonstrations [28]. Given an instruction command (e.g., ObtainDi amond) task T, prompt generator (ProG) will translate T into prompt text. We also added two DEP examples in the prompt as demonstrations to make the LLM output familiar the -f-houh thinking and structural output. We also design a chain-of siiinn-ffthhous thought code-comments-lyeee planning prompt to better demonstrate the capabilities of LLM. All messages are modified to suitable prompts through the prompt-generator before being input to LLM, including task 7" and description d, The full prompt sentences and interaction logs are listed Appendix H.
D Comparison with other LLM-based Planners
The architectures of the different M-LM-based planners are illustrated in Figure 7. Where (b) describes the information in the environment into LLM via scene descriptor and success detector, and directly
23

---

Soene Description
Instrcsion
Instruction
Instruction
PLANNER LLM
EXPLAINETY LLM
PLANNER LLM
plan
idssoniation
Scare Description
Quaue
Suaccaeee Detector
SELECTOR VLM
DESCRIFTCR VLM
Ignal
N gost gft)
gil+1l
CONTROLL ER
ski
CONTROLL ER
CONTROLL ER
action
ENV
ENV
ENV
octiom
(a) Ours
(b) Short-horizon action planning
(c) Long-Horizon one-shot planning
(d) Soong-
planning & re-planning
Figure 7: Comparison of LLM- planner architecture. (a), (b), (c), (d) represents planner of ours, Inner
Monologue [17], Zero-Shot Planner [16] and Zero-Shot Planner with re-planning process, respectively.
plans the goal/action, (c) is Zero-Shot planner [16], which generates the step-by-step goal sequences as plan and ignores the environment state and execution feedback, (d) is the Zero-Shot planner augmented with textual feedback and re-planning process. DEPS further rethink and explain the feedback of previous plans explicitly with the descriptor and explainer. The LLM-based planner will re-plan the task according the explanation, demonstrated Figure 7(a). In addition, the goal Selector further improves the executability of the LLM plan.
E Discussion on ObtainDiamond Task
nine log
craft planks
craft crafting table
craft stick ccaft wooden pickare
nine stone
Peraft stone pickaxe
wine fron
craft furnace
sselt fron Ingot Ficraft iron pickaxe
nine diasond
Figure 8: The milestone goals of the ObtainDiaman amond task.
task formidable task within the open-ended Minecraft environment. Given the necessity to explore infinitely expansive world, efficient plan can prove advantageous, as shown Figure The task is allotted maximum 12,000 steps to interact with the environment, which is comparable to that of human performance [14]. Rather than manually devising explicit hicrarchical rewards, we opt to utilize DEPS for generating a hierarchical plan, which is then transferred the downstream controller to progressively achieve each goal. When equipped with an Oracle Controller, DEPS yields success rate of 60% for ObtainDiamond, In our experimentation, we employed Behavior Cloning train Controller agent [6]. DEPS+BC Controller achieved a success rate of 0.6% in randomly generated Minecraft worlds. The primary bottleneck impeding overall agent success rate lies within the goal-conditioned Controller, not the plans generated by DEPS. Thus, it worth exploring the development of a data-efficient Controller capable of accepting Language goals.
As outlined in Section 4.4, Obt
Another rationale for using DEPS that, akin to reality, materials in Minecraft possess quantity con- straints, and durability for tools. In aaaannDiaoonn task, an iron pickaxe is typically insufficient

---

to support the agent, given the rarity of diamonds within the environment (which are predominantly found between depths of 2- 16 layers and appear only 0.0846% the time). The robust re-planning capabilities of DEPS can facilitate the generation of a feasible plan (initiating with crafting an iron-pickaxe) based on the agent's current state.
Additionally, we report the milestones, which demonstrate the decreasing success rate of subsequent tasks in Figure 9 attributable the task's inherent complexity and Controller constraints.
100
96.7
89.0
89.0
89.0
96.5
63.1
80
Success Rate
40
20
8.8
61
0
Milestone Items
Figure 9: Success rate of milestone items for mining diamond.
F Success Rates of ALL Tasks in Minecraft
We report the complete and detailed success rate table of all tasks for different methods in Table 21, including Zero-shot Planner [16], ProgPrompt [42], Chain-of-Thougeee [45], Inner Monologue [17], Code Policies [20], and proposed methods (i.e., DEP w/o Selector, and DEPS).
tasks are executed for least times across different world seeds, given the same initial conditions. The birth positions of the world are random according to the seed. The average success rates are listed in Table Our approach state-of-the-a on almost all tasks, especially on difficult tasks that require more skills.
25

---

Table 21: Success rate comparison of various methods MineDojo [10] environments.
Meta-Taa
Task Object
GPT [16]
PP|42]
Co5
IM [M7]
CaP [20]
DEP
DEPS
planks
56.7
56.7
81.3
83.3
83.3
83.3
81.3
stick
0.0
83.3
83.3
83.3
83.3
86.7
wooden_stal
26.7
83.3
83.3
wooden wooden_button button
23.3
50.0
73.3
73.3
73.3
73.3
96.7
wooden, pressure_plafe
80.0
80.0
53.3
80.0
80.0
80.0
86.7
chest
76.7
Basic MT1
staistt
20.0
40.0
36.7
16.7
36.7
56.7
60.0
sign
23.3
0.0
43.3
0.0
43.3
63.3
86.7
lence
20.0
20.0
20.0
43.3
63.3
80.0
fence gatt
63.3
63.3
63.3
63.3
93.3
73.3
boat trapdoor
0.0
0.0
0.0
26.7
56.7
83.3
73.3
26.7
26.7
26.7
56.7
56.7
83.3
76.7
bowl wooden_ door
0.0
23.3
46.7
70.0
23.3
23.3
46.7
46.7
46.7
66.7
80.0
crafting table
70.0
23.3
70.0
70.0
70.0
70.0
90.0
wooden, wooden__pckaad pickaxe
80.0
80.0
80.0
80.0
80.0
80.0
80.0
wooden_
46.7
70.0
70.0
70.0
70.0
wooden_hre hrne
86.7
56.7
86.7
30.0
86.7
86.7
86.7
wooden sword
83.3
83.3
83.3
83.3
83.3
83.3
86.7
Toop(Si
wooden furnace
76.7
76.7
MT2
20.0
20.0
0.0
40.0
40.0
60.0
66.7
stone stone_piccee pickaxe stone hoe stone shovel
16.7
16.7
36.7
36.7
53.3
53.3
73.3
46.7
20.0
20.0
36.7
36.7
56.7
56.7
66.7
56.7
$6.7
40.0
36,7
36.7
56.7
66.7
stone_sword
16.7
53.3
53.3
36.7
53.3
80.0
pointing
33.3
0.0
16.7
16.7
53.3
86.7
Hunt and Food
carpet
0.0
33.3
0.0
0.0
13.3
33.3
43.3
irem_frame
23.3
0.0
23.3
73.3
MT3
cooked porkchop
0.0
0.0
40.0
50.0
cooked, beef
0.0
0.0
0.0
D.0
0.0
$3.3
63.3
cooked_ motton
0.0
0.0
0.0
43.3
66.7
stone slub
36.7
56.7
16.7
50.0
16.7
33.3
50.0
73.3
Dig down MT4
cobblestone_co wall
16.7
16.7
16.7
16.7
43.3
43.3
63.3
lever
0.0
46.7
46.7
70.0
coal
0.0
16.7
6.7
16.7
20.0
torch
0.0
6.7
0.0
6.7
0.0
23.3
13.3
leather boots
0.0
13.3
0.0
13.3
13.3
36.7
60.0
chotspr_ehcsplltee
0.0
16.7
0.0
6.7
23.3
leather helmet
16.7
6.7
6.7
26.7
70.0
leather, leagber_leggings
0.0
0.0
0.0
20,0
0.0
30.0
56.7
Equipment MTS
chossthht
0.0
0.0
0.0
0.0
leggings
iron_ helmet
0.0
0.0
0.0
0.0
0.0
0.0
3.3
iron, hoots
0.0
0.0
0.0
0.0
0.0
6.7
20.0
shicld
bucket
0.0
0.0
3.3
10.0
shears
0.0
0.0
0.0
0.0
0.0
10.0
30.0
Toal Complex
iron, pickaxe
0.0
0.0
6.7
0.0
6.7
10.0
MT6
axe
iron hoc
0.0
0.0
0.0
0.0
0.0
3,3
13.3
iron, shovel
0.0
0.0
0.0
0.0
6.7
13.3
fron_swod,
0.0
iron_nugaet
0.0
6.7
0.0
6.7
6.7
23.3
40.0
minecart
0.0
0.0
0.0
0.0
3,3
10.0
hopper
peoper_minnecatt
0.0
0.9
0.0
furnace_ minecart
0.0
0.0
0.0
0.0
0.0
3.3
3.3
Iron Stage
chest_minccart minccart
0.0
0.0
3.3
MT7
iron, doue
0.0
0.0
0.0
3.3
iron, trapdoce tripwire, book
0.0
0.0
0.0
0.0
0.0
3.3
16.7
0.0
0.0
6.7
0.0
16.7
30.0
beavy_ beavy_weighted weighted wwwpsprreeppseeee rail
0.0
0.0
0.0
0.0
0.0
0.0
6.7
cauldron
0.0
0.0
0.0
0.0
0.0
Challenge
diamond
0.0
0.0
0.0
0.0
0.0
0.5
0.6
26

---

G Prompt for Different Tasks and Different Methods
G.1 Prompt for Minecraft Tasks
G.1.1 DEPS
User: You are a helper agent in Minecraft. You reed to generate the sequences of goals for a certain task in Kinecraft. Planner: OK.
User:
How obtain 1 rooden_button Minecraft stap-by-step? pooden_button is fallows:
Planner: The code for obtaining
def watooooeenettttoooeoeoaoooooaeee
mine (1 ([' :1,, rell); step 1: mine leq without tool
craft (1' (I'planks' :4), {² ('log':1) :1), mull); step 2: craft planks from 1 log :1,, l'planks' :1), mull); t step 3: craft wooden_button
craft ((ww:l(wwww_tttteee from. planks
returr: "wooden_batton
Descriptor: succeed in step 1, User: Good. finish all steps and obtain atone_pickaxe successfully. Planner: OK.
User: How to obtain 1 stone_sword in Minecraft step-by-step?
Planner: The code for obtaining stcne_swodd is as follows:
def sotii____stooe_ooo(ibbeeee
= (1):
mine (I'log'' :31, mull); step 1: mine log without tool
planks 3 log craft (raft(1"stee (['stick's :4), ('planks':21 nulli; step 3: craft stick from planks mine ( cobblestooee :2), mull); step mine 2 cobblestone without tool craft (""stone_sword":lll :1], ('cobllestooe' :2, sstick'll "raftiin_ttabee'>
craft (['plank''
('1og' :31, null); step
5: craft 1 stone_sword from 2 cobblestone and 1 stick, crafting_table
returr 'stone_sword'
Descriptor: I succeed on step 1, 2, 3, but fail on step 4 "rine cobbl cobblestone estone ':2), null);", My inventory now has 10 planks, stick. wooden pickexe.
Explainer: Because mining coablestene needs to
User: Please fix the above errors
replan the task "obtain 1 stone_sword", RePlanner: The code for crafting stome_sword
belloxs:
def satia___ssoone_sorddiaaee
('planks":10) 'stick':
craft ("axoooclcode :1;, (' ('planks' :3, 'stick' 'crafting_table"); # step
1: craft wooden_pickexe from 3 planks and 2 stick, on tabble mine (1' cobblestonn' :2), wooden_pickase
step 2: mine cobblestone with
xooden_pickaxe craft (['stone_sword :1), (' c'obllleston'' :2, setick':l;; "crafting_table" step 3: craft 1 stone_swoze from 2 cobblestone and 1 stick, on crafting_table
'stone_skord'
Descriptor: I fail step 1 *craft htt.:oooopcccaaaeeeee :1l, l'planks' :3, 'stick' :2), scrafting_table">: My inventory now has 10 planks, 4 stick.
Explainer: Because the action needs to use the tool crafting_table, but do not have it.
User: Please fix the above errors and replan the task "obtain 1 stone_sword",
RePlanner: The code for crafting stone_sword
as bellows:
def
['planks' :10, 'stick' :4)); crafting_table from planks
craft (""rraftinng_ttatee :1;, ('planks':4], null); step 1: craft
craft ("ww"""deeee_ppccee"""" :1;, ('planks':3, "stick' :2), (rrattingttable)) f step
2: craft 1 wooden_pickaxe from 3 planks and 2 stick, on crafting_table
mine (i' (l'cobblesstoee" :2), 'xooden_pickaxx""; step 3: nine
astoneston with
pooden_pickaxe
craft ['sston__soodd" :1), ('cobblestone' :2, 'stick' :2), 'orafting
step 4: craft 1 stone_sword from 2 cobblestane and 1 stick, on crafting
returr: 'stone_sword'
Descriptor: succeed in step 1, 3, Good. finish all steps and obtain stone_pickaxe successfully.
27

---

Planner: OK.
Listing Prompt DEPS Minecraft tasks
G.1.2
Baselines
User: You are a helper agent in Minecraft. You need to generate the sequences of goals for a certain task in Kinecraft.
Planner: OK.
User: How to obtain 1 wooden_button Minecraft step-by-step? follows:
Planner: plan step mine log
step 2: craft planks
step 3: craft 1 button
ttt
User: How to obtain stone_sword in Minecraft step-by-step?
Planner: The plan is as follows:
step 1: mine log
step
planks
step craft stick
step craft crafting_table
step craft
maoden_pickaxe
step 6: mine 2 cobblestone
step 7: craft 1 stone_sword
##
Listing 4: Prompt for Zero-shot Planner [16] in Minecraft tasks.
User: You are a helper agent in Minecraft. You need to generate the sequences of goals for a certain task in Kinecraft. Planner: OK.
User: How to obtain 1 wooden obtaining wacden_button is follows:
Minecraft step-by-step?
Planner:
from actions import mine Logn craft_planks, (iaventory []]:
setet__wodde__butee
def
mine_log (num = 11; # step 1: mine log
craft_ppannk (num 4); step 2: craft 4 planks
craft_wooden_ (num = 11; # step 3: craft 1 wooden_button
reture "wooden_button"
User: obtain 1 stone_ sword Minecraft step-by-step? Planner: The code for obtaining stene_sword is follows:
from actions import mine_log, craft planks, craft_sticd craft_craattinggtable scaft_wocden_pichee
cone_coobbestone, staft_stone_skork
def dttaiin_tooe__sant iinventtorr - :)):
mine_log (num 3); step mine log
craft_planks (num 12); # step craft planks craft_stick
4); step craft stick
craft_craftinggtabbe (num 1); step craft crafting_table
craft_wooden_pie pickaxe (num }); step 5: craft wooden_pickase
tne_cobbleston (num 2); step 6: mine 2 cobblestone
astone
craft_stone_ wtw.sttoeewwwrdauae - 1); # step 7: craft 1 stone_sword
returr "stone_skord"
Listing 5: Prompt for ProgPrompt [42] in Minecrafi tasks.
User:
You are a helper agent in Minecraft. You need to generate the sequences of goals for a certain task in Mi Minerafft.
Planner: Ox.
You are helper agent in Minecraft. need generate sequences goals certain task in Kinecraft.
28

---

Planner: OK.
User: How to obtain 1 gooden_button in Minecraft step-by-step?
Planner: The plan is as folloks:
step 1: mine 1 loq without tool
Thinking: Inventory will add 1 log, as inventery 1+log step 2: craft 4 planks from 1og Thinking: Inventory will step craft wooden_button from planks
planks
delete log, as inventory 4xplanks
Thinking: Inventory will add 1 wooden_button and delete 1 planks, inventory 3 planks Lutto
#tf
User: How to obtain stone_sword in Minecraft step-by-step?
Planner:
The plan is as follows:
step 1: mine log without tool
Thinking: Inventory will
3 log, inventory 3+log,
planks Thinkinq: Inventory will add 12 planks and delete log, inventory 12+planks step 3: craft 4 stick from 2 planks
step
log
Thinking: Inventory will add 4 stick and delete 2 planks, inventory = 10+planks
+ 4*stick
step 4: craft 1 crafting_table from 4 planks Thinking: Inventory will add 1 crafting_table and delete planks, as inventory = 6-planks 4+stick + Lecrafting_tade table
step craft 1 wooden_pickaxe from 3 planks and stick, on crafting_tabll table Thinking: Inventory will
wooden_pickaxe deleta planks and delete stick, as inventory 3+planks 2+stick Lecrafting_table L-wooden L-xooden_pickaxe step mine cobblestone with wooden_pickax.
Thinking: Inventory will add 2 cobblestone, as inventory 3xplanks 2*stick 1 crafting_table + 1=xooden_pickaxe
+ 2+cobblestone
step 7: craft 1 stone_swordd from 2 cobblestone and 1 stick, on crafting_tabl table Thinking: Inventory will add 1 stone_sword, delete 2 cobblestene, and delete 1
stick, as inventory 3xplanks 1+stick + 1xcrafting inatffe T 1+stone_eword
#
Listing 6: Prompt for Chain-of-Thought [45] in Minecraft tasks
User:
You are a helper agent in Minecraft. You need to generate the sequences goals certain task Minecraft.
Planner:
User:
Obtain 1 botten in Minecraft step-by-step.
Scene: My Inventory has nothing.
Planner: mine 1 log
Robot Thought: mining leg needs na tool
Successful Action: True
Scene: My inventory has log.
Planner: craft planks
Robot Thought: erafting planks needs log
Successful Action: True
Scene: My inventory has planks
Planner: craft xooden_button
Robot Thought): crafting wooden_buttonn needs planks Successful Action: True
User: Obtain swor__swerd in Minecraft step-by-step.
Scene: My inventory has nothing,
Planner: mine 3 log
Robot Thought: mining log needs no tool
Successful Action: True
Scene: My inventory has log.
Planner: craft 12 planks
Robot Thought: erafting 12 planks needs 3 log
29

---

Successful Action: True
Scene: My inventory has 12 planks
Planner: craft stick
Robot Thought: crafting stick needs 2 planks
Successful Action: True
Scene: My inventory
planks, stick
Planner: craft kooden_pickaxe Robot Thought: crafting 1 wooden_pickaxe needs 3 planks and 2 stick on crafting_table
Successful Action: False
Scene: My inventory has 10 planks, 4 stick
Planner: craft crafting_table
Robot Thought: crafting crafting_table needs 4 planks Successful Action: True
planks, 4 stick, crafting_table Planner: craft gooden_pickaxe Robot Thought: crafting 1 picken_ needs 3 planks and 2 stick, c-afting_table
Scene: My inventory
Successful Action: True
Scene: My inventory has planks, stick, grafting_table, wooden_pickaxe
Planner: mine 2 cobblestone
Robot Thought: mining cobblestone needs wooden_pickaxe as tool
Successful Action:
Scene: My Lnventory has planks, stick, crabting_table, pocken, 2 cobblestone
Planner: craft stone_sword
Robot Thought: crafting 1 stone_sword needs 2 cobblestone and 1 atick, on crafting_table Successful Action: True
Listing 7: Prompt for Inner Monologue [17] in Minecraft tasks
User: You are a helper agent in Minecraft. You need generate sequences goals certain task in Kinecraft.
Planner: OK.
obtain weoden_butee User: The code for obtaining wocden_button is follows:
Minecraft step-by-step?
Planner:
def craft_stone, sttee_sttteessedd_nnnee sxord
- +)):
I step 1: mine 1 log without tccl
while
mine Luventozy.has((1100 :1)):
*log", tool null);
# step 2: craft planks from 1 log
while not inventory. haw.ooor..aa(laooo (I''lanks :41):
inventory Ceaentorr.haa((13333 :1)) "I do have enough log for crafting planks"
crb]t "planka", tool null);
# step 3: craft woodea_button from 1 planks
while inventory haw ("kodd""ceebbbtttttoo :1:):
assert inventory
haanttory.asslee has (L'plankks :1]), "I do not have enough planks for crafting wooder_buttoa"
crait "wooden_button", tool null);
"wooden_button"
iff
User: obtain stone_sword Minecraft step-by-step?
Planner:
The code for obtaining stane_sword sword is follows:
def craft_stone (xtone_sxxordiaiee (xvventnnyytory - ()):
It step 1: mine 3 log without tocl
while
mine
"log", tool null);
# step 2: craft 12 planks from 3 log
while not haatttoryyaaa(lpllll
:12:1:
assert inventory has :1]), "I do have enough log for crafting planks"
(obj "planks", tool = null);
# step 3: craft stick from planks
while inventory has (l'sttitcck" :41)
30

---

assert inventory haa plankkss :2)), "I do not have enough planks for crafting stick"
(rbft "stick", tool null);
# step craft crafting_table from planks
while not inventory hha wetaaaiinggtaalell.eeeeeee
assert inventory (tt......aaallaaaeeee :4H), "I do not have enough planks for crafting crafting_table"
craft "crafting_table"" tool muil)
# step 5: craft wooden_pickewt from planks and 2 stick, crafting_table while not inventory.has(( (( :1)):
assert inventory.han (f :3)),
not enough planks
crafting wooder_pi.kaxe"
assert (aa.nttor..as((tttt..... :2)), "I de not have enough stick for crafting wacden_pickase
assert inventory. www.aaaaaaa (f [able':l)) do have the tool (aaennttor.aaa((aaaee
crafting_table for srafting wasden_pickexe"
(rbj = "crafting_table", tool "crafting_table""
# step 6: mine 2 cobblestone with tool gooder__iickaee
while inventory cfobbllsnntt :2]):
assert inventory. (td(:/ooode___ccceee===
"I do not have the tool
wooden_ (obe "cobblestone", tool piooaen_
for mining cobblestone"
# step craft stone_swore while not inventory
2 cobblestone and stick crafting
:11):
assert inventory has cobblesttt :21), "I do not have enough cobblestone for crafting stone_sworee
inventory haa :11), do have enough stick for crafting stone_sword"
assert inventory.hag ((' wwwaarratttaaabilaaaeee :1)), "I do not have the tool crafting_table
crafting waeden "roden_pickexe" craft "stone_sword", tcol "eraftire_table") reture "stone_sword" sword"
tfft
Listing 8: Prompt for Code as Policies [20] in Minecraft tasks
G.2 Prompt for ALFWorld Tasks
G.2.1 DEPS
Human:
Welcome TextWorld, ALFRED!
Descriptor: You are in the middle of room. Looking quickly around you, see cabinet 1, a diningtable 1 and a fridge 1. put
in cabinet. Please generate the sequences of sub-goals to complete the task.
task
planner: The task is to put cool tomato in cabinet. The plan is below:
#1 pick a tomato at fridge
cool tomato
fridge
#3 Put tomato in cabinet
cabinet
Descriptor: I fail on step 1 pick a tomato at fridge 1. I'm not carrying a tomato
and there is no tomato, so the goal can't be accomplished.
Explainer: It's probably because there isn't tomato fridge The task put cool tomato in cabinet. so should pick tomato somewbere else. I'm most likely to pick a tomato at diningtable 1.
Planner: So the new plan below:
pick tomato
diningtable
#2 put tomato in cabinet at cabinet
Descriptor: I sacceed on step 1.
Descriptor:
failed on step 2 put tomato in cabinet at cabinet 1. put a tomato 2 may be cool tomato, the goal
tonato in cabinet,
accomplished shed.
Explainer: It's probably because the tomato isn't cool enough. The task is to put tomato. I'm most likely cool
cool temato cabinet.
should
tomato at fridge 1.
Planner: So the new plan is below:
#2 ccol a tomato at fridge
#3 put tomato
cabinet
cabinet
Descriptor:
succeed on step 2,
Descriptor:
I sacceed on step 3, I finish the task!!!
Human:-= Welcome TextWorld, ALFRED!
Descriptor: You
in the middle of desk 1, and a shelf 1.
Looking quickly around you,
Human: Your task is to: look at mug under the desklamp, Please generate the sequences of sub-goals complete the task.
31

---

Planner:
The task is look mug under the desklamp. The plan is below: #1 pick up a mug at shelf 1
use the desklamp at desk 1 Descriptor: succeed on step
Descriptor:
succeed on step 2, finish the task!!!
Listing 9: Prompt for DEP in ALFWorld tasks
Human: You are in the middle of room. You are not carrying anything. Your goal
to: pick plate at diningtable
The admissible actions of the current sicuation are
1. to cabinet
2. go to diningtable
go fridge
4. inventory
5. look
AI: The goal is to pick plate at diningtable I'm not at diningtable
should go to diningtable first. So the answer 2.
Human: You arrive at loc 17. on the diningrable 1, see plate 2, and tomato 2.
The acmissible actions the current situation are
1. examine diningtable
2. to cabinet
3. go to fridge
inventory
5. look
6. take plate 2 from diningtable 1
7. take tomato from diningtable:
diningtable and there is plate 2. So I should take plate 2 from diningtable 1. So the answer is 6. Human: You pick up the plate 2 from the diningtable 1. The admissible actions of the currest sicuation are examine diningtable
The goal is pick plate diningtable
2. to cabinet
3. go to fridge
4. inventory
look
6. put plate 2 in/or diningtable 1
AI: The goal is to pick plate at diningtable 1. According to Human, pick up the plate 2
diningtable 1. So the goal accomplished.
Human:
You are facing fridge 1. You are carrying plate 2. Your goal is to: cool plate at fridge 1,
admissible actions the current situation are
1. examire fridge
2. go to cabinet
3. go to diningtable
4. inventory
5. look
6. cool plate 2 with fridge
AI: The goal is to cool plate at fridge 1, z'm at fridge 1. So should cool plate 2 with fridge 1. So the answer 6.
Human: You cool the plate using the fridge 1.
The admissible actions of the current sicuation are
1. examine fridge
2. go cabinet
3. to diningtable
4. inventory
5. look
AI: The goal is cool plate fridge According Human, cool the plate
2 using the fridge 1. So the goal is accompliss ished.
Human:
facing diningtable to: pick plate diningtable 1.
are carrying anything, geal
The admissible actions of the current sicuation are
1. examine diningtable
2. go cabinet
3. go to fridge
4. inventory
5. look
5. take tomato from diningtable
AI: The goal is to pick plate at diningtable 1. I'm at diningtable 1, but there
is no action to pick a plate. So the answer is (none) So the goal can't t be accomplished, probably because there no plate at diningtable 1.
32

---

Human: You facing fridge 1. Yeu carrying anything. Your goal is to: cool plate at fridge 1.
The admissible actions of the current situation are
examine fridge
2. to cabinet
3. go to diningtable
4. inventory
look
AI: The goal is to cool plate at fridge 1, I'm at fridge 1, but there is no action to cool a plate. So the answer is 0 (none). So the goal can't be accompl shed, probably because T'm not carrying plate.
Human:
You are facing fridge 1. You are carzying a plate. Your goal is to: pick tonato at fridge 1.
The admissible actions the current sicuation
1. examine fridge
2. to cabinet
3. go to diningtable
inventory
5. look
AI: The goal is to pick tomato at fridge 1. I'm at fridge 1, but there is no
action to pick a temato. So the answer (none). So the goal can't t be while carrying plate.
accomplished, probably because can't pick
Listing 10: Prompt for the controller DEP in ALFWorld tasks
G.2.2 Baselines
Human: Welcome TextMorld, ALFRED!
Your task is to: cool
plate and put it in cabinet.
Scene: You are in the middle of a room. Looking quickly around you, you see cabinet 1, diningtable 1 and fridge 1. admissible actions of the currest sicuation are
1. to cabinet
2. go to diningtable
3. go to fridge
inventory
look
Robot Thought:
The task is to cool some plate and put it in cabinet. I'm carrying nothing. So
to find plate first. There are cabinet, diningtable and fridge, but no plate. I'm likely find plate diningtable. should go to diningtable, 1. Sa the answer is 2. Robot Action: go to diningtable
Scene: You arrive lac 17. On the diningtable 1, you plate 2, and tomato 2.
The admissible act ions of the current sicuation are
examine diningtable
2. to cabinet
3. go to fridge
4. inventory
5. look
6. take plate 2 from diningtable 1
7. take tomato from diningtable
Robot Thought: The task is to cool some plate and put in cabinet. I'm carrying find plate first. There plate 2. should take plate from diningtable 1. So the answer is 6.
aothing.
Robot Action: zake plate 2 from diningtable
the plate 2 from diningtable 1. The admissible actions of the current sicuation are
Scene:
1. examire diningtable
2. examine place
3. to cabinet
4. go to fridge
5. inventory
6. look
7. put plate 2 in/or diningtable
Robot Thought: The task is to cool some plate and put it in cabinet. I'm carrying plate 2. So need to cool plate first. There are cabinet, diningtable, likely cool plate fridge. should go fridge 1. So the anawer is 4.
and fridge. I'm
Robot Action: go to fridge
Scene: arrive loc 41. The fridge 1 is closed.
33

---

The admissible actions of the currest situation are
1. cool plate 2 with fridge
2. examine fridge 1
examine plate
to cabinet
5. go to diningtable
6. inventory
7. lock
8. open fridge
Robot Thought: The task is to cool some plate and put it in cabinet. I'm carrying to seol plate first. There fridge So should 1.
plate
cool plate with fridge 1. So the
Robot Action:
cool plate 2 with fridge
cool plate using fridge The admissible actions of the current sicuation are
Scene:
1. cool plate 2 with fridge
2. examine fridge 1
examire plate
4. to cabinet
5. go to diningtable
6. inventory
7. lock
open fridge
Robot Thought: The task is to cool aome plate and put it in cabinet. I'm carrying cool plate 2. So I need find a cabinet first. There cabinet 1. So should go cabinet 1. the
is
Robot Action: to cabinet
Scene: arrive
cabinet closed.
The ssible actions of the current situation
1. examine cabinet
2. examire place
go diningtable
4. go to fridge 1
5. inventory
6. look
open cabine:
Robot Thought: The task is cool some plate and put it in cabinet. I'm carrying cool plate 2. So I need to find a cabinet first. There is a cabinet 1. The cabinet is closed. So should open cabinet 1. So the answer 7. open cabinet
Robot Action:
Scene: You open the cabinet 1. The cabinet 1 is open. In it, you see nothing. The acmissible actions the current situation are close cabinet
1.
2. examire cabinet
3. examire place
4. go diningtable
5. to fridge
6. inventory
7. look
8. put plate in/en. cabinet
Robot Thought: The task is to cool plate and put it in cabinet. I'm carrying cool plate 2. So I need to find a cabinet first. There is cabinet 1. So should put plate 2 in/on cabinet 1. so the answer is 8.
Robot Action: put plate in/on cabinet 1
Scene: You put the plate 2 in/on the cabinet 1. You won!!!
Listing 11: Prompt for Inner Monologue [17] in ALFWorld tasks
G.3 Prompt for Tabletop Manipulation Tasks
G.3.1 DEPS
system:
The template of the lang geal is follow: put the <eelor> blocks in color> bowl, For example, "put the green blocks in blue bowl" is a valid lang goal. Do not provide any additional explanations or instructions beyond writing lang goals.
user: There are 4 bowls of blue, green, pink, yellow on the table. There are 3 blocks of red, yellow, blue on the table. How match the blocks and the boxls?
34

---

assistant:
1. put the yellow blocks yellow bowl
2. put blue blocks blue bowl
Listing 12: Prompt for DEP CLIPort tasks
G.3.2 Basclines
system:
The template the Language goal is as follows: put the <color> blocks <color> bowl. lang goal. Do not provide any additional explanations writing lang goals and determing the game is DONE. Please write aingle lang goal in your response!
example, "put green blecks in blue bowl" valid
instructions beyond
user:
There are 4 bowls of blue, green, pink, yellow the table. There are 3 blocks of red, yellow, blue on the table. What should I do as the first step match the blocks with the
the literally same color? Note not be matched and you determine the game DONE anywhere. assistant: put the yellow blocks in yellow bowl
all the blocks
user: OK. What should do next?
assistant: the blue blocks in blue bowl
user: OK. finish the task!
Listing 13: Prompt for Inner Monologue [17] in Tabletop Manipulation tasks
H Full Dialogue
User: You helper agent in Minecraft. You need generate sequences goals for certain task in Kinecraft.
Planner: OK.
User: Now to obtain 1 diamond in Kinecraft step-by-step!
Planner: The code for taak ostaining diamond is as bellows: def obtain_l_diamone (inventover
(1):
mine (I'log' :3), rell); step 1: mine log without tool
craft (I'planks':12), ('log' :31, null); step
planks 3 log craft (['stick' :4), ('planks' :2), nulll; step 3: craft stick from planks ('planks':4) :4), mull); step 4; craft
craft (wwrraaattiinnettalleeeeaeo craftinggtable from planks
craft ("acccddeee_p :1;, l'planks' :3, 'stick' :2), 'crafting_table"); # step
5: craft 1 wooden_picken from 3 planks and 2 stick, on crafting_table mine (1' cobblestone' :2), 'wooden pickaxe'); ; step 6: mine cobblestone with wrodeen_pickaee))
wooden_pickaxe
craft ((" pessone_piccxx"" :1,, (' l'cobblestooee :3, "stickr:2; 'crafting_table") step 7: craft 1 stone_pickake from cobblestone and 2 stick, on crafting_tabi table
mine (l'diamond' ;1), 'stone_pickaxe"); step 8: mine diamond with stone_pickaxe return 'diamond'
Descriptor: I succeed on step 1, 2, 3, 4, 5, 6, but fail on step 7 "craft (1' sicone_pickaxe'. ('cobblestone'' :3, 'stick' :2), saattiingttabll My inventory now has planks, 2 stick, 1 crafting_table, wooden_pickaxe cobblestene, dirt.
Explainer: Because crafting stone needs cobblestone, but only have cabblestone inventory.
and re-plan task *obtain diamond". RePlanner: The code for task obtaining diamond is follows:
User: Please fix above
def (iaii__i__dmonnnniieeeeee
('planks' 17, 'stick' 'crafting wooden_rickaxe" :1, aetonelsstee''11 :2, 'dirt':5)):
mine cobblestbbe :1), 'wooden (ondeen_iccaae]) ); step 1: mine cobblestone with xooden_pickaxe craft (('sttoe_ppccke ckexe' :1,, I'cobblestoone :3, 'stickr:2 :2;, 'crafting_table" step 2: craft 1 stone_pickare from cobblestone and 2 stick, on crafting_table
mine (1 aioadd 'stone_pickaxe") step 3: diamend with stone_pickaxe
35

---

reture 'diamond'
Descriptor: I succeed on step 1, 2 in new plan, but fail on step 3 "mine (f'
diamond' :1), stton__picaxeees );" My inventory now has 7 planks, crafting_table, wcoden_pickaxe, 1 cobblestone, dirt, 1 stone_pickaxe.
Explainer: Because mining diamond needs to use the tool troe_pickaxe, but do not have it.
User: Please fix above errors and re-plan
task *obtain diamond".
RePlanner: The code for obtaining diamond is follows: (inventory ('planks' 'erafting_table"=l 'xooden_pickaks
def
';i, 'cobblestone' :1, 'dirt':5, "stone_pickaxe" :1]):
( [['iroo_ore':l); 'stone_pickaxe") # step 1: mine 1 iren_ore with
stone_ pickaxe
smelt (pliii((ioo ;1), ("l),((i ('iron_ore' ;1), 'furnace') 1 step 2: craft 1
pron_ingot from 1 iron_pre, or furnace
craft (1" wtc.kreeecckeee :11, !' ircon_iggot/ 'stick' :2), corafting_tables step 3: craft 1 iron_pickaxe from 3 iron_inget and 2 stick, on crafting_table
mine (I'di :1ond" 'iron_pickaxe"); step mine diamend with iren_pickaxe return 'diamond'
Descriptor: I success on step in new plan, but I fail on step 2 "smelt(("
Izon ison_ingot':l] :1), ('iron_ore oroo_ooe':;;; :14, "furnace');"; My inventory now has 7 planks, crafting_table, 1 sooden_pickaxe, 6 cobblestone, 1 iron_ore, 20 dirt, 64 stone.
Explainer: Because smelting iron ingot needs to use the tool furnace, but do not have
User: Please fix
errors replan the task "obtain diamond".
RePlanner: The code for obtaining diamond is follows: def mine_ i_m___ddammond (inventory - l'planks' 'crafting_table"i.. 'cobblestone' :6, 'iron_ore's :1, 'dirt':20, 'stone' :64]1:
wooden_pickaxee :1,
craft (1' furnace' :1,, ('cobblestene' :8), mail); step 1: craft 1 furnace from 8 cobbleatone
craft i ron hom_ingot :1), ('iron_are':l) "uurrace)) ); step 2: craft
iron iron_ingot from 1 oren on furnace
craft (1' iroa__icckaeee :1), ['iren_ingot' 'stick' 12), 'erafting_table") iron_inget 2 stick, on
step craft tron_pickaxe crafting table mine (I' [''diamond' :1), 'iron_pickaxe") step mine diamond with iron_pickaxe return 'diamond'
Descriptor: I fall on step 1 "craft furnace' :1), (' cobblestene' :8), null);", My inventory now has planks, 1 crafting_table, 1 wooden_pickaxe, cobblestone, 1 Liron_ere, dirt, stone.
Explainer: Because crafting furnace needs 8 cobblestone, but only have 6 cobblestone.
User:
fix
errors re-plan
*obtain diamond".
RePlanner: The code for obtaining diamond is follows:
daf obtain_l_diamme (aaaond(inveetee (inventory ('planks' "crafting_table"" 'xooden_pickaxe
":i, 'cobblestone's :6, 'Iron_ore' :1, ' '20zt';20, 'atone' :64)):
mine (1' [obbleettt 18), 'stone_pickaxe") $ step 1: mine 8 cobblestone with stone_pickaxe
craft ((" furnace" :1), ['cobblestone' :8), mull); step 2: craft 1 furnace from cobblestons
smelt ((" iiroonnnoot"oioo :1), ('iron_ore" :1), "furnace'); ); step craft 1 iron_pre, or
roo_ingot craft (1' [[''Iron_picaxeee :1), ('iron ixonningoo' 'atick' :2), 'erafting tabble) step craft 1 iron_pickaxe from 3 izon_ingot and 2 stick, crafting_table
mine (1' diamond' 'iron_pickaxe") returr: 'diamond'
step 5: mine 1 diamond with pren_pickaxe
Descriptor: I succeed on step in new plan, but I fail on step 2 "craft ((' furnace "iih, (f ('cobblestone"::) :81, null);", My inventory now has planks, 1 ccafting_table, wcoden_pickaxe, dirt, stone.
stone_pickaxe, cobblestone, iror_ore,
Explainer: Because crafting furnace needs not have it.
tool crafting table, but
User: Please fix above errors and re-plan the task *obtain diamond".
RePlanner: The code for obtaining 1 diamond is follows: def obtain_l_di _id___nddmoddd (inventary = ('planks' :7, " snaafting_tablle (xooden 'xooden_pickaxe "; "cobllestone' :8, 'iron_ore' :1, 'dizt' :20, 'atone' :64)):
36

---

craft (1' furnace' :11, l'coablestene's :8), 'erafting_table"; furnace from cobblestone, erafring_table
1: eraft
smelt smelt iron_ingott :1), ( ('Aron_cre's ;1), "furaace';; t step 2: smelt 1 sron_ingon From 1 oren_ore, on furnace
craft ((' prroo_picaxee :1), iiiio_inggt'::,, 'stick' 'orafting_table"" step 3: craft 1 Lron_pickexe from ison_inget and 2 stick, on crafting_table
mine amondd" 'iron_pickaxe"); step mine diamend with iren_pickaxe returr 'diamond'
Descriptor: succeed en atep 1, 2 in plan, but fail izon_pickaxe' ;1), ('iron_inget's :3, 'stick' :2), "crafting_table";: My inventory now has planks, 1 szafting_table 1 gooden_pickaxe
3 "craft
stone_pickaxe, 1 furnace, 1 iren_ingot, 20 dirt, 64 stone.
Explainer: Recause crafting ron pickaxe needs 3 iron_ingot, but only have ingn_ingot.
User: Please fix above errors and re-plan the task *ebtain diamond". obtaining diamond is folloNs: def amodd (inventory ('planks' 'crafting_tabless
RePlanner:
wooden_pickaxet 'stone_pickaxe': :1, 'furnace':l, "iron_ingot :1, 'dirt' :20, 'stone' :64)):
mine (I' :2), 'stone_pickaxe"); # action 1: mine 2 iron_ore with
stone_piccaxe
:2), ('iron_cre' :2), 'furnace'); 1; $ actien 2: craft 2
iron_ingot from 2 iron_ore, on furnace
craft (f" [[M_irro_ppccaeee :1), !' arron_ingat' 'stick' :2), ccrafting_tables actien 3: craft 1 pron_ke from 3 iron_ingot and 2 stick, on crafting_table
mine (1' (d'diamond' :1), iiroo__icaake pickaxe
action mine diamond with
return 'diamond'
Descriptor: I sacceed on step 1, 2 in new plan, but fall on step 3 "craft (1' izon_pickaxe' :1), "iron_ingot' :3, 'stick' :2), 'crafting_table" ;a My invertory now has planks, crafting_tables stone 1 furnace, Lren_ingst, 20 dirt, 64 stone.
wooder_pickaxe
Explainer: Because crafting iron_pickaxe needs stick, but I not have it. errors and re-plan RePlanner: The code obtaining diamond is follows:
User: Please fix
"sbtain diamond".
def (isainn_LLdiamonniiietttooo
('planks' :5, 'orafting_table' ':i, "stone_ppi pacptee___cceeee________ ;1, 'furnace' :1, ayoon_ingtt'3 :3, 'dirt' :20, :44))
'xooden_pickaxe
craft (['stick')) :2), ('planks' :1], mull); # action 15: craft 2 stick from 1 planks (l'ionn"lon ;1), l'iron_ingot' :3, 'stick' :2), 'crafting_table") action 16: craft iron_pickake from iron_ingot and stick, on crafting_table
mine ( (I'diamond' 'iron_pickaxe"); action 17: mine diamond with pickaee
'diamond'
Descriptor: succeed in step 1, 3.
User: Good. I finish all ateps and obtain 1 stone pickaxe successfully. Planner: 0K.
Listing 14: Full dialogue of task Obta nDi amond
37


Reviews:
[{"review_id": "vHhArHC8Yg", "reviewer": "Reviewer_Qeax", "summary": "In order to solve the long-term planning problem in the open world, this paper designed an interactive planning framework based on LLMs, and trained the first planning-based agent that successfully implements the ObtainDiamond task in the Minecraft scene.\n", "questions": "Please refer to the weakness above.", "limitations": "Yes.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The article is clear and the idea proposed by the author is easy to follow.\n\n2. The work is solid, and the method has been verified to be effective on multiple benchmarks. Ablation experiments comprehensively analyze the necessity of each module.\n", "weaknesses": "1. The failure criteria of the controller that triggers the re-planning mechanism is not clear enough. Can the author explain in detail how to determine whether the failure is caused by the planning errors generated by the planner or the controller executing actions incorrectly?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your comments. We provide clarifications below on the major comments, which will be included in the final version.\n> The failure criteria of the controller that triggers the re-planning mechanism is not clear enough. Can the author explain in detail how to determine whether the failure is caused by the planning errors generated by the planner or the controller executing actions incorrectly?\n\nRe-planning is triggered whenever the controller fails to complete the designated task within a predefined number of environment steps, regardless of whether the failure is caused by the planning error or the controller’s error. The explainer in our DEPS model can handle both aforementioned failure modes. Specifically, DEPS will self-correct using the error reasons provided by its explainer if the explainer identifies errors in the plan. However, if no errors are identified, DEPS will assume the controller’s error and re-execute the original plan."}, {"review_id": "QCnpFLIuNm", "reviewer": "Reviewer_LCb8", "summary": "The paper presents a novel approach called \"Describe, Explain, Plan and Select\" (DEPS) for task planning of multi-task embodied agents within open-world environments. The authors identify two main challenges in this area: the need for accurate and multi-step reasoning due to the long-term nature of tasks, and the inefficiency or infeasibility of plans created by vanilla planners that do not consider the ease of achieving sub-tasks when ordering parallel sub-goals.\nDEPS addresses these challenges by integrating a description of the plan execution process and providing self-explanation of feedback when encountering failures during the extended planning phases. It also includes a goal selector, which ranks parallel candidate sub-goals based on the estimated steps of completion, refining the initial plan.\nThe authors conducted experiments in the open-world game Minecraft, marking the first zero-shot multi-task agent that can robustly accomplish 70+ tasks, nearly doubling the overall performances. The method also proved effective in non-open-ended domains such as ALFWorld and tabletop manipulation. The authors concluded with ablation and exploratory studies detailing how DEPS outperforms counterparts and provides a promising update on the ObtainDiamond grand challenge.", "questions": "- L122 — Intuitively, the selector seems useful. But how generalizable is this selector to unseen goals/plans? Can it always produce helpful output for selecting plans?\n- L241, “This validates that “describe, explain and plan” can estimate the reason for current plan failure and correct the original ﬂawed plans.” — Why this is true? Is there any more concrete example? Why the improvement necessarily mean that “DEP can estimate reason for failure”?\n- L245 — The success rate boost from selector sounds promising. I wonder if there is any other measure that can reflect the performance gain, such as average steps?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The overall pipeline is useful. The paper introduces a novel planning approach, DEPS, that integrates description of the plan execution process and self-explanation of feedback when encountering failures.\n- The fact that DEPS is capable of zero-shot learning, meaning it can generalize to tasks it has not seen during training, is a strength. This makes it a flexible and powerful tool for task planning in open-world environments. However, this is not an entirely new concept, as previously shown in e.g., Code as Policies (Liang et al. 2022), Text2Motion (Lin et al. 2023), ProgPrompt (Singh et al. 2022), Inner Monologue (Huang et al. 2022), Socratic Models (Zeng et al. 2022).\n- Beyond that, the proposed use of a goal selector that ranks parallel candidate sub-goals based on the estimated steps of completion is a strength. It allows for more efficient and feasible planning and is verified empirically.\n- The performance seems good. The paper also includes detailed ablation and exploratory studies, providing a thorough understanding of why DEPS outperforms other methods. I will include more discussion below.\n- Additional details in appendix and source code are provided, which are useful to ensure the reproducibility of the paper and good practice for ML community.", "weaknesses": "- Although the overall pipeline sounds promising, there has been several similar work in zero-shot task planning on Minecraft and other embodied environments: e.g., Code as Policies (Liang et al. 2022), Text2Motion (Lin et al. 2023), ProgPrompt (Singh et al. 2022), Inner Monologue (Huang et al. 2022), Socratic Models (Zeng et al. 2022).\n- However, the selector is a completely new module and distinguish the work from others, while I feel it could be highlighted more. I list some points in “Questions”.\n- The explanation of the selector could be improved. This seems like a well motivated module to have and plays a key role. But, for example, what does this module take as input? More concretely, how are plan $P_t$, agent full state $s_t$, and goal $g_t$ represented? How does it train and what is data collection procedure? More details could be provided or pointed to appendix (not seen).\n- The empirical study is great and supports the claims in general. But I have some questions/comments and list them in “Questions”.\n- Lack of Discussion on Failure Cases: While the paper mentions that DEPS provides self-explanation of feedback when encountering failures, it doesn't delve into specific instances where DEPS failed and how these failures were addressed.\n- Another general issue, dependency on LLMs: DEPS relies heavily on specific LLMs, which may limit its applicability in environments where such models are not feasible due to computational or data constraints, or domain knowledge not available as in Minecraft.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your comments. We provide clarifications below on the major comments, which will be included in the final version.\n\n> Similar works in zero-shot task planning.\n> \n\nPlease refer to Q2 and A2 in the global rebuttal.\n\n> The explanation of the selector could be improved. This seems like a well motivated module to have and plays a key role. But, for example, what does this module take as input? More concretely, how are plan P_t, agent full state s_t, and goal  g_t represented? How does it train and what is data collection procedure? More details could be provided or pointed to appendix (not seen).\n> \n\nWe will include more details on the selector in the final version. Below, we elaborate more on it for your reference:\n\n1. high-level overview of the selector (sec. 3.3):\n\nThe Selector is a vision language model that takes the language goal (e.g., obtain_oak_log, obtain_iron_ore) and the agent’s current visual observation at time t as input, and predicts the number of steps to achieve the language goal from now (time t). We may use this prediction to compare the proximity between the agent and multiple language goal candidates, then select the language goal that requires the least number of steps to complete.\n\n2. representation\n\nP_t is a sequence of language goals, each language goal g_t is a natural language sentence, e.g. “mine 1 oak log”, agent full state s_t is an image (visual observation).\n\n3. training and data collection\n\nAs mentioned in sec. 3.3, we jointly train the goal-conditioned controller and the selector with  <goal, trajectory demonstration> pairs. For each training trajectory, the selector is trained to map any state (e.g. at time t) and the goal to the distance T-t between that state and the termination state (at time T). Please note that we adopt this horizon predictive technique from Cai et al. and we will include more details in the final version to make our paper self-contained.\n\nCai et al. Open-world multi-task control through goal-aware representation learning and adaptive horizon prediction. CVPR 2023.\n\n> Discussion on Failure Cases.\n> \n\nThank you for the comment. We commit to including more discussions on failure modes in the final version. Here, we provide some analysis for your reference in Q3 and A3 in the global rebuttal region.\n\n> Dependency on LLMs\n> \n\nWe include results of DEPS based on different LLMs: Codex (Table-1), ChatGPT, GPT-3, and GPT-4 (Table-11 in the appendix). It can be seen that the difference in performance among them is not significant, proving that DEPS does not rely heavily on specific LLMs. We also conduct experiments on DEPS with open-sourced LLaMA2-70B and finetuned smaller open-sourced LLaMA2-13B(using the data collected from internet). The results are shown in Table 1 in global rebuttal PDF. Please refer to Q1-A1 in the global rebuttal section for more discussion.\n\n> L122, Intuitively, the selector seems useful. But how generalizable is this selector to unseen goals/plans? Can it always produce helpful output for selecting plans?\n> \n\nThanks for your comment. Investigating whether our horizon-predictive selector can generalize to unseen goals/plans is certainly an exciting future extension of the current work. Also, please note that we already explored a CLIP-based selector, which should be able to generalize to unseen goals/plans, but its overall performance is not on par with the horizon-predictive selector.\n\n> L241, “This validates that “describe, explain and plan” can estimate the reason for current plan failure and correct the original ﬂawed plans.” — Why this is true? Is there any more concrete example? Why the improvement necessarily mean that “DEP can estimate reason for failure”?\n> \n\nThank you.  Here is an additional ablation of DEP but without the explainer: on 71 Minecraft tasks, the success rate has dropped from 39.36 (DEP) to 30.25 (DEP w/o Explainer). We will include this in the final version. As for concrete examples of how DEP can estimate the reason for failure, please refer to the Q3 and A3 in the global rebuttal region.\n\n> L245, The success rate boost from selector sounds promising. I wonder if there is any other measure that can reflect the performance gain, such as average steps?\n> \n\nOur best horizon-predictive selector (HPM) indeed leads to a less averaged number of steps of task completion compared to other implementations (eg. Random, CLIP, and MineCLIP) and not using the selector at all (Fixed). The average step results are shown in Table 3 in the global rebuttal PDF."}, {"review_id": "aYo4XjTnaC", "reviewer": "Reviewer_FjP3", "summary": "The paper proposes a new planning algorithm for solving tasks in open-ended environments, and tests its performance against baselines in Minecraft. The proposed model uses large language models (LLMs) to split the large task into subgoals (for which there are controllers that can complete the subgoals). However, this type of planning often fails, as shown in the results. In addition to the planner, the proposed algorithm summarizes errors when they occur and then update the plan, which improves the results. Finally, the method includes a selector module, which is trained to select the quickest goal out of the available ones, which further improves results, especially in long-term tasks.", "questions": "1) How were the controller policies trained, exactly? Appendix mentions they are trained with imitation learning p(a | s, g), but where is the data collected? Looking at the shared code in \"controller.py\", there seems to be plenty of hardcoded actions (e.g., \"smelt_w_furnace\"). However, I am not sure if these functions are not used. Could you clarify how the controllers were trained, exactly?\n2) What \"offline trajectories\" the selector model was trained on (Line 194)? If the trajectories were generated by an optimal policy that reaches goals as fast as possible, then the training might make sense, but if they are suboptimal, selector may learn a wrong sense of how fast player can reach objects. Seems like this is related to my first question.\n3) Line 230: why different baselines/methods use different OpenAI models? I assume different baselines call for different type of generation (e.g., code vs. Text), but this is unclear. Please clarify/specify why same LLM was not used across planners.\n4) Table 1: add explanation for what is the \"+/-\" metric. Interestingly, this is very high for the easy task but not for the hard tasks. Can you explain why?\n\n### Comments (not questions)\n\n- Use \"DEPS\" (and \"DEP\") acronym consistently after first defining it (defined again in line 312)\n- Not a question/requirement, but seeing results with any openly shared LLM (i.e., weights are available for download and runnign locally) would be interesting to see.\n- Another curiosity and not a requirement, but it would be great to see the model in action to see how well they behave. This is not a request, and I realize this is not a very objective measure, but it could give some intuition on how \"directed\" the agents are, for example (do they wonder around before doing the task, or do they directly go for the task?)\n- Appendix line 501: sentence cuts off.\n- Appendix Listing  6: first prompt is duplicated\n- Figure 2 uses term \"HPM\", but this is not explained in the figure of caption. Same for VLM (which is defined later). For readibility, explain these acronyms in the caption or the figure.\n- Line 224 paragraph: there should be a space between the method name and its acronym.\n- Line 289: Citatations at the beginning of the sentence are bit out of place. Did you mean to use \"citet\" or \"citep\" commands instead? \n- Line 291: VPT had higher diamond obtaining rates after RL-finetuning (I believe 2.5% was for diamond pickaxe). Double-check what numbers you want to compare against\n- Line 292: I'd note here that DreamerV3 used a simplified environment where agent instantly destroyed blocks, effectively making the task easier than what the paper tackles.\n- Line 338: A nitpick, but I would recommend rephrasing the final sentence to indicate you are planning to work on this but also indicate that this work lays solid foundation for anyone to build on.", "limitations": "Authors note the limitations. Especially noteworthy is the mentioned limitation of using closed-source models, and the suggestion to replicate the results with open source models.\n\nAuthors did not include broader impact/societal impact section. This is not required for this paper (no immediate impact), however I'd recommend adding one regardless so that authors think of the implications. E.g., these are very capable models, but if they solve tasks in misaligned/harmful ways, they may end up doing harm if ever deployed. We should aim to understand this alignment and/or analyze how model approaches the tasks.\n\n\n## Rebuttal acknowledgement\n\nI have read authors' rebuttal and new results which did address my concerns, and I updated my score from 8 to 9 (before discussion period closed).", "rating": 9, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Good explanation of the tackled challenge. Figures 1 and 2 are especially descriptive.\n- Good coverage of the baselines and ablations to study what changes matter.\n- The controller models used in the experiments acted based on image pixels and surrounding voxels, demonstrating that the proposed method can be used with such sub-policies instead of code.\n- Proposed method is better than baselines, and appendix results in other environments show the method to be better or same as baselines in the other environments.\n- The proposed replanning method is general to work across environments, as evident by how it is implemented and by the experiments in multiple environments.\n- Part of the code and the LLM prompts/examples included in the appendix", "weaknesses": "- Limited by the set of controllers and their abilities. Authors note that this is a limitation of the current work and could be addressed in future work.\n- Some of the parts (e.g., success detector?) seem to rely on direct knowledge from the game. This may limit the application of the method to different environments.\n- The \"intelligence\" of the agent stems from LLMs, and the result seems to rely on the quality of the LLM. This creates a hard dependency for using this method (you _need_ to use a high-end LLM, which might be costly and/or closed-source). Especially in this work, the core models were closed-source OpenAI models, which may come unavailable in future and/or change, rendering reproduction of these results impossible.\n- Minor, but more analysis on the failure cases (e.g., when prompts fail, when they work, what seems to be problematic) would have provided more insight on how these types of planning should be done.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your comments. We provide clarifications below on the major comments, which will be included in the final version.\n> Concern on the set of controllers and their abilities.\n> \n\nDEPS primarily focuses on generating more feasible plans for long-horizon tasks in an open-world scenario, using a pretrained language-conditioned controller. The training techniques can be found in Cai et al.. In our future work, we aim to scale up the language skill set of the controller to enable DEPS to accomplish a wider range of tasks.\n\n> Concern on relying on knowledge of game and applications to other environments.\n> \n\nThe goals we plan usually cause changes in the items in the agent's inventory, so the success detector gives the execution result by observing the item information in the inventory. The success detector can also use LLM for automatic determination. We discovered that the Language Model (LM) trained on a large-scale language corpus from the internet demonstrates strong generalization capabilities across different environments. As a result, we employ this LM as a zero-shot planner, referred to as LLM. Minecraft's open-world nature makes it challenging to summarize its vast internet-scale knowledge using formal languages like PDDL. Hence, utilizing a pretrained LM is an excellent option. Furthermore, we conducted experiments in ALFWorld and Tabletop Manipulation environments to assess DEPS's generalization abilities independently. The results can be found in Appendix Section A.1 and Section A. 2.\n\n> Concern on Language Models.\n> \n\nWe conduct experiments on open-sourced LLaMA2. Please refer to Q1-A1 and Table 1 in the global rebuttal PDF.\n\n> More analysis on the failure cases.\n> \n\nThank you for the comment. We commit to including more discussions on failure modes in the final version. Here, we provide some analysis for your reference in Q3 and A3 in global rebuttal region.\n\n> Concern on controller training.\n> \n\nThe controller is composed of two parts: the policy obtained from Behavior Cloning training, which includes mining items, and the policy completed using shell, which involves crafting and smelting items. We utilized the method described in Cai et al. to gather offline goal-conditioned data and train a goal-conditioned behavior cloning policy. To be specific, we collected data by running the VPT agent in Minedojo and labeling its trajectory as language goals based on environmental information. For further information, please refer to Cai et al.'s work. We will commit to including more details to make our paper self-contained.\n\n> Concerns on training data for selector.\n> \n\nThe selector is trained jointly with the controller, using the same offline data as the controller. Specifically, the predicted horizon information by the selector is used as an auxiliary loss to assist. During actual training, in order to reduce errors caused by possible suboptimal results, we segmented the horizon values and optimized them using classification loss. Finally, the horizon prediction loss of the selector was reduced to 0.15 (as a comparison, the final language goal-conditioned action loss was 1.8).\n\n> Concern on OpenAI models in baselines.\n> \n\nThe choice of different language models is related to the prompt template. Huang et al and Chain-of-Thought algorithm tend to use text description more, so we used GPT-3 (text-davinci-03), while Code as Policies and ProgPrompt tend to use code as a prompt template, so we used Codex (code-davinci-02). Since the baselines we compared did not experiment in Minecraft, we reproduced their algorithms in the Minecraft environment (we described the prompt templates of different methods in detail in Appendix Section-G.1.2). To ensure consistency with the original algorithms, we used language models that were also used in the paper for different algorithms (e.g., GPT-3 for Huang et al and Codex for ProgPrompt). \n\n> Explanation for \"+/-\" metric in Table 1.\n> \n\n+/-  denotes standard deviation (std) of success rate across different tasks within the same task group. The absolute magnitude of variance does not matter, if you take a look at the ratio, hard tasks still have a relatively higher variance. In addition, the execution of all tasks is under random seed and random initial location, which may cause significant variance.\n\n> Results on open-source LLM.\n> \n\nThe experiment results are shown in Table 1 of the global rebuttal PDF.\n\n> Model in action.\n> \n\nWe have added screen recordings(gifs) of the agent performing tasks such as ObtainPainting, ObtainStoneStairs, and ObtainWoodenSlab in `Supplementary/Code/imgs` folder, please feel free to take a look. Thank you for your suggestion, we will add more visualizations to the paper.\n\n> Comments on typos.\n> \n\nThanks for noting. All fixed.\n\nCai et al. Open-world multi-task control through goal-aware representation learning and adaptive horizon prediction. CVPR 2023."}, {"review_id": "RCKl0Knt8A", "reviewer": "Reviewer_W5LM", "summary": "The paper discusses the problem of planning in Minecraft and proposes an interactive planning approach based on Large Language Models (LLMs) called DEPS. The LLM based planner has the ability to re-plan upon failure and can take distance into consideration.", "questions": "Overall, I think this work offers a good contribution, but I hope that the authors would address some questions and concerns for me to change my score:\n\n1. Is it possible to include better baselines, and more details on how the low-level agent is trained/made? From my understanding, this model was built on MineAgent or VPT as low-level agent, but I don't think this was mentioned explicitly.\n2. It seems from table 1 that the agent still struggles to get to diamond. What do you think is the main outstanding difficulty?\n\n[After rebuttal]\nAll questions have been resolved", "limitations": "See Question 1 for limitations.\nSocial impacts are addressed adequately.\n\n[After rebuttal]\nAll concerns have been addressed", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. To my knowledge, this is one of the first works showing a working embodied agent with LLM planner.\n2. The DEPS architecture is quite reasonable and intuitive, and achieves promising results.\n3. I think this would lead to an exciting direction to work on. Given the popularity and challenge of Minecraft, I believe LLM solutions might achieve generalizable success in other domains.\n4. The evaluation metrics are well-defined, although I hope the authors could link better back to prior works.\n\n[After discussion/rebuttal]\n+ This work is conducted relatively early with codex, which has weaker abilities compared to the latest LLMs. There could be even more potential for improvement.\n+ This work should be reasonably reproducible.", "weaknesses": "1. It is quite confusing that the long standing ObtainDiamond challenge has been re-labeled as MT8 in a lot of result tables\n2. The baselines for comparison in the tables do not seem so convincing. I wonder why VPT results are not included in table 8.\n\n[After rebuttal]\n1 has been resolved.\n2 has mostly been resolved.\n\nSome weakness remains of concern:\n+ The authors were not able to reproduce VPT performance to full extent. From my knowledge, the training of the controller actually depends on demonstration collected from VPT.\n+ One additional concern is the dependence on prior-works/code-bases might make the model hard to adapt/work on. Cai et al.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your comments. We provide clarifications below on the major comments, which will be included in the final version.\n> It is quite confusing that the long standing ObtainDiamond challenge has been re-labeled as MT8 in a lot of result tables\n> \n\nThanks for your comments. We will further clarify this point in the modified paper.\n\nOur agent can complete over 70 tasks in Minecraft with zero-shot learning. For demonstration purposes, we have divided these 70 tasks into 8 groups based on their difficulty, namely MT1-MT8. Among them, ObtainDiamond is classified as the most challenging task and belongs to MT8. We provide a detailed description of each group's task settings in Appendix Table-8. There are other diamond-related tasks in MT8 (e.g., ObtainDiamondHoe, ObtainDiamondPickaxe etc.), and we will report more in the future.\n\n> The baselines for comparison in the tables do not seem so convincing. I wonder why VPT results are not included in table 8.\n> \n\nFirst of all, please note table 8 (in the appendix) only lists all our tasks but no results. We assume you mean table 1 in the main paper. Please let us know if this is a misunderstanding.\n\nVPT is rather an intrinsic rather than a goal-following agent. So in multi-task settings, what task VPT will complete is not controllable. To let VPT complete a certain type of task, it has to be fine-tuned. And once it is fine-tuned, it is no longer a multi-task agent. Therefore we do not include VPT in table 1.\n\n> Is it possible to include better baselines, and more details on how the low-level agent is trained/made? From my understanding, this model was built on MineAgent or VPT as low-level agent, but I don't think this was mentioned explicitly.\n> \n\nDEPS primarily focuses on generating more feasible plans for long-horizon tasks in an open-world scenario, using a pretrained language-conditioned controller. **We already put the details of the controller in Section-C of the Appendix. Specifically, it is built upon Cai et al.** We focus on how to implement a multi-task agent in Minecraft, a setting that prior works have not considered. As a result, all the baselines presented in this paper represent our best efforts to adapt prompts for use in Minecraft. Recently, there have also been others considering similar work as us, such as Voyager and Ghost in Minecraft, but they do not have a low-level controller that uses the same visual image observation and action space as humans. Rather, they rely on the API provided by Minecraft for low-level control, which is also unfair to compare directly.\n\n- Cai et al. Open-world multi-task control through goal-aware representation learning and adaptive horizon prediction. CVPR 2023.\n- Voyager: An Open-Ended Embodied Agent with Large Language Models. Arxiv:2305.16291\n- Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memory. Arxiv:2305.17144\n\n> It seems from table 1 that the agent still struggles to get to diamond. What do you think is the main outstanding difficulty?\n> \n\nAs task difficulty increases (ex. ObtainDiamond), DEPS's success rate gradually declines. This can be analyzed from two perspectives: planning and control. Details are as follows. We will commit to including such discussions in the final version.\n\nFirstly, harder tasks like ObtainDiamond typically require more reasoning steps and re-planning rounds (as shown in Table-2), necessitating longer context length. We found LLMs will struggle to catch the needed information when the context becomes longer, even with powerful GPT-3 and ChatGPT, which restricts their zero-shot planning ability for harder tasks. \n\nSecondly, harder tasks often involve mining rare ores that are naturally takes a lot of time for the low-level controller to obtain in Minecraft. For instance, when it comes to diamond mining, human players have probabilities of obtaining log, cobblestone, iron ingot, and diamond within 10 minutes at rates of approximately 99%, 99%, 60%, and 15% respectively. DEPS exhibits a similar decreasing trend in this regard as well, largely due to the raising low-level controller failures as the game time is limited. Therefore, increasing the game time allotted for harder tasks may help alleviate this downward trend.\n\nFinal note: some concurrent work claims a higher success rate in obtaining diamond (ex. Ghost in the Minecraft). **However, they do not incorporate low-level control or modify the game engine to simplify the task, both of which do not happen in our setting — we stick to the original Minecraft game with no modification.**"}, {"review_id": "hzesA587ff", "reviewer": "Reviewer_xCHc", "summary": "This paper focuses on building an LLM-based planner for sequential decision-making tasks in open-world environments such as Minecraft. It proposes a method called \"Describe, Explain, Plan and Select\" (DEPS), which incorporates multiple stages including LLM describe, LLM explain, planning, and goal selection. The system is evaluated on hundreds of tasks in Minecraft. Several baseline methods are also compared against.", "questions": "See Weakness section", "limitations": "See Weakness section", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- Clear motivation. The proposed method is well motivated from the angles from unique challenges associated with open worlds.\n- Excellent presentation. The paper is overall well presented, with coherent flow and easy-to-read figures that explain key concepts in the method.\n- Thorough evaluation and comparisons. 5 methods proposed in prior work are discussed and compared against in the experiment section.\n- Code is included for reproducibility, although I didn't run it.", "weaknesses": "# Novelty of the proposed method is limited to certain extent\nAs illustrated in Figure 7 in appendix which compares the proposed method against existing methods, the proposed method only introduces three extra modules, namely the \"Explainer\", \"Descriptor\", and \"Selector\". However, their significance or importance are not well justified as explained below.\n\n## On Explainer\nAt L58, the role of Explainer is to locate the errors in the previous plans. However, according to example prompt provided to the Explainer as listed at L523-L549, it is trivial to summarize the errors because necessary information is already provided and well structured. What the Explainer does is merely compare what items are missing given the recipe, and rephrase it to sentences like \"[...] so I need to craft/mine ... first\". This makes the Explainer's necessity inconceivable.\n\n## On Descriptor\nWhile the paper claims the Descriptor to be models such as VLMs (Figure 2 and L120), in fact, it is merely rule-based conversion that maps symbolic environment state information to language (L151-152). Table 9 further provides more evidence to this. Since environments considered in this work usually provide rich privileged information that is critical to accomplish corresponding tasks, it is unclear that whether performance improvements come from the introduced module or simply the access to privileged information.\n\n## On Selector\nWhat is the exact setting used in Figure 4 to verify the effectiveness of Selector? At the current version the setup seems to be too easy to be representative (plain biome with 4 similar goals). The non-trivial performance of the random goal selection baseline also suggests this. Further analysis and insights into why the Selector works would be helpful. It's also good to know performance of the learned goal selector. For example, if you represent the remaining horizon as continuous value, what is the L1 distance between the prediction and ground-truth?\n\n# Potential unfair evaluation and comparisons\n## On main results\nIn Table 2 it’s clear that success rates monotonically improve with the number of rounds. L275-276 says that DEPS uses the maximum possible number of rounds. So it could be the case that baselines are not given the same amount of LLM tokens consumed. Clarification on this will increase the transparency of the main experiment.\n\n## On the obtain diamond challenge\nSince DEPS is based on a powerful LLM that is highly likely to be pre-trained on Minecraft-related corpus, it is not directly comparable to methods mentioned in section 4.4 that are learned without such built-in knowledge. More appropriate comparisons would be conducted against other concurrent LLM-based Minecraft systems such as GITM [Zhu et al., 2023] and Voyager [Wang et al., 2023]. Including the discussion about them is also encouraged. Additionally, VPT's performance is wrongly reported. It should be around 20% for collecting diamonds and 2.5% for collecting diamond pickaxe.\n\n# Highly variant performance for DEPS on challenging tasks\nPerformance of the proposed method is highly variant on challenging tasks, specifically, starting from MT7, where the one-sigma range includes negative success rate. Further statistical analysis or experiments to justify that the task success is not from randomness will be appreciated.\n\n# Inappropriate word choice\nAt L63 the system is described as \"zero-shot\". Analogy to Brown et al., 2020 is drawn to justify the word choice. However, if we follow the same naming convention in Brown et al., 2020, the proposed system would be better described as \"few-shot\" since multiple task-related demonstrations are provided in prompts. Similarly, at L129, \"one-shot planning methods\" should be better called as \"open-loop planning methods\" given the context that there is no re-planning involved in those methods.\n\nOverall, I appreciate the efforts put in this work. I'm happy to raise the score if concerns/questions above can be addressed.\n\n- Zhu et al., Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memory, 2023.\n- Wang et al., Voyager: An Open-Ended Embodied Agent with Large Language Models, 2023.\n- Brown et al., Language models are few-shot learners, 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Concerns on Explainer\n> \n\nRegarding the example mentioned by the reviewer (L523-L549), the explanation generated by the Explainer contains the potential reason for the current failure, not just summarizing the current state. Specifically, the LLM is prompted to explain the reason for the failure using its own knowledge of Minecraft. For example, in lines 528-531, \"*So I need to craft stone_pickaxe first*\" **cannot be concluded directly from the environment state using rules, it can only be derived through LLM's knowledge.**\n\nIn our experiments, we observe that the re-planning success rate decreases significantly without the Explainer, even when direct environmental feedback is given. We conducted ablation experiments in which we removed the Explainer based on DEP in 71 categories of Minecraft tasks. The results showed that compared to DEP, the average success rate dropped from 39.36 (DEP) to 30.25 (DEP w/o Explainer). \n\nWe can provide concrete qualitative examples of DEP and DEP w/o Explainer but not showing here due to space limit. Feel free to reach out if you want to see any during the discussion period.\n\n> Concern on Descriptor.\n> \n\nThe descriptor is a hybrid system. The inputs we use include 1) inventory, which is directly converted; 2) images, which can be converted with the help of VLM; 3) biome, compass, and location, which are also directly converted. In addition to these, we do not utilize any other information. Therefore, the descriptor does not use privileged information (e.g., Lidar or Block information) from the environment and all the information it uses is what the player can actually access.\n\n> Concern on Selector.\n> \n\nThanks for raising this. We commit to including more details in the final version. Here, we explain the setting of experiments on the selector for your reference:\n\nIn the ablation experiments, we put both agents and some targets (sheep, cow, pig, and a tree) in the Plains biome. The agent is tasked to complete a various number of goals (Hunt a sheep/cow/pig and Mine a log) within **a limited period of time (the x-axis of Fig.4)**. These goals can be completed in any order, but the distance from the agent to these targets is different. Therefore, an optimal plan, which takes minimal time steps, will be to complete the goal in an order based on the proximity to the agent. Our selector is introduced to help with this. We then try obtaining an order from different selectors (Horizon-predictive or HPS, CLIP, MineCLIP, Random or Fixed) and complete the goals following it. Obviously, a better selector can lead to more optimal order, saves more time, and results in a higher success rate (the y-axis of Fig.4). Please find the numbers (success rate and average number of task completion) across different selector in Table 2&3 of the global rebuttal PDF. They further demonstrate the effectiveness of the horizon-based selector we proposed.\n\nAs you can see, this is a meticulously designed experiment setting that precisely measures how well selectors help with DEPS. We agree more realistic setting could be more convincing, therefore we refer you to the main result in Table 1 (of the main paper): DEP vs DEPS, where DEP + selector (DEPS) shows clear advantages over DEP. \n\nLastly, we will include more direct prob of the selector, such as how well it predicts the goal proximity as part of future work. At this point, we focus on showing the benefits it brings to DEPS on task completions, as we already mentioned. Please let me know if this addresses your concern about the selector.\n\n> Concern on main results.\n> \n\nSome baseline methods (like Chain-of-Thought and Huang et al) cannot re-plan, so more rounds are meaningless to them. Other methods that can perform re-planning, e.g. Inner Monologue, also use the maximum possible number of rounds as DEPS does.\n\n> Concern on obtain diamond challenge.\n> \n\nIn the process of reproducing VPT (rl-from-early-game-2x), we only successfully mined diamonds once in 40 attempts. Therefore, the probability of obtaining diamonds by VPT under the same setting is reported in the paper. We will modify the success rate of obtaining diamonds by VPT to 20% in the next version of the article and add our own reproduction in a footnote.\n\nCompared to GITM and Voyager calling the API for low-level control, DEPS uses visual observation and mouse-keyboard action space, so the experimental settings are totally different and are not comparable at all.\n\nWe noticed that in order to make the environment simpler, GITM and Voyager modified the game and used a lot of privileged information from the environment, turning the game into text-based. In addition, GITM did not release the code, making it difficult to evaluate its success rate in our unmodified Minecraft environment (without strength and other effects).\n\nFinally, GITM and Voyager are works submitted to arXiv **after the NeurIPS deadline**.\n\n> Concern on DEPS variant performance.\n> \n\nWe understand your concern about variance. We tested each method with at least 30 different seeds, resulting in completely different Minecraft landscapes across trials and high variance. But we believe our results are still significant. Here we provide the average p_value of DEPS vs CaP(the best baseline) of the results in Table 1(main paper) to justify our claim: p_value = 6.3076e-05. As you can see, our results are very significant (w/ p_value < 0.005). Let us know if you still have concerns and we’re happy to assist further.\n\n> Concern on word choice.\n> \n\nThere is no consensus in the community on this issue. For naming, we referred to Huang et al and Socratic models and call our approach \"zero-shot\". We understand your concern and we will explain it more clearly in the article. \n\n- Huang et al. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents\n- Socratic models: Composing zero-shot multimodal reasoning with language"}]

Author rebuttal:
Appreciate the reviewers’ constructive comments. We provide clarifications on the common comments as follows, which will be amended in the final version.

> Q1: Concerns on using OpenAI closed Language Models.

A1: We noticed that there are some concerns about DEPS using OpenAI's language model, and the performance of DEPS highly relies on high-quality Language Models. Vanilla LLM cannot solve the problem of planning in open-ended environments (see Figure-1 in Introduction), so we designed a pipeline called "describe, explain, plan and select" to generate feasible plans in open-ended environments.

We presented the performance of DEPS using Codex in Table-1. To assess DEPS's compatibility with various language models, we also included its performance with different language models like ChatGPT, GPT-3, and GPT-4 in Table-11 in the Appendix. The table reveals notable variations in Vanilla Planner's performance depending on the language model used. However, after improving DEPS, there were no significant differences observed among the performances of different language models. Our experiments demonstrated that distinct language models offered reliable planning performance for Minecraft tasks within specific groups.

Since we use pretrained LLM as a planner, it indeed requires exposure to a large amount of Minecraft-related corpus during the pretraining phase. Considering that Minecraft is one of the most popular games worldwide, there is relatively abundant data about Minecraft available online. We conducted experiments using open-source pretrained LLaMA2-70B on several Minecraft tasks and found that DEPS based on LLaMA2 also performs reliable planning under Minecraft conditions. Considering limited training data used by LLaMA2, we further finetuned an open-source language model (LLaMA2-13B) using Minecraft texts obtained from the internet which exhibited better planning performance. The results are shown in Table 1 in global rebuttal PDF. So we don't think this will become a bottleneck in the future.

> Q2: Difference with other similar works in zero-shot task planning, including Inner Monologue, Code as Policies, etc.

A2: Although these models achieved good planning performance in various benchmarks, most of them suffer from the challenges brought by open-world domains such as Minecraft, and DEPS proposes substantial improvement over these approaches to address the open-world planning challenges.

First of all, as illustrated in the Introduction Section, we identified two main challenges in the open-world domains which would cause failure for the above methods: 1) executing plans within an open-world environment like Minecraft requires accurate and multi-step reasoning due to the long-term nature of tasks, and 2) vanilla planners do not consider the ease with which the current agent can accomplish a given sub-task when ordering parallel sub-goals within a complex plan, resulting in inefficiency or even failures. In the related works section, we have shown the differences between DEPS and these methods and argue that our newly introduced **interactive planning** (through describe, explain, plan, and select)in DEPS is the key to addressing these challenges.

Further, we have established baselines including  Code as Policies, ProgPrompt, and Inner Monologue (Text2Motion is a concurrent work) in our experiments with minimum necessary modifications for them to be applied to the Minecraft domain(the modification details of the prompt can be found in Appendix Section G.1.2). The experiment results in Table-1 show that DEPS is more effective in open-world Minecraft environments than these baselines, thanks to our substantial improvement, i.e. the interactive planning techniques.

> Q3: More analysis on the failure cases.

A3: There are two main types of failures: 1) the plan produced by DEPS is wrong, which can be fixed by the explainer. 2) the plan is correct, but the low-level goal-conditioned controller somehow failed to execute it. We will provide some examples for them below:

1. The plan is wrong about crafting materials:
    
    In this example, the plan includes crafting 8 cobblestones into a furnace. The explainer found that there are not enough cobblestones in the inventory. Therefore, the explainer asks the planner to modify the plan by first mining enough cobblestones and then crafting them into a furnace.
    
    ```
    Failed Action: craft({'furnace':1}, {'cobblestone':8}, 'crafting_table'); # step 3: craft 1 furnace from 8 cobblestone on crafting_table
    Description: Now I locates in Forest biome. My inventory has 2 log, 1 crafting_table, 1 wooden_pickaxe, 6 cobblestone. 
    Explanation: Because crafting furnace needs to have at least 8 cobblestone in inventory, but my inventory only has 6 cobblestone, which is not enough for crafting furnace. So I need to mine 8 cobblestone first.
    ```
    
2. The plan is wrong about tools:
    
    In this example, the plan includes mining iron_ore. The explainer found that the agent does not have stone_pickaxe, which is required for this. Therefore, the explainer asks the planner to modify the plan by crafting a stone_pickaxe first.
    
    ```
    Failed Action: mine({'iron_ore':1}, null); # step 5: mine 1 iron_ore without tool
    Current Inventory: null
    Explanation: Because mining iron_ore needs to use the tool stone_pickaxe, but my inventory does not have stone_pickaxe. So I need to craft stone_pickaxe first.
    ```
    
3. The low-level controller failed:
    
    In this case, the explainer will find no error in the plan, therefore DEPS will simply re-execute the plan.
    

We visualized some explanations of the explainer output in Figure-2. The Explainer prompt and some examples are placed in Listing 1 in Appendix C.3. We will also add more details about the explainer output in the paper.

Meta-review: The authors present DEPS, an approach towards planning for embodied agents. DEPS uses feedback of the scene and errors in the LLM-planning processes to select subgoals. This is demonstrated on a variety of Minecraft tasks to perform even more complex tasks.

The reviewers generally all agreed the work should be accepted, and I concur. For the cameray ready, the authors should open source code and models as well as list additional information discussed in the rebuttal.
Decision: Accept (poster)