Title: Describe, Explain, Plan and Select: Interactive Planning with LLMs Enables Open-World Multi-Task Agents
Venue: NeurIPS 2023
Accepted: yes
Keywords: open-ended learning

multi task

large language models

zero-shot planning
Abstract: In this paper, we study the problem of planning in Minecraft, a popular, democratized yet challenging open-ended environment for developing multi-task embodied agents. We've found two primary challenges of empowering such agents with planning: 1) planning in an open-ended world like Minecraft requires precise and multi-step reasoning due to the long-term nature of the tasks, and 2) as vanilla planners do not consider the achievability of the current agent when ordering parallel sub-goals within a complicated plan, the resulting plan could be inefficient. To this end, we propose ``$\underline{D}$escribe, $\underline{E}$xplain, $\underline{P}$lan and $\underline{S}$elect'' ($\textbf{DEPS}$), an interactive planning approach based on Large Language Models (LLMs). Our approach helps with better error correction from the feedback during the long-haul planning, while also bringing the sense of proximity via goal $\textbf{Selector}$, a learnable module that ranks parallel sub-goals based on the estimated steps of completion and improves the original plan accordingly. Our experiments mark the milestone of the first zero-shot multi-task agent that can robustly accomplish 70+ Minecraft tasks and nearly double the overall performances. Further testing reveals our method's general effectiveness in popularly adopted non-open-ended domains as well (i.e., ALFWorld and tabletop manipulation). The ablation and exploratory studies detail how our design beats the counterparts and provide a promising update on the $\texttt{ObtainDiamond}$ grand challenge with our approach.

Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agents

Abstract

We investigate the challenge of task planning for multi-task embodied agents in open-world environments. 2 Two main difficulties are identified: 1) executing plans in an open-world environment (e.g., Minecraft) necessitates accurate and multi-step reasoning due to the long-term nature of tasks, and 2) as vanilla planners do not consider how easy the current agent can achieve a given sub-task when ordering parallel sub-goals within a complicated plan, the resulting plan could be inefficient or even infeasible. To this end, we propose 'Describe, Explain, Plan and Select' ( DEPS ), an interactive planning approach based on Large Language Models (LLMs). DEPS facilitates better error correction on initial LLM-generated plan by integrating description of the plan execution process and providing selfexplanation of feedback when encountering failures during the extended planning phases. Furthermore, it includes a goal selector , which is a trainable module that ranks parallel candidate sub-goals based on the estimated steps of completion, consequently refining the initial plan. Our experiments mark the milestone of the first zero-shot multi-task agent that can robustly accomplish 70+ Minecraft tasks and nearly double the overall performances. Further testing reveals our method's general effectiveness in popularly adopted non-open-ended domains as well (i.e., ALFWorld and tabletop manipulation). The ablation and exploratory studies detail how our design beats the counterparts and provide a promising update on the ObtainDiamond grand challenge with our approach. The code is released at https://github.com/CraftJarvis/MC-Planner .

1 Introduction

Developing multi-task agents that can accomplish a vast and diverse suite of tasks in complex domains has been viewed as one of the key milestones towards generally capable artificial intelligence [36, 1, 5, 10, 25]. To enable such capabilities, earlier works have suggested employing a hierarchical goal execution architecture [2, 4], where a planner generates action plans that would then be executed by low-level goal-conditioned controllers. This architecture has been delivering promising progress in

Reachable region

within 3 mins

Unachievable

routes

Optional routes

Challenge #2: State-dependent Task Feasibility

Challenge #1: Complex Sub-task Dependency

Mine diamond in Minecraft environment

99%

42%

23%

80%

9%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1

2

3

Planner w/ Learned Controller

Planner w/ Oracle Controller

Tabletop Environment

ALFWorld

Minecraft

Success Rate

Manipulation in Tabletop environment

Planning success plummet in open worlds due to new challenges

“put the blue blocks

many robotics domains, including table-top and mobile manipulation [46, 4], 2D shape drawing [20] and table rearrangement [17]. However, whether such success can be transferred to a more open-ended world with unlimited exploration areas and internet-scale knowledge remains open [14, 10, 13, 12, 19].

To understand the gap, we run Inner Monologue[17], a general and competitive hierarchical goal execution model on a typical open-world domain Minecraft [18, 14, 10] and two classical robotic environments ALFWorld [41] and Tabletop environments [40, 4]. The algorithm uses a Large Language Model (LLM) based planner that contains domain-specific knowledge for all three environments. In all environments, we use either an Oracle goal-conditioned controller or a learned one. Results are shown in the bar plot in Figure 1. First, even when the Oracle controller is used, the success rate of executing Minecraft tasks is much less than that of the other environments. Next, the task failure rate becomes even higher in Minecraft when the learned controller is substituted. Both failures originate from unique challenges brought by open-world environments, which we identify in the following.

First, compared to canonical environments (e.g., Atari [29] and robotic control suite [40]), open worlds have highly abundant object types with complex dependency and relation. As a result, groundtruth plans typically involve a long sequence of sub-goals with strict dependencies. As Figure 1 challenge #1 suggests, it requires at least 13 sub-goals executed in proper order to obtain a diamond in Minecraft, while in Tabletop a task is typically no more than a few consecutive sub-goals.

Another challenge brought by the complicated tasks in an open-ended world is the feasibility of the produced plans. Consider the example shown in Figure 1 (challenge #2). To craft a bed in Minecraft, the fastest way is by either slaughtering a sheep to obtain wool, which can be used to craft beds, or collecting beds from a village. However, since no sheep or village is reachable by the agent within 3 minutes of gameplay, to craft a bed efficiently, the agent should choose to slaughter a spider and use materials (e.g., string) it drops to craft wool, and then a bed. That is, when dealing with a task that can be completed by executing multiple possible sequences of sub-goals, the planner should be able to select the best route based on the current state of the agent. However, the complex and diverse state distribution of open-world environments makes state awareness hard to achieve.

To tackle these problems, we propose 'Describe, Explain, Plan and Select' ( DEPS ), an interactive planning approach based on Large Language Models (LLMs) to alleviate the aforementioned issues. The key to tackling the first challenge is to effectively adjust the generated plan upon failure. Specifically, whenever the controller fails to complete a sub-goal, a descriptor will summarize the current situation as text and send it back to the LLM-based planner. We then prompt the LLM as an explainer to locate the errors in the previous plan. Finally, a planner will refine the plan using information from the descriptor and explainer. To improve the feasibility of generated plans conditioned on the current state, which is the second identified challenge, we use a learned goalselector to choose the most accessible sub-task based on the proximity to each candidate sub-goal.

Our experiments are conducted on 71 tasks in open-ended Minecraft without any demonstration. Given the goal-conditioned controller for atom sub-tasks (i.e., mine log and mine stone), our zero-

LLM

*

Controller

Goal-conditioned Policy

Selector

Descriptor

VLM

LLM

*

Instruction

plan

𝑃

!

goal

𝑔

!

feedback

action

obs

description

𝑑

!

explain

Environment

obs

Task instruction: Obtain a diamond

in Minecraft survival mode step-by-step?

Candidate goals:

Selected Goal

𝒈

𝟏

:

×

4

The agent locates in the birch forest,

which only has birch wood.

Description

𝒅

𝒕

: I

succeed

on goal 1-5. I

fail

on goal 6, mining 3

with

.

Now my inventory has 5 planks, …

Initial Plan

𝑷

𝟎

:

4

16

1

8

3

3

1

1

1

Explanation:

Because mining       needs to

use at least

, which I do not have.

So I need to craft

first.

…

Finished Plan

𝑃

!

:

3

3

1

1

1

1

Selected Goal

𝑔

$

×

3

no other choices

Description

𝒅

𝒕

: I succeed on goal …. I

fail on smelting 3

from 3

, on

.

My inventory now has 3 iron ore, …

Updated Plan

𝑷

𝒕

:

3

1

3

1

1

3

Explanation:

Because smelting       needs

to use

and

, which I do not have.

So I need to craft

first.

Task Finished !

Agent

Env

𝑎

%

𝑠

%

Mine acacia wood

Craft acacia planks

Craft crafting table

Craft wood pickaxe

Craft stick

Mine cobblestone

Craft stone pickaxe

Mine diamond

Mine coal

Mine iron ore

Craft furnace

Smelt iron ingot

Craft iron pickaxe

Mine birch wood

Mine oak wood

Craft birch planks

Craft oak planks

米K个一个

→→

>→>→

shot 3 LLM-based planner can finish all tasks within a limited horizon (3000-12000 steps for different tasks). We find DEPS outperforms all language planner baselines by nearly doubling the overall success rate, with the same initial state and goal-conditioned controller. Our ablation and exploratory studies then explain how our approach beats the counterparts and becomes the first planning-based agent that accomplishes the challenging ObtainDiamond task. DEPS does not require any planning training for the environment. Additionally, DEPS achieves between on-par and more than 50% relative improvement over existing or concurrent LLM-based planning methods on non-open-ended robotics domains such as ALFWorld [41] and Tabletop environments [40].

2 Background

We aim to develop an agent capable of solving long-horizon goal-reaching tasks using image observations and language goals. To accomplish this, we propose a combined approach involving goal-conditioned policies (termed controllers) and a planner. The goal-conditioned policies are trained to complete sub-goals, while the planner decomposes long-horizon tasks into a series of K short-horizon sub-goals, g 1 , . . . , g K , to be executed by the controller. At each time step t , the goal-conditioned policy π ( a t | s t , g k ) generates an action a t based on the current state s t and the specified sub-goal g k .

Planning with Large Language Models Previous works have shown that LLMs such as InstructGPT [32] and Codex [8] can be used as zero-shot planners to generate sub-goal sequences for various tasks in embodied environments [16, 42]. Formally, given the task description T as prompt p , LLM acts as a planner to decode T into K sub-goals, g 1 , . . . , g K , which are then executed one by one by the low-level controller π ( a t | s t , g k ) to accomplish the task.

However, the above pipeline suffers from both challenges identified in Section 1. Regarding the first challenge, the probability of generating a flawless plan directly from the task description decreases significantly as the required number of sub-goals increases. Moreover, even when the LLM generates a correct plan, it is very likely that the plan is highly inefficient given the agent's current state (challenge #2). Prior works mostly focus on solving the first challenge by providing environmental feedback to the LLM through affordance functions [4], success detector [20] or scene descriptor [17]. However, although these approaches work well on many non-open-ended domains, they still suffer from high failure rates in open-world environments.

3 Towards Reliable Planning in Embodied Open-World Environments

In this section, we first give an overview of our proposed interactive planning framework 'Descibe, Explain, Plan, and Select' (DEPS) for solving complex and long-horizon tasks in open-world environments (Sec. 3.1). Next, in Section 3.2, we elaborate how DEPS iteratively refines its plan to combat the first identified challenge. Section 3.3 introduces the selector module that is used to identify efficient plans in response to the second identified challenge.

3.1 DEPS Overview

As demonstrated in Figure 2, our agent (DEPS) consists of an event-triggered Descriptor, a Large Language Model (LLM) as Explainer and Planner, a goal Selector based on horizon prediction and a goal-conditioned controller. In the following, we use Minecraft as a running example to better elaborate our agent. Note that DEPS can be directly applied to other (non-)open-ended tasks.

We take a large language model (LLM) as a zero-shot planner of the agent to complete tasks. Given a goal command (e.g., ObtainDiamond ) as task T , the LLM-based planner decomposes this high-level task into a sequence of sub-goals { g 1 , . . . , g K } , as the initial plan P 0 . The goals are instructions in natural language, such as mine oak wood (in Minecraft), find two cups (in ALFWorld), put block A on top of block B (in Tabletop Manipulation).

As described in Section 2, a controller is then invoked to execute the provided sub-goals sequentially through a goal-conditioned policy π ( a | s, g ) . However, the initial plan provided by the planner often contains errors, which results in execution failures of the controller. For example, the goal can not be finished only with a wooden pickaxe as shown in Figure 2. When failure pops up, the descriptor will summarize the current state s t and execution outcome of the most recent goal into text d t and send it to the LLM. The LLM will first try to locate the errors in the previous plan P t -1 by self-explanation , e.g., the goal need to be executed with a stone pickaxe . Then it will re-plan the current task T and generate a revised plan P t according to the explanation. In this process, the LLM is also treated as an explainer in addition to the planner role. The Descriptor, Explainer, and Planner will be detailed in Section 3.2.

As shown in Equation (1), DEPS will iteratively update the plan P t until the task is finished, where f DESC is the descriptor model, f LM denotes the language model as explainer and planner, f S is the selector model, π is goal-conditioned policies from the controller.

To filter out inefficient plans, the selector is trained to predict the number of time steps remaining to achieve every goal g k in a set of parallel goals given the current state s t . When the generated plan contains alternative routes, the selector uses this information to choose a suitable goal as the current goal g t . For example, the horizon predicted by the selector of goal acacia tree is less than goal oak tree in Savanna biome, which leads to chop acacia tree as current goal g t .

3.2 Describe, Explain and Plan with LLM Generates Executable Plans

Current LLM-based planners usually query the LLM once at the beginning of every episode and use the output plan throughout the episode [16, 42]. However, as demonstrated by Figure 1, such one-shot planning methods often fail on long-horizon tasks that require many sub-goals. This is caused by two major issues. First, since the correct plan for long-horizon tasks needs to respect various complex preconditions, it is extremely hard for the LLM to generate a flawless plan directly from the task instructions, resulting in failure when simply following the initial plan. Additionally, due to the unpredictable transition dynamics, some incidents may happen during the execution and make the initial plan non-executable. To remedy these problems, existing methods introduce feedback (e.g.,

Prompt 1 Planner prompt template, Python-like code

from success detector or scene descriptor) to reflect on the results of previous executions [17, 20, 4]. However, merely informing the LLM whether a sub-goal is completed is often insufficient to correct the planning error.

To remedy this, we propose 'describe, explain and plan', a new interactive planning method to generate more executable and explainable plans. We start with rewriting the prompt into an interactive dialogue format as in ChatGPT [32] so that subsequent feedback can be passed to the LLM effectively. The produced plan is also augmented with the preconditions and effects of each goal. The structured prompt improves the readability and interpretability of the plan and facilitates error-locating when the execution fails later, as demonstrated in Prompt 1.

The descriptor will then collect the feedback generated by the agent during the execution of the task. The feedback can be practically obtained either by a person (human feedback [4]), or by a pre-trained vision-language model CLIP [35]. While the previous type of feedback needs intensive human involvement, the latter from the pre-trained model needs to be fine-tuned for the specific domain, which decreases the automation and generalization of the agent. On the contrary, Minecraft returns the 'info' and other high-level observations (such as biome, GPS, and compass), we can easily translate the unstructured information into structured language. Therefore we take the symbolic information available in the game and translate it into feedback description d t in this work. To avoid carrying unrelated information in the prompt, we further distill plan-related messages (e.g., inventory information, biome) as final event-level description d t as demonstrated in Figure 2.

Notably, we also treat the LLM as an explainer to explain why the previous plans P t -1 failed. Specifically, by analyzing the current state from description d t and precondition of current goal g t , the explainer can identify the reason why the current goal cannot be executed successfully. As shown in Figure 2, the reason may be the current goal requires the use of an iron pickaxe, but the tool is not prepared in advance, or the current goal requires the use of 3 planks, but the currently available planks are not enough . To implement this, we provide few-shot demonstrations to the LLM as in chain-of-thoughts prompting [45], as shown in Prompt 1. Finally, the LLM goes back to its role as a planner and re-plans the task with the explicit explanation of existing bugs in the previous plan P t -1 , ultimately generating an updated plan P t according to the explanation.

3.3 Horizon-Predictive Selector Yields Efficient Plans

Due to the abundance of objects and the compositional nature of their functionalities, there often exist multiple feasible plans to complete a task, i.e., there are usually multiple paths for the completion of a particular goal. However, despite the feasibility of all such plans, most of them are highly inefficient to execute in the current episode. For example, as shown in Figure 2, obtaining a wood can be done by chopping oak trees , birch trees , or acacia trees . But only oak trees are available in the plains biome. So the planner needs to choose oak trees since it is more efficient, as the agent does not need to travel to another biome.

On the other hand, there is no strict sequential requirement for some goals in the plan P t , i.e., g i , g j ∼ P t enjoy the same precondition, which means g i and g j can be executed in any order. As shown in Figure 1, the choice of different paths (sequences) may affect the execution efficiency of the plan P t as one goal might be closer to the agent. Always choosing the closer goal to execute first could yield more efficient plans and improve the final success rate under a limited episode length. Moreover, the dynamic nature of open-world environments further amplifies the impact of efficient

4999800000

Goal : Meat*3 Candidate Skill : Kill Sheep OR Cow OR Pig Selection : Kill Sheep Explanation : Meet sheep first.

CC4444444A

Goal : Coal*1 AND Iron_Ore*1 Candidate Skill : Mine Coal AND Iron_Ore Selection : Mine Iron_Ore Explanation : Meet iron_ore first.

Goal : Survive in Night. Candidate Skill : Sleep in bed OR Dig down. Selection : Sleep_in_bed Explanation : Village has beds.

plans on the success rate. For example, in Minecraft, if the agent chooses to execute a further goal like collect wood first, the much closer target sheep may disappear and be hard to find again.

In order to improve the efficiency of our plans, we propose to use a selector that selects the most efficient path with the highest execution success rate as the final plan. Specifically, we design a state-aware selector to choose the nearest goal under state s t as the current goal g t from the candidate goal sets in plan P t . It predicts the goal distribution p ( g t | s t , P t ) under the current state s t and plan P t , where g t ∈ G t , G t describes all current executable goals in P t . A straight way to implement the selector is to leverage the semantic similarity between the current state and the goal text using a vision-language model (VLM) such as CLIP [35]. Nevertheless, this may not exactly reflect the difficulty of completing the goal since VLM lacks practical experience. For example, an 'oak tree' in front of the agent could lead to high semantic similarity for the 'chopping tree' goal, but it may be inefficient to achieve this goal if a canyon is in the middle between the agent and the oak tree.

To mitigate this, we implement a horizon-predictive selector that embeds practical task experience to accurately rank the goals based on their efficiency and feasibility. Here, we define the horizon of a goal h t ( g ) := T g -t as the remaining time steps to complete the given goal, where T g is the time of completing goal g . This metric accurately reflects how quickly we can achieve the given goal from the current state. To estimate the horizon, we learn a neural network µ to fit the offline trajectories by minimizing the entropy loss -log µ ( h t ( g ) | s t , g ) , where h t is the ground-truth horizon in trajectories of completing goal g . Therefore, the goal distribution can be formulated as follows:

We set goal-sensitive Impala CNN [6] as the backbone of the selector. In practice, the horizon predictive selector can be jointly trained with the controller policies and share the backbone parameters [6].

4 Experiments

This section analyzes and evaluates our proposed 'describe, explain, plan, and select" (DEPS) method. To minimize performance variation caused by the low-level controller, we standardize all experiments with one controller learned by behavior cloning. We refer to the details of this controller in Appendix C. In Section 4.1, we introduce our testing environments and our evaluation task set, consisting of the hardest 71 tasks from MCU SkillForgeChain [22]. In Section 4.2, we report our performance in the context of existing LLM-based planners. Ablation studies are conducted in Section 4.3. Finally, we pay close attention to the hardest task, ObtainDiamond , which is long-hailed as a major challenge in the community. The experiments on ALFWorld and Tabletop Manipulation environments are shown in Appendix A.

4.1 Experimental Setup

Environment and Task Setting We first evaluate our proposed method in Minecraft, a popular open-world environment with both challenges discussed in Section 1. For better reflecting the performance of DEPS, we choose three Minecraft environments with different versions for better

evaluation, including Minedojo [10] with Minecraft 1.11.2, MineRL [3] with Minecraft 1.16.5, and MC-TextWorld [22] with Minecraft 1.19.2. Rules and items have something different in the above three Minecraft environments, which can better evaluate the dynamic and interactive planning abilities of DEPS.

We choose 71 tasks from the Minecraft Universe Benchmark SkillForgeChain [22] for evaluation. These tasks are related to items that can be obtained in the Minecraft overworld. To better present the results, we divide the 71 Minecraft tasks into 8 meta groups according to the ingredients and function of the tasks, i.e., MT1-MT8. The instruction for every task is written in natural language, e.g., make a wooden door in MT1 (Basic group) and obtain a diamond in MT8 (Challenge group), as illustrated in Table 1. Considering how long it typically takes human players to complete each task as a ballpark [14], we set different maximum episode steps for different meta tasks from 3000 (for easiest Basic tasks) to 12000 (for the hardest Challenge tasks). The names, number of required skills, and functions of all tasks are listed in Appendix B. We give an empty inventory for every task in Survival mode and require the agent to obtain every item from the environment by itself. Note that our agent will be summoned in different environments randomly for each evaluation. Biomes and initial positions are also different each time. Following the previous work [18], we take the success rate as the evaluation metric.

Baselines We compare DEPS with other language-based planners, including GPT as Zero-shot Planner(GPT) [16], ProgPrompt(PP) [42], Chain-of-Thought(CoT) [45], Inner Monologue(IM) [17], and Code as Policies(CaP) [20]. For all baseline models, we use the same demonstration example in the prompt, the same LM model from OpenAI, and the same controller in all tasks for a fair comparison. Since these methods were not originally experimented with Minecraft, we reproduce them to conform to the Minecraft specification based on prompt and feedback template design. All planner methods access the LLM model through OpenAI API ( text-davinci-03 model [32] for GPT, CoT, and IM, and code-davinci-02 model [8] for PP, CaP, and Ours). All hyperparameters of LLM (including the temperature and best_of , etc.) are kept as default. We also list the full prompt of all different methods in Appendix G.

4.2 Main Results

Every task is executed 30 times and the average results in Minedojo [10] for every meta task are listed in Table 2. Our approach achieves the best performance with all meta tasks. As the complexity of the task increases from MT1-MT8, the planner usually needs to give more accurate task steps (i.e., longer goal sequence) to achieve the final task. Therefore the success rate of all agents decreases with the reasoning steps increasing. Starting from MT6, almost all existing LLM-based planners fail (nearly 0 success rate). DEP (w/o Selector) already consistently beats existing LLM-based planners in all meta tasks with a significant margin. This validates that 'describe, explain and plan' can estimate the reason for current plan failure and correct the original flawed plans. Due to the limited maximum episode length and restricted control success rate for a hard goal (e.g., Mine diamond with iron_pickaxe ), the final success rate is still capped.

In addition, selector also greatly improves the final task success rate of the agent (from DEP w/o Selector to DEPS ). Hard meta tasks usually require the completion of multiple sub-goals (up to dozens of goals), thus bringing more flexibility and providing more candidate goals for the Selector. At the same time, as the agent conducts experiments with limited episode length, it also places high demands on the efficiency of the plan. Therefore, the Selector brings a significant improvement on efficiency-sensitive tasks such as MT7 (up to +2.7 times success rate).

Robustness on different controller and different Minecraft versions We also evaluate DEPS on MineRL [3] and MC-Textworld [22]. Note that DEPS is a planning method, which needs to equip the goal-conditioned controller for interacting with the Minecraft environments. We choose MC-Controller [6] and Steve-1 [21] as controllers to interact with Minedojo and MineRL, respectively. These two methods are all control policies that perceive visual partial observations and produce mouse and keyboard actions. While MC-Textworld is a text world, which only keeps the Minecraft crafting recipes and mining rules. So MC-Textworld does not require the controller. The DEPS results of the task set MT1-MT8 on different Minecraft environments are shown in Table 3. The results report that DEPS can generate effective plans in various Minecraft environments. The results on MC-Textworld [22] also show that the performance drops on more difficult task sets from MT6 to MT8 are mainly from the controller limitation.

4.3 Ablation Study

We conduct ablation experiments to investigate the number of candidate executable goals for different Selector models and the specific impact of the rounds of DEPS.

4.3.1 Ablation on Selector

We verify the robustness of our proposed Selector under different parallel goals. The agent is asked to complete 2, 3, and 4 candidate goals (the precondition is consistent for all goals), respectively. The goals of the task correspond to different kinds of mobs or materials.

We report the final success rate of our method (DEP) with different selector implementations, including using a fixed sequence of goals, a random sequence of goals, and selecting a goal based on MineCLIP [10], CLIP [35], and our horizon-predictive Selector (HPS). As Figure 4 shows, in one round of parallel candidate goals, an improvement of ∆ =+22.3%, +29.2%, +32.6% is obtained using our horizon-predictive Selector compared to not any selector (i.e., fixed plan), respectively.

At a limited episode length, e.g., 1000 steps, goal-model shows a greater advantage, which proves that goal-model can improve the execution efficiency of the plan in embodied environments. In addition, compared to using vision-language models such as CLIP [35] and MineCLIP [10] as a goal model, horizon-predictive has the best performance due to better estimation of the horizon information. The curve trend also demonstrates that agents with Selector scale up under large amounts of goals in an open-world environment.

Maximum Episode Length

Success Rate

Selector Model

HPS

CLIP

MineCLIP

Fixed

Random

Parallel Goals = 2

Parallel Goals = 3

Parallel Goals = 4

Success Rate

Goals Setting

Biome:

Plains

2:

3:

4:

0.8

0.6

0.4

0.2

0.0

0

500

1000

15002000

2500

0000

0

500

1000150020002500

3000

0.8

0.6

0.4

0.2

中正

0.0

0

500

10001500200025003000

4.3.2 Ablation on Re-Planning Rounds

We evaluate our agent on all tasks with increasing maximum rounds of DEPS. The round is defined as a cycle of interactive LLM-based planning with description, explanation, and planning and selecting, i.e., an updated plan. All tasks for every maximum round are executed 30 times and the average success rate is reported in Table 4. We take the vanilla LLM planner as the baseline, in which the model takes the initially generated plan as the final execution plan, without involving any description, re-planning, or self-explanation processes during the task execution. Our results in the previous subsection utilize the maximum rounds possible under maximum tokens capped by OpenAI. We also report the success rate increment from vanilla planner to DEPS of every meta task in column ∆ in Table 4. This set of experiments demonstrates that DEPS can iteratively improve its plan in open-world environments. More description, self-explanation, and re-planning rounds produce better results, especially for hard tasks.

4.4 ObtainDiamond Challenge

Mining diamonds in the open-world game Minecraft, i.e. MT8 in Table 2, has been a long-standing challenge for the community [14]. It is challenging because mining diamonds from scratch in Minecraft involves acquiring a sequence of difficult-to-obtain items that require complex planning on goals like mining, inventory management, crafting with and without a crafting table, tool use, smelting iron ingot in a furnace, and mining at the lowest depths. We take the ObtainDiamond task as a bonus experiment to show the capabilities of our zero-shot planner on complex tasks in embodied environments. Previous methods' success rates on this challenge further vouch for its difficulty. [43, 34] leverages domain-secific reward functions and RL fine-tuning to achieve ∽ 0.1% success rate in 15 minutes of game play. VPT further boosts the success rate to 20% within 20 minutes of play through pre-training on collects ∽ 70k hours human demonstrations and finetuning with human-designed reward function [3]. DreamerV3 is trained from scratch to collect diamonds in a modified Minecraft environment (easier to break blocks) with world models to achieve a success rate of 2% [15].

Our DEPS manages to achieve on-par performance in this grand challenge; our agent achieves a 0.59% success rate within 10 minutes of gameplay. Note our method does not specifically fine-tune for this challenge. It is designed to be multi-task in its nature. Furthermore, considering our planner operates with demonstration prompts on a fixed Large Language Model, it can be straightforwardly adapted to other open-ended environments with modifications.

5 Related Works

Task planning with LLMs There have been some methods leveraging the large language model to generate action plans for high-level tasks in embodied environments [46, 9, 11]. [16] decompose natural language commands into sequences of executable actions by text completion and semantic

translation, while SayCan generates feasible plans for robots by jointly decoding an LLM weighted by skill affordances from value functions [4]. For better executing the plan in embodied environments, some methods use an object detector describing the initial environment into the language prompt to produce environment-suitable plans and adopt success detectors to check that each step is executed successfully [17, 20]. [42] and [20] use the pythonic-style prompt to produce more executable plans. However, all of the above methods assume that the initial plan from the LLM is correct. When there are bugs in the initial plan, it's difficult for the agent to finish the task successfully.

Interactive Planning with LLMs Inner Monologue [17] pilots the front of interactive planning with LLMs, which introduces the feedback (including success detection and scene description) to the planner. However, we found it could still suffer from accumulative planning error, especially in long-horizon open-world tasks. Rather, our ' Describe, Explain, Plan and Select ' (DEPS) method can produce more reliable plans by leveraging chain-of-thought thinking and explanation to locate the errors in previous plans. Moreover, we also propose a goal Selector to further improve the efficiency of the plan, thereby yielding much better performances. Readers are encouraged to refer to the comparative results in Section 4.2 between DEPS and these prior arts. There are also some concurrent works on planning with LLMs [39, 26, 23, 33, 47].

Agents in Minecraft Some previous works have employed the hierarchical architecture to solve long-horizon tasks in Minecraft [30, 27, 24]. Recently, based on the internet-scale corpus, [10] pre-trains a language-conditioned reward function and learns multi-task MineAgent. [3] collects a vast amount of human demonstrations to train a behavior cloning agent. More recently, [15] utilized a learned world model to distill a policy that can efficiently explore in Minecraft. There are also some works focus on learning goal-conditioned policies for better instruction-following [6, 7, 21]. While these efforts all focus on improving the low-level controller. Rather, the planner in our architecture emphasizes applying domain knowledge to propose and arrange the sub-goals. It significantly influences the complexity and breadth of tasks that the agent can handle. Moreover, our planner is zero-shot, making it possible to generalize to other long-horizon open worlds.

6 Limitations

Albeit the impressive results of our approach, we believe there are at least two major limitations within our approach. First of all, our framework relies on privately-held LLMs like GPT-3 and ChatGPT, which makes it less accessible to those who cannot afford or access the service. However, we're fully committed to ensuring a more democratized method and will explore using open-sourced models including OPT [48] and BLOOM [38]. Another issue is the explicit step-by-step planning in our system. Although it brings us superior performances over the baselines, the planning bottleneck can also prevent our model from being further scaled up. A more appealing approach will be amortizing the planning within an end-to-end trainable goal-conditioned policy, which is worth exploring next. Furthermore, some previous fundamental challenges in planning (e.g., dead ends) may not prevalent in our adopted environments and hence could be inadvertently overlooked by our paper. We are dedicated to addressing more fundamental challenges present in building a multi-task generalist agent in our series of following work.

7 Conclusion

We investigate the problem of planning in open worlds. We identify two major challenges unique to these environments: 1) long-term planning requires precise and multi-step reasoning, and 2) planning efficiency could be compromised since canonical planners do not take the agent's proximity to parallel goals/subtasks into consideration. We propose 'Describe, Explain, Plan and Select' ( DEPS ), an interactive approach based on Large Language Models (LLMs) to tackle them both. Our experiments in the challenging Minecraft domain verify the advantages of our approach over counterparts by marking the milestone of robustly accomplishing 70+ Minecraft tasks and nearly doubling the overall performances. DEPS also is the first planning-based agent that can reach the diamond in this game.

References

J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y . Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, et al. Flamingo: a visual language model for few-shot learning. arXiv preprint arXiv:2204.14198 , 2022. 1

P.-L. Bacon, J. Harb, and D. Precup. The option-critic architecture. In Proceedings of the AAAI conference on artificial intelligence , 2017. 1

B. Baker, I. Akkaya, P. Zhokhov, J. Huizinga, J. Tang, A. Ecoffet, B. Houghton, R. Sampedro, and J. Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos. arXiv preprint arXiv:2206.11795 , 2022. 7, 8, 9, 10

A. Brohan, Y. Chebotar, C. Finn, K. Hausman, A. Herzog, D. Ho, J. Ibarz, A. Irpan, E. Jang, R. Julian, et al. Do as i can, not as i say: Grounding language in robotic affordances. In 6th Annual Conference on Robot Learning , 2022. 1, 2, 3, 5, 10, 21

T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020. 1, 3, 15, 21, 22

S. Cai, Z. Wang, X. Ma, A. Liu, and Y. Liang. Open-world multi-task control through goal-aware representation learning and adaptive horizon prediction. arXiv preprint arXiv:2301.10034 , 2023. 6, 8, 10, 21, 24

S. Cai, B. Zhang, Z. Wang, X. Ma, A. Liu, and Y. Liang. Groot: Learning to follow instructions by watching gameplay videos. arXiv preprint arXiv:2310.08235 , 2023. 10

M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 , 2021. 3, 7, 21, 22

I. Dasgupta, C. Kaeser-Chen, K. Marino, A. Ahuja, S. Babayan, F. Hill, and R. Fergus. Collaborating with language models for embodied reasoning. In NeurIPS Foundation Models for Decision Making Workshop , 2022. 9

L. Fan, G. Wang, Y. Jiang, A. Mandlekar, Y. Yang, H. Zhu, A. Tang, D.-A. Huang, Y. Zhu, and A. Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowledge. Advances in Neural Information Processing Systems Datasets and Benchmarks , 2022. 1, 2, 7, 8, 10, 18, 21, 26

R. Gong, Q. Huang, X. Ma, H. Vo, Z. Durante, Y. Noda, Z. Zheng, S.-C. Zhu, D. Terzopoulos, L. Fei-Fei, et al. Mindagent: Emergent gaming interaction. arXiv preprint arXiv:2309.09971 , 2023. 9

W. H. Guss, M. Y. Castro, S. Devlin, B. Houghton, N. S. Kuno, C. Loomis, S. Milani, S. P. Mohanty, K. Nakata, R. Salakhutdinov, J. Schulman, S. Shiroshita, N. Topin, A. Ummadisingu, and O. Vinyals. The minerl 2020 competition on sample efficient reinforcement learning using human priors. arXiv: Learning , 2021. 2

W. H. Guss, C. Codel, K. Hofmann, B. Houghton, N. Kuno, S. Milani, S. Mohanty, D. P. Liebana, R. Salakhutdinov, N. Topin, et al. Neurips 2019 competition: the minerl competition on sample efficient reinforcement learning using human priors. arXiv preprint arXiv:1904.10079 , 2019. 2

W. H. Guss, B. Houghton, N. Topin, P. Wang, C. Codel, M. Veloso, and R. Salakhutdinov. Minerl: A large-scale dataset of minecraft demonstrations. arXiv preprint arXiv:1907.13440 , 2019. 2, 7, 9, 24

D. Hafner, J. Pasukonis, J. Ba, and T. Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104 , 2023. 9, 10

W. Huang, P. Abbeel, D. Pathak, and I. Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. ICML , 2022. 3, 4, 7, 8, 9, 15, 16, 17, 21, 24, 25, 26, 28

W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. Florence, A. Zeng, J. Tompson, I. Mordatch, Y. Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.05608 , 2022. 2, 3, 5, 7, 8, 10, 15, 16, 17, 24, 25, 26, 30, 34, 35

M. Johnson, K. Hofmann, T. Hutton, and D. Bignell. The malmo platform for artificial intelligence experimentation. In Ijcai , pages 4246-4247. Citeseer, 2016. 2, 7

A. Kanervisto, S. Milani, K. Ramanauskas, N. Topin, Z. Lin, J. Li, J. Shi, D. Ye, Q. Fu, W. Yang, W. Hong, Z. Huang, H. Chen, G. Zeng, Y. Lin, V. Micheli, E. Alonso, F. Fleuret, A. Nikulin, Y. Belousov, O. Svidchenko, and A. Shpilman. Minerl diamond 2021 competition: Overview, results, and lessons learned. neural information processing systems , 2022. 2

J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng. Code as policies: Language model programs for embodied control. arXiv preprint arXiv:2209.07753 , 2022. 2, 3, 5, 7, 8, 10, 25, 26, 31

S. Lifshitz, K. Paster, H. Chan, J. Ba, and S. McIlraith. Steve-1: A generative model for text-to-behavior in minecraft. arXiv preprint arXiv:2306.00937 , 2023. 8, 10

H. Lin, Z. Wang, J. Ma, and Y. Liang. Mcu: A task-centric framework for open-ended agent evaluation in minecraft. arXiv preprint arXiv:2310.08367 , 2023. 6, 7, 8, 18

K. Lin, C. Agia, T. Migimatsu, M. Pavone, and J. Bohg. Text2motion: From natural language instructions to feasible plans. arXiv preprint arXiv:2303.12153 , 2023. 10

Z. Lin, J. Li, J. Shi, D. Ye, Q. Fu, and W. Yang. Juewu-mc: Playing minecraft with sampleefficient hierarchical reinforcement learning. arXiv preprint arXiv:2112.04907 , 2021. 10

X. Ma, S. Yong, Z. Zheng, Q. Li, Y. Liang, S.-C. Zhu, and S. Huang. Sqa3d: Situated question answering in 3d scenes. arXiv preprint arXiv:2210.07474 , 2022. 1

J. Mai, J. Chen, B. Li, G. Qian, M. Elhoseiny, and B. Ghanem. Llm as a robotic brain: Unifying egocentric memory and control. arXiv preprint arXiv:2304.09349 , 2023. 10

H. Mao, C. Wang, X. Hao, Y. Mao, Y. Lu, C. Wu, J. Hao, D. Li, and P. Tang. Seihai: A sampleefficient hierarchical ai for the minerl competition. In Distributed Artificial Intelligence: Third International Conference, DAI 2021, Shanghai, China, December 17-18, 2021, Proceedings 3 , pages 38-51. Springer, 2022. 10

S. Min, X. Lyu, A. Holtzman, M. Artetxe, M. Lewis, H. Hajishirzi, and L. Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202.12837 , 2022. 23

V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 , 2013. 2

J. Oh, S. Singh, H. Lee, and P. Kohli. Zero-shot task generalization with multi-task deep reinforcement learning. In International Conference on Machine Learning , pages 2661-2670. PMLR, 2017. 10

OpenAI. Gpt-4 technical report, 2023. 21, 22

L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155 , 2022. 3, 5, 7, 8

J. S. Park, J. C. O'Brien, C. J. Cai, M. R. Morris, P. Liang, and M. S. Bernstein. Generative agents: Interactive simulacra of human behavior. arXiv preprint arXiv:2304.03442 , 2023. 10

V. P. Patil, M. Hofmarcher, M.-C. Dinu, M. Dorfer, P. M. Blies, J. Brandstetter, J. A. ArjonaMedina, and S. Hochreiter. Align-rudder: Learning from few demonstrations by reward redistribution. ICML , 2020. 9

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning , pages 8748-8763. PMLR, 2021. 5, 6, 8

S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikov, G. Barth-Maron, M. Gimenez, Y. Sulsky, J. Kay, J. T. Springenberg, et al. A generalist agent. arXiv preprint arXiv:2205.06175 , 2022. 1

N. Reimers and I. Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084 , 2019. 23

T. L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ili´ c, D. Hesslow, R. Castagné, A. S. Luccioni, F. Yvon, M. Gallé, et al. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100 , 2022. 10

N. Shinn, B. Labash, and A. Gopinath. Reflexion: an autonomous agent with dynamic memory and self-reflection. arXiv preprint arXiv:2303.11366 , 2023. 10

M. Shridhar, L. Manuelli, and D. Fox. Cliport: What and where pathways for robotic manipulation. In Conference on Robot Learning . PMLR, 2022. 2, 3, 15, 16, 17, 23

M. Shridhar, X. Yuan, M.-A. Côté, Y. Bisk, A. Trischler, and M. Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. arXiv preprint arXiv:2010.03768 , 2020. 2, 3, 15, 16

I. Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg. Progprompt: Generating situated robot task plans using large language models. arXiv preprint arXiv:2209.11302 , 2022. 3, 4, 7, 8, 10, 25, 26, 28

A. Skrynnik, A. Staroverov, E. Aitygulov, K. Aksenov, V. Davydov, and A. I. Panov. Forgetful experience replay in hierarchical reinforcement learning from expert demonstrations. Knowledge-Based Systems , 218:106844, 2021. 9

H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023. 22

J. Wei, X. Wang, D. Schuurmans, M. Bosma, E. Chi, Q. Le, and D. Zhou. Chain of thought prompting elicits reasoning in large language models. 36th Conference on Neural Information Processing Systems (NeurIPS 2022) , 2022. 5, 7, 8, 15, 25, 26, 29

A. Zeng, A. Wong, S. Welker, K. Choromanski, F. Tombari, A. Purohit, M. Ryoo, V. Sindhwani, J. Lee, V. Vanhoucke, et al. Socratic models: Composing zero-shot multimodal reasoning with language. arXiv preprint arXiv:2204.00598 , 2022. 2, 9

C. Zhang, K. Yang, S. Hu, Z. Wang, G. Li, Y. Sun, C. Zhang, Z. Zhang, A. Liu, S.-C. Zhu, et al. Proagent: Building proactive cooperative ai with large language models. arXiv preprint arXiv:2308.11339 , 2023. 10

S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068 , 2022. 10

Contents

Appendix

A Additional Experiments

Additional experiments are conducted on the ALFWorld [41] and Tabletop Manipulation environments [40] to showcase the generalization capabilities of DEPS.

A.1 ALFWorld

ALFWorld [41] is an interactive learning environment that aligns text and embodiment, allowing agents to acquire abstract, text-based policies in TextWorld, and subsequently execute goals from the ALFRED benchmark in a visually rich environment.

A.1.1 Tasks

The ALFWorld framework contains six types (namely Pick & Place , Examine in Light , Clean & Place , Heat & Place , Cool & Place , Pick Two & Place ) of tasks with various difficulty levels. Tasks involve first finding a particular object, which often requires the agent to open and search receptacles like drawers or cabinets. Subsequently, all tasks other than Pick & Place require some interaction with the object such as heating (place the object in a microwave and start it) or cleaning (wash the object in a sink). To complete the task, the object must be placed in the designated location. We sample 10 tasks from ALFWorld randomly and list all the task names, types, and the number of receptacles in Table 5. We classify them into 6 groups based on their functionality. For all tasks, the maximum number of steps is set as 50.

We select the GPT as Zero-Shot Planner (GPT) [16] and Inner Monologue (IM) [17] as baseline methods. For the Inner Monologue, the planning goal is the next goal among all candidate goals. For the GPT and DEP, which produce the full plan at once, the planning goal is the full plan (a goal sequence). Then the plan will be executed step-by-step, i.e., the current goal will be given to the controller and select suitable action according to the current state. The goal termination module is also employed with the LLM. For better demonstrate the effectiveness of self-explanation in DEP, we also augment the zero-shot planner with re-planning ability (GPT+RP). All planner methods access the LLM model through OpenAI API ( text-davinci-03 model [5]). Since ALFWorld is a text world, the environment will be given a literal description and candidate language-conditioned actions for each state, so the controller under ALFWorld is also LLM-based. Chain-of-Thought [45] is also employed in the controller for better decision-making. All prompts for planner and controller in ALFWorld are listed in Section G.2.

A.1.2 Results

Each task is executed five times, and the average results for each task group are presented in Table 6. BUTLER is the a training-based method, the results are sourced from [41]. Re-planning is a crucial capability in complex and exploratory environments. The short-horizon planning approach (IM) with

initial frame

Goal : pick a soapbottle at cabinet 4 Action : go to cabinet 4

Goal : pick a soapbottle at cabinet 4 Action : open cabinet 4

Goal : pick a soapbottle at cabinet 4 Action : take soapbottle 2 from cabinet 4

Goal : put a soapbottle on garbagecan at gar- bagecan 1 Action : go to garbage- can 1

Goal

: put a soapbottle

on garbagecan at gar-

bagecan 1

Action

: put soapbottle

2 in/on garbagecan 1

Task : put some soapbottle on garbagecan

厂

re-planning capability outperforms the long-horizon planning approach (GPT) without re-planning capability with a large margin. Furthermore, the long-horizon planning method augmented with re-planning capability (GPT+RP) achieves superior performance ranging from 10% (GPT) to 52%. DEP further enhances the feasibility of planning with descriptions and self-explanation. Notably, all planning methods fail on Place Two & Place tasks, which is attributable to LLM's lack of requisite knowledge for this task. It is worth investigating how to effectively incorporate the distinctive knowledge of an environment into LLM.

A.2 Tabletop Manipulation

The Tabletop Manipulation experiments are conducted on a Universal Robot UR5e with a suction gripper in the simulated environments [40].

A.2.1 Tasks

The assessment of all methods is conducted in five seen tasks, as illustrated in Table 7, wherein the seen tasks are employed for training the CLIPort [40] as the controller. The task involves a robotic arm equipped with a gripper, which is tasked with rearranging a number of blocks and bowls on a table to achieve a desired configuration specified via natural language (e.g., "putting the blocks in the bowls with matching colors").

Task:assembling-kits

initial frame

1.put the cyan letter

2. put the green

3.put the blueletter

4. put the brown ring

5.put the red letterE

L shape in the letter L

hexagon in the

M shape in the letter

in the ring hole

shape in the letter E

shapehole

M shape hole

shape hole

Task: put-block-in-bowl

initial frame

1. put the blue blocks

2. put the red blocks

3. put the yellow

in a blue bowl

in a red bowl

blocks in a yellow

bowl

Task:packing-shapes

initial frame

1. pack the unicorn

2. pack the

3.pack the android

4. pack the lion figure

5. pack the alarm

toy in the brown box

butterfinger chocolate

toy in the brown box

in the brown box

clock in the brown

in the brown box box

We utilized Inner Monologue (IM) [17] and Zero-shot Planner (GPT) [16] as planning baselines, in addition to comparing with a multi-task CLIPort policy directly trained on long-horizon task instructions (i.e., without utilizing LLM for planning). As CLIPort is a single-step policy that does not spontaneously terminate during policy rollout, we report CLIPort evaluations with Oracle termination (i.e., repeat until the Oracle indicates task completion) and fixed-step termination (i.e., repeat for k steps). For Inner Monologue, which directly produces the next-step goal and terminates when the LLM ceases to generate new steps, we similarly set the maximum number of steps to be k for practical considerations. For the zero-shot planner [16] and our DEP, which produce the full plan at once, they are augmented with the LLM-based termination. DEP also involves the description, explanation, and re-planning process. The same k step is suitable for these two methods. In practice, k is set as 15. The prompts for all methods are listed in Section G.3. We use the checkpoints provided by CLIPort as the controller and all planner methods access the ChatGPT (as LLM) through OpenAI API ( gpt-3.5-turbo model). Each task is evaluated 5 times with different seeds.

A.2.2 Results

The results of each method are listed in Table 8. All LLM-based planning methods perform well on tabletop rearrangement tasks. Given the compact nature of the tabletop environment, the performance gap among the various LLM-planning methods is not as pronounced as in the open-ended Minecraft. This observation underscores the robust generalization capabilities of LLM-based planning methods across diverse environments.

B Minecraft Task Details

To fully validate the multitask planning and execution capability of our agent, we choose over 70 tasks from the Minecraft Universe Benchmark [22] as the set of evaluation tasks. These tasks are related to items that can be obtained in the Minecraft overworld. These tasks are also a subset of MineDojo [10] programmatic tasks. Minedojo exists some programmatic tasks sharing the same object item given different conditions (e.g., obtain wool given shear or obtain wool given nothing). Minedojo expands the richness of the same tasks (sharing the same Minecraft item as an object) by giving different initial conditions (e.g., obtain wool given shears or obtain wool given nothing ). We keep only the 71 hardest conditions (i.e. given nothing ) as tasks.

We list all task names, objects, and their required skills number for planning from Table 9 to Table 16. Object item is used as the basis for the successful completion of the task. These objects cannot be obtained directly from the environment, and usually require multiple goals (i.e., reasoning steps) to be constructed. Here we only consider the number of required goal types, and multiple identical goals are unified into 1 reasoning step. Note that the reasoning steps for each task are not fixed, and as the initial state of the agent and the biome is in change, more reasoning steps may be required to complete it, we only report the most basic case here.

As shown in Figure 4, for each task, a relaxed (longer) maximum episode steps will increase the success rate of the task. To fully test the efficiency of our method, we set an upper limit on the episode length for each task. Since different tasks have different difficulty levels, we double the average completion time of human players for different meta-tasks as the upper limit of the episode. The play time are computed as corresponding maximum steps (i.e., Max. Steps in Table 1) of episode length at 20Hz.

C DEPS Implementation Details

We study three different implementations of DEPS for each of the experimental settings. While each version incorporates description and self-explanation to improve planning of LLM, there are differences in the internal components of each system, as seen in Table 17.

C.1 Controller

As the name implies, tasks in Minecraft are usually related to mine and craft goals. Mine goals require the agent to collect raw materials from the environment using the appropriate tools. Craft goals ask the agent to synthesize using existing materials. Any raw material used requires the agent to collect through suitable tools (e.g., diamonds can only be collected by an iron pickaxe or a better pickaxe). So a task usually requires dozens of step-by-step mine and craft goals, as the required skills in Table 9. Note that the successful execution of a task needs to satisfy certain exact numerical constraints due to the presence of strict generation recipes in the environment (e.g., a log can craft 4 planks, so harvesting 6 planks requires at least 2 logs). When the number of materials collected is not enough, the goal cannot be completed successfully. When more materials are collected than actually needed, the execution success rate of the task could also be reduced because the plan can not be finished under the maximum action steps.

We designed the agent's skill space based on these goals, as shown in Table 18, with a total of 262 goals. Every goal is designed with an objective item (e.g., 1 minecraft:cobblestone for skill ' Mine 1 cobblestone with pickaxe '), which is used to evaluate the achievement of the goal. The skill, as a goal-conditioned policy π ( a | s, g ) for decision-making, maps the current state s and goal g to action a . The goal is specified as natural language instructions here, which is similar to [4].

When training the controller, we adopt the observation space provided by MineDoJo [10], which includes an RGB camera view, yaw/pitch angle, GPS location, and the type of 3 × 3 blocks surrounding the agent. We discretize the original multi-discrete action space provided by MineDojo into 42 discrete actions. We use the proposed imitation learning method proposed by [6] in training. To be specific, a modified goal-sensitive Impala CNN is used as the backbone network. The success rate under a fixed episode length of every skill is listed in Table 18.

C.2 LLMas Planner

DEPS relies on Large Language Models (LLMs) to generate language-based plans. In our Minecraft experiment, we chose Codex [8] as the LLM Planner because it can accept longer input tokens and is cost-effective. However, DEPS is compatible with various types of LLMs. Therefore, we used GPT3 [5] and ChatGPT as LLM Planners in the ALFWorld and Tabletop Manipulation experiments, respectively. Due to the effective planning and error correction performance of DEPS, the initial plan generated by the LLM has little impact on the final performance of the Agent. We also conduct ablation experiments on even if the initial plan generated by the LLM has low accuracy, DEPS can generate a final feasible plan through self-explanation and re-planning. Therefore, we conducted ablation experiments on LLM in Minecraft.

We choose Codex [8], ChatGPT, GPT3 [5], and recent GPT-4 [31] as Planners. We used Vanilla Planner [16] as baselines and excluded the re-planning process. Given the same prompt with DEPS, the performance of baseline models reflects the planning ability of different LLMs. The success rate of baseline and DEPS on different LLMs are reported in Table 19.

The success rate of Vanilla Planner varies on the LLMs. The GPT-4 baseline achieved an initial plan accuracy twice as high as the baselines on other LLMs, demonstrating superior planning ability. After being augmented by Descriptor, Explainer, and Selector, DEPS based on different LLMs showed almost identical success rates. This indicates that DEPS-augmented LLMs can generate more feasible plans in open-world environments even if the initial plan is less successful.

It is noteworthy that DEPS is constrained by the maximum token limits of various models, which dictate the maximum re-planning rounds that can be supported. Longer re-planning rounds tend to yield superior performance, particularly in long-horizon tasks requiring more skills (in MT6-MT7), as detailed in the Section 4.3.

Since we use pretrained LLM as a planner, it indeed requires exposure to a large amount of Minecraftrelated corpus during the pretraining phase. Considering that Minecraft is one of the most popular games worldwide, there is relatively abundant data about Minecraft available online. We conducted experiments using open-source pretrained LLaMA2-70B on several Minecraft tasks and found that DEPS based on LLaMA2 also performs reliable planning under Minecraft conditions. Considering limited training data used by LLaMA2, we further finetuned an open-source language model (LLaMA2-13B) using Minecraft texts obtained from the internet which exhibited better planning performance. The results are shown in Table 20.

C.3 LLMas Explainer

Given the description and previous plan, the explainer can generate a self-explanation of the failure of the current plan and give instructions to fix the bugs. The explainer is implemented with the OpenAI completion mode based on text-davinci-03 models. The prompt for the explainer is listed in Listing 1.

C.4 Other modules

Goal Parser We need to map the plan expressed in free-form language to the pre-defined controller skills set. We use the LLM as an automatic parser to parse the language plan first. For the goals not following pre-defined code expression, we calculate its semantic distance to the skills by cosine similarity with pre-trained Sentence-Bert model [37] and select the most similar skill as the corresponding goal. All executable goals are listed in Appendix C. The LLM-based parser is general and can be transferred to other domains easily by modifying the prompt. The prompt for Minecraft parser is listed in Listing 2.

Success Detector The successful execution of a plan is contingent upon the agent's perception of the current goal's completion status, which is assessed by the success detector. In Minecraft, agents possess an inventory that contains all pertinent information regarding the agent's current state. Thus, the Success Detector can be implemented by monitoring changes in object information within the item inventory. In other scenarios, we can query the LLM to ascertain whether the agent has accomplished a general goal by describing the agent's current state. Alternatively, in certain environments [40], the execution of a goal is linked to the agent's current reward, signifying that these rewards can serve as automatic success detectors.

Prompt The generalization of the LLM to different tasks relies on well-designed prompts and related demonstrations [28]. Given an instruction command (e.g., ObtainDiamond ) as task T , a prompt generator (ProG) will translate T into prompt text. We also added two DEP examples in the prompt as demonstrations to make the LLM output familiar to the chain-of-thought thinking and structural output. We also design a chain-of-thought code-comments-type planning prompt to better demonstrate the capabilities of LLM. All messages are modified to suitable prompts through the prompt-generator before being input to LLM, including task T and description d t . The full prompt sentences and interaction logs are listed in Appendix H.

D Comparison with other LLM-based Planners

The architectures of the different LLM-based planners are illustrated in Figure 7. Where (b) describes the information in the environment into LLM via scene descriptor and success detector, and directly

(a)

Ours

(b) Short-horizon

(c) Long-Horizon

(d) Long-Horizon

action planning

one-shot planning

planning & re-planning

Scene

Instruction

Instruction

Instruction

Description

Instruction

PLANNER

EXPLAINER

PLANNER

PLANNER

LLM

explanation

LLM

LLM

LLM

PLANNER

LLM

plan

plan

plan

description

goal

Description

Scene

Queue

Detector

Success

SELECTOR

VLM

DESCRIPTOR

VLM

goal

state

goal g(t+1)

L

N

CONTROLL

ER

CONTROLL

CONTROLL

goal&skill

runninglogs

ER

ER

action

obs

action

action

→

obs

4

obs

CONTROLL

obs

ENV

ENV

ER

ENV

ENV

ractiom

plans the next goal/action, (c) is Zero-Shot planner [16], which generates the step-by-step goal sequences as plan and ignores the environment state and execution feedback, (d) is the Zero-Shot planner augmented with textual feedback and re-planning process. DEPS further rethink and explain the feedback of previous plans explicitly with the descriptor and explainer. The LLM-based planner will re-plan the task according to the explanation, as demonstrated in Figure 7(a). In addition, the goal Selector further improves the executability of the LLM plan.

E Discussion on ObtainDiamond Task

mine log

craft planks

craft crafting table

craft stick

craft wooden pickaxe

mine stone

craft stone pickaxe

mine iron ore

craft iron pickaxe

smelt iron ingot

craft furnace

mine diamond

→

As outlined in Section 4.4, ObtainDiamond task is a formidable task within the open-ended Minecraft environment. Given the necessity to explore an infinitely expansive world, an efficient plan can prove advantageous, as shown in Figure 8. The task is allotted a maximum of 12,000 steps to interact with the environment, which is comparable to that of human performance [14]. Rather than manually devising explicit hierarchical rewards, we opt to utilize DEPS for generating a hierarchical plan, which is then transferred to the downstream controller to progressively achieve each goal. When equipped with an Oracle Controller, DEPS yields a success rate of 60% for ObtainDiamond. In our experimentation, we employed Behavior Cloning to train a Controller agent [6]. DEPS+BC Controller achieved a success rate of 0.6% in randomly generated Minecraft worlds. The primary bottleneck impeding overall agent success rate lies within the goal-conditioned Controller, not the plans generated by DEPS. Thus, it is worth exploring the development of a data-efficient Controller capable of accepting Language goals.

Another rationale for using DEPS is that, akin to reality, materials in Minecraft possess quantity constraints, and durability for tools. In ObtainDiamond task, an iron pickaxe is typically insufficient

to support the agent, given the rarity of diamonds within the environment (which are predominantly found between depths of 2-16 layers and appear only 0.0846% of the time). The robust re-planning capabilities of DEPS can facilitate the generation of a feasible plan (initiating with crafting an iron-pickaxe) based on the agent's current state.

Additionally, we report the milestones, which demonstrate the decreasing success rate of subsequent tasks in Figure 9 attributable to the task's inherent complexity and Controller constraints.

100

96.7

89.0

89.0

89.0

86.5

83.1

80

68.7

67.9

Success Rate

60

40

31.

.6

20

7

K

8.8

87

0.6

0

MilestoneItems

F Success Rates of ALL Tasks in Minecraft

We report the complete and detailed success rate table of all tasks for different methods in Table 21, including Zero-shot Planner [16], ProgPrompt [42], Chain-of-Thought [45], Inner Monologue [17], Code as Policies [20], and proposed methods (i.e., DEP w/o Selector, and DEPS).

All tasks are executed for at least 30 times across different world seeds, given the same initial conditions. The birth positions of the world are random according to the seed. The average success rates are listed in Table 21. Our approach is state-of-the-art on almost all tasks, especially on difficult tasks that require more skills.

G Prompt for Different Tasks and Different Methods

G.1 Prompt for Minecraft Tasks

G.1.1 DEPS

G.1.2 Baselines

G.2 Prompt for ALFWorld Tasks

G.2.1 DEPS

G.2.2 Baselines

G.3 Prompt for Tabletop Manipulation Tasks

G.3.1 DEPS

G.3.2 Baselines

H Full Dialogue

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on building an LLM-based planner for sequential decision-making tasks in open-world environments such as Minecraft. It proposes a method called \"Describe, Explain, Plan and Select\" (DEPS), which incorporates multiple stages including LLM describe, LLM explain, planning, and goal selection. The system is evaluated on hundreds of tasks in Minecraft. Several baseline methods are also compared against.\n\nSTRENGTHS:\n- Clear motivation. The proposed method is well motivated from the angles from unique challenges associated with open worlds.\n- Excellent presentation. The paper is overall well presented, with coherent flow and easy-to-read figures that explain key concepts in the method.\n- Thorough evaluation and comparisons. 5 methods proposed in prior work are discussed and compared against in the experiment section.\n- Code is included for reproducibility, although I didn't run it.\n\nWEAKNESSES:\n# Novelty of the proposed method is limited to certain extent\nAs illustrated in Figure 7 in appendix which compares the proposed method against existing methods, the proposed method only introduces three extra modules, namely the \"Explainer\", \"Descriptor\", and \"Selector\". However, their significance or importance are not well justified as explained below.\n\n## On Explainer\nAt L58, the role of Explainer is to locate the errors in the previous plans. However, according to example prompt provided to the Explainer as listed at L523-L549, it is trivial to summarize the errors because necessary information is already provided and well structured. What the Explainer does is merely compare what items are missing given the recipe, and rephrase it to sentences like \"[...] so I need to craft/mine ... first\". This makes the Explainer's necessity inconceivable.\n\n## On Descriptor\nWhile the paper claims the Descriptor to be models such as VLMs (Figure 2 and L120), in fact, it is merely rule-based conversion that maps symbolic environment state information to language (L151-152). Table 9 further provides more evidence to this. Since environments considered in this work usually provide rich privileged information that is critical to accomplish corresponding tasks, it is unclear that whether performance improvements come from the introduced module or simply the access to privileged information.\n\n## On Selector\nWhat is the exact setting used in Figure 4 to verify the effectiveness of Selector? At the current version the setup seems to be too easy to be representative (plain biome with 4 similar goals). The non-trivial performance of the random goal selection baseline also suggests this. Further analysis and insights into why the Selector works would be helpful. It's also good to know performance of the learned goal selector. For example, if you represent the remaining horizon as continuous value, what is the L1 distance between the prediction and ground-truth?\n\n# Potential unfair evaluation and comparisons\n## On main results\nIn Table 2 it’s clear that success rates monotonically improve with the number of rounds. L275-276 says that DEPS uses the maximum possible number of rounds. So it could be the case that baselines are not given the same amount of LLM tokens consumed. Clarification on this will increase the transparency of the main experiment.\n\n## On the obtain diamond challenge\nSince DEPS is based on a powerful LLM that is highly likely to be pre-trained on Minecraft-related corpus, it is not directly comparable to methods mentioned in section 4.4 that are learned without such built-in knowledge. More appropriate comparisons would be conducted against other concurrent LLM-based Minecraft systems such as GITM [Zhu et al., 2023] and Voyager [Wang et al., 2023]. Including the discussion about them is also encouraged. Additionally, VPT's performance is wrongly reported. It should be around 20% for collecting diamonds and 2.5% for collecting diamond pickaxe.\n\n# Highly variant performance for DEPS on challenging tasks\nPerformance of the proposed method is highly variant on challenging tasks, specifically, starting from MT7, where the one-sigma range includes negative success rate. Further statistical analysis or experiments to justify that the task success is not from randomness will be appreciated.\n\n# Inappropriate word choice\nAt L63 the system is described as \"zero-shot\". Analogy to Brown et al., 2020 is drawn to justify the word choice. However, if we follow the same naming convention in Brown et al., 2020, the proposed system would be better described as \"few-shot\" since multiple task-related demonstrations are provided in prompts. Similarly, at L129, \"one-shot planning methods\" should be better called as \"open-loop planning methods\" given the context that there is no re-planning involved in those methods.\n\nOverall, I appreciate the efforts put in this work. I'm happy to raise the score if concerns/questions above can be addressed.\n\n- Zhu et al., Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memory, 2023.\n- Wang et al., Voyager: An Open-Ended Embodied Agent with Large Language Models, 2023.\n- Brown et al., Language models are few-shot learners, 2020.\n\nQUESTIONS:\nSee Weakness section\n\nLIMITATIONS:\nSee Weakness section", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper discusses the problem of planning in Minecraft and proposes an interactive planning approach based on Large Language Models (LLMs) called DEPS. The LLM based planner has the ability to re-plan upon failure and can take distance into consideration.\n\nSTRENGTHS:\n1. To my knowledge, this is one of the first works showing a working embodied agent with LLM planner.\n2. The DEPS architecture is quite reasonable and intuitive, and achieves promising results.\n3. I think this would lead to an exciting direction to work on. Given the popularity and challenge of Minecraft, I believe LLM solutions might achieve generalizable success in other domains.\n4. The evaluation metrics are well-defined, although I hope the authors could link better back to prior works.\n\n[After discussion/rebuttal]\n+ This work is conducted relatively early with codex, which has weaker abilities compared to the latest LLMs. There could be even more potential for improvement.\n+ This work should be reasonably reproducible.\n\nWEAKNESSES:\n1. It is quite confusing that the long standing ObtainDiamond challenge has been re-labeled as MT8 in a lot of result tables\n2. The baselines for comparison in the tables do not seem so convincing. I wonder why VPT results are not included in table 8.\n\n[After rebuttal]\n1 has been resolved.\n2 has mostly been resolved.\n\nSome weakness remains of concern:\n+ The authors were not able to reproduce VPT performance to full extent. From my knowledge, the training of the controller actually depends on demonstration collected from VPT.\n+ One additional concern is the dependence on prior-works/code-bases might make the model hard to adapt/work on. Cai et al.\n\nQUESTIONS:\nOverall, I think this work offers a good contribution, but I hope that the authors would address some questions and concerns for me to change my score:\n\n1. Is it possible to include better baselines, and more details on how the low-level agent is trained/made? From my understanding, this model was built on MineAgent or VPT as low-level agent, but I don't think this was mentioned explicitly.\n2. It seems from table 1 that the agent still struggles to get to diamond. What do you think is the main outstanding difficulty?\n\n[After rebuttal]\nAll questions have been resolved\n\nLIMITATIONS:\nSee Question 1 for limitations.\nSocial impacts are addressed adequately.\n\n[After rebuttal]\nAll concerns have been addressed", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new planning algorithm for solving tasks in open-ended environments, and tests its performance against baselines in Minecraft. The proposed model uses large language models (LLMs) to split the large task into subgoals (for which there are controllers that can complete the subgoals). However, this type of planning often fails, as shown in the results. In addition to the planner, the proposed algorithm summarizes errors when they occur and then update the plan, which improves the results. Finally, the method includes a selector module, which is trained to select the quickest goal out of the available ones, which further improves results, especially in long-term tasks.\n\nSTRENGTHS:\n- Good explanation of the tackled challenge. Figures 1 and 2 are especially descriptive.\n- Good coverage of the baselines and ablations to study what changes matter.\n- The controller models used in the experiments acted based on image pixels and surrounding voxels, demonstrating that the proposed method can be used with such sub-policies instead of code.\n- Proposed method is better than baselines, and appendix results in other environments show the method to be better or same as baselines in the other environments.\n- The proposed replanning method is general to work across environments, as evident by how it is implemented and by the experiments in multiple environments.\n- Part of the code and the LLM prompts/examples included in the appendix\n\nWEAKNESSES:\n- Limited by the set of controllers and their abilities. Authors note that this is a limitation of the current work and could be addressed in future work.\n- Some of the parts (e.g., success detector?) seem to rely on direct knowledge from the game. This may limit the application of the method to different environments.\n- The \"intelligence\" of the agent stems from LLMs, and the result seems to rely on the quality of the LLM. This creates a hard dependency for using this method (you _need_ to use a high-end LLM, which might be costly and/or closed-source). Especially in this work, the core models were closed-source OpenAI models, which may come unavailable in future and/or change, rendering reproduction of these results impossible.\n- Minor, but more analysis on the failure cases (e.g., when prompts fail, when they work, what seems to be problematic) would have provided more insight on how these types of planning should be done.\n\nQUESTIONS:\n1) How were the controller policies trained, exactly? Appendix mentions they are trained with imitation learning p(a | s, g), but where is the data collected? Looking at the shared code in \"controller.py\", there seems to be plenty of hardcoded actions (e.g., \"smelt_w_furnace\"). However, I am not sure if these functions are not used. Could you clarify how the controllers were trained, exactly?\n2) What \"offline trajectories\" the selector model was trained on (Line 194)? If the trajectories were generated by an optimal policy that reaches goals as fast as possible, then the training might make sense, but if they are suboptimal, selector may learn a wrong sense of how fast player can reach objects. Seems like this is related to my first question.\n3) Line 230: why different baselines/methods use different OpenAI models? I assume different baselines call for different type of generation (e.g., code vs. Text), but this is unclear. Please clarify/specify why same LLM was not used across planners.\n4) Table 1: add explanation for what is the \"+/-\" metric. Interestingly, this is very high for the easy task but not for the hard tasks. Can you explain why?\n\n### Comments (not questions)\n\n- Use \"DEPS\" (and \"DEP\") acronym consistently after first defining it (defined again in line 312)\n- Not a question/requirement, but seeing results with any openly shared LLM (i.e., weights are available for download and runnign locally) would be interesting to see.\n- Another curiosity and not a requirement, but it would be great to see the model in action to see how well they behave. This is not a request, and I realize this is not a very objective measure, but it could give some intuition on how \"directed\" the agents are, for example (do they wonder around before doing the task, or do they directly go for the task?)\n- Appendix line 501: sentence cuts off.\n- Appendix Listing  6: first prompt is duplicated\n- Figure 2 uses term \"HPM\", but this is not explained in the figure of caption. Same for VLM (which is defined later). For readibility, explain these acronyms in the caption or the figure.\n- Line 224 paragraph: there should be a space between the method name and its acronym.\n- Line 289: Citatations at the beginning of the sentence are bit out of place. Did you mean to use \"citet\" or \"citep\" commands instead? \n- Line 291: VPT had higher diamond obtaining rates after RL-finetuning (I believe 2.5% was for diamond pickaxe). Double-check what numbers you want to compare against\n- Line 292: I'd note here that DreamerV3 used a simplified environment where agent instantly destroyed blocks, effectively making the task easier than what the paper tackles.\n- Line 338: A nitpick, but I would recommend rephrasing the final sentence to indicate you are planning to work on this but also indicate that this work lays solid foundation for anyone to build on.\n\nLIMITATIONS:\nAuthors note the limitations. Especially noteworthy is the mentioned limitation of using closed-source models, and the suggestion to replicate the results with open source models.\n\nAuthors did not include broader impact/societal impact section. This is not required for this paper (no immediate impact), however I'd recommend adding one regardless so that authors think of the implications. E.g., these are very capable models, but if they solve tasks in misaligned/harmful ways, they may end up doing harm if ever deployed. We should aim to understand this alignment and/or analyze how model approaches the tasks.\n\n\n## Rebuttal acknowledgement\n\nI have read authors' rebuttal and new results which did address my concerns, and I updated my score from 8 to 9 (before discussion period closed).", "TITLE": "Official Review", "RECOMMENDATION": 9, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a novel approach called \"Describe, Explain, Plan and Select\" (DEPS) for task planning of multi-task embodied agents within open-world environments. The authors identify two main challenges in this area: the need for accurate and multi-step reasoning due to the long-term nature of tasks, and the inefficiency or infeasibility of plans created by vanilla planners that do not consider the ease of achieving sub-tasks when ordering parallel sub-goals.\nDEPS addresses these challenges by integrating a description of the plan execution process and providing self-explanation of feedback when encountering failures during the extended planning phases. It also includes a goal selector, which ranks parallel candidate sub-goals based on the estimated steps of completion, refining the initial plan.\nThe authors conducted experiments in the open-world game Minecraft, marking the first zero-shot multi-task agent that can robustly accomplish 70+ tasks, nearly doubling the overall performances. The method also proved effective in non-open-ended domains such as ALFWorld and tabletop manipulation. The authors concluded with ablation and exploratory studies detailing how DEPS outperforms counterparts and provides a promising update on the ObtainDiamond grand challenge.\n\nSTRENGTHS:\n- The overall pipeline is useful. The paper introduces a novel planning approach, DEPS, that integrates description of the plan execution process and self-explanation of feedback when encountering failures.\n- The fact that DEPS is capable of zero-shot learning, meaning it can generalize to tasks it has not seen during training, is a strength. This makes it a flexible and powerful tool for task planning in open-world environments. However, this is not an entirely new concept, as previously shown in e.g., Code as Policies (Liang et al. 2022), Text2Motion (Lin et al. 2023), ProgPrompt (Singh et al. 2022), Inner Monologue (Huang et al. 2022), Socratic Models (Zeng et al. 2022).\n- Beyond that, the proposed use of a goal selector that ranks parallel candidate sub-goals based on the estimated steps of completion is a strength. It allows for more efficient and feasible planning and is verified empirically.\n- The performance seems good. The paper also includes detailed ablation and exploratory studies, providing a thorough understanding of why DEPS outperforms other methods. I will include more discussion below.\n- Additional details in appendix and source code are provided, which are useful to ensure the reproducibility of the paper and good practice for ML community.\n\nWEAKNESSES:\n- Although the overall pipeline sounds promising, there has been several similar work in zero-shot task planning on Minecraft and other embodied environments: e.g., Code as Policies (Liang et al. 2022), Text2Motion (Lin et al. 2023), ProgPrompt (Singh et al. 2022), Inner Monologue (Huang et al. 2022), Socratic Models (Zeng et al. 2022).\n- However, the selector is a completely new module and distinguish the work from others, while I feel it could be highlighted more. I list some points in “Questions”.\n- The explanation of the selector could be improved. This seems like a well motivated module to have and plays a key role. But, for example, what does this module take as input? More concretely, how are plan $P_t$, agent full state $s_t$, and goal $g_t$ represented? How does it train and what is data collection procedure? More details could be provided or pointed to appendix (not seen).\n- The empirical study is great and supports the claims in general. But I have some questions/comments and list them in “Questions”.\n- Lack of Discussion on Failure Cases: While the paper mentions that DEPS provides self-explanation of feedback when encountering failures, it doesn't delve into specific instances where DEPS failed and how these failures were addressed.\n- Another general issue, dependency on LLMs: DEPS relies heavily on specific LLMs, which may limit its applicability in environments where such models are not feasible due to computational or data constraints, or domain knowledge not available as in Minecraft.\n\nQUESTIONS:\n- L122 — Intuitively, the selector seems useful. But how generalizable is this selector to unseen goals/plans? Can it always produce helpful output for selecting plans?\n- L241, “This validates that “describe, explain and plan” can estimate the reason for current plan failure and correct the original ﬂawed plans.” — Why this is true? Is there any more concrete example? Why the improvement necessarily mean that “DEP can estimate reason for failure”?\n- L245 — The success rate boost from selector sounds promising. I wonder if there is any other measure that can reflect the performance gain, such as average steps?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
In order to solve the long-term planning problem in the open world, this paper designed an interactive planning framework based on LLMs, and trained the first planning-based agent that successfully implements the ObtainDiamond task in the Minecraft scene.

STRENGTHS:
1. The article is clear and the idea proposed by the author is easy to follow.

2. The work is solid, and the method has been verified to be effective on multiple benchmarks. Ablation experiments comprehensively analyze the necessity of each module.

WEAKNESSES:
1. The failure criteria of the controller that triggers the re-planning mechanism is not clear enough. Can the author explain in detail how to determine whether the failure is caused by the planning errors generated by the planner or the controller executing actions incorrectly?

QUESTIONS:
Please refer to the weakness above.

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-26