SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zihao Wang $^{1,2}$ , Shaofei Cai $^{1,2}$ , Guanzhou Chen $^{3}$ , Anji Liu $^{4}$ , Xiaojian Ma $^{4}$ , Yitao Liang $^{1,5*}$   
Team CraftJarvis
$^{1}$ Institute for Artificial Intelligence, Peking University
$^{2}$ School of Intelligence Science and Technology, Peking University
$^{3}$ School of Computer Science, Beijing University of Posts and Telecommunications
<sup>4</sup>Computer Science Department, University of California, Los Angeles
$^{5}$ Beijing Institute for General Artificial Intelligence (BIGAI)
{zhwang,caishaofei}  $@$ stu.pku.edu.cn, rayment@bupt.edu.cn
liuanji@cs.ucla.edu, xiaojian.ma@ucla.edu, yitaol@pku.edu.cn
We investigate the challenge of task planning for multi-task embodied agents in open-world environments. Two main difficulties are identified: 1) executing plans in an open-world environment (e.g., Minecraft) necessitates accurate and multi-step reasoning due to the long-term nature of tasks, and 2) as vanilla planners do not consider how easy the current agent can achieve a given sub-task when ordering parallel sub-goals within a complicated plan, the resulting plan could be inefficient or even infeasible. To this end, we propose "Describe, Explain, Plan and Select" (DEPS), an interactive planning approach based on Large Language Models (LLMs). DEPS facilitates better error correction on initial LLM-generated plan by integrating description of the plan execution process and providing self-explanation of feedback when encountering failures during the extended planning phases. Furthermore, it includes a goal selector, which is a trainable module that ranks parallel candidate sub-goals based on the estimated steps of completion, consequently refining the initial plan. Our experiments mark the milestone of the first zero-shot multi-task agent that can robustly accomplish  $70+$  Minecraft tasks and nearly double the overall performances. Further testing reveals our method's general effectiveness in popularly adopted non-open-ended domains as well (i.e., ALFWorld and tabletop manipulation). The ablation and exploratory studies detail how our design beats the counterparts and provide a promising update on the ObtainDiamond grand challenge with our approach. The code is released at https://github.com/CraftJarvis/MC-Planner.
Developing multi-task agents that can accomplish a vast and diverse suite of tasks in complex domains has been viewed as one of the key milestones towards generally capable artificial intelligence [36, 1, 5, 10, 25]. To enable such capabilities, earlier works have suggested employing a hierarchical goal execution architecture [2, 4], where a planner generates action plans that would then be executed by low-level goal-conditioned controllers. This architecture has been delivering promising progress in
2We borrow the term "open world" from the game community. It highlights that the agent can navigate inside a diverse environment and accomplish open-ended tasks freely.
Challenge #1: Complex Sub-task Dependency
Figure 1: Planning success rates plummet in open worlds due to new challenges.
Challenge #2: State-dependent Task Feasibility
many robotics domains, including table-top and mobile manipulation [46, 4], 2D shape drawing [20] and table rearrangement [17]. However, whether such success can be transferred to a more open-ended world with unlimited exploration areas and internet-scale knowledge remains open [14, 10, 13, 12, 19].
To understand the gap, we run Inner Monologue[17], a general and competitive hierarchical goal execution model on a typical open-world domain Minecraft [18, 14, 10] and two classical robotic environments ALFWorld [41] and Tabletop environments [40, 4]. The algorithm uses a Large Language Model (LLM) based planner that contains domain-specific knowledge for all three environments. In all environments, we use either an Oracle goal-conditioned controller or a learned one. Results are shown in the bar plot in Figure 1. First, even when the Oracle controller is used, the success rate of executing Minecraft tasks is much less than that of the other environments. Next, the task failure rate becomes even higher in Minecraft when the learned controller is substituted. Both failures originate from unique challenges brought by open-world environments, which we identify in the following.
First, compared to canonical environments (e.g., Atari [29] and robotic control suite [40]), open worlds have highly abundant object types with complex dependency and relation. As a result, ground-truth plans typically involve a long sequence of sub-goals with strict dependencies. As Figure 1 challenge #1 suggests, it requires at least 13 sub-goals executed in proper order to obtain a diamond in Minecraft, while in Tabletop a task is typically no more than a few consecutive sub-goals.
Another challenge brought by the complicated tasks in an open-ended world is the feasibility of the produced plans. Consider the example shown in Figure 1 (challenge #2). To craft a bed in Minecraft, the fastest way is by either slaughtering a sheep to obtain wool, which can be used to craft beds, or collecting beds from a village. However, since no sheep or village is reachable by the agent within 3 minutes of gameplay, to craft a bed efficiently, the agent should choose to slaughter a spider and use materials (e.g., string) it drops to craft wool, and then a bed. That is, when dealing with a task that can be completed by executing multiple possible sequences of sub-goals, the planner should be able to select the best route based on the current state of the agent. However, the complex and diverse state distribution of open-world environments makes state awareness hard to achieve.
To tackle these problems, we propose "Describe, Explain, Plan and Select" (DEPS), an interactive planning approach based on Large Language Models (LLMs) to alleviate the aforementioned issues. The key to tackling the first challenge is to effectively adjust the generated plan upon failure. Specifically, whenever the controller fails to complete a sub-goal, a descriptor will summarize the current situation as text and send it back to the LLM-based planner. We then prompt the LLM as an explainer to locate the errors in the previous plan. Finally, a planner will refine the plan using information from the descriptor and explainer. To improve the feasibility of generated plans conditioned on the current state, which is the second identified challenge, we use a learned goal selector to choose the most accessible sub-task based on the proximity to each candidate sub-goal.
Our experiments are conducted on 71 tasks in open-ended Minecraft without any demonstration. Given the goal-conditioned controller for atom sub-tasks (i.e., mine log and mine stone), our zero-
Figure 2: Overview of our proposed interactive planner architecture.
shot $^3$  LLM-based planner can finish all tasks within a limited horizon (3000-12000 steps for different tasks). We find DEPS outperforms all language planner baselines by nearly doubling the overall success rate, with the same initial state and goal-conditioned controller. Our ablation and exploratory studies then explain how our approach beats the counterparts and becomes the first planning-based agent that accomplishes the challenging ObtainDiamond task. DEPS does not require any planning training for the environment. Additionally, DEPS achieves between on-par and more than  $50\%$  relative improvement over existing or concurrent LLM-based planning methods on non-open-ended robotics domains such as ALFWorld [41] and Tabletop environments [40].
We aim to develop an agent capable of solving long-horizon goal-reaching tasks using image observations and language goals. To accomplish this, we propose a combined approach involving goal-conditioned policies (termed controllers) and a planner. The goal-conditioned policies are trained to complete sub-goals, while the planner decomposes long-horizon tasks into a series of  $K$  short-horizon sub-goals,  $g_{1},\ldots ,g_{K}$ , to be executed by the controller. At each time step  $t$ , the goal-conditioned policy  $\pi (a_t\mid s_t,g_k)$  generates an action  $a_{t}$  based on the current state  $s_t$  and the specified sub-goal  $g_{k}$ .
Planning with Large Language Models Previous works have shown that LLMs such as Instruct-GPT [32] and Codex [8] can be used as zero-shot planners to generate sub-goal sequences for various tasks in embodied environments [16, 42]. Formally, given the task description  $T$  as prompt  $p$ , LLM acts as a planner to decode  $T$  into  $K$  sub-goals,  $g_{1},\ldots ,g_{K}$ , which are then executed one by one by the low-level controller  $\pi (a_t\mid s_t,g_k)$  to accomplish the task.
However, the above pipeline suffers from both challenges identified in Section 1. Regarding the first challenge, the probability of generating a flawless plan directly from the task description decreases significantly as the required number of sub-goals increases. Moreover, even when the LLM generates a correct plan, it is very likely that the plan is highly inefficient given the agent's current state (challenge #2). Prior works mostly focus on solving the first challenge by providing environmental feedback to the LLM through affordance functions [4], success detector [20] or scene descriptor [17]. However, although these approaches work well on many non-open-ended domains, they still suffer from high failure rates in open-world environments.
Similar to [5, 16], "zero-shot" here means no gradient updates are performed. However we provide some related demonstrations as prompts during inference time.
3 Towards Reliable Planning in Embodied Open-World Environments
In this section, we first give an overview of our proposed interactive planning framework "Describe, Explain, Plan, and Select" (DEPS) for solving complex and long-horizon tasks in open-world environments (Sec. 3.1). Next, in Section 3.2, we elaborate how DEPS iteratively refines its plan to combat the first identified challenge. Section 3.3 introduces the selector module that is used to identify efficient plans in response to the second identified challenge.
3.1 DEPS Overview
As demonstrated in Figure 2, our agent (DEPS) consists of an event-triggered Descriptor, a Large Language Model (LLM) as Explainer and Planner, a goal Selector based on horizon prediction and a goal-conditioned controller. In the following, we use Minecraft as a running example to better elaborate our agent. Note that DEPS can be directly applied to other (non-)open-ended tasks.
We take a large language model (LLM) as a zero-shot planner of the agent to complete tasks. Given a goal command (e.g., ObtainDiamond) as task  $T$ , the LLM-based planner decomposes this high-level task into a sequence of sub-goals  $\{g_1, \dots, g_K\}$ , as the initial plan  $P_0$ . The goals are instructions in natural language, such as mine oak wood (in Minecraft), find two cups (in ALFWorld), put block A on top of block B (in Tabletop Manipulation).
As described in Section 2, a controller is then invoked to execute the provided sub-goals sequentially through a goal-conditioned policy  $\pi(a \mid s, g)$ . However, the initial plan provided by the planner often contains errors, which results in execution failures of the controller. For example, the goal cannot be finished only with a wooden pickaxe as shown in Figure 2. When failure pops up, the descriptor will summarize the current state  $s_t$  and execution outcome of the most recent goal into text  $d_t$  and send it to the LLM. The LLM will first try to locate the errors in the previous plan  $P_{t-1}$  by self-explanation, e.g., the goal need to be executed with a stone pickaxe. Then it will re-plan the current task  $T$  and generate a revised plan  $P_t$  according to the explanation. In this process, the LLM is also treated as an explainer in addition to the planner role. The Descriptor, Explainer, and Planner will be detailed in Section 3.2.
$$
\text {D e s c r i p t}: d _ {t} = f _ {\text {D E S C}} \left(s _ {t - 1}\right),
$$
$$
\text {E x p l a n a t i o n}: e _ {t} = f _ {\mathrm {E X}} (d _ {t}),
$$
$$
\begin{array}{l} \text {P r o p :} p _ {t} = \operatorname {C O N C A T} \left(p _ {t - 1}, d _ {t}, e _ {t}\right), \\ \text {P r o p :} P = f _ {t - 1} (\cdot) \end{array} \tag {1}
$$
$$
\text {P l a n}: P _ {t} = f _ {\mathrm {L M}} \left(p _ {t}\right),
$$
$$
\text {G o a l}: g _ {t} \sim f _ {\mathrm {S}} \left(\mathrm {P} _ {t}, s _ {t - 1}\right),
$$
$$
\text {A c t i o n}: a _ {t} \sim \pi \left(a _ {t} \mid s _ {t - 1}, g _ {t}\right)
$$
As shown in Equation (1), DEPS will iteratively update the plan  $P_{t}$  until the task is finished, where  $f_{\mathrm{DESC}}$  is the descriptor model,  $f_{\mathrm{LM}}$  denotes the language model as explainer and planner,  $f_{\mathrm{S}}$  is the selector model,  $\pi$  is goal-conditioned policies from the controller.
To filter out inefficient plans, the selector is trained to predict the number of time steps remaining to achieve every goal  $g_{k}$  in a set of parallel goals given the current state  $s_t$ . When the generated plan contains alternative routes, the selector uses this information to choose a suitable goal as the current goal  $g_{t}$ . For example, the horizon predicted by the selector of goal acacia tree is less than goal oak tree in Savanna biome, which leads to chop acacia tree as current goal  $g_{t}$ .
3.2 Describe, Explain and Plan with LLM Generates Executable Plans
Current LLM-based planners usually query the LLM once at the beginning of every episode and use the output plan throughout the episode [16, 42]. However, as demonstrated by Figure 1, such one-shot planning methods often fail on long-horizon tasks that require many sub-goals. This is caused by two major issues. First, since the correct plan for long-horizon tasks needs to respect various complex preconditions, it is extremely hard for the LLM to generate a flawless plan directly from the task instructions, resulting in failure when simply following the initial plan. Additionally, due to the unpredictable transition dynamics, some incidents may happen during the execution and make the initial plan non-executable. To remedy these problems, existing methods introduce feedback (e.g.,
Prompt 1 Planner prompt template, Python-like code
def craft Wooden_axe(initial_inventory  $\equiv$  {）： #step1: mine3 logs mine(obj  $=$  {"log":3},tool  $=$  None)# step 2: craft 12 planks from 3 logs craft(obj  $=$  {"planks":12},materials  $=$  {"log":3},tool  $=$  None)# step 3: craft 4 sticks from 2 planks craft(obj  $=$  {"stick":4},materials  $=$  {"planks":2},tool  $=$  None)# step 4: craft 1 crafting_tablefrom4planks craft(obj  $=$  {"crafting_table":1},materials  $=$  {"planks":4},tool  $=$  None)# step 5: craft 1 wooden_axe from 3 planks and 2 sticks on crafting table craft(obj  $=$  {"wooden_axe":1},{"planks":3,"stick":2},tool  $=$  "crafting_table") return"wooden_axe"
from success detector or scene descriptor) to reflect on the results of previous executions [17, 20, 4]. However, merely informing the LLM whether a sub-goal is completed is often insufficient to correct the planning error.
To remedy this, we propose "describe, explain and plan", a new interactive planning method to generate more executable and explainable plans. We start with rewriting the prompt into an interactive dialogue format as in ChatGPT [32] so that subsequent feedback can be passed to the LLM effectively. The produced plan is also augmented with the preconditions and effects of each goal. The structured prompt improves the readability and interpretability of the plan and facilitates error-locating when the execution fails later, as demonstrated in Prompt 1.
The descriptor will then collect the feedback generated by the agent during the execution of the task. The feedback can be practically obtained either by a person (human feedback [4]), or by a pre-trained vision-language model CLIP [35]. While the previous type of feedback needs intensive human involvement, the latter from the pre-trained model needs to be fine-tuned for the specific domain, which decreases the automation and generalization of the agent. On the contrary, Minecraft returns the 'info' and other high-level observations (such as biome, GPS, and compass), we can easily translate the unstructured information into structured language. Therefore we take the symbolic information available in the game and translate it into feedback description  $d_{t}$  in this work. To avoid carrying unrelated information in the prompt, we further distill plan-related messages (e.g., inventory information, biome) as final event-level description  $d_{t}$  as demonstrated in Figure 2.
Notably, we also treat the LLM as an explainer to explain why the previous plans  $P_{t-1}$  failed. Specifically, by analyzing the current state from description  $d_t$  and precondition of current goal  $g_t$ , the explainer can identify the reason why the current goal cannot be executed successfully. As shown in Figure 2, the reason may be the current goal requires the use of an iron pickaxe, but the tool is not prepared in advance, or the current goal requires the use of 3 planks, but the currently available planks are not enough. To implement this, we provide few-shot demonstrations to the LLM as in chain-of-thoughts prompting [45], as shown in Prompt 1. Finally, the LLM goes back to its role as a planner and re-plans the task with the explicit explanation of existing bugs in the previous plan  $P_{t-1}$ , ultimately generating an updated plan  $P_t$  according to the explanation.
3.3 Horizon-Predictive Selector Yields Efficient Plans
Due to the abundance of objects and the compositional nature of their functionalities, there often exist multiple feasible plans to complete a task, i.e., there are usually multiple paths for the completion of a particular goal. However, despite the feasibility of all such plans, most of them are highly inefficient to execute in the current episode. For example, as shown in Figure 2, obtaining a wood can be done by chopping oak trees, birch trees, or acacia trees. But only oak trees are available in the plains biome. So the planner needs to choose oak trees since it is more efficient, as the agent does not need to travel to another biome.
On the other hand, there is no strict sequential requirement for some goals in the plan  $P_{t}$ , i.e.,  $g_{i}, g_{j} \sim P_{t}$  enjoy the same precondition, which means  $g_{i}$  and  $g_{j}$  can be executed in any order. As shown in Figure 1, the choice of different paths (sequences) may affect the execution efficiency of the plan  $P_{t}$  as one goal might be closer to the agent. Always choosing the closer goal to execute first could yield more efficient plans and improve the final success rate under a limited episode length. Moreover, the dynamic nature of open-world environments further amplifies the impact of efficient
Goal: Meat*3 Candidate Skill: Kill Sheep OR Cow OR Pig Selection: Kill Sheep Explanation: Meet sheep first.
Goal: Log*2 Candidate Skill: Chop Oak OR Birch OR Acacia Tree Selection: Chop Acacia Tree Explanation: Savanna biome only has Acacia tree.
Figure 3: Selection Demonstration from "Selector". Given parallel sub-goals, i.e. candidate skills, our Selector will determine the sequence in which to carry out these sub-goals based on their current proximity to the agent and modify the original plan produced by the LM planner.
Goal: Coal\*1 AND Iron_Ore\*1 Candidate Skill: Mine Coal AND Iron_Ore Selection: Mine Iron_Ore Explanation: Meet iron_ore first.
Goal: Survive in Night. Candidate Skill: Sleep in bed OR Dig down. Selection: Sleep in bed Explanation: Village has beds.
plans on the success rate. For example, in Minecraft, if the agent chooses to execute a further goal like collect wood first, the much closer target sheep may disappear and be hard to find again.
In order to improve the efficiency of our plans, we propose to use a selector that selects the most efficient path with the highest execution success rate as the final plan. Specifically, we design a state-aware selector to choose the nearest goal under state  $s_t$  as the current goal  $g_t$  from the candidate goal sets in plan  $P_t$ . It predicts the goal distribution  $p(g_t | s_t, P_t)$  under the current state  $s_t$  and plan  $P_t$ , where  $g_t \in G_t$ ,  $G_t$  describes all current executable goals in  $P_t$ . A straight way to implement the selector is to leverage the semantic similarity between the current state and the goal text using a vision-language model (VLM) such as CLIP [35]. Nevertheless, this may not exactly reflect the difficulty of completing the goal since VLM lacks practical experience. For example, an "oak tree" in front of the agent could lead to high semantic similarity for the "chopping tree" goal, but it may be inefficient to achieve this goal if a canyon is in the middle between the agent and the oak tree.
To mitigate this, we implement a horizon-predictive selector that embeds practical task experience to accurately rank the goals based on their efficiency and feasibility. Here, we define the horizon of a goal  $h_t(g) \coloneqq T_g - t$  as the remaining time steps to complete the given goal, where  $T_g$  is the time of completing goal  $g$ . This metric accurately reflects how quickly we can achieve the given goal from the current state. To estimate the horizon, we learn a neural network  $\mu$  to fit the offline trajectories by minimizing the entropy loss  $-\log \mu(h_t(g) \mid s_t, g)$ , where  $h_t$  is the ground-truth horizon in trajectories of completing goal  $g$ . Therefore, the goal distribution can be formulated as follows:
$$
f \left(g _ {t} \mid s _ {t}, P _ {t}\right) = \frac {\exp \left(- \mu \left(g _ {t} , s _ {t}\right)\right)}{\sum_ {g \in G _ {t}} \exp \left(- \mu \left(g , s _ {t}\right)\right)}. \tag {2}
$$
We set goal-sensitive Impala CNN [6] as the backbone of the selector. In practice, the horizon predictive selector can be jointly trained with the controller policies and share the backbone parameters [6].
This section analyzes and evaluates our proposed "describe, explain, plan, and select" (DEPS) method. To minimize performance variation caused by the low-level controller, we standardize all experiments with one controller learned by behavior cloning. We refer to the details of this controller in Appendix C. In Section 4.1, we introduce our testing environments and our evaluation task set, consisting of the hardest 71 tasks from MCU SkillForgeChain [22]. In Section 4.2, we report our performance in the context of existing LLM-based planners. Ablation studies are conducted in Section 4.3. Finally, we pay close attention to the hardest task, ObtainDiamond, which is long-hailed as a major challenge in the community. The experiments on ALFWorld and Tabletop Manipulation environments are shown in Appendix A.
Environment and Task Setting We first evaluate our proposed method in Minecraft, a popular open-world environment with both challenges discussed in Section 1. For better reflecting the performance of DEPS, we choose three Minecraft environments with different versions for better
evaluation, including Minedojo [10] with Minecraft 1.11.2, MineRL [3] with Minecraft 1.16.5, and MC-TextWorld [22] with Minecraft 1.19.2. Rules and items have something different in the above three Minecraft environments, which can better evaluate the dynamic and interactive planning abilities of DEPS.
Table 1: Attributes of 8 meta tasks covering Task101: We evaluate the algorithm on Minecraft Task101. We group the consisted 71 task into 8 different meta groups, with each focusing on testing a different aspect of our proposed method.
Meta	Name	Number	Example Task	Max. Steps	Initial Inventory	Given Tool
MT1	Basic	14	Make a wooden door.	3000	Empty	Axe
MT2	Tool (Simple)	12	Make a stone pickaxe.	3000	Empty	Axe
MT3	Hunt and Food	7	Cook the beef.	6000	Empty	Axe
MT4	Dig-Down	6	Mine coal.	3000	Empty	Axe
MT5	Equipment	9	Equip the leather helmet.	6000	Empty	Axe
MT6	Tool (Complex)	7	Make shears and bucket.	6000	Empty	Axe
MT7	IronStage	13	Obtain an iron sword.	6000	Empty	Axe
MT8	Challenge	1	Obtain a diamond!	12000	Empty	Axe