SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We choose 71 tasks from the Minecraft Universe Benchmark SkillForgeChain [22] for evaluation. These tasks are related to items that can be obtained in the Minecraft overworld. To better present the results, we divide the 71 Minecraft tasks into 8 meta groups according to the ingredients and function of the tasks, i.e., MT1-MT8. The instruction for every task is written in natural language, e.g., make a wooden door in MT1 (Basic group) and obtain a diamond in MT8 (Challenge group), as illustrated in Table 1. Considering how long it typically takes human players to complete each task as a ballpark [14], we set different maximum episode steps for different meta tasks from 3000 (for easiest Basic tasks) to 12000 (for the hardest Challenge tasks). The names, number of required skills, and functions of all tasks are listed in Appendix B. We give an empty inventory for every task in Survival mode and require the agent to obtain every item from the environment by itself. Note that our agent will be summoned in different environments randomly for each evaluation. Biomes and initial positions are also different each time. Following the previous work [18], we take the success rate as the evaluation metric.
Baselines We compare DEPS with other language-based planners, including GPT as Zero-shot Planner(GPT) [16], ProgPrompt(PP) [42], Chain-of-Thought(CoT) [45], Inner Monologue(IM) [17], and Code as Policies(CaP) [20]. For all baseline models, we use the same demonstration example in the prompt, the same LM model from OpenAI, and the same controller in all tasks for a fair comparison. Since these methods were not originally experimented with Minecraft, we reproduce them to conform to the Minecraft specification based on prompt and feedback template design. All planner methods access the LLM model through OpenAI API (text-davinci-03 model [32] for GPT, CoT, and IM, and code-davinci-02 model [8] for PP, CaP, and Ours). All hyperparameters of LLM (including the temperature and best_of, etc.) are kept as default. We also list the full prompt of all different methods in Appendix G.
Every task is executed 30 times and the average results in Minedojo [10] for every meta task are listed in Table 2. Our approach achieves the best performance with all meta tasks. As the complexity of the task increases from MT1-MT8, the planner usually needs to give more accurate task steps (i.e., longer goal sequence) to achieve the final task. Therefore the success rate of all agents decreases with the reasoning steps increasing. Starting from MT6, almost all existing LLM-based planners fail (nearly 0 success rate). DEP (w/o Selector) already consistently beats existing LLM-based planners in all meta tasks with a significant margin. This validates that "describe, explain and plan" can estimate the reason for current plan failure and correct the original flawed plans. Due to the limited maximum episode length and restricted control success rate for a hard goal (e.g., Mine diamond with iron_pickaxe), the final success rate is still capped.
Table 2: Success rates of DEPS and existing LLM planners on Minecraft Task101. The full task-by-task list is in Appendix F.
Methods	MT1	MT2	MT3	MT4	MT5	MT6	MT7	MT8	AVG
GPT[16, 32]	25.85±24.8	47.88±31.5	10.78±14.6	7.14±9.0	1.98±5.9	0.0±0.0	0.0±0.0	0.0±0.0	15.42
PP[42]	30.61±23.6	40.09±30.6	17.13±19.1	16.00±17.3	3.21±4.9	0.47±1.3	0.60±2.2	0.0±0.0	16.88
CoT[45]	40.24±30.8	55.21±26.8	6.82±11.6	4.76±8.2	1.73±5.2	0.0±0.0	0.0±0.0	0.0±0.0	18.89
IM[17]	46.89±31.4	53.73±20.8	3.64±6.9	18.41±17.4	4.57±7.4	0.64±1.7	1.02±2.5	0.0±0.0	21.64
CaP[20]	60.08±17.3	60.11±20.24	8.72±9.7	20.33±21.0	2.84±4.6	0.63±1.3	0.60±2.2	0.0±0.0	25.77
DEP	75.70±10.4	66.13±13.4	45.69±16.2	43.35±20.2	15.93±13.9	5.71±3.7	4.60±7.1	0.50±0.5	39.36
DEPS	79.77±8.5	79.46±10.6	62.40±17.9	53.32±29.3	29.24±27.3	13.80±8.0	12.56±13.3	0.59±0.5	48.56
In addition, selector also greatly improves the final task success rate of the agent (from DEP w/o Selector to DEPS). Hard meta tasks usually require the completion of multiple sub-goals (up to dozens of goals), thus bringing more flexibility and providing more candidate goals for the Selector. At the same time, as the agent conducts experiments with limited episode length, it also places high demands on the efficiency of the plan. Therefore, the Selector brings a significant improvement on efficiency-sensitive tasks such as MT7 (up to +2.7 times success rate).
Robustness on different controller and different Minecraft versions We also evaluate DEPS on MineRL [3] and MC-Textworld [22]. Note that DEPS is a planning method, which needs to equip the goal-conditioned controller for interacting with the Minecraft environments. We choose MC-Controller [6] and Steve-1 [21] as controllers to interact with Minedojo and MineRL, respectively. These two methods are all control policies that perceive visual partial observations and produce mouse and keyboard actions. While MC-Textworld is a text world, which only keeps the Minecraft crafting recipes and mining rules. So MC-Textworld does not require the controller. The DEPS results of the task set MT1-MT8 on different Minecraft environments are shown in Table 3. The results report that DEPS can generate effective plans in various Minecraft environments. The results on MC-Textworld [22] also show that the performance drops on more difficult task sets from MT6 to MT8 are mainly from the controller limitation.
Table 3: Success rates of DEPS under different Minecraft environments.
Environment	Version	Controller	MT1	MT2	MT3	MT4	MT5	MT6	MT7	MT8
MineDojo [10]	1.11.2	[6]	79.77	79.46	62.40	53.32	29.24	13.80	12.56	0.59
MineRL [3]	1.16.5	[21]	84.05	80.32	24.25	36.21	9.16	17.22	16.79	1.84
MC-Textworld [22]	1.19.2	-	100.00	90.00	80.00	56.25	64.71	57.14	69.57	50.00
We conduct ablation experiments to investigate the number of candidate executable goals for different Selector models and the specific impact of the rounds of DEPS.
4.3.1 Ablation on Selector
We verify the robustness of our proposed Selector under different parallel goals. The agent is asked to complete 2, 3, and 4 candidate goals (the precondition is consistent for all goals), respectively. The goals of the task correspond to different kinds of mobs or materials.
We report the final success rate of our method (DEP) with different selector implementations, including using a fixed sequence of goals, a random sequence of goals, and selecting a goal based on MineCLIP [10], CLIP [35], and our horizon-predictive Selector (HPS). As Figure 4 shows, in one round of parallel candidate goals, an improvement of  $\Delta = +22.3\%$ ,  $+29.2\%$ ,  $+32.6\%$  is obtained using our horizon-predictive Selector compared to not any selector (i.e., fixed plan), respectively.
At a limited episode length, e.g., 1000 steps, goal-model shows a greater advantage, which proves that goal-model can improve the execution efficiency of the plan in embodied environments. In addition, compared to using vision-language models such as CLIP [35] and MineCLIP [10] as a goal model, horizon-predictive has the best performance due to better estimation of the horizon information. The curve trend also demonstrates that agents with Selector scale up under large amounts of goals in an open-world environment.
Figure 4: The success rates of DEPS with different selectors under varying numbers of parallel goals and maximum episode lengths.
Table 4: Success Rate of DEPS under different maximum rounds of re-planning. Round 0 represents the vanilla Planner w/o the re-planning process.  $\infty$  represents the re-planning process will not end until task success or reaching the maximum horizon, which is still limited by the maximum tokens of LLMs. The maximum number of rounds for Codex is around 7-8 rounds.
Rounds	0	1	3	5	∞	Δ (0 → ∞)
MT1	28.6	50.6	68.1	79.8	79.8	+51.2
MT2	37.1	71.2	71.4	79.2	79.5	+42.4
MT3	15.1	20.1	40.3	40.8	62.4	+47.3
MT4	15.9	17.4	48.3	50.7	53.3	+37.4
MT5	3.2	3.2	3.2	15.2	29.2	+26.0
MT6	0.5	0.5	1.1	1.9	13.8	+13.3
MT7	0.6	2.3	2.9	2.9	12.6	+12.0
MT8	0.0	0.0	0.0	0.0	0.6	+0.6
4.3.2 Ablation on Re-Planning Rounds
We evaluate our agent on all tasks with increasing maximum rounds of DEPS. The round is defined as a cycle of interactive LLM-based planning with description, explanation, and planning and selecting, i.e., an updated plan. All tasks for every maximum round are executed 30 times and the average success rate is reported in Table 4. We take the vanilla LLM planner as the baseline, in which the model takes the initially generated plan as the final execution plan, without involving any description, re-planning, or self-explanation processes during the task execution. Our results in the previous subsection utilize the maximum rounds possible under maximum tokens capped by OpenAI. We also report the success rate increment from vanilla planner to DEPS of every meta task in column  $\Delta$  in Table 4. This set of experiments demonstrates that DEPS can iteratively improve its plan in open-world environments. More description, self-explanation, and re-planning rounds produce better results, especially for hard tasks.
4.4 ObtainDiamond Challenge
Mining diamonds in the open-world game Minecraft, i.e. MT8 in Table 2, has been a long-standing challenge for the community [14]. It is challenging because mining diamonds from scratch in Minecraft involves acquiring a sequence of difficult-to-obtain items that require complex planning on goals like mining, inventory management, crafting with and without a crafting table, tool use, smelting iron ingot in a furnace, and mining at the lowest depths. We take the ObtainDiamond task as a bonus experiment to show the capabilities of our zero-shot planner on complex tasks in embodied environments. Previous methods' success rates on this challenge further vouch for its difficulty. [43, 34] leverages domain-secic reward functions and RL fine-tuning to achieve  $\sim 0.1\%$  success rate in 15 minutes of game play. VPT further boosts the success rate to  $20\%$  within 20 minutes of play through pre-training on collects  $\sim 70\mathrm{k}$  hours human demonstrations and finetuning with human-designed reward function [3]. DreamerV3 is trained from scratch to collect diamonds in a modified Minecraft environment (easier to break blocks) with world models to achieve a success rate of  $2\%$  [15].
Our DEPS manages to achieve on-par performance in this grand challenge; our agent achieves a  $0.59\%$  success rate within 10 minutes of gameplay. Note our method does not specifically fine-tune for this challenge. It is designed to be multi-task in its nature. Furthermore, considering our planner operates with demonstration prompts on a fixed Large Language Model, it can be straightforwardly adapted to other open-ended environments with modifications.
Task planning with LLMs There have been some methods leveraging the large language model to generate action plans for high-level tasks in embodied environments [46, 9, 11]. [16] decompose natural language commands into sequences of executable actions by text completion and semantic
translation, while SayCan generates feasible plans for robots by jointly decoding an LLM weighted by skill affordances from value functions [4]. For better executing the plan in embodied environments, some methods use an object detector describing the initial environment into the language prompt to produce environment-suitable plans and adopt success detectors to check that each step is executed successfully [17, 20]. [42] and [20] use the pythonic-style prompt to produce more executable plans. However, all of the above methods assume that the initial plan from the LLM is correct. When there are bugs in the initial plan, it's difficult for the agent to finish the task successfully.
Interactive Planning with LLMs Inner Monologue [17] pilots the front of interactive planning with LLMs, which introduces the feedback (including success detection and scene description) to the planner. However, we found it could still suffer from accumulative planning error, especially in long-horizon open-world tasks. Rather, our "Describe, Explain, Plan and Select" (DEPS) method can produce more reliable plans by leveraging chain-of-thought thinking and explanation to locate the errors in previous plans. Moreover, we also propose a goal Selector to further improve the efficiency of the plan, thereby yielding much better performances. Readers are encouraged to refer to the comparative results in Section 4.2 between DEPS and these prior arts. There are also some concurrent works on planning with LLMs [39, 26, 23, 33, 47].
Agents in Minecraft Some previous works have employed the hierarchical architecture to solve long-horizon tasks in Minecraft [30, 27, 24]. Recently, based on the internet-scale corpus, [10] pre-trains a language-conditioned reward function and learns multi-task MineAgent. [3] collects a vast amount of human demonstrations to train a behavior cloning agent. More recently, [15] utilized a learned world model to distill a policy that can efficiently explore in Minecraft. There are also some works focus on learning goal-conditioned policies for better instruction-following [6, 7, 21]. While these efforts all focus on improving the low-level controller. Rather, the planner in our architecture emphasizes applying domain knowledge to propose and arrange the sub-goals. It significantly influences the complexity and breadth of tasks that the agent can handle. Moreover, our planner is zero-shot, making it possible to generalize to other long-horizon open worlds.
Albeit the impressive results of our approach, we believe there are at least two major limitations within our approach. First of all, our framework relies on privately-held LLMs like GPT-3 and ChatGPT, which makes it less accessible to those who cannot afford or access the service. However, we're fully committed to ensuring a more democratized method and will explore using open-sourced models including OPT [48] and BLOOM [38]. Another issue is the explicit step-by-step planning in our system. Although it brings us superior performances over the baselines, the planning bottleneck can also prevent our model from being further scaled up. A more appealing approach will be amortizing the planning within an end-to-end trainable goal-conditioned policy, which is worth exploring next. Furthermore, some previous fundamental challenges in planning (e.g., dead ends) may not prevalent in our adopted environments and hence could be inadvertently overlooked by our paper. We are dedicated to addressing more fundamental challenges present in building a multi-task generalist agent in our series of following work.
We investigate the problem of planning in open worlds. We identify two major challenges unique to these environments: 1) long-term planning requires precise and multi-step reasoning, and 2) planning efficiency could be compromised since canonical planners do not take the agent's proximity to parallel goals/subtasks into consideration. We propose "Describe, Explain, Plan and Select" (DEPS), an interactive approach based on Large Language Models (LLMs) to tackle them both. Our experiments in the challenging Minecraft domain verify the advantages of our approach over counterparts by marking the milestone of robustly accomplishing  $70+$  Minecraft tasks and nearly doubling the overall performances. DEPS also is the first planning-based agent that can reach the diamond in this game.
This work is funded in part by the National Key R&D Program of China #2022ZD0160301, a grant from CCF-Tencent Rhino-Bird Open Research Fund, NSF grants #IIS-1943641, #IIS-1956441, #CCF-1837129, an SRA from Meta and a research gift from Amazon Alexa AI, and a gift from RelationalAI. We thank Dai Zhixiang from NVIDIA and Xu Hongming from BIGAI on training LLMs and infrastructure supports, respectively.
[1] J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, et al. Flamingo: a visual language model for few-shot learning. arXiv preprint arXiv:2204.14198, 2022. 1
[2] P.-L. Bacon, J. Harb, and D. Precup. The option-critic architecture. In Proceedings of the AAAI conference on artificial intelligence, 2017. 1
[3] B. Baker, I. Akkaya, P. Zhokhov, J. Huizinga, J. Tang, A. Ecoffet, B. Houghton, R. Sampedro, and J. Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos. arXiv preprint arXiv:2206.11795, 2022. 7, 8, 9, 10
[4] A. Brohan, Y. Chebotar, C. Finn, K. Hausman, A. Herzog, D. Ho, J. Ibarz, A. Irpan, E. Jang, R. Julian, et al. Do as i can, not as i say: Grounding language in robotic affordances. In 6th Annual Conference on Robot Learning, 2022. 1, 2, 3, 5, 10, 21
[5] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020. 1, 3, 15, 21, 22
[6] S. Cai, Z. Wang, X. Ma, A. Liu, and Y. Liang. Open-world multi-task control through goal-aware representation learning and adaptive horizon prediction. arXiv preprint arXiv:2301.10034, 2023. 6, 8, 10, 21, 24
[7] S. Cai, B. Zhang, Z. Wang, X. Ma, A. Liu, and Y. Liang. Groot: Learning to follow instructions by watching gameplay videos. arXiv preprint arXiv:2310.08235, 2023. 10
[8] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. 3, 7, 21, 22
[9] I. Dasgupta, C. Kaeser-Chen, K. Marino, A. Ahuja, S. Babayan, F. Hill, and R. Fergus. Collaborating with language models for embodied reasoning. In NeurIPS Foundation Models for Decision Making Workshop, 2022. 9
[10] L. Fan, G. Wang, Y. Jiang, A. Mandlekar, Y. Yang, H. Zhu, A. Tang, D.-A. Huang, Y. Zhu, and A. Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowledge. Advances in Neural Information Processing Systems Datasets and Benchmarks, 2022. 1, 2, 7, 8, 10, 18, 21, 26
[11] R. Gong, Q. Huang, X. Ma, H. Vo, Z. Durante, Y. Noda, Z. Zheng, S.-C. Zhu, D. Terzopoulos, L. Fei-Fei, et al. Mindagent: Emergent gaming interaction. arXiv preprint arXiv:2309.09971, 2023. 9
[12] W. H. Guss, M. Y. Castro, S. Devlin, B. Houghton, N. S. Kuno, C. Loomis, S. Milani, S. P. Mohanty, K. Nakata, R. Salakhutdinov, J. Schulman, S. Shiroshita, N. Topin, A. Ummadisingu, and O. Vinyals. The minerl 2020 competition on sample efficient reinforcement learning using human priors. arXiv: Learning, 2021. 2
[13] W. H. Guss, C. Codel, K. Hofmann, B. Houghton, N. Kuno, S. Milani, S. Mohanty, D. P. Liebana, R. Salakhutdinov, N. Topin, et al. Neurips 2019 competition: the minerl competition on sample efficient reinforcement learning using human priors. arXiv preprint arXiv:1904.10079, 2019. 2
A Additional Experiments 15
A.1 ALFWorld 15
A.1.1 Tasks 15
A.1.2 Results 15
A.2 Tabletop Manipulation 16
A.2.1 Tasks 16
A.2.2 Results 17
B Minecraft Task Details 18
C DEPS Implementation Details 20
C.1 Controller 20
C.2 LLM as Planner 21
C.3 LLM as Explainer 22
C.4 Other modules 23
D Comparison with other LLM-based Planners 23
E Discussion on ObtainDiamond Task 24
F Success Rates of ALL Tasks in Minecraft 25
G Prompt for Different Tasks and Different Methods 27
G.1 Prompt for Minecraft Tasks 27
G.1.1 DEPS 27
G.1.2Baselines 28
G.2 Prompt for ALFWorld Tasks 31
G.2.1 DEPS 31
G.2.2Baselines 33
G.3 Prompt for Tabletop Manipulation Tasks 34
G.3.1 DEPS 34
G.3.2Baselines 35
H Full Dialogue 35
Additional experiments are conducted on the ALFWorld [41] and Tabletop Manipulation environments [40] to showcase the generalization capabilities of DEPS.
A.1 ALFWorld
ALFWorld [41] is an interactive learning environment that aligns text and embodiment, allowing agents to acquire abstract, text-based policies in TextWorld, and subsequently execute goals from the ALFRED benchmark in a visually rich environment.
A.1.1 Tasks
The ALFWorld framework contains six types (namely Pick & Place, Examine in Light, Clean & Place, Heat & Place, Cool & Place, Pick Two & Place) of tasks with various difficulty levels. Tasks involve first finding a particular object, which often requires the agent to open and search receptacles like drawers or cabinets. Subsequently, all tasks other than Pick & Place require some interaction with the object such as heating (place the object in a microwave and start it) or cleaning (wash the object in a sink). To complete the task, the object must be placed in the designated location. We sample 10 tasks from ALFWorld randomly and list all the task names, types, and the number of receptacles in Table 5. We classify them into 6 groups based on their functionality. For all tasks, the maximum number of steps is set as 50.
Table 5: Task list in ALFWorld.
Group	No.	Task	Number of Receptacles
Pick & Place	1	put some soapbottle on garbagecan	13
2	put a tissuebox in dresser	26
3	put some soapbar on drawer	15
Clean & Place	4	put a clean soapbar in bathtubbasin	16
5	clean some tomato and put it in fridge	35
Cool & Place	6	put a cool tomato in countertop	30
7	put a cool bread in countertop	27
Heat & Place	8	heat some cup and put it in cabinet	36
Pick Two & Place	9	find two cup and put them in cabinet	36
Examine in Light	10	look at mug under the desklamp	18
We select the GPT as Zero-Shot Planner (GPT) [16] and Inner Monologue (IM) [17] as baseline methods. For the Inner Monologue, the planning goal is the next goal among all candidate goals. For the GPT and DEP, which produce the full plan at once, the planning goal is the full plan (a goal sequence). Then the plan will be executed step-by-step, i.e., the current goal will be given to the controller and select suitable action according to the current state. The goal termination module is also employed with the LLM. For better demonstrate the effectiveness of self-explanation in DEP, we also augment the zero-shot planner with re-planning ability (GPT+RP). All planner methods access the LLM model through OpenAI API (text-davinci-03 model [5]). Since ALFWorld is a text world, the environment will be given a literal description and candidate language-conditioned actions for each state, so the controller under ALFWorld is also LLM-based. Chain-of-Thought [45] is also employed in the controller for better decision-making. All prompts for planner and controller in ALFWorld are listed in Section G.2.
A.1.2 Results
Each task is executed five times, and the average results for each task group are presented in Table 6. BUTLER is the a training-based method, the results are sourced from [41]. Re-planning is a crucial capability in complex and exploratory environments. The short-horizon planning approach (IM) with
Task: put some soapbottle on garbagecan
initial frame
Goal: pick a soapbottle at cabinet 4
Action: go to cabinet 4
Figure 5: Planning in the ALFWorld experiments.
Goal: pick a soapbottle at cabinet 4
Action: open cabinet 4
Goal: pick a soapbottle at cabinet 4
Action: take soapbottle
2 from cabinet 4
Goal: put a soapbottle on garbagecan at garbagecan 1
Action: go to garbage can 1
Goal: put a soapbottle on garbagecan at garbagecan 1
Action: put soapbottle 2 in/on garbagecan 1
re-planning capability outperforms the long-horizon planning approach (GPT) without re-planning capability with a large margin. Furthermore, the long-horizon planning method augmented with re-planning capability (GPT+RP) achieves superior performance ranging from  $10\%$  (GPT) to  $52\%$ . DEP further enhances the feasibility of planning with descriptions and self-explanation. Notably, all planning methods fail on Place Two & Place tasks, which is attributable to LLM's lack of requisite knowledge for this task. It is worth investigating how to effectively incorporate the distinctive knowledge of an environment into LLM.
Table 6: Success rates of tasks in ALFWorld.
Group	BUTLER [41]	GPT [16]	GPT+RP	IM [17]	DEP
Pick & Place	46.0%	33.3%	100.0%	33.3%	93.3%
Clean & Place	39.0%	0.0%	10.0%	50.0%	50.0%
Cool & Place	100.0%	0.0%	30.0%	50.0%	100.0%
Heat & Place	74.0%	0.0%	40.0%	0.0%	80.0%
Pick Two & Place	24.0%	0.0%	0.0%	0.0%	0.0%
Examine in Light	22.0%	0.0%	100.0%	0.0%	100.0%
Average	37.0%	10.0%	52.0%	30.0%	76.0%
Average	.0%	10.0%	52.0%	30.0%	76.0%
A.2 Tabletop Manipulation