SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The Tabletop Manipulation experiments are conducted on a Universal Robot UR5e with a suction gripper in the simulated environments [40].
A.2.1 Tasks
The assessment of all methods is conducted in five seen tasks, as illustrated in Table 7, wherein the seen tasks are employed for training the CLIPort [40] as the controller. The task involves a robotic arm equipped with a gripper, which is tasked with rearranging a number of blocks and bowls on a table to achieve a desired configuration specified via natural language (e.g., "putting the blocks in the bowls with matching colors").
Table 7: Task list in CLIPort.
No	Task Name	Instruction
1	Assembling Kits	Put the objects in the corresponding holes.
2	Towers of Hanoi	Move the rings to the darker brown side.
3	Put Block in Bowl	Match the blocks and the bowls.
4	Packing Shapes	Pack the objects in the brown box.
5	Stack Block Pyramid	Stack the blocks into a pyramid.
Task: assembling-kits
initial frame
1. put the cyan letter L shape in the letter L shape hole
2. put the green hexagon in the hexagon hole
3.put the blue letter M shape in the letter M shape hole
4. put the brown ring in the ring hole
5. put the red letter E shape in the letter E shape hole
Task: put-block-in-bowl
initial frame
1. put the blue blocks in a blue bowl
2. put the red blocks in a red bowl
3.put the yellow blocks in a yellow bowl
Task: packing-shapes
initial frame
1. pack the unicorn toy in the brown box
Figure 6: Planning in the Tabletop Manipulation experiments.
2. pack the butterfinger chocolate in the brown box
3 pack the android toy in the brown box
4. pack the lion figure in the brown box
5. pack the alarm clock in the brown
box
We utilized Inner Monologue (IM) [17] and Zero-shot Planner (GPT) [16] as planning baselines, in addition to comparing with a multi-task CLIPort policy directly trained on long-horizon task instructions (i.e., without utilizing LLM for planning). As CLIPort is a single-step policy that does not spontaneously terminate during policy rollout, we report CLIPort evaluations with Oracle termination (i.e., repeat until the Oracle indicates task completion) and fixed-step termination (i.e., repeat for  $k$  steps). For Inner Monologue, which directly produces the next-step goal and terminates when the LLM ceases to generate new steps, we similarly set the maximum number of steps to be  $k$  for practical considerations. For the zero-shot planner [16] and our DEP, which produce the full plan at once, they are augmented with the LLM-based termination. DEP also involves the description, explanation, and re-planning process. The same  $k$  step is suitable for these two methods. In practice,  $k$  is set as 15. The prompts for all methods are listed in Section G.3. We use the checkpoints provided by CLIPort as the controller and all planner methods access the ChatGPT (as LLM) through OpenAI API (gpt-3.5-turbo model). Each task is evaluated 5 times with different seeds.
A.2.2 Results
The results of each method are listed in Table 8. All LLM-based planning methods perform well on tabletop rearrangement tasks. Given the compact nature of the tabletop environment, the performance gap among the various LLM-planning methods is not as pronounced as in the open-ended Minecraft. This observation underscores the robust generalization capabilities of LLM-based planning methods across diverse environments.
Table 8: Success rates for various methods across different tasks in Tabletop Manipulation environment.
Task	CLIPort [40] +oracle	GPT [16]	IM [17]	DEP
Assembling Kits	60.0%	60.0%	60.0%	60.0%
Towers of Hanoi	100.0%	100.0%	40.0%	100.0%
Put Block in Bowl	100.0%	100.0%	82.0%	100.0%
Packing Shapes	40.0%	40.0%	60.0%	40.0%
Stack Block Pyramid	80.0%	100.0%	40.0%	100.0%
Average	76.0%	80.0%	56.4%	80.0%
B Minecraft Task Details
To fully validate the multitask planning and execution capability of our agent, we choose over 70 tasks from the Minecraft Universe Benchmark [22] as the set of evaluation tasks. These tasks are related to items that can be obtained in the Minecraft overworld. These tasks are also a subset of MineDojo [10] programmatic tasks. Minedojo exists some programmatic tasks sharing the same object item given different conditions (e.g., obtain wool given shear or obtain wool given nothing). Minedojo expands the richness of the same tasks (sharing the same Minecraft item as an object) by giving different initial conditions (e.g., obtain wool given shears or obtain wool given nothing). We keep only the 71 hardest conditions (i.e. given nothing) as tasks.
We list all task names, objects, and their required skills number for planning from Table 9 to Table 16. Object item is used as the basis for the successful completion of the task. These objects cannot be obtained directly from the environment, and usually require multiple goals (i.e., reasoning steps) to be constructed. Here we only consider the number of required goal types, and multiple identical goals are unified into 1 reasoning step. Note that the reasoning steps for each task are not fixed, and as the initial state of the agent and the biome is in change, more reasoning steps may be required to complete it, we only report the most basic case here.
As shown in Figure 4, for each task, a relaxed (longer) maximum episode steps will increase the success rate of the task. To fully test the efficiency of our method, we set an upper limit on the episode length for each task. Since different tasks have different difficulty levels, we double the average completion time of human players for different meta-tasks as the upper limit of the episode. The play time are computed as corresponding maximum steps (i.e., Max. Steps in Table 1) of episode length at  $20\mathrm{Hz}$ .
Table 9: Task details on MT1 Basic set.
Meta-Task	ID	Task Name	Required Skills	Object	Initial Inventory	Instruction
MT1 Basic	1	CraftPlanks	2	planks	Obtain a plank.
2	CraftSticks	3	stick	Obtain a stick.
3	CraftWoodenSlab	4	wooden_slab	Obtain a wooden slab.
4	CraftWoodenPressure	3	wooden_pressure	Obtain a wooden pressure plate.
5	CraftBowl	4	bowl	Obtain a bowl.
6	CraftWoodenButton	3	wooden_button	Obtain a wooden button.
7	CraftChest	4	chest	null	Obtain a chest.
8	CraftOakStairs	4	oak_stairs	Obtain an oak stair.
9	CraftSign	5	sign	Obtain a sign.
10	CraftFence	5	fence	Obtain a fence.
11	CraftFenceGate	5	fence_gate	Obtain a fence gate.
12	CraftBoat	4	boat	Obtain a boat.
13	CraftTrapdoor	4	trapdoor	Obtain a trap door.
14	CraftWoodenDoor	4	door	Obtain a door.
Table 10: Task details on MT2 Tool (Simple) set.
Meta-Task	ID	Task Name	Required Skills	Object	Initial Inventory	Instruction
MT2 Tool (Simple)	15	CraftCraftingTable	3	crafting_table	Obtain a crafting table.
16	CraftWoodenPickaxe	5	wooden_pickaxe	Obtain a wooden pickaxe.
17	CraftWoodenAxe	5	wooden_axe	Obtain a wooden axe.
18	CraftWoodenHoe	5	wooden_hoe	Obtain a wooden hoe.
19	CraftWoodenSword	5	wooden_sword	Obtain a wooden sword.
20	CraftWoodenShovel	5	wooden_shovel	null	Obtain a wooden shovel.
21	CraftFurnace	7	furnace	Obtain a furnace.
22	CraftStonePickaxe	7	stone_pickaxe	Obtain a stone pickaxe.
23	CraftStoneAxe	7	stone_axe	Obtain a stone axe.
24	CraftStoneHoe	7	stone_hoe	Obtain a stone hoe.
25	CraftStoneShovel	7	stone_shovel	Obtain a stone shovel.
26	CraftStoneSword	7	stone_sword	Obtain a stone sword.
Table 11: Task details on MT3 Hunt and Food set.
Meta-Task	ID	Task Name	Required Skills	Object	Initial Inventory	Instruction
MT3 Hunt & Food	27	CraftBed	5	bed	Obtain a bed.
28	CraftPainting	6	painting	Obtain a painting.
29	CraftCarpet	5	carpet	Obtain a carpet.
30	CraftItemFrame	6	item_frame	null	Obtain an item frame.
31	CookPorkchop	9	cooked_porkchop	Cook the porkchop.
32	CookBeef	9	cooked_beef	Cook the beef.
33	CookMutton	9	cooked_mutton	Cook the mutton.
Table 12: Task details on MT4 Dig-Down set.
Meta-Task	ID	Task Name	Required Skills	Object	Initial Inventory	Instruction
MT4 Dig-Down	34	CraftStoneStairs	7	stone_stairs	Obtain a stone stair.
35	CraftStoneSlab	7	stone_slab	Obtain a stone slab.
36	CraftArmorStand	10	armor_stand	Obtain an armor stand.
37	CraftCobblestoneWall	7	cobblestone_WALL	Obtain a cobblestone wall.
38	CraftQuartzBlock	10	quartz_block	Obtain a quartz block.
39	CraftStoneBrick	9	stone_brick	Obtain a stone brick.
40	SmeltStone	9	stone	null	Smelt a stone.
41	CraftTorch	9	torch	Obtain a stone brick.
42	ObtainCoal	8	coal	Mine coal.
43	CraftStoneBrickStairs	10	stonebrick_stairs	Obtain a stone brick.
44	CraftStonePressurePlate	9	stone_pressurePlate	Obtain a stone brick.
45	CraftStoneButton	7	stone_button	Obtain a stone brick.
46	CraftLever	7	level	Obtain a stone brick.
Table 13: Task details on MT5 Equipment set.
Meta-Task	ID	Task Name	Required Skills	Object	Initial Inventory	Instruction
MT5 Equipment	47	EquipLeatherBoots	5	leather_boots	Equip the leather boot.
48	EquipLeatherChestplate	5	leather_chestplate	Equip the leather chestplate.
49	EquipLeatherHelmet	5	leather_helmet	Equip the leather helmet.
50	EquipLeatherLeggings	5	leather-leggedness	Equip the leather leggings.
51	EquipShield	11	shield	null	Equip the shield.
52	EquipIronChestplate	11	iron_chestplate	Equip the iron chestplate.
53	EquipIronLeggings	11	iron-leggedness	Equip the iron leggings.
54	EquipIronHelmet	11	iron_helmet	Equip the iron helmet.
55	EquipIronBoots	11	iron_boots	Equip the iron boots.
Table 14: Task details on MT6 Tool (Complex) set.
Meta-Task	ID	Task Name	Required Skills	Object	Initial Inventory	Instruction
MT6 Tool (Complex)	56	CraftBucket	11	bucket	Obtain a bucket.
57	CraftShears	11	shears	Make shears.
58	CraftIronPickaxe	11	iron_pickaxe	Obtain an iron pickaxe.
59	CraftIronAxe	11	iron_axe	null	Obtain an iron axe.
60	CraftIronHoe	11	iron_hoe	Obtain an iron hoe.
61	CraftIronShovel	11	iron_shovel	Obtain an iron shovel.
62	CraftIronSword	11	iron_sword	Obtain an iron sword.
Table 15: Task details on MT7 Iron-Stage set.
Meta-Task	ID	Task Name	Required Skills	Object	Initial Inventory	Instruction
MT7 Iron-Stage	63	CraftIronBars	11	iron-bars	Obtain an iron bar.
64	CraftIronNugget	11	iron_nugget	Obtain an iron nugget.
65	CraftMinecart	11	minecart	Obtain a minecart.
66	CraftHopper	12	hopper	Obtain a hopper.
67	CraftHopperMinecart	14	hopper_minecart	Obtain a hopper minecart.
68	CraftFurnaceMinecart	12	furnace_minecart	Obtain a furnace minecart.
69	CraftCauldron	11	cauldron	null	Obtain a cauldron.
70	CraftChestMinecart	13	chest_minecart	Obtain a chest minecart.
71	CraftIronDoor	11	iron大门	Obtain an iron door.
72	CraftIronTrapdoor	11	iron_trapdoor	Obtain an iron trapdoor.
73	CraftTripwireHook	11	tripwire hook	Obtain a tripwire hook.
74	CraftHWPressurePlate	11	heavy_weightedplate	Obtain a heavy weighted plate.
75	CraftRail	11	rail	Obtain a rail.
Table 16: Task details on MT8 Challenge set.
Meta-Task	ID	Task Name	Required Skills	Object	Initial Inventory	Instruction
Challenge MT8	76	ObtainDiamond	12	diamond	null	Obtain a diamond.
C DEPS Implementation Details
We study three different implementations of DEPS for each of the experimental settings. While each version incorporates description and self-explanation to improve planning of LLM, there are differences in the internal components of each system, as seen in Table 17.
Table 17: Comparison between different versions of DEPS implemented in three different environments.
Minecraft	ALFWorld	Tabletop Manipulation
LLM	code-davinci-02	text-davinci-03	gpt-3.5-turbo
Controller	Behavior Cloning Learned	LLM-based	CLIPort
Descriptor	Inventory Description	Env Support	heuristics
Explainer	LLM-based	LLM-based	LLM-based
Selector	Horizon Prediction Module	N/A	N/A
C.1 Controller
As the name implies, tasks in Minecraft are usually related to mine and craft goals. Mine goals require the agent to collect raw materials from the environment using the appropriate tools. Craft goals ask the agent to synthesize using existing materials. Any raw material used requires the agent to collect through suitable tools (e.g., diamonds can only be collected by an iron pickaxe or a better pickaxe). So a task usually requires dozens of step-by-step mine and craft goals, as the required skills in Table 9. Note that the successful execution of a task needs to satisfy certain exact numerical constraints due to the presence of strict generation recipes in the environment (e.g., a log can craft 4 planks, so harvesting 6 planks requires at least 2 logs). When the number of materials collected is not enough, the goal cannot be completed successfully. When more materials are collected than actually needed, the execution success rate of the task could also be reduced because the plan can not be finished under the maximum action steps.
Table 18: The success rate of different skill/goal with imitation learning controller.
ID	Skill Description	Success Rate	Episode Length
0	Mine 1 oak wood	0.39	600
1	Mine birch wood	0.29	600
2	Mine 1 cobblestone with pickaxe	0.95	600
3	Mine 1 stone with pickaxe	0.70	600
4	Mine 1 seed	0.18	600
5	Mine 1 leaves with shears	0.68	600
6	Mine 1 dirt	0.54	600
7	Mine 1 iron ore with stone pickaxe	0.40	3000
8	Mine 3 iron ore with stone pickaxe	0.16	3000
9	Mine 1 diamond with iron pickaxe	0.35	12000
10	Mine 1 diamond with stone pickaxe	0.00	12000
11	Kill 1 sheep with axe	0.44	600
12	Kill 1 cow with axe	0.60	600
13	Kill 1 chicken with axe	0.46	600
14	Kill 1 pig with axe	0.49	600
15	Kill 1 llama	0.50	600
16	Equip tool on mainhand	1.00	600
Craft w/o crafting_table	1.00	600
17-261	Craft w/ crafting_table	0.90	600
Smelt w/ furnace	0.80	600
We designed the agent's skill space based on these goals, as shown in Table 18, with a total of 262 goals. Every goal is designed with an objective item (e.g., 1 minercraft: cobblestone for skill "Mine 1 cobblestone with pickaxe"), which is used to evaluate the achievement of the goal. The skill, as a goal-conditioned policy  $\pi(a|s, g)$  for decision-making, maps the current state  $s$  and goal  $g$  to action  $a$ . The goal is specified as natural language instructions here, which is similar to [4].
When training the controller, we adopt the observation space provided by MineDoJo [10], which includes an RGB camera view, yaw/pitch angle, GPS location, and the type of  $3 \times 3$  blocks surrounding the agent. We discretize the original multi-discrete action space provided by MineDojo into 42 discrete actions. We use the proposed imitation learning method proposed by [6] in training. To be specific, a modified goal-sensitive Impala CNN is used as the backbone network. The success rate under a fixed episode length of every skill is listed in Table 18.
C.2 LLM as Planner
DEPS relies on Large Language Models (LLMs) to generate language-based plans. In our Minecraft experiment, we chose Codex [8] as the LLM Planner because it can accept longer input tokens and is cost-effective. However, DEPS is compatible with various types of LLMs. Therefore, we used GPT3 [5] and ChatGPT as LLM Planners in the ALFWorld and Tabletop Manipulation experiments, respectively. Due to the effective planning and error correction performance of DEPS, the initial plan generated by the LLM has little impact on the final performance of the Agent. We also conduct ablation experiments on
even if the initial plan generated by the LLM has low accuracy, DEPS can generate a final feasible plan through self-explanation and re-planning. Therefore, we conducted ablation experiments on LLM in Minecraft.
We choose Codex [8], ChatGPT, GPT3 [5], and recent GPT-4 [31] as Planners. We used Vanilla Planner [16] as baselines and excluded the re-planning process. Given the same prompt with DEPS, the performance of baseline models reflects the planning ability of different LLMs. The success rate of baseline and DEPS on different LLMs are reported in Table 19.
Table 19: Success rates for different LLMs on Minecraft tasks.
Group	Codex [8]	GPT-3 [5]	ChatGPT	GPT-4 [31]
baseline	DEPS	baseline	DEPS	baseline	DEPS	baseline	DEPS
MT1	28.6	79.8	27.2	75.4	20.3	70.2	49.2	89.3
MT2	37.1	79.5	42.1	76.3	28.2	68.5	48.3	85.0
MT3	15.1	62.4	7.8	58.7	3.2	50.4	38.04	63.4
MT4	15.9	53.3	6.7	50.2	4.8	47.8	27.0	55.7
MT5	3.2	29.2	2.7	17.2	0.8	16.3	15.7	32.2
MT6	0.5	13.8	0.3	7.9	0.3	6.0	4.9	16.19
MT7	0.6	12.6	0.4	5.3	0.5	5.2	3.1	16.41
The success rate of Vanilla Planner varies on the LLMs. The GPT-4 baseline achieved an initial plan accuracy twice as high as the baselines on other LLMs, demonstrating superior planning ability. After being augmented by Descriptor, Explainer, and Selector, DEPS based on different LLMs showed almost identical success rates. This indicates that DEPS-augmented LLMs can generate more feasible plans in open-world environments even if the initial plan is less successful.
It is noteworthy that DEPS is constrained by the maximum token limits of various models, which dictate the maximum re-planning rounds that can be supported. Longer re-planning rounds tend to yield superior performance, particularly in long-horizon tasks requiring more skills (in MT6-MT7), as detailed in the Section 4.3.
Since we use pretrained LLM as a planner, it indeed requires exposure to a large amount of Minecraft-related corpus during the pretraining phase. Considering that Minecraft is one of the most popular games worldwide, there is relatively abundant data about Minecraft available online. We conducted experiments using open-source pretrained LLaMA2-70B on several Minecraft tasks and found that DEPS based on LLaMA2 also performs reliable planning under Minecraft conditions. Considering limited training data used by LLaMA2, we further finetuned an open-source language model (LLaMA2-13B) using Minecraft texts obtained from the internet which exhibited better planning performance. The results are shown in Table 20.
Table 20: Results of DEPS based on open-sourced LLaMA language models.
Language Model	CraftingTable	WoodenPickaxe	Furnace	StonePickaxe
Pretrained LLaMA2-70B [44]	60.0	50.0	40.0	50.0
Finetuned LLaMA2-13B [44]	90.0	80.0	70.0	80.0
OpenAI Codex [8]	90.0	80.0	66.7	73.3
C.3 LLM as Explainer
Given the description and previous plan, the explainer can generate a self-explanation of the failure of the current plan and give instructions to fix the bugs. The explainer is implemented with the OpenAI completion mode based on text-davinci-03 models. The prompt for the explainer is listed in Listing 1.
Here are some actions that the agent fails to perform in Minecraft. Please give the explanation of action execution failure according to the current inventory information of the agent.   
#   
Failed Action: mine({'iron_ore':1}, null); # step 5: mine 1 iron_ore without tool Current Inventory: null   
Explanation: Because mining iron_ore needs to use the tool stone_pickaxe, but my inventory does not have stone_pickaxe. So I need to craft stone_pickaxe first.   
#   
Failed Action: craft({'stone_pickaxe':1},{'cobblestone':3,'stick':2}, ' crafting_table'); # step 1: craft 1 stone_pickaxe from 3 cobblestone and 2 stick, on crafting_table   
Current Inventory: null   
Explanation: Because crafting stone_pickaxe needs to have 3 cobblestone and 2 stick in inventory, but my inventory does not have cobblestone and stick. So I need to mine cobblestone and craft stick first.
Listing 1: Prompt for Explainer in Minecraft tasks
## 
Failed Action: craft({'stick':4},{'planks':2},null); # step 3: craft 4 stick from 2 planks first
Current Inventory: null
Explanation: Because crafting stick needs to have planks in inventory, but my inventory does not have planks. So I need to craft planks first.
##
C.4 Other modules
Goal Parser We need to map the plan expressed in free-form language to the pre-defined controller skills set. We use the LLM as an automatic parser to parse the language plan first. For the goals not following pre-defined code expression, we calculate its semantic distance to the skills by cosine similarity with pre-trained Sentence-Bert model [37] and select the most similar skill as the corresponding goal. All executable goals are listed in Appendix C. The LLM-based parser is general and can be transferred to other domains easily by modifying the prompt. The prompt for Minecraft parser is listed in Listing 2.
Listing 2: Prompt for Goal Parser in Minecraft tasks
Extract the action name, action type, goal object, tool and action rank from the input text.  
input: mine({'log':3}, null); # step 1: mine 3 log without tool  
name: mine_log  
action: mine  
object: {'log':3}  
tool: null  
rank: 1  
##  
input: craft({'planks':12}, {'log':3}, null); # step 2: craft 12 planks from 3 log  
name: craft_planks  
action: craft  
object: {'planks':12}  
materials: {'log':3}  
tool: null  
rank: 2  
##
Success Detector The successful execution of a plan is contingent upon the agent's perception of the current goal's completion status, which is assessed by the success detector. In Minecraft, agents possess an inventory that contains all pertinent information regarding the agent's current state. Thus, the Success Detector can be implemented by monitoring changes in object information within the item inventory. In other scenarios, we can query the LLM to ascertain whether the agent has accomplished a general goal by describing the agent's current state. Alternatively, in certain environments [40], the execution of a goal is linked to the agent's current reward, signifying that these rewards can serve as automatic success detectors.
Prompt The generalization of the LLM to different tasks relies on well-designed prompts and related demonstrations [28]. Given an instruction command (e.g., ObtainDiamond) as task  $T$ , a prompt generator (ProG) will translate  $T$  into prompt text. We also added two DEP examples in the prompt as demonstrations to make the LLM output familiar to the chain-of-thought thinking and structural output. We also design a chain-of-thought code/comments-type planning prompt to better demonstrate the capabilities of LLM. All messages are modified to suitable prompts through the prompt-generator before being input to LLM, including task  $T$  and description  $d_t$ . The full prompt sentences and interaction logs are listed in Appendix H.
D Comparison with other LLM-based Planners
The architectures of the different LLM-based planners are illustrated in Figure 7. Where (b) describes the information in the environment into LLM via scene descriptor and success detector, and directly
(a) Ours
(b) Short-horizon action planning
Figure 7: Comparison of LLM-based planner architecture. (a), (b), (c), (d) represents planner of ours, Inner Monologue [17], Zero-Shot Planner [16] and Zero-Shot Planner with re-planning process, respectively.
(c) Long-Horizon one-shot planning
(d) Long-Horizon planning & re-planning
plans the next goal/action, (c) is Zero-Shot planner [16], which generates the step-by-step goal sequences as plan and ignores the environment state and execution feedback, (d) is the Zero-Shot planner augmented with textual feedback and re-planning process. DEPS further rethink and explain the feedback of previous plans explicitly with the descriptor and explainer. The LLM-based planner will re-plan the task according to the explanation, as demonstrated in Figure 7(a). In addition, the goal Selector further improves the executability of the LLM plan.
E Discussion on ObtainDiamond Task
mine log
craft planks
craft crafting table
craft stick
craft wooden pickaxe
mine stone
craft stone pickaxe
Xine iron ore
Figure 8: The milestone goals of the ObtainDiamond task.
craft furnace
smelt iron ingot
craft iron pickaxe
mine diamond