SlowGuess/NeurIPS_2023_OCR huggingface dataset:
As outlined in Section 4.4, ObtainDiamond task is a formidable task within the open-ended Minecraft environment. Given the necessity to explore an infinitely expansive world, an efficient plan can prove advantageous, as shown in Figure 8. The task is allotted a maximum of 12,000 steps to interact with the environment, which is comparable to that of human performance [14]. Rather than manually devising explicit hierarchical rewards, we opt to utilize DEPS for generating a hierarchical plan, which is then transferred to the downstream controller to progressively achieve each goal. When equipped with an Oracle Controller, DEPS yields a success rate of  $60\%$  for ObtainDiamond. In our experimentation, we employed Behavior Cloning to train a Controller agent [6]. DEPS+BC Controller achieved a success rate of  $0.6\%$  in randomly generated Minecraft worlds. The primary bottleneck impeding overall agent success rate lies within the goal-conditioned Controller, not the plans generated by DEPS. Thus, it is worth exploring the development of a data-efficient Controller capable of accepting Language goals.
Another rationale for using DEPS is that, akin to reality, materials in Minecraft possess quantity constraints, and durability for tools. In ObtainDiamond task, an iron pickaxe is typically insufficient
to support the agent, given the rarity of diamonds within the environment (which are predominantly found between depths of 2-16 layers and appear only  $0.0846\%$  of the time). The robust re-planning capabilities of DEPS can facilitate the generation of a feasible plan (initiating with crafting an iron-pickaxe) based on the agent's current state.
Additionally, we report the milestones, which demonstrate the decreasing success rate of subsequent tasks in Figure 9 attributable to the task's inherent complexity and Controller constraints.
Figure 9: Success rate of milestone items for mining diamond.
F Success Rates of ALL Tasks in Minecraft
We report the complete and detailed success rate table of all tasks for different methods in Table 21, including Zero-shot Planner [16], ProgPrompt [42], Chain-of-Thought [45], Inner Monologue [17], Code as Policies [20], and proposed methods (i.e., DEP w/o Selector, and DEPS).
All tasks are executed for at least 30 times across different world seeds, given the same initial conditions. The birth positions of the world are random according to the seed. The average success rates are listed in Table 21. Our approach is state-of-the-art on almost all tasks, especially on difficult tasks that require more skills.
G Prompt for Different Tasks and Different Methods
G.1 Prompt for Minecraft Tasks
G.1.1 DEPS
User: You are a helper agent in Minecraft. You need to generate the sequences of goals for a certain task in Minecraft.  
Planner: OK.
User: How to obtain 1 wooden_button in Minecraft step-by-step?  
Planner: The code for obtaining 1 wooden_button is as follows:  
def obtain_1 Wooden_button inventory = {}):  
    mine({'log':1}, null); # step 1: mine 1 log without tool  
    craft({'planks':4}, {'log':1}, null); # step 2: craft 4 planks from 1 log  
    craft({'wooden_button':1}, {'planks':1}, null); # step 3: craft 1 wooden_button from 1 planks  
    return "wooden_button"
Descriptor: I succeed in step 1, 2, 3.  
User: Good. I finish all steps and I obtain 1 stone_pickaxe successfully.  
Planner: OK.
User: How to obtain 1 stone_sword in Minecraft step-by-step?  
Planner: The code for obtaining stone_sword is as follows:  
def obtain_1_stone_sword inventory = {}):  
    mine({'log':3}, null); # step 1: mine 3 log without tool  
    craft({'planks':12}, {'log':3}, null); # step 2: craft 12 planks from 3 log  
    craft({'stick':4}, {'planks':2}, null); # step 3: craft 4 stick from 2 planks  
    mine({'cobblestone':2}, null); # step 4: mine 2 cobblestone without tool  
    craft({'stone_sword':1}, {'cobblestone':2, 'stick':1}, 'crafting_table'); #  
    step 5: craft 1 stone_sword from 2 cobblestone and 1 stick, on  
    crafting_table  
return 'stone_sword'
Descriptor: I succeed on step 1, 2, 3, but I fail on step 4 "mine({'cobblestone ':2}, null)"; My inventory now has 10 planks, 4 stick.
Explainer: Because mining cobblestone needs to use the tool wooden_pickaxe.  
User: Please fix the above errors and replan the task "obtain 1 stone_sword".
RePlanner: The code for crafting stone_sword is as bellows:  
def obtain_1_stone_sword inventory = {'planks':10, 'stick':4} :  
    craft{'wooden_pickaxe':1}, {'planks':3, 'stick':2}, 'crafting_table'}; # step 1: craft 1 wooden_pickaxe from 3 planks and 2 stick, on crafting_table  
    mine{'cobblestone':2}, 'wooden_pickaxe'}; # step 2: mine 2 cobblestone with wooden_pickaxe  
    craft{'stone_sword':1}, {'cobblestone':2, 'stick':1}, 'crafting_table'}; # step 3: craft 1 stone_sword from 2 cobblestone and 1 stick, on crafting_table  
    return 'stone_sword'
Descriptor: I fail on step 1 "craft({'wooden_pickaxe':1}, {'planks':3, 'stick':2}, 'crafting_table')"; My inventory now has 10 planks, 4 stick.
Explainer: Because the action needs to use the tool crafting_table, but I do not have it.
User: Please fix the above errors and replan the task "obtain 1 stone_sword".
RePlanner: The code for crafting stone_sword is as bellows:  
def obtain_1_stone_sword inventory = {'planks': 10, 'stick': 4}):  
    craft({'crafting_table': 1}, {'planks': 4}, null); # step 1: craft 1  
    crafting_table from 4 planks  
    craft({'wooden_pickaxe': 1}, {'planks': 3, 'stick': 2}, 'crafting_table'); # step 2: craft 1 wooden_pickaxe from 3 planks and 2 stick, on crafting_table  
    mine({'cobblestone': 2}, 'wooden_pickaxe'); # step 3: mine 2 cobblestone with wooden_pickaxe  
    craft({'stone_sword': 1}, {'cobblestone': 2, 'stick': 2}, 'crafting_table'); # step 4: craft 1 stone_sword from 2 cobblestone and 1 stick, on crafting_table  
    return 'stone_sword'
Descriptor: I succeed in step 1, 2, 3, 4.  
User: Good. I finish all steps and I obtain 1 stone_pickaxe successfully.
Planner: OK.
Listing 3: Prompt for DEPS in Minecraft tasks
G.1.2 Baselines
Listing 4: Prompt for Zero-shot Planner [16] in Minecraft tasks.
User: You are a helper agent in Minecraft. You need to generate the sequences of goals for a certain task in Minecraft.  
Planner: OK.  
User: How to obtain 1 wooden_button in Minecraft step-by-step?  
Planner: The plan is as follows:  
step 1: mine 1 log  
step 2: craft 4 planks  
step 3: craft 1 wooden_button  
##  
User: How to obtain stone_sword in Minecraft step-by-step?  
Planner: The plan is as follows:  
step 1: mine 3 log  
step 2: craft 12 planks  
step 3: craft 4 stick  
step 4: craft 1 crafting_table  
step 5: craft 1 wooden_pickaxe  
step 6: mine 2 cobblestone  
step 7: craft 1 stone_sword  
##
Listing 5: Prompt for ProgPrompt [42] in Minecraft tasks.
User: You are a helper agent in Minecraft. You need to generate the sequences of goals for a certain task in Minecraft.   
Planner: OK.   
User: How to obtain 1 wooden_button in Minecraft step-by-step?   
Planner: The code for obtaining wooden_button is as follows:   
from actions import mine_log, craft_planks, craft_wooden_button def obtain_wooden_button inventory  $=$  {): mine_log(num  $= 1$  ); # step 1: mine 1 log craft_planks(num  $= 4$  ); # step 2: craft 4 planks craft_wooden_button(num  $= 1$  ); # step 3: craft 1 wooden_button return "wooden_button"   
User: How to obtain 1 stone_sword in Minecraft step-by-step?   
Planner: The code for obtaining stone_sword is as follows:   
from actions import mine_log, craft_planks, craft_stick, craft_crafting_table, craft_wooden_pickaxe, mine_cobblestone, craft_stone_sword   
def obtain_stone_sword inventory  $=$  {): mine_log(num  $= 3$  ); # step 1: mine 3 log craft_planks(num  $= 12$  ); # step 2: craft 12 planks craft_stick(num  $= 4$  ); # step 3: craft 4 stick craft_crafting_table(num  $= 1$  ); # step 4: craft 1 crafting_table craft_wooden_pickaxe(num  $= 1$  ); # step 5: craft 1 wooden_pickaxe mine_cobblestone(num  $= 2$  ); # step 6: mine 2 cobblestone craft_stone_sword(num  $= 1$  ); # step 7: craft 1 stone_sword return "stone_sword"
User: You are a helper agent in Minecraft. You need to generate the sequences of goals for a certain task in Minecraft.  
Planner: OK.  
User: You are a helper agent in Minecraft. You need to generate the sequences of goals for a certain task in Minecraft.
Listing 6: Prompt for Chain-of-Thought [45] in Minecraft tasks
Planner: OK.   
User: How to obtain 1 wooden_button in Minecraft step-by-step?   
Planner: The plan is as follows:   
step 1: mine 1 log without tool   
Thinking: Inventory will add 1 log, as inventory  $= 1$  .log   
step 2: craft 4 planks from 1 log   
Thinking: Inventory will add 4 planks and delete 1 log, as inventory  $= 4$  .planks   
step 3: craft 1 wooden_button from 1 planks   
Thinking: Inventory will add 1 wooden_button and delete 1 planks, as inventory  $= 3$  . planks + 1\*wooden_button   
```   
```   
User: How to obtain stone_sword in Minecraft step-by-step?   
Planner: The plan is as follows:   
step 1: mine 3 log without tool   
Thinking: Inventory will add 3 log, as inventory  $= 3$  .log.   
step 2: craft 12 planks from 3 log   
Thinking: Inventory will add 12 planks and delete 3 log, as inventory  $= 12$  .planks   
step 3: craft 4 stick from 2 planks   
Thinking: Inventory will add 4 stick and delete 2 planks, as inventory  $= 10$  .planks + 4\*stick   
step 4: craft 1 crafting_table from 4 planks   
Thinking: Inventory will add 1 crafting_table and delete 4 planks, as inventory  $= 6$  .planks + 4\*stick + 1\*crafting_table   
step 5: craft 1 wooden_pickaxe from 3 planks and stick, on crafting_table   
Thinking: Inventory will add 1 wooden_pickaxe, delete 3 planks and delete 2 stick, as inventory  $= 3$  .planks + 2\*stick + 1\*crafting_table + 1\*wooden_pickaxe   
step 6: mine 2 cobblestone with wooden_pickaxe   
Thinking: Inventory will add 2 cobblestone, as inventory  $= 3$  .planks + 2\*stick + 1\*crafting_table + 1\*wooden_pickaxe + 2\*cobblestone   
step 7: craft 1 stone_sword from 2 cobblestone and 1 stick, on crafting_table   
Thinking: Inventory will add 1 stone_sword, delete 2 cobblestone, and delete 1 stick, as inventory  $= 3$  .planks + 1\*stick + 1\*crafting_table + 1\*wooden_pickaxe + 1\*stone_sword
User: You are a helper agent in Minecraft. You need to generate the sequences of goals for a certain task in Minecraft.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
User: Obtain 1 wooden_button in Minecraft step-by-step.
Scene: My inventory has nothing.
Planner: mine 1 log
Robot Thought: mining log needs no tool
Successful Action: True
Scene: My inventory has 1 log.
Planner: craft 4 planks
Robot Thought: crafting 4 planks needs 1 log
Scene: My inventory has 4 planks
Planner: craft 1 wooden_button
{Robot Thought}: crafting 1 wooden_button needs 1 planks
User: Obtain 1 stone_sword in Minecraft step-by-step.
Planner: mine 3 log
Scene: My inventory has 3 log.
Planner: craft 12 planks
Robot Thought: crafting 12 planks needs 3 log
Listing 7: Prompt for Inner Monologue [17] in Minecraft tasks
Successful Action: True   
Scene: My inventory has 12 planks   
Planner: craft 4 stick   
Robot Thought: crafting 4 stick needs 2 planks   
Successful Action: True   
Scene: My inventory has 10 planks, 4 stick   
Planner: craft 1 wooden_pickaxe   
Robot Thought: crafting 1 wooden_pickaxe needs 3 planks and 2 stick on crafting_table   
Successful Action: False   
Scene: My inventory has 10 planks, 4 stick   
Planner: craft 1 crafting_table   
Robot Thought: crafting 1 crafting_table needs 4 planks   
Successful Action: True   
Scene: My inventory has 6 planks, 4 stick, 1 crafting_table   
Planner: craft 1 wooden_pickaxe   
Robot Thought: crafting 1 wooden_pickaxe needs 3 planks and 2 stick, on crafting_table   
Successful Action: True   
Scene: My inventory has 3 planks, 2 stick, 1 crafting_table, 1 wooden_pickaxe   
Planner: mine 2 cobblestone   
Robot Thought: mining cobblestone needs wooden_pickaxe as tool   
Successful Action: True   
Scene: My inventory has 3 planks, 2 stick, 1 crafting_table, 1 wooden_pickaxe, 2 cobblestone   
Planner: craft 1 stone_sword   
Robot Thought: crafting 1 stone_sword needs 2 cobblestone and 1 stick, on crafting_table   
Successful Action: True
User: How to obtain 1 wooden_button in Minecraft step-by-step?   
Planner: The code for obtaining wooden_button is as follows:   
def craft_stone_sword inventory  $=$  {): # step 1: mine 1 log without tool while not inventory.has({'log':1}): mine(obj  $=$  "log",tool  $=$  null); # step 2: craft 4 planks from 1 log while not inventory.has({'planks':4}): assert inventory.has({'log':1}),"I do not have enough log for crafting planks" craft(obj  $=$  "planks",tool  $=$  null); # step 3: craft 1 wooden_button from 1 planks while not inventory.has({'wooden_button':1}): assert inventory.has({'planks':1}),"I do not have enough planks for crafting wooden_button" craft(obj  $=$  "wooden_button",tool  $=$  null); return "wooden_button"   
##
User: How to obtain 1 stone_sword in Minecraft step-by-step?   
Planner: The code for obtaining stone_sword is as follows:   
def craft_stone_sword inventory  $=$  {}): #step 1: mine 3 log without tool while not inventory.has({'log':3}): mine(obj  $=$  "log",tool  $=$  null); #step 2: craft 12 planks from 3 log while not inventory.has({'planks':12}): assert inventory.has({'log':1}), "I do not have enough log for crafting planks" craft(obj  $=$  "planks",tool  $=$  null); #step 3: craft 4 stick from 2 planks while not inventory.has({'stick':4}):
Listing 8: Prompt for Code as Policies [20] in Minecraft tasks
assert inventory.has({'planks':2}), "I do not have enough planks for crafting stick"  
craft(obj = "stick", tool = null);  
# step 4: craft 1 crafting_table from 4 planks  
while not inventory.has({'crafting_table':1}):  
    assert inventory.has({'planks':4}), "I do not have enough planks for crafting table"  
    craft(obj = "crafting_table", tool = null)  
# step 5: craft 1 wooden_pickaxe from 3 planks and 2 stick, on crafting_table  
while not inventory.has({'wooden_pickaxe':1}):  
    assert inventory.has({'stick':2}), "I do not have enough planks for crafting wooden_pickaxe"  
    assert inventory.has({'crafting_table':1}), "I do not have the tool crafting_table for crafting wooden_pickaxe"  
    craft(obj = "crafting_table", tool = "crafting_table")  
# step 6: mine 2 cobblestone with tool wooden_pickaxe  
while not inventory.has({'cobblestone':2}):  
    assert inventory.has({'wooden_pickaxe':1}), "I do not have the tool wooden_pickaxe for mining cobblestone"  
    mine(obj = "cobblestone", tool = "wooden_pickaxe")  
# step 7: craft 1 stone_sword from 2 cobblestone and 1 stick on crafting_table  
while not inventory.has({'stone_sword':1}):  
    assert inventory.has({'stick':1}), "I do not have enough stick for crafting stone_sword"  
    assert inventory.has({'crafting_table':1}), "I do not have the tool crafting_table for crafting wooden_pickaxe"  
    craft(obj = "stone_sword", tool = "crafting_table")  
return "stone_sword"
G.2 Prompt for ALFWorld Tasks
G.2.1 DEPS
Human:  $- =$  Welcome to TextWorld, ALFRED!  $= -$
Descriptor: You are in the middle of a room. Looking quickly around you, you see a cabinet 1, a diningtable 1 and a fridge 1.
Human: Your task is to: put a cool tomato in cabinet. Please generate the sequences of sub-goals to complete the task.
Planner: The task is to put a cool tomato in cabinet. The plan is below:
1 pick a tomato at fridge 1
2 cool a tomato at fridge 1
3 Put a tomato in cabinet at cabinet 1
Descriptor: I fail on step 1 pick a tomato at fridge 1. I'm not carrying a tomato and there is no tomato, so the goal can't be accomplished.
Explainer: It's probably because there isn't a tomato at fridge 1. The task is to put a cool tomato in cabinet. So I should pick a tomato somewhere else. I'm most likely to pick a tomato at diningtable 1.
Planner: So the new plan is below:
1 pick a tomato at diningtable 1
2 put a tomato in cabinet at cabinet 1
Descriptor: I succeed on step 1.
Descriptor: I failed on step 2 put a tomato in cabinet at cabinet 1. I put a tomato in cabinet, but tomato 2 may not be a cool tomato, so the goal is not accomplished.
Explainer: It's probably because the tomato isn't cool enough. The task is to put a cool tomato in cabinet. So I should cool the tomato. I'm most likely to cool a tomato at fridge 1.
#3 put a tomato in cabinet at cabinet 1
Descriptor: I succeed on step 2.
Descriptor: I succeed on step 3. I finish the task!!!
Human:  $\equiv$  Welcome to TextWorld, ALFRED!  $\equiv -$
Descriptor: You are in the middle of a room. Looking quickly around you, you see a desk 1, and a shelf 1.
Human: Your task is to: look at mug under the desklamp. Please generate the sequences of sub-goals to complete the task.
Planner: The task is to look at mug under the desklamp. The plan is below:
1 pick up a mug at shelf 1
2 use the desklamp at desk 1
Descriptor: I succeed on step
Descriptor: I succeed on step 2. I finish the task!!!
Listing 9: Prompt for DEP in ALFWorld tasks
Human: You are in the middle of a room. You are not carrying anything. Your goal is to: pick a plate at diningtable 1.
The 5 admissible actions of the current situation are :
1. go to cabinet 1
2. go to diningtable 1
3. go to fridge 1
4. inventory
5. look
AI: The goal is to pick a plate at diningtable 1. I'm not at diningtable 1. So I should go to diningtable 1 first. So the answer is 2.
Human: You arrive at loc 17. On the diningtable 1, you see a plate 2, and a tomato 2.
The 7 admissible actions of the current situation are :
1. examine diningtable 1
2. go to cabinet 1
6. take plate 2 from diningtable 1
7. take tomato 2 from diningtable 1
AI: The goal is to pick a plate at diningtable 1. I'm at diningtable 1, and there is a plate 2. So I should take plate 2 from diningtable 1. So the answer is 6.
Human: You pick up the plate 2 from the diningtable 1.
The 6 admissible actions of the current situation are :
6. put plate 2 in/on diningtable 1
AI: The goal is to pick a plate at diningtable 1. According to Human, I pick up the plate 2 from diningtable 1. So the goal is accomplished.
Human: You are facing fridge 1. You are carrying plate 2. Your goal is to: cool a plate at fridge 1.
1. examine fridge 1
3. go to diningtable 1
6. cool plate 2 with fridge 1
AI: The goal is to cool a plate at fridge 1. I'm at fridge 1. So I should cool
plate 2 with fridge 1. So the answer is 6.
Human: You cool the plate 2 using the fridge 1.
AI: The goal is to cool a plate at fridge 1. According to Human, I cool the plate 2 using the fridge 1. So the goal is accomplished.
Human: You are facing diningtable 1. You are not carrying anything. Your goal is to: pick a plate at diningtable 1.
6. take tomato 2 from diningtable 1
AI: The goal is to pick a plate at diningtable 1. I'm at diningtable 1, but there is no action to pick a plate. So the answer is 0 (none). So the goal can't be accomplished, probably because there is no plate at diningtable 1.
Human: You are facing fridge 1. You are not carrying anything. Your goal is to: cool a plate at fridge 1.
AI: The goal is to cool a plate at fridge 1. I'm at fridge 1, but there is no action to cool a plate. So the answer is 0 (none). So the goal can't be accomplished, probably because I'm not carrying a plate.
Human: You are facing fridge 1. You are carrying a plate. Your goal is to: pick a tomato at fridge 1.
AI: The goal is to pick a tomato at fridge 1. I'm at fridge 1, but there is no action to pick a tomato. So the answer is 0 (none). So the goal can't be accomplished, probably because I can't pick a tomato while carrying a plate.
Listing 10: Prompt for the controller of DEP in ALFWorld tasks
G.2.2 Baselines
Human:  $- =$  Welcome to TextWorld, ALFRED! --
Your task is to: cool some plate and put it in cabinet.
Scene: You are in the middle of a room. Looking quickly around you, you see a cabinet 1, a diningtable 1 and a fridge 1.
Robot Thought: The task is to cool some plate and put it in cabinet. I'm carrying nothing. So I need to find a plate first. There are cabinet, diningtable and fridge, but no plate. I'm most likely to find a plate near diningtable. So I should go to diningtable 1. So the answer is 2.
Robot Action: go to diningtable 1
Scene: You arrive at loc 17. On the diningtable 1, you see a plate 2, and a tomato 2.
Robot Thought: The task is to cool some plate and put it in cabinet. I'm carrying nothing. So I need to find a plate first. There is a plate 2. So I should take plate 2 from diningtable 1. So the answer is 6.
Robot Action: take plate 2 from diningtable 1
Scene: You pick up the plate 2 from the diningtable 1.
2. examine plate 2
3. go to cabinet 1
4. go to fridge 1
5. inventory
6. look
7. put plate 2 in/on diningtable 1
Robot Thought: The task is to cool some plate and put it in cabinet. I'm carrying a plate 2. So I need to cool a plate first. There are cabinet, diningtable, and fridge. I'm most likely to cool a plate near fridge. So I should go to fridge 1. So the answer is 4.
Robot Action: go to fridge 1
Scene: You arrive at loc 41. The fridge 1 is closed.
Listing 11: Prompt for Inner Monologue [17] in ALFWorld tasks
The 8 admissible actions of the current situation are :   
1. cool plate 2 with fridge 1   
2. examine fridge 1   
3. examine plate 2   
4. go to cabinet 1   
5. go to diningtable 1   
6. inventory   
7. look   
8. open fridge 1   
Robot Thought: The task is to cool some plate and put it in cabinet. I'm carrying a plate 2. So I need to cool a plate first. There is a fridge 1. So I should cool plate 2 with fridge 1. So the answer is 1.   
Robot Action: cool plate 2 with fridge 1   
Scene: You cool the plate 2 using the fridge 1.   
The 8 admissible actions of the current situation are :   
1. cool plate 2 with fridge 1   
2. examine fridge 1   
3. examine plate 2   
4. go to cabinet 1   
5. go to diningtable 1   
6. inventory   
7. look   
8. open fridge 1   
Robot Thought: The task is to cool some plate and put it in cabinet. I'm carrying a cool plate 2. So I need to find a cabinet first. There is a cabinet 1. So I should go to cabinet 1. So the answer is 4.   
Robot Action: go to cabinet 1   
Scene: You arrive at loc 20. The cabinet 1 is closed.   
The 7 admissible actions of the current situation are :   
1. examine cabinet 1   
2. examine plate 2   
3. go to diningtable 1   
4. go to fridge 1   
5. inventory   
6. look   
7. open cabinet 1   
Robot Thought: The task is to cool some plate and put it in cabinet. I'm carrying a cool plate 2. So I need to find a cabinet first. There is a cabinet 1. The cabinet is closed. So I should open cabinet 1. So the answer is 7.   
Robot Action: open cabinet 1   
Scene: You open the cabinet 1. The cabinet 1 is open. In it, you see nothing..   
The 8 admissible actions of the current situation are :   
1. close cabinet 1   
2. examine cabinet 1   
3. examine plate 2   
4. go to diningtable 1   
5. go to fridge 1   
6. inventory   
7. look   
8. put plate 2 in/on cabinet 1   
Robot Thought: The task is to cool some plate and put it in cabinet. I'm carrying a cool plate 2. So I need to find a cabinet first. There is a cabinet 1. So I should put plate 2 in/on cabinet 1. So the answer is 8.   
Robot Action: put plate 2 in/on cabinet 1   
Scene: You put the plate 2 in/on the cabinet 1. You won!!!
G.3 Prompt for Tabletop Manipulation Tasks
G.3.1 DEPS
system: The template of the lang goal is as follow: put the <color> blocks in a < color> bowl. For example, "put the green blocks in a blue bowl" is a valid lang goal. Do not provide any additional explanations or instructions beyond writing lang goals.
user: There are 4 bowls of blue, green, pink, yellow on the table. There are 3 blocks of red, yellow, blue on the table. How to match the blocks and the bowls?
assistant:
1. put the yellow blocks in a yellow bowl  
2. put the blue blocks in a blue bowl
Listing 12: Prompt for DEP in CLIPort tasks
G.3.2 Baselines