SlowGuess/NeurIPS_2023_OCR huggingface dataset:
1. Easy: Simple tasks where the horizon is short (10-30 steps) and fully described by the textual instruction, e.g. OpenDoors and PutNext. The short horizon makes them relatively easy for a wide range of HRL algorithms. The instructions for these tasks generally spell out each individual skill, making them particularly easy for high-level planners based on language modeling.
2. Medium: Combination of short and long horizon tasks (up to 80 steps) with step-by-step textual instructions, e.g. LockedRoom. While being significantly longer, these tasks also tend to have instructions that spell out the low-level tasks (see Table 7).
3. **Hard:** Complex, long horizon instructions (over 100 steps) with short, ambiguous instructions that necessitate multi-step reasoning and efficient exploration, e.g. MultiRoom and LinkedUnlock. In addition to being long-horizon, the instructions in this case tend to be ambiguous and under-specified, e.g. "traverse through the rooms to get to the goal", which does not provide enough context for any blind planning agent.
Difficulty	Task Name	Example Instruction
Easy	OpenDoors	open door blue, then open door red
PutNext	move the red ball next to the green box
Medium	LockedRoom	get the red key from the purple room, open the red door and go to the goal
Hard	MultiRoom	traverse the rooms to get to the goal
BlockedUnlock	pick up the blue box
Table 7: Example Instructions in Minigrid
A.3.3 Language Model
We use InstructGPT [54] (text-davinci-002), accessed through OpenAI API. The prompts used can be found in Section A.5.
We found the prompts to be generally sufficient for solving the "seen" tasks, as well as "unseen" tasks, i.e. tasks that do not have an example in the context. Empirically, we did not find any improvements by including more than 3 example tasks in the prompt — we hypothesize that this is likely due to the shared low-level primitives across tasks. For all Minigrid experiments presented in this paper, we used the prompt shown in Section A.5.
A.3.4 Low-level Primitives
To train low-level primitives, we train an RL agent to solve a wide range of short-horizon subtasks (under 10 steps) that are shared across the various Minigrid tasks — go to  $\langle \mathrm{obj} \rangle$ , pick up  $\langle \mathrm{obj} \rangle$ , drop  $\langle \mathrm{obj} \rangle$ , open  $\langle \mathrm{obj} \rangle$ . Rather than training individual skills for each of them [65], we train a single multi-task policy that is conditioned on the CLIP embeddings [57] of the task strings. This scheme allows some robustness to synonyms and ambiguous task specifications, and has been widely used in learning language-grounded policies [32, 68].
We train these primitives using PPO [62], as recommended by the environment developers [10]. Each of these skills are trained with a sparse outcome reward  $(+1$  if a trajectory is successful, 0
otherwise). In addition to these low-level skills, we perform a form of hindsight relabeling where "strings" of the task strings are masked to allow generalization to partial strings, e.g. "go to red" may be interpreted as "go to red key" or "go to red door", and our masking strategy allows the multi-task policy to execute tasks specified by partially complete strings, if necessary.
A.3.5 Additional Qualitative Results
Traverse the rooms to get to the goal.
Step 1: go to the purple door and open it  
Step 2: go to the green door and open it  
Step 3: go to the purple door and open it  
Step 4: go to the green door and open it  
Step 5: go to the green door and open it  
Step 6: go to the goal
Get the green key from the purple room, unlock the green door and go to the goal.
Step 1: go to the purple door and open it  
Step 2: go to the green key  
Step 3: pick up the green key  
Step 4: go to the green door and open it  
Step 5: go to the goal
Pick up the purple box.
Step 1: go to the obstacle
Step 2: pick up the obstacle  
Step 3: place the obstacle  
Step 4: go to the blue key  
Step 5: pick up the blue key  
Step 6: go to the blue door and open it  
Step 8: drop the blue key  
Step 9: go to the purple box  
Step 10: pick up the purple box
Figure 8: Minigrid Domain
A.4 Implementation Details of Real-World Mobile Manipulation
A.4.1 Tasks
put an energy bar and water bottle on the table  
bring me a lime soda and a bag of chips  
Can you throw away the apple and bring me a coke  
bring me a 7up can and a tea  
move an multigrain chips to the table and an apple to the far counter  
move the lime soda, the sponge, and the water bottle to the table  
bring me an apple, a coke, and water bottle
Table 8: List of unambiguous SayCan instructions
Instruction
I want to wipe some spill.Bring me a fruitBring me a snackBring me a bag of chipsBring me a bag of snackBring me a bag of chips and something to wipe a spillBring me a bag of chips and something to drinkBring me a bag of chips and a sodaHuman: I want a soda that is not coke, and a fruit.I want a fruit and a soda
Table 9: List of ambiguous SayCan instructions
A.4.2 Language Model
For planning, we use PaLM [11], a 540B parameter language model trained on a large datasets that include high-quality web documents, books, Wikipedia, conversations, and GitHub code. Before planning, we use InstructGPT [54] (text-davinci-002), accessed through OpenAI API. to generate the (grounded) chain of thought.
We used square bracket to indicate grounded decoding, as illustrated in Fig. 5. The prompts are shown in Listing 3.
A.4.3 Low-level Primitives
We use a combination of learned and scripted control policies for navigation and manipulation, following the implementation described in SayCan [2] and RT-1 [7]. The manipulation policies for the picking action are learned using Behavior Cloning (BC) on 68000 demonstrations and 12000 autonomous successes that were collected over the course of 11 months using a fleet of 10 robots. The demonstrations are collected by teleoperators using VR headset controllers to track the motion of their hand, which is then mapped onto the robot's end-effector pose. The navigation policies are scripted, based on a ground-truth map as well as a learned perception module for collision avoidance and planning. The placing actions follow pre-computed motions only when preceded by a navigation policy. The Value Functions used by SayCan for affordance grounding are provided by the  $Q$ -networks of trained RL agents; we follow the RL training setup described in [2].
A.4.4 Open-Vocabulary Detector Grounding Function
We use owl-vit [50] as our grounding model. It takes in an image and a natural language query, and returns a list of bounding boxes with scores. We take the maximum score a the grounding function.
More examples of object detection as a grounding function can be found in Fig. 9.
Human: Bring me a bag of chips.  
Robot thought: I will bring the [kettle chips]
Human: I want to wipe some spill.  
Robot thought: I will bring the [paper towel]
Human: I want a soda that is not coke, and a fruit.  
Robot thought: I will find the [pepsi] and the [apple]
Figure 9: Additional examples of using open-vocabulary object detection as a grounding function in Real-World Kitchen Mobile Manipulation Domain.
A.5 Prompts
Listing 1: Grounded Decoding Prompt in Simulated Tabletop Rearrangement Domain
Task: Pack all letter objects on the brown box  
Step 1: pick up the e and place it on the brown box  
Step 2: pick up the g and place it on the brown box  
Step 3: done  
Task: Put the letters on the tables in alphabetical order  
Step 1: pick up the c and place it on the bottom left side  
Step 2: pick up the d and place it on the right of c  
Step 3: pick up the i and place it on the right of d  
Step 4: pick up the l and place it on the right of i  
Step 5: pick up the w and place it on the right of l  
Step 6: done
Task: Spell as much of "blue" as you can  
Step 1: pick up the l and place it on the bottom left side  
Step 2: pick up the the u and place it on the right of l  
Step 3: pick up the the e and place it on the right of u  
Step 4: done
Task: Separate the vowels from the remaining letters  
Step 1: pick up the i and place it on the bottom side  
Step 2: pick up the o and place it on the bottom side  
Step 3: done
Task: Stack all the blocks  
Step 1: pick up the brown block and place it on the pink block  
Step 2: pick up the cyan block and place it on the brown block  
Step 3: pick up the orange block and place it on the cyan block  
Step 4: pick up the gray block and place it on the orange block  
Step 5: done
Task: Put all the blocks on the bottom left corner  
Step 1: pick up the white block and place it on the bottom left corner  
Step 2: pick up the yellow block and place it on the bottom left corner  
Step 3: pick up the green block and place it on the bottom left corner  
Step 4: pick up the blue block and place it on the bottom left corner  
Step 5: pick up the purple block and place it on the bottom left corner  
Step 6: done
Task: Put all the blocks in the bowls with matching colors  
Step 1: pick up the cyan block and place it on the cyan bowl  
Step 2: pick up the purple block and place it on the purple bowl  
Step 3: pick up the brown block and place it on the brown bowl  
Step 4: pick up the pink block and place it on the pink bowl  
Step 5: done
Task: Pack the items into any box  
Step 1: pick up the donut stick and place it on the red box  
Step 2: pick up the pepsi and place it on the brown box  
Step 3: pick up the peach and place it on the brown box  
Step 4: pick up the strawberry and place it on the red box  
Step 5: done
Task: Pack the items on the table into the brown box  
Step 1: pick up the knife and place it on the brown box  
Step 2: pick up the plum and place it on the brown box  
Step 3: pick up the Pepsi and place it on the brown box  
Step 4: pick up the cupcake and place it on the brown box  
Step 5: done
Task: Pack the items on the table into the brown box  
Step 1: pick up the i and place it on the brown box  
Step 2: pick up the green block and place it on the brown box  
Step 3: pick up the l and place it on the brown box  
Step 4: done
Task: Can you put some snacks on the right side for me? Step 1: pick up the plum and place it on the right side Step 2: done
Task: Can you pack my picnic box for me?  
Step 1: pick up the orange and place it on the picnic box  
Step 2: pick up the diet pepsi and place it on the picnic box  
Step 3: pick up the knife and place it on the picnic box  
Step 4: done
Listing 2: Grounded Decoding Prompt in MiniGrid 2D Maze Domain
You are a 2D maze-solving agent with access to a variety of low-level skills such as picking up or dropping objects, navigating to doors/keys/boxes, and opening/closing doors. Here are some example tasks:
Task: get the green key from the purple room, unlock the green door and go to the goal
Step 1: go to the purple door and open it
Step 2: go to the green key
Step 3: pick up the key
Step 4: go to the green door and open it
Step 5: go to the goal.
Task: pick up the purple box
Step 1: go to the green obstacle
Step 2: pick up the obstacle
Step 3: place the obstacle
Step 4: go to the blue key
Step 5: pick up the blue key
Step 6: go to the blue door and open it
Step 8: drop the blue key
Step 7: go to the purple box
Step 8: pick up the purple box.
Task: traverse the rooms to get to the goal
Step 2: go to the green door and open it
Step 3: go to the purple door and open it
Step 5: go to the green door and open it
Step 6: go to the goal.
Now your turn.
Listing 3: Grounded Decoding Prompt in Real-World Kitchen Mobile Manipulation Domain
The following objects are in the scene: 7up, apple, banana, mango, tea, multigrain chips, kettle chips, jalapeno chips, rice chips, coke, grapefruit soda, pepsi, redbull, energy bar, lime soda, sponge, paper towel, and water bottle.   
The following locations are in the scene: close counter, far counter, table, trash, bowl.   
The robot will always put object name in brackets ].   
Robot: I am a robot that can bring objects to you.   
Human: I am hungry.   
Robot thought: I will find the [multigrain chips].   
Robot plan: 1. Find the multigrain chips   
2. Pick up the multigrain chips   
3. Bring it to you   
4. Put it down   
5. Done   
Robot: I am a robot that can bring objects to you.   
Human: Throw away the fruit.   
Robot thought: I will find the [mango] and move it to the trash.   
Robot plan: 1. Find the mango   
2. Pick up the mango   
3. Go to the trash   
4. Put it down   
5. Done   
Robot: I am a robot that can bring objects to you.   
Human: (inject instruction).   
Robot thought: