SlowGuess/NeurIPS_2023_OCR huggingface dataset:
If needed, hints are located at visible locations to unblock the user to complete the mission. For the above example, the happy path of the robot to complete the mission without any impediments involves toggling the robotic arm to move the wooden block, gaining access to the other side of the room. The robot then picks up the trophy which is now accessible, and navigates to the manager's desk to deliver it. Such a mission tests the visual and compositional reasoning of the agent. The agent must understand that the path to the trophy is blocked, and that the robotic arm controls the wooden block, which can be moved to clear a path.
The remaining 13 missions are described briefly below:
Assemble and fire the laser cannon. The agent is spawned in the quantum lab, is required to find a control panel and place it on the laser (which looks very similar to the freeze ray). The laser is then turned on using the laser monitor (one of two monitors disambiguated by color and position).
Turn a white bowl red. The agent is spawned in the quantum lab, is required to pick up one of 3 bowls from the breakroom and use the color changer to change the color of the bowl.
Disinfect the computer. The agent is required to find the floppy disk in the manager's office, navigate to the main office and use the disinfecting floppy on the computer with an infection (indicated by a skull on the monitor among other monitors).
Freeze the soda can with a freeze ray. The agent needs to find the soda can on a table and freeze it using the freeze ray. This requires finding the right monitor to toggle.
Defrost the cake with laser. The agent needs to find a frozen cake in the freezer and defrost it using the laser in the quantum lab, which requires disambiguating between tables and monitors.
Print action figure. The agent needs to print an action figure using the 3D printer. Before doing this, the agent needs to turn on the power in the room (as the room is initially dark) to power the 3D printer on.
Turn a red apple green. The agent needs to find a red apple in the lower fridge and use the color changer on it to turn it green.
Make a bowl of cereal. The agent is required to make a bowl of cereal by pouring milk and cereal into a bowl in the breakroom. The milk is initially in the fridge, the cereal is on the breakroom table, and the bowl is initially broken. The agent needs to fix the broken bowl before making the cereal.
Pour coffee and deliver to reception. The agent is required to pour coffee into a mug from the breakroom and deliver it to the reception table.
Heat a cup of coffee using the laser machine. The agent needs to find the mug with the coffee and heat it using the laser machine in the quantum lab. This requires finding the laser monitor and toggling it.
Scan items. The agent is required to scan a computer, a light switch and a vending machine.
Find prorotype and toggle. The agent needs to find the prototype machine with a face on it, and toggle it repeatedly until a sad face appears.
Print mug. The agent is required to find the mug printer cartridge on a table, insert it in the 3D printer and use it to print a 3D mug.
Figure 8: Process of generating game missions and expert demonstrations.
A.3 AID Dataset Collection
In this section, we describe the details of the data collection process for the Arena Interactive Dialogue (AID) dataset. The game mission and expert demonstration generation process is illustrated in Figure 8. After generating the missions and expert demonstrations, we use Amazon Mechanical Turk (AMT) for collecting human instructions as well as questions and answers necessary for task completion. Annotators receive  $1 compensation for each annotation task which roughly takes 3 minutes to complete, and the hourly wage is around$ 20. All annotators agree to the Amazon Mechanical Turk Participation Agreement before starting the annotation.
A.3.1 Human Instruction Annotation
To collect natural language dialogue for the missions, we design a two-stage data annotation process. The first stage is for collecting instructions and the second is for questions and answers. In both stages, the annotator watches a video containing an expert demonstration for the mission and provides the annotations by text data or multiple choice questions.
In the first stage, annotators are told to write instructions to tell a "smart robot" how to accomplish a task. During the process, an annotator first watches the video of the ground-truth robot actions, then writes instructions for each highlighted video segments. Fig. 9a shows the user interface for collecting language instructions.
After all the instructions are collected, we start the second stage of annotation, where the annotators are asked to raise questions to help them complete the task, as if they are controlling the robot to follow the instructions. They also need to answer their own questions. Fig. 9b shows the interface for collecting questions and answers. Here are the steps annotators need to follow:
1. Annotator watches a video: A 10 second video clip is played till the beginning of a sub-task, and the respective instruction is shown on screen. The video clip helps the human annotator to understand the initial environment states.
2. Annotator selects a question: The annotator selects one pertinent question that they think may help task completion. The questions are generated from several predefined templates. Annotators also have the options to type in their own questions.
3. Annotator answers the question: The same annotator watches the video segment of an expert agent working on the sub-task specified by the language instruction, and then providing the answer to the question in text.
4. Annotator indicates whether asking the question is necessary: In some cases, the instruction and the visual context is already clear enough, and asking a question is not necessary.
To ensure the annotation quality, in practice, we collected 3000 data sessions for the first data collection stage. Then we asked additional annotators to identify annotation errors and try to correct them, e.g. erroneous language instructions, grammatical errors, misalignment between instructions with robot actions, etc. For data sessions where most collected annotations do not make sense and are not worth fixing, the additional annotators are asked to simply ignore them. As a result,  $20.3\%$  sessions are ignored and  $35.7\%$  instructions are corrected. From the data verification results, we identified a subset of AMT workers (around 300 workers) who generated high-quality data, and used them for our subsequent data collection process.
<sup>4</sup>https://www.mturk.com/participation-agreement
1. Imagine you are guiding a robot via language instructions to finish a task.
2. Click each of the "Play Step#" button right of the video and watch the video clip.
3. Write step-by-step instructions that you would give to the robot in order to guide it to achieve what is shown in the video clips.
4. Write one instruction per step, and please pay close attention to correctly align each instruction with the corresponding step in the video.
Imagine you are guiding a robot via language instructions to finish this task: Find the floppy disk and deliver it to the desk. Write down what were your step-by-step instructions that lead to the robot actions in the video.
Play Step 1 write instruction here:
Play Step 2 write instruction here:
Play Step 3 write instruction here:
Play Step 4 write instruction here:
Thank you for completing this HIT! Do you have any feedback (optional) for us?
Please provide your feedback (optional) here:
(a) User interface for collecting human instructions. Annotators watch the expert demonstrations and write down instructions to guide the robot.
View instructions
Figure 9: User interface for data collection.
Please read the instructions at the top-left corner before starting.
Begin Show Demonstration
Step 1. Hit the "Begin" button and watch the video till it stops.
Step 2. Now you are shown the following command:
" Pour the cereal into the bowl "
Select a question that can help you execute the above command:
Which direction should I turn to?
What does the cereal look like?
Where is the cereal?
What does the bowl look like?
Where is the bowl?
Other (type in below)
If no question apply, type in your own question
Step 3. Hit the "Show Demonstration" button to watch the video that executes the command. Answer the above question:
Step 4. Do you think the question is necessary to execute the command?
Yes No
Thank you for completing this HIT! Do you have any feedback for the task?
(Optional) Please provide your feedback here.
(b) User interface for collecting questions and answers. Annotators input their questions and answers to help the robot complete the mission.
A.3.2 Questions Generation
Similar to [11], the questions choices in the second stage of annotation are generated using predefined templates. Given an instruction, we extract nouns and insert them into the templates to construct questions. The nouns extracted correspond to query objects that the agent need more information to complete the mission.
In particular, we consider three types of questions, related to the location and appearance of the query object  $o$ , and the relative direction between the agent and the target position. The templates for each question are defined below:
1. Location: where is  $o$ ?
# scene	# task	# session	# instr	# dialog
Train	6	2661	7983	40443	40105
Valid	6	383	1149	6120	6128
Total	6	3044	9132	46563	46233
Table 4: AID Dataset Breakdown. # session represent the number of data sessions. Each data session corresponds to one language annotation on a given task. # instr stands for the total number of instructions in each data split. # dialog is for the total number of question and answers.
(a) Frequencies of action types.
(b) Steps in expert demonstrations.
(c) Length of human utterances.
Figure 10: Statistics of the AID dataset.
2. Appearance: what does  $o$  look like?
3. Direction: which direction should I turn to?
4. Reference: which  $o$  are you referring to?
A.3.3 Oracle Answers
Human language has a lot of variety and complexity, and thus can be quite challenging for the agent to understand. Alternatively, we can provide templated language to the agent as a starting point for language understanding. To this end, in addition to asking human annotators to answer the questions, we also build an oracle that can answer the questions by extracting information from the simulator metadata: (i) To answer the location question, we compute the direction of the object relative to the agent and the viewpoint that is closest to the object in the room. To help detect small objects, if the object is held by a container, we also provide the information of the container and other objects that are also held by the same container as landmarks; (ii) To answer the appearance question, we directly extract object shape, color and material information from the simulator metadata; (iii) For direction question, we compare the agent's location at the end of the sub-task with its initial location to provide answers for the agent's moving direction. After we have the necessary information, we use language templates to generate the answers. Example templates include:
1. Location: The  $o$  is to your [direction] in/on the [container] next to the [landmark] in the [room]. It is closest to [viewpoint].
2. Appearance: The  $o$  is [shape] and of [color]. It is made of [material].
3. Direction: You should turn [direction] / You don't need to move.
A.3.4 Dataset statistics
Table 4, Figure 10 and Table 5 show the statistics of the AID dataset.
A.4 Vision Dataset
Visual perception is integral for EAI agents to navigate and interact within their environments. Visually intelligent robots use visual cues to set navigational targets and build visual representations based on perceived object state transitions. To facilitate visual AI research within and outside the context of EAI, we are releasing a vision dataset based on Arena. The dataset is composed of 600k training images and 60k validation images spanning 336 unique objects from more than 160 semantically grouped object groups. This can be used for large scale computer vision research on Arena, e.g. object detection, object state classification, etc. The overall structure and contents of the dataset are illustrated in Fig. 11. The vision dataset is collected by programmatically configuring
Mission Type	Train	Valid
breakObject	196	30
clean&deliver	161	29
color&deliver	29	4
fill&deliver	108	16
freeze&deliver	76	8
heat&deliver	42	13
insertInDevice	381	59
pickup&deliver	669	95
pourContainer	265	39
repair&deliver	131	18
scanObject	324	37
toggleDevice	279	35
Table 5: Distributions of mission types across training and validation splits of the AID dataset.
Figure 11: Overview of the vision dataset. The dataset consists of RGB and ground truth segmentation images in png format, along with a json file for each image with the metadata information.
CDFs and initializing scenes with all objects spawnable in the Arena environment. We then navigate to each of them to capture images from different perspectives and distances.
A.4.1 Vision Data Generation
The image data collection strategy was simple and straightforward - the goal was to collect data spanning all possible objects spawnable in Arena from different distances, views and perspectives of the said object. This was done by following the below steps:
1. Initialize a game mission using the CDF file with required objects.
2. After initialization, we issue commands to navigate to all the different objects present in the scene using the primitive navigation actions and capture the first party view.
3. We then capture images by issuing interactive commands as per the affordances of that object at the time of capture, which are retrieved from the game metadata response.
4. We also capture additional perspectives by taking a random walk around the object using basic trigonometry.
We repeat the above steps with various missions and objects with lights on and off for different lighting conditions. All the commands, object states and their locations in the scene, along with a host of other useful information are logged in metadata files. We provide the RGB color images, segmentation maps and associated metadata for each image. The statistics of the dataset are illustrated in Fig. 12.
Dataset Validation. To validate the quality of the ground truth segmentation and bounding box annotations for modeling, we collect 1245 images containing instances spanning all object classes
(a) Vision data object size distribution.
(b) Vision data class frequency.
Figure 12: Vision dataset statistics
and annotate them with bounding boxes and associated class names. Since Arena features fantastical objects (like Gravity Flippers), we also provide example images for each object class to familiarize the annotator with the appearance of all classes. The images are then validated by annotators according to the class names into one of the following 4 categories (with results provided in parenthesis):
1. Yes, the object is fully contained within the bounding box  $(79.7\%)$
2. Yes, the object is partially contained within the bounding box  $(16.3\%)$
3. No, the object is not within the bounding box, but is in the image  $(1.6\%)$
4. The object is not in the image  $(2.4\%)$
For category 2, the objects are present only partially in the image because they are only partially visible in the robot's egocentric view at the time of image collection. Overall results show that the ground truth segmentation and bounding box annotations have very little noise and are of high quality to be used for training and evaluation of visual perception models.
A.5 Baseline Models Details
A.5.1 Neural-symbolic Model
The architecture is shown in Figure 13. In particular, the multi-modal Transformer model uses the previous actions, previous visual observations and the full utterances of the episode to predict the next action type and object class before a stop token is predicted as the action. For the visual observations, we first extract features using the backbone of the vision model before sending them into the transformer. During training, after each look around action, the model uses the observations from different directions to predict rotations: the number of rotation depends on which image out of the four panoramic images is used by the following actions in the expert demonstrations. During
Figure 13: Model architecture for the Neural-Symbolic approach which uses unshared modules to predict actions and masks, and heuristic rules for visual grounding.
inference, the vision model detects all the objects from the current image, and uses the predicted object class to generate the mask for the corresponding object. Note that if the vision model detects multiple object instances of the same class, we select the one with the highest likelihood. We trained the transformer model for 20 epochs with a batch size of 2. The training takes around 10 hours on 1 NVIDIA A10 GPU. We use the Adam optimizer with a learning rate of  $1e^{-4}$  for the first 10 epochs, and  $1e^{-5}$  for the remaining epochs.
The Neural-Symbolic model is composed of a separate visual component that performs instance level segmentation of the first party view and/or surrounding images of the agent. We use the vision dataset to train a Mask-RCNN image segmentation model [14]. We process the object classes in the vision dataset to group them into 86 semantic classes including the background class. The model takes as input an RGB image and predicts masks for all object instances (across the 86 object classes) present in the image along with their class labels and confidence scores. The masks and object class predictions are then used in conjunction with the action and object predicted from the multi-modal transformer to give a single mask prediction for the interactable object. The vision model is trained for 22 epochs with a learning rate of 0.00125 and a weight decay of 0.001 with the SGD optimizer, decaying the learning rate after 15 epochs by a factor of 0.1. We use a global batch size of 16 and trained the model on 2 Tesla V100 GPUs, and the training takes around 24 hours. We evaluate the instance segmentation model using the standard COCO evaluation metric, the Mean Average Precision (mAP). The mAP metric is calculated by averaging the precision at Intersection over Union (IoU) thresholds ranging from 0.5 to 0.95 in steps of 0.05. The score of the predicted instance class is not taken into consideration, and instead a cap is applied on the number of maximum detections per image (chosen to be a 100). To be able to choose an operating point for our end-to-end model and also tune for recall, in addition to the standard COCO mAP metric, we also provide a score and iou thresholded precision metric, which we call  $t$ -mAP. This is calculated by thresholding the output instance class scores at [0.05, 0.1, 0.3, 0.5, 0.7], and thresholding the IoUs at [0.1, 0.3, 0.4, 0.5, 0.75, 0.8] for all 30 (score, IoU) combinations and then averaging them. We present both metrics for all objects, as well as metrics categorized for small, medium and large objects. The results are provided in Table 6.
A.5.2 VL Model
For training, we use a learning rate of  $1e^{-6}$  for the backbone and mask prediction head, and  $1e^{-4}$  for the action prediction head (chosen empirically). We use the Adam optimizer with a weight decay of  $1e^{-5}$  and a learning rate decay of 0.1 after 35 epochs. The model is trained for 65 epochs with a global batch size of 64 (16 per GPU) on 4 Tesla V100 GPUs, and the training takes about 7 hours.
Category	Area (px2)	COCO mAP	t-mAP
Small	0 - 1296	37.63	91.49
Medium	1296 - 9216	60.41	92.15
Large	9216 - 90000	64.72	84.91
Overall	0 - 90000	46.03	89.5
Table 6: Image segmentation results for the vision model used in the neural-symbolic approach. We present the metrics for small, medium and large objects.
Figure 14: There are two tables in this image, a blue table on the left and a green one on the right, leading to challenging disambiguation tasks for the agent if not guided adequately.
A.5.3 Encoding Questions and Answers
We use two tokens to efficiently encode the predefined questions, including (i) a token representing the question type (i.e. loc for questions asking about object location, app for object appearance, dir for direction and ref for reference), and (ii) a token for the target object in location, appearance, referential questions. For example, a question asking about the appearance of a microwave will be encoded as app microwave. Since the direction question is asking about the direction and not related to an object, the second token is not necessary and thus omitted for this type of question. For questions that are not of the predefined types, we include the whole question and answers.
A.6 VL Model Error Analysis
While the VL model demonstrates improved performance using an end-to-end method that generates actions and masks simultaneously, we identify key insights, sources of errors and failure points of the end-to-end architecture by analyzing the robot trajectory for 25 missions:
1. Some missions fail because of failure to visually ground the natural language instruction to a single object instance for interaction in the case of multiple instance candidates. E.g. in Figure 14, there are two desks - blue on the left and green on the right. If the natural language instruction is not descriptive enough, the model fails to visually ground the desk of interest. On analysis, we notice that in the  $\langle x\rangle \&$  deliver mission type, the agent is often able to complete the  $\langle x\rangle$  goal and fails at the deliver goal because of ambiguity in language ("The desk is right in front of you with the monitor on it"), or model visual perception failures to disambiguate between multiple instances of the same object.
2. Sometimes, if the agent veers off its path due to a wrong prediction (e.g. if it goes to a wrong room), the subsequent instructions to find an object in the room steers the agent down a path to infinitely rotate about its position to try finding the object, i.e., the agent is trying to respond to the next instruction that assumes successful attainment of the correct previous state. This can
potentially be solved by better error correction mechanisms in the model to identify such loopy states and exit them to explore other options in the trajectory, or to execute a strategy to backtrack to previous instructions. Since the current baseline models use the readily available answers in the dataset, this can also be improved in models that are designed to ask questions based on its real-time current visual observations.