SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The baseline models are trained on the training split, and evaluated on the validation split of the AID dataset. The overall mission completion results are displayed in Table 1. Both models are evaluated with a cap of 50 maximum allowed steps and 10 failed steps per mission, beyond which if the mission goal is not completed, the agent stops execution leaving the goal incomplete. Below, we analyze the results for both models and provide insights on the model performances.
Neural-Symbolic Model. Overall, adding QA leads to a marginal improvement in the performance for the Neural-Symbolic model. The model performs well in MSR for missions with short horizons (e.g. toggleDevice). As an ablation study, we also evaluate the multi-modal transformer on the validation set by providing the ground-truth visual observations in the dataset at each time step. As a result, the model can correctly predict all the actions and objects for  $71.3\%$  of the missions without QA, and  $76.9\%$  with QA. Since it is common that there are multiple instances of the same class in a room, the model can choose an incorrect interactable mask even if the transformer model correctly predicts the object class and the vision model correctly predicts all the object masks of that class in the image. This can also explain why the improvement with QA is limited, since the additional information does not participate in the grounding once the action and object pairs are predicted.
Vision Language Model. The VL model shows a  $6.88\%$  absolute improvement in MSR when trained and evaluated with QA compared to training and evaluating without QA. The agent is also able to complete the missions more efficiently with QA, demonstrating that asking the right questions
Team	Week 1	Week 2	Week 3	Week 4 (Unseen)	Offline
SlugJARVIS	3.98±0.18 (52.82)	4.10±0.19 (67.35)	4.19±0.18 (49.56)	3.90±0.20 (55.56)	(3.66)
ScottyBot	3.48±0.21 (46.46)	3.67±0.21 (40.40)	3.97±0.15 (49.04)	3.63±0.21 (57.95)	(8.36)
EMMA	3.31±0.20 (47.66)	3.90±0.18 (54.44)	3.88±0.18 (57.61)	4.28±0.14 (56.12)	(30.55)
SEAGULL	3.57±0.19 (33.78)	3.77±0.18 (50.00)	4.31±0.16 (51.72)	3.63±0.21 (37.86)	(30.98)
GauchoAI	4.05±0.17 (62.36)	4.31±0.16 (68.38)	4.48±0.15 (64.35)	4.33±0.17 (66.67)	(36.47)
Table 2: For online evaluation results, we show the weekly user satisfaction on a 5-point scale and missions success rate in percentage (in the parenthesis). We report the mean and standard deviation for the user satisfaction. For each team, we also show the MSR achieved in the offline evaluation for comparison.
improves task execution efficiency and accuracy. For tasks like toggling which few objects can afford, the instructions tend to already be descriptive enough in natural language ("Turn on the red computer", "Press the blue button"), which explains why adding QA shows a mere  $3\%$  improvement in MSR. For all other task types which involve numerous objects, the QAs provide crucial additional information that is not naturally provided with the instruction in the first turn, thus leading to significant performance improvement both in MSR and NRA. From analysis, the color & deliver and freeze & deliver missions are prone to failure (as evidenced by low MSRs) because of relatively longer contexts than other missions and difficulty in disambiguating the receptacle of delivery.
5.2 Online Human evaluation
Alexa Arena is used as the interactive EAI platform in the Alexa Prize Simbot Challenge, in which human users can perceive the environment through the agent's egocentric view, and communicate with the embodied agent via natural language commands. To understand how additional user interaction data can help model development, the challenge schedule follows a continual learning setting in a four-week duration: in the first three weeks, the teams can freely update their model using interaction data on 9 missions, and in the final week the models are tested on 5 unseen missions for generalizability.
5.2.1 Modeling Methodologies
In total, 10 teams participated in the challenge and implemented their own models for Arena. The models output physical actions for the agent and engages in dialog with users to successfully complete missions. We briefly describe the design of some representative models. SlugJARVIS [43] use a hybrid approach for action prediction, with a rule-based parser to process most requests and an end-to-end Pythia [5] model to handle corner cases. They also design a progressive task experience retrieval module which helps make use of the existing successful user interaction data on seen tasks to enhance performance. ScottyBot [40] design a modularized system containing a neural NLU module for action prediction, named entity recognition, and action/object names standardization. The proposed design also incorporates semantic maps for efficient navigation and object localization. SEAGULL [25] use a rule-based system to detect user intent, and a PDDL planner to generate robot actions. They also design a hierarchical vision model to detect environment states, including fine-grained object categories, object states and objects spatial relations. EMMA [41] propose an encoder-decoder architecture to encode the multi-modal input and generate outputs as textual and visual tokens. The model is first pretrained on multiple vision-language tasks including masked language modeling and image captioning, before finetuning on embodied planning tasks. GauchoAI [42] also use a modular framework for their agent, with a hybrid rule-based and neural action prediction module, an agent state tracking module, a preprocessing module to reject non-plausible actions, and a MaskFormer [8] based visual grounding model. We refer readers to the technical reports for additional details<sup>3</sup>.
5.2.2 Results and analysis
In Table 2, we show human evaluation results for each team on a weekly-basis. For the first three weeks, we can see a clear upward trend for user satisfaction within each week across all the models. For the unseen missions, most teams are able to achieve similar performance as seen ones, showing the potential to generalize to novel tasks. Compared to the offline performance, we observe that the teams are able to achieve a much higher mission success rate with human evaluation despite the tasks being more complicated. This shows the effects of having interactive dialog during task execution, which helps the agent understand the task goal and the user to understand the model's limitations. This is validated qualitatively by playing with the virtually deployed bots, where we observe critical
<sup>3</sup>https://www.amazon.science/alexa-prize/proceedings
contributors to a higher user satisfaction and MSR. While there is a positive correlation between MSR and user ratings across different teams, users tend to give a higher rating to bots that are transparent and collaborative, able to give clear feedback, let their limitations be known, and are good at anticipating user actions in a balanced manner, as opposed to bots taking actions without appropriate feedback. For example, from Week 2 to Week 3 for certain models such as EMMA and GauchoAI, the user ratings increased while the overall MSR decreased. The increase in user satisfaction is attributed to a variety of factors, including that these models became more user-centric, were more interactive and feedback friendly to the users. The teams were able to update their models with real-time human feedback signals to align their bots with the users that interact with them.
We introduce Alexa Arena, a user-centric EAI platform, featuring user-friendly graphics, animations and control mechanisms to facilitate user-centric EAI research. We provide baseline results for a dialogue-guided task completion use-case, and also release the AID dataset. While the Arena platform is designed to improve user-experience for EAI systems, the proposed online models cannot be evaluated in a straightforward fashion without a human-in-the-loop to supply interactive dialog and feedback. To overcome this limitation, we aim to incorporate a model-based user-simulator in future work, to which an oracle provides relevant answers during task execution. Another direction to explore is enabling the human and the robot to control different embodied agents and create more scenarios for HRI.
Ethical Considerations. Via introducing Arena as a public embodied AI platform built for human-robot communications, we mainly target English language and do not explicitly look at the issues innate to natural language understanding such as comprehending under-represented languages. The missions, tasks and game designs can be controlled by the platform users as needed, and is backed by a framework that has been validated quantitatively and qualitatively, to the best of our abilities, by carefully constructed and robust user studies as presented in the paper. Since Arena is to be open-sourced, it is possible for users to modify the platform for unintended use cases, which is a common risk for all open-sourced embodied AI platforms. But we believe the risk for Arena is low comparing to other platforms since Arena uses a gamified setting and the released recourses are not directly applicable for real-world malicious applications. As for real world transferability of the trained agents, robust testing and continual safety monitoring mechanisms should be put in place to minimize risks of unpredictable robot behaviours in unexpected scenarios.
[35] S. Srivastava, C. Li, M. Lingelbach, R. Martin-Martin, F. Xia, K. E. Vainio, Z. Lian, C. Gokmen, S. Buch, K. Liu, et al. Behavior: Benchmark for everyday household activities in virtual, interactive, and ecological environments. In Conference on Robot Learning, pages 477-490. PMLR, 2022.
[36] A. Suhr, C. Yan, C. Schluger, S. Yu, H. Khader, M. Mouallem, I. Zhang, and Y. Artzi. Executing instructions in situated collaborative interactions. arXiv preprint arXiv:1910.03655, 2019.
[37] A. Suhr, C. Yan, C. Schluger, S. Yu, H. Khader, M. Mouallem, I. Zhang, and Y. Artzi. Executing instructions in situated collaborative interactions, 2022.
[38] A. Szot, A. Clegg, E. Undersander, E. Wijmans, Y. Zhao, J. Turner, N. Maestre, M. Mukadam, D. S. Chaplot, O. Maksymets, et al. Habitat 2.0: Training home assistants to rearrange their habitat. Advances in Neural Information Processing Systems, 34:251-266, 2021.
[39] Y. Tian, Q. Gong, W. Shang, Y. Wu, and C. L. Zitnick. Elf: An extensive, lightweight and flexible research platform for real-time strategy games. Advances in Neural Information Processing Systems, 30, 2017.
[40] C. M. University. Scottybot: A modular embodied dialogue agent. In *Alexa Prize SimBot Challenge Proceedings*, 2023.
[41] H.-W. University. Emma: A foundation model for embodied, interactive, multimodal task completion in 3d environments. In *Alexa Prize SimBot Challenge Proceedings*, 2023.
[42] S. B. University of California. Gauchoi: A proactive modular embodied ai that collaborates with humans. In *Alexa Prize SimBot Challenge Proceedings*, 2023.
[43] S. C. University of California. Slugjarvis: Multimodal commonsense knowledge-based embodied ai for simbot challenge. In *Alexa Prize SimBot Challenge Proceedings*, 2023.
[44] Z. Wang, S. Cai, A. Liu, X. Ma, and Y. Liang. Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents. arXiv preprint arXiv:2302.01560, 2023.
[45] Z. Wang, Y. Lu, Q. Li, X. Tao, Y. Guo, M. Gong, and T. Liu. Cris: Clip-driven referring image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022.
[46] F. Xia, A. R. Zamir, Z. He, A. Sax, J. Malik, and S. Savarese. Gibson env: Real-world perception for embodied agents. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9068-9079, 2018.
[47] F. Xiang, Y. Qin, K. Mo, Y. Xia, H. Zhu, F. Liu, M. Liu, H. Jiang, Y. Yuan, H. Wang, et al. Sapien: A simulated part-based interactive environment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11097-11107, 2020.
[48] L. Yuan, X. Gao, Z. Zheng, M. Edmonds, Y. N. Wu, F. Rossano, H. Lu, Y. Zhu, and S.-C. Zhu. In situ bidirectional human-robot value alignment. Science robotics, 7(68):eabm4183, 2022.
[49] M. Ángeles Quiroga, S. Escorial, F. J. Roman, D. Morillo, A. Jarabo, J. Privado, M. Hernández, B. Gallego, and R. Colom. Can we reliably measure the general factor of intelligence (g) through commercial video games? yes, we can! Intelligence, 53:1-7, 2015.
Supplementary Materials Overview:
- Appendix A.1 contains details of the platform.
- Appendix A.2 contains definitions of tasks.
- Appendix A.3 contains details of data collection process for the AID dataset.
- Appendix A.4 contains details of the vision dataset.
- Appendix A.5 contains details of the baseline models.
- Appendix A.6 contains error analysis of the VL model.
Our code and data are available at https://github.com/amazon-science/alexa-arena.
A.1 Alexa Arena Platform Details
A.1.1 Architecture
The architecture of the Alexa Arena framework is illustrated in Figure 5. The core components of the platform are the arena wrapper and model wrapper. The arena wrapper houses the arena orchestrator, which takes as input the configuration file (described in Section A.1.2), and communicates with the arena engine, which contains the the arena simulator and the streaming server. The streaming server in the arena engine streams the robot's first party camera view to human users. The model wrapper contains the inference model and the model executor that interfaces between the model and the orchestrator. The model wrapper can be modified to develop other models and execution strategies. In addition, we also provide a web interface (the webtool) for interacting with a robot model in a chatbot like interface, while also watching the game play out in streaming on a web page.
A.1.2 Challenge Definition Format
To define the mission configurations, we introduce a schema called Challenge Definition Format (CDF). A CDF file is a json-format file that is readable by Arena. It contains the necessary information to configure, initialize and run a game mission in Arena. These include:
1. Game initial state: game scene setting, robot initial location, object initial location and initial states (such as cabinet door being open or closed).
2. Game goal condition: robot and object states that need to be satisfied. Once the goal condition is met, the game mission is considered completed.
3. Game-related text data: textual data that help guide the user to finish the game, including game mission description text, sub-goal descriptions, system prompts during game play, etc.
The motivation for designing and utilizing the CDF is to allow programmatic generation of large numbers of game missions, which is essential for data collection. Since manually defined game missions are not scalable for generating large-scale training data for ML models, with the easily configurable CDF files, we can generate thousands of game missions by permuting scenes, robot location & states, object types, object location & states, and mission goal states. The other advantage is that since the CDFs are human readable, we can specify new game missions flexibly without writing code.
A.1.3 Environment Metadata
The environment metadata in Arena contains robot egocentric view images, agent state, objects state, scene metadata, goal condition status, and previous action execution result. To facilitate different task and modeling settings, the robot camera view images support different modalities, including RGB image, depth image, instance segmentation image and normals image. The metadata response from the arena simulator is updated after each action execution.
Figure 5: Overview of Alexa Arena Platform Architecture. The core modules are the arena wrapper and model wrapper. The arena engine contains the simulator and the streaming server. The metadata from the Arena Engine are sent to the inference model, which generates actions to be executed in the simulator. We have also built infrastructure to generate results submission files for EvalAI challenges.
A.1.4 Objects and State Transitions
There are 336 unique objects in Arena. Each object has a set of properties, called affordances, which specify if a certain type of robot-object interaction is possible. To further mimic the complexities of real-world tasks and execution strategies, along with numerous object types and properties, Arena also provides a variety of constraints defined by commonsense causal events. For example, the power system enables a series of causal constraints on using electric appliances, e.g., the fuse box needs to be reset before lights being turned on, the microwave needs to be plugged in to the outlet before it can be powered on. Agents in Arena also have multiple ways to complete a task, e.g., To heat something, the agent can make use of either the Laser or the Microwave. As a result, the agent needs to reason based on its current state for efficient task completion. As an example, consider the task of heating a bowl and delivering it to a receptacle in the Quantum Lab, and the agent is currently in the Quantum Lab. If the agent's execution path to heat the bowl using the Laser in the Quantum Lab is physically obstructed, it is more efficient for the agent to go to the Breakroom, heat the bowl quicker using the Microwave and return to deliver the bowl in the Quantum Lab. A full list of of objects and their properties and state transitions are available in the "arena/documentation.html" file in the supplementary materials.
A.1.5 Layouts
Alexa Arena contains 10 large game-ready interactive multi-room layouts. Each layout features an office-like environment with modular design, allowing the contents within rooms to be rearranged and repositioned. The layouts were hand designed to be visually different and aesthetically interesting. The office setting allows both mundane and fantastical interactive elements: users would be accustomed to seeing everyday objects in more conventional rooms (and would know instinctively how to interact with them), but entering rooms clearly labeled as "labs" they would expect to find futuristic devices and prototype machines that would require experimentation to discover their function. As a result of the above bipolar content, we are able to create game missions with imaginative situations sitting alongside normal everyday objects. The layouts in Arena are large: on average, each house has 190 square meters of navigable area, which is significantly larger than the scenes in other EAI platforms (Figure 6). Figure 7 shows the top down images of 4 houses in Alexa Arena.
A.1.6 Related platforms
See Table 3 for a comparison between Alexa Arena and other embodied AI platforms.
Figure 6: Bar plot of the navigable areas for Alexa Arena, comparing to ArchitecTHOR, AI2-iTHOR, RoboTHOR and ProcTHOR.
Figure 7: Top-down images of 4 multi-room game-ready layouts in Alexa Arena.
A.2 Task definition
We designed two set of tasks: 1) the AID dataset include 12 simple tasks that require the agent to perform some basic object interactions, 2) the 14 compositional missions are used for online evaluation, each mission requires the agent to complete multiple sub-goals.
A.2.1 Basic tasks
There are 12 types of basic tasks. For most tasks, the agent is required to find the target object, use a specified tool to change its state and deliver it to the designated receptacle.
toggleDevice. The agent is required to find the device and toggle it. For some devices (e.g. color changer), the agent needs to interact with the correct object parts (e.g. red button vs green button) in order to finish the task.
Platforms	Scenes	Object Types	Multi Room	Object States	3D Env	Real Scenes	Human Control	Game Ready
Malmö [19]	17*	1000+	✓	✓	MK	✓
Overcooked [7]	6*	5	✓	MK	✓
Deepmind Lab [4]	28*	-	✓	MK	✓
AI2-Thor [20]	120	125+	✓	✓	✓	MK,VR
iGibson [46]	15	390+	✓	✓	✓	✓	MK,VR
Habitat 1.0 [32]	1000	-	✓	✓	✓	MK
Habitat 2.0 [38]	105	90+	✓	✓	✓	✓	MK
VirtualHome [28]	7	170+	✓	✓	✓	✓	NL
TDW [10]	15	200+	✓	✓	✓	VR
VRKitchen [12]	16	200+	✓	✓	✓	VR
Alexa Arena	10	335+	✓	✓	✓	✓	MK,NL	✓
Table 3: Comparison with other platforms. Scenes and objects: the number of scenes and interactable object types the platform provide. Multi-room: multi-room scenes. Some platforms marked by * are customizable and new scenes can be added. States: agent can change object states via interactions. 3D env: whether the platform supports 3D virtual environment. Real scenes: whether the scenes in the virtual environment are realistic. Human Control: the interface for the human to control the agent. MK: mouse and keyboard. NL: natural language. VR: virtual reality. Game ready: whether the platform has game-ready features to enhance user experience, including environmental and interaction sounds, action animations, smooth navigation, fantastical objects, hazards, mini-map, scoring mechanism and hints.
pickup&deliver. The agent is required to first find the target object, pick it up, navigate to the target receptacle and put the object on the receptacle.
breakObject. The agent is required to pick up a hammer, navigate to the target object, and use the hammer to break the object.
clean&deliver. The agent is required to pick up the dirty object, navigate to the sink, turn on the faucet and clean the object using the water. The agent can then deliver the cleaned object to the receptacle.
color&deliver. The agent is required to pick up the object, go to the color changer and put the object on the color changer. The agent can then push a button on the color changer to change the color of the object, and deliver it to the receptacle.
fill&deliver. The agent is required to pick up a container, go to the sink, and turn on the faucet. The agent can then fill the container with water and deliver it to a receptacle.
freeze&deliver. The agent is required to pick up the target object and use either the fridge or the freeze ray to freeze the object. The agent can then deliver it to the receptacle.
heat&deliver. The agent is required to pick up the target object and use either the laser or the microwave to heat up the object. The agent can then deliver it to the receptacle.
insertInDevice. The agent is required to pick up some small object (e.g. floppy disk, printer cartridge, laser component) and insert it into the corresponding device to assemble the device.
pourContainer. The agent is required to pick up the container (e.g. milk carton, cereal box, coffee pot), navigate to the receptacle and pour the things in the container into the receptacle.
repair&deliver. The agent is required to pick up a broken object, and use the time machine to repair it. The agent can then deliver the intact object to the receptacle.
scanObject. The agent is required to go to a place near the target object and scan it.
A.2.2 Compositional tasks
For the online human evaluation, a suite of 14 long-horizon missions are designed to test compositionality and generalizability. Each mission includes multiple sub-goals that need to be achieved to complete the overall task. The human users can use the sticky notes placed in the scene to guide themselves during mission play. An example mission is provided below with the overall setup, goals and subgoals. Only the goals and subgoals are visible to the users during mission play.
Scene: The robot is spawned in the Robotics Lab, a trophy is kept on one of the tables in the corner, but access to that side of the room is blocked by a wooden block. The block can only be moved by toggling a robotic arm, which lifts and displaces the block and clears the path.
Find the trophy and deliver it to the manager's desk.
Pick up the trophy
Deliver it to the manager's desk