Alexa Arena: A User-Centric Interactive Platform for Embodied AI
Authors: Qiaozi Gao
Govind Thattai
Suhaila Shakiah
Xiaofeng Gao
Shreyas Pansare
Vasu Sharma
Gaurav S. Sukhatme
Hangjie Shi
Bofei Yang
Desheng Zhang
Lucy Hu
Karthika Arumugam
Shui Hu
Matthew Wen
Dinakar Venkateswar Guthy
Shunan Cadence Chung
Rohan Khanna
Osman Ipek
Leslie Ball
Kate Bland
Heather Rocker
Michael Johnston
Reza Ghanadan
Dilek Hakkani-Tur
Prem Natarajan
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Human-robot interaction
Embodied AI
Natural Language Processing

Abstract: We introduce Alexa Arena, a user-centric simulation platform to facilitate research in building assistive conversational embodied agents. Alexa Arena features multi-room layouts and an abundance of interactable objects. With user-friendly graphics and control mechanisms, the platform supports the development of gamified robotic tasks readily accessible to general human users, allowing high-efficiency data collection and EAI system evaluation. Along with the platform, we introduce a dialog-enabled task completion benchmark with online human evaluations.

Alexa Arena: A User-Centric Interactive Platform for Embodied AI
Qiaozi Gao
Govind Thattai*t
Suhaila Shakiah"
Xiaofeng Gao"
Shreyas Pansare
Vasu Sharma
Gaurav Sukhatme
Hangjie Shi
Bofei Yang!
Desheng Zhang
Lucy Hu
Karthika Arumugam
Shui Hu
Matthew Wen
Dinakar Guthy
Cadence Chung
Rohan Khanna
Osman Ipek
Leslie Ball
Kate Bland
Heather Rocker
Michael Johnston
Reza Ghanadan
Dilek Hakkani-Tur
Prem Natarajan'
Amazon Alexa AI
Abstract
We introduce Alexa Arena, a user-centric simulation platform to facilitate research building assistive conversational embodied agents. Alexa Arena features multi- room layouts and an abundance of interactable objects. With user-friendly graphics and control mechanisms, the platform supports the development of gamified robotic tasks readily accessible to general human users, allowing high-efficiency data collection and EAI system evaluation. Along with the platform, we introduce a dialog-enabled task completion benchmark with online human evaluations.
Introduction
A longstanding goal of AI is to develop autonomous robotic agents that can assist humans in day-to- day activities. Experiments with robots are often conducted controlled environments, limiting their variety and scale of operation. To mitigate this problem, several EAI simulation platforms have been proposed, which support virtual scenes that can be either manually designed, synthetically generated, captured from real scenes. An embodied agent can freely navigate and interact with objects in these scenes to complete tasks. However, the current EAI platforms suffer from a set of limitations that curtail the ability to build generalizable assistive AI agents.
Facilitating HRI data collection by gamification. persistent challenge in Human-Robbe Interac- tion (HRI) is collecting human interaction data comprising of natural language instructions along with visual content and actions. Currently available EAI platforms, however, are not designed for humans effectively interact with the agent, making the data collection process expensive and time consuming [21] 35). Games have been historically utilized to encourage wide-spread user partici- pation and engagement with services |6 49 30). To case the data collection effort, introduce gamification EAI, which is achieved by presenting tasks or user interactions in the form of games,
"Equal contribution. Correspondence arena-adis adminss amazon. com Work done while at Amazon Alexa AI
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

and by introducing scoring mechanisms, achievements and streaks to stimulate and engage users, promoting user participation.
Reasoning based on visual observations. Tasks in current EAT platforms have limited in-class variabilityy: each task type requires the agent to make use of the same objects repeatedly. As result, agents may resort to over-engineering (over-fitting to specific and simplistic tasks) and memorization to converge to trivial and non-generalizable solutions, or use predefined task decomposition mappings [22]. One way to mitigate this is to design missions that can be completed in multiple ways by interacting with varied objects in the environment with compositional and causally interconnected state changes, forcing the agent to then use common-sense reasoning to understand its current state from visual observations, past actions and associated state changes.
Reconciling offiine and online execution scenarios. Real-world operation of assistive agents is characterized by object interactions and collaborative engagement with other agents humans in the form of visual cues or dialog. In the offline setting, the agent passively follows human instructions. Most currently proposed dialog guided interactive benchmarks and platforms operate in this fashion 11 2 26 Online execution features an agent interacting actively with users that have control, interfacing through natural language in continuously evolving environment. For example, if the agent veers off its path in the online phase, the human user can regain control of the situation and guide the robot to success, sometimes accomplishing some of the the tasks themselves, contributing a learning opportumity for the agent [37]. Such corrective mechanism that incorporates the elements of real-time user interaction can be implemented limitedly in an offline setting, to the extent of augmenting datasets with interactive elements like dialogs. With our platform, we aim to bridge the fundamental operational gap between the offline and online scenarios by demonstrating the use of offline interaction data to bootstrap an EAT agent, following which runtime data and interactions could be used to quantitatively and qualitatively improve user experience and task execution success rates, leading to sucessful demonstration of a human-in-the loop EAI system that more closely resembles real-world deployment scenarios.
To this end, we propose Alexa Arena, user-centric EAI platform. For better user experience, the platform design has commonalities with games, with features like task-guiding UI elements and engaging visual effects. The platform boasts many variations of objects and state transitions, on top of which variety of missions are designed (Figure l}. We show an example use case of Arena on dialog-guided guided task completion, where the embodied agent can communicate with the user through natural language to finish an indoor mission. To assist developing models for this task, we release a dataset in which each mission is annotated with an expert demonstration and corresponding human- annotated language instructions and dialogues. To evaluate the tialog-guided agent in real-time, we also release web-based user interface (UT), where user can communicate with the agent in text and receive visual observations of the agent from Arena. This paper makes the following contributions:
1. Alexa Arena as a new user-centric EAI platform which focuses on building assistive conversational agents that can assist humans through reasoning and procedural learning.
2. A dataset of over 9K dialog sessions and 46K human annotated instructions. Each data session simulates a human user communicating with an embodied agent to complete household activity.
3. In addition to offline evaluation of the dialogue-guided embodied agents, we also provide an interactive evaluation protocol. Using the web-based interface, the user can provide instructions to the agent and observe the progress in real-time. We also present results from baseline models for both evaluations.
2 Related Work
Embodied Al platforms. In recent years, many platforms have been proposed for embodied agents
to perform household activities in indoor virtual environments 28
10 47]. Most simulators are Although some simulators have built-in infrastructure for human data collection 112 46 the user interactions are generally not the focus of their designs. On the contrary, Alexa Arena is user-centric embodied AI platform with features designed specifically to enhance user experience. There are also several EAI platforms that
designed with performance and realism the top priorities [38]
inspired by video games [19 4 39 13 48 but the agent observations are often simplified.
Language-guided navigation and task completion. Most existing work for learning language- guided embodied agents focuses navigation tasks (3, 366 24 9 31, For increased task complexity,

---

Figure 1: The Alexa Arena EAI platform has variety of object categories, like set of fantastical objects as engagement enhancers, such the freeze ray (3), the time machine (6) and the color changer (7). The agent (1) can make use different tools change the state the object. For example, ray (3) or the fridge (5) to cool an object. Arena also has more interaction actions and state changes compared to existing platforms. For instance, the agent can use the time machine (6) to repair broken objects, or use the color changer (7) change the color of objects.
agent can use either the freeze
enable the agent follow natural language instructions and complete household activities, where both navigation and object manipulation actions are required. Most recently. [26 [1] propose new datasets and benchmarks for training and evaluating task-oriented embodied agents that can engage in dialogue. However, both works use offline settings where the dialogues are pre-collected from humans or generated with templates. In comparison, the Arena platform enables embodied agents to communicate with human users in real-time interactive fashion.
23 33
Task planning with large language models. Recently, there is growing trend of using large language models (LLMs) for assisting robot task planning in learning novel activities or completing LLMs have been shown to be good at providing high-level semantic knowledge about the physical world and common human activities [16] 34). When combined with the sensory perception of the embodied agent, such knowledge can often substantially improve the agent's capability of solving complex tasks in unseen scenarios [18] [7 The Arena platform provides good testbed for this line of work. With numerous object types, properties and state changes, along with various environmental causal events, Arena supports the creation of robot tasks that require adequate reasoning capability on common world knowledge.
complex tasks D 44
The Arena Platform
In this section, we describe the Alexa Arena Platform. We briefly describe the attributes of the simulator and tools that we are releasing as part of the platform.
3.1 Objects Properties and States
There are 336 unique objects in Arena. Each object has set properties, called affordances, which specify if certain type of robot-object interaction is possible. For example, the agent can toggle the 3-D printer since it has an object property toggleable. the same time, the agent cannot pick up since it does not have the pickupable property. In total, there are 14 object properties, including pickupable, openable, breakable, receptacle, toggleable, powerable, dirtyable, heatable, eatable, chillable, fillable, cookable, decor and infectable. Each object property has corresponding action and subsequent object state upon taking that action. For example, break is the corresponding action for breakable, and broken is the corresponding state after the action has been performed. The states of objects will change as a consequence of the action, given that the pre-conditions are met. For example, if powered on (Fig. (2)). the 3-D printer can be used to make toys (Fig. (4)).

---

Arosa Seraddr
Chat with Robel
BRRACKCOM
wuth
Figure 2: The web-based user interface. user can use the chat box to communicate with the agent Arena for task completion. Meanwhile, video from Arena is streamed in real time to show the progress. The minimap is displayed on the lop right corner of the game UI, showing the room layout and the robot location and orientation. The mission goal and subgoals displayed to the users in text the top left corner of the UI.
3.2 Robot Action Space
In general, Arena supports two kinds of actions: 1) user interaction actions for communicating with the user, and 2) robot physical actions to interact with the simulation environment. There are two types of robot physical actions navigation actions and object interaction actions. For better user experience, all the navigation and interaction actions are animated associated appropriate environmental sounds also being played during the animation.
continuous fashion with
User interaction. To communicate with the user, the robot can initiate dialog, the contents of which are displayed on the user interface (Figure The robot can also highlight objects for real-time visual feedback and use for confirmation or user instruction disambiguation.
The goal of Arena is to aid in compositional task learning and reasoning. instead of indoor navigation. To this end, simplify the navigation in Arena by enabling the robot directly navigate to a viewpoint in room by specifying the viewpoint name or the room name, or to an object by specifying the object mask. Meanwhile, if preferred, the robot can also perform step-by-step navigation by combination of parametrized local primitive actions like MoveFowarl and Rotate.
Navigation.
Object interaction. Arena supports 11 actions for object interaction, including Examine, Pickup, Place, Open, Close, Break, Pour, Toggle, Fill, Scan and Clean. Each action is associated with a set of objects in specific state in which the objects can afford that action to be performed upon them. E.g., toggling can be performed on a Time Machine when it is in a closed state. Interaction actions are accompanied by a change in the associated object's state, which gets updated in the metadata.
3.3 Enhanced User Experience
We incorporate into Arena elements that enhance application usability and improve engagement quality: 1) the platform includes engage-enhancers, such as fantastical objects and a scoring system for each mission; 2) for better visual effects, actions in Arena are animated in a continuous played-out fashion for both manipulation and navigation actions. makes the scenes more natural, user-friendly and engaging; 3) the platform features a unique user interface with several elements to provide users with better task guidance, including minimap and sub-task hints, all of which work in an integrated fashion to improve user-experience.
4 Dialog-guides Task Completion
We now present the Dialog--uided Task Completion benchmark. The benchmark is designed to evaluate dialog-guided agents for indoor object interaction tasks. To support model development, we release a hybrid dataset where ground-truth robot action trajectories are paired with human annotated language. We also set up two evaluation protocal: an offline evaluation on the validation split of the dataset and an online evaluation with human users.

---

Tasks
Heat & Deliver
Freeze & Deliver
Repair & Deliver
Color & Deliver
Scene
104
Objects
Bowl, microwave,
Mug, freeze ray, table
Hlopyy disk, time machine, fridge
Apple, color changer, desk
desk
Instructions
Take bow!
Go to the freeze ray target
Carry floppy disk time machine
color changer station
Questions
Where the bowl?
What does the target look It the blue tray attached to the wall
Where is the time machine?
Where is the station?
Answers
the tan table next to the fan
The time machine in the corner of the countertop
your left the southwest corner of the room
demonstratiooss
Figure 3: Examples in the AID dataset, Each data session corresponds to one language annotation on mission
that needs to be completed in specific scene. Each language annotation includes sequence of instructions guiding the robot to complete the task, as well as questions and answers for clarification. Expert demonstrations are provided for each data session.
4.1 Task settings
Each task in the benchmark is specified by the initial states and the goal states of the scene, and a sequence of human instructions. The agent is required to understand the language instructions and interact with the environment through series of object manipulation and navigation actions to achieve the goal states.
4.2 Arena Interactive Dialogue (AID) Dataset
To enable training and evaluation of the dialog- guided embodied agents, we collect the Arena Interactive Dialogue (AID) dataset via crowd-soureing. The dataset contains expert action trajectories. human language instructions, and associated questions and answers for missions to simulate dialog. We split the dataset into training and validation folds. There are 2661 tasks in training and 383 in validation, in which each task is annotated by 3 annotators. Each human annotation is considered one session, giving total of 7983 training sessions and 149 validation sessions. There are in total 12 unique tasks types in the dataset. Examples data points are shown in Figure 3
Expert demonstrations. The game mission and expert demonstration generation follows a two-step process. In step one, game missions are programmatically generated via sampling from initial environment states and mission goals. The second step generates expert demonstrations using planner. Along with the game definitions, we also generate PDDL (Planning Domain Definition Language) planning problem definitions for each game mission. A symbolic planner is used solve the planning problem and the output action sequence is collected as the expert demonstration. One thing to note that the planner has access to the game metadata, which is not available the agents during inference time. For tasks that can be completed in different ways (e.g., by using different tools), we pick one unique way for each task to generate expert demonstration. For example, for freeze and deliver, in some missions the fridge is used to cool the object, while in other cases the freeze ray is used.
Human Language Annotation. To collect natural language dialogue for the missions, we design a two-stage data annotation process on Amazon Mechanical Turk (AMT). In both stages, the annotator watches video containing an expert demonstration for the mission and provides the annotations in free-form text data or answers to multiple choice questions. In the first stage, annotators write instructions to tell "smart robot" how to accomplish task. During the process, an annotator first watches the video of the ground-trutt -truth robot actions, then writes instructions for each highlighted video segment. After all the instructions are collected, we start the second stage of annotation, where the annotators are asked to raise questions to better complete the task, as they are controlling the robot to follow the instructions. They also need to subsequently answer their own raised questions. Similar to IT. the question choices the second stage of annotation are generated using predefined templates. For more details on the human language annotation process, see Appendix.
5

---

Instruction +
Planning
Object
"Dien troge"
Transfmer
Language
LSTM
Ackin Cota
Image
Hidden
Disect magg
"coppp: The Indge is yellow in color"
Language
Languoge
Adiont Doen.
Output
Ofted Fortgn.
Image
Join VL Encoder
Goto
Shared
Both
Heods
Hlidken
Open
Restet VIT
Image
Language
Mask
Image
Referemtial
Hidden
Mask
Language Imagi 2
Predictor
Image
Mask Prediction Head Based on Language.
Image and Task Planning Outputs
Figure Model architecture of vision language model for end-to-end action prediction and visual grounding.
4.3 Offline evaluation
We expect that good embodied agent should be able finish the missions efficiently. To achieve this goal, the agent should understand the human instructions and generate a corresponding sequence of actions. Thus, we evaluate the agent using the following metries:
Mission success rate (MSR). For each mission, there is mission success variable m indicating whether the goal conditions have been met for the mission. m 1 all the goal conditions are met. Mission completion rate is calculated by averaging across all the missions.
Average number of robot actions (NRA). To measure the efficiency of task completion, we also record the number of actions taken by the agent complete each mission. Average number of robot actions is calculated by averaging the number of actions across all the missions.
4.4 Online Human evaluation
To demonstrate the potential of Alexa Arena for real-time user interactions, we design a human evaluation protocol for dialog-guided task completion. Different from the offline evaluation where the agent only passively follows the user instructions, in this online setting, the agent can also engage in multi-turn dialog with the user communicate its internal states and clarify the mission goals.
Missions. On top of the 12 task types presented in the AID dataset, we design and curate 14 more complex Longe-horizon missions, each requiring the agent complete several sub-goals by following the user instructions, aiming to test compositional learning capabilities of the agents. The missions designed be gamified engage the and the environment also contains hints guide the users to complete the tasks. The setting encourages dialog, where the human and the robot share the goal to complete the mission. An example of such task is provided in the Appendix. To test the agent's 's capability to generalize across tasks, we keep 5 missions as the unseen test set.
Metrics. For online evaluation, use the MSR as the task performance metric. As subjective evaluation metric, we collect users' overall satisfaction toward the game on a 5-point likert scale after each interaction.
5 Experiments
5.1 Offline evaluation
We use the AID dataset to train two baseline models. The inputs the models are the agent egocentric view and natural language conversations. The outputs are a series of executable action sequences and associated interactable masks (where needed) to complete the task hand. For both models, we experiment using only the human instructions, as well as the instructions appended with the questions and answers (QA) as input to simulate human-agent dialog during the mission in offline evaluation.
Neural-Symbolic Approach. Episodic Transformer (ET) is a neural-symbolic model for visual- language task completion [27]. ET uses multi-modal transformer to predict actions and object pairs based on the visual and language information in the entire episode, with separate object detector to generate object masks.

---

Mission
NS
NS
VL
VL
Type
w/o QA
w/ QA
w/o QA
w/ QA
0.00#0.00
0.00=00000
28.88=3..22
37.41=3.66
breakObject
(10.93+0,22)
(9.90+0005))
(8.90=0442)
(9.07+0.56)
14.18+1.443
13.79++0.00
11.4++248
16.85+3.01
clean&deliver
(.0666:00.06
(15.14:10.09)
(16.12:00.6)
(13.63:00.3))
0,0010.00
0.00100.00
0.0 0.0
2.7713992
color&deliver
(14,4411.44)
(13.22:10.17)
(16.6910.45)
(18.0520.67)
14.5810.00
20.1440.998
118014.28
23.61
fill&deliver
(20.88=0.27)
(17.12+0.08)
(15,43+1.277)
(13.79=1.80 1.66)
31.94+1.96
19.44
1,38=1.96
5.55+5.19
freeze&deliver
(17.17+0.07)
(17.42+0.322
(21.48+1.66)
(19.71+2.79)
5.13+0.00
5.13+0.00
23.07+5.53
28.20
heat&deliver
(14.7920.36)
(18.22+0.45)
(19.24=1.97)
(16.9140.57
14.6910.00
14.8
15.6312..2
22.03
insertnnDevice
(11.53+0.15)
(11.45+0.04)
(7.54-0.46
(1.22+043)
9.47:10.00
11.9310.29
21,28:0.92
26,5421.67
pickup&delliver
(11.15:00.10)
(12.66-10.08)
(8.03 0.36)
(7.8910.40)
13.10=0.40
14.1
25.92+5.004
31.05=1006
pourContainer
(12.1910.04)
(12.7110.12)
(11.09-0.90)
(10.07-0.29)
11.11+0.00
10.06
11.72+087
18.01+3.02
repair&deliver
(17.19=0.28)
(15.56+0.35)
(21.69+1.14)
(18.39=1.23)
41.44100.00
41.4410.00
41.4413.20
57.6514,55
scanObject
(4.90+0.00)
(4.84+0.00)
(3.72-0.19)
(3.96+0.36)
57.1420.00
36.19110:00
84.12:3,23
87.6224.85
toggleDevice
(5.0910.00)
10.08
(4.13 0.22)
(4.4910.68)
18.42=0.000
18.97=0.00
07.00=
33.88=0.71
average
(1.89900.01)
(11.62:0.03)
(9,88_0229)
(9.20:10228)
Table Experimental results on the validation datasct for the Neural-Symbolt (NS) and Vision-Language (VL)
models. For cach task, we show mission success rate and average number of robot actions (in the parenthesis). For each metric, we report the mean and standard deviation from three runs.
End-to-end Vision Language Model. We experiment with an end-to-end vision language (VL. model shown in Figure A transformer model encodes the natural language instruction, and the egocentric view is encoded using ResNet [15], both of which are initialized with pre-trained CLIP weights 29]. The encodings are passed through a joint VL encoder get combined vision-language embedding. This is then channelled through two heads: 1) The task planning head is an LSTM module encoding the context of previous predicted outputs, followed by two linear classifiers to predict the (action, object) pairs. 2) The mask prediction head uses the current first party view, the encoded language and the action-object prediction hidden state from the task planning head to predict single mask referring to the interactable object using the referential image segmentation architecture described in [45]. The above procedure is repeated for a natural language instruction and a series of first party views until an End-of-Sequent (EOS) flag is emitted by separate binary classifier.
5.1.1 Results and analysis
The baseline models are trained on the training split, and evaluated on the validation split of the AID dataset. The overall mission completion results are displayed in Table Both models are evaluated with a cap of 50 maximum allowed steps and 10 failed steps per mission, beyond which if the mission goal is not completed, the agent stops execution leaving the goal incomplete. Below, we analyze the results for both models and provide insights on the model performances.
Neural-Symbolic Model. Overall, adding QA leads to marginal improvement in the performance for the Neural-Symbolic model. The model performs well in MSR for missions with short horizons (e.g. toggleDevice). As an ablation study, we also evaluate the multi-modal transformer on the validation set by providing the ground-trute visual observations in the datasct at each time step. As a result, the model can correctly predict all the actions and objects for 71.3% of the missions without QA, and 76.9% with QA. Since it is common that there are multiple instances of the same class in a room, the model can choose incorrect interactable mask even if the transformer model correctly predicts the object class and the vision model correctly predicts all the object masks of that class in the image. This can also explain why the improvement with QA is limited, since the additional information does not participate in the grounding once the action and object pairs are predicted.
Vision Language Model. The VL model shows 6.88% absolute improvement in MSR when trained and evaluated with QA compared to training and evaluating without QA. The agent also able to complete the missions more efficiently with QA, demonstrating that asking the right questions
7

---

Team
Week 1
Week 2
Week 3
Week 4 (Unseen)
Offline
SlugJARVIS
18..0.118522882
4.10:1019 (67.335)
4.19:0.18 (49.56)
2.90 (55.56)
(3.66)
ScottyBot
3.48+00211
(46.46)
3.67=0021 (40.40)
3.97=0.15 (49.04)
3.63=0.21 (57.95)
(8.36)
EMMA
3.31+0.20 (47.66)
3.90+00188 (54.44)
3.88+0.18 (57.61)
4.28+00.4
(56.12)
(30.55)
SEAGULL
3.57:40.19 (33.78)
3.77:40.18 (50.00)
4.31 (51.72)
0.63+0.21 (37.86)
(30.98)
GauchoAl
4.05:10.17 (62.36)
4.3110.16 (68.38)
4.4810.15 (64.35)
1.33 (66.67)
(36,47)
Table 2: For online evaluation results, we show the weekly user satisfaction on S-point scale and missions
success rate percentage (in the parenthesis). We report the mean and standard deviation for the user satisfaction. For each team, we also show the MSR achieved the offline evaluation for comparison.
improves task execution efficiency and accuracy. For tasks like toggling which few objects can afford, the instructions tend to already be descriptive enough in natural language ("Turn on the red computer". "Press the blue button"), "), which explains why adding QA shows a mere 3% improvement in MSR. For all other task types which involve numerous objects, the QAs provide crucial additional information that not naturally provided with the instruction in the first turn, thus leading to significant performance improvement both in MSR and NRA. From analysis, the color & deliver and freeze deliver missions are prone to failure (as evidenced by low MSRs) because of relatively longer contexts than other missions and difficulty in disambiguating the receptacle of delivery.
5.2 Online Human evaluation
Alexa Arena is used as the interactive EAI platform in the Alexa Prize Simbot Challenge, in which human users can perceive the environment through the agent's egocentric view, and communicate with the embodied agent via natural language commands. To understand how additional user interaction data can help model development, the challenge schedule follows continual learning setting in four- week duration: in the first three weeks, the teams can freely update their model using interaction data on 9 missions, and in the final week the models are tested on 5 unseen missions for generalizability.
5.2.1 Modeling Methodologies
In total, 10 teams participated in the challenge and implemented their own models for Arena. The models output physical actions for the agent and engages in dialog with users to successfully complete missions. We briefly describe the design of some representative models. SlugJARVIS [ use hybrid approach for action prediction, with a rule-based parser to process most requests and an end-to- end Pythia [5] model to handle corner cases. They also design a progressive task experience retrieval module which helps make use of the existing successful user interaction data on seen tasks to enhance performance. ScottyBot [40) design modularized system containing a neural NLJ module for action prediction, named entity recognition, and action/object names standardization. The proposed design also incorporates semantic maps for efficient navigation and object localization. SEAGULL [25] use a rule-based system to detect user intent, and PDDL planner to generate robot actions. They also design hierarchical vision model to detect environment states, including fine-grained object categories, object states and objects spatial relations. EMMA 41 propose an encoder-decoder architecture to encode the multi-modal input and generate outputs as textual and visual tokens. The model is first pretrained on multiple vision languala tasks including masked language modeling and image captioning, before finetuning on embodied planning tasks. GauchoAl [42 also use modular framework for their agent, with a hybrid rule-based and neural action prediction module, an agent state tracking module, a preprocessing module to reject non-plausible actions, and a MaskFormer based visual grounding model. We refer readers to the technical reports for additional details [1]
5.2.2 Results and analysis
In Table we show human evaluation results for each team on a weekly-basis. For the first three weeks, we can see clear upward trend for user satisfaction within each week across all the models. For the unseen missions, most teams are able to achieve similar performance as seen ones, showing the potential to generalize to novel tasks. Compared the offline performance, we observe that the teams are able to achieve much higher mission success rate with human evaluation despite the tasks being more complicated. This shows the effects of having interactive dialog during task execution, which helps the agent understand the task goal and the user to understand the model's limitations. This is validated qualitatively by playing with the virtually deployed bots, where we observe critical
http://www..mmzoon.cccceeeeepppp
8

---

contributors to higher user satisfaction and MSR. While there is positive correlation between MSR and user ratings across different teams, users tend to give a higher rating bots that are transparent and collaborative, able to give clear feedback, let their limitations be known, and are good at anticipating user actions balanced manner, as opposed bots taking actions without appropriate feedback. For example, from Week 2 to Week 3 for certain models such as EMMA and GauchoAI, the user ratings increased while the overall MSR decreased. The increase in user satisfaction is attributed to variety of factors, including that these models became more user-centric, were more interactive and feedback friendly to the users. The teams were able to update their models with real-time human feedback signals to align their bots with the users that interact with them.
6 Conclusion and future work
We introduce Alexa Arena, user-centric EAI platform, featuring user-friendly graphics, animations and control mechanisms facilitate user-centric EAI research. We provide baseline results for a dialogue-guided task completion use-case, and also release the AID dataset. While the Arena platform is designed to improve user-experience for EAI systems, the proposed online models cannot be evaluated in straightforward fashion without a human in- tn-the-loop to supply interactive dialog and feedback. To overcome this limitation, we aim to incorporate model-based user-simulator in future work, to which an oracle provides relevant answers during task execution. Another direction to explore is enabling the human and the robot to control different embodied agents and create more scenarios for HRI.
Ethical Considerations. Via introducing Arena as public embodied Al platform built for human- robot communications, we mainly target English language and do not explicitly look at the issues innate natural language understanding such comprehending under-represented languages. The missions, tasks and game designs can be controlled by the platform users as needed, and is backed by framework that has been validated quantitatively and qualitatively, to the best of our abilities, by carefully constructed and robust user studies as presented in the paper. Since Arena is to be open-sourced, it is possible for users to modify the platform for unintended use cases, which is common risk for all open-sourced embodied Al platforms. But we believe the risk for Arena is low comparing to other platforms since Arena uses a gamified setting and the released recources are not directly applicable for real-world malicious applications. As for real world transferability of the trained agents, robust testing and continual safety monitoring mechanisms should be put in place to minimize risks of unpredictable robot behaviours in unexpected scenarios.

---

References
[1] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, et al. Do as can, not say; Grounding language in robotic affordances. arXiv preprint WiXiv:204.00199, 2022.
[2] J. Albrecht, A. J. Fetterman, B. Fogelman, E. Kitanidis, B. Wróblewski, N. Seo, M. Rosenthal, M. Knutins, Z. Polizzi, J. B. Simon, and K. Qiu. Avalon: benchmark for RL generaliza- tion using procedurally generated worlds. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
[3] Anderson, Q. Wu, D. Teney, J. Bruce, M. Johnson, N. Sünderhauf, Reid, S. Gould, and A. Van Den Hengel. Vision-and sgiion-ad-language navigation: Interpreting -oouull- navigation instructions in real environments. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3674-3683, 2018.
[4] C. Beattie, J. Z. Leibo, D. Teplyashin. T. Ward, M. Wainwright, H. Küttler, A. Lefrancq. Green, V. Valdés, A. Sadik, et al. Deepmind lab. arXiv preprint arXiv:1612.03800 2016.
[5] S. Biderman, H. Schoelkopf, Q. Anthony, H. Bradley, K. O'Bren, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, A. Skowron, L. Sutawika, and O van der Wal. Pythia: A suite for analyzing large language models across training and scaling, 2023.
[6] F. Brandle, K. Allen, J. R. Tenenbaum, and S. Eric. Using games to understand intelligence. Proceedings of the Annual Meeting of the Cognitive Science Society, 43, 2021.
[7] M. Carroll, R. Shah, M. K. Ho, T. Griffiths, S. Seshia, P. Abbeel, and A. Dragan. On the utility of learning about humans for human-ai coordination. Advances in neural information processing systems, 32, 2019.
[8] B. Cheng, A. G. Schwing, and A. Kirillov. Per-pixel classification is not all you need for semantic segmentation, 2021.
[9] T.-C. Chi, M. Shen, M. Eric, S. Kim, and D. Hakkani-tur. Just ask: An interactive learning framework for vision and language navigation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 2459-2466, 2020.
[10] C. Gan, J. Schwartz, Alter. M. Schrimpf, Traer, De Freitas, J. Kubilius, A. Bhandwaldar, N. Haber, M. Sano, et al. Threedworld: A platform for interactive multi-modal physical (tt1:2000.049554 2020.
simulation. arXiv preprint
[11] X. Gao, Q Gao, R. Gong, K. Lin, G. Thattai, and G. S. Sukhatme, Dialfred: Dialogue-enabled agents for embodied instruction following. arXiv preprint arXiv: 13330, 2022.
[12] X. Gao, R. Gong, T. Shu, X. Xie, S. Wang, and S.-C. Zhu. Vrkitchen: interactive 3d virtual environment for orientr learning. arXiv preprint arXiv: 2019.
[13] X. Gao, R. Gong, Y. Zhao, S. Wang, T. Shu, and S.-C. Zhu. Joint mind modeling for explanation generation in complex human-robot collaborative tasks. In 2020 29th IEEE international conference on robot and human interactive communication (RO-MAN), pages 11 1119-1126. IEEE, 2020.
[14] K. He, G. Gkioxari, Dollár, and R. B. Girshick. Mask R-CNN. CoRR, 0abs/1703.0668 2017.
[15] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. arxiv 2015. arXiv preprint arXiv:1512.03385, 14, 2015.
[16] W. Huang, P. Abbeel, D. Pathak, and I. Mordatch. Language models zero-shot planners: Extracting actionable knowledge for embodied agents. In International Conference on Machine Learning, pages 91 18-9147. PMLR, 2022.
[17] W. Huang, F. Xia, T. Xiao, H. Chan, Liang, P. Florence, A. Zeng, J. Tompson, Mordatch, Y. Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.0560 056000 2022.
10

---

[18] Y. Inoue and H. Ohashi, Prompter: Utilizing large language model prompting for a data efficient embodied instruction following. arXiv preprint arXiv:2211.03267, 2022.
[19] M. Johnson, K. Hofmann, T. Hutton, and D. Bignell. The malmo platform for artificial intelligence experimentation. In Ijcai, pages 4246-4247. Citeseer, 2016.
[20] E. Kolve, R. Mottaghi, W. Han, E. VanderBilt, L. Weihs, A. Herrasti, D. Gordon, Y. Zhu,
A. Gupta, and A. Farhadi. Ai2-thor: An interactive 3d environment for visual ai. arXiv preprint arXiv:1712.054444 2017.
[21] A. Mandlekar, Y. Zhu, A. Garg, J. Booher, M. Spero, A. Tung. J. Gao, J. Emmons, A. Gupta, E. Orbay, et al. Roboturk: A crowdsourcing platform for robotic skill learning through imitation. PMLR, 2018.
In Conference on Robot Learning, pages
[22] S. Y. Min, D. S. Chaplot, P. K. Ravikumar, Y. Bisk, and R. Salakhutdinov. FILM: Following instructions in language with modular methods. In International Conference on Learning Representations, 2022.
[23] D. Misra, A. Bennett, V. Blukis, E. Niklasson, M. Shatkhin, and Y. Artzi Mapping instructions to actions in 3d environments with visual goal prediction. arXiv preprint (00K:018000007888 2018.
[24] K. Nguyen and H. Daumé III. Help, anna! visual navigation with natural multimodal assistance via retrospective curiosity -encuurtyenncouraginn imitation learning. In Proceedings of the Conference Empirical Methods in Natural Language Processing (EMNLP), November 2019.
[25] U. of Michigan. Seagull: An embodied agent for instruction following through situated dialog. In Alexa Prize SimBot Challenge Proceedings, 2023.
[26] A. Padmakumar, J. Thomason, A. Shrivastava, P. Lange, A. Narayan-Chen, S. Gella, R. Pi- ramithu, G. Tur, and D. Hakkani-Tur. TEACh: Task-driven Embodied Agents that Chat. In Conference Artificial Intelligence (AAAI), 2022.
[27] A. Pashevich, C. Schmid, and C. Sun. Episodic transformer for vision-and-language navigation. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 15922-15932, 2021.
[28] X. Puig, K. Ra, M. Boben, J. Li, T. Wang, S. Fidler, and A. Torralba. Virtualhome: Simulating household activities via programs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages $494-8502 2018.
[29] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pages 8748-8766 PMLR, 2021.
[30] A. Ram, S. Ontañón, and M. Mehta. Artificial intelligence for adaptive computer games. The Florida Al Research Society, 2007.
[31] H. R. Roman, Y. Bisk, J. Thomason, A. Celikyilmaz, and J. Gao. Rmm: A recursive mental model for dialog navigation. In Findings of the 2020 Conference on Empirical Methods in Natural Language Processing, 2020.
[32] M. Savva, A. Kadian, O. Maksymets, Y. Zhao, E. Wijmans, B. Jain, J. Straub, J. Liu, V. Koltun,
J. Malik, et al. Habitat: A platform for embodied ai research. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9339-9347, 2019.
[33] M. Shridhar, J. Thomason, D. Gordon, Y. Bisk, W. Han, Mottaghi, L. Zettlemoyer, and D. Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10740-10749, 2020.
[34] I. Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg. Progprompt: Generating situated robot task plans using large language models. arXiv preprint arXiv: 2022.
11

---

[35] S. Srivastava, C. Li, M. Lingelbach, R. Marttn-Marrtnn F. Xia, K. E. Vainio, Z. Lian, C. Gokmen, S. Buch, K. Liu, et al. Behavior: Benchmark for everyday household activities in virtual, interactive, and ecological environments. In Conference Robot Learning, pages 477-490. PMIRR 2022.
[36] A. Suhr, C. Yan, C. Schluger, S. Yu, H. Khader, M. Mouallem, I. Zhang, and Y. Artzi Executing instructions in situated collaborative interactions. arXiv preprint arXiv:1910.0365. 2019.
[37] A. Suhr, C. Yan, C. Schluger, S. Yu, H. Khader, M. Mouallem, I. Zhang, and Y. Artzi Executing instructions in situated collaborative interactions, 2022.
[38] A. Szot, A. Clegg, E. Undersander, E. Wijmans, Y. Zhao, J. Tumne, N. Maestre, M. Mukadam, D. S. Chaplot, O. Maksymets, et al. Habitat 2.0: Training home assistants to rearrange their habitat. Advances in Neural Information Processing Systems, 34:251-2660 2021.
[39] Y. Tian, Q. Gong, W. Shang, Y. Wu, and C. I.. Zitnick. Elf: An extensive, lightweight and flexible research platform for real-time strategy games. Advances Neural Information Processing Systems, 30, 2017.
[40] C. M. University. Scottybot: A modular embodied dialogue agent. In Alexa Prize SimBot Challenge Proceedings, 2023.
[41] H.-W. University. Emma: A foundation model for embodied, interactive, multimodal task completion in 3d environments. In Alexa Prize SimBot Challenge Proceedings, 2023.
[42] S. B. University of California. Gauchoai: proactive modular embodied ai that collaborates with humans. In Alexa Prize SimBot Challenge Proceedings, 2023.
[43] S. C. University of California. Slugjarvis: Multimodal commonsense knowledge-basse based embod- ied ai for simbot challenge, In Alexa Prize SimBot Challenge Proceedings. 2023.
[44] Z. Wang, S. Cai, A. Liu, X. Ma, and Y. Liang. Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents. arXiv preprint arXiv:2302.01566 2023.
[45] Z. Wang, Y. Lu, Q. Li, X. Tao, Y. Guo, M. Gong, and T. Liu. Cris: Clip-driven referring image segmentation. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, 2022.
[46] F. Xia, A. R. Zamir, Z. He, A. Sax, J. Malik, and S. Savarese. Gibson env: Real-world perception for embodied agents. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9068-9077 2018
[47] F. Xiang, Y. Qin, K. Mo, Y. Xia, H. Zhu, F. Liu, M. Liu, H. Jiang, Y. Yuan, H. Wang, et al. Sapien: A simulated bast- interactive environment. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 11097-11107, 2020.
[48] L. Yuan, X. Gao, Z. Zheng, M. Edmonds, N. WU, F. Rossano, H. Lu, Y. Zhu, and S.-C. Zhu. In situ bidirectional human-robot value alignment. Science robotics, 7(68):eabm4183 2022.
[49] M. Ángeles Quiroga, S. Escorial, F. J. Román, D. Morillo, A. Jarabo, J. Privado, M. Hemández, B. Gallego, and R. Colom. Can we reliably measure the general factor of intelligence (g) through commercial video games? yes, we can! Intelligence, 53:1-7, 2015.
12

---

Supplementary Materials Overview:
Appendix A.J
contains details of the platform.
Appendix A
contains definitions of tasks.
Appendix
contains details of data collection process for the AID dataset.
Appendix
contains details of the vision dataset.
Appendix
contains details of the baseline models.
Appendix
contains error analysis of the VL model.
Our code and data available at http://gghhaa.cccmmmoooooonseeet
A
Appendix
A.1
Alexa Arena Platform Details
A.1.1 Architecture
The architecture of the Alexa Arena framework is illustrated in Figure The core components the platform are the arena wrapper and model wrapper. The arena wrapper houses the arena orchestrator, which takes as imput the configuration file (described in Section A.I.2), and communicates with the arena engine, which contains the the arena simulator and the streaming server. The streaming server in the arena engine streams the robot's first party camera view human users. The model wrapper contains the inference model and the model executor that interfaces between the model and the orchestrator. The model wrapper can modified to develop other models and execution strategies. In addition, we also provide a web interface (the webtool) for interacting with robot model a chatbot like interface, while also watching the game play out in streaming on web page.
A.1.2 Challenge Definition Format
To define the mission configurations, we introduce schema called Challenge Definition Format (CDF). A CDF file a json-format file that is readable by Arena. contains the necessary information to configure, initialize and run game mission in Arena. These include:
Game initial state: game scene setting, robot initial location, object initial location and initial states (such as cabinet door being open or closed).
2. Game goal condition: robot and object states that need to be satisfied. Once the goal condition is met, the game mission is considered completed.
3. Game-related text data: textual data that help guide the user to finish the game, including game mission description text, sub-goal descriptions, system prompts during game play, etc.
The motivation for designing and utilizing the CDF is to allow programmatic generation of large numbers of game missions, which is essential for data collection. Since manually defined game missions are not scalable for generating large-scale training data for ML models, with the easily configurable CDF files, we can generate thousands of game missions by permuting scenes, robot location & states, object types, object location & states, and mission goal states. The other advantage that since the CDFs are human readable, we can specify new game missions flexibly without writing code.
A.1.3 Environment Metadata
The environment metadata in Arena contains robot egocentric view images, agent state, objects state, scene metadata, goal condition status, and previous action execution result. To facilitate different task and modeling settings, the robot camera view images support different modalities, including RGB image, depth image, instance segmentation image and normals image. The metadata response from the arena simulator is updated after each action execution.
13

---

Auxiliary Arena
Development Tools
Submission
Arena
Eval Al Test Runner
Arena Wrapper
Model Wrapper
CDF
Arena Orchestrator
Arena Model
Task Metadata
Engine Arena Simuliator
Streaming Server
Inference Model
Figure Overview of Alexa Arena Platform Architecture, The core modules are the arena wrapper model wrapper. The arena engine contains the simulator and the streaming server. The metadata from the Arena Engine are sent to the inference model, which generates actions to be executed in the simulator. We have also built infrastructure to generate results submission files for EvalAI challenges.
A.1.4 Objects and State Transitions
There are 336 unique objects in Arena. Each object has a set of properties, called affordances, which specify if a certain type of robot-object interaction is possible. To further mimic the complexities of real-world tasks and execution strategies, along with numerous object types and properties, Arena also provides variety of constraints defined by commonsense causal events. For example, the power system enables a series of causal constraints on using electric appliances, e.g., the fuse box needs be reset before lights being turned on, the microwave needs to be plugged to the outlet before it can be powered on. Agents in Arena also have multiple ways to complete a task, e.g., To heat something, the agent can make of either the Laser or the Microwave. As result, the agent needs to reason based on its current state for efficient task completion. As an example, consider the task of heating a bowl and delivering the Quantum Lab. If the agent's execution path to heat the bowl using the Laser in the Quantum Lab is physically obstructed, it is more efficient for the agent to go the Breakroom, heat the bowl quicker using the Microwave and return to deliver the bowl in the Quantum Lab. A full list of of objects and their properties and state transitions are available in the "arena "arena_dooumen dowumentation.nnee html" file in the supplementary materials.
receptacle in the Quantum Lab, and the agent currently in
A.1.5 Layouts
Alexa Arena contains large game-ready interative multi-room layouts. Each layout features an office-like environment with modular design, allowing the contents within rooms to be rearranged and repositioned. The layouts were hand designed be visually different and aesthetically interesting. The office setting allows both mundane and fantastical interactive elements: users would be accus- tomed to seeing everyday objects in more conventional rooms (and would know instinctively how to interact with them), but entering rooms clearly labeled "labs" they would expect to find futuristic devices and prototype machines that would require experimentation to discover their function. As result of the above bipolar content, are able to create game missions with imaginative situations sitting alongside normal everyday objects. The layouts in Arena are large: on average, each house has 190 square meters of navigable area, which is significantly larger than the scenes in other EAI platforms (Figure 6). Figure / shows the top down images houses in Alexa Arena.
A.1.6 Related platforms
See Table for comparison between Alexa Arena and other embodied Al platforms.
14

---

Navigable Area of Scenes
175
150
125
Inw")
100
25
Alexa
ArchitecTHORR 412-TTHOR RoboTHOR ProcTHOR
Figure 6: Bar plot of the navigable areas for Alexa Arena, comparing to ArchitecTHOR., A12-ITHOR. RoboTHOR and ProeTHOR.
(a)
(c)
Figure 7: Top-down images 4 multi-room game-ready layouts in Alexa Arena.
A.2
Task definition
We designed two set of tasks: 1) the ATD dataset include 12 simple tasks that require the agent to perform some basic object interactions, 2) the 14 compositional missions are used for online evaluation, each mission requires the agent to complete multiple sub-goals.
A.2.1 Basic tasks
There are 12 types of basic tasks. For most tasks, the agent is required to find the target object, use specified tool to change its state and deliver it to the designated receptacle.
teggleDevice. The agent is required to find the device and toggle it. For some devices (e.g. color changer), the agent needs to interact with the correct object parts (e.g. red button vs green button) in order to finish the task.
15

---

Object
Multi Room
Object States
3D
Real Scenes
Human Control
Game Ready
Platforms
Scenes
Types
Env
Malmo [9
17
1000+
MK
Overcooked
f
6*
5
MK
Deepmind Lab
28
MK
A12-Thor 20
120
125+
MK,VR VR
iGibson [46
15
390+
MK, VR
Habitat 1.0 32
1000
V
MK
Habitat 2.0
38
105
90+
MK
VirtualHome 28
7
170+
V
NL
TDW [10]
15
200+
V
VR
VRKitchen
12
16
200+
V
VR
Alexa Arena
10
335+
MK,NL
Table 3: Comparison with other platforms. Scenes and objects: the number of scenes and interactable object types the platform provide. Multi-room: multi-room scenes, Some platforms marked by * are customizable and new scenes can be added. States: agent can change object states via interactions. 3D env: whether the platform supports 3D virtual environment. Real scenes: whether the scenes in the virtual environment are realistic. Human Control: the interface for the human control the agent. MK: mouse and keyboard. NL: natural language. VR: virtual reality. Game ready: whether the platform game-ready features to enhance user experience, including environmental and interaction sounds, action animations, smooth navigation, faattastcal objects, hazards, mini-map, scoring mechanism and hints.
The agent is required first find the target object, pick it up, navigate the target receptacle and put the object on the receptacle.
pickup&delivedr
breakObject. The agent is required to pick up hammer, navigate to the target object, and use the hammer to break the object.
dleive. The agent is required to pick up the dirty object, navigate to the sink, turn on the faucet and clean the object using the water. The agent can then deliver the cleaned object to the receptacle.
collr&deliver. The agent is required pick up the object, go to the color changer and put the object on the color changer. The agent can then push a button on the color changer to change the color of the object, and deliver
the receptacle.
dillvveliver The agent is required to pick up a container, go to the sink, and turn on the faucet. The agent then fill the container with water and deliver it to receptacle.
freeze&deliver. The agent is required to pick up the target object and use either the fridge or the freeze ray to freeze the object. The agent can then deliver it to the receptacle.
delive. The agent required to pick up the target object and use either the laser or the microwave to heat the object. The agent can then deliver it to the receptacle.
insertInDevice. The agent is required to pick up small object (e.g. floppy disk, printer cartridge, laser component) and insert it into the corresponding device to assemble the device.
PourContainer: The agent is required to pick up the container (e.g. milk carton, cereal box, coffee pot), navigate to the receptacle and pour the things the container into the receptacle.
repair&delivee The agent is required to pick up broken object, and use the time machine to repair The agent can then deliver the intact object the receptacle.
scanObject. The agent is required to go to a place near the target object and scan it.
A.2.2 Compositional tasks
For the online human evaluation, suite of 14 long-horizon missions designed to test compo- sitionality and generalizability. Each mission includes multiple sub-goals that need to achieved to complete the overall task. The human users use the sticky notes placed in the scene to guide themselves during mission play. An example mission is provided below with the overall setup, goals and subgoals. Only the goals and subgoals are visible the users during mission play.
16

---

Scene: The robot is spawned in the Robotics Lab, a trophy is kept on one of the tables in the corner, but access to that side of the room is blocked by a wooden block. The block can only be moved by toggling a robotic arm, which lifts and displaces the block and clears the path.
Find the trephy and deliver it to the manager's desk.
Pick up the trophy
Deliver it to the manager's desk
needed, hints are located at visible locations to unblock the user to complete the mission. For the above example, the happy path of the robot to complete the mission without any impediments involves toggling the robotic arm to move the wooden block, gaining access to the other side of the room. The robot then picks up the trophy which is now accessible, and navigates to the manager's desk to deliver it. Such a mission tests the visual and compositional reasoning of the agent. The agent must understand that the path the trophy is blocked, and that the robotic arm controls the wooden block, which can be moved to clear path.
The remaining 13 missions are described briefly below:
Assemble and fire the laser cannon. The agent is spawned in the quantum lab, required to find control panel and place it on the laser (which looks very similar to the freeze ray). The laser is then turned on using the laser monitor (one of two monitors disambiguated by color and position).
Turn a white bowl red. The agent is spawned in the quantum lab, is required to pick up one of 3 bowls from the breakroom and use the color changer change the color of the bowl.
Disinfect the computer. The agent is required to find the floppy disk in the manager's office, navigate to the main office and use the disinfecting floppy on the computer with infection (indicated by skull on the monitor among other monitors).
Freeze the soda can with freeze ray. using the freeze ray. This requires finding the right monitor to toggle.
The agent needs to find the soda can on a table and freeze it
Defrost the cake with laser. The agent needs to find a frozen cake the freezer and defrost it using the laser in the quantum lab, which requires disambiguating between tables and monitors.
Print action figure. The agent needs to print an action figure using the 3D printer. Before doing this, the agent needs to turn on the power in the room (as the room is initally dark) to power the 3D printer on.
Turn a red apple green. The agent needs to find red apple in the lower fridge and use the color changer on it turn green.
Make a bowl of cereal. The agent is required to make a bowl of cereal by pouring milk and cereal into bowl in the breakroom. The milk is initially in the fridge, the cereal on the breakroom table, and the bowl is initially broken. The agent needs to fix the broken bowl before making the cereal.
Pour coffee and deliver to reception. The agent is required pour coffee into mug from the breakroom and deliver it to the reception table.
Heat a cup of coffee using the laser machine. The agent needs to find the mug with the coffee and heat it using the laser machine in the quantum lab. This requires finding the laser monitor and toggling it.
Scan items. The agent is required sean computer, light switch and vending machine.
Find prorotype and toggle. The agent needs find the prototype machine with a face on it, and toggle it repeatedly until sad face appears.
Print mug. The agent is required find the mug printer cartridge on a table, insert it in the 3D printer and use it to print a 3D mug.
17

---

Game mission generation
PDDL Problem Definition
Action Sequence
Planner
Type: heat & deliver Objects:
Objects: mug, laser,
gotoffridge) open(fridge) pickup(mug)...
mug, laser, Init: In(mug, fridge),
Init: In(mug, fridge),
Goal: IsHot(mug),..
Figure 8: Process of generating game missions and expert demonstrations.
A.3 AID Dataset Collection
In this section, describe the details of the data collection process for the Arena Interactive Dialogue (AID) dataset. The game mission and expert demonstration generation process is illustrated Amazon Mechanical
in Figure 8 After generating the missions and expert demonstrations, Turk (AMT) for collecting human instructions as well as questions and answers necessary for task completion. Annotators receive $1 compensation for each annotation task which roughly takes 3 minutes to complete, and the hourly wage is around $20. All annotators agree to the Amazon Mechanical Turk Participation Agreement" before starting the annotation.
A.3.1 Human Instruction Annotation
To collect natural language dialogue for the missions, we design two-stage data annotation process. The first stage is for collecting instructions and the second is for questions and answers. In both stages, the annotator watches a video containing an expert demonstration for the mission and provide the annotations by text data or multiple choice questions.
In the first stage, annotators are told to write instructions to tell a "smart robot" how to accomplish task. During the process, annotator first watches the video of the ground-truth robot actions, then writes instructions for each highlighted video segments. Fig. 9a shows the user interface for collecting language instructions.
After the instructions are collected, we start the second stage annotation, where the annotators asked to raise questions to help them complete the task, if they are controlling the robot to follow the instructions. They also need to answer their own questions. Fig. shows the interface for collecting questions and answers. Here the steps annotators need to follow:
1. Annotator watches a video: and the respective instruction is shown screen. The video clip helps the human annotator understand the initial environment states.
10 second video clip is played till the beginning of a sub-task,
Annotator selects a question: The annotator selects one pertinent question that they think may help task completion. The questions generated from several predefined templates. Annotators also have the options to type in their own questions.
Annotator answers the question: The same annotator watches the video segment of an expert agent working on the sub-task specified by the language instruction, and then providing the answer to the question in text.
Annotator indicates whether asking the question necessary: In some cases, the instruction and the visual context is already clear enough, and asking a question is not necessary.
To ensure the annotation quality, in practice, we collected 3000 data sessions for the first data collection stage. Then we asked additional annotators to identify annotation errors and try to correct them, e.g. erroneous language instructions, grammatical errors, misalignment between instructions with robot actions, etc. For data sessions where most collected annotations do not make sense and are not worth fixing, the additional annotators are asked to simply ignore them. As a result, 20.3% sessions are ignored and 35.7% instructions are corrected. From the data verification results, we identified a subset of AMT workers (around 300 workers) who generated high-quality data, and used them for subsequent data collection process.
18

---

Instructions:
1. Imagine you are guiding robot via language instruciios firihh task.
right
video
Vinte srep-tty-step instructions that you wmild give to the enhvot in onder arde in arhieve whet shown in the video rlips
per stop, plaase pay
correctly align
with comnepondinng stop
Imagine you are guiding robot via far guage instruciion fiiish this tade: Find the flapey diikk and delbver It the dealt Writ down what were your step by step instructions that lead the robot actions the
PRATIM
SAON
write instruction here:
AUP
Play Step 2
wilk instructionn hore.
ACCORA
Play Step
welte instruction hire:
Thank you for completing this HITI Do have feedbadk (optional) for us?
Step1
Pheass provide feellbuck topi trall hrre
Submit
(a) User interface for collecting human instructions, Annotators watch the expert demonstrations and write down instructions to guide the robot.
View intructions
Pleass the instruciio nu top feft corner beforn starting
Begin
Show Demnnettttion
Step "tacir" button and wiich wide sill stopp.
Stpp 2 how the toklowing commanct
CEREAL
CEREAL
TASTER
into
question
theobowe conniardd
Which netion doout
C Whheer in the cevell wherr
Odwer Oppe bakow)
Uru questic upp
"Show Denanitaateer wetch ousctas moner Anoser dus sirve priction
Stpp4 Dayoo thenk the quatiion eeceosery conccutee commondd
you completing
[Cattona prodide your hure.
Subnk
(b) User interface for collecting questions and answers. Annotators input their questions and answers to help the robot complete the mission.
Figure 9: User interface for data collection.
A.3.2 Questions Generation
Similar to [II] the questions choices in the second stage of annotation are generated using predefined templates. Given instruction, we extract nouns and insert them into the templates construct questions. The nouns extracted correspond to query objects that the agent need more information to complete the mission.
In particular, we consider three types of questions, related to the location and appearance the query object o, and the relative direction between the agent and the target position. The templates for each question are defined below:
1. Location: where o?
19

---

# scene
# task
# session
# instr
# dialog
Train
6
2661
7983
40443
40105
Valid
6
383
1149
6120
6128
Total
6
3044
9132
46563
46233
Table 4: AID Dataset Breakdown. session represent number of data sessions. Each data session corresponds to one language annotation on a given task. # instr stands for the total number of instructions in each data split. # dialog is for the total number of question and answers.
Action frequencies
expert demonstrations
Human utterances length
Action Eypes
Number of Steps
NNmmer wiadh
(a) Frequencies of action types.
(b) Steps in expert demonstrations.
(c) Length of human utterances.
Figure 10: Statistics of the AID dataset.
2. Appearance: what does look like?
3. Direction: which direction should turn to?
4. Reference: which o are you referring to?
A.3.3 Oracle Answers
Human language has a lot of variety and complexity, and thus can be quite challenging for the agent to understand. Alternatively, we can provide templated language to the agent as a starting point for language understanding. To this end, in addition to asking human annotators answer the questions, we also build an oracle that can answer the questions by extracting information from the simulator metadata: (i) To answer the location question, we compute the direction of the object relative the agent and the viewpoint that is closest to the object in the room. To help detect small objects, if the object is held by a container, we also provide the information of the container and other objects that are also held by the same container as landmarks; (ii) To answer the appearance question, we directly extract object shape, color and material information from the simulator metadata; (iii) For direction question, compare the agent's location at the end of the sub-task with its initial location to provide answers for the agent's 's moving direction. After we have the necessary information, we use language templates to generate the answers. Example templates include:
1. Location: The o is to your [direction] in/on the [container] next to the [landmark] in the [room]. It is closest to [viewpoinr].
2. Appearance: The is [shape] and of [color]. It is made of [material].
3. Direction: You should turn [direction] / You don't io need to move.
A.3.4 Dataset statisties
Table Figure 10 and Table 5
show the statisties of the AID dataset.
A.4
Vision Dataset
Visual perception is integral for EAI agents to navigate and interact within their environments. Visually intelligent robots visual cues navigational targets and build visual representations based on perceived object state transitions. To facilitate visual Al research within and outside the context of EAI, 600k training images and 60k validation images spanning 336 unique objects from more than 160 semantically grouped object groups. This can be used for large scale computer vision research on Arena, e.g. object detection, object state classification, etc. The overall structure and contents of the dataset illustrated in Fig. The vision dataset is collected by programmatically configuring
releasing vision dataset based on Arena. The dataset is composed of
20

---

Mission Type
Train
Valid
breakObject
196
30
clean&deliver
161
29
color&deliver
29
4
filldelliver
108
16
freeve&deliver
76
8
heat&deliver
42
13
insertinDevice
381
59
pickup&deliver
669
95
pourContainer
265
39
repair&deliver
131
18
scanObject
324
37
toggleDevice
279
35
Table 5: Distributions of mission types across training and validation splits of the AID dataset.
Vision Dataset
Raw Files
metadata
RGB Color Image
Segmentation image
CDF
RGB Color Image File
Segienntttio Image Fill
IImage Annotations
Deptth Image
Object States
Object
and
Metadata Response
Commands
Figure 11:
Overview of the vision dataset. The dataset consists RGB and ground truth segmentation images png format, along with a json file for each image with the metadata information.
CDFs and initializing scenes with all objects spawnable in the Arena environment. We then navigate each of them capture images from different perspectives and distances.
A.4.1 Vision Data Generation
The image data collection strategy was simple and straightforward the goal was to collect data spanning all possible objects spawnable in Arena from different distances, views and perspectives of the said object. This was done by following the below steps:
1. Initialize game mission using the CDF file with required objects.
2. After initialization, we issue commands to navigate to all the different objects present in the scene using the primitive navigation actions and capture the first party view.
3. We then capture images by issuing interactive commands per the affordances of that object at time of capture, which are retrieved from the game metadata response.
4. We also capture additional perspectives by taking random walk around the object using basic trigonometry.
We repeat the above steps with various missions and objects with lights on and off for different lighting conditions. All the commands, object states and their locations in the scene, along with a host of other useful information logged in metadata files. We provide the RGB color images, segmentation maps and associated metadata for each image. The statistics of the dataset are illustrated in Fig. 12
Dataset Validation. To validate the quality of the ground truth segmentation and bounding box annotations for modeling, collect 1245 images containing instances spanning all object classes
21

---

0.7
0.3
0.2
0.1
0.0
Small
Medium
Object Size
Vision data object size distribution.
Fraction of maximum
Maximum "980000 Minimum "20
Absolute Scale
Log Scale
(b) Vision data class frequency.
Figure 12: Vision dataset statistics
and annotate them with bounding boxes and associated class names. Since Arena features fantastical objects (like Gravity Flippers), we also provide example images for each object class familiarize the annotator with the appearance of all classes. The images are then validated by annotators according the class names into one of the following 4 categories (with results provided parenthesis):
1. Yes, the object is fully contained within the bounding box (79.7%)
2. Yes, the object is partially contained within the bounding box (16.3%)
3. No, the object is not within the bounding box, but is in the image (1.6%)
4. The object not the image (2.4%)
For category 2, the objects are present only partially in the image because they are only partially visible in the robot's egocentric view at the time image collection. Overall results show that the ground truth segmentation and bounding box annotations have very little noise and are of high quality to be used for training and evaluation visual perception models.
A.5
Baseline Models Details
A.5.1
Neural-symbolic Model
The architecture is shown in Figure 13 In particular, the multi-modal Transformer model uses the previous actions, previous visual observations and the full utterances of the episode to predict the next action type and object class before stop token is predicted as the action. For the visual observations, we first extract features using the backbone of the vision model before sending them into the transformer. During training, after each look around action, the model uses the observations from different directions to predict rotations: the number rotation depends on which image out of the four panoramic images is used by the following actions in the expert demonstrations. During
22

---

Previous Actions
Multi-modal Transformer
Actions and Objects
Goto
Goto
Lab1
Look
Look
Goto
Goto
Desk
Object Mask
Pickup
Bowl
NEW BEBT
TTER
Utterances
AMENCC
the robotics lab turn left and grab the bowl the corner... place bowl the desk
Vision Model
Visual Observations
Figure 13: Model architecture for the Neural-Symbolit approach which uses unshared modules to predict actions and masks, and hcuristic rules for visual grounding.
inference, the vision model detects all the objects from the current image, and uses the predicted object class to generate the mask for the corresponding object. Note that if the vision model detects multiple object instances of the same class, we select the one with the highest likelihood. We trained the transformer model for 20 epochs with batch size of 2. The training takes around 10 hours 1 NVIDIA A10 GPU. We use the Adam optimizer with a learning rate le-4 for the first 10 epochs, and le
for the remaining epochs.
The Neural-Symbolic model composed of a separate visual component that performs instance level segmentation of the first party view and/or surrounding images of the agent. We use the vision dataset to train a Bask-RCNN image segmentation model [14]. We process the object classes in the vision dataset group them into 86 semantic classes including the background class. The model takes as input an RGB image and predicts masks for all object instances (across the 86 object classes) present in the image along with their class labels and confidence scores. The masks and object class predictions are then used in conjunction with the action and object predicted from the multi-modal transformer give single mask prediction for the interactable object. The vision model is trained for 22 epochs with a learning rate of 0.00125 and a weight decay of 0.001 with the SGD optimizer, decaying the learning rate after 15 epochs by a factor of 0.1. We use global batch size of 16 and trained the model on 2 Tesla V100 GPUs, and the training takes around 24 hours. We evaluate the instance segmentation model using the standard COCO evaluation metric, the Mean Average Precision (mAP). The mAP metric is calculated by averaging the precision at Intersection over Union (IoU) thresholds ranging from 0.5 to 0.95 in steps of 0.05. The score of the predicted instance class is not taken into considcration, and instead a cap is applied on the number of maximum detections per image (chosen be 100). To be able choose an operating point for our end-to-end model and also tune for recall, in addition to the standard COCO mAP metric, we also provide score and thresholded precision metric, which we call 1-mAP. This is calculated by thresholding the output instance class scores [0.05, 0.1, 0.3, 0.5, 0.7]. and thresholding the loUs at 10.1, 0.3, 0.4, 0.5. 0.75, 0.8] for all 30 (score, IoU) combinations and then averaging them. We present both metrics for all objects, well as metrics categorized for small, medium and large objects. The results are provided in T
A.5.2 VL Model
For training, we use a learning rate of 1e for the backhone and mask prediction head, and 1e for the action prediction head (chosen empirically). We use the Adam optimizer with a weight decay of 1e-5 and learning rate decay of 0.1 after 35 epochs. The model is trained for 65 epochs with global batch size of 64 (16 per GPU) on Tesla V100 GPUs, and the training takes about 7 hours.
23

---

Category
Area (px
COCO mAP
t-mAP
Small
0- 1296
37.63
91.49
Medium
1296 9216
60.41
92.15
Large
9216 90000
64.72
84.91
Overall
0 90000
46.03
89.5
Table Image segmentation results for the vision model used in the neural-symbolic approach. We present the metrics small, medium and large objects.
Figure 14: There are two tables in this image, blue table on the left and green one on the right, leading to challenging disambiguation tasks for the agent if guided adequately.
A.5.3 Encoding Questions and Answers
We use two tokens to efficiently encode the predefined questions, including (i) a token representing the question type (i.e. loc for questions asking about object location, app for object appearance, dir for direction and ref for reference), and (ii) a token for the target object in location, appearance, referential questions. For example, a question asking about the appearance of microwave will be encoded as app microwave. Since the direction question is asking about the direction and not related an object, the second token is necessary and thus omitted for this type of question. For questions that are not of the predefined types, we include the whole question and answers.
A.6
VL Model Error Analysis
While the VL model demonstrates improved performance using end-to-end method that generates actions and masks simultaneously, we identify key insights, sources of errors and failure points the end-to-end architecture by analyzing the robot trajectory for 25 missions:
1. Some missions fail because of failure to visually ground the natural language instruction to single object instance for interaction the case of multiple instance candidates. E.g. in Figure there are two desks blue on the left and green on the right. If the natural language instruction is descriptive enough, the model fails visually ground the desk of interest. On analysis, we notice that in the <.>> Exllveeiver mission type, the agent is often able to complete the <x> goal and fails the deliver goal because of ambiguity language ('The desk is right in front of you with the monitor on it"), model visual perception failures to disambiguate between multiple instances the same object.
Sometimes, if the agent veers off its path due wrong prediction (c.g. if goes wrong room), the subsequent instructions to find an object in the room steers the agent down a path to infinitely rotate about its position to try finding the object i.c.. the agent is trying respond to the next instruction that assumes successful attainment of the correct previous state. This can

---

potentially be solved by better enror correction mechanisms in the model to identify such loopy states and exit them explore other options the trajectory, or execute strategy to backtrack to previous instructions. Since the current baseline models use the readily available answers in the dataset, this can also be improved in models that are designed ask questions based on its real-time current visual observations.
3. Since this is an end-to-end model, the language and visual features are jointly encoded and trained end-to-end from input language and images output masks and actions. Some errors arise because the model is attending more to the visuals in front of it rather than the language input. E.g. if the robot is standing very close a cabinet door, or shelf with a coffee pot, even though the language instruction was to "go the laser cannon" it will try to Open and Close the cabinet door, or Pickup the coffee pot, respectively, which the model has learnt from the data.
Some more sources of error distributed in the model across all evaluation tasks include ambiguity in language, inability to model longe-horizon task instructions like "turn right, to the breakroom, heat the bowl and place it the table" " which consists of at least 9 steps, inability detect masks for some very small objects, or for objects too up close, and failure to distinguish between very similar looking objects (like Laser Cannon and Freeze Ray). These are common issues between the Neural-Syy Seural-Symbolic and the VL models.
We also did an ablation study by training only the mask prediction head and using the ground truth action predictions. Evaluating this model the validation dataset yields MSR of 71.89%. This simulates the scenario when the perfect actions are predicted to situate the robot in a position predict mask for interactable object, thus evaluating only object detection and visual grounding of the natural language instruction. This result shows the model's difficulties in finding object locations in end-to-end fashion, requiring the incorporation of better path planning modules that learn from multi-modal inputs.
25


Reviews:
[{"review_id": "fLt7GphZBU", "reviewer": "Reviewer_WUL3", "summary": "", "questions": "", "limitations": "My biggest concern is that this work is impossible to use because the main contribution, the platform, is closed-source under an extremely restrictive license that gives away all IP to Amazon and prevents any future research and any scrutiny of the current work.\n\nWhile the abstract states \"We make Alexa Arena publicly available to facilitate research in building assistive conversational embodied agents.\" This is false. The platform is not public. The code that wraps the closed-source by-request-only executable for the platform is made available.\n\nMoreover, this fact is hidden and not revealed in the publication. Nowhere in the publication is it made clear that a closed-source executable is needed to run this work. That the executable is only available by request. And that it involves an extreme, never before used, and unprecedented license. It is stated in one field in OpenReview. But, the implications of the \"executable\" being closed are not obvious: one would expect this to be a minor component, with the rest being open as advertised, not the entire platform itself.\n\nA license which virtually all researchers, academic or industrial, would be forbidden by their legal office from using. Practically, it means that while this work is interesting, it is of extremely limited utility. No one can build on it but the original authors and everyone who works on it gives their IP to amazon.\n\n1. The license gives away all IP to Amazon\n\nWhile the data of the dataset is under a standard license, CC-BY-NC 4.0, the code of the dataset (equally essential) appears to be under a very non-standard license. One which has never appeared before in any project or dataset; indeed, this is the only time it has appeared online at all.\n\nTo use the platform one must get an \"Arena executable\". This is not open source, it is governed by the https://raw.githubusercontent.com/amazon-science/alexa-arena/main/ARENA_EXECUTABLE_LICENSE and it can only be obtained by personal request.\n\nSection 5 states:\n\n> 5. Reservation of Rights; Feedback\n\n> The Materials are the intellectual property of Amazon or its licensors. Except for the rights explicitly granted to you in this License Agreement, all right, title, and interest in and to the Materials are reserved and retained by us and our licensors. If you make suggestions, ideas, or other feedback available to us relating to the Materials, we will be free to exercise all rights in such feedback without restriction and without compensating you.\n\nA publication is by definition a suggestion, idea, or feedback related to the dataset. This means that anything anyone publishes includes a license to Amazon. Using this dataset would violate existing IP agreements that most researchers have. And most researchers would not agree to do research on this dataset given that it means Amazon has their IP. Moreover, most companies and universities would not allow their researchers to do this.\n\nWorse. This isn't made clear anywhere in the publication or on the website. Indeed, GitHub states that the work is provided under an LGPL-2.1 license. This is highly misleading. The only note is that a separate non-commercial license exists, but this statement falls well short of revealing what the restrictions of this license are.\n\n2. The license also prevents any future work\n\n> 3. Limitations\n\n> You may use the Materials only as expressly authorized under this License Agreement. You may not: (a) distribute, sub-license, sell, offer for sale, lease, transfer, publicly display, publicly perform, or otherwise provide access to any portion of the Materials to any third party; (b) modify, adapt, or create derivative works of the Materials; (c) circumvent or disable any copy protection, security, or other controls in the Materials; (d) reverse engineer, disassemble, or decompile the Materials (except to the extent applicable law doesn’t allow this restriction); (e) use the Materials with any software or other materials that are subject to licenses or restrictions (e.g., open source software licenses) that, when combined with the Materials, would require you or us to disclose, license, distribute, or otherwise make all or any part of such Materials available to anyone; (f) remove, modify, or obscure any copyright, patent, trademark, or other proprietary or attribution notices on or in any Materials; or (g) direct, encourage, or assist any other party to take any action prohibited by this License Agreement.\n\nThis means that no researcher can modify this platform. How can one add assets?  Modify robot actions? What if there is a bug? How can we change the physic model? The answer is that one cannot. The key contribution of this work, the platform, is in truth not released and not accessible.\n\nI was dismayed to have to discover this and not have it clearly stated in the manuscript.\n\n3. The main contribution is not available\n\nIt is hard to see the scientific utility of this platform if one cannot actually use it. Unless it is made available to researchers, both practically (not just as an executable) and under a license that researchers can use, the manuscript essentially describes what research only the authors can perform and no one else.\n\n4. Limited and confusing object properties\n\nWhy are object properties so restricted? Virtually anything can break. Almost anything can be moved. Why for example art dart boards are not movable?\n\nIt looks like some objects are arranged in a hierarchy? \"Computer_Monitor_01\", \"Computer_Monitor_Broken\", \"Computer_Monitor_New\" are these all instances of computer monitor? Why is \"Computer_Monitor_01\" a receptacle and infectable while the others are not?\n\nSimilarly with paper trays, there are empty and full ones. How does the transition between them work?\n\nOther object properties, while, in a game-world sense, are not logical to a human. A dart board is not a receptacle. Dart boards do not contain darts.\n\nIn general, the object and object interaction model appears to be very arbitrary and poorly specified. This very much limits how generic the dataset is and its long term applications to very narrow settings.\n\n5. Reasoning\n\nVery little reasoning is required. Almost all of the tasks are of the form \"get object, perform action with object\". This makes the questions very plain and simple to parse into a formalism like PDDL (which is precisely what the authors do). But it's hard not to see this as a missed opportunity. In a rich game-like environment why restrict yourself to such simple questions? While the dataset goes beyond previous work in raw scale, it does not innovate in terms of the complexity of the language used.\n\nInstead of restricting this platform to a dozen simple mission types, why not define bigger missions like you see in a computer game (the need to avert some catastrophe or get through a dangerous lab), and have humans describe in natural language how a robot could solve those missions? Humans could describe the actions of the robot at different levels from the highest like \"get through the lab without being electrocuted\", to \"that means avoiding using the broken coffee makers\", to \"touch the 2nd panel on the right as you enter the lab\" assuming that one is not broken. This would lead to a far richer dataset without these predefined categories. One could of course imagine many similar setups.\n", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "Current platforms are very limited in scope.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "C400KR3mLE", "reviewer": "Reviewer_A3RV", "summary": "", "questions": "", "limitations": "Yes", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "Significance: This is a new platform for studying language use in interactive, embodied scenarios. In comparison to existing platforms (e.g., ALFRED, CerealBar), it adds a few additional features, such as gamification to make the task more interesting for human annotators, a realistic environment, more hierarchical \"missions\" to complete, and more complex world dynamics (e.g., multiple ways to accomplish a single goal).\n\nRelevance: This is very relevant to much of the current work in embodied language use. The later experiments on continual learning through interaction a relevant to a few recent works on learning from human feedback in embodied, interactive scenarios.\n\nQuality of research: The game is well designed, with additional features on top of what existing platforms already provide. Some of the interaction affordances (e.g. highlighting an object) are also novel for these platforms. The paper also includes several experiments; I am particularly interested in the experiments on continual learning through interaction.\n\nEthical and social implications: This is not discussed much in the paper, beyond the common motivation that agents which can collaborate with people in embodied scenarios have many applications, e.g. in accessibility.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "T7OAsRoajO", "reviewer": "Reviewer_PEw7", "summary": "", "questions": "", "limitations": "The authors discuss the limitations of this work, including the difficulty of evaluating online models without a human-in-the-loop.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The simulation environment seems user-friendly and engaging, which can help with better data collection.\n2. This framework provides the ground work to test compositional learning capabilities of embodied AI models by considering states for object.\n3. The problem this work tackles is important since it enables easier data collection in human-robot interaction scenarios where it is usually difficult and requires controlled environments to collect data.\n4. The paper is presented very well and is easy to follow.\n\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "ZW4SvRuyz3", "reviewer": "Reviewer_zBVW", "summary": "", "questions": "", "limitations": "* The authors have effectively taken into consideration the shortcomings and possible adverse consequences of their research.", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "The \"Alexa Arena” includes a valuable dataset and an interactive evaluation protocol, encouraging robust and authentic research. With the focus on assistive conversational agents, it has significant societal implications, bolstering the development of AI designed for human assistance. The provision of the platform and its rich dataset for public use highlights a commitment to open and collaborative research.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "MWVnhpvi9k", "reviewer": "Reviewer_Yxqr", "summary": "", "questions": "", "limitations": "see above.", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "+ Interesting and significant idea. Indeed, natural language is reasonable for human to control/guide robotics to accomplish a task as a collaboration. \n\n+ Both language input and key board/mouse action control.\n\n+ Very rich accessories with carefully designed properties. Some are even unusual in daily life.  \n\n+ Wide-range task compositions and action space.\n\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: This is a controversial paper, and reviewers (and ethics reviewers) do not agree on appropriate disposition.

- Pros: several of the reviewers have a positive impression of the paper, with the general sense that the presented testbed (the "Alexa arena") for exploring embodied AI in simulation has the potential to be a useful benchmark. The gamification of the human-robot interaction has the potential to improve uptake or at least participant experience, and the experiments presented make a baseline for future learning approaches. Overall, the game is relevant to current work on embodied language, and has some advantages over existing testbeds. If the license and ethical concerns weren't there (see below), this paper would be a suitable fit for the track.

- Cons: In its current form, the actual arena is essentially unavailable. What the developers have released is a wrapper around code which is accessible only by request from the developers, and use of that code is controlled by a rather draconian license. The terms of this license make reviewers unwilling/unable to experiment further with the testbed. Furthermore, the actual paper does not acknowledge this limitation, and instead describes the license which the author are currently seeking approval for. The authors point out, correctly, that in "select cases, requiring solid motivation," it is acceptable to release materials up to a year after acceptance; however, in this case the paper should acknowledge the current state.

- Ethical concerns included lack of acknowledgement of the potential problems with being English-centric; although the recommendation is that this be further discussed in the ethical considerations section, the changes consist of a single sentence stating that the authors have not looked at these issues. They also included concerns about the potential misuse of the arena to support unethical behavior on the part of the agents. One possible recommendation for this is, unfortunately, a _more_ restrictive license than that currently being sought. The suggestion that 'guard rails' be added to minimize the AI agent's ability to unknowingly violate the boundaries of ethics is a significant proposed change; the authors' response is that this is less necessary for their testbed, which does not contain items or scenarios of concern (e.g., bombs). This response doesn't address the ethical quandaries inherent to agents' language use (see, for example, Tay).

This is overall a good paper with significant flaws, and the question is whether those flaws represent dealbreakers for acceptance. The more substantial ethical recommendations are not within scope for a camera-ready revision, but represent possible failures that are rather more hypothetical; the nearer-term ethical concerns can be discussed by putting more attention into the Ethical Considerations section. The fact that the paper does not accurately represent the current state of licensing is a major flaw, but one that can be addressed for a camera-ready. I strongly encourage the authors to take the time necessary to address these points in any final or subsequent version of this manuscript, as well as other points and clarifications raised by reviewers.
Decision: Accept (Poster)