SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Qiaozi Gao* Govind Thattai* Suhaila Shakiah* Xiaofeng Gao* Shreyas Pansare
Vasu Sharma† Gaurav S. Sukhatme Hangjie Shi Bofei Yang‡ Desheng Zhang
Lucy Hu Karthika Arumugam Shui  $\mathbf{H}\mathbf{u}^{\dagger}$  Matthew Wen Dinakar Guthy
Cadence Chung 王汉斯·Oshan Klandt
Heather Rocker†
Reza Ghanadan†
Dilek Hakkani-Tur† Prem Natarajan†
Amazon Alexa AI
We introduce Alexa Arena, a user-centric simulation platform to facilitate research in building assistive conversational embodied agents. Alexa Arena features multi-room layouts and an abundance of interactable objects. With user-friendly graphics and control mechanisms, the platform supports the development of gamified robotic tasks readily accessible to general human users, allowing high-efficiency data collection and EAI system evaluation. Along with the platform, we introduce a dialog-enabled task completion benchmark with online human evaluations.
A longstanding goal of AI is to develop autonomous robotic agents that can assist humans in day-to-day activities. Experiments with robots are often conducted in controlled environments, limiting their variety and scale of operation. To mitigate this problem, several EAI simulation platforms have been proposed, which support virtual scenes that can be either manually designed, synthetically generated, or captured from real scenes. An embodied agent can freely navigate and interact with objects in these scenes to complete tasks. However, the current EAI platforms suffer from a set of limitations that curtail the ability to build generalizable assistive AI agents.
Facilitating HRI data collection by gamification. A persistent challenge in Human-Robot Interaction (HRI) is collecting human interaction data comprising of natural language instructions along with visual content and actions. Currently available EAI platforms, however, are not designed for humans to effectively interact with the agent, making the data collection process expensive and time consuming [21, 35]. Games have been historically utilized to encourage wide-spread user participation and engagement with services [6, 49, 30]. To ease the data collection effort, we introduce gamification to EAI, which is achieved by presenting tasks or user interactions in the form of games,
*Equal contribution. Correspondence to arena-admins@amazon.com
$^{\dagger}$ Work done while at Amazon Alexa AI
and by introducing scoring mechanisms, achievements and streaks to stimulate and engage users, promoting user participation.
Reasoning based on visual observations. Tasks in current EAI platforms have limited in-class variability: each task type requires the agent to make use of the same objects repeatedly. As a result, agents may resort to over-engineering (over-fitting to specific and simplistic tasks) and memorization to converge to trivial and non-generalizable solutions, or use predefined task decomposition mappings [22]. One way to mitigate this is to design missions that can be completed in multiple ways by interacting with varied objects in the environment with compositional and causally interconnected state changes, forcing the agent to then use common-sense reasoning to understand its current state from visual observations, past actions and associated state changes.
Reconciling offline and online execution scenarios. Real-world operation of assistive agents is characterized by object interactions and collaborative engagement with other agents or humans in the form of visual cues or dialog. In the offline setting, the agent passively follows human instructions. Most currently proposed dialog guided interactive benchmarks and platforms operate in this fashion [11, 2, 26]. Online execution features an agent interacting actively with users that have control, interfacing through natural language in a continuously evolving environment. For example, if the agent veers off its path in the online phase, the human user can regain control of the situation and guide the robot to success, sometimes accomplishing some of the tasks themselves, contributing a learning opportunity for the agent [37]. Such a corrective mechanism that incorporates the elements of real-time user interaction can be implemented limitedly in an offline setting, to the extent of augmenting datasets with interactive elements like dialogs. With our platform, we aim to bridge the fundamental operational gap between the offline and online scenarios by demonstrating the use of offline interaction data to bootstrap an EAI agent, following which runtime data and interactions could be used to quantitatively and qualitatively improve user experience and task execution success rates, leading to a successful demonstration of a human-in-the-loop EAI system that more closely resembles real-world deployment scenarios.
To this end, we propose Alexa Arena, a user-centric EAI platform. For better user experience, the platform design has commonalities with games, with features like task-guiding UI elements and engaging visual effects. The platform boasts many variations of objects and state transitions, on top of which a variety of missions are designed (Figure 1). We show an example use case of Arena on dialog-guided task completion, where the embodied agent can communicate with the user through natural language to finish an indoor mission. To assist in developing models for this task, we release a dataset in which each mission is annotated with an expert demonstration and corresponding human-annotated language instructions and dialogues. To evaluate the dialog-guided agent in real-time, we also release a web-based user interface (UI), where a user can communicate with the agent in text and receive visual observations of the agent from Arena. This paper makes the following contributions:
1. Alexa Arena as a new user-centric EAI platform which focuses on building assistive conversational agents that can assist humans through reasoning and procedural learning.
2. A dataset of over 9K dialog sessions and 46K human annotated instructions. Each data session simulates a human user communicating with an embodied agent to complete a household activity.
3. In addition to offline evaluation of the dialogue-guided embodied agents, we also provide an interactive evaluation protocol. Using the web-based interface, the user can provide instructions to the agent and observe the progress in real-time. We also present results from baseline models for both evaluations.
Embodied AI platforms. In recent years, many platforms have been proposed for embodied agents to perform household activities in indoor virtual environments [28, 10, 47]. Most simulators are designed with performance and realism as the top priorities [38]. Although some simulators have built-in infrastructure for human data collection [12, 46], the user interactions are generally not the focus of their designs. On the contrary, Alexa Arena is a user-centric embodied AI platform with features designed specifically to enhance user experience. There are also several EAI platforms that are inspired by video games [19, 4, 39, 7, 13, 48], but the agent observations are often simplified.
Language-guided navigation and task completion. Most existing work for learning language-guided embodied agents focuses on navigation tasks [3, 36, 24, 9, 31]. For increased task complexity,
Figure 1: The Alexa Arena EAI platform has a variety of object categories, like a set of fantastical objects as engagement enhancers, such as the freeze ray (3), the time machine (6) and the color changer (7). The agent (1) can make use of different tools to change the state of the object. For example, the agent can use either the freeze ray (3) or the fridge (5) to cool an object. Arena also has more interaction actions and state changes compared to existing platforms. For instance, the agent can use the time machine (6) to repair broken objects, or use the color changer (7) to change the color of objects.
[23, 33] enable the agent to follow natural language instructions and complete household activities, where both navigation and object manipulation actions are required. Most recently, [26, 11] propose new datasets and benchmarks for training and evaluating task-oriented embodied agents that can engage in dialogue. However, both works use offline settings where the dialogues are pre-collected from humans or generated with templates. In comparison, the Arena platform enables embodied agents to communicate with human users in a real-time interactive fashion.
Task planning with large language models. Recently, there is a growing trend of using large language models (LLMs) for assisting robot task planning in learning novel activities or completing complex tasks [1, 44]. LLMs have been shown to be good at providing high-level semantic knowledge about the physical world and common human activities [16, 34]. When combined with the sensory perception of the embodied agent, such knowledge can often substantially improve the agent's capability of solving complex tasks in unseen scenarios [18, 17]. The Arena platform provides a good testbed for this line of work. With numerous object types, properties and state changes, along with various environmental causal events, Arena supports the creation of robot tasks that require adequate reasoning capability on common world knowledge.
3 The Arena Platform
In this section, we describe the Alexa Arena Platform. We briefly describe the attributes of the simulator and tools that we are releasing as a part of the platform.
3.1 Objects Properties and States
There are 336 unique objects in Arena. Each object has a set of properties, called affordances, which specify if a certain type of robot-object interaction is possible. For example, the agent can toggle the 3-D printer since it has an object property toggleable. At the same time, the agent cannot pick it up since it does not have the pickupable property. In total, there are 14 object properties, including pickupable, openable, breakable, receptacle, toggleable, powerable, dirtyable, heatable, eatable, chillable, fillable, cookable, decor and infectable. Each object property has a corresponding action and a subsequent object state upon taking that action. For example, break is the corresponding action for breakable, and broken is the corresponding state after the action has been performed. The states of objects will change as a consequence of the action, given that the pre-conditions are met. For example, if powered on (Fig. 1 (2)), the 3-D printer can be used to make toys (Fig. 1 (4)).
Figure 2: The web-based user interface. The user can use the chat box to communicate with the agent in Arena for task completion. Meanwhile, video from Arena is streamed in real time to show the progress. The minimap is displayed on the top right corner of the game UI, showing the room layout and the robot location and orientation. The mission goal and subgoals are displayed to the users in text on the top left corner of the UI.
3.2 Robot Action Space
In general, Arena supports two kinds of actions: 1) user interaction actions for communicating with the user, and 2) robot physical actions to interact with the simulation environment. There are two types of robot physical actions - navigation actions and object interaction actions. For better user experience, all the navigation and interaction actions are animated in a continuous fashion with associated appropriate environmental sounds also being played during the animation.
User interaction. To communicate with the user, the robot can initiate a dialog, the contents of which are displayed on the user interface (Figure 2). The robot can also highlight objects for real-time visual feedback and use it for confirmation or user instruction disambiguation.
Navigation. The goal of Arena is to aid in compositional task learning and reasoning, instead of indoor navigation. To this end, we simplify the navigation in Arena by enabling the robot to directly navigate to a viewpoint in a room by specifying the viewpoint name or the room name, or to an object by specifying the object mask. Meanwhile, if preferred, the robot can also perform step-by-step navigation by a combination of parametrized local primitive actions like MoveFoward and Rotate.
Object interaction. Arena supports 11 actions for object interaction, including Examine, Pickup, Place, Open, Close, Break, Pour, Toggle, Fill, Scan and Clean. Each action is associated with a set of objects in a specific state in which the objects can afford that action to be performed upon them. E.g., toggling can be performed on a Time Machine when it is in a closed state. Interaction actions are accompanied by a change in the associated object's state, which gets updated in the metadata.
3.3 Enhanced User Experience
We incorporate into Arena elements that enhance application usability and improve engagement quality: 1) the platform includes engage-enhancers, such as fantastical objects and a scoring system for each mission; 2) for better visual effects, actions in Arena are animated in a continuous played-out fashion for both manipulation and navigation actions, makes the scenes more natural, user-friendly and engaging; 3) the platform features a unique user interface with several elements to provide users with better task guidance, including minimap and sub-task hints, all of which work in an integrated fashion to improve user-experience.
4 Dialog-guided Task Completion
We now present the Dialog-guided Task Completion benchmark. The benchmark is designed to evaluate dialog-guided agents for indoor object interaction tasks. To support model development, we release a hybrid dataset where ground-truth robot action trajectories are paired with human annotated language. We also set up two evaluation protocols: an offline evaluation on the validation split of the dataset and an online evaluation with human users.
Tasks	Heat & Deliver	Freeze & Deliver	Repair & Deliver	Color & Deliver
Scene #	101	105	110	104
Objects	Bowl, microwave, desk	Mug, freeze ray, table	Floppy disk, time machine, fridge	Apple, color changer, desk
Instructions	Take the bowl	Go to the freeze ray target	Carry floppy disk to time machine	Go to the color changer station
Questions	Where is the bowl?	What does the target look like?	Where is the time machine?	Where is the station?
Answers	On the tan table next to the fan	It is the blue tray attached to the wall	The time machine is in the corner of the countertop	To your left in the southwest corner of the room
Expert demonstrations
Figure 3: Examples in the AID dataset. Each data session corresponds to one language annotation on a mission that needs to be completed in a specific scene. Each language annotation includes a sequence of instructions guiding the robot to complete the task, as well as questions and answers for clarification. Expert demonstrations are provided for each data session.
4.1 Task settings
Each task in the benchmark is specified by the initial states and the goal states of the scene, and a sequence of human instructions. The agent is required to understand the language instructions and interact with the environment through a series of object manipulation and navigation actions to achieve the goal states.
4.2 Arena Interactive Dialogue (AID) Dataset
To enable training and evaluation of the dialog-guided embodied agents, we collect the Arena Interactive Dialogue (AID) dataset via crowd-sourcing. The dataset contains expert action trajectories, human language instructions, and associated questions and answers for missions to simulate dialog. We split the dataset into training and validation folds. There are 2661 tasks in training and 383 in validation, in which each task is annotated by 3 annotators. Each human annotation is considered one session, giving total of 7983 training sessions and 1149 validation sessions. There are in total 12 unique tasks types in the dataset. Examples data points are shown in Figure 3.
Expert demonstrations. The game mission and expert demonstration generation follows a two-step process. In step one, game missions are programmatically generated via sampling from initial environment states and mission goals. The second step generates expert demonstrations using a planner. Along with the game definitions, we also generate PDDL (Planning Domain Definition Language) planning problem definitions for each game mission. A symbolic planner is used to solve the planning problem and the output action sequence is collected as the expert demonstration. One thing to note is that the planner has access to the game metadata, which is not available to the agents during inference time. For tasks that can be completed in different ways (e.g., by using different tools), we pick one unique way for each task to generate expert demonstration. For example, for freeze and deliver, in some missions the fridge is used to cool the object, while in other cases the freeze ray is used.
Human Language Annotation. To collect natural language dialogue for the missions, we design a two-stage data annotation process on Amazon Mechanical Turk (AMT). In both stages, the annotator watches a video containing an expert demonstration for the mission and provides the annotations in free-form text data or answers to multiple choice questions. In the first stage, annotators write instructions to tell a "smart robot" how to accomplish a task. During the process, an annotator first watches the video of the ground-truth robot actions, then writes instructions for each highlighted video segment. After all the instructions are collected, we start the second stage of annotation, where the annotators are asked to raise questions to better complete the task, as if they are controlling the robot to follow the instructions. They also need to subsequently answer their own raised questions. Similar to [11], the question choices in the second stage of annotation are generated using predefined templates. For more details on the human language annotation process, see Appendix.
Figure 4: Model architecture of the vision language model for end-to-end action prediction and visual grounding.
4.3 Offline evaluation
We expect that a good embodied agent should be able to finish the missions efficiently. To achieve this goal, the agent should understand the human instructions and generate a corresponding sequence of actions. Thus, we evaluate the agent using the following metrics:
- Mission success rate (MSR). For each mission, there is a mission success variable  $m$  indicating whether the goal conditions have been met for the mission.  $m = 1$  if all the goal conditions are met. Mission completion rate is calculated by averaging  $m$  across all the missions.
- Average number of robot actions (NRA). To measure the efficiency of task completion, we also record the number of actions taken by the agent to complete each mission. Average number of robot actions is calculated by averaging the number of actions across all the missions.
4.4 Online Human evaluation
To demonstrate the potential of Alexa Arena for real-time user interactions, we design a human evaluation protocol for dialog-guided task completion. Different from the offline evaluation where the agent only passively follows the user instructions, in this online setting, the agent can also engage in multi-turn dialog with the user to communicate its internal states and clarify the mission goals.
**Missions.** On top of the 12 task types presented in the AID dataset, we design and curate 14 more complex longe-horizon missions, each requiring the agent to complete several sub-goals by following the user instructions, aiming to test compositional learning capabilities of the agents. The missions are designed to be gamified to engage the users and the environment also contains hints to guide the users to complete the tasks. The setting encourages dialog, where the human and the robot share the goal to complete the mission. An example of such a task is provided in the Appendix. To test the agent's capability to generalize across tasks, we keep 5 missions as the unseen test set.
Metrics. For online evaluation, we use the MSR as the task performance metric. As a subjective evaluation metric, we collect users' overall satisfaction toward the game on a 5-point likert scale after each interaction.
5.1 Offline evaluation
We use the AID dataset to train two baseline models. The inputs to the models are the agent egocentric view and natural language conversations. The outputs are a series of executable action sequences and associated interactable masks (where needed) to complete the task at hand. For both models, we experiment using only the human instructions, as well as the instructions appended with the questions and answers (QA) as input to simulate human-agent dialog during the mission in offline evaluation.
Neural-Symbolic Approach. Episodic Transformer (ET) is a neural-symbolic model for visual-language task completion [27]. ET uses a multi-modal transformer to predict actions and object pairs based on the visual and language information in the entire episode, with a separate object detector to generate object masks.
Mission Type	NS w/o QA	NS w/ QA	VL w/o QA	VL w/ QA
breakObject	0.00±0.00 (10.93±0.22)	0.00±0.00 (9.90±0.05)	28.88±3.62 (8.90±0.42)	37.41±3.66 (9.07±0.56)
clean&deliver	14.18±1.43 (21.66±0.06)	13.79±0.00 (15.14±0.09)	11.49±2.48 (16.12±0.86)	16.85±3.01 (13.63±0.93)
color&deliver	0.00±0.00 (14.44±1.43)	0.00±0.00 (13.22±0.17)	0.0±0.0 (16.69±0.45)	2.77±3.92 (18.05±0.67)
fill&deliver	14.58±0.00 (20.38±0.27)	20.14±0.98 (17.12±0.08)	11.80±4.28 (15.43±1.27)	23.61±5.47 (13.79±1.86)
freeze&deliver	31.94±1.96 (17.17±0.07)	19.44±1.96 (17.42±0.32)	1.38±1.96 (21.48±1.65)	5.55±5.19 (19.71±2.79)
heat&deliver	5.13±0.00 (14.79±0.36)	5.13±0.00 (18.22±0.45)	23.07±5.53 (19.24±1.97)	28.20±2.09 (16.91±0.57)
insertInDevice	14.69±0.00 (11.53±0.15)	14.88±0.53 (11.45±0.04)	15.63±2.62 (7.54±0.46)	22.03±1.22 (1.22±0.43)
pickup&deliver	9.47±0.00 (11.15±0.10)	11.93±0.29 (12.66±0.08)	21.28±0.92 (8.03±0.36)	26.54±1.67 (7.89±0.40)
pourContainer	15.10±0.40 (12.19±0.04)	14.81±1.07 (12.71±0.12)	25.92±5.04 (11.09±0.90)	31.05±1.06 (10.07±0.29)
repair&deliver	11.11±0.00 (17.19±0.28)	12.96±0.00 (15.56±0.35)	11.72±0.87 (21.69±1.14)	18.51±3.02 (18.39±1.23)
scanObject	41.44±0.00 (4.90±0.00)	41.44±0.00 (4.84±0.00)	41.44±3.20 (3.72±0.19)	57.65±4.59 (3.96±0.36)
toggleDevice	57.14±0.00 (5.09±0.00)	56.19±0.00 (5.08±0.00)	84.12±3.23 (4.13±0.22)	87.62±4.85 (4.49±0.68)
average	18.42±0.00 (11.89±0.01)	18.97±0.00 (11.62±0.03)	27.00±0.39 (9.88±0.29)	33.88±0.71 (9.20±0.28)
Table 1: Experimental results on the validation dataset for the Neural-Symbolic (NS) and Vision-Language (VL) models. For each task, we show mission success rate and average number of robot actions (in the parenthesis). For each metric, we report the mean and standard deviation from three runs.
End-to-end Vision Language Model. We experiment with an end-to-end vision language (VL) model shown in Figure 4. A transformer model encodes the natural language instruction, and the egocentric view is encoded using a ResNet [15], both of which are initialized with pre-trained CLIP weights [29]. The encodings are passed through a joint VL encoder to get a combined vision-language embedding. This is then channelled through two heads: 1) The task planning head is an LSTM module encoding the context of previous predicted outputs, followed by two linear classifiers to predict the (action, object) pairs. 2) The mask prediction head uses the current first party view, the encoded language and the action-object prediction hidden state from the task planning head to predict a single mask referring to the interactable object using the referential image segmentation architecture described in [45]. The above procedure is repeated for a natural language instruction and a series of first party views until an End-of-Sequence (EOS) flag is emitted by a separate binary classifier.
5.1.1 Results and analysis