SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Christopher Rawles* Google Research
Alice Li* Google Research
Daniel Rodriguez Google Research
Oriana Riva Google Research
Timothy Lillicrap  
Google DeepMind
There is a growing interest in device-control systems that can interpret human natural language instructions and execute them on a digital device by directly controlling its user interface. We present a dataset for device-control research, Android in the Wild (AITW), which is orders of magnitude larger than current datasets. The dataset contains human demonstrations of device interactions, including the screens and actions, and corresponding natural language instructions. It consists of 715k episodes spanning 30k unique instructions, four versions of Android (v10-13), and eight device types (Pixel 2 XL to Pixel 6) with varying screen resolutions. It contains multi-step tasks that require semantic understanding of language and visual context. This dataset poses a new challenge: actions available through the user interface must be inferred from their visual appearance, and, instead of simple UI element-based actions, the action space consists of precise gestures (e.g., horizontal scrolls to operate carousel widgets). We organize our dataset to encourage robustness analysis of device-control systems, i.e., how well a system performs in the presence of new task descriptions, new applications, or new platform versions. We develop two agents and report performance across the dataset. The dataset is available at https://github.com/google-research/google-research/tree/master/android_in_the_wild.
Users complete tasks on mobile devices via a sequence of touches and gestures on the screen. Tasks can often be succinctly described using natural language commands, and, in many situations, it is valuable to be able to speak or type commands rather than interacting directly with the device. This has important implications for users who are unable to physically operate a device due to a physical (e.g., visual or motor disabilities) or situational (e.g., driving, cooking, etc.) impairment. It is therefore beneficial to build device-control systems that can interpret natural language instructions and execute them on a device without any manual intervention.
Instead of using application-specific APIs, which are not generally available for any given application or function, these systems directly manipulate user interface (UI) elements on a screen, exactly as a human does [1, 28, 29, 35, 21]. Hence, to work correctly, it is essential for such systems to understand the screen, which usually means detecting position and inferring semantics of its UI elements. Device-control systems must also be able to map high-level commands to execution plans that can be carried out on the device. For example, understanding that the command "open my recent email with Jane" involves opening an email app, potentially tapping the search icon, typing "Jane", etc. Further, to be useful, they must be able to generalize across a variety of task instructions and UIs.
*Equal contribution. Contact: crawles@google.com and lialice@google.com
Figure 1: AITW data pipeline. Raters are given a randomly selected instruction. The raters execute the task by interacting with the device in a natural way. We capture precise gestures in addition to typing and the home and back button interactions (we plot swipes with the arrow pointing where the finger moves to). Hindsight relabeling of high-level episodes is used to generate single-step tasks.
The rapid development of general-purpose large foundation models (LLMs) [8, 6, 13] makes device-control systems more viable. Yet, there is a lack of datasets for training, fine-tuning, and evaluating these systems. Existing datasets [28, 9, 42, 37, 4] are limited in terms of number of human demonstrations and the diversity of task instructions, and they are platform specific (either Android or web). They also assume a tree-based representation of an application UI can be derived from platform-specific UI metadata (e.g., the View Hierarchy for Android and the DOM tree for the web). This assumption simplifies the problem, but limits the resulting systems to work in environments where high-quality UI metadata is available<sup>2</sup>. Finally, some popular datasets (e.g., MiniWoB++ dataset [29] and UIbert [4]) assume task instructions are specified as step-by-step commands referring to specific UI elements appearing on the screen ("Click the button in the dialog box labeled Cancel"), while users may use short commands that describe high-level goals (e.g., "turn on airplane mode") or pose questions (e.g., "Is it going to rain tomorrow?").
To drive research in this field, we release AITW (Figure 1), an Android device-control dataset which is orders of magnitude larger than existing datasets. It consists of 715k episodes spanning 30k unique task instructions collected across hundreds of Android apps and websites. Each episode consists of a goal instruction provided in natural language and a sequence of observation-action pairs describing the execution of the task. Observations consist of screenshots of the application UI. Gesture actions are represented as taps and drags at arbitrary  $\langle x,y\rangle$  coordinates in the screen. Agents trained on this dataset can be evaluated using AndroidEnv [40], an open-source platform for developing and testing Android agents with the Android Emulator<sup>3</sup>.
A key feature of our dataset is the diversity of task instructions and execution paths we collected, aimed to emulate real-world scenarios. We used multiple sources to collect high-level goal instruc
<sup>2</sup>Since most users do not use UI metadata for interactions it tends to be poor quality or missing altogether. On Android, only applications registered as Accessibility tools can access the View Hierarchy [45]. On Windows, in many cases (e.g., Electron apps like Teams), UI trees are not easily accessible. Moreover, screen representations derived from UI metadata can be incomplete. On Android, Websites and Canvas are not captured in the View Hierarchy, and many websites render directly to a Canvas, which does not contain any tree structure.
<sup>3</sup>https://developer.android.com/studio/run/ emulator
Dataset	Platform	# Human demos	# Apps or websites	# Task steps	Observation format	Screen features	Real	High-level instruction
RicoSCA [28]	Android (apps)	0	n/a	1.0	VH, screen	x	x	x
UIBert [4]	Android (apps)	16,660	n/a	1.0	VH, screen	x	✓	x
MiniWoB++ [37, 29]	synthetic web	17,971	100	2.3	DOM, screen	x	x	x
PixelHelp [28]	Android (apps)	187	4	4.2	VH, screen	x	✓	✓
UGIF [42]	Android (apps)	523	12	5.3	VH, screen	x	✓	✓
Mind2Web [14]	web	2,350	137	7.3	DOM, screen	x	✓	✓
MoTIF [9]	Android (apps)	4,707	125	4.5	VH, screen	x	✓	✓
AITW	Android (apps+web)	715,142	357+	6.5	screen	✓	✓	✓
Table 1: Comparison of AITW to existing datasets. We consider platform, format of screen observations, presence of synthetic UIs or synthetic instructions ("Real"), and whether instructions are expressed as goals (high-level). For size comparison, we report the number of human demonstrations, apps/websites, and average task steps. AITW collects observations as screenshots and includes screen features (OCR and icon labels), which can be used to augment them.
tions: humans (both crowdsourced raters and us authors), LLM-generated prompts, and technical documentation (as in PixelHelp [28]). During crowdsourcing, raters were asked to both demonstrate full tasks and annotate sequences of screenshots (hindsight language relabeling [31, 32]), which allowed us to collect both multi-step and single-step task trajectories. We made the execution paths more varied by randomizing the application state, which forced the raters to demonstrate how to navigate to the relevant screens. Finally, we collected demonstrations on four versions of Android (v10-13) and eight device types (Pixel 2 XL to Pixel 6) with varying screen resolutions.
Device-control systems need to work on rapidly evolving software platforms, so an important metric for their success is generalizability to new tasks and applications. We organize our dataset to enable analysis of how trained systems perform in the presence of previously-seen tasks and applications, but also in the presence of new task descriptions, new Android versions, and new applications. Due to the lack of off-the-shelf pixel-based device-control models, to establish new state-of-the-art results on this dataset, we implement two agents: one trained from scratch using behavioural cloning (BC) and a second based on a pre-trained LLM.
We make the following contributions: (i) we collect and release a dataset for device-control research, AITW, which is larger and more varied than existing datasets; (ii) we report performance for two models, which can serve as baselines for future work and (iii) we show how to use the dataset to conduct a generalization analysis.
2.1 Device-control datasets
Table 1 provides a comparison of device-control datasets. Some datasets (top part of the table) target the problem of grounding referring expressions to UI elements on a screen. Every data instance in these datasets includes a screen, a low-level command (e.g., "click the menu button at the top left"), and a UI element corresponding to the command. In the RicoSCA dataset [28], commands are synthetically generated, while in MiniWoB++ [37, 29] sequences of low-level UI commands describe multi-step tasks (e.g., "find and click on the center of the circle, then press submit").
A second group of datasets contain instructions expressed as task goals. Each episode in these datasets is a sequence of action-observation pairs. The observations include screenshots and tree-based representations: View Hierarchy (VH) for Android and Document Object Model (DOM) for web-based applications. For instance, the PixelHelp dataset [28] comprises 187 high-level task goals and step-by-step instructions sourced from Pixel Phone Help pages. The UGIF dataset [42] contains similar queries but extends to multiple languages. The largest dataset to date is MoTIF [9], which consists of 4.7k task demonstrations<sup>4</sup> with an average number of 6.5 steps and 276 unique task instructions. AITW is two orders of magnitude larger than MoTIF. In total, AITW consists of 715,142 episodes, spanning 30,378 unique prompts, with a small subset of the prompts inspired by
This represents the number of "feasible" tasks. We do not consider tasks without a valid demonstration.
Name	Task type	Description	Episodes	Screens	Prompts
GOOGLEAPPS	Multi-step	Tasks with Google apps (Gmail, Photos, Settings, etc.)	625,542	4,903,601	306
INSTALL	App installation and login tasks	25,760	250,058	688
WEBSHOPPING	Web shopping tasks	28,061	365,253	13,473
GENERAL	Misc web/app tasks and Q&A	9,476	85,413	545
SINGLE	Single-step	Mostly shopping tasks from WEBSHOPPING	26,303	85,668	15,366
Total	715,142	5,689,993	30,378
Table 2: Composition of the AITW dataset.
the PixelHelp dataset. Observations are represented by screenshots along with pixel-based screen features.
2.2 UI representation and automation models
Research on device control is mainly focused on two problems: understanding UIs and automating tasks. Existing work on the first problem utilizes self-supervised [21, 4, 5, 27] and supervised methods [10, 30, 11, 46] to train UI understanding models. In some cases, these models are fine-tuned for simple grounding tasks (e.g., referring expression component retrieval [4]), along with widget captioning or question answering tasks [27, 4].
For task automation, Li et al. [28] decompose the problem in two stages: an action phrase-extraction stage to transform step-by-step instructions into actionable phrases, and a grounding stage that executes these instructions. Venkatesh et al. [42] utilize an LLM to parse the instruction before executing "macros" (e.g., tap(), toggle())) during the grounding phase. AppBuddy [38] train an RL agent to interact with on-screen UI elements to achieve tasks. LLMs can also understand and operate UI screens [43]. On the web front, previous studies have developed RL [20, 25, 29, 18], behavioral cloning [24], and LLM-based models [19, 26, 17]. These approaches utilize Document Object Model (DOM) inputs and often evaluate results on a simulated environment, MiniWob++ [37, 29]. Finally, LLMs have shown impressive results leveraging APIs, when they are available, for performing higher-level tasks [36, 33, 34].
3 Android in the Wild (ARTW)
Table 2 shows the composition of AITW in terms of category and type of tasks. Overall, AITW consists of four multi-step datasets, GOOGLAPPS, INSTALL, WEBSHOPPING, and GENERAL, along with a single-step dataset SINGLE.
The dataset is collected in a two-stage pipeline shown in Figure 1. First, we ask the raters to perform end-to-end tasks on emulators. Then the raters apply hindsight language relabeling [31, 32] to the trajectories that were collected in the first stage. We ask the raters to identify and label simple action sequences. We refer to these as single-step tasks.
Our recording system uses AndroidEnv [40] with the Android Emulator. The environment supports 3 action types {TOUCH, LIFT, REPEAT} with an  $(x,y)$  tuple indicating the on-screen position of the action. We record the TOUCH and LIFT actions. In response to an action, the environment returns an RGB screenshot, along with additional metadata such as the opened application. Raters interact with the emulated device using a mouse and keyboard on a desktop computer. Click events are logged as touch events. We provide dedicated buttons for Home, Back and Enter actions along with a field for entering text. We encourage the raters to use the dedicated buttons when necessary, however we require them to use a dedicated input text field for typing; we do not allow them to use the on-screen keyboard. We also ask the raters to indicate when they have completed the task or if they deem the task to be impossible to complete by pressing a button on our data collection UI.
The system captures the raw observations and actions at  $10\mathrm{Hz}$ . Mouse presses and releases are recorded as TOUCH and LIFT, respectively. For touch events, we log the start and end position of the virtual finger's gesture, which we call a "dual-point" gesture. A scroll is represented by a start and end position, and a tap is a special case where the start and end are approximately equal ( $<= 0.04$  Euclidean distance away). Figure 1 contains an example of a tap and horizontal scroll gesture using
this formulation. We found the dual-point gesture abstraction to be a good trade-off between data compression and precision, allowing us to represent arbitrary drags that are needed to operate widgets, including scrolling through a menu and operating carousel widgets. After identifying dual-point gestures, we drop LIFT actions. Button presses and type events are logged as additional actions types. For type events, we log the typed text.
In summary, AITW's actions are described by four fields: type, touch_point, lift_point (only for gesture actions), and typed_text (only for typing actions). The type field can be one of the following: dual-point gesture, type, go_back, go_home, enter, task Complete, or task_impossible.
We post-process RGB screenshots to map them to a set of detected UI elements. Each element has a bounding box and either OCR-detected text or an icon class label (one of the possible 96 icon types detected using IconNet [39]). The OCR outputs describe most of the text on the screen, although certain characters can be misidentified and text blocks are not always grouped as desired. Although this screen representation inferred from pixels is noisy and not as comprehensive as that obtained from UI metadata, we provide these features for convenience and expect developers will replace them with more powerful screen understanding models. We use these features for training and evaluating our models.
3.1 Multi-step task trajectories
We first create high-level task instructions from various sources: (1) the authors, (2) a subset of PixelHelp [28] instructions that were deemed achievable, and (3) an LLM prompted to generate instructions. Next, we randomly assign instructions to raters and they follow them to complete tasks. Every task requires multiple steps to be performed. For example, the task "show my schedule for next week in Google Calendar" could correspond to the following steps: 1) opening Google calendar, 2) selecting "week view", and 3) opening next week. For each episode, we reset the environment to a random starting screen.
We ask the raters to interact with the device in a natural way, to avoid clicking on anything unrelated to the task, and to avoid unnecessary scrolling. To help guide the raters we prompt them with the following "Imagine a friend is asking you to perform the task on their phone..." The raters end a task with a special "status" action: either task Complete or task Impossible. A task is deemed impossible when an invalid or unavailable instruction is given, e.g., "turn on flashlight" on an emulator or "show my starred emails" when the Internet is not available. For instructions that result in verification rather than a state change (e.g., if the prompt is "Turn wifi off" and WiFi is found to be already off), we ask the raters to mark the task as successful.
3.2 Hindsight language relabeling
Single-step task demonstrations cannot be collected in the usual way of giving raters instructions and asking them to solve end-to-end tasks, since they require the relevant preceding steps to be executed. For example, in the task we described above, we cannot ask the raters to demonstrate "go to next week" unless they are already in the week view of the calendar app. Rather than asking raters to manually perform the single steps, we utilize event-selectable hindsight language relabeling [31, 32] to label previously collected trajectories.
To collect single-step demonstrations, we provide the raters observation-action sequences of multi-step task trajectories and ask them to identify and annotate shorter sequences (around two to five frames). We instruct them to label single steps, e.g., "add item to cart", "show the settings", "show me my bookmarks". We ask that they label at least K subsequences  $(\mathrm{K} >= 3$  in our case) per video.
We instruct the raters to avoid the following words: "click", "select", "tap", "touch" or "scroll down/up/left/right", since these can be easily synthetically created, and instead ask them to write descriptive phrases that describe the result of the action (e.g., instead of "tap airplane mode", write the label "disable airplane mode").
3.3 Dataset summary
With reference to Table 2, we describe the 5 sub-categories of AITW.
Figure 2: Statistics for AITW. a) Episode length distribution. b) Episode length distribution by dataset group. c) Frequency of Android apps in the dataset. d) Token analysis including distribution of instruction length and token frequency for GOOGLAPPS and GENERAL.
GOOGLEAPPS contains high-level tasks with some overlap from PixelHelp [28] which involve various Google applications such as Gmail, Calendar, Photos, Settings, etc.
INSTALL contains high-level tasks related to installing and uninstalling apps, app login, and app login support (e.g., "forgot password") for 88 different apps available on the Google Play store.
WEBSHOPPING contains tasks related to shopping on E-commerce websites. Example tasks include searching for an item, adding an item to the cart, viewing the shopping cart, etc.
GENERAL contains miscellaneous tasks (e.g., "play the new Taylor Swift video on YouTube"), mostly centered around question and answering (Q & A) (e.g., "How much does a 2 bedroom apartment rent cost in San Francisco?") and interacting with third-party apps and websites.
SINGLE contains single-step tasks manually annotated using hindsight relabeling, mostly from WEBSHOPPING (e.g., "Close the pop-up then add first item to cart", "clear items from cart"). It also contains a smaller amount of episodes (560) from a variety of Google apps and third-party websites.
In Figure 2, we report statistics about AITW. The episode length distribution (Figure 2a), measured as number of steps required to complete the task, shows that tasks are of moderate length (between 2 and 16 steps for the 5th to 95th percentile, respectively) and that WEBSHOPPING tasks are generally the longest (Figure 2b). Chrome and Google apps are the most commonly used apps (Figure 2c). Overall, the dataset spans 159 Android apps and  $198+$  websites.
(Figure 2d) shows summary statistics of the instructions. Instructions lengths fall between 4 and 24 for the 5th to 95th percentile, respectively, and are not overloaded with technical terms such as "click", "tap", "menu", "button", etc. which is generally the case for low-level UI commands provided in existing datasets [37, 4].
4 Experimental setup
With the ultimate goal of building automation systems that can generalize to new scenarios, we use a standard test split and also design four experimental setups to evaluate Out-of-Distribution (OOD) generalization.
This number is a conservative estimate computed using heuristics.
Table 3: Examples of subject templates.
Instruction	Subject template	Split
open app grab and go to login screen	open {subject1} and	train
open Walmart and go to shopping cart	go to {subject2}	train
search newegg.com on google	search {subject1}	val
search usb-c to usb-a on ebay	on {subject2}	val
add jbl flip 4 to the cart on bestbuy	add {subject1} to the	test
add acer nitro to the cart on target	cart on {subject2}	test
Standard. We randomly split each dataset (the four multi-step datasets and SINGLE) episode wise into a training, validation, and test set  $(80 / 10 / 10\%)$ . Because the datasets different sizes, we evaluate each of them separately, then take the average score across them; we do the same for OOD setups.
Unseen Android version. To evaluate a system's performance on an unseen Android version — which contains unseen graphical components and execution flows — we partition our data as follows: We put episodes collected on Android versions 10, 11, and 12 into the training and validation sets, maintaining a  $90/10\%$  split respectively. Then, we create a separate test set comprising entirely of episodes captured on Android version 13 devices.
Unseen subject and unseen verb. This setup is aimed at evaluating generalization to unseen instructions. Due to the large number of prompts in AITW, it is infeasible to manually group similar tasks together. Simply splitting based on exact match of the raw instructions would be the most straightforward way to automatically assign splits. However, similar instructions with minor changes in language would potentially be seen in both training and testing.
To better differentiate the training and test sets, we develop instruction templates by masking out either verb or subject phrases (examples provided in Table 3). By splitting data based on these templates, we can assess a system's ability to generalize to unseen language patterns, and occasionally to entirely new tasks. For instance, all instructions following the template add {subject1} to the cart on {subject2}" are grouped together, ensuring they are not represented in both training and testing sets. Similarly, verb-based templates such as open the shopping cart" and "view the shopping cart" would be assigned to the same split.
We extract the templates for each instruction, by prompting a few-shot LLM [13]. In total, we extract 6,111 subject templates and 22,122 verb templates. For both types, we randomly assign each template to a train, validation or test split (with  $80/10/10\%$ ). Then for each episode, we determine its template based on its instruction, and map the episode to a split.