SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Unseen domain. This split is designed to test an agent's ability to generalize to unseen apps and websites, which we refer to as domains. For WEBSHOPPING and GENERAL, we perform the split based on the web domain, as inferred from the instructions. For INSTALL tasks, we divide the data based on the app name, but we restrict these tasks to only those that require interaction with the installed app (e.g., performing a 'forgot password' request). Each domain, along with all associated episodes, is randomly assigned to a train/Validation/test split (80/10/10%). We exclude SINGLE, as there are no distinguishable domains across tasks, and GOOGLEAPPS, due to the limited number of distinct apps.
In this section, we report results of two device-control agent models evaluated on AITW. Both models take as input a task instruction, the current screen's pixel-derived features (included in the dataset), and (optionally) a stacked history of screen observations and actions.
5.1 Models
BC. We implement a Transformer-based [41] Behavioural Cloning (BC) agent. The agent's output is in line with the AITW's data format. It outputs an action type and a gesture. The action type can be dual-point gesture, type, go_back, go_home, enter, task_COMPLETE, or task_impossible. The gesture
action includes two spatial points, a touch and a lift position. This approach gives this agent a large and flexible action space, as it is able to predict taps and scrolls at arbitrary locations, rather than at specific UI elements as in existing work [4, 28]. We consider two variants of the agent, depending on whether it takes as input the screen-action history (2 prior steps), BC-history, or not, BC-single. Appendix B.1 provides more implementation details.
LLM. We feed to PaLM 2 [3] a textual description of the screen and ask it to predict an action among the supported actions in AITW. We adopt a previously-proposed LLM-based design for device control [43], where the input screen (represented by an Android VH) is converted to HTML syntax. We use a modified version of their prompt (see Appendix B.2), and convert the OCR and detected icons to HTML. We create a zero-shot (LLM-0) and a 5-shot Chain-of-Thought (CoT) [44] (LLM-history-5-CoT) version, which also contains history on prior actions taken by the agent, as we observed improves model performance. This model takes the same inputs as the BC model, but as in the original implementation [43], it can only click on detected UI elements, rather than at arbitrary locations and scrolling at precise locations. Since AITW was collected by humans performing precise gestures, some of the recorded gestures are not associated with OCR/Icon-detected UI elements, thus being not feasible for the LLM-based model. This could potentially be ameliorated in future versions by outputting a  $<x,y>$  output, rather than tapping specific elements.
5.2 Evaluation methodology and metrics
Online evaluation of device-control systems is hard because the execution environment generally does not provide a reward signal. Human validation of such systems can be leveraged, however watching and judging an agent's behaviour in real-time requires constant attention and is error prone. We propose an offline evaluation method which is cheaper and reproducible at the expense of accuracy.
We devise and release the code for action matching to evaluate an agent's action's alignment with the ground truth. Two actions can match if their action types are equal. For dual-point taps, they are considered equal if they fall within a  $14\%$  screen distance from each other. Alternatively, if the tap actions occur within the same detected bounding box (augmented to  $240\%$  of their total size during action matching) they are considered equal. Finally, two dual-point scrolls are equal if they have the same primary scroll axis (vertical or horizontal).
Using action matching, we compute partial and complete action matching scores (originally proposed by Li et al. [28]). A partial score is defined as the number of correct actions divided by the episode length, and the complete score is defined as a partial match of 1.0.
To validate offline evaluation results, for subsets of the data, we also perform online evaluation. A human marks an episode as failed if any of the agent actions are incorrect, and correct when the agent performs a correct action on every step and achieves the expected goal. Human validation scores typically outperform complete action matching scores due to the multiple valid action alternatives one can take to complete a task. For instance, pressing the navigation bar's back button is functionally similar to using an app-specific back button. As action matching relies on distance-based measures, these actions are deemed distinct.
5.3 Results
We evaluate the four agents on the five AITW splits described in §4. For the BC agent, we train and test using all the data. For the LLM agent, due to the high computational overhead, we test on a random sample of 288 episodes for each split. Table 4 reports the average partial matching scores.
The BC agent performs the best across all splits. It performs reasonably well on the OOD tasks, particularly on the subject and verb template splits, indicating the model is generalizing to unseen language instructions and tasks. The LLM-based model only sees a small amount (only those k-shot that are in the prompt) of the training distribution for the OOD experiments. Making use of fine-tuning for future experiments would allow us to leverage more of the training data.
The performance of the LLM-based models suffers due to its element-based action space. For the standard test set, for example,  $33\%$  of the episodes have some non-element tap actions (i.e., only  $< x, y>$  location), which are infeasible for this modelling approach. Across the feasible actions, LLM-hist-5-CoT has a partial match score of  $58\%$ .
Model	Standard	Out-of-domain generalization
Version	Subject	Verb	Domain
BC-single	68.7	59.2	64.2	66.4	52.2
BC-history	73.1	63.2	68.5	70.4	59.7
LLM-0 [43]	30.9 [25.6, 36.6]	31.6 [26.3, 37.3]	33.7 [28.2, 39.5]	32.6 [27.3, 38.4]	25.3 [20.4, 30.8]
LLM-hist-5-CoT	39.6 [33.9, 45.5]	29.5 [24.3, 35.1]	44.4 [38.6, 50.4]	41.7 [35.9, 47.6]	35.8 [30.2, 41.6]
Table 4: Partial match scores across standard and OOD generalization splits. For the LLM agent, the estimated score and binomial proportion  $95\%$  confidence interval are shown. BC evaluation is on the entire test sets; confidence intervals are  $< 0.1\%$  and are excluded for brevity.
Figure 3: True complete match (estimated using human evaluation), and partial and complete match (both estimated using automated evaluation) for BC-history. True complete match is based on a subset of episodes;  $95\%$  confidence bounds are reported. Partial match is correlated with true complete match, while the complete match heuristic is a lower bound score.
We perform human evaluation for BC-history on a small subset from GOOGLAPPS (on average 86.5 episodes from each split). We use this dataset portion because it has the largest training set, but we exclude the domain split due to the limited number of apps. As shown in Figure 3, we find that action matching is a reasonable approximation of true success rates.
As expected, the agent performs the best on the standard test split. Compared to what is observed across all dataset portions (Table 4) its performance on the standard set is higher, but on subject and verb OOD splits is lower. This is due to the nature of GOOGLAPPS data (see Table 2) where the tasks are rather distinct (few unique prompts) which makes the verb and subject generalization hard, but at the same time every prompt has many demonstrations, which makes the standard test easier.
Although the automated complete match is low, we note the agent is correct for the majority of steps as indicated by the partial match scores  $>0.5$ . We confirmed this was the case by visual inspection. The agent typically performs many of the initial steps correct, but it is more error prone farther in the trajectory.
In summary, across the four splits, partial match tends to be correlated with true complete match. It is a reliable approximation especially if the number of steps in a task is small. Automated complete metrics represent a lower bound score of the true value.
User Demographics Distribution. The raters are not a representative sample of the entire world population. The screens they visit, containing dynamic content from the Internet, are not representative of the rich variety of content and languages of the world. Similarly, the dataset prompts are exclusively in English, although they could potentially be translated and evaluated using multilingual models.
Rater device interaction. Raters use a mouse and keyboard rather than the native touch-based interface. This may result in somewhat different user patterns.
Form factor. The dataset is derived from mobile phone user interactions. The dataset could be augmented with more form factors, such as tablets, to increase generalization.
UI Drift/Evolution. Our dataset includes an unseen domain split, containing new and unseen UIs, but it may not fully represent the continuous evolution of a given app or website's UI. This dynamic change is an essential aspect of real-world interfaces but is a complex phenomenon to capture comprehensively. However, we do capture some of this drift through the unseen Android version split, reflecting changes in Google apps' UI over various Android versions.
6.2 Ethical considerations
Privacy. The raters were instructed not to enter any Personal Identifiable Information (PII) during collection. The dataset does not contain any interactions from real users.
Malicious use. Malicious actors could use the dataset for undesired purposes such as overriding anti-fraud mechanisms like CAPTCHAs. Malicious actors could also manipulate prompts and/or screen representations of deployed models to achieve undesirable goals.
7 Future Work
Multimodal modeling. The LLM-based model, adapted from prior work [43], is not as performant as the bespoke BC model. This model consumes a text-based screen representation and cannot output a  $< x, y >$  coordinate-based output. A multimodal foundation model [2, 12] that consumes raw pixels and outputs gestures at arbitrary points would be a natural next model type to investigate. Furthermore, any foundation models may benefit from fine-tuning on the AITW training sets.
Multiple ways to achieve a task. There are often multiple ways to achieve a task. Future evaluation methods could be more "lenient" and not penalize correct agent actions that do not match human demonstrations. Furthermore, constraining agents to achieve goals in "optimal" ways, however that may be defined, may increase user satisfaction with trained models.
Mobile device control via natural language commands has broad application. It requires translating high-level instructions into execution plans that operate the device interface as a human would. Recent advancements in general-purpose large foundation models have opened doors for creating such device-control systems, however there remains a substantial void due to the dearth of large, comprehensive datasets essential for training and evaluating these systems.
Addressing these gaps, we present AITW, which is significantly larger and more diverse than existing device-control datasets. AITW consists of 715k episodes across more than 350 Android applications and websites, and a variety of task instructions and execution paths, a realistic representation of real-world system interactions.
Through dataset structure, we provide experimental setups for evaluation under varying conditions, including novel tasks and language, Android versions, and applications and websites. We trained and ran models on the data and demonstrated how to evaluate model performance under novel conditions. We hope AITW will spur research to create more powerful device automation models.
The authors thank Gabriel Taubman, James Stout, Gregory Wayne, and Max Lin for insightful discussions throughout. Thanks to Elisabeth Chauncey for help with dataset release. Thank you to JD Chen for helpful feedback on early manuscript versions. Daniel Toyama, Philippe Hamel, and Anita Gergely provided essential Android environment assistance. We also thank our raters for collecting our data.
[1] Adept. ACT-1: Transformer for Actions, 2022. https://www.adept.ai/act.
[2] J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, R. Ring, E. Rutherford, S. Cabi, T. Han, Z. Gong, S. Samangooei, M. Monteiro,
[43] B. Wang, G. Li, and Y. Li. Enabling conversational interaction with mobile ui using large language models. In Proc. of the 2023 CHI Conference on Human Factors in Computing Systems, CHI '23. Association for Computing Machinery, 2023.
[44] J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, and D. Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023.
[45] XDA. Google is trying to limit what apps can use an Accessibility Service (again), 2021. https://www.xda-developers.com/google-trying-limit-apps-accessibility-service/.
[46] X. Zhang, L. de Greef, A. Swearngin, S. White, K. Murray, L. Yu, Q. Shan, J. Nichols, J. Wu, C. Fleizach, A. Everitt, and J. P. Bigham. Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels. In Proc. of the 2021 CHI Conference on Human Factors in Computing Systems, CHI '21, 2021.
Appendix A Dataset collection
A.1 Crowdsourcing
This work was carried out by participants who are paid contractors. Those contractors received a standard contracted wage, which complies with living wage laws in their country of employment. Due to global privacy concerns, we cannot include more details about our participants, e.g., estimated hourly wage or total amount spent on compensation.
We provided raters with a detailed instructional document and a video tutorial, followed by having them perform test demonstrations using our system. For multi-step task trajectories, we ensured quality and diversity through manual inspection of a subset of demonstrations. Tasks were marked as complete with the task Complete action once a rater completed an assignment, including cases where the task was already completed. In contrast, the task Impossible action was used to indicate infeasible tasks, such as turning on a flashlight in an emulator.
For hindsight-language relabeling, we conducted manual reviews of a sample of labeled trajectories due to the nuanced nature of natural language use. The aim was to encourage the creation of descriptive, unambiguous labels and discourage the use of oversimplified technical terms or vague language in order to collect clear and useful task descriptions that cannot be captured using automatic heuristics.
A.2 Prompt generation
We use the following prompt to extract the subject templates and a similar prompt for the verb templates:
Identify subject variables in commands.   
phrase  $=$  ["show me popular videos on youtube", "whats the weather?", "go toESPN.com", "click the top result", "open calendar and show me the fourth week of next month", <INPUT_INSTRUCTIONS>]   
result  $=$  ["show me {subject1} on {subject2}", "whats {subject1}?", "go to {subject1}", "click {subject1}", "open {subject1} and show me {subject2} of {subject3}",
A.3 Examples
Example of episodes from AITW are show in Figures 4, 5, and 6.
Appendix B Experiment details
B.1 Behavioral Cloning
The Behavioral Cloning (BC), shown in Figure 7, is a Transformer-based architecture [41] that takes a task instruction, the current screen, and a stacked history of screen observations and actions as input. The model is conditioned on BERT [15] embeddings of the natural language instruction. For the screen input the model embeds each detected OCR and detected icon to a vector of size 512 using the following procedure. We embed the text using a pre-trained BERT model taking the output from the CLS token, which is then linearly projected from size 732 to 512. For the icons, we learn an embedding from the ID, which we add element-wise to the BERT embedding. Following similar approaches [28, 16], we add to this the spatial information by learning four embeddings for each of the bounding box points, which are binned into 200 and 96 elements vertically and horizontally. For the screen history (excluding the current screen), we embed the  $<x,y>$  positions of the touch and lift actions, which are added to the element encoding, using a dummy value for non-gesture actions. We found that including action history improves performance.
turn on bluetooth scan
Figure 4: Example episode from the dataset.
What's the news in Argentina?
Figure 5: Example episode from the dataset.
clear history in the chrome app
Figure 6: Example episode from the dataset.
Screen embeddings
Figure 7: Architecture diagram of the BC agent.
Table 5: Partial match scores across generalization splits and datasets for BC-history.
Dataset	Standard	Version	Subject	Verb	Domain
GOOGLEAPPS	75.7	63.4	48.4	57.4	n/a
GENERAL	63.7	52.5	65.4	64.1	n/a
WEBSHOPPING	68.5	56.7	69.5	68.4	49.6
INSTALL	77.5	66.5	76.4	77.7	66.9
SINGLE	80.3	77.0	82.8	84.6	62.6
For gesture actions, the agent outputs a dual-point output. We found such a formulation useful for interacting with many common widgets (e.g., carousel widgets, switching months in a calendar, controlling sliders), which require precise scrolling.
We train the agent using the standard cross-entropy loss using a 2x2 slice of a V2 Tensor Processing Unit (TPU). The agent is implemented using Acme [23], Haiku [22], and JAX [7]. The Transformer has 4 layers, a dropout rate of 0.1, and we train use the AdamW optimizer with a learning rate of 0.0001, and a batch size of 128.
For evaluation we train and perform a hyperparameter search via grid search on the validation set. We choose the best performing model and run it on the test set for the final numbers.
Table 5 reports a breakdown of the performance of the BC-history agent (our best performing agent) across the different dataset splits and portions.
B.2 LLM
We use the following prompt for LLM-0:
Given a mobile screen and a question, provide the action based on the screen information.   
Available Actions: {"action_type": "click", "idx": <element_idx>} {"action_type": "type", "text": <text>} {"action_type": "navigate_home"} {"action_type": "navigate_back"} {"action_type": "scroll", "direction": "up"} {"action_type": "scroll", "direction": "down"} {"action_type": "scroll", "direction": "left"} {"action_type": "scroll", "direction": "right"}   
Screen: <SCREEN Representation> Instruction: <GROUNDING_GOAL> Answer:
We use the following prompt for LLM-hist-5-CoT:
Given a mobile screen and a question, provide the action based on the screen information.   
Available Actions: {"action_type": "click", "idx": <element_idx>} {"action_type": "type", "text": <text>} {"action_type": "navigate_home"} {"action_type": "navigate_back"} {"action_type": "scroll", "direction": "up"} {"action_type": "scroll", "direction": "down"} {"action_type": "scroll", "direction": "left"} {"action_type": "scroll", "direction": "right"}
Previous Actions:
{"stepidx":0,"action_description":"press [HOME key]"}\{ "stepidx":2, "action_description": "click [Google Icon]"} {"stepidx":3, "action_description": "click [search for hotels]"}
Screen:
<img id=0 class="IconGoogle" alt="Google Icon"></img>  
<img id=1 class="IconX" alt="Close Icon"></img>  
<p id=2 class="text" alt="search for hotels"> search for hotels </p>  
<p id=3 class="text" alt="in"> in </p>  
<p id=4 class="text" alt="mexico city mexico"> mexico city mexico </p>  
<img id=5 class="IconMagnifyingGlass" alt="Search Icon"></img>  
<p id=6 class="text" alt="Share"> Share </p>  
<p id=7 class="text" alt="Select allI"> Select allI </p>  
<p id=8 class="text" alt="Cut"> Cut </p>  
<p id=9 class="text" alt="Copy"> Copy </p>  
<p id=10 class="text" alt="hotel in mex"> hotel in mex </p>  
<img id=11 class="IconMagnifyingGlass" alt="Search Icon"></img>  
<p id=12 class="text" alt="best hotel"> best hotel </p>  
<p id=13 class="text" alt="mexico city"> mexico city </p>  
<p id=14 class="text" alt="in"> in </p>  
<img id=15 class="IconMagnifyingGlass" alt="Search Icon"></img>  
<p id=16 class="text" alt="K"> K </p>  
<p id=17 class="text" alt="hotel ciudad"> hotel ciudad </p>  
<p id=18 class="text" alt="de mexico"> de mexico </p>  
<p id=19 class="text" alt="gran"> gran </p>  
<img id=20 class="IconVBackward" alt="Left Icon"></img>  
<img id=21 class="IconNavBarCircle" alt="Home Icon"></img>  
<img id=22 class="IconNavBarRect" alt="Overview Icon"></img>
Instruction: What time is it in Berlin?
Answer: Let's think step by step. I see unrelated search results in the Google app, I must clear the search bar, so the action is {"action_type": "click", "idx": 1}
Previous Actions:
stepidx: 0
action_description: click [DISMISS]
Screen:
<p id=0 class="text" alt="Update your"> Update your </p>
<p id=1 class="text" alt="Gmail app"> Gmail app </p>
<p id=2 class="text" alt="attach files from"> attach files from </p>
<p id=3 class="text" alt="To"> To </p>
<p id=4 class="text" alt="download the"> download the </p>
<p id=5 class="text" alt="Drive,"> Drive, </p>
<p id=6 class="text" alt="latest"> latest </p>
<p id=7 class="text" alt="version"> version </p>
<p id=8 class="text" alt="of"> of </p>
<p id=9 class="text" alt="Gmail"> Gmail </p>
<p id=10 class="text" alt="UPDATE"> UPDATE </p>
<p id=11 class="text" alt="DISMISS"> DISMISS </p>
<p id=12 class="text" alt="Got"> Got </p>
<p id=13 class="text" alt="it"> it </p>
<img id=14 class="IconVBackward" alt="Left Icon"> </img>
Instruction: see creations saved in the google photos
Previous Actions:
Answer: Let's think step by step. I see a popup, I need to open Google Photos, so the action is {"action_type": "click", "idx": 11}
Screen:
<p id=0 class="text" alt="M"> M </p>
<p id=1 class="text" alt="New in Gmail"> New in Gmail </p>
<p id=2 class="text" alt="All the features you"> All the features you </p>
<p id=3 class="text" alt="love with"> love with </p>
<p id=4 class="text" alt="a fresh"> a fresh </p>
<p id=5 class="text" alt="look"> look </p>
<p id=6 class="text" alt="new"> new </p>
<p id=7 class="text" alt="GOT IT"> GOT IT </p>
Instruction: open app "Google Play services"  
Answer: Let's think step by step. I see the GMail app, I need to open the app drawer, so the action is {"action_type": "navigate_home"}
Previous Actions:
Screen:  
<p id=0 class="text" alt="Tuesday, Aug"> Tuesday, Aug </p>  
<p id=1 class="text" alt="9"> 9 </p>  
<img id=2 class="IconChat" alt="Chat Icon"> </img>  
<img id=3 class="IconGoogle" alt="Google Icon"> </img>
Instruction: open app "Messenger Lite" (install if not already installed)  
Answer: Let's think step by step. I see the home screen, I need to open the app drawer, I should swipe up, so the action is {"action_type": "scroll", "direction": "down"}
Previous Actions: {"stepidx": 0, "action_description": "scroll down"}