AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents

## Abstract

Abstract Autonomous agents that execute human tasks by controlling computers can enhance human productivity and application accessibility. However, progress in this field will be driven by realistic and reproducible benchmarks. We present AndroidWorld , a fully functional Android environment that provides reward signals for 116 programmatic tasks across 20 real-world Android apps. Unlike existing interactive environments, which provide a static test set, AndroidWorld dynamically constructs tasks that are parameterized and expressed in natural language in unlimited ways, thus enabling testing on a much larger and more realistic suite of tasks. To ensure reproducibility, each task includes dedicated initialization, success-checking, and tear-down logic, which modifies and inspects the device’s system state. We experiment with baseline agents to test AndroidWorld and provide initial results on the benchmark. Our best agent can complete 30.6% of AndroidWorld ’s tasks, leaving ample room for future work. Furthermore, we adapt a popular desktop web agent to work on Android, which we find to be less effective on mobile, suggesting future research is needed to achieve universal, cross-platform agents. Finally, we also conduct a robustness analysis, showing that task variations can significantly affect agent performance, demonstrating that without such testing, agent performance metrics may not fully reflect practical challenges. AndroidWorld and the experiments in this paper are available at https://github.com/google-research/android_world .

## 1 Introduction

Autonomous agents that interpret natural language instructions and operate computing devices can provide enormous value to users by automating repetitive tasks, augmenting human intelligence, and accomplishing complex workflows. However, a key research challenge remains the realistic evaluation of these agents in real-world settings. Despite growing enthusiasm for building autonomous agents [, , , , , , , , , , ], most existing approaches for evaluation compare an agent’s actions at each step to a previously collected human demonstration [, , , , , , , ]. Measuring performance in this way can be misleading because when performing tasks online in real environments agents can take multiple paths to solve tasks, environments may behave non-deterministically, and agents can dynamically learn from mistakes to correct their actions [, , , ]. For this reason, online evaluation of agents in realistic environments able to reward task outcome provides a gold standard for evaluation. While there is an emerging body of work to address this need across different environments [, , , , , ], there is no comprehensive solution for mobile platforms, such as Android, which are used by billions of users and therefore represent environments in which automation agents may be very productively employed. We introduce AndroidWorld to address this.

At its core, AndroidWorld offers a reliable means of obtaining reward signals for tasks performed by agents in realistic mobile environments. Reward signals are quantitative metrics that indicate functional correctness of a task, i.e. is the stated goal achieved? For example, for the task “Send a text message to Jane confirming I’ll be there,” a positive reward indicates that the relevant message has been sent. Unlike simulated environments [, ] or games [, , , , , ], real-world apps and websites do not inherently offer explicit reward signals. While human [, , , ] or LLM-based [, , , , , , ] judges can be employed to reward the outcome of a task, these approaches scale poorly or are not fully reliable, respectively. Alternatively, environments for autonomous agents which provide automated ground-truth rewards for complex workflows have been developed  [, , , , ]. We find two problems with these environments. First, they are constrained to desktop computing environments, overlooking the mobile domain, which is of paramount importance given the ubiquity and diversity of mobile devices in the real world. Secondly, they are limited in their real-world diversity and scale. Crucially, unlike in real-world scenarios where conditions and task inputs vary widely, these environments support only static test specifications, meaning that when task parameters deviate, the reward signal is likely to break.

Figure: Figure 1: AndroidWorld is an environment for building and testing autonomous agents.
Refer to caption: x1.png

We seek to develop a comprehensive benchmark that addresses the limitations of the existing approaches above for evaluating automation agents in mobile environments. AndroidWorld does this by spanning 20 Android apps on a total of 116 programmatic tasks to provide ground truth-rewards. Unlike existing test environments (with MiniWoB++ [] being a notable exception), each task in AndroidWorld is dynamically instantiated using randomly-generated parameters, challenging agents with millions of unique task goals and conditions. While MiniWob++ consists of simple, synthetic websites, AndroidWorld leverages actual Android applications. A main challenge that AndroidWorld must address is how to ensure that reward signals are durable when using real-world applications and varying task parameters dynamically. AndroidWorld’s key insight is to leverage the extensive and consistent state management capabilities of the Android operating system, using the same mechanisms that the apps themselves utilize to store and update data.

In addition to providing a comprehensive benchmark, AndroidWorld is lightweight, requiring only 2 GB of memory and 8 GB of disk space, and is designed with convenience in mind. It connects agents to the Android OS by leveraging the Python library AndroidEnv [] to connect to the freely available Android Emulator.(^1^11The Android Emulator is packaged as part of Android Studio, which can be downloaded from https://developer.android.com/studio) In addition to the 116 Android tasks, we extend AndroidWorld with web tasks by integrating the MiniWoB++ [, ] benchmark into it.

To demonstrate AndroidWorld’s usefulness as a benchmark, we build and release a multi-modal agent, *M3A* (Multimodal Autonomous Agent for Android), and establish state-of-the-art results on AndroidWorld. We analyze M3A’s performance using both multimodal and text-only input, and we observe that while multimodal perception can improve performance in some cases, it generally does not outperform the text-only approach. On AndroidWorld, M3A achieves a 30.6% success rate, which surpasses that of a web agent adapted for Android but remains significantly lower than the human success rate of 80.0%. In pursuit of building robust UI control agents, our study includes comprehensive tests under varied real-world conditions, demonstrating significant performance variations primarily driven by changes in intent parameters.

Overall, we make the following contributions: (i) the creation of a new, highly diverse and realistic mobile UI control agent environment; (ii) establishment of benchmark performance with a state-of-the-art multimodal agent, and (iii) a careful analysis demonstrating the need to evaluate agents across variable task parameters and conditions due to the inherent stochasticity in both models and environments.

## 2 Related Work

Table 1 compares existing evaluation environments for autonomous UI agents.

### 2.1 Interactive evaluation environments

Effective evaluation of autonomous agents requires benchmarks that mimic real-world scenarios, but also interactive environments that provide reward signals upon successful task completion [, , , , ]. Many existing benchmarking environments target web browsing. MiniWoB++ [, ] consists of small, synthetic HTML pages with parameterizable tasks which allow for unlimited task variability.
WebShop [] provides a simulated e-commerce environment, whereas WebArena [] and VisualWebArena [] consist of simulated websites across up to six domains. WorkArena [] consists of 29 tasks for enterprise software. GAIA [] is a static dataset that tests an agent’s ability to interact with live web environments. MMInA [] is a multihop and multimodal benchmark designed to evaluate agents for compositional Internet tasks.

Towards building generalist agents that control operating systems (OSes), OSWorld [] and WindowsAgentArena [] provide a test suite of tasks for desktop computer use cases and custom execution-based evaluation scripts across across 9 and 11 apps, respectively. In the mobile domain, existing benchmarks are limited and do not capture the diversity of real-world mobile interactions, containing low-complexity tasks or on a limited number of applications. B-MoCA’s [] evaluation is based on 6 simple tasks (e.g., ”Call 911”, ”turn on airplane mode”) across 4 apps(^2^22Based on what reported in the Experiments Section of the B-MoCA manuscript as of October 1^st, 2024.), validated using regular expressions. Mobile-Env [] offers task reproducibility limited to 13 task templates for a single app (WikiHow).

While AndroidWorld shares the mobile platform focus of B-MoCA and Mobile-Env, it is more comparable to OSWorld (and WindowsAgentArena, which builds on top of OSWorld) in terms of task complexity and the diversity of interactions it supports. AndroidWorld enhances OSWorld’s approach by dynamically constructing the start states of an agent’s run and varying the task parameters in unlimited ways, thus allowing for a new type of evaluation under varying real-world conditions.

Other studies leverage human evaluation [, , ] for tasks where automatic evaluation is not available. Lastly, emerging research [, , ] explores the potential of multimodal models to generalize agent evaluations to new settings, though this area requires further research to achieve accuracy comparable to manually-coded rewards.

AndroidEnv [] provides a mechanism to manage communication with the Android emulator, similar to Playwright and Selenium for web environments. While AndroidWorld leverages this functionality, it diverges in its reward system. AndroidEnv’s approach requires modifying application source code and implementing task-specific logging statements, making it well-suited for gaming environments with easily verifiable success criteria. In contrast, AndroidWorld implements a non-invasive reward mechanism, allowing it to create a benchmark suite for apps where source code access is unavailable and to reuse validation components across different apps. This approach enables AndroidWorld to cover a broader range of real-world mobile tasks.

### 2.2 Static datasets for UI automation

Datasets derived from human interactions provide proxy metrics that correlate with real-world agent performance [, , , ]. On mobile platforms, AitW [], AndroidControl [], PixelHelp [], AndroidArena [], LlamaTouch [], UGIF [], and MoTIF [] consist of demonstrations across Android apps and mobile websites, with screens often represented via accessibility trees. In contrast, desktop web environments typically utilize the DOM for representing website content, with Mind2Web [], OmniAct [] and others, across various desktop websites. Mobile-based datasets frequently involve more complex actions, such as scrolling, which are not as useful in DOM-based desktop interactions where the entire action space is readily accessible. Additionally, API-centric datasets like API-Bank [], ToolTalk [], and ToolBench [] assess agents’ capabilities to manipulate computer systems via APIs.

### 2.3 Interactive agents

Prior to today’s foundation models, traditional approaches to developing user interface-operating agents primarily used reinforcement learning and behavioral cloning to simulate interactions like mouse clicks and keyboard typing [, , , , ]. More recent work tends to leverage off-the-shelf foundational models [, , ] with in-context learning (ICL) and fine-tuning applied to mobile [, , , , , , ], desktop web [, , , , , ], and desktop OS [, , ]. Recent work explores agents that reflect on system state [, , ] by leveraging exploration, self-evaluation, and retry-capabilities enabling continual learning and adaptation [, , , , , ].

**Table 1: Comparison of different datasets and environments for benchmarking computer agents.**
|  | Env? | # of apps | # task | Avg # task | Reward | Platform |
| --- | --- | --- | --- | --- | --- | --- |
|  |  | or websites | templates | instances | method |  |
| GAIA | ✗ | n/a | 466 | 1 | text-match | None |
| Mind2Web | ✗ | 137 | 2350 | 1 | None | Desktop Web |
| WebLINX | ✗ | 155 | 2337 | 1 | None | Desktop Web |
| WebVoyager | ✗ | 15 | 643 | 1 | LLM judge | Desktop Web |
| PixelHelp | ✗ | 4 | 187 | 1 | None | Android |
| MetaGUI | ✗ | 6 | 1125 | 1 | None | Android |
| MoTiF | ✗ | 125 | 4707 | 1 | None | Android (Apps+Web) |
| AitW | ✗ | 357+ | 30378 | 1 | None | Android (Apps+Web) |
| AndroidControl | ✗ | 833 | 15283 | 1 | None | Android (Apps+Web) |
| OmniAct | ✗ | 60+ | 9802 | 1 | None | Desktop (Apps+Web) |
| AndroidArena | ✗ | 13 | 221 | 1 | Action match/LLM | Android (Apps+Web) |
| LLamaTouch | ✗ | 57 | 496 | 1 | Screen match | Android (Apps+Web) |
| MiniWoB++ | ✓ | 1 | 114 | $\infty$ | HTML/JS state | Web (synthetic) |
| WebShop | ✓ | 1 | 12k | 1 | product attrs match | Desktop Web |
| WebArena | ✓ | 6 | 241 | 3.3 | url/text-match | Desktop Web |
| VisualWebArena | ✓ | 4 | 314 | 2.9 | url/text/image-match | Desktop Web |
| WorkArena | ✓ | 1 | 29 | 622.4 | cloud state | Desktop Web |
| Mobile-Env | ✓ | 1 | 13 | 11.5 | regex | Android (Apps) |
| B-MoCA | ✓ | 4 | 6 | 1.9 | regex | Android (Apps+Web) |
| MMInA | ✓ | 14 | 1050 | 1 | text-match | Desktop web |
| OSWorld | ✓ | 9 | 369 | 1 | device/cloud state | Desktop (Apps+Web) |
| WindowsAgentArena | ✓ | 11 | 154 | 1 | device state | Desktop (Apps+Web) |
| AndroidWorld | ✓ | 20 | 116 | $\infty$ | device state | Android (Apps+Web) |

## 3 AndroidWorld

### 3.1 Android for autonomous agents

Android is an ideal environment for developing autonomous agents. It is the most widely-used OS globally(^3^33[https://gs.statcounter.com/os-market-share](https://gs.statcounter.com/os-market-share)) and is highly flexible for research, while providing an open world of the Web(^4^44Mobile is the most popular platform for accessing the web; [https://gs.statcounter.com/platform-market-share/desktop-mobile/worldwide/](https://gs.statcounter.com/platform-market-share/desktop-mobile/worldwide/)) and over 2M apps for agents to operate in. Using emulation, an Android environment is easy to deploy, does not require specialized hardware, and can be run on a laptop. Android Virtual Devices or emulator images are well suited for research as they are self-contained, easy to distribute, and configurable.

Compared to desktop environments, mobile environments, like Android, pose unique research challenges for computer control agents. On one hand, mobile UIs tend to be simpler than their desktop counterparts because of their smaller screen size. On the other hand, the action space on mobile devices is more complicated and more actions can be required to complete tasks. Precise gestures are needed to fully operate the UI, such as when navigating a carousel widget, long-pressing on a widget, or performing multi-finger gestures to zoom in. Since it is an OS, Android is a fully open environment compared to web-browser-only environments. Android’s flexibility is also reflected in its action space; in addition to UI actions (click, scroll, type, etc.), Android provides function-calling APIs, such as sending a text message, for example, which allow computer control agents to utilize a broader action space.

### 3.2 The observation and action space

AndroidWorld provides an interface for agents to receive observations and execute actions on Android. It uses AndroidEnv [] and the Android Device Bridge to facilitate interaction between Android and the agent. The observation space consists of a full-resolution screenshot and a UI tree representation developed for accessibility purposes. The action space is similar to that which humans use, consisting of gestures (i.e., tapping, long-press, and swiping), typing, and navigation buttons (i.e., go home and go back). In addition to these naturalistic actions, AndroidWorld exposes a limited set of function calling APIs, such as send_text_message, to help agents accomplish goals. Appendix B provides more details on the observation format and action space.

### 3.3 Reproducible and parameterized tasks

Figure: (a)
Refer to caption: extracted/5947147/difficultycounts.png

AndroidWorld consists of a suite of 116 tasks, spread across 20 diverse applications (see Appendix C for more details). These tasks simulate practical, everyday activities, including note-taking, scheduling appointments, communicating through messaging, and interacting with system utilities. The suite consists of open-source apps and built-in Android system apps, such as Settings and Contacts. As rated by humans, the tasks vary in difficulty, duration, and categories (Figure 2).

To achieve a high degree of reproducibility in real-world scenarios, AndroidWorld precisely controls the OS and app states in several ways. The Android OS is fixed, consisting of a Pixel 6 emulator running Android 13 with a fixed time on the date October 15th, 2023. All applications in AndroidWorld are fully-functional and consists of both open-source apps and OS-level apps included with Android. For the open-source apps, AndroidWorld maintains a constant environment by installing a fixed version of each app, acquired from F-Droid.(^5^55[https://f-droid.org/](https://f-droid.org/))
OS-level apps’ versions are determined by the Android OS, which is also fixed. To maintain a reproducible environment, AndroidWorld utilizes apps that do not require login/authentication and can store their application data on device.

In addition to managing the states of apps and operating systems, AndroidWorld precisely defines and controls the state during task execution. Each task has its own unique setup, reward determination logic, and teardown procedures (see Appendix C.2 and C.3 for more details), ensuring a fully reproducible suite of tasks.

Automatic task parameterization is a critical mechanism, unique to AndroidWorld, to evaluate agents on a much larger and more realistic suite of tasks than current benchmarks support. Achieving this requires significantly more effort than randomly generating new task parameters because it involves developing evaluation logic that remains valid across different task instantiations. It is exactly through its careful state management that in addition to reproducibility AndroidWorld ensures that the reward mechanisms function correctly. Task parameters, initialized randomly at the start of each task based on a controlled random seed, dictate the initial state and influence reward outcomes. Similarly to MiniWoB++ [, ], AndroidWorld consists of a practically infinite set of varying initial conditions and success criteria.

This approach provides more granular analyses of agents’ adaptability — a vital attribute for real-world deployment. Beyond testing agent robustness, the dynamic construction of tasks supports the use of online learning methodologies, particularly reinforcement learning [, , , ]. It also simplifies the generation of distinct train/test datasets, facilitating supervised learning experiments [, , ].

**Table 2: Selected tasks with code describing validation logic.**
| Task | Validation code |
| --- | --- |
| In Simple Calendar Pro, create a calendar event on {event.year}-{event.month}-{event.day} at {event.hour}h with the title ‘{event.title}’ and the description ‘{event.description}’. The event should last for {event.duration} mins. | event_exists(event) |
| Send a text message to {phone_number} with message: {message}. | message_exists(phone_number, message, messaging_db) |
| Create a new drawing in Simple Draw Pro. Name it {file_name}. Save it in the Pictures folder. | file_exists(file_path) |
| Create a timer with {hours} hours, {minutes} minutes, and {seconds} seconds. Do not start the timer. | timer_displays(time, ui_hierarchy) |
| Create a new note in Markor named {file_name} with the following text: {text}. Share the entire content of the note with the phone number {number} via SMS. | (file_exists(file_name, content=text) + message_exists(phone_number, message)) / 2.0 |
| Turn on WiFi and open {app_name}. | (wifi_enabled() + app_launched(app_name))/2.0 |

### 3.4 Durable rewards from system state

AndroidWorld provides reward signals by managing application state using the Android Debug Bridge (adb). With the adb tool AndroidWorld has complete access to system resources including the file system, application databases, and system settings. Determining reward signals from system state has several benefits. It is highly accurate because an application’s state can be quickly inspected and manipulated using the same mechanisms that the app itself utilizes. Using the underlying system state is much more durable than matching superficial UI changes. Additionally, it facilitates easy re-use across disparate apps, which tend to use the same underlying caching mechanisms. For instance, logic for checking existence of a specific file is used across many unrelated applications, including those for file management, note-taking, and media playback. For applications leveraging SQLite databases, a common pattern, AndroidWorld implements evaluators that verify the existence of new and deleted rows. Table 2 shows examples of the validators in AndroidWorld. For more examples see Table 5.

### 3.5 Task composability

In addition to facilitating accurate and reusable evaluations, inferring a task’s success from system state makes it easy to create composite tasks by combining together existing tasks. For example, the task “Create a calendar event with details and text the details to contact” was created by combining together two existing tasks for creating a calendar event and for sending a text message, which is possible because each task initialization and success detection logic is hermetic. Composite tasks tend to be more challenging because of their complexity, although they provide partial rewards based on completion of sub tasks, to help facilitate hill climbing. The last two rows of Table 2 show the validation code for composite tasks.

### 3.6 Integrating MiniWob++

We implement MiniWoB++ in the AndroidWorld framework and term it MobileMiniWoB++. Each MobileMiniWoB++ task is instantiated using the standard AndroidWorld interface, inheriting from TaskEval base class, and contains methods like initialize_state and is_successful. Since MiniWoB++ leverages JavaScript for task configuration and success detection, we built a WebView app to communicate between Python and the app. For instance, the is_successful method of each task retrieves the reward value from the WebView app via an Android intent.

MobileMiniWoB++ introduces modifications in both observations and actions compared to the original benchmark. For example, HTML5 <input> elements are rendered with native Android UI widgets like the date-picker (see Figure  4), enhancing the realism of the tasks. MobileMiniWoB++ uses the same observation space as the Android tasks (accessibility tree and screenshot). Notably, it does not include the DOM as in the original implementation. The action space from AndroidWorld is retained. We manually review and test each task to ensure they are solvable. We excluded twelve of the original tasks that failed to render correctly on Android, presented compatibility issues with the touch interface, or required near real-time interaction, which poses challenges on emulators. Overall, AndroidWorld supports 92 MiniWoB++ tasks. See Appendix B.3 for more details.

## 4 AndroidWorld as a computer-control benchmark

To test AndroidWorld’s applicability for autonomous agents, we develop and test a state-of-the-art agent and its variants across all 20 apps and 116 tasks, as well as on MobileMiniWoB++.

### 4.1 Computer control agents

#### 4.1.1 M3A

We develop a multimodal autonomous agent for Android, M3A. It is zero-shot, integrating ReAct-style [] and Reflexion-style [] prompting to consume user instructions and screen content, reason, take actions, and update its decision-making based on the outcome of its actions.

In the first stage, M3A generates an action, represented in JSON, and reasoning for that action. To generate this output, the agent is provided with a list of available action types, guidelines for operating the phone, and a list of UI elements derived from the Android accessibility tree’s leaf nodes. The agent receives the current screenshot and a Set-of-Mark (SoM) [] annotated screenshot, which includes bounding boxes with numeric labels on the top-left corner for each UI element (see screenshot in Figure 5). The agent attempts to execute outputted action by referencing the specific mark (if applicable). In addition to the multimodal agent, we have developed a text-only variant that consumes the screen represented using the accessibility tree and selects the relevant action in JSON format.

After executing an action, M3A reflects on its effect by observing any state changes that may have occurred. During this stage, the agent is provided with available action types, general operating guidelines, the actual action taken, and its reasoning, as well as before-and-after UI states, represented by UI element representations and screenshots with SoM annotations. We request the agent to provide a concise summary of this step, including the intended action, success or failure, potential reasons for failure, and recommendations for subsequent actions. This summary will serve as the action history and be used for future action selection. See Appendix D for more details on the agent.

#### 4.1.2 SeeAct baseline

We implement a baseline agent based on SeeAct [], which was originally designed for GPT-4V for web navigation. Specifically, we implement the best-performing variant, SeeActchoice, which grounds actions via textual choices. We implement SeeAct for the Android environment to evaluate how an existing model that performs well on web tasks [] can be adapted and applied to Android.

To accommodate the Android environment, we adapt SeeAct in several ways. Firstly, we augment the action space from the original SeeAct implementation to support actions needed for mobile, including scroll, long press, navigate home and back, and open app actions. Secondly, in lieu of the DOM, which is not available for Android apps, we utilize the accessibility tree to construct candidate UI actions. Due to the lack of the DOM representation, we do not use the bespoke ranker model from the original implementation. However, we observe that after applying a filtering heuristic to remove non-interactable elements, the majority of screens contains less than 50 candidate elements. See Appendix D.5 for more details on the implementation.

**Table 3: Success Rates (SR) on AndroidWorld and MobileMiniWoB++.**
| Agent | Input | Base model | SRAndroidWorld | SRMobileMiniWoB++ |
| --- | --- | --- | --- | --- |
| *Human* | *screen* | *N/A* | *80.0* | *100.0* |
| M3A | a11y tree | GPT-4 Turbo | 30.6 | 59.7 |
| M3A | a11y tree | Gemini 1.5 Pro | 19.4 | 57.4 |
| M3A | SoM (screen + a11y tree) | GPT-4 Turbo | 25.4 | 67.7 |
| M3A | SoM (screen + a11y tree) | Gemini 1.5 Pro | 22.8 | 40.3 |
| SeeAct [] | SoM (screen + a11y tree) | GPT-4 Turbo | 15.5 | 66.1 |

### 4.2 Experimental results

We evaluate M3A and SeeAct on AndroidWorld and MobileMiniWoB++. We set the seed to 30 and provide a task-specific step budget. We use Gemini 1.5 Pro and GPT-4 Turbo as base models. For MobileMiniWoB++, we evaluate on a subset of 62 tasks, consistent with recent studies [, , ].

Table 3 presents the success rates (SR) for the agents and human performance on both task suites. Although the agents have far from human performance, they demonstrate out-of-the-box capabilities in operating mobile UIs, exhibiting basic understanding and control capabilities of UIs. They can perform a variety of actions, including long-press, scrolling to search for information, and revising their plan if actions do not work out. The best performance is obtained for M3A when using GPT-4. On AndroidWorld the SoM-based variant is less performant, while on MobileMiniWoB++ it performs best. A similar result was obtained in recent work in the context of computer agents for desktop applications []. We posit SoM plays a more critical role in MobileMiniWoB++ tasks due to the often incomplete accessibility tree, compared to that of native Android apps.

### 4.3 Analysis

Agents have difficulty understanding mobile UIs, often failing to detect visual cues that are essential for task completion (see Figure 6(a)). Additionally, agents struggle with certain UI patterns and affordances, and when they make reasoning mistakes (see Figure 6(b)), they often lack the capability to explore and adapt as humans do (see Figure 6(c)). Moreover, agents sometimes struggle with tasks that simply involve confirming system states, e.g., confirming the WiFi is turned on, suggesting challenges in both task and screen understanding.

The agents struggle with grounding, particularly when executing precise interactions, such as manipulating text (see Figure 7) or operating sliders, and they are often unable to recover from mistyping errors. In addition, for tasks that demand memory, such as performing transcriptions across apps, multiplying numbers, or scrolling, the agents struggle as they are unable to “remember” content.

SeeAct performs less effectively than M3A on the AndroidWorld task suite and similarly on MobileMiniWoB++, reflecting its optimization for web rather than mobile environments. It struggles with mobile-specific actions like long-presses and swipes, and often fails to select appropriate actions due to not incorporating screen elements during action generation. Memory-intensive tasks are particularly challenging, as SeeAct only caches actions without remembering outcomes, leading to repetitive, ineffective behaviors such as endless scrolling. This lack of quick error recovery often results in task termination once maximum steps are reached.

Finally, we note that large foundation models significantly increase latency, taking three times longer than humans on average to complete tasks. On average, M3A takes 3.9 minutes to complete a task, with the text-only version taking 2.5 minutes.

### 4.4 Agent robustness under random seeds

We evaluate agent robustness under two conditions: (1) identical tasks with the same parameters and (2) tasks with different parameter combinations, which change the initial state and task definition. Due to computational constraints, we perform this analysis on a representative subset of AndroidWorld tasks (listed in Appendix D.4). We use the strongest agent, M3A using the accessibility tree and GPT-4, for this analysis. Our results are shown in Figure 3.

Figure: Figure 3: Success rate variation across tasks due to the parametrization built into AndroidWorld. Using a fixed seed, the agent appears completely incapable of solving some tasks due to “bad luck” with the seed. In contrast, under different task parameterizations, we observe the agent is capable of solving the tasks fairly often. Wilson binomial proportion confidence intervals (95%) are shown for the different seed group (orange) and the same seed group (blue). The different seed group has higher variance than the same seed group. Significant differences, with p-value < 0.05, are indicated by “*”.
Refer to caption: extracted/5947147/robustness.png

In the baseline experiment with a constant seed, the agent fails the add and edit tasks, and rarely solves the two delete tasks. For the add and edit tasks, the agent struggles with UI operations, while for the delete tasks, it often gets confused before the step budget is consumed. Surprisingly, the agent’s performance varies even with a fixed seed, suggesting the model’s non-determinism affects agent reliability.

Agent performance varies much more using different seeds, with statistically significant differences for the add expense and edit note tasks. The high intra-task variation indicates the model’s sensitivity to the seed and its task parameters, stemming from a lack of robustness from the underlying model.

Research in RL environments [, , ] has noted similar sensitivity to seeds. Consistent with these studies, we observe agent performance is best represented by the mean across random seeds. Notably, the observation of non-zero rewards under some seeds points to potential enhancements through RL-like mechanisms in future work.

This experiment underscores the importance of extensively testing agents under varied task parameters to ensure they can handle real-world variability, a capability that AndroidWorld supports effectively.

## 5 Limitations

AndroidWorld currently supports tasks from open-source Android apps with at least 1 million downloads and from built-in Android system apps. While testing on more trending apps is desirable, we found open-source apps to be equally realistic and, in some cases, more challenging than apps with larger user bases. Trending apps tend to have UIs which are heavily optimized for smooth user experience, offering more UI functionality and shortcuts. Testing on less-optimized UIs makes the agent’s task harder. In an example failure we show in Figure 6(c), the agent needed to delete all notes in a list and failed by repeatedly searching for a “delete-all” button. Instead, an agent with stronger reasoning capabilities would have probably searched once for that functionality, realized it was not available, and deleted the notes one by one.

## 6 Conclusion

We introduced AndroidWorld, a realistic and robust agent environment for Android that enables the development and evaluation of autonomous agents across a wide range of tasks and apps. AndroidWorld provides a reproducible task suite consisting of 116 tasks across 20 apps, with each task dynamically generated using random parameters to challenge agents with millions of unique goals. By releasing AndroidWorld and establishing benchmark performance with M3A, we aim to accelerate research and development in this area, ultimately leading to the creation of computer control agents capable of operating effectively in real-world environments. Further, the dynamic nature of AndroidWorld opens up new research opportunities for online learning algorithms in computer control agents.

## Appendix A Ethical considerations

##### Malicious use

There is a risk that malicious actors could engineer agents to bypass security measures like CAPTCHAs or engage in activities like spamming. Additionally, they could alter prompts or screen outputs to further harmful objectives.

##### Societal impact

Automation agents may transform societal norms, disrupt employment, and modify human behavior. While they can enhance efficiency, this improvement could pose risks if exploited by malevolent forces.

## Appendix B AndroidWorld environment

### B.1 Observation Space

In AndroidWorld, the Android screen is represented using a State class, which includes the following attributes:

- •
Pixels: An RGB array representing the current screen capture of the device. The screenshot resolution is $2400\times 1080\times 3$.
- •
Accessibility tree: A raw representation of the accessibility tree.(^6^66Represented using all current windows; [https://developer.android.com/reference/android/view/accessibility/AccessibilityWindowInfo](https://developer.android.com/reference/android/view/accessibility/AccessibilityWindowInfo)) This UI tree provides a detailed snapshot of all UI elements currently displayed on the screen. We utilize an accessibility forwarding app from AndroidEnv [], which leverages gRPC to transmit the accessibility tree data efficiently to the device.
- •
UI elements: A list of processed UI elements extracted from the children of the accessibility tree. Each UIElement contains attributes such as text, content description, bounding boxes, and various state flags (e.g., clickable, scrollable, focused).

Since Android observations and actions are asynchronous, changes resulting from actions may take some time to manifest. Therefore, instead of using an RL-based interface, which assumes a tight coupling between actions and observations, we design an interface for the agent tailored for asynchronous interaction. This interface implements a get_state method responsible for capturing the current state of the environment, typically after executing an action. This method includes an optional wait_to_stabilize flag, which, when enabled, employs heuristics to ensure the UI elements are not in a transient state, thus providing a stable and accurate snapshot of the environment.

### B.2 Action space

Actions are stored using a Python dataclass (shown below) and are executed using adb. Each action type corresponds to specific ADB commands that interact with the Android environment. For click-based actions (click, double tap, long press), we use ADB to simulate touch events at specified coordinates on the screen. For text input actions, we first focus on the text input field and then use ADB to type the desired text and press the enter button. Navigation actions (home, back) involve sending corresponding key events to the device. Scrolling and swiping actions, which are essentially inverse operations, are both implemented by generating and issuing swipe commands through ADB to simulate these gestures. To launch applications, we use ADB to start the desired app.

Figure: Listing 1: Pseudo-code representation of the action space.

### B.3 MobileMiniWoB++

Authors manually completed all tasks in MiniWoB++ to verify solvability on a mobile interface. MobileMiniWoB++ differs from MiniWoB++ due to the touch-based interface, which required different approaches for certain tasks. For instance, highlighting text from the highlight-text tasks involves using Android’s long-press and cursor-moving functionalities. HTML5 <input> elements are natively rendered with native Android UI widgets like the date-picker (see Figure 4).

Our implementation of MiniWoB++ contains 92 tasks in total. We exclude the following tasks: chase-circle (requires near-realtime movement, unachievable by humans on emulators), moving-items (too hard to click in emulator), drag-cube (drags will scroll the screen, moving the task out of view), drag-items-grid (elements are not interactable on Android), drag-items (elements are not interactable on Android), drag-shapes (drags will scroll the screen, moving the task out of view), drag-sort-numbers (elements are not interactable on Android), text-editor (cannot underline everything, weird glitch), number-checkboxes (not correctly rendered: only three columns), use-slider-2 (slider implementation not working), use-spinner (slider implementation not working), and click-menu (the menu responsiveness breaks and the task does not behave as intended).

Figure: Figure 4: Native Android UI widget rendering for HTML5 <input> element.
Refer to caption: extracted/5947147/android_widget.png

## Appendix C AndroidWorld Benchmark Details

### C.1 App selection

Our selection of apps (summarized in Table 4) was guided by three main factors: use case, popularity, and the need for consistency and reproducibility.

##### Use case and categories

We analyzed popular app categories in app stores, focusing on productivity, communication, and multimedia. Selected apps had to meet criteria such as not requiring a login and storing data locally on the device. Additionally, we considered apps from categories that the authors commonly used, ensuring the selection was representative of real-world Android usage.

##### Popularity

We used download statistics from the Google Play Store to gauge app popularity, selecting apps with over 1 million downloads. Most of the selected apps exceeded this threshold. Less popular apps were also included if they featured common UI patterns and affordances, ensuring they are indicative of typical Android app usage. For instance, Simple Calendar Pro, though less downloaded, has a UI comparable to the widely-used Google Calendar app.

##### Consistency and reproducibility

All apps were sourced from F-Droid, an open-source Android app repository. This allowed us to manage app versions precisely by selecting and distributing specific APKs. We use the newest version of each app at the time of download.

**Table 4: List of AndroidWorld apps and number of tasks for each one.**
| App name | Description | # tasks |
| --- | --- | --- |
| Simple Calendar Pro | A calendar app for creating, deleting, and managing events and appointments. | 17 |
| Settings | The Android system settings app for managing device settings such as Bluetooth, Wi-Fi, and brightness. | 15 |
| Markor | A note-taking app for creating, editing, deleting, and managing notes and folders. | 14 |
| Broccoli - Recipe App | A recipe management app for adding, deleting, and organizing recipes. | 13 |
| Pro Expense | An expense tracking app for adding, deleting, and managing expenses. | 9 |
| Simple SMS Messenger | An SMS app for sending, replying to, and resending text messages. | 7 |
| OpenTracks | A sport tracking app for recording and analyzing activities, durations, and distances. | 6 |
| Tasks | A task management app for tracking tasks, due dates, and priorities. | 6 |
| Clock | An app with stopwatch and timer functionality. | 4 |
| Joplin | A note-taking app. | 4 |
| Retro Music | A music player app. | 4 |
| Simple Gallery Pro | An app for viewing images. | 4 |
| Camera | An app for taking photos and videos. | 3 |
| Chrome | A web browser app. | 3 |
| Contacts | An app for managing contact information. | 3 |
| OsmAnd | A maps and navigation app with support for adding location markers, favorites, and saving tracks. | 3 |
| VLC | A media player app for playing media files. | 3 |
| Audio Recorder | An app for recording and saving audio clips. | 2 |
| Files | A file manager app for the Android filesystem, used for deleting and moving files. | 2 |
| Simple Draw Pro | A drawing app for creating and saving drawings. | 1 |

### C.2 Task classification and generation

We categorize tasks into two types: those with side-effects and those without. Tasks with side-effects are those that modify the internal state of the device or applications, such as turning off Wi-Fi or creating a calendar event. These tasks are implemented as distinct Python classes, each with its own parameter generation, initialization, evaluation, and teardown methods.

Below we show an example of the task evaluation for a SendSms task, which involves sending and validating a text message. The pseudocode illustrates the task initialization, success check, and parameter generation methods. Each task has its own random parameter generation method and success logic.

### C.3 Information retrieval tasks

Tasks without side-effects are Information Retrieval tasks, requiring the agent to answer a question based on the device or app’s current state. For these tasks, instead of a Python class, we create a protobuf structure to specify the prompt, parameter values, and initialization and validation logic. We decided to use a structured data format with the belief that it would allow us to define new information retrieval tasks by simply adding new entries, making it easier to scale up the number of tasks without needing to write and maintain Python classes for each one.

Initialization is defined per app, including only the state relevant to the prompt’s answer and exclusion conditions for generating random states. This ensures that no random state contains information that could alter the expected answer. The initial state and prompt are parameterized using random values from the specified task parameters. For validation, we define the expected answer format within the prompt and use a few supported functions (“count”, “sum”, “identity”) to generate the answer from the initial state.

Once an app and its specific logic are programmed, new tasks can be generated using an LLM to generate the task’s protobuf. The process is not automatic and requires human review. Common issues with LLM-generated tasks include missing fields, hallucinated fields, incompatible parameter generation, insufficient parameter usage, and non-specific task prompts. We observed that the complexity of the proto structure correlates with an increase in generated task issues. Despite these challenges, we found that editing LLM-generated protobufs can be more efficient than writing a complete task from scratch.

Below we show a simplified version of the task definition for the SimpleCalendarEventsOnDate task which involves checking which events are on a certain date. It specifies the relevant event, the exclusion conditions for any noisy event, how to determine success, and possible parameter values to be chosen at random that will be used to fill out the task definition.

### C.4 Humans for task analysis

During development, we recruited six volunteers with proficient programming skills to analyze task difficulty, duration, and category. Each human was assigned an equal portion of tasks and tasked with identifying bugs during this annotation phase. This process resulted in the discovery and resolution of over 30 bugs.

To evaluate human performance, we enlisted two software engineers to complete the tasks using an Android emulator. Participants were provided with task descriptions and attempted to achieve the goals based on their interpretations. Each participant had one attempt per task. The majority of errors stemmed from misinterpretations or minor errors, such as entering an incorrect file extension. Other errors occurred when participants encountered unfamiliar user interfaces, impeding their ability to solve the tasks on their first attempt.

In both exercises, we informed participants about the intended use of the collected data. Participants were not required to enter any personal information in the tested tasks.

### C.5 Task examples

Table 5 lists some additional examples of tasks and highlights which task attributes can be parameterized in unlimited ways.

**Table 5: Examples of AndroidWorld tasks. We list the task nickname, the task template indicating which task attributes can be parameterized, the initialization logic that is executed before the task starts and pseudo code describing the success evaluation.**
| Task nickname | Task template | Initialization logic | Success evaluation code |
| --- | --- | --- | --- |
| VlcCreatePlaylist | Create a playlist in VLC, titled “{playlist_name}” with the following files, in order: {files} | Create new mpeg files: files + “noise” files that should not be added. Add them to VLC videos folder. | execute_sql(vlc_query) == files |
| RecipeAddMultiple RecipesFromImage | Add the recipes from recipes.jpg in Simple Gallery Pro to the recipe app. | Write a receipt file with recipes to Simple Gallery. | sql_rows_exist(expected_recipes) |
| MarkorEditNote | Edit {file_name} in Markor. {file_operation}. | Generate file with starting content, along with “noise” files not relevant to goal. Note: file_operation can be to add a footer, header, or update note content. | file_exists(file_name, content=expected_content) |
| ExpenseAddSingle | Add the following expenses into pro expense: {expense_csv} | Add to the app’s SQLite database the expense that should be deleted, along with “noise” expenses that should not be deleted. | sql_rows_exist(expense_obj) |
| SimpleCalendarDelete EventsOnRelativeDay | In Simple Calendar Pro, delete all events scheduled for this {day_of_week}. | add to the app’s SQLite database calendar events on specified day, along with “noise” events that should not be deleted. | !sql_rows_exist(expected_events) |
| FilesDeleteFile | Delete the file {file_name} from the Android filesystem located in the {subfolder} folder within the sdk_gphone_x86_64 storage area. | Generate specified file, along with “noise” files that should not be deleted. | !file_exists(file_name) |
| SportsTrackerActivities CountForWeek | How many {category} activities did I do this week in the OpenTracks app? Express your answer as a single integer. | add to the app’s SQLite database activities for the specified category, along with “noise” activities. | int(agent_response) == expected_count |

## Appendix D AndroidWorld agent details

### D.1 M3A observations

AndroidWorld consumes the raw screen pixels, the screen shot with Set-of-Mark (SoM) [] annotations, and a list of UI elements on screen.

Figure: Listing 2: The prompt format pertaining to screen representation with UI elements.

### D.2 M3A actions

For the SoM prompting, the screen is annotated based on the UI elements extracted from the accessibility tree, which form the agent’s action space. Figure 5 shows one example.

Figure: Figure 5: Set-of-marks overlaid on an Android screen.
Refer to caption: extracted/5947147/som_example.jpg

### D.3 Error analysis

We analyze M3A errors based on broader categories we observe during evaluation.

##### Perceptual errors

Perceptual errors are caused when the model fails to recognize crucial elements on the screen necessary for task completion.

For the task below, the model does not recognize that the “All-day” checkbox is currently not checked (see Figure 6(a)):

> *In Simple Calendar Pro, create a recurring calendar event titled ’Review session for Budget Planning’ starting on 2023-10-15 at 14h. The event recurs weekly, forever, and lasts for 60 minutes each occurrence. The event description should be ’We will understand software updates. Remember to confirm attendance.’*

Figure: (a) Perceptual error. Red square highlights issue.
Refer to caption: extracted/5947147/perceptual_error.png

##### Reasoning errors

Reasoning errors occur when the model misinterprets the task requirements or the current state, leading to incorrect actions.

For example, in the task below, the model mistakenly believes the note name has already been entered, so it types the note text into the “Name” field and cannot recover from this error (see Figure 6(b)):

> *Create a new note in Markor named ipsum_mNcKd36pDq.txt with the following text: …*

Here is the agent’s reasoning trace:

> *The user wants to create a new note with a specific name and text. The name of the note has already been entered, so all that needs to be done is to enter the text and click on the OK button.*

##### Missing knowledge errors

Missing knowledge errors occur when the model lacks the necessary understanding of the application’s UI to complete a task efficiently.

For example, in the task below, the agent does not know how to delete all files at once. It looks for an option to do that instead of deleting each file one by one (see Figure 6(c)):

> *Delete all files in the current directory.*

##### Grounding errors

Grounding errors occur when the model fails to correctly interact with the UI elements based on their spatial or contextual positioning.

For the task below, the agent needs to update the Markor note by prepending text to the existing text. Figure 7 illustrates the errors the agent makes. It clicks the entire text field area, highlighted in green, which automatically places the cursor after the current text, resulting in the new text being appended after the current content.

> *Update the Markor note ‘2023_08_10_neat_wolf.txt‘ by adding the following text, along with a new blank line before the existing content: ”ETBM2jAP6vXqhbpUsfVm”, and rename it to ‘sure_ocean_uRnI.txt‘.*

Figure: (a) Error on initial click.
Refer to caption: extracted/5947147/ground_error.jpg

Then, in the next screen, the text has been entered after the existing content, and the agent clicks the save button.

### D.4 Agent robustness experiments

We ran the agent on the following tasks (the nicknames shown in the figures in parentheses):

- •
MarkorEditNote (EditNote)
- •
ExpenseAddSingle (AddExpense)
- •
SimpleCalendarDeleteEventsOnRelativeDay (DeleteEvent)
- •
FilesDeleteFile (DeleteFile)
- •
SportsTrackerActivitiesCountForWeek (CountActivities)

More details about these tasks can be found in Table 5.

### D.5 SeeAct details

We modify the SeeAct prompt [] to reflect that the environment is Android by inputting elements from the accessibility tree and supporting additional actions (e.g., scrolling). Below we include the updated prompt. We annotate the system, user, and assistant roles that are each provided to the OpenAI API.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: - How might AndroidWorld be expanded or adapted to include tasks that require specific contextual understanding, such as personalized interactions with app data?
- How might agent performance improve if AndroidWorld incorporated multimodal feedback mechanisms, such as sound or haptic responses, beyond the visual and text-based cues currently used?
- B-MoCA, like AndroidWorld, operates in Android environments and also employs various randomization schemes for task variability. How does your approach to task diversity, randomization, and reproducibility compare to B-MoCA? What unique advantages does AndroidWorld offer regarding its benchmarking scope, real-world applicability, and parameterized task generation? Additionally, are there specific aspects of your design that provide it with a distinct edge in evaluating agent robustness?
- Could the authors provide a full list of the tasks included in AndroidWorld, along with the initialization logic and success evaluation codes for each task?
soundness: 3
strengths: - AndroidWorld’s dynamic task construction introduces extensive variability in task conditions, offering a realistic and reproducible environment for testing autonomous agents on Android.
- The paper provides a robust baseline evaluation and a thorough performance analysis across real-world conditions.
- Extensive experiments provide essential insights into current agents' limitations and suggest potential pathways for improvement in future cross-platform agent designs.
summary: The paper introduces AndroidWorld, a benchmarking environment for autonomous agents interacting with Android devices. This environment enables testing on various apps by dynamically constructing parameterized tasks expressed in natural language. AndroidWorld supports a more realistic and varied testing experience than static environments, enabling agents to engage with numerous unique task goals and environmental conditions. Initial experiments using baseline agents indicate significant room for improvement in task completion rates, with top-performing agents achieving a 30.6% success rate. This highlights the challenges and research opportunities in building robust, cross-platform autonomous agents.
weaknesses: - The agents achieve a low overall success rate (30.6%), which, while reflecting the environment’s complexity, suggests that current methods may need significant refinement to handle mobile platforms effectively.
- Though this paper introduces extensive task parameterization, it does not provide a detailed explanation of how specific parameter variations impact agent performance. There is also no analysis of performance changes across different task components or parameters.
- Large foundation models integrated into the agents exhibit high latency, taking significantly longer than humans to complete tasks, which may limit the practical applicability of these agents in real-time scenarios.

---

confidence: 4
contribution: 4
initial_rating: 8
presentation: 3
questions: Q1 - Could the authors explain the rationale for integrating MiniWoB++ into a mobile environment, as this incorporation seems highly auxiliary? What unique challenges do these MiniWob++ tasks present, especially compared to tasks involving typical applications in AndroidWorld?

Q2 - While the unique aspects of the mobile action space are described, not all are represented in the current action space (e.g., multi-finger actions). Could the authors comment on the extensibility of M3A/AndroidWorld to accommodate different action spaces, or do they plan to incorporate a broader range of actions in the future?

Q3 - Could the authors provide a guideline on the training and testing configuration, such as a split of task categories? Additional details would aid in understanding the optimal configuration for achieving reliable results.

Q4 - (Minor) Would the authors consider providing results using Claude’s newly proposed models (considering that the relevant developers are interested in digital device usage)? This addition would offer a timely perspective on how recent advancements perform within this framework.

Q5 - The authors mention that using multiple random seeds and averaging the results would provide a fair representation of the results (line 493). Could the authors confirm if the results in Table 3 are also based on averaged results across different seeds? If not, might it be beneficial to conduct the experiments across multiple seeds, as task randomness can significantly impact agent performance?
soundness: 4
strengths: S1 - This work presents a foundational contribution that advances the AI community's understanding of **mobile device control**, offering a robust framework for evaluating LLM agents in interactive environments.

S2 - The related work and comparisons to existing benchmarks are well-organized and comprehensive. This provides a clear context of how this benchmark builds upon and differentiates itself from prior studies.

S3 - The benchmark introduces an important challenge related to task generalization by implementing parametrized tasks. This effectively assesses the generalizability of agents across varying task instructions.
summary: A benchmark for LLM (or LMM) agents controlling mobile devices is introduced, focusing on the necessity of interactive evaluation (as common benchmarks in this domain have mainly focused on static datasets so far) and the agents’ generalization ability in terms of task instructions. The tasks regard applications in Android commonly used in daily life as well as tasks in MiniWob++, which have been a common benchmark in digital device control. The study also proposes a new prompting approach to enhance agentic capabilities in foundation models and presents a detailed analysis of agent performance. The analyses include comparisons across different input modalities, robustness across varied task parameters, and their common errors. 

The main contributions of this work are the introduction of a new benchmark, tasks that can be randomized via instruction parametrization, and a new algorithm enhancing the capability of LLM agents for mobile device control.
weaknesses: W1 - The explanation of the action space could be more detailed. For instance, in the ACTION_TYPE section described in Appendix B.2, further clarification on the purpose of the "STATUS" action would be helpful. Additionally, what is the rationale behind the necessity of a "SWIPE" action, despite the existence of "SCROLL", could be further justified. 

W2 - Although line 257 mentions a limited set of high-level APIs, Appendix B lacks specific details on these APIs. More explanation would help us understand the scope of actions.

W3 - The rationale behind the task categorization tags could be elaborated on further. Expanding on the justification for the categorization approach would clarify its significance and enhance the robustness of the framework.

W4 - I believe that this work could benefit from the inclusion of more baseline comparisons. Presenting results from a basic version of the agent, without any prompting methods for example, would provide valuable reference points. While the current results demonstrate that M3A is a strong baseline, additional baselines would enrich the study and offer further insights into performance.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. How was the confidence interval in Figure 3 calculated? I couldn’t find details on how these intervals were derived. Did you run the same model on these tasks multiple times to obtain them?
soundness: 3
strengths: 1. Solid contribution: this benchmarks fills the gap for solid, reproducible, and executable benchmarks for Android device control tasks.
2. Good presentation: the presentation is clear, discussions in related works are comprehensive
3. Interesting experiments: the experiments presents a few good baselines and shows how human performs on the tasks. The robustness analysis also provides new information to the community.
summary: This paper introduces AndroidWorld, an executable, reproducible, and realistic benchmark for evaluating Android device control agents. The authors also present results from several baseline agents and human.
weaknesses: 1. Lack of many real-world apps and tasks: The benchmark lacks many real-world applications and tasks, as ensuring full reproducibility and automated evaluation makes it impossible to include closed-source apps like YouTube, Twitter (X), Amazon, or actual real-web browsing. This results in inherent sim-to-real gaps.
2. Lack of device diversity on android device/OS: The emulated device is fixed as a Pixel 6 running Android 13. However, in practice, what we care about is agents' performance across various devices and OS versions. Adding more OS/device options would potentially make the benchmark more correlated with real world use cases.
3. Lack of open-source baselines: All baselines in the paper appear to be based on proprietary models. Including some open-source models as baselines would be beneficial, as it would provide researchers a baseline they can iterate on.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. What is the number of human hours required to develop the environment and configure the tasks?
2. In your initialization and teardown process, have you considered scenarios where the agent performs actions outside the defined teardown process, potentially altering system states and resulting in non-identical states across different experiment trials?
soundness: 3
strengths: 1. The paper is overall clear and well-written.
2. The environment is lightweight and compatible with task random parameterization.
3. Detailed examples and error analysis are provided in the appendix.
4. I appreciate the effort to implement the initialization and teardown process for each task to ensure reliability and reproducibility.
summary: This paper presents AndroidWorld, a realistic and reproducible Android benchmark that provides functional rewards for 116 dynamically constructed tasks across 20 real-world Android apps. The experiments show that the developed agent M3A can complete only 30.6% of the tasks, and existing web agents struggle to transfer well in Android environments.
weaknesses: 1. The experimental setup is not very clear. Specifically:
- What specific stop criteria are used for the experiments (e.g., step limits or time limits)? How is the task-specific step budget set for each task?
- What decoding temperature is applied?
- How is human performance measured?

2. It would be helpful to include results from more models, including recent APIs and open-source models. The ablation study only investigates the observation space and does not consider the impact of other agent design choices, such as Reflexion.

3. Are all tasks and apps offline? If not, how to ensure reproducibility? For example:
- Does the SendSms task send actual messages?
- Could real-world time and date affect the evaluation results and reproducibility in calendar/timer tasks within AndroidWorld and MobileMiniWoB++?

4. More discussion of related work, e.g.:
- Zheng et al., "Agentstudio: A toolkit for building general virtual agents," arXiv preprint arXiv:2403.17918.
- You et al., "Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs." arXiv preprint arXiv:2404.05719.

Meta-review: The reviewers unanimously agree that AndroidWorld provides a novel, challenging benchmark for the computer-use abilities of LLMs. Importantly, reviewers point out that this work also reports the various aspects of these tasks with which modern LLMs struggle, providing a clear indication of what improvement on this benchmark means for LLMs capabilities.

Decision: Accept (Poster)