AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents
Authors: Christopher Rawles
Sarah Clinckemaillie
Yifan Chang
Jonathan Waltz
Gabrielle Lau
Marybeth Fair
Alice Li
William E Bishop
Wei Li
Folawiyo Campbell-Ajala
Daniel Kenji Toyama
Robert James Berry
Divya Tyamagundlu
Timothy P Lillicrap
Oriana Riva
Venue: ICLR 2025 Poster (2025)
Keywords: Computer Control
Autonomous Agents
LLMs
Multimodal
Primary area: datasets and benchmarks
TL;DR: An environment for benchmarking Android UI agents which provides durable rewards for different task parameterizations across 20 apps.

Abstract: Autonomous agents that execute human tasks by controlling computers can enhance human productivity and application accessibility. However, progress in this field will be driven by realistic and reproducible benchmarks. We present AndroidWorld, a fully functional Android environment that provides reward signals for 116 programmatic tasks across 20 real-world Android apps. Unlike existing interactive environments, which provide a static test set, AndroidWorld dynamically constructs tasks that are parameterized and expressed in natural language in unlimited ways, thus enabling testing on a much larger and more realistic suite of tasks. To ensure reproducibility, each task includes dedicated initialization, success-checking, and tear-down logic, which modifies and inspects the device’s system state.

We experiment with baseline agents to test AndroidWorld and provide initial results on the benchmark. Our best agent can complete 30.6% of AndroidWorld's tasks, leaving ample room for future work. Furthermore, we adapt a popular desktop web agent to work on Android, which we find to be less effective on mobile, suggesting future research is needed to achieve universal, cross-platform agents. Finally, we also conduct a robustness analysis, showing that task variations can significantly affect agent performance, demonstrating that without such testing, agent performance metrics may not fully reflect practical challenges. AndroidWorld and the experiments in this paper are available at https://github.com/google-research/android_world.

Published as a conference paper at ICLR 2025
ANDROIDWORLD: A DYNAMIC BENCHMARKING ENVIRONMENT FOR AUTONOMOUS AGENTS
Christopher Rawles*1 Sarah Clinckemaillie Yifan 2 Jonathan Waltz", Gabrielle Lau², Marybeth Fair², Alice Li", William Bishop', Wei Li", Folawiyo Campbell-Ajala" Daniel Toyama', Robert Berry'. Divya Tyamagundlu', Timothy Lillicrap. and Oriana Riva'
'Google DeepMind Google
ABSTRACT
Autonomous agents that execute human tasks by controlling computers hance human productivity and application accessibility. However, progress in this field will be driven by realistic and reproducible benchmarks. We present INDROIDWORLD. a fully functional Android environment that provides reward signals for 1 116 programmatic tasks across 20 real-world Android apps. Unlike ex- isting interactive en environments, which provide a static test set, INDROIDWORLL dynamically constructs tasks that are parameterized and expressed in natural lan- guage in unlimited ways, thus enabling testing much larger and more realistic suite tasks. To ensure reproducibility, each task includes dedicated initializa- tion, success-checking, and tear-down logic, which modifies and inspects the de- vice's 's system state.
We experiment with baseline agents to test ANDROIDWORLD and provide initial results on the benchmark. Our best agent can complete 30.6% of ANDROID- WORLD'S tasks, leaving ample room for future work. Furthermore, we adapt popular desktop web agent to work on Android, which we find to be less effective on mobile, suggesting future research is needed to achieve universal, cross-platfoni agents. Finally, we also conduct a robustness analysis. showing that task variations can significantly affect agent performance, demonstrating that without such testing, agent performance metries may not fully reflect practical challenges. ANDROIDWORLD and the experiments in this paper are available at https://gittnei com/ google/ www.eercreeeeceeeeeeeeeeeeee andarndooo aadroodd_wor cl world.
INTRODUCTION
Autonomous agents that interpret natural language instructions and operate computing devices can provide enormous value to users by automating repetitive tasks, augmenting human intelligence, and accomplishing complex workflows. However, key research challenge remains the realistic evalu- ation of these agents real-world settings. Despite growing enthusiasm for building autonomous agents (Deng et al., 2023; Rawles et al., 2023; Zheng et al., 2024a; Koh et al., 2024; Kim et al., 2024: He et al., 2024; Gravitas, 2023; Wu et al., 2023; Xie et al., 2023) most existing approaches for evaluation compare agent's 's actions at each step to previously collected human demonstra- tion (Deng et al., 2023; Rawles et al., 2023; Yang et al., 2023b; Zhang & Zhang, 2023; Là et al., 2024; Zhang et al., 2024c; Yan et al., 2023; Li et al., 2024). Measuring performance in this way can be mislcading because when performing tasks online in real environments agents can take multiple paths solve tasks, environments may behave non-deterministialll, and agents can dynamically leam from mistakes to correct their actions (Shinn et al., 2023; Liu et al., 2018b; Li et al., 2023b; Pan et al., 2024). For this reason, online evaluation of agents in realistic environments able to reward task outcome provides a gold standard for evaluation. While there is an emerging body of work to address this need across different environments (Zhou et al., 2023; Koh et al., 2024; Drouin al.,
"Lead contributor. Contact: crwwles@@ooggee.com
I Equal contribution.

---

Published as a conference paper at ICLR 2025
Simple Calerdar Pro, crease calendar on
(bfiee and the description (desciition. The avent should last for (ouration minss minss
Android Emulator
TaskEval
goal
gen
Apps
OS State
reward
insEinlizs task task
1x autiowoolul
Add location for Docalion) the Oaminnd maps apo.
teardown
or APls
Agent
task
goal
116 Android & MiniWob++ tasks
1
rocomplete Parve do (date) Tasks app?
AndroidWorld
CU 10/2017 AA the dalle and hi! submit
Figure I: ANDROIDWORLD is environment for building and testing autonomous agents.
2024; Lee et al., 2024; Xie et al., 2024; Bonatti et al., 2024; Zheng et al., 2024b), there is no com- prehensive solution for mobile platforms, such as Android, which are used by billions of users and therefore represent environments in which automation agents may be very productively employed. We introduce ANDROIDWORD
to address this.
At its core, ANDROIDWORLD offers reliable means of obtaining reward signals for tasks per- formed by agents in realistic mobile environments. Reward signals are quantitative metrics that indicate functional correctness of task. i.c. is the stated goal achieved? For example, for the task "Send a text message to Jane confirming I'll be there," a positive reward indicates that the relevant message has been sent. Unlike simulated environments (Tassa al., 2018; Shridhar al., 2020) or games (Mnih et al., 2013; Silver et al., 2016; Vinyals et al., 2019; Wang et al., 2023b; Tan et al., 2024; Toyama et al., 2021), real-world apps and websites do not inherently offer explicit reward sig- nals. While human (Rawles et al., 2023; Zheng et al., 2024a; Pan et al., 2024; Kinniment et al., 2023) or LLM-based (Chiang al., 2024; Zheng et al., 2023; Liu et al., 2023; Du et al., 2023; Ma al., 2023; Pan et al., 2024; He et al., 2024) judges can be employed to reward the outcome of task, these approaches scale poorly or are not fully reliable, respectively. Alternatively, environments for autonomous agents which provide automated ground-truth rewards for complex workflows have been developed (Yao et al., 2023; Zhou et al., 2023; Koh et al., 2024; Xie et al., 2024; Bonatti et al., 2024). We find two problems with these environments. First, they constrained to desktop computing environments, overlooking the mobile domain, which is of paramount importance given the ubiquity and diversity of mobile devices in the real world. Secondly, they are limited in their real-world diversity and scale. Crucially, unlike in real-world scenarios where conditions and task inputs vary widely, these environments support only statie test specifications, meaning that when task parameters deviate, the reward signal is likely to break.
We seek to develop a comprehensive benchmark that addresses the limitations of the existing ap- proaches above for evaluating automation agents in mobile environments. ANDROIDWORLD does this by spanning 20 Android apps on total of 116 programmatic tasks to provide ground truth- rewards. Unlike existing test environments (MiniWoB++ (Shi et al., 2017) being a notable excep- tion), each task in ANDROIDW ORLD is dynamically instantiated using mandomly-generatee param- eters, challenging agents with millions of unique task goals and conditions. While MiniWob++ consists of simple, synthetic websites, ANDROID W ORLDD leverages actual Android applications. A main challenge that ANDROIDW WORLD must address is how to ensure that reward signals are durable when using real-world applications and varying task parameters dynamically. ANDROIDWORLDD solves this by leveraging the extensive and consistent state management capabilities of the Android OS, using the same mechanisms that the apps themselves utilize to store and update data.
In addition to providing comprehensive benchmark, ANDROIDWORLD is lightweight, requiring only 2 GB of memory and 8 GB of disk space, and is designed with convenience in mind. connects agents to the Android OS by leveraging the Python library AndroidEny (Toyama et al.,
2

---

Published as a conference paper at ICLR 2025
Table 1: Comparison of different datasets and environments for benchmarking computer agents.
Env?
@f of apps or websites
task templates
Avg task
Reward
Platform
instances
method
GATA
n/a
466
text-match
None
MIND2WEB
137
2350
None
Desktop Web
WEBLINX
155
2337
None
Desktop Web
WEB VOYAGER
15
643
LLM judge
Desktop Web
PIXELHELP
4
187
None
Android
METAGUI
X
6
1225
None
Android
MOTIF
125
4707
None
Android (Apps+ Web)
AITW
357+
30378
None
Android (Apps+Wor Web)
ANDROIDCONTROL
None
Android (Apps+ Web)
OMNIACT
60+
9802
None
Desktop (Apps+Wett Web)
ANDROIDARENA
221
Action mateh/LLM
Android (Apps+ Web)
LLAMATOUCH
57
496
Screen match
Android (Apps+ Web)
++++
1
114
HTMI JJS state
Web (synthetic)
WEBSHOP
1
12k
product altrs match
Desktup Web
WEBARENA
6
241
3.3
urlttcxt-macch
Desktop Web
ASUALWEBARENS
4
314
2.9
urlhext/image-mate
Desktop Web
WORK ARENA
622.4
cloud state
Desktup Web
MOBILE-ENV
I
13
11.5
regex
Android (Apps)
B B-MOCA
4
1.9
regex
Android (Apps+ Web)
MMINA
14
1050
text-match device/cloud state
Desktop web
OSWORLD
9
369
Desktop (Apps+Wel) Web)
WINDOWSAGENTAREN
11
154
device state device state
Desktop (Apps+ Web)
AGENTŠTUDIO
205
Desktop (Apps+ Web)
WORRD
20
116
00
device state
Android (Apps+ Web)
2021) to connect to the freely available Android Emulator.' In addition to the 116 Android tasks, we extend ANDROIDWORLD with web tasks by integrating the MiniWoB++ (Shi al., 2017; Liu et al., 201 8a) benchmark into it.
To demonstrate INDROIDWORLDY usefulness a benchmark, build and release a multi-modaa agent, M3A (Multimodal Autonomous Agent for Android), and establish state-of-the-ard results on ANDROIDWORLL We analyze M3A's performance using both multimodal and text-only input, and we observe that while multimodal perception improve performance in some cases, it gen- crally does not outperform the text-only approach. On ANDROIDWORLD, M3A achieves 30.6% success rate, which surpasses that of web agent adapted for Android but remains significantly lower than the human success rate of 80.0%. In pursuit of building robust UI control agents, our study includes comprehensive tests under varied real-world conditions, demonstrating significant performance variations primarily driven by changes in intent parameters.
We make the following contributions: (i) the creation of new, highly diverse and realistic mobile UI control agent environment; (ii) establishment of benchmark performance with a state-of-the-an mul- timodal agent, and (iii) careful analysis demonstrating the need evaluate agents across variable task parameters and conditions due to the inherent stochasticity in both models and environments.
2 RELATED WORK
Table compares existing evaluation environments for autonomous UI agents.
2.1 INTERACTIVE EVALUATION ENVIRONMENTS
Effective evaluation of autonomous agents requires benchmarks that mimic real-world scenarios, but also interactive environments that provide reward signals upon successful task completion (Rawles al., 2023: Deng et al., 2023; Abramson al., 2022; Ruan al., 2023; Chen al., 2021). Many existing benchmarking environments target web browsing, MiniWoB++ (Shi et al., 2017; Liu et al., 2018b) consists of small, synthetic HTML pages with parameterizable tasks which allow for un-
'The Android Emulator is packaged as part of Android Studio, which can he downloaded from http:///oeoooprraandooo..c///.td
3

---

Published as a conference paper at ICLR 2025
limited task variability. WebShop (Yao et al., 2023) provides a simulated e-commerce environment, whereas WebArena (Zhou et al., 2023) and VisualWebArena (Koh et al., 2024) consist of simulated websites across up to six domains. WorkArena (Drouin et al., 2024) consists of 29 tasks for enter- prise software. GAIA (Mialon et al., 2023) is a static dataset that tests an agent's ability to interact with live web environments. MMInA (Zhang et al., 2024e) is multihop and multimodal benchmark designed to evaluate agents for compositional Internet tasks.
Towards building computer use agents, OSWorld (Xie et al., 2024), WindowsAgentAreen (Bonatti et al., 2024), and AgentStudio (Zheng et al., 2024b) provide a test suite of tasks for desktop computer interfaces and custom execution-baas basedd evaluation scripts across 9, 11, and 9 apps, respectively. In the mobile domain, existing benchmarks are limited and do not capture the diversity of real-world mobile interactions, containing low-complexity tasks or on a limited number of applications. B- MoCA's (Lee et al., 2024) evaluation is based on 6 simple tasks (e.g., "Call 911", "turn on airplane mode") across apps', validated using regular expressions. Mobile-Ent (Zhang et al., 2024b) offers task reproducibility limited to 13 task templates for a single app (WikiHow).
shares the mobile OS focus of B-MoCA and Mobile-Env, it more com- parable to OSWorld (and WindowsAgentArenn, which builds on top of OSWorld) in terms of task complexity and the diversity of interactions it supports. ANDROID WERRODDWORR WORLD enhances OSWorld's ' ap- proach by dynamically constructing the start states of an agent's run and varying the task parameters in unlimited ways, thus allowing for a new type of evaluation under varying real-world conditions.
While ANDROIDWORD
Other studies leverage human evaluation (Rawles al., 2023; Zheng et al., 2024a; Bishop et al., 2024) for tasks where automatic evaluation is not available. Lastly, emerging research (Pan et al., 2024; He et al., 2024; Xing al., 2024; Zheng et al., 2024b) explores the potential of multimodal models to generalize agent evaluations to new settings, though this area requires further research to achieve accuracy comparable to manually-coder rewards.
AndroidEnv (Toyama et al., 202.) provides a mechanism to manage communication with the An- droid emulator, similar to Playwright and Selenium for web environments. While ANDROIDWORLE leverages this functionality, it diverges in its reward system. AndroidEnv's s approach requires mod- ifying application source code and implementing tpsk-specific logging statements, making it well- suited for gaming environments with easily verifiable success criteria. In contrast, ANDROID- WORLD implements a non-invasive reward mechanism, allowing it to create a benchmark suite for apps whose source code is unavailable and to reuse validation components across different apps. This approach enables ANDROIDWORLD to cover a broader range of real-world mobile tasks.
2.2 STATIC DATASETS FOR UI AUTOMATION
Datasets derived from human interactions provide proxy metrics that correlate with real-world agent performance (Li et al., 2020; Burns et al., 2021; Deng et al., 2023; Rawles et al., 2023). On mo- bile platforms, AitW (Rawles al., 2023), AndroidControl (Li et al., 2024), PixelHelp (Li et al., 2020), AndroidArens (Xing et al., 2024). LlumaTouch (Zhang et al., 2024d), UGIF (Venkatesh et al., 2022), and MoTIF (Burns et al., 2021) consist of demonstrations across Android apps and mobile websites, with screens often represented via accessibility trees. In contrast, desktop web environments typically utilize the DOM for representing website content, with Mind2Web (Deng et al., 2023), OmniAct (Kapoor et al., 2024) and others, across various desktop websites. Mobile- based datasets frequently involve more complex actions, such as scrolling, which are not as useful in DOM-based desktop interactions where the entire action space is readily accessible. Additionally, API-centric datasets like API-Bank (Li et al., 2023a), ToolTalk (Farn & Shin, 2023), and ToolBench (Xu et al., 2023) assess agents' capabilities to manipulate computer systems via APIs.
2.3 INTERACTIVE AGENTS
Prior to today's foundation models, traditional approaches to developing user interface-operating agents primarily used reinforcement learning and behavioral cloning to simulate interactions like mouse clicks and keyboard typing (Liu et al., 201 80;8; Li et al., 2020; Shvo et al., 2021; Gur et al., 2022a; Humphreys et al., 2022). More recent work leverages off-the-shelf foundational models (Gemini, 2023; OpenAI, 2023; Touvron et al., 2023) with in-context learning (ICL) and fine-tuning
Baaed on what reported in the Experiments Section of the B-MoCA manuscript as of October 1 2024.
4

---

Published as a conference paper at ICLR 2025
entry
reading
info retrieval
data edit
complex ui
tosks
repetition
multi app
transcription
requires setup
*
math counting
memorization
10 15
20
25
30
0
20 of tasks
30
medium
hard
of steps
(a)
(h)
(c)
Figure 2: Annotators performed the tasks assigned to them, assigned a difficulty level (2a) and estimated the number of steps required to complete each task (2b), using the action space available to an agent. For each task, they selected relevant category tags from a predefined list (2c).
applied to mobile (Rawles et al., 2023; Hong al., 2023; Wang et al., 2023a; Yan al., 2023; Zhang & Zhang, 2023; Bishop et al., 2024; Zhang et al., 2023), desktop web (Zheng et al., 2024a; Deng al., 2023; Zhou et al., 2023; Koh al., 2024; Cheng al., 2024; Lai et al., 2024; You al., 2024), and desktop OS (Wu et al., 2024; Zhang et al., 2024a; Xie et al., 2024). Recent work explores agents that reflect on system state (Shinn al., 2023; Yao al., 2022; Madaan al., 2024) by leveraging exploration, self-evaluation, and retry-capabilities for continual learning and adaptation (Li et al., 2023b; Yang et al., 2023b; Pan et al., 2024; Wu al., 2024; Gao al., 2023; Murty al., 2024).
ONDRO
3.1 ANDROID FOR AUTONOMOUS AGENTS
Android is ideal environment for developing autonomous agents. It is the most widely-used OS globally² and is highly flexible for research, while providing an open world of the Web and over 2M apps for agents operate in. Using emulation, not require specialized hardware, and can be rum on a laptop. Android Virtual Devices or emulator images well suited for research they self-contained, easy to distribute, and configurable.
Android environment is easy to deploy, does
Compared to desktops, mobile environments like Android present unique challenges for computer- agents. While mobile UIs simpler due smaller screens, their action space requiring intricate gestures (e.g., navigating carousels, long-press oong-pressing, multi-finger zooming) and often more steps complete tasks. Unlike web-browser-only environments, Android, an OS, offers greater flexibility, including function-calling APIs (e.g., sending texts) alongside standard UI actions (click, scroll, type).
complex,
3.2 THE OBSERVATION AND ACTION SPACE
ANDROIDWORLD provides an interface for agents to receive observations and execute actions on Android. uses AndroidEny (Toyama et al., 2021) and the Android Device Bridge to facilitate inter- action between Android and the agent. The observation space consists of full-resolution screenshot and UI tree representation developed for accessibility purposes. The action space is similar to that which humans use, consisting of gestures (i.c., tapping, swiping), typing, and navigation buttons (i.e.. go home and go back). In addition these naturalistic actions, ANDROIDW WOLD exposes a limited set of function calling APIs, such as send text. www.tttttttttteeeeeeeeee _message, to help agents accomplish goals. Appendix C provides more details the observation format and action space.
3.3 REPRODUCIBLE AND PARAMETERIZEI TASKS
ANDROIDWORLD consists of a suite of 116 tasks, spread across 20 diverse applications (see Ap- pendix for more details). These tasks simulate practical, everyday activities, including note-
https: / wtt...statt.errr...m wwwpnntterrrcoo/oooeeeoeeeee com/os-moooooooooooooooooo...o markt-ahare Mobile is the most popular platform for accessing the web; http::///..aaacccameee..omm form-marrre shwre/ wearrr/ddsttooommee we/worldwicc
5

---

Published as a conference paper at ICLR 2025
Table 2: Selected tasks with code describing validation logic.
Task
Validation code
Simple Calendar Pro, creale calendar event
event eeteeeeisttee (ovent)
(avent -year) yyar)-(avant mont (aet--)-eeeeet.. day) at (event hour with title "(event. tiilee" and description weweeet.eeacrippee..... description]: The event should last for fevent. aevent.durate duratiion ionj mins.
Scnd a text message to [shone_number] nusbbe)) with message: {message).
message.existe (phone number, message, messaging db)
Create new drawing in Simple Draw Pro. Name it (file name Save it in the Pictures folder.
file exwste (file path)
Create timer with [hours) hours, {minutes) minutes, and [seconds) seconds. Do not start the timer.
timer displeys (time, ui ui.hicrarchy)
note Markor named { Eileeneme) } with following text: ftex:] J. Share the entire content of the note with the phone number [number) via SMS.
(file.existe (file.name, contentsteaxt) dessage.exist (pteeeepeuniiie number, message)) 2.0
Turn on WiFi and open (app_namc).
(xifi_enabled app_launched (app_name))
taking, scheduling appointments, communicating through messaging, and interacting with system utilities. The suite consists of open-source apps and built-in Android system apps, such Settings and Contacts. As rated by humans, the tasks vary in difficulty, duration, and categories (Figure 2).
To achieve high degree of reproducibilityy in real-world scenarios, ANDROIDWORLD precisely controls the OS and app states in several ways. The Android OS is fixed, consisting of Pixel emulator running Android 13. At the start of each task, ANDROIDW ORLD resets the device times- tamp to October 15th, 2023 at 15:34 UTC, ensuring consistent dimendependent behaviors across all executions. All applications in ANDROIDWORLD WORLD are fully-functional and consists of both open- source apps and OS-level apps included with Android. For the open-soure apps, ANDROIDWORD WORLD maintains constant environment by installing a fixed version of each app. acquired from F-Droid," OS-level apps' versions are determined by the Android OS, which is also fixed. To maintain reproducible environment, ANDROIDWORDD utilizes apps that do not require login/authentication and can store their application data on device.
In addition to managing the states of apps and operating systems, INDROIDWORLD precisely de- fines and controls the state during task execution. Each task has its unique setup, reward deter- mination logic, and teardown procedures (see Appendix D.2 and D.3 for more details), ensuring a fully reproducible suite of tasks.
Automatic task parameterization is a critical mechanism, unique to ANDROIDWORLD, to evaluate agents much larger and more realistic suite of tasks than current benchmarks support. Achiev- ing this requires significantly more effort than randomly generating new task parameters because involves developing evaluation logic that remains valid across different task instantiations. It is exactly through its careful state management that in addition to reproducibility AndroidWorld en- sures that the reward mechanisms function correctly. Task parameters, initialized randomly at the start of each task based on a controlled random seed, dictate the initial state and influence reward outcomes. Similar MiniWoB++ (Shi et al., 2017; Liu al., 2018a), ANDROIDWORKS consists of practically infinite set of varying initial conditions and success criteria.
This approach enables finer- arei-grained analyses of agent adaptability. essential for real-world deploy- ment. Beyond robustness testing, dynamic task construction supports online learning, particularly reinforcement learning (Shi al,, 2017; Liu al, 2018a; Humphreys et al., 2022; Gur at., 2022a), while also streamlining train/test dataset generation for supervised learning (Humphreys et al., 2022; Shaw et al., 2023; Furuta et al., 2023).
3.4 DURABLE REWARDS FROM SYSTEM STATE
ANDROIDWORLD provides reward signals primarily by managing application state using the An- droid Debug Bridge (adb), while also incorporating UI element validation where appropriate. With

---

Published as a conference paper at ICLR 2025
adb, ANDROIDWORLD has complete access to system resources including the file system, ap- plication databases, and system settings. For tasks where system state inspection is impractical, ANDROIDWORKS validates task completion by examining UI elements screen. Determining ward signals from system state has several benefits. is highly accurate because an application's state be quickly inspected and manipulated using the same mechanisms that the app itself uti- lizes. Using the underlying system state is much more durable than matching superficial UI changes. Additionally, it facilitates easy re-use across disparate apps, which tend to use the same underlying caching mechanisms. For instance, logic for checking existence of specific file is used across many unrelated upplications, including those for file management, note-taking, and media playback. For applications leveraging SQLite databases, common pattern, ENDROIDWORLE ators that verify the existence new and deleted rows. Table 2 shows examples of the validators in A AnDrOID W WORLD. See Table 6 for comprehensive list of all tasks in the suite. Table 5 provides selected examples with additional implementation details.
implements evalu-
3.5 TASK COMPOSABILITY
Inferring task success from system state enables accurate, reusable evaluations and simplifies creat- ing composite tasks by combining existing ones. For instance, "Create calendar event with details and text the details to contact" merges two standalone tasks, facilitated by hermetic initialization and success detection. Composite tasks are more challenging due to their complexity but provide partial rewards for subtask completion, aiding hill climbing. The last two rows Table show validation code for composite tasks.
3.6 INTEGRATING MINIWOB++
We implement MiniWoB++ in the ANDROIDWORLD framework and term it MobileMiniWoo++: Each MobileMiniWoB++ task instantiated using the standard ANDROIDWORLD interface, in- heriting from TaskEval base class, and contains methods like initialis settiializeesaar and is ss.success Since MiniWoB++ leverages JavaScript for task configuration and success de- tection, we built WebView app to communicate between Python and the app.
MobileMini Wo+++ introduces modifications both observations and actions compared the orig- inal benchmark. For example, HTML5 <input>> elements rendered with native Android UI wid- gets like the date-picker (see Figure 4), enhancing the realism the tasks. MobileMiniWoBHt uses the same observation space the Android tasks (accessibility tree and screenshot). Notably, it does not include the DOM as in the original implementation. The action space from HEDROIDWORLLS retained. We manually review and test each task ensure they are solvable. We excluded twelve of the original tasks that failed to render correctly on Android, presented compatibility issues with the touch interface, required near real-time interaction, which poses challenges on emulators. Overall, ANDROIDWORLD supports 92 MiniWoB++ tasks. See Appendix C.3 for more details.
4 ANDROIDWORLD AS A COMPUTER-CONTROS BENCHMARK
To test AODRO'DWORLD'S applicability for autonomous agents, we develop and test state-of-the- art agent and variants across all 20 apps and 116 tasks, as well on MobileMiniWoB+4.
4.1 COMPUTER USE AGENTS
4.1.1 M3A
We develop a multimodal autonomous agent for Android, M3A. It is zero-shot, integrating ReAct- style (Yao al., 2022) and Reflexion-style (Shinn et al., 2023) prompting to consume user instruc- tions and screen content, reason, take actions, and update its decision-making based on the outcome of its actions.
In the first stage, M3A generates an action, represented in JSON, and reasoning for that action. To generate this output, the agent is provided with list of available action types, guidelines for operating the phone, and a list of UI elements derived from the Android accessibility tree's leaf nodes. The agent receives the current screenshot and Set-of-Mark (SoM) (Yang et al., 2023a)

---

Published as a conference paper at ICLR 2025
Table Success Rates (SR) on ANDROIDWORLD and MobileMiniWoB++..
Agent
Input
Base model
SRANDKOIWWOLLD
SR SRMabileMaWWoor
Human
N/A
80.0
100.0
SceAct (Zheng et ul., 2024a)
SoM (screen ally tre)
GPT-4 Turbo
15.5
66.1
MBA-Simple
al ly tree
Gemma 2
3.4
35.5
M3A-Simple
al ly tree
Gemini 1.5 Pro
14.7
55.2
M3A-Simple
ul ly tree
GPT-4 Turbo
19.8
67.7
M3A
al ly tree
Gemma 2
9.5
45.6
M3A
al ly tree
Gemini 1.5 Pro
19.4
57.4
M3A
SoM (screen al y tree)
Gemini 1.5 Pro
22.8
40.3
M3A
ally tree
GPT-4 Turbo
30.6
59.7
MBA
SoM (screen al tree)
GPT-4 Turbo
25.4
67.7
annotated screenshot, which includes bounding boxes with numeric labels the top-left corner for each UI element (see screenshot in Figure 5). The agent attempts to execute outputted action by referencing the specific mark (if applicable). In addition the multimodal agent, we have developed a text-only variant that consumes the screen represented using the accessibility tree and selects the relevant action in JSON format.
After executing an action, M3A reflects on its effect by observing any state changes that may have occurred. During this stage, the agent provided with available action types, general operating guidelines. the actual action taken, and its reasoning, sented by UI element representations and screenshots with SoM annotations. We request the agent to provide a concise summary of this step, including the intended action, success or failure, potential reasons for failure, and recommendations for subsequent actions. This summary will serve as action history and be used for future action selection. See Appendix E for more details the agent.
well as before-and-afiee UI states, repre-
In addition to the full agent, we develop M3A-SIMPLE to measure the performance that can be achieved with minimal prompting, without guidelines reflection mechanisms. This helps quantify the impact of more advanced prompting techniques and domain-speciff guidance.
4.1.2 SRRACT BASELINE
We implement baseline agent based on SeeAct (Zheng al., 2024a), which was originally de- signed for GPT-4V for web navigation. Specifically, we implement the best-performing variant, SeeACtehoize. which grounds actions via textual choices. We implement SeeAct for the Android en- vironment to evaluate how existing model that performs well on web tasks (Deng et al., 2023) can be adapted and applied to Android.
To accommodate the Android environment, adapt SceAct in several ways. Firstly, augment the action space from the original SeeAct implementation to support actions needed for mobile, including scroll, long press, navigate home and back, and open app actions. Secondly, in lieu of the DOM, which is not available for Android apps, we utilize the accessibility tree to construct candidate UI actions. Due the lack of the DOM representation, we do not the bespoke ranker model from the original implementation. However, we observe that after applying a filtering heuristic to remove -on-interactable elements, the majority of screens contains less than 50 candidate elements. See Appendix E.6 for more details on the implementation.
4.2 EXPERIMENTAL RESULTS
We evaluate M3A, M3A-SIMPLER and SeeAct on ANDROIDWORLD and MobileMiniWooB+t We set the seed to 30 and the temperature to 0 aid reproducibility. Each task has a maximum allowed number of steps (dctailed in Appendix F), typically set to twice the number of steps needed by human annotators complete the task. We Gemini 1.5 Pro, GPT-4 Turbo, and the open-source Gemma 2 27B (Team et al., 2024) as base models. For MobileMiniWoB+++ we evaluate on a subset of 62 tasks, consistent with recent studies (Zheng et al., 2024c; Kim et al., 2024; Gur et al., 2022b).
Table 3 presents the success rates (SR) for the agents and human performance on both task suites. Although the agents have far from human performance, they demonstrate out-of-the-box capabilities
8

---

Published as a conference paper at ICLR 2025
in operating mobile Uls, exhibiting basic understanding and control capabilities of Uls. They can perform a variety of actions, including long-press, scrolling to search for information, and revising their plan if actions do not work out. The best performance is obtained by M3A when using GPT- 4. On ANDROIDWoRD WORLD the SoM-based variant is less performant, while on MobileMiniWoB-t performs best. A similar result was obtained in recent work on computer agents for desktop applications (Xie et al., 2024). We posit SoM plays a more critical role in MobileMiniWooee tasks due to the often incomplete accessibility tree, compared that of native Android apps.
The simplified agent variant WORLD tasks (19.8% 30.6% with GPT-4), indicating that additional prompting techniques and domain-specifis guidance are beneficial for navigating the complexity of Android interactions. How- ever, on MobileMiniWooB+ tasks, A3A-STMPI, achieves comparable performance (67.7%), sug- gesting that these simpler tasks may not benefit as much from sophisticated prompting strategies. The open-source Gemma model's lower performance (9.5% on ANDROIDWORLDD 45.6% on Mo- bileMinnWoBB++) compared to proprietary models likely stems from its smaller parameter count, though exact comparisons are difficult as the parameter counts for GPT-4 and Gemini are not public.
shows significant performance drop on ANDROID-
4.3 ANALYSIS
Agents have difficulty understanding mobile UIs, often failing to detect visual cues that are essential for task completion (see Figure 6a). Additionally, agents struggle with certain UI patterns and affordances, and when they make reasoning mistakes (see Figure 6b), they often lack the capability to explore and adapt as humans do (see Figure 6c). Moreover, agents sometimes struggle with tasks that simply involve confirming system states, e.g., confirming the WiFi is turned on, suggesting challenges in both task and screen understanding.
The agents struggle with grounding, particularly when executing precise interactions, such as manip- ulating text (see Figure 7) or operating sliders, and they are often unable to recover from mistyping errors. In addition, for tasks that demand memory, such performing transcriptions across apps, multiplying numbers, or scrolling, the agents struggle as they are unable to "remember" content.
SceAct performs less effectively than M3A on the ANDROIDWORLD task suite and similarly on MobileMiniWoo++. reflecting its optimization for web rather than mobile environments. It strug- gles with mobile-specifi actions like long-pro peegspressssseee and swipes, and often fails select appropriate actions due to not incorporating screen elements during action generation. Memory-intensive tasks are particularly challenging, as SeeAct only caches actions without remembering outcomes, leading to repetitive, ineffective behaviors such as endless scrolling. This lack of quick eπror recovery often results in task termination once maximum steps are reached.
Finally, we note that large foundation models significantly increase latency, taking three times longer than humans on average to complete tasks. On average, M3A takes 3.9 minutes complete a task, with the text-only version taking 2.5 minutes.
5 ROBUSTNESS ANALYSIS
To understand agent robustness, we analyze M3A's performance across different random seeds, which generate different task parameters (e.g., calendar appointments, expense categories) and can consequently require different UI interaction patterns (e.g., scrolling to access hidden elements, handling varying numbers of elements modify, or adapting to different input types and lengths). Across three seeds, we observe significant performance variations: 27.6%, 26.3% and 33.2% (mean 29.0%), obtained using M3A with GPT-4 Turbo with accessibility trees as input. Note that for consistency with existing literature we maintain the single-seed results in Table 3.
To better understand the sources of this variability, we evaluate agent robustness under two condi- tions: (1) identical tasks with the same parameters and (2) tasks with different parameter combina- tions, which change the initial state and task definition. We perform this analysis on a representa- tive subset of ANDROIDWORLLE tasks that span different interaction patterns and complexity levels (listed in Appendix E.4). Due to computational constraints, we conduct 20 trials for each task using our strongest agent configuration M using the accessibility tree and GPT-4.

---

Published as a conference paper at ICLR 2025
100
Same seed Different seeds
80
60
Success
40
20
AddExpense
DeleteFile
EditNote
DeleteEvent
CountActivites
Figure 3: Success rate variation across tasks due to the parametrization built into ANDROID Using fixed seed, the agent appears completely incapable of solving some tasks due to "bad luck" with the seed. In contrast, under different task parameterizations, we observe the agent can solve the tasks fairly often. Wilson binomial proportion confidence intervals (95%) are shown for the different seed group (orange) and the same seed group (blue). The different seed group has higher variance than the same seed group. Significant differences, with p-value 0.05, are indicated by "g"
Figure shows our results. With a constant seed, the agent fails on add and edit tasks and rarely solves delete tasks, primarily due to UI operation challenges. Surprisingly, performance varies even with a fixed seed, suggesting model non-determinism affects reliability. Performance varies significantly more with different seeds, with statistically significant differences for add expense and edit note tasks. The high intra-task variation indicates the model's sensitivity to task parameters. Section E.5 provides analysis how specific parameter variations impact agent performance.
This sensitivity aligns with observations in RL research (Henderson et al., 2018; Raffin et al., 2021; Colas al., 2018), suggesting performance best represented by the mean across seeds. We believe ANDROID WORLD'S 's support for such analysis will become increasingly valuable as more efficient models are developed. Finally, we note the observation of non-zere rewards under some seeds points to potential enhancements through RL-like mechanisms in future work.
To assess AndroidWorld's robustness to OS variations, we tested on Pixel 5 (Android 12) alongside our primary setup (Pixel 6, Android 13). The agent achieved 28.4% success rate, with performance variations akin those from random seed changes, suggesting it maintained its capabilities despite differing UI layouts and device types.
These experiments underscore the importance testing agents under varied conditions, a capability that ANDROIDW ORR.D ORLD effectively supports.
CONCLUSION
We introduced ANDROIDWORLD WOLDD realistic and robust agent environment for Android that enables the development and evaluation of autonomous agents across a wide range tasks and apps. AN- DROIDWORLD provides reproducible task suite consisting of 116 tasks across 20 apps, with each task dynamically generated using random parameters to challenge agents with millions of unique goals. By releasing ANDROIDWORLD and establishing benchmark performance with M3A, we aim to accelerate research and development in this area, ultimately leading to the creation of com- puter use agents capable of operating effectively real-world environments. Further, the dynamic nature of ANDROIDW WORLD WORLD opens up new research opportunities for online learning algorithms in computer use agents.
REFERENCES
Josh Abramson, Arun Ahuja, Federico Carnvvale, Petko Georgiev, Alex Goldin, Alden Hung, Jes- sica Landon, Timothy Lillicrap, Alistair Muldal, Blake Richards, Adam Santoro, Tamara von
10

---

Published as a conference paper at ICLR 2025
Glehn, Greg Wayne, Nathaniel Wong, and Chen Yan. Evaluating multimodal interactive agents, 2022.
William E Bishop, Alice Li, Christopher Rawles, and Oriana Riva. Latent state estimation helps ui agents to reason, 2024.
Rogerio Bonatti, Dan Zhao, Francesco Bonacci, Dillon Dupont, Sara Abdali, Yinheng Li, Yadong Lu, Justin Wagle, Kazuhito Koishida, Arthur Bucker, Lawrence Jang, and Zack Hui. Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale, 2024. URL sttps://arxiv. org/abs/ 2409. 08264.
Andrea Burns, Deniz Arsan, Sanjna Agrawal, Ranjitha Kumar, Kate Saenko, and Bryan A. Plum- mer. Mobile app tasks with iterative feedback (motif): Addressing task feasibility in interac- tive visual environments. CoRR, abs/2104.08566) 2021. URL https:://axte org/ /bs/ 2104. 08560.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fo- tios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino. Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji. Shantanu Jain, William Saun- ders, Christopher Hesse, Andrew N Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code. July 2021.
Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiy- ong Wu. Seeclick: Harnessing gui grounding for advanced visual gui agents. arXiv preprint arXiv:2401.10935 2024.
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng. Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E Gonzalez, al. Chathot arena: An open platform for evaluating llms by human preference. arXiv preprint a3Xiv:240..04132, 2024.
Cédric Colas, Olivier Sigaud, and Pierre- Yves Oudeyer. How many random seeds? statistical power analysis in deep reinforcement learning experiments. arXiv preprint arXiv: (/XX::/8006082955 2018.
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2Web: Towards a generalist agent for the web, 2023.
Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H Laradji, Manuel Del Verme, Tom Marty, Léo Boisvert, Megh Thakkar, Quentin Cappart, David Vazquez, al. Workarena: How capable are web agents at solving common knowledge work tasks? arXiv preprint arXiv:2403.077116 2024.
Yuqing Du, Ksenia Konyushkova, Misha Denil, Akhil Raju, Jessica Landon, Felix Hill, Nando de Freitas, and Serkan Cabi. Vision-Language models success detectors. March 2023.
Nicholas Farn and Richard Shin. Tooltalk: Evaluating tool-usage in conversational setting. arXiv preprint arxiv:2311. 10775, 2023.
Hiroki Furuta, Kuang-Huel Lee, Ofir Nachum, Yutaka Matsuo, Aleksandra Faust, Shixiang Shane Gu, and Izzeddin Gur. Multimodal web navigation with Fineeted foundation models. May 2023.
Difei Gao, Lei Ji, Zechen Bai, Mingyu Ouyang, Peiran Li, Dongxing Mao, Qinchen Wu, Weichen Zhang, Peiyi Wang, Xiangwu Guo, et al. Assistgui: Task-oriented desktop graphical user interface automation. arXiv preprint arXiv:2312.13108 2023.
Gemini. Gemini: A family of highly capable multimodal models, 2023.

---

Published as a conference paper at ICLR 2025
Significant Gravitas. AutoGPT. https//agpt..co, 2023. https://agpt.
Izzeddin Gur, Natasha Jaques, Yingjie Miao, Jongwook Choi, Manoj Tiwari, Honglak Lee, and Aleksandra Faust. Environment generation for zero-shot compositional reinforcement learning. 2022a.
Izzeddin Gur. Ofir Nachum, Yingjie Miao, Mustafa Safdari, Austin Huang, Aakanksha Chowdhery, Sharan Narang, Noah Fiedel, and Aleksandra Faust. Understanding html with large language models. arXiv preprint urXiv:2210.03995, 2022b.
Hongliang Hc, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. Webvoyager: Building an end-to-end web agent with large multimodal models. arXiv preprint arXiv:2401.33911 2024.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Wenyi Hong, Weihan Wang. Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang. Zihan Wang, Yuxiao Dong, Ming Ding, al. Cogagent: A visual language model for gui agents. arXiv preprint arXiv:2312.08911 2023.
Peter C Humphreys, David Raposo, Tobias Pohlen, Gregory Thornton, Rachita Chhaparia, Alistair Muldal, Josh Abramson, Petko Georgiev, Adam Santoro, and Timothy Lillicrap. A data-driven approach for learning to control computers. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Con- ference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 9466-9482. PMLR, 17-23 Jul 2022. URL https: /ate::/prrooeee ppwocoooceeiccoeeeooooooooooeece .gs.mlrr........ng......mmm....m www.mmll.pprr.r www.ssrssvvlll vv16// 6 humphreys22a. html.
Raghav Kapoor, Yash Parag Butala, Melisa Russak, Jing Yu Koh, Kiran Kamble, Waseem Alshikh, and Ruslan Salakhutdinov. Omniact: dataset and benchmark for enabling multimodal generalist autonomous agents for desktop and web. arXiv preprint 10Xiv:2402.17553, 2024.
Geunwoo Kim, Pierre Baldi, and Stephen McAleer. Language models can solve computer tasks. Advances in Neural Information Processing Systems, 36, 2024.
Megan Kinniment, Lucas Jun Koba Sato, Haoxing Du, Brian Goodrich, Max Hasin, Lawrence Chan, Luke Harold Miles, Tao R Lin, Hjalmar Wijk, Joel Burget, et al. Evaluating language-model agents realistic autonomous tasks. arXiv preprint mrXiv:2312.11671 2023.
Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks. arXiv preprint arXiv:2401.13649 2024.
Hanyu Lai, Xiao Liu, lat Long long, Shuntian Yao, Yuxuan Chen, Pengho Shen, Hao Yu, Hanchen Zhang, Xiaohan Zhang, Yuxiao Dong, et al. Autowebglm: Bootstrap and reinforce large lan- guage madel-based web navigating agent. arXiv preprint arXiv:2404.03645 2024.
Juyong Lee, Taywon Min, Minyong An, Changyeon Kim, and Kimin Lee. Benchmarking mo- bile device control agents across diverse configurations. In ICLR 2024 Workshop on Generative Models for Decision Making, 2024.
Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang. and Yongbin Li. API-Bank: A comprehensive benchmark for Togl-Augeetted LLMs. April 2023a.
Tao Li, Gang Li, Zhiwei Deng, Bryan Wang, and Yang Li. A Zero-Shot language agent for computer control with structured reflection. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 11261-11274 Singapore, December 2023b. Association for Computational Linguistics.
12

---

Published as a conference paper at ICLR 2025
Wei Li, William Bishop, Alice Li, Chris Rawles, Folawiyo Campbell-Ajala, Divya Tyamagundlu, and Oriana Riva. On the effects of data scale on computer control agents. In Advances in Neural Information Processing Systems (NeurIPS 2024), 2024. URL https://aaxtti .rrg/ / 2406. 03679.
Yang Li, Jiacong He. Xin Zhou, Yuan Zhang, and Jason Baldridge. Mapping natural language instructions to mobile UI action sequences. In Proc. of the 58th Annual Meeting of the Associa- tion for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pp. 8198-8210. Associa- tion for Computational Linguistics, 2020. URL https: www. ac www.wwweb...... wtt..boooo/aalgggg////ooo 2020. acl maiin..e 729/.
Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, and Percy Liang. Reinforcement learning web interfaces using workflow-guided exploration. In 6th International Conference on Learn- ing Reprexentations (ICLR '88, 2018a. URL https:/ /openreviewwweeeeeee net forum?id- ΣYТp3f-0-
Thomas Liu, Mark Craft, Jason Situ, Ersin Yumer, Radomir Mech, and Ranjitha Kumar. Learning design semanties for mobile apps. In Proc. of the 31 Annual ACM Symposium User Interface Software and Technology, UIST 18, pp. 569-579, New York. NY. USA, 2018b. Association for Computing Machinery. ISBN 978 450039981. doi: 04L145//244558.322266656 URL https:
(/doi.org//il 11 11 3212522225883 3242650.
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-Eval: NLG evaluation using GPT-4 with better human alignment, 2023.
Xing Han Lå, Zdenčk Kasner, and Siva Reddy. WebLINX: Real-World website navigation with Multi-Turn dialogue. February 2024.
Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayara- man, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Eureka: Human-level reward design via coding large language models, 2023.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36, 2024.
Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. GAIA: a benchmark for general Al assistants. November 2023.
Volodymyr Mnih, Koray Kavukcuoglu. David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning, 2013.
Shikhar Murty, Christopher Manning, Peter Shaw, Mandar Joshi, and Kenton Lec, BAGEL: Boot- strapping agents by guiding exploration with language, 2024. URL https://arxiv.cr abs/2403 08140.
OpenAl. GPT-4 technical report, 2023.
Jiayi Pan, Yichi Zhang, Nicholas Tomlin, Yifei Zhou, Sergey Levine, and Alane Suhr. Autonomous evaluation and refinement of digital agents. April 2024.
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dor- mann. Stable-baselines:: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268): 2021.
Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. Android in the wild: large-scale dataset for android device control. arXiv preprint arXiv:2307. 10088, 2023.
Yangjun Ruan, Honghua Dong, Andrew Wang, Silviu Pitis, Yongchao Zhou, Jimmy Ba, Yann Dubois, Chris Maddison, and Tatsunori Hashimoto. Identifying the risks of LM agents with an LM-Emulated sandbox. September 2023.
13

---

Published as a conference paper at ICLR 2025
Peter Shaw, Mandar Joshi, James Cohan, Jonathan Berant, Panupong Pasupat, Hexiang Hu, Urvashi Khandelwal, Kenton Lee, and Kristina Toutanova, From pixels to UI actions: Learning to follow instructions via graphical user interfaces. May 2023.
Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hermandez, and Perey Liang. World of bits: An open-domain platform for web-based agents. In Doina Precup and Yee Whye Teh (eds.), Proc. of the 34th International Conference on Muchine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 3135-3144, PMLR, 06-11 Aug 2017. URL htLp: httt::://proceedingg.
mlr. malr.pressv/i shil 7a. html.
Noah Shimn, Beck Labash, and Ashwin Gopinath. Reflexion: autonomous agent with dynamic memory and self-reflection. arXiv preprint arXiv:2303. 1366, 2023.
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks, 2020.
Maayan Shvo, Zhiming Hu, Rodrigo Toro Icarte, Iqbal Mohomed, Allan Jepson, and Sheila A. Mcllraith. Appbuddy: Learning to accomplish tasks in mobile apps via reinforcement learning, 2021. URL hwtpt:tt//s/ee.t.m //arxiv .rgg abs / 2106.00 00133.
David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schritwieser,, Joannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, llya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the Game of Go with Deep Neural Networks and Tree Search. Nature, 529(7587):484 489, January 2016. doi: 10.1038/nature16661
Weihao Tan, Ziluo Ding, Wentao Zhang, Boyu Li, Bohan Zhou, Junpeng Yue, Haochong Xia, Jiechuan Jiang, Longtao Zheng, Xinrun Xu, Yifei Bi. Pengjie Gu, Xinrun Wang, Börje F. Karls- son, Bo An, and Zongqing Lu. Towards General Computer Control: A Multimodal Agent For Red Dead Redemption II As A Case Study. arXiv preprint arXiv:2403.03//6, 2024.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy Lillicrap, and Martin Ried- miller. DeepMind control suite. January 2018
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhu- patiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Fer- ret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Char- line Le Lan, Sammy Jerome, Anton Tsitsulin, Nino Vieillard, Piotr Stanczyk, Sertan Girgin, Nikola Momchev, Matt Hoffman, Shantanu Thakoor, Jean-Bastien Grill, Behnam Neyshabur, Olivier Bachem, Alanna Walton, Aliaksei Severyn, Alicia Parrish, Aliya Ahmad, Allen Hutchi- son, Alvin Abdagic, Amanda Carl, Amy Shen, Andy Brock, Andy Coenen, Anthony Laforge, Antonia Paterson, Ben Bastian, Bilal Piot, Bo Wu, Brandon Royal, Charlie Chen, Chintu Kumar, Chris Perry, Chris Welty, Christopher A. Choquette-Choo, Danila Sinopalnikov, David Wein- berger, Dimple Vijaykumar, Dominika Rogozińska, Dustin Herbison, Elisa Bandy, Emma Wang, Eric Noland, Erica Moreira, Evan Senter, Evgenii Eltyshev, Francesco Visin, Gabriel Rasskin, Gary Wei, Glenn Cameron, Gus Martins, Hadi Hashemi, Hanna Klimczak-Plucifikk. Harleen Batra, Harsh Dhand, Ivan Nardini, Jacinda Mein, Jack Zhou, James Svensson, Jeff Stanway, Jetha Chan, Jin Peng Zhou, Joana Carrasqueira, Joana Iljazi, Jocelyn Becker, Joe Fernandez, Joost van Amersfoort, Josh Gordon, Josh Lipschultz. Josh Newlan, Ju yeong Ji, Kareem Mohamed, Kar- tikeya Badola, Kat Black, Katie Millican, Keelin McDonell, Kelvin Nguyen, Kiranbir Sodhia, Kish Greene, Lars Lowe Sjoesund, Lauren Usui, Laurent Sifre, Lena Heuermann, Leticia Lago, Lilly McNealus, Livio Baldini Soares, Logan Kilpatrick, Lucas Dixon, Luciano Martins, Machel Reid, Manvinder Singh, Mark Iverson, Martin Görner, Mat Velloso, Mateo Wirth, Matt Davidow, Matt Miller, Matthew Rahtz, Matthew Watson, Meg Risdal, Mehran Kazemi, Michael Moyni- han, Ming Zhang, Minsuk Kahng, Minwoo Park, Mofi Rahman, Mohit Khatwani, Natalie Dao, Nenshad Bardoliwalla. Nesh Devanathan, Neta Dumai, Nilay Chauhan, Oscar Wahltinez, Pankil Botarda, Parker Barnes, Paul Barham, Paul Michel, Pengchong Jin, Petko Georgiev, Phil Culli- ton, Pradeep Kuppala, Ramona Comaneseu, Ramona Merhej, Reena Jana, Reza Ardeshir Rokni,
14

---

Published as a conference paper at ICLR 2025
Rishabh Agarwal, Ryan Mullins, Samaneh Saadat, Sara Mc Carthy, Sarah Cogan, Sarah Perrin, Sébastien M. R. Amold, Sebastian Krause, Shengyang Dai, Shruti Garg, Shruti Sheth, Sue Ron- strom, Susan Chan, Timothy Jordan, Ting Yu, Tom Eccles, Tom Hennigan, Tomas Kocisky, Tulsee Doshi, Vihan Jain, Vikas Yadav, Vilobh Meshram, Vishal Dharmadhikari, Warren Barkley, Wei Wei, Wenming Ye, Woohyun Han, Woosuk Kwon, Xiang Xu, Zhe Shen, Zhitao Gong, Zichuan Wei. Victor Cotruta, Phoebe Kirk, Anand Rao, Minh Giang, Ludovic Peran, Tris Warkentin, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, D. Sculley, Jeanine Banks, Anca Dra- gan, Slav Petrov, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Fara- bet, Elena Buchatskaya, Sebastian Borgeaud, Noah Fiedel, Armand Joulin, Kathleen Kenealy, Robert Dadashi, and Alek Andreev. Gemma 2: Improving open language models at practical size, 2024. URL https: /ette///xxxive /ttp//aawivv.g////////// wwtp:///gg////o///oooooooosss//s 2408 00118.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint 2xXi:2337.002888 2023.
Daniel Toyama, Philippe Hamel, Anita Gergely, Gheorghe Comanici, Amelia Glaese, Zafarali Ahmed, Tyler Jackson, Shibl Mourad, and Doina Precup. Androidenv: A reinforcement learning platform for android, 2021. URL https: / org/ abs
Sagar Gubbi Venkatesh, Partha Talukdar, and Srini Narayanan. Ugif: Ui grounded instruction fol- /
lowing, 2022. /RL/ittoee////aarrr.rrra///////1
Oriol Vinyals, Igor Babuschkin, Wojeiech M Czamecki, Michaël Mathieu, Andrew Dudzik, Juny- oung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang. Laurent Sifre. Trevor Cai, John P Agapiou, Max Jaderberg, Alexander S Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom L Paine, Caglar Gulcehre, Ziyu Wang, Tobias Pfaff, Yuhuai Wu, Roman Ring, Dani Yogatama, Dario Wünsch, Katrina McKinney, Oliver Smith, Tom Schaul, Timothy Lillicrap, Koray Kavukcuoglu, Demis Hassabis, Chris Apps. and David Silver. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature, 575(7882): 350-354, November 2019.
Bryan Wang, Gang Li, and Yang Li. Enabling conversational interaction with mobile ui using large language models. In Proc. of the 2023 CHI Conference on Human Factors in Computing Systems, CHI '23, Association for Computing Machinery, 2023a. ISBN 978 9781450394215. doi: 10.1145/354454883558000 URL https:/:/// /oi. org 10. 111115/344444 3585544343888888 3580895.
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291, 2023b.
Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W White, Doug Burger, and Chi Wang. Autogen: Enabling next-gen Ilm applications via multi-agent conversation framework. 2023.
Zhiyong Wu, Chengeheng Han, Zichen Ding, Zhenmin Weng, Zhoumianze Liu, Shunyu Yao, Tao Yu, and Lingpeng Kong. Os-copilot: Towards generalist computer agents with seproimmpoveeennt arXiv preprint arXiv:2402.07456 2024.
Tianbao Xie, Fan Zhou, Zhoujun Cheng, Peng Shi, Luoxuan Weng, Yitao Liu, Toh Jing Hua, Jun- ning Zhao, Qian Liu, Che Liu, Leo Z. Liu, Yiheng Xu, Hongjin Su, Dongchan Shin, Caiming Xiong, and Tao Yu. Openagents: An open platform for language agents in the wild, 2023.
Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li. Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. arXiv preprint arXiv:2404.0797. 2024.
Mingzhe Xing, Rongkai Zhang, Hui Xue, Qi Chen, Fan Yang. and Zhen Xiao. Understanding the weakness of large language model agents within complex android environment. In Proceedings
15

---

Published as a conference paper at ICLR 2025
of the 30th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pp. 6061- 6072, 2024.
Qiantong Xu, Fenglu Hong, Bo Li, Changran Hu, Zhengyu Chen, and Jian Zhang. On the tool manipulation capability of open-source large language models. May 2023.
An Yan, Zhengyuan Yang, Wanrong Zhu, Kevin Lin, Linjie Li, Jianfeng Wang. Jianwei Yang. Yiwu Zhong, Julian McAuley, Jianfeng Gao, Zicheng Liu, and Lijuan Wang. GPT-4V in wonderland: Large multimodal models for Zero-Shot smartphone GUI navigation. November 2023.
Jianwei Yang. Hao Zhang, Feng Li. Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v, arXiv preprint arXiv:2310.
2023a.
Zhao Yang, Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. Appagent: Multimodal agents as smartphone users. arXiv preprint arXiv:2312.13771, 2023b.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. October 2022.
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable
real-world web interaction with grounded language agents, 2023.
Keen You, Haotian Zhang, Eldon Schoop, Floris Weers, Amanda Sweargin, Jeffirey Nichols, Yinfei Yang, and Zhe Gan. Ferret-ui: Grounded mobile ui understanding with multimodal llms, 2024. URL /aattss:/aazxte
org/ abs/ 2404. 05719.
Chaoyun Zhang, Liqun Li, Shilin He, Xu Zhang, Bo Qiao, Si Qin, Minghua Ma, Yu Kang, Qingwei Lin, Saravan Rajmohan, et al. Ufo: ui-focused agent for windows os interaction. arXiv preprint arXiv:2402.07939, 2024a.
Danyang Zhang, Zhennan Shen, Rui Xie, Situo Zhang, Tianbao Xie, Zihan Zhao, Siyuan Chen, Lu Chen, Hongshen Xu, Ruisheng Cao, and Kai Yu. Mobile-env: Building qualified evalua- tion benchmarks for Ilm-gui interaction, 2024b. URL https:://aaatt wrw. ab/////s///5 08144.
Jiwen Zhang, Jihao Wu, Yihua Teng, Minghui Liao, Nuo Xu, Xiao Xiao, Zhongyu Wei, and Duyu Tang. Android in the zoo: Chain-of-action-thouuht for gui agents. arXiv preprint arXiv:2403.02713. 2024c.
Li Zhang, Shihe Wang, Xianqing Jia, Zhihan Zheng, Yunhe Yan, Longxi Gao, Yuanchun Li, and Mengwei Xu. Llamatouch: A faithful and scalable testbed for mobile ui task automation, 2024d. URL https:ttp/// https:://axxvvvoo org/ abs/2404 16054.
Zhuosheng Zhang and Aston Zhang. You only look at screens: Multimodal chain-of-action agents, 2023.
Zhuosheng Zhang, Yao Yao, Aston Zhang, Xiangru Tang, Xinbei Ma. Zhiwei He, Yiming Wang, Mark Gerstein, Rui Wang, Gongshen Liu, et al. Igniting language intelligence: The hitchhiker's guide from ofain-of-thought reasoning to language agents. arXiv preprint 1..1:2311.11197,, 2023.
Ziniu Zhang. Shulin Tian, Liangyu Chen, and Ziwei Liu. MMInA: Benchmarking multihop multi- modal internet agents. arXiv preprint arXiv:2404, 09992, 2024e.
Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-4v(ision) is a generalist web agent, if grounded. arXiv preprint urXiv:2401.01614 2024a.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E Gonzalez, and lon Stoica. Judging L.LM-as-a-Judge with MT-Bench and chatbot arena. June 2023.
Longtao Zheng, Zhiyuan Huang. Zhenghai Xue, Xinrun Wang. Bo An, and Shuicheng Yan. Agentstudio: A toolkit for building general virtual agents, 2024b. URL https: //rxiv.. hte/abb/2400: 17918.
16

---

Published as a conference paper at ICLR 2025
Longtao Zheng, Rundong Wang, Xinrun Wang. and Bo An. Synapse: Trajectory-as- exeemlaa prompting with memory for computer control, 2024c.
Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Webarena: A realistic web environment for building autonomous agents, 2023.
APPENDIX A LIMITATIONS
ANDROIDWORLD currently supports open-source Android apps (> 1M downloads) and built-in sys- tem apps. While testing on trending apps would be desirable, we found open-source apps often present harder challenges due to their less-optimized Uls. Popular apps typically offer more short- cuts and UI affordances, while open-source apps may require more complex interaction patterns. For example, in Figure 6c, the agent fails by repeatedly searching for non-existent "delete-all" button instead of recognizing the need to delete notes individually.
APPENDIX B ETHICAL CONSIDERATIONS
There is risk that malicious actors could engineer agents to bypass security mea- sures like CAPTCHAs or engage in activities like spamming. Additionally, they could alter prompts or screen outputs further harmful objectives.
Malicious use
Societal impact Automation agents may transform societal norms, disrupt employment, and mod- ify human behavior. While they can enhance efficiency, this improvement could pose risks ex- ploited by malevolent forces.
APPENDIX C ANDROIDWORLD ENVIRONMENT
C.I OBSERVATION SPACE
In ANDROIDWORLD, the Android screen is represented using State class, which includes the following attributes:
Pixels: An RGB array representing the current screen capture of the device. The screenshot resolution is 2400 x 1080 x 3.
Accessibility tree: raw representation of the accessibility tree.* This UI tree provides detailed snapshot of all UI elements currently displayed on the screen. We utilize an acces- sibility forwarding app from AndroidEnv (Toyama et al., 2021), which leverages gRPC to transmit the accessibility tree data efficiently to the device.
UI elements: A list of processed UI elements extracted from the children of the accessibil- ity tree. Each UIEl ement contains attributes such text, content description, bounding boxes, and various state flags (e.g., clickable, scrollable, focused).
Since Android observations and actions are asynchronous, changes resulting from actions may take some time to manifest. Therefore, instead of using an RL-based interface, which assumes a tight coupling between actions and observations, we design an interface for the agent tailored for asyn- chronous interaction. This interface implements get_state method responsible for capturing the current state of the environment, typically after executing action. This method includes an optional wa sitt.s t.too... t.t.ttstaabbbot..... wt..aiallli.. lie ze flag, which, when enabled, employs heuristics to ensure the UT el- ements not in transient state, thus providing stable and accurate snapshot of the environment.
'Represented using all current windows; https::/dest http::///eeeloper..aaa. andrra wtw.r.androdd.comm wttpr/oooorrii.ccc//eeeeeeer.eee androite/iiew/accsssiiiii
17

---

Published as a conference paper at ICLR 2025
C.2
ACTION SPACE
Actions are stored using Python dataclass and executed using adb. The action space includes:
Direct UT Actions:
Clsck actions (click, long press): Simulates touch events at specified coordi- nates
Text input: Simulates typing in focused text fields
Navigation: Sends home/back key events
Scrolling: Executes swipes in four directions (up, down, left, right)
App launching: Starts specified applications
Task Management Actions:
Status: Reports if task is in in-progress, complete, infeasible
Answer: Provides responses, which are needed for information retrieval tasks
System Actions:
Wait: No-op useful for loading screens and UI transitions
Unknown: No-op for handling internal errors
ACTIOILTYPES
Manigul at "CLICK": "ul cilikk",
"SCROLL*: "screlin,
"INPOT "EX""; "ingut SASFIGAT_ECNE". Fravigalla_
"NAVTGATF: RaCK"; fravigee previgate back*, "KEYBOARD_ENTTER "keyboarri wnter"
"oPENLAPE"; "open_aps"
DNG PRESS"; "long press".
11
12 Concrol Plow
13
Reports completion
"WAI?"; "aail",
Bandlex Lzansition iona
15
"ANSMER"; *anewer",
For intormation retrieval tasks
16
"UNINION"; "unknown"
for internel arrors
18
19
2dw:ael aclssess asses.detann detaclass allaasss...
20
class J50tikl lon:
**"Represents parsed
23
Exemple
24
result json l'action typef; falick', fxfs 4d, fy'r Rdi JSONAct Contexreull
26
27
Actributes:
28
The act lon
index to click, action click. an <x, y> should be provided. See x, y astributes belov,
30
31
x: The it to click, if the action is a click.
32
position click, the action is click. type, sirection: Tae direction to scroll, if action scroll.
type.
34
35
goal_statre It the status is "statue" type, indicates the status or the gosl. launch, the ior is apenneep"..
36
The
38
action action_type: type: str
19
Index: int Kone
40
ys
12
text: stz Hene
43
disection: stz None
44 poatssaatss: None
apo_name:
Listing 1: Pseudo-code representation of the action space.
In addition to the Ul-based action space described above, AndroidWorld provides set of high- level APIs for direct device interaction (i.e., sending SMS messages, opening web pages, managing contacts). While the core action space focuses on fundamental UI control capabilities, these supple- mentary APIs found in env/too wtt.soolss..oooooooooo..oooooltl .py enable future research into hybrid interaction approaches that combine both Ul-based and programmatic device control.
18

---

Published as a conference paper at ICLR 2025
aaracooti scith anst
2013
Mon, Dec 18
Desedber
Figure 4: Native Android UI widget rendering for HTML5 <input> element.
C.3 MOBILEMINWWOB++
Authors manually completed all tasks in MobileMiniWoo++, implemented as a Web app. to verify solvability on mobile interface. MobileMiniWoo++ differs from MiniWoB++ due to the touch-based interface, which required different approaches for certain tasks. For instance, highlight- ing text from the highlight-tex tasks involves using Android's long-puo long-press and cursor-moving functionalities. HTML5 <input> elements are natively rendered with native Android UI widgets like the date-picker (see Figure 4).
Our implementation of MiniWoB++ contains 92 tasks in total. We exclude the following tasks: chase-ciroee (requires near-realtime movement, unachievable by humans on emulators), moving-items (too hard to click in emulator), drag- cube (drags will scroll the screen, moving the task out view), drag-items seseams--riid (elements are not interactable on Android), drag-items (elements not interactable Android), drag-shapes (drags will scroll the screen, moving the task out view), drag-sort--uubber Android), text-editon (cannot underline everything, weird glitch), number-cher cereecbbxxess (not correctly rendered: only three columns), use-sll der-22 (slider implementation not work- ing), use-spinnt (slider implementation not working), and click-menu (the menu respon- siveness breaks and the task does not behave as intended).
(elements are not interactable on
APPENDIX D ANDROIDWORLD BENCHMARK DETAILS
D. APP SELECTION
Our selection of apps (summarized in Table 4) guided by three main factors: case, popular- ity, and the need for consistency and reproducibility.
Use case and categories We analyzed popular app categories in app stores, focusing on produc- tivity, communication, and multimedia. Selected apps had to meet criteria such not requiring login and storing data locally on the device. Additionally, we considered apps from categories that the authors commonly used, ensuring the selection was representative of real-world Android usage.
Popularity We used download statistics from the Google Play Store to gauge app popularity, lecting apps with over million downloads. Most of the selected apps exceeded this threshold. Less popular apps were also included if they featured common UI patterns and affordances, ensuring they are indicative of typical Android app usage. For instance. Simple Calendar Pro, though less downloaded, has UI comparable the widely-used Google Calendar app.
19

---

Published as a conference paper at ICLR 2025
Table List of ANDROIDWORLD apps and number of tasks for each one.
App name
Description
tasks
Simple Calendar Pro
calendar app for creating, deleting, and managing events and appoint- ments.
17
Settings
The Android system settings app for managing device settings such as Bluctooth, Wi-Fi, and brightness.
15
Markor
A note-taking app for creating, editing. deleting, and managing notes and folders.
14
Broccoli Recipe App
A recipe management app for adding, deleting, and organizing recipes. An expense tracking app for adding, deleting, and managing expenses. SMS app sending, replying to, and resending text messages. sport tracking app for recording and analyzing activities, durations, and distances.
13
Pro Expense
Simple SMS Messenger
Open'rracs
Tasks
A task management upp for tracking tasks, due dates, and priorities.
6
Clock
An app with stopwatch and timer functionality.
4
Joplin
A note-taking app.
4
Retro Music
A music player app.
4
Simple Gallery Pro
An app for viewing images.
4
Camera
app for taking photos and videos.
3
Chrome
web browser app.
Contacts
An app for managing contact information.
OsmAnd
A maps and navigation app with support for adding location markers,
3
favorites, and saving tracks.
VLC
A media player app for playing media files.
3
Audio Recorder
An app for recording and saving audio clips.
2
Files
A file manager app for the Android filesystem, used for deleting and moving files.
2
Simple Draw Pro
drawing app for creating and saving drawings.
Consistency and reproducibility All apps were sourced from F-Droid, open-source Android app repository. This allowed us to manage app versions precisely by selecting and distributing specific APKs. We use the newest version of each app at the time of download.
D.2 TASK CLASSIFICATION AND GENERATION
We categorize tasks into two types: those with side-effects and those without. Tasks with sidc-effeects are those that modify the internal state the device or applications, such as turming off Wi-Fi or creating calendar event. These tasks are implemented as distinct Python classes, each with its own parameter generation, initialization, evaluation, and teardown methods.
Below show an example of the task evaluation for SendSms task, which involves sending and validating a text message. The pscudocode illustrates the task initialization, success check, and parameter generation methods. Each task has its random parameter generation method and success logic.
elass Bendins (ZaskEval): wengwok sending and validet ing Lext mmssagg has been aent.
In chocks the tolophony database, shich is locatad ac:
adata/date/ cum.andreidd www.eaadddddd.diiiiii. providera, Lelpphony/ bouuun/ddaa abasssassatssaas abases httpw/////mmmssemm di,
tergl
"und Lext measage using Simple ENC Hessenger Lu [nunbez] with nessages [massagu)*
10
11
12
matiializo_takkeee envs interface.Aayaeee >> Nones aeegets up the initial state of the teak, super (1 iinit Lask
13
14
15
ane
16
17.
del Iaaanoossaa Cul (welf, savt Casscfaae.Aayyceae floaes ae teschecke if the SHS was sent successful super suaunossafll messages get get_ansss (env.base_env) return cbeck ne spage exi sxists!
18
19
20
21
22
chhone_numbereee perans ("munber")), body=selt. parane poo--selt.paooe ""ecssge"),
20

---

Published as a conference paper at ICLR 2025
26
def teardowaisels eav: intarfacs .AyyncBnv) +> None:
27
eeccic the SNS databors. *n*
28
super 12 beardown
ciear_sns_dattttee bavebbase_anv)
30
31
Bclassmethed
32
ctspss__oorooooparssccaaaaa dictistt,, number Sanorata_iandoonneaaa
36
messags
35
seturn
36
"nunbes";
*massacu": nessaqe,
38
D.3 INFORMATION RETRIEVAL TASKS
Tasks without side-effects are Information Retrieval tasks, requiring the agent to answer a question based on the device or app's current state. For these tasks, instead of Python class, we create protobuf structure to specify the prompt, parameter values, and initialization and validation logic. We decided to use a structured data format with the belief that would allow us define new information retrieval tasks by simply adding entries, making it casier to scale up the number of tasks without needing to write and maintain Python classes for each one.
Initialization is defined per app, including only the state relevant to the prompt's answer and exclu- sion conditions for generating random states. This ensures that no random state contains information that could alter the expected answer. The initial state and prompt parameterized using random values from the specified task parameters. For validation, we define the expected answer format within the prompt and use few supported functions ("count", "sum", "identity") to generate the answer from the initial state.
Once app and its specific logic programmed, new tasks be generated using LLM to generate the task's protobuf. The process is not automatic and requires human review. Common issues with LLM-generated tasks include missing fields, hallucinated fields, incompatible parame- ter generation. insufficient parameter usage, and non-specific task prompts. We observed that the complexity of the proto structure correlates with an increase in generated task issues. Despite these challenges, we found that editing L.M L.L--generated protobufs can be more efficient than writing complete task from scratch.
simplified version of the task definition for the Simpleealeeddrrvver sOnDate task which involves checking which events are on certain date. It specifies the relevant event, the exclusion conditions for any noisy event, how to determine success, and possible parameter values to chosen at random that will be used out the task definition.
Below we show
Lasks
prompt: "What events do have (date] in simpla Calendar Fro? AnDwer with the only. there multiple Litles, Zornat your anaves as conna sepacated list."
complaxity: 1
relevani_xiade
// Defines infernation the goal eventa.
state:
calendas
start_dite: "[dato]"
11
siaet_Liae: [LLiN]]
12
duration: "]duration]"
13
*[titlej*
15
16
17
18
vxclusio_ocaidieen
19
field: astart dlece"
20
uperations PC
"idate)*
25
24
criterie 1 espectationa
fiald
27
operations IDENTITY nimid "LiElw"
28
30
types STATNG
21

---

Published as a conference paper at ICLR 2025
32
33
34
nana:
36
poscible_velaes: "11:00an"
37
//
38
40
parana
41
canz: "dato"
42
passible_valleg *Oclobes 2023"
46
45
perana
46
"durstion"
poxsibla_ vele
48
//
49
parkapatan
nane: "Litia"
52
possiblewecless: *Data Dive*
13
D.4 HUMANS FOR TASK ANALYSIS
During development, recruited six volunteers with proficient programming skills analyze task difficulty, duration, and category. Each human was assigned an equal portion of tasks and tasked with identifying bugs during this annotation phase. This process resulted in the discovery and reso- lution of over 30 bugs.
To evaluate human performance, enlisted two software engineers to complete the tasks using an Android emulator. Participants were provided with task descriptions and attempted to achieve the goals based on their interpretations. Each participant had one attempt per task. The majority of errors stemmed from misinterpretations or minor errors, such as entering incorrect file extension. Other errors occurred when participants encountered unfamiliar user interfaces, impeding their ability to solve the tasks on their first attempt.
In both exercises, we informed participants about the intended use of the collected data. Participants were not required to enter any personal information in the tested tasks.
D.5
TASK EXAMPIES
Table 5 lists some additional examples tasks and highlights which task attributes can parame- terized in unlimited ways.
APPENDIX E ANDROIDWORLD AGENT DETAILS
E
M3A OBSERVATIONS
ANDROIDWORLD consumes the raw screen pixels, the screen shot with Set-of-Mark (SoM) (Yang et al., 2023a) annotations, and list of UI elements screen.
Sere
descriptions
elemenze
0: UTPI enent etvet,, content deser pertiitooonnneee fonetore,, class namee faneee"nddrdddwiiie widgee Text boox www.ell-Boundingg... memin-99, 886, y_nia-16e, y_nax-3:1),, ...)
Utel eeeentl: UtElenent (oene- content description "ller search box", class www.eereedddddddd..oooo.o widget.
Imagellutt
bbox boo__pixel--Bunee Boond-nooxx 886, 10n5, y_min-160, 2_nax-291), ...)
UTe ement2: utElement (cext=*1b:l1t concent desoriptione" ioneitttiii" cass__nrre-aandree andoesadddoooodddd iine Text
wow.iii--unddddeeee (x_min-30, 11nax-148, yuniar1, y_nax-128), ...
...
10
11
Suidelines ca action colection onitced
12
cucput
above
fornat, following why you
Your ansver should Look like:
14
15
Action: ("action_eyperr eypsso...y
Listing 2: The prompt format pertaining to screen representation with UI elements.
22

---

Published as a conference paper at ICLR 2025
Table 5: Examples ANDROIDWORLD tasks. We list the task nickname, the task template indi- cating which task attributes can be parameterized, the initialization logic that is executed before the task starts and pseudo code describing the success evaluation.
Task nicknume
template
Initialization logic
Plyylit
Create playlist in VLC, titled following
Create new mpeg liles: "noise" should not be added.
execute.sg c.c.urrrr) ==
"(playlist name)" files, in order: (files)
Add them VLC videos folder.
RecipeAddMullippc RecipesFrom/mage
Add the recipes from recipes.jpg Sim- ple Gallery the recipe app.
Write receipt file with recipes Simple Gallery,
sqp.ccows.exial texpecled.ceeipes
EdrNoeEdinNote
Edit [file.name] opeeatpnettion..
in
Markor.
Generate file with start-
fi eeee_existaa (f1 (ft" renname, contans-exxeerrod.com
ing content, along "noise" files not rele- vant goal. Note:
file operation can be add a footer, header, or update
AudSnnee
Add the following expenses into ex- pense: (expense csv)
Add to the app's SQLite sq-.rows.cot rx.isst exist lexpense.ols ob)) dstabase
expense
that should be deleted. along with "noise" penses that should not deleted.
SimpleCalendarDDlltte EvemtsOnRelativeDay
app's SQI !sg)_"ows_ex (expectadd.events databuse calendar events specified day, along with "noise" events that should not deleted. Delete lile {ille name) from the An-
Simple Calendar all events
scheduled for this (day.of_ week).
FilesDeleteFile
Generate specilied lile, !filee exists (file name) along with "noise" files
droid fiesystem located the (subfolder) within sik yphone age area.
deleted.
Trcckrrraacs CountFoWeeck
many {category) activitics this week in the OpenTracks app? Express your single integer.
the app's SQLite database activities for the specified category, along with "noise" activities.
int expected_count
E.2 M3A ACTIONS
For the SoM prompting, the screen is annotated based on the UI elements extracted from the acces- sibility tree, which form the agent's action space. Figure 5 shows one example.
E.3 ERROR ANALYSIS
We analyze M3A errors based broader categories observe during evaluation.
Perceptual errors Perceptual errors are caused when the model fails to recognize crucial elements on the screen necessary for task completion.
For the task below, the model does not recognize that the "All-day" checkbox is currently not checked (see Figure 6a):
In Simple Calendar Pro, create a recurring calendar event titled 'Review session for Budget Planning' starting on 2023-10-15 at 14h. The event recurs weekly, forever, and lasts for 60 minutes each occurrence. The event description should be 'We will understand software updates. Remember to confirm attendance."
Reasoning errors Reasoning occur when the model misinterprets the task requirements the current state, leading to incorrect actions.
For example, in the task below, the model mistakenly believes the note name has already been entered, so it types the note text into the "Name" field and cannot recover from this error (see Figure 6b):
23

---

Published as a conference paper at ICLR 2025
Setup
Blue Gray
Nammin Recordh mis
Phopsh by lomre
Semplente
15irts
1ile
kaps
Skeve
Moro
sndorrooooooo gax Cuiliyy cndil she (pcormnnnddd
Rewet
Apply
Figure 5: Set-of marks overlaid Android screen.
1537
Markor
Asterage remuaeed Documente
www.gg/emmiieei//// Documents
Review session for Budget Planning
10/00 1925
16/15/0021, 1935
Location
follile
nfx4YyEX
5/200/ 1524
10/16/3021 1531
undernttand aoftware updatees. confem
folder, 20240402204000 2206 18.md 5/200 15
nfxbWwEX
nfxEX91m
Ire.
All day
Tpee
nfxEXB19
Markdown
October
16:00
October (Bun))
16:00
Hoikde costed suttoee exderione
FOLDER
CAMOEL
www.deep77000
W
E
R
T
U
P
D
X
C
B
N
M
1123
@
ToDo
Quiodier
Mon
(a) Perceptual error.
Red square
(b) Reasoning error. The agent's (c) Missing knowledge error. To next action is to start entering the delete all notes, the agent mistak- note's contents, which is incorrect enly looks for option to delete all because it needs to enter the note's the notes at once, rather than trying name first.
highlights issue.
to do it note-by-note.
Figure 6: Screens illustrating perceptual, reasoning and missing knowledge errors.
Create a new note in Markor named ipsum mNcKd336Dq.tar with the following text:
24

---

Published as a conference paper at ICLR 2025
15:44
2023
10,
10.00..
GH
W
F
X
C
B
N
M
G
(a) Error on initial click.
(b) Error with text entered and saving.
Figure 7: Screens illustrating grounding errors.
Here the agent's reasoning trace:
The user wants to create a new note with a specific name and text. The name of the note has already been entered, all that needs be done is to enter the text and click on the OK button.
Missing knowledge errors Missing knowledge errors occur when the model lacks the necessary understanding of the application's UI to complete a task efficiently.
For example, in the task below, the agent does know how delete all files at once. It looks for an option to do that instead of deleting each file one by one (see Figure 6c);
Delete all files in the current directory.
Grounding errors Grounding errors oceur when the model fails correctly interact with the UI elements based on their spatial or contextual positioning.
For the task below, the agent needs update the Markor note by prepending text to the existing text. Figure 7 illustrates the errors the agent makes. It clicks the entire text field area, highlighted in green, which automatically places the cursor after the current text, resulting in the new text being appended after the current content.
Update the Markor note '2023 2023_08_1 08_10811100 1ea.1nnnttttate peannenalooll wolfixx" by adding the fol- lowing text, along with a new blank line before the existing content: "ETBM2jAPGxxqhbpusss
and rename it to "uure_oceannaa txt.
Then, in the next screen, the text has been entered after the existing content, and the agent clicks the save button.
E.4 AGENT ROBUSTNESS EXPERIMENTS
We the agent the following tasks (the nicknames shown in the figures in parentheses):
MarkorEdi tNote (EditNote)
25

---

Published as a conference paper at ICLR 2025
ExpensekddSinngle (AddExpense)
Shn.geece.eeaaaa...ceeetttttooer aateaaaeeeeaaaaeeaaaeeeeeeeaaaaa (DeleteBvent) DllesDDeltteFFleeeeeeeet (DeleteFile))
SperteeTaccerrccttttteeCoootttoo (ooe (CountActivittis)
More details about these tasks can be found in Table 5.
E.5 AGENT STRUGGLES DUE TO TASK PARAMETERIZATION
15:4
Expense Entry
Figure 8: The expense entry interface features a horizomtally scrollable category selector. When cer- tain parameterization seeds require selecting categories that are not initially visible (e.g.. "Food""", the agent fails to discover the scrolling interaction required to access them.
The variance success rates (Figure 3) across different seeds demonstrates how task parameteri- zation fundamentally changes task difficulty. For instance, in the ExpenseAddSings seed determines which expense category must be selected (see UI in Figure 8). When the seed specifies readily on-screen visible categories (e.g., "Housing", "Social"), the agent can complete the task. However, when the seed requires categories that are only accessible via horizontal scrolling (e.g.. "Food", "Other"), the agent consistently fails due its inability discover and execute this UI interaction pattern.
task, the
Similarly, the MarkorEditNoks task's difficulty varies based the reed-determined variant: adding text to the top of a note, adding text to the bottom, or replacing existing text. The "replace" variant requires more complex sequence of UI interactions (long-press, text selection, deletion, then text entry) compared to the simpler "header" variant. This explains both the complete failure under fixed seeds that happen to select challenging variants, and the higher but variable success rates when using different seeds that allow the agent to encounter various task parameterizations.
E.6 SEEACT DETAILS
We modify the SeeAct prompt (Zheng et al., 2024a) to reflect that the environment is Android by inputting elements from the accessibility tree and supporting additional actions (e.g., scrolling). Below we include the updated prompt. We annotate the system, user, and assistant roles that are each provided the OpenAl API.
26

---

Published as a conference paper at ICLR 2025
Rolo: SYSTEM
Imagine that you are initating hunans operating an lndroid device ter tack step stop, M each stape, you screanshot
the Android step dacided by yeurself threach zecorded history. You need dacide on the first folloxing astion to take, You can tap on an cleneat, loag-press an clement, svipe, input text, apen an app, or use the kayboard anter, hene, back key. (For your underatandiae, they are like 'adb shell inpet taps, 'adb shell iapuz suipe',
like
the previous actiona before
'adb sheli ingut kayevent'). One next step menns one operation within these adciona. Uelike hunana, for Lyping 1e.9.. Lext areas, taxt boxest, you should try directly typing the input oz selecting the choise, bysassing the nsed for an initial click. You should net actempt to create accounts, 1oz in do the riasl cubniscion. Terinate when you
shell
text', 'adb
-a',
the cumplete
it requites polentially haznful
> Rolo: USER
You are saked to complete the following Laaks <GOAL>
Pswvious iona::
<ERVVIOSS ACTION3>
10
The screenshot below shows Andraid you Pollow the Following guidance co chink step by slep before outlinang the next act step at the cuscent stage:
12
13 (Curswel Screen Cicatiiii
14 Firatly, think about what the
15
16 (eevvious Aet iun Analysia)
17 Becondly, coniiined with the screanshot, analyze each step of the previcus action history and their insention
by one,
anlarly, pay attention the step, chich ney last act anvolved INDUT TEXT, always avaluaae
related
you
should do now as next step. Specifically, whethez it cacessitates a concismation step, because typically a single INFU7 TEXI action does not make the default elanens inval wed in the lest action,
offect. loften, simply presxing "Pater', unlezs ather cleas slements presen. for opesacion).
18
19 (Screenshot betaila Anelysis)
20 Clasely examine the screenahot Lo check the stalus of evesy of the screen Lo understand what completed.
should close y exanine sew what steps have been completed by previous acs ions even thaugh you are given the text previnus actions. Because the Lextual histoty may not gleatly and autficisntly secord some effwels of poeviuus actions, you should closely eva maste the status of every part of the screen to understand what you have
operate
screershot details
21
22 (Next Action Resed on Android screen and Analysiat
23
Then, baxed your anelysia, conjunet ion with Lunan phone opecation ion labita and the logic of design, decide on the tollowing actioo. operate with ee the first next target element, it' detailed location, and the correspondl ng operat on.
clestly outline which element Android sareen users xill
24
25 be succesetul, is importart to tollow the tolloving rules:
26 should only
given
27 You whould only isaue one act fun at Line
28
3. For handling the select dropoown elenents on screen, it's not necessary for you to provide completely accurate opt fors right
The full list of tons for these elenents xill be supplied later,
30
Role: ASSISTANT
31
XAGRET IRSPONSP: TO ARCVES
32
34
[Deiteration)
35
First, ceiterate your next target element, its detailed location, and the correspancing operation.
36
Mfuttchoiize Qusstion)
38 Belox 15 multi-choice question, xhere the choices are eiements on the screen, All elements are arranged in the order based their height or the screen, from top to botton (ans fron left to right). This the scresnchot, fird
where and each is content and details. Then, determine whether one natches
ecreen, taxing into account both their your target ementt Please exemire the chot ces one by one, Choose the natching one. If mileiple opt tons match your answer, choose the most likely one re-cxamining the serecnshet, the caoices, and your turther reassning, It you would like to pestorn svipe action, you optionally select the choice you swige.
39
40 6. "Hone" icon
41
"Ehone"
"#essages" icon
43 D. *Chrome" icon
44 "Scazch" 12on
45
ot these elenents match your target elenent, please select E, Kone the other options
ccrrect cieneat.
47
48 (Final Ansxer)
Finally, canclude your answer using the sormat below. Ensure your answer strictly achering fornet provided belsw, Please do not Leave any explanstion in your answere of the cinal standardized format part, and this final part should be clear and certain. The elanen: choics, action, and valua should be in three separate lines.
50
51 Fornat:
52
BLEKENT: uppercase letter your chiicce. reed for TERIINATE, KEYBOARD ENTER, ANSMER, OPEN APP NAVIGATE HOME, NNVIGHTE BACKj and optional for sWIe3.)
54
55 ACTION: Choose an action fron (CLICK, INPUT TEXT, LANG PRESS, NAVIGATE BACK, TERMINATE, KEYBOARD ENTER, SWIPE, ANSKER, OPEN APF, RAVIGMIE HOME).
27

---

Published as a conference paper at ICLR 2025
additional Impuz ACTION.
58
59 The VALUE means:
ACTION INFIC TEXT, specity
ACTION
specity disection: up, dowt, lett, zight.
62 12 NCTION OPEN A2e, pEovIda the name ot the spp to be opened.
63
2f ACTION AMEKER, specify the sex: of your answez to zeapond directly LO question request Zoz
For CLICK, 2KESS,
ENTER, NAVIGATE HOME,
BACK, MAID, and TERNINATE, "Nore",
APPENDIX F ANDROIDWORLD TASK LIST
The table below lists all tasks in ANDROIDWORLA The maximum number of steps per task ("S") were determined based on human performance analysis, allowing agents approximately twice the number of steps typically required by human annotators complete each task while preventing infinite loops.
Task completion tasks (e.g., send message or edit note) are abbreviated as "TC" and information retrieval tasks are abbreviated as "IR".
Name
Template
Task
Validation method
Apps
type
Audio Recorder
Record an audio clip using Audio Recorder app and save it.
TC
Filesystem 12 audio recorder
Record Audio
Audio Recorder
Record an audio clip and save it with name "[file_name)" using Audio Recorder app.
TC
Filesystem 20
audio recorder
Record Audio
With File Name
Browser Draw
Open the the file manager; when prompted open it with Chrome. Then create drawing using the three colors shown at the top and hit submit.
html Downloads
TC
UI- elements
20
files, chrome
Browser Maze
Open the file hask Downloads in the file manager, when prompted open with Chrome. Then navigate the to the bottom-right cell, by using the direction buttons.
TC
UI- elements
20
files, chrome
Browser Multi- ply
Open the file haml in Downloads in the file manager; when prompted open it with Chrome. Then click the button 5 times, remember the numbers displayed, and enter their product in the form.
TC
UI- elements
22
files, chrome
Camera Take Photo
Take one photo.
Filesystem
camera
Camera Take
Take one video.
TC
Filesystem
camera
Video
Clock Stop
Pause the stopwatch.
TC
UI-
10 clock
Watch Paused
elements
Verify
Clock Stop Watch Running
Run the stopwatch.
TC
UI-
10
clock
elements
Clock Timer
Create timer with {hours} hours, {minutes) minutes, and {seconds) sec- onds. Do not start timer.
TC
UI-
10
clock
Entry
elements
Contacts Add Contact
Create contact for (name). Their number (number).
TC
Datahase query
12
contacts
28

---

Published as a conference paper at ICLR 2025
Name
Template
Task Validation type
Apps
method
Contacts New
Go the ter the following details: First Name:
and
TC
UI-
12
contacts
Contact Draft
elements
(first), Last Name: {last), Phone:
{phone). Phone Label: {phone label].
Do NOT hit save.
Expense Add
Add the following expenses into the pro expense: (expense list
TC
Database query
40
expense
Multiple
Expense Add Multiple From Gallery
Add the expenses from expenses. in
TC
Databasc query
20
gallery,
Simple gallery to pro expense.
expense
Expense Add Multiple From Markor
Go through the transactions my eexpensees.xt Markor. Log
TC
Database query
30
markor, expense
the reimbursable transactions in the pro expense.
Expense Add Single
Add the following expenses into the pro expense: fexpense info)
TC
Datahase query
12
expense
Expense Delete
Delete all but one of any expenses in pro expense that are exact duplicates, ensur-
TC
Database query
12
expense
Duplicates
ing least one instance each unique expense remains.
Expense Delete Duplicates2
Delete one of any expenses in pro expense that are exact duplicates, ensur- ing at least one instance of each unique expense remains.
TC
Database query
expense
Expense Delete Multiple
Delete the following expenses from pro expense: {expense_list) list).
TC
Database query
20
expense
Expense Delete Multiple2
Delete the following expenses from pro expense: (expense.list).
TC
Database
34
expense
query
Expense Delete Single
Delete the following expenses from pro expense: (expense.name))
TC
Database
10
expense
query
Files Delete File
Delete the file [file name} from
TC
Filesystem
10
files
the Android filesystem located in the [subfolder] folder within the sdk gphone x86 64 storage area.
Files Move File
Move the file (file name) from TC within
Filesystem 20
files
(source folder) sdk gphone x86 64 storage area to the iilestinationfolderr within the same _66-66_ storage area in the Android filesystem.
the
Markor Note Header
Update the Markor note [file_name) by adding the following text, along with a new blank line before the existing con- tent: "(header)".
TC
Filesystem
markor
Markor Change
Update the content of {file_ name} to TC poppdated_conentt)) in Markor.
Filesystem 12
markor
Note Content
Markor Create
Create new folder in Markor named {folder name).
TC Filesystem 10 markor
Folder
Markor Create
Create new note in Markor named (file name) with the following text: [text)
TC
Filesystem 16 markor
Note
29

---

Validation method Published as a conference paper at ICLR 2025
Name
Template
Task
Apps
type
Markor Create
Create note in Markor named
TC
Filesystem, 18 markor,
Note Sms
{file name) with the following text:
database query
sms
(text]. Share the entire content of the note with the phone number [number)
via SMS using Simple SMS Messenger
Markor Create
Create a note in Markor named (file name). Perform paste operation the note and save note.
TC
Filesystem 14
markor
Note From
Clipboard
Markor Delete
Delete all my notes in Markor.
TC
Filesystem 14
markor
Notes
Markor Delete
Delete the newest note in Markor.
TC
Filesystem
markor
Newest Note
Markor Delete Note
Delete the note
Markor named TC
Filesystem
markor
(file.name)
Markor Edit Note
Edit (file name) in Markor. TC (ubiossubc)mmand
Filesystem
markor
Markor Merge Notes
Merge the contents Markor notes (filel .name), [file_.name] and [fime3 (in the same order) into new Markor note named (new_fie_namp and save it. Add new line between the content of each note.
TC
Filesystem 78
markor
Markor Move Note
In Markor, move the note (file_name) from foldrce_folder) to (destination_folder)
TC
Filesystem 14 markor
Markor Tran- scribe Receipt
Create file in Markor, called re- ceipt.md mnd with the transactions from the receipt.png. Use Simple Gallery view the receipt. Please enter transactions csv format including the header "Date, Item, Amount".
TC
Filesystem 18 gallery, markor
Markor Tran- scribe Vidco
Transcribe the contents of video {video_named
TC
Filesystem 20 markor,
watching it in VLC player (located in Download) and writ- ing the sequence strings shown on each frame to the text file {file name] in Markor as comma separated list. For example, if the first frame shows the text "edna" and second frame shows the text "pineapple", then the text file should contains only the following text: "edna, pincapple".
vlc
Notes Is Todo
Is the note titled '(title)' in the Joplin IR app marked todo item? Respond with either "True" if it is todo 'False' if not.
String match
10
joplin
Notes Meeting Attendee Count
How many attendees were present the IR meeting titled [title)' the Joplin app?
String match
joplin
Express your answer as just single number.
30

---

Published as a conference paper at ICLR 2025
Name
Template
Task
Validation
Apps
type
method
Notes Recipe Ingredient Count
What quantity of fingredient] do need for the recipe '(title}' the Joplin app? Express your answer in the format (amount) (unit) without using abbrevia- tions.
IR
String match
10
joplin
Notes Todo Item Count
How many to-dos do I have in the IR [[older]' folder in the Joplin app? Ex- press your answer as just a single num- ber.
String match
10 joplin
Open App Task Eval
Open the [app_name) app. Clear any pop-ups that may appear by granting all
TC
System API
camera, clock,
permissions that are required.
contacts, settings, dialer
Osm And Fa- vorite
Add favorite location marker for TC
Filesystem 13
osmand
[location] in the OsmAnd maps app.
Osm And Marker
Add location marker for (location)
TC
Filesystem 20
osmand
the OsmAnd maps app.
Osm And Track
Save track with waypoints Ruggell, Liechtenstein, Bendern, Liechtenstein the OsmAnd maps app in the same order as listed.
TC
Filesystem 120
osmand
Recipe Add
Add the following recipes into the Broc- coli app: (recipe_list Jiist
TC
Database query
68
recipe
Multiple Recipes
Recipe Add Multiple
Add the recipes from recipes. in Sim- ple gallery to the Broccoli recipe app.
TC
Datahase query
26
markor, recipe
Recipes From
Image
Recipe Add
Add the recipes from recipes.txt Markor to the Broccoli recipe app.
TC
Database query
48
gallery, recipe
Multiple Recipes From
Markor
Recipe Add
Add the recipes from recipes.txt in Markor that take 10 mins to prepare into the Broccoli recipe app.
TC
Databasc query
52
recipe
Multiple Recipes From
Markor2
Recipe Add Single Recipe
following recipes into Broc- coli app: (recipe list]
TC
Datahase query
recipe
Recipe Delete
Delete all one any recipes in the Broccoli app that are exact duplicates, ensuring least one instance of each unique recipe remains
TC
Database query
recipe
Duplicate
Recipes
Recipe Delete
Delete all but one of any recipes in the TC Broccoli app that are exact duplicates, ensuring least one instance of each unique recipe remains
Databasc query
24
recipe
Duplicate Recipes2
Recipe Delete Duplicate Recipes3
Delete all but one any recipes in the TC Broccali app that are exact duplicates, ensuring least one instance of each unique recipe remains
Database query
34
recipe
Recipe Delete Multiple
Delete the following recipes from Broc- coli app: poccieelist!
TC
Datahasc query
24
recipe
Recipes
31

---

Published as a conference paper at ICLR 2025
Name
Template
Task
Validation
Apps
type
method
Recipe Delete
Delete the recines from Broccoli app that use (ingredient) the directions.
TC
Datahase query
40
recipe
Multiple
Recipes With
Constraint
Recipe Delete
Delete the following recipes from Broc- coli app: (recipe list]
TC
Database query
34
recipe
Multiple
Recipes With
Noise
Recipe Delete
Delete the following recipes from Broc-
TC
Database query
10
recipe
Single Recipe
coli app: (recipe_list}
Recipe Delete
Delete the following recipes from Broc- coli app: (recipe_list)
TC
Database query
20
recipe
Single With
Recipe With
Noise
Retro Create
Create playlist in Retro Music titled "[title}" with the following songs, in or- der: {song list)
TC
Datahase query
24
music
Playlist
Retro Playing Queue
Add the following songs, in order, {song. list) to my playing queue Retro music.
TC
Databasc query
32
music
Retro Playlist Duration
Create playlist in Retro Music titled "[title}" with a duration between 45 and 50 minutes using the provided songs.
TC
Database query
30
music
Retro Save Playlist
Create playlist in Retro Music titled "(title)" with the following songs, in order: {song_list] Then export the playlist to the Downloads directory on the device.
TC
Database query
50
music
Save Copy Of Receipt Task Eval
Copy {file_name, in DCIM and copy with the same name Download
TC
Filesystem
16
gallery
Simple Calen- dar Add One Event
In Simple calendar, create a cal- endar event on (yonh)-(month)- {day} at {hour)h with the title (event_title) and the description (event description] The event should last for [duration mins} mins.
TC
Database query
34
calendar
Simple Calen- dar Add One Event In Two Weeks
Simple calendar, create a cal- endar event two weeks from today (hour]h with the title (event title)' and the description (event deverrttscrip The event should last for [duration_ .mins} mins.
Datahase query
calendar
Simple Calen- dar Add One Event Relative Day
Simple calendar, create calendar event for this [day_of.weed) week) at [hour]h with the title "(event.title) and the deseription "fevent_dese desercctccpppioo The event should last for (duration mins) mins.
TC
Database query
calendar
Simple Calen- dar Add One Event Tomor- row
Simple calendar, create calendar event for tomorrow at (hour)h with the title {cvent_titlc)* and the description "(event description).. . The event should last for (duration mins) mins.
TC
Database query
26
calendar
32

---

Published as a conference paper at ICLR 2025
Name
Template
Task
Validation
Apps
type
method
Simple Calen- Add Re-
In Simple calendar, calendar event titled "(event title]' starting on {year)-(month)-(day The event recurs
recurring
TC
Datahase query
28
calendar
peating Event
at [hour}h. (repeat_uule). forever, and lasts for {duration_mins) minutes each occur- rence. The event description should be '(event description).
Simple Calen- dar Any Events On Date
Do have any events }date} in Simple IR calendar? Answer with the titles only. If there multiples titles, format your answer in comma separated list.
Database query
10
calendar
Simple Cal- endar Delete
In Simple calendar, delete all the calen- dar events on
TC
Database query
14
calendar
Events
Simple Cal- endar Delete
Simple calendar, delete events scheduled for this wdee_...week).
TC
Database query
calendar
Events On Rel-
ative Day
Simple Calen- dar Delete One Event
In Simple calendar, delete the calen- TC dar event (ycott-(month)-(dayy {hour]h with the title "[eventtitlee""
Database query
12
calendar
Simple Calen- Event Date At Time
What {time} in Simple calendar? Answer with the titles only. If there are multiples titles, format your answer in comma separated list.
my schedule for [date] IR Database query
10
calendar
Simple Calen- dar Events In Next Week
What events do have in the next week in Simple calendar? Answer with the ti- tles only. there are multiples titles, format your answer in a comma sepa- rated list.
IR
Database query
10
calendar
Simple Calen- dar Events In Time Range
Do have any events between {start.time and 8pm {date} in Simple calendar? Answer with the titles only. If there multiples titles, format your answer in comma separated list.
IR
Database query
10
calendar
Simple Calen- Events On Date
What events do have (date) in Simple calendar? Answer with titles only. If there are multiple titles, format your answer as comma separated list.
IR
Datahase query
10
calendar
Simple Calen- dar First Event After Start Time
What is my first event after [time} IR {date) in Simple calendar? Answer with the titles only. If there multiples titles, format your answer comma separated list.
Database query
10
calendar
Simple Calen- dar Location Of Event
What is the location my [title] event in Simple calendar? Answer with the lo- cation only.
Database query
calendar
Simple Calen- dar Next Event
What is my next upcoming event in Sim- ple calendar? Answer with the title only. If there multiples titles, format your answer in comma separated list.
IR
Datahase query
10
calendar
33

---

Published as a conference paper at ICLR 2025
Name
Template
Task
Validation method
Apps
type
Simple Cal-
When is my {person} Simple calendar? press your answer in the format (month name) (day) (year) (hour in 24-hour (minut)))
meeting with IR
Datahase query
10
calendar
endar Next Mceting With Person
Simple Draw Pro Create
Create drawing in Simple Draw Pro. Name it (file name). Save it Pictures folder within the sdk sphone_x86.oo storage area.
TC
Filesystem 18
simpledrawpro
Drawing
Simple Sms Reply
Reply [number) with message: (message) in Simple SMS Messenger
TC
Database
sms
query
Simple Sms
Reply to most recent text message using Simple SMS Messenger with mes- sage: {message}
TC
Datahase query
SMN
Reply Most
Recent
Simple Sms
Resend the message just sent to
TC
Datahase query
12
sms
Resend
{name) in Simple SMS Messenger
Simple Sms
Scnd text message using Simple SMS Messenger to {number) with message:
TC
Database query
12
sms
Send
{message}
Simple Sms Send Clipboard Content
Send message (number) with the clipboard content Simple SMS Mes- senger Simple Sms Send Received
TC
Databasc query
12
sms
Text address the event to {namel} that {name2) just sent me in Simple SMS Messenger
TC
Database query
sms
Address
Sports Tracker
How many {category) activities did do IR
String match
10
sportstracker
Activities Count For Week
this week in the OrenTracks app? Ex- press your answer single integer.
Sports Tracker Activities On Date
What activities did do [date) in the IR OpenTracks app? Answer with the egory only. If there are multiples cate- gories, format your answer in comma separated list.
String match
20
sportstracker
Sports Tracker Activity Dura- tion
How long was my {category) } activity IR {date) in the OpenTracks app? Express your answer in minutes single inte- ger.
String match
12
sportstracker
Sports Tracker Longest Dis- tance Activity
What was longest distance covered IR in {category) activity the Open- Tracks app this week? Express your an- swer in meters single integer.
String match
sportstracker
Sports Tracker Total Distance
What was the total distance covered for IR {category) activities in the OpenTracks app from date) {end_date)? Express your answer meters as sin- gle integer.
String match
22
sportstracker
For Category
Interval
Sports Tracker
What was the total duration (category) activities in the Open- Tracks app this week? Express your in minutes as single integer.
IR
String match
16
sportstracker
Total Duration
For Category
This Week
System Blue- tooth Turn Off
Turn bluetooth off.
TC
System API
10
settings
34

---

Published as a conference paper at ICLR 2025
Name
Template
Task type
Validation
Apps
method
System Blue-
Turn bluctonth off.
TC
System API
10
settings
tooth Turn
Verify
System Blue- tooth Turn On
Turn bluctooth on.
TC
System API
settings
System Blue-
Turn bluctooth on.
TC
System API
10
settings
tooth Turn On Vorify
System Bright- ness Max
Turn brightness to the max value.
TC
System API
10
settings
System Bright-
Turn brightness to the max value.
TC
System API
10
settings
ness Max Ver-
ity
System Bright- ness Min
Turn brightness to the max value.
TC
System API
10
settings
System Bright-
Turn brightness to the max value.
TC
System API
10
settings
ness Min Verify
System Copy To Clipboard
Copy the following text the clipboard:
TC
System API
n/a
{clipboard
System Wifi
Turn wifi off.
TC
System
settings
Turn Off
API
System Wifi
Turn wifi
TC
System API
settings
Turn Off Verify
System Wifi
Turn on.
TC
System API
settings
Turn On
System Wifi
Turn on.
TC
System API
settings
Turn On Verify
Tasks Com-
Which tasks have completed for (date) in Tasks app? Answer with the titles only. If there are multiples titles, format your answer in comma separated list.
IR
String match
10
tasks
pleted Tasks
For Date
Tasks Due Next Week
How many tasks do I have due next IR week in Tasks app? Express your swer as a single integer.
String match
12
tasks
Tasks Due On Date
What tasks have due [date) Tasks IR app? Answer with titles only. there are multiples titles, format your answer in comma separated list.
String match
Tasks High Priority Tasks
What are my high priority tasks in Tasks IR app? Answer with the titles only. If there are multiples titles, format your answer in comma separated list.
String match
10
tasks
Tasks High Priority Tasks
Which tasks with high priority due IR {date} } in the Tasks app? Answer with the title only. If there are multiples titles, format your answer in a comma sepa- rated list.
String match
tasks
Due On Date
Tasks Incom- plete Tasks On Date
What incomplete tasks do I have still have do by (date) in Tasks app? An- swer with the titles only. If there are multiples titles, format your answer in a comma separated list.
IR
String match
10
tasks
35

---

Published as a conference paper at ICLR 2025
Name
Template
Task type
Validation
Apps
method
Turn Off Wifi
Turn off WiFi, then enable bluetooth
TC
String match
20
settings
And Turn On
Bluetooth
Turn
Turn on Wifi, then open the (app_name)
TC
String
settings
And Open App
app
match
Create Playlist
Create a playlist titled {title}" with the
TC
String match
28
vlc
following files in VLC (located in In- ternal Mcmory/VLCVideos)) in order: ]
Vic Create Two Playlists
Create playlist titled "[titlel)" with the following files VLC (located Internal MemoryVV.CVideoo))
TC
String match
48
vlc
or-
der: [video namesl). ). And then, cre- ate playlist titled "(title2)" with the following files in VLC, in order: (video_nn 3..
36


Reviews:
[{"review_id": "lH7BdLj9uA", "reviewer": "Reviewer_1Fiv", "summary": "The paper introduces AndroidWorld, a benchmarking environment for autonomous agents interacting with Android devices. This environment enables testing on various apps by dynamically constructing parameterized tasks expressed in natural language. AndroidWorld supports a more realistic and varied testing experience than static environments, enabling agents to engage with numerous unique task goals and environmental conditions. Initial experiments using baseline agents indicate significant room for improvement in task completion rates, with top-performing agents achieving a 30.6% success rate. This highlights the challenges and research opportunities in building robust, cross-platform autonomous agents.", "questions": "- How might AndroidWorld be expanded or adapted to include tasks that require specific contextual understanding, such as personalized interactions with app data?\n- How might agent performance improve if AndroidWorld incorporated multimodal feedback mechanisms, such as sound or haptic responses, beyond the visual and text-based cues currently used?\n- B-MoCA, like AndroidWorld, operates in Android environments and also employs various randomization schemes for task variability. How does your approach to task diversity, randomization, and reproducibility compare to B-MoCA? What unique advantages does AndroidWorld offer regarding its benchmarking scope, real-world applicability, and parameterized task generation? Additionally, are there specific aspects of your design that provide it with a distinct edge in evaluating agent robustness?\n- Could the authors provide a full list of the tasks included in AndroidWorld, along with the initialization logic and success evaluation codes for each task?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- AndroidWorld’s dynamic task construction introduces extensive variability in task conditions, offering a realistic and reproducible environment for testing autonomous agents on Android.\n- The paper provides a robust baseline evaluation and a thorough performance analysis across real-world conditions.\n- Extensive experiments provide essential insights into current agents' limitations and suggest potential pathways for improvement in future cross-platform agent designs.", "weaknesses": "- The agents achieve a low overall success rate (30.6%), which, while reflecting the environment’s complexity, suggests that current methods may need significant refinement to handle mobile platforms effectively.\n- Though this paper introduces extensive task parameterization, it does not provide a detailed explanation of how specific parameter variations impact agent performance. There is also no analysis of performance changes across different task components or parameters.\n- Large foundation models integrated into the agents exhibit high latency, taking significantly longer than humans to complete tasks, which may limit the practical applicability of these agents in real-time scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qYVI21Wesq", "reviewer": "Reviewer_gUrJ", "summary": "A benchmark for LLM (or LMM) agents controlling mobile devices is introduced, focusing on the necessity of interactive evaluation (as common benchmarks in this domain have mainly focused on static datasets so far) and the agents’ generalization ability in terms of task instructions. The tasks regard applications in Android commonly used in daily life as well as tasks in MiniWob++, which have been a common benchmark in digital device control. The study also proposes a new prompting approach to enhance agentic capabilities in foundation models and presents a detailed analysis of agent performance. The analyses include comparisons across different input modalities, robustness across varied task parameters, and their common errors. \n\nThe main contributions of this work are the introduction of a new benchmark, tasks that can be randomized via instruction parametrization, and a new algorithm enhancing the capability of LLM agents for mobile device control.", "questions": "Q1 - Could the authors explain the rationale for integrating MiniWoB++ into a mobile environment, as this incorporation seems highly auxiliary? What unique challenges do these MiniWob++ tasks present, especially compared to tasks involving typical applications in AndroidWorld?\n\nQ2 - While the unique aspects of the mobile action space are described, not all are represented in the current action space (e.g., multi-finger actions). Could the authors comment on the extensibility of M3A/AndroidWorld to accommodate different action spaces, or do they plan to incorporate a broader range of actions in the future?\n\nQ3 - Could the authors provide a guideline on the training and testing configuration, such as a split of task categories? Additional details would aid in understanding the optimal configuration for achieving reliable results.\n\nQ4 - (Minor) Would the authors consider providing results using Claude’s newly proposed models (considering that the relevant developers are interested in digital device usage)? This addition would offer a timely perspective on how recent advancements perform within this framework.\n\nQ5 - The authors mention that using multiple random seeds and averaging the results would provide a fair representation of the results (line 493). Could the authors confirm if the results in Table 3 are also based on averaged results across different seeds? If not, might it be beneficial to conduct the experiments across multiple seeds, as task randomness can significantly impact agent performance?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "S1 - This work presents a foundational contribution that advances the AI community's understanding of **mobile device control**, offering a robust framework for evaluating LLM agents in interactive environments.\n\nS2 - The related work and comparisons to existing benchmarks are well-organized and comprehensive. This provides a clear context of how this benchmark builds upon and differentiates itself from prior studies.\n\nS3 - The benchmark introduces an important challenge related to task generalization by implementing parametrized tasks. This effectively assesses the generalizability of agents across varying task instructions.", "weaknesses": "W1 - The explanation of the action space could be more detailed. For instance, in the ACTION_TYPE section described in Appendix B.2, further clarification on the purpose of the \"STATUS\" action would be helpful. Additionally, what is the rationale behind the necessity of a \"SWIPE\" action, despite the existence of \"SCROLL\", could be further justified. \n\nW2 - Although line 257 mentions a limited set of high-level APIs, Appendix B lacks specific details on these APIs. More explanation would help us understand the scope of actions.\n\nW3 - The rationale behind the task categorization tags could be elaborated on further. Expanding on the justification for the categorization approach would clarify its significance and enhance the robustness of the framework.\n\nW4 - I believe that this work could benefit from the inclusion of more baseline comparisons. Presenting results from a basic version of the agent, without any prompting methods for example, would provide valuable reference points. While the current results demonstrate that M3A is a strong baseline, additional baselines would enrich the study and offer further insights into performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aGW4yKXPSo", "reviewer": "Reviewer_5yqj", "summary": "This paper introduces AndroidWorld, an executable, reproducible, and realistic benchmark for evaluating Android device control agents. The authors also present results from several baseline agents and human.", "questions": "1. How was the confidence interval in Figure 3 calculated? I couldn’t find details on how these intervals were derived. Did you run the same model on these tasks multiple times to obtain them?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Solid contribution: this benchmarks fills the gap for solid, reproducible, and executable benchmarks for Android device control tasks.\n2. Good presentation: the presentation is clear, discussions in related works are comprehensive\n3. Interesting experiments: the experiments presents a few good baselines and shows how human performs on the tasks. The robustness analysis also provides new information to the community.", "weaknesses": "1. Lack of many real-world apps and tasks: The benchmark lacks many real-world applications and tasks, as ensuring full reproducibility and automated evaluation makes it impossible to include closed-source apps like YouTube, Twitter (X), Amazon, or actual real-web browsing. This results in inherent sim-to-real gaps.\n2. Lack of device diversity on android device/OS: The emulated device is fixed as a Pixel 6 running Android 13. However, in practice, what we care about is agents' performance across various devices and OS versions. Adding more OS/device options would potentially make the benchmark more correlated with real world use cases.\n3. Lack of open-source baselines: All baselines in the paper appear to be based on proprietary models. Including some open-source models as baselines would be beneficial, as it would provide researchers a baseline they can iterate on.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ljGOLm0SZL", "reviewer": "Reviewer_EDMT", "summary": "This paper presents AndroidWorld, a realistic and reproducible Android benchmark that provides functional rewards for 116 dynamically constructed tasks across 20 real-world Android apps. The experiments show that the developed agent M3A can complete only 30.6% of the tasks, and existing web agents struggle to transfer well in Android environments.", "questions": "1. What is the number of human hours required to develop the environment and configure the tasks?\n2. In your initialization and teardown process, have you considered scenarios where the agent performs actions outside the defined teardown process, potentially altering system states and resulting in non-identical states across different experiment trials?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is overall clear and well-written.\n2. The environment is lightweight and compatible with task random parameterization.\n3. Detailed examples and error analysis are provided in the appendix.\n4. I appreciate the effort to implement the initialization and teardown process for each task to ensure reliability and reproducibility.", "weaknesses": "1. The experimental setup is not very clear. Specifically:\n- What specific stop criteria are used for the experiments (e.g., step limits or time limits)? How is the task-specific step budget set for each task?\n- What decoding temperature is applied?\n- How is human performance measured?\n\n2. It would be helpful to include results from more models, including recent APIs and open-source models. The ablation study only investigates the observation space and does not consider the impact of other agent design choices, such as Reflexion.\n\n3. Are all tasks and apps offline? If not, how to ensure reproducibility? For example:\n- Does the SendSms task send actual messages?\n- Could real-world time and date affect the evaluation results and reproducibility in calendar/timer tasks within AndroidWorld and MobileMiniWoB++?\n\n4. More discussion of related work, e.g.:\n- Zheng et al., \"Agentstudio: A toolkit for building general virtual agents,\" arXiv preprint arXiv:2403.17918.\n- You et al., \"Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs.\" arXiv preprint arXiv:2404.05719.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)