SPA-BENCH: A COMPREHENSIVE BENCHMARK FOR SMARTPHONE AGENT EVALUATION
Authors: Jingxuan Chen
Derek Yuen
Bin Xie
Yuhao Yang
Gongwei Chen
Zhihao Wu
Li Yixing
Xurui Zhou
Weiwen Liu
Shuai Wang
Kaiwen Zhou
Rui Shao
Liqiang Nie
Yasheng Wang
Jianye HAO
Jun Wang
Kun Shao
Venue: ICLR 2025 Spotlight (2025)
Keywords: AI Agent
LLM
MLLM
Benchmark
Smartphone Control
Primary area: datasets and benchmarks
TL;DR: A comprehensive benchmark for smartphone agents, evaluating multiple agents across smartphone control tasks in various scenarios.

Abstract: Smartphone agents are increasingly important for helping users control devices efficiently, with (Multimodal) Large Language Model (MLLM)-based approaches emerging as key contenders. Fairly comparing these agents is essential but challenging, requiring a varied task scope, the integration of agents with different implementations, and a generalisable evaluation pipeline to assess their strengths and weaknesses. In this paper, we present SPA-Bench, a comprehensive SmartPhone Agent Benchmark designed to evaluate (M)LLM-based agents in an interactive environment that simulates real-world conditions. SPA-Bench offers three key contributions: (1) A diverse set of tasks covering system and third-party apps in both English and Chinese, focusing on features commonly used in daily routines; (2) A plug-and-play framework enabling real-time agent interaction with Android devices, integrating over ten agents with the flexibility to add more; (3) A novel evaluation pipeline that automatically assesses agent performance across multiple dimensions, encompassing seven metrics related to task completion and resource consumption. Our extensive experiments across tasks and agents reveal challenges like interpreting mobile user interfaces, action grounding, memory retention, and execution costs. We propose future research directions to ease these difficulties, moving closer to real-world smartphone agent applications.

Published as a conference paper at ICLR 2025
SPA-BENCH: A COMPREHENSIVE BENCHMARK FOR SMARTPHONE AGENT EVALUATION
Jingxuan Chen',, Derek Yuen", Bin Xie², Yuhao Yang', Gongwei Chen', Zhihao Wu',
Yixing Li², Xurui Zhou², Weiwen Liu Shuai Wang', Kaiwen Zhou', Rui Shao
Liqiang Nie², Yasheng Wang', Jianye Hao Jun Wang", Kun Shao
Huawei Noah's Ark Lab, Harbin Institute of Technology, Shenzhen, TTinjjn University, + "I Centre, University College London
ABSTRACT
Smartphone agents are increasingly important for helping users control devices efficiently, with (Multimodal) Large Language Model (MI.LM))-basdd approaches emerging key contenders. Fairly comparing these agents is essential but chal- lenging, requiring a varied task scope, the integration of agents with different im- plementations, and generalisable evaluation pipeline to assess their strengths and weaknesses. In this paper, we present SPA-BENCIA a comprehensive SmartPhont Agent Benchmark designed to evaluate (M)LLM- M-bLLMd agents in interactive environment that simulates real-world conditions. SPA-BENCES offers three key contributions: (1) A diverse set of tasks covering system and third-party apps in both English and Chinese, focusing on features commonly used in daily routines; (2) A plug-and-play framework enabling real-time agent interaction with Android devices, integrating over ten agents with the flexibility to add more: (3) A novel evaluation pipeline that automatically assesses agent performance across multiple dimensions, encompassing seven metrics related to task completion and resource consumption. Our extensive experiments across tasks and agents reveal challenges like interpreting mobile user interfaces, action grounding, memory retention, and execution costs. We propose future research directions to ease these difficulties, moving closer to real-world smartphone agent applications. PEA-BENCH is avail- able at hhtphhs:://// //aii agent s- 20000000000 qithub. 10/ S Bench/
INTRODUCTION
The growing capabilities Large Language Models (LLMs) and Multimodal Large Language Models
(MLLMs) have broadened the application of AI agents across various domains (Gur al., [023] Gou
et al. 2023] Cai et al. 2023 Li et al. 2023a Wang et al. 2023 Wu et al.) 2023a). One promising
area is smartphone control. where agents assist users in tasks like booking hotels or setting alarms.
These agents can be broadly categorised into two main types: (1) agent-as -s-a-eee--- (Lai et al.) (024)).
where fine-tuned or pre-trained (M)LLMs are customised for agentic tasks (Zhan & Zhang 2023
Hong et al. 2024 Bai et al. 22024] Lu et al. 2024; Christianos et al.) 2024 Wang et al. (024d)), and (2) agentic workflow (Shang al. 2024), which typically relies on off the-slll models and modular designs to support agentic functionality (Yang et 2023b. Wen et al.]
20224] Wang et al. 2024b Rawles et al. 2024a). In both cases, these models act as the "brains" " for decision-making The information these agents to interact with smartphones
vary, with common methods involving
direct screen observation (Wang et al. 2024b a Zhan & Zhang 2023 HHng et al.) 2024 Bai et al.
2024
Lu et al. 2024), accessing non-visible data via Android View Hierarchy or Extensible Markup
Equal contribution.
Corresponding authors: shaonui @ hit.edu.en, shaokun2 @ huawei.com.

---

Published as a conference paper at ICLR 2025
Agent Pool
Task Pool
Evaluation Pipellne
Off-th--Sheff
Fine-tured (M)LLM based Agents
Single-App Tasks
Cross-App Tasks El Languages
(M)LLMi- based Agents
Languages
Succees Signal
API Cost
Dificuily Levels
2 Difficuily Levels
Step Ratio
Sport
(1 Select Agent Task
Agent Framework dea
2 Agent + Task Deser plion
END
Termination Reason
Worker Machine
Performance
(M)LMM based Agents
Premature Termination Signal
Multiwarker Process
Overall Performance
Android Devices
T Screcrshots Exaistion Information
Task Description Screanshots
Overdue Termination Signal
Execution Information
Figure 1: An overview of SPA- SPA-BENCH. The worker machine iterates through the task and agent pools, assigning tasks to agents within the framework for execution, and then passes the execution results to the evaluation pipeline for measuring task completion and resource consumption performance.
Language (XML.) (Wen et al. 2024), or a combination of both (Yang et al. 2023b Rawles et al. 2024a).
As the number of (M)LLM-basee smartphone agents grows, fair performance comparisons become crucial for identifying their strengths and shortcomings. leading to an increasing need for bench-
marking (Chan et al. 2023] Liu et al., 2023a b. Wu et al., 2023b). Regarding smartphone agent
benchmarks, existing studies use three main approaches to evaluate agents: actions-based (Xing
ctt al. 2024), states-bases based (Rawles et al.) 2024a| Zhang et al. 2024 Lee et al. 2024 Wang et al. 2024c). or a hybrid of both (Wang et al. 2024c). Each method faces specific difficulties: action-based evaluation may involve multiple correct sequences, while state-hased methods struggle to determine the appropriate post-action state. A hybrid approach could mitigate these limitations, but the challenge lies in effectively utilising both action and state information.
Despite these efforts, current research (Rawles et al., 2024a] Xing et al. 2024 Zhang et al. 2024 Lee et al.] [024] Wang et al. (024c) still has several key limitations: (1) The focus remains primarily on system and Google suite applications (apps) in English, which are often free from distractions like ads and pop-ups that could introduce complexity and randomness; (2) The number of evaluated agents typically fewer than five, with some studies including only similar variants of the same agent; (3) Automated success detection methods frequently require human intervention (e.g., handcrafted validation logic for each task) or rely on data that may be inaccessible in certain cases (e.g., Android View Hierarchy data, which is unavailable in Web View apps (Xing et al. 2024)).
In this paper, we introduce SPA-BENCH, a SmartPhone Agent Benchmark designed to evaluate more than 10 smartphone control agents in daily tasks. As illustrated in Figure I1 SPA-BENCH comprises 340 tasks, including 150 single-app tasks and 20 cross-app tasks, in both English and Chinese apps, as well third-party ones. It integrates 11 agents into a unified framework based on their original implementations. This framework is linked to an automated evaluation pipeline that measures agent performance and can be applied to additional tasks beyond this benchmark, without requiring human inputs. Our experiments show that agents following the agentic workflow outperform those in the caant-as-a-mode category, although the former one remain impractical for real-world deployment due to time and cost constraints. We also provide a detailed discussion on the challenges and future directions for smartphone agents, covering topies such as building perceptive mobile interfaces, reasoning mechanisms, and user-friendly applications.
In summary, our comprehensive benchmark makes several key contributions: (1) a diverse task collection of 340 tasks with increasing difficulty, accompanied by human trajectory annotations. It covers both English and Chinese apps, including single-app and cross-app scenarios, and featuring 58 third-party apps (Section 3); (2) plug-and agent framework supporting 11 agents, which allows for easy integration of new agents with minimal adaptation and offers features like automatic Android setup and multi-devie emulator support (Section 4 (3) an automated and scalable evaluation pipeline assesses agent performance using task completion and resource consumption metrics. It employs success detection methods that achieve average F1 scores of 90.5% for single-app
2

---

Published as a conference paper at ICLR 2025
Table 1: Comparison of SPA-BENCE and other smartphone agent benchmarks. Agents that differ
only in their base models are not counted as separate agents.
Benchmark
Third patty app
Cross app?
Chinese app?
Difticulty level?
Number taaks
Number of agents
Number of metnies
Free of hand cralted validation?
Information for success detection
AndrodAAreaa
Xing et clal.ge 2005
X
221
Action only
And ndWolll
Roree ea 20144
X
116
State ouly
Dimin diy
X
495 x
1
State only
B-McCA
T Tarana mi 2074
X
60
1
K
Stat only
MobuleAgehtzeess
Wall et 2024C
x
100
6
K
Action and State
SPA BENCH
340 11
7
Action and State
tasks and 84.5% for cross-app tasks compared to human evaluators (Section 5 and (4) extensive experiments across agents and tasks, providing a detailed analysis of current smartphone agent capabilities and limitations, while also offering directions for future research (Section 6
RELATED WORK
Smartphone Agent. Smartphone agents aim to automate tasks on mobile apps in a human- like way. Early agents, like Siri and Google Assistant, relied on system- APIs and customisation, limiting their generality. Recently, (M))LM- agents have emerged, using the user interface (UI) to achieve a more general approach. These agents, with (M)LLMs as their "brains", also require "hands" (actions) and "eyes" (observations) to interact with smartphones. They are based on either off-the-shelf or fine-tuned models and perform human-like actions (e.g., tapping, typing, and swiping). According to how they observe the UI, recent works are categorised into text-based, vision-based, and combined approaches. Text-based methods (Wen et al. 2024 Rawles al. 2024a) rely UI document data (e.g.. XML) or convert visual information into text, vision-based methods (Wang et al.)
2024b a Zhan & Zhang 2023 Hong al.. 2024 Bai al. 2024 Lu al. 2024) use sereenshots to capture the complete visual context, while combined approaches Yang et al. 2023b Rawles et al.
2024a) integrate both text and vision inputs for greater informativeness. SPA-BENCH evaluates all
three types of agents to provide comprehensive comparison of their capabilities.
Smartphone Agent Evaluation. Effective evaluation of smartphone agents crucial for identifying limitations and guiding improvements. Success rate, which measures task completion, is the most commonly used metric, with some studies also considering efficiency. Success detection methods are generally classified into two types: human detection (Yang et al. [0236] Wang et al., 2024b a), which is accurate but resource-intensive. and automated detection, which less costly but varies in accuracy. Current automated methods primarily rely on hand-crafted validation logic, making them unscalable without human intervention. They are restricted to evaluating tasks involving apps that are limited to English-only and simpler apps (c.g., system, Google Suite, and open-source apps), with minimal coverage of other third-party ones. These automated methods can be further divided into action-based state-based, and hybrid approaches. Action- based methods (Xing et al.
2024 compare
agents' actions to human demonstrations but struggle with the non-unique nature of correct action sequences. State-based methods (Rawles et al. 2024a Zhang et al. 2024 Lee et al.) 2024 assess whether essential states are reached but may miss minor actions. Hybrid approaches (Wang et al. 2024c) combine state and action data for more accurate success detection. SPA-BENCH introduces two hybrid approaches for evaluating single-app and cross-app tasks. Compared to other automated methods, our approaches support wider range of apps and tasks. They do not rely on hand-crafted validation logic, making them adaptable without human intervention. Table Jableepreeentts a comparison between our work and other automated evaluation- based smartphone agent benchmarks, highlighting our comprehensive evaluation of various agents in diverse tasks across multiple dimensions.
3 SPA-BENCH TASK
3.1 OVERVIEW
SPA-BENCE builds a collection of smartphone agent tasks across both English and Chinese apps, featuring 39 English and 29 Chinese apps divided into eight categories based on features (see Appendix B.I. The collection includes 150 single-app tasks and 20 cross-app tasks for each language. These tasks focus core app functions that reflect everyday use, providing realistic assessment of
3

---

Published as a conference paper at ICLR 2025
smartphone agents' performance. The inclusion of diverse Chinese and third-party apps increases complexity, primarily due to the difficulties agents encounter in understanding Chinese and navigating more intricate Uls. A complete list of tasks is provided in Appendix B.2
The single-app tasks are grouped into sets, with progres- sively increasing complexity across three difficulty levels. In each set, Level tasks serve as foundational and straight- forward activities. while Level and Level tasks introduce more complex requirements, such as handling intricate UI elements or animations. Gener- ally, Level tasks require fewer than five actions, while Level 2 tasks typically involve fewer than ten, and Level 3 tasks fewer than fifteen. While each set follows similar instructions, the tasks are designed to use distinet entities (c.g., creating folders with different names) to prevent any influence from earlier tasks. Examples of single-app tasks are shown Figure 2 For cross-app tasks, we refer to the recent work GUI Odyssey (Lu et al., 2024).
componente
(iee weachh crack by Vill auilts
mecuur huve to tu dus Hen
Figure 2: A sample set of tasks within the Deliveroo app, annotated by human. In this example, simpler tasks form the foundation for more complex ones, resulting in shared trajectories in the initial stages. The final screenshots for tasks of all three difficulty levels are highlighted in corresponding colours. Each final screenshot highlights the key components used coarse detection (explained further in Section 5 with the zoomed-in versions available in Appendix B.3
which defines six task types: General Tool, Information Management, Web Shopping, Media Entertainment, Social Sharing, and Multi-Apps. Unlike single-app tasks, cross-app difficulty levels are determined by the number of apps involved in a task. Level I tasks require interactions between two apps, while Level 2 tasks necessitate switching between three. As the number of apps increases, complexity arises not only from additional steps but also from inter-app dependencies and coordination. Our cross-app app tasks include three Level tasks for each of the first five types, and five Level 2 tasks for the Multi-Apps type. Appendix B.4 provides examples.
3.2 TASK CONSTRUCTION
Our tasks were primarily constructed by human annotators. For single-app tasks, we selected commonly used apps and supplemented them with apps from related works (Yang et al.) 2023b Wang et al. 2024b). Based on each app's 's core features, tasks were created following an annotation guideline specifying: (1) A clear task description that reflects the task's goal and difficulty level. For descriptions inspired by prior works, we standardised them and assigned appropriate difficulty levels. (2) A human-executed trajectory presented as a series of screenshots. Between any two adjacent screenshots, only one action (e.g., tap, type) is allowed. The total number of actions in the human execution serves as the "golden steps" in our experiments. To ensure a reproducible and unbiased baseline, we instruct human annotators to avoid irrelevant actions and refrain from using shortcuts that are inherently dynamic, influenced by factors such as recommendation algorithms or user-specific history. (3) Key components of the final state, which are pieces of text that must appear in the final screenshot the task successfully completed. We focus only on the final state because there may be multiple correct paths to complete the task, but they typically converge to the same final state (Wang ct al. 2024c). These key components are designed for future use, as detailed in Section 5.2 For cross-app tasks, annotations include only task descriptions and human-executed trajectories due to the flexibility of final states. Most cross-app English tasks were drawn from GUI Odyssey (Lu et al.) 2024], and we reformatted descriptions and recollected trajectories where necessary.
To ensure task quality, a validation process followed task annotation. Annotators cross-checked all tasks for clarity, trajectory accuracy, and key component quality. The tasks were also tested
4

---

Published as a conference paper at ICLR 2025
across different Android devices, Android versions, and app versions to verify feasibility. The same validation was repeated before experiments.
In total, SPA-BENCH includes 300 single-app app and 40 cross-app tasks, evenly split between English and Chinese. Each task may consist of multiple subtasks (c.g., adding, modifying, deleting. searching). The distribution of steps performed by humans for these tasks, categorised by task type, is illustrated in Appendix B.3
4
AGENT FRAMEWORK
4.1 A UNIFIED PLUG-AND-PLAY FRAMEWORK
Our framework facilitates the execution of autonomous smart- phone agents and tasks. As shown in Figure 3] the worker machine manages communica- tion, providing task descriptions and receiving outcomes (trajec- tories and logs). It hosts multi- ple worker processes, each con- necting Android emulator and an agent. Each agent in- teracts with the Android device by performing actions based on observations, such taking screenshots and generating tions like taps or swipes. The snapshot state is restored at the start of each experimental cycle. The framework is highly scalable. Unlike existing research (Rawles et al. 2024a Xing al. 2024 Zhang et al., 2024; Lee et al., 20244, Wang et al.) (024c), which integrates a limited number of agents tightly into the framework,
Configusstion
S) Exacution Trajectry
Worker Process
Astion
2 AosignAAgeml
Aulo
Scrwer
Log
Siepshot
Observation
Null-nocal Iiset
Server
Monual
Emuiator
Agont
Docison
Figure 3: An overview of the agent framework using a multi- processing architecture. Each worker process connects agent to an Android emulator, and they interact multiple times throughout the task (i.e., step is repeated) until completion. The emulators are reset after the agent has executed all assigned tasks.
allows easy addition of new agents with minimal integration, ensuring each agent operates independently within an isolated environment. Details about the agents integrated into our framework are provided in Appendix C
4.2 SNAPSHOT-BASFI EMULATOR FOR CONSISTENT TESTING
The framework integrates Android emulators scalable alternative physical devices, replicating most Android functions for parallel testing and rapid experiment deployment. For instance, a 24- core CPU with 64GB RAM support up eight emulators worker processes simultaneously, depending on the agents' resource needs.
To ensure consistency, emulators be quickly loaded from snapshots, which capture and restore system states (e.g., installed apps, login credentials, and local settings). This eliminates repetitive setup processes by preserving pre-configured settings (e.g., a pre-existing contact for messaging tasks). However, since some app data is stored externally, manual intervention is required after each experiment cycle, such as unsubscribing from channels post-task completion.
AUTOMATED EVALUATION PIPELINE
5.1 METRICS
We define seven key metrics for comprehensive evaluation:
Completion-related Metrics. (1) Success signal a binary indicator of task success. For single-app
and cross-app tasks, we develop two different hybrid approaches that leverage both action and state information, allowing for multiple valid execution paths. These approaches eliminate the need for human evaluators and handcrafted evaluation logic (details are provided in Section 5.2 (2) Step
5

---

Published as a conference paper at ICLR 2025
ratio measures execution efficiency by comparing agent steps with human steps (the "golden steps" from (3.2 This is considered only when the task is successful (i.e., success signal is "true"). A higher ratio indicates more unnecessary actions and lower efficiency. (3) Termination reason explains why the task was terminated, including self-reported completion (i.e., an agent proactively terminates a task based on its belief that the task has been completed successfully), reaching the configured maximum step limit, or execution errors (e.g., invalid actions).(4) Premature termination signal binary indicator applicable only when the termination reason is self-reported completion. It is set to "true" when the success signal is "false", indicating the agent mistakenly believed the task was completed. This premature stopping reduces success rates by causing the agent to assume success before finishing the task. (5) Overdue termination signal - a binary indicator applicable only when the termination reason reaching the maximum step limit. It is set to "true" when the success signal is "true", meaning the agent mistakenly thought the task was incomplete. This results in unnecessary steps, reducing efficiency as the agent takes extra actions before concluding the task.
Consumption-relatee Metries. (6) Time spent the time taken for task execution, recorded in seconds. (7) API cost - the monetary cost incurred by API usage, measured in US dollars. However, these two metrics apply only to agents using proprietary MLLMs, as for locally hosted fine-tunds models, the time taken heavily depends on computational resources, and there are no monetary costs from external API calls.
5.2 SUCCESS DETECTION
Single-App Success Detection. We employ a coarse-to-fing suc- cess detection pipeline that uses key component matching fol- lowed by MILM evaluation. As shown in Figure 4 for each agent-task pair, the pipeline first applies coarse detection using the annotated key components introduced in Section [.2] filter- ing out trajectories irrelevant the task. If passed, fine detec- tion follows, using an MILM evaluator for final success de- termination. This approach motivated by the need to balance scalability and cost efficiency, addressing the limitations of relying on extensive human labour or expensive MLLM-based evaluations in large-scale benchmarks. We compared our single-app success detection approach with human evaluations and found it achieves an Fl score of 0.926 for English tasks and 0.884 for Chinese tasks. Further details on the single-app success detection and its performance can be found in Appendix D
and -ktby.
Detoctoo
Pap Call poeeds
Tuxk
MLM Brok
Coata Dete-
Taok Facled
Teok Suzzaen
Figure 4: An example of our single-app success detection pipeline. It features coarse detection through key component matching on execution screenshots and pre-annotated key components, followed by fine detection using MLLMM evaluation given action information.
Cross-App Success Detection. Unlike single-app success detec- tion which processes the entire task at once, our cross-app ap- proach splits tasks into subtasks and evaluates them sequentially. This is because cross-app tasks are usually longer than single- app tasks and require switching between multiple apps, increas- ing the complexity of success detection. As illustrated in Fig- ure 5 MLLM first generates subtasks based on the involved apps, followed by a human re- view. During evaluation, another MLLM splits the trajectory into multiple segments based solely on each app in the ordered list. If the segmentation is valid, each subtask is then evaluated sequentially
Bubbiook Gonoration
Suttaic
Subtenn
Stage
Tanc Twited
Sultaak 2
Toss
Figure 5: An example of our cross-app success detection pipeline that is based on subtasks instead of the entire task. The first stage involves splitting the full trajectory into segments, while the second stage checks the subtasks sequentially.

---

Published as a conference paper at ICLR 2025
until either the final subtask is checked earlier subtask fails. Our cross-app success detection method closely aligns with human evaluations, achieving an Fl score of 0.845. More details on the cross-app success detection and its performance are provided in Appendix E
EXPERIMENTS
In this paper, the success rate results were derived using the automated success detection methods outlined in Section 5.2 with GPT-4o serving as the MLLM. To account for agents with multiple variants, detailed configurations for each agent are provided in Appendix F.T Furthermore, case studies illustrating various agent behaviors presented in Appendix H
6.1 OVERVIEW OF SUCCESS RATE
Table shows the overall success rates. Notably, M3A consistently achieved the highest success rates across all task sets. We found that agents generally performed English tasks better than Chinese tasks, agents with the agentic workflow outperformed those categorised as agent-as-a-mode and cross-app tasks were more challenging than single-app tasks for agents.
Comparison in Single-App Tasks. For single-app En- glish tasks, M3A. T3A, and MobileAgentV2 were the best- performing ones, with suc- cess rates ranging from 0.640 to 0.433. These agents are equipped with reflection mod- ules that help prevent them from stalling. AppAgent and Au- toDroid performed less well, though they would likely had performed better with access to external knowledge documents, as in their original implementa- tions. For single-app Chinese tasks, MobileAgentV2 outper- formed T3A, while its perfor- mance was more comparable to M3A. potential factor could be the overly complex accessi- bility (al (ally) tree layout used by T3A. MobileAAeentV2, relying
Table 2: Success rates across all tasks and agents in this benchmark, categorised by task type. The first seven agents fall under the category of agentic workflow, while the last four belong to agent- as-a-model. AutoDroid was tested only on single-app tasks as its agent framework, Droidbot (Li et al. 2017), supports only these tasks.
Single-App
Cross-App
Agent
Overall
English
Chinese
Overall
English
Chinese
Agontic Workflow (GPT-4n)
AppAgent
0.294
0.340
0.247
0
AutoDroid
0.257
0.327
0.187
MobileAgent
0.314
0,387
0,240
0.075
0.050
0.100
Mobile/gemtV2
M3A
0.544
0,640
0.447
0.150
0,200
0.100
T3A SeeAct
0.434
0.487
0.380
0.100
0.100
0.100
0.360
0.393
0.327
0,075
0,100
0.050
Agent ax
Auto-UI UI
0.010
0.013
0.007
CogAgent DigiRL
0.027
0.027
0.027
0.010
0.020
0
0
OdysseyAgent
0,053
0.020
on OCR and raw screenshots, averaged 12,400 prompt tokens per step in Chinese single-app tasks, compared to T3A's 22,000 tokens using only al ay trees, indicating larger more intricate structures in Chinese apps. potentially contributing to the agent's degraded performance. A similar trend was observed in English single-app tasks, with lower token usage across both agents: 11,200 for MobileAgentV2 V2 and 19,700 for T3A. In general, a decrease in success rates for Chinese tasks was observed due the limited capabilities of (M)LLMs in Chinese, compounded by the increased complexity of Chinese apps. These apps often feature more intricate layouts, frequent animations, and distracting elements such ads and pop-ups.
Impact of Core Models and Input Modalities. There was a significant gap in success rates between agents using proprietary models like GPT-4o and those based on fine-tuned models. Agents following the agentic workflow significantly outperformed those in the agent-as-a-model category, the latter of which often struggled to complete any tasks. This contrasts with the high action matching scores reported in prior studies (Zhan & Zhang 2023 Hong et al. 2024 Bai et al.) 2024 Lu et al. 2024), indicating that tneettuned agents are often optimised for generating textual actions based on fixed UI scenarios. While such optimisations may achieve high accuracy in offline environments, they often fail in dynamic, real- world settings. For example, a tap action is deemed successful if its coordinates fall within 14% of the screen distance to the ground truth (Rawles al. 2024b), but this tolerance
7

---

Published as a conference paper at ICLR 2025
Table 3: Task performance on single-app English tasks. SRC and MSR refer Self-Reported Completion and Maximum Steps Reached, respectively. The execution time and token costs of the last four agents are omitted because they use locally hosted open-source models.
Agent
Success Rate
Mean Step Ratio Success
SRC Rate
Tenmination Reason
MSR Rate
Error
Temiinatio
Premature Rate
Innccuracy
Overdue Rate
Mean Exec Time per Step
Mean Token Cost per Step (USD)
Agentic Workflow (GPT-4o)
AppAgent
0.340
1.33
0.327
0.507
0.166
0.347
0.197
26.5
0.014
AutoDroid
0.327
1.10
0.593
0.340
0.067
0..994
0.078
34.0
0.008
MobileAgent
0.387
0.367
0.109
MobileAgentV2
0.433
1.05
0,580
0.420
0
0,333
0.111
56.1
0.067
M3A
0.640
0.847
0.244
T3A
0.487
1.04
0.707
0.293
0,368
0.136
9.6
0.116
AceAct
0.393
1.60
0.200
0.773
0.027
0.100
0.276
41.2
0.046
Agent-araa-Model
0.060
1,000
CogAgent
0,020
1.67
0.147
0.820
0.033
1,000
0.024
DigiRI.
0.020
1.52
0.227
0.607
0.166
0,971
0.022
OdysseyAgent
0.053
2.00
0
1,000
0.013
can cause inaccuracies with actionable elements in practice. Furthermore, reliance on predefined scenarios limits the agents' ability to generalise unseen UI contexts or recover from detoured states caused by mistaken actions. On the other hand, agents utilising agentic workflow are typically equipped with input from visual modules, such as mark-up documents and set-of-marks (Yang it al. 2023a). These layout documents are sometimes incomplete, failing to capture all available UI elements on the interface. other cases, they are unnecessarily complex for models to handle, as seen in the case of T3A mentioned above. This highlights a critical gap in grounding capabilities, which are essential for end-to-end task completion but remain challenging especially for fine-tuned models (Zheng et al. 2024).
Complexity and Memory Retention in Cross-App Task. For cross-app tasks, most agents, except M3A, completed more than 4 tasks in total across both English and Chinese apps. Although M3A performed better, completing 6 out of 40 tasks, overall performance was still low, reflecting the complexity of cross-app tasks. These tasks require more steps. reasoning, and the ability to switch between apps while retaining memory of previous actions. In some cases, agents might nearly complete the task but fail in the end due to minor mistakes missed requirements, especially in long sequences or multi-context scenarios. Even Odyssey Agent (Lu et al./ 22024), specifically designed for cross-app tasks, faced difficulty completing them end-to-end. It sometimes handled subtasks within a single app well but faltered when transitioning between apps. illustrating the challenge of maintaining context and reasoning across environments. These findings suggest that current agents, including the best-performing ones, struggle with multi-step cross-app tasks, often losing context or forgetting prior actions. This highlights the need for better memory mechanisms, enhanced inter-app reasoning, and advanced handling of complex, multi-context environments (Shinn et al.) 2023 [i] jet al., 22023b, Pan et al., 2024)). These capabilities are essential for tasks users expect autonomous agents to manage.
6.2 COMPLETION- AND CONSUMPTION-RELATI METRICS
When comparing completion- and consumption-related metrics across agents, we observed consistent trends across single-app and cross-app tasks in both English and Chinese. Since the single-app English results are the most comprehensive, this section focuses primarily on those results, with additional details available in Appendix F.2 Table 3 shows full task performance for single-app English scenarios.
Step Efficiency and Success Rate. As discussed in Section 6. agents with the agentic workflow substantially outperformed those belong to agent-as-a-model. Higher success rates correlate with lower mean step ratios, indicating more efficient task completion with fewer unnecessary actions or errors. Conversely, agents facing difficult tasks tend to make more errors, even if they ultimately succeed. M3A exhibited notably low mean step ratio of 0.92, indicating it used fewer steps than a human. This efficiency is partly achieved through combined actions specifically defined by the agent itself, where single action encompasses multiple operations, such typing in the search box
8

---

Published as a conference paper at ICLR 2025
and pressing "enter" in step. Agents may also exploit strategic shortcuts, such clicking recommended item instead of using the search bar. Thus, both approaches allow agents to reduce the steps needed to complete task.
Task Termination and Success Rate. Regarding task termination, a higher success rate generally aligns with higher Sepf-Reported Completion (SRC) rate and lower Maximum Steps Reached (MSR) rate. Agents terminated tasks either when they believed the task was complete or when they reached the step limit encounter errors. However, agents did not always accurately determine task completion, leading to discrepancies between success rates and SRC rates. This can be further analysed by examining the premature termination rate (PTR) and overdue termination rate (OTR). As mentioned in Section 5.I PTR can affect the success rate, while OTR can influence task efficiency. Notably, a pattern emerges where agents with a lower PTR tend to have higher OTR. This compro- mise likely arises from the agent's 's internal decision thresholds. For instance, SeeAct exhibited the lowest PTR (0. 100) but the highest OTR (0.276). This demonstrates a trade-off in the sensitivity of the agent's internal success detector, balancing the risk of premature termination with the tendency to extend task completion unnecessarily, An ideal success detector should minimise both premature and overdue terminations to optimise both task accuracy and efficiency.
mechanisms are cru- cial for improving success rates and ensuring reliable performance. Agents lacking these mechanisms were more prone to failure, which led to early termination when execution errors occurred. Common issues include parsing errors arising from the agents' inability to correctly interpret model outputs as valid actions. For example, the output may be missing specific phrases, such as "Thought: ", that are required by the agent's parsing module. Some agents also encountered failures when necessary inputs (e.g., XML files) could not be accessed. These failures highlight the need for better error detection and recovery strategies, allowing agents to correct mistakes and improve their overall success rates.
Enhancing Robustness through Error Handling Mechanisms.
Limitations in Cost and Efficiency for Real-World Use. While agents categorised as agent-as-a- model do not incur token costs and their execution time varies with device power, their low success rates make them impractical for deployment. Among agents with the agentic workflow, ArtdDroid is the most cost-effective, using only S0.008 per step due to text-based input. However, it has a long execution time (34 seconds per step) and success rate of only 0.327. M3A and T3A, though faster (under 20 seconds per step) and more successful, have higher token costs at around $O. 10 per step due to the complexity of inputs generated by UI elements. Mobile/geentV2, while more affordable at S0.067 per step. suffers from a complex visual perception pipeline, resulting in the longest execution time ( seconds per step). These results highlight the trade-off between efficiency and effectiveness. Agents like T3A, despite achieving relatively high success rates and faster execution times, still fall short of human-level usability due to their monetary cost. Such limitations stem from major factors. One is the delay between UI information collection and action execution, which can cause inaccuracies especially when dynamic content appears. The other is the agents' slower speeds and higher costs compared to human users. Users are unlikely to rely on an autonomous agent to complete task if they have wait for extended periods or pay several dollars, especially when they could complete it in few steps themselves.
Performance Variation Across Difficulty Levels, Open-Ended Task Experiments, and Case Studies. We compared agent performance across difficulty levels, showing that easier tasks are exccuted more successfully, as demonstrated in Appendix F.3 To further explore the scalability of our success detection approaches, we conducted initial experiments on "open-ended" single-app English tasks, detailed in Appendix G In Appendix H representative scenarios of agent task execution.
we present three case studies that illustrate
6.3
KEY INSIGHTS
To enhance the performance of autonomous smartphone agents, future research may need to address several core dimensions, including UI understanding and action grounding, dataset diversity, memory retention, reflection and error-handling mechanisms, internal task termination recognition, and execution efficiency.
First, integrating more advanced visual perception modules is essential for enhancing agents' under- standing of complex UI layouts and precise action grounding across various scenarios. Although agents using al ly trees and OCR have shown relatively good performance in English tasks, their

---

Published as a conference paper at ICLR 2025
effectiveness is still limited in Chinese tasks, which often feature more visually complex and dynamic content. Currently, some agents struggle to ground actions in these dynamic environments, often failing to recognise actionable elements map generated actions to the correct coordinates. Future designs should focus on building more robust visual models that can accurately interpret these environments and perform end-to-end task completion in interactive setrings.
Diversifying fine-tuning datasets is also essential for making agents more generalisable. Datasets should include various task instruction formats, languages, and both single-app and cross-app scenarios to better simulate real-world conditions. This would ensure that agents are prepared to handle broader range of interactions, particularly in multilingual environments where language and UI complexity vary.
Memory retention mechanisms can be improved well, especially for handling long, multi-step tasks that span multiple apps. Current agents often lose context during complex tasks or app transitions, which leads to incomplete task execution. Memory-augmented networks or episodic memory architectures could enable agents to retain context across transitions, which is particularly valuable in cross-app scenarios where agents usually struggle. These scenarios closely resemble real-world tasks that require continuity and context recall over extended sequences.
Reflection and error- prror-handling capabilities are another critical area for improvement. Many agents fail to learn from mistakes, repeatedly making the same errors without self-correction. Implementing robust reflection modules, similar to those found in M3A, would allow agents to better assess their past actions and adjust their strategies dynamically. Additionally, error-handling mechanisms, such as error identification, recovery loops, self-correction, and fallback strategies, are vital for maintaining performance in unpredictable, dynamic environments. Agents need be able detect and resolve issues such as invalid model outputs, unactionable UI elements, or parsing errors, rather than terminating prematurely or getting stuck in unproductive actions.
In task termination, agents must carefully balance premature and overdue termination. Some agents still struggle accurately determine when a task is truly complete. For example, while SecAct showed a low premature termination rate, it also exhibited high overdue termination rate. This indicates that although SeeAct avoided ending tasks prematurely, it often failed to recognise when tasks were completed, leading to inefficiencies. A well well-designed internal success detector can minimise both types of termination inaccuracies, thereby improving task accuracy and efficiency.
Finally, execution time and cost need to be optimised for real-world deployment. Agents such as MobileAgentV2, which rely on multiple modules, need to reduce overhead and streamline execution to minimise task completion time. MLLM-basee agents, in contrast to T3A, may also focus reducing input context size to lower token costs while preserving critical information for task completion. A hybrid model approach that combines the speed and efficiency of lightweight models with the robustness of more complex ones could provide a promising solution for balancing performance and
7 CONCLUSION
In this paper, we introduced agents across diverse tasks. The evaluation covers English and Chinese apps, single-app and cross-app scenarios, and varying difficulty levels. Our experiments reveal that even the best-performing agents complete less than 70% of tasks successfully, and there are significant performance gaps between agents following the agentic workflow and those in the agent-as-a-mout action grounding and generalisation within complex Chinese apps. While some agents excel in simpler tasks, their long execution times and high costs limit their practicality for real-world use. Our findings highlight the need for better memory mechanisms, robust error handling, accurate self-evaluator, improved integration of reasoning with UI understanding, and optimising execution time and cost for real-world deployment. Additionally, agents based on fine-tuned models should be adapted to diverse scenarios and focus on long-sequence decision-making makinn rather than isolated actions. as a fair and scalable benchmark. we aim to accelcrate the development
a comprehensive benchmark for evaluating smartphone
category, particularly in
By developing of more efficient, practical, and user- friendly smartphone agents.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This work was supported by the National Natural Science Foundation of China (Grant Nos. 62422605, 92370132).
REFERENCES
Hao Bai, Yifei Zhou, Mert Cemri, Jiayi Pan, Alane Suhr, Sergey Levine, and Aviral Kumar. Digirl: Training in-the-wild device-control agents with autonomous reinforcement learning, arXiv preprint arXiv:2406. 1896, 2024.
Tianle Cai, Xuezhi Wang, Tengyu Ma. Xinyun Chen, and Denny Zhou. Large language models as tool makers, arXiv preprint arXiv: 17122,,,,,,,,, 2023.
Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. Chateval: Towards better llm-based evaluators through multi-agent debate. arXiv preprint arXiv:2308.07201 2023.
Filippos Christianos, Georgios Papoudakis, Thomas Coste, Jianye Hao, Jun Wang, and Kun Shao. Lightweight neural app control. arXiv preprint arXiv:2410. 11.224400.7888.,. 2024.
Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Minlie Huang, Nan Duan, and Weizhu Chen. Tora: A tool-integrated reasoning agent for mathematical problem solving, arXiv preprint ar arXiv:2309.17452 2023.
Izzeddin Gur, Hiroki Furuta, Austin Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. A real-world webagent with planning, long context understanding, and program synthesis. arXiv preprint arXiv:2307 12856, 2023.
Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. Cogagent: visual language model for gui agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14281-14299 2024.
Hanyu Lai, Xiao Liu, lat Long long, Shuntian Yao, Yuxuan Chen, Pengho Shen, Hao Yu, Hanehen Zhang, Xiaohan Zhang, Yuxiao Dong, al. Autowebglm: Bootstrap and reinforce large language model-based web navigating agent. arXiv preprint wxXiv:2404.03644 2024.
Juyong Lee, Taywon Min, Minyong An, Changycon Kim, and Kimin Lee. Benchmarking mobile device control agents across diverse configurations. arXiv preprint arXiv:2404. 16660, 2024.
Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Com- municative agents for" mind" exploration of large language model society. Advances in Neural Information Processing Systemx, 0555199--5200. 2023a.
Tao Li, Gang Li, Zhiwei Deng, Bryan Wang, and Yang Li. A zero-shot language agent for computer 2023b.
control with structured reflection. arXiv preprint arXiv:2310.088:
Yuanchun Li, Ziyue Yang, Yao Guo, and Xiangqun Chen. Droidbot: lightweight ui guide test input generator for android. In 2017 IEEE/ACM 39th International Conference on Software Engineering Companion (CCEEC), pp. 23-26. IEEE, 2017.
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agenthench: Evaluating llms as agents. arXiv preprint arXiv:2308. 03688, 2023a.
Zhiwei Liu, Weiran Yao, Jianguo Zhang, Le Xue, Shelby Heinecke, Rithesh Murthy, Yihao Feng, Zeyuan Chen, Juan Carlos Niebles, Devansh Arpit, et al. Bolaa: Benchmarking and orchestrating llm aug-enteented autonomous agents. arXiv preprint arXiv:2308.0599 08.660600 2023b.
Quanfeng Lu, Wenqi Shao, Zitao Liu, Fanqing Meng, Boxuan Li. Botong Chen, Siyuan Huang, Kaipeng Zhang, Yu Qiao, and Ping Luo. Gui odyssey: A comprehensive dataset for cross-app gui navigation mobile devices. arXiv preprint arXiv:2406 2024.
11

---

Published as a conference paper at ICLR 2025
Jiayi Pan, Yichi Zhang, Nicholas Tomlin, Yifei Zhou, Sergey Levine, and Alane Suhr. Autonomous evaluation and refinement of digital agents. In First Conference on Language Modeling, 2024.
Christopher Rawles, Sarah Clinckemailiie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, et al. Androidworld; dynamic benchmarking environment for autonomous agents. arXiv preprini arXiv:2405. 14573, 2024a.
Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. An- droidinthewild: A large-scale dataset for android device control. Advances in Neural Information Processing Systems, 36, 2024b.
Yu Shang, Yu Li, Keyu Zhao, Likai Ma, Jiahe Liu, Fengli Xu, and Yong Li. Agentsquare: Automatic Ilm agent search in modular design space, arXiv preprint arXiv:2410.06153, 2024.
Noah Shinn, Beck Labash, and Ashwin Gopinath. Reflexion: an autonomous agent with dynamic memory and self-reftection. arXiv preprint arXiv:2303.11366 2(5):9, 2023,
Guanzhi Wang. Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint ar arXiv:2305.1629/ 2023.
Junyang Wang, Haiyang Xu, Haitao Jia, Xi Zhang, Ming Yan, Weizhou Shen, Zhang, Fei Huang, and Jitao Sang. Mobile-agent-v2: Mobile device operation assistant with effective navigation via multi-agent collaboration. arXiv preprint arXiv:2406. 01014, 2024a.
Junyang Wang, Haiyang Xu, Jiabo Ye, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. Mobile-agent: Autonomous multi-modal mobile device agent with visual perception. arXiv preprint wwxiv:2401.161555 2024b.
Luyuan Wang, Yongyu Deng. Yiwei Zha, Guodong Mao, Qinmin Wang, Tianchen Min, Wei Chen, and Shoufa Chen. Mobileagentbench: An efficient and user-friendly benchmark for mobile Ilm agents. arXiv preprint w:xiv:2406008/84 2024c.
Taiyi Wang, Zhihao Wu, Jianheng Liu, Jianye Hao, Jun Wang, and Kun Shao. Distrl: An asyn- chronous distributed reinforcement learning framework for on-device control agents. arXiv preprint arXiv:2410.1445 /4803, 2024d.
Hao Wen, Yuanchun Li, Guohong Liu, Shanhui Zhao, Tao Yu, Toby Jia-Jun Li, Shiqi Jiang, Yunhao Liu, Yaqin Zhang, and Yunxin Liu. Autodroid: Llm-powered task automation in android. In Proceedings of the 30th Annual International Conference on Mobile Computing and Networking, pp. 543-557, 2024.
Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. Autogen: Enabling next-gen I1m applications via multi-agent conversation framework. arXiv preprint arXiv:2308.08/55, 2023a.
Yue Wu, Xuan Tang, Tom M Mitchell, and Yuanzhi Li. Smartplay: A benchmark for llms as intelligent agents. arXiv preprint arX1v:2310.01557, 2023b.
Mingzhe Xing, Rongkai Zhang, Hui Xue, Qi Chen, Fan Yang, and Zhen Xiao. Understanding the weakness of large language model agents within complex android environment. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 6061-6072, 2024.
Jianwei Yang. Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Sct-of-man mark prompting unleashes extraordinary visual grounding in gpt-4v. arXiv preprint a:X1v:2310.11441, 2023a.
Zhao Yang, Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang. Bin Fu, and Gang Yu. Appagent: Multimodal agents as smartphone users. arXiv preprint a3Xiv:23/2.13771, 2023b.
Zhuosheng Zhan and Aston Zhang. You only look at screens: Multimodal Chain-of-action agents.
arXiv preprint arXiv:2309.1 1436, 2023.
12

---

Published as a conference paper at ICLR 2025
Li Zhang, Shihe Wang. Xianqing Jia, Zhihan Zheng, Yunhe Yan, Longxi Gao, Yuanchun Li, and Mengwei Xu. Llamatouch: A faithful and scalable testbed for mobile automation task evaluation. arXiv preprint arXiv:2404.16054 2024.
Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-4v (ision) is generalist web agent, if grounded. arXiv preprint 1exiv:2401.0160 2024.
13

---

Published as a conference paper at ICLR 2025
APPENDIX
A LIMITATION AND FUTURE WORK
Given that constructing tasks is both time-consuming and resource-intensive, SPA-BENCI currently includes 300 single-app tasks and 40 cross-app tasks, evenly split between English and Chinese. We plan to expand the scope of our task collection and increase the diversity of task presentation (e.g.. as explored in the initial examples in Appendix G by adding vague task descriptions and mimicking various human tones). Since some apps are difficult operate using emulators, we also aim to design tasks that can be more easily experimented with. Additionally, will execute experiments multiple times to ensure robustness.
Regarding agent framework, intend to expand the scope of our research by supporting broader range of agents. This will include, but is not limited to, locally deployable agents, cloud-base agents, and agents capable of operating other smart devices, such as Android tablets and iOS devices.
In terms of our evaluation method, particularly for single-app success detection, we plan to introduce more accurate approach and extend support for cross-app success detection. Furthermore, will define a more fine-grained metric to assess how agents complete tasks, moving beyond a simple binary success signal.
B
TASK COLLECTION
B.1
TASK APPS
The distribution and categories of apps for the 300 single-app tasks are presented in Figure
00
Bnd
Bock
03 07
80
goog
dlay
opaia
losal Becem
80
000
00
000
0000
axdin
000
Figure 6: Distribution of apps and their categories. Left: English apps. Right: Chinese apps. The circle size is proportional to the number of tasks.
B.2 LIST OF TASKS
The 340 tasks, encompassing single-app English, single-app Chinese, cross-app English, and cross- app Chinese categories, are detailed in Tables 5 respectively.
B.2.1 SINGLE-APP APP ENGLISH TASKS
14

---

Published as a conference paper at ICLR 2025
Table 4: Single-app English tasks.
App
Diff'
Golden Step
Key Components
Task Description
Level
guest
Get seurch resulls stay for aduli anywhere any week. Get the search results for stay tonight near wembley stadium' for I adult.
Airbab
2
1. guest, wembleey stadium
Airbub
3
13
1. guest, wemblee, stadium
Get the search results for stay lonight near "wembley stadium" for adult. Add one result to wishlist. Confirm that this item is in the wishlist.
Amazon
1
sunglasses
Get the scarch results for 'sunglasses".
Amazon
2
sunglasses, checkout
Get the search results for "sunglasses". Filter with "kids". Add one result to cart.
Confirmn that this
in the cart.
Amazon
3
11
goggles, checkout
Get the search results for 'goggles'. Filter with 'adult'. Add one result to cart. Confirm this item is the cart. Compare with similar items. Add one the similar items cart.
BBC
1
save save
Navigate to 'Innovation' section. Select 'Technology' tab Open any news article. Go to app settings, Change
BBC
2
10
size 'Smaller'. Navigate to 'Innovation' section. Select "Technology' lab. Open any news article. Go app settings. Change the Text size 'Larger'. Navigate to 'Business' block. Select Technologg of Business' tab. Open news article. Save this article. Go to Saved Items to confirm the article was added.
BBC
3
saved, items
Bolt
1
ciffel, lower,
Select Fiffel Tower my destination. Select Louvre museum Paris as my pick-up location. Seleet Eiffel Tower as my destination.
Bolt
2
ciffel, tower
Bolt
3
10
are. de, triomphe, holt, cash
Select Louvre museum Paris as my pick up location. Select Eiffel Tower as my destination. Add 'Are de Triomphe' as the final destination and Fiffel Tower stopping point.
Booking
1
berlin
Get the search results for stays Berlin. Select any date, rooms and guests. Navigate Flights section. Select any date. Choose flight from Manchester Airport to CDG Paris. Get the search results for reund trip.
Booking
2
11
man, odg
Booking
3
london, shunghai
Navigule to Flights section. Select one way llgght. Choose the any month the flight date. Get the search results from Shanghai to London.
Booking
1
settings settings, celsius, metric notifications
Navigate to app settings.
Booking
2
Navigate to app settings. Change Temperature 'Degrecs in Colsius'. Change Units to "Metric (km, m)'.
Booking
3
12
Navigate to app settings. Change Currency to 'Pound Sterling', Disable all notifica- tions.
Calculator
2
Get result '1+1".
Calculator
2
10
2, 3
Get the result for In))
Calculator
3
5, 040
Get the result (ogg20)+In(e)). Clear the results. Get the result for factorial 7. Check the upcoming 31 October. Click the eveat for that day.
Calendar
halloween. 31
Calendar
2
16, haircul
Set up an all all-day event tilled 'Haircul' on the 16th of any munth.
Calendar
3
14
17, dental, check, 7,
Set up event titled 'Dental Check' the 17th of any month. Set the time to from
9. pm laylor, swift taylor, swift, wiki, bookmark
7pm to 9pm.
Chrome
Get search results Taylor Swift. Get the search results for Taylor Swift. Go to her Wikipedin page. Add to bookmarks. Cheek the Bookmarks for confirmation.
Chrome
2
10
Chrome
3
16
taylor, swift. wiki, reading
Get the search results for Taylor Swift Go her Wikipedia page, Add bookmarks. Move this bookmark to Reading List. Check the Realing List for confirmation.
Clack
1
8
Set un alarm for Sam.
Clack
2
Set an alarm 9am weekdays. Set an alarm for 10am woskdays. Disable vibration for this alarm. Set another alarm 11am on weekeads.
Clock
3
15
1I
Clock
1
clock, london clock, home, hong, kong settings, analog
Add current time London (UK) to clock. Set Home time zone to 'Hong Kong".
Clock
2
Clock
3
12
Add current time at Melhourn (Australia) to clock. Change style to Analog for clock. Change style Analog for screen saver. Create contact named ''gent'. The phone number is +44 1234 567 890. Create contact named 'Ageat Two". The phone number is +44 1234 567 890, The email is benchmark @ @gmail.ccoo
Contacts Contacts
1
agent, contact agent, two. contact, grail
2
11
Contacts
3 15
thres, contact, work, gmail, huawei
Madify the last name of one the contacts to 'Three'. Update the label for the contact's 's phone number Work. Set the company to 'Huawei', Add email agent hww.m.annhmakkkaaam 2024 ymail.com. Lahel the email as Work. Get the search results for McDonald's
Deliveroo
medonald
Deliveroo
2
fries
Get the search results for McDonald's Enter McDonald's restaurant. Search for fries there.
Deliveroo
3
10
onder, fries
Get the search results for McDonald's Enter McDonald's 's restaurant. Search for fries there. Add small fries to the basket. Add two medium fries to the basket. View the basket for confirmation.
Merriam-
definition, dictio- nary, thesaurus definition. dictio- nary. thesaurus saved, words
Look up the definition of the word 'agen"".
Webster
Merriam-
2
Look up the detinition of the word 'agent". Switch to Thesaurus tab find its symonyms. Click on one of its synonyms. Switch back to Dictionary lab. 'agent". Switch Thesaurus
Webster
Merriam-
3
Look up the definition of the synonyms. Click on one of its synonyms. Switch back to Dictionary tab. Save this synouym. Confirm that synenym Get the search results for 'Klay Thompson'. Get the search results for 'Klay Thompson'. See all the articles. Open one of the articles.
find its
Webster
the saved words.
ESPN
1
klay, thompson
ESPN
2
klay, thompson, like
15

---

Published as a conference paper at ICLR 2025
ESPN
3
11
thompson
Get the seurch results for "Klay Thompson' See all the articles. Open one of the articles, Retum to the player's search results. Select the player. Turn on player news notification. Follow the player.
Evemote
1
ageut, cookbook agent, first, note
Create new notebook 'Ageat Cookbook'.
Evernote
2
Create new notebook 'Agent'. Create a new note notebook with "First note', Return to the 'Agcot' notebook to confirm the note.
Evernote
3
13
agent2, lirst. note, hello, world, lest
Create new notebook 'Agemt2' Create new note in the notebook. Write content "Helll World!' and title "Fistt note'. Create
lag 'test". Apply the lag "test" to the note. Save the note. Return to the 'Agem2" notebook.
Evernote
literature, review paper, writing. to- morrow recurring main, git, repo rome,
Create task "Literature Review".
Evernote
2
Create new task 'Paper Weiting' .Set the due date to tomorrow. Navigate to the Tasks for confirmation
Evernote
3
12
Create new task 'Maintain Git Repo' Set it to repeat daily. Navigate to the Tasks tab. Apply the recurring tasks lilter. Confirm that task exists.
Expedia
1
Check stays in Rome. The dates do not matter. Get the search results for room
and 2 people.
Expodia
2
paris, 25, 28,
Cheek stays in Paris. Choose from 25th to 28th any month. Get the scarch results for room for people.
Expedia
3
12
hong, kong, 25, 28,
Check stays in Hong Kong. Choose from 25th to 28th any month. Get the search
2
results for room for 2 people. Filter hotels with parking.
Expedia
1
paris, 25, 28
Check things to do in Paris. Get the search results for 25th to 28th of any month.
Expedia
2
11
26,
Check things to do in Rome. Get the search results for 26th 29th of any month.
Save it to my trips.
Expodia
3
paris, 25, save
Check things to
to 28th of any month. Save it to my trips. Confirm that by checking the saved Paris trip.
Paris. the search results
Facebook
1
hello, world morning honne, muit, eiffel, tower, paris settings birthday, notifica- tions notifications, email.
Create new post saying 'Hello World". Post
Facebook
2
Create new Public post saying 'Mrrnig!". Change black background, Post
Facebook
3
11
Create new Public post saying 'Bonne Nuit". Add the location as Eiffel Tower. Post it.
Facebook Facebook
1
Navigate to scttings.
2
Navigate to settings, Disallow notifications Birthdays.
Facebook
3
11
Navigate to settings. Disallow notifications for Marketplace from Email and SMS. Disallow notifications for Memories from Emnail and SMS.
Files Files
1
deim
Go to the 'DCTM' folder in the internal storage. Go the 'DCIM' folder the internal storage. Create subfolder named 'Ageet created'.
2
deim, agent, created
Files
3
agent, created
Go 'DCIM" folder the
storage.
subfolder named 'Agent
created 2', Create another suhfollde named 'Agent created 3', Then move the folder 'Agent created 2' into the 'Documents' folder in the internal storage.
Gmail
1
paper
Draft email to agent benchmark, 2024 gmail.com asking them about their paper.
Gmail
2
paper
Scnd email agent.benchmatt 2024@ gmail.com asking them about their new paper. Navigate to the Sent tab Check the email details for confirmation after sending.
Gmail
3
11
paper, scheduled
Draft an email tn agent www.heark.chccccmmmmmm 2024 gmail.com asking them about their new paper. Schedule it to be seat tomorrow morning. Navigate Check the email details for confirmation for confirmation after scheduling. Navigate to settings.
Scheduled tab,
Gmail
1
settings
Gmail
2
gmail, notification
Navigatc settings. Check current setting detifications. Turn off notification for Attachments.
Gmail
3
inbox
Navigate to settings. Check current setting notifications. Turn off notification for Miscellaneous. Disable 'notification dot', Return to Inhox. Get the search results for nearby hotel rooms.
Google
hotel
Maps
Google
2
hotel, 4
Get the search results for oearby hotel rooms. Filter the results to show only those that can accommodatee adults. Get the search results for nearby botel ruoms. Filter the results to show only those that can accommodate adults. Further filter resalts with ratings higher Get the search results for nearby gas stations.
Maps
Google
3
10
hotel, 4
Maps
Google
gas, station
Maps
Google
2
your, location
Get the search results for a nearby gas station that open now, Get driving to it.
Maps
Google
your, location, me- donald whatsapp
Get search results a nearby gas station open now, a driving route with the gas station as the first stop. Set McDonald's as the final destination. Get the search results WhatsApp.
Maps
Google
Play
Google
2
review
Gett the search results for Facebook. Leave a 5-star review on its app store page,
Play
Google
3
14
whatsapp. review, re- cent settings
Get the search results for WhatsApp. Leave a 3-star review on its app store page. Sort the reviews by most recent.
Play
Google
Check the details of General settings.
Play
Google
2
settings
Check the details of General settings. Switch dark theme.
Play
Google
3
notification, settings
Check the dctails of General settings. Turn off notifications. Confirm that notification settings for this device are off. Create new list 'Wrrk'.
Play
Google
work,
Tasks
16

---

Published as a conference paper at ICLR 2025
Google
2
lusks, buy, groceries, weekend
Create new list 'Weekend". Add task "Buy groceries'.
Tasks
Google
3
tasks, 12, visa, travel
Create list "Travel', Add task "Visa', Set date the 12th of any month.
Tasks
Instagram
messi, posts cristiono. following, message minions. notifica- tions, all odit, profile it
Get Get the search results for Cristiano Ronaldo', Follow one account.
results 'Messi".
Instagram
2
Instagram
3
10
Get the search results for 'Minions', Fullow account. Set get all notifications when they goes live. Tum on notifications for their posts. Navigate to the page edit my profile.
Instagram
Instagram
2
10
Navigate to the page to odit my prefile. Add bio 'Hello World!". Change pronouns 'it'.
Instagram
3
17
account, privacy, pri- vale note, hello,
Navigate the page edit my profile, Add link 'ttps://gttubb.cmmm gender to Custom "Them', Switch to private account.
Change
Keep
Create
Write 'Hello this note the content.
Notes
Keep
2
Create notc. Write 'Hello this a note2' the content. Write" 'Written by note, ageut, hello,
Notes
Agent2' as the note title.
Keep
3
11
agent, python, java
Create new chockliitt. Add two items 'Leam Python' and 'Leam Java'. Write
Notes
'Goal of agent' the checklist title. Label this checklist 'Agent".
Linkedln
1
following, ogenai
Get the seurch resulls for 'OpenAI.. Follow their page.
Linkedln
2
join, huuwei, groups
Get the search resalts for 'Huweii, Follow their page. Filter the search resalts Groups, Join one of the Huawei groups.
LinkedIn
3
huawei, reposted.
Get the scarch results for 'Huawei HKRC". Follow their page, Leave 'Choers!'
hkre
comment on one of its posts. Like the post. Repost the post instanily. View the repost to confirm.
Linkedla
1
engineer. jobs engineer, jobs, spain engineer, jobs, spain, saved agent, benchmark
Get the search results for 'Engineer' job.
Linkedln
2
Get the seurch resulls for 'Engineer' job in Spain. Get the search results for 'Engineer' jobs Spain. Save of the johs. Confirm it is saved in My Jobs.
LinkedTn
3
10
Microsoft
Create new page titl 'Benchmark' and content "Test Agent'.
OneNote
Microsoft
2
beachmark2, appa- geni, mobile, agent prompts, test, pages
Create new page with title "Benchmark2" and content TODO 'AppAgont" and 'Mubile Agent'.
OneNote
Microsoft
3
11
Create new notebook 'test'. Create a new section 'prompts' in "test notebank. Enter section prompes' for confirmation.
OneNote
Quora
1
scarch, openai search, openai worth, thinking
Get the scarch results for 'OpenAl'.
Quora
Get search resulis 'OpenAl'. Filter show only questions. Get the search resalts for 'OpenAl.. Filter to show only questions. Select one question or answer from the results to see more details. Add a comment "Worth thinking" the
Quora
11
Quora
following questions, follow, answer following,
Discover any Space. Follow that space. Discover any Space. Follow that space. Go to questions in the space. Filter unanswered questions. Hollow one question.
Quora
2
Quora
Discover any Technology Spaces. Follow that space. Also follow of suggested spaces. Turn off notification for the suggested space. Follow one of the contributors of the suggested space.
Reddit Reddit
1
worldnews joined premierleague, liver-
Get the search resalts for 'rhworldhews'. Join the group.
2
Get the search results for /PPremietL_aague Filter posts for Liverpool. Join the group. Click one of the posts.
pool blackmyyliwekoong
Reddit
3
11
Get the search results for wiBlackMythWukknn Join the group. Set community alerts to frequent.
one of the posts.
Settings Settigs
1
screen, timeout screen. timcout, 5, min
Check the current screen timeout.
2
Check the current screen timcout, Sot to minutes.
Settings Settings
3
10
dark, theme notification, history store, notifications instagram, storage
Chock the current screen timeout. Set it to 10 minutes. Then turn the dark theme on. Go notification settings. Tur on Notification History. Go to notilication seltings. Turn off the notification from Google Play Store. the 'Alerts' and 'Likes' notification
Settings
2
Settings
Go to notification settings. Turn Instagram. Clear the eache from storage.
Spotify Spotify
1
taylor, swift taylor, swift
Get the search results for the artist Taylor Swift.
2
Get the search results for the artist Taylor Swift. Enter her artist page. Shuffle play har playlist.
Spotily
agent, playlist. love,
the search results for the song 'Love Story' by Taylor Swift. Add song the new playlist namely 'Ageni Playlist'. Then add another song "The Scientist by Coldplay to the same playlist. Check the playlist for confirmation.
story. the, scientist
Temu
1
gaming. beadset gaming. headset, checkout checkout cal cute, cat
Get the scarch results for gaming headset.
Temu
2
Get the search results for gaming headset. Sort the result by the lowest price to my shopping cart. Confirm that this item is in the cart.
highost. Add
Temu
3
12
Get the search results for gaming mouse. Filter items priced above 10. Add one to cart.
cart. Confimm that this item
TikTok
1
Get the search results for videos about pet cats.
TikTok
2
Get search results for videos about pot cats, Comment on video with 'Such a cute cat.'
TikTok
3
13
cal
Get the search results for videos about pel cats. Comment on video with "Such Swipe through another two vidcos and like them.
WhatsApp
1
hi, you, message mark, bench, contact smart, agent, hi, message
Send a message 'Hi' to myself.
WhatsApp
new contact with the name Add new contact with the name 'Smart Agent' and (+44y7746953749, Send message 'Hi' to 'Smart Agent'.
Bench' (+44)7437321230.
WhatsApp
3
13
17

---

Published as a conference paper at ICLR 2025
x
1
agent, post,
Drall post with the content 'Written by AgentI".
X
2
agent, post, 2, reply
Create post with the content 'Written by Agen22.. Tag "aanimllrrussing" Post it.
Chesk from the profile.
X
3
12
ageut, post, 3, roply, amazing
Create post with the content' 'Written by Agent3.. Tag. "annimllrossiing Post it.
Check from the profile. Then Like Reply to
'Amazing post'.
X
1
mayday, following nintendo, super,
Search for the account @ Mayday EN, Follow it.
X
2
Search for the account @ Nintendo, Follow it, Search its post about "Super Mario'.
mario animal, crossing, timmy, tommy, post restaurants restaurants, chinese
x
3
15
Search for the account @animalerssing Follow it. Search its post ahout "Timmy and Tommy'. Repost one result. Check from the profile for confirmation.
Yelp
1
Get the search results for nearby restaurants.
Yelp
2
Get search results for nearhy restaurants. Filter to include only Chinese restau- rants that offer takcout, Sort them distance.
Yelp
3
10
review
Get the search resulls for nearby restaurants. Filter to include only Chinese restau- runts that offer takeratt Sort them by distance. Select one result. Filter for Star reviews.
YouTube
tesla, subseribed subscribed
Get search results for the channel 'Ø Tosla'. Subseribe the channel.
YouTube
2
Get the search results for the channel "ØBMW". Subscribe to the channel. Get the scarch results the channel @ Mereodes'. Subscribe to the channel.
YouTube
3
12
all. subscriptions, microsofl, google
Get the search results for the channel "Gooll". Subscribe the channel. Get the seurch results for the chamnel '6 Microsoll'. Subseribe to the channel. Navigule to the Suhscription tah. Show all suhscriptions. Sort the subscriptions from A to
Yobeube
lebron
Get the search results for videos about LeBron James.
YouTube
2
10
lebron, views comment
Get scarch results for vidcos about LeBron James. Filter vidcos under 4 minutes.
YouTube
3
14
Get the search results for videos about Lethron James. Filter videos under 4 minutes.
Select any one of the results. Leave comment 'great performance!"
Zoom
1
smartphone, agent, henchmark smartphone, agent, benchmark
Schedule meeting litled 'Smartphone Agent Benchmark Use personal meeting ID..
Zoom
2
Schedule meeting titled Smartphhoe Agent Benehmark' Use personal meeting ID. Change the timezone to Hawaii. Repeat the meeting every day.
Zoom
3
smartphone, agent.
Schedule a meeting titled 'Smartphone Agent Benchmark. Use personal meeting ID. Change the timezoue to laaaiii Repeal the meeting every day. Disable waiting room. Tum on host and participant video.
benchmurk
B.2.2 SINGLE-APE CHINESE TASKS
Table 5: Single-app Chinese tasks.
App
Diff
Golden Step
Key Components
Task Description
Level
W Mi 18t IE, BXCT,
2
3
13
SHEOTTC BT 100 TUULE
1
2
ft, Hie
3
12
1
A.
2
A-FF UR, 4 18.
A A E
3
35 2
#
57
1EBE
HiR [i)
HE 1 A
D WZ 1 A R 1UT. OF
iF.. H9149
NZ
A M
MIR. 1F AMFEN 1000
12 18"
I
ZL DJ DL S 1E
2
Ht A.
Mi
10 BREAA, AERAKE F
N 123456789
3
13
4K 11.
4: Di Mi 13 AA
XI BRARA AE FALE
1234557789,
AJE, 12111 MERE
1 Hiii
1
NE 18 HIf
A 4
#EAT
25
2
2599335
3
HH JE. ME>
A # 4
A
*
"
M
1
2
19 -
AT 1
1
[ 51.
3
11
HPP
3 9h, 12
ME (##)) 20
18

---

Published as a conference paper at ICLR 2025
# 1.
*ASHA (1
BB
2
*H (A
4 10
3
# EE. HE
MASHK (7
* 31
イ
41
1
28.09, 1E
2
5G 37000
2.
3
A.
32 13
2 E. 10
1
2
1 ft: :
# HEST. A A HUST. 11 #E
3
13
1
=
11
2
Il MA
101
1::0 Mi.
R258
3
11
# A S.C
#1
14
10000
DT, 3L 09
Fi /
12A""
2 2
1
#4 13
27 2
2
#
2
3
10
30 1911
#
877 2
1
T 2
2
10 00
#
Inl.
#M 1T
2
3
A
* W5 #
AM
1
ft B9-5 11:7
2
#M ST
10
10
1
8 tit H
2
3
10
f2 12 M. MAHHAS $.
2IR MEETR
41
1
# 13
A
M. 10 15 K
2
AAR
3
32 #
# m01
1A H 2 3t &
A M 20
1
#| 15630 A HX 1216
16
2
11
Pf BE 12:00 RE
3
1.5
BU F 16 AA T
11 ML K.HL.
1
0712. HA1 HR NF-FF 078
1H F] 16
2
10
A A
R THI, X2. A 1
3
18
A
12
1 15
*
A AM HBST4
33
2
Te f A MST H. [415]
4
10 /
14
A A
$
43 B
1
BE 11 N... BE
H 3 mL
2
A 3I I..
A M
3
11
MN A A IT.
M +
5 - SE
4 WL
1
25 0
bbbidii.com
2
# Shilibliiccom com.
21 FUJ 11 a0 aE UP
3
#5 (16
Sbiliilii com.
19

---

Published as a conference paper at ICLR 2025
10:00 Hi
1
D1 ME ELL itt Aitttobbooccom
19OD0U HE
hi, 940b BERR
20 ME EEL aljtnohaao.coo comm
70mate50pr
3
10
$ H
DAT 59 EE (R) H4 Ritaabaa.ccom
71mat660pro.
HA 7F Mo P ffr
1
# WH 2 WH $ W11,
P8 DE WH-10000 7H-10000M4
2
AO &N WH- 1000M4
ME 2000
5% 35
WIL 1000XM4
2000 Mt.
TRO NA 11
JL
10 AAA
4.
1
2
1F 1111
3
12
NI- A
1
16,60199
2
#
3
13
38 17.
2
3
1T4,311
28-44 45 D9162 M0e6 S AA
I
RE
2
"f world" "Hello world".
3
13
70
B #0
Q0 AR
1
MA M.
00
2
QQ
3
MAA 1712
TO MM
Thllo 30
QQ
I
1 A -
2
00
3
_ :A
A
2
QQ
00
2
M 1X , 33
DI M è 3T
QQ
3
#1
(x (T])
2: JL
1
B. 13.1 A
ZE. 22
AR.. 13
JL
2
* At, TE.
# 12 12
69.19 के. 190 08. 2AA 2 1.53
BH1GH 10
3
# DT 12
1
AL 42.951 18:
BR-BIA
2
81.77 12:3. 50. 19
HH
AL.
3
12
Wetil 1.X.,, 18:4
AF
M. & HE
19
10
61155 #
1
, 14 A..
2
3
# 14:1
A di
A
F.S.
1
2
94-0h
3
10
AJE 3
1
H.
2
$1 g1
#
-8 AE
3
10
AL 54
STI MA UOR
R. #1
1
#1
2
A1
* E4 tu
#
#
,
1
A i1:0
X
2
K5 1:3F
WEELA WR BEXBAN
3
10
4 $6:88
:
RUE A
15
2
$ 110
#
20

---

Published as a conference paper at ICLR 2025
82
3
4
1
PURM
2
T 0 A4
#0
3
10
S REE X 18.8. M
$1
*T
$H
2
#T W.
i.... KME. #112
R. A
1
AA
2 1U
3
15
4
MARTER REB. JE
#
2
MAIO
3
13
3 16
PA /LET"
3E (1 3 SAMIS 15, RELEF
4:
1219
1
7
A1JFF1
2
24
* A #
3
10
BM. Al
S1 20 4100 11.0..
16 FT "hello world" (1
1
ME M. A.0. FI
2
1 -
32 12
-
RE 30
9512 2
2
7
# DE
2
# #0
3
#1
RIVEE 12 (1 Mid
# 10
2
3
10
HA
3
1
#4
1
2
1 31
#
3
*
IN 5T
x
Fi dl
2
3a... 32 10
#A
10
hello, world
AA 1
1P bello
301-2
1
10 30 13 2:25
2.
3
13
5 |0 1012
32 "ello world"
3012
1
A $1
2
10
11 TF "hello world"
3
13
A0 "eello world". #155
B.2.3 CROSS-APP ENGLISH TASKS
Table 6: Cross-app English tasks.
Category
App
Golden Level Step
Task Description
General Tool
Google Play
13
Open Google Play Store, uninstall the Alibaba.com app, than go to Settings and verify if app still listed under app resources.
Setting
General Tool
Keep Notes,
1
12
Use the LinkedIn upp to search for customer service representative position. Select a joh, open Keep Nates, create new note, record the company is name, set note's 's title to 'customer service representative'
LinkedIn
General Taol
Clock, Setting
the Settings app, enable "Data Saver" mode. Open the Clock app and set an alarm for 6:00 AM.
Information Management
Faccbook, Set- ting
17
Open Facebook, search for tropical pictures, save one picture to your phone, go to the Wallpuper section Settings app, set the saved picture your wallpaper.
21

---

Published as a conference paper at ICLR 2025
Information
Calendar,
1
16
Using Chrome, search for the date of the next Winter Olympics opening ceremoon and then set a reminder for that date in your Calendar.
Management
Chrome
Information
Spotify,
Open Chrome, search for the top Country songs of 2023, identify song. from the scarch results, then switch to Spotify and add that song to your playlist.
Management
Chrome
Media Antertain-
Google
Watch YouTube video about fitess tracking app recommendation check the vidco's is description for the suggosted apps, then use Google Play Store download one of the suggested apps.
mont
Play
Store,
Youtube
Media Entertain-
Gougle Play
Utilize Chrome research different Recipe Organizer apps, and then proceed Google Play Store, download one of your choice.
ment
Store, Chrome
Media Entertain-
Clock,
Search for relaxing soundscape video Tube, the Clock app to set a timer for 3 hours, thea go back to YouTube and play the video.
ment
Youtube
Multi Apps
Quorn, eBay, Chrome
20
Utilize Chrome to search for biography book, then use Quora to read reviews about the book, and finally add the book to watchlist eBay. Organize movie night by choosing a horror film using Chrome, sending an invita- tion of your friends via Instagram, and setting reminder the Clock upp for 8:35 PM on Sunday.
Multi Apps
Clock,
2
20
Chrome,
Instagram
Multi Apps
Triller, Sct-
First, install the Triller app from the Google Play Store. After the installation, open the Trille app. navigate to the Setting app to check current battery status, reopen the Triller app.
ting. Google
Play Store
Multi Apps
Clock, What-
2
23
Arrange business meeting using Zoom, copy the sharing text, go to Whats.App send the copied text to a contact, set un alarm using the Clock app at the meeting time.
sApp. Zoom
Multi Apps
AccuWeather,
2
25
Utilize Expedia to search for Things to do in Beijing on 18-20th, choose one and record the sharing text using Evernete, open Accu Weather chock daily weather Beijing.
Everoote
Expedia
Social Sharing
X, Facebook
20
Use the social media platform to post photo, copy the link to your post, then open Facebook and send the link friend Use the BBC News app to search for Artificial Intelligence news, read an article, share via Gmail, send agent henchmankcc.akk..m 20244 @ gmail.com.
Social Sharing
BBC
News,
10
Gmail
Social Sharing
Spotify, Facz-
19
Listen to Reggacton album on Spotify, then share the albumå name with a friend Facebook.
book
Web Shopping
eBay. book Amazon.
Face-
15
Search for "Circe by Madeline Miller' on Facebook, read one of the posts, head over to eBay, search the book, add it watchlist.
Web Shopping
15
Investigale the prices for Catan bourd gume across Amazon and Temm, then pruceed to add the cheaper option into your cart.
Tema Airbnb, Insta-
Web Shopping
Use Instagram to search for itinerary for Venice, Italy, and then proceod Airbnb, book accommodations at Venice, Italy.
gram
B.2.4 CROSS-APP CHINESE TASKS
Table 7: Cross-app Chinese tasks.
Category
App
Diff
Golden
Task Description
Level Step
General Tool
827 22, REM
#1 2.
MPPP. #MHTRAX 2,
Hi FARMA MRL
General Taol
B. 34:66 ei
6
General Tuol
H H W. DASTTE
3y KEEt"
Information
4 4
18 83.
1
9
#E44 ANN - A Abblibii (4 11
Management
bilibili
Information
4 24 100
12 88.
#: Ja It
22 (2 * 2024 1224 #1
2X d
Monagement
QQ 95
BIQQ
Information
18
/
#8 11
Management
Media Entertain
#6 94
[3)
12
<E* 11
1100
ThE
ment
QQ #4 1
F
Media Entertain-
16
81 4 # 2 18' "BLACCKINNK
33 #0 H.
ment
ACLKIIKINKK
Media Entertain-
QQ 15. bill- bili
12
HE 13
HH
ment
Zbblibili 4 BUN 13. 2. AEA HEAE
Muli Apps
4 5 300 K2 H9
2 2 - 19 ME 4A 11
2008 1E
bilibili, QQ
Multi Apps
My 2t S *.
2
18
RUJ S: 1U 5F M3Mate60Pmm 389 23.3 SR 87 9.000
Maro 13
Multi Apps
38
# [T M Ae
EL N
Multi Apps
1 JL, BLA
2
21
HLA #
Multi Apps
80) H. #9
2
16
#F 4I
Hots HE-EE 1o it OT 1TM ae
# E
& RI ha 41 10 (9 EE
Al 4 E3 12 44
Social Sharing
bilibili, QQ
9
Zbilibil 41 M E 1D
A
#l $2
Social Sharing
KC QO
#
#FQU F HAnk- Mi ME Ai
5
22

---

Published as a conference paper at ICLR 2025
0H4
WiDoomed
Srae
X
Dhopping
28 results for "ines
Delivery
All results for MicDonolde *
Langeirimiive Jv13
Dellwwr 15
NcDonaldss
20
Smail Fries WFF
P1.50
Pepulrved (nderi bassuu
Smaall Frine (V])
2u77 aull
chicken
Hardelss
disporable
W
e
y
U
Fees
f
9
9
h
Stril Inder
12.00
+
z
b
n
m
P123
@
(a) Level I: "medonald"
(b) Level 2: "fries"
(c) Level 3: "order" and "fries"
Figure 7: A visualised example of key components across three diffieulty levels, with subcaptions indicating the key components for each level and highlighted key components in the corresponding screenshots.
Social Sharing
1. 6 30 #
A - 1:
Web Shopping
#M 1000
-
Web Shopping
14
AG
Shopping
4: _
(00 AZ
Armate Pi F.Lm) #
#
B.3 EXAMPLE OF KEY COMPONENTS
Figure 7 shows an example of key components.
B.4 CROSS-APP EXAMPLE TASK DEMO
Figure illlstrrates two examples of English cross-app tasks, each with a different difficulty level.
B.5 STEPS OF TASKS
Refer to Figure for box plot illustrating the distribution of steps across tasks.
C INTEGRATED AGENTS
The benchmark includes 11 statc-of-the-ar autonomous agents, shown in Table These agents differ in core models, input modalities, action spaces, and additional training or prompting modules. They fall into two categories: those leveraging off-the-shelf MILMS (e.g., GPT, Qwen), and those using
23

---

Published as a conference paper at ICLR 2025
Level Task: Using Chrome, search for the date the next Winter Olympics opening ceremony then set reminder for that date in your Calendar.
Chrome
Calendar
Task difficulty
Level
Level
Cross Apps
Level Task: Arrange business meeting using Zoom, copy the sharing text, go WhatsApp, send the copied text to contact. set an alarm using the Clock app at the meeting time.
Apps
Apps
Zoom
WhatsApp
Clock
Figure 8: Example cross-app tasks with trajectories collected by human annotators.
Single-App Tasks
Cross-App Tasks
25.
Number of Steps
10-
ENG Diff CHN Diff
ENG Diff
CHN piff
ENG piff
CHN Diff
ENG Diff
CHN DIK
ENG DIT
CHN Diff 2
Figure 9: Distribution of steps taken by humans to execute tasks, categorised by difficulty level and task type.
fine-tuned models with parameter counts ranging from 1.3 billion 18 billion. Fine-tuned models, trained primarily on the offline AITW (Rawles et al. 2024b) dataset, focus on action prediction, with DigiRL additionally employing online RL training. In our benchmarks, unlike their offline training settings, all agents are tested in real-world scenarios that require precise action grounding and long-sequence task execution.
C.1 AGENT INPUT MODALITIES
Input modalities and action spaces define an agent's ability to interact with mobile user interfaces.
Screenshot input is intuitive, capturing everything human would see, but MLLMs often struggle to identify actionable UI elements and link them with screen coordinates (Zheng et al. 2024). To
address this, some agents enhance input with XML files, accessibility trees, information obtained through Optical Character Recognition (OCR). For instance, AppAgent (Yang et al. 2023b) and AutoDroid (Wen al. 2024) element IDs and coordinates, M3A (Rawles et al. (024a) annotates screenshots with key UI elements, while MobileAgent (Wang et al., 2024b) first identifies interaction elements and then uses OCR icon recognition locate them.
24

---

Published as a conference paper at ICLR 2025
Table 8: Comparison of agents integrated into SPA-BENCE framework across key dimensions.
Agent
Core Model
UT Representation
Touch Point Localisation
AppAgent
Yang al.
2024
GP1-44
Screenshot XMI.
Coordinates from XMI. Coordinates from HTML
AutoDroid Web
2024
GPT-4o
HTML
AobileAgent Wang ct al
2124b
GPT-4o
Screemshot
OCR + Icon Recognition
MohileAgentV
Wain el
20248
GPT-4o
Screenshot
OCR + Icon Recognition
Screenshot
Coordinates
MBA
Rawles et
2024a
GPT-4o
Accessibility Tree
Accessibiitt Tree Coordinates from Accessibility Tree Cooedinates from Accessiilitty Tree
T3A
Rawles ctal.
2024a
GPT-4o
Accessibility Tree
Screenshot
2024a
Zheng
2024
Accessibility Tree
Auto-LI
Zhan & Zhung
2023
Fine-taned FLAN-Alpsca- Basc --.-I-2-TS.Imsteee (I.IB)
Screenshot
Normalized coordinates Model
Normalized coardinatos
CogAgent Hong al.
2024
CogAgent-188
Screenshot
Model
Fine tuned FL.AN-Alacu-- Base (200M) BLIP-2-TS Instrct (I.IB)
Normalized coordinates
DigiRI. (ai al.
2024
Screenshot
from Model
Normalized coordinates
Odyssey Agent Lu et al
2024
Fine- tuned Qwen- Wwenn-L((C (9.68)
Screenshot
from Model
C.2 ADOPTION OF AGENTS INTO FRAMEWORK
Integrating agents into the framework required several adaptations. We used their original open- source implementations, with the exception of SeeAct (Zheng et al. (2024), for which we adopted AndroidWorld's s action grounding module. For agents using fine-tuned models (i.e., Auto-U DigiRI., Agysteyggent, CogAgent), which lacked direct Android interaction capabilities, we used UlAutomator end-to-end task execution.
C.3
LOGS AND ERRORS
While task descriptions and screenshot trajectories remain the primary inputs/outputs, we also logged executed actions, performance metrics (steps, time, API costs), and errors. Errors were categorised expected (e.g., invalid responses) or unexpected (e.g., network failures). Expected eиrоrs arise from the agent's limitations, such as failing to generate valid actions or when certain functionalities are restricted. Unexpected errors refer unforesceable issues like network fallures, Android malfunctions, or CAPTCHA challenges. The framework automatically re-runs such tasks to avoid penalising agents for unexpected errors, ensuring fair and accurate assessment of their capabilities and limitations.
C.4 SCOPE OF USING ANDROID EMULATOR
Certain English tasks involving WhatsApp and OneNote, as well as most Chinese tasks, were executed exclusively physical Android devices rather than emulators] This decision due to strict app control measures, such as restrictions on logging in across multiple devices and compatibility issues with emulator system images. While physical Android devices replace the emulator, doing so would eliminate the snapshot functionality described Section 4.2
D SINGLE-APP SUCCESS DETECTION
D.1 COARSE DETECTION: KEY COMPONENT MATCHING
Given a single screenshot, PaddleOCK is used to extract text, which is then lowercased and con- catenated to minimise inaccuracies. This text matched against key components of the final state (defined by human annotators in Section 3.2 Matching starts from the last screenshot and moves
Auto-UI UI has been renamed to Auto-GUI, but in this paper, we use Auto-UI as it is more commonly referenced in previous works.
25

---

Published as a conference paper at ICLR 2025
Table The proportion reduction in MLLM evaluation times through key component matching. and the Fl score performance of our MLLM evaluator (without key component matching) across reasoning and action modes. Bold values indicate the best performance for each task and language pair.
Task
Language
Reduction Rate
No Acticn
Text Action
Image Action
Result-unly
Reason and Result
Resull-ont
Reason and Resali
Result-unt
Reason- and Result
Single-rp
English Chinese
0,313
0.911 (-0.003
0.922 (-0.035)
0.919 (-0.016)
0,903 (-0.00))
0.926 (-1.006)
0.915 (-0.050)
0.670
(40.776)
(40.092)
(-0.113)
0.864
backward until match is found the first screenshot is reached. If no match is found, the task is marked as failed, skipping fine detection.
D.2 FINE DETECTION: MILM EVALUATION
If coarse detection is successful, fine detection is performed using a MLLM evaluator (based on GPT-4o). The evaluator receives task descriptions, screenshots, and executed actions to assess task success. Action information be presented either text concatenated screenshots. Prompts used for the MLLM evaluator are detailed Appendix D.4
D.3 APPROACH EVALUATION AND RESULTS
To validate the single-app success detection pipeline, we compared its detection against human evaluations for AppAgent and M3A (English tasks), and CogAgent and MobileAgentV2 (Chinese tasks). Two reasoning and three action modes were tested to prompt the MILM, and an ablation study was conducted to assess the impact of coarse detection.
Table presents the proportion of fine detection time reduced before and after applying coarse detection, along with the Fl scores for each reasoning and action mode across English and Chinese tasks, both with and without coarse detection. The results demonstrate that coarse detection effectively enhances performance by reducing the frequency of fine detection calls and improving the success detection F1 score, particularly in Chinese tasks where MILMM evaluation struggles. While significant differences were found between reasoning modes, incorporating action data improved decision-making but also increased token length, which sometimes led hallucinations.
Overall, in the best test-performing evaluation modes, our pipeline achieved Fl scores of 0.926 for English tasks and 0.884 for Chinese tasks, demonstrating its effectiveness in aligning with human evaluations. For further task evaluations, we use these modes to detect success: result-only reasoning with image action for English tasks, and reason-and and-result with text action for Chinese tasks.
D.4 PROMPTING TEMPLATES
D.4.1 SYSTEM PROMPT
You are an expert in eva pvaluating smartphone oparation tasks. Your primary role la has successfully completed screenshots (provided in order of execution) and the corresponding task description. 44 Guidelines:
determine whether
secies
1. AANo Assumptionswv: Evaluste solely based the provided screenshots. not infer er assume details that aren't explicitly shown.
is sucnesaful only whan all its subtasks sucoesatully completed. For example, for the taak "Go to the webaite althub.com. Adc. this website to resding list, it is successful only
2. **Subtook Completionne,
screenshots show fithub.com has been navigated to and then added to the reading list.
3. a*Common Reasons for Subtask Failuress:
subtask is not successful if it is net performed schieved. Same task example abave, visiting the website but not adding it to the reading list results task fat lure.
->.Insompetess:
Incorrrett Execationaw: à aublask fails 1f the screenshots do not align with any part instruction.
26

---

Published as a conference paper at ICLR 2025
website github. com.. but screenshots show google.com, the subtask fails. Similar entities (e.g., (ishone 11' ve. "ipaone or "driving direstions' va. 'walking directions") are considered different, leading to task failure if not correctly executed, +*Rrong Verb/Action-<; If the subtask is "Like a post," bat the screenshote show the reposted inatead, 4. vxddiitional Actionsas; II intermediate scieenshels show all sublaska ace successful, consider
vilrong Houn/Entityww:
subtask
f due 'ncorreat action.
success, even additienal actions are This applies as long these actions do not impact task completion or cause the original taak
afterward.
5. *filttering Subtaskes: II aubtask involves fittering based specific critezla, ensuze the filter has been applied (i.e., specific app festure). If the filter is additional
condition, the subtask 6. AxOrdez of Subtasksea: Subtasks can be completed in any order unless they are explicitly dependent each other.
7. **Bubraaks Completed Midway==: Schtaaks complated in the middle of the process not reflected the final screenshot; these shoule st be conaidered successful If they align with the task requirements.
8, **Corrective Actionsse; Subtasks that initially appear to fail but are corrected should considered successfui only aligas with the criginel task.
by subsequent
correction Tully
isn't completed one go, long ss the final result neets the task rements; consider this success.
*nintermediate Stepa*r: it's acceptable
10. A*Focus on Overviewrs: Pay attention the overall objective and avoid letting minor, irrelevant details distract from the main evalustion.
11. tvul Differenceswe: Be mindful of subtle ui differences le.g., different font styles ptrprpsss_ooomp_eeppp_e (ectionnacti
indicating selected tabs).
*+These guidelines serve as general framework. Apply them thoughtfully avoid overfitting edge cases not covered. Be atrict and caut tous when determining whether Lask has been auccessfully completed or net. Use to indicate success and indicate failure...
D.4.2 SYSTEM PROMPT WITH ACTION
quicx pop-upe may captured by screenshots provided. If zeeded, consider the action information when evalusting the
AvUse Action Informationns
13. **Single Action for Multiple subtoskaev: Some subtasks can be completed with single action, such aa clicking an icon that shuffles playlist.
44 Commoa Actions: Click/Tap: The selects activates specific point the screen, triggering an event or interaction.
point Lrigger secondazy sction swipe/Scroll: The user drags their finger acrozs the screen to scrol. or navigate; the content screen positien changes according
Long Press:
user presses
direction.
Type/Tsput Text: The user types or inputs text into field.
Back: The user presses the back button return to the previous screen,
D.4.3 BASE PROMPT
New, here is smartphone operaticn tosk description:
ketask_deaer www.errprtiionee history info
Please carefully detarmina whether the task been correctly and completely exceuted sccording to the provided screanshots. Iise 1 to ind cate success and to indi cate failure.
sction_prospt(]]
reasoning_prompe
Remember:
not make assumptions based on information not presented in the screenahots. Only evaluste what is explicitly shoun.
Ensure that every entity and action in the taak description is precisely matched and fulfilled.
Consider additionsl actions taken after task is successfully completed as part tha success, as lang as those actions dan's impact the teak's complation or cause fai lare,
A filtering subtask only correct when apeciñic filter is agplied as feature of keyword search wil" Subtasks can be completed in any order unless they are explicitly degendent on each other.
the app. daing ariteria
the subtask fail.
Subtasks completed correetly mid-process, even if not reflected in the final screenghot, should ba considered successful.
27

---

Published as a conference paper at ICLR 2025
initislly appear
corrected by subsequent actions should
be considered successful.
successful even some subtasks not completed ane ge, long as the tinal reault neets the taak requi rements. Pocus on the everall objective of the Lesk without being distracted by minoz, trralevant 6etaf
Pay Lent to subile ur differences that might indicate task completion or failuze, changes in feat.
highlighted prtmon_ [1]
D.4.4 BASE PROMPT WITH TEXT ACTION
assist you determining whether is provided. Jae this information only when cannct datermine succesa purely 1-th screensho: nay contain detaila that change the
successful,
information
based the screenshota. screenshot from the i-th to the i+1-th, while the last screenshot contains action Informat ion as the task ends afterward. Tn some sereenshats, red dot may ind care specific action occuired le.g., clicked or Lateraction. If there isn't a red dat, the action is more complex than single details these
long pressed), trişgering
pesiticn operation. (e.g., a swipe text imput). actiona bel if applicable. extra_action
information only necessary. Pop-ups that appear immedistely after an actien may not be captured in the screenshots; do consider this Some subtasks can be completed with aingle action, such elicking an fcon that shuffles playlist.
Censider
D.4.5 BASE PROMPT WITH IMAGE ACTION
To aasist in determining whether the cask succesatul, action information is provided. use this information only when you connot determine succasa purely based on the screenshots. from the i th screenshot to the 1il th screenshot, while the last screenshot contains afterward.
sction information
describea the changes
action information as white strip attachad to the original screenshot, saperated by blue line. In some screenahots,
information presented
or dot may indieste where specífic occurred le.g., leng pressed), triggering
interaction.
Consider the action infermation only when necessary.
Pop-ups appear immediately after may captured screenshots; do not bonsider this foilure. Some subtasks can be completed with aingle action, such as clicking an lcon that shuffles playlist.
D.4.6 RESULT-ONLY PROMPT
Please provide your decision using the follewing template without any reasoning:
Result:
D.4.7 REASON-AND-RER RESULT PROMPT
following
your respense:
Reason: <Briet description of why believe the task auccessful falled, including the aligament or misaligmment between the task description end screenshoss, starting with "Σ believe this task mucosssull/aiiedd. Result: OR 0>
28

---

Published as a conference paper at ICLR 2025
W
Coarse Detection
wembley
stadium
Extracted Text
guest
Matched e
Fine Detection
Image-acion
Image Trajectory
FINEH
Result:
Figure 10: Evaluation of the "airbnb_ 1" task executed by M3A. All four annotated key components were successfully matched in the OCt-extracted text from the final screenshot, allowing the task to pass both coarse and fine detection.
D.5 EXAMPLE OF SUCCESS DETECTION
Figure 1O illustrates a coarse-to-fine evaluation of the "airbnb_1" task executed by M3A, which corresponds to the Airbnb Level task listed in T
E CROSS-APP SUCCESS DETECTION
E. SUBTASK GENERATION
For crpss-app task, each subtask is tied to single app, and any adjacent subtasks must use different apps. However, the same app can appear multiple times as long as there is at least one different app between occurrences. Beyond "app" and "task description", each subtask also includes the fields "history" and "memory". The "history" field is boolean value indicating whether the subtask requires information from previous tasks, highlighted as phrases in the task description. This information, referred to as "memory", consists of phrases that will be matched with the highlighted "history" phrases. Such subtasks are generated by MILM and then reviewed by humans to ensure quality. Examples of subtasks are provided below, and detailed prompts can be found in the Appendix E.5
E.2 STAGE 1: TRAJECTORY SPLIT
Stage splits the entire trajectory into segments based solely on app transitions as preparation for detecting subtask success. The previous subtask generation step provides an ordered list of apps for each task, indicating the sequence in which they should be operated for successful completion. A MLLM processes this app list along with the complete series of execution sereenshots, segmenting the trajectory that each part includes only sereenshots related to the corresponding app's operations. the segmentation is invalid, such as when an app is missing or the sequence is incorrect, the task is marked as unsuccessful due errors in more apps.
29

---

Published as a conference paper at ICLR 2025
E.3 STAGE 2: SEQUENTIAL SUBTASK SUCCESS DETECTION
Stage 2 is activated when the segmentation is valid, meaning each app in the ordered list has unique series of screenshots. Subtasks are checked sequentially, with each subtask evaluated only if its predecessor is marked as successful. If a subtask is marked as successful, the phrases in its "memory" field (unless the field empty), will be required as historical references for subsequent subtasks. This memory is generated by another MLLM, which summarises the current screenshots based on the required phrases and appends the relevant information to the memory set for future use, If a subsequent subtask's "history" field is marked true, the necessary phrases are then extracted and matched with the stored information to assist in evaluating success. Such historical data, combined with partial task screenshots and action details, is used to determine the subtask's ' success. Since each subtask involves only a single app, it uses the same MLLM evaluation method applied in single-app success detection. The entire task is considered successful only if all subtasks pass. Otherwise, fails as soon as any subtask is marked unsuccessful.
E.4 APPROACH EVALUATION AND RESULTS
To validate the cross-app success detection pipeline, we com- pared its results against human evaluations using four different agents per language. For English tasks, the agents were M3A, T3A, Auto-Ul, and Odyssey Agent, while for Chinese tasks, we used AppAgent, MobileAgent, MobileAgentVV2 and CogAgent. TOble|[0ppreeeett the F1 scores our cross-app app success detection pipeline for both English and Chinese tasks. The performance is lower compared to single-app success detection due to the in- creased complexity of cross-app tasks. With over 90% of tasks being true negatives, even small number of errors significantly impacts the overall performance. Additionally, we observed that for each agent, false positives and false negatives occurred at similar rate. Thus, despite relatively modest F1 score, the pipeline's success detection still reflects each agent's performance.
Table 10: The score perfor- mance of our cross-app success detection pipeline.
Crossapp
English
Chinese
Score
0.833
E.5 PROMPTING TEMPLATES
E.5. SYSTEM PROMPT OF STAGE 1
You provided with sequence of screenshets representing agent performing tasks across multiple apps on a smartphene. Each screenshot correspands to a specific action. You also given 1ss apps that should used in the taak. *+Your task is to:*a
1. Split the screenshota 'rto segments based on tranaitions between apps the gf ven list, Do not change the order of apps, even if they do not match the acreenshot order. Output the results based on the provided app 1ist order.
2. each app, identify where the agent cpens and operates within the app. Each app interaction requires at least two screanchot.s: one for opaning the app and one for quitting or awitching another, except for final ap9, which may not requive quit action.
3. **Enaure that the start and end indices you provice are within the range of screenshols sent to you. You will zeceive certain number of screenshots, and you must zepest how many screenshots you received before processing. Any indices previded should excead the total number of screanshots.
4. Tf an app from the 1sst is missing in the screanshota, returr 1-1 for both the screenshot Indices 5. Ignore screenshots that show isrelevent actiens (e.g., the hone screen or unrelsted apps). You may mention them in the analysia but do not include then in the final result.
and
that app.
6. An app may appear more than once in the list (e.g., ''"AppA", "App=", "Appa*) N, but there must anather app between repeated instances of the app.
7. There might be distractors (e.g., advert isementa and popups! in the screenahots;
you should not interpret then as transitions between apps.
** example Input:
<ppp list:va "["AppA", "AppB", "Aopā"]"
**Scrennhotss: sequeace of numbered screenshets.
#t example Reasoring: 1. **Screenshets 1-3:.. The agent opens Apps, and eperates within it. v*Serensshats 4-51++ The agent opens App5 and operates within it. 3. ** Sczeenshot 6:se The agent interacta with the home screen, which is ircelevant. 4. Screenshots 7-3:*+ The agent opens Appå again and operates within it.
30

---

Published as a conference paper at ICLR 2025
Final Output: "A"pA_1"; "start screen";
*nd screen"; h, "App3*;
"start screen": 4, *end screen": 5 b "Appa_2:: 1
"start screen": 7, "end screen";
wttak_cossrlpptonn
E.5.2 USER PROMPT OF STAGE
Hera ta the app 1 task_app Ensure the order appa your final output is exactly the same as the order provided in ny app 1ist.
E.5.3 SYSTEM PROMPT OF STAGE 2 MEMORY
You an MLAM tasked with ana lyzing screenshots and summarizing the relevant information based on a description provided by the user. Only summarize information from screenshots that relate
the description, ignoring that unrelated. If the acreenshots ahow a 1ist of results (e.g., search page), summarise or 11at all be step-by-step details, or line breaks.
relevant results.
summary
concise, without bulles peints,
E.5.4 USER PROMPT OF STAGE 2 MEMORY
Here is the description: memory_text
E.5.5 SUBTASK GENERATION
You tasked with splitting smartphone control instruction into series subtasks, each corresponding to specific interactions. For each subtask, should define:
1. **pppet: The name of the being used in the subtask.
2. istaskee: A string describing the actien
performed. Do not include the the task description unless necessary (e.g., if the task the app). use "[PREVIOUS MENORY]' if the task depends en informstion from a previous previous subtask's memory (1.e.,
app
enly open
subrask. This should he exactly if history is True].
phrase
('Trus" or 'False') esting whather relles data Trom previous aubtask.
sshistorye:: A boolean
1. *Amеmozyas: If applisable, specify a piece of information that the current subtesk generates retrieves, which will be passed the subtask. if no memory needed, set this to 'Kone'. .*Guidellnesww
there are several consecut ivve subtaska for the aame app, comatne them into a single use the same language for the split task as the task description.
app) Sublesks for the sane spp are acceptable if there is at least one subtask fer different app in bezween. default, each subtask should independent unless explicitly needing dats from prior aubtaak (In which case, set vehlatory": True'l. Flexibly
sublask (i.e., adjacent sublaska should not have the
whether any information to subsequent tasks, based on the task's naturel requizements.
**memozy**
Output the aubtaska "subtasx_i":f "ApP" "TASK""""'S", "history":"B00L" "memory":"MEMOR"" ), "subtask_2:: "Ape":"A""", "APe", "cask":""ASk"" "history":"soe" "memory" "KENORY" 1, ##
structured format like the following:
*vTakk.:: Adjust the notification settings for the YouTube app on your phone vsing Settings, proceed open YouTube. **Resultv::
"subtask_i": "app", "Settingg" "Lask";"Adjuet the notification settings for the YouTube app your phone", "history"ffalse, "menery": "Nonn"" }, "subtask_2": "app":"Youtuba" "task*;"Upen Youlube", "sistory":falses, "memory"; "Hone" #* 5xample
TTaaken; Utilize the X app to research and identiiy aighly reconmended roboLic go
vacuum cleener,
purchase one.
31

---

Published as a conference paper at ICLR 2025
Table 11: Task performance single-app Chinese tasks. SRC and MSR refer Self-Reported Completion and Maximum Steps Reached, respectively. The token costs of four agents are omitted because they use locally hosted open-source models.
Agent
Success Rate
Mean Step Ratio
Success
SRC
Tenmination Reason
MSR
Error
Temiinatio Innceuracy
Premature
Overdue
Mean Exec Time per Step
Mean Token Cost per Step (USD)
Agentic Workflow (GPT-4o)
AppAgent
0.247
1.66
0.100
0.393
0.507
0.600
0.407
25.6
0.013
AutoDroid
0.187
1.25
0.567
0.360
0.073
0.729
0.111
48.8
0.011
MobileAgent
MobileAgentV2
0.440
1.28
0.460
0.487
0.053
0.333
0.274
104.5
0,075
0.447
0.640
0.323
20.8
T3A
0,380
1.31
0.507
0.493
0.408
0.162
12.6
0.128
AceAct
0.327
0.067
0,927
0.006
0.300
0.302
23.0
0.050
Agent-as-a-Model
CogAgent
0,027
1.79
0.060
0.893
0.047
1.000
0.030
DigiRI.
0
0.387
0.520
0.093
1,000
0
OdysseyAgent
0.007
2.00
0
1,000
0.007
Rasult+*:
"subtask_1"; "app";"X"; "task"; "Reseerch and identify a highly recommended zebotic vacuum cleaner", "history" [alse, "memory"; :"rbbttic vacuum cleanes" h, "subtask_2": "app": ""mazoo", ""Go to Amazen to purchase (rchotic vacuum cleoner]*, "history":trus "memory" "Nene" New, for amartphone control instruction ion, decompose the task into subtasks using the format above.
F EXPERIMENT DETAILS
F.1 AGENT CONFIGURATION
The agents in this benchmark include variations in core models and optional modules. Of the agents, 7 originally used off-the-shelf (M)LLMs such as GPT-4V and Qwen- VL-Max. For consistency, these agents were upgraded to GPT-4o, including replacing MobileAgentt22's Qwen-VL-Chat with GPT-4o-mini for icon recognition. For Auto-UI and DigiRL (fine-tuned), the Auto-UI-Ba core model was selected.
Agent-specific configurations include:
AppAgent, SeeAct, M3A, and T3A: Added AdbKeyboard for Chinese character input, following the MobileAgent setup.
Auto-UI: Enabled "action history" and "chain of actions" features.
OdysseyAgent: Enabled action and screenshot history.
AppAgent and AutoDroid: No additional knowledge or exploration was allowed before experiments.
For all other settings, the default configurations provided by the developers were used. Agents were allowed to execute up to twice the number of "golden steps" for a task, after which execution was halted.
F.2 EXPERIMENTAL RESULTS
12 for the detailed experiment results of single-app Chinese, cross-app English, and cross-app Chinese tasks respectively.
See Tables
https:///iittub..comm///
32

---

Published as a conference paper at ICLR 2025
Table 12: Task performance on cross-app English tasks. SRC and MSR refer Self-Reported Completion and Maximum Steps Reached, respectively. The token costs of four agents are omitted because they use locally hosted open-source models.
Agent
Success Rate
Mean Step Ratio
Success
SRC
Tonmination Reason
MSR
Error
Teminnatio Innccuracy
Premature
Overdue
Mean Exec Time per Step
Mean Token Cost per Step (USD)
Agentic Workflow (GPT-4o)
AppAgent
0
0.200
0.550
0.250
1,000
22.9
0.014
MobileAgent
0,050
2.00
0.100
0.900
1,000
0.056
25.3
0.089
MobileAgem
0.100
0.250
M3A
0,200
1.16
0,700
0.300
0.714
0
17.3
0.082
0.100
0.600
0.833
12.1
ScoAct
0.100
1.52
0.150
0,850
0,333
0
19.9
0.043
Agent dir-d
Auto-UI
0
0.100
0,800
0. 100
1,000
0
CogAgent
0.050
DigiRL
0
0.050
0.550
0.400
1,000
0 0.007
OdysseyAgent
0.650
0.350
Table 13: Task performance cross-app Chinese tasks. SRC and MSR refer to Self-Reported Completion and Maximum Steps Reached, respectively. The token costs of four agents are omitted because they use locally hosted open-source models.
Termination Reasom
Termination Inaccuracy
Success Rate
Step on Success
Exec Time per Step (sec)
Token per Step (USD)
Agent
SRC
MSR
Errur
Premature
Overdue Rate
Rate
Rate
Rate
Rate
Agoutic Workyloww (GPT-4n)
AppAgent
0
0.450
23.5
MobileAgent
0.100
1.62
0.150
0.750
0.100
0.667
0.067
53.4
0.064
Mabile/gemtV2
0.100
1.89
0.200
0,750
0.050
1.000
0.133
104.1
0.075
M3A
0.100
1.32
0.500
0.500
0.800
17.8
0.091
T3A
0.100
1.08
0.750
0.250
0.867
0
13.4
0.110
0,050
1.000
0.056
Ageni-oa--- -oid---------e
AutoU!
0
1,00
1.000
CogAgcat
0
0.050
0,850
0.100
1,000
0
DigirlAgent
0.800
0.150
1,000
GUI GUrr_ddyssey Odyssey
D
0
0.500
0.500
0
F.3 PERFORMANCE ACROSS TASK DIFFICULTY LEVELS
Table 14 shows agent performance across different difficulty levels. As expected, agents perform better easier tasks, confirming that tasks designed with increasing difficulty, where lower- level tasks serve as subtasks for higher-level ones. The overall trend in performance across difficulty levels aligns with each agent's general success rate discussed in Section 6.1
G EXPERIMENTS ON OPEN-ENDED SINGLE-APP ENGLISH TASKS
To further explore the scalability of our success detection approaches, we designed an initial set of ten "open-ended" single-app English tasks across distinct apps, as detailed in Table 15
Table 15: Open-enden single-app English tasks.
App
Task Description
Airbnb
I'm traveling to London with three Iriends and need accommedation. I'll munage the checkout process myself.
Amazon
I'd like buy wedding gifts for friend and their pariner. take care of the checkout myself.
Calculator
I want to show my friend the multiplication two negative numbers is indeed a positive number.
Chrome
mm planning trip to
would
save blug
Clock
Please set two alamss one for weekdays and another for weekends. I prefer waking later on weekends.
Merriam- Webster
I'd expand my vocabulary in political idcologies. aim to learn two new terms today. and I'm also feeling hungry.
Google Maps
My car low
Settings
Sometinees have trouble reading the screen cleatly.
Spotify
Create music playlist list for
recommended gente. Just songs will do.
YouTube
I'm interested in watching tech tutorial videos recently.
33

---

Published as a conference paper at ICLR 2025
Table 14: Success rates on single-app English, single-app Chinese, cross-app English and cross-app Chinese tasks, categorised by difficulty level. AutoDroid was tested only on single-app tasks as its agent framework, Droidbot (Li et al. 2017). supports only these tasks.
Single-app Eiglish Ticks
Single-app Chineso Tasks
Cross-app Englissh Tasks
Cross-app Chiness Tisks
Agent
Level 1
Level
Level 3
Level
Level 2
Level 3
Level
Level 2
Level
Level 2
Agentic Workflow (GPT-4o)
AppAgent
0.540
0.340
0.140
0,410)
0.180
0.160
D
AutoDroid
0.560
0.300
0.120
0.360
0.120
0,080
AobileAgent
0.620
0.380
0.160
0.300
0.240
0.180
0.067
0.067
0,200
MobilcAgeat
0.700
0.400
0.200
0,580
0,420
0.320
0.133
0.133
M2A
0.133
T3A
0.720
0.480
0,260
0,480
0.460
0.200
0.133
0.133
SccAct
0.067
Agent-ar-a Modeel
0,020
CogAgent 0.060
0.040
0.040
DignRI.
0.020
0.040
0
OdysseyAgent
0.140
0.020
0.004
0.020
0
As discussed in Section 3.2 when a task description is clearly defined with a specific goal, its executions typically converge the same final state. Such tasks can be treated as "closed-ended" tasks, which form the basis for human-annotater key components. In contrast, for a more vague task description, the task is considered "open-ended". The final state may result in multiple possible outcomes, making it challenging to define key components explicitly. While the coarse detection phase may be limited in such cases, we hypothesised that our fine detection approach, relying on the MLLM evaluator, remains effective and can still be applied to "open-ended" tasks.
In this initial experiment, we tested the seven agents that follow the agentic workflow the ten "open-ended" tasks. Given the open-ended nature of these tasks and the absence of predefined golden steps, agents were allowed maximum of 20 steps to complete each task. We compared the alignment of success detec- tion results between human evaluations and MILMM evaluator. Using the same MLLM evaluator introduced in Section 5.2 we identified 22 true positives, 2 false positives, false negatives, and 44 true negatives. This resulted in an F1 score of 0.917, con- sistent with the corresponding results for "closed-ended" tasks reported in Table 9 These findings demonstrate the potential of applying our MILM evaluator to a broader range of tasks, both "open-ended"" and "close-ended", highlighting its scalability to tasks beyond our benchmark.
Table 16: Success rates on open- ended single-app English tasks.
Success
Agent
Agentic Workflow (GPT-4o)
AppAgent
0,200
AutoDroid
0.300
MobileAgent
0.200
MohileAgentV
0.200
M3A
0,700
T3A
0.400
SeeAct
Table 16 presents the success rates of the seven agents these ten tasks. M3A consistently outper- formed the other agents. However, compared to the success rales reported in Table MobileAgentVV exhibited the largest performance gap. suggesting its limitations in handling "open-ended" tasks.
In future work, we aim to expand this initial experiment with more comprehensive task collection to further improve and assess the fcasibility of our MLLM evaluator for wider range of tasks, and to investigate agent performance on "open-ended" tasks.
34

---

Published as a conference paper at ICLR 2025
H CASE STUDY
Three case studies presented to illustrate representative scenarios of task execution by agents. These include: (1) an invalid action taken by AppAgent due to misinterpretation of the UI structure in the XML file, (2) dynamically changing screen without any action execution, repetitive actions due to the lack of reflection, and unrelated behaviours to the task description in MobileAgent, and (3) the combined actions employed by M3A.
H.1 APPAGENT ON CONTACT 2 TASK
of deenie tec-n "wource Gie vdgee FOREAAAR
10:00
Yng
X
Create new contact
SAVE
disends "rass" endtae feameda- Ialae locass hiss prstlata anpectmu pennarr
"raue beea slggneyssyys toduin de- Gate adgee Laniiio ausage tar
andtet-oaet cenaatt "Mata" Dacas "aur"
Log wwweetaaaataeaaaa Train Geactae-1soo
Mobile
mhe lelte Sodte hut encacce AeCoerna scttaat
enture des deeckitee-er "erua credes "tnat dleabe moder "ire"
Work
jezzens Thlae aclente "Maa"" bunds "pids (87 (67"
cadr--narrs ddger Oodiufeoitt pacage eecag--cc catric croaets
Main
gigmmell
Con-'virat Deckifooertr fom cronds
"Man"" enpertdee "aado"" Pecnoo"ibbe sintteterfiis
Work fex
*laie" nuat "Mla
asos bac get Hoaed-- ioersdeddoreee
Home Fax
choae "Taa"" dustale
tee rlasete Moche" pereer "fatee wiletes/fill P
Pager
usen teleong tod-hhar fel roseree (ePoerttrdootet
Other
entore dape dedkktte-trr coddss-fkbac clobatt-
line diceats-fiae sesaers-11is"* stlette Taae
Custom
8B
Three
There
O
tegett setterett weraate setectes "etl""
q
W
r
t
U
i
O
p
Can- Oechuffueltor pacige
makus cou-d--- "Las! Trzae- "Mlae srrilaee
a
S
d
g
h
k
aua--100 (200 2200)/2..00
+
z
x
v
b
m
Cane' addect Oeckufoattor caceee cainos crrtect"
?123
metred tha" uode raceac "Tara" ero lena "rato"
(a) Annotated screenshot
(b) Parsed XML file
Figure 11: The screenshot and XML file before the last action for AppAgent executing task contact_2 ___ The model generated invalid action tap(2). Task description: "Modify the last name of one of the contacts to "Three'. Update the label for the contact's phone number to Work. Set the company to 'Huawei'. Add an email agent. www.bbennhhhhhhre... @ gmail.com. Label the email as Work".
As shown in Figure 11 in the final step of task contact_2, 22, AppAgent encountered critical error due to a misinterpretation of the UI structure. The model incorrectly parsed the XML, treating the entire pop-up menu as single element instead of recognizing each individual operable component, which reduced the number of widgets the agent could interact with. In addition, the agent executed an invalid action, tap(2), targeting non-elickable element. This issue highlights that imperfect operable action detection mechanism may limit the agent's 's ability to navigate complex UI hierarchies and execute fine-grained interactions.
35

---

Published as a conference paper at ICLR 2025
H.2 MOBILEAGENT ON EXPEDIA 3 TASK
As shown in Figure 12 and Figure 13 MobileAgent's execution of task expedia_3 reveals several noteworthy points: (I) Although the transition between the second and third screenshots (highlighted with a red border) lacks valid actions, the interface still changes, indicating that content is loading during a waiting period (i.c., a dynamically changing screen). (2) The agent generates repetitive actions despite no changes in the interface, but after several iterations, a correction occurs (highlighted with blue border). (3) Interestingly, the beginning of task execution, the agent initially attempted to chat with ChatGPT, which was unrelated to the task description. By the time the agent attempted to execute something relevant, several steps had already been wasted, leaving insufficient opportunities to complete the task properly.
Expardoo
Hi Siaar
you can dream it, Expedia can help plan
you can dream it, Expodia can help plan it.
you can dream it, Expedia can help plan
cen cocldiing longgr coorts
oyr monng mn pager
rudtins
Rome
Heme
losteh by funs
Autken click (backkidketikk vill ag0
diet "oo
Figure 12: Trajectory of MobileAgent on expedia_3 (Part 1). Task description: "Check things to do in Paris. Get the search results for 25th to 28th of any month."
36

---

Published as a conference paper at ICLR 2025
If you can dream it, Expedia help plan it.
If you can dream it, Expedia help plan
you can dream it, Expedia help plan
you can
Expedia can help plan
yetoroyee
Betee agiht
you your optriee mutten
Tyu macoage
Vite finet
uodeon cohire
cafine
clic pet "Ne
Activti didc "Ne" dur anddogge
Activs didy No"
10:0
Expedio
you it, Expedia can help plan
you dream it, Expedia can help plan
Refore mattenn
Your recent searches
Recent searches
Mendodin County Squue
Augh Applt
Explone your destinations
Agp- raaee
Active
Nette diik Loveiiee leff
(eapxiiing better)
dick tnd Elling bode
The roll ataliges.
Thensl andelgee spece
The rod det adtel intelipee
Thenslike veaseee
LMa bortan..
Movenaa Collento debet
ne $597
New
L- $60
diA tou Ang 1-Ag
Calibontis Untudd State.
wariited spas.
Figure 13: Trajectory of MobileAgent expedia 3 (Part 2). Task description: "Check things to do in Paris. Get the search results for 25th to 28th of any month."
37

---

Published as a conference paper at ICLR 2025
H.3 M3A vs HUMAN ON GOOGLE TASKS O TASK
Takks
Toaks
1355S yot
pod de
tooks
A D
G
pol and
X
V
N
M
@
Acdio Deal:: The tes beahees
disk Thered
(a) Trajectory of M3A on google tasks
No sos
No tusls
exree Comple Vereeper
Iara
McB
Gondag
NOME
smess Coegin Moviqpoee
O
W
E
I
U
O'
W
X
V
TUX
(b) Trajectory of Human moogle_tasks___
Figure 14: Trajectory of M3A vs human on google_tasks_ 00, Task description: "Create a new list "Work"."
By comparing Figure 14a Figure 14b is evident that M3A employed a combined action strategy, encapsulating text input and pressing the "enter" key within single-step operation. This approach led to a more concise execution, requiring one fewer step compared to the human trajectory.
38


Reviews:
[{"review_id": "wrqh3idHH2", "reviewer": "Reviewer_K4f8", "summary": "The paper introduces Spa-Bench, a comprehensive benchmark designed for evaluating smartphone agents across a variety of tasks in both English and Chinese languages. This benchmark supports over 340 tasks that span single-app and cross-app scenarios, covering different app types and real-world complexities such as third-party app interactions and varying difficulty levels. SPA-BENCH also integrates a plug-and-play framework that supports 11 agents and features a scalable, automated evaluation pipeline. This pipeline assesses agent performance based on success rates, efficiency, resource consumption, and reliability, eliminating the need for handcrafted evaluation logic. Experiments reveal that while proprietary (M)LLM-based agents outperform open-source counterparts, they still face challenges in real-world application due to time and cost inefficiencies. The paper emphasizes the need for advancements in UI understanding, memory retention, and error handling to improve the practicality of autonomous smartphone agents. I raise an acceptance of this paper.", "questions": "1. Paper can be better organized. For example, there is a very large blank space between line 188 and 191. These large blank spaces need to be fixed.\n\n2. In the paper, authors mention the human-executed trajectory to avoid shortcuts. Why do we need to avoid the shortcuts? I think it is not a bad choice if the agent is able to find a shortcut solving the problem.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Diverse task collection of 340 tasks\n\n2. Automated evaluation pipeline consists of multiple evaluation metrics", "weaknesses": "Paper can be better organized.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p8Gr2wFK9X", "reviewer": "Reviewer_UgAV", "summary": "The present work introduces a new smartphone agent benchmark called “SPA-BENCH.” The motivation for this research stems from the rapid development and increasing adoption of (multimodal) LLM-powered smartphone agents. SPA-BENCH seeks to address the challenges of evaluating these agents by providing a structured evaluation environment that simulates real-world conditions. SPA-BENCH includes a diverse set of tasks that cover both single-app and cross-app tasks on system and third-party applications, catering to both English and Chinese users. The authors also conduct extensive experiments with 7 off-the-shelf LLM agents and 4 finetuned LLM agents, highlighting the key insights derived from these experiments.", "questions": "Please refer to the Weaknesses above.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.This paper studies an important and timely problem. Developing a comprehensive benchmark is important to advance LLM agents in smartphone applications. \n\n2.The present work includes comprehensive experiments with 11 LLM agents on diverse tasks. The findings derived from these experiments could provide insights for future LLM agent design.\n\n3.This paper introduces a plug-and-play framework, which could facilitate real-time interaction with Android devices. It is important to support diverse tasks and minimize the gap to real-world usage.\n\n4.The design of the proposed benchmark is adequately motivated. In general, the paper is well-written.", "weaknesses": "1.The evaluation tasks are constructed by human annotators. It is fine, but I think the present work will benefit a lot if the authors could discuss more about the process of recruiting human annotators and protocols for quality control. It will also be great if the authors can discuss the possibility of synthesizing behavior trajectory, which might extend the impact of present work from solely evaluation to fine-tuning or even pre-training.\n\n2.In addition to off-the-shelf LLM agents and fine-tuned agents, it is also important to evaluate locally deployable agents and cloud-based agents [1], which is particularly relevant for mobile agent setting. I suggest the authors to re-organize the taxonomy of agents as two main categories, i.e., agent-as-a-model (fine-tuned or pre-trained models customized for agentic tasks) [2] and agentic workflow (usually based on off-the-shelf model, possibly with modular design) [3]. \n\n3.Can SPA-bench be applied to a broader range of smart devices? Such as tablets with Android systems or IOS devices. I think it is important to discuss the potential and challenges here.\n\n4.The authors should also elaborate on the main differences of smartphone agents compared to PC agents or cloud-based agents. It provides important insights for context of application.\n\n[1] Gunter, Tom, et al. \"Apple intelligence foundation language models.\" arXiv preprint arXiv:2407.21075 (2024).\n\n[2] Lai, Hanyu, et al. \"AutoWebGLM: Bootstrap And Reinforce A Large Language Model-based Web Navigating Agent.\" arXiv preprint arXiv:2404.03648 (2024).\n\n[3] Shang, Yu, et al. \"AgentSquare: Automatic LLM Agent Search in Modular Design Space.\" arXiv preprint arXiv:2410.06153 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oIjAtckNon", "reviewer": "Reviewer_VmmX", "summary": "This paper presents a benchmark for mobile device control agents, featuring the incorporation of Chinese applications in addition to English apps and newly proposed evaluation metrics regarding task completion. The benchmark includes a set of tasks that involve single and multi-app usage. The authors assess the performance of state-of-the-art agents developed through prompting or fine-tuning (e.g., agent fine-tuned with reinforcement learning). The valuable analyses in this work reveal the limitations of existing algorithms, particularly when handling tasks that span multiple applications or incorporate Chinese-language interfaces. \n\nThe key contributions of these works are benchmark task suites with both English and Chinese apps, additional evaluation metrics, and experiments with in-depth analyses of various agents.", "questions": "Q1 - Could the authors elaborate on the unique challenge introduced by this benchmark? While it is clear that this work provides valuable references and novel attributes (e.g., evaluation metrics), a more explicit articulation of its key contributions would be helpful.\n\nQ2 - Could the authors explain the rationale for the coarse-to-fine evaluation design? While this sounds helpful, I couldn't find a rationale for this design choice. Can the authors provide the reason for choosing this method? Also, what happens if the agent meets the second fine-grained part but misses the first coarse-grained part (or is it ensured that these cases don't usually occur)? A deeper insight into the reasoning and clarification behind these would offer a better understanding of the effectiveness of this approach.\n\nQ3 - (Minor) For some experiments, English and Chinese languages are treated separately (e.g., human annotation in line 318). Could the authors clarify the motivation for this separation?\n\nQ4 - In line 430, the term \"shortcut\" used by the M3A agent is somewhat ambiguous. Does this mean the agent generates a new action outside the predefined action space? If so, how is this interface managed within the environment? If not, clearer phrasing could avoid potential misunderstandings, such as stating, \"The predefined action space includes combinations of actions that humans typically perform independently\". \n\nQ5 - Could the authors confirm if the fine-tuned agents were trained specifically on the dataset proposed in this benchmark? If they were fine-tuned, what factors might explain their low success rates? If they were not, wouldn't it be fair to fine-tune them when comparing the algorithmic performance? Although it might not be necessary to fine-tune them in this benchmark specifically, noting how they are trained (e.g., whether they are trained in this benchmark or not) seems necessary to be noted to avoid misleading the readers.\n\n\nI note that I am highly willing to raise the overall scores.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "S1 - This benchmark offers a notable addition to the community by employing Chinese applications. This expands the scope and applicability of benchmarking in multilingual contexts.\n\nS2 - The benchmark includes diverse tasks, levels, and metrics. It also introduces several new evaluation metrics. Furthermore, a novel coarse-to-fine evaluation approach has been proposed.  \n\nS3 - The authors provided comprehensive benchmark references and included many of the existing baselines. Hence, this study offers a comprehensive comparison, enabling valuable insights. For example, it reveals the necessity of improved memory capabilities and lack of understanding in task termination.", "weaknesses": "W1 - This benchmark’s unique challenge is unclear. The authors should highlight the specific interesting challenge or novel difficulties it poses compared to existing benchmarks.\n\nW2 - While the newly introduced termination metrics are intriguing and valuable, they seem to require more careful design. Could the authors propose a strategy for applying these metrics to agents that do not produce language-based rationales for termination? (Minor) Additionally, it would be helpful if the authors could underscore the significance of these metrics (e.g., why the ability to understand the termination of the tasks would be important when developing agents) while introducing the metric (i.e., Section 5.1). \n\nW3 - The criteria for levels 1, 2, and 3 appear somewhat ambiguous. If the levels are based solely on the number of steps required, the division doesn't sound necessary. Could the authors clarify this distinction if these levels also account for the complexity of cross-app usage?\n\nW4 - Several claims in the paper require further elaboration. For instance, the qualitative complexity of the a11y tree could be clarified by providing additional metrics, such as the number of tokens, as referenced in lines 367-371. Additionally, could the authors further elaborate on what they mean by \"adequately grounding them in real-world context\", as mentioned in lines 386-397?\n\nW5 - It would be helpful to clarify the source of parsing errors. Are these issues arising from the agent's operations, or are they related to the environment? Additionally, there is a minor typographical error with a misplaced comma in line 449.\n\nW6 - The paper lacks details on the specific prompts used for agents, and supplementary materials would facilitate the reproducibility of the experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)