MobileAgentBench: An Efficient and User-Friendly Benchmark for Mobile LLM Agents

## Abstract

Abstract Large language model (LLM)-based mobile agents are increasingly popular due to their capability to interact directly with mobile phone Graphic User Interfaces (GUIs) and their potential to autonomously manage daily tasks. Despite their promising prospects in both academic and industrial sectors, little research has focused on benchmarking the performance of existing mobile agents, due to the inexhaustible states of apps and the vague definition of feasible action sequences. To address this challenge, we propose an efficient and user-friendly benchmark, MobileAgentBench , designed to alleviate the burden of extensive manual testing. We initially define 100 tasks across 10 open-source apps, categorized by multiple levels of difficulty. Subsequently, we evaluate several existing mobile agents, including AppAgent and MobileAgent, to thoroughly and systematically compare their performance. All materials are accessible on our project webpage: https://MobileAgentBench.github.io , contributing to the advancement of both academic and industrial fields.

## 1 Introduction

With the emergence of large language models (LLMs) [], researchers have developed various autonomous agents across fields such as robotics [, ], games [, ], and mobile phones [, ]. Among these, mobile agents have attracted significant attention due to their potential to enhance user experiences and provide intelligent assistance on-the-go.

People have been dreaming of Intelligent Personal Assistants (IPAs) [] that can fully automate daily tasks for decades. Since Apple introduced its digital assistant, Siri [] in 2011, almost all the leading technology companies have launched their own IPAs, including Microsoft Cortana [], Amazon Alexa [], and Google Assistant [].
While these digital assistants provide a hands-free human-computer interaction experience using Natural Language Interface (NLI), they can only fulfill relatively simple tasks, such as setting an alarm clock or sending a text message [].
For third-party apps, developers have to follow and implement the application programming interfaces and protocols, such that when a user issues a very specific command, the system can invoke the corresponding functionality. This limits the usability of those digital assistants.

**Table 1: Comparison between the proposed and existing benchmarks**
| Benchmark | Fully<br>Autonomous | Realistic<br>Environment | Success Condition<br>Flexibility | Low Code<br>Invasiveness |
| --- | --- | --- | --- | --- |
| AppAgent [] | ✗ | ✓ | ✓ | ✓ |
| AITW [] | ✓ | ✗ | ✗ | ✗ |
| AndroidArena [] | ✓ | ✓ | ✗ | ✗ |
| MobileAgentBench (ours) | ✓ | ✓ | ✓ | ✓ |

LLMs contribute significantly to resolving the persistent challenge of understanding user intent. The demonstrated reasoning ability [] highlights the potential of LLM-based autonomous agents as next-generation Intelligent Personal Assistants (IPAs), which are not limited by the programming interfaces since they directly operate on the Graphic User Interface (GUI) [, ]. The GUI can either be represented by a text-based view tree to be consumed by a LLM or a screenshot image that can leverage a Multi-modal LLM (MLLM) []. The action space of the agents composes a series of functions to simulate human operations, such as click, type, swipe, etc. In this way, LLM agents can theoretically achieve whatever human users can do, without any modification of the existing apps.

The promising future of LLM-based smartphone agents attracts more and more researchers to study this topic. However, the scope of benchmarks available for evaluating the performance of these agents remains constrained. Among the existing benchmarks, several prevalent issues are evident:
1. Scalability and usability. Researchers need to fully understand complicated data structures and tools before extending the benchmark with customized tasks or integrating it into their own codebases [].
2. Robustness and Flexibility. Only the annotated task completion path is considered [, ]. However, there might be multiple paths to successfully complete a task, which may break the task success judgment logic.
3. Realistic environment. Some benchmarks evaluate the agent’s performance based on a collection of screenshots but not real devices. It fails if the agent performs an abnormal action and goes to an undefined state [].

To address the issues described, we propose a robust benchmark, MobileAgentBench, designed to evaluate the capabilities of mobile LLM agents within the Android ecosystem. MobileAgentBench offers several advantages over previous benchmarks due to its ease of use and minimally invasive nature. Specifically, for standard agents, the integration process requires fewer than ten lines of additional code. The benchmark excels in usability and versatility, supporting a broad spectrum of testing tasks across various Android operating system versions and executing on actual devices.

In this initial release, we offer 100 built-in benchmarking tasks spanning ten open-source applications. Notably, MobileAgentBench diverges from traditional approaches by simplifying the extension process. Third-party developers can specify the conditions for task success using just a few lines of Python code, without needing extensive knowledge in Android development. This accessibility makes MobileAgentBench more conducive to developers and researchers from non-Android Development communities. Furthermore, we introduce an innovative method for determining the task-terminating state, rendering the benchmark resistant to the complexities of tracking multiple potential success pathways. This approach ensures that MobileAgentBench provides reliable and precise benchmarking outcomes.

The comparison between the proposed and existing benchmarks is listed in Tab. 1, where fully autonomous represents if the benchmark does not need human supervision or judgment, realistic environment represents the tasks can be run on real devices, success condition flexibility represents it takes all possible success paths into consideration, low code invasiveness represents integrating the benchmark into existing agents does not need significant code changes.

Our contributions are summarized as follows:

- •
We propose a benchmark framework for mobile LLM agents. The new approach addresses common issues of existing benchmarks, making the evaluation process fully autonomous and reliable.
- •
We implement and test 100 benchmarking tasks with different levels of difficulty. The benchmark is plug-and-play and easy for both developing new agents and evaluating existing agents.
- •
We evaluate the performance of state-of-the-art mobile LLM agents and perform a solid and systematic comparison with our new benchmark, providing baseline data to the community.

## 2 Related Work

### 2.1 Mobile LLM Agents

Studies before the LLM-era employed reinforcement learning (RL) to solve the autonomous GUI navigation problem []. The recent advancement of LLM and MLLM becomes the dominant agent paradigm becaust of the greater ability of UI understanding and reasoning. Early studies focused on web agents, which achieve task automation within browsers [, ]. Recently, more and more studies started to investigate agents on mobile devices, especially on the Android platform, as Android smartphones are the most widely used personal computing devices.

Mobile LLM agents share a similar algorithm. The full input prompt often consists of four main components: the user prompt (task description), the current UI view hierarchy (VH) description, the action function list, and historical information. Specifically, the action list mainly includes click, swipe, type, and other common UI operations. If MLLM is used, the current screenshot is also a part of the input. The LLM/MLLM is asked to think of the next action based on the current and historical states and call the correct function to perform the given task step by step. The agent finally parses the model response and sends control signals to the Android device using Android Debug Bridge (ADB) (^1^11https://developer.android.com/tools/adb), UIAutomator (^2^22https://developer.android.com/training/testing/other-components/ui-automator), or other higher-level UI automation frameworks.

Despite the similarity of the high-level ideas, researchers have developed different techniques to improve performance and efficiency. Among these works, AndroidArena [] transforms the long and overwhelming view hierarchy XML into a compressed representation and assigned UI elements with unique node IDs, which shortens the prompt and makes the system more efficient. MobileAgent [] observes that GPT-4V lacks the capability of UI element localization, and employs an Optical Character Recognition (OCR) model to locate and localize text views. Moreover, it uses the CLIP [] and Grounding DINO [] models to detect icons. AppAgent [] uses SoM [] prompts to localize UI elements and breaks tasks into two phases, exploration, and deployment. During the exploration phase, the agent automatically interacts with the apps and summarizes the observations into a document. In the deployment phase, it employes the Retrieval Augmented Generation (RAG) [] technique to utilize the summarized knowledge and improve success rate. CogAgent [] proposes its own highly efficient 18B-parameter MLLM, which can be loaded on a single commercial GPU. Furthermore, Octopus v2 [] proposes a compact 3B-parameter model, which unlocks the potential to run mobile LLM agents on-device in an efficient and privacy-preserving manner.

### 2.2 Benchmarks for Mobile LLM Agents

Since the Mobile LLM agent is a newly emerging research field, the choice of benchmarks is very limited. Some studies rely on verifying the task execution status manually to evaluate the performance [], which is tedious and time-consuming. To expedite the agent development, we need a fully autonomous benchmarking system to report various metrics, especially the task success rate. However, automatically judging if a task is completed successfully is non-trivial. The main challenge is caused by the dynamic nature of the GUI navigation task – the agent may perform random actions and drive the app to an unknown state.

AITW [] is a popular benchmark for mobile LLM agents. It has a large scale, but it’s based on static screenshot images. Thinking of each app state (screenshot) as a node, and each action as an edge, we can build a State Transferring Graph (STG) based on the screenshots and the human-annotated actions. It fails immediately if the agent performs actions in a non-considered sequence and leads the STG to a non-existent node, even if the agent can eventually complete the task.

The only solution is to identify task successes on real devices. One approach is to match the agent’s actions with the annotated ground truth (GT). A step-wise matching algorithm is not accurate, because the agent may not finish the task exactly in the same order with GT. AndroidArena [] proposes an adaptive method of calculating the longest common subsequence of the agent and GT action sequences, which is illustrated as follows, where $a$ and $\hat{a}$ are the GT and actual actions, respectively. The common subsequences are marked in red.

$$ $\displaystyle a$ $\displaystyle={\color[rgb]{1,0,0}ABC}$ (1) $\displaystyle\hat{a}$ $\displaystyle={\color[rgb]{1,0,0}A}XY{\color[rgb]{1,0,0}B}U{\color[rgb]{1,0,0} C}VW$ (2) $$

AndroidArena [] treats a task as successful if the GT is a subsequence of the actual action sequence. It addresses the issue of redundant actions but is still not optimal. A simple counter-example can be navigating from the page 1 to page 3 by clicking the next page button two times. If the agent performs the following sequence of actions: clicking the next page button, clicking the previous page button, and clicking the next page button, it doesn’t navigate to the correct page but is still a subsequence of the GT. This method gives false positive results if the redundant action has a side effect.

A concurrent work, LlamaTouch [], addresses this problem by examining the final UI state, which is similar to our approach. We observe that despite the infinite feasible action sequences, the final success state convergence to one. The success or failure can be determined by checking the final UI state. An edge case is that some tasks may not have a direct UI representation, for example, the result of a network request triggered by a button may not be directly reflected on the current UI page. Thus, only checking the UI state is not sufficient and we need to incorporate actions, such as the clicking event, into consideration. LlamaTouch [] matches the click action by mapping the coordinate to a UI element, based on the view bounding boxes. However, it may not always be accurate. The process of finding the correct view to respond to a clicking event is called a hit test, and it’s only accurate if performed by the Android UI system. This is because app developers can modify the touchable area, making it different from the view border to get better user experience.

Figure: Figure 1: Extended touchable area.
Refer to caption: x1.png

Fig. 1 shows an example of enlarging the touchable area of a button view. In Fig. 1, touching point 1, the button does not respond because it’s outside of the touchable area. Touching point 2, the button responds because it’s inside the button view. Touching point 3, although it’s outside of the visible button view, the button still responds because it’s inside the extended touchable area. To overcome this difficulty, we utilize the Android Accessibility Service (^3^33https://developer.android.com/reference/android/accessibilityservice/AccessibilityService) to capture app events faithfully and forward them to the benchmark server. The details of our implementation are described in Sec. 3.1.

## 3 MobileAgentBench

Figure: Figure 2: Overview of the MobileAgentBench architecture.
Refer to caption: x2.png

### 3.1 Method

MobileAgentBench runs on real Android devices, supporting both physical devices and emulators. It sets up environment and then invokes the agent execution function. While the agent is operating the device, MobileAgentBench judges the task success status in real time without any side effects. After the agent stops or exceeds the maximum steps, MobileAgentBench automatically switches to the next task. The whole process is fully automated and requires no human supervision.

The task success judgment mechanism is implemented by matching the final UI state, instead of examining the action sequence. This is because there might be multiple paths towards task completion, but the final success state converges to one. For example, if the task is to go to the settings page, agents may mistakenly open random pages before they correctly find the desired settings page. Matching the action sequence is difficult because of the randomness. On the contrary, checking if the top page is the settings page is easy and reliable. No matter what operations the agent does, we treat it as a success as long as the settings page is detected. ADB and UIAutomator are used to fetch the VH information. For each task, there is a Python file that defines the task success criteria, making it easy to extend and customize tasks for third-party developers.

As some tasks may not have a direct reflection on the current UI page, only checking the VH information is not sufficient. An example can be editing a note and saving the changes. Clicking the save button, the app may only pop up a temporary alert to indicate the save action has succeeded and stays on the current page. When the benchmark checks the current view state, it doesn’t know if the save button is clicked or not. As a benchmark, it cannot go to other pages because it may change the app state and affect t he next action of the agent. Since we want to determine the task success in real-time to collect how many steps the agent takes, it is not feasible to check UI states of other pages after the agent stops. Besides, some agents have the problem of not being able to stop gracefully even the task is completed. We address this issue by incorporating app events, especially button click action signals. For the above-mentioned task, we can use the view hierarchy to check if the note is edited correctly, and then mark the task as a success if the save button clicking signal is received afterwards.

To faithfully receive the app event signals, we make an Android app using the Android Accessibility Service. Android Accessibility Service was originally designed to help people with disabilities. It runs in the background and invokes a callback function when the Android system fires an accessibility event. Such events include most UI state transitions by the user (agent) interactions, such as button clicking, window changing, etc, which fulfills the needs of the proposed benchmark.

The overview of MobileAgentBench is shown in Fig. 2. The benchmarking apps run on real devices from a device farm, which can either be a physical device or an emulator. The device talks to the host machine of the benchmark and the agent via ADB. The benchmark and the agent use ADB to retrieve app state information, such as screenshots, view hierarchy, and send control signals. The benchmark invokes the agent with the current benchmarking task prompt and collects runtime information from the agent, such as the LLM input and output. Whenever the event listener app receives an app event, it forwards the event to the benchmark server via a socket, so the benchmark can assess the task success status using both VH and the actions. At the top of Fig. 2, we show a sample task workflow, “Create a new task Laundry” with the Calendar app. The agent needs to perform 4 actions to complete the task: clicking the add button, clicking the task button, typing the task name “Laundry”, and clicking the save button. The benchmark checks the content of the task name input box view and listens to the save button clicking event to determine if the task is finished successfully.

### 3.2 Task Description

In our initial version of the benchmark, we implement 100 tasks over 10 daily apps. The 10 apps are from SimpleMobileTools (^4^44https://simplemobiletools.com, GPL-3.0), an open-source project of Android apps. These apps have simple and straightforward user interfaces, without any advertisements or unnecessary permissions, and thus are great for benchmarking use cases. The full list of app names is shown in Fig. 3b.

We carefully design the benchmarking tasks, so that they can simulate a normal user’s daily activity and have multi-level difficulties. The distribution of task difficulty levels is shown in Fig. 3. Difficulties are defined by the minimum steps to finish a task, which is cross-verified by 3 human experts independently. A task would be classified as an *easy* task if it can be done within 2 steps, *medium* if greater or equal to 3 while less than 6, and otherwise, *hard*.

Figure: (a) Distribution over all tasks.
Refer to caption: x3.png

### 3.3 Usage

The benchmark APIs are designed to be user-friendly and as less invasive as possible. For a standard agent, it takes less than 10 lines of additional code to integrate. List 1 shows the pseudo-code of the benchmark usage. First, we need to import the benchmark Python library and initialize the benchmark orchestrator. Next, the main agent entrance function should be defined. It takes and only takes one parameter, the task prompt. The agent iteratively performs actions to finish the task. Before and after the agent performs one action, the orchestrator functions are called, so information such as the time spent and LLM output can be collected. The program starts with the orchestrator run function. It calls the agent entrance function with each task prompt and automatically switches to the next task after the task finishes. The task success status is judged on the fly.

Figure: Listing 1: Pseudo code of integrating MobileAgentBench into an existing Mobile LLM Agent.

## 4 Experiments and Agent Evaluations

### 4.1 Metrics

We define 6 metrics to comprehensively benchmarking mobile agents:

- •
Success Rate (SR): ${SR}=N_{success}/M_{tasks}$, where $N_{success}$ is the number of successful tasks, judged by the benchmark system. $M_{tasks}$ is the number of total benchmarking tasks. This metric reflects the agent’s ability to correctly finish a task end-to-end.
- •
Step-wise Efficiency (SE). $SE=S_{actual}/S_{min}$, where $S_{actual}$ is the number of actual steps the agent takes to successfully finish a task, and $S_{min}$ is the minimum number of steps. This metric tells us if the agent performs unnecessary or redundant actions and reflects the efficiency of the agent. Failure tasks are not taken into account.
- •
Latency. The average time spent in seconds before and after one action. This metric tells us how long a user needs to wait between two actions.
- •
Tokens. The number of LLM input and output tokens. For simplicity, we use the GPT-4V (gpt-4-vision-preview) standard [] to calculate the number of tokens for all models, which gives us a rough estimation of the LLM cost. For text, 1 token is 4 characters. For an image, it’s divided into multiple 512$\times$ 512 tiles, and each tile is 170 tokens. 85 base tokens are applied to each image as well.
- •
False Negative (FN) Rate. $FN=N_{early}/M_{failure}$, where $N_{early}$ is the number of early stopped tasks and $M_{failure}$ is the total number of failure tasks. This metric represents how likely the agent falsely thinks it has finished the task and stopped early.
- •
False Positive (FP) Rate. $FP=N_{late}/M_{success}$, where $N_{late}$ is the number of late stopped tasks and $M_{success}$ is the total number of successful tasks. Symmetricly to FN Rate, this metric reveals how likely the agent falsely thinks the task is not finished successfully.

### 4.2 Environment Setup

Five popular mobile LLM agents, AndroidArena [], AutoDroid [], AppAgent [], CogAgent [], and MobileAgent [] are evaluated with the proposed benchmark. We choose the Google Pixel 3a emulator and Android 14 operating system to run the benchmarking apps. Besides, the Android 9 operating system is used for AutoDroid as some of the dependency libraries do not support the newer Android systems.

As there are no local neural networks used in AndroidArena, AutoDroid, and AppAgents, these agents are executed on an Apple Macbook Pro with the M1 Max chip. CogAgent and MobileAgent require local model referencing, so they are executed on a workstation equipped with a single Nvidia RTX 4090 GPU, with 24 GB GPU memory.

The self-exploration feature is turned on for AppAgent. When performing a task, it can reference the previously summarized document. For CogAgent, we use 4-bit quantization to load the model due to GPU memory limitation. CogAgent is implemented in its vanilla flavor, *i.e.*, given the current screenshot, ask for the next action. No history information is provided.

### 4.3 Results

**Table 2: Agent performance results with multiple metrics**
| Agent | SR $\uparrow$ | SE $\downarrow$ | Latency (s) $\downarrow$ | Tokens $\downarrow$ | FN Rate $\downarrow$ | FP Rate $\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| AndroidArena [] | 0.22 | 1.13 | 18.61 | 750.47 | 0.09 | 0.33 |
| AutoDroid [] | 0.27 | 3.10 | 4.85 | 963.48 | 0.93 | 0.01 |
| AppAgent [] | 0.40 | 1.29 | 26.09 | 1505.09 | 0.17 | 0.40 |
| CogAgent [] | 0.08 | 2.42 | 6.76 | 579.84 | 1.0 | 0.04 |
| MobileAgent [] | 0.26 | 1.33 | 15.91 | 1236.88 | 0.19 | 0.31 |

The main experiment results are shown in Tab. 2. We observe that AppAgent has the highest success rate, benefiting from the self-exploration mechanism. CogAgent has the lowest success rate, most likely caused by the naive agent implementation, which limits the usage of history information. Although AutoDroid has a similar success rate to MobileAgent, the step-wise efficiency is significantly lower, possibly caused by the weaker reasoning capability of the GPT-3.5 model used by AutoDroid. Latency-wise, both AutoDroid and CogAgent have very low latency, indicating the high inferencing cost with the GPT-4V model. AppAgent needs to look up the app document, thus consuming more tokens than others. On the other hand, because of the naive agent implementation of CogAgent, it consumes the least number of tokens. AutoDroid and CogAgent have very high FN rates, indicating they always stop early when the task is not finished yet. AppAgent, although having the highest task success rate, is not good at determining the task success task, it cannot stop gracefully after finishing a task and has a high FP rate.

Figure: (a) Task success rate with difficulty levels.
Refer to caption: x5.png

The task success rates for each agent with difficulty levels are shown in Fig. 4a. From Fig. 4a, we can see most agents have higher success rates when handling easier tasks, which is as expected. Interestingly, AppAgent has a higher success rate when performing medium-level tasks. This is because we set the maximum execution steps as twice the minimum steps, which would make the agents have very limited steps to correct their earlier steps for easy tasks. For example, an agent can only use 1 additional step to correct and finish the task for a 1-step easy task. However, for medium and hard tasks, there is significantly more space to correct the previous actions.

Fig. 4b shows the averaged task success rate over the backbone LLM models. It is interesting that AutoDroid, although using a text-based GPT-3.5 model, outperforms some other agents that use the more advanced GPT-4V model. This reveals that the textual view hierarchy contains the most important information for GUI navigation tasks. However, we believe that visual screenshots are helpful for other types of tasks, for example, if the task involves recognizing an image on the screen, or if the textual view hierarchy is not available.

## 5 Limitations and Future Work

While the proposed MobileAgentBench is efficient, user-friendly, and addresses many issues of the existing benchmarks for mobile LLM agents, there are two main limitations that the authors would like to improve in the future. Firstly, although the use of Python code snippets as the configuration of task success conditions is easy for researchers in the AI/ML community, it is still difficult for people without a technical background. We will explore new methods to automatically build the task configuration without writing any code in the future work. Additionally, the usage of UIAutomator makes it infeasible to dump the view hierarchy with dynamic screen contents. UIAutomator dumps the view hierarchy only if the main thread is free. It fails if the app contains persistent animations or videos. Therefore, replacing the UIAutomator with other frameworks to retrieve view information would be a promising direction.

## 6 Conclusion

In this paper, we propose a new benchmark, MobileAgentBench, for mobile LLM agents on the Android platform. With the 100 built-in benchmarking tasks, researchers can test and evaluate existing and new agents automatically on real Android devices. Extending the benchmark to support customized tasks is also easy, as only basic Python coding skills are needed. Leveraging the Android Accessibility Services and only checking the final app state, MobileAgentBench can detect task completion status faithfully. We report the evaluation results of 5 popular agents across multiple metrics, and they can be used as strong baselines to advance future mobile LLM agent development.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: How does MobileAgentBench handle tasks that require interactions with system-level UI elements (e.g., notifications, permission requests)?

What is the specific implementation of the "hit test" used by the benchmark to determine successful button clicks?

How does the framework handle cases where the agent crashes or the app under test becomes unresponsive?

How does the choice of UIAutomator as the backend for AndroidViewClient impact performance and reliability?  Have other backends been considered?
soundness: 2
strengths: The authors convincingly identified important dimensions in current mobile agent benchmarks, particularly regarding scalability, robustness to diverse action sequences, realistic device-based testing, and ease of integration. Here are some strengths.

Automated evaluation: The framework automates the evaluation process, reducing manual effort and increasing reproducibility.
Accessibility and ease of integration: The benchmark is designed to be easily integrated with existing mobile agent frameworks, requiring minimal code changes.
Open-source and reproducible:  The authors commit to making the benchmark open-source, promoting transparency and further development by the community.
Baseline data provided: The evaluation of five existing agents offers valuable baseline data for future research.
summary: This paper introduces MobileAgentBench, a new benchmark for evaluating Large Language Model (LLM)-based mobile agents on the Android platform. The authors argue that existing benchmarks suffer from limitations in scalability, robustness, flexibility, and realism.  MobileAgentBench aims to address these issues by providing 100 built-in tasks across 10 open-source Android apps, facilitating automated evaluation on real devices, and incorporating a flexible task success judgment mechanism based on final UI state and app event signals.  The benchmark also allows for easy customization and integration with existing agents, requiring minimal code modifications.  The authors evaluate five popular mobile LLM agents (AndroidArena, AutoDroid, AppAgent, CogAgent, and MobileAgent) using their benchmark and provide baseline performance data.
weaknesses: I think main motivation for building a new benchmark should ultimately be about "can we evaluate better" or "can we evaluate more complex tasks". While suggested benchmark seem more "user-friendly" than the referenced ones, I'm not quite convinced if MobileAgentBench is moving us forward.

Limited app diversity results limited agent behaviors. The benchmark currently relies on 10 open-source apps from a single developer (SimpleMobileTools).  While understandable for initial development, this limits the diversity of UI elements, interaction patterns, and complexities that agents face. The provided tasks, while covering basic functionalities, might not adequately capture the complexity of real-world mobile interactions.  I expect a new benchmark that encompasses the referenced ones to involve more intricate, multi-step tasks involving data input, navigation across multiple apps, and handling errors are needed.

Limited metric depth:  While the proposed metrics (SR, SE, Latency, Tokens, FFR, OER) are relevant, they could be expanded to capture aspects like agent robustness to unexpected UI changes, error recovery, and efficiency in terms of actions taken.

Limited explanation of the agent event listener app: The functionality and implementation details of the Android Accessibility Service-based event listener app are not thoroughly explained. A more detailed description is crucial for understanding the robustness and reliability of the event capture mechanism.

---

confidence: 3
contribution: 3
initial_rating: 3
presentation: 2
questions: 1. Writing of the paper can be further improved. For example, we do not need such detailed illustrations on related work as within this paper. Save the space for more benchmark analysis is preferable.

2. Your limitations and further work does not sufficiently comprise a single section. Merging it into conclusion is better.

3. Unnecessary blank space should be fixed such as from line 69 to line 74 and line 246 to line 249, etc.
soundness: 3
strengths: 1. It is a rather new benchmark designed for mobile use cases.

2. The benchmark designs multiple tasks ranging from different difficulty levels.

3. It is easy-to-use, and can be integrated within few lines
summary: The paper introduces MobileAgentBench, an efficient and user-friendly benchmarking framework designed for evaluating large language model (LLM)-based mobile agents within the Android environment. Addressing limitations in existing benchmarks, MobileAgentBench allows for the autonomous testing of agents on real devices, with minimal code requirements for integration. The benchmark supports 100 predefined tasks across ten open-source applications, covering various difficulty levels. It evaluates agent performance based on metrics such as success rate, efficiency, and latency, and incorporates Android Accessibility Services to capture real-time app events. This design facilitates customizable and accurate testing, providing a robust platform for developing and evaluating intelligent mobile agents. The benchmark is interesting for mobile use cases. The writing needs to be further improved before acceptance consideration.
weaknesses: 1. Writing can be further improved

2. Large blank space should be fixed

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: Please refer to the Weaknesses above.
soundness: 3
strengths: 1.The proposed benchmark addresses an important gap in evaluating mobile LLM agents, which is essential for the advancement of this field.

2.The authors implement 100 benchmarking tasks categorized by different levels of difficulty. It could provide a more nuanced evaluation of LLM agents.

3.The paper provides a systematic comparison of multiple agents, establishing a foundation for understanding their performance capabilities and limitations.
summary: This paper introduces MobileAgentBench, a benchmark specifically designed to evaluate mobile LLM agents. The present work is motivated by the growing popularity of LLM agents in mobile setting, and the difficulty to develop a common platform to evaluate these diverse agents. The authors propose to alleviate the burden of manual setting by allowing LLM agents directly interact with GUIs. They 100 tasks across 10 open-source applications, categorizing them by difficulty to facilitate thorough evaluation. The experiment results highlight the strengths and weaknesses of existing agents, including AppAgent and MobileAgent.
weaknesses: 1.The scale and scope of designed tasks are generally smaller than other existing benchmarks. For example, SPA-bench [1] contains 340 tasks with both single APP and cross APP settings. Comprehensive comparison and discussion are needed to justify the unique advantage of the present work. 

2.The paper lacks a detailed explanation of the underlying protocol of implementing MobileAgentBench, which could hinder reproducibility and applicability.

3.The evaluation metric of computation cost is limited. The authors only evaluate the token cost of using cloud-based LLMs, which do not consider the constrained computation resource in mobile devices. 

4.The experiments do not include agents run on locally deployed LLMs like Phi-3 [2]. It misses an important setting for mobile agent, which has unique advantage for privacy preservation, low latency, etc. 

[1] Chen, Jingxuan, et al. "SPA-Bench: A Comprehensive Benchmark for SmartPhone Agent Evaluation." NeurIPS 2024 Workshop on Open-World Agents. 2024.

[2] Abdin, Marah, et al. "Phi-3 technical report: A highly capable language model locally on your phone." arXiv preprint arXiv:2404.14219 (2024).

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. As the suceess rates of agents on MobileAgentBench are low, a detailed and thorough error analysis is needed to illustrate why they perform not well when doing the tasks and what is the bottleneck of improving on the benchmark. I recommend presenting some failure examples to better illustrate it.
2. How do you design and formulate the action space of MobileAgentBench? Could you explain more baout it?
3. In the part when you discuss about the digital assistant and human-computer interaction, I think one missing citation would be OSWorld[1], which is a benchmark of real-world computer tasks in a unified and interactive computer environment for multimodal agents. The task simulation and evaluation pipeline are quite related to your work.

[1] Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. CoRR, abs/2404.07972, 2024. doi: 10.48550/ARXIV.2404.07972. URL https://doi.org/10.48550/arXiv.2404.07972.
soundness: 3
strengths: 1. Compared to the existing mobile LLM agent benchmarks, MobileAgentBench can be run on real Android devices to test mobile digital agents, making the evaluation process dynamic and realistic.
2. The benchmark can be extended and integrated with more task instances easily, with only several lines of Python codes needed. It is possible that the benchmark will attract more people in the community to contribute to scale and enrich it.
3. The evaluation process only checks the final state in the app system to detect if it is successfully completed, while allows the variance of trajectories/steps. It is effective and efficient way of evaluate digital tasks.
summary: The paper presents MobileAgentBench, a benchmark for mobile LLM agents within the Android system, with a fully autonomous and reliable evaluation process. MobileAgentBench features itself for it can be run on real devices and needs no significant code changes to integrate agents into the framework.

The authors evaluate the performance of current SOTA mobile LLM agents on the new benchmark and find the success rates are relatively low, leaving space for further exploration.
weaknesses: 1. MobileAgentBench consists of 100 tasks totally, spanning across 10 simple use mobile apps with simple and straightforward user interfaces, which may damage the diversity and of the benchmark. The apps might need more careful selection and filtering, and the tasks could be more realistic as it would run into various situations in the wild environment.
2. The analysis part within the article is found to be inadequate and lacking in robustness. To enhance its credibility and depth, a more thorough comparative evaluation of agent performance is required. Furthermore, incorporating an examination of task types that span multiple applications could provide valuable insights.