Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration
Authors: Junyang Wang
Haiyang Xu
Haitao Jia
Xi Zhang
Ming Yan
Weizhou Shen
Ji Zhang
Fei Huang
Jitao Sang
Venue: NeurIPS 2024 poster (2024)
Keywords: multi-agent
multi-modal agent
multi-modal large language model
mobile operation
UI assistant
Primary area: natural_language_processing
TL;DR: In this paper, we propose a mobile device operation assistant based on a multi-agent architecture, which is capable of effectively navigating through historical information and self-reflection.

Abstract: Mobile device operation tasks are increasingly becoming a popular multi-modal AI application scenario. Current Multi-modal Large Language Models (MLLMs), constrained by their training data, lack the capability to function effectively as operation assistants. Instead, MLLM-based agents, which enhance capabilities through tool invocation, are gradually being applied to this scenario. However, the two major navigation challenges in mobile device operation tasks — task progress navigation and focus content navigation — are difficult to effectively solve under the single-agent architecture of existing work. This is due to the overly long token sequences and the interleaved text-image data format, which limit performance. To address these navigation challenges effectively, we propose Mobile-Agent-v2, a multi-agent architecture for mobile device operation assistance. The architecture comprises three agents: planning agent, decision agent, and reflection agent. The planning agent condenses lengthy, interleaved image-text history operations and screens summaries into a pure-text task progress, which is then passed on to the decision agent. This reduction in context length makes it easier for decision agent to navigate the task progress. To retain focus content, we design a memory unit that updates with task progress by decision agent. Additionally, to correct erroneous operations, the reflection agent observes the outcomes of each operation and handles any mistake accordingly. Experimental results indicate that Mobile-Agent-v2 achieves over a 30% improvement in task completion compared to the single-agent architecture of Mobile-Agent. The code is open-sourced at https://github.com/X-PLUG/MobileAgent.

Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration
Junyang Wang
Haiyang Xu
Haïtao Jia
Xi Zhang"
Ming Yan
Weizhou Shen Ji Zhang"
Fei Huang?
Jitao Sang"!
[junyangwang, 23120356, jtsang) eeub.tu.edu.com cn
(shuofeng.xxr xhy, zx443053, yml 19608, shenweizhou.sve zj122146, f.huang) @alibaba-ine.com
Beijjnn Jiaotong University
Allbaba Group
Abstract
Mobile device operation tasks are increasingly becoming a popular multi-modes AI application scenario. Current Multi-modal Large Language Models (MLLMs), constrained by their training data, lack the capability to function effectively as operation assistants. Instead, MILM-based agents, which enhance capabilities through tool invocation, are gradually being applied to this scenario. However, the two major navigation challenges in mobile device operation tasks task progress navigation and focus content navigation - are difficult to effectively solve under the single-agent architecture of existing work. This is due the overly long token sequences and the interleaved text-image imate data format, which limit performance. To address these navigation challenges effectively, we propose Mobile-Agent-v2 v2, multi-agent architecture for mobile device operation assistance. The architecture comprises three agents: planning agent, decision agent, and reflection agent. The planning agent condenses lengthy, interleaved image-text history operations and screens summaries into pure-text task progress, which is then passed on to the decision agent. This reduction in context length makes it easier for decision agent to navigate the task progress. To retain focus content, we design memory unit that updates with task progress by decision agent. Additionally, correct erroneous operations, the reflection agent observes the outcomes of each operation and handles any mistake accordingly. Experimental results indicate that Mobile-Agent-ve achieves over a 30% improvement in task completion compared the single-agent architecture of Mobile-Agent The code is open-sourced at httpss: //tt:////tth wttpaa..coo/-PP/////A//AAA Ccm//--PLLC//tooieeggen
Introduction
Multi-modal Large Language Models (MLLMs), represented by GPT-4v OpenAl (2023). have
demonstrated outstanding capabilities in various domains Bai et al. (2023): Liu et al. (203c 6); Dai
al. (203)) Zhhu et al. (2023); Chen al. (2023); Ye (2023a b): Wang al. (2023c Hu et al.
(2023 2024); Zhang et al. (2024b). With the rapid development of agents based on Large Language
Models (LLMs)) /Zha al. (2024) Liu al. (202f)) Talebirad and Nadiri (2023): Zhang al. (203bb); Wu et al. (2023); Shen et al. (2024); Li et al. (2023a), MLLM-based agents, which can overcome the limitations of MLLMs in specific application scenarios by various visual perception tools, have become focal point of research attention Liu et al. (023d)
Work done during internship at Alibaba Group. Cooresponning author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

History Screens & Operations
Focus Content Navigation
141
132
Google
roller
Sule
Task Progress Navigation
Open Chrome app
Search the game results
Back to home page
Batee
241)
pone Laker'
Iotar's LEOAT
Sad parlay
Tap grttu
Open Notes app
Create type
Instruction: Search tie recull for 'ocay's Lakers game, and then create a note Notes to te a news for this resut
Image-text Interleave Input [12K Token]
Planning Decision Reflection
Trae the
Accuracy
System
Iteration #
lteratioo 7
86.5
Instruction:
Screen: Image taken)
Screen:
[image tokenn)
60.7
Operation
Operation: Open app (Chrome)
Operation:
Space:
Tap the icon (yellow circle with plus)
Single-ag Architecture
Mobile Operation Assistant
Single-agent Multi-agent
Figure I: Mobile device operation tasks require navigating focus content and task progress from history operation sequences, where the focus content comes from previous screens. As the number of operations increases, the length of the input sequences grows, making extremely challenging for a single-agent architecture to manage these two types of navigation effectively.
Automated operations on mobile devices, as a practical multi-modal application scenario, are emerg- ing as a major technological revolution in AI smartphone development Yao et al. (2022); Deng et al.
(2023) : Gur et al. (2024): /Zhng et al. (2024); Zhang et al. (2023a); Wang al. (2024): Chen and Li (2024a b c Zhang et al. (2024a); Wen et al. (2023); Cheng et al. (2024). However, due to the limited screen recognition, operation, and location capabilities. existing MLLMs face challenges in this scenario. To address this, existing work leverages MLLM-based agent architecture to endow MLLMs with various capabilities for perceiving and operating mobile device UI. AppAgent Zhang et al. 2023a) tackles the limitation of MLLMs in localization by extracting clickable positions from device XML files. However, the reliance on UI files limits the applicability of this method to other platforms and devices. To eliminate the dependency on underlying UI files, Mobile-Agent Wang et al. (2024) proposes a solution for localization through visual perception tools. perceives the screen through an MILM and generates operations, locating their positions by visual perception tools.
Mobile device operation tasks involve multi-step sequential processing. The operator needs to perform a series of continuous operations on the device starting from the initial screen until the instructions are fully executed. There are two main challenges in this process. First, plan the operation intent, the operator needs to navigate the current task progress from the history operations. Second, some operations may require task-relevant information in the history screens, for example, writing sports news in Figure requires using the match results queried earlier. We refer to this important information as the focus content. The focus content also needs to be navigated out from the history screens. However, as the task progresses, the lengthy history of interleaved image and text history operations and screens as input can significantly reduce the effectiveness of navigation in a single-agent architecture, as shown in Figure
In this paper, we propose Mobile-Agent-v2, mobile device operation assistant with effective navi- gation via multi-agent collaboration. Mobile-Agent has three specialized agent roles: planning agent, decision agent, and reflection agent. The planning agent needs to generate a task progress based on the history operations. To save the focus content from the history screens, we design a memory unit to record task-related focus content. This unit will be observed by the decision agent when generating an operation, simultaneously checking if there is any focus content on the screen and updating it to the memory. Since the decision agent cannot observe the previous screen to reflect, we design the reflection agent to observe the changes in the screen before and after the decision agent's 's operation and determine whether the operation meets the expectations. If it finds that the operation does not meet expectations, it will take appropriate measures to re-cxecute the operation. The entire

---

process is illustrated in Figure 3 The three agent roles work respectively in the progress, decision, and reflection stages, collaborating alleviate the difficulty of navigating.
Our summarized contributions are as follows:
We propose multi-agent architecture Mobile-Agent-v2 to alleviate various navigating difficulties inherent in the single-agent framework for mobile device operation tasks. We design a planning agent to generate task progress based the history operations, ensuring effective operation generation by the decision agent.
To avoid the loss of focus content navigating and reflection capability, we design both a memory unit and reflection agent. The memory unit is updated by the decision agent with focus content. The reflection agent assesses whether the decision agent's 's operation meets expectations and generates appropriate remedial measures if expectations are not met.
We conducted dynamic evaluations of Mobile-Agent-V across various operating systems, language environments, and applications. Experimental results demonstrate that Mobile- Agent-v2 achieves significant performance improvements. Furthermore, we empirically validated that the performance of Mobile-Agent-v2 can be further enhanced by manual operation knowledge injection.
2 Related Work
2.1 Muiti-agent Application
The powerful comprehension and reasoning capabilities of Large Language Models (LLMs) enable LLM-based agents to demonstrate the ability to independently execute tasks Brown et al. 20020;;
Achiam et al. (2023); Touvron et al. (2023a by; Bai all (2023). Inspired by human-team collabora- tion, the multi-agent framework has been proposed. Park et al. (2023) constructs Smallville consisting of 25 agents in a sandbox environment. Li et al. (2023b) proposes a role- setee-llynnnbbbneeee based multi-agent collaborative framework to enable two agents playing different roles to autonomously collaborate.
Chen et al.) (024) innovatively propose an effective multi-agent framework for coordinating the col- laboration of multiple expert agents. HHng al. 2024) presents a groundbreaking meta- reta-programming
multi-agent collaboration framework. Wu et al. (2024) proposes generic multi-agent framework that allows users configure the number of agents, interaction modes, and toolsets. CCan et al. (2024);
Subramaniam et al. (2024); Tao et al. (2024) investigate the implementation of a multi-agent debating framework, aiming evaluate the quality of different texts generated content. Abdelnabi al. (022)); Xu et al. (2024); Mukobi et al. (2024) integrate multi-agent interaction with game theoretic strategies, aiming to enhance both the cooperative and decision abilities.
2.2 LLM-based UI Operation Agent
Webpages, as a classic application scenario for UI agents, have attracted widespread attention to research on web agents. Yao et al. (2022) and Deng al. (2023) aim to enhance the performance of agents on real-world webpage tasks by constructing high-quality website task datasets. Gur et al. (2024) utilizes pre-trained LLMs and self-experience learning to automate task processing on real-world websites. Zheng et al. (2024) leverages GPT-4V for visual understanding and webpage manipulation. Simultaneously, research on LLM--bsed UI agents for mobile platforms has also drawn significant attention. /Wen et al. (2023) converts Graphical User Interface (GUT) information into HTML representations and then leverages LLM in conjunction with application-specffic domain knowledge. (aan et al. (2023) proposes a multi-modal intelligent mobile agent based on GPT-4V, exploring the direct utilization of GPT-4V to perceive screen screenshots with annotations. Unlike the former approach that operates on screens with digital labels, Zhang et al. (2023a) combines the application's 's XML files for localization operations, mimicking human spatial autonomy operating mobile applications. Wang et al. 2024) eliminates the dependency on the application' 's XML files and leverages visual module tools for localization operations. Additionally, Hong et al. (2023) designed a GUI agent based on pre-trained vision-
models. [Chen and Li (2024a blc) propose small-scale client-side models for deployment on actual devices. /Zhang et al. (2024a) proposed a UI multi-agent framework tailored for the Windows operating system. Despite the significant performance improvements achieved by multi-agent architectures in many tasks, currently, there no work that employs multi-agent architectures in mobile device operation tasks. To address the
3

---

VPM
S
TP
Planning Agent
Decision Agent
Spt
Reflection Agent
R₂
Ins, Out
Ins, Ry
FC
FC.
FC,1
Ins, O
FC,
Memory Unit
Figure 2: Illustration of the overall framework of Mobile-Agent-v2.
challenges of long-contex. navigation in mobile device operation tasks, in this paper, introduce the multi-agent architecture Agbite-Agent-v..
3 Mobile-Agent-v2
In this section, we will provide detailed overview of the architecture of Mobile-Agent-v2 The operation of Mobile-Agent--22 is iterative, and its process is depicted in Figure Mobilc-Agent-v2 has three specialized agent roles: planning agent, decision agent, and reflection agent. We also design the visual perception module and memory unit to enhance the agent's screen recognition capability and the capability to navigate focus content from history. Firstly, the planning agent updates the task progress, allowing the decision agent to navigate the progress of the current task. The decision agent then operates based on the current task progress, current screen state, and the reflection (if the last operation is erroneous). Subsequently, the reflection agent observes the screens before and after the operation to determine the operation meets expectations.
3.1 Visual Perception Module
Screen recognition remains challenging even for ofate-of-the-art MILMs when processed end-to- end. Therefore, we have incorporated visual perception module enhance the screen recognition capability. In this module, we utilize three tools: text recognition tool, icon recognition tool, and icon description. Inputting a screenshot into this module will ultimately yield the text and icon information present the screen, along with their respective coordinates. This process is represented by the following formula:
P. VPM(S.)
(1)
where P. represents the perception result of the screen in the 1-th iteration.
3.2 Memory Unit
Due to the task progress generated by the planning agent being in textual form, the navigation of focus content from history screens is still challenging. To address this issue, we design memory unit to store the focus content related to the current task from history screens. The memory unit serves as short-term memory module that is updated as the task progresses. The memory unit is crucial for scenarios involving multiple apps. For instance, as shown in Figure weather information observed by the decision agent will be utilized in subsequent operations. At this point, the information related to the weather app's page will be updated in the memory unit.
3.3 Planning Agent
We aim to reduce the reliance on lengthy history operations during decision-making by employing separate agent. We observe that although each round of operation occurs on different pages and different, often the goals of multiple operations are the same. For example, in the example illustrated

---

Planning Stage
Decision Stage
Reflection Stage
Operation
open app seen from the provious stop
Screenshot
The wealher apo currentty apon. need recard the weather information the page: Then return the deoktop next
Last Screenshof
can be seen from the screen before and alter the operation that the device return the desklop. Therefore operation
Operation
11:34
Heal Uhis look has
Back Retry
boen completed.
Or
Completed Tasks
Planning Agent
Completed
Decision Agent
Next Screenshot
11:95
Reflection Agent
Instfective Oparation
strrt
skart
Back Torm
Open
Update
have checked the today's s
Thoughht
Reflection
Retry Or
Useer iistrctiion
User Instruction
Check txday's weather
Memory
Queration
then go to hotes
Last Oparsiors
Memory
Operation
Home
Actioo:
Mensory Licc The woother 6o0
Updarte
Alemooy:
Mamory
Oparation
Reteection 2
Correct
Open app (Vensthre)
Update Memory
Figure 3: Illustration of the operation process and interaction of agent roles in Mobile-Agent-v2.
in Figure ! the first four operations are all about scarching for match results. Therefore, we design a planning agent summarize the history operations and track task progress.
We define the operation generated by the decision agent at the t-th iteration O.. Before the decision agent makes decision, the planning agent observes the decision agent's 's operation O,-1 from the last iteration and updates the task progress TP- to T... The task progress includes the sub-tasks that have already been completed. After generating the task progress, the planning agent passes it to the decision agent. This aids the decision agent considering the content of tasks that have not yet been completed, thereby facilitating the generation of the next operation. As shown in Figure 3 the planning agent's inputs consist of four parts: user instruction Ins, the focus content FC, in memory unit, the previous operation Oz-1, and the previous task progress TP--1 Based above information, the planning agent generates the TP. This process is represented by the following formula:
TP: PA(Ins, O,-1, FC,-1)
where the PA represents the LLM of the planning agent.
3.4 Decision Agent
The decision agent operates during the decision stage, generating operations O and implementing them on the device, while also being responsible for updating the focus content FC in memory unit. This process is illustrated in the Decision Stage shown in Figure and represented by the following formula:
O, DA(Tns, TP--1, FC--1, Re R,,
where the DA represents the MLLM of the decision agent and the R& represents the reflection result from reflection agent.
To reduce the complexity of operations, design operation space and restricted the decision agent to selecting operations only from within this space. For operations with higher degrees of freedom, such tapping and swiping, incorporate additional parameter space to locate or handle specific content. Below is a detailed description of the operation space:
Operation Space.
Open app (app name). the current page is the home page, this operation can be used to open the app named "app name".
Tap (x. y). This operation is used to tap on the position with coordinates (x, y).
Swipe (x/, y/), (x2, y2). This operation is used to swipe from the position with coordinates (xl, to the position with coordinates (x2, y2).
Type (tex)). If the current keyboard is in an active state, this operation can be used input the content of text" in the input box.
5

---

Home. This operation is used to return to the home page from any page.
Stop. If the decision agent thinks that all requirements have been fulfilled, it can use this
operation to terminate the entire operation process.
Memory Unit Update. As each operation made by the decision agent is highly task-relevant and based the visual perception results of the current page, it is well-suited to observe task-related focus content within the screen pages. Accordingly, we have endowed the decision agent with the ability update the memory unit. When making decisions, the decision agent is prompted observe whether there task-related focus content within the current screen page. If such information is observed, the decision agent updates it in memory for reference in subsequent decisions. This process is represented by the following formula:
FC, FCIIns,FC-11:
3.5 Reflection Agent
Even with the visual perception module, Mobile-Agent-v2 may still generate unexpected operations. In some specific scenarios, MLLMs may produce severe hallucinations Liu et al. (2023a) ; Li
et al. (202ch;; Gunjal et al. [2024) Wang et al. (2023b); Zhou et al. (2023). even the most advanced
MLLM GPT-4V Cui et al. (203)) Wang et al. (2023a). Therefore, we design the reflection agent
to observe the screen state before and after decision agent's operation to determine whether the
current operation meets expectations. This process represented by the following formula:
R₂ RA(Ins,,FF On, St. Pt, St41, Pc+1)
(5)
where the RA represents the MILM of the reflection agent.
As shown in Figure 3 the reflection agent generates three types of reflection results after operation execution: erroneous operation, ineffective operation, and correct operation. The following will describe these three reflection results:
Erroneous operation refers to operation that leads the device to enter page unrelated to the task. For example, the agent intends to chat with contact A in a messaging app, but accidentally opens the chat page of contact B instead.
Ineffective operation refers to an operation that does not result in any changes to the current page. For example, the agent intends to tap on an icon, but it taps on the blank space next to the icon instead.
Correct operation refers to an operation that meets the decision agent's expectations and serves as a step towards fulfilling the requirements of the user instruction.
If erroneous operation, the page will revert to the state before the operation. If ineffective operation, the page will remain in its current state. Neither erroneous nor ineffective operations are recorded in the operation history to prevent the agent from following these operations. If correct operation, the operation will be updated in the operation history, and the page will be updated the current state.
4 Experiments
4.1
Model
Visual Perception Module. For the text recognition tool, we use the document OCR recognition model WwwNNextViTTdooument from ModelScopd"] For the icon recognition tool, we employ GroundingDINO Liu al. (2023e). a detection model capable of detecting objects based on natural language prompts. For the icon description tool, we utilize the Qwen-VL-
MLLMMs. For the planning agent, it does require screen perception, we utilize the text-only GPT-4 OpenAl (023). For the decision agent and reflection agent, we employ GPT-4V OpenAl 2023). All calls made through the official API method provided by the developers.
summary

---

4.2 Evaluation
Evaluation Method. To evaluate the performance of Mubile-Agent-Ve on real mobile devices, employed dynamic evaluation method. This evaluation method requires the operation tool to implement the agent's operations in real time on an actual device. We use two mobile operation systems, Harmony OS and Android OS, to assess capabilities in non-English and English scenarios, respectively. We used Android Debug Bridge (ADB) as the tool for operating mobile devices. ADB can simulate all operations of Mobile-Agent-22 in the operation space. In both scenarios, select system apps and 5 popular external apps for evaluation. For each app, we devise two basic instructions and two advanced instructions. Basic instructions are relatively simple operations with clear instructions within the app interface, while advanced instructions require a certain level of experience with app operations to complete. Additionally, to evaluate multi-app operation capability, we design two basic instructions and two advanced instructions involving multiple apps. In total, there were 88 instructions for non-Enge non-English and English scenarios, comprising 40 instructions for system apps, 40 instructions for external apps, and 8 instructions for multi-app operations [5] The apps and instructions used for evaluation in non-English and English scenarios are presented the appendix.
Metries. We design the following four metries for dynamic evaluation:
Success Rate (SR): When all the requirements of user instruction are fulfilled, the agent is considered to have successfully executed this instruction. The success rate refers to the proportion of user instructions that successfully executed.
Completion Rate (CR): Although some challenging instructions may not be successfully executed, the correct operations performed by the agent still noteworthy. The completion rate refers to the proportion of correct steps out of the ground truth operations.
Decision Accuracy (DA): This metric reflects the accuracy of the decision by the decision agent. It the proportion of correct decisions out of all decisions.
Reflection Accuracy (RA): This metric reflects the accuracy of reflection by the reflection agent. It the proportion of correct reflections out all reflections.
Implementation Details. We use Mobile-Agent as the baseline. Mobile-Agent is a single-agent architecture based on GPT-4V end-to-end screen recognition. We fix the seed for GPT-4V invocation and set the temperature introduce the scenario of knowledge injection. This involves providing the agent with some operation hints in addition to the user instructions to aid the agent. It's worth noting that we only injected knowledge for instructions that Aobile-Ageet--e couldn't accomplish. For instructions that could be completed without additional assistance, we keep the input unchanged.
to avoid randomness. In addition to Mobile-Agent-v2, we further
4.3 Results
4.3.1 Evaluation
Evaluation on Task Completion. Table and respectively illustrate the performance of Mobile- Agent-v2 in non-English and English scenarios. Compared to Mobile-Agent, Mobile-Agent-v2 exhibits significant improvements in both basic and advanced instructions. With the multi-agent architecture, even in highly challenging advanced instructions, the success rate can still reach 55% compared to only 20% with Mobile-Agem Even in the English scenario, Mobile-Agent-V still achieves a significant performance improvement. Although Mobile-Agenl performs better in English scenarios compared to Chinese scenarios, Mobile-Agent-v2 still achieves average improvement of 27% in success rate.
Evaluation on Reflection Capability. In the case of knowledge injection, even though the decision accuracy does not reach 100%, the completion rate can still reach 100%. This indicates that even with knowledge injection, Mobile-Agent-v2 still makes erroneous decisions. Errors in decision-making are difficult to avoid, even for humans. Therefore, the importance of the reflection agent is highlighted.
'e did not use Mobile-Ewer because the difficulty of this benchmark is relatively low and Mobile-Agent-ve achieve an accuracy of 99%. In this work, we have redesigned more challenging evaluation tasks.
7

---

Basic Instruction
Advanced Instruction
Method
SR
CR
DA
RA
SR
CR
DA
RA
System app
Mobile-Agent
5/10
41.2
37.6
3/10
37.3
32.9
Mobile-Agent-v2
9/10
86.8
82.5
93.3
6/10
82.7
78.2
84.4
Mobile-Agent-v2 + Know.
10/10
100
98.2
98.9
8/10
88.9
87.2
91.4
External app
Mobile-Agent
2/10
38.3
35.4
1/10
29.2
27.0
Mobile-Agent-v2
8/10
97.9
94.0
92.5
5/10
77.9
74.1
78.8
Mobile-Agent-v2 Know.
10/10
100
95.6
97.3
8/10
87.8
83.0
85.9
Multi-app
Mobile-Agent
1/2
52.8
50.0
0/2
33.3
31.4
Mobile-Agent-v2
2/2
100
92.9
91.6
2/2
100
93.8
92.9
Mobile-Agemt-v2+ Know.
Table 1: Dynamic evaluation results on non-English scenario, where the Know. represents manually
injected operation knowledge.
Basic Instruction
Advanced Instruction
Method
SR
CR
DA
RA
SR
CR
DA
RA
System app
Mobile-Agent
9/10
92.5
89.7
4/10
62.0
71.3
Mobile-Agent-v2
9/10
95.0
92.9
96.5
6/10
76.0
77.6
88.4
Mobile-Agent-v2 Know.
10/10
100
96.2
98.7
8/10
85.3
87.9
92.0
External app
Mobile-Agent
7/10
79.7
72.0
3/10
45.3
38.7
Mobile-Agent-v2
9/10
97.1
93.8
96.2
7/10
89.7
91.0
93.4
Mobile-Agent-v2 + Know.
10/10
100
98.2
97.4
9/10
97.1
94.2
98.5
Multi app
Mobile-Agent
2/2
100
91.2
1/2
86.7
92.9
Mobile-Agent-v2 Mobile-Agent-v2 Know.
2/2
100
97.4
100
1/2
93.3
93.3
80.0
2/2
100
100
100
Table 2: Dynamic evaluation results on English scenario, where the Know: represents manually
injected operation knowledge.
Evaluation on App Type. From all metrics, it can be observed that the performance of all methods on system apps exceeds that of external apps. From the results of multiple apps, it can be seen that Mobile-Agent-v2 achieves improvements of 37.5% and 44.2% in SR and CR, respectively, compared Mobile-Agent Compared to single-app app tasks, multi-app tasks rely more on the retrieval of history operations and focus content. The significant performance improvement indicates that the multi-agent architecture and memory unit of Mobile-Agent-v2 play an important role.
Evaluation on Operation Knowledge Injection. From the results knowledge injection in Table T and 2 it can be observed that operation knowledge can effectively enhance the performance of Agente-Ag which suggests that manually injected operation knowledge can mitigate the limitations of an agent's 's operation capability. This finding implies that knowledge injection can broaden the application scenarios of Mobile-Agent by manually written operation tutorials to instruct agents. This finding may offer new insights for automated script testing on mobile devices and suggests that to enhance the operation capabilities of MILMS their limits, automating the generation high-quality operation knowledge can further improve the performance of Mobile-Agent-v2 Moreover, the success brought about by knowledge injection also opens up new avenues future mobile app testing. Existing mobile app testing solutions still limited to manual script writing, which restricts the universality of testing and raises
because even complex tasks can be guided

---

Basic
Advanced
Model
SR&DA
SR&DA
GPT-4V w/o agent
2.7
0.9
Gemini-1.5-Pro
38.2
29.8
Qwen-VIee VL.-Max
42.1
33.6
GPT-4Y
92.7
83.5
Table 3: Performance results of Mobile-Agent-v2 with different MLLMs. To better illustrate the differences, we converted all instructions to single-step forms and evaluated the success rate (which the same as decision accuracy) of each single-step task.
Ablation Sctting
Basic
Advanced
Planning Agent
Reflection Agent
Memory Unit
SR
CR
DA
SR
CR
DA
59.1
63.7
58.9
29.5
43.8
42.6
77.3
83.6
84.0
45.5
72.3
69.8
86.4
89.2
85.7
54.5
75.9
72.4
88.6
93.9
89.4
61.4
82.1
80.3
Table 4: The results of the ablation study planning agent, reflection agent, and memory unit.
the threshold for users. To address the aforementioned issues, one can inject natural language testing procedures into Aobile-Agent-v.. After injecting accurate testing procedures, the system can operate normally regardless of changes in the size or color of the mobile interface. Additionally, language descriptions eliminate the need for knowledge base in script writing.
Evaluation on MILMs. In Table 3 we evaluate the performance of different MILMs within the Mobile-Agent- framework. Since some models are not well-suited for handling sequential inputs, we selected specific instructions and modified each step to function as a single-step task. Therefore, we only evaluate the SR (which is the same DA). We also evaluate the direct use of GPT-4V, bypassing the agent architecture for end-to-end operation. The results indicate that using GPT-4V directly as a mobile device operation assistant almost infeasible. GPT-4V combined with the agent architecture remains the most effective configuration for operational capabilities.
4.3.2 Ablation Study
We conduct ablation studies on Mobile- Agentt-- including the planning agent, reflection agent, and memory unit. From the results in Table it is evident that the impact of the planning agent the overall framework is the most significant. This further demonstrates the challenging nature of navigation in long sequences for current MLLMs. Additionally, performance declines are observed after removing the reflection agent and memory unit. The reflection agent is essential for correcting erroneous operations. It enables the decision agent to avoid operating incorrect pages getting stuck in loops of invalid operations. The memory unit crucial for successful execution in multi-ap aalti-app scenarios. Even in scenarios involving multiple sub-tasks, the memory can sometimes record the positions of critical UI elements, aiding better localization for the next sub-task execution.
4.3.3 Analysis of Operation Sequence Length
As shown in Figure we analyze the positions of erroneous or ineffective operations within failed instructions in English scenarios, dividing the relative sequence positions into three equal parts. The results indicate that in Mobile-Agent, such enrors or ineffective operations predominantly occur in the later stages of the tasks. In contrast, Mobile-Agent-v2 does not exhibit any obvious pattern. This indicates that the multi-agent architecture is better equipped handle the challenges posed by long sequences in UI operation tasks.

---

Mobile Agent
Mobile Agent-v2
10
Eronnou or inffecttive Openstoon
6-10
1/3-2/
20-1
0-10
10-20
-0-11
Raratrve Tound
Pudative Roand
Figure 4: The relative positions of erroneous or ineffective operations in the operation sequence.
TOP
Figure 5: The cases of the complete operation process and reflection of Mobile-Agent-v2.
4.3.4 Case Study
Figure (a) illustrates a complete operational process of Mobite-Agent--2. Under the planning of the planning agent, the decision agent can navigate the task progress correctly in the case of single-image input. Meanwhile, the memory unit accurately stores the chat content needed for the task and, when required for search purposes, the decision agent can effectively navigate to it. Figure (b) illustrates an example of correcting ineffective operations through reflection. After the failure of the previous operation, the reflection agent promptly detects the error and communicates the reflection result to the decision agent. Based on this, the decision agent rethinks and implements the correct operation.
Conclusion
Existing single-agent architectures for mobile device operation assistants experience significant reduction in navigation effectiveness when dealing with long sequences of interleaved text and images, thereby limiting their performance. To address this issue, in this paper, propose Mobile-Agent-v2 a mobile device operation assistant with efficient navigation via multi-agent collaboration. We address the aforementioned navigating challenges through the planning agent and memory unit, respectively. Additionally, we design reflectors to ensure the smooth progress of tasks. Experimental results demonstrate that Mobile-Agent-v2 achieves significant performance improvements compared the single-agent Mobile Agent. Furthermore, we find that performance can be further enhanced through the injection of manual operation knowledge, providing new directions for future work.
10

---

Acknowledgements
This work is supported by the National Key R&D Program of China (No. 2023YFC3310700.
References
Abdelnabi, S., Gomaa, A., Sivaprasad, S., Schönherr, L., and Fritz, M. (2024). LLM-deliberation: Evaluating LLMs with interactive multi-agent negotiation game.
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et (2023). Gpt-4 technical report. arXiv preprint arXiv:2303.0877
Bai, J., Bai, S., Yang. S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., and Zhou, J. (2023). Qwen-vl: frontier large vision-language model with versatile abilities. arXiv preprini urXiv:2308. 12966.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020). Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
Chan, C.-M., Chen, W., Su, Y., Yu, J., Xue, W., Zhang, S., Fu, J., and Liu, Z. (2024). Chateval: Towards better LLM-based evaluators through multi-agent debate. In The Twelfth International Conference on Learning Representations.
Chen, J.. Li, D. Z. X. S. X., Zhang, L. Xiong, R. V. C. Y., and Elhoseiny, M. (2023). Minigpt-v2: Large language model as unified interface for vision- languae multi-task learning. arXiv preprint arXiv:2310.0944
Chen, W. and Li, 7. (2024a). Octopus v2: On-device language model for super agent. arXiv preprint arXiv:2404.01744
Chen, W. and Li, Z. (2024b). Octopus v3: Technical report for on-device sub-billion multimodal ai agent. arXiv preprint arXiv:2404.1145)
Chen, W. and Li, Z. (2024c). Octopus v4: Graph of language models. 2404v:2404. 19296.
arXiv preprint
Chen, W., Su, Y., Zuo, J., Yang. C.. Yuan, C., Chan, C.-M, Yu, H., Lu, Y., Hung, Y.-.,, Qian, C., Qin. Y., Cong, X., Xie, R., Liu, Z., Sun, M., and Zhou, J. (2024). Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors. In The Twelfth International Conference on Learning Representations.
Cheng, K., Sun, Q., Chu, Xu, F., Li, Y., Zhang, J., and Wu, Z. (2024). Seeclick: Harnessing gui grounding for advanced visual gui agents. arXiv preprint arXiv:2401.
Cui, C., Zhou, Y., Yang, X., Wu, S., Zhang, L., Zou, J., and Yao, H. (2023). Holistic analysis of hallucination in gpt-4v (ision): Bias and interference challenges. arXiv preprini arXiv:2311.03227.
Dai, W., Li, J., Li, D., Tiong, A. M. H., Zhao, J., Wang, W., Li, B., Fung, P., and Hoi, S. (2023). Instructblip: Towards general-purpose vision vision-language models with instruction tuning. arXiv preprint arXiv:2305.066 065000
Deng, X., Gu, Y., Zheng, B., Chen, S., Stevens, S., Wang, B., Sun, H., and Su, Y. (2023). Mind2web: Towards a generalist agent for the web. CoRR, abs/2306.06070.
Gunjal, A., Yin, and Bas, E. (2024). Detecting and preventing hallucinations in large vision language models. In Proceedings the AAAI Conference on Artificial Intelligence, volume 38, pages 33135-18144
Gur, I., Furuta, H., Huang, A. V., Safdari, M., Matsuo, Y., Eck, D., and Faust, A. (2024). A real-world webagent with planning, long context understanding, and program synthesis. In The Twelfth International Conference Learning Representations.
11

---

Hong, S., Zhuge, M., Chen, J., Zheng, X., Cheng, Y., Wang, J. Zhang, C., Wang, Z., Yau, S. K. S., Lin, Z., Zhou, L., Ran, C., Xiao, L., Wu, C., and Schmidhuber, (2024). MetaGPT: Meta programming for a multi-agent collaborative framework. In The Twelfth International Conference on Learning Representations.
Hong, W., Wang. W., Lv, Q. Xu, J., Yu, W. Ji. J., Wang, Y., Wang, Z., Zhang, Y., Li, J., Xu, B., Dong, Y., Ding, M., and Tang, (2023). Cogagent: A visual language model for gui agents.
Hu, A., Shi, Y., Xu, H., Ye, J., Ye, Q. Yan, M., Li, C., Qian, Q. Zhang, J., and Huang, F. (2023). ppllug-pperrow:: Scientific diagram analysis with the multimodal large language model. arXiv preprint arXiv:2311.1824
Hu, A., Xu, H., Ye, J., Yan, M., Zhang, L., Zhang, B., Li, C.,, Zhang, J., Jin, Q.. Huang, F., et al. (2024). mplug-docowl 1.5: Unified structure learning for ocr-free document understanding. arXiv preprint arXiv:2403. 128.9
Li, C., Chen, H., Yan, M., Shen, W. Xu, H., Wu, Z., Zhang, Z., Zhou, W., Chen, Y., Cheng, C., et al. (2023a). Modelscope-agent Building your customizable agent system with open-source large language models. arXiv preprint arXiv:2309.00986
Li, G., Hammoud, H. A. A. K., Itani, H., Khizbullin, D., and Ghanem, B. (2023b). CAMEL: Communicative agents for "mind" exploration of large language model society. In Thirty-seventh Conference on Neural Information Processing Systems.
Li, Y., Du, Y., Zhou, K., Wang, J., Zhao, W. X., and Wen, J.-R. (2023c). Evaluating object hallucination in large rision-langual models. arXiv preprint 23Xx::2305.1035
Liu, F., Lin, K., Li, L., Wang, J., Yacoob, Y., and Wang, L. (2023a). Aligning large multi-modal model with robust instruction tuning. arXiv preprint arXiv:2306. 14565.
Liu, H., Li, C.,, Li, Y., and Lee, J. (2023b). Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744.
Liu, H., Li, C., Wu, Q. and Lee, Y. J. (2023c). Visual instruction tuning. arXiv preprint arXiv:2304.000 230:23304.8885. 08485.
Liu, S., Cheng. H., Liu, H., Zhang, H., Li, F., Ren, T., Zou, X., Yang, J., Su, H., Zhu, J., et al. (2023d). Llava-plus: Learning to use tools for creating multimodal agents. arXiv preprint arXiv:2311.05457
Liu, S., Zeng, Z., Ren, T., Li, F., Zhang, H., Yang. J., Li, C., Yang. J.. Su, H., Zhu, J., et al. (2023e). Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint 233v:23033.05499
Liu, Z., Zhang, Y., Li, P., Liu, Y., and Yang, D. (2023f). Dynamic Ilm-agent network: An Ilm Ilm-agent collaboration framework with agent team optimization. arXiv preprint arXi::2310002170.
Mukobi, G., Erlebach, H., Lauffer, N., Hammond, L., Chan, A., and Clifton, J. (2024). Welfare diplomacy: Benchmarking language model cooperation.
OpenAl (2023). Gpt-4 technical report. ArXiv, abs/2303.08774.
Park, J. S., O' O'Brien. Cai, C. J., Morris, M. R., Liang, P., and Bernstein, M. (2023). Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, UIST 223 ACM.
Shen, W., Li, C., Chen, H., Yan, M., Quan, X., Chen, H., Zhang, J., and Huang, (2024). Small llms are weak tool learners: A multi-llm agent. arXiv preprint arXiv:2401.07324.
Subramaniam, V., Torralba, A., and Li, S. (2024). DebateGPT: Fine-tuning large language models with multi-agent debate supervision.
Talebirad, and Nadiri, A. (2023). Multi-agent collaboration: Harnessing the power of intelligent Il agents. arXiv preprint arXih:2306.03314.
12

---

Tao, M., Zhao, D., and Feng, Y. (2024). Stain-of-discussion A multi- framework for complex evidence-based question answering.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F, et al. (2023a). Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13977
Touvron, H., Martin, L.., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023b). Llama 2: Open foundation and fine-tuned chat models. arXiv preprint urXiv:2377.09280
Wang. J., Wang, Y., Xu, G., Zhang, J., Gu, Y., Jia, H., Yan, M., Zhang, J., and Sang. J. (2023a).
An lim-free multi-dimensional benchmark for milms hallucination evaluation. arXiv preprint arXiv:2311.07397.
Wang. J., Xu, H., Ye, J., Yan, M., Shen, W., Zhang, J., Huang, F., and Sang, J. (2024). Mobile- agent: Autonomous multi-modal mobile device agent with visual perception. arXiv preprint arXiv:2401.1615.
Wang. J., Zhou, Y., Xu, G., Shi, P., Zhao, C.,, Xu, H., Ye, Q., Yan, M., Zhang, J., Zhu, J., al. (2023b). Evaluation and analysis of hallucination in large vision-language models. arXiv preprint arXiv:2308. 15/26.
Wang. W., Lv, Q., Yu, W. Hong, W., Qi. J., Wang, Y.. Ji. J., Yang, Z., Zhao, L.. Song, X., al. (2023c). Cogvlm: Visual expert for pretrained language models. arXiv preprint arXiv:231.03079)
Wen, H., Li, Y., Liu, G., Zhao, S., Yu, T., Li, T. J.-.,, Jiang, S., Liu, Y., Zhang, Y., and Liu, Y. (2023). Autodroid: Lim-powered task automation android.
Wu, Q., Bansal, G., Zhang, J., Wu, Y., Li, B., Zhu, E., Jiang, L., Zhang, X., Zhang, S., Liu, J., Awadallah, A. H., White, R. W., Burger, D., and Wang, C. (2024). Autogen: Enabling next-gen LLM applications via multi-agent conversation.
Wu, Q. Bansal, G., Zhang, J., Wu, Y., Zhang, S., Zhu, E., Li, B., Jiang, L., Zhang, X., and Wang, C. (2023). Autogen: Enabling next-gen Ilm applications via multi-agent conversation framework. arXiv preprint arXiv:2308.081:5
Xu, Z., Yu, C., Fang, F., Wang, Y., and Wu, Y. (2024). Language agents with reinforcement learning for strategic play the werewolf game.
Yan, A., Yang, Z., Zhu, W., Lin, K., Li, L., Wang, J., Yang, J., Zhong, Y., McAuley, J., Gao. J., Liu, Z., and Wang, L. (2023). Gpt-4v in wonderland: Large multimodal models for zero-shot smartphone gui navigation.
Yao, S., Chen, H., Yang, J., and Narasimhan, K. R. (2022). Webshop: Towards scalable real-world web interaction with grounded language agents. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K., editors, Advances in Neural Information Processing Systems.
Ye, Q. Xu, H., Xu, G., Ye, J., Yan, M., Zhou, Y., Wang, J., Hu, A., Shi, P., Shi, Y., et al. (2023a). mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint ar0iv:2304. 14178.
Ye, Q. Xu, H., Ye, J., Yan, M.. Liu, H., Qian, Q., Zhang, J., Huang, F,, and Zhou, J. (2023b). owll: Revolutionizing multi-modal large language model with modality collaboration. arXiv preprint arX1v.2311.04257.
Zhang, C., Li, L., He, S., Zhang, X., Qiao, B., Qin, S., Ma, M., Kang, Y., Lin, Q., Rajmohan, S., Zhang, D., and Zhang, Q. (2024a). Ufo: A ui-focused agent for windows os interaction.
Zhang, C.. Yang, Z., Liu, J., Han, Y., Chen, X., Huang, Z., Fu, B., and Yu, G. (2023a). Appagent: Multimodal agents as smartphone users.
Zhang, J., Xu, X., and Deng, S. (2023b). Exploring collaboration mechanisms for Ul agents: social psychology view. arXiv preprint arXiv:2310.02112..
13

---

Zhang, L., Hu, A., Xu, H., Yan, M.. Xu, Y., Jin, Q. Zhang, J., and Huang, F. (2024b). Tinychart: Efficient chart understanding with visual token merging and program-of thetuuhss learning. arXiv preprint arXiv:2404. 16635.
Zhao, A., Huang, D., Xu, Q. Lin, M., Liu, Y.J., and Huang, G. (2024). Expel: Llm agents are experiential learners. Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19632-19642
Zheng, B., Gou, B., Kil, J., Sun, H., and Su, Y. (2024). Gpt-4v(ision) is generalist web agent, if grounded. CoRR, abs/2401.01614.
Zhou, Y., Cui, C., Yoon, J., Zhang, L., Deng, Z., Finn, C., Bansal, M., and Yao, H. (2023), An- alyzing and mitigating object hallucination in large vision-language models. arXiv preprint arXiv:2310.00754.
Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. (2023). Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.1053 10592.
14

---

A Appendix supplemental material
A.1 Evaluation Application and Instruction
Table and 6 present the applications and instructions used for dynamic evaluation in both English and -nngiingshh scenarios. Basic instructions are relatively simple operations with clear instructions within the app interface, while advanced instructions require certain level of experience with app operations to complete.
A.2 Agent Prompt
In Table 7 and TO we present the system and user prompts for the planning agent, decision agent, and reflection agent. Notably, since the history of operations is empty the beginning of the task, we display the initial planning and subsequent planning prompts for the planning agent separately Table 7 and g
For image inputs, we differentiated based on the task characteristics of each agent. For the planning agent, navigating historical operations and generating task plans is a purely textual process, screenshots are required. For the decision agent, we input the screenshot of the current device state. For the reflection agent, we additionally retain the screenshot from the previous operation and input alongside the current device state screenshot in chronological order.
Since some instructions need to be specially tailored for specific mobile applications, we have hidden these details to protect privacy. the Table and "xxx" represents the redacted information. For certain celebrities or locations mentioned in the instructions, we retained them as they do not involve privacy concerns.
User Instruction
Reply
messages WhatsApp
Figure 6: A case of replying to messages in chat platform WhatsApp based on the content of unread messages.
15

---

# User Instruction
Search for Elon Musk on X app and follow him
Eorple
Mux dutc
WD Hoboiic 1134 WD
Figure 7: A case of searching for celebrity social media platform X and following him.
User Instruction
Search for "Musk" on Tok and open the relevant videos, then comment on the relevant content
RETE SUP PEPINES COL DOAN
Figure 8: A case of searching for celebrity's 's video on the short video platform Tik and commenting on relevant content.
16

---

# User Instruction
E 13 + H2 3
1
eame PURRS Bons
OH 0 E
Ore O
Figure 9: A case of searching for posts with specific content on the Little Red Book.
# User Instruction
1 Wi es 43
7
11:42 14:42
MONRED
Figure 10: A case of commenting on post on the social media platform Weibo.
17

---

# User Instruction
1
11:22
AMM
BROIC
SPREOO
Figure 11: A case of searching for celebrity and following them on the long-video platform Bilibili.
18

---

App
Basic Instruction
Advanced Instruction
System app
Setting
1. Turn dark mode.
1. Switch the system theme.
2. Set system sound to "Do Disturb".
2. Tum the real-time network speed display in the notification bar.
Cinema
1. Open the and lake photo.
1. Open the camera and lake photo with lelepholo lens, and then view the photo.
2. Open the camera and record a video.
2. Open the camera and record a video with telephoto lens, and then view video.
Contact
1. Call the user Phone with the phone numher '1".".
1. Call the xxx from the Contacts app.
2. Reply to the unread texi message according to the content text message.
2. Send a greeting message to XXX.
Notes
1. Create new note and write something.
1. Delete the existing note in the note, then create new note record the current phone power and network speed and save it.
2. Create a new note to record the current phone power and network speed.
2. Create a new note, write something and return to the previous page, then create another to record the cur- reni phone power and network speed.
System app
1. Install "I" in the App Market.
then create note to record the current phone power and network speed and save
1. Delete existing
2. Create new to record current phone power and network speed.
2. Create new note, write something and return to the previous page, then create another note to record the cur- rent phone power and aetwork speed.
External app
X
1. Like post on the homepage of app.
1. Comment post on the homepage of X app with content.
2. Search for Flon Musk on X app and follow him.
2. Search for Flon Musk on X app and comment on post of him.
TikTok
1. Swipe up on TikTok to see cat-related video and like
1. Swipe up on TikTok to see cat-related video and comment on the relevant content.
it.
2. Swipe up TikTok cat-related video and share
2. Scarch for "Musk" Tik and open the relevant
it with other users.
videos, then comment on the relevant content.
YouTube
1. Like video "Shorts" on YouTube.
1. Like Sharts video You Yube and then relevant content.
2. Search for "Stephen Curry" on You Tube and subscribe him.
2. Search for a video about "LeBron James", then like and comment on the relevant content.
Google
1. Navigate to gus station on Maps.
1. Write an email and send to "abc@cbb.ccmm
2. Install Facebook on Play store.
2. Search for singer Taylor Swift on Chrome and open her introduction page.
WhatsApp
1. Send a hello message to "xxx" on WhatsApp.
1. Reply to unread messages on WhatsApp.
2. Find the contact xxx and open the chat interface.
2. Reply to all the unread messages on WhatsApp.
Multi-app
1. View the contacts and create a new note in Notes to record these contacts.
1. Check your unread messages in WhatsApp and search YouTube for videos related to that message.
2. View content of the in Notes, then scarch for videos about it on Tik.ok.
2. Search the resalt for today's Thunder game, and then create note in Notes to write a sport news for this result.
Table 5: The applications and instructions used for the evaluation on English scenario, where the "xxx" represents the redacted information.
19

---

App
Basic Instruction
Advanced Instruction
System app
1. TT
2. WHRRE
1. IT> M
L.TT
2. 1T:F1100. 1
2. #1 Kill
1. A
1. 21
2. #1 201
@r 1ligg
1. ME
: & SRR MUSITALL ~
10 F109 RIRTE BE OF
2. Wi
AA DE
1
1. T (L1
2. A. MA FEE"
2 AE 19.
Externol app
1. ERl 22 19777
2. www.:
2.
1. E 1
2. 12
2 -
#
(AINE
1. 2
1. 71 AE
2. E
2
1. #:: H1
# -1
2. ## -1
#0
19
2.
CALB
Multi-app
1. サ
2.
M
Table 6: The applications and instructions used for the evaluation non Engllish scenario, where the "xxx" represents the redacted information..
20

---

System
You are a helpful mobile phone operating assistant.
User
### Background ###
There user's instruction which is: (User's instruction). You mobile phone operating assistant and are operating the user's mobile phone.
### Hint ###
There are hints to help you complete the user's instructions. The hints are as follow: you want to tap an icon of app, use action "Open app"
#Hif Current operation ###
To complete the requirements of user's 's instruction, you have performed an operation. Your operation thought and action of this operation are as follows:
Operation thought: [Last operation thought)
Operation action: [Last operation)
#if Response requirements #ff
Now you need to combine all of the above to generate the "Completed contents". Completed contents is a general summary of the current contents that have been completed. You need to first focus on the requirements of user's instruction, and then summarize the contents that have been completed.
#### Output format ###
Your oulput format
### Completed contents ###
Generated Completed contents. Don't output the purpose of any operation. Just summarize the contents that have been actually completed in the ### Current operation ##lf..
(Please use English to output)
Table 7: The prompt for the planning agent during the first operation.
System
You are a helpful Al mobile phone operating assistant.
User
#illi Background #N
There an user's instruction which (User's instruction). are mobile phone operating assistant and are operating the user's mobile phone.
#### Hint ###
There are hints to help you complete the user's instructions. The hints are as follow: you want to tap an icon of an app, use action "Open app"
### History operations ###
To complete the requirements of user's instruction, you have performed a scries of operations. These
operations are follow:
Step-l: (Operation thought: (operation thought 1): Operation action: (operation 1}] Step-2: [Operation thought: (operation thought 2); Operation action: (operation 2)
#MH Progress thinking ####
After completing the history operations, you have the following thoughts about the progress of user's
instruction completion:
Completed contents:
[Last "Completed contents")
##HH Response requirements ###
Now you need to update the "Completed contents". Completed contents a general summary of current contents that have been completed based on the ### History operations ###..
#### Output format ### Your output format is: ##ff Completed contents #
Updated Completed contents. Don't output the purpose of any operation. summarize the contents that have been actually completed the ### History operations ###
Table 8: The prompt for the planning agent during subsequent operations.
21

---

System
You are a helpful Al mobile phone operating assistant. You need to help me operate the phone to complete the user's instruction.
User
#iHl Background fifl
This image is phone screenshot. Its width is (Lateral resolution) pixels and its height is ( (Vertical resolution)
pixels. The user's instruction is: [User's instruction].
### Screenshot information ####
In order to help you better perceive the content in this screenshot, we extract some information on the current screenshot through system files. This information consists of two parts: coordinates; content. The format of the coordinates [x. yl. is the pixel from left to right and y is the pixel from top bottom; the content is text an icon description respectively. The information as follow: (x1, yl); text or icon: text content icon description
##lf Keyboard status ###
We extract the keyboard status of the current scrcenshot and it is whether the keyboard of the current screenshot is activated.
The keyboard status as follow:
The keyboard has not been activated and you can't can'tun type. or The keyboard has been activated and you can type.
#ilfi Hint fiil#
There are hints help you complete the user's instructions. The hints are as follow: you want to tap an icon of app, use the action "Open app"
### History operations ###
Before reaching this page, some operations have been completed. You need to refer to the completed operations decide the next operation. These operations as follow: Step-1: (Operation thought: (operation thought J:: Operation action: {operation 1/]
#iifi Progress ###ff
After completing the history operations, you have the following thoughts about the progress of user's instruction completion:
Completed contents:
(Task progress from planning agent)
##ff Response requirements ####
Now you need to combine all of the above to perform just one action on the current page. You must choose
one of the six actions below:
Open app (app name): If the current page is desktop. you can use this action to open app named "app name" on the desktop.
Tap (x, y): Tap the position (x, y) in current page.
Swipe (x1. yl), (x2. y2): Swipe from position (x1, 1) to position (x2, y2).
Unable to Type. You cannot use the action "Type" because the keyboard has not been activated. If you want type, please first activate the keyboard by tapping on the input box the screen. or Type (lext): Type the
"text" in the input box.
Home: Return to home page.
Stop: If you think all the requirements of user's instruction have been completed and no further operation is required, you can choose this action to terminate the operation process.
### Output format ####
Your output consists of the following three parts: ##H Thought ### Think about the requirements that have been completed in previous operations and the requirements that need to be completed in the next one operation.
###f Action ###
You can only choose one from the actions above. Make sure that the coordinates or text in "("". ### Operation ###
Please generate a brief' natural language description for the operation in Action based on your Thought.
Table 9: The prompt for the decision agent.
22

---

System
You helpful Al mobile phone operating assistant.
User
These images two phone screenshots before and after operation. Their widths are (Lateral resolution) ) pixels and their heights are Vertical resolution) pixels.
order to help you better perceive content this screenshot. we extract some information on current screenshot through system files. The information consists of two parts, consisting format: coordinates; content. The format of the coordinates is (x, y), x is the pixel from left to right and is the pixel from top to bottom; the content is text icon description respectively The keyboard status is whether the keyboard of the current page is activated.
### Before the current operation ### Screenshot information:
yl); text or icon: text content or icon description
Keyboard status: The keyboard has not been activated. The keyboard has been activated.
## After the current operation #tili
Screenshot information:
(x1, y)); text icon: text content icon description
Keyboard status:
The kcyboard has not been activated. or The keyhoard has been activated.
### Current operation ###
The user's instruction is: (User's instruction). You also need the following requirements: If you want to tap an icon an app. use the action "Open app". process completing the requirements of instruction, an operation performed on the phone. Below are the details this operation:
Operation thought: [Last operation thought)
Operation action: aLast operation)
#### Response requirements ###
Now you need output the following content based the screenshots before and after the current operation: Whether the result of the "Operation action" meets your expectation of "Operation thought"?
A: The result of "Operation action" meets my expectation of "Operation thought".
B: The "Operation action" results in wrong page and need to return to the previous page.
C: The "Operation action" produces no changes.
### Output format ###
Your output format is:
### Thought ###
Your thought about the question
### Answer ###
Table 10: The prompt for the reflection agent.
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction clearly state the paper's contributions and scope.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitations in the Appendix.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The focus of our work is to explore the application research of the multi-modal large models based agents in special fields. Thus theoretical results are involved.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have provided necessary information for reproduction in Section 4.2
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We the model in modelscope and we provide the code address of this model.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have provided the training and test details necessary to understand the results Section
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have provided the compute workers details in Appendix.
9. Code Of Ethics
24

---

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://pppp.//..o........o////aa
Answer: [Yes]
Justification: The research conforms with the NeurlPS Code of Ethics in every respect.
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The paper prioritizes discussing technical methodologies and results rather than societal impacts.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper exploring mobile device operation assistant poses no such risks.
Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have properly credited the used assets. All the datasets introduced in the paper are available for free researchers for non-commereial use.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Our code and relevant assets including usage details are provided in the submission.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing research within human subjects. 15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does involve crowdsoureing nor research with human subjects.
25


Reviews:
[{"review_id": "nWtWQtbmRk", "reviewer": "Reviewer_c2St", "summary": "The paper titled \"Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration\" presents a multi-agent architecture designed to address the challenges of mobile device operation tasks, specifically focusing on task progress navigation and focus content navigation. The proposed system includes three agents: a planning agent, a decision agent, and a reflection agent. The architecture aims to improve task completion rates and operational efficiency compared to existing single-agent architectures.", "questions": "- How does the proposed system handle the increased computational overhead associated with running multiple agents concurrently on resource-constrained mobile devices?\n- Can the system be extended to support more complex multi-app operations that require intricate coordination between different agents?\n- How does the reflection agent determine the appropriate corrective measures for erroneous operations, and what is the success rate of these corrections in practice?", "limitations": "see weakness", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Innovative Multi-Agent Architecture:** The introduction of a multi-agent system to handle different aspects of mobile device operation is a novel approach. It effectively distributes the workload among specialized agents, which likely contributes to improved performance.\n\n**Comprehensive Experimental Evaluation:** The paper includes a detailed evaluation across different operating systems, language environments, and applications, providing robust evidence of the system's effectiveness.\n\n**Significant Performance Improvements:** Experimental results show that Mobile-Agent-v2 achieves over a 30% improvement in task completion compared to a single-agent architecture, demonstrating the practical benefits of the proposed system.", "weaknesses": "**Limited Discussion on Scalability:** There is insufficient discussion on the scalability of the proposed system. The paper does not address potential challenges when scaling the architecture to more complex tasks or larger datasets.\n\n**Potential Overhead of Multi-Agent Coordination:** While the multi-agent approach shows improved performance, the paper does not discuss the potential computational overhead and complexity introduced by coordinating multiple agents, which could be a significant drawback in resource-constrained environments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## **Question 1: Overhead of multi-agent.**\n## **Response:**\n* For time overhead, although multiple agents work serialize, there are still phases to parallel. The tables below compare the operation times for the Mobile-Agent single-agent framework and the Mobile-Agent-v2 multi-agent framework. \"Screenshot\" represents obtaining a screenshot and corresponding image processing from the device, while \"Tool Call\" represents the use of visual perception tools. It is evident that the increased operation time of the multi-agent framework is acceptable under parallel design.\n\n* For memory overhead, Mobile-Agent-v2 calls the large model via API method, so there is no additional memory overhead.\n\n**Mobile-Agent (Single-Agent Framework)**\n|Phase|Preparation|Planning|Decision|Operation|Reflection|Total|\n|-|-|-|-|-|-|-|\n|Task|Screenshot|-|Decision|Tool Call, Grounding|-||\n|Time|2s||20s|12s|-|**34s**||\n\n**Mobile-Agent-v2 (Multi-Agent Framework)**\n|Phase|Preparation|Planning|Decision|Operation|Reflection|Total|\n|-|-|-|-|-|-|-|\n|Task|Parallel with Planning|Screenshot, Tool Call, Planning, Reflection|Decision, Memory|-| Parallel with Planning||\n|Time|-|18s|21s| -| -|**39s**||\n\n## **Question 2: Can Mobile-Agent-v2 be extended to more complex multi-app scenarios?**\n## **Response:**\n* Yes, Mobile-Agent-v2 can be extended to more complex multi-app scenarios. Mobile-Agent-v2 is a purely visual general framework that is not restricted by app type or operating system, allowing it to be freely extended to any scenario. Mobile-Agent-v2 inherently possesses the capability to plan and make decisions in complex scenarios. Benefiting from the planning agent not being affected by long sequence and image-text interleaved inputs, the decision agent can ensure decision accuracy. Additionally, the memory unit can store key information from previously opened apps' screenshot for use in subsequent operations.\n\n* For extremely complex multi-app scenarios, Mobile-Agent-v2 also supports custom extensions to the operation space. By simply adding the functional description of the operation to the operation space, Mobile-Agent-v2 can perform the operation when needed. For example, if the user needs to use the Mobile-Agent-v2 to crawl the screenshot of the mobile app in batches, the user can add the \"screenshot\" operation, and request the Mobile-Agent-v2 in the instruction to use the operation to intercept the screen after completing the specified operation task. This extension can effectively improve accuracy without affecting operation efficiency.\n\n## **Question 3: How does the reflection agent determine if an operation is correct and the success rate of error correction?**\n## **Response:**\n* The Multi-modal Large Language Model (MLLM) itself has multi-image understanding capabilities, which supports multi-image input reflection. While the decision agent outputs an operation, it also outputs the operation intent. The internal contextual reasoning ability of the MLLM can be used to determine whether the operation result meets expectations. Additionally, the MLLM has some operation knowledge, so even if the decision agent's operation intent is inaccurate, it can still make a judgment based on the user's task.\n\n* We have compiled statistics on the success rate and average steps required for the reflection agent's error correction in the table below, where \"Reflection SR,\" \"Correction SR,\" and \"Average Steps\" respectively represent the success rate of reflection, the success rate of error correction, and the average steps for error correction. The results show that over **60%** of operational errors can be successfully corrected, with the cost being less than **2** steps.\n|Reflection SR|Correction SR|Average Steps|\n|-|-|-|\n|94%|62%|1.63|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "udJahkD2JE", "reviewer": "Reviewer_kbfL", "summary": "In this paper, the authors present Moble-Agent-v2, a  multi-agent architectruure deisgned to assist mobile device operation. Mobile-Agent-v2 comprises three agents: planning agent, decision agent, and reflection agent. Frist, planning agent summarizes the task progress based on the operation hsistories. Then, the decision agent generates next operation to be executed. Finally, the reflextion agent classifies whether the excuted operation as correct, errorneous, or ineffective. For evaluation, the authors conduct experiments on real mobile devices in various mobile applications.", "questions": "- Can the authors provide more information about knowledge injection? It would be helpful if the authors included examples to illustrate this process.", "limitations": "- Including some discussion on the scope limitations of Mobile-Agent-v2 would be beneficial. For instance, what happens if the required application is missing—can Mobile-Agent-v2 suggest an alternative method? Additionally, for ambiguous instructions, is Mobile-Agent-v2 capable of resolving these by seeking clarification from the user?", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The application of multi-modal LLMs in mobile operation is intriguing\n- The framework that uses multiple agents demnostrates superior perfornamce compared to the single-agent framework\n- The paper is easy to follow and evaluation is conducted on the real mobile devices", "weaknesses": "- The primary concern is the novelty of the paper. While using multiple agents for mobile operations appears novel, the complexity of tasks used in the evaluation suggests that the implementation of each component within Mobile-Agent-V2 is relatively simple. It would be benifical if the authors evalute the proposed framework in diverse user scenarios (refer to the limitation section).\n- It appears that the authors assume that LLMs possess the knowledge about mobile applications. For example, if the user instruction is \"turn on dark mode,\" how does the LLM determine which setting (general setting, display setting) the agent should execute? Because, sometimes even for the human need the trial and error to execute certain task.  Without grounding the available execution in the given context, the proposed method might require multiple trial and error until it successfully completes the task.\n- Regarding to the upper question, it would be better if the authors provide how many interactions are required to complete a  task compared to the optimal execution.\n- It seems to be neither erroneous nor ineffective operations are recorded in the operation history. If this is the case, isn’t there a potentional risk that the decision agent might repeat the same operation? Woudn’t it be better to use these operation histories to prevent the LLM from repeating erroneous or ineffective executions?\n- There seems some potential risks associated with inaccuracies in every agents within the proposed framework, especially since it utilizes LLMs. For instance, the planning agent might inaccurately summarize operation histories, the decision agent could misunderstand visual inputs, or it might generate incorrect executions. Can Mobile-Agent-v2 automatically identify which component is responsible for an error? or could the authors provide more statistical information about this?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## **Question 1: Lack of evaluation in special scenarios, such as unachievable or ambiguous tasks.**\n## **Response:**\n\n* For tasks that cannot be directly completed through the given instructions, Mobile-Agent-v2 can still attempt to complete them. Due to the high flexibility of Mobile-Agent-v2's operation space, it can simulate almost any operation on a mobile device. Therefore, even if the device does not have the conditions to complete the task, Mobile-Agent-v2 can try to create the conditions through mobile operations. For example, if the task is to open \"Facebook\" but the app is not installed on the device, Mobile-Agent-v2 will open the app store, search for \"Facebook,\" and click to install it. Then it will return to the home screen and open the app.\n* For ambiguous instructions that require explicit user input, Mobile-Agent-v2 can resolve them by expanding the operation space. Mobile-Agent-v2 supports custom operations within the operation space. For example, an operation \"seek help\" can be added with the description: \"Use this operation when you have multiple operation paths to get clearer instructions from the user.\" When multiple choices can fulfill the instruction's requirement, Mobile-Agent-v2 can determine that the current state is ambiguous and proactively invoke the corresponding operation to obtain user guidance.\n* Due to inherent biases in Multi-modal Large Language Models (MLLMs), some operations with strict triggering conditions may be difficult for the MLLMs to use. We are also working on improving the MLLMs‘ contextual learning and operation invocation abilities through alignment training and reinforcement learning.\n\n## **Question 2: How will the agent operate if the agent lacks operation knowledge?**\n## **Response:**\n\n* Mobile-Agent-v2 can leverage the operation knowledge within the MLLMs and the multi-agent cooperation to complete complex tasks. Existing mainstream MLLMs, such as GPT-4, Gemini, and Qwen-VL, have operational experience with many apps. Even if the MLLMs lacks knowledge of certain apps, the decision agent can infer possible operations based on page content and the output of the planning agent. Even if errors occur, they can be corrected by the reflection agent. Additionally, operational capabilities can be acquired through training, such as Apple's Ferret-UI.\n\n* We propose knowledge injection to explore whether external operation knowledge can compensate for the agent's operational deficiencies. If a task is too difficult for the MLLMs' internal knowledge, knowledge injection can generate tutorials. Details on knowledge injection will be addressed in the next question.\n\n* If knowledge injection is not used, the agent can acquire operation knowledge through self-exploration. The agent will perform possible operations and use the reflection agent to observe the results, determining whether the operation was correct. The knowledge gained from exploration is then added to an external knowledge base as input for the decision agent. This way, the agent will not need to explore when facing the same task again. We selected samples from the test set with exploration processes and recorded the steps required to complete the task in the table below, where \"KI\" represents knowledge injection. We can see that the efficiency of the additional exploration process is acceptable.\n|w/o KI|w/ KI|\n|-|-|\n|8.6|6.4|\n\n## **Question 3: How to implement knowledge injection?**\n## **Response:**\n\nOperation knowledge is often a tutorial for a specific app. It is input into the decision model as part of the prompt:\n```python\nif add_info != \"\":\n    prompt += f'''\n### Hint ###\nThere are hints to help you complete the user's instructions. The hints are as follows:\n{add_info}\n'''\n```\nwhere \"add_info\" is the tutorial.\n\nFor example, on TikTok, sharing a video requires clicking the fourth icon on the right. However, the LLM may not have this knowledge. In this case, knowledge can be injected: \"The share icon is a white arrow pointing to the right on the right side of the screen.\" When Mobile-Agent-v2 reaches the step where it needs to click the share icon, it will complete the operation based on the injected knowledge. Since the use of knowledge is determined by the decision agent, other steps are not affected by the presence of the knowledge.\n\n## **Question 4: No record of incorrect operations in operation history.**\n## **Response:**\nTo simplify the structure of the prompt, we have omitted the situation when an incorrect operation occurs. We save the incorrect operations in another part of the decision agent's inputs:\n```python\nif error_flag:\n    prompt += f'''\n### Last operation ###\nYou previously wanted to perform the operation \\\"{last_summary}\\\" on this page and executed the Action \\\"{last_action}\\\". But you find that this operation does not meet your expectation. You need to reflect and revise your operation this time.'''\n```\n\n## **Question 5: Can the framework automatically attribute errors to specific modules or provide error statistics?**\n## **Response:**\n* Thank you for your question. This is an important capability for UI interaction agents. However, since the modules within the framework are interdependent, it is challenging to attribute errors to a specific module solely through global reflection after each operation.\n\n* We manually analyzed the causes of errors in Mobile-Agent-v2 on the evaluation set. The results show that errors can occur at any stage.\n|Planning Agent|Visual Tool Results|Decision Agent|\n|-|-|-|\n|12|10|13|\n\nTherefore, to achieve automated error detection, independent reflections need to be designed for each module. For improvement, we will add reflection on the previous module's results while generating results for each module, preventing errors from propagating between modules. This will not increase latency due to the parallel design. Moreover, errors can be intercepted at the point of occurrence. This will be a key direction for our future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uhNPQlA1EV", "reviewer": "Reviewer_Xdcr", "summary": "This paper introduces an agentic framework, Mobile-Agent-v2, designed to address the challenges of planning and sequential function/tool invocation for Large Language Models (LLMs) in mobile operation scenarios. Mobile-Agent-v2 comprises three agents: a planner, an actioner, and a reflector, which jointly enhance the performance of LLMs in mobile operation tasks. Experimental results demonstrate the advantage of Mobile-Agent-v2 over the single-agent framework: Model-Agent, across various real-world applications.", "questions": "This paper proposes an agentic workflow, comprising a planner, actioner, and reflector, to enhance the planning and sequential tool-calling capabilities of Large Language Models (LLMs) in mobile operations. The motivation behind this approach is sound, and the experimental results appear promising. However, I have several concerns primarily regarding the technical novelty and the insights derived from the experiments.\n\n- The method of decomposing the responsibilities of a single agent into multiple components—planner, actioner, and reflector—is not novel. This approach has been previously proposed by [1] and is widely used in other contexts. Therefore, it is essential to clarify the key challenges and new insights associated with applying this framework to mobile operation scenarios. Without such clarification, it appears that the study merely leverages an existing method and applies it to a specific scenario without significant innovation.\n\n- The experimental results across various mobile applications are promising. However, the sequence length and type are critical factors that warrant detailed discussion. While some analysis is provided, the paper lacks comprehensive results regarding task completion accuracy when sequence length and type vary. For instance, it would be beneficial to understand, across different apps, which types of operations are more successfully completed with different tool sequences. These ablation studies are necessary to offer deeper insights.", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- It is well-written and the structure is clear.\n- Its studied problem of how to design agentic workflow for LLM to improve planning and sequential tool calling is beneficial to the community.\n- The experimental results on real-world mobile apps look promising.", "weaknesses": "- The technical contribution seems limited. Its proposed framework of using planner, actioner and reflector for agentic workflow is a common practice.\n- The evaluation misses some important ablation studies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## **Question 1: Similar multi-agent architectures have been applied in other scenarios.**\n## **Response:**\n\nMobile-Agent-v2 is the first work to use a multi-agent architecture in the mobile domain. The required capabilities of each agent, the tasks they perform, the interaction between agents, and their independent units are significantly different from existing work. Below are the challenges faced by Multi-modal Large Language Models (MLLMs) in the mobile domain and the solutions provided by our framework:\n\n1. **Input type and sequence length:** In the mobile domain, MLLMs' operation decisions face the problem of multi-image long sequences and interleaved text and images input format, which limits MLLMs' decision accuracy. We propose a planning agent to maintain the stability of input sequence length and format in the single image. This input can improve the context learning of MLLMs.\n2. **Operation grounding:** In the mobile domain, MLLMs need to generate decisions and their locations. Currently, mainstream MLLMs (even GPT-4) lack grounding capabilities. To address this, we included a visual perception tool to assist decision-making. This purely visual solution overcomes the limitations of the operating platform and offers greater versatility.\n3. **Error gandling in long sequences:** In the mobile domain, long sequence operation tasks inevitably lead to errors during intermediate operations. When errors occur, MLLMs often struggle to reflect effectively and correct mistakes. We introduced a reflection agent that uses the MLLMs' multi-image understanding capabilities and contextual learning to judge the accuracy of operations and correct errors.\n4. **Following historical screen information:** In the mobile domain, following and navigating history screen information is difficult due to input length limitations and interleaved text and images. We proposed using an independent memory unit to store this information, significantly enhancing the agent's ability to navigate key information.\n\nRegarding the design of the Mobile-Agent-v2 framework, we note positive feedback from other reviewers, such as **c2St** and **cAKE**, who endorsed our approach: **\"The introduction of a multi-agent system to handle different aspects of mobile device operation is a novel approach\"**; **\"Multi-agent framework for mobile device operation is novel in the literature\"**. This confirms the novelty of applying a multi-agent framework in the mobile domain.\n\nMoreover, Mobile-Agent-v2 is a practical framework. Our code, released in the open-source community, has received widespread acclaim, garnered thousands of stars, and has been applied in various real-world scenarios. This practical application stands out among the many works on multi-agent frameworks.\n\n## **Question 2: The relationship between sequence length and operation type requires further study.**\n## **Response:**\n\nWe have compiled statistics on the relationship between operation sequence length and operation accuracy in the table below.\n\n|Sequence Length|Open|Click|Swipe|Type|Back|Home|Stop|\n|-|-|-|-|-|-|-|-|\n|[1, 4)|86.4%|91.5%|100%|75.0%|-|100%|100%|\n|[4, 7)|100%|81.3%|100%|60.0%|100%|100%|88.4%|\n|[7, )|-|80.8%|-|75.0%|-|-| 86.2%|\n\nHere are the conclusions:\n1. **Minimal impact of sequence length in simple operations:** Operations that do not require precise coordinates or parameters, such as \"Swipe\", \"Back\", and \"Home\", are minimally affected by sequence length.\n2. **Success rate variation in complex operations:** For click and stop operations, the success rate is higher at the beginning of the task than in the middle or later stages. This is because early-stage operations usually involve primary pages with better guidance, while advanced pages require the agent to have more robust operational knowledge. However, with the multi-agent architecture, there is no significant decline in the middle and later stages despite the sequence length.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qUyq4Y3kGi", "reviewer": "Reviewer_cAKE", "summary": "This paper proposes mobile-agent-v2, a multi-agent framework for mobile device operation. The framework includes a planning agent, decision agent, reflection agent and memory unit. The multi-agent framework exhibits advantages in reducing errors in long-horizon tasks and achieves better results than single-agent framework on the evaluations designed in the aper.", "questions": "1. See weakness, the evaluation details needs further clarification.", "limitations": "1. This paper relies on commercial models such as GPT-4 and GPT-4V to complete the tasks. The cost of completing a task is uncaculated or discussed in the paper, but is expected to be more cost than the single agent framework.\n2. The efficiency and accuracy trade-off. After introducing mutli-agent framework, the inference time should be much slower.\n\n--------\nAfter the rebuttal, the reviewer think the heavy reliance on commercial models and the time and token cost should be discussed in the revision.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Multi-agent framework for mobile device operation is novel in the literature, improving the overall performance of single agent framework, with the effectiveness of each component verified.", "weaknesses": "1 The paper lacks a comparison of the evaluation benchmark with previous works such as AppAgent and works mentioned in Sec. 2.2. This makes it difficult to evaluate the overall effectiveness of the framework in comparison with previous works.\n2. The evaluation benchmark is relatively small and simple, and how they are evaluated is not very clear. Does the evaluation relies on human evaluator to check the success of each component in the trajectory?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## **Question 1: Lack of other baselines such as CogAgent and AppAgent.**\n## **Response:**\n\n**CogAgent**\n\nCogAgent is a QA model that does not possess the capability to perform concrete operations on real devices through tool invocation. Therefore, our dynamic evaluation framework is not applicable to such QA models.\n\n**AppAgent**\n\nFirst, AppAgent requires an additional exploration phase for each app, whereas Mobile-Agent-v2 can perform app operations without the exploration phase. Secondly, AppAgent relies on XML, restricting it to the Android platform. In contrast, Mobile-Agent-v2 uses a purely visual solution, making it platform-independent. Thus, comparing Mobile-Agent-v2 with AppAgent is not fair.\n\nNevertheless, we manually evaluated the AppAgent and the results are shown in the below table. With the same base model, Mobile-Agent-v2 outperforms AppAgent.\n\n|Model|Success Rate|Completion Rate|\n|-|-|-|\n|AppAgent|66.7%|74.3%|\n|Mobile-Agent-v2|77.8%|82.1%|\n\n## **Question 2: Small and simple evaluation set with unclear evaluation method.**\n## **Response:**\n\n* Mobile-Agent-v2 uses dynamic evaluation, requiring the agent to directly invoke mobile operation tools and connect to a real device for evaluation. This method is more complex and challenging compared to static app screenshots used in existing work, which only evaluate single-step operations.\n\n* Mobile-Agent-v2's dynamic evaluation involves 20 apps, including both system and third-party apps, across various operating systems and languages. Each app has 4 tasks, with an average of 7 steps per task. Each step undergoes three evaluations: planning, decision, and reflection, totaling 1500+ evaluations. This evaluation scale is the largest among works using dynamic evaluation, covering the most apps and having the most comprehensive task coverage.\n\n* We also received endorsements from Reviewer **c2St** and **Xdcr** for our evaluation method: **\"The paper includes a detailed evaluation across different operating systems, language environments, and applications, providing robust evidence of the system's effectiveness.\"** and **\"The experimental results on real-world mobile apps look promising.\"** We also recognize the community's lack of a general benchmark based on dynamic evaluation. We are currently working on creating this benchmark and providing a rich set of operation tools to encourage more models to participate in dynamic evaluation.\n\n## **Question 3: Token cost of usage.**\n## **Response:**\nAssuming an average of 7 steps per task, Mobile-Agent-v2, using a multi-agent architecture, consumes a relatively fixed number of tokens per step. In contrast, a single-agent architecture requires inputting a long sequence of operation history and screenshots with each call, leading to increased token consumption as the number of task steps rises.\n\n|Architecture|Token per Step|Total Token|\n|-|-|-|\n|Mobile-Agent (single-agent)|(1.3k) * steps + 0.4k|43.4k|\n|Mobile-Agent-v2 (multi-agent)|4.4k|30.8k|\n\n## **Question 4: Efficiency reduction in multi-agent systems.**\n## **Response:**\n\n* For time overhead, although multiple agents work serialize, there are still phases to parallel. The tables below compare the operation times for the Mobile-Agent single-agent framework and the Mobile-Agent-v2 multi-agent framework. \"Screenshot\" represents obtaining a screenshot and corresponding image processing from the device, while \"Tool Call\" represents the use of visual perception tools. It is evident that the increased operation time of the multi-agent framework is acceptable under parallel design.\n\n* For memory overhead, Mobile-Agent-v2 calls the large model via API method, so there is no additional memory overhead.\n\n**Mobile-Agent (Single-Agent Framework)**\n|Phase|Preparation|Planning|Decision|Operation|Reflection|Total|\n|-|-|-|-|-|-|-|\n|Task|Screenshot|-|Decision|Tool Call, Grounding|-||\n|Time|2s||20s|12s|-|**34s**||\n\n**Mobile-Agent-v2 (Multi-Agent Framework)**\n|Phase|Preparation|Planning|Decision|Operation|Reflection|Total|\n|-|-|-|-|-|-|-|\n|Task|Parallel with Planning|Screenshot, Tool Call, Planning, Reflection|Decision, Memory|-| Parallel with Planning||\n|Time|-|18s|21s| -| -|**39s**||", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ynDyJ6QStb", "reviewer": "Reviewer_4qiL", "summary": "This paper proposes a multi-agent architecture for mobile device operation, Mobile-Agent-v2. Mobile-Agent-v2 includes three agents : planning agent, decision agent and reflection agent. To retain focus content, this paper designs a memory unit to record task-related focus content. The planning agent generates task progress based on the history operations, the reflection agent corrects erroneous operations and the decision agent outputs operations. Extensive experiments conducted on various operating systems, language environments, and applications shows that MobileAgent-v2 achieves significant performance improvements compared to single-agent architecture of Mobile-Agent.", "questions": "1. Can the Mobile-Agent-v2 be generalized to IOS systems?\n2. How operations knowledge is injected into the model?", "limitations": "The authors describe limitations in the paper.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper introduces a multi-agent architecture Mobile-Agent-v2, which can alleviate various navigating difficulties inherent in the single-agent framework for mobile device operation tasks.\n2. This paper designs a memory unit and reflection agent, which can avoid the loss of focus content navigating and reflection capability.\n3. Experimental results demonstrate that Mobile-Agent-v2 achieves significant performance improvements.", "weaknesses": "1. Insufficient baselines. More powerful baselines should be considered, such as AppAgent [1] and CogAgent [2].\n2. This paper lack description of how the decision agent retrieves the memory unit. In addition,  the content or format of the memory unit storage is not included.\n\n[1] AppAgent: Multimodal Agents as Smartphone Users. 2023.\n\n[2] CogAgent: A Visual Language Model for GUI Agents. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## **Question 1: Can Mobile-Agent-v2 be transferred to iOS or other platform?**\n## **Response:**\n\n* Mobile-Agent adheres to a purely visual solution, making it universally applicable across platforms since Mobile-Agent-v1. Mobile-Agent-v2 continues this approach, allowing it to be transferred to any GUI-based system, such as tablets, PCs, TVs, and automotive systems.\n\n* Specifically, we have implemented it on PCs, facilitating operations such as \"downloading papers in the browser\" and \"modifying font and formatting in Word.\" The relevant code will also be open-sourced on GitHub recently.\n\n## **Question 2: How to implement knowledge injection.**\n## **Response:**\n\nOperation knowledge is often a tutorial for a specific app. It is input into the decision model as part of the prompt:\n```python\nif add_info != \"\":\n    prompt += f'''\n### Hint ###\nThere are hints to help you complete the user's instructions. The hints are as follows:\n{add_info}\n'''\n```\nwhere \"add_info\" is the tutorial.\n\nFor example, on TikTok, sharing a video requires clicking the fourth icon on the right. However, the LLM may not have this knowledge. In this case, knowledge can be injected: \"The share icon is a white arrow pointing to the right on the right side of the screen.\" When Mobile-Agent-v2 reaches the step where it needs to click the share icon, it will complete the operation based on the injected knowledge. Since the use of knowledge is determined by the decision agent, other steps are not affected by the presence of the knowledge.\n\n## **Question 3: Lack of other baselines such as CogAgent and AppAgent.**\n## **Response:**\n\n**CogAgent**\n\nCogAgent is a QA model that does not possess the capability to perform concrete operations on real devices through tool invocation. Therefore, our dynamic evaluation framework is not applicable to such QA models.\n\n**AppAgent**\n\nFirst, AppAgent requires an additional exploration phase for each app, whereas Mobile-Agent-v2 can perform app operations without the exploration phase. Secondly, AppAgent relies on XML, restricting it to the Android platform. In contrast, Mobile-Agent-v2 uses a purely visual solution, making it platform-independent. Thus, comparing Mobile-Agent-v2 with AppAgent is not fair.\n\nNevertheless, we manually evaluated the AppAgent and the results are shown in the below table. With the same base model, Mobile-Agent-v2 outperforms AppAgent.\n\n|Model|Success Rate|Completion Rate|\n|-|-|-|\n|AppAgent|66.7%|74.3%|\n|Mobile-Agent-v2|77.8%|82.1%|\n\n## **Question 4: Memory unit storage format and retrieval method**\n## **Response:**\n\n* Memory is stored as natural language descriptions of task-related content from historical screenshots. For example, in the task \"Check the weather and write a dressing guide in notes,\" after opening the weather app, the agent needs to remember the weather information from the screenshot for subsequent use. Before exiting the weather app, the agent will add the weather information to the memory unit in plain text.\n\n* The stored memory is shared among all agents. When the agent needs to input the dressing guide, the decision agent will automatically retrieve the relevant information from the memory during inference.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely appreciate the reviewers for their valuable and constructive feedback, which will be pivotal in enhancing the quality of our work. 

We are encouraged by the following reviewers' perceptions:
* Innovative and interesting multi-agent architecture in the mobile domain (c2St, kbfL, cAKE).
* Comprehensive experiments are promising (c2St, Xdcr).
* Significant performance improvements (c2St, kbfL, cAKE, 4qiL).
* Well-written and easy to follow (kbfL, Xdcr).

We appreciate the valuable suggestions and questions raised by the reviewers regarding Mobile-Agent-v2. These insights are significantly important for refining our work and guiding future research. We have diligently addressed all concerns and questions from the reviewers in individual responses. Below, we highlight some of the key issues or frequently asked questions raised by the reviewers.

## **Question 1: Overhead of multi-agent.**
## **Response:**
* For time overhead, although multiple agents work serialize, there are still phases to parallel. The tables below compare the operation times for the Mobile-Agent single-agent framework and the Mobile-Agent-v2 multi-agent framework. "Screenshot" represents obtaining a screenshot and corresponding image processing from the device, while "Tool Call" represents the use of visual perception tools. It is evident that the increased operation time of the multi-agent framework is acceptable under parallel design.

* For memory overhead, Mobile-Agent-v2 calls the large model via API method, so there is no additional memory overhead.

**Mobile-Agent (Single-Agent Framework)**
|Phase|Preparation|Planning|Decision|Operation|Reflection|Total|
|-|-|-|-|-|-|-|
|Task|Screenshot|-|Decision|Tool Call, Grounding|-||
|Time|2s||20s|12s|-|**34s**||

**Mobile-Agent-v2 (Multi-Agent Framework)**
|Phase|Preparation|Planning|Decision|Operation|Reflection|Total|
|-|-|-|-|-|-|-|
|Task|Parallel with Planning|Screenshot, Tool Call, Planning, Reflection|Decision, Memory|-| Parallel with Planning||
|Time|-|18s|21s| -| -|**39s**||

## **Question 2: How to implement knowledge injection?**
## **Response:**

Operation knowledge is often a tutorial for a specific app. It is input into the decision model as part of the prompt:
```python
if add_info != "":
    prompt += f'''
### Hint ###
There are hints to help you complete the user's instructions. The hints are as follows:
{add_info}
'''
```
where "add_info" is the tutorial.

For example, on TikTok, sharing a video requires clicking the fourth icon on the right. However, the LLM may not have this knowledge. In this case, knowledge can be injected: "The share icon is a white arrow pointing to the right on the right side of the screen." When Mobile-Agent-v2 reaches the step where it needs to click the share icon, it will complete the operation based on the injected knowledge. Since the use of knowledge is determined by the decision agent, other steps are not affected by the presence of the knowledge.

## **Question 3: Similar multi-agent architectures have been applied in other scenarios.**
## **Response:**

Mobile-Agent-v2 is the first work to use a multi-agent architecture in the mobile domain. The required capabilities of each agent, the tasks they perform, the interaction between agents, and their independent units are significantly different from existing work. Below are the challenges faced by Multi-modal Large Language Models (MLLMs) in the mobile domain and the solutions provided by our framework:

1. **Input type and sequence length:** In the mobile domain, MLLMs' operation decisions face the problem of multi-image long sequences and interleaved text and images input format, which limits MLLMs' decision accuracy. We propose a planning agent to maintain the stability of input sequence length and format in the single image. This input can improve the context learning of MLLMs.
2. **Operation grounding:** In the mobile domain, MLLMs need to generate decisions and their locations. Currently, mainstream MLLMs (even GPT-4) lack grounding capabilities. To address this, we included a visual perception tool to assist decision-making. This purely visual solution overcomes the limitations of the operating platform and offers greater versatility.
3. **Error gandling in long sequences:** In the mobile domain, long sequence operation tasks inevitably lead to errors during intermediate operations. When errors occur, MLLMs often struggle to reflect effectively and correct mistakes. We introduced a reflection agent that uses the MLLMs' multi-image understanding capabilities and contextual learning to judge the accuracy of operations and correct errors.
4. **Following historical screen information:** In the mobile domain, following and navigating history screen information is difficult due to input length limitations and interleaved text and images. We proposed using an independent memory unit to store this information, significantly enhancing the agent's ability to navigate key information.

## **Question 4: Lack of other baselines such as CogAgent and AppAgent.**
## **Response:**

**CogAgent**

CogAgent is a QA model that does not possess the capability to perform concrete operations on real devices through tool invocation. Therefore, our dynamic evaluation framework is not applicable to such QA models.

**AppAgent**

First, AppAgent requires an additional exploration phase for each app, whereas Mobile-Agent-v2 can perform app operations without the exploration phase. Secondly, AppAgent relies on XML, restricting it to the Android platform. In contrast, Mobile-Agent-v2 uses a purely visual solution, making it platform-independent. Thus, comparing Mobile-Agent-v2 with AppAgent is not fair.

Nevertheless, we manually evaluated the AppAgent and the results are shown in the below table. With the same base model, Mobile-Agent-v2 outperforms AppAgent.

|Model|Success Rate|Completion Rate|
|-|-|-|
|AppAgent|66.7%|74.3%|
|Mobile-Agent-v2|77.8%|82.1%|

Meta-review: This paper describes Mobile-Agent-v2, an MLLM-based multi-agent architecture designed to assist with mobile device operations, incorporating the multi-agent collaboration with three agents: a planning agent, a decision agent, and a reflection agent.

The paper was reviewed by five reviewers with four inclined to suggest for acceptance and one not. The majority of reviewers highlighted the strengths, such that (1) the proposed fmobile-agent-v2 framework is an interesting contribution to the mobile domain, (2) the paper is clearly written and easy-to-follow, and (3) the experiments with real-world mobile apps demonstrate promising and significant performance. One reviewer commented several concerns about limited operational knowledge of existing MLLMs and error-correcting capabilities of the agent, which the authors argued are beyond the scope of current research work. However, since this research addresses the challenges in practical mobile device operation tasks, those concerns could be discussed further in the final version. 

Considering the strengths above and the novel approach toward agentic workflow in the mobile domain, this paper is recommended for acceptance. 
The authors are encouraged to utilize their constructive and detailed discussion with reviewers towards further raising the quality and impact of their work.
Decision: Accept (poster)