Title: Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration
Venue: NeurIPS 2024
Accepted: yes
Keywords: multi-agent

multi-modal agent

multi-modal large language model

mobile operation

UI assistant
Abstract: Mobile device operation tasks are increasingly becoming a popular multi-modal AI application scenario. Current Multi-modal Large Language Models (MLLMs), constrained by their training data, lack the capability to function effectively as operation assistants. Instead, MLLM-based agents, which enhance capabilities through tool invocation, are gradually being applied to this scenario. However, the two major navigation challenges in mobile device operation tasks — task progress navigation and focus content navigation — are difficult to effectively solve under the single-agent architecture of existing work. This is due to the overly long token sequences and the interleaved text-image data format, which limit performance. To address these navigation challenges effectively, we propose Mobile-Agent-v2, a multi-agent architecture for mobile device operation assistance. The architecture comprises three agents: planning agent, decision agent, and reflection agent. The planning agent condenses lengthy, interleaved image-text history operations and screens summaries into a pure-text task progress, which is then passed on to the decision agent. This reduction in context length makes it easier for decision agent to navigate the task progress. To retain focus content, we design a memory unit that updates with task progress by decision agent. Additionally, to correct erroneous operations, the reflection agent observes the outcomes of each operation and handles any mistake accordingly. Experimental results indicate that Mobile-Agent-v2 achieves over a 30% improvement in task completion compared to the single-agent architecture of Mobile-Agent. The code is open-sourced at https://github.com/X-PLUG/MobileAgent.

Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration

Abstract

Mobile device operation tasks are increasingly becoming a popular multi-modal AI application scenario. Current Multi-modal Large Language Models (MLLMs), constrained by their training data, lack the capability to function effectively as operation assistants. Instead, MLLM-based agents, which enhance capabilities through tool invocation, are gradually being applied to this scenario. However, the two major navigation challenges in mobile device operation tasks - task progress navigation and focus content navigation - are difficult to effectively solve under the single-agent architecture of existing work. This is due to the overly long token sequences and the interleaved text-image data format, which limit performance. To address these navigation challenges effectively, we propose Mobile-Agent-v2, a multi-agent architecture for mobile device operation assistance. The architecture comprises three agents: planning agent, decision agent, and reflection agent. The planning agent condenses lengthy, interleaved image-text history operations and screens summaries into a pure-text task progress, which is then passed on to the decision agent. This reduction in context length makes it easier for decision agent to navigate the task progress. To retain focus content, we design a memory unit that updates with task progress by decision agent. Additionally, to correct erroneous operations, the reflection agent observes the outcomes of each operation and handles any mistake accordingly. Experimental results indicate that Mobile-Agent-v2 achieves over a 30% improvement in task completion compared to the single-agent architecture of Mobile-Agent. The code is open-sourced at https://github.com/X-PLUG/MobileAgent .

1 Introduction

Multi-modal Large Language Models (MLLMs), represented by GPT-4v OpenAI (2023), have demonstrated outstanding capabilities in various domains Bai et al. (2023); Liu et al. (2023c,b); Dai et al. (2023); Zhu et al. (2023); Chen et al. (2023); Ye et al. (2023a,b); Wang et al. (2023c); Hu et al. (2023, 2024); Zhang et al. (2024b). With the rapid development of agents based on Large Language Models (LLMs) Zhao et al. (2024); Liu et al. (2023f); Talebirad and Nadiri (2023); Zhang et al. (2023b); Wu et al. (2023); Shen et al. (2024); Li et al. (2023a), MLLM-based agents, which can overcome the limitations of MLLMs in specific application scenarios by various visual perception tools, have become a focal point of research attention Liu et al. (2023d).

History Screens & Operations

FocusContentNavigation

141-132

中

Google

中

All notes·

Lakers

Bulls

(23 - 23)

(21-25)

Team

23

LAL

31423929141

CHI

32253639132

TaskProgressNavigation

132

OpenChromeapp

Searchthegameresults

Backtohomepage

Open app (Chrome)

Tap the text(Search

or type URL)

Type(today’s Lakers

game results)

Lakers game results)

Tap the text (today's

Home

Open app (Notes)

Tap the icon (yellow

circle with a plus)

OpenNotesapp

Createanotetotypenews

Instruction:

Searchtheresult fortoday'sLakersgame,and thencreatea note inNotes towritea sportnewsforthisresult.

?

Image-textInterleaveInput[12KToken]

Decision

Planning

Type(TheFocus

Content)

#System#

#Iteration1#

# Iteration 7 #

Reflection

86.5

Instruction:

Screen:

Screen:

......

[image token]

[image token]

50.7

Operation

Operation:

Operation:

Space:

Openapp

Tap theicon

(Chrome)

(yellow circle

......

(snid e ym

Single-agent Architecture

MobileOperationAssistant

Single-agent Multi-agent

Automated operations on mobile devices, as a practical multi-modal application scenario, are emerging as a major technological revolution in AI smartphone development Yao et al. (2022); Deng et al. (2023); Gur et al. (2024); Zheng et al. (2024); Zhang et al. (2023a); Wang et al. (2024); Chen and Li (2024a,b,c); Zhang et al. (2024a); Wen et al. (2023); Cheng et al. (2024). However, due to the limited screen recognition, operation, and location capabilities, existing MLLMs face challenges in this scenario. To address this, existing work leverages MLLM-based agent architecture to endow MLLMs with various capabilities for perceiving and operating mobile device UI. AppAgent Zhang et al. (2023a) tackles the limitation of MLLMs in localization by extracting clickable positions from device XML files. However, the reliance on UI files limits the applicability of this method to other platforms and devices. To eliminate the dependency on underlying UI files, Mobile-Agent Wang et al. (2024) proposes a solution for localization through visual perception tools. It perceives the screen through an MLLM and generates operations, locating their positions by visual perception tools.

Mobile device operation tasks involve multi-step sequential processing. The operator needs to perform a series of continuous operations on the device starting from the initial screen until the instructions are fully executed. There are two main challenges in this process. First, to plan the operation intent, the operator needs to navigate the current task progress from the history operations. Second, some operations may require task-relevant information in the history screens, for example, writing sports news in Figure 1 requires using the match results queried earlier. We refer to this important information as the focus content. The focus content also needs to be navigated out from the history screens. However, as the task progresses, the lengthy history of interleaved image and text history operations and screens as input can significantly reduce the effectiveness of navigation in a single-agent architecture, as shown in Figure 1.

In this paper, we propose Mobile-Agent-v2, a mobile device operation assistant with effective navigation via multi-agent collaboration. Mobile-Agent-v2 has three specialized agent roles: planning agent , decision agent , and reflection agent . The planning agent needs to generate a task progress based on the history operations. To save the focus content from the history screens, we design a memory unit to record task-related focus content. This unit will be observed by the decision agent when generating an operation, simultaneously checking if there is any focus content on the screen and updating it to the memory. Since the decision agent cannot observe the previous screen to reflect, we design the reflection agent to observe the changes in the screen before and after the decision agent's operation and determine whether the operation meets the expectations. If it finds that the operation does not meet expectations, it will take appropriate measures to re-execute the operation. The entire

process is illustrated in Figure 3. The three agent roles work respectively in the progress, decision, and reflection stages, collaborating to alleviate the difficulty of navigating.

Our summarized contributions are as follows:

We propose a multi-agent architecture Mobile-Agent-v2 to alleviate various navigating difficulties inherent in the single-agent framework for mobile device operation tasks. We design a planning agent to generate task progress based on the history operations, ensuring effective operation generation by the decision agent.

To avoid the loss of focus content navigating and reflection capability, we design both a memory unit and a reflection agent. The memory unit is updated by the decision agent with focus content. The reflection agent assesses whether the decision agent's operation meets expectations and generates appropriate remedial measures if expectations are not met.

We conducted dynamic evaluations of Mobile-Agent-v2 across various operating systems, language environments, and applications. Experimental results demonstrate that MobileAgent-v2 achieves significant performance improvements. Furthermore, we empirically validated that the performance of Mobile-Agent-v2 can be further enhanced by manual operation knowledge injection.

2 Related Work

2.1 Muiti-agent Application

The powerful comprehension and reasoning capabilities of Large Language Models (LLMs) enable LLM-based agents to demonstrate the ability to independently execute tasks Brown et al. (2020); Achiam et al. (2023); Touvron et al. (2023a,b); Bai et al. (2023). Inspired by human-team collaboration, the multi-agent framework has been proposed. Park et al. (2023) constructs Smallville consisting of 25 agents in a sandbox environment. Li et al. (2023b) proposes a role-playing-based multi-agent collaborative framework to enable two agents playing different roles to autonomously collaborate. Chen et al. (2024) innovatively propose an effective multi-agent framework for coordinating the collaboration of multiple expert agents. Hong et al. (2024) presents a groundbreaking meta-programming multi-agent collaboration framework. Wu et al. (2024) proposes a generic multi-agent framework that allows users to configure the number of agents, interaction modes, and toolsets. Chan et al. (2024); Subramaniam et al. (2024); Tao et al. (2024) investigate the implementation of a multi-agent debating framework, aiming to evaluate the quality of different texts or generated content. Abdelnabi et al. (2024); Xu et al. (2024); Mukobi et al. (2024) integrate multi-agent interaction with game theoretic strategies, aiming to enhance both the cooperative and decision abilities.

2.2 LLM-based UI Operation Agent

Webpages, as a classic application scenario for UI agents, have attracted widespread attention to research on web agents. Yao et al. (2022) and Deng et al. (2023) aim to enhance the performance of agents on real-world webpage tasks by constructing high-quality website task datasets. Gur et al. (2024) utilizes pre-trained LLMs and self-experience learning to automate task processing on real-world websites. Zheng et al. (2024) leverages GPT-4V for visual understanding and webpage manipulation. Simultaneously, research on LLM-based UI agents for mobile platforms has also drawn significant attention. Wen et al. (2023) converts Graphical User Interface (GUI) information into HTML representations and then leverages LLM in conjunction with application-specific domain knowledge. Yan et al. (2023) proposes a multi-modal intelligent mobile agent based on GPT-4V, exploring the direct utilization of GPT-4V to perceive screen screenshots with annotations. Unlike the former approach that operates on screens with digital labels, Zhang et al. (2023a) combines the application's XML files for localization operations, mimicking human spatial autonomy in operating mobile applications. Wang et al. (2024) eliminates the dependency on the application's XML files and leverages visual module tools for localization operations. Additionally, Hong et al. (2023) designed a GUI agent based on pre-trained vision-language models. Chen and Li (2024a,b,c) propose small-scale client-side models for deployment on actual devices. Zhang et al. (2024a) proposed a UI multi-agent framework tailored for the Windows operating system. Despite the significant performance improvements achieved by multi-agent architectures in many tasks, currently, there is no work that employs multi-agent architectures in mobile device operation tasks. To address the

VPM

TP

TP

t-1

Planning Agent

Decision Agent

ReflectionAgent

O

Rt

Ins, Rt-1

Ins, Ot-1

FCt-1

FCt-1

FC

Ins, Ot

FCt

Memory Unit

challenges of long-context navigation in mobile device operation tasks, in this paper, we introduce the multi-agent architecture Mobile-Agent-v2.

3 Mobile-Agent-v2

In this section, we will provide a detailed overview of the architecture of Mobile-Agent-v2. The operation of Mobile-Agent-v2 is iterative, and its process is depicted in Figure 2. Mobile-Agent-v2 has three specialized agent roles: planning agent, decision agent, and reflection agent. We also design the visual perception module and memory unit to enhance the agent's screen recognition capability and the capability to navigate focus content from history. Firstly, the planning agent updates the task progress, allowing the decision agent to navigate the progress of the current task. The decision agent then operates based on the current task progress, current screen state, and the reflection (if the last operation is erroneous). Subsequently, the reflection agent observes the screens before and after the operation to determine if the operation meets expectations.

3.1 Visual Perception Module

Screen recognition remains challenging even for state-of-the-art MLLMs when processed end-toend. Therefore, we have incorporated a visual perception module to enhance the screen recognition capability. In this module, we utilize three tools: text recognition tool, icon recognition tool, and icon description. Inputting a screenshot into this module will ultimately yield the text and icon information present on the screen, along with their respective coordinates. This process is represented by the following formula:

where P t represents the perception result of the screen in the t -th iteration.

3.2 Memory Unit

Due to the task progress generated by the planning agent being in textual form, the navigation of focus content from history screens is still challenging. To address this issue, we design a memory unit to store the focus content related to the current task from history screens. The memory unit serves as a short-term memory module that is updated as the task progresses. The memory unit is crucial for scenarios involving multiple apps. For instance, as shown in Figure 3, weather information observed by the decision agent will be utilized in subsequent operations. At this point, the information related to the weather app's page will be updated in the memory unit.

3.3 Planning Agent

We aim to reduce the reliance on lengthy history operations during decision-making by employing a separate agent. We observe that although each round of operation occurs on different pages and is different, often the goals of multiple operations are the same. For example, in the example illustrated

Planning Stage

Decision Stage

ReflectionStage

LastOperation

11:34

Ineedtoopen

theweatherapp

Screenshot

currently open. SoI

Theweatherappis

LastScreenshot

It canbe seen from

thescreenbefore

ErroneousOperation

first.It can be

23

need to record the

23

andafterthe

seenfrom the

weatherinformation in

operation that the

previous step

the page.Then return

devicereturnstothe

that this task has

to the desktop to open

desktop.Therefore

GoBack

been completed.

the next app.

thisoperationwas

andRetry

successful.

Or

Completed Tasks

Planning Agent

Completed Tasks

DecisionAgent

NextScreenshot

11:36

ReflectionAgent

IneffectiveOperation

Record

Process

Process

weather)

start

start

Backto

Open

home

"weather”

Update

app

#Thought:

Reflection

Ihave checked the today's

Retry

UserInstruction

then go to Notes and

Check today's weather,

#UserInstruction

UserInstruction

weather andIneed togoHome

page to open the next app

#Action:

#Memory:

UserInstruction

Reflection

CorrectOperation

Or

Memory

Unit

Operation

History

write a dressing guide.

#LastOperations

Memory

n

Operation

History

Home

#Memory Unit

Update theweather

Memory

Unit

Operation

History

Open app(Weather)

The weatherisxxx.

Continue

Global Information

UpdateMemory

in Figure 1, the first four operations are all about searching for match results. Therefore, we design a planning agent to summarize the history operations and track task progress.

We define the operation generated by the decision agent at the t -th iteration as O t . Before the decision agent makes a decision, the planning agent observes the decision agent's operation O t -1 from the last iteration and updates the task progress TP t -1 to TP t . The task progress includes the sub-tasks that have already been completed. After generating the task progress, the planning agent passes it to the decision agent. This aids the decision agent in considering the content of tasks that have not yet been completed, thereby facilitating the generation of the next operation. As shown in Figure 3, the planning agent's inputs consist of four parts: user instruction Ins , the focus content FC t in memory unit, the previous operation O t -1 , and the previous task progress TP t -1 . Based on above information, the planning agent generates the TP t . This process is represented by the following formula:

where the PA represents the LLM of the planning agent.

3.4 Decision Agent

The decision agent operates during the decision stage, generating operations O and implementing them on the device, while also being responsible for updating the focus content FC in memory unit. This process is illustrated in the Decision Stage shown in Figure 3 and represented by the following formula:

where the DA represents the MLLM of the decision agent and the R t represents the reflection result from reflection agent.

Operation Space. To reduce the complexity of operations, we design an operation space and restricted the decision agent to selecting operations only from within this space. For operations with higher degrees of freedom, such as tapping and swiping, we incorporate an additional parameter space to locate or handle specific content. Below is a detailed description of the operation space:

Open app ( app name ). If the current page is the home page, this operation can be used to open the app named " app name ".

Tap ( x , y ). This operation is used to tap on the position with coordinates ( x , y ).

Swipe ( x1 , y1 ), ( x2 , y2 ). This operation is used to swipe from the position with coordinates ( x1 , y1 ) to the position with coordinates ( x2 , y2 ).

Type ( text ). If the current keyboard is in an active state, this operation can be used to input the content of " text " in the input box.

Home. This operation is used to return to the home page from any page.

Stop. If the decision agent thinks that all requirements have been fulfilled, it can use this operation to terminate the entire operation process.

Memory Unit Update. As each operation made by the decision agent is highly task-relevant and based on the visual perception results of the current page, it is well-suited to observe task-related focus content within the screen pages. Accordingly, we have endowed the decision agent with the ability to update the memory unit. When making decisions, the decision agent is prompted to observe whether there is task-related focus content within the current screen page. If such information is observed, the decision agent updates it in memory for reference in subsequent decisions. This process is represented by the following formula:

3.5 Reflection Agent

Even with the visual perception module, Mobile-Agent-v2 may still generate unexpected operations. In some specific scenarios, MLLMs may even produce severe hallucinations Liu et al. (2023a); Li et al. (2023c); Gunjal et al. (2024); Wang et al. (2023b); Zhou et al. (2023), even the most advanced MLLM GPT-4V Cui et al. (2023); Wang et al. (2023a). Therefore, we design the reflection agent to observe the screen state before and after a decision agent's operation to determine whether the current operation meets expectations. This process is represented by the following formula:

where the RA represents the MLLM of the reflection agent.

As shown in Figure 3, the reflection agent generates three types of reflection results after operation execution: erroneous operation, ineffective operation, and correct operation. The following will describe these three reflection results:

Erroneous operation refers to an operation that leads the device to enter a page unrelated to the task. For example, the agent intends to chat with contact A in a messaging app, but it accidentally opens the chat page of contact B instead.

Ineffective operation refers to an operation that does not result in any changes to the current page. For example, the agent intends to tap on an icon, but it taps on the blank space next to the icon instead.

Correct operation refers to an operation that meets the decision agent's expectations and serves as a step towards fulfilling the requirements of the user instruction.

If erroneous operation, the page will revert to the state before the operation. If ineffective operation, the page will remain in its current state. Neither erroneous nor ineffective operations are recorded in the operation history to prevent the agent from following these operations. If correct operation, the operation will be updated in the operation history, and the page will be updated to the current state.

4 Experiments

4.1 Model

Visual Perception Module. For the text recognition tool, we use the document OCR recognition model ConvNextViT-document from ModelScope 3 . For the icon recognition tool, we employ GroundingDINO Liu et al. (2023e), a detection model capable of detecting objects based on natural language prompts. For the icon description tool, we utilize the Qwen-VL-Int4 4 .

MLLMs. For the planning agent, as it does not require screen perception, we utilize the text-only GPT-4 OpenAI (2023). For the decision agent and reflection agent, we employ GPT-4V OpenAI (2023). All calls are made through the official API method provided by the developers.

4.2 Evaluation

Evaluation Method. To evaluate the performance of Mobile-Agent-v2 on real mobile devices, we employed a dynamic evaluation method. This evaluation method requires the operation tool to implement the agent's operations in real time on an actual device. We use two mobile operation systems, Harmony OS and Android OS, to assess capabilities in non-English and English scenarios, respectively. We used Android Debug Bridge (ADB) as the tool for operating mobile devices. ADB can simulate all operations of Mobile-Agent-v2 in the operation space. In both scenarios, we select 5 system apps and 5 popular external apps for evaluation. For each app, we devise two basic instructions and two advanced instructions. Basic instructions are relatively simple operations with clear instructions within the app interface, while advanced instructions require a certain level of experience with app operations to complete. Additionally, to evaluate multi-app operation capability, we design two basic instructions and two advanced instructions involving multiple apps. In total, there were 88 instructions for non-English and English scenarios, comprising 40 instructions for system apps, 40 instructions for external apps, and 8 instructions for multi-app operations. 5 The apps and instructions used for evaluation in non-English and English scenarios are presented in the appendix.

Metrics. We design the following four metrics for dynamic evaluation:

Success Rate (SR): When all the requirements of a user instruction are fulfilled, the agent is considered to have successfully executed this instruction. The success rate refers to the proportion of user instructions that are successfully executed.

Completion Rate (CR): Although some challenging instructions may not be successfully executed, the correct operations performed by the agent are still noteworthy. The completion rate refers to the proportion of correct steps out of the ground truth operations.

Decision Accuracy (DA): This metric reflects the accuracy of the decision by the decision agent. It is the proportion of correct decisions out of all decisions.

Reflection Accuracy (RA): This metric reflects the accuracy of reflection by the reflection agent. It is the proportion of correct reflections out of all reflections.

Implementation Details. We use Mobile-Agent as the baseline. Mobile-Agent is a single-agent architecture based on GPT-4V end-to-end screen recognition. We fix the seed for GPT-4V invocation and set the temperature to 0 to avoid randomness. In addition to Mobile-Agent-v2, we further introduce the scenario of knowledge injection. This involves providing the agent with some operation hints in addition to the user instructions to aid the agent. It's worth noting that we only injected knowledge for instructions that Mobile-Agent-v2 couldn't accomplish. For instructions that could be completed without additional assistance, we keep the input unchanged.

4.3 Results

4.3.1 Evaluation

Evaluation on Task Completion. Table 1 and 2 respectively illustrate the performance of MobileAgent-v2 in non-English and English scenarios. Compared to Mobile-Agent, Mobile-Agent-v2 exhibits significant improvements in both basic and advanced instructions. With the multi-agent architecture, even in highly challenging advanced instructions, the success rate can still reach 55%, compared to only 20% with Mobile-Agent. Even in the English scenario, Mobile-Agent-v2 still achieves a significant performance improvement. Although Mobile-Agent performs better in English scenarios compared to Chinese scenarios, Mobile-Agent-v2 still achieves an average improvement of 27% in success rate.

Evaluation on Reflection Capability. In the case of knowledge injection, even though the decision accuracy does not reach 100%, the completion rate can still reach 100%. This indicates that even with knowledge injection, Mobile-Agent-v2 still makes erroneous decisions. Errors in decision-making are difficult to avoid, even for humans. Therefore, the importance of the reflection agent is highlighted.

Evaluation on App Type. From all metrics, it can be observed that the performance of all methods on system apps exceeds that of external apps. From the results of multiple apps, it can be seen that Mobile-Agent-v2 achieves improvements of 37.5% and 44.2% in SR and CR, respectively, compared to Mobile-Agent. Compared to single-app tasks, multi-app tasks rely more on the retrieval of history operations and focus content. The significant performance improvement indicates that the multi-agent architecture and memory unit of Mobile-Agent-v2 play an important role.

Evaluation on Operation Knowledge Injection. From the results of knowledge injection in Table 1 and 2, it can be observed that operation knowledge can effectively enhance the performance of Mobile-Agent-v2, which suggests that manually injected operation knowledge can mitigate the limitations of an agent's operation capability. This finding implies that knowledge injection can broaden the application scenarios of Mobile-Agent-v2 because even complex tasks can be guided by manually written operation tutorials to instruct agents. This finding may offer new insights for automated script testing on mobile devices and suggests that to enhance the operation capabilities of MLLMs to their limits, automating the generation of high-quality operation knowledge can further improve the performance of Mobile-Agent-v2. Moreover, the success brought about by knowledge injection also opens up new avenues for future mobile app testing. Existing mobile app testing solutions are still limited to manual script writing, which restricts the universality of testing and raises

the threshold for users. To address the aforementioned issues, one can inject natural language testing procedures into Mobile-Agent-v2. After injecting accurate testing procedures, the system can operate normally regardless of changes in the size or color of the mobile interface. Additionally, language descriptions eliminate the need for a knowledge base in script writing.

Evaluation on MLLMs. In Table 3, we evaluate the performance of different MLLMs within the Mobile-Agent-v2 framework. Since some models are not well-suited for handling sequential inputs, we selected specific instructions and modified each step to function as a single-step task. Therefore, we only evaluate the SR (which is the same as DA). We also evaluate the direct use of GPT-4V, bypassing the agent architecture for end-to-end operation. The results indicate that using GPT-4V directly as a mobile device operation assistant is almost infeasible. GPT-4V combined with the agent architecture remains the most effective configuration for operational capabilities.

4.3.2 Ablation Study

We conduct ablation studies on Mobile-Agent-v2, including the planning agent, reflection agent, and memory unit. From the results in Table 4, it is evident that the impact of the planning agent on the overall framework is the most significant. This further demonstrates the challenging nature of navigation in long sequences for current MLLMs. Additionally, performance declines are observed after removing the reflection agent and memory unit. The reflection agent is essential for correcting erroneous operations. It enables the decision agent to avoid operating on incorrect pages or getting stuck in loops of invalid operations. The memory unit is crucial for successful execution in multi-app scenarios. Even in scenarios involving multiple sub-tasks, the memory can sometimes record the positions of critical UI elements, aiding better localization for the next sub-task execution.

4.3.3 Analysis of Operation Sequence Length

As shown in Figure 4, we analyze the positions of erroneous or ineffective operations within failed instructions in English scenarios, dividing the relative sequence positions into three equal parts. The results indicate that in Mobile-Agent, such errors or ineffective operations predominantly occur in the later stages of the tasks. In contrast, Mobile-Agent-v2 does not exhibit any obvious pattern. This indicates that the multi-agent architecture is better equipped to handle the challenges posed by long sequences in UI operation tasks.

Mobile-Agent

Mobile-Agent-v2

10

10

Erroneous or Ineffective Operations

8

Erroneous or Ineffective Operations

8

6

6

4

4

2

2

0

0-1/3

1/3-2/3

2/3 - 1

0

0-1/3

1/3-2/3

2/3 - 1

Relative Round

Relative Round

#User Instruction

Check tne i

video on Bilibili.

9:46

Decision

?

poeeChrat acp.

# Last Operations

Open app (微信)

Action:

(36.1026)

口

?

sion

Ope

[36,1028]

# Last Operatior

nation, the r

the

fron

Decision

# Last Operations

x00.

Opened the WeChat app and

Ueho$ owoy g o Suo6 fq

#Action

content *Moble

9:54

助

Plan

Last Op

Action

avpto.

Action:

Tap (614, 235)

Last Operations

Thoe

qa gaexqpues

dtothe

perd is not

cand.

WeChat app

Reflection

f'. This

Reflect

aigns

ows the

#Last Opera

Last Oper

Planning

I the V

ed the Bibii

VeChat,

1 9ig pe

box

boer

Decision

Decision

Type (Mob

Act

on:

Tap (1225, 25)

#Action:

Tap (964, 1124)

#Action

#Action:

Stop

. There :

search

Reflect

Oinl

an) dde uado

Action:

(348.,09)

Planning

?

typed i

# Last Operat

into the

P

# Last Opera

Action:

Though:

Decision

Tap (348, 209)

（346.

Decision

Tap (346, 788)

Action

×

TOP

pag

(b)

4.3.4 Case Study

Figure 5 (a) illustrates a complete operational process of Mobile-Agent-v2. Under the planning of the planning agent, the decision agent can navigate the task progress correctly in the case of single-image input. Meanwhile, the memory unit accurately stores the chat content needed for the task and, when required for search purposes, the decision agent can effectively navigate to it. Figure 5 (b) illustrates an example of correcting ineffective operations through reflection. After the failure of the previous operation, the reflection agent promptly detects the error and communicates the reflection result to the decision agent. Based on this, the decision agent rethinks and implements the correct operation.

5 Conclusion

Existing single-agent architectures for mobile device operation assistants experience a significant reduction in navigation effectiveness when dealing with long sequences of interleaved text and images, thereby limiting their performance. To address this issue, in this paper, we propose Mobile-Agent-v2, a mobile device operation assistant with efficient navigation via multi-agent collaboration. We address the aforementioned navigating challenges through the planning agent and memory unit, respectively. Additionally, we design reflectors to ensure the smooth progress of tasks. Experimental results demonstrate that Mobile-Agent-v2 achieves significant performance improvements compared to the single-agent Mobile-Agent. Furthermore, we find that performance can be further enhanced through the injection of manual operation knowledge, providing new directions for future work.

References

Abdelnabi, S., Gomaa, A., Sivaprasad, S., Schönherr, L., and Fritz, M. (2024). LLM-deliberation: Evaluating LLMs with interactive multi-agent negotiation game.

Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. (2023). Gpt-4 technical report. arXiv preprint arXiv:2303.08774 .

Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., and Zhou, J. (2023). Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966 .

Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020). Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901.

Chan, C.-M., Chen, W., Su, Y., Yu, J., Xue, W., Zhang, S., Fu, J., and Liu, Z. (2024). Chateval: Towards better LLM-based evaluators through multi-agent debate. In The Twelfth International Conference on Learning Representations .

Chen, J., Li, D. Z. X. S. X., Zhang, Z. L. P., Xiong, R. K. V. C. Y., and Elhoseiny, M. (2023). Minigpt-v2: Large language model as a unified interface for vision-language multi-task learning. arXiv preprint arXiv:2310.09478 .

Chen, W. and Li, Z. (2024a). Octopus v2: On-device language model for super agent. arXiv preprint arXiv:2404.01744 .

Chen, W. and Li, Z. (2024b). Octopus v3: Technical report for on-device sub-billion multimodal ai agent. arXiv preprint arXiv:2404.11459 .

Chen, W. and Li, Z. (2024c). Octopus v4: Graph of language models. arXiv preprint arXiv:2404.19296 .

Chen, W., Su, Y., Zuo, J., Yang, C., Yuan, C., Chan, C.-M., Yu, H., Lu, Y., Hung, Y .-H., Qian, C., Qin, Y., Cong, X., Xie, R., Liu, Z., Sun, M., and Zhou, J. (2024). Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors. In The Twelfth International Conference on Learning Representations .

Cheng, K., Sun, Q., Chu, Y., Xu, F., Li, Y., Zhang, J., and Wu, Z. (2024). Seeclick: Harnessing gui grounding for advanced visual gui agents. arXiv preprint arXiv:2401.10935 .

Cui, C., Zhou, Y., Yang, X., Wu, S., Zhang, L., Zou, J., and Yao, H. (2023). Holistic analysis of hallucination in gpt-4v (ision): Bias and interference challenges. arXiv preprint arXiv:2311.03287 .

Dai, W., Li, J., Li, D., Tiong, A. M. H., Zhao, J., Wang, W., Li, B., Fung, P., and Hoi, S. (2023). Instructblip: Towards general-purpose vision-language models with instruction tuning. arXiv preprint arXiv:2305.06500 .

Deng, X., Gu, Y., Zheng, B., Chen, S., Stevens, S., Wang, B., Sun, H., and Su, Y. (2023). Mind2web: Towards a generalist agent for the web. CoRR , abs/2306.06070.

Gunjal, A., Yin, J., and Bas, E. (2024). Detecting and preventing hallucinations in large vision language models. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 18135-18143.

Gur, I., Furuta, H., Huang, A. V ., Safdari, M., Matsuo, Y ., Eck, D., and Faust, A. (2024). A real-world webagent with planning, long context understanding, and program synthesis. In The Twelfth International Conference on Learning Representations .

Hong, S., Zhuge, M., Chen, J., Zheng, X., Cheng, Y., Wang, J., Zhang, C., Wang, Z., Yau, S. K. S., Lin, Z., Zhou, L., Ran, C., Xiao, L., Wu, C., and Schmidhuber, J. (2024). MetaGPT: Meta programming for a multi-agent collaborative framework. In The Twelfth International Conference on Learning Representations .

Hong, W., Wang, W., Lv, Q., Xu, J., Yu, W., Ji, J., Wang, Y., Wang, Z., Zhang, Y., Li, J., Xu, B., Dong, Y., Ding, M., and Tang, J. (2023). Cogagent: A visual language model for gui agents.

Hu, A., Shi, Y., Xu, H., Ye, J., Ye, Q., Yan, M., Li, C., Qian, Q., Zhang, J., and Huang, F. (2023). mplug-paperowl: Scientific diagram analysis with the multimodal large language model. arXiv preprint arXiv:2311.18248 .

Hu, A., Xu, H., Ye, J., Yan, M., Zhang, L., Zhang, B., Li, C., Zhang, J., Jin, Q., Huang, F., et al. (2024). mplug-docowl 1.5: Unified structure learning for ocr-free document understanding. arXiv preprint arXiv:2403.12895 .

Li, C., Chen, H., Yan, M., Shen, W., Xu, H., Wu, Z., Zhang, Z., Zhou, W., Chen, Y., Cheng, C., et al. (2023a). Modelscope-agent: Building your customizable agent system with open-source large language models. arXiv preprint arXiv:2309.00986 .

Li, G., Hammoud, H. A. A. K., Itani, H., Khizbullin, D., and Ghanem, B. (2023b). CAMEL: Communicative agents for 'mind' exploration of large language model society. In Thirty-seventh Conference on Neural Information Processing Systems .

Li, Y., Du, Y., Zhou, K., Wang, J., Zhao, W. X., and Wen, J.-R. (2023c). Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355 .

Liu, F., Lin, K., Li, L., Wang, J., Yacoob, Y., and Wang, L. (2023a). Aligning large multi-modal model with robust instruction tuning. arXiv preprint arXiv:2306.14565 .

Liu, H., Li, C., Li, Y., and Lee, Y. J. (2023b). Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744 .

Liu, H., Li, C., Wu, Q., and Lee, Y. J. (2023c). Visual instruction tuning. arXiv preprint arXiv:2304.08485 .

Liu, S., Cheng, H., Liu, H., Zhang, H., Li, F., Ren, T., Zou, X., Yang, J., Su, H., Zhu, J., et al. (2023d). Llava-plus: Learning to use tools for creating multimodal agents. arXiv preprint arXiv:2311.05437 .

Liu, S., Zeng, Z., Ren, T., Li, F., Zhang, H., Yang, J., Li, C., Yang, J., Su, H., Zhu, J., et al. (2023e). Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499 .

Liu, Z., Zhang, Y., Li, P., Liu, Y ., and Yang, D. (2023f). Dynamic llm-agent network: An llm-agent collaboration framework with agent team optimization. arXiv preprint arXiv:2310.02170 .

Mukobi, G., Erlebach, H., Lauffer, N., Hammond, L., Chan, A., and Clifton, J. (2024). Welfare diplomacy: Benchmarking language model cooperation.

OpenAI (2023). Gpt-4 technical report. ArXiv , abs/2303.08774.

Park, J. S., O'Brien, J., Cai, C. J., Morris, M. R., Liang, P., and Bernstein, M. S. (2023). Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology , UIST '23. ACM.

Shen, W., Li, C., Chen, H., Yan, M., Quan, X., Chen, H., Zhang, J., and Huang, F. (2024). Small llms are weak tool learners: A multi-llm agent. arXiv preprint arXiv:2401.07324 .

Subramaniam, V., Torralba, A., and Li, S. (2024). DebateGPT: Fine-tuning large language models with multi-agent debate supervision.

Talebirad, Y. and Nadiri, A. (2023). Multi-agent collaboration: Harnessing the power of intelligent llm agents. arXiv preprint arXiv:2306.03314 .

Tao, M., Zhao, D., and Feng, Y. (2024). Chain-of-discussion: A multi-model framework for complex evidence-based question answering.

Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023a). Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 .

Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023b). Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 .

Wang, J., Wang, Y., Xu, G., Zhang, J., Gu, Y., Jia, H., Yan, M., Zhang, J., and Sang, J. (2023a). An llm-free multi-dimensional benchmark for mllms hallucination evaluation. arXiv preprint arXiv:2311.07397 .

Wang, J., Xu, H., Ye, J., Yan, M., Shen, W., Zhang, J., Huang, F., and Sang, J. (2024). Mobileagent: Autonomous multi-modal mobile device agent with visual perception. arXiv preprint arXiv:2401.16158 .

Wang, J., Zhou, Y., Xu, G., Shi, P., Zhao, C., Xu, H., Ye, Q., Yan, M., Zhang, J., Zhu, J., et al. (2023b). Evaluation and analysis of hallucination in large vision-language models. arXiv preprint arXiv:2308.15126 .

Wang, W., Lv, Q., Yu, W., Hong, W., Qi, J., Wang, Y., Ji, J., Yang, Z., Zhao, L., Song, X., et al. (2023c). Cogvlm: Visual expert for pretrained language models. arXiv preprint arXiv:2311.03079 .

Wen, H., Li, Y., Liu, G., Zhao, S., Yu, T., Li, T. J.-J., Jiang, S., Liu, Y ., Zhang, Y ., and Liu, Y . (2023). Autodroid: Llm-powered task automation in android.

Wu, Q., Bansal, G., Zhang, J., Wu, Y., Li, B., Zhu, E., Jiang, L., Zhang, X., Zhang, S., Liu, J., Awadallah, A. H., White, R. W., Burger, D., and Wang, C. (2024). Autogen: Enabling next-gen LLM applications via multi-agent conversation.

Wu, Q., Bansal, G., Zhang, J., Wu, Y., Zhang, S., Zhu, E., Li, B., Jiang, L., Zhang, X., and Wang, C. (2023). Autogen: Enabling next-gen llm applications via multi-agent conversation framework. arXiv preprint arXiv:2308.08155 .

Xu, Z., Yu, C., Fang, F., Wang, Y., and Wu, Y. (2024). Language agents with reinforcement learning for strategic play in the werewolf game.

Yan, A., Yang, Z., Zhu, W., Lin, K., Li, L., Wang, J., Yang, J., Zhong, Y., McAuley, J., Gao, J., Liu, Z., and Wang, L. (2023). Gpt-4v in wonderland: Large multimodal models for zero-shot smartphone gui navigation.

Yao, S., Chen, H., Yang, J., and Narasimhan, K. R. (2022). Webshop: Towards scalable real-world web interaction with grounded language agents. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K., editors, Advances in Neural Information Processing Systems .

Ye, Q., Xu, H., Xu, G., Ye, J., Yan, M., Zhou, Y., Wang, J., Hu, A., Shi, P., Shi, Y., et al. (2023a). mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304.14178 .

Ye, Q., Xu, H., Ye, J., Yan, M., Liu, H., Qian, Q., Zhang, J., Huang, F., and Zhou, J. (2023b). mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration. arXiv preprint arXiv:2311.04257 .

Zhang, C., Li, L., He, S., Zhang, X., Qiao, B., Qin, S., Ma, M., Kang, Y., Lin, Q., Rajmohan, S., Zhang, D., and Zhang, Q. (2024a). Ufo: A ui-focused agent for windows os interaction.

Zhang, C., Yang, Z., Liu, J., Han, Y., Chen, X., Huang, Z., Fu, B., and Yu, G. (2023a). Appagent: Multimodal agents as smartphone users.

Zhang, J., Xu, X., and Deng, S. (2023b). Exploring collaboration mechanisms for llm agents: A social psychology view. arXiv preprint arXiv:2310.02124 .

Zhang, L., Hu, A., Xu, H., Yan, M., Xu, Y., Jin, Q., Zhang, J., and Huang, F. (2024b). Tinychart: Efficient chart understanding with visual token merging and program-of-thoughts learning. arXiv preprint arXiv:2404.16635 .

Zhao, A., Huang, D., Xu, Q., Lin, M., Liu, Y.-J., and Huang, G. (2024). Expel: Llm agents are experiential learners. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 19632-19642.

Zheng, B., Gou, B., Kil, J., Sun, H., and Su, Y. (2024). Gpt-4v(ision) is a generalist web agent, if grounded. CoRR , abs/2401.01614.

Zhou, Y., Cui, C., Yoon, J., Zhang, L., Deng, Z., Finn, C., Bansal, M., and Yao, H. (2023). Analyzing and mitigating object hallucination in large vision-language models. arXiv preprint arXiv:2310.00754 .

Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. (2023). Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592 .

A Appendix / supplemental material

A.1 Evaluation Application and Instruction

Table 5 and 6 present the applications and instructions used for dynamic evaluation in both English and non-English scenarios. Basic instructions are relatively simple operations with clear instructions within the app interface, while advanced instructions require a certain level of experience with app operations to complete.

A.2 Agent Prompt

In Table 7, 8, 9, and 10, we present the system and user prompts for the planning agent, decision agent, and reflection agent. Notably, since the history of operations is empty at the beginning of the task, we display the initial planning and subsequent planning prompts for the planning agent separately in Table 7 and 8.

For image inputs, we differentiated based on the task characteristics of each agent. For the planning agent, as navigating historical operations and generating task plans is a purely textual process, no screenshots are required. For the decision agent, we input the screenshot of the current device state. For the reflection agent, we additionally retain the screenshot from the previous operation and input it alongside the current device state screenshot in chronological order.

Since some instructions need to be specially tailored for specific mobile applications, we have hidden these details to protect privacy. In the Table 5 and 6, "xxx" represents the redacted information. For certain celebrities or locations mentioned in the instructions, we retained them as they do not involve privacy concerns.

#UserInstruction

Replyto all theunread messagesonWhatsApp

WhatsApp

@Q

4

Egyptian pyramids

whichareamong the

worlds largest

WhatsApp

Introducethe Musk forme.

and

CEO, and lead de

lead designer of SpaceX

and co-founder and initial

n of OpenAI]

#UserInstruction

Search for Elon Musk on X app and follow him

17:28

+

1731 @

1734 e

17.36

?

Search )

For you

Following

Natu

Tangers atAthletics

Trends for you

#cosplay

D

O562t314.4K86K3.6M

Xinjiang

2,579 posts

?

小

TrendingiHong Kng SAR Chn

The Chinese

76.9K posts

10.5K posts

28

28

Q

日

Subscribe

Elon Musk

Elon Musk

@elonmusk

Elon Musk (Parody)

Suggested

NASA

Joined June 2009

182MF

BORED

There's space foreverybody.

150 St

Elon Musk (Parody)

Not fol

Binance

Subs

Highlights

ElonMusk News

The world's leading block

ecosystem

ThestandardTeslaModelYnowhasa

Pinned

AappleCEOAubu

National Parks

@tim_cook

Follc

precisonoflsts

Athough itisrerwheel-rie, the

means it sillhasgreatraction on snow

320 mile range!

Elon Musk (Parody)

Elon Musk Quotes

Ice with all-season tres.

593 Following 182M Fll

Joined June 2@

Elon Musk -Parody

150 Subscriptions

#UserInstruction

Searchfor"Musk"onTikTokand open therelevant videos,then commentonthe relevant content

←Q Ms

HAZI

20:17 @

HATER:YOURPERFUMES

B.T.WTHEY'RE FROM ARABIA

B.T.WTHEY'RE FROM ARABIA

LOOKWATEREDDOWN

马斯克的惊人论断

# User Instruction

在小红书中搜索一个机器学习相关的帖子并评论相关内容

413

415

关注

发现

附近

<6小象发

搜索

机器学习

提你患接

321看镇头

中国石油大学北京

O

C机建燃油费

Q机票

地铁10号线

*19O

红书出点

Q机票怎么买最划算

我化昆凌仿妆去看周杰伦演唱会

Q机智的羊耳啡

线遇到个精神病人

2我有学习责妇病

Q机长

全车人都被吓醒

机票盲盒

K3

sC

主当然要戴中国冠

5阿如耶

6劝你多

不止读小红书

发明马克食的人给我出来

机车推荐一万以内

半夜隔新房小情倡邻居能

不能别调情了

起猛了.

儿箱

学校里的黑人无

19

机械键盘

首页

喝分享AI

立关注

分享A

明分享A

?

盗即关注

机器学习十大经典算法

机器学习十大经典算法

机器学习十大经典算法

机器学习十大经典算法

前法解析

九大回归算法解析

学路动

如何自己周刷完

机器学习十大经典算

机器学习十大经兵

全解：目机器学习

习有银多构

拿常有用的帖子，谢谢分享！

机器学习十大经典算法，全解：机器学习有很多种

机器学习十大经典算法，全解！川机器学习

存下吧.比

《机器习》

相好太多了！

比段团通，今天给大家整理出了10个最经典算法，

算法，对于算法

说点什么

21622410527

?

鱼

比较图难，今天给大家整理出了10个最经典算法，

算法，对于算法小白来说，全部都学习的话，可能会

学习十大算法全

九大园归算法分析

#User Instruction

根据帖子内容评论微博首页中的一个帖子

11:42

11:44

图

11:46

11:42

关注·推荐

关注

推荐

福

4月24日星期三三月十六丨*23°℃

航天民真

24

区余承东和比亚迪汽车等关注了

#

华为终端

吴员

人再车手2022年原星计划排主

+关注

321看镇头！看超模随晓雯、赵磊、赵佳丽与前华为

Purna70#Pro+一起，以先锋角度，解锁时尚态度，华

小书

为P70#

小虹书

二违一，你选谁？

理想MEGA

G 14

16 白

口

11:56

11:53

11:50

关注

推荐

关注

推存

关注

外汇美金

华为终

华为

华为终瑞业

321看镇头41看超模建绕雯、起磊、赵佳丽与华为

Pura70#iPro+一起，以先锋角度，解锁时尚态度，华

为P70#

321看镇头#！看超模雎晓雯、赵磊、赵佳丽与华为

一起，以先锋角度，解锁时尚态度，华

321看镜头！看超模晓变、赵，赵佳丽与华为

Pura70#Pro+一起，以先锋角度，

锁时尚态度，华

# User Instruction

根据帖子内容评论微博首页中的一个帖子

11:22

11:26

11:28

11:22

Q12306回应

<Q12306回应开售即秒光

搜索

直指

热门

新征程

=

bi热损

完整榜单）

4月25日星期四三月十七120°℃

悔辱袁隆平院士的网.

外媒曝卡截图园应

王者诸葛亮马超大改

社保缴15年就能坐等..

25

原神仆人是伤害天花.

坎公骑冠剑3岁啦

小红书

目历

的橘猫，数天后从废墙中走.

取息！台湾女子地震时舍命款

搜索历史

周鸿祎探馆北京..

领克07EM-P预售.

原神仆人数值怪

自

Mobile-Agent

无名的人

小米销量

四肌

忠佐

节奏大师

徐静雨

1:37

搜索发现

@ rcFPV

19万点赞|盟算

维络

1小时前更新

时前更新

小自测评·1小时前更新

阿维塔11

城市智驾大PK！

单走

冲前更新

6GHz-23小时前更新

无名的人

华为·11小时前更新

手机管家

应用市场

2:22

12306候补成功率

12306

123b

应开售即秒光

万点货|紧屏

11:33

11:32

11:30

<a雷军

搜索

<a雷军

搜索

<Q雷军

综合

番剧

直播（99+)

(+66)里

影视

图

综合

番别

直播(99+)

(+66)田

影视

雷军

默认排序

新发布

摄放多

弹幕多

默认排序

新发布

播放多

弹幕多

筛选

雷军直播

are you ok

名场面

are you ok

鬼图

名场面

雷军are you ok

雷军

66个视质

雷军

间沟伟约架

事长CEO

雷军演讲

雷军发布会

【需军】我们对造车

怀敬提之心

的能姆美保时捷和...

【雷军】小米SU7真

【雷军】小米SU7开

售在即，此刻最担.

【需军】我们对造车

调怀敬提之心，

的

【雷军]小米

【雷军】小米SU7开

售在即，此刻最担..

雷军周湾祎

3月28日

3月27日

查看全部66个视频）

3月26日

3月28日

查看全部66个视频>

3月26日

雷军金凡

需军：关于AreYou OK，我

雷军：关于Are You OK，我

雷军早期直播

的看法是这样的。

的看法是这样的。

雷军约架

合营军等联合创作

需军 等联合创作

小来公司：雷总，真的不是我们干的（您看我明天进公司先迈

哪条腿比较安全？）带话题需军演讲国观季国

哪条腿比较安全？）

是人民的企业家吧

泪目这就是人民的企业家吧

关礼物，开口全是百，

直疆关礼物，开口全是百

16.27万 - 4月18日

视花：整个视频我购得都是非常富欢的话，没有什么科技方面

视施：整个视频我购得都是非常富欢的话，没有什么科技方面

为本人不太懂科技，用手机电脑什么的也就是

为本人不太懂科技，用手机电脑什么的也就是

System

You are a helpful AI mobile phone operating assistant. You need to help me operate the phone to complete the user's instruction.

User

### Background ###

This image is a phone screenshot. Its width is {Lateral resolution} pixels and its height is {Vertical resolution} pixels. The user's instruction is: {User's instruction}.

### Screenshot information ###

In order to help you better perceive the content in this screenshot, we extract some information on the current screenshot through system files. This information consists of two parts: coordinates; content. The format of the coordinates is [x, y], x is the pixel from left to right and y is the pixel from top to bottom; the content is a text or an icon description respectively. The information is as follow:

(x1, y1); text or icon: text content or icon description

......

### Keyboard status ###

We extract the keyboard status of the current screenshot and it is whether the keyboard of the current screenshot is activated.

The keyboard status is as follow:

The keyboard has not been activated and you can't type. or The keyboard has been activated and you can type.

### Hint ###

There are hints to help you complete the user's instructions. The hints are as follow:

If you want to tap an icon of an app, use the action "Open app"

### History operations ###

Before reaching this page, some operations have been completed. You need to refer to the completed operations to decide the next operation. These operations are as follow:

Step-1: [Operation thought: {operation thought 1}; Operation action: {operation 1}]

......

### Progress ###

After completing the history operations, you have the following thoughts about the progress of user's instruction completion:

Completed contents:

{Task progress from planning agent}

### Response requirements ###

Now you need to combine all of the above to perform just one action on the current page. You must choose one of the six actions below:

Open app (app name): If the current page is desktop, you can use this action to open the app named "app name" on the desktop.

Tap (x, y): Tap the position (x, y) in current page.

Swipe (x1, y1), (x2, y2): Swipe from position (x1, y1) to position (x2, y2).

Unable to Type. You cannot use the action "Type" because the keyboard has not been activated. If you want to type, please first activate the keyboard by tapping on the input box on the screen. or Type (text): Type the "text" in the input box.

Home: Return to home page.

Stop: If you think all the requirements of user's instruction have been completed and no further operation is required, you can choose this action to terminate the operation process.

### Output format ###

Your output consists of the following three parts:

### Thought ###

Think about the requirements that have been completed in previous operations and the requirements that need to be completed in the next one operation.

### Action ###

You can only choose one from the six actions above. Make sure that the coordinates or text in the "()". ### Operation ###

Please generate a brief natural language description for the operation in Action based on your Thought.

System

You are a helpful AI mobile phone operating assistant.

User

These images are two phone screenshots before and after an operation. Their widths are {Lateral resolution} pixels and their heights are {Vertical resolution} pixels.

In order to help you better perceive the content in this screenshot, we extract some information on the current screenshot through system files. The information consists of two parts, consisting of format: coordinates; content. The format of the coordinates is (x, y), x is the pixel from left to right and y is the pixel from top to bottom; the content is a text or an icon description respectively The keyboard status is whether the keyboard of the current page is activated.

### Before the current operation ###

Screenshot information:

(x1, y1); text or icon: text content or icon description

......

Keyboard status:

The keyboard has not been activated. or The keyboard has been activated.

### After the current operation ###

Screenshot information:

(x1, y1); text or icon: text content or icon description

......

Keyboard status:

The keyboard has not been activated. or The keyboard has been activated.

### Current operation ###

The user's instruction is: {User's instruction}. You also need to note the following requirements: If you want to tap an icon of an app, use the action "Open app". In the process of completing the requirements of instruction, an operation is performed on the phone. Below are the details of this operation:

Operation thought: {Last operation thought}

Operation action: {Last operation}

### Response requirements ###

Now you need to output the following content based on the screenshots before and after the current operation:

Whether the result of the "Operation action" meets your expectation of "Operation thought"?

A: The result of the "Operation action" meets my expectation of "Operation thought".

B: The "Operation action" results in a wrong page and I need to return to the previous page.

C: The "Operation action" produces no changes.

### Output format ###

Your output format is: ### Thought ### Your thought about the question ### Answer ###

A or B or C

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction clearly state the paper's contributions and scope.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations in the Appendix.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The focus of our work is to explore the application research of the multi-modal large models based agents in special fields. Thus no theoretical results are involved.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have provided necessary information for reproduction in Section 4.2.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We use the ocr model in modelscope and we provide the code address of this model.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We have provided the training and test details necessary to understand the results in Section 4.2.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We have provided the compute workers details in Appendix.

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conforms with the NeurIPS Code of Ethics in every respect.

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: The paper prioritizes discussing technical methodologies and results rather than societal impacts.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper exploring mobile device operation assistant poses no such risks.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have properly credited the used assets. All the datasets introduced in the paper are available for free to researchers for non-commercial use.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: Our code and relevant assets including usage details are provided in the submission.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research within human subjects.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Peer reviews:
comments: SUMMARY:
This paper proposes a multi-agent architecture for mobile device operation, Mobile-Agent-v2. Mobile-Agent-v2 includes three agents : planning agent, decision agent and reflection agent. To retain focus content, this paper designs a memory unit to record task-related focus content. The planning agent generates task progress based on the history operations, the reflection agent corrects erroneous operations and the decision agent outputs operations. Extensive experiments conducted on various operating systems, language environments, and applications shows that MobileAgent-v2 achieves significant performance improvements compared to single-agent architecture of Mobile-Agent.

STRENGTHS:
1. This paper introduces a multi-agent architecture Mobile-Agent-v2, which can alleviate various navigating difficulties inherent in the single-agent framework for mobile device operation tasks.
2. This paper designs a memory unit and reflection agent, which can avoid the loss of focus content navigating and reflection capability.
3. Experimental results demonstrate that Mobile-Agent-v2 achieves significant performance improvements.

WEAKNESSES:
1. Insufficient baselines. More powerful baselines should be considered, such as AppAgent [1] and CogAgent [2].
2. This paper lack description of how the decision agent retrieves the memory unit. In addition,  the content or format of the memory unit storage is not included.

[1] AppAgent: Multimodal Agents as Smartphone Users. 2023.

[2] CogAgent: A Visual Language Model for GUI Agents. 2023.

QUESTIONS:
1. Can the Mobile-Agent-v2 be generalized to IOS systems?
2. How operations knowledge is injected into the model?

LIMITATIONS:
The authors describe limitations in the paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-11

comments: SUMMARY:
This paper proposes mobile-agent-v2, a multi-agent framework for mobile device operation. The framework includes a planning agent, decision agent, reflection agent and memory unit. The multi-agent framework exhibits advantages in reducing errors in long-horizon tasks and achieves better results than single-agent framework on the evaluations designed in the aper.

STRENGTHS:
1. Multi-agent framework for mobile device operation is novel in the literature, improving the overall performance of single agent framework, with the effectiveness of each component verified.

WEAKNESSES:
1 The paper lacks a comparison of the evaluation benchmark with previous works such as AppAgent and works mentioned in Sec. 2.2. This makes it difficult to evaluate the overall effectiveness of the framework in comparison with previous works.
2. The evaluation benchmark is relatively small and simple, and how they are evaluated is not very clear. Does the evaluation relies on human evaluator to check the success of each component in the trajectory?

QUESTIONS:
1. See weakness, the evaluation details needs further clarification.

LIMITATIONS:
1. This paper relies on commercial models such as GPT-4 and GPT-4V to complete the tasks. The cost of completing a task is uncaculated or discussed in the paper, but is expected to be more cost than the single agent framework.
2. The efficiency and accuracy trade-off. After introducing mutli-agent framework, the inference time should be much slower.

--------
After the rebuttal, the reviewer think the heavy reliance on commercial models and the time and token cost should be discussed in the revision.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces an agentic framework, Mobile-Agent-v2, designed to address the challenges of planning and sequential function/tool invocation for Large Language Models (LLMs) in mobile operation scenarios. Mobile-Agent-v2 comprises three agents: a planner, an actioner, and a reflector, which jointly enhance the performance of LLMs in mobile operation tasks. Experimental results demonstrate the advantage of Mobile-Agent-v2 over the single-agent framework: Model-Agent, across various real-world applications.\n\nSTRENGTHS:\n- It is well-written and the structure is clear.\n- Its studied problem of how to design agentic workflow for LLM to improve planning and sequential tool calling is beneficial to the community.\n- The experimental results on real-world mobile apps look promising.\n\nWEAKNESSES:\n- The technical contribution seems limited. Its proposed framework of using planner, actioner and reflector for agentic workflow is a common practice.\n- The evaluation misses some important ablation studies.\n\nQUESTIONS:\nThis paper proposes an agentic workflow, comprising a planner, actioner, and reflector, to enhance the planning and sequential tool-calling capabilities of Large Language Models (LLMs) in mobile operations. The motivation behind this approach is sound, and the experimental results appear promising. However, I have several concerns primarily regarding the technical novelty and the insights derived from the experiments.\n\n- The method of decomposing the responsibilities of a single agent into multiple components—planner, actioner, and reflector—is not novel. This approach has been previously proposed by [1] and is widely used in other contexts. Therefore, it is essential to clarify the key challenges and new insights associated with applying this framework to mobile operation scenarios. Without such clarification, it appears that the study merely leverages an existing method and applies it to a specific scenario without significant innovation.\n\n- The experimental results across various mobile applications are promising. However, the sequence length and type are critical factors that warrant detailed discussion. While some analysis is provided, the paper lacks comprehensive results regarding task completion accuracy when sequence length and type vary. For instance, it would be beneficial to understand, across different apps, which types of operations are more successfully completed with different tool sequences. These ablation studies are necessary to offer deeper insights.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors present Moble-Agent-v2, a  multi-agent architectruure deisgned to assist mobile device operation. Mobile-Agent-v2 comprises three agents: planning agent, decision agent, and reflection agent. Frist, planning agent summarizes the task progress based on the operation hsistories. Then, the decision agent generates next operation to be executed. Finally, the reflextion agent classifies whether the excuted operation as correct, errorneous, or ineffective. For evaluation, the authors conduct experiments on real mobile devices in various mobile applications.\n\nSTRENGTHS:\n- The application of multi-modal LLMs in mobile operation is intriguing\n- The framework that uses multiple agents demnostrates superior perfornamce compared to the single-agent framework\n- The paper is easy to follow and evaluation is conducted on the real mobile devices\n\nWEAKNESSES:\n- The primary concern is the novelty of the paper. While using multiple agents for mobile operations appears novel, the complexity of tasks used in the evaluation suggests that the implementation of each component within Mobile-Agent-V2 is relatively simple. It would be benifical if the authors evalute the proposed framework in diverse user scenarios (refer to the limitation section).\n- It appears that the authors assume that LLMs possess the knowledge about mobile applications. For example, if the user instruction is \"turn on dark mode,\" how does the LLM determine which setting (general setting, display setting) the agent should execute? Because, sometimes even for the human need the trial and error to execute certain task.  Without grounding the available execution in the given context, the proposed method might require multiple trial and error until it successfully completes the task.\n- Regarding to the upper question, it would be better if the authors provide how many interactions are required to complete a  task compared to the optimal execution.\n- It seems to be neither erroneous nor ineffective operations are recorded in the operation history. If this is the case, isn’t there a potentional risk that the decision agent might repeat the same operation? Woudn’t it be better to use these operation histories to prevent the LLM from repeating erroneous or ineffective executions?\n- There seems some potential risks associated with inaccuracies in every agents within the proposed framework, especially since it utilizes LLMs. For instance, the planning agent might inaccurately summarize operation histories, the decision agent could misunderstand visual inputs, or it might generate incorrect executions. Can Mobile-Agent-v2 automatically identify which component is responsible for an error? or could the authors provide more statistical information about this?\n\nQUESTIONS:\n- Can the authors provide more information about knowledge injection? It would be helpful if the authors included examples to illustrate this process.\n\nLIMITATIONS:\n- Including some discussion on the scope limitations of Mobile-Agent-v2 would be beneficial. For instance, what happens if the required application is missing—can Mobile-Agent-v2 suggest an alternative method? Additionally, for ambiguous instructions, is Mobile-Agent-v2 capable of resolving these by seeking clarification from the user?", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper titled \"Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration\" presents a multi-agent architecture designed to address the challenges of mobile device operation tasks, specifically focusing on task progress navigation and focus content navigation. The proposed system includes three agents: a planning agent, a decision agent, and a reflection agent. The architecture aims to improve task completion rates and operational efficiency compared to existing single-agent architectures.\n\nSTRENGTHS:\n**Innovative Multi-Agent Architecture:** The introduction of a multi-agent system to handle different aspects of mobile device operation is a novel approach. It effectively distributes the workload among specialized agents, which likely contributes to improved performance.\n\n**Comprehensive Experimental Evaluation:** The paper includes a detailed evaluation across different operating systems, language environments, and applications, providing robust evidence of the system's effectiveness.\n\n**Significant Performance Improvements:** Experimental results show that Mobile-Agent-v2 achieves over a 30% improvement in task completion compared to a single-agent architecture, demonstrating the practical benefits of the proposed system.\n\nWEAKNESSES:\n**Limited Discussion on Scalability:** There is insufficient discussion on the scalability of the proposed system. The paper does not address potential challenges when scaling the architecture to more complex tasks or larger datasets.\n\n**Potential Overhead of Multi-Agent Coordination:** While the multi-agent approach shows improved performance, the paper does not discuss the potential computational overhead and complexity introduced by coordinating multiple agents, which could be a significant drawback in resource-constrained environments.\n\nQUESTIONS:\n- How does the proposed system handle the increased computational overhead associated with running multiple agents concurrently on resource-constrained mobile devices?\n- Can the system be extended to support more complex multi-app operations that require intricate coordination between different agents?\n- How does the reflection agent determine the appropriate corrective measures for erroneous operations, and what is the success rate of these corrections in practice?\n\nLIMITATIONS:\nsee weakness", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-17"}