Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration

Authors: junyang  wang
haiyang  xu
haitao  jia
xi  zhang
ming  yan
weizhou  shen
ji  zhang
fei  huang
jitao  sang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 04:30:40

Abstract: Mobile device operation tasks are increasingly becoming a popular multi-modal AI application scenario. Current Multi-modal Large Language Models (MLLMs), constrained by their training data, lack the capability to function effectively as operation assistants. Instead, MLLM-based agents, which enhance capabilities through tool invocation, are gradually being applied to this scenario. However, the two major navigation challenges in mobile device operation tasks — task progress navigation and focus content navigation — are difficult to effectively solve under the single-agent architecture of existing work. This is due to the overly long token sequences and the interleaved text-image data format, which limit performance. To address these navigation challenges effectively, we propose Mobile-Agent-v2, a multi-agent architecture for mobile device operation assistance. The architecture comprises three agents: planning agent, decision agent, and reflection agent. The planning agent condenses lengthy, interleaved image-text history operations and screens summaries into a pure-text task progress, which is then passed on to the decision agent. This reduction in context length makes it easier for decision agent to navigate the task progress. To retain focus content, we design a memory unit that updates with task progress by decision agent. Additionally, to correct erroneous operations, the reflection agent observes the outcomes of each operation and handles any mistake accordingly. Experimental results indicate that Mobile-Agent-v2 achieves over a 30% improvement in task completion compared to the single-agent architecture of Mobile-Agent. The code is open-sourced at https://github.com/X-PLUG/MobileAgent.

Main paper: 
 introduction: 
 Multi-modal Large Language Models (MLLMs), represented by GPT-4v OpenAI (2023) , have demonstrated outstanding capabilities in various domains Bai et al. (2023) ; Liu et al. (2023c,b) ; Dai et al. (2023) ; Zhu et al. (2023) ; Chen et al. (2023) ; Ye et al. (2023a,b) ; Wang et al. (2023c) ; Hu et al. (2023 Hu et al. ( , 2024)) ; Zhang et al. (2024b) . With the rapid development of agents based on Large Language Models (LLMs) Zhao et al. (2024) ; Liu et al. (2023f) ; Talebirad and Nadiri (2023) ; Zhang et al. (2023b) ; Wu et al. (2023) ; Shen et al. (2024) ; Li et al. (2023a) , MLLM-based agents, which can overcome the limitations of MLLMs in specific application scenarios by various visual perception tools, have become a focal point of research attention Liu et al. (2023d) .Figure 1 : Mobile device operation tasks require navigating focus content and task progress from history operation sequences, where the focus content comes from previous screens. As the number of operations increases, the length of the input sequences grows, making it extremely challenging for a single-agent architecture to manage these two types of navigation effectively.Automated operations on mobile devices, as a practical multi-modal application scenario, are emerging as a major technological revolution in AI smartphone development Yao et al. (2022) ; Deng et al. (2023) ; Gur et al. (2024) ; Zheng et al. (2024) ; Zhang et al. (2023a) ; Wang et al. (2024) ; Chen and Li (2024a,b,c) ; Zhang et al. (2024a) ; Wen et al. (2023) ; Cheng et al. (2024) . However, due to the limited screen recognition, operation, and location capabilities, existing MLLMs face challenges in this scenario. To address this, existing work leverages MLLM-based agent architecture to endow MLLMs with various capabilities for perceiving and operating mobile device UI. AppAgent Zhang et al. (2023a) tackles the limitation of MLLMs in localization by extracting clickable positions from device XML files. However, the reliance on UI files limits the applicability of this method to other platforms and devices. To eliminate the dependency on underlying UI files, Mobile-Agent Wang et al. (2024) proposes a solution for localization through visual perception tools. It perceives the screen through an MLLM and generates operations, locating their positions by visual perception tools.Mobile device operation tasks involve multi-step sequential processing. The operator needs to perform a series of continuous operations on the device starting from the initial screen until the instructions are fully executed. There are two main challenges in this process. First, to plan the operation intent, the operator needs to navigate the current task progress from the history operations. Second, some operations may require task-relevant information in the history screens, for example, writing sports news in Figure 1 requires using the match results queried earlier. We refer to this important information as the focus content. The focus content also needs to be navigated out from the history screens. However, as the task progresses, the lengthy history of interleaved image and text history operations and screens as input can significantly reduce the effectiveness of navigation in a single-agent architecture, as shown in Figure 1 .In this paper, we propose Mobile-Agent-v2, a mobile device operation assistant with effective navigation via multi-agent collaboration. Mobile-Agent-v2 has three specialized agent roles: planning agent, decision agent, and reflection agent. The planning agent needs to generate a task progress based on the history operations. To save the focus content from the history screens, we design a memory unit to record task-related focus content. This unit will be observed by the decision agent when generating an operation, simultaneously checking if there is any focus content on the screen and updating it to the memory. Since the decision agent cannot observe the previous screen to reflect, we design the reflection agent to observe the changes in the screen before and after the decision agent's operation and determine whether the operation meets the expectations. If it finds that the operation does not meet expectations, it will take appropriate measures to re-execute the operation. The entire process is illustrated in Figure 3 . The three agent roles work respectively in the progress, decision, and reflection stages, collaborating to alleviate the difficulty of navigating.Our summarized contributions are as follows:• We propose a multi-agent architecture Mobile-Agent-v2 to alleviate various navigating difficulties inherent in the single-agent framework for mobile device operation tasks. We design a planning agent to generate task progress based on the history operations, ensuring effective operation generation by the decision agent. • To avoid the loss of focus content navigating and reflection capability, we design both a memory unit and a reflection agent. The memory unit is updated by the decision agent with focus content. The reflection agent assesses whether the decision agent's operation meets expectations and generates appropriate remedial measures if expectations are not met. 2023) designed a GUI agent based on pre-trained vision-language models. Chen and Li (2024a,b,c) propose small-scale client-side models for deployment on actual devices. Zhang et al. (2024a) proposed a UI multi-agent framework tailored for the Windows operating system. Despite the significant performance improvements achieved by multi-agent architectures in many tasks, currently, there is no work that employs multi-agent architectures in mobile device operation tasks. To address the 3 Mobile-Agent-v2In this section, we will provide a detailed overview of the architecture of Mobile-Agent-v2. The operation of Mobile-Agent-v2 is iterative, and its process is depicted in Figure 2 . Mobile-Agent-v2 has three specialized agent roles: planning agent, decision agent, and reflection agent. We also design the visual perception module and memory unit to enhance the agent's screen recognition capability and the capability to navigate focus content from history. Firstly, the planning agent updates the task progress, allowing the decision agent to navigate the progress of the current task. The decision agent then operates based on the current task progress, current screen state, and the reflection (if the last operation is erroneous). Subsequently, the reflection agent observes the screens before and after the operation to determine if the operation meets expectations. 
visual perception module: 
 Screen recognition remains challenging even for state-of-the-art MLLMs when processed end-toend. Therefore, we have incorporated a visual perception module to enhance the screen recognition capability. In this module, we utilize three tools: text recognition tool, icon recognition tool, and icon description. Inputting a screenshot into this module will ultimately yield the text and icon information present on the screen, along with their respective coordinates. This process is represented by the following formula:EQUATIONwhere P t represents the perception result of the screen in the t-th iteration.Due to the task progress generated by the planning agent being in textual form, the navigation of focus content from history screens is still challenging. To address this issue, we design a memory unit to store the focus content related to the current task from history screens. The memory unit serves as a short-term memory module that is updated as the task progresses. The memory unit is crucial for scenarios involving multiple apps. For instance, as shown in Figure 3 , weather information observed by the decision agent will be utilized in subsequent operations. At this point, the information related to the weather app's page will be updated in the memory unit.We aim to reduce the reliance on lengthy history operations during decision-making by employing a separate agent. We observe that although each round of operation occurs on different pages and is different, often the goals of multiple operations are the same. For example, in the example illustrated in Figure 1 , the first four operations are all about searching for match results. Therefore, we design a planning agent to summarize the history operations and track task progress.We define the operation generated by the decision agent at the t-th iteration as O t . Before the decision agent makes a decision, the planning agent observes the decision agent's operation O t-1 from the last iteration and updates the task progress T P t-1 to T P t . The task progress includes the sub-tasks that have already been completed. After generating the task progress, the planning agent passes it to the decision agent. This aids the decision agent in considering the content of tasks that have not yet been completed, thereby facilitating the generation of the next operation. As shown in Figure 3 , the planning agent's inputs consist of four parts: user instruction Ins, the focus content F C t in memory unit, the previous operation O t-1 , and the previous task progress T P t-1 . Based on above information, the planning agent generates the T P t . This process is represented by the following formula:EQUATIONwhere the P A represents the LLM of the planning agent.The decision agent operates during the decision stage, generating operations O and implementing them on the device, while also being responsible for updating the focus content F C in memory unit. This process is illustrated in the Decision Stage shown in Figure 3 and represented by the following formula:EQUATIONwhere the DA represents the MLLM of the decision agent and the R t represents the reflection result from reflection agent.Operation Space. To reduce the complexity of operations, we design an operation space and restricted the decision agent to selecting operations only from within this space. For operations with higher degrees of freedom, such as tapping and swiping, we incorporate an additional parameter space to locate or handle specific content. Below is a detailed description of the operation space:• Open app (app name). If the current page is the home page, this operation can be used to open the app named "app name".• Tap (x, y). This operation is used to tap on the position with coordinates (x, y).• Swipe (x1, y1), (x2, y2). This operation is used to swipe from the position with coordinates (x1, y1) to the position with coordinates (x2, y2).• Type (text). If the current keyboard is in an active state, this operation can be used to input the content of "text" in the input box.• Home. This operation is used to return to the home page from any page.• Stop. If the decision agent thinks that all requirements have been fulfilled, it can use this operation to terminate the entire operation process.Memory Unit Update. As each operation made by the decision agent is highly task-relevant and based on the visual perception results of the current page, it is well-suited to observe task-related focus content within the screen pages. Accordingly, we have endowed the decision agent with the ability to update the memory unit. When making decisions, the decision agent is prompted to observe whether there is task-related focus content within the current screen page. If such information is observed, the decision agent updates it in memory for reference in subsequent decisions. This process is represented by the following formula:EQUATIONEven with the visual perception module, Mobile-Agent-v2 may still generate unexpected operations.In some specific scenarios, MLLMs may even produce severe hallucinations where the RA represents the MLLM of the reflection agent.As shown in Figure 3 , the reflection agent generates three types of reflection results after operation execution: erroneous operation, ineffective operation, and correct operation. The following will describe these three reflection results:• Erroneous operation refers to an operation that leads the device to enter a page unrelated to the task. For example, the agent intends to chat with contact A in a messaging app, but it accidentally opens the chat page of contact B instead. • Ineffective operation refers to an operation that does not result in any changes to the current page. For example, the agent intends to tap on an icon, but it taps on the blank space next to the icon instead. • Correct operation refers to an operation that meets the decision agent's expectations and serves as a step towards fulfilling the requirements of the user instruction.If erroneous operation, the page will revert to the state before the operation. If ineffective operation, the page will remain in its current state. Neither erroneous nor ineffective operations are recorded in the operation history to prevent the agent from following these operations. If correct operation, the operation will be updated in the operation history, and the page will be updated to the current state. MLLMs. For the planning agent, as it does not require screen perception, we utilize the text-only GPT-4 OpenAI (2023) . For the decision agent and reflection agent, we employ GPT-4V OpenAI (2023). All calls are made through the official API method provided by the developers. 
evaluation: 
 Evaluation Method. To evaluate the performance of Mobile-Agent-v2 on real mobile devices, we employed a dynamic evaluation method. This evaluation method requires the operation tool to implement the agent's operations in real time on an actual device. We use two mobile operation systems, Harmony OS and Android OS, to assess capabilities in non-English and English scenarios, respectively. We used Android Debug Bridge (ADB) as the tool for operating mobile devices. ADB can simulate all operations of Mobile-Agent-v2 in the operation space. In both scenarios, we select 5 system apps and 5 popular external apps for evaluation. For each app, we devise two basic instructions and two advanced instructions. Basic instructions are relatively simple operations with clear instructions within the app interface, while advanced instructions require a certain level of experience with app operations to complete. Additionally, to evaluate multi-app operation capability, we design two basic instructions and two advanced instructions involving multiple apps. In total, there were 88 instructions for non-English and English scenarios, comprising 40 instructions for system apps, 40 instructions for external apps, and 8 instructions for multi-app operations. 5 The apps and instructions used for evaluation in non-English and English scenarios are presented in the appendix.Metrics. We design the following four metrics for dynamic evaluation:• Success Rate (SR): When all the requirements of a user instruction are fulfilled, the agent is considered to have successfully executed this instruction. The success rate refers to the proportion of user instructions that are successfully executed.• Completion Rate (CR): Although some challenging instructions may not be successfully executed, the correct operations performed by the agent are still noteworthy. The completion rate refers to the proportion of correct steps out of the ground truth operations.• Decision Accuracy (DA): This metric reflects the accuracy of the decision by the decision agent. It is the proportion of correct decisions out of all decisions.• Reflection Accuracy (RA): This metric reflects the accuracy of reflection by the reflection agent. It is the proportion of correct reflections out of all reflections.Implementation Details. We use Mobile-Agent as the baseline. Mobile-Agent is a single-agent architecture based on GPT-4V end-to-end screen recognition. We fix the seed for GPT-4V invocation and set the temperature to 0 to avoid randomness. In addition to Mobile-Agent-v2, we further introduce the scenario of knowledge injection. This involves providing the agent with some operation hints in addition to the user instructions to aid the agent. It's worth noting that we only injected knowledge for instructions that Mobile-Agent-v2 couldn't accomplish. For instructions that could be completed without additional assistance, we keep the input unchanged.Evaluation on Task Completion. Table 1 and 2 respectively illustrate the performance of Mobile-Agent-v2 in non-English and English scenarios. Compared to Mobile-Agent, Mobile-Agent-v2 exhibits significant improvements in both basic and advanced instructions. With the multi-agent architecture, even in highly challenging advanced instructions, the success rate can still reach 55%, compared to only 20% with Mobile-Agent. Even in the English scenario, Mobile-Agent-v2 still achieves a significant performance improvement. Although Mobile-Agent performs better in English scenarios compared to Chinese scenarios, Mobile-Agent-v2 still achieves an average improvement of 27% in success rate.Evaluation on Reflection Capability. In the case of knowledge injection, even though the decision accuracy does not reach 100%, the completion rate can still reach 100%. This indicates that even with knowledge injection, Mobile-Agent-v2 still makes erroneous decisions. Errors in decision-making are difficult to avoid, even for humans. Therefore, the importance of the reflection agent is highlighted. Evaluation on App Type. From all metrics, it can be observed that the performance of all methods on system apps exceeds that of external apps. From the results of multiple apps, it can be seen that Mobile-Agent-v2 achieves improvements of 37.5% and 44.2% in SR and CR, respectively, compared to Mobile-Agent. Compared to single-app tasks, multi-app tasks rely more on the retrieval of history operations and focus content. The significant performance improvement indicates that the multi-agent architecture and memory unit of Mobile-Agent-v2 play an important role.Evaluation on Operation Knowledge Injection. From the results of knowledge injection in Table 1 and 2, it can be observed that operation knowledge can effectively enhance the performance of Mobile-Agent-v2, which suggests that manually injected operation knowledge can mitigate the limitations of an agent's operation capability. This finding implies that knowledge injection can broaden the application scenarios of Mobile-Agent-v2 because even complex tasks can be guided by manually written operation tutorials to instruct agents. This finding may offer new insights for automated script testing on mobile devices and suggests that to enhance the operation capabilities of MLLMs to their limits, automating the generation of high-quality operation knowledge can further improve the performance of Mobile-Agent-v2. Moreover, the success brought about by knowledge injection also opens up new avenues for future mobile app testing. Existing mobile app testing solutions are still limited to manual script writing, which restricts the universality of testing and raises 7 58.9 29.5 43.8 42.6 77.3 83.6 84.0 45.5 72.3 69.8 86.4 89.2 85.7 54.5 75.9 72.4 88.6 93.9 89.4 61.4 82.1 80.3 Table 4: The results of the ablation study on planning agent, reflection agent, and memory unit.the threshold for users. To address the aforementioned issues, one can inject natural language testing procedures into Mobile-Agent-v2. After injecting accurate testing procedures, the system can operate normally regardless of changes in the size or color of the mobile interface. Additionally, language descriptions eliminate the need for a knowledge base in script writing.Evaluation on MLLMs. In Table 3 , we evaluate the performance of different MLLMs within the Mobile-Agent-v2 framework. Since some models are not well-suited for handling sequential inputs, we selected specific instructions and modified each step to function as a single-step task. Therefore, we only evaluate the SR (which is the same as DA). We also evaluate the direct use of GPT-4V, bypassing the agent architecture for end-to-end operation. The results indicate that using GPT-4V directly as a mobile device operation assistant is almost infeasible. GPT-4V combined with the agent architecture remains the most effective configuration for operational capabilities.We conduct ablation studies on Mobile-Agent-v2, including the planning agent, reflection agent, and memory unit. From the results in Table 4 , it is evident that the impact of the planning agent on the overall framework is the most significant. This further demonstrates the challenging nature of navigation in long sequences for current MLLMs. Additionally, performance declines are observed after removing the reflection agent and memory unit. The reflection agent is essential for correcting erroneous operations. It enables the decision agent to avoid operating on incorrect pages or getting stuck in loops of invalid operations. The memory unit is crucial for successful execution in multi-app scenarios. Even in scenarios involving multiple sub-tasks, the memory can sometimes record the positions of critical UI elements, aiding better localization for the next sub-task execution.As shown in Figure 4 , we analyze the positions of erroneous or ineffective operations within failed instructions in English scenarios, dividing the relative sequence positions into three equal parts. The results indicate that in Mobile-Agent, such errors or ineffective operations predominantly occur in the later stages of the tasks. In contrast, Mobile-Agent-v2 does not exhibit any obvious pattern. This indicates that the multi-agent architecture is better equipped to handle the challenges posed by long sequences in UI operation tasks. Figure 5 (a) illustrates a complete operational process of Mobile-Agent-v2. Under the planning of the planning agent, the decision agent can navigate the task progress correctly in the case of single-image input. Meanwhile, the memory unit accurately stores the chat content needed for the task and, when required for search purposes, the decision agent can effectively navigate to it. Figure 5 (b) illustrates an example of correcting ineffective operations through reflection. After the failure of the previous operation, the reflection agent promptly detects the error and communicates the reflection result to the decision agent. Based on this, the decision agent rethinks and implements the correct operation. 
conclusion: 
 Existing single-agent architectures for mobile device operation assistants experience a significant reduction in navigation effectiveness when dealing with long sequences of interleaved text and images, thereby limiting their performance. To address this issue, in this paper, we propose Mobile-Agent-v2, a mobile device operation assistant with efficient navigation via multi-agent collaboration. We address the aforementioned navigating challenges through the planning agent and memory unit, respectively. Additionally, we design reflectors to ensure the smooth progress of tasks. Experimental results demonstrate that Mobile-Agent-v2 achieves significant performance improvements compared to the single-agent Mobile-Agent. Furthermore, we find that performance can be further enhanced through the injection of manual operation knowledge, providing new directions for future work. 
acknowledgements: 
 This work is supported by the National Key R&D Program of China (No. 2023YFC3310700).A Appendix / supplemental material A.1 Evaluation Application and Instruction Table 5 and 6 present the applications and instructions used for dynamic evaluation in both English and non-English scenarios. Basic instructions are relatively simple operations with clear instructions within the app interface, while advanced instructions require a certain level of experience with app operations to complete. 
 
 Appendix: In Table 7, 8, 9 , and 10, we present the system and user prompts for the planning agent, decision agent, and reflection agent. Notably, since the history of operations is empty at the beginning of the task, we display the initial planning and subsequent planning prompts for the planning agent separately in Table 7 and 8. For image inputs, we differentiated based on the task characteristics of each agent. For the planning agent, as navigating historical operations and generating task plans is a purely textual process, no screenshots are required. For the decision agent, we input the screenshot of the current device state. For the reflection agent, we additionally retain the screenshot from the previous operation and input it alongside the current device state screenshot in chronological order.Since some instructions need to be specially tailored for specific mobile applications, we have hidden these details to protect privacy. In the Table 5 and 6 , "xxx" represents the redacted information. For certain celebrities or locations mentioned in the instructions, we retained them as they do not involve privacy concerns. Table 5 : The applications and instructions used for the evaluation on English scenario, where the "xxx" represents the redacted information.https://modelscope.cn/models/iic/cv_convnextTiny_ocr-recognition-document_damo/ summaryhttps://modelscope.cn/models/qwen/Qwen-VL-Chat-Int4/summaryWe did not use Mobile-Eval because the difficulty of this benchmark is relatively low and Mobile-Agent-v2 can achieve an accuracy of 99%. In this work, we have redesigned more challenging evaluation tasks.Search the result for today's Thunder game, and then create a note in Notes to write a sport news for this result. 


Decision: This paper describes Mobile-Agent-v2, an MLLM-based multi-agent architecture designed to assist with mobile device operations, incorporating the multi-agent collaboration with three agents: a planning agent, a decision agent, and a reflection agent.

The paper was reviewed by five reviewers with four inclined to suggest for acceptance and one not. The majority of reviewers highlighted the strengths, such that (1) the proposed fmobile-agent-v2 framework is an interesting contribution to the mobile domain, (2) the paper is clearly written and easy-to-follow, and (3) the experiments with real-world mobile apps demonstrate promising and significant performance. One reviewer commented several concerns about limited operational knowledge of existing MLLMs and error-correcting capabilities of the agent, which the authors argued are beyond the scope of current research work. However, since this research addresses the challenges in practical mobile device operation tasks, those concerns could be discussed further in the final version. 

Considering the strengths above and the novel approach toward agentic workflow in the mobile domain, this paper is recommended for acceptance. 
The authors are encouraged to utilize their constructive and detailed discussion with reviewers towards further raising the quality and impact of their work.

Reviews: review: paper_summary: paper_summary: The paper titled "Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration" presents a multi-agent architecture designed to address the challenges of mobile device operation tasks, specifically focusing on task progress navigation and focus content navigation. The proposed system includes three agents: a planning agent, a decision agent, and a reflection agent. The architecture aims to improve task completion rates and operational efficiency compared to existing single-agent architectures.
strength_weakness: strength_weakness: **Limited Discussion on Scalability:** There is insufficient discussion on the scalability of the proposed system. The paper does not address potential challenges when scaling the architecture to more complex tasks or larger datasets.

**Potential Overhead of Multi-Agent Coordination:** While the multi-agent approach shows improved performance, the paper does not discuss the potential computational overhead and complexity introduced by coordinating multiple agents, which could be a significant drawback in resource-constrained environments.
questions: questions: - How does the proposed system handle the increased computational overhead associated with running multiple agents concurrently on resource-constrained mobile devices?
- Can the system be extended to support more complex multi-app operations that require intricate coordination between different agents?
- How does the reflection agent determine the appropriate corrective measures for erroneous operations, and what is the success rate of these corrections in practice?
limitations: limitations: see weakness
paper_summary: paper_summary: In this paper, the authors present Moble-Agent-v2, a  multi-agent architectruure deisgned to assist mobile device operation. Mobile-Agent-v2 comprises three agents: planning agent, decision agent, and reflection agent. Frist, planning agent summarizes the task progress based on the operation hsistories. Then, the decision agent generates next operation to be executed. Finally, the reflextion agent classifies whether the excuted operation as correct, errorneous, or ineffective. For evaluation, the authors conduct experiments on real mobile devices in various mobile applications.
strength_weakness: strength_weakness: - The primary concern is the novelty of the paper. While using multiple agents for mobile operations appears novel, the complexity of tasks used in the evaluation suggests that the implementation of each component within Mobile-Agent-V2 is relatively simple. It would be benifical if the authors evalute the proposed framework in diverse user scenarios (refer to the limitation section).
- It appears that the authors assume that LLMs possess the knowledge about mobile applications. For example, if the user instruction is "turn on dark mode," how does the LLM determine which setting (general setting, display setting) the agent should execute? Because, sometimes even for the human need the trial and error to execute certain task.  Without grounding the available execution in the given context, the proposed method might require multiple trial and error until it successfully completes the task.
- Regarding to the upper question, it would be better if the authors provide how many interactions are required to complete a  task compared to the optimal execution.
- It seems to be neither erroneous nor ineffective operations are recorded in the operation history. If this is the case, isn’t there a potentional risk that the decision agent might repeat the same operation? Woudn’t it be better to use these operation histories to prevent the LLM from repeating erroneous or ineffective executions?
- There seems some potential risks associated with inaccuracies in every agents within the proposed framework, especially since it utilizes LLMs. For instance, the planning agent might inaccurately summarize operation histories, the decision agent could misunderstand visual inputs, or it might generate incorrect executions. Can Mobile-Agent-v2 automatically identify which component is responsible for an error? or could the authors provide more statistical information about this?
questions: questions: - Can the authors provide more information about knowledge injection? It would be helpful if the authors included examples to illustrate this process.
limitations: limitations: - Including some discussion on the scope limitations of Mobile-Agent-v2 would be beneficial. For instance, what happens if the required application is missing—can Mobile-Agent-v2 suggest an alternative method? Additionally, for ambiguous instructions, is Mobile-Agent-v2 capable of resolving these by seeking clarification from the user?
paper_summary: paper_summary: This paper introduces an agentic framework, Mobile-Agent-v2, designed to address the challenges of planning and sequential function/tool invocation for Large Language Models (LLMs) in mobile operation scenarios. Mobile-Agent-v2 comprises three agents: a planner, an actioner, and a reflector, which jointly enhance the performance of LLMs in mobile operation tasks. Experimental results demonstrate the advantage of Mobile-Agent-v2 over the single-agent framework: Model-Agent, across various real-world applications.
strength_weakness: strength_weakness: - The technical contribution seems limited. Its proposed framework of using planner, actioner and reflector for agentic workflow is a common practice.
- The evaluation misses some important ablation studies.
questions: questions: This paper proposes an agentic workflow, comprising a planner, actioner, and reflector, to enhance the planning and sequential tool-calling capabilities of Large Language Models (LLMs) in mobile operations. The motivation behind this approach is sound, and the experimental results appear promising. However, I have several concerns primarily regarding the technical novelty and the insights derived from the experiments.

- The method of decomposing the responsibilities of a single agent into multiple components—planner, actioner, and reflector—is not novel. This approach has been previously proposed by [1] and is widely used in other contexts. Therefore, it is essential to clarify the key challenges and new insights associated with applying this framework to mobile operation scenarios. Without such clarification, it appears that the study merely leverages an existing method and applies it to a specific scenario without significant innovation.

- The experimental results across various mobile applications are promising. However, the sequence length and type are critical factors that warrant detailed discussion. While some analysis is provided, the paper lacks comprehensive results regarding task completion accuracy when sequence length and type vary. For instance, it would be beneficial to understand, across different apps, which types of operations are more successfully completed with different tool sequences. These ablation studies are necessary to offer deeper insights.
limitations: limitations: N/A
paper_summary: paper_summary: This paper proposes mobile-agent-v2, a multi-agent framework for mobile device operation. The framework includes a planning agent, decision agent, reflection agent and memory unit. The multi-agent framework exhibits advantages in reducing errors in long-horizon tasks and achieves better results than single-agent framework on the evaluations designed in the aper.
strength_weakness: strength_weakness: 1 The paper lacks a comparison of the evaluation benchmark with previous works such as AppAgent and works mentioned in Sec. 2.2. This makes it difficult to evaluate the overall effectiveness of the framework in comparison with previous works.
2. The evaluation benchmark is relatively small and simple, and how they are evaluated is not very clear. Does the evaluation relies on human evaluator to check the success of each component in the trajectory?
questions: questions: 1. See weakness, the evaluation details needs further clarification.
limitations: limitations: 1. This paper relies on commercial models such as GPT-4 and GPT-4V to complete the tasks. The cost of completing a task is uncaculated or discussed in the paper, but is expected to be more cost than the single agent framework.
2. The efficiency and accuracy trade-off. After introducing mutli-agent framework, the inference time should be much slower.

--------
After the rebuttal, the reviewer think the heavy reliance on commercial models and the time and token cost should be discussed in the revision.
paper_summary: paper_summary: This paper proposes a multi-agent architecture for mobile device operation, Mobile-Agent-v2. Mobile-Agent-v2 includes three agents : planning agent, decision agent and reflection agent. To retain focus content, this paper designs a memory unit to record task-related focus content. The planning agent generates task progress based on the history operations, the reflection agent corrects erroneous operations and the decision agent outputs operations. Extensive experiments conducted on various operating systems, language environments, and applications shows that MobileAgent-v2 achieves significant performance improvements compared to single-agent architecture of Mobile-Agent.
strength_weakness: strength_weakness: 1. Insufficient baselines. More powerful baselines should be considered, such as AppAgent [1] and CogAgent [2].
2. This paper lack description of how the decision agent retrieves the memory unit. In addition,  the content or format of the memory unit storage is not included.

[1] AppAgent: Multimodal Agents as Smartphone Users. 2023.

[2] CogAgent: A Visual Language Model for GUI Agents. 2023.
questions: questions: 1. Can the Mobile-Agent-v2 be generalized to IOS systems?
2. How operations knowledge is injected into the model?
limitations: limitations: The authors describe limitations in the paper.

Comments: title: Response to the Official Comment by Reviewer Xdcr
Thanks for the author response
Response to the Official Comment by Reviewer cAKE
A few further questions
Response to the Official Comment by Reviewer kbfL
Official Comment by Reviewer 4qiL
Response to the Official Comment by Reviewer 4qiL
Official Comment by Reviewer 4qiL
comment: Thank the authors for their rebuttals. I will raise my score to 5. However, the results on the multi-modal scenarios suggest that the performance of the proposed agentic framework can be satisfactory only when it has been applied on GPT-4V. This may limit the generalization of the proposed agent frameworks. In the next version of this paper, I would like to see whether the gap between applying Mobile-agent-V2 on GPT-4V and on other open-source models, especially on small models (less than 7B) can be narrowed down. This is because small models are more practical choice when deploying LLMs on the mobile device.
Thank you very much for your feedback and the time you dedicated to reviewing our paper. We greatly appreciate your insights and are pleased to hear that the clarifications we provided were helpful in addressing your concerns. If you have any further questions or suggestions, please do not hesitate to reach out. We would be happy to discuss them with you.
Thank you very much for your feedback and the time you dedicated to reviewing our paper. We greatly appreciate your insights and are pleased to hear that the clarifications we provided were helpful in addressing your concerns. If you have any further questions or suggestions, please do not hesitate to reach out. We would be happy to discuss them with you.
Thanks for the clarification.
The reviewer tends to raise the score to 5 since most of the questions are resolved.
However, the concern about it heavy reliance on GPT-4V and the time cost still exists. It is not very straightforward to think about the scenarios that accept the slow time and GPT-4V token cost, which should be added in the discussion or limitation section in the next revision.
### **Question 1: How to determine whether the operation is successful or not?**
### **Response:**
Due to the lack of dynamic evaluation benchmarks in the mobile field, all our success or failure judgments are made through manual evaluation. 

### **Question 2: What is the relationship between the 1500+ evaluation with the 88 instructions?**
### **Response:**
The instructions used in the Mobile-Agent-v2 evaluation require multiple steps to complete, with an average of 7 steps per instruction. For each operation, we manually evaluated the accuracy of planning, decision-making, and reflection. Therefore, there are about 88 x 7 x 3 = 1848 evaluations in total.

### **Question 3: Mobile Agent-v2 seems to increase the overall time to finish the task in comparison with Mobile agent-v1.**
### **Response:**
* First although Mobile-Agent-v2 consumes about 15% more time, it achieves an operation success rate of more than 30%. Both are based on the same MLLM, and Mobile-Agent-v2 can significantly improve its operational capabilities by virtue of its architectural advantages.
* The main reason that currently limits the operation time of Mobile-Agent-v2 is the inference speed of GPT-4. This is not a flaw in the framework itself. As the inference speed of MLLMs continues to accelerate and more advanced inference acceleration methods are used, or the use of local MLLMs, the inference speed will be further shortened. This will also significantly reduce the time-consuming gap between single-agent architecture and multi-agent architecture, while allowing the operation speed to be close to that of humans.
* It is worth noting that although the operation speed of Mobile-Agent is not as fast as that of humans, there are still many practical application scenarios where such operation delays can be accepted. In addition, thanks to the parallel logic between agents, Mobile-Agent is also the fastest architecture that can currently achieve real machine operation. We are currently working on using local models to replace GPT-4 and have achieved initial results. Currently, the operation time per operation can be as low as 10 seconds.

### **Question 4: The time calculation.**
### **Response:**
The time we calculate is the time of the complete operation, which includes the network delay and the communication consumption on the pipeline. If Mobile-Agent-v2 is used in a real device or app, it will not increase the operation delay.
Thank the authors for the detailed response.
However, the reviewer is still confused about whether the successful judgment is done by humans or GPT-4V. If it is by GPT-4V, to what degree is the assessment reliable?
As mentioned in lines 215-216 of the main paper, there are 88 instructions tested in total, with only 8 instructions for multi-app operations, what is the relationship between the so-called 1500+ evaluation with the 88 instructions?

Mobile Agent-v2 seems to increase the overall time to finish the task in comparison with Mobile agent-v1, not to mention it is impractical to use GPT-4V for mobile devices. Further, the time calculation does not consider the information transmission time through the web or cable in real-world applications when using GPT-4V.
Thank you for your continued evaluation and thoughtful feedback. We appreciate your recognition of the significance of configuring an agentic workflow within the mobile application domain.

We understand your concern regarding the practical utility of Mobile-Agent-v2 compared to existing methods. It is worth emphasizing that our approach has demonstrated a substantial improvement in task completion rates and a significant reduction in errors during navigation tasks. These results directly translate to an enhanced user experience, particularly in complex mobile operation scenarios. The ability of Mobile-Agent-v2 to efficiently manage long sequences and interleaved data in real-world applications is a notable advantage over other related methods.

The framework's design is specifically optimized to minimize user intervention and exploration during mobile operations, which is critical for maintaining a smooth and intuitive user experience.

* **Automatically attribute errors.** We agree that automatic error resolution and handling ambiguous tasks are critical aspects. This study is focused on establishing the foundational architecture of Mobile-Agent-v2, specifically addressing the challenges of task progress navigation and content focus in mobile operations. 

* **end-user evaluation.** The area of mobile GUI agent is still at very early stage. The current time and resource constraints make end-user evaluations challenging. Our primary focus at this stage has been on establishing the core architecture and demonstrating its effectiveness through technical benchmarks, which we believe are essential first steps before moving on to broader end-user evaluations.

We believe that Mobile-Agent-v2 is well-positioned to significantly advance mobile operation tasks, and we are committed to improving the framework based on your valuable suggestions.
I thank authors for their detailed responses to my questions.

While each element of Mobile-Agent-v2 may not be novel in itself, its significance lies in configuring the agentic workflow through multiple agents within the mobile application domain. Therefore, I believe that this paper should be evaluated from an end-user perspective. From this viewpoint, it is essential for the framework to automatically identify and resolve the errors, as highlighted in question 5. Additionally, minimizing the exploration, as discussed in question 2, is crucial for enhancing the user experience. Moreover, regarding question 1, the proposed framework might handle ambiguous tasks, but there is a lack of experimental proof.

As these details crucial to enhancing the user experience are missing in Mobile-Agent-v2, I will maintain my current score. I hope these aspects can be addressed in future version of the paper.
Thanks for your reply, things are clear now. I hope that these experiments details and results can be reflected in a future version of the paper.
Thank you for your response. Below is our response to the above two questions.

### **Question 1: Could you provide more details about the experiments?**
### **Response:**
* Environment: We keep the evaluation environment consistent with the our paper, namely multi-platform (Android OS and Harmony OS), multi-app type (system app and third-party app), and multi-language (English and non-English). All evaluation tasks start from the mobile phone's desktop, and the app used in the instructions is ensured to exist on the desktop.
* Setting: We used AppAgent's official open source project for evaluation. During the evaluation, we only conducted an exploration phase, that is, running the official code “learn.py”. All other settings, including hyper-parameters and MLLM selection, were defaulted using the official code.
* Number: We hope that AppAgent can be consistent with the paper in terms of apps and evaluation tasks. However, since AppAgent relies on XML, some pages involved in instructions cannot obtain XML files. We found that this may be because the app page has permissions or the page is dynamic (such as video), which makes the XML unable to be obtained through ADB. We also removed these tasks from the evaluation results of AppAgent and Mobile-Agent-v2. In the end, a total of 63 instructions were used for evaluation.

### **Question2: How is the storage and retrieval costs of memory cells?**
### **Response:**
* The storage of memory unit depends on the maximum input length of MLLMs. For example, the most advanced MLLM GPT-4o can support an input length of 128K. This means that the memory unit can definitely hold the entire app's tutorials. These tutorials can come from the official instructions of the app, manually written operating experience, or from key information recorded during the historical tasks of Mobile-Agent-v2.
* Since memory retrieval is done at the decision stage by decision agent, we cannot directly quantify the retrieval efficiency. However, we have experimentally found that, while keeping the output length unchanged, every 1k tokens added to the memory unit will increase the time it takes the decision agent to make a decision by 200ms to 1s. This means that even if the memory length reaches 3k tokens (the length of a general app tutorial), the decision stage will only increase the time by 8% (the average decision time increased from 21s to 22.8s.).
Thank you for taking the time and effort to address all questions.

There are a few follow-up questions.

a) Could you provide more details about the experiments comparing the performance of Mobile-Agent-v2 and AppAgent, such as the environment, setting, number of evaluation tasks, etc. 

b) How is the storage and retrieval costs of memory cells?