Mobile-Agent-v2: Mobile Device Operation Assistant with Effective
  Navigation via Multi-Agent Collaboration
Authors: name: Junyang Wang; name: Haiyang Xu; name: Haitao Jia; name: Xi Zhang; name: Ming Yan; name: Weizhou Shen; name: Ji Zhang; name: Fei Huang; name: Jitao Sang
Published: 2024-06-03 05:50:00

Abstract: Mobile device operation tasks are increasingly becoming a popular multi-modal
AI application scenario. Current Multi-modal Large Language Models (MLLMs),
constrained by their training data, lack the capability to function effectively
as operation assistants. Instead, MLLM-based agents, which enhance capabilities
through tool invocation, are gradually being applied to this scenario. However,
the two major navigation challenges in mobile device operation tasks, task
progress navigation and focus content navigation, are significantly complicated
under the single-agent architecture of existing work. This is due to the overly
long token sequences and the interleaved text-image data format, which limit
performance. To address these navigation challenges effectively, we propose
Mobile-Agent-v2, a multi-agent architecture for mobile device operation
assistance. The architecture comprises three agents: planning agent, decision
agent, and reflection agent. The planning agent generates task progress, making
the navigation of history operations more efficient. To retain focus content,
we design a memory unit that updates with task progress. Additionally, to
correct erroneous operations, the reflection agent observes the outcomes of
each operation and handles any mistakes accordingly. Experimental results
indicate that Mobile-Agent-v2 achieves over a 30% improvement in task
completion compared to the single-agent architecture of Mobile-Agent. The code
is open-sourced at https://github.com/X-PLUG/MobileAgent.