A3: Android Agent Arena for Mobile GUI Agents
Authors: name: Yuxiang Chai; name: Hanhao Li; name: Jiayu Zhang; name: Liang Liu; name: Guozhi Wang; name: Shuai Ren; name: Siyuan Huang; name: Hongsheng Li
Published: 2025-01-02 09:03:56

Abstract: AI agents have become increasingly prevalent in recent years, driven by
significant advancements in the field of large language models (LLMs). Mobile
GUI agents, a subset of AI agents, are designed to autonomously perform tasks
on mobile devices. While numerous studies have introduced agents, datasets, and
benchmarks to advance mobile GUI agent research, many existing datasets focus
on static frame evaluations and fail to provide a comprehensive platform for
assessing performance on real-world, in-the-wild tasks. To address this gap, we
present Android Agent Arena (A3), a novel evaluation platform. Unlike existing
in-the-wild systems, A3 offers: (1) meaningful and practical tasks, such as
real-time online information retrieval and operational instructions; (2) a
larger, more flexible action space, enabling compatibility with agents trained
on any dataset; and (3) automated business-level LLM-based evaluation process.
A3 includes 21 widely used general third-party apps and 201 tasks
representative of common user scenarios, providing a robust foundation for
evaluating mobile GUI agents in real-world situations and a new autonomous
evaluation process for less human labor and coding expertise. The project is
available at https://yuxiangchai.github.io/Android-Agent-Arena/.