DigiData: Training and Evaluating General-Purpose Mobile Control Agents
Authors: name: Yuxuan Sun, name: Manchen Wang, name: Shengyi Qian, name: William R. Wong, name: Eric Gan, name: Pierluca D'Oro, name: Alejandro Castillejo Munoz, name: Sneha Silwal, name: Pedro Matias, name: Nitin Kamra, name: Satwik Kottur, name: Nick Raines, name: Xuanyi Zhao, name: Joy Chen, name: Joseph Greer, name: Andrea Madotto, name: Allen Bolourchi, name: James Valori, name: Kevin Carlberg, name: Karl Ridgeway, name: Joseph Tighe
Organization: AI at Meta

AI agents capable of controlling user interfaces have the potential to
transform human interaction with digital devices. To accelerate this
transformation, two fundamental building blocks are essential: high-quality
datasets that enable agents to achieve complex and human-relevant goals, and
robust evaluation methods that allow researchers and practitioners to rapidly
enhance agent performance. In this paper, we introduce DigiData, a large-scale,
high-quality, diverse, multi-modal dataset designed for training mobile control
agents. Unlike existing datasets, which derive goals from unstructured
interactions, DigiData is meticulously constructed through comprehensive
exploration of app features, resulting in greater diversity and higher goal
complexity. Additionally, we present DigiData-Bench, a benchmark for evaluating
mobile control agents on real-world complex tasks. We demonstrate that the
commonly used step-accuracy metric falls short in reliably assessing mobile
control agents and, to address this, we propose dynamic evaluation protocols
and AI-powered evaluations as rigorous alternatives for agent assessment. Our
contributions aim to significantly advance the development of mobile control
agents, paving the way for more intuitive and effective human-device
interactions.