OS-ATLAS: A Foundation Action Model for Generalist GUI Agents
Authors: name: Zhiyong Wu; name: Zhenyu Wu; name: Fangzhi Xu; name: Yian Wang; name: Qiushi Sun; name: Chengyou Jia; name: Kanzhi Cheng; name: Zichen Ding; name: Liheng Chen; name: Paul Pu Liang; name: Yu Qiao
Published: 2024-10-30 17:10:19

Abstract: Existing efforts in building GUI agents heavily rely on the availability of
robust commercial Vision-Language Models (VLMs) such as GPT-4o and
GeminiProVision. Practitioners are often reluctant to use open-source VLMs due
to their significant performance lag compared to their closed-source
counterparts, particularly in GUI grounding and Out-Of-Distribution (OOD)
scenarios. To facilitate future research in this area, we developed OS-Atlas -
a foundational GUI action model that excels at GUI grounding and OOD agentic
tasks through innovations in both data and modeling. We have invested
significant engineering effort in developing an open-source toolkit for
synthesizing GUI grounding data across multiple platforms, including Windows,
Linux, MacOS, Android, and the web. Leveraging this toolkit, we are releasing
the largest open-source cross-platform GUI grounding corpus to date, which
contains over 13 million GUI elements. This dataset, combined with innovations
in model training, provides a solid foundation for OS-Atlas to understand GUI
screenshots and generalize to unseen interfaces. Through extensive evaluation
across six benchmarks spanning three different platforms (mobile, desktop, and
web), OS-Atlas demonstrates significant performance improvements over previous
state-of-the-art models. Our evaluation also uncovers valuable insights into
continuously improving and scaling the agentic capabilities of open-source
VLMs.