Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction
Authors: name: Yiheng Xu; name: Zekun Wang; name: Junli Wang; name: Dunjie Lu; name: Tianbao Xie; name: Amrita Saha; name: Doyen Sahoo; name: Tao Yu; name: Caiming Xiong
Published: 2024-12-05 18:58:26

Abstract: Graphical User Interfaces (GUIs) are critical to human-computer interaction,
yet automating GUI tasks remains challenging due to the complexity and
variability of visual environments. Existing approaches often rely on textual
representations of GUIs, which introduce limitations in generalization,
efficiency, and scalability. In this paper, we introduce Aguvis, a unified pure
vision-based framework for autonomous GUI agents that operates across various
platforms. Our approach leverages image-based observations, and grounding
instructions in natural language to visual elements, and employs a consistent
action space to ensure cross-platform generalization. To address the
limitations of previous work, we integrate explicit planning and reasoning
within the model, enhancing its ability to autonomously navigate and interact
with complex digital environments. We construct a large-scale dataset of GUI
agent trajectories, incorporating multimodal reasoning and grounding, and
employ a two-stage training pipeline that first focuses on general GUI
grounding, followed by planning and reasoning. Through comprehensive
experiments, we demonstrate that Aguvis surpasses previous state-of-the-art
methods in both offline and real-world online scenarios, achieving, to our
knowledge, the first fully autonomous pure vision GUI agent capable of
performing tasks independently without collaboration with external
closed-source models. We open-sourced all datasets, models, and training
recipes to facilitate future research at https://aguvis-project.github.io/.