Agent S2: A Compositional Generalist-Specialist Framework for Computer
  Use Agents
Authors: name: Saaket Agashe, name: Kyle Wong, name: Vincent Tu, name: Jiachen Yang, name: Ang Li, name: Xin Eric Wang
Organization: Simular

Computer use agents automate digital tasks by directly interacting with
graphical user interfaces (GUIs) on computers and mobile devices, offering
significant potential to enhance human productivity by completing an open-ended
space of user queries. However, current agents face significant challenges:
imprecise grounding of GUI elements, difficulties with long-horizon task
planning, and performance bottlenecks from relying on single generalist models
for diverse cognitive tasks. To this end, we introduce Agent S2, a novel
compositional framework that delegates cognitive responsibilities across
various generalist and specialist models. We propose a novel
Mixture-of-Grounding technique to achieve precise GUI localization and
introduce Proactive Hierarchical Planning, dynamically refining action plans at
multiple temporal scales in response to evolving observations. Evaluations
demonstrate that Agent S2 establishes new state-of-the-art (SOTA) performance
on three prominent computer use benchmarks. Specifically, Agent S2 achieves
18.9% and 32.7% relative improvements over leading baseline agents such as
Claude Computer Use and UI-TARS on the OSWorld 15-step and 50-step evaluation.
Moreover, Agent S2 generalizes effectively to other operating systems and
applications, surpassing previous best methods by 52.8% on WindowsAgentArena
and by 16.52% on AndroidWorld relatively. Code available at
https://github.com/simular-ai/Agent-S.