Scaling Computer-Use Grounding via User Interface Decomposition and
  Synthesis
Authors: name: Tianbao Xie; name: Jiaqi Deng; name: Xiaochuan Li; name: Junlin Yang; name: Haoyuan Wu; name: Jixuan Chen; name: Wenjing Hu; name: Xinyuan Wang; name: Yuhui Xu; name: Zekun Wang; name: Yiheng Xu; name: Junli Wang; name: Doyen Sahoo; name: Tao Yu; name: Caiming Xiong
Published: 2025-05-19 15:09:23

Abstract: Graphical user interface (GUI) grounding, the ability to map natural language
instructions to specific actions on graphical user interfaces, remains a
critical bottleneck in computer use agent development. Current benchmarks
oversimplify grounding tasks as short referring expressions, failing to capture
the complexity of real-world interactions that require software commonsense,
layout understanding, and fine-grained manipulation capabilities. To address
these limitations, we introduce OSWorld-G, a comprehensive benchmark comprising
564 finely annotated samples across diverse task types including text matching,
element recognition, layout understanding, and precise manipulation.
Additionally, we synthesize and release the largest computer use grounding
dataset Jedi, which contains 4 million examples through multi-perspective
decoupling of tasks. Our multi-scale models trained on Jedi demonstrate its
effectiveness by outperforming existing approaches on ScreenSpot-v2,
ScreenSpot-Pro, and our OSWorld-G. Furthermore, we demonstrate that improved
grounding with Jedi directly enhances agentic capabilities of general
foundation models on complex computer tasks, improving from 5% to 27% on
OSWorld. Through detailed ablation studies, we identify key factors
contributing to grounding performance and verify that combining specialized
data for different interface elements enables compositional generalization to
novel interfaces. All benchmark, data, checkpoints, and code are open-sourced
and available at https://osworld-grounding.github.io.