Grounding Computer Use Agents on Human Demonstrations
Authors: name: Aarash Feizi; name: Shravan Nayak; name: Xiangru Jian; name: Kevin Qinghong Lin; name: Kaixin Li; name: Rabiul Awal; name: Xing Han Lù; name: Johan Obando-Ceron; name: Juan A. Rodriguez; name: Nicolas Chapados; name: David Vazquez; name: Adriana Romero-Soriano; name: Reihaneh Rabbany; name: Perouz Taslakian; name: Christopher Pal; name: Spandana Gella; name: Sai Rajeswar
Published: 2025-11-10 17:35:21

Abstract: Building reliable computer-use agents requires grounding: accurately
connecting natural language instructions to the correct on-screen elements.
While large datasets exist for web and mobile interactions, high-quality
resources for desktop environments are limited. To address this gap, we
introduce GroundCUA, a large-scale desktop grounding dataset built from expert
human demonstrations. It covers 87 applications across 12 categories and
includes 56K screenshots, with every on-screen element carefully annotated for
a total of over 3.56M human-verified annotations. From these demonstrations, we
generate diverse instructions that capture a wide range of real-world tasks,
providing high-quality data for model training. Using GroundCUA, we develop the
GroundNext family of models that map instructions to their target UI elements.
At both 3B and 7B scales, GroundNext achieves state-of-the-art results across
five benchmarks using supervised fine-tuning, while requiring less than
one-tenth the training data of prior work. Reinforcement learning post-training
further improves performance, and when evaluated in an agentic setting on the
OSWorld benchmark using o3 as planner, GroundNext attains comparable or
superior results to models trained with substantially more data,. These results
demonstrate the critical role of high-quality, expert-driven datasets in
advancing general-purpose computer-use agents.