ICML 2025

UI-Vision: A Desktop-centric GUI Benchmark for
Visual Perception and Interaction
Shravan Nayak * 1 2 3 Xiangru Jian * 4 3 Kevin Qinghong Lin 5 Juan A. Rodriguez 1 3 6 Montek Kalsi 3
Nicolas Chapados 3 M. Tamer ¨Ozsu 4 Aishwarya Agrawal 1 2 7 David Vazquez 3 Christopher Pal 1 8 3 7
Perouz Taslakian 3 Spandana Gella 3 Sai Rajeswar 3 1 2
https://uivision.github.io
Abstract
Autonomous agents that navigate Graphical User
Interfaces (GUIs) to automate tasks like docu-
ment editing and file management can greatly
enhance computer workflows. While existing re-
search focuses on online settings, desktop environ-
ments, critical for many professional and everyday
tasks, remain underexplored due to data collec-
tion challenges and licensing issues. We intro-
duce UI-Vision, the first comprehensive, license-
permissive benchmark for offline, fine-grained
evaluation of computer use agents in real-world
desktop environments. Unlike online benchmarks,
UI-Vision provides: (i) dense, high-quality an-
notations of human demonstrations, including
bounding boxes, UI labels, and action trajecto-
ries (clicks, drags, and keyboard inputs) across
83 software applications, and (ii) three fine-to-
coarse grained tasks—Element Grounding, Lay-
out Grounding, and Action Prediction—with
well-defined metrics to rigorously evaluate agents’
performance in desktop environments. Our evalu-
ation reveals critical limitations in state-of-the-art
models like UI-TARS-72B, including issues with
understanding professional software, spatial rea-
soning, and complex actions like drag-and-drop.
These findings highlight the challenges in develop-
ing fully autonomous computer-use agents. With
UI-Vision, we aim to advance the development of
more capable agents for real-world desktop tasks.
*Equal contribution
1Mila - Quebec AI Institute 2Universit´e
de Montr´eal
3ServiceNow Research
4University of Water-
loo 5National University of Singapore 6 ´Ecole de Technologie
Sup´erieure 7CIFAR AI Chair 8Polytechnique Mon...