UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interaction
Authors: Shravan Nayak
Xiangru Jian
Kevin Qinghong Lin
Juan A. Rodriguez
Montek Kalsi
Nicolas Chapados
M. Tamer Özsu
Aishwarya Agrawal
David Vazquez
Christopher Pal
Perouz Taslakian
Spandana Gella
Sai Rajeswar
Venue: ICML 2025 poster (2025)
Keywords: Large language models
Multimodal models
LLMs
VLMs
Autonomous agents
GUI
Primary area: applications->everything_else
TL;DR: We introduce UI-Vision, a comprehensive desktop-centric and license-permissive GUI understanding benchmark

Abstract: Autonomous agents that navigate Graphical User Interfaces (GUIs) to automate tasks like document editing and file management can greatly enhance computer workflows. While existing research focuses on online settings, desktop environments, critical for many professional and everyday tasks, remain underexplored due to data collection challenges and licensing issues. We introduce UI-Vision, the first comprehensive, license-permissive benchmark for offline, fine-grained evaluation of computer use agents in real-world desktop environments. Unlike online benchmarks, UI-Vision provides: (i) dense, high-quality annotations of human demonstrations, including bounding boxes, UI labels, and action trajectories (clicks, drags, and keyboard inputs) across 83 software applications, and (ii) three fine-to-coarse grained tasks—Element Grounding, Layout Grounding, and Action Prediction—with well-defined metrics to rigorously evaluate agents' performance in desktop environments. Our evaluation reveals critical limitations in state-of-the-art models like UI-TARS-72B, including issues with understanding professional software, spatial reasoning, and complex actions like drag-and-drop. These findings highlight the challenges in developing fully autonomous computer-use agents. With UI-Vision, we aim to advance the development of more capable agents for real-world desktop tasks.

UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interaction
Shravan Nayak 122 Xiangru Jian *43 Kevin Qinghong Lin Juan A. Rodriguez Montek Kalsi
Nicolas Chapados
M. Tamer Özsu
Aishwarya Agrawal 127 David Vazquez² Christopher Pal 1837
Perouz Taslakian
Spandana Gella 3 Sai Rajeswar 312
https://uivision github.io
Abstract
"Create sittch 5:00mm created
Autonomous agents that navigate Graphical User Interfaces (GUIs) to automate tasks like docu- ment editing and file management can greatly enhance computer workflows. While existing re- search focuses on online settings, desktop environ- ments, critical for many professional and everyday tasks, remain underexplored due to data collec- tion challenges and licensing issues. We intro- duce UI-Vision, the first comprehensive, license- permissive benchmark for offline, fine-grained evaluation of computer use agents in desktop environments. Unlike online benchmarks, UI- Vision provides: (i) dense, high-quality an- notations of human demonstrations, including bounding boxes, UI labels, and action trajecto- ries (clicks, drags, and keyboard inputs) across 83 software applications, and (ii) three fine-to- coarse grained Easment Grounding, Lay- out Grounding, and Action Prediction- with well-defined metrics rigorously evaluate agents' performance in desktop environments. Our evalu- ation reveals critical limitations in state-of-the-art models like UI-TARS-72B, including issues with understanding professional software, spatial rea- soning, and complex actions like
MIVL 121374, 4753
CL2CK(293, $29)
CLICK(PTR,
1881
Figure 1: UI-Vision's Action Prediction task. Given a
task instruction, a screenshot of the current UI state, and a history of previous interactions, the agent must predict the next action necessary to progress toward task completion. We show here an example of successful episode from task based on FreeCAD software!.
1. Introduction
Graphical User Interfaces (GUIs) have become the dominant way users interact with digital worlds, replacing command- line interfaces with visually intuitive environments that en- hance usability across desktops, web browsing, and mobile devices (Jansen, 1998). To accelerate digital workflows and assist users, there has been rapid progress in developing intelligent GUI agents-Al systems capable of automating GUI interactions, from simple tasks like auto-filling forms to complex operations such as configuring software set- tings. Recent advances in Large Language Models (LLMs) have significantly expanded the capabilities of these agents, allowing them to reason and follow natural language instruc- tions (Wei et al., 2022; Yao et al., 2022b; Ouyang et al., 2022; OpenAI, 2021; Schick et al., 2023), However, LLMs that rely only on text struggle with GUI automation, as they lack the ability to interpret visual layouts, spatial re- lationships, and non-textual UI elements like icons (Gou
These findings highlight the challenges in develop- ing fully autonomous computer-use agents. With UI- Vision, aim advance the development of more capable agents for real-world desktop tasks.
Equaa contribution 'Mila- Quebec AI Institute Univeersité de Montréal "ServiceNow Research *University Water- 1o0 'National University of Singapore 'École de Technologie Supérieure "CIFAR AI Chair "Polytechnique Montréal. Corre- spondence to: Shravan Nayak <shravan www.annnnaaakkaa @ www.a.ammiaae.......... Xiangru Jian <x2jian uwaterloo.ca>>
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

UI- Vi-Vision
UL-Vision Benchmark
Data Creation Process
Key Features
Tested Skills in UI-Vision
Human Annotator
UI Task Execution Software
Screen Recording
Desklop Open-Source Platforms plutiomss:
Ul Grounding
Training
Action Logs
Thal Percepton and Strztual Peedittion
Densety Annotated Screen
Lpyaut Relation Reasoning Locase Bernents using Spassi Re ators
KeyFnimes.
insraction Tojectories
Navigation na To,...
Annotating Keytrames Bounding Boxes, OCR, Interactions
Quality Assurance Expert Human Review
Human Annotated & Verified Muti Step Expert Vellcoddon
Human Annotations
Ul Data & Benchmark Tasks
W Action Prediction
Element Grounding
? Layout Grounding
Instruction
Instruction: Predict the culegory and bounding box cocrdinates the folowing elements.
Instruction
Add lext "COOL" to the
region cantains navigation controls: back, forward, refresh, and home bunons.
image
Ruler
Freeform selector
TOC123, 258),
ealegwry: Savapeiion 155,
eniegary: Bucion
eatepary: Teolbor
"cs" ax,
boundiagizs" ('xt", 12, "ee":
HOVE T0(610,
dt.
DRAG TOE510, 294, 2013
tyal: "yot
TYPrNG "
KOVE To(767, 548),
the region contains user interaction elements: search bar, liter options. and sorting controls.
CLUCK( ror, 14ft)
Instruction: Open color selection dislog.
catupory: Zepuc Clanest
category: Tooiber
beandinnghas ("x8": 184, "satt 474,
Issastrriioe" (faxts 0ss, taat: 199,
"yt":
Figure 2: VI-Vision Benchmark Overview. Human annotators perform GUI tasks across various desktop software platforms, generating raw trajectories through screen recordings. The collected data undergoes expert verification and is then annotated with multiple layers of information, including keyframe detection, bounding box annotations with OCR, and action logs capture user interactions.
et al., 2024b). Humans navigate software visually, recog- nizing elements by appearance and position. To replicate this, automation must incorporate vision, allowing mod- els to process visual information beyond text. Multimodal LLMs have demonstrated effectiveness this domain, suc- cessfully automating web browsing (Shi al., 2017; Yao et al., 2022a; Gou et al., 2024a; Lü et al., 2024) and soft- ware control (Rodriguez et al., 2023; Li et al., 2023; Hong et al., 2024a; Qin et al., 2025), opening new possibilities for human-computer interaction.
unlike web automation, which benefits from tools like Sele- nium² and Playwright', This lack of standardization hinders large-scale data collection and automation. Given these gaps, a dedicated desktop-centric benchmark is needed to evaluate models on real- world software variability, visual perception, and open-ended GUI interactions.
To bridge the gap in evaluating GUI agents for desktops, we introduce UI-Vision a benchmark spanning 83 software ap- plications across 6 domains designed to assess GUI agents' ability to perceive and interact with desktop environments in an offline setting. To build UI-Vision participants de- sign computer use tasks across these applications that reflect world software interactions. They record demonstra- tions for these tasks and annotate them by labeling all UI elements (Figure 13) and capturing action sequences such as clicking, dragging, and typing, creating a rich human- annotated dataset. For an agent to take meaningful actions within a GUI, must first understand the interface in- teracts with. This involves two key abilities: recognizing functional regions where UI elements are grouped together and identifying specific elements within these regions. Our evaluation framework builds on these foundational skills by assessing an agent's 's ability to perceive, interpret, and interact with a GUI through three core tasks. First, Layout Grounding evaluates how well an agent identifies func- tional groupings within a GUI, helping it understand the
Despite progress in LLM- and multimodal-driven automa- tion, evaluating GUI agents for desktop remains challenging as existing benchmarks primarily focus on web and mobile environments. Web-based benchmarks like MiniWoB++ (Shi et al., 2017), Mind2Web (Deng et al., 2024), WebArena (Là et al., 2024; Zhou et al., 2023) and WorkArena (Drouin et al., 2024) assess agent performance using Document Ob- ject Model (DOM) structures and HTML metadata. While effective for web interfaces, these approaches fail for desk- top GUTs, which lack standardized text-based representa- tions like HTML. Accessibility (A1ly) trees provide mantic alternative but are often inconsistent and incomplete, limiting their reliability for UI understanding (Gou et al., 2024a; Dardouri et al., 2024). Similarly, mobile-focused benchmarks (Li et al., 2020b: Toyama et al., 2021) empha- size touchscreen interactions, making them less applicable to desktop environments where interactions rely on pre- cise mouse control and keyboard interactions. Furthermore, desktop applications lack standardized automation APIs,
'htps:/wwww.seleeimm.dew
hhtps:/ppaaywrihh.dew/
2

---

UI- UI-Vision
broader structure of interface (Figure 4 bottom right). Next, Element Grounding measures an agent's is ability to precisely recognize and locate individual UI components (Figure 4). Finally, Action Prediction builds on these skills, testing whether agents can execute interactions like click- ing, dragging, and typing based on their UI understanding (Figure 1). Unlike web and mobile interfaces, these tasks remain largely unexplored in desktop environments. UI- Vision fills this gap by providing a large-scale benchmark with extensive UI element coverage, spanning 83 platforms, 450 recorded videos with dense annotations, and 8267 query-label pairs, creating a fine-grained and robust evalu- ation framework for multimodal GUI agents.
2. Related Work
2.1. GUI Agents
Recent research highlights the expanding capabilities of large language models (LLMs) beyond conventional lin- guistic tasks including their application in GUI environ- ments. A key milestone in task-driven GUI navigation is MiniWoB++ (Shi et al., 2017), a web-based environment. Interest in this problem has been renewed with the emer- gence of large language models, which have demonstrated remarkable capabilities in orchestrating complex workflows autonomously (Yao et al., 2023; Yang et al., 2024a; Gao et al., 2023). driving progress in GUI automation. Current methodologies broadly fall into two categories.
We evaluate ofate-of-thh-arr GUT agents on our UI-Vision benchmark, which reveals significant gaps across all three tasks. Element Grounding, core ability for performing actions on GUI, remains particularly difficult where even the best-performing model, UI-TARS (Qin et al., 2025), achieves only 25.5% accuracy. This becomes more chal- lenging when an interface is functionality- and visually complex, with an increased number of UI elements. Layout Grounding remains challenging task, with Gemini-1.5-Pro (Team al., 2024) achieving only 30.8 IoU, indicating that models lack high-level visual comprehension to recognize structured UT regions. Lastly, Action Prediction is conse- quently challenging, with UI-TARS achieving only 19.7% recall on click actions. The lack of accurate grounding naturally limits action prediction, as models struggle to as- sociate UI elements with the correct interactions. Moreover, all models struggle with drag actions, exposing weaknesses in handling notion-based interactions, capability not typi- cally required in web-based GUI tasks.
Pure Language Agents extract UI metadata from HTML structures, accessibility trees, or OCR (Lee et al., 2023) and spatial-semantic models (Yang et al., 2023; Yan et al., 2023; Zheng et al., 2024; Lu et al., 2024). LLMs then process this structured data to generate task-specific actions. While flexible, these approaches rely on closed-source models applications
and struggle with generalization, as often operate on raw visual inputs rather than structured metadata, which is not always available.
Multimodal Agents attempt to overcome these limitations by using multimodal datasets, pairing images with textual descriptions (Hong et al., 2023; Cheng et al., 2024; You et al., 2024; Li et al., 2024; Gou al., 2024a; Lin al., 2024b; Yang et al., 2024b; Wu et al., 2024; Qin et al., 2025). This approach enables pixel-level element grounding and context- aantett-wware interface navigation. However, progress remains constrained by the lack of large-scale visual data and standardized benchmarks for real-world UI interactions in desktop environments. Our work fills this gap by introducing a benchmark tailored to desktop applications.
In summary, our major contributions include:
We introduce UI- Vision, the largest and most diverse desktop GUT benchmark, spanning 83 real-world envi- ronments. It enables a comprehensive evaluation of mod- els in an offline setting across three core tasks: Element Grounding, Layout Grounding, and Action Prediction.
2.2. GUI Benchmarks
Existing benchmarks evaluate different aspects of GUI agent capabilities but remain fragmented in their focus. Element grounding benchmarks (Cheng et al., 2024; Li al., 2025) test agent's 's ability to locate UI elements like icons and text. Action prediction datasets (Deng et al., 2024; Rawles et al., 2023; Zhang et al., 2024) assess task understanding and next-step inference based on screenshots and history. However, layout grounding is often overlooked- coordinate-bast based methods (Li et al., 2025) fail to capture structural relationships in GUT tasks (Wu et al., 2023; Ro- driguez al., 2024). Most benchmarks specialize in either grounding (Cheng et al., 2024; Li et al., 2025), naviga- tion (Rawles et al., 2023; Deng et al., 2024), or understand- ing (Hsiao et al., 2024), lacking a comprehensive evaluation framework. Furthermore, GUI benchmarks are dispropor- tionately skewed towards Web (Hao et al., 2011; Wu et al.,
Our dataset provides dense annotations with unmatched UI element coverage, allowing models to be tested spatial relationships and UI layout areas often overlooked in prior benchmarks. Built from open-source and per- missive data, it ensures accessibility and reproducibility.
We identify major weaknesses in state-of-the-ar models across core GUI tasks. Grounding of elements and lay- out remains highly challenging, as models struggle with fine-grained understanding and lack proper recognition of UI regions and their significance. These limitations ultimately hurt action execution, where GUI agents fail at precise click and drag operations.
3

---

UI- Ul-Vision
Enviromments
Tasks
Statisties
Data Collection
Benchmarks
Platform
SW/App Settings
Ground. (multi)
Action Layout
Sample Avg. Ele Avg. Steps
Human Open License
MiniWoB++ (Shi sl., 2017)
Web
N/A
Online
X
X
125
NIA
2.3
N/A
N/A
Mind2Web (Deng et al., 2024)
Web
N/A
Offline
x
X 2,350
1
7.3
V
AITW (Rawles et al., 2023)
Mobile
159
Oflline
X 30,378
1
6.5
NIA
OmniAct (Kapoor al., 2024)
Desklop, Weh
Offine
9,802
NIA
OS-ld (Xie 2024)
Desktop
8
Online
X
369
NIA
N/A
X(Windows)
VideoGUI (Lin all, 20244a
Desktop
11
Offine
X
/
X
86
N/A
22.7
N/A
ScreenSpot (Cheng et all, 2024)
Desktop. Web, Mobile
N/A
Offiine
((K)
X
X
1,272
1
N/A
N/A
ScreenSpot- Pro (1 et al., 2025)
Deskiop
23
Offine
(00)
X
X
1,581
N/A
UI- Vision (ours)
Desktop
83 Ofline
J(C)
8227
7.3
Table I: Comparison of existing GUI benchmarks with our UI-Vision. We evaluate their inclusion of desktop platforms, noting a gap in recent works. Additionally, we examine permissive licensing (Open License), highlighting that UI- Vision focuses on open-source platforms. Lastly, we assess the presence of relevant annotations, particularly those related to grounding, text, and OCR. Multi. refers to Multi purli- query for grounding.
2023; Zhou et al., 2023; Koh et al., 2024) and Mobile (Deka et al., 2017; Li et al., 2020a;b; Toyama et al., 2021) plat- forms, primarily due to their accessibility. Desktop environ- ments, despite their significance in professional work- flows, remain underexplored. Existing desktop bench- marks (Xie al., 2024; Bonatti et al., 2024) primarily focus on online interactions, while others are constrained by small- scale datasets (Kapoor et al., 2024; Lin et al., 2024a).
Education
Table compares existing benchmarks, highlighting the lack of comprehensive desktop datasets with diverse applications, annotations and real-world complexity. To address these gaps, we introduce UI-Vision, the largest desktop-centric benchmark date, spanning 83 software environments and featuring rich human annotations, With 8227 query- label pairs across three tasks and a broad software and do- main coverage, UI-Vis Vision establishes a new standard for evaluation in desktop environments.
grayoic
Drayfr La
Productivy an
Greatueey
Entertatiennd Srone
enty
bniyoli Fante
capa
Figure 3: Distribution of Software Platforms in UI- UI-Vision. The outer ring displays 83 different software platforms span- ning six categories shown in the inner ring.
3. UI- Vision O
This section introduces UI-Vision, a large-scale benchmark for GUI navigation and visual grounding across 83 desktop applications. We describe the data collection and annotation in Section 3.1 and in Section 3.2 explain how these annota- tions are used to construct the UI-Vision benchmark tasks for model evaluation.
involved in data collection.
Selecting Desktop Applications. We curate data from 83 open-source desktop platforms across six domains: Pro- ductivity, Development, Creativity, Education, Browsers, and Social Media/Entertainmenn. ensuring diverse platforms for comprehensive benchmarking and permissive licensing (Figure 3 and Table 6).
3.1. Data Collection
Designing Computer Use Tasks. Computer use tasks are designed around real-world workflows, ranging from basic tasks (e.g., renaming a folder) to complex operations (e.g., applying subtitles to a video). We ensure that each task well-defined, and comprehensive. Each platform includes 5-7 computer tasks.
We first collect data from users interacting with desktop software to achieve goal, capturing their actions and an- notating critical elements. This involves recording user interactions, extracting action trajectories that document step-by-step decisions to achieve a goal, and labeling UI components with bounding boxes and functional descrip- tions. We partner with data- cota-sourcing company for this process, ensuring a diverse and well-trained annotator pool. Details about the annotators and their qualifications are pro- vided in the Appendix A. Below, we describe the key steps
Capturing and Annotating User Interactions. Expert an- notators perform computer use tasks while capturing (i) task videos, (ii) logged actions (10 predefined types; Table 7 in Appendix B) with metadata such as mouse click type, click frequency and exact timestamp of the action in the video,
4

---

UI- Ul-Vision
Grounding
Spatial Grounding
He Exit
+
Tron
Actions
Bus
Applisation: Linu Temiiaal
Application: GMU Cash
Ayplisaton: Windows Explorer
Appltcation:
Edi
Grounding Instruciio:: Tax"
Grounding Instruction: What sienent that uericaly dicwerr the wes" and abore o? Ul Type: ingut Beement
Grounding Inair What alament horizorialy to she right vilplay selected coosole?? Ul Type: Button Bousnging
*Ul Type: Buttan
HUl Sultan
-Bounging Blosc (20, 1o2, 36, 242)
-Bounging Box: (102. 22, 222
BBonging (36
551]
Functional Grounding
Layout Grounding
f Save
Application: GNU Cash
Application: GMP
Application: LibreCffcee Witter
Application: Danstatle
Opwn
Grounding Region
Grounding snapshots, histeyy dusicaae
Buston
UI InputElemmn
tomaatngg firs, size, aigunere
*Bounglng Box: (87, 63, 382,
+Bounging Box: 218, 416)
-ui Typec syyut
+UI Type: Button
+Bounging Box: (220. 97. 817. 128]
* BBounging Box (18, 517, 325, 600)
Figure 4: Examples of Grounding Tasks Vi-visoon. The dataset features three GUI grounding tasks: Basic, involving locating elements' bounding boxes; Spatial, focusing on determining positions relative to other elements; and Functional, requiring identification of click locations for specific actions. Layout Grounding locates larger regions given a description.
and (iii) keyframe screenshots, which are screenshots taken just before an action, such as a click, is performed. Post- task, annotators label UI elements with bounding boxes and descriptions for the keyframme using a proprictary tool. A multi-stage quality check by separate annotators and peri- odic verification by authors ensures completeness, accuracy, and consistency. The final dataset consists of 450 high- quality demonstrations across 83 applications. Figure 2 provides an overview of the data collection process.
bilities: Element Grounding, to identify and localize UI elements from textual queries; Layout Grounding, to rec- ognize structural relationships and group UI elements into functional clusters; and Action Prediction, to determine the correct interaction needed to achieve goal. The first two address perception and structural prediction, while the third connects perception to interaction. See Figure 4 for exam- ples of Flement and Layout Grounding and Figure 1 for an example on the Action Prediction task.
Dataset Statistics. In Figure 3, we present the taxonomy and software distribution within UI- Vision, with the largest portions corresponding to 33.7% productivity. 25.3% de- velopment, and 19.3% creativity. Figure 12 in Appendix C presents key dataset statistics, including bounding box distribution per keyframe, task video durations, and action steps required for completion. Each keyframe contains be- tween 5 to 200 labeled bounding boxes, with an average of 71, resulting nificantly higher than previous works (Table 1), highlighting the richness of our dataset. Most videos are under 50 sec- onds, averaging 37.6 seconds. Finally, annotators complete most tasks within 25 steps, with average of 14 steps. This makes the dataset significantly complex, as agents must plan and predict actions over long-horizon tasks, which pose a major challenge.
TASK 1: ELEMENT GROUNDING
This task evaluates a model's ability to predict the bounding box of a UI component in a screenshot given short query (e.g., "Add New Tab"). We leverage the dense bounding box annotations collected during the initial data collection stage to introduce three grounding subtasks basic, functional, and spatial-to assess different aspects of GUI understand- ing beyond simple textual queries. To create these subtasks, we implement multi-stage process that involves sampling challenging UI elements followed by thorough human eval- uation of selected UI elements (see details in Appendix B)
dense annotation setup. This count is sig-
The basic setting evaluates model's ability to predict the location of UI element (e.g.. button, sidebar, input field) given minimal textual description, such as "Actions" (Fig- ure 4). The functional setting requires identifying UI ele- ments based on their function rather than direct labels; for instance, instead of "Save Button," the query specifies "Save the current document or data." Functional descriptions are generated using GPT-4o (OpenAI, 2023) and then validated by human reviewers. The spatial setting requires locating
3.2. UI- Vision Benchmark
Building on the rich annotations from the previous section, we focus on three key tasks essential for an agent's capa-
5

---

UI- Ui-Vision
Action
Description
our action spaces and their corresponding parameters.
Bove DG, y)
mouse to specified cvordinates.
click y, button)
Click the specifed button at the given coordinates.
Typing ('Hlllo')
Types spacified string.
4. Experiments
Hotkey ('otrl,, "ery
Performs individual or combination hotkeys.
Drag ( (12,y11, (22,y21)
Drags the mouse from start (s11, g4) end (x2, v2).
4.1. Baselines
Table 2: Action Prediction task, action space.
We tested several open-source VLMs with GUI capabilitis including Qwen2-VI--7B and Qwen2.5VI.-7B (Wang et al., 2024), MiniCPM-V-2.6-888 (Yao et al., 2024), Intern 8B and IntemnL2..588B (Chen et al., 2024); open source GUI agents, including CogAgent-915 (Hong al., 2023), SeeClick-9 9.6B (Cheng et al., 2024), UGround-7B and 72B (Gou et al., 2024b), UI-TARS 7B and 72B (Qin et al., 2025), OSAtlas-7B (Wu et al., 2024), ShowUl-2B (Lin al., 2024b), Aria-U1-25.33B5 23..5.333..9 active parameters) (Yang et al., 2024b) and Aguvis-7B (Xu et al., 2024). We also considered closed source models such as GPT-4o (Ope- nAI, 2024). Claude-3.5-Sonnet and Claude-3.7-Sonme (An- thropic, 2024), and Gemini-1.5-pr and Gemini-Flash- 2.0 (Team et al., 2024). Each model used its recommended format for prompting.
UI elements based on their spatial relationships with neigh- boring components. Queries are generated by detecting the nearest bounding boxes in four directions (left, right, top, bottom), e.g.. "What is directly above the 'Exit' button?" Both basic and functional grounding subtasks have 1,772 query-label pairs, while the spatial setting has 1,935.
TASK 2: LAYOUT GROUNDING
1
GUI layouts define how interface components, such as but- tons, text fields, and containers, are arranged into cohe- sive regions (e.g.. "Formatting Tools" or "Main Navigation Bar"). This novel task evaluates model's 's ability to clus- ter UI elements into funetional and semantic groups and predict bounding boxes that encapsulate them, capabil- ity not explicitly explored in previous GUI-related bench- marks (Figure 4, bottom right). We again leverage the dense bounding box annotations collected during the initial data collection phase and provide them as input to I.LAMA- 3.3-70B (Meta, 2024), which generates non-overlapping functional or semantic clusters along with textual descrip- tions (e.g.. "Formatting Tools"). These textual descriptions serve as queries for the model, which must predict bounding boxes that encapsulate entire functional or semantic areas. The dataset comprises 311 I from 77 platforms, with each functional or semantic cluster containing 5-10 UI elements (see details in Appendix B.2).
4.2. Evaluation Metries
Element Grounding. Following prior works (Li et al., 2025; Cheng et al., 2024; Wu et al., 2024), prediction is considered correct if the predicted point (r,, 34) falls within the ground-truth bounding box: Correct = (2 I x; S Tmax A Ymnin S 3/s S 4max)) We compute accuracy across all UI components. As prior analysis (Cheng et al., 2024) showed minimal gains from bounding-box boo-bbaed evaluation, we adopt point-based evaluation throughout.
query-label pairs
Layout Grounding. We assess predicted bounding boxes using Intersection over Union (IoU), precision, and recall, measuring aligmment with ground truth.
TASK 3: ACTION PREDICTION
Action prediction builds on the previous perception tasks, requiring the agent to solve computer use tasks through a structured sequence of interactions. Unlike static recogni- tion tasks, this agentic task evaluates a model's capability to interpret given goal (e.g., "Apply a transparency of 45.9% to the layer") and determine the correct sequence of actions. We use the action trajectories collected during the data col-
Action Prediction. Inspired by action evaluation metries used by (Lin et al., 2024a), we develop metries for each action type in our dataset:
Click & Move: We evaluate coordinate predictions [r,y]] using two metrics: Distance (Dist.) - normalized Euclidean distance (D) between predicted and ground ground-truth locations: Dist == D where L is the maximum possible distance to any corner. Recall@d d click is correct if within d pixels of the ground truth. We calculate d as the average bounding box size across the dataset.
lection stage to create this task. Each step in the trajectory is treated as an independent prediction, with the model re- ceiving the screenshot of the current UI state and previous actions as input. More concretely, given a task query Q and screenshot Zi at the i-th state, along with the previous action history H {aglisi-i, the model is required to output à. Each action a; includes both the action type and its parameters. The details of creating the task including choosing the right screenshots from videos for prediction and processing action trajectories are detailed in Appendix B.3. As a result, we obtain 3191 action-anne annotation pairs over 442 computer use tasks. In Table 2, we explicitly define
Drag: Evaluated using: Distance (Dist.) measures average displacement error by computing the mean Fu- clidean distance between the predicted and actual start and end coordinates of the drag action: Dist == 1 ( + 2:). Recall d checks if both start and end positions are within d pixels.
Typing & Hotkey:
Evaluated using Correctness
6

---

UI- Ul-Vision
Functional
Spotial
Final Ang
Model
Ed
Fr.
Do
Pr
Cr
Ex
Owalll
Ed
Frr
Die
Pr
Cr
En
Overdl
Fai
De
Py
Cr
Overall (1935)
(215)
(56)
(376)
(503)
(438)
(82)
(1772)
(215)
(376)
(503)
(438)
(82)
(1772)
(212)
(31)
(338)
(740)
(586)
(28)
Claced-Nonree VL.Ms
GPT-4s (OpenAI,
2.23
1.86
1,16
9.88
1.58
1.92
5.19
0.91
3.66
0.94
1,48
0.51
Geninii-1.5 qmu (Team al. 2023)
0.47
0.00
1.60
0.63
0.46
0.00
0.79
0.00
1,79
0.27
0.17
0.46
0.00
0.28
0.94
0.00
0.59
0.54
0.34
0.00
0.57
0.55
Gemini-Paal-200 (Team et sl., 2023)
0.00
0.00
0.27
0.66
0.66
0.00
0.45
0.47
1.79
0.000
0.66
0.23
0.00
01.40
0.00
0.00
0.00
0.14
0.00
0.00
0.06
0.30
(hhaaee-.--Soomdl 1.5-So (Anthropic 1014
3.26
5.52
5.94
1.81
1.88
5.08
5.12
19.6
4.70
5.98
2.51
4.88
5.19
2.53
9.68
5.0%
2.41
2.56
2.14
1.15
4.42
Claste-377 Sonnet (Asthropi. 30244)
6.51
12.5
11.34
7.88
Open-Source VLAS
Moec-22.VVL-78 (Wong al. 2036)
0:47
0.00
1.58
1.65
0.663
1:22
1:34
0:44
0.00
0.:00
1.16
0.46
1:22
0.75
0:47
0.00
1.46
0.00
1.51
0.00
0.51
0LSS
ImmmV11. 2-08 (C hem 2004)
0:00
0.00
(.00)
0.00
(1:00
11.14
0.11
11:00
0.00
0.22
0.00)
0.00
1.27
0111
0.001
01:00
0.00
0.14
11:00
0.00
0.08
0.00
Im 5 8B (Chen et oll, 2004)
0.93
3.93
3.46
2.51
1.32
4.88
2.48
1.40
7.14
3.72
2.61
1.00
0.10
2.82
0:94
3.23
1.28
1..8
13.68
3.57
2.09
Ques 2VL-7B (Wang al. 2024)
2.22
1.72
3.07
12.2
3,45
2.72
12.5
3.19
3.22
0.65
6.10
3.21
0.47
3.23
2.37
0.51
2.00
MiaCCP5: V-8B (Yao 2024)
4.19
18.3
4.19
6.38
2.97
1.78
Opens-onree GEV Agents
Show 2B (LU et 202489
5.12
16.1
9.54
9.09
3.42
19.5
5.07
5.12
12.5
9.31
5,00
4,31
1559
7.67
0.94
9.68
2.96
2.79
1.68
3.57
2.07
5.94
AddUl 25, 3B (Yang et el., 20248)
10.7
25.2
13.0
12.9
8.22
20.7
12.2
12.6
19.6
15.4
14.6
10.5
22:0
1410
3.77
9.68
4.44
4.85
2.22
7.34
3.96
10.1
UGmondd11-7 (Ceu et al, 20244)
11.6
19.7
15.0
18.3
15.4
15.4
22.3
16.5
11.6
19.5
17.1
4.25
6.45
9.76
6:35
4.44
14:3
6:35
12.9
0SAilse-BB (Wir 2024)
10.7
13.3
12.6
12.2
11.4
3.77
2.22
U.Goord-18 (Gon et Ni0Ma)
16.7
2N.2
14.9
10.6
7:51
19.5
11.5
17.3
25.0
15.2
11.2
7.99
20/7
17.2
3.16
0.00
4.54
2.20
2.20
7.04
7.79
6.55
Agunts TB (hi ol., 2004)
16.7
11.5
12.3
16.1
12.4
76.
12.8
12.2
15.1
31.5
1811
13.0
24.1
18.1
5.10
0.68
0.51
4.115
6.74
11.1
5,00
11.2
UL TURSSRSTT 7B (Qto et 2025)
15.4
41.1
21.31
21.3
13.2
39.0
20.1
20.5
41.1
25.5
26.5
16.0
45.1
34.3
6.60
12.9
13.0
9.19
5.89
17.9
5.37
17.6
CagAgeni-98 (Hony 202480
12.5
15.9
12.0
11.4
3.50
SeCCick--AR oChing al. 2004)
7.44
25.2
13.0
8.43
17.1
9.42
4.65
7.84
5.50
3.97
4.34
7.50
4.68
0.47
6.45
3.25
1.22
2.71
3.57
2.07
5.19
UGnoodd-1-122 (Gou er at., 2014a)
27.0
42.9
31.7
20.6
20.8
40.2
27.9
25.1
33.9
30.6
26.6
21.0
40.2
26,7
15.1
25.8
19.8
13.4
12.8
25.0
14.9
15.2
ULTTARS 72B (Qun et 2025)
30.7
483
33.7
33.6
21.9
51.2
31.4
29.5
46.4
30.9
34.1
22.6
36.6
30.5
13.7
16.1
19.2
15.4
11.1
25.0
14.7
25.6
Table 3: Element Grounding results by category for Basic, Functional, and Spatial settings. For each setting, the first six columns (Ed, Br, De. Pr, Cr, En) present the fine-grained breakdown, followed by an overall score column. The number of samples in each category is noted under the column name. The final column shows the overall average. Abbreviated category labels: Ed (Education), Br (Browsers), De (Development), Pr (Productivity), Cr (Creativity), En (Entertainment). The best model within each size category is highlighted in bold, and the runner-up is underlined. Models are categorized by size: gray marks closed-source models, green indicates open-source VLM models, blue represents open-source GUI Agent models below 8B (active) parameters, orange denotes open-source GUI Agent models above (active) 8B parameters.
(Corr.), which verifies exact string keystroke matches. We also compute the Step Success Rate (Deng et al., 2024) as an overall performance measure across the dataset. A step is considered successful only if the predicted action and its associated metadata are correct. This includes bounding boxes for click and drag actions, as well keyboard inputs for typing and hotkey actions.
mance remains low, highlighting the challenges of ground- ing in desktop environments. Functional grounding yields similar scores to basic grounding for GUI agents, as they are explicitly trained on such functional tasks (Gou et al., 2024b; Qin et al., 2025). In contrast, both open-source and closed-source VLMs perform worse in this setting. The biggest challenge lies in spatial grounding, where even the best model achieves only 14.9%, underscoring the difficulty of understanding spatial relationships in GUIs.
4.3. Results
A closer analysis reveals two major factors influencing model performance. First, as shown in Figure 5(a), models perform better when grounding larger UI elements, sug- gesting that smaller elements are harder to predict. Second, Figure 5(b) indicates that accuracy drops sharply when more elements are present in screenshot. This pattern is evident across domains creativiit platforms such as Blender and GIMP exhibit the lowest performance, likely due to their densely packed GUT interface, with an average of 1 bound- ing boxes per screen and an UI element area averaging 4 418 square pixels. In contrast, entertainment platforms like VLC Media Player achieve the highest performance, benefiting from larger and more spaced-out UI elements, averaging 62.8 bounding boxes per screen and an average area of 875 square pixels per element. While it is commonly assumed
Performance on Element Grounding. Table presents model performance on the three Element Grounding sub- tasks. Both closed-source and open-source VLMs struggle with this task. The best closed-source model, Claude 3.7 Sonnet (Anthropic, 2024), achieves only 8.7% accuracy, while the best open-source model, MiniCPM-V-ae V-8B (Yao et al., 2024), performs even worse at 4.3%. This poor per- formance likely stems from these models being trained for broad visual understanding, including grounding objects in natural scenes, rather than the fine-grained grounding required for desktop environments, where UI elements are smaller and context-dependdet. GUI agents trained on large- scale GUT data for UI understanding and grounding tasks perform best, with UI-TARS achieving 25.5% and UGround closely following at 23.2%. However, their overall perfor-
7

---

UI- UI-Vision
Layout Grounding
1.-
Model
IoUt
Precisiont
Recall
0.0
Closed-Source VLMs
GPT-4o (OpenAI, 2024)
20.0
59.6
24.1
Accuracy
0.66
Claude-3.5- (Anthropic, 2024)
22.4
64.3
26.8
0.4
Claude-3.7-Somme (Anthropic, 2024)
17.6
31.5
34.1
Gemini-1.5-po (Team et al,, 2023)
30.8
67.8
36.9
0.2
Gemini-2.0-faas (Team et al., 2023)
28.3
63.0
34.2
Open-Source VLMs
25 50
75 120 12551517772
Qwen-2VL-7B (Wang ct al., 2024)
24.3
65.7
33.4
Area GT Bounding Box (10")
4 of Elements per Screenshot
MiniCPM-V-8B (Yao et al., 2024)
16.3
25.7
43.6
(a)
(b)
Open Source GUI Agents
0.7
CogAgent-9B (Hong et al., 2024b)
6.22
7.99
42.9
0.6
SeeClick-9.61 (Cheng et al., 2024)
5.11
6.32
30.1
0.5
0.4
OSArlas-BB (Wu et al., 2024)
28.2
66.4
41.6
0.3
0.3
Table 4: UI-Vision Leaderboard on Layout Grounding. We present the performance results of state-of-the-art UI understanding models that can predict bounding boxes. Re- sults are averaged across IoU, precision, and recall.
0.2
0.1
15
0.0
50
100
150
200
Area of GT Bounding Box (10%)
Elements per Screenshot
(d)
that screenshot resolution significantly impacts grounding accuracy, our analysis (see Appendix F. reveals that it is not a key factor empirically.
Figure 5: Analysis Element Grounding (UI-TARS-72B
under Basic setting) in terms of (a) Area of GT. Bbox and (b) # Element per screenshot. Layout Grounding (Gemini- 1.5-pro) in terms of (c) Area of GT. Bbox and (d) # Element per screenshot. To enhance clarity, we do not display long- tail examples below 5%.
Error Analysis on Element Grounding. Our analysis highlights three key challenges in model performance (see Appendix D.I). (i) Fine-graine ambiguity: Models often understand the query but struggle distinguish the correct target when multiple similar-looking elements are present (Figure 14). (ii) Lack of domain knowledge: Agents fail to interpret certain symbols, such as the letter "F," due to missing context, suggesting the need for external knowledge sources like software documentation (Figure 15). (iii) Small elements: Models have difficulty recognizing small UI com- ponents (Figure 16), especially in high-resolution interfaces, indicating that strategies like iterative zooming (Wu & Xie, 2024) could improve performance. (iv) Cross-platform gen- eralization: Models incorrectly apply UI layout assump- tions from one platform to another, such as confusing button locations between Windows and iOS systems (Figure 17).
2VL and does not show significant improvement compared to the latter. This might be due to the fact that current GUI agent frameworks focus primarily on element grounding, such as improving click accuracy, rather than layout under- standing and bounding box prediction.
Similar grounding, IoU increases with the ground-truth bounding box area (Figure 5 (c)) and decreases as clement density rises (Figure 5 (d)). Hence, complex and cluttered UI layouts make precise layout generation more challenging and remains the primary bottleneck in layout grounding.
Error Analysis on Layout Grounding. Models exhibit dis- tinct failure patterns in layout grounding (see Appendix D.2). (i) Inaccurate bounding box placement: Closed-source mod- els often fail to return the minimal bounding box for a target region, even when it is contained within their prediction (see Figure 18a, 19a). This suggests weak understanding of lay- out partitioning. (ii) Poor functional grouping: Open-source GUI agents struggle to interpret UI structures at higher level, failing even on queries that explicitly mention rele- vant elements (see Figure 18b). This indicates quite weak understanding of functional groups with coarser granularity maybe due to a lack of such training data. (iii) Superficial semantic matching: When uncertain, open-source models tend to select smaller elements based on partial keyword matches rather than true semantic relevance, leading to in- correct predictions (Figure 19b). These limitations highlight
Performance on Layout Grounding. Table 4 summarizes the evaluation results on Layout Grounding. We only eval- uate models capable of generating bounding boxes across closed-source VLMs, open-source VLMs, and GUI agents. Unlike Element Grounding, VLMs-both open and closed- source-outee outperfrm GUI agents in detecting UI layouts. Closed-source VLMs achieve stateoof-the-ars performance, while open-source VLMs perform comparably. This is likely due to their strong general visual understanding, which aids in semantic comprehension of the UI screen which is neces- sary for this task. Additionally, their training on grounding, even in natural image settings, may contribute to their suc- cess. In contrast, GUI agents perform poorly on this task. Both CogAgent-999 and SeeClick-99 9.66 show subpar results. is the exception, it is trained upon Qwen-
While
8

---

UI- Ul-Vision
Brag Recal
Typlng
Hutley
screen. This approach substantially improves click action recall by 2x 5x across all models. Specifically, GPT--4o achieves 26.7% recall, Claude reaches 26.9%, and Gemini follows with 25.0%, bringing their performance to near par- ity. These results indicate that while closed-source models are good at predicting the action, their main limitation is accurately grounding them within the UI.
Dia..
Dia,
Cerr. T
Cars.
1
Wel Bookses
Raslen Gr5--b iaagst
30
94.
NiA
N/A
Chund NEAD
GPT-tb 1001)
41-3
4-4
614
1-5
X2.1
56.
11.
Gesaee 1.5 (Teanic ot al, 2004)
36.7
13.0
63.1
16
24.7
453
160
Claske5.5 S.e-eec (Aallookk 2024)
41,0
4.8
61.4
E.I
29.0
302
99
Showtt--0 at 30046)
42.5
11.
NIA
104
152
625
151
U1-TARS-I (gje 3025)
19.7
648
318
21.4
Table 5: Vision Action Prediction Leaderboard. We com- pare the performance of three model categories: Closed-Source VL.Ms, Open-Source VL.Ms, and Agentic solutions across four action types-Cllickan Cliek/Mvvee, Drag, Typing, and Hotkey. The eval- uation metrics include Distance (Dist.) and @ecall@d for Click and Drag actions, while Correctness (Corr.) is reported for Typing and Hotkey. Step Success Rate (SSR) represents the overall per- formance of models on the Action Prediction task.
Open-souree GUI agent UI-TARS, achieves the highest over- all performance, particularly in click actions with recall of 19.7% However, these models often fail to follow instruc- tions accurately, likely due to rigid behavior from training on specific instruction formats. For instance, Show fails to generate drag actions entirely, while UI-TARS occasion- ally inserts wait () and fini mnshhh () actions, even when they are not specified in the prompt. This suggests that while open-source models exhibit stronger grounding, their instruction adherence remains a challenge.
the need for better spatial reasoning and structural awareness in GUT agents.
Performance on Action Prediction. Table 5 summarizes the results of the Action Prediction task. The random base- line, where actions and bounding box coordinates are se- lected at random for click and drag actions, performs very poorly, achieving 0% recall for both highlighting the chal- lenging nature of the setup. In all our settings, we provide action histories to the model. To assess its impact, we set up an experiment where only task descriptions and action his- tories are given to GPT-4o without the image (GPT-4o (w/ image) in Table 4). The results suggest that action history only significantly improves performance compared to the random baseline. Notably, the model also performs well for typing and hotkey actions. This may be because prompts explicitly state information like emails or passwords for typ- ing, while hotkeys sometimes follow predictable sequences, such as pressing the 'enter' key after typing.
Error Analysis on Action Prediction. We identify three major sources of errors in the Action Prediction task (see Appendix D.3). (i) Poor grounding: Models often predict the correct action but fail to execute it in the correct loca- tion on the screen, leading to grounding errors. (ii) Lack of platform knowledge: Some models lack platform-speciffe knowledge, causing them to hallucinate or predict incorrect actions when they do not fully understand the UI. (iii Com- plexity of the platforms: Performance tends to be lower on visually dense platforms with many small UI elements, making action prediction more challenging. For example, UI-TARS has the highest error rate (85%) on creativity platforms, where UI elements are tightly packed, whereas education platforms, with simpler layouts, show the lowest error rate (72%).
More Analysis. For further insights into factors affecting model performance on all three tasks, we provide additional experiments in Appendix E, including the impact of screen- shot resolution (Appendix E. cross-software generaliza- tion (Appendix E.2), latency trade-offs (Appendix E.3), and an ablation study on planner vs. grounding contributions (Appendix E.4).
Among closed-source models, Gemini- Semini--155-Pa P.5-Pro outperforms GPT-4o and Souude-3.5-Somee on click and drag actions. Gemini's strong performance in the Layout Grounding task suggests that a high-level understanding of UT structures aids in predicting actions more accurately. In contrast, GPT-4o achieves significantly higher accuracy on keyboard-related actions. However, all models struggle with click and drag actions, indicating inadequate grounding- trend also ob- served in the Element Grounding task. Moreover, poor drag action performance reveals weakness in handling motion- based interactions, likely due to their limited presence in web-based tasks, which dominate training data.
5. Conclusion
We present UI- (i-Visso,, a large-scale GUI benchmark cov- ering 83 desktop applications, making it one of the most extensive and diverse dataset of its kind. We use UI-Vision to build benchmarks supporting three structured evaluation tasks: (i) Element Grounding; (ii) Layout Grounding; and (iii) Action Prediction. Our experiments with state-of-the. art VLMs highlight significant challenges, such as the diffi- culty in grounding UI elements and predicting actions ac- curately. The results underscore key gaps in current model capabilities, emphasizing the need for further research in UI interaction modeling for desktops.
Closed-source models are known for generating effective plans for GUI-related tasks (Lin et al., 2024a). To better un- derstand their performance, we evaluate setup that isolates their planning ability from grounding. Instead of requiring models to predict both an action and its exact coordinates, we prompt them to generate only a textual description of the action (c.g.. "click the file button"). A strong ground- ing model ( (UGround-vl) then maps these actions onto the

---

UI- UIVissoo
Impact Statement
Chen, Z., Wang, W., Tian, H., Ye, S., Gao, Z., Cui, E., Tong, W., Hu, K., Luo, J., Ma, Z., Ma, J., Wang, J., Dong, X., Yan, H., Guo, H., He, C., Shi, B., Jin, Z.. Xu, C., Wang, B., Wei, X., Li, W., Zhang, W., Zhang, B., Cai, P., Wen, L.,, Yan, X., Dou, M., Lu, I.,, Zhu, X., Lu, T., Lin, D., Qiao, Y., Dai, J., and Wang, W. How far are we to gpt-4v? closing the gap to commercial multimodal models with open- suites, 2024. URL https: httpss://rxxvvvvo org/ abs/ /2404 16821.
This research benchmark and analysis offers contributions that are crucial for the advancement of visual-centric UI assistants, specifically targeting agents for Desktop-based software and their applications. The rise of these agents promises to revolutionize how humans engage with digital software and transform the execution of digital tasks by workers. In the following sections, we will discuss both the potential positive and negative societal impacts stemming from this research.
Cheng, K., Sun, Q.. Chu, Y., Xu, F,, Li, Y., Zhang, J., and Wu, Z. Seeclick: Harnessing gui grounding for advanced visual gui agents. arXiv preprint arXiv:2401.
Positive Impacts: Enabling GUI agents automate repet- itive and complex desktop tasks allows users to achieve higher productivity and focus on more creative and strate- gic activities. With more studies and capable GUI agents, software developers can easily finish unit-tests and collect feedback to develop better software or an interface. UI- Vision also provides standardized desktop benchmark to drive innovation in GUI automation and advance the devel- opment of GUI models.
2024.
Dardouri, T., Minkova, L.., Espejel, J. L.., Dahhane, W., and Ettifouri, E. H. Visual grounding for desktop graphical user interfaces. arXiv preprint arXiv:2407.01556, 2024.
Deka, B., Huang, Z., Franzen, C., Hibschman, J., Afergan, D., Li, Y., Nichols, J., and Kumar, R. Rico: mobile app dataset for building data-driven design applications. In Proceedings of the 30th annual ACM symposium on user interface software and technology, pp. 245-854, 2017.
Potential Negative Impacts: While our work advances GUI automation, it also presents several challenges. First, protecting user privacy is crucial when developing, evaluat- ing or deploying GUI agents on personal devices, as these systems may inadvertently expose sensitive information. Second, the high computational cost of vision-language models (VLMs) during inference raises significant envi- ronmental concerns due to increased energy consumption, which must be addressed for sustainable deployment. Lastly, reliance on highly capable GUI agents may lead to user over- dependence, diminishing manual proficiency, and critical problem-solving skills over time.
Deng, X., Gu, Y., Zheng, B., Chen, S., Stevens, S., Wang, B., Sun, H., and Su, Y. Mind2web: Towards general- ist agent for the web. Advances in Neural Information Processing Systems, 36, 2024.
Drouin, A., Gasse, M., Caccia, M., Laradji, I. H., Verme, M. D., Marty, T., Boisvert, I.,, Thakkar, M., Cappart, Q.. Vazquez, D., Chapados, N., and Lacoste, A. Workarena: How capable are web agents at solving common knowl- edge work tasks?, 2024.
Acknowledgments
Gao, D., Ji, L., Zhou, L., Lin, K. Q., Chen, J., Fan, Z., and Shou, M. Z. Assistgpt: A general multi-modal assistant that can plan, execute, inspect, and learn. arXiv preprint D.Xiv:2306086-00 2023.
Authors would like to sincerely thank Ghazwa Darwiche, Christian Hudon, Tom Murray, Ryan Ghiselli, Pierre-Andre, Chao Wang, and Fanny Rancourt for their valuable adminis- trative and technical support. We also thank Megh Thakkar for his valuable feedback on the paper. Furthermore, we acknowledge MITACS for supporting the recruitment of visiting student researchers who played significant role in this project.
Gou, B., Wang, R., Zheng, B., Xie, Y., Chang, C., Shu, Y., Sun, H., and Su, Y. Navigating the digital world as humans do: Universal visual grounding for gui agents. arXiv preprint 2400022410 2024a. URL https: // //axxiv. www.iii.org//aba.. 2520.
Gou, B., Wang, R., Zheng, B., Xie. Y., Chang, C., Shu, Y., Sun, H., and Su, Y. Navigating the digital world as humans do: Universal visual grounding for gui agents. arXiv preprint arXiv:2410.0524 2024b.
References
Anthropic, A. The claude 3 model family: Opus, sonnet, haiku. 3laude-3 Model Card, 2024.
Hao, Q., Cai, R., Pang, Y., and Zhang. L. From one tree to a forest: a unified solution for structured web data extrac- tion. In Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval, pp- 775-784, 2011.
Bonatti, R., Zhao, D., Bonacci, F., Dupont, D., Abdali, S., Li, Y., Lu, Y., Wagle, J., Koishida, K., Bucker, A., et al. Windows agent arena: Evaluating multi-modal os agents at scale, arXiv preprint arXiv:2409.0822 2024.
10

---

UI- UI-Vision
Hong, S., Zhuge, M., Chen, J., Zheng, X., Cheng, Y.,
Li, Y., Li, G., He, L., Zheng, J., Li, H., and Guan, Z. Widget captioning: Generating natural language descrip- tion for mobile user interface elements. arXiv preprint arXiv:2010.04295, 2020b.
Wang, J., Zhang, C., Wang, Z., Yau, S. K. S., Lin, Z Zhou, L.. Ran, C., Xiao, L.,, Wu, C., and Schmidhuber, J. MetaGPT: Meta programming for a multi-agent collabo- rative framework. In The Twelfth International Confer- ence on Learning Representations, 2024a. URL https: //openreview net forum? id=VEMSAGNN7..
Li, Z., You, K., Zhang, H., Feng, D., Agrawal, H., Li, X., Moorthy, M. P. S., Nichols, J., Yang, Y., and Gan, Z. Ferret-ui 2: Mastering universal user interface understand- ing across platforms. arXiv preprint 2024.
Hong, W., Wang, W., Lv, Q., Xu, J., Yu, W., Ji, Wang, Y., Wang, Z., Dong, Y., Ding, M., al. Cogagent: A visual language model for gui agents. arXiv preprint A.Xiv:23/2008// 2023.
Lin, K. Q., Li, L.. Gao, D., Wu, Q. Yan, M., Yang, Z., Wang, L.,, and Shou, M. Z. Videogui: A benchmark for gui automation from instructional videos. arXiv preprint arXiv:2406 10227, 2024a.
Hong. W., Wang, W., Lv, Q., Xu, J., Yu, W., Ji. Wang, Y., Wang. Z., Dong, Y.. Ding, M., et al. Cogagent: A visual language model for gui agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14281-14290, 2024b.
Lin, K. Q.. Li, L., Gao, D., Yang, Z., Wu, S., Bai, Z., Lei, W., Wang, L., and Shou, M. Z. Showui: One vision- language-action model for gui visual agent, 2024b. URL org/ abs / 2411.17466
Hsiao, Y-C.,, Zubach, F., Wang, M., and Chen, Screenqa: Large-scale question-answer pairs mobile app screenshots, 2024.
Lå, X. H., Kasner, Z., and Reddy, Weblinx: Real- world website navigation with multi-turn dialogue. arXiv preprint 2rXiv:2402.059330 2024.
Jansen, B. J. The graphical user interface. ACM SIGCHI Bull., 30:22-26, 1998. URL /sttps://ap1. semant www.iisschhhlr....o wttp:oor///oooooo 18T::::14130000
Lu, Y., Yang. J., Shen, Y., and Awadallah, A. Omniparser for pure vision based gui agent, 2024. URL https: / //arxivv.e wwwp:ii.orrr..oooooo oor.orgg//ao abs 200 00203.
Kapoor, R., Butala, Y. Russak, M., Koh, J. Y., Kamble, K., Alshikh, W., and Salakhutdinov, R. Omniact: A dataset and benchmark for enabling multimodal generalist autonomous agents for desktop and web. arXiv preprint arXiv:2402.17553 2024.
Meta. Introducing meta Ilama 3: The most capable openly available 1lm to date, 2024. URL https: //ai meta. com/blog /oeog//ette a-1lama-3/ Accessed: 2024-04- 18.
Koh, J. Y. Lo, R., Jang, L., Duvvur, V., Lim, M. C., Huang, P.-Y, Neubig, G., Zhou, S., Salakhutdinov, R., and Fried, D. Visualwebarena: Evaluating multimodal agents on re- alistic visual web tasks. arXiv preprint arXiv:2401.13649 2024.
OpenAI. Introducing chatgpt. OpenAl Blog, 09 2021. URL https:/ http://ppaaaa..cc/lloooooooo wtt.ccom///loo. chatgpt.
OpenAI. Gpt-4 technical report, 2023.
Lee, K., Joshi, M., Ture, R., Hu, H., Liu, F., Eisensch- los, J. M., Khandelwal, U., Shaw, P., Chang, M.-W., and Toutanova, K. Pix2struct: Screenshot parsing pretraining for visual language understanding. In Inter- national Conference on Machine Learning, pp. 18893- 18912. PMLR, 2023.
OpenAl. Hello gpt-4o, May 2024. URL https:// // openai. com/ http//inde//eelloooooo pateeelllpppp---e- gpt.4o/. Accessed: 2024-06-06.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., al. Training language models follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27741 2022.
Li, H., Su, J., Chen, Y., Li, Q., and ZHANG, Z.-X. Sheet- copilot: Bringing software productivity to the next level through large language models. In Advances in Neural Information Processing Systems, pp. 4952-4984. Curran Associates, Inc., 2023.
Qin, Y., Ye, Y., Fang. J., Wang, H., Liang, S., Tian, S., Zhang, J., Li, J., Li. Y., Huang, S., Zhong, W., Li, K., Yang, J., Miao, Y., Lin, W., Liu, L.. Jiang, X., Ma, Q.. Li, J., Xiao, X., Cai, K., Li, C., Zheng, Y., Jin, C., Li, C., Zhou, X., Wang, M., Chen, H., Li, Z., Yang, H., Liu, H., Lin, F., Peng, T., Liu, X., and Shi, G. Ui-tars: Pioneering automated gui interaction with native agents. arXiv preprint arXiv:2501.12326 2025. URL https: //github cow/bytedancc//uetttt.cctcc
Li, K., Meng, Z., Lin, H., Luo. Z., Tian, Y., Ma, J., Huang, Z., and Chua, T.-S. Screenspot-pre pro: Gui grounding for professional high-resolution computer use, 2025.
Li, Y., He, J., Zhou, X., Zhang, Y., and Baldridge, J. Map- ping natural language instructions to mobile action sequences. arXiv preprint XXXx:2005.00777 2020a.

---

UI- Vi-Vis
Rawles, C., Li, A., Rodriguez, D., Riva, O., and Lillicrap, T. Android in the wild: A large-scale dataset for android device control. arXiv preprint 2rxi::2307.
Wu, J., Wang, S., Shen, S., Peng. Y.-H., Nichols, J., and Bigham, J. P. Webui: A dataset for enhancing visual ui understanding with web semantics. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pp. 1-14, 2023.
2023. Rodriguez, J., Jian, X., Panigrahi, S. S., Zhang. T., Feizi, A., Puri, A., Kalkunte, A., Savard, F., Masry, A., Nayak, S., al. Bigdocs: An open and permissively- -iceened dataset for training multimodal models document and code tasks. arXiv preprint arXiv:2412. 046666, 2024.
Wu, P. and Xie, S. V Guided visual search as a core mechanism in multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13084-13094, 2024.
Rodriguez, J. A., Agarwal, S., Laradji, I. H., Rodriguez, P. Vazquez, D., Pal, C., and Pedersoli, M. Starvector: Generating scalable vector graphies code from images. arXiv preprint ar arXiv:2312. 1156, 2023.
Wu, Z., Wu, Z., Xu, F., Wang, Y., Sun, Q. Jia, C., Cheng, K., Ding, Z., Chen, L., Liang. P., et al. Os-atlas: A foundation action model for generalist gui agents. arXiv preprint arXiv:2410.232165 2024.
Schick, T., Dwivedi-Yit J., Dessi, R., Raileanu, R., Lomeli, M., Hambro, E., Zettlemoyer, L., Cancedda, N., and Scialom, T. Toolformer: Language models can teach themselves to use tools. Advances Neural Information Processing Systems, 36:68539-6855 2023.
Xie, T., Zhang, D., Chen, J., Li, X., Zhao, S., Cao, R., Hua, T. J., Cheng, Z., Shin, D., Lei, F., et al. Os- world: Benchmarking multimodal agents for open-ended tasks in real computer environments. arXiv preprint arXiv:2404.07977 2024.
Shi, T., Karpathy, A., Fan, L., Hernandez, J., and Liang, P. World of bits: An open-domain platform for web- based agents. In International Conference on Machine Learning, pp. 3135-3144. PMLR, 2017.
Xu, Y., Wang. Z., Wang. J., Lu, D., Xie, T., Saha, A., Sa- hoo, D., Yu, T., and Xiong, C. Aguvis: Unified pure vision agents for autonomous gui interaction, 2024. URL https: /atp:///aaivv.v/aa www.wawwwwwwwwaaeeeeeeeeeee.eeee /ttp://bs//21..... 04454.
Team, G., Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., et al. Gemini: a family of highly capable multimodal models. arXiv preprint ar 2iv: /2..2///2///0.. 2023.
Yan, A., Yang. Z., Zhu, W., Lin, K., Li, L., Wang, J., Yang, J., Zhong, Y., McAuley, J., Gao, J., et al. Gpt-4v in won- derland: Large multimodal models for zero-shot smart- phone gui navigation. arXiv preprint arXiv:2311.07562 2023.
Team, G., Georgiev, P. Lei, V. I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang, S., Mariooryad, S., Ding, Y., Geng, X., Alcober, F., Frostig, R., Omernick, M., Walker, L., Paduraru, C., Sorokin, C., Tacchetti, A., Gaffney, C., Daruki, S., Sercinoglu,
Yang, J., Zhang, H., Li, F., Zou, X., Li, C., and Gao, J. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v, 2023. URL
O., Gleicher, Z., and Others. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403..0 05530, 2024.
/arxiv.
org/abs/22 2210. 11441.
Yang, R., Song, L., Li, Y., Zhao, S., Ge. Y., Li, X., and Shan, Y. Gpt4tools: Teaching large language model to use tools via self-instruction. Advances in Neural Information Processing Systems, 36, 2024a.
Toyama, D., Hamel, P., Gergely, A., Comanici, G., Glaese, A., Ahmed, Z., Jackson, T., Mourad, S., and Precup. D. Androidenv: A reinforcement learning platform for android. arXiv preprint arXiv:2 13231, 2021.
Yang, Y., Wang, Y., Li, D., Luo, Z., Chen, B., Huang, C.. and Li, J. Aria-ui: Visual grounding for gui instructions. arXiv preprint arXiv:2412.16256 2024b.
Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W. Fan, Y., Dang. K., Du, M.. Ren, X., Men, R., Liu, D., Zhou, C., Zhou, J., and Lin, J. Qwen2-vl: Enhancing vision-language model's perception the world at any resolution. arXiv preprint arXiv:2409.122 2024.
Yao, S., Chen, H., Yang, J., and Narasimhan, K. Web- shop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Informa- tion Processing Systems, 35:20744-20757 2022a
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F. Chi, E., Le, Q V., Zhou, D., et al. Chain-of-thougle prompting elicits reasoning in large language models. Advances in neural information processing systems, 11:24824-48871 2022.
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03622 2022b.
12

---

UI- Vision
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, Narasimhan, K., and Cao, Y. React: Synergizing reasoning and act- ing in language models. In International Conference Learning Representations (ICLR), 2023.
Yao, Y., Yu, T., Zhang, A., Wang, C.,, Cui, J., Zhu, H., Cai, T., Li, H., Zhao, W., He, Z.,, al. Minicpm-v: A gpt-4v level milm on your phone, arXiv preprint arXiv:2408.0188 2024.
You, K., Zhang, H., Schoop, E., Weers, F., Swearngin, A., Nichols, J., Yang, Y., and Gan, Z. Ferret-ui: Grounded mobile ui understanding with multimodal llms. arXiv preprint ar arXiv:2404.057/9 2024.
Zhang. J., Wu, J., Teng, Y., Liao, M., Xu, N., Xiao, X., Wei, Z., and Tang, D. Android in the zoo: Chain-of-action- thought for gui agents. arXiv preprint arXiv:2403.02713. 2024.
Zheng, B., Gou, B., Kil, Sun, H.. and Su, Y. Gpt- 4v(ision) is generalist web agent, if grounded. Forty- first International Conference Machine Learning, 2024. URL /ttp:s///ppeeeeieew/ee ner/forum? id-piecKJ2D1B.
Zhou, S., Xu, F. F., Zhu, H., Zhou, X., Lo, R., Sridhar, A., Cheng, X., Bisk, Y., Fried, D., Alon, U., et al. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.1385. 2023.
13

---

UI- Ul-Vision
Appendix
Table of Contents
Page
A. Human Annotation
15
UI- UI-Vision Benchmark Tasks
15
B.1 Element Grounding
15
B.2 Layout Grounding
B.3 Action Prediction
18
C. Dataset Statistics and Examples
19
Error Analysis
D.I Element Grounding
19
D.2 Layout Understanding
21
D.3 Action Prediction
22
E. More Experimental Analysis
22
E.1 Effect Screenshot Resolution on Element Grounding Accuracy
E.2 Cross-software Generalization Analysis
22
E.3 Latency Analysis
22
E.4 Analysis of Planner vs. Grounding Model Contributions
F. Limitation and Future works
25
14

---

UI- VisVis
Category
Software Applications
Education
Anki, Zotero, GrassGIS, Calibre, Audacity, QGIS, OpenBoard, Mendeley
Browsers
Brave, Chromium, Mozilla Firefox, DuckDuckGo
Development
VSCode, Atom, FreeCAD, Eclipse, NetBeans, PyCharm, IntelliJ IDEA, Brackets, Geany, Bluefish, KDevelop, Komodo Edit, Code::llckss QI Creator, Arduino IDE, Spyder, Ubuntu Terminal, Conky, Bash, gedit
Productivity
LibreOffice Calc, LibreOffice Draw, LibreOffice Impress, LibreOffice Writer, draw.io, Joplin, OpenProject, Affine, Zulip, PDFedit, OnlyOffice Calendar, OnlyOffice Document Editor, OnlyOffice Forms, OnlyOffice PDF Forms, OnlyOffice Presentation, OnlyOffice Spread- sheet, Nexteloud, Gnumeric, Simplenote, Cryptomator, WeKan, 7-Zip, GnuCash, Bitwarden. Metabase, Jitsi, Flameshot, Nemo
Creativity
Blender, GIMP, Inkscape, Krita, darktable, FontForge, MuseScore, Scribus, OpenShot, OBS Studio, Lightworks, Shotcut, Natron, Open'Toozz, WordPress
Entertainment
VLC Media Player, Kodi, Element, Signal, Mastodon, Lemmy, Matrix, Emby
Table 6: Mapping from coarse-grain categories to platforms. This table lists the categories and their corresponding platforms used in our study.
A. Human Annotation
We partnered with Turing* (referred to as the "Vendor"), data labeling company that specializes in data curation for AI applications. The annotation process spanned roughly over three three-month periods, with the initial month dedicated to a pilot phase. During this pilot period, we worked closely with the vendor annotation team conducted detailed reviews, and provided extensive feedback to help annotators better grasp the task requirements.
The Vendor annotation team consisted of 70 annotators. This included multi-tiered team comprising annotators, quality assurance specialists, and managers. The majority of the team was based in India and Latin America. The annotators were in the 20-35 year-old age group, with equal distribution of male and female participants. They all had strong proficiency in technical writing and English. All of the annotators hold bachelor's degrees in Engineering, Computer Science, and related disciplines. Annotators also had prior experience in data labeling and UI research.
To ensure high-quality annotations, annotators underwent training process to familiarize themselves with the platforms and the applications. Annotators were paid per hour, and each task took an average of 60 90 minutes to complete, from task creation to quality check. The annotators started with creating computer highlighted in Table 6. These computer use tasks diversity and coverage. This was followed by the execution of computer use tasks and screen recording. A proprietary tool captured action trajectories and logged the precise coordinates of actions as the annotators performed the task. Finally, the annotators used a custom annotation tool to capture screenshots (keyframes) of each user interaction and refine the action trajectories. Each of the captured keyframes was annotated by drawing bounding boxes around all interface elements visible on the screen. The entire process underwent rigorous quality assurance, including review by human annotators and custom annotation evaluation scripts.
tasks for 83 different software applications verified by quality assurance specialists and the authors to ensure
B. UI-Vision Benchmark Tasks
B. Element Grounding
To create a high-quality benchmark, we implement a systematic data workflow to ensure accuracy, diversity, and reliability for evaluating GUI models. The following steps outline this process:
We start with the bounding box annotations obtained during the initial data collection, as highlighted in Section 3.1. Figure 13 illustrates an example screenshot with annotations. Given the large number of annotations, we sample a subset for the
hhtpss/wwww.turingcoom
15

---

UI-Vis Vi-Vision
Synthetic Predictions of SOTA Multimodal Models
Post Processing Deduplication Based on loU and Annotation
Curated OUI-Vision Grounding Datasets
Human Verification of Annotations
Source Raw Ul Data
UI- UI-Vision's Element Grounding Data Curation Workflow. Step-by-step process from sourcing raw UI data to creating curated datasets. The workflow includes synthetic predictions using state-of-the-art (SOTA) multimodal models, post-processing for deduplication, and human verification to ensure high-quality annotations.
Figure 6:
benchmark to ensure comprehensive platform coverage while maintaining case of evaluation.
Our analysis of state-of-the-ar models like UI-TARS (Qin et al., 2025) reveals that they perform well on naive data samples, particularly those with textual information, but struggle significantly with icon-based elements and smaller Ul components. Additionally, models show lower performance on platforms with high bounding box ratio, such as those the creativity category like Blender and GIMP. To address these issues, we sample bounding boxes from different image resolutions and sizes, with focus smaller screen elements.
To maximize diversity, apply an Intersection Union (IoU) threshold within each platform, filtering redundant samples with similar functionality spatial positioning. This ensures that the selected samples vary in function and placement within the UI. After the initial sampling, we use an ensemble of four cop-performing GUI models (c.g., (Qin et al., 2025; Gou et al., 2024b; Yang et al., 2024b)) to identify additional challenging cases. Samples are classified as hard (where all models fail or predictions disagree) or easy (where least one model provides a correct prediction), ensuring a balanced difficulty distribution. Reviewers verify uncertain instances by consulting documentation or online sources.
To expand the evaluation beyond basic queries, we introduce variations assess model capabilities in functional and spatial settings. This approach allows to reuse existing annotations while benchmarking different aspects of GUI understanding.
For the functional setting, we use each element's annotation and corresponding screenshot (with the bounding box highlighted) as input to GPT-4o to generate function-based descriptions. These descriptions reflect how the element is used rather than just its label.
Prompt used with GPT-4o generate annotation for Element Grounding with functional setting
You are an assistant tasked with generating a single, precise functional annotation for a UI element based on its basic textual description. Your job is to:
1. Ensure the functional annotation is accurate, straight and directly aligns with the provided description and bounding box context before adding any complexity or diversity.
2. Focus on clarity and correctness first, avoiding ambiguous or overly complex annotations.
3. Use the textual description and bounding box context to infer the most likely purpose or role the element.
Try best not to directly copy the textual description, but rather provide functional interpretation based the context.
Below is the input for generating functional annotation:
1. Screenshot: Please refer to the provided image. The UI element is
highlighted by red bounding box.
2. Description of the UI element (basic textual content): "(annotation for basic setting)".
16

---

UI- Vi-Vision
Ensure the annotation meets the following requirements:
It directly reflects the UI element's functionality as described in the textual content.
It is clear, concise, and free of unnecessary complexity.
It avoids speculative ambiguous statements.
- It clearly reflects how the user interacts with the UI element.
Provide only the JSON output without additional explanations.
For the spatial setting, we iterate over four directions- [up, bottom, left, right]- generate queries about the closest neighboring element. For example, given an element labeled "previous annotation options" corresponding query might be: "What is the element that is inmediately prevvious annotation options' hori zortaaltally We retrieve the correct bounding box for the queried element using the bounding boxes collected during initial data creation (Section 3.1). To ensure accuracy and reduce ambiguity, we remove samples where no valid closest element exists or where the distance exceeds set threshold. The processing scripts will be released alongside the code for reproducibility.
the right
B.2. Layout Grounding
GUI layouts define how interface components, such as buttons, text fields, and containers, are arranged into cohesive regions (e.g., "Formatting Tools" or "Main Navigation Bar"). In our layout grounding task, we start with raw UI element annotations collected during the initial data collection phase, which include dense bounding box information for individual elements. These annotations are then fed into 1...A--3.3-000 (Meta, 2024) using a predefined prompt (provided below) that instructs the model to cluster UI elements into non-overlapping functional and semantic groups. For each group, the model generates corresponding textual description and computes an aggregated bounding box that encompasses all the elements within that group, thereby capturing the higher-level structure of the GUI.
Following the automated model inference, the predicted clusters and bounding boxes are subjected to rigorous human verification process. Expert annotators in our research team review and validate each functional group to ensure that the clusters accurately represent the intended UI regions and that the aggregated bounding boxes fully cover the grouped elements. Specifically, we remove samples that do not meet the requirements and retain those that do. This two-stage procedure yields a high-quality dataset of 311 human-verified query-label pairs spanning platforms, with each functional group typically containing 5-10 UI elements. The process thus extends individual element grounding layout grounding task, emphasizing both spatial organization and semantic coherence in desktop interfaces. To the best of our knowledge, this is the first work to introduce this setting and systematically evaluate GUI agents in this context.
comprehensive
Prompt used with LLAMA-3.3-700 to generate samples layout grounding
Below is list of dictionaries representing UT elements for the software (software n'me)'. Please use the knowledge you have on this software to analyze these elements and group them based on their functionality.
Your task is to:
1. Analyze each element for text, category, and boundingBox attribute.
2. Identify the elements that belong to a particular functional group (e.g.,
all the file edit tools). Name as many as you possibly can, but make sure the functional groups make sense.
Combine the bounding boxes of all elements belonging to that functional group into single bounding box that encompasses them all.
3.
4.
Return structured response (JSON or similarly structured text) that includes: The name of the group (e.g., "Edit Tools Region").
- An ioplanation of the group's functionality. Please be as specific as
17

---

UI- VisVos
possible but do not simply repeat the names of the UI elements in the group. The coarse bounding box (x1, yl, x2, y2, etc.).
Here is the list of dictionaries you should work with: (raw data for the given screenshot]
Please produce your final answer in structured JSON-like format, such as:
[
"name": "Edit Tools Region", [1ex] "explanation": "This region contains all the tools related to editing the files.",
"boundingBox": M
"x1" :
"yl ":
"x2 :
"y2"
V
V,
Some additional requirements:
1. Please try your best to make functional groups that cover all the elements
in the list.
2. Please make the functional group not overlap with each other. Try to prioritize non-overlapping bounding boxes containing more elements.
3. Make sure the name of each group is descriptive as well as meaningful to the human user and relevant to the elements it contains. Please make sure the name is not too generic.
4. Make the bounding box complete, based the bounding boxes of the elements it contains.
Make to provide the accurate bounding box that encloses all relevant elements for each functioral group. And return only this JSON object.
B.3. Action Prediction
We begin with the raw action trajectories collected and annotated by humans during the initial data collection stage. Table 7 lists all recorded actions. Our analysis revealed that current GUI models are limited their ability predict wide range of actions, as they are typically trained on restricted set. To simplify evaluation, we standardized actions into five categories: click, move to, drag, typing, and hotkey. These actions common across all models. We excluded tasks involving key down, key up, and scroll actions, reducing the total number of tasks to 442. Additionally, we applied the following preprocessing steps to refine the action data:
1. Removing Redundant Move Actions: a "move to" action directly preceded "click" action, we removed it. Since the "click" action already contains positional data, preceding movement was unnecessary.
2. Grouping Drag Actions: A drag action always preceded by a mouse-down event and followed by a mouse-up event. We merged these three actions into a single "drag" action. Additionally, since a drag operation requires moving the cursor to the starting position, we included this movement, resulting in standardized format: drag(x1, y1, x2, y2) where x1, yl is the start of the drag and x2, y2 the end coordinates of the drag.
3. Merging Press and Hotkey Actions: In the raw trajectories, "press" action referred pressing single key, whereas
18

---

UI- Ul-Vision
Action Type
Parameters
Deseription
MOVE_TO CLICK
x, y
Move the cursor to the specified position
button,
Click the mouse button at a specified location
y, num_cliccs
HOUSELDOWT
button
Press and hold the specified mouse button
MOUSE UP
button
Release the specified mouse button
DRAG_TO
x, y
Drag the cursor to the specified position with the left button pressed
SCROLL
dx, dy
Scroll the mouse wheel up down
TYPING
text
Type the specified text
KEY KEY_DOWN
key
Press and hold the specified key
KEY-UP
key Release the specified key
HOTKEY
key/key
Press the specified key or key combination
Table 7: Table of original Mouse and Keyboard Actions during human demonstration.
"hotkey" involved multiple keys. We merged both into single "hotkey" category maintain consistency.
C. Data Statistics and Examples
Data Statistics. Figure. 12 presents key statistical distributions within the UI-Vision. (a) Shows the distribution of bounding box number per screenshot, indicating mean of 74.3, reflecting the dense UI elements present in each frame. (b) Illustrates the distribution of human recording durations in UI- Vision,, with a mean of 38. seconds, highlighting the variation in the GUI navigation task. (c) Depicts the distribution of the number of action steps required to complete tasks, with mean of 14.0, indicating the long-horizon challenges and interaction complexity within the dataset. (d) Shows the distribution of different screenshot resolutions in our dataset. We have more than 100 screenshot resolutions offering rich diversity for benchmarking.
Visualization. Figure 7 to 11 illustrate concrete examples highlighting different aspects and challenges of UI- IL-Viion'ss 's tasks across multiple software applications. In Figure 7, we demonstrate examples from the Element Grounding task with the basic setting, showing typical UI elements identified by straightforward textual descriptions. Models must accurately localize UI components based solely on labels like "crop tool" or "skip all breakpoints". Figure presents cases from Element Grounding task with the functional setting. emphasizing the need for models understand UI elements based on their functionality rather than simple visual labels, such as "Remove data series from the chart" "Open the DoxyBlocks plugin menu". Figure provides challenging examples from the Element Grounding task with spatial setting, requiring models to precisely interpret spatial relations among UI components, such as elements directly adjacent positioned relative to specific reference elements. Lastly, Figure 10 demonstrates examples from Layout Grounding, showcasing how models must group UI components into meaningful semantic regions, like identifying comprehensive areas for menu options, editing tools, or formatting tool clusters. Figure shows an example of Action Prediction task along with the task and raw actions collected.
D. Error Analysis
D.1. Element Grounding
Below, we present several failure cases predicted by the top-performing UI-TARS model (Qin et al., 2025) to examine the limitations of current models. We highlight the ground truth with a red box. The prediction of the model is indicated by a red arrow surrounding it.
A. Fine-grained ambiguity: As shown in Fig. 14, the agent effectively understands the query and primarily responds the
19

---

UI- Vision
(b) Query: "constrain are or cirsle", The ground truth element bounded by red bounding box. The platform of the example is FreeCAD.
(a) Query: "crop tool". The ground truth element is bounded by red bounding box. The platform of the example is GIMP.
(c) Query: "skip all breakpoints". The ground truth element is bounded by red bounding box. The platform of the example is Eclipse.
(d) Query: "code intelligence", The ground truth element is bounded by red bounding box. The platform of the example is Komodo Edit.
Figure 7: Examples of Element Grounding with basic setting. The ground truth element is bounded by red bounding box. We demonstrate examples from the Element Grounding task with the basic setting, showing typical UI elements identified by straightforward textual descriptions. Models must accurately localize UI components based solely on labels like "crop tool" or "skip all breakpoints".
nearby region but struggles to recognize the correct target among several similar candidates. This highlights the need for existing agents to develop advanced validation strategies.
B. Lack of domain knowledge: In Fig. 15, we observe that the model's difficulty in identifying the most efficient approach to task completion underscores a deficiency in domain-specific knowledge, such as understanding what the letter "F" signifies. This challenge prompts us to consider developing solutions based on Retrieval-Augmented Generation (RAG) or integrating external databases, such as software documentation or tutorials, to enhance the model's performance.
C. Small elements: 1ig. we find that the model struggles with small visual elements, especially in an interface with high resolution, and more importantly, dense distribution of UI elements. To resolve this issue, one potential strategy is to develop an iterative zoom-in strategy as V-Search (Wu & Xie, 2024).
D.
The model sometimes incorrectly transfers layout assumptions across platform. As in Fig. 17, the query given to the model mimmiee window. The model fails to generalize to the "minimize" button for IOS system and points to the place where "minimize" button is located in the Windows operating system. We highlight the ground truth with red box. The prediction of the model indicated by red arrow surrounding it.
generalization:
20

---

UI- Visiin
(a) Query: "Remove d series from the chart". The ground truth element is bounded by bounding box. The platform of the example is LibreOffice Draw.
(b) Query: "Create Java package". The ground truth element is bounded by red bounding box. The platform of the example is Eclipse.
Clos
Shotuen Tou ghote Potrooee: Tychotben Tenteheccome Powe Gone
(c) Query: "Adjusts path curvature by pinching vector points". The ground truth element is bounded by red bounding box. The platform of the example is Open'oonzz
(d) Query: "Open the DoxyBlock ocks plugin menu". The ground truth element is bounded by red bounding box. The plat- form of the example is Code:Blocks.
Figure 8: Examples of Element Grounding with functional setting. The ground truth element bounded by red bounding box. We present cases from the Element Grounding task with the functional setting, emphasizing the need for models to understand UI elements based on their functionality rather than simple visual labels, such as "Remove data series from the chart" or "Open the DoxyBlocks plugin menu".
D.2. Layout Grounding
Below, we present some failure cases predicted by the top-performing closed-source model Gemini-1.5-pp 1.5-pr (Team et al., 2024) and open-source GUI agent OSAtlas-BB (Wu et al., 2024) to examine the limitations of both types of models as their error patterns are not identical. We highlight the prediction of the model with a blue box. The ground truth is indicated by a red arrow surrounding it.
A. Inaccurate bounding box placement: As shown in Figure 18a and Figure 19a, it is quite typical for closed-soures models to fail to return the minimal bounding box of the ground truth region, though it is usually contained in the predicted box. This indicates models can not understand the partition of the layout well.
B. Poor functional grouping: As shown in Figure 18b, open-source GUI agents sometimes struggle even with a very specific query that explicitly mentions the names of almost all the elements. Usually it should not be very difficult for the model to ground those elements one at time in the basic Element Grounding setting. This indicates quite a weak understanding of functional groups with coarser granularity. Instead, they may rely solely learning the correspondence between element's label and its position. This limitation is likely due to the lack of training data that enables models to perceive interfaces at higher and more generalized level.
C. Superficial semantic matching: As shown in Figure 19b, when an open-source agent lacks sufficient knowledge or confidence in locating the requested region, it sometimes predicts only one or two items whose labels share words with the query but are not semantically equivalent. the example, the predicted item labeled "Design" appears in the query but an
21

---

UI-Vis UI-Vision
incorrect selection. This suggests that open-source GUI agents require deeper semantic understanding of interface layouts beyond simple keyword matching and grounding.
D.3. Action Prediction
We analyze UI-TARS (Qin al., 2025), the best-performing open-source model, and Gemini 1.5-Pro (Team et al., 2024), the highest-performing closed-source model, to identify common failure cases in Action Prediction prediction. Our analysis highlights three major sources of errors that impact model performance:
A. Poor Grounding: As discussed in Section 4.3, both open-souree and closed-source models struggle with accurately grounding actions. In many cases, models correctly predict the intended action but fail to execute it on the correct UI element. Figure 20 illustrates an example where Gemini-1.5 P.5-Pro identifies the right action but applies it to the wrong location, leading to failed execution.
B. Lack of Platform Knowledge: Each platform has unique UI elements and interactions. To perform tasks effectively, models need an understanding of platform-specifie keywords and the major operations each platform supports. Our analysis shows that models often lack this knowledge, likely due to insufficient exposure to diverse desktop environments during training. As a result, models sometimes hallucinate actions or generate nonsensical predictions. Figures 21 and 22 showcase cases where the models' limited platform knowledge leads to incorrect or implausible actions.
C. Complexity of the Platforms: Models struggle more with platforms that have dense and intricate UI layouts. Smaller Ul elements and tightly packed interfaces make it harder for models perceive and interact accurately, Additionally, platforms with extensive functionalities allow users to perform complex workflows, increasing the difficulty of predicting appropriate actions. This challenge is reflected in UI-TARS' highest error rate (85%) creativity platforms, where elements are densely packed, while education platforms, with simpler layouts, exhibit the lowest error rate (72%).
E. More Experimental Analysis
E.1. Effect of Screenshot Resolution on Element Grounding Accuracy
Figure 23 presents the grounding accuracy of the two best-perforning models, UI-TARS-72B and UGround-v1-72B: across different sereenshot areas (i.e., resolutions). Observing the accuracy distributions, there is no clear trend that indicates a strong correlation between the resolution of the screen and the grounding accuracy. The accuracy values fluctuate across different resolution levels for both models, suggesting that factors other than resolution may have a more significant impact on performance.
E.2. Cross-software Generalization Analysis
We compare model performance common apps (e.g., VSCode) vs. less common ones (e.g., FreeCAD, QGIS) using the element grounding subset in R1. Results are included in Table 8. While we cannot confirm the exact training data used in several models, this serves as proxy for generalization analysis. We observe that all models show significant accuracy drops on less common apps, confirming consistent generalization challenges.
E.3. Latency Analysis
We report latency per query, average output tokens, and GPU usage across all three tasks using default Hugging Face implementations for consistency. Table 9, 10 and 11 for Element Grounding.Layout Grounding and Action Prediction, respectively. Token efficiency was measured with GPT-4 tokenization. Models like UI-TARS, trained on action-heavy tasks, generate longer outputs due to detailed step-by-step reasoning.
E.4. Analysis Planner vs. Grounding Model Contributions
Our experiments demonstrate a significant improvement in recall when large language model planners are combined with grounding models, gain we attribute primarily to better action selection since the grounding model only supplies coordinates. We further analyzed error rate reductions across platform categories (Fig. 24), finding that productivity tools experienced the largest decrease (26%) despite entertainment tools having higher baseline grounding ecuuacy--uderscoriinn the planner's whereas creativity platforms saw the smallest gain (14%), highlighting challenges in planning and grounding
22

---

UI- UI-Vision
Basic
Functional
Spatial
Model
Acc (Common) (206)
Acc (Ranc) (173)
Delta (3)
Overall (1772)
Ace (Common) (205)
Ace (Ranc)
Delia (5)
Owalll (1772)
Acc (Commoni)
Ace (Ranc)
Delta (8)
Overal (1935)
(173)
(289)
(156)
Cloved-Source Models
Claude-.S-Somme
100.00
2.56
Cladde3.7-Somee
3.47
62.53
8.78
1.73
$0.30
7.73
1.92
72.25
7.60
Open-Source VI.Ms
Qwen2VL-7B
3.41
0.58
$2.99
3.44
2.44
0.00
100.00
3.22
0.69
0.64
7.25
1.45
MiniCPM-V.SB
6.34
4.62
27.13
7.11
3.41
2.31
32.26
5.30
0.35
0.00
100.00
1.45
Open-Source GUT Agents
ShewUL-2B
7.80
5.20
33.33
8.07
8.78
5.20
40.77
7.67
2.08
1.28
38.46
2.07
AriAU1-25.31
13,70
9.82
28.32
12.20
13.20
8.09
38.71
14,00
4.50
3.20
28.89
3.98
OSAtlas-BB
11.70
6.93
40.77
12.20
9.76
8.09
17.11
11.20
3.11
3.85
23.779
3.67
U-Grund-v1-7B
15.10
10.40
31.13
15.40
16.10
13.30
17.39
17.10
6.23
5.13
17.66
6.25
Aguvis-78
15.60
10.40
33.33
17,80
17.10
10.40
39.18
18.30
3,46
3.21
7.23
5.06
UI-TARS-7B
20.50
11.60
43,41
20.10
23.40
15.60
33.33
24.30
8,30
5.13
38.19
8.37
C9gAgem24-98
19.50
6.36
67.38
12.00
16.10
4.05
74.84
12.20
6.23
0.64
89.73
2.63
UKGround-11-22
26.80
17.90
33.21
27.90
26.80
14.50
45.90
26.70
15.90
12.80
19.50
14.90
UI-TARS-72B
34,60
20.80
39.88
31.40
33.70
18.50
45.10
30.50
16.60
11.50
30.72
14.70
Table 8: Cross-software generalization analysis for Element Grounding. Since directly retraining models for evaluation beyond the scope of this study, we approximate cross-software generalization by comparing model performance on common apps (e.g., VSCode, draw.io, OnlyOffice suite) versus rare apps (e.g. FreeCAD, QGIS, Qu Creator, darktable, Mastodon), the latter presumably unseen during training. We report accuracy scores on these subsets along with the relative accuracy degradation (delta percentage). Positive deltas indicate better performance common apps, reflecting drop when generalized to rare apps. We also add the overall results in Table 3 reference. From the results, all models show significant accuracy drops on less common apps, confirming consistent generalization challenge. Notably. larger and extensively trained models (e.g., UI-TARS-722 -7BB) exhibit higher overall accuracy but still encounter noticeable performance reductions on rare apps, highlighting the challenges generalization in practical scenarios. Models are categorized by size for clarity: gray (closed-source models), green (open-source VLMs), and blue orange for open-source GUT Agents below/above 8B (active) parameters, respectively.
23

---

UI-V VisVis
Model
Avg Latency per Query (s)
Avg # Output Tokens
# GPUs (H100)
Closed-Source Models (API-based)
Claude-3.5-Sonee
8.28
Claude-3.7 3.ndd--37-SSonet
47.5
Open-Source VLMs
Qwen2VL-7B
1.36
116.1
1
MiniCPM-V-8B
0.84
28.6
Open-Source GUI Agents
ShowUl-2B
4.73
25.9
AriaU1-25.3B
2.38
14.0
4
UI-TARS-7B
4.97
66.1
2
Aguvis-7B
1.38
18.3
1
OSAtlas-7B
0.76
40.6
1
UGround-77
0.60
6.42
CogAgent24-9B
6.14
173.2
1
UI-TARS-72B
12.1
57.6
4
Table Efficiency metrics for the Element Grounding task under the Basic setting. Latency measurements are based on default Hugging Face implementations via the Transformers library, ensuring fair comparison (not applicable closed- source models). Output tokens are counted using the GPT-4 tokenizer. Models are grouped into three categories: gray
for closed-source models (APl-based), green for open-source VLMss, and blue / orange for open-source GUT Agents below/above 8B (active) parameters, respectively.
Model
Avg Latency per Query (s)
Avg # Output Tokens
# GPUs (H100)
Closed-Source Models (API-based)
GPT-40
33.9
Claude-3.5-Sonne
166.2
Gemini-1.5-pro
20.1
Open-Source VLMs
MiniCPM-V-SR
0.66
18.8
Qwcn2VL-7B
0.95
25.7
Open-Source GUI Agents
OSAtlas-7B
0.95
39.8
CogAgen(24-9B
6.71
192.1
Table 10: Efficiency metrics for the Layout Grounding task. Latency measurements based default Hugging Face implementations via the Transformers library, providing fair comparison across models (not applicable to closed-source models), Output tokens are counted using the GPT-4 tokenizer. Models are grouped into three categories: gray for closed-
source models (API-based), green for open-source VIMMs, and blue orange for open-source GUT Agents below/above 8B parameters, respectively.
24

---

UI-Vis UI-Visoon
Model
Avg Latency per Query (s)
Avg # Output Tokens
# GPUs (H100)
Closed-Source Models (API-based)
GPT-4o
59.6
Claude-3.5-Sonnct
65.3
Gemini-1.5-pro
55,9
Open-Source GUI Agents
ShowUl-2B
0.6
25.3
UI-TARS-7B
1.7
58.0
Table 11: Efficiency metrics for the Action Prediction task. Latency measurements are based on default Hugging Face implementations via the Transformers library, providing fair comparison across models (not applicable closed-source models). Output tokens are counted using the GPT-4 tokenizer. Models are grouped into two categories: gray for closed-source models (API-based), and blue for open-source GUI Agents. We use 2 in implementation for all open-source models.
within functionally dense interfaces featuring small UI elements.
F. Limitation and Future works
Our benchmark tasks evaluate models an offline setting. However, expanding an online environment would enable comprehensive assessment of real-world interactions. We currently use heuristic sampling methods select challenging UI elements for the benchmark. Future work should explore improved strategies to ensure representative UI elements are sampled for evaluating agentic abilities. In the action task, we rely single human demonstration, which may not fully capture the range of possible user interactions. Since tasks can often be completed in multiple ways, single demonstration might not provide a complete assessment of model capabilities. Expanding the dataset to include multiple action trajectories per task would offer a more robust evaluation and reduce potential biases. Additionally, we have not yet incorporated hucondrecorded videos, which are crucial for assessing GUI action understanding. Providing video clips and querying models about possible actions could offer deeper insights into their reasoning and decision- makiin processes. Finally, our benchmark does not explicitly evaluate combined actions involving both mouse and keyboard inputs, such as holding the Control key while dragging an item. Since most models have limited exposure to such interactions, developing reliable evaluation methods for these complex actions remains an open challenge.
25

---

UI- UI-Vision
(h) Query: "What the element that is vertica 11y closest to the "change case" and above it?". The element in the blue box is the reference one referring to "medium eraser" while the red one is the ground truth. The platform of the example is OnlyOffice Document Editor.
(a) Query: "Di rectly vertically, 1y below the "baseSize", can you find the closest
element?". The element in the blue box is the refer- ence referring "medium eraser" while the red one is the ground truth. The platform of the example is Creator.
Egrabl
"What is the element that is vertically closest to the "Activate fill tool" and above it?". The element in the blue box is the reference referring "medium eraser" while the red one is the ground truth. The platform of the example Darktable.
(c) Query:
(d) Query: "What the element that is horizontally closest to the "Image Frame" and to the right of it". The element in the blue box is the reference referring "medium eraser" while the red the ground truth. The platform of the example is Scribus.
Figure 9: Examples of Element Grounding with spatial setting. The ground truth element bounded by red bounding box. We provide challenging examples from the Element Grounding task with spatial setting, requiring models to precisely interpret spatial relations among UI components, such elements directly adjacent positioned relative to specific reference elements.
26

---

UI- UI-Vision
Firefox
"shortcut Region: this region contains shortcuts to varicus actions and features such bookmarks, history, and settings. The red the ground truth. The platform of the example is Mozilla Firefox.
(h) Query:
(a) Query: "Menu Bar: this region contai nas the main menu options for the opplication, Ancluding file, edit, view, and The red the ground truth. platform the example is Krita.
N
DS
(c) Query:
"Cover Management Region: this
regien centains all the tools and options related to managing book covers, such changing, removing, and generating covers." The red is the ground truth. The platform the example Calibre.
(d) Query: "Top Menu Bar: this region contains the main menu options for the application, including Media, Playback, Audio, Video, and Help. " The red the ground truth. The platform the example is VLC Media Player.
Figure 10: Examples of Layout Grounding. The ground truth element is bounded by red bounding box. We demonstrate examples from Layout Grounding, showcasing how models must group UI components into meaningful semantic regions, like identifying comprehensive areas for menu options, editing tools, or formatting tool clusters.
27

---

UI- UI-Vision
"Apply transparency of 45.9 to the layer"
CKA
C
GGIS
RGIS
RGIS
the satellite the location
Properties
Transpanncy the location [275, 257]
Move the Slider (660,
the location
[122, 716)
[310, 70419]
Hold
Drag
QalS
C
Click
QGIS
5
Hold dosn the left mouse bution
6
Drag curser to [487, 196]
Release the left mouse button
Left chick on the OK the location
Figure 11: The figure illustrates the raw action trajectories recorded for a computer use task in QGIS software. To simplify evaluation, certain actions, such as drag, are grouped with related actions like mouse up and mouse down, while redundant actions, such as move, are removed. The recorded data consists only of action coordinates, which have been translated into descriptive terms for clarity.
0.025
Median:
Median:
0.020
0.008
Density
Density
0.015
0.006
0.004
0.002
100
200
300
400
50
100
150
200
Number of bounding box notaations per screenshot
Duration of the video in seconds
(a) Distribution of bounding boxes
(b) Video duration distribution
0.08
0.07
Median: 13.0
omner resolution
0.06
0.05
0.04
1900 1280 99)
ADD 1322 ANJ
720
0.03
200 $231
768 7RI
0.02
10
20
3.
40
50
60
70
80
Number of steps to complete the task
(c) Action steps distribution
Distribution of different screenshot resolutions
Figure 12: Dataset Statistics. (a) Distribution the number of bounding boxes present per keyframe. (b) Distribution of
video durations UI-Vision. (c) Distribution of the number of actions required to complete the task.
28

---

UI- Vision
BB M
Figure 13: Examples of raw annotation screenshot in Uisiiso,, is a representative of dense and complete element
coverage of bounding box annotations.
drroppod Imeget
Nomsol grdda
Figure 14: Typical error case (A) of Element Grounding with basic setting. The query given to the model is hori i zont all grids. The model fails identify the fine -raiied differences in those cases. We highlight the ground truth with red box. The prediction of the model indicated by a red arrow surrounding it. The ground truth element is cropped and attached below the screenshot of each interface.
29

---

UI- UI-Vision
Do
Gronn Truth Strepped Inscer
F
Nams
Figure 15: Typical error case (B) of Element Grounding with basic setting. The query given to the model insert font work text. The model shows the obvious lack of domain knowledge of the software/platform in question. We highlight the ground truth with red box. The prediction of the model is indicated by red arrow surrounding it. The ground truth element is cropped and attached below the screenshot of each interface.
30

---

UI- UI-Vision
Ground Truth Bemmen jcropped imagel
Figure 16: Typical error case (C) of Element Grounding with basic setting. The query given to the model is Ca Ca Calligraphy. The model fails to locate and understand small items within interface with a dense distribution of UI elements. We highlight the ground truth with red box. The prediction of the model is indicated by a red arrow surrounding it. The ground truth element is cropped and attached below the screenshot of each interface.
ArT Artilttiii LDc
Staecc
Help
You DUBAC 6515
setuol)
your here,
lospt1
wttp//tttteeeeee wwwdeedde heray Pat ryp
Figure 17: Typical error case (D) of Element Grounding with basic setting. The query given to the model is minimi ze window. The model fails to generalize to the "minimize" button for IOS system and points the place where the "minimize" button is located in the Windows operating system. We highlight the ground truth with red box. The prediction of the model is indicated by red arrow surrounding it.
31

---

UI- Vi-Vision
(a) Case from Gemini- 1.5-pro
Case from OSAtlas-7B
Figure 18: Layout Grounding error case. Query: "Menu Bar: this region contains the main menu options for the application, including file, edit, view, and more. " The red is the ground truth. blue box the model prediction. The platform of the example LibreOffice Impress.
(a) Case from Gemini-1.5-pro
(b) Case from OSAtlas-7B
Figure 19:
Layout Grounding error case. Query: "Widget Toolbox: This region contains collection of widgets that can be used to design and build user interfaces, including buttons, sliders, and input fields." " The red is the ground truth. blue box is the model prediction. The platform the example is Qt Creator.
32

---

UI- Vi-Vision
Welcome Qt Creator
10 Aiiiion
Data apmn
Trijeta
bebugge
Drikka 1hibad
bela
Dettal bark
Scn matrocss
Ondtra CONTE
Hinae Peer Acdlontenooee Comole Open Nenie
Figure 20: Gemini was instructed to "Increase the parallel job count to 5 in analyzer settings." " At this step, it correctly predicts the action: Click on Analyzer in the left sidebar menu to open the settings. However, as shown by the yellow arrow, fails to ground the action in the correct location, clicking on an incorrect element instead. This highlights a common issue where models generate accurate plans but struggle with precise execution.
Muant
Figure 21: Gemini was instructed to "In the Sketcher workbench of FreeCAD, set the orientation offset to 10, and draw a 2D ellipse." However, instead clicking the Sketcher button (indicated by the arrows in the figure), predicts the action "Click on the new sketch button to create new sketch." " Since sketch is already present, this action is unnecessary and incorrect. This suggests that Gemini may lack knowledge the Sketcher icon and its intended functionality.
33

---

UI- Vi-Vision
Welcome to Qt Creator
X Inti
T5
12
Obie
Loboe
Mama
Diarke pursad
Defta Sarting
Dr Cuice
Iusue Seenthheettt Cedlettenrrrdd Conde Dde
Figure 22: UI-TARS was instructed to "Enable timeout for 60 seconds for testing." Instead of dragging through the dialog box's left menu to locate the Timeout setting, it predicts the action click on the "Manage" button. Although the action is accurately grounded (yellow arrows on the screen). it is incorrect because the button in the background and inaccessible without closing the dialog. This suggests that UI-TARS may lack familiarity with the Qt Creator platform and does not recognize the need scroll within the dialog find the testing option.
1.0
1.0
0.8
0.8
Accuracy
0.6
Accuracy
0.6
0.4
0.4
0.2
0.2
0.0
0.0
1
2
5
1
4
5
Area of the Screenshot (10%)
Area of the Screenshot (106)
(a) UI-Tars-72B
(b) UGround-v1-728
Figure 23: Analysis Element Grounding accuracy on top two performing models in terms of the of screenshot (i.e. resolution of screenshot).
34

---

UI- UI-Vision
Decrease in Error with Grounding Model by Category
Productivity
Education
Entertainment
Browsers
Development
Creativity
Decrease in Emor (%)
Figure 24: Percentage decrease in error when using GPT-4o as a planner, paired with UGround-v1-72 for grounding the predicted action and target UI element.
35


Reviews:
[{"review_id": "eQB0JdGKve", "reviewer": "Reviewer_3NZB", "summary": "This paper introduces a desktop GUI benchmark (i.e., UI-Vision) that spans 83 real-world environments with open-source and permissive data. It enables three key tasks evaluation, including element grounding, layout grounding, and action prediction. The evaluation reveals the limitations of existing works to handle desktop environments.\n\n## update after rebuttal\n\nI appreciate the authors' clarifications. My main concerns have been addressed by the rebuttal. I would lean to accept the paper by involving the additional discussions in the revised version.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing the value of our desktop-centric benchmark, its utility for comprehensive analysis of GUI agents, and the clarity of the paper. Below, we address the concerns raised.\n\n**R1: Clarification on “largest desktop-centric benchmark” claim**\n\nWe appreciate the reviewer’s point and agree that “largest” can be interpreted in different ways. While OmniAct includes 9802 samples (7639 desktop-related), it focuses solely on action prediction. In contrast, our benchmark includes 6484 samples across 83 applications and supports three tasks—action prediction, layout and element grounding—with dense frame-level UI annotations. Considering the range of platforms, tasks, and annotation details, we believe UI-Vision is the most comprehensive desktop GUI benchmark, and we are happy to revise the claim for clarity.\n\n\n**R2: Clarification on 442 vs. 450 video count inconsistency**\n\nTo clarify, our dataset contains 450 densely annotated videos across 83 applications. However, for the action prediction task, we use 442 videos, excluding 8 that involved complex actions (e.g., press and hold Ctrl + drag + release), which most models cannot yet handle reliably making evaluations difficult. This is noted in the Limitations section (L795–797), and we will ensure both this and the earlier typo are clarified in the camera-ready version.\n\n**R3: Qualitative Evaluation**\n\nBelow we highlight qualitative error analysis for all three benchmark tasks:\n\n**Element Grounding:** Figures are available at [](https://bit.ly/43xRn7f)<[link](https://bit.ly/43xRn7f)>. Observations are on SOTA UI-TARS model\n>**Fine-grained ambiguity:** The model fails to recognize the correct target among several visually similar candidates (Fig. 9), highlighting the need for improved disambiguation strategies.\n\n>**Lack of domain knowledge:** The model misinterprets platform-specific elements, such as  “Fontwork” represented by “F” symbol  (Fig. 10). Incorporating external knowledge could help improve performance.\n\n>**Small element detection:** The model struggles with small UI elements, particularly in high-resolution or dense interfaces (Fig. 11). Iterative zoom-in strategies may address this limitation.\n\n>**Cross-platform generalization:** The model incorrectly transfers layout assumptions across platforms—eg., predicting the minimize button's position on macOS as it would appear on Windows (Fig. 12). This suggests memorization and overfitting.\n\n**Layout Grounding:** Example cases below are available at <[link](https://bit.ly/4iNTBnL)>\n>**Inaccurate bounding box placement:** Closed-source models often predict bounding boxes that loosely cover the correct region without precisely matching its boundaries (Fig. 13a, 14a). This suggests difficulty in precise layout partitioning\n\n>**Poor functional grouping:** Open-source models sometimes fail to group elements correctly, even when the query explicitly mentions them (Fig. 13b).\n\n>**Superficial semantic matching:** Models sometimes default to grounding smaller elements that share surface-level keywords with the query but are semantically unrelated (e.g., predicting a “Design” button for a different design-related query, Fig. 14b)\n\n**Action Prediction:** Example cases below are available at <[link](https://bit.ly/42gIBID)>\n>**Poor grounding:** Models often predict the correct action type but fail to ground it to the appropriate UI element (Fig. 15). This reflects challenges in bridging perception and execution.\n\n>**Lack of platform knowledge:** We observe that models sometimes hallucinate actions (eg., refer to non-existent elements) or misinterpret platform-specific elements (Figs. 16, 17), likely due to limited training exposure to diverse desktop environments.\n\n>**High interface complexity:** Dense and feature-rich platforms pose greater challenges for accurate action prediction. UI-TARS exhibits the highest error rate (85%) on creativity platforms (112 elements/frame) while performing better (72% error) on simpler education platforms (62 elements/frame). \n\n**R4: Clarification on offline/static benchmark setting**\n\nOur benchmark is offline and static by design. The controlled setup allows for a fine-grained evaluation of how perception and grounding errors affect downstream actions. By structuring tasks from perception to action prediction, UI-Vision helps isolate failure modes and provides insights crucial for building more robust agents before deployment in dynamic environments.\n\n**R5: Annotator consistency during dataset creation**\n\nWe partnered with a for-profit company that provided experienced annotators (L-604-605). All annotators underwent training on the software and were required to pass several assessments related to the tasks to proceed. Those who failed were excluded. Additionally, we followed a detailed annotation protocol, which was refined during a month-long pilot phase where annotators received detailed feedback to ensure high-quality and consistent data (L-676-677).", "claims_and_evidence": "1. Claiming that the proposed benchmark is the largest desktop-centric benchmark is a little bit unconvincing. As shown in Table 1, the proposed benchmark contains 6484 samples, while OmniAct (Kapoor et al., 2024) contains 9802 samples.\n2. L209 states that the final dataset consists of 442 high-quality demonstrations across 83 applications. But L055 (right) states that the proposed method contains 450 recorded videos spanning 83 platforms.", "methods_and_evaluation_criteria": "The proposed benchmark dataset is useful for the comprehensive evaluation of autonomous GUI agents, covering essential agent capabilities.", "theoretical_claims": "Yes.", "experimental_designs_or_analyses": "It would be better to add qualitative evaluations to the paper.", "relation_to_broader_scientific_literature": "The key contribution of this paper is the proposed desktop-centric GUI benchmark. It is useful for comprehensive element grounding, layout grounding, and action navigation tasks.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "The paper is well-organized with a clear structure. The proposed benchmark dataset could be useful for future research in the community. However, this benchmark still belongs to offline and static scenarios.", "other_comments_or_suggestions": "The number of videos is inconsistent in the paper.", "questions_for_authors": "How to ensure the consistency of human annotators during the dataset creation process?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "E2f0M0WbGt", "reviewer": "Reviewer_PYJp", "summary": "The authors introduce UI-Vision, a comprehensive desktop GUI benchmark with 83 open-source applications, focusing on three tasks: Element Grounding, Layout Grounding, and Action Prediction. Built from human demonstrations and expert annotations, it evaluates GUI agents’ visual perception and interaction capabilities. Tests on top VLMs reveal poor spatial grounding (e.g., 18% accuracy) and action execution (e.g., 4.4% recall on clicks), highlighting gaps in desktop GUI automation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing the value of our desktop-focused benchmark, task diversity beyond web/mobile settings, dense annotations, and actionable insights. We address the concerns below.\n\n**R1: Failures cases**\n\nWe summarize key error patterns below and refer the reviewer to our response to Reviewer 3NZB (R3) for more details and examples.\n\n- **Element Grounding:** Models often confuse visually similar elements, fail to recognize platform-specific icons and struggle with small elements in dense layouts.\n\n- **Layout Grounding:** Common issues include bounding boxes that are too large or loosely defined and failure to group related UI elements correctly\n\n- **Action Prediction:** Models frequently fail to ground actions correctly, hallucinate UI elements, and perform poorly on complex interfaces with many interactive elements.\n\n**R2: LLAMA-3.3-70B layout grounding validation process**\n\nThe authors conducted a manual validation process to ensure quality using three criteria: **(i)** bounding boxes must tightly enclose relevant UI elements without including unrelated regions; **(ii)** grouped elements must form a semantically meaningful and visually coherent unit; and **(iii)** labels and descriptions must accurately reflect the group's function. Groups failing any check were discarded. The authors had a detailed protocol and access to the platform and its documentation to ensure consistency. We will include these validation details in the camera-ready version.\n\n**R3: Latency metrics**\n\nWe report latency per query, average output tokens, and GPU usage across all three tasks in Tables at  <[link](https://bit.ly/4jeHdNC)>, using default Hugging Face implementations for consistency. Token efficiency was measured with GPT-4 tokenization.  Models like UI-TARS, trained on action-heavy tasks, generate longer outputs due to detailed step-by-step reasoning.\n\n**R4: *Recall@d metric and value selection***\n\nWe choose the *d* values based on the average bounding box size across the dataset after resizing all screenshots to a standard resolution (800×700) yielding a base value of (25, 35). This base value is then rescaled based on the original resolution of each sample to ensure consistent evaluation across varying interface sizes. \n\n**R5: Ablation on annotation density and task design impact**\n\nWe perform a detailed ablation analysis (Fig. 18 at <[link](https://bit.ly/4c8La3M)>) to understand factors affecting performance on 3 tasks. Across all three tasks, we find that densely packed applications with smaller UI elements like GIMP (112 elements/frame, area of 418 px/element) show lower performance compared to entertainment platforms with simpler layouts like VLC (63 elements/frame, area of 875 px/element).\n\nRegarding task design, GUI agents perform comparably or better on functional grounding than basic grounding, likely due to alignment with their training data. In contrast, both open- and closed-source VLMs perform worse on functional tasks. Spatial grounding is the most challenging, as it requires identifying the correct element and reasoning about its relative position—an area where VLMs generally struggle due to limited spatial reasoning.\n\nFor a more comprehensive analysis with detailed settings, we refer the reviewer to our detailed response to Reviewer 7jPV (R1).\n\n**R5: annotator qualifications and training process**\n\nBeyond academic background, annotators were selected through technical assessments, language tests, and task-specific bootcamps. Those who didn’t meet the criteria were excluded. We used a detailed annotation protocol refined during a month-long pilot with feedback to ensure consistency (L-676–677). Quality was further ensured through manual reviews and ongoing performance monitoring. We will clarify these details in the camera-ready version.\n\n**R6: Clarification on UI-Vision’s unique contribution vs. existing benchmarks**\n\nWe appreciate the reviewer’s concern and would like to clarify the distinct contribution of our work. While recent efforts [1–4] have advanced GUI agent evaluation, they primarily focus on web environments or online interaction. Specifically, [1] includes limited desktop data and annotations (OmniAct: 5412 training samples across 38 platforms only for action prediction). [3] and [4] focus mostly on web-based tasks. While [2] (OSWorld) targets desktop platforms, it evaluates across a limited number of platforms in an online setting using broad task completion metrics.\n\nIn contrast, our benchmark supports offline evaluation across 450 real-world desktop tasks spanning 83 applications. It provides a complete pipeline of three benchmark tasks, along with detailed evaluation metrics. This setup allows models to be assessed from basic perception to planning and execution, all within a single structured benchmark. Unlike existing works, UI-Vision offers fine-grained insights into where and how models fail, making it a valuable tool for diagnosing and improving GUI agents.", "claims_and_evidence": "- UI-Vision’s focus on desktop GUIs fills a critical gap, with its diverse tasks (e.g., layout grounding) offering a fresh approach beyond web/mobile benchmarks. \n- Fig. 1 shows a task example but lacks failure cases to illustrate model struggles.  \n- Sec. 3.2: Layout grounding generation via LLAMA-3.3-70B is mentioned, but validation process details are missing.", "methods_and_evaluation_criteria": "- The open-source, densely annotated dataset (83 apps, 2,072 tasks) with real-world complexity is a valuable resource for GUI agent research.\n- Table 5 lacks latency metrics, vital for practical GUI automation, despite extensive action evaluation.\n- Sec. 4.2: Action metric definitions (e.g., Recall@d) lack specific $d$ values, muddying interpretation.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "- Evaluations (Tables 3-5) expose clear model weaknesses (e.g., spatial grounding at 18%, drag action struggles), providing actionable insights for future development.\n- No ablations test the impact of annotation density (e.g., 71 boxes/frame) or task design (e.g., spatial vs. functional grounding). How do these affect performance?  \n- Sec. 3.1: How are \"expert annotators\" qualified beyond degrees? Training details are vague, affecting reproducibility.", "relation_to_broader_scientific_literature": "Good to discuss recent work.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "- I understand that this is a Desktop-centric GUI Benchmark, but now many benchmarks or datasets [1-4] have already covered these, and I think the unique contribution of this work is a bit limited.\n\n[1] Xu Y, Wang Z, Wang J, et al. Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction[J]. arXiv preprint arXiv:2412.04454, 2024.\n\n[2] Xie T, Zhang D, Chen J, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments[J]. Advances in Neural Information Processing Systems, 2024, 37: 52040-52094.\n\n[3] Zheng B, Gou B, Kil J, et al. GPT-4V (ision) is a Generalist Web Agent, if Grounded[C]//International Conference on Machine Learning. PMLR, 2024: 61349-61385.\n\n[4] Koh J Y, Lo R, Jang L, et al. VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks[C]//Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024: 881-905.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3pANJuIqyL", "reviewer": "Reviewer_7jPV", "summary": "This paper introduces UI-Vision, a benchmark for evaluating AI agents’ ability to interact with desktop Graphical User Interfaces (GUIs). Unlike existing benchmarks that focus on web or mobile environments, UI-Vision is designed specifically for desktop platforms and is claimed to be the largest of its kind. It includes 6,484 tasks across 83 software applications, spanning categories such as productivity, development, creativity, education, browsers, and entertainment. The benchmark assesses models on three key tasks: element grounding, layout grounding, and action prediction. The study evaluates multiple state-of-the-art models, including GPT-4o, Gemini, Claude, and various open-source alternatives, highlighting their limitations in handling complex desktop interactions.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comments and address them below.\n\n**R1: Fine-grained error analysis, ablations on task difficulty, and generalization across software categories**\n\nTo deepen our understanding of model performance, we conducted an error analysis through a human study for element grounding and found that models often struggle with small UI elements and platforms with high functional density. To investigate further, we leveraged the dense bounding box annotations in our dataset and sampled a diverse subset of elements that were consistently challenging for top-performing models such as UI-TARS, UGround-v1, and Aria UI. We selected cases where one or more models failed, resulting in a subset of 5479 samples spanning basic, functional, and spatial categories. We report detailed results across software and categories in Table in <[link](https://bit.ly/4iLiu3j)>, and summarize key findings below:\n\n**Error Types:** Models often confuse visually similar elements, miss platform-specific icons, and fail to detect small elements in dense layouts.\n\n**Task Difficulty:** GUI agents perform better on functional grounding than basic grounding, likely due to alignment with their training data. In contrast, VLMs underperform functional tasks, revealing task-specific weaknesses.\n\n**Category Generalization:** Performance varies significantly across software categories. Creativity tools like Blender and GIMP (112 elements/frame, 418 px/element) show the lowest accuracy, while simpler platforms like VLC (63 elements/frame, 875 px/element) perform best. Notably, screenshot resolution had little impact.\n\nWe refer the reviewer to our response to Reviewer 3NZB (R3) for qualitative study and analysis on **Layout Grounding** and **Action Prediction** and more details and examples on **Element Grounding**.\n\n**R2: cross-software generalization analysis**\n\nWe compare model performance on common apps (e.g., VSCode) vs. less common ones (e.g., FreeCAD, QGIS) using the element grounding subset in R1. results are included in Table in <[link](https://bit.ly/4iLEnQ5)>. While we cannot confirm exact training data used in several models, this serves as a proxy for generalization analysis. We observe all models show significant accuracy drops on less common apps, confirming consistent generalization challenges. \n\n**R3: Concern about dataset bias due to focus on open-source software**\n\nMany open-source applications closely emulate proprietary counterparts, capturing similar complexity. For instance, LibreOffice mirrors Microsoft Office features and serves over 200 million users[1]. Given this overlap in functionality and interface design, we believe our benchmark remains meaningful and representative of real-world GUI scenarios. We refer the reviewer to our response to Reviewer aEXy (R3) for more details on our choice to focus on open-source software\n\n[1]Wikipedia page for LibreOffice. https://en.wikipedia.org/wiki/LibreOffice\n\n**R4: Evaluation of inference speed, latency and token efficiency**\n\nWe perform a detailed analysis of the inference speed, token efficiency and latency for different models and different benchmark tasks and report the numbers in <[link](https://bit.ly/4jeHdNC)>.\n\n**R5: Ensuring generalisation vs memorisation**\n\nSince we do not have access to the training data recipe of the most of models, we are not able to carry out a comprehensive study on this point. However, the failure case of Element Grounding in Fig. 12 in <[link](https://bit.ly/43xRn7f)> indicates merely memorization can not ensure accurate grounding. Models need good generalization ability to excel on the task.\n\n**R6: Analysis of spatial grounding and drag actions**\n\nIn the spatial setting, models must first correctly identify the reference element and then reason about its spatial relation—both steps are required for success. Also, VLMs are known to struggle with spatial reasoning, limiting performance.\n\nFor drag actions, models are rarely trained on such interactions in web data, making them difficult to execute. Also, success depends on accurately predicting both start and end points, increasing the chance of error.\n\n**R7: Failure cases in layout grounding**\n\nAs shown in Fig. 13(a) and 14(a) in <[link](https://bit.ly/4iNTBnL)>, Gemini-1.5-Pro often fails to return a minimal bounding box for the ground truth region, although the correct region is usually contained within the predicted box. \n\n**R8: Domain shift between open and closed software**\n\nYes, we do. However, open-source systems are built to provide functionalities similar to those of closed-source counterparts, so evaluating model capabilities in these scenarios will directly correlate with those of closed-source ones.\n\n**R9: Potential for automated annotation**\n\nYes, we do. We have applied LLMs in the annotation of layout grounding in Sec 3.2. However, the major challenge is that UI tasks are quite fine-grained, so it will be hard to control the quality during automatic annotation at scale.", "claims_and_evidence": "The paper claims that UI-Vision is the largest and most diverse benchmark for evaluating desktop GUI agents. This is well-supported by the dataset size (6,484 tasks, 83 applications) and comparisons with existing benchmarks. Claims about performance gaps in SOTA models are backed by experiments.", "methods_and_evaluation_criteria": "The evaluation criteria align well with the problem: GUI automation requires visual perception and interaction, and the paper assesses models across grounding, layout recognition, and action prediction. The dataset is large and diverse, covering real-world applications, which enhances credibility. However, the evaluation could be improved by including human performance baselines to contextualize model failures. The choice of IoU, recall@d, and accuracy metrics makes sense, though more fine-grained analysis (e.g., error types in action prediction) would be useful.", "theoretical_claims": "The paper does not include formal theoretical claims or proofs. The methodology is empirical, focusing on dataset construction, benchmarking, and performance analysis. No verification of proofs is required.", "experimental_designs_or_analyses": "The experimental setup is generally sound, using diverse state-of-the-art models (GPT-4o, Gemini, Claude, and open-source alternatives). The dataset is well-documented, and comparisons with existing benchmarks are thorough. However, there are limitations in the experimental design: 1) The effect of dataset bias, e.g., open-source software selection, is not discussed; 2) No cross-software generalization analysis, e.g., how well models trained on some applications transfer to unseen ones; 3) The study lacks enough error analysis, e.g., why do models fail at certain tasks?", "relation_to_broader_scientific_literature": "The paper correctly situates UI-Vision within the broader field of GUI automation, multimodal learning, and AI-driven interaction systems. It builds on prior GUI benchmarks like MiniWoB++, WebArena, and OmniAct but extends focus to desktop environments. It also connects with multimodal learning research, citing relevant vision-language models.", "essential_references_not_discussed": "N/A, no obvious omissions were identified from closely related literature.", "other_strengths_and_weaknesses": "Strengths:\n•\tThe first large-scale benchmark specifically designed for desktop GUI automation, addressing a gap in existing benchmarks that focus on web and mobile platforms.\n•\tEvaluates models on element grounding, layout recognition, and action prediction, providing a multi-faceted assessment of GUI interaction.\n•\tThe dataset spans 83 diverse applications across various categories (e.g., productivity, development, creativity), enhancing its applicability to real-world use cases.\n•\tBenchmarks state-of-the-art multimodal models, revealing critical performance gaps in GUI perception and interaction.\nWeaknesses:\n•\tDataset bias: The dataset focuses exclusively on open-source software, limiting its applicability to commercial tools.\n•\tLimited analysis of model failures: While performance limitations are discussed, there is no in-depth analysis of specific error patterns (e.g., types of misclicks, confusion between similar UI elements, or issues in handling dynamic interfaces).\n•\tThe benchmark focuses on accuracy but does not assess inference speed, latency, or token efficiency for real-world deployment.\n•\tWhile the paper evaluates multiple models on UI-Vision, it does not include ablation studies to analyze which task components contribute most to difficulty. Similarly, there is no systematic analysis of how models generalize across different software categories, such as productivity vs. creative tools. Such studies could provide deeper insights into model weaknesses and potential benchmark improvements.", "other_comments_or_suggestions": "I look forward to the open-source release of the dataset and resources, which will allow for a more comprehensive review of the benchmark’s reproducibility, cross-benchmark comparisons, and dataset extensibility.", "questions_for_authors": "1.\tIf models were previously trained on applications similar to those in UI-Vision, how do you ensure the benchmark fairly evaluates generalization rather than memorization?\n2.\tYour experiments reveal significant performance drops in spatial element grounding (best model: 18%) and drag actions (near-zero recall). Have you analyzed why these tasks are particularly difficult?\n3.\tGiven that the best-performing model (Gemini-1.5-Pro) achieves only 30.8 IoU in layout grounding, what are the most common failure cases? Do models struggle more with complex, nested UI layouts (e.g., tabbed interfaces) or dense interfaces with many overlapping elements?\n4.\tDo you anticipate domain shift issues between open-source and closed-source UI designs?\n5.\tDo you envision automated UI annotation tools or self-supervised learning techniques to reduce reliance on human annotators? What challenges do you foresee in ensuring annotation consistency at scale?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GXkR80YkZg", "reviewer": "Reviewer_aEXy", "summary": "The paper introduces UI-Vision, a large-scale, desktop-centric benchmark for evaluating Graphical User Interface (GUI) agents in visual perception and interaction. Unlike existing benchmarks that focus on structured web and mobile interfaces, UI-Vision targets desktop environments, which lack standardized automation APIs and require direct screen interpretation. The benchmark spans 83 open-source applications, providing 6,484 tasks with dense human-annotated data across three key tasks: Element Grounding, Layout Grounding, and Action Prediction. Evaluation of leading Vision-Language Models (VLMs) and GUI agents highlights major limitations, such as poor spatial reasoning (best model achieves only 18.0% accuracy in element grounding) and difficulty in action execution, particularly dragging actions. However, combining LLMs with GUI-grounding models significantly improves action recall (2×–5× gains). The findings emphasize the need for better multimodal training, stronger screen parsing, and enhanced planner-grounding integration for AI-driven GUI interaction. To advance research, UI-Vision is fully open-source, aiming to set a new standard for desktop GUI automation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**R1: Failures cases**\n\nClaude 3.7 Sonnet was released on Feb 24, 24 days after the deadline, and Qwen 2.5-VL on Jan 28, just two days prior—making it infeasible to include them in the review version. However, we have now evaluated both models on our grounding benchmarks, and results are available at <[link](https://bit.ly/4iLiu3j)>.\n\nWe summarize key error patterns below and confirm that they hold for the above models and refer the reviewer to our responses to Reviewer 3NZB (R3) and Reviewer 7jPV (R1) for more details and examples:\n\n**Element Grounding:** Models struggle with visually similar elements, platform-specific icons, and small elements in dense UIs. GUI agents perform better on functional tasks due to training alignment, while VLMs underperform, revealing task-specific gaps.\n\n**Layout Grounding:** Frequent issues include overly large or loosely defined bounding boxes and failure to group related UI elements correctly.\n\n**Action Prediction:** Models often fail to ground actions accurately, hallucinate elements/actions, and perform poorly on dense, complex interfaces.\n\n**R2: Concern about generalizing findings from open-source to proprietary desktop environments**\n\nWe appreciate the reviewer’s concern. However, many open-source apps closely emulate proprietary counterparts, capturing similar complexity. For instance, LibreOffice mirrors Microsoft Office features and serves over 200 million users[1]. Given this overlap in functionality and interface design, we believe our benchmark remains meaningful and representative of real-world GUI scenarios.\n\n[1]Wikipedia page for LibreOffice. https://en.wikipedia.org/wiki/LibreOffice\n\n**R3: Limited real-world applicability due to the exclusion of closed-source applications**\n\nWe agree that automation is widely used in proprietary desktop environments; however, including such applications poses legal and licensing challenges, as many restrict redistribution of software interfaces, recordings, or interaction data, making it difficult to build a publicly shareable dataset around them. To ensure accessibility and reproducibility, we focus on open-source software.\n\nMore importantly, our benchmark includes widely-used open-source applications like LibreOffice, VSCode, GIMP, Firefox, Brave, and Shotcut, which serve as strong counterparts to commercial software. These applications support complex, real-world workflows, share high functional similarity with proprietary software and are backed by active communities. We believe this focus ensures UI-Vision remains both practical and widely applicable.\n\n**R4: Planner + VLM grounding analysis**\n\nWe clarify that the planner selects the action and target element, while the grounding model only provides coordinates. Thus, the improvement is primarily due to better action selection. To analyze this further, we compared the decrease in error rates across platform categories (Fig. <[link](https://bit.ly/3RimGMd)>) in this setup. Productivity tools show the largest improvement (26%), even though Entertainment tools have the highest baseline grounding accuracy—highlighting the planner’s effectiveness. In contrast, Creativity platforms show the smallest improvement (14%), reflecting the challenges of planning and grounding in functionally rich interfaces with small UI elements.\n\n**R5: Lack of multi-step planning evaluation**\n\nWe agree that long-term planning is important, but current models still struggle with single-step actions (nearly 0% recall on drag and only 20% on click actions). Strengthening performance on these core tasks is a necessary first step, with multi-step planning as a valuable direction for future work.\n\n**R6: Missing support for hybrid or cross-platform GUI environments**\n\nWe agree that hybrid environments are a valuable direction for future work. However, our current benchmark already presents significant challenges for existing models showing that even desktop settings require substantial progress. We believe UI-Vision offers ample opportunities to advance model capabilities.\n\n**R7: Anonymous repo with full benchmark**\n\nWe apologize for the inconvenience but as per ICML policy, we are only permitted to share links to figures and tables. However, to give reviewers a clearer sense of the benchmark, we have included detailed failure cases in our response to Reviewer 3NZB (R3) and provided additional task examples at <[link1](https://bit.ly/4clZMx6)> and <[link2](https://bit.ly/3G06lsO)> (5s loading). All code, data, and evaluation scripts will be soon released.\n\n**R8: Low performance of InternVL-8B**\n\nThe main reason is that InternVL2-8B fails to consistently generate meaningful bounding boxes, often producing arbitrary outputs like [0, 0, 50, 50] or [0, 0, 200, 200]. This suggests limited training on grounding tasks, particularly for small UI elements. Its poor performance is also reflected in the ScreenSpot benchmark [1] (Table 2).\n\n[1] Wu, et al. OS-ATLAS: A foundation action model for generalist GUI agents.", "claims_and_evidence": "1. The paper presents quantitative results showing that state-of-the-art models (GPT-4o, Gemini, etc.) achieve low accuracy on grounding and action tasks. However, the specific reasons for failure (e.g., lack of spatial reasoning, reliance on textual cues) are asserted rather than deeply analyzed. Additional qualitative error analyses or ablation studies could better support this claim. In addition, some new models including Qwen2.5-VL, InternVl-2.5, Cluade-3.7, etc, should be evaluated.\n\n2. While the dataset covers a broad range of applications, it focuses primarily on open-source software, which may not fully capture the complexities of proprietary desktop environments (e.g., Windows/MacOS enterprise applications). The assumption that findings generalize across all GUI platforms could be problematic.\n\n3. The experiments show a 2×–5× improvement in recall when LLMs are paired with grounding models. However, it is unclear whether this improvement is due to better action selection or simply improved recall on basic UI elements. Further analysis of failure cases and different LLM planner strategies would clarify the robustness of this conclusion.", "methods_and_evaluation_criteria": "1. While UI-Vision is license-permissive, it excludes proprietary applications (e.g., Microsoft Office, Adobe Photoshop, enterprise software). Desktop automation is often used in closed-source environments, so the benchmark might not fully reflect real-world constraints where API restrictions and security settings affect interactions. A more balanced dataset, incorporating at least some closed-source applications, would improve applicability.\n\n2. The Action Prediction task assumes step-by-step task execution, but real GUI agents often require long-term reasoning (e.g., navigating through multiple menus before completing a task). Introducing multi-step action planning evaluation would be beneficial, as current metrics mostly assess immediate action correctness rather than goal completion efficiency.\n\n3. UI-Vision is focused exclusively on desktop applications, whereas many GUI automation challenges also involve cross-platform scenarios (e.g., web-based applications embedded in desktop environments). Incorporating some hybrid environments (e.g., web-based GUIs within desktop software) would better reflect modern GUI automation challenges.", "theoretical_claims": "This work focuses on introducing a new benchmark for evaluating Graphical User Interface (GUI) agents, specifically designed for desktop environments. The paper discusses the creation of the UI-Vision benchmark, its tasks, and the evaluation of various models on this benchmark.", "experimental_designs_or_analyses": "1. The data collection process involved human annotators, which could introduce variability. However, the multi-stage quality checks and periodic verification by separate annotators and authors aim to mitigate this.\n\n2. The results indicate that even state-of-the-art models struggle with spatial grounding and precise action execution. This highlights the need for further research and development in this area.", "relation_to_broader_scientific_literature": "The paper's evaluation reveals specific limitations in current models' spatial understanding and action execution capabilities. It highlights critical areas for future research that align with growing recognition in the literature of the need for improved visual-spatial reasoning in multimodal agents.\nIn summary, UI-Vision advances the field by addressing documented limitations in existing benchmarks while building on recent progress in multimodal AI and GUI automation research. Its comprehensive approach provides a new standard for evaluating desktop GUI agents, filling important gaps identified in prior literature.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "I hope authors provide anonymous repo containing full benchmarsk and eval scripts. This is extremely important for reviewers to valid the quality of the proposed benchmark.", "other_comments_or_suggestions": "1. In table 3, based on the performance of open-source VLMS, why InternVL2-8b achieves such low performance in terms of basic setting?\n\n2. Can authors provide more failure case analysises for VLMs? It is important to understand the significance of the proposed benchmark.", "questions_for_authors": "None.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_flag": "Flag this paper for an ethics review.", "ethics_expertise_needed": ["Privacy and Security"], "ethical_review_concerns": "None."}}]

Author rebuttal:

Meta-review: This paper presents a comprehensive benchmark for understanding complex desktop GUIs. The benchmark features high-quality, human-annotated action trajectories, diverse layout and interaction annotations, and spans 83 software applications to capture broad real-world variability. Experimental results reveal that current models struggle with desktop environments, highlighting the need for further research. After discussion, three reviewers recommended a weak accept, while Reviewer 7jPV maintained a weak reject. The Area Chair acknowledges that some of Reviewer 7jPV’s concerns were addressed in the rebuttal, though questions remain about the suitability of a benchmark-focused paper for ICML. As a result, the Area Chair recommends a weak accept, and encourages the authors to incorporate the necessary revisions in the final version.
Decision: Accept (poster)