Grounding Computer Use Agents on Human Demonstrations
Authors: Aarash Feizi
Shravan Nayak
Xiangru Jian
Kevin Qinghong Lin
Kaixin Li
Rabiul Awal
Xing Han Lù
Johan Obando-Ceron
Juan A. Rodriguez
Nicolas Chapados
David Vazquez
Adriana Romero-Soriano
Reihaneh Rabbany
Perouz Taslakian
Christopher Pal
Spandana Gella
Sai Rajeswar
Venue: ICLR 2026 Poster (2026)
Keywords: computer use agents
dataset
multimodal large language models
Primary area: datasets and benchmarks
TL;DR: We introduce GroundCUA, the largest, most diverse, human-annotated GUI grounding dataset and the GroundNext series of models.

Abstract: Building reliable computer-use agents requires grounding: accurately connecting natural language instructions to the correct on-screen elements. While large datasets exist for web and mobile interactions, high-quality resources for desktop environments are limited. To address this gap, we introduce GroundCUA, a large-scale desktop grounding dataset built from expert human demonstrations. It covers 87 applications across 12 categories and includes 56K screenshots, with every on-screen element carefully annotated for a total of over 3.56M human-verified annotations. From these demonstrations, we generate diverse instructions that capture a wide range of real-world tasks, providing high-quality data for model training. Using GroundCUA, we develop the GroundNext family of models that map instructions to their target UI elements. At both 3B and 7B scales, GroundNext achieves state-of-the-art results across five benchmarks using supervised fine-tuning, while requiring less than one-tenth the training data of prior work. Reinforcement learning post-training further improves performance, and when evaluated in an agentic setting on the OSWorld benchmark using o3 as planner, GroundNext attains comparable or superior results to models trained with substantially more data. These results demonstrate the critical role of high-quality, expert-driven datasets in advancing general-purpose computer-use agents.

Published as a conference paper at ICLR 2026
GROUNDING COMPUTER USE AGENTS ON HUMAN DEMONSTRATIONS
Aarash F2i
Shravan Nayak 1.3
Xiangru Jian
Kevin Qinghong Lin
Kaixin Li²
Rabiul Awal 1,3,4
Xing Han Lu 1.
Johan Obando-Ceron 13 Juan A. Rodriguez 19
Nicolas Chapados
David Vazquez
Adriana Romero-Soriano 1.2 Reihaneh Rabbany 1.2
Perouz Taslakian 1224
Christopher P2.4
Spandana Gella 4t Sai Rajeswar
Mila- Quebec AI Institute
MccGill University Université de Montréal
*ServiceNow Research
'University of Waterloo "Polytechnique Montréal
University of Oxford
"National University of Singapore
fccoe de Technologie Supérieure
"CIFAR AI Chair
Project Page: https: github. 1o
ABSTRACT
Building reliable computer-use agents requires grounding: accurately connect- ing natural language instructions to the correct on-screen elements. While large datasets exist for web and mobile interactions, high-quality resources for desktop environments are limited. To address this gap, we introduce GROUNDCUA, a large-scale desktop grounding dataset built from expert human demonstrations. It covers 87 applications across 12 categories and includes 56K screenshots, with every on-screen element carefully annotated for a total of over 3.56M human- verified annotations. From these demonstrations, tions that capture a wide range of real-world tasks, providing high-quality data for model training. Using GROUNDCUA, we develop the GROUNDNEXT family of models that map instructions to their target UI elements. At both 3B and 7B scales, GROUNDNEXT achieves state-of-the-a results across five benchmarks us- ing supervised fine-tuning, while requiring less than one-tenth the training data of prior work. Reinforcement learning post-training further improves performance, and when evaluated in an agentic setting on the OSWorld benchmark using o3 as planner, GROUNDNEXT attains comparable or superior results models trained with substantially more data. These results demonstrate the critical role of high- quality, expert-driven datasets in advancing general-purpose computer-use agents.
generate diverse instruc-
INTRODUCTION
The vision of computer-use agents (CUA) that operate software on behalf of users has gained signif- icant momentum with recent progress in multimodal large language model-based agents (OpenAI, 2025: Anthropic, 2024; Qin al., 2025; Wang et al., 2025a). These agents promise to automate routine work and make complex digital tools more accessible. For such agents to succeed, they must first plan the next step task, then ground the plan to the exact on-screen element to click, type, or drag. Accurate grounding is critical: without correctly identifying the right button or menu item, even flawless plan cannot be executed. In FreeCAD, for instance, when asked to "open the color picker" (Figure 1), the agent must distinguish a small palette icon from look-alike tools, one of which it must precisely click. When grounding fails, the plan quickly veers off course, small errors compound, and tasks ultimately fail (Nayak et al., 2025). Moreover, grounding in desktop applications is challenging due to their complexity and diversity. These applications often feature righ-resolution displays with dense layouts and visually similar elements, making precise localiza- tion difficult. Additionally, desktop applications can contain user-specific artifacts (c.g., documents or spreadsheets) that may have been seen during training, adding variability and unseen contexts. Finally, collecting automated datasets for desktop environments with strong coverage is also chal- lenging, as highlighted by recent datasets (Gou et al., 2024; Wu et al., 2024; Xie et al., 2025). To this
Equal contribution. Equal supervision.

---

Published as a conference paper at ICLR 2026
GroundcUA Dataset
GroundNext Models
Change
30
green
Elements & MetaData
Computer User Instrutions
Training on 7e8k SFT samples and 10K samples
Categocy; fectoe Mox: (4), 121, 245, 234]
Select bright green
SFT
RL
Seb Gale Flaeest Saee Screen Color s3oc: 1252, 196, 149, 1:1]
Prese the button below bastc celers.
Hight ight the mor goows cute face.
Lianert kaza Campeerancy Stider
Platiosm: FreetAD
Expert arentatet's
6.8]
Mea: (18 2, 150 2, 594, 592)
X
Clase the left propectios pane.
Elerents: 152
Open Display properties
Resstatton: 1366378B
Catesety: Sede Ricila
(38. M.... 554,
loft sidebaz.
Figure I: Overview of the GROUNDCUA dataset and GROUNDNEXT models. Human demonstra- tions of computer-use tasks are recorded as screenshots (example from FreeCAD) with UI meta- data, which are processed into high-quality natural language instruction tasks for UI grounding. GRoUNDNEXT is trained in two stages: SFT (700K samples) followed by RL (10K samples), achieving state-of-the-art grounding performance with efficient training.
end, we introduce GROUNDCUA, a large-scale, human-annotacie dataset for desktop grounding. The dataset spans 87 applications across 12 categories, with 56K screenshots and 3.56M+ element annotations. These annotations are collected from task demonstrations by trained annotators, ensur- ing high-quality and densely labeled data that provides rich context for effective model training. also reflects the pixel diversity of desktops, with resolutions ranging from 500K to 7M pixels and a substantial proportion of very small bounding boxes. highlighting the fine- fine-grained challenges agents must overcome. Furthermore. GROUNDCUA includes fine-graine category information (menus, buttons, etc.) for 50% of the UI elements and includes multiple variants of related applications (e.g., LibreOffice and OnlyOffice), directly addressing the difficulty of similar yer distinct applica- tions and enabling agents to learn robust, application-specitii grounding strategies. Key highlights of GROUNDCUA compared to other datasets are: Scale: 56K annotated screenshots and 3.56 mil- lion elements; Resolution, Element Size, and Density: High-resolution images with maximum annotation density, covering almost every visible element, including small elements like icons and controls; Expert Quality: Human-verified annotations for high accuracy; Application Diversity: 87 desktop applications for broad real-world coverage. Using this dataset, we construct 700K image-instruction pair instruction-tuning set that mimics real-world semantic interactions.
We introduce the GROUND series of vision-language models, designed for precise grounding across desktop applications. The series includes models at 3B and 7B scales, offering a balance between efficiency and accuracy. Each model is trained in two stages: first, supervised fine-tuning (SFT) on 700K curated datapoints from GROUNDCUA, and second, reinforcement learning (RL) to further refine performance. This approach enables GROUNDNEXT to achieve state-of-the of-the-art results on key desktop benchmarks, including ScreenSpotPro (Li et al., 2025), OSWorld-G (Xie et al., 2025), and UI-Vision (Nayak et al., 2025). Despite using significantly fewer SFT datapoints than state-of-the-arr models like JEDI (which are trained on 9M datapoints), GROUND outperforms existing models, demonstrating its efficiency in training and proving that high-quality, well-curated data can outperform larger, less precise datasets. In the RL stage, GROUNDNEXT further refines its grounding accuracy, achieving significant improvements without relying on complex reward strategies, unlike many RL-tuned models, which typically incorporate specialized reward functions and additional objectives. This shows the effectiveness of combining supervised fine-tuning (SFT) with high-quality data. This efficiency is further highlighted in agentie, multi-step tasks; evaluated not only significantly outperforms its 3B peers but also surpasses many larger models. including OpenCUA-72B and proprictary APIs. Notably, our 3B model achieves performance comparable to the much larger JEDI-7B, demon- strating significant practical utility for real-world, resource-constrained systems. Additionally, GROUNDNEXT excels in gross-platfors generalization, delivering strong performance across desktop, mobile, and web environments; even though we only train on desktop dataset. Evaluated on benchmarks like MMBench-GUI (L2) and ScreenSpot-v2, in addition to desktop-specific tasks, GROUNDNEXT showcases its ability to generalize across a wide range of user interfaces and platforms. We plan to release both GROUNDCUA the trained GROUNDNEXT models to support open research, providing a solid foundation for the development of reliable, adaptable computer-use agents across diverse environments.
on the OSWorld-Verified benchmark, GROUNDNEXT-D
2

---

Published as a conference paper at ICLR 2026
In summary, contributions are follows:
We introduce GROUNDCUA. large-scale, human-annotated desktop grounding dataset with over 3.56 million annotations across 56K screenshots from 87 applications in 12 categories, providing dense, high-resolution, and fine-grained supervision for robust computer-use agents.
We present the GROUNDNEXT series, vision-language models at 3B and 7B scales, trained on GROUNDCUA with SFT and RI,, achieving state-of-the-art performance across desktop bench- marks and multi-step agentic tasks, with significantly fewer datapoints than prior models.
We provide a comprehensive analysis of SFT and RL roles, evaluate our dataset's cross-domain impact and generalization beyond desktop, and study the benefits of open-source software for grounding performance. We release both GROUNDCUA and the GROUNDNEXT models sup- port open research. port open research.
RELATED WORK
Computer-Use Agents. Recent advancements computer-use agents have focused on enhanc- ing their ability to understand and interact with user interfaces, ranging from simple commands to complex, multi-step tasks. Supervised fine-tuned models such CogAgent (Hong et al., 2023), ShowUI (Lin et al., 2024), and Ferret-UI (You et al., 2024) have improved interaction capabilities by enabling zero-shot instruction-following across desktop, web, and mobile interfaces, combining vi- sion, language, and action. Benchmarks like ScreenSpot-Pro (Li et al., 2025) and UI-Vision (Nayak et al., 2025) have emphasized the challenges of grounding natural language instructions in high- resolution desktop environments, particularly with dense screens and small elements. Grounding- focused agents, such as OS-ATLAS (Wu et al., 2024), UGround (Gou et al., 2024), and JEDI (Xie al., 2025), have made significant progress by scaling training data to map language specific UI elements. However, these methods often face challenges with data efficiency, particularly in complex desktop environments. Furthermore, recent RL-based approaches, inspired by DeepSeek-RI (Guo et al., 2025), such as GUI-RI (Luo et al., 2025), GUI-G" (Tang et al., 2025), and InfiGUI-GI (Liu al., 2025a), have addressed grounding through both simplistic and complex distance- based reward approaches. Despite these advancements, reliably grounding instructions to the correct on-screen el- ements remains a persistent bottleneck. To address this, we focus on high-quality, expert-annotated data to enhance grounding through both SFT and RI. training, while prioritizing data-efficient fine- tuning to improve model performance.
Table I: Comparison of grounding datasets. Columns: H = human provided instructions and labels; Desk includes desktop data; E/ Desk-E/S S number of elements, desktop elements, and screenshots; Res Range screenshot resolution range (MP); EleArea average element area (% of screenshot); #AvgE average elements per screen; Perm = permissive OSI-style license (e.g.. Apache-2.0. MIT), not clearly reported. Datasets marked with are grounding-specific versions constructed from the OS-ATLAS (Wu et al., 2024).
Annotation
Scale
Avg. Data Stats
Grounding Datasets
Perm?
Desk
Desk-E
Res Range EleArea #AvgE
UGround (Gou al., 2024)
X
9M
773k
(0.4, 1.9)
11.6
/
JEDI (Xie al., 2025)
V
4M
2.4M
575k
(0.9, 2.1)
?
AGUVIS-G (Xu al., 2024)
M
452k
2..5
?
AS-ATLAS (Wu et al., 2024)
V
14.5M
1.2M
1.85M
(0.5, 5.2)
0.53%
7.8
X
RICOSCA" (Li et al., 2020a)
X
X
170K
18K
(0.5,
0.28%
9,4
?
UIBert al., 2021)
X
166K
57K
(0.5,
0.24%
Widget Caption (Li et al., 2020b)
X
101K
14K
(0.5,
4.2%
7.0
V
AMEX'(Chai et al., 2025)
X
X
1.2M
101K
(0.9.4.5)
2.1%
11.8
?
SeeClick (Cheng al., 2024)
X
3M
270K
(2.1,2.1)
0.33%
11.2
?
AriaUI (Yang al., 2024)
/
4.1M
150K
295K
(1.3,
13.9
Fineweh's (Penedo et al., 2024)
X
9.9M
1.4M
2.11,
0.29%
6.9
GROUNDCUA(our
V
3.56M
3.56M
55k
(0.4,
0.13%
64.1
GUI Grounding Datasets. Training datasets for GUI grounding span mobile, web, and desk-
top platforms. Mobile datasets like RICO (Deka et al., 2017), UlBert (Bai et al., 2021), and
3

---

Published as a conference paper at ICLR 2026
EFICE
(a) GIMP
(b) FreeCAD
(c) LibreOffice Cale
Figure 2:
Examples of screenshots from different applications in GROUNDCUA. Red bounding
boxes indicate the annotated UI elements within each screenshot.
AMEX (Chai et al., 2025) provide element-level supervision within standardized layouts, sim- plifying extraction but limiting exposure to desktop-style density and iconography. Web-focused datasets, including SeeClick (Cheng et al., 2024) and UGround (Gou et al., 2024), scale ground- ing through automated harvesting from HTML/DOM, while Aguvis-G (Xu et al., 2024) broadens coverage across platforms. However, these automated pipelines likely overemphasize text-bearing elements while underrepresenting small icon-only controls, which are standard in desktop software. Desktop resources remain limited and challenging. OS-ATLAS (Wu et al., 2024) and AriaUl (Yang et al., 2024) assembles desktop splits via accessibility -tree tree traversal, yet accessibility signals are of- ten incomplete or inconsistent, leading to missing or imprecise element labels (Muryn et al., 2025; Gou et al., 2024). JEDI (Xie al., 2025) achieves scale through synthetic interface generation, but these simplified screens underrepresent genuine desktop complexity.
How is ours different? GROUNDCUA is the largest expert-annotated dataset for desktop ground- ing, comprising 55,568 screenshots across 87 open-source applications with over 3.56M human- verified UI elements. Compared existing datasets, GROUNDCUA features denser screens, wider resolution range, and smaller average element areas (see Table 1). It uniquely captures small desk- top components, such as icons, toolbars, and controls, that are difficult capture using automated tools. Its high-resolution images (ranging from 0.39 to 7.0M pixels) are substantially higher than other datasets and are the only ones to include very high-resolution images (see Figure 5, left). Additionally, GROUNDCUA features dense annotations that support semantics- and context-aware instructions, averaging 64 per screenshot, more than three times that of OS-Atlas (Desktop) and much higher than Aguvis-G (9) or UGround (11). Together, these properties make GROUNDCUA comprehensive and challenging dataset for training robust desktop grounding agents.
3 GROUNDCUA DATASET
This section introduces GROUNDCUA, the largest and most diverse desktop-specit dataset an- notated by human experts. We provide overview of the data collection pipeline and annotation process, as well as our high-quality fine-tuning data below.
Collecting demonstrations from human experts We record real-world interactions of expert users performing tasks with desktop applications and annotated interface elements scale. This ap- proach captures user-driven interactions, resulting in more realistic distribution of screenshots that better reflects real-world usage, compared to prior work that often relies on depth-firs or breadth- first search to generate random interface states (Wu et al., 2024). Our pipeline consists of three main steps: selecting diverse applications, designing and executing practical tasks, and annotating screen- shots. We partnered with a data labeling company for this process, with details on the annotator pool and training in Section A.2.
Selecting diverse desktop applications To support general-purpose computer-use agents, we se- lected 87 open-source applications across 12 categories (Table 5). Most applications are drawn from UI-Vision (Nayak et al., 2025), with four additional ones covering finance and scientific ap- plications. By focusing on open- sourcsource applications with permissive licenses, we ensure the dataset can be freely released while encompassing wide range of domains. These applications mirror the functionality of popular closed-source software (e.g., LibreOffice vs. Microsoft Office), making the dataset broadly applicable. Further details provided in Section A.I.
Designing and executing computer-use tasks. We asked annotators to design everyday computer-use tasks that reflect common goals (e.g., drafting document, editing spreadsheet,
4

---

Published as a conference paper at ICLR 2026
running a simulation) and then carry them out. This approach produces natural interaction trajecto- ries, unlike random clicking, and yields screenshots that closely mirror real-world usage. In total, annotators completed over 10, 000 task demonstrations across 87 applications !
Dense annotation of screenshots. From the recorded demonstrations, we extracted keyframes that capture the state of the interface immediately before a user action (c.g.. a mouse click or text entry) that would trigger change in the application. Annotators labeled every visible element in each keyframe using bounding boxes. For each element, they provided a textual label. This label was the element's 's name when available, the displayed text for shorter strings, the case of long passages such as source code or detailed descriptions. We also extracted OCR using PaddleOCR (Cui al., 2025) extract raw text specifically for these longer segments. In addition, around 50% of the elements were assigned to one of cight high-level categorics(sce Table 6). In total, this process produced over 3.56 million annotated elements, making GROUNDCUA the largest and most diverse human- auman-annotated grounding dataset for desktop environments to date. Examples of the annotations are provided in Appendix A.5, and further details of the annotation process are described in Appendix A.2.
concise summary in
Constructing high- finetuning instructions User queries in real-world settings can take various forms, from explicit references to UI elements (e.g.. Click 'Save') ) functional commands (e.g., Open a new tab), or spatial descriptions (e.g., Select the icon left of 'Files".. To handle this diversity, we design pipeline that leverages our dense annotations, which include bounding boxes, labels, categories, and OCR text, to construct diverse instruction-tuning data. These annotations enable the generation of highly contextual instructions, grounded directly in annotated screenshots. Unlike prior works that rely pretrained models, our approach involves prompting a multimodal LLM with annotated bounding boxes, application names, element labels, and surrounding context. This ensures that the instructions are tightly linked to both the visual and textual content, making them semantically and contextually relevant. By leveraging nearly every visible element on the screen, are able create UI context-aware and challenging instructions. We generate three pri- mary types of instructions: Direct, which describe an element's 's attributes, position, and surrounding context (e.g., Click the magnifying-en taggsifyinggglas icon next to the search bar for visual elements or Click the button that has the text 'Save' for DCR-based textual elements): Functional, which focus on the intended action of an element (e.g.. Open new tab instead of Click the '+' button); and Spatial, which guide the model based on the relative positioning of elements (e.g., Click the element to the left of 'Files' or Select the icon between 'Undo' and 'Redo"), "). We describe these instruction types in more detail in Section B and provide examples in Section B.6. These diverse instruction types, grounded both visual and semantic context, provide a comprehensive foundation for training more effective and context-aware GUI agents.
Dataset Statistics GROUNDCUA consists of 56K screenshots, totaling 3.56 million annotated elements, On average, each screenshot contains 64 annotations, with some images having as many as 542. The images have a mean resolution of 2.03 megapixels, with range from 0.39 to 7 megapixels. Bounding boxes are relatively small, covering just 0.13% of the image area on average, underscoring the fineggrained nature of the annotations. This results in high-quality fine-tuning data, with 700K samples for SFT and 10K for RI., extracted from the densely annotated screenshots and metadata. Detailed distribution plots of resolution, bounding box sizes, and category-level statistics for both screenshots and annotations are provided in Appendix A.3.
4
TRAINING GROUNDNEXT MODELS ON GROUNDCUA
4.1 MODEL TRAINING
We use Owen2.55-V--Instrruee VL-Instruct as the base model for all experiments, considering both the 3B and 7B parameter variants. We finetune both the vision encoder and the language model, as preliminary experiments indicated that this leads to better grounding performance.
We will release both the tasks and videos as part of the dataset.
5

---

Published as a conference paper at ICLR 2026
SFT We first train the models with standard supervised finetuning. Training is performed single node with 8 H100 GPUs, using global batch size of 128. Additional hyperparameter details provided in Appendix C.2. For training data, Section 3. From this dataset, we use subset of 700k instructions that balances coverage and diver- sity. This choice keeps the experiments practical and reproducible, while still being large enough to demonstrate the effectiveness of our dataset for grounding tasks. Further details on the composition of this subset, along with the choices made in its construction, are provided in Appendix C.I.
the instruction tuning dataset introduced in
RL Post-training. In the next stage, we adopted RL post-training and explored several heuristics for constructing training data. GROUNDCUA allows us to sample from a much larger pool than the one used for SFT, so we selected 10K new elements not included in the original 700K SFT training set. This approach yielded the strongest generalization across benchmarks in our initial experiments, and we adopted it for the final model.
For policy optimization, we employed the Relative Leave-One-Out (RLOO) method (Ahmadian et al., 2024), which compares the reward of each rollout to the average reward of other samples within the same group. avoiding the need for training a separate critic model. Concretely, for a group of n rollouts (.1....,, 3. the gradient is given
VOJ(TO) - Σe (R(y..)
Σ 3
R(y,, Vo log on (v(2z),
n2
where R(y;, z) is the reward assigned to output y; given the input z. In our grounding setup, each Yi corresponds sequence of tokens representing the predicted coordinates (f,) on the image and x corresponds to the input prompt and image.
Reward Function. We designed a customized discrete reward based on the normalized distance
-1.0 D Dnoom < -0.5,
-0.5 if - 0.5 < D Dnom < 0.1,
-0.1 if - 0.1 < Dnorm < 0,
Recore (p. B,I)=
0.1 if0 S Drorm 0.1,
0.5 if < D Doorm < 0.5,
1.0 if D Dnorm 2 0.5.
The normalized distance is defined as Dnorm = P(A.)) where Dref diam B) if E and Dinf
2
Dref Dmmrr (B, I) otherwise. D(g, B) is the signed distance between the predicted coordinate á and the ground-truth bounding box B, with positive values inside. We use half the bounding box diameter if B because that is the maximum distance point inside B can have from the boundary. This results in a Drorm value between -I and 1.
predictions just outside the box receive milder penalty, while predictions far outside receive a stronger one, and predictions inside the box encouraged to move toward the center. We exclude reward model-based approaches due to the unreliable nature of current judges (Feizi et al., 2025; Là et al., 2025). We experimented with alternative reward formulations (e.g., continuous and binary schemes), but ultimately adopted this discrete variant due to its superior empirical performance (see Appendix C.4 for details). We set the group size 8, the batch size 64, and trained for one epoch on a single H100 node (8 GPUs), consistent with the SFT setup.
This discrete scheme captures dominant error modes:
4.2 EVALUATION
Task Definition. Given a screenshot / and user instruction x, the model predicts a 2D point (Ū, 6) in image coordinates. Let B denote the axis-aligned ground-truth bounding box for the target element. A prediction is marked correct if p E B, and incorrect otherwise. We report the accuracy metric.
We evaluate GROUNDNEXT on five key benchmarks that cover a wide range of grounding scenarios. For desktop applications, we ScreenspotPro (Li et al., 2025), OSWorld-G
Benchmarks.

---

Published as a conference paper at ICLR 2026
Table SFT-only results on five challenging benchmarks. Results are shown for both 3B and 7B model scales. Only por-perfognnng models are presented here; see Section D for full comparisons with additional baselines. Our GROUNDNEXT (SFT) consistently achieves the best average perfor- mance across all benchmarks, demonstrating the effectiveness of our high-quality data.
Model
SSPro
OSW-G
MMB-GUI
SSv2
UI-V
Avg
3B
Qwen2.5-VI.-BB (Bai et al., 2025)
16.1
27.3
60.8
80.9
6.3
38.3
Qwen2.5-VI.33B (Agent mode)
29.0
37.4
60.8
81.8
6.3
43.1
PhiGround-4B-77 (Zhang et al., 2025)
22.8
51.4
60.3
80.8
20,5
47.2
JEDI-3B (Xie al., 2025)
36.1
50.9
66.5
88.6
18.7
52.2
GUI-Actor-3B (Wu et al., 2025)
42.2
48.9
69.8
91.0
19.7
54.3
GROUNDNEXT-3B (SFT)
48.6
62.2
75.5
87.3
58.2
66.4
Qwen2.5-VL--7B (Bai et al., 2025)
26.8
31.4
33.9
88.8
0.9
36.4
Owen2.5-VL-7B (Agent mode)
29.7
42.7
67.7
86.4
16.5
48.6
OS-Atlas-7B 7B (Wu ct al., 2024)
18.9
27.7
41.4
85.1
9.0
36.4
UGround-Ve V1-7B (Gou al., 2024)
16.5
36.4
65.7
87.6
12.9
43.8
Aguvis-7B (Xu et al., 2024)
39.5
38.7
45.7
86.0
13.7
44.7
GUI-Actor-7B (Wu et al., 2025)
44.6
47.0
70.9
92.1
21.9
55.3
JEDI-7B (Xie et al., 2025)
39.5
54.1
70.4
91.7
24.8
56.1
GROUNDNEXT-7B (SFT)
50.2
67.2
80.4
89.3
58.7
69.2
(Xie et al., 2025), and Vision (Nayak et al., 2025), which focus on desktop interactions. To test cross-platform performance, also MMBench-GUUI (L2) (Wang al., 2025b) and Screenspot- v2 (Cheng et al., 2024), which include mobile and web splits in addition to desktop. This mix of benchmarks lets us evaluate performance across desktops, mobile, and web environments. Since UI- Vi-Vision overlaps with our dataset in platform coverage, we treat it as in-domain benchmark, while the others are out-of-domain. We make efforts minimize overlap during training, but due to annotation differences and the repetitive nature of desktop software, perfect separation isn' always possible.
Baselines. We compare GROUNDNEXT against two main types of baselines. First, we evaluate GROUNDNEXT (SFT) alongside several SFT-only variants to measure the impact of instruction data (see Table 2). Then, we compare GROUNDNEXT (RL) with recent reinforcement learning- based models to assess the effectiveness of RL fine-tuning (see Table 3).
5 RESULTS
5.1 EFFICIENT SUPERVISED FINE-TUNING WITH HIGH-QUALITY DATA
We present the performance results of our models trained using SFT across five benchmarks in Table 2. Our models achieve the highest average performance for both 3B and 7B model sizes. For <3B, GEX-ND (SFT) ranks first most datasets (except SSv2) and leads the SFT--nly group by clear margin with an average performance of 68.4 vs. 63.0 for the next best (GUI-Actor- 3B) without considering UI-V. and 66.4 vs. 54.3 with UI-V (i.e., +5.4 and +12.1 points, respectively). Notably, our 3B SFT average also surpasses all RL-tuned 3B baselines. Adding the RL stage yields small, consistent lift 68.4 Avg 70.0 (w/o UI-V), setting the best overall results in this size range. For 27B, GROUNDNEXT-77 (SFT) also leads among SFT-only models with 71.8 Avg without UI- and 69.2 with, outperforming the next best SFT baseline (JEDI-7B) by +7.9 and +13.1 points, respectively. Among RL-tuned systems, GROUNDNEXT-7B (RL) attains the top Avg (w/o UI-V) 73.0. Overall, these results indicate the efficacy our high quality data. Notably, our results achieved with substantially less data and modest compute. We train only 700K instructions, which far below outperform larger SFT baselines and remain competitive with RI--tuned systems. This suggests that
salllion-samlle corpora used by prior work (e.g.. JEDI 9M). Yet. we

---

Published as a conference paper at ICLR 2026
Table 3: RL-tuned results. We present results for the 3B and 7B model scales. We highlight the top-performing models here and refer readers to Section D for full comparisons with additional baselines. Our GROUNDNEXT(RL) achieves the highest average performance.
Model
SSPro
OSW-G
MMB-GUI
SSv2
UI-V
Avg
3B
UI-RI-E-3B (Lu al., 2025)
17.8
48.8
68.4
88.6
16.5
48.0
SE-GUI-3B (Yuan et al., 2025)
35.9
46.1
66.3
86.8
15.0
50.0
InfiGUI-RI-3B (Liu et al., 2025a)
35.7
42.9
70.6
89.5
17.8
51.3
GUI G²-3B (Tang et al., 2025)
36.4
53.5
66.3
87.6
18.7
52.5
GUI-G1-3B (Zhou et al., 2025)
37.1
49.5
71.0
89.5
20.3
53.5
InfiGUI-G1-3B (Liu et al., 2025b)
45.2
49.6
73.4
91.1
22.0
56.3
GROUNDNEXT-3B (SFT)
48.6
62.2
75.5
87.3
58.2
66.4
GROUNDNEXT-3B (RL)
49.8
64.2
77.1
88.8
62.1
68.4
2 7B
SE-GUI-7B (Yuan et al., 2025)
47.3
33.9
34.5
68.9
16.7
40.3
UI-TARS-1.5 1.5-7B (Qin et al., 2025)
49.6
64.2
64.3
90.3
20.8
57.8
GUI G2-7B (Tang al., 2025)
47.5
61.9
79.5
93.3
25.6
61.7
InfiGUI-GI-7B (Liu al., 2025b)
51.9
59.9
80.8
93.5
26.1
62.4
GTA1-7B (Yang al., 2025)
50.1
67.7
79.4
92,4
25.7
63.1
GROUNDNEX1-7B (SFT)
50.2
67.2
80.4
89.3
58.7
69.2
GROUNDNEXT-7B (RL)
52.9
67.7
81.1
90.4
60.3
70.5
high-quality, densely grounded supervision and targeted instruction design can substitute for raw scale, delivering strong gains without escalating data volume or compute.
GROUNDCUA compared to other SFT training corpora To make fair comparison and high- light the quality of our dataset, we train the same base model -0ween..5VV.-3B--nnniti
on 100K
samples from each of the following datasets: Aguvis, UGround, OS-Atlas (Desktop), JEDI, and GROUNDCUA. We use identical hyperparameters and preprocessing for experiments (details in the Appendix). Figure 3 (yellow bars) summarizes the average performance across benchmarks, cluding UI- Vision. We observe that GROUNDCUA yields significantly higher SFT averages than all other training sources, demonstrating the benefits of its high-quality, densely grounded supervision.
5.2 REINFORCEMEN LEARNING POST-TRAINING RESULTS
RL post-training on top the SFT models results in consistent but modest improvements for both the 3B and 7B models shown in Table For the 3B (RL) achieves aver-
SCT bugl
jain bwth qualomundt
00.0
model. GROUNDNEXT-2 age of 70.0 (without UI- ()-V) and 68.4 overall, surpass- ing the SFT-only model, GROUNDNEXT-3B (SFT), which scores 68.4 and 66.4, respectively. For the
57.5
+4.3
50.0
7B model, GROUNDNEXT-7B (RL) achieves 70.5, improving upon GROUNDNEXT-TR (SFT)'s 69.2 (with UI-V). These results suggest that SFT, when trained with high-quality data, captures the major- ity of the model's performance, with RI. offering targeted fine-tuning that provides incremental im- provements. In practice, high-quality SFT can es- tablish strong baselines, and RL can serve an optional refinement step to further enhance perfor- mance. While reward design is simple, knowledge that more sophisticated reward functions, such as those Liu et al. (2025b), could lead more substantial RL gains, which leave for future work.
47.5
40.0
05.0m (Dcsktop)
JA3
Agavn
Utroard
Figure 3: Mean SFT scores (orange)
across benchmarks, with RL gains from 10k GROUNDCUA samples shown in blue.
8

---

Published as a conference paper at ICLR 2026
Table Agentic performance comparison OSWorld-Veriltte, Bold and underline indicate the best-per pest-performing open-source model in each category. Our 3B model, GROUND is among the oop-performing open-source models, surpassing larger and proprietary models, highlighting its practical utility and efficiency for real-world agentic tasks.
Model
OS
Office
Daily
Pro
Workflow
Overall
Proprietary Models
OpenAl o3 (OpenAI, 2025)
62.5
14.5
21.4
38.8
16.5
23.0
CUA (OpenAI, 2025)
23.9
34.6
55.1
18.3
18.3
31.4
Claude-4-Sonne (Anthropic, 2025a)
45.8
39.3
48.1
59.2
27.9
41.4
Qwen3-VL-Flash (Bai et al., 2025)
40.9
53.6
55.1
22.0
22.0
41.6
T1-TARS-250700 (Qin et al., 2025)
41.7
50.4
55.7
51.0
14.7
41.8
Claude-4.5 4.5nSonnet (Anthropic, 2025b)
70.8
72.6
61.4
63.3
49.0
62.9
Open-source Models
Qwen2.5-VL- VL-32BB (Bai et al., 2025)
8.3
1.7
6.4
6.1
2.2
3,9
Qwen2.5-VL VL-72B (Bai et al., 2025)
16.7
4.3
6.4
2.0
3,2
5.0
Kimi- (Kimi Team, 2025)
12.5
6.0
21.7
18.4
1.1
10.3
OpenCUA-A3B (Wang et al., 2025a)
12.5
16.3
21.7
46.9
2.2
17.7
UI-TARS-72B-DPO (Qin et al., 2025)
37.5
19.0
34.6
63.3
8.3
27.1
OpenCUA-77 (Wang et al., 2025a)
41.7
22.2
37.1
49.0
9.3
27.0
UI-TARS- 1.5-BB (Qin et al., 2025)
33.3
29.9
37.9
53.1
9.1
29.6
OpenCUA-72B (Wang al., 2025a)
58.3
47.0
53.8
73.5
20,4
46.1
JEDI-7B w/ (Xie al., 2025)
46.1
61.9
75.5
35.3
51.0
GROUNDNEET-3B (RL) w/ o3 (ours)
62.5
47.0
55.0
73.5
36.5
50.6
Analyzing RL Gains. We investigate the modest gains from RL (see Figure 3). In this setup, we start with SFT models D0wen2.5-VLL-3-Innse
trained on Aguvis, UGround, OS-Atlas (Desk- top), JEDI, and GROUNDCUA, and then apply RL using 10K examples exclusively from GROUND- CUA. find that models trained with GROUNDCUA during SFT show the smallest performance gains from RL, while models trained GROUNDCUA. This suggests that SFT with GROUNDCUA already provides highly informative supervision, leaving fewer enrors for RI. correct. Moreover, the magnitude of RL. improvements correlates with the initial SFT performance: stronger SFT models yield smaller absolute gains be- cause they start with fewer remaining errors. We explore this phenomenon in greater detail Sec- tion D.6.
other datasets benefit more from RL fine-tuning with
5.3 FURTHER ANALYSIS
Agentic Performance We evaluate GROUNDNEXT's performance in an agentic setting to assess its ability ground in realistic, multi-step tasks. Experiments conducted the OSWorld- Verified benchmark using OpenAl o3 as the planner, which consumes task instructions and action history generate grounding commands that GRoUNDNEXT executes to locate target UI elements on the screen. Following the setup of (Xie et al., 2025; Yang et al., 2025; Wang et al., 2025a), we evaluate 361 tasks (excluding Google Drive-related ones) on an Ubuntu system with a 1920x1080 resolution, running on Microsoft Azure within 10 Docker environments.
The results in Table 4 highlight GROUNDNEXT-3B's strong performance. Within its 3B param- eter class, GROUNDNEXT-33 (50.6 Overall) significantly outperforms peers like OpenCUA-A3B (17.7) and Kimi-VL--A3K (10.3). Notably, it surpasses many larger models, including OpenCUA- 72B (46.1) and proprietary APIS such as Qwen3-VL--Flash (41.6) and Claude-4-Sonnet (41.4). The comparison with JEDI-7B, which also uses the o3 planner, particularly notable. Despite being less than half the size, our 3B model achieves comparable overall score (50.6 vs. 51.0) and demon- strates superior performance in 3 out of 5 categories (OS, Office, and Workflow). This performance from compact 3B model underscores GROUNDNEXT-3B significant practical utility, presenting as an effective and efficient solution for real-world agentic systems where inference speed and resource constraints critical factors.

---

Published as a conference paper at ICLR 2026
Gains from GROUNDCUA. We investigate where GROUNDCUA yields the greatest gains by studying the performance of GROUNDNEXT. Since GROUNDCUA primarily covers desktop soft- ware, we expect the largest gains on desktop benchmarks. Our results confirm this: GROUND- NEXT-7B (RL) achieves the best performance on UI-V, OSW-G, and SSPro. For mixed datasets such as MMBench-GUU, GROUNDNEXT shows 3.66% improvement the recond-best model, InfiGUI-GI, with notable gains coming from Linux and macOS (see Sec- tion D.3). At the element level, the most significant improvements are observed in icon recognition. For example, on SSPro, we outperform most models by an average of 10.7% in icon recognition (see Table 11). This reflects the high density of icons in desktop applications and suggests that the diversity in GROUNDCUA provides richer knowledge, leading to better performance on icons.
desktop platforms over
GROUNDNEXT generalization across domains. Next, we evaluate the generalization ability of GROUNDNEXT, trained primarily on desktop software, to mobile and web interfaces using SSv2 and MMBench-GUI. On MMBench-GUI, GROUNDNEXT-7B (RL) performs competitively across both domains, achieving 89.2% on mobile and 81.9% on web, compared to the next best model, i.., InfiGUI-G1-7B. at 90.9% and 85.3%, respectively. On SSv2, GROUNDNEXT achieves compa- rable results on mobile but falls behind on web. A detailed error analysis is provided in Section E. These results suggest that while GROUNDCUA enables strong cross-domain generalization, future work could explore augmenting desktop data with web and mobile sources to further enhance per- formance.
Effects of using open source applications. To study the impact of open-source software, we ex- amine SSPro performance across various categories, focusing particularly on icon recognition. Icons often require application-speet
knowledge, unlike text, which more general in nature. As shown in Table 11, GROUNDNEXT achieves the best icon performance in the Office Suite, Development, Creative, Scientific, and CAD categories, and ranks second in OS. The presence of open-source of- fiee software, such as LibreOffice, likely contributes to the strong results in the Office Suite category. Similarly. the diversity of open-source development tools and creative software, such as video and image editing programs, results in significant improvements, with our model outperforming the next best model, i.c. InfiUU--G-77B by 15.9% in Development and 8.4% in Creative for icon accu- racy. Future work could further analyze the impact of application similarity to determine whether applications more similar to those in our dataset lead to higher performance.
CONCLUSION & DISCUSSION
We introduced GROUNDCUA, a human-annotaie desktop grounding dataset spanning 87 applica- tions (56K screenshots, 3. 3.56M+ elements) with dense keyframe labels that reflect real interaction states. From these annotations, we constructed real-world computer-use use instruction tasks for ground- ing. We developed the GROUNDNEXT family of models and following recent trends, trained it first with SFT and then RL on verifiable rewards. Across five challenging benchmarks, GROUNDNEXT achieves state-of thefthh-att results despite using substantially less SFT training data than many prior works. The key takeaway is that high-quality data drives reliable desktop grounding more effectively than sheer data volume. By releasing both the dataset and other research artifacts, we aim unlock grounding as a core capability, laying the foundation for end-to-end computer-use agents that can perform complex tasks across diverse desktop applications.
While this work advances desktop grounding and demonstrates the value of high-quality expert demonstrations, it also opens up new opportunities and raises important questions. First, we train models with limited scale and compute, but the dataset can support variable-sized fine-tuning sets to further scale model performance. Second, our dense annotations should enable the develop- ment of precise and expressive reward signals for RL, moving beyond the simplistic one used in this paper. This creates opportunities to systematically study how different reward designs impact grounding accuracy. Third, cross-domain generalization remains key frontier. Desktop environ- ments involve complex, multi-window workflows, whereas mobile and web tasks are lighter and more context-specific. Mixing data across these domains could yield models that operate seamlessly across platforms, though balancing these domains and addressing transfer bottlenecks will require careful study. Finally, GROUNDCUA includes platform- and category- level metadata, enabling search on continual learning and adaptation, evaluating how agents adapt to unseen applications and continually improve new interaction paradigms emerge.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
Our work focuses on the responsible development of computer-use agents through transparent dataset curation and model training. We have taken significant steps to protect user privacy by ensuring all desktop applications used are open-source with permissive licenses, and no personally identifiable information (PII) was collected during screenshot annotation. All human annotators were fairly compensated and worked under proper data protection protocols. While have fil- tered potentially harmful content from our dataset, we cannot fully guarantee that models trained on GROUNDCUA will not generate inappropriate instructions interact with sensitive interface elements inappropriately. Users and developers are strongly encouraged to implement appropriate safeguards and human oversight when deploying computer-use agents in production environments. Additionally, all human evaluation studies were conducted by collaborating researchers following established ethical guidelines, with no PII collected during the evaluation process.
We disclose that there are no conflicts of interest that would bias this work. Any funding sources will be listed in the camera-ready version per ICLR policy.
REPRODUCIBILITY STATEMENT
We are committed to ensuring full reproducibility of our work by providing comprehensive im- plementation details and releasing all necessary resources. All artifacts, including the complete GROUNDCUA dataset, GROUNDNEXT model weights, training code, evaluation scripts, and de- tailed data sheets, will be publicly released. We have thoroughly documented all hyperparameters, training procedures, data preprocessing steps, and evaluation metrics in the appendices to enable accurate replication of our results. The human annotation guidelines, quality assurance protocols fully described to ensure transparency in dataset creation. The instruction generation prompts, model architectures, and benchmark evaluation procedures are detailed to facilitate consistent repro- duction across different research groups.
REFERENCES
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learn- ing from human feedback in llms. arXiv preprint arXiv:2402.14744 2024.
Anthropic.
Introducing computer use, new claude 3.5 sonnet, and claude 3.5 haiku, October 2024. URL https:////// / www. anttroppiiii. com/ news - and- wttpeelssa-accccoo-t--------
Anthoopic. Introducing claude 4, May 2025a. URL claude-4.
Anthropic. Introducing claude sonnet 4.5, September 2025b. com/newe / claude sonnet 5-
Chongyang Bai, Xiaoxue Zang, Ying Xu, Srinivas Sunkara, Abhinav Rastogi, Jindong Chen, et al. Uibert: Learning generic multimodal representations for ui understanding. arXiv preprint 1IXi::2107.13731. 2021.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, al. Qwen2. 5-vl technical report. arXiv preprint 13923, 2025.
Yuxiang Chai, Siyuan Huang, Yazhe Niu, Han Xiao, Liang Liu, Dingyu Zhang, Shuai Ren, and Hongsheng Li. Amex: Android multi-annotation expo dataset for mobile gui agents, 2025. URL
orgooo////aoo.o alts////////// 2407 17490.
Wentong Chen, Junbo Cui, Jinyi Hu, Yujia Qin, Junjie Fang, Yue Zhao, Chongyi Wang, Jun Liu, Guirong Chen, Yupeng Huo, et al. Guicourse: From general vision language models to versatile gui agents. arXiv preprint arXiv:2406.1 1317, 2024.

---

Published as a conference paper at ICLR 2026
Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiy- ong Wu. Seeclick: Harnessing gui grounding for advanced visual gui agents. arXiv preprint arXiv:2401.10935, 2024.
Cheng Cui, Ting Sun, Manhui Lin, Tingquan Gao, Yubo Zhang, Jiaxuan Liu, Xueqing Wang, Zelun Zhang, Changda Zhou, Hongen Liu, Yue Zhang, Wenyu Lv, Kui Huang, Yichao Zhang, Jing Zhang, Jun Zhang, Yi Liu, Dianhai Yu, and Yanjun Ma. Paddleocr 3.0 technical report, 2025. URL wwtpt:tpp///... // //wxivvvvvvv v. org/ abs /2077 05595.
Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar. Rico: A mobile app dataset for building data-driven design ap- plications. In Proceedings of the 30th amnual ACM symposium on user interface software and technology, pp. 845-854, 2017.
Aarash Feizi, Sai Rajeswar, Adriana Romero-Soriano, Reihaneh Rabbany, Valentina Zantedeschi, Spandana Gella, and João Monteiro. Pairbench: Are vision-language models reliable at compar- ing what they see?, 2025. URL https::t///aat /ttp:///axivv.vii org/org//////o ht/p/bbs/255000 2502 15210.
Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. Navigating the digital world as humans do: Universal visual grounding for gui agents. arXiv preprint arXiv:2410005233 2024.
Daya Guo, Dejian Yang, Haowei Zhang. Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint (ttt:::2501.194444 2025.
Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. Cogagent: A visual language model for gui agents. arXiv preprint wwx.x:23122.881222 2023.
Kimi Team. Kimi-vl technical report, 2025. URL
Kaixin Li, Ziyang Meng, Hongzhan Lin, Ziyang Luo, Yuchen Tian, Jing Ma, Zhiyong Huang, and Tat-Seng Chua. Screenspot-pro: Gui grounding for professional high-resolution computer use, 2025.
Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge. Mapping natural language instructions to mobile ui action sequences. arXiv preprint arXiv:2005.03776, 2020a.
Yang Li, Gang Li, Luheng He, Jingjie Zheng, Hong Li, and Zhiwei Guan. Widget caption-
ing: Generating natural language description for mobile user interface elements. arXiv preprint arXiv:2010.04295 2020b.
Kevin Qinghong Lin, Linjie Li, Difei Gao, Zhengyuan Yang, Shiwei Wu, Zechen Bai, Weixian Lei, Lijuan Wang, and Mike Zheng Shou. Showui: One Assion-anguaage-cttinn model for gui visual agent, 2024. URL httpp:://axxi..rrrr/./.. 2b111 17465.
Yuhang Liu, Pengxiang Li, Congkai Xie, Xavier Hu, Xiaotian Han, Shengyu Zhang, Hongxia Yang, and Fei Wu. Infigui-rl: Advancing multimodal gui agents from reactive actors to deliberative reasoners. arXiv preprint arXiv:2504. /4239, 2025a.
Yuhang Liu, Zeyu Liu, Shuanghe Zhu, Pengxiang Li, Congkai Xie, Jiasheng Wang, Xueyu Hu, Xiaotian Han, Jianbo Yuan, Xinyao Wang, Shengyu Zhang, Hongxia Yang, and Fei Wu. Infigui- gl: Advancing gui grounding with adaptive exploration policy optimization, 2025b. URL https:/ htt.xvivvvv. org/ alos/ / 2208 05731.
Zhengxi Lu, Yuxiang Chai, Yaxuan Guo, Xi Yin, Liang Liu, Hao Wang, Han Xiao, Shuai Ren, Guanjing Xiong, and Hongsheng Li. Ui-r1: Enhancing efficient action prediction of gui agents by reinforcement learning, 2025. URL https:/ / arxiv. org/ /abs//////// 2503 620..
Run Luo, Lu Wang, Wanwei He, and Xiaobo Xia. Gui-rl: A generalist rl-style langugle action model for gui agents. arXiv preprint ar0iv:2504. 10458, 2025.
12

---

Published as a conference paper at ICLR 2026
Xing Han Lü, Amirhossein Kazemnejad, Nicholas Meade, Arkil Patel, Dongchan Shin, Alejan- dra Zambrano, Karolina Starczakk, Peter Shaw, Christopher Pal, and Siva Reddy. Agentre- wardbench: Evaluating automatic evaluations of web agent trajectories, 2025. URL https:
//arxii.orr///aaaeaaa../.a /a/s//b//// 22504. 08942.
Lu Ma, Hao Liang, Meiyi Qiang, Lexiang Tang, Xiaochen Ma, Zhen Hao Wong, Junbo Niu, Chengyu Shen, Runming He, Yanhao Li, Bin Cui, and Wentao Zhang. Learning what inforcement learning can't: Interleaved online fine-tuning for hardest questions, 2025. URL
https://arxiv. org/ abs 2502500... 07527.
Viktor Muryn, Marta Sumyk, Mariya Hirna, Sofiya Garkot, and Maksym Shamrai. Screen2ax: Vision-based approach for automatic macos accessibility generation, 2025. URL https://
arxiv.org/ abs//abs///// 2507 16704.
Shravan Nayak, Xiangru Jian, Kevin Qinghong Lin, Juan A. Rodriguez, Montek Kalsi, Rabiul Awal, Nicolas Chapados, M. Tamer Özsu, Aishwarya Agrawal, David Vazquez, Christopher Pal, Perouz Taslakian, Spandana Gella, and Sai Rajeswar. Ui-vision: A desktop-centric gui benchmark for visual perception and interaction, 2025. URL https:://///cc.t http///xxiv..o org/ abcpgg//bb//5550 2503. 200../15611..1111.... 15661.
OpenAL. Introducing openai o3 and 04-mini, April 2025.
URL htttt::///peeeaaa com/
OpenAI. Computer-using agent, January 2025. URL http:://opeeaa..com/inn
computer-using-aggeee
Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale, 2024. URL https: // /arxiv.. org/ abs /2406. 17557.
Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jia- hao Li, Yunxin Li, Shijue Huang, Wanjun Zhong, Kuanye Li, Jiale Yang, Yu Miao, Woyu Lin, Longxiang Liu, Xu Jiang, Qianli Ma. Jingyu Li, Xiaojun Xiao, Kai Cai, Chuang Li, Yaowei Zheng, Chaolin Jin, Chen Li, Xiao Zhou, Minchao Wang, Haoli Chen, Zhaojian Li, Haihua Yang, Haifeng Liu, Feng Lin, Tao Peng, Xin Liu, and Guang Shi. Ui-tars: Pioneering tomated gui interaction with native agents. arXiv preprint arXiv:2501.12322 2025. URL https: / / / thub. com / wwwpyyedecceeeeeeeeeeeeoe /UII-A/IITTT..TTA
Fei Tang, Zhangxuan Gu, Zhengxi Lu, Xuyang Liu, Shuheng Shen, Changhua Meng, Wen Wang, Wenqi Zhang, Yongliang Shen, Weiming Lu, et al. Gui-g Gaussian reward modeling for gui grounding. arXiv preprint arXiv:2507.1 /5846, 2025.
Xinyuan Wang, Bowen Wang, Dunjie Lu, Junlin Yang. Tianbao Xie, Junli Wang, Jiaqi Deng, Xiaole Guo, Yiheng Xu, Chen Henry Wu, Zhennan Shen, Zhuokai Li, Ryan Li, Xiaochuan Li, Junda Chen, Boyuan Zheng, Peihang Li, Fangyu Lei, Ruisheng Cao, Yeqiao Fu, Dongchan Shin, Martin Shin, Jiarui Hu, Yuyan Wang. Jixuan Chen, Yuxiao Ye, Danyang Zhang, Yipu Wang, Heng Wang, Diyi Yang. Victor Zhong, Y. Charles, Zhilin Yang, and Tao Yu. Opencua: Open foundations for computer-use agents, 2025a. URL https::///// /opencua x allang.aa.
Xuehui Wang, Zhenyu Wu, JingJing Xie, Zichen Ding, Bowen Yang, Zehao Li, Zhaoyang Liu, Qingyun Li, Xuan Dong, Zhe Chen, Weiyun Wang, Xiangyu Zhao, Jixuan Chen, Haodong Duan, Tianbao Xie, Chenyu Yang, Shiqian Su, Yue Yu, Yuan Huang, Yiqian Liu, Xiao Zhang, Yant- ing Zhang, Xiangyu Yue, Weijie Su, Xizhou Zhu, Wei Shen, Jifeng Dai, and Wenhai Wang- Mmbench-gui: Hicrarchical multi-platfomn evaluation framework for gui agents, 2025b. URL https: / arxiv. org/ alos/ 2207. 19478.
Qianhui Wu, Kanzhi Cheng, Rui Yang, Chaoyun Zhang, Jianwei Yang, Huiqiang Jiang, Jian Mu, Baolin Peng, Bo Qiao. Reuben Tan, Si Qin, Lars Liden, Qingwei Lin, Huan Zhang, Tong Zhang, Jianbing Zhang. Dongmei Zhang, and Jianfeng Gao. Gui-actor: Coordinate- free visual grounding for gui agents, 2025. URL https::////....... /aaaxivvoorrgg.eeaaae..a.eeee... www./as//////.oooooooeo.......ae 22... 6. 03143.
Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, and Yu Qiao. Os-atlas: A foundation action model for generalist gui agents, 2024. URL https: // wxw...rg/eeeee..... /21 23218.
13

---

Published as a conference paper at ICLR 2026
Tianbao Xie, Jiaqi Deng. Xiaochuan Li, Junlin Yang, Haoyuan Wu, Jixuan Chen, Wenjing Hu, Xinyuan Wang, Yuhui Xu, Zekun Wang, et al. Scaling computer-use grounding via user interface decomposition and synthesis. arXiv preprint 13227 2025.
Yiheng Xu, Zekun Wang, Junli Wang. Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, and Caiming Xiong. Aguvis: Unified pure vision agents for autonomous gui interaction, 2024.
URL https: arxiv. org/ abw//422511 04454.
Yan Yang, Dongxu Li, Yutong Dai, Yuhao Yang. Ziyang Luo, Zirui Zhao, Zhiyuan Hu, Junzhe Huang, Amrita Saha, Zeyuan Chen, Ran Xu, Liyuan Pan, Caiming Xiong, and Junnan Li. Gtal: Gui test-time scaling agent, 2025. URL https: arxiv. org/ abs 2507.05791.
Yuhao Yang, Yue Wang, Dongxu Li, Ziyang Luo, Bei Chen, Chao Huang, and Junnan Li. Aria-ui: Visual grounding for gui instructions. arXiv preprint arXiv:2412.16256, 2024.
Keen You, Haotian Zhang, Eldon Schoop, Floris Weers, Amanda Sweamgin, Jeffrey Nichols, Yinfei Yang, and Zhe Gan. Ferret-ui: Grounded mobile ui understanding with multimodal llms. arXiv preprint arXiv:2404. 057/1, 2024.
Xinbin Yuan, Jian Zhang, Kaixin Li, Zhuoxuan Cai, Lujian Yao, Jie Chen, Enguang Wang. Qibin Hou, Jinwei Chen, Peng- Tao Jiang, et al. Enhancing visual grounding for gui agents via self- evolutionary reinforcement learning. arXiv preprint arXiv:2505. 12370, 2025.
Miaosen Zhang, Ziqiang Xu, Jialiang Zhu, Qi Dai, Kai Qiu, Yifan Yang, Chong Luo, Tianyi Chen, Justin Wagle, Tim Franklin, et al. Phi-ground tech report: Advancing perception in gui grounding. 2025.
arXiv preprint arXiv:2507.23777
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye. Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Pro- ceedings of the 62nd Annual Meeting of the Association for Computational Linguisties (Volume 3: System Demonstrraionss, Bangkok, Thailand, 2024. Association for Computational Linguis- tics. URL /ttt:///rxiv.. 2ttp://aaa22223333333
Yuqi Zhou, Sunhao Dai, Shuai Wang, Kaiwen Zhou, Qinglin Jia, and Jun Xu. Gui-gl: Un- derstanding r1-zero-like training for visual grounding in gui agents, 2025. URL https: //aaxiv.crrt wttp://.wwww/////.......2 15810.
14

---

Published as a conference paper at ICLR 2026
APPENDIX
Table of Contents
Page
A. GROUNDCUA Creetion
16
A.I Platforms
16
A.2 Human Annotation
16
A.3 Dataset Statisties
A.4 Comparision with Prior Works
17
A.5 Dataset Examples
19
B. Instruction Tuning Data
19
B.1 Direct Instructions
19
B.2 Functional Instructions
B.3 Spatial Instructions
22
B.4 Examples
23
B.5 Need and Impact of Different Instruction Types
22
B.6 Quality of Instruction Tuning Data
23
Training
C.1 SFT Data
23
C.2 SFT Training Details
C.3 RL Data
25
C.4 RL Training Details
25
C.5 RL Design Choices and Hyperparameters
27
D. Evaluation
28
D.1 ScreenSpot-PPe Results
28
D.2 OSWorld-G Results
28
D.3 MMBench- GUI Results
28
D.4 ScreenSpot-v2 Results
28
D.5 UI-Vision Results
28
D.6 Discussion on RL Gains
30
D.7 SFT Scaling Behavior
32
E. GROUNDNEXT Error Analysis
33
G. Limitations
H. LLM Usage
34
15

---

Published as a conference paper at ICLR 2026
A GROUNDCUA- CREATION
A.1 PLATFORMS
Table 5: Categories of desktop applications and their corresponding applications.
Category
Platforms
Education
Anki, Zotero, Calibre, OpenBoard, Mendeley
Browsers
Brave, Chromium, Mozilla Firefox, DuckDuckGo
Development
VSCode, Atom, Eclipse, NetBeans, PyCharm, IntelliJ IDEA, Brackets, Geany, Bluefish, KDevelop, Komodo Edit, Code::Blocks, Qt Creator, Arduino IDE
Productivity
LibreOffice Cale, LibreOffice Draw, LibreOffice Impress, LibreOffice Writer, draw.io, Joplin, OpenProject, Affine, PDFedit, OnlyOffice Calendar, OnlyOffice Document Ed- itor, OnlyOffice Forms, OnlyOffice PDF Forms, OnlyOf- fice Presentation, OnlyOffice Spreadsheet, Nexteloud, Gnu- meric, Simplenote, WeKan
Graphics and Design
Blender, GIMP, Inkscape, Krita, darktable, FontForge. Scribus, WordPress
Video and Audio Production
OpenShot, OBS Studio, Lightworks, Shotcut, Natron, OpenToonz, Audacity, MuseScore
Communication
Element, Signal, Mastodon, Lemmy, Matrix, Zulip, Jitsi
Entertainment
VLC Media Player, Kodi, Emby
System Utilities
Ubuntu Terminal, Conky, Bash, 7-Zip, Flameshot, Nemo, gedit
Security
Bitwarden, Cryptomator
Finance and Business Analytics
GnuCash, Frappe Books, Metabase
Scientific
RStudio, Veusz, GNU Octave, GrassGIS, QGIS, FreeCAD, Spyder
We select 87 platforms, focusing on open-source software with permissive licenses. These ap- plications span 12 diverse categories, detailed in Table 5. Our selection is motivated by the under-representation
such platforms in existing datasets and the flexibility provided by permis- sive licensing, which enables dataset release with minimal restrictions. We primarily rely UI- Vision (Nayak et al., 2025) as the source for platforms, as they motivated their platform selection similarly. We additionally include platforms improve coverage across finance and scientific cat- egories. We further show that this choice does not compromise generalization (see Section 5.3), as the open-source software usually shares UI elements and layout with its closed-souree counterparts. For instance, LibreOffice and Office Suite share many interface elements, layout, and functionality. This ensures broader applicability of GROUNDCUA.
A.2 HUMAN ANNOTATION
We collaborated with a professional data labeling vendor that specializes in dataset curation for Al applications. The annotation effort spanned three phases, beginning with a pilot study where we worked closely with the annotation team to refine task instructions and provide iterative feedback. The annotation team consisted of around 70 individuals, organized into multiple tiers of annotators, quality assurance specialists, and project managers. The majority of the team was located India and Latin America, with participants in the 20-35 year age group and a balanced gender distribution. All annotators held at least bachelor's 's degree in technical fields such as Computer Science or Engineering and had prior experience in data labeling and user interface research.
16

---

Published as a conference paper at ICLR 2026
Table UI element categories in GROUNDCUA with descriptions and representative examples.
Category
Description and Common UT Elements
Input Element
Interactive fields where users enter or modify data, like text boxes, checkboxes, radio buttons, etc.
Sidebar
Vertical horizontal panels that provide quick access tools or navigation. Examples include tool palettes, folder trees, settings sidebars.
Information Display
Regions that primarily present textual numerical information. Examples in- clude labels, console outputs, document text, and code blocks.
Button
Clickable controls that trigger an action like submit button, "OK/Cancel" but- tons, play/pause buttons
Navigation
Elements that help users move within or across applications. Examples: tabs, hack/forward arrows etc.
Visual Elements
Non-textual graphical elements that convey information functionalityy Ex- amples include icons, thumbnails, images, charts, and progress bars.
Menu
Structured lists of commands or options, often hierarchical. Examples: file menu, context menu, dropdown menus.
Others
Elements not covered by the above categories, often decorative or container elements like spacers.
Annotators underwent training process become familiar with the platforms and annotation guidelines. They were compensated hourly, with each task requiring on average 60-90 minutes to complete, including quality checks. The process began with the creation of computer-use tasks for 87 software applications (see Table 5). Annotators then executed these tasks while screen record- ings were collected. From these recordings, we extracted keyframes corresponding to major user interactions. Each keyframe was annotated using custom tool, where annotators drew bounding boxes around all visible interface elements. For each bounding box, annotators assigned a label corresponding to the element's name, in the case of textual elements, the text was also provided in addition to the element name. For long text segments such as source code or lengthy descrip- tions, annotators provided a concise summary that captured the main theme. To supplement these summaries, we also applied OCR using PaddleOCR (Cui et al., 2025) to extract the full text when available. In addition, every element rigorous quality assurance at multiple stages. Annotations were reviewed by dedicated quality spe- cialists, cross-checked by the authors, and validated using custom evaluation scripts. This pipeline allowed us to construct a large-scale dataset of grounded user interface interactions with high diver- sity and reliable annotation quality.
assigned one of six high-level categories. We applied
A.3 DATASET STATISTICS
We provide detailed statistics for GROUNDCUA. Figure 4 presents the overall dataset statistics. Fig- ure 4a shows the number of annotations across the 12 categories, while Figure 4b reports the number of screenshots per category. We also analyze the pixel distribution of screenshots in Figure 4c, ob- serving a wide range from roughly 0.3 megapixels to 7 megapixels. The distribution of bounding box areas, shown in Figure 4d, highlights the prevalence of small UI elements in the dataset. Finally, Figure 4e shows the number of bounding boxes per screenshot, with some screenshots containing up to 500 annotated elements and Figure 4f shows the distribution of desktop applications across 12 different categories.
A.4 COMPARISION WITH PRIOR WORKS
Comparative Analysis with Existing Datasets We compare GROUNDCUA against four recent grounding datasets: UGround (Gou et al., 2024), Aguvis (Xu et al., 2024), OS-Atlas (Wu et al., 2024), and JEDI (Xie et al., 2025). For OS-Atlas and JEDI, which are much larger, we sample 200k images for screenshot-level analysis, with bounding- box statistics computed over all annotations. As shown in Figure (left). GROUNDCUA's screenshots range from 0.5M-7M pixels, averaging 2.0M, capturing high-resolution desktop environments. UGround and OS-Atlas (Desktop) have lower res- olutions (I.IM and 1.6M), limiting their detail. Figure 5 (right) highlights GROUNDCUA 's smaller median element size, with many fine- mneiner targets like icons and small controls, typical of desktop
17

---

Published as a conference paper at ICLR 2026
1.0
14000
Number of Annotations
Number Screenshots
12000
0.8
0.6
0.4
0000
4000
0.2
2000
Graphics and Design
Productivity
Video Audio Production
0.0
Scentiti
Deyelopnent
Communication
Finance and Arattricc
Education
Browsers
Entertainment
Sydtem Utiities
Securit
Productivit
Graphics and Desiggn
Developmenr
Video Austo Productton
Communicato
Fitance and Enterturmnn Uniities
Sdentiic
Educatio
Anaby
Sydt orem Browoer
Software Categories
Software Categories
(a) Number of annotations in different software categories
(b) Number of screenshots in different software categories
Density Curve
Density Curve
2.0
Median: 1.94
Median: 0.032%
Mean: 2.03
Mean: 0.135%
95th percentlle:
percentile:
99th percentle: 6.28
95th percentile: 0.372%
15
15
Density
Density
0.0
8.00
0.05
0.10
0.15
0.20
0.25
0.30
0.35
Image Size (Megapixels)
Relative Bbox Area (% of Image)
Pixel distribution of images
(d) Relative bounding box
Density Curve
0.012
Mean:
95th percentile: 122
percentile: 155
Density
0.008
Padlab ZUR
0.006
Doagy PG
Peclattel 31%
0.004
0.002
MIA and Dui
0.000
Fyshi
Number Annotations per Screenshot
(e) Distribution of number of annotations in an image
(f) Desktop application across different categories
Figure 4: Dataset Statistics
18

---

Published as a conference paper at ICLR 2026
Medien
10%
Hest CONOD Whileans
Resolution (Megapixels
Meen
Whiskers
Relative Araa (lop scale)
[K02,06M
(K.1.GOM
Atlas
Aguiis
UGround
(Desktap)
(Al) Datasets
(Dellted
241
LScoure
Datasets
Figure 5: Comparison across different datasets. (Left) Pixel distribution for different datasets. (Right) Relative bounding box area in log scale.
interfaces. In contrast, other datasets focus on larger, more salient elements. GROUNDCUA also has denser annotations, averaging 64 per screenshot, more than three times that OS-Atlas (Desktop) and much higher than Aguvis (9) or UGround (1 1). JEDI, despite its scale, has sparser annotations due to its reliance on synthetic data. UGround and Aguvis cover web interfaces, while OS-Atlas uses automated accessibility- 2024; Muryn et al., 2025), resulting in less precise annotations. JEDI impressive in scale but lacks dense, real-world coverage due the synthetic pipeline involved in creating the dataset. GROUND- CUA, with its nigh-resolution, numan-verified annotations, and extensive platform diversity, fills a crucial gap by providing
traversal, which is often incomplete and prone to errors (Gou al.,
accurate and detailed representation of desktop environments.
A.5 DATASET EXAMPLES
Figure shows examples of screenshots from several software platforms with bounding boxes over- laid the images.
B INSTRUCTION TUNING DATA
GROUNDCUA contains over 3.5M annotated elements. Desktop screens are highly redundant, with many UI elements repeating across views. To reduce duplication before building instructions, we deduplicate elements using text matching on labels and perceptual image similarity (pHash) com- puted on crops defined by each element's bounding box. This produces roughly 900k unique ele- ments. We strict thresholds during filtering. While this may remove some valid cases, it yields a diverse, non-redundant pool overall. We also randomize selection across screenshots so that no single interface is rerreeepreen..d. The filtered elements form the base for constructing the instruc- tion tuning data. We detail the different types of instructions we have created below and provide examples Figure 7.
B. DIRECT INSTRUCTIONS
Direct instructions explicitly refer to the element (Click the "File" button) that the model should These the most common types of instruction CUA would encounter. We first create a class of descriptive instructions for every element, which incorporates attributes such as color, shape, position, and nearby context. These descriptions provide richer context for the model and help reduce ambiguity. We generate these instructions by prompting Qwen2.5-VL-7225 with the element's s bounding box, platform name, annotated label, the full screenshot, and an optional zoomed crop. We also ask the model to provide the location of the element if there are other similar elements disambiguate. We additionally use category information create three specific types of direct instructions:
19

---

Published as a conference paper at ICLR 2026
(a) 7-Zip
(b) GIMP
(c) Brave
(d) Audrino
(c) FreeCAD
(1) GNU Octave
(g) Cryptomator
(h) OpenShot
(i) Frappe Books
E
(i) Emby
(k) Mastadon
(1) LibreOffice Cale
(m) R Studio
(n) VLC Media Player
(a) Zotero
Figure 6: Examples of screenshots from different platforms in GROUNDCUA. Red bounding boxes indicate the annotated UI elements within each screenshot.
Description Instruction Prompt
You are an expert UI analyst. You are given a screenshot with a target element in a red bounding box, cropped image containing the target element in red bounding box, the name of the element and the platform name.
Can you find it? Is it visible from the sereenshot? Can you write a concise description that is sufficient for humans to locate it from the screenshot? The response must be relevant to the platform and element name provided. Do not reference the red bounding box and that it is highlighted.
you find other identical elements, your description must include specific details about the
20

---

Published as a conference paper at ICLR 2026
target element's location and other unique attributes to differentiate it from the others. Only output what you are sure about. Do not make assumptions. Return the response in the following JSON format:
"visible": true, "description": "your description here"
Platform:
(platform name}
Target Element Label: (text)
Textual elements. We identify textual elements by matching OCR output with the human-annotated label and by selecting items from the Information Display category. We then embed the extracted text into about 100 templates that directly instruct the model to move to these labels. Some templates used to generate instructions provided below.
Textual Elements Instruction Templates
1. Do you see the 'text'? Please click it.
2. Please locate the user interface component marked with the text "text' and then proceed to
click on it.
3. Make your way to the 'text' label with your cursor.
You are required to find the element associated with the text 'text" and then move your cursor
to hover over it.
Visual elements. For icon-based or other visual elements (e.g., tool icons, shapes, images), generate concise captions that highlight distinctive features and local context (c.g.. "Click the magnifying-glae icon next to the search bar"). These are produced using Qwen2.5-VL-72B by providing the element crop, its bounding box in the full screenshot, the platform name, and the annotated label.
General templates. In addition to text and visual elements, design a set of general instructions that apply to any element. These are created heuristically using about 120 templates (e.g.. "Click on the following element:") generated by prompting MLLM.
General Instruction Prompt
You expert UI analyst. You are given screenshot with target element in red bounding box, a cropped image containing the target element in a red bounding box, the name of the element and the platform name.
Is it visible from the screenshot? Generate concise, imperative instruction a user would give to operate or interact with the target element.
The response must be relevant to the platform and element name provided. Do not reference the red bounding box and that it highlighted.
If you find other identical elements, your description must include specific details about the target element's location and other unique attributes to differentiate from the others. Only output what you are sure about. Do not make assumptions. Return the response in the following JSON format:
"visible": true,
"instruction": "your description here"
Platform:
{platform}
Target Element Label: (text)
21

---

Published as a conference paper at ICLR 2026
B.2 FUNCTIONAL INSTRUCTION PROMPT
Functional instructions describe an element by its purpose rather than its name (e.g., "Open new tab"). We focus on Buttons and Menus since these most often encode actions. For each candidate element, we prompt Owen2.5-VL-722 with the full screenshot, the element crop and bounding box, platform name, and the annotated label, asking for a concise functional instruction (c.g., "Open new tab").
Functional Instruction Prompt
You are an expert UI analyst. You are given screenshot with a target element in red bound- ing box, cropped image containing the target element red bounding box, the name of the element and the platform name. Is it visible from the screenshot? Generate task-oriented in- struction that describes user's goal. The instruction must implicitly identify the target element by describing what it helps the user accomplish (not the name of the element).
The response must relevant to the platform and element name provided. should also be concise and to the point. Do not reference the red bounding box and that is highlighted.
Include the location or other unique attributes if there are other identical elements.
Only output what you sure about. Do not make assumptions. Return the response in the following JSON format:
"visible": true,
"function": "your description here" }
Platform:
(platform}
Target Element Label: (text}
B.3 SPATIAL INSTRUCTIONS
Spatial instructions locate a target element by its position relative to another element (anchor), using relations such as left, right, above, below, and between. We leverage dense annotations to choose anchors that are close to the target and have reliable labels (e.g.. "Click the icon to the left of "Files""). We generate these with simple templates that insert the anchor's 's label and the relation. Some templates used produce instructions provided below.
Spatial Instructions Templates
1. Place your mouse the element directly to the right of of "(clement)""
Hover your mouse on the element immediately to the left of "(element)".
3. Hover your mouse the element between "felement_ and "felement.2)"
4. Place your mouse on the element directly above "(element)".
B.4 EXAMPLES
Figure shows different kinds of instructions generated by our data generation pipeline.
B.5 NEED AND IMPACT OF DIFFERENT INSTRUCTION TYPES
As mentioned above have three main instruction types: Direct, Functional, and Spatial. The "Direct" category is itself broad, encompassing instructions based on descriptions, text/visuals, as well general/heuristic templates. To analyze their impact of different instruction types and their subtypes, we sampled 100k datapoints for each distinct instruction subtype and trained Qwen2.5- VL-IIstrruct3BB model. The results are presented in Table 7.
As shown in Table 7, Functional instructions yield the highest average performance. We hypothesize this is because these goal-oriented instructions (e.g., "Open tab") closely match the tasks in
22

---

Published as a conference paper at ICLR 2026
Table 7: Ablation study on different instruction types. and train Qwen2.5-VL-33 model.
We sample 100k data points for each type
Data Type
SS-Pro
OSW-G
UI-V
MMB-GUI
SSv2
Avg
Functional
43.0
51.0
27.6
73.4
88.1
56.6
Direct Description
36.5
58.5
24.5
70.1
85.0
54.9
Direct General templates
37.9
52.8
26.4
73.8
86.1
55.4
Direct- Text and visual
35.3
51.0
20.3
64.2
85.1
51.2
Direct- Miscellaneous
32.5
56.3
24.7
67.1
85.4
53.2
Spatial
20.5
52.5
22.4
67.8
74.1
47.5
the evaluation benchmarks. However, the other instruction types are crucial for building a robust, well-rounded agent for two main reasons:
1. Complementary Strengths: While "Functional" best average, other types excel specific tasks. For example, in the OSWorld-G text-recognition category. the "Direct Text and visual" split achieves 0.64 score, outperforming the "Functional" split's 0.60.
2. Preventing Overfitting:
We observed that models trained on only one instruction type become brittle. For example, model trained only on "Description" instructions sees its performance drop by 4% on ScreenSpot-Pry when the prompt "Click on the element with the following description:" is not prepended to the benchmark's 's instructions.
Our final 700K SFT dataset is chosen mix to ensure the model is both high-pent gigh-performing and less sensitive to prompts used. Hence, when evaluating our final models, we do not provide any prefix prompts and evaluate directly on the instructions provided by the benchmarks.
B.6 QUALITY OF INSTRUCTION TUNING DATA
Our MILM pipeline for generating the instructions is highly robust for key reason: not de- pendent on the MLLM's open-ended knowledge. Our pipeline is highly constrained. It provides the MILM with strong, ground-truth context, including the element's name, its bounding box, and addi- tional screenshot context (e.g., elements around the target element for spatial instructions). The task one of grounded rephrasing description, not open-ended task creation. This also clearly reflects the outcome of the trained model using generated instructions across a wide variety of benchmarks and the reported result in the paper.
To further verify this systematically, we performed a human evaluation on the generated instruction- tuning set. Three annotators who are not the authors of the paper annotated 100 randomly selected instructions to check for validity (i.e., whether the instruction accurately describes the element being grounded). Using a majority vote to aggregate these annotations, we found an error rate of 4%. We believe this represents low error rate for a training dataset, highlighting the quality and reliability of our -ennrredee instruction data.
C TRAINING
In this section, we describe the training process for GROUNDNEXT. We outline the key design choices behind our SFT and RL setups, including data selection and filtering strategies, hyperpa- rameter configurations, and other relevant details. We also report experimental observations, high- lighting the impact of these choices and the insights gained during development.
C.I
SFT DATA
From the instruction-tune tutnnnntttnnngnggo corpus, we curate a split of 700K size with 50% direct instructions, 35% functional instructions, and 15% spatial instructions.
23

---

Published as a conference paper at ICLR 2026
Human Label: Constrain lock
Category: Button
Platform: FreeCAD
Human Labeb Anki
Category: Visumi Elements
Platform: Anki
by paclock symbol,
Instruction (lcon/lago]:
located constroint
hortaantal conseraint
Functional Instruction:
Spatial Instructio: Click above Edit option
current geomatry to prevent accidiental ficationn
WELCOME
Homan Label: Undo
Category: Button
Platformm OnlyOifice Document Editor
Direct Instruction (bedjj:
Briig your mouss pointer the component that says tdeck"
Functional Instruction:
Direct Instruction (desorlptton):
Revert the last action or change made the document.
The Dock' secson is under the Necifocaions oottings. is positioned below the Calencar. contacts and tasks' section and containn options related to changes bcards, listrr cards.
Direct Instruction (Ganerad):
you could, plason selact 'Undo'
Category:
Human Labet: Borders
Category: Menu
Platform: LibrOOffice Cate:
Apply enharce Instruction:
your spreadeheet.
Spatial together managing roles permissions.
Borders button located tootbar Calo, specifccaly
formatting section. rspresented
gnd
Click betwoen Forma
other formatinng optiona aligiment styles.
Figure 7: Instruction tuning data examples.
C.2 SFT TRAINING DETAILS
We use LlamaFactory (Zheng et al., 2024) to train our SFT models with learning rate 3e-6, cosine decay, and warmup ratio of 0.05. Models are trained for two epochs, as this consistently outperforms training for a single epoch. Preliminary experiments also show that training the entire model, rather than only the LLM. is more effective; we therefore adopt this configuration throughout. All models are trained on a single H100 node with H100 GPUs, using a global batch size of 128, gradient accumulation of 16, and per-device batch size of 1.
24

---

Published as a conference paper at ICLR 2026
C.3 RL DATA
For RL training, we first performed rejection sampling on the SFT training set using the SFT model itself. Specifically, we extracted the model's errors and sampled 10K instances, which were then used to run RI. While this yielded modest improvements, the SFT model was already strong, and many of the extracted errors corresponded to noisy or ambiguous datapoints (e.g., prompts with mul- tiple valid answers or inconsistent labels). These issues limited the effectiveness of this approach.
We next applied RL on top of the SFT model using 10K previously unseen samples from GROUND- CUA. This strategy avoided noise from ambiguous training points and yielded more significant performance boost. Consequently, our final setup exclusively used the 10K samples unseen during SFT from GROUNDCUA.
We also explored incorporating a small amount of out-of-distribution data to encourage generaliza- tion to web and mobile domains. Specifically, we added 10K samples from GUIAct (Chen et al., 2024), in addition to 10K samples from GROUNDCUA, split evenly between mobile (5K) and web (5K). Unlike the gains observed when adding in-distribution samples from GROUNDCUA, this pre- liminary attempt did not yield consistent improvements. We note, however, that our setup was limited in scope and did not include rejection sampling or other analysis. A more systematic in- vestigation of combining our dataset with complementary sources, particularly in the context of RL training improve cross-platforn performance, an exciting direction for future work.
C.4 RL TRAINING DETAILS
For our RL training, we compared two rule-based optimization methods, Group Relative Policy Optimization (GRPO) and Relative Leave-One-Oue (RLOO). Empirically, and as pointed out in previous literature (Zhang et al., 2025), we found that RLOO produced more stable learning and better results. The RLOO objective can be written
T
)
(mo) = ETTWTT
Vo log xo (aclo))
R(T)
1
Σ R(Ts) J
(1)
where R(T) is the reward of trajectory т. and the baseline is computed as the average reward of all other trajectories in the same group (excluding the i-th trajectory). This avoids training a critic model and instead uses relative group comparisons. In our case, the trajectories are the predicted coordinates by the model, and the reward is defined based on where the predicted point is relative to the bounding box. For the grounding task, 7 is sequence of tokens, which represents the predicted coordinate.
Reward Formulation.
1. Continuous reward: Based on the normalized distance between the predicted point p and the ground-truth bounding box B, we defined:
10 D"1
r 1 dd, d MaxDist B, W, H)
where p" is the closest point B, and MaxDist B, W, H) is the maximum possible distance point inside image of width W and height H have. However, this suffered from sparsity and weak gradient signals.
2. Binary reward: A simple scheme assigning
1 p E B,
otherwise.
This proved more stable than continuous rewards but lacked sensitivity to error magnitude. Customized Discrete Reward (final choice): To distinguish between predictions that miss the bounding box by small or large margin, and encourage predictions inside the box to closer
25

---

Published as a conference paper at ICLR 2026
2202
1800
180
Drequeeeyy
632
300
Ner melloed Stetence to Geound Thith Bounding Dos
Norma tred D Extance Sround T-th Bounding Ros
Figure 8: Dorm distribution of the errors made by GROUNDNEXT-3B (SFT) and GROUNDNEXT- 7B (SFT) on the training set. 50% of the errors lie within Dnorm 0.1, highlighting the motivation of our reward function.
to the center, we designed a customized discrete reward based on a normalized signed distance. The reward function defined as:
-1. D norm ~0.5,
0.5 if 0.5 < D Dnorm < 0.1,
0.1 if - 0.1 < D Dnorm < 0,
Recore B,
0.1 if S Dnoom x011,
0.5 if D Dorm < 0.5,
1.0 D'norm 2 0.5.
The normalized distance calculated Doorm D( between the predicted coordinate P and the ground-truth bounding box (with positive values denoting the interior). The reference distance Dref adapts based on the prediction's location to ensure D Dnorm E 1-1
Drof where D(p, B) is the signed distance
0.5 x diam B) Dref
if B, D Dmaa (B,1) otherwise.
Here, we use half the bounding box diameter when P E B, as this represents the maximum possible distance a point inside B can have from the boundary. Conversely, DD Pmaz(B, represents the maximum distance in the image context for exterior points.
In summary, we adopt RLOO with this shaped reward formulation, as it effectively balances penal- ties for misses with incentives for precise centering. Our level-wise reward is motivated by the large proportion of predicted points that miss the bounding box by only small margin. We highlight this characteristic in Figure 8, where we compute D Dnorm for 10K errors made by GROUNDNEXT-3B (SFT) and GROUNDNEXT-78 (SFT) the training set they were trained on. This figure shows the imbalance in the error distance of the predicted points and the prevalence of "near misses". which directly motivates choice of reward function.
To better demonstrate how the rewards behave for different predicted points, Figure shows a screenshot from FreeCAD where the ground-truth bounding box encloses the "sketch in progress". In this toy example, we illustrate six predicted points, each with a corresponding Dnorm value that falls into one of the ranges defined by reward function. As result, each point receives a different reward. Higher rewards correspond to predicted points that are closer to the center of the bounding box.
26

---

Published as a conference paper at ICLR 2026
bes
Mere
Help
Pare +0.00
Pure
m-2.05
Puant
P1:
0.76
Pare +0
Sounding Six Caster
Pd: +0.1
Poim -0.5
Ph: +0.5
Purr 0.24
PM: +1.0
Figure 9: Rewards of 6 predicted points with respect to the ground truth bounding box in screenshot of FreeCAD.
C.5 RL DESIGN CHOICES AND HYPERPARAMETERS
C.5.1 DISCRETE REWARD DESIGN
We investigate the impact of reward granularity on model performance during the RLOO stage. As shown in Table 8, compared reward configurations ranging from binary feedback 8-level quantization using a preliminary 3B checkpoint. We observe that the 6-level design achieves the best performance granularity settings. Based on these results, we selected the 6-level configuration (formulated as {-1.0, 0.5, 0.1, 0.1, 0.5, 1.03) for all subsequent experiments.
OSWorld-G (63.1%) and yields the highest average score (65.2%) across all
Table 8: Performance of different reward designs for the RLOO stage using a preliminary 3B check- point. The "Levels" column indicates the number of discrete reward values used within the range [11.1].
Reward Granularity
MMBench GUI
OSWorld-G
ScreenSpot-PPe Pro
Avg
8 Levels
80.6
61.5
52.3
64.8
Levels
81.0
63.1
51.6
65.2
4 Levels
80.4
62.6
52.6
65.2
2 Levels
81.0
62.1
52.4
65.1
C.5.2 IMPACT OF GROUP NUMBER
We also ablated the group size n (the number of generations per prompt) to balance performance with training efficiency. This study was conducted using preliminary GROUNDNEXT-335 3-SSFT checkpoint trained via GRPO on a subset of 5.1K RL data points. As shown in Table 9, increasing the group size to not justify the significant increase in computational cost and training time. While n 4 slightly outperforms 8, consider such small group size potentially unstable for gradient estimation.
32 yields the best overall performance; however, the marginal gains do
27

---

Published as a conference paper at ICLR 2026
Consequently, choose for our final experiments, choice consistent with settings used in recent prior works (Liu et al., 2025a; Tang et al., 2025).
Table 9: Ablation the group number n using a preliminary 3B checkpoint with GRPO (5K sam- ples). All scores reported as percentages.
ScreenSpot-PP Pro
ScreenSpot-v2
OSWorld-G
UI-Vision
Avg
4
48.1
88.1
59.4
58.11
63.4
8
48.6
87.7
59.0
57.8
63.3
32
49.5
88.2
58.3
58.2
63.5
C.5.3 RLOO Vs. GRPO
Further, conducted an ablation on RLOO vs. GRPO. While GRPO is widely used, we selected RLOO for its simplicity and successful application in related work such as InfiUU--G1 (Liu al., 2025a). To validate this choice, conducted minimal comparison using early SFT checkpoint (Table 10). We observed that RLOO achieves performance parity with GRPO, yielding a slightly higher average score (65.7% 65.3%) and stronger results OSWorld-G. We emphasize that do not claim RLOO is inherently superior to GRPO; rather, these results indicate that RLOO was a reliable and robust configuration for our dataset-centric experiments.
Table 10: Comparison of RLOO and GRPO algorithms early SFT checkpoint. All scores reported as percentages.
Algorithm
ScreenSpot-Pry
ScreenSpot-v2
OSWorld-G
Avg
GRPO
49,5
88.2
58.3
65.3
RLOO
49.3
88.3
59.4
65.7
D EVALUATION
D.1 SCReeNSPOTPRO RESULTS
Table 11 summarises the results for different models on ScreenSpot-Pro (Li al., 2025).
D.2 OSWORLD-G RESULTS
Table 12 summarises the results for different models on OSWorld-G (Xic et al., 2025).
D.3 MMBENCH-GUL RESULTS
Table 13 summarises the results for different models on MMBench-GUT (Wang et al., 2025b).
D.4 ScREENSPOT-v2 RESULTS
Table 14 summarises the results for different models on ScreenSpot-v2 (Cheng et al., 2024).
D.5 UI- VISION RESULTS
Table 15 summarises the results for different models UI-Vision (Nayak et al., 2025).
28

---

Published as a conference paper at ICLR 2026
Table 11: Performance of different models on SSPro across categories (CAD, Dev, Creative, Scien- tific, Office, OS). Text and Icon refer to different input types.
Model
CAD
Dev
Creative
Selentißi
Office
Text
Icon
Text
Icon
Text
Icon
Text
Icon
Text
Icon
Text
Icon
Text
Icon
GPT-4o
2.0
0.0
1.3
0.0
1.0
0.0
2.1
0.0
1.1
0.0
0.0
1.3
0.0
0.8
Clande Computer Use
14.5
3.7
22.0
3.9
25.9
3.4
33.9
15.8
30.1
16.3
11.0
4.5
23.4
7.1
17.1
Qwen2.5 VLBBBB
9.1
7.3
22.1
1.4
26.8
38.2
7.3
33.9
15.1
10.3
1.1
23.6
16.1
Qwen2.5-V1.77
FOCUS-28
7.6
22.8
1.7
23.7
7.1
23.2
2.5
19.8
13.3
ShowUI-28
2.5
0.0
16.9
1.4
9.1
0.0
13.2
7.3
15.3
7.5
10.3
2.2
10.8
2.6
7.7
UI-TARS-28
15.8
1.2
51.9
2.8
47.5
9.7
57.6
14.3
60.5
13.2
38.3
7.9
45.2
8.1
31.1
JEDI-3B
27.4
9.4
61.0
13.8
53.5
54.2
18.2
64.4
32.1
38.3
9.0
49.8
13.7
36.1
SccClick-9.6B
2.5
1.0
0.0
2.8
0.0
Aria-U1
7.6
1.6
16.2
0.0
23.7
2.1
27.1
6.4
20.3
1,9
4.7
0.0
17.1
11.3
OS-Atlas
12.2
4.7
33.1
1.4
28.8
2.8
37.5
7.3
33.9
5.7
27.1
4.5
28.1
4.0
18.9
UGround-7B
14.2
1.6
26.6
2.1
27.3
2.8
31.9
2.7
31.6
11.3
17.8
0.0
25.0
16.5
U1-TARS-7B
5.6
JEDI-78
38.0
14.1
42,9
50.0
11.9
25.5
75.1
47.2
33.6
16.9
52.6
18.2
39.5
GUI-Acttr-7B
44.6
OpenCUA-77B
50.0
1gnnt----888
3.1
9.6
13.0
0.0
UI-TARS-72B
18.8
12.5
57.1
15.4
20.9
63.3
26.4
42.1
15.7
50.9
38.1
UI-RI-3B
11.2
6.3
22.7
4.1
27.3
3.5
42.4
11.8
32.2
11.3
13.1
4.5
24.9
6.4
17.8
UI-RI-E-3B
37.1
12.5
46.1
6.9
41.9
4.2
56.9
21.8
65.0
26.4
32.7
10.1
33.5
GUI-RI-3B
26.4
7.8
4.8
40.9
61.8
17.3
53.6
17.0
28.1
5.6
InfiGU-RI-3B
GUI-G1-3B
39.6
9.4
50.7
10.3
36.6
11.9
61.8
30.0
67.2
32.1
23.5
10.6
49.5
16.8
37.1
SE GUI-3D
38.1
12.5
55.8
7.6
47.0
4.9
61.8
16.4
59.9
24.5
40.2
12.4
50.4
11.8
35.9
InAGUI-GI-3B
50.8
25.0
64.9
20.0
51.5
16.8
68.8
32.7
70.6
32.1
49.5
15.7
45.2
GUI-RI-7B
23.9
38.9
58.7
42.1
16.9
SE. GUI-BB
51.3
42.2
68.2
19.3
57.6
75.0
28.2
78.5
43.4
49.5
25.8
63.5
21.0
47.3
Phi-Grrund-7B-10c..DD
70.8
16.7
56.6
13.3
26.9
17.2
58.0
29.1
76.4
44.0
55.1
25.8
56.4
21.8
43.2
GUT-GF-7B
55.8
12.5
68.8
17.2
57.1
15.4
77.1
24.5
74.0
32.7
57.9
21.3
64.7
19.6
47.5
GTA1-7B
66.9
20.7
18.2
53.3
17.2
76.4
82.5
50.9
48.6
25.9
65.5
25.2
50.1
29.2
Models
GROUNDNEXT-3B (SFT)
50.3
26.6
65.6
36.6
48.5
22.4
66.0
38.2
76.3
54.7
41.1
28.1
58.3
32.8
48.6
GROUNDNEXT-3H (RL)
55.3
32.8
65.6
36.6
50.0
24.5
66.0
37.3
74,6
50.9
45.8
29.2
59.9
33.6
49.8
GROUNDNEXT-7B (SFT)
46.2
32.8
38.6
54.5
20.3
37.3
76.8
49.1
45.8
33.7
59.9
33.6
50.2
GROUNDNEXT-2B
Table 12: Performance comparison of models on OSWORLD-G across multiple capability dimen- sions.
Matching
Recognition
Layont Understanding
Fine-grained Manipulation
0S 18--Alls-BB
44.1
29.4
35.2
16.8
744
27.7
UGround-V1-78 VV-BB
51.3
40.3
43.5
24.8
0.0
36.4
Agavis-7B
UI UI-TARS-7B
60.2
51.8
51.9
35.6
0.0
47.5
Se
18.5
U1-TARS-T2B
69.4
60.6
62.9
45.6
0.0
57.1
Gemini 2.5.P Pro
59.8
45.5
49.0
33.6
38,9
45.2
Operator
$1.3
42.4
46.6
31.5
0.0
40.6
Qwen2 5-VL-38
41.4
28.8
34.8
13.4
0.0
27.3
Qwen2.5-VL-7B
Q-e.2.5-V1-328
63.2
47.3
49.0
36.9
0.0
46.5
JEDI-38
67.4
53.0
53.8
44.3
7.4
50.9
JEDI-7B
65.9
55.5
57.7
46.9
7.4
54.1
IaAGUL-GI-38
65.5
53.0
56.1
34.2
0.0
49.6
InRGCL-GG1-7
GTA-L-7B
63.2
82.1
74.2
70.5
0.0
67.7
Our Models
CB (SFT)
67.4
68.8
68.4
43.0
0.0
62.2
(RI.)
70.9
71.2
70.8
43.6
0.0
64.2
GROUNDNENT-78 (SFT)
73.1
GROUNDNEXT-7H (RI.)
74,3
23.9
23.5
51.7
0.0
67.7
29

---

Published as a conference paper at ICLR 2026
Table 13: MMBench-GUI: Cross- Cross-platform performance of models across Windows, MacOS, Linux, iOS, Android, and Web.
Model
Windows
MacOS
Linux
iOS
Android
Web
Avg
GPT-4o
1.5
1.1
4.3
1.1
1.0
5.1
3.3
2.5
1.4
3.2
2.9
2.9
Claude-3.7
1.5
0.7
7.5
1.1
10.6
1.4
2.3
4.7
Qwen Owen-Max-VI.
56.1
79.5
ShowUI-2B
17.4
12.7
LVL-7B
31.4
16.5
22.0
21.5
10.2
66.6
55.2
35.1
35.2
40.3
32.5
33.9
Qwen2.5 72.
55.7
33.8
49.9
30.1
40.3
20.9
56.1
28.2
55.6
25.4
68.4
45.8
41.8
DS-AAtla-Basee-2
36,9
18.8
44.4
21.7
31.4
13.3
74.8
48.8
69.6
46.8
61.3
35.4
41.4
Aguvis-7B-720P
37.3
21.7
48.
33.3
33.5
25.0
67.5
65.2
61.0
51.0
61.6
45.5
45.7
UI-TARS 1.5-7B
68.3
39.0
69.0
44.5
64.4
37.8
88.5
69.4
90.5
69.3
81.0
56.5
64.3
U1-1ARS-72B-DPO
78.6
51.8
80.3
62.7
68.6
51.5
90.8
81.2
93.0
80.0
88.1
68.5
74.3
UGrrund -1-77
66.8
39.0
71.3
48.6
56.5
31.1
92.7
70.9
93.5
71.0
88.7
64.6
65.7
VL-77B
70.1
42.6
75.7
52.3
59.2
41.3
93.6
80.6
92.7
78.6
90.7
65.9
72.2
Naive RLVR-3B
68.6
44.5
50.0
61.3
92.4
76.4
91.3
63.0
IniGUU--G133
64.3
In6GUI-GI-7B
82.7
61.8
83.8
63.9
72.3
52.0
89.4
95.2
85.6
93.5
76.3
80.8
Models
GROUNDNENT-38 (SFT)
73.8
GROUNDNENT-3B (RL)
80.4
52.6
87.2
64.5
70.7
78.5
91.9
78.0
90.6
64.3
77.1
GROUNDNENT-7B (SFT)
83.8
60.7
86.7
69.9
75.4
61.2
94.3
83.3
94.9
79.4
91.0
70.5
80.4
GROUNDNENI-7B (RL)
81,5
60.7
87.8
73.1
75.4
59.2
95.2
86.1
95.5
80.3
90.97
72.7
81.1
ScreenSpot-V2 V2: Cross-platform breakdown by device and modality. "Icon/Widget" indi- cates icon- or widget-based queries. "Avg." is across all devices and modalities.
Table 14:
Model
Mobile
Desktop
Web
Avg.
Text
Icon/Widget
Text
Icon/ (con/Widget
Text
Icon/Widget
SceClick
78.4
50.7
70.1
29.3
55.2
32.5
55.1
OS-Atlas-Basee
95.2
75.8
90.7
63.6
90.6
77.3
85.1
UI-TARS-7B
96.9
89.1
95.4
85.0
93.6
85.2
91.6
UI-TARS-72B
94.8
86.3
91.2
87.9
91.5
87.7
90.3
Qwen2.5-VL.
93.4
73.5
88.
58.6
88.0
71.4
80.9
Qwcn2.5-VL-7B
97.6
87.2
90.2
74.2
93.2
81.3
88.8
Qwen2.5-V1L-32R
97.9
88.2
98.5
79.3
91.2
86.2
91.3
InfGUU--G1-BB
99.3
88.2
94.8
82.9
94.9
80.3
91.1
InfGUU-GG-7B
99.0
91.9
94.3
97.9
89.2
93.5
Our Models
GROUNDNEXT-3B (SFT)
95.2
80.6
93.8
84.3
87.6
78.8
87.3
GROUNDNEXT-3B (RL)
96.4
93.9
87.1
79.3
88.5
GROUNDNEXT-7B (SFT)
97.2
84.8
94.3
90.0
91.5
74.9
89.3
GROUNDNEXT-7B (RL)
96.6
88.2
95.4
87.9
94.9
75.9
90.4
D.6 DISCUSSION ON RI. GAINS
We hypothesize that the observed "limited" improvement is not failure of the RL. step, but rather a finding that highlights the interaction between strong SFT baselines and RL gains. We also demon- strate that GROUNDCUA is an effective dataset for RL fine-tuning. We detail our analysis below.
D.6.1 HIGH-QUALITY SET CREATES A "STRONG CEILING"
We hypothesize that stronger GUI models contain fewer actionable errors after the SFT stage, sulting in lower marginal benefits from subsequent RL fine-tuning, especially when the RL data is drawn from the same distribution the SFT data. This is supported by Figure 3, where observe
30

---

Published as a conference paper at ICLR 2026
Table 15: UI-Vision: Performance grouped by category (Edu., Browser, Dev., Prod., Creative, En- tert.) and by setting (Basic, Functional, Spatial).
Model
Grouped by Setting
Overall
Basic
Func.
Spatial
GPT-4o
1.6
1.5
1.0
1.4
Claude-3.7-onneet
9.5
7.7
7.6
8.3
Qwen-2.5VL-7B
1.2
0.8
0.5
0.9
IntemVL2.5588
2.5
2.8
1.0
2.1
MiniCPM- V-BB
7.1
5.3
4.3
SceClick-9.6B
9.4
4.7
2.1
5.4
ShowU1-2B
8.1
7.7
2.1
5.9
CogAgent-9B
12.0
12.2
8.9
OSAtlas-7B
12.2
11.2
3.7
9.0
12.2
14.0
4.0
10.1
UGround -l-BB
15.4
17.1
6.3
12.9
UGround-v1-72h
27.9
26.7
14.9
23.2
Ag
17.8
18.3
5.1
13.7
UI-TARS-77 7B
20.1
24.3
8.4
17.6
UI-TARS-72B
31.4
30.5
25.5
InfiGUI-G1-3B
31.2
28.0
8.2
22.0
InfGUU-GG--B
36.2
31.9
11.5
26.1
Our Models
NEX-ND (SFT)
70.9
59.8
45.1
58.2
GROUNDNEXT-3B (RL)
72.9
63.9
62.1
GROUNDNEXT-7B (SFT)
67.1
60.0
49.9
58.7
GROUNDNEXI-77 (RL)
70.1
62.0
49.9
60.3
that RL provides significantly larger gains for SFT models trained with other datasets compared to GROUNDCUA. We attribute this to the fact that GROUNDCUA yields much stronger initial model; consequently, the RL stage serves as minor refinement rather than a primary performance driver in current setting.
Table 16: Performance comparison between the base model (UI-Tars-1.. 1.5-7B) and the RL-tuned model (GTA-1-7B). Note that the results for UI-TARS-1.5-7B are reported using our own evaluation setup and differs from (Yang al., 2025).
Benchmark
Ul-Tars-1.5-7B
GTA-1-7B
Improvement
SS-Pro
47.9
50.1
+2.2%
OSW-G
64.2
67.7
+3.5%
MMB-GUI
75.4
79.4
+4.0%
SSv2
90.3
92.4
+2.1%
UI-V
20.8
25.7
+4.9%
We also see this trend related work, GTAI (Yang al., 2025). GTA1-7B initializes its training from UI-TARS-L.5-BB, which is powerful GUI grounding model. We observe that the average improvement across five benchmarks 3.3% (see Table 16). This demonstrates that, while RL provides consistent lift, the gains moderate, not universally massive. The limited marginal return observed GROUNDNEXT is consistent with the observations for GTA-1-7B.
We emphasize that do not make general claim that stronger GUI models cannot be effectively RL-tuned. Our results merely provide evidence supporting the hypothesis that the marginal return is lower when robust, high-quality SFT initialization is used, particularly
Important Note:
31

---

Published as a conference paper at ICLR 2026
for models in the 3-7B size range under the reward formulations employ. A more detailed study spanning different architectures, various reward functions, different model sizes, and deeper RL fine-tuning is required to fully understand these interactions. This comprchensive investigation is beyond the scope of our current paper, but our results provide the initial evidence for an interesting phenomenon.
D.6.2 GROUNDCUA IS GREAT SOURCE FOR RL FINE-TUNING
Table 17: RL ablation study: Performance of the Qwen2.5-VL-3B baseline trained with 10k RL samples from different datasets.
Dataset
SSPro
OSWorld-G
SSv2
MMBench
UI-V
Baseline
29.0
37.4
81.8
60.8
6.3
Aguvis
31.2
45.6
86.01
67.01
14.7
OSAtlas
30.4
46.4
62.0
67.7
14.1
UGround
33.6
43.8
89.0
68.8
16.7
GroundCUA
36.8
48.8
88.9
70.5
19.2
Imp. over baseline
+7.8%
+11.4%
+7.1% %
+9.7%
+12.9%
We clarify that the limited marginal gain observed in the final GROUNDNEXT models is not flaw of the GROUNDCUA dataset itself. We validate this through a controlled experiment where we sampled 10k data points from GROUNDCUA and three competing datasets with available bounding boxes: Aguvis, OS-Atlas, and UGround. We trained a hyperparameters described Section 4.1 and Appendix C.4, with two modifications: we adopted the simpler binary (0/1) reward formulation described in the GTA paper and extended training to epochs, as we observed rewards continuing to increase after the first epoch. We report the perfor- mance across various benchmarks in Table Table 17
baseline using the
We observe that GROUNDCUA provides substantial gains of 9.8% on average over the Qwen2.5 VL-3B baseline. Furthermore, it achieves an average gain of 1.9% over the next best baseline (and 2.5% we exclude SSv2 benchmark). This validates that GROUNDCUA data is highly effective for RL. We attribute this to our human-annotated labels and bounding boxes (less noise) and the rich diversity of platforms covered by our dataset.
The strength of GROUNDCUA lies in its platform diversity (87 applications across 12 categories) and dense annotations, which offer a huge variety of UI elements for agents to learn. RL training on very large scale (e.g., 700k samples in case) is computationally expensive, especially in resource-constrained settings (e.g., we used 8 H100 GPUs for our experiments). Hence, we believe the diversity of GROUNDCUA can be effectively exploited through a careful combination of SFT, which teaches new knowledge, and RL, which helps with generalization. Future research could focus on optimizing this SFT/RL mix, which has shown promise in works like Ma al. (2025). By releasing our data, we provide the necessary resource to explore this path.
D.7 SFT SCALING BEHAVIOR
To study how performance scales with additional supervised data, we trained three versions of Dween..5-VL-Instmuce
model using subsets of 100k, 350k, and 700k instructions sampled from GROUNDCUA. All runs used identical hyperparameters. The results are reported in Table 18.
We observe steady gains across almost all benchmarks the amount of training data increases. The improvements indicate that the training setup remains stable across scales and that the additional samples provide useful signal. Since GROUNDCUA undergoes rigorous deduplication, larger sub- sets introduce new visual and semantic variety rather than repeated patterns, which directly strength- grounding performance.
The most pronounced improvements appear on UI- Vision, where performance increases from 29.8 100k 58.2 at 700k. UI-Vision is closely aligned with the layouts and element styles present
32

---

Published as a conference paper at ICLR 2026
Table 18: Scaling ablation on GROUNDCUA. Performance improves steadily more SFT data is used.
Size
SS-Pro
OS-World-G
MMBench-GUI
UI-Vision
SS-v2
Average
100k
41.7
54.6
72.5
29.8
87.4
57.2
350k
44.5
57.3
72.9
39.5
87.0
60.2
700k
48.6
62.2
75.5
58.2
87.3
66.4
in GROUNDCUA, allowing the model to leverage broader coverage as more data is included. OSWorld-G G and ScreenSpot-PPe show similar positive trends, reflecting consistent benefits in dense desktop scenarios.
Overall, the scaling results show that high-quality, non-redundant supervision continues to improve grounding accuracy throughout this data range, suggesting room for further gains with larger curated subsets.
E
GROUNDNEXT ERROR ANALYSIS
Prompt: join twitch server
Prompt: view details
Create Your Server
serve wheryyye your hang
Make yoars and talking
Iepaarn pessalatiio sad wisy.
Discover Home
tamplitee
Gaming
Home Upgrade Available
Horee nen untelyne performentt your your horm.
Learn Moze
Study
Artists Creators
Ascocsair
Have an invite siready?
Join Server
Figure 10: Errors made by GROUNDNEXT on mobile devices. Examples are chosen from the SSv2 benchmark.
In this section, we examine the errors made by GROUNDNEXTS and categorise them into broad categories:
Limited Domain Knowledge (Generalization to Web/Mobile): We observe that some errors stem from limited knowledge of web and mobile platforms, as GROUNDCUA predominantly cov- ers desktop software applications. This is most prevalent when generalizing to out-of-domain plat- forms. While GROUNDNEXT performs competitively on Mobile and Web benchmarks, emrors often arise due to distribution shifts. Mobile interfaces, for instance. utilize vastly different aspect ratios, resolutions, and distinct UI patterns that are absent in our desktop-centri training data. Figure 10 illustrates examples where GROUNDNEXT fails relatively simple queries for the mobile platform,
33

---

Published as a conference paper at ICLR 2026
New Note
Figure 11: (Left) Example of near miss for the prompt "Click on the marker already added to the map." (Right) Example from OSWorld-G where the model clicks the wrong cell for the prompt "Move your mouse to the cell in the 3rd column and 12th row (this cell is labeled C12), then press the left mouse button."
such as 'Join a twitch server.' We attribute these errors to a combination of factors, primarily the domain shift inherent in mobile screenshots compared to desktop training data, as well lack of specific application knowledge, which we discuss in greater detail below.
Localization Precision (Near-Misses); We analyzed the magnitude of grounding errors and found that many "failures" are actually correct semantics with imperfect localization. As shown in Fig- 8, over 50% of the errors have relative distance of less than 10% from the ground truth bounding box. This suggests the model successfully identifies the correct target region but occasionally lacks -ixel-perfect precision. We visualize "near-miss" example in Figure Figure 11 (left).
Application-Speciffe Semantics: We observe errors when the model encounters specialized termi- nology or icons unseen software applications. For example, our analysis of Platform Ware in ScreenSpot-Pro (which is not present in our training data), the model struggles with niche tools that require specific software knowledge to identify (eg. "restart from CD", "snapshot details"), whereas it remains robust on generic UI elements like "Refresh", "font size".
Spatial Reasoning Limitations: Despite inclusion of spatial data in our instruction mix (es 13%). the model shows notable performance gap when handling complex relative instructions. This is quantified in the UI-Vision benchmark, where performance drops from x 70.1% on the "Basic" category to 49.9% on the "Spatial" category. We suspect that while the current data helps, solving this fully may require a base model with stronger inherent spatial reasoning capabilities or higher ratio of spatial instruction tuning. In Figure 11 (right), we show error in localising the correct cell in the Libre Office Calc platform (from the OSWorld-G benchmark)
F LIMITATIONS
While our work makes significant progress in desktop GUI grounding, there are a few key limita- tions. Although it covers 87 applications across 6 categories, the dataset may not fully represent the diversity of desktop software, as it is biased toward commonly used applications. Our keyframe- based annotations capture static UT states but miss dynamic elements like animations and real-time updates. While we' ve taken steps to ensure annotation consistency, human labeling at scale can still introduce some inconsistencies, and the time and cost of annotation limit scalability. Additionally, our evaluation focuses on benchmark accuracy, but real-world applications require robustness to changes in distribution, new app versions, and UI updates; issues that need further exploration. Fi- nally, we do not perform end-to-end agentic testing for task completion, which remains an important area for future work.
G LLM USAGE
In our work, LLMs are used for the following aspects:
34

---

Published as a conference paper at ICLR 2026
Using an LLM to help with paper writing. We use GPT5 to help optimize language, correct grammar, and write ETEX table code.
Using an LLM as a research assistant. We use GPT5 to help search related works.
Using LLMs in our methods and experiments. This is described in the paper.
35


Reviews:
[{"review_id": "jgBG9YkgCQ", "reviewer": "Reviewer_heBV", "summary": "This paper introduces GROUND-CUA, a large-scale, human-annotated grounding dataset, and demonstrates its effectiveness using two-stage, state-of-the-art grounding models evaluated on several desktop benchmarks. The dataset offers diverse applications, detailed annotations, and broad coverage of action types and pixel densities. The paper’s introduction of the concept of phased rewards—which encourages models to gradually move toward the ground truth—is particularly innovative and interesting.", "questions": "1. Why choose RLOO rather than the normally used GRPO? Are there any preliminary experiments that show RLOO is better?\n\n2. When does $D_{norm}$ in the reward definition exceed 0.5? Does that imply the grounding box is approximately the same size as the screenshot? Are you using the bounding box center as the reference point? Could you provide concrete examples for the different reward types?\n\n3. I would be grateful if the authors could provide some analysis on the dataset’s scaling trend: if we sample datasets of different sizes and perform SFT on the model, will we observe a similar scaling curve in the grounding domain?\n\n4. It would be better if the test results in the main table had confidence intervals.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. A new SOTA performance on grounding tasks: with only one-tenth of the training data compared to previous methods, GROUNDNEXT achieves state-of-the-art performance on desktop benchmarks while also generalizing to OOD categories, demonstrating the effectiveness and huge contribution of this grounding dataset.\n2. Action and task types that match real-world computer tasks: demonstrations are collected from real-world user experiences, and the applications span diverse real-world use cases, helping to narrow the gap between synthetic and real-world grounding tasks.\n3. The instructions generated for fine-tuning target diverse descriptions — including directness, intent, and location — thereby enhancing the generalization ability of the fine-tuned models.", "weaknesses": "1. Although the authors demonstrate their dataset's effectiveness on the grounding task, they did not show whether this grounding ability transfers to improved performance on computer-use tasks.\n2. After the model underwent reinforcement learning on 10K samples, the improvement was limited, casting doubt on the authors' motivation for using the reinforcement learning step.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IWou2zioyw", "reviewer": "Reviewer_AXGb", "summary": "This paper tackles the grounding problem for computer-use agents. Basically helping AI systems correctly identify which UI element to click when given natural language instructions. The main contribution is GROUNDCUA, a massive dataset with 56K desktop screenshots from 87 applications. The dataset coverage is pretty broad, from office software to creative tools to development environments.", "questions": "Please see the weakness section above for more detailed analysis.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "GROUNDCUA fills a major gap in desktop grounding data. Most existing datasets focus on web or mobile interfaces, but desktop apps are way more complex with tiny icons and dense layouts. The human expert approach is smart - instead of automated scraping, they had people actually use the software and annotate everything they see.\n\nThe results show that data quality beats data quantity. The two-stage approach with supervised fine-tuning plus reinforcement learning is pretty straightforward. No complex reward engineering needed.\n\nEven though the models only see desktop data during training, they transfer surprisingly well to mobile and web interfaces. This suggests that learning good grounding skills on complex desktop environments helps with simpler interfaces too.", "weaknesses": "The paper describes three instruction types but doesn't analyze how they affect model performance differently. The instruction generation relies heavily on prompting Qwen2.5-VL-72B. But there's no discussion of prompt sensitivity or failure cases.\n\nWhat happens when the LLM generates wrong instructions? The paper mentions using \"about 100 templates\" for textual elements and \"120 templates\" for general ones. But it doesn't say which templates work best or how template diversity impacts training.\n\nThe discrete reward function with six levels seems somewhat arbitrary. Why these specific thresholds (-0.5, -0.1, 0, 0.1, 0.5)? The paper mentions trying continuous and binary rewards but doesn't provide detailed comparisons.\n\nThe RLOO method choice over GRPO is mentioned briefly but lacks analysis. How sensitive is the approach to the group size (n=8) and batch size (64) choices? The modest RL improvements suggest the reward signal might not be optimal. More sophisticated reward formulations could yield better gains.\n\nThe error analysis in Appendix E is brief and doesn't systematically categorize failure modes across domains. The paper mentions some errors come from \"limited domain knowledge\", it would be helpful to explore this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "374mo9VV2w", "reviewer": "Reviewer_aAn7", "summary": "The paper introduces GROUNDCUA, a large, expert annotated desktop UI grounding dataset built from human demonstrations across 87 applications and 56K screenshots with over 3.56M element annotations. The authors convert these dense annotations into contextual instruction. Based on this dataset, a series of 3B/7B models are also trained, which are termed GROUNDNEXT. They are trained with supervised fine-tuning, followed by a light RL. On five benchmarks spanning desktop, web, and mobile, GROUNDNEXT achieves the top average SFT performance at both model sizes, with RL offering small but consistent gains.", "questions": "Listed above.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "GROUNDCUA features high quality, human verified supervision data. The elements are hand labeled from expert task demonstrations, giving reliable targets rather than noisy accessibility or synthetic signals.\n\nThe dataset contains dense and fine grained coverage of samples, with screens averaging 64 labeled elements to support precise grounding at pixel-level granularity.\n\nThe SFT model has demonstrated strong performance with modest data volume. GROUNDNEXT tops SFT baselines across five benchmarks for both 3B and 7B sizes, highlighting data quality over raw scale.", "weaknesses": "As a dataset paper, it would be great that the authors can provide a link to the dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BtiIjb8LLR", "reviewer": "Reviewer_Xw9S", "summary": "This paper introduces GROUNDCUA, a large-scale human-expert-annotated dataset for desktop UI grounding. The dataset is comprehensive, covering 56K screenshots from 87 applications with over 3.56M dense annotations. The authors also present GROUNDNEXT, a series of models (3B and 7B) trained on this dataset using a two-stage SFT-RL process. GROUNDNEXT achieves state-of-the-art performances on five grounding benchmarks, significantly outperforming models trained on datasets 10x larger. The paper argues for the superior value of high-quality, dense, expert-driven data over sheer data quantity for this task.", "questions": "Please see weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The GROUNDCUA dataset. It collects grounding samples during human-driven task execution. The inclusion of complex and diverse desktop screen states directly addresses a major gap in existing resources.\n- Sample efficiency. This is another valuable feature of the dataset. Training with significantly fewer samples, GROUNDNEXT can outperform models with much more samples.", "weaknesses": "- Although during data collection, synthetic tasks are written by human and conducted by human in real environments, only the grounding samples automatically extracted from each step is collected for training a grounding-only model. Can the authors elaborate more on the reasons and rationales behind such design?\n- A highly relevant work on scaling desktop UI data and training a grounding model is lacking discussion in the paper: Aria-UI: Visual Grounding for GUI Instructions.\n- Limited RL impact: 66.4->68.4 and 69.2->70.5. Considering that in GTA1, RL training brings very significant improvements, what's the underlying factors that lead to limited RL impact on GROUNDNEXT after SFT? Would it be relevant to the data? Also, it might be valuable to see how the GTA1-style heavy RL training for grounding would work with the GROUNDCUA dataset.\n- Is there a timeline for open-sourcing the tasks and trajectories?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)