GUI-Actor: Coordinate-Free Visual Grounding for GUI Agents
Authors: Qianhui Wu
Kanzhi Cheng
Rui Yang
Chaoyun Zhang
Jianwei Yang
Huiqiang Jiang
Jian Mu
Baolin Peng
Bo Qiao
Reuben Tan
Si Qin
Lars Liden
Qingwei Lin
Huan Zhang
Tong Zhang
Jianbing Zhang
Dongmei Zhang
Jianfeng Gao
Venue: NeurIPS 2025 poster ()
Keywords: GUI Agents
cooridinate-free visual grounding
VLM
VLA
Primary area: deep_learning
TL;DR: We propose GUI-Actor, a VLM-based, coordinate-free GUI grounding method with an attention-based action head and verifier, achieving state-of-the-art results and strong generalization.

Abstract: One of the principal challenges in building VLM-powered GUI agents is visual grounding—localizing the appropriate screen region for action execution based on both the visual content and the textual plans. Most existing work formulates this as a text-based coordinate generation task. However, these approaches suffer from several limitations: weak spatial-semantic alignment due to lack of explicit spatial supervision; inability to handle ambiguous supervision targets, as single-point predictions penalize valid variations; and a mismatch between the dense nature of screen coordinates and the coarse, patch-level granularity of visual features extracted by models like Vision Transformers. In this paper, we propose **GUI-Actor**, a VLM-based method for coordinate-free GUI grounding. At its core, **GUI-Actor** introduces an attention-based action head that learns to align a dedicated `<ACTOR>` token with all relevant visual patch tokens, enabling the model to propose one or more action regions in a single forward pass. In line with this, we further design a grounding verifier to evaluate and select the most plausible action region from the candidates proposed for action execution. Extensive experiments show that **GUI-Actor** outperforms prior state-of-the-art methods on multiple GUI action grounding benchmarks, with improved generalization to unseen screen resolutions and layouts. Notably, **GUI-Actor-7B** achieves scores of **40.7** with Qwen2-VL and **44.6** with Qwen2.5-VL as backbones, outperforming **UI-TARS-72B (38.1)** on ScreenSpot-Pro, with significantly fewer parameters and training data. Furthermore, by incorporating the verifier, we find that fine-tuning only the newly introduced action head (~100M parameters for 7B model) while keeping the VLM backbone frozen is sufficient to achieve performance comparable to previous state-of-the-art models, highlighting that **GUI-Actor** can endow the underlying VLM with effective grounding capabilities without compromising its general-purpose strengths. Project page: [https://aka.ms/GUI-Actor](https://aka.ms/GUI-Actor)

GUI-Actor: Coordinate-Free Visual Grounding for GUI Agents
Qianhui Wu"
Kanzhi Cheng"
Rui Yang"
Chaoyun Zhang'
Jianwei Yang'
Huiqiang Jiang'
Jian Mu²
Baolin Peng'
Bo Qiao'
Reuben Tan'
Si Qin'
Lars Liden'
Qingwei Lin'
Huan Zhang'
Tong Zhang"
Jianbing Zhang"
Dongmei Zhang'
Jianfeng Gao
Microsoott
Nanjjng University
Unniversity of Illinois Urhana-Champaign
Abstract
One of the principal challenges in building VLM-powered GUI agents is visual grounding-
the appropriate screen region for action execution based both the visual content and the textual plans. Most existing work formulates this as a text-based coordinate generation task. However, these approaches suffer from several limitations: weak spatial-semantic alignment due lack of explicit spatial supervision; inability to handle ambiguous supervision targets, as single- point predictions penalize valid variations; and mismatch between the dense nature of screen coordinates and the coarse, patch-level granularity of visual features extracted by models like Vision Transformers. this paper, we propose GUI-Actor. a VLM-based method for coordinate- free GUI grounding. At its core, octor introduces an attention-based action head that learns to align a dedicated <ACTOR> token with all relevant visual patch tokens, enabling the model to propose one more action regions further design grounding verifier to evaluate and select the most plausible action region from the candidates proposed for action execution. Extensive experiments show that GUI-Actor outperforms prior state-of-the-at methods multiple GUI action grounding benchmarks, with improved generalization to unseen screen resolutions and layouts. Notably, GUI-Actor-7B achieves scores of 40.7 with Qwen2- VL and 44.6 with Qwen2.5VVL as backbones, outperforming UI-TARS RS- 72B (38.1) ScreenSpot-Pir with significantly fewer parameters and training data. Furthermore, by incorporating the verifier, we find that fine-tuning only the newly introduced action head (~100M parameters for 7B model) while keeping the VLM backbone frozen is sufficient to achieve performance comparable to endow the
single forward pass. In line with this, we
previous state-of-the models, highlighting that GUI-Actor underlying VLM with effective grounding capabilities without compromising its general-purpose strengths. Project page: https: fttts://abee //aka. ma/ Pese/UUIActoon
Introduction
With the rapid advancement of large language models (LLMs) and vision-language models (VLMs), there is increasing interest in building GUI (Graphical User Interface) agents that understand natural language instructions and autonomously interact with software interfaces across platforms such as desktops [ 2) mobile devices 33) and web applications 4 Effective GUI agents require two core capabilities: (i) multimodal perception to interpret visual and linguistic cues, and (ii) action
'Equal contribution: cotpiiivwmmmmrrooof....ommm
ry21 @ illinois.cdu
t "Leadership.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Gul-Actor-78
40
Performance Screen
GUI-Actor-28
UI-1ARS-75
30
UGround-7B &
UGround 28
TARS 29
05-Arlas-78
20
AGUVS-78
Araall-3..n
20
ShowLi-26 'r
SeeClik 8I
1M
2M
Data Volume (# of Screenshots)
Instruction: Restart from CDs
Figure I: Left: Model performance vs. training data scale on the ScreenSpot-Pro benchmark. Higher and more left is better; larger points indicate models with more parameters. We only show GUI-Actor models built upon WL here for fair comparison. With Qwen2.5-VV VL as the backbone, GUI- Actor-3B/7B reaches scores up to 42.2/44.6 (without Verifier). Right: Illustration of action attention. GUI-Actor grounds target elements by attending to the most relevant visual regions.
execution to interact with digital environments via mouse, keyboard, or touchscreen 15 6) While carly systems relied on structured metadata (e.g. HTML, DOM trees, or view hicrarchies) [. such data is often noisy, inconsistent, or unavailable across platforms. Recent work thus emphasizes visual GUI agents that perceive interfaces directly from rendered screenshots, akin to human users (7]. A central challenge in this paradigm is visual grounding: mapping natural language plans to screen regions. Most existing methods treat this as a coordinate generation task, producing screen positions 1x= y=0.23") through the same text generation mechanisms used by LLMs (8).
However, representing GUT actions through coordinate generation, where models output screen positions text tokens (e.g.. x=.... y=...) introduces several intrinsic limitations. First, spatial- semantic alignment is weak: generating discrete coordinate tokens requires the model to implicitly map visual inputs to numeric outputs via language modeling head, without any explicit spatial inductive bias. This process is inefficient, data-intensive, and prone to errors due to the lack of direct supervision linking visual features to action locations. Second, supervision signals are ambiguous: many GUI actions, such as clicking within a button, allow for a range of valid target positions. However, coordinate-basee methods typically treat the task as fingle-point prediction, penalizing all deviations even reasonable ones-ane failing to capture the natural ambiguity of human interaction. Finally, there is a granularity mismatch between vision and action space: while coordinates are continuous and high-resolution, vision models like Vision Transformers (ViTs) (9] operate on patch- level features. This mismatch forces the model infer dense, pixel-level actions from coarse visual tokens, which undermines generalization to diverse screen layouts and resolutions.
Although some recent approaches [10] attempt to enrich spatial grounding by predicting bounding boxes instead of single points, they still represent these regions as raw coordinate strings (e.g.x_mix min, y y_min, x_maa max. y_ __max) that are detached from the visual features. Without architectural components such as ROI pooling [11 or spatial attention mechanisms [22. such methods fall short of bridging the gap between linguistic intent and grounded visual action.
Rethinking how humans interact with digital interfaces reveals key insight: humans do not calculate precise screen coordinates before acting-thi perceive the target element and interact with it directly. Motivated by this observation, we propose GUI-Actor, a VLM augmented with an attention- based action head, enabling coordinate-free visual grounding that more closely mimics human behavior. Unlike prior methods that treat action grounding as a coordinate prediction task, GUI-Actor learns to attend directly to relevant visual regions without relying numeric coordinate generation. At the core of GUI-Actor a dedicated <ACTOR> token, which encodes the grounding context by jointly processing visual input and natural language instructions. An attention mechanism then learns to align this token with the most relevant GUI regions by attending over visual patch tokens from the screenshot. The resulting attention map naturally identifies actionable regions on the interface.
To address the inherent ambiguity in GUT interactions, where multiple points within a UI element (e.g.a button) may all be valid, GUI-Actor is trained using multi-patcl supervision. All visual

---

patches overlapping with ground-truth bounding boxes are labeled as positives, while others are treated as negatives. This supervision strategy allows the model to tolerate spatial ambiguity and reduces per-ppealliztionn of reasonable action variations. Furthermore, because GUI-Actor grounds actions directly at the vision backhone's 's native spatial resolution, it avoids the granularity mismatch of previous methods and generalizes more robustly across different screen sizes, resolutions, and layouts. Finally, to support decision refinement, we further enhance GUI-Actor by presenting a lightweight grounding verifier that evaluates multiple candidate regions and selects the most plausible one for action execution.
Our contribution can be summarized as follows:
1. We revisit recent coordinate generation-base approaches for visual grounding in GUI agents, identify their Limitations-sue as weak spatial-semantic alignment, ambiguous supervision targets, and mismatched feature granularity and propose GUI-Actor, novel coordinate-free method that effectively addresses these issues.
We design an attention- treettion-based action head, which can generate multiple candidate regions in a single forward pass, offering flexibility for downstream modules such as search strategies.
3. We introduce grounding verifier to select the most likely action region among the candidates proposed from the action attention map. We show that this verifier can be easily integrated with other grounding methods for a further performance boost.
Extensive experiments demonstrate that GUI-Actor outperforms the state-of-the-art methods trained on a similar scale of data across multiple GUI action grounding benchmarks, and ex- hibits greater robustness to unseen screen sizes and resolutions. Remarkably, the 2B version of GUI-Actor even surpasses several competing 7B models. Furthermore, by leveraging the verifier, GUI-Actor with lightweight training (i.e. freezing the backbone LLM and fine-tuning only the newly introduced ~000M parameters in the action head) can effectively equip the underlying strengths.
VLM with grounding capabilities without compromising its
2 Related Work
LLM/VLM-PoWern GUI Agents. The advent of LLMs and VLMs has catalyzed the development of GUI agents that can understand natural language instructions and perform complex tasks across mobile [13,, web [5), and desktop environments [16] I 2]. Early research focused on designing
autonomous agent frameworks 4 17
that prompt commercial models interact with operating
systems via code generation [19 20 or tool use 21
22). With rising demand for open-sourg and customizable agents, a parallel line of work focuses on training LLMs/VLMs for enhanced
agentic capabilities, including GUI understanding, planning, and execution 7 TO 23 24]. The key
to these efforts collecting GUT-specific training data, such as OCR annotations
25 interface
summaries (26. QA pairs [23] and large-scale task demonstrations [15 22
central requirement agent development is the ability to interact with realistic GUI environments deployed in virtual machines and aarome-based browsers. While early agents operated over structured metadata like HTML or accessibility trees 14 33 such representations are brittle and inconsistent across platforms (3 77 Consequently, recent trends have shifted toward vision-centric paradigm, where agents interact with raw screenshots using mouse and keyboard inputs [35] 36), closely mimicking human behavior. Within this setting, central challenge emerges: grounding natural language instructions to specific GUI regions, referring as GUI Visual Grounding.
GUT Visual Grounding. Given a GUT screenshot and a natural language instruction, GUI visual grounding aims to locate the target region for interaction. Although conceptually related grounding in natural images, this task presents unique challenges due to the semantic density and structural regularity of GUI layouts [34] []. A common approach frames GUI grounding as a text-based coordinate prediction task, where models generate point positions (e.g., x= .) as output language tokens [37 38). This formulation has led to widespread adoption due to its simplicity and compatibility with existing LLMs/VLMs. To improve performance, prior works have scaled both models and training data [39,4] 10 44 UGround g proposes a data pipeline for synthesizing
45 offers multi-platform dataset and a unified GUI action model. More recently, Xu et al. (4 introduced training- approach that performs GUI grounding by leveraging the internal attention of VLMs.
diverse GUT grounding examples, while OS-Atlas
3

---

Action Head
<ACTORS
Self-Atta XLF-VV
MIPP-
LM Head
Backbone
Tokenizer
Encoder
Tokeniser
System Prompt
Screensbot
User Prompe
Text Response
(a) Illustration attention- action head.
(b) Image Patch Labels
Figure 2: Overview of GUI-Actor. (a) Illustration of how the action head works with VLM for coordinate- free GUI grounding. (b) Illustration of the apat multi-patch supervision for model training. We label all image patches that are partially or fully covered by the ground-truth bounding box as positive (1) and all others as negatives (0).
Despite their success, coordinate- boodeinate-based methods suffer from key limitations, including weak spatial inductive bias, ambiguous point supervision, and resolution mismatches between visual features and action targets. This paper presents compelling alternative to the prevailing coordinate-based method: GUI-Actor, novel coordinate-free grounding framework for GUI agents. introduces <ACTOR> token that attends directly to relevant image patches via an attention- based action head, enabling more human-like grounding while mitigating the limitation of coordinate-bazt methods.
3 The Design of GUI-Actor
Considering the limitations of text-based coordinate generation, e-g., weak spatial-semantic alignment and ambiguous supervision targets, we draw inspiration from how humans interact with GUIs. Rather than computing precise coordinates, humans typically visually identify the intended element and then directly act on it, by tapping with a finger or positioning a mouse cursor. Motivated by this, GUI-Actor explores novel architecture for GUI visual grounding: token <ACTOR> as the contextual anchor, and then train an action attention head to attend from this anchor to image patches corresponding to the target element. Finally, present grounding verifier to select the most semantically appropriate target among the multiple candidates derived from the attention map.
first introduce special
<ACTOR> Token Contextual Anehor Given screenshot image I and instruction coordinate generation based methods typically train the VLM to produce a sequence of (#1. Fnitm (0+mm++m--- Wjjttn 2 Bfjn+tii I:N], where m, 0, N is the total length of the output sequence, > 1, and 1 + 1+m m + 1. For example, in pyautogui... cetpgiiillllcixllleeeeeeoo 123, y=0. 234) Xu al. 17). the segments Fii+m and Fzs++n correspond the tokenized x- and y-coordinates, respectively. The segment 3i+m+1.j-1 represents the separator between them, while the rest captures the surrounding context. In this work, we replace the coordinate span (mai+mm Litm+1lj-1, {zjttn with three special tokens to facilitate coordinate- grounding and better context integration from both the visual input and textual instruction:
VI. ④) = (01:-11,
<ACTOR>, <ACTOR_ END>, 23-3:N)
(1)
We use the final-layer hidden state of <ACTOR>, i.e., hEACTORS contextual anchor for computing action attention over the visual patch tokens.
Attention-Bas -aaeid Action Head Let Ui,... UM denote the visual patch features extracted by Vision Encoder of the VLM from the input screenshot, where each E Rdd, The action head computes an attention distribution from the <ACTOR> token over these visual patches identify the target action region.

---

To incorporate GUl-aware contextual information, we first apply a self- celf-attention layer over the visual patch features. This allows the model aggregate semantics across spatially related patches, enabling patches that belong to the same GUT element to share coherent representations:
U
UM SelfAttn(UI,....... UM);
(2)
where 6 R& denotes the contextualized feature for the i-th patch after self self-attention module. Note that we do not introduce additional positional embeddings in this self-attention layer, as the input features from the VIT backbone already incorporate positional information.
Next, we project both the <ACTOR> token representation h<AACTOR> and the contextualized patch features u1...., UM into a shared embedding space via two separate MLPs, and obtain and 24 E IR
2 = MLPr (h<aaToOCT)))>) 2; = (i.)
(3)
Finally, we compute attention scores between the <ACTOR> token and each visual patch. Let M denote the total number of image features that are input to the LM backbone, the resulting attention weights 01..... ay form an attention map over the screen, indicating the most relevant region for grounding the action.
2T, 0, =
04 = exp(as) E exp(a))'
for i = 1, 1,..., ...,,... M.
(4)
Vd
Spatial-Aware Multi-Patch Supervision key advantage our approach the ability to leverage dense and spatially structured learning signals from bounding-box supervision. Rather than relying on a single, potentially ambiguous click point as in traditional coordinate-based methods, GUI-Actor treats all image patches that are partially or fully covered by the ground-truth truth bounding box positive examples (y,=l) and all others as negatives (3/- = 0), where y, denotes the label associated with v/. This allows the model to more effectively capture the full spatial extent of actionable elements. An illustration is provided Figure 2b
For more details, please refer to Appendix B
We train the model using combination of next-token prediction (NTP) loss and action attention loss:
L = LNTP + Lccttio_AAm Aatt:
(5)
The action attention loss is defined as:
Ps CAction Attn Σ Pa
Vi
P.
1 1, 1,..., M,
(6)
a,
Σ j=1 Dj +
where is small constant for numerical stability.
4 Grounding Verifier
A key advantage of our attention based action grounding model is its ability to produce multiple candidate action regions in a single forward pass, without incurring additional inference cost. This is a natural consequence of the attention-base based design, where the model assigns scores to all visual patches simultaneously. This efficiency creates an opportunity: rather than relying solely on the top-scoring patch, we can introduce a lightweight verification step select the most semantically appropriate target among the candidates.
With the insight that verification is often easier than generation 47 lightweight VLM module which takes as input the instruction and a screenshot with a visual marker placed at the proposed location, and predicts whether the marked region correctly fulfills the task intent. This verifier serves as a decision refinement layer, allowing the system to reflect on its action before execution.
we propose a Grounding Verifier,
Data Training Training data for the verifier is constructed from the OS-Atlas dataset [45]. which spans desktop. mobile, and web domains. This dataset provides triplets of the form (image, query, bounding box), where each image is associated with multiple queries and their cor- responding bounding boxes. For each triplet, we generate positive example by placing visual
5

---

marker (i.e., hollow red circle) at the center of the bounding box, treating it as the correct grounding point for the given query. To create negative examples, we apply two strategies: (1) selecting the center of an incorrect bounding box from the same image; (2) randomly sampling point outside the target region. Each resulting training instance formatted as a tuple (I, x, y), where I is an image with marked point, is the corresponding language instruction, and E { (TTrue?, 'False') } is the ground-truth label indicating whether the point correctly satisfies the instruction. More details are deferred to Appendix F.I
We fine-tune the verifier from a lightweight VLM using standard supervised learning. The model takes (I, x) as input and is trained to generate the correct token y. The training objective is the cross-entropy loss:
Veeiiierr log Pe. (y I,x,,
where Pe. denotes the output probability from the verifier model with parameters 0,.
Inference At inference time, GUI predicts the final action location by combining natural language generation with visual grounding. Given the current GUI state and user instruction, GUI-Actor first generates an agent response via standard decoding, for example, producing a string like pyautogui. click click(<ACTOn) (<ACTOR_ (<ACTOR_START> START><<CTOR>><ACTO ACTOOREND>>>) that includes the special <ACTOR> token. We then extract the hidden state corresponding to <ACTOR> and use the action head to compute attention over all visual patches. This attention distribution serves spatial activation map, identifying the most relevant screen region for executing the predicted action.
To identify the most semantically appropriate region among the top-K attention attention-weighted patches, use the verifier 8, to score each candidate by marking it on the image I and evaluating its alignment with the instruction For each marked image, the verifier outputs probabilities for 'True" and 'False' tokens, and we define the selection score as:
P I, [I,x) + Po. ('False' I,x)
s(I, x)
Candidates are evaluated in descending order of attention weights, and we return the first one exceeding a confidence threshold (c.g., s(I,x) > 7) without further evaluation.
Experiments
Implementation Details We implement GUI-Actor using PyTorch and Huggingface Transformers. Unless otherwise specified, we adopt Qwen-2-VL. VL--BBInnstrue [38 as the backbone VLM for both GUI-Actor and the baselines ensure a fair comparison with previous state-of-the-arr methods. For the re-implementation of baseline Aguvis-7B with both point supervision (point sup.) and bounding-box supervision (bbox sup.). we directly use the official source code provided by Aguvis (7). The number of attention heads in the self-attention layer is set to 8; Both MLP components are two-layer feedforward networks with a GELU activation in between. We use the same dimensionality as the backbone VLM for all configurations of the action head. The grounding verifier is finetuned from UI-TARS-2B-S3 [T0]. During inference, we construct a pool of K = 20 candidates and apply confidence threshold of 7 0.95 for tasks in ScreenSpot-Pro and 9 0.8 for ScreenSpot and ScreenSpot-v2. Following Aguvis 77. we structure our training data as sequences of pyautogui- style operations, but replace the original coordinates with the special tokens, described in Section 3 Our full training recipe is built from several public GUI datasets, comprising ~-MM screenshots. Both GUI-Actor and the two baseline models trained using the data recipe summarized in Table for 1 epoch. Additional dataset details are provided in Appendix [ To train GUI-Actor, we begin by freezing all backbone VLM parameters and training only the newly introduced components of the action head 1~20M/~100M parameters for 2B/7B backbone). After this warm-up up phase, we finetune the entire model using standard supervised learning.
Evaluation Benchmarks & Metric We evaluate GUI-Actor and other baseline methods on three well-established GUI visual grounding benchmarks: ScreenSpot [44, ScreenSpot-v2 [45), and ScreenSpot-Pro 48 with the last featuring higher-resolution interfaces and greater domain shift (e.g., industrial software, mindi-window layouts), serving as a practical testbed for generalization. For the evaluation metric, use Element Accuracy, which measures the proportion of predictions

---

Table Performance comparison on ScreenSpot-Pra which features nigher-resolution interfaces and greater domain shift (e.g., industrial software, multi-window layouts), serving as practical testbed generalization.
Dev
Creative
CAD
Scientific
Office
OS
Avg-Text
Avg-Icon
Avg
GPT-40
0.7
0.6
1.5
1.2
0.9
0
1.3
0
0.8
Claude Compute
12.6
16.8
11.9
25.8
26,9
8.1
23.4
7.1
17.1
OS-Atlas-IB
2.3
1.5
7.5
4.8
3.1
5.0
1.7
3.7
U1-2B
5.3
1.9
10.6
13.5
6.6
10.8
2.6
7.7
Soround-V1-222
27.4
26.7
14.6
34.3
38.3
17.9
26.6
UI-TARS-2B
26.4
27.6
14.6
39.8
42.6
14.3
39.6
8,4
27.7
GUI-Actor-2B
34.8
35.5
28.4
38.2
53.9
30.6
50.7
14.1
36.7
GUI-Actor-2B Verifier
41.8
36.7
34.5
41.3
62.6
36.2
57.6
16.1
41.8
Qwen2-VL-7B
1.3
0.9
0.4
3.5
3.0
0.5
2.5
0.2
1.6
SceClick-9.6B
0.3
0.6
1.9
2.0
0.9
1.5
1.8
0
1.1
Aria-U1-2-5.3B
8.4
14.7
6.1
18.1
16.1
2.6
17.1
2.0
11.3
OS-Atlas-7B
17.7
17.9
10.3
24.4
27.4
16.8
28.1
4.0
18.9
Aguvis-7B
16.1
21.4
13.8
34.6
34.3
19.4
22,9
UGround-V1-1
28.1
31.7
14.6
39
49,6
24.5
31.1
UI-TARS-7B
32.8
18.0
53.5
24.5
16.2
35.7
GUI-Actor-7B
38.8
40.2
29.5
44.5
56.5
36.2
55.8
16.4
40.7
GUI-Actor-7B Verifier
38.8
40.5
37.2
44.5
64.8
43.9
60.7
17.6
44.2
CogAgent-1890
5.6
6.1
13.4
10.0
3.1
12.0
0.8
7.7
WGround-72B-V
35.8
13.8
$1.3
25.5
34.5
UI-TARS-72B
40.8
39.6
17.2
45.7
54.8
30.1
50.9
17.5
38.1
Table 2: Performance comparison on ScreenSpot.
Mobile-Texx
Mobile-Icon
Desktop-Text
Desktop-Icon
Web-Text
Web-Icon
Avg
GPT44 GPT-4o
22.6
24.5
20.2
11.8
9.2
8.8
16.2
20.2
24.9
21.1
23.6
12.2
7.8
18.3
Claude Computer Use
83.0
Gemini 2.0
84.0
UGround-Vi G-Grund-V1-28
89.4
72.0
88.7
65.7
81.3
68.9
77.7
UI-TARS-2B
93.0
75.5
90.7
68.6
84.3
74.8
82.3
GUI-Actor-2B
93.0
79.9
88.1
78.6
90.9
84.0
86.5
GUI-Actor-2B Verifier
93.8
81.2
89.7
80.7
91.3
80.6
86.9
Qwen2-V1.22
75.5
60.7
76.3
54.3
35.2
25.7
55.3
CogAgent-778
67.0
24.0
74.2
20.0
70.4
28.6
47.4
SteClick-9.6B
78.0
$2.0
72.2
30.0
55.7
32.5
53.4
Magma-8B
60.4
58.5
75.3
52.9
69.1
52.0
60.3
Aguvis-G-7B
88.3
78.2
88.1
70.7
85.7
74.8
81.8
OS-Atlas-7kt
93.0
72.9
91.8
62.99
90.9
74.3
82.5
Aguvis-7B
95.6
77.7
93.8
67.1
88.3
75.2
84.4
UGround-v1-7B
93.0
79.9
93.8
76.4
90.9
84.0
86.3
UI-TARS-78
94.5
85.2
95.9
K5.7
90.11
K3.5
89.5
GUI-Actor-7B
94.9
82.1
91.8
80.0
91.3
85,4
88.3
GUI-Actor-7B Verifier
96.0
83.0
93.8
82.1
92.2
87.4
89.7
UI-TARS-72B
94.9
82.5
89.7
88.6
88.7
85.0
88
Aguvis-72B 72B
94.5
85.2
95.4
77.9
91.3
85.9
89.2
UGround-V1-727
94.1
83.4
94.9
85.7
90.4
87.9
89.4
where the click point falls within the ground-truth bounding box of the target element. Please refer to Appendix E for more details the benchmark information.
Baselines We primarily compare our approach against models of comparable scale (~7B parame- ters). The baselines include (i) Mlsed-souuce models like GPT-4o [9]. Claude for Computer Use
[50]. and Gemini 2.0 [5I]. well as (ii) open- source models like SeeClick 3. Show [39].
and Magma [52]. We particularly highlight several baselines that share the same backbone as ours, including the backbone Qwen2-VI. 38,, Aguvis-7B 7. UGround-v1- 1-BB X8) and UI-TARS-7B (00
We also conduct performance comparison among Qwen-2.5-VI and models using it as backbone like Jedi 44. Unless otherwise specified, all numbers are reported from the original paper or from the
UI-TA TI-TARSS benchmark[ID

---

Table 3: Performance comparison on ScreenSpot-v2 v2. indicates results obtained from our own
evaluation of the official model on Huggingface.
Moxile-Text
Mobile-Icon
Desktop-Text
Desktop-Icon
Web-Text
Web-Icon
Avg
Operator
47.3
41.5
90.2
80.3
92.8
84.3
70.5
GPT-4o OmniParser-v2
95.5
74.6
92.3
60.9
88.0
59.6
80.7
OS-Atlas-4B
87.2
59.7
72.7
46.4
85.9
63.1
71.9
UI-TARS-2B
95.2
79.1
90.7
68.6
87.2
78.3
84.7
GUI-Actor-2B
95.0
82.2
92.2
81.8
92.9
82.7
88.6
GUI-Actor-2B- + Verifler
95.9
81.5
94.3
82.9
93.6
82.8
89.3
SceClick-9.6B
78.4
50.7
70.1
29.3
55.2
32,5
55.1
Magma-8B
62.8
53.4
80.0
57.9
67.5
47.3
61.5
OS-Atlas-7B
95.2
75.8
90.7
63.6
90.6
77.3
84.1
Aguvis-7Bf
95.5
77.3
95.4
77.9
91.0
72.4
86.0
UGround-V -1-7Bt
95.0
83.3
95.0
77.8
92.1
77.2
87.6
UI-TARS-7B
96.9
89.1
95.4
85.0
93.6
85.2
91.6
GUI-Actor-7B
96.5
84.3
91.7
84.1
93.9
82.3
89.5
GUI-Actor-7B + Verifier
97.2
84.8
94.3
85.0
94.0
85.2
90.9
UI-TARS-72B
94.8
86.3
91.2
87.9
91.5
87.7
90.3
Table 4: Performance comparison of models based on the Qwen-2.5-VL backbone.
SereenSpot-Pro:
Dev
Creative
CAD
Scientific
Office
Avg
Qwen2.5-VL-3B
21.4
18.4
29.5
40,9
20.4
25.9
Qwen2.5.VI.7B
29.1
24.9
13.8
31.1
45.7
22.4
27.6
Jedi-3B
38.
34,6
23.0
38.6
$7.0
25.0
36.1
Jedi-7B
27.4
32.2
52.4
68.7
26.0
39.5
GUI-Actors3B
39.8
36.7
34.1
49.6
61.3
35.2
42.2
GUI-Actor-3B- Verifler
43.8
37.8
43.3
48.4
63.5
42,9
45.9
GUI-Actor-7B
38.1
41.4
38.3
50.8
63.0
38.8
44.6
GUI-Actor-7B- Verifier
43.1
41.9
48.3
47.2
65.7
43.4
47.7
SereenSpot-r2:
Mobile-Text
Mobile-Icon
Desktop-Text
Desktop-lcon
Web-Text
Web-Icon
Avg
Qwen2.5-VL-3B
93.4
73.5
88.1
58.6
88.0
71.4
80.9
Qwen2.5.VL.77
97.6
87.2
90.2
74.2
93.2
81.3
88.8
Iedi-3B
96.6
96.9
78.6
88.5
83.7
88.6
Jedi-7B
96.9
87.2
95.9
87.9
94.4
84.2
91.7
GUI-Actors3B GUI-Actor-3B- Verifier
97.6
83.4
96
83.6
94.0
85.7
91.0
98.3
96.9
87.9
95.3
86.7
92.4
GUI-Actor-7B
97.6
88.2
96.9
85.7
93.2
86.7
92.1
GUI-Actor-7B- Verifier
96.9
89.6
97.4
86.4
95.7
84.7
92.5
Main Results Table I and presentt performance comparisons on ScreenSpot-Prie ScreenSpot, and ScreenSpot-v2 benchmarks. Our models, GUI-Actor-22 2B and GUI-Actor-7B, consistently outper- form existing state-of-the -ft--o--thhrrrr- methods across all benchmarks, with the 2B model even surpassing many competing 7B models. While there is one exception UI-TARS-7B achieving stronger performance, benefits from training significantly larger dataset that includes both public and proprietary data (see Figure [ Additionally, it undergoes a more extensive training pipeline, including continual pre-training. annealing phase, and final stage of direct preference optimization (DPO). Although our models are trained solely with supervised fine-tuning, they achieve competitive or even superior results on ScreenSpot-Prr demonstrating its strong capability and potential.
Robust contof-DDisribbution Generalization As shown in Table both GUT-Actor-2B and GUI- Actor-7B demonstrate strong performance ScreenSpot-Pro- out-of-distribution benchmark with higher resolutions and substantial domain shifts- Surpas-nu the previous state-of-the-aar UI- TARS model by +9.0 and +5.0 points with 2B and 7B parameters, respectively. We attribute this gain to explicit spatial-semea -eetial-semantic alignment: unlike coordinate-baue GUI-Actor leverages an attention- based action head that grounds semantic cues directly in discrete visual regions. This design embeds stronger spatial inductive bias and naturally aligns with the patch- based representations of modern vision backbones. As result, GUI-Actor better equipped to reason over localized visual content, enabling robust generalization across diverse screen resolutions and layouts. Further evidence of this robustness shown in the Figure 3c): as training progresses, GUI-Actor-BB and (UUl-Actor-BB show sustained overfitting on the out-of-domain ScreenSpot benchmark: its accuracy recovers from early dips, then gradually increases before stabilizing. In contrast, baseline performance stcadily declines after peaking early in training.
approaches such as UI-TARS,

---

QUTACTOR
Adoo
Aguss sup3
Aguil 5MD.)
11
Aonumyy
Percentagg ng Data
Percenfage of (%)
Parcaniage Wrainin Data
Accurwcy prograsion Screentpot
(b)
Screen5pal-12
Accuracy Screanspaterro
Figure 3: Accuracy Progression Over Training Steps.
Improved Sample Efficiency Figure illustrates how GUI-Actor 's design leads to improved sample efficiency compared to coordinate-based baselines. GUI-Actor reaches its final accuracy on both ScreenSpot and ScreenSpot-v2 using only ~60% of the training data, outperforming the point and box-supervised models of AGUVIS, which plateau after 80-90% of the data. This efficiency stems from GUI-Actor 's explicit spatial-semantic alignment through its action head, which enables grounding directly at the vision backbone's 's native patch resolution, avoiding the granularity mismatch that hampers baseline methods. Additionally, our multi-pate supervision strategy gracefully handles the supervision ambiguity in coordinate generation based methods, offering dense, spatially structured supervision signals.
Enabling backbone VLM grounding on GUIs without sacrificing general-purpor strengths. We introduce variant, GUI-Actor-Lite where freeze all backbone VLM parameters and train only the newly introduced components for the action head and special tokens. This setup explores how GUI-Actor can impart GUT grounding capabilities without diminishing the VLM's general-purpose strengths. As shown in T3] GUT-Actor-L Litira yields substantial performance improvements over the unmodified backbone VLM. With the help of grounding verifier, it even rivals fully fine-tuned coordinate generation models. These results suggest that the backbone VLM already exhibits strong perceptual understanding of UI screenshots. As such, training the model generate coordinates in text format may primarily focus on coordinate mapping, offering limited contribution to the semantic understanding of UI elements. More importantly, GUI-Actor- Litor-LitTTaiinn retains the backbone's 's original language and vision- language capabilities, demonstrating that lightweight integration can enable grounding without compromising generality.
Table 5: Analysis on lightweight training (-LiteTrain), where the backbone Qwen-2- Bieen2--LLL is frozen, and only the newly introduced parameters for the action head and special tokens are updated training.
Method
Updated # of Params
ScreenSpot-Pro
ScreenSpot
ScreenSpot-r2
19M
25.4
71.4
73.9
GUI-Actor Mul-AActo-2B-iieTTeee 2iteetara + Verifier
19M
79,2
82.3
GUI-Actor-7BB 7B-LiteTTaini
103M
22.9
73.5
74.9
GUL-Actor:B--LieTraaa + Verifier
103M
35.8
81.3
83.8
Boosting Performance via Grounding Verifier The results in Tables 1 3 and demonstrate that the grounding verifier consistently improves performance. highlighting its effectiveness in enhancing grounding accuracy. The improvement especially significant on ScreenSpot-Pro. where it boosts GUI- Actor----- by nearly 4 points and GUI-Actor-78 7B-LitB TriiinTTrain by an impressive 13 points. Additionally, we investigate the benefits of the Verifier Self-Aggregation strategy in Appendix G. and evaluate the verifier's applicability to other baseline models in Appendix G.2) Our findings suggest that our verifier highly robust and well-suited to GUI-Actor, as requires only a single forward pass to generate diverse region proposals.
Ablation Study present the results of our ablation study, where "bbox sup." and "point sup." denote models trained to predict the ground-truth bounding box or action point coordinates in natural language format, respectively. The results show that models trained with coordinate generation (both bounding box and point supervision) consistently underperform compared to GUT-Actor-7B

---

across the benchmarks, highlighting the effectiveness and necessity of explicit spatial- semantic alignment achieved through our proposed action head. Interestingly, despite having access more spatial information, Aguvis-7B (bbox sup.) performs similarly to or worse than Aguvis-7B (point sup.), suggesting that without architectural mechanisms or spatial inductive bias, these coordinate generation based methods remain disconnected from the underlying visual representation, limiting their generalization and grounding capabilities. For more detailed results on different domains and tasks, see Table 10 in the Appendix.
Table 6: Ablation study: averaged performance across benchmarks.
Method
ScreenSpot-Pro
SereenSpot
ScreenSpot-v2
GUI-Actor-BB
40.7
88.3
89.5
Aguvis-7B 7B (bbox sup.)
13.8
84.4
84,4
Aguvis-7B (point sup.)
15.6
85.1
87.0
grounding mechanism, GUI-Actor can generate multiple candidate action regions in single forward pass, incurring no extra inference cost. To evaluate the effectiveness of these high-probability regions, we the Hit@k k metric, where k represents the number of top-ranked predictions considered. Figure shows that GUI-Actor exhibits a substantial improvement from Hit@ to H, whereas the gap for baselines is relatively marginal. In our analysis, we observed that for norddiaae-geenraatinn- baselines, even when multiple predictions are sampled, the outputs are mostly identical, e.g. shifting slightly from (0.898, 0.667) to (0.899, 0.666). In contrast, our model simultaneously produces multiple candidate regions from the attention distribution in a single forward pass. These candidates are mutually exclusive, naturally promoting diversity and increasing the chance of capturing all valid action regions. Figure 4b) provides a qualitative example where our approach successfully identifies all ground-truth regions required for action execution.
Multi-Region Prediction Without Extra Inference Cost Due to its
100
113
60
One Stop Mar ket
S0/e0rpptt
pro
G340m220118
GA4CONT B180
Apar--3 bent mal
(4
(a) Hit@| and Hit@3 for different models.
(b) GUI-Actor can capture multiple potential regions.
Figure 4: (a) @it@1 1 and @it@3 for different methods. For Aguvis baselines, we run inference 3 times with temperature- 1.0, top_p= 0.95. (b) Illustration of multi-region prediction. In this example, the instruction is "check shopping cart"' and the central "shopping cart" text is clickable, while the ground truth is only the top-right icon.
Conclusion
We present GUI-Actor, a novel coordinate-free visual grounding framework for GUT agents. Depart- ing from prevailing text-based coordinate generation paradigms, GUI-Actor introduces dedicated <ACTOR> token that attends to target visual patches to directly localize GUI elements on the screen. This mechanism explicitly aligns spatial visual features with the semantic signals from instructions, and naturally supports bounding-box box-based multi multi-patch supervision, which helps mitigate the ambi- guity inherent in single-poin predictions. Benefiting from its ability to propose multiple candidate regions in single pass, GUI- further employs a lightweight verifier to select the most plausible target at inference time. Experiments across diverse benchmarks demonstrate that GUI-Actor out- performs state-of-the-art methods and exhibits stronger generalization to unseen layouts and screen resolutions. We conduct extensive analyses to understand the effectiveness of each component in our framework, highlighting its promising potential for advancing visual GUI agents.
10

---

Acknowledgments
We would like to thank Boyu Gou for providing the bounding box training data, and Yiheng Xu, Qiushi Sun, Zichen Ding, and Fangzhi Xu for their valuable discussions and insightful suggestions.
References
Chaoyun Zhang, Liqun Li, Shilin He, Xu Zhang, Bo Qiao, Si Qin, Minghua Ma. Yu Kang, Qingwei Lin. Saravan Rajmohan, et al. Ufo: A ui-focused agent for windows os interaction. arXiv preprint urXiv:2402.07939, 2024.
[2] Chaoyun Zhang, He Huang, Chiming Ni, Jian Mu, Si Qin, Shilin He, Lu Wang, Fangkai Yang, Pu Zhao, Chao Du, et al. Ufo2: The desktop agentos. arXiv preprint arXiv:2504. 2025.
[3] Junyang Wang, Haiyang Xu, Jiabo Ye, Ming Yan. Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. Mobile-agent: Autonomous multi-modal mobile device agent with visual perception. In ICLR 2024 Workshop on Large Language Model (LLM) Agents.
[4] Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-4v (ision) is a generalist web agent, grounded. arXiv preprint arXiv:2401.01644 2024.
[5] Chaoyun Zhang. Shilin He, Jiaxu Qian. Bowen Li, Liqun Li, Si Qin, Yu Kang, Minghua Ma. Guyue Liu, Qingwei Lin, et al. Large language model- gui agents: A survey, arXiv preprint 1..i::4411..22777 2024.
[6] Lu Wang, Fangkai Yang. Chaoyun Zhang. Junting Lu, Jiaxu Qian, Shilin He, Pu Zhao, Bo Qiao, Ray Huang. Si Qin, et al. Large action models: From inception implementation. arXiv preprint (xXx:22412.10047 2024.
[7] Yiheng Xu, Zekun Wang, Junli Wang, Dunjic Lu, Tianbao Xic, Amrita Saha, Doyen Sahoo, Tao Yu, and Caiming Xiong. Aguvis: Unified pure vision agents for autonomous gui interaction. arXiv preprint arXiv:2412.04454. 2024.
[8] Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. Navigating the digital world humans do: Universal visual grounding for gui agents. arXiv preprint arXiv:2410.05243 05.444.... 2024.
[9] Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URKKtttps://ppporreeeeeeett Id-YiebFaNTty
[10] Yujia Qin, Yining Ye, Junjie Fang. Haoming Wang, Shihao Liang. Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, et al. Ui-tars: Pioneering automated gui interaction with native agents. arXiv preprint 250v 12326, 2025.
[11] Yuxuan Sun, Chong Sun, Dong Wang, You He, and Huchuan Lu. Roi pooled correlation filters for visual tracking. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 5783-5791, 2019.
[12] Xizhou Zhu, Dazhi Cheng, Zheng Zhang, Stephen Lin, and Jifeng Dai. An empirical study of spatial attention mechanisms deep networks. In Proceedings of the IEFF/CVF international conference on computer vision, pages 6688-6697, 2019.
[13] Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Al- ice Li, William Bishop. Wei Li, Folawiyo Campbell-Ajala, et al. Androidworld: A dynamic benchmarking environment for autonomous agents. arXiv preprint MrXiv:2405.14577 2024.
[14] Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk. Daniel Fried, et Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307. 13854, 2023.
[15] Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang. Huan Sun, and Yu Su. Mind2web: Towards generalist agent for the web. Advances in Neural Information Processing Systems. 36:28091 181114, 2023.
11

---

[16] Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao. Toh Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, al. Osworld: Benchmarking multimodal agents for open-ended tasks real computer environments. Advances in Neural Information Processing Systems, 37: 52040-52094 2024.
[17] Zhiyong Wu, Chengeheng Han, Zichen Ding, Zhenmin Weng, Zhoumianze Liu, Shunyu Yao, Tao Yu, and Lingpeng Kong. Os-copilot: Towards generalist computer agents with improventt arXiv preprint arXiv: arXiv:2402.07456 2024.
[18] Chengyou Jia, Minnan Luo, Zhuohang Dang, Qiushi Sun, Fangzhi Xu, Junlin Hu, Tianbao Xic, and Zhiyong Wu. Agentstore: Scalable integration of heterogeneous agents specialized generalist computer assistant. arXiv preprint arXiv:2410. 18603, 2024.
[19] Qiushi Sun, Zhirui Chen, Fangzhi Xu, Kanzhi Cheng, Chang Ma, Zhangyue Yin, Jianing Wang. Chengcheng Han, Renyu Zhu, Shuai Yuan, et al. A survey of neural code intelligence: Paradigms, advances and beyond. arXiv preprint arXiv:2403. 14734, 2024.
[20] Qiushi Sun. Zhoumianze Liu, Chang Ma, Zichen Ding, Fangzhi Xu, Zhangyue Yin, Haiteng Zhao, Zhenyu Wu, Kanzhi Cheng, Zhaoyang Liu, et al. Scienceboard: Evaluating multimodal autonomous agents in realistic scientific workflows. arXiv preprint wrXi:2555.118977 2025.
[21] Chi Zhang, Zhao Yang, Jiaxuan Liu, Yanda Li, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. Appagent: Multimodal agents as smartphone users. In Proceedings of the 2025 CHIl Conference on Human Factors in Computing Systems, pages 1-20, 2025.
[22] Chaoyun Zhang, Shilin He, Liqun Li, Si Qin, Yu Kang, Qingwei Lin, and Dongmei Zhang, Api agents vs. gui agents: Divergence and convergence. arXiv preprint arXiv:2503.1 /069, 2025.
[23] Wentong Chen, Junbo Cui, Jinyi Hu, Yujia Qin, Junjie Fang, Yue Zhao, Chongyi Wang, Jun Liu, Guirong Chen, Yupeng Huo, Guicourse: From general vision language models to versatile gui agents. arXiv preprint ar Xiv:24006 1131, 2024.
[24] Yuxiang Chai, Siyuan Huang, Yazhe Niu, Han Xiao, Liang Liu, Dingyu Zhang, Peng Gao, Shuai Ren. and Hongsheng Li. Amex: Android multi-annotatinn expo dataset for mobile gui agents. arXiv preprint arXiv:2407.17490 2024.
[25] Quanfeng Lu, Wenqi Shao, Zitao Liu, Fanqing Meng, Boxuan Li, Botong Chen, Siyuan Huang, Kaipeng Zhang, Yu Qiao, and Ping Luo. Gui odyssey: A comprchensive dataset for cross-app gui navigation on mobile devices. arXiv preprint arXiv:2406.004551 08451, 2024.
[26] Keen You, Haotian Zhang, Eldon Schoop, Floris Weers, Amanda Swearngin, Jeffrey Nichols, Yinfci Yang. and Zhe Gan. Ferret-ui: Grounded mobile ui understanding with multimodal Ilms. In European Conference on Computer Vision, pages 240-255. Springer, 2024.
[27] Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. Androidinthewild: A large-scale dataset for android device control. Advances in Neural Information Processing Systems, 36: 09708-99788, 2023.
[28] Jiwen Zhang, Jihao Wu, Yihua Teng, Minghui I Liao, Nuo Xu, Xiao Xiao, Zhongyu Wei, and Duyu Tang. Android in the zoo: Chain-of -at--------------eeeeeeeee thouuht for gui agents. arXiv preprint 4rXiv:2403.02713, 2024.
[29] Jiani Zheng, Lu Wang, Fangkai Yang, Chaoyun Zhang, Lingrui Mci, Wenjie Yin, Qingwei Lin, Dongmei Zhang, Saravan Rajmohan, and Qi Zhang. Vem: Environment free exploration for training gui agent with value environment model. arXiv preprint arXiv:2502. 18906, 2025.
[30] Qiushi Sun, Kanzhi Cheng. Zichen Ding, Chuanyang Jin. Yian Wang, Fangzhi Xu, Zhenyu Wu, Chengyou Jia, Liheng Chen, Zhoumianze Liu, al. Os-genesis: Automating gui agent trajectory construction via reverse task synthesis. arXiv preprint arXiv:2412. 19723, 2024.
[31] Kanzhi Cheng, Yantao Li, Fangzhi Xu, Jianbing Zhang. Hao Zhou, and Yang Liu. Vision-language models can self-improve reasoning via reflection. arXiv preprint arXiv:2411.0083 2024.
[32] Junlei Zhang, Zichen Ding, Chang Ma, Zijie Chen, Qiushi Sun, Zhenzhong Lan, and Junxian He. Breaking the data barricr-building gui agents through task generalization. arXiv preprint arXiv:2504. 10127, 2025.
[33] Hongliang He, Wenlin Yao, Kaixin Ma. Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. Webvoyager: Building an end-to-ent web agent with large multimodal models. arXiv preprint arXiv:2401.1391. 2024.
12

---

[34] Kanzhi Cheng. Qiushi Sun, Yougang Chu. Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. Seeclick: Harnessing gui grounding for advanced visual gui agents. arXiv preprint arXiv:2401.10955. 2024.
[35] OpenAL. Computer-using agent. Available at: http:://wwaaaaaaooooolocccooooom 2025.
[36]
Anthropic. Claude computer use. Available at: http://wwwwaaaanpcccccooo/////es 2024.
[37] Keqin Chen, Zhao Zhang, Weili Zeng. Richong Zhang. Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal Ilm's referential dialogue magic. arXiv preprint 230i::23061199955 2023.
[38] Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen. Xuejing Liu. Jialin Wang. Wenbin Ge. et al. Qwen2-vl: vl: Enhancing rision-languen model's perception of the world any resolution, arXiv preprint arXiv:2409. 2024.
[39] Kevin Qinghong Lin, Linjie Li, Difei Gao, Zhengyuan Yang. Shiwei Wu, Zechen Bai, Weixian Lei, Lijuan Wang, and Mike Zheng Shou, Showui: One aasion-anguage-actin action model for gui visual agent. arXiv preprint arXiv:241 17465 2024.
[40] Yuhao Yang. Yue Wang, Dongxu Li, Ziyang Luo, Bei Chen, Chao Huang, and Junnan Li, Aria-ui: Visual grounding for gui instructions. arXiv preprint arXiv:2412.16256, 2024.
[41] Pawel Pawlowski, Krystian Zawistowski, Wojeiech Lapacz, Marcin Skorupa, Adam Wiacek, Sehastien Postansque, and Jakub Hoscilowicz. Tinyelick: Single-turn agent for empowering gui automation. arXiv preprint arX:v:2410. 1871, 2024.
[42] Fei Tang, Yongliang Shen, Hang Zhang. Sigi Chen, Guiyang Hou, Wenqi Zhang. Wenqiao Zhang, Kaitao Song, Weiming Lu, and Yueting Zhuang. Think twice, click once: Enhancing gui grounding via fast and slow systems. arXiv preprint 2rXiv:2533.06777 2025.
[43] Shuai Bai, Keqia Chen, Xuejing Liu, Jialin Wang, Weabin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, et al. Qwen2. 5-v1 technical report. arXiv preprint arXiv:2502. 2025.
[44] Tianbao Xie, Jiaqi Deng, Xiaochuan Li, Junlin Yang. Haoyuan Wu, Jixuan Chen, Wenjing Hu. Xinyuan Wang, Yuhui Xu, Zekun Wang. Yiheng Xu, Junli Wang. Doyen Sahoo, Tao Yu, and Caiming Xiong, Scaling computer-use grounding via user interface decomposition and synthesis, 2025.
[45] Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, et al. Os-atlas: A foundation action model for generalist gui agents. arXiv preprint ar312:2410.23218. 2024.
[46] Hai-Ming Xu, Qi Chen, Lei Wang, and Lingqiao Liu. Attention-driven gui grounding: Leveraging pretrained multimodal large language models without fine-tuning. Proceedings of the AAAI Conference on Artificial Intelligence, pages 8851-8859, 2025.
[47] Karl Cohhe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacoh Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems, arXiv preprint arXiv:21 14168, 2021.
[48] Kaixin Li, Ziyang Meng, Hongzhan Lin, Ziyang Luo, Yuchen Tian, Jing Ma. Zhiyong Huang, and Tat-Seng Chua. Screenspot-pros Gui grounding for professional high-resolution computer use. arXiv preprint arXiv:2504.07981, 2025.
[49] OpenAI. Introducing gpt-4o. Available at: http:://ooemiicccmmmmnnmeeellllp
2024.
[50] Siyuan Hu, Mingyu Ouyang, Difei Gao, and Mike Zheng Shou. The dawn of gui agent: A preliminary case study with claude 3.5 computer use, 2024. URL RRRLLaappe://axxx..orrr/aaa/////
10323
[51] Google. Introducing gemini 2.0. Available at: htpp://lggggggecccoooo///////eeg cenii--a-upplaae-deemmbe------- 2024.
[52] Jianwei Yang, Reuben Tan, Qianhui Wu, Ruijie Zheng, Baolin Peng. Yongyuan Liang, Yu Gu. Mu Cai. Seonghyeon Ye, Joel Jang, et al. Magma: A foundation model for multimodal ai agents. arXiv preprint 1313000 2025.
[53] Wei Li, William E Bishop, Alice Li, Christopher Rawles, Folawiyo Campbell-Ajala, Divya Tyamagundlu. and Oriana Riva On the effects of data scale on ui control agents. Advances in Neural Information Processing Systems, 37:92130- 92154, 2024.
13

---

[54] Guilherme Penedo, Hynek Kydliček, Anton Lozhkov, Margaret Mitchell, Colin A Raffel, Leandro Von Werra, Thomas Wolf, al. The fineweb datasets: Decanting the web for the finest text data scale, Advances Neural Information Processing Systems, 37:30811-30849, 2024.
[S5] Chongyang Bai, Xiaoxue Zang, Ying Xu, Srinivas Sunkara, Abhinav Rastogi, Jindong Chen, et al. Uibert: Learning generic multimodal representations for ui understanding. arXiv preprint arXiv:2107. 2021.
[56] Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge. Mapping natural language instructions to mobile ui action sequences. arXiv preprint arXiv:2005. 03776, 2020.
[57] Yang Li, Gang Li, Lubeng He, Jingjie Zheng, Hong Li, and Zhiwei Guan. Widget captioning: Generating natural language description for mobile user interface elements. arXiv preprint arXi::2010.04299. 2020.
[58] Rogerio Bonatti, Dan Zhao, Francesco Bonacci, Dillon Dupont, Sara Abdali, Yinheng Li, Yadong Lu, Justin Wagle, Kazuhito Koishida, Arthur Bucker, Windows agent arena: Evaluating multi-modal os agents at scale. arXiv preprint arXiv:2409.0822 2024.
[59] Yadong Lu, Jianwei Yang, Yelong Shen, and Ahmed Awadallah. Omniparser for pure vision based gui agent. arXiv preprint ar 2408 00203, 2024.
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes].
Justification: The abstract and introduction clearly articulate our key contributions, including the proposal of coordinate- free grounding framework and its empirical advantages prior coordinate-buus based methods.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss some limitations in the analysis section and Appendix A
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
14

---

The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
Our theoretical assumptions are largely aligned with established consensus.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We provide the reproducible experimental setting in our experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
15

---

reproducible, they While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
We plan to release all data and code upon passing internal review and approval processes.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.c cc/
Polic)) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
/ //nips https/ccccuulcccguddcccocc /etggdddee/ooeduemieieane
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: All experimental details are provided in our paper.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail
that is necessary to appreciate the results and make sense of them.
16

---

The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Most of our experiments are deterministic, and we follow standard reporting practices commonly adopted in this domain.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
This information is provided in the experimental setup section the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http:////we.aappaccc////ta/ttttt
Answer: [Yes]
Justification:
Our research complies with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
17

---

If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader societal impacts of GUI agents in the paper.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deploymentt of technologics that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: Our research does have such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
18

---

Answer: [Yes]
Justification: Datasets and models used in this work comply with their respective licenses and terms of use.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwwppsppppeaaaaeeeee thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We have appropriately documented the new assets introduced in our research.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [No]
Justification:
We do not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
19

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [No]
Justification: Our paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non non-standard components.
Please refer to our LLM policy //tp:::////urraie cc. Cc//oonferencee/2255////// for what should or should not be described.
20

---

A Limitations
Our attention-base based action generation particularly well-suited for GUI environments, where interface elements such as icons and text lines often exhibit regular geometric patterns that align natually with patch-based detection. However, limitation stems from the base model employ: the backbone VLM (e.g., Qwen2-VL) adopts Naive Dynamic Resolution strategy with a fixed patch size of 28 x pixels. This poses challenges when dealing with very small interface elements (e.g.. icons smaller than 10 x 10 pixels), as such fine- eee-giaieed details may be insufficiently represented. Although this challenge not unique to our method, it can be pronounced in tasks demanding high-precision control, such as those encountered in professional software like CAD tools. While we introduce a simple mitigation strategy this work, fully addressing this limitation may require more substantial advancements in the future, such as improving the visual encoder's perceptual resolution or incorporating offset-based spatial refinement.
B Details on Multi-Patch Supervision
A key advantage of our approach lies in its ability to leverage dense, spatially structured supervision from bounding boxes. Unlike traditional coordinate- methods that depend on a single, potentially ambiguous click point, GUI-Actor enables supervision extent actionable elements more effectively. An example is illustrated Figure 2b
entire target regions, capturing the spatial
Specifically, we convert each ground-trun truth bounding box into binary mask over the W x H image patch grid. Given normalized bounding box b [left, top, right, bottom] [0, 1]*, we scale the normalized coordinates to the patch grid resolution:
(Lleft WJ, top HJ, [right W7, [bottom H))
This rounding ensures that all patches partially or fully covered by the bounding box are included in the supervision region. All patches whose indices fall within the resulting grid-aligned region are labeled as positives (i.e., mask value 1), while all others are assigned a value of This yields a binary vector y {0, )M aligned with the image patch sequence, where M W x H.
We define the action head loss as KL divergence between the predicted attention distribution {a)...., (..) and normalized target distribution p derived from the binary mask {0, 1)::
M
3/4
Pi
for i = 1....,,,. M; LEACTOR>
log
U/ +E
1=1
where is small constant for numerical stability.
C Visualization of Attention Maps from GUI-Actor
To better understand the grounding behavior of GUI-Actor, we provide additional examples visualiz- ing its attention maps overlaid on the original input images in Figure
The following Python code outlines the visualization process: starting from the raw attention scores, normalize and reshape them to match the image dimensions, apply colormap for clearer interpretation, and finally blend the attention heatmap with the original image. This produces an intuitive overlay that highlights regions the model attends when making decisions.
width, height = image. size
W, H attention_map_size # This may not equal width // patch_size due image reshaping, and you may need W // 2 and H // 2 due to the Naive Dynamic Resolution operation Quen2-VI.
scores = np. array (json_ [ paatt_ssoores" [0]) reshape W)
Normalize the attention weights for coherent visualization scores_norm (scores scores ())
iscores.max scores m)n
# Resize the attention map to match the image size. You can choose different resize strategies, such NEAREST and BILINEAR.
21

---

score_map Image. fromarray ( norr * 255) astype (np (np. uint8)).
resize ((width, height), resample=lmage BILINEAR)
12
Apply colormap
13
colormap - plt. plt.get_cmmppe ('tjee)
colored_score_ map colormap (up. array (rcay_ecc___map) 255.01 0) # returns
RGBA
15
colored_score__ng
scoorred_score.map :, :3] 255) astype (np.
uint8)
16
colored_overlay Image. fromarray ccolerrarcclloordd_ccrro.....
18
Blend with original image
19
blended Image blend (image, colored_overray alpha=0.3)
Listing 1: Python code for overlaying the attention map the image.
(a) ScotenSpot: "click the button create project"
(b) ScreenSpot-Pro: "restart from CD"
(c) ScreenSpot-Pro: "confirm sort"
(d) ScreenSpot-Proe "select the legend the plot"
Figure 5: Example visualizations from (a) ScreenSpot and (b) ScreenSpot-Par Each image shows the original interface with an overlaid attention map indicating regions of interest GUI-Actor. The attention maps largely overlap with the ground truth areas (red bounding boxes), demonstrating that the model can effectively capture the accurate UI elements.
D Training Datasets used for GUI-Actor
We compile our training data from several publicly available, high-quality GUI datasets. Summary statistics are provided Table Note that exclude samples from Wave-UI that overlap with downstream task test sets.
22

---

Table Overview of training datasets used for GUI-Actor.
Datasct
# of Elements
# of Screenshots
Platform
Uground Web-Hybrid 8
8M
775K
Web
GUI-Env
25
262K
70K
Web
GUT-Act
23
42K
13K
Weh
AndroidControl
47K
47K
Android
AMEX 24
1.2M
100K
Android
Wave-U
50K
7K
Hybrid
Total
9.6M
IM
E GUI Visual grounding Benchmarks
In these benchmarks, each screenshot is paired with a natural language instruction written by human annotators, typically describing the content or function of the target element, e-g., "the new project button" "switch to weekly view in the calendar." The agent is required identify the location of the corresponding element on the screen based on the given instruction. ScreenSpot the first benchmark specifically designed for GUI visual grounding, containing 1,272 single-step instructions paired with corresponding target elements. It covers wide range of GUI platforms, including mobile (Android and iOS). desktop (macOS and Windows), and web environments, and categorizes elements into text-based or icon elements. ScreenSpot-v2 is a corrected version of ScreenSpot that fixes annotation errors and ambiguous instructions, while keeping the total number of samples unchanged.
ScreenSpot-PPr is recently introduced challenging benchmark tailored for high-resolution profes- sional scenarios. It contains 1,581 tasks annotated by experts across 23 professional applications span- ning three operating systems. Compared to ScreenSpot, ScreenSpot-PPe features higher-resolution screenshots and larger domain gap from most grounding pretraining data, e.g., industrial software and mulli-window interfaces. We view its performance as a practical estimate of generalization for GUI visual grounding models.
F
More Detailed on Grounding Verifier
F.1
Data Construction
We construct the verifier training dataset from the OS-Atlas dataset [45] which spans desktop, mobile, and web domains. The original data consists triplets the form of (image, query, bounding box). where each image is paired with multiple queries and their corresponding bounding boxes. For each triplet, we generate a positive example by placing a marker at the center the bounding box, treating it as the correct grounding point for the given query. To create negative examples, we apply two strategies: (1) selecting center of different bounding box from the same image to simulate semantically plausible but incorrect location; (2) randomly sampling point outside the correct bounding box to simulate an unrelated action. As shown in Figure 6 each proposed point is marked on the image with hollow red circle. This process produces two labeled examples per query: one positive and one negative, formatted as:
{image with correct point, query, 'True'), {image with wrong point, query, 'False')
In total, we construct a balanced training set containing 730K examples, equally split between positive and negative cases. The overview of our dataset is listed in Table 8
F.2
Patch Selection
Given the top M candidate patches from the attention map, our goal is to select the one that best aligns with the user instruction in the image. A straightforward approach is to draw marker at the center of each patch and use a verifier to score how well each position satisfies the instruction æ. Specifically, for each candidate image I with marked point, predicting tokens 'True' or 'False': Parue Po. (True''Ze x) and Pralse = Pa., ('False' | x). We then define the score for each position the normalized probability of the "True' token: s(I,x) Pi,.
the verifier 9, compute the probability of
PoratP
23

---

Table 8: Overview of the dataset used to train our Grounding Verifier, including both positive and negative examples, Since multiple positive and negative samples can be generated from a single screenshot, the size of our dataset can exceed that the original dataset.
Dataset
# of Screenshots
Platform
SeeClick
34
147K
Web
Wib -4
123K
Web
Ulbert 155
17K
Mobile
AMEX 24
155K
Android
RICOSCA [66]
30K
Android
Widget Captioning
57
22K
Mobile
Linux-desktop
45
9K
Linux
Windows-desktop
220K
Windows
MacOS-desktop
45
7K
MacOS
Total
730K
Positive Sample
X Negative Sample
Stric O
333.6 O
Please observe the screenshot and exame whether the hollow red circle accurately placed on the intended position in the image: "Voice search'. Answer True False.
Red circle for verification
Ground Truth: True
Ground Truth: False
Figure 6: Illustration of positive and negative examples used train the grounding verifier.
key limitation this approach is that each patch (typically 28 x 28 pixels) may miss small icons located between two neighboring patches, leading inaccurate target localization. To address this, we introduce simple yet effective refinement. We first cluster 4-connected neighboring patches and compute weighted center based the verifier scores of the individual patches. This enables the generation of candidate points that lie between adjacent patches and improves localization accuracy without directly modifying the patch size of the base model.
In our implementation, we use up to M = 20 top-scoring patches, filtering out those with attention weights below 20% of the maximum attention weight. We then apply clustering to the neighboring patches, compute the weighted centers of these clusters, and add them to the set of M candidate positions. Each candidate position is scored using s(I, x), and select the one with the highest score. Given a candidate coordinate (æ,, y). we crop the image using a square region of size leoop x looo centered (xr, y). This is implemented
image crop (( max (0, x 1__rop//22 //2),
max (0,
1_crop //2)
min (image size [0] 1_crop//2), min (image size [1], y 1_crop//2)
)
24

---

Table 9: Verifier aggreegignn on ScreenSpot-Per
Creative
CAD
Scientific
Office
OS
Avg-Text
Avg-Icon
Avg
GUI-Aclor-7B Verifier GUI-Acter-7B Verifier self-aggreggation
40.1
39.0
37.2
47.2
63.5
41,8
61.1
16.7
44.2
39.5
40.2
38.3
44.9
63.9
44,9
61.3
17.4
44.5
Screendpot
Screen5pet
ScreerSpot Pro
nrVerfer
wherter EoC wil
vivetter wits
verterr
95-
70
Appels
0ut k7
Ag
021 10t
Figure 7: Comparison with AGUVIS using the verifier. AGUVIS inferences 21 times for verification. In contrast, GUI-Actor performs a single inference step, requiring only about 5% of the computation during inference.
We set lesop 1000 pixels for all tasks. To reduce the computational cost, if a candidate position achieves a high confidence score (e.g.. s(I, 2) 0.95), we immediately return that position without evaluating the remaining candidates. In our experiments, we set the threshold to 0.95 for tasks in ScreenSpot-Pro and 0.8 for ScreenSpot and ScreenSpot-v2. A lower threshold reduces reliance the verifier and instead trusts the grounding model's output, which is suitable when the grounding carefully assess
model is highly accurate. In contrast, higher threshold prompts the verifier each candidate, which is beneficial when the grounding model is less reliable, as in ScreenSpot-Pro
G Improving Grounding with Verifier
G.1 Enhancing Generation with Verifier Self-Aggregation
In this section, explore how to further leverage the verifier's capability through simple yet effective technique that we call Verifier Self- begr-Agggegation (VS). The idea is to crop the input image at multiple scales and compute the verifier scores for each crop, then average these scores to obtain more robust final prediction. This approach balances the trade-off between capturing detailed local information (with smaller crops) and maintaining a broader context (with larger crops). Specifically, we use two crop sizes in our experiments: lerop 1200 and lerop 1400 for ScreenSpot-pre The results, shown in Table demonstrate that verifier self-aggregation leads to improved performance on ScreenSpot-Prn Pro. Verifier self-aggregation provides a simple yet effective strategy to enhance verifier robustness, while also highlighting the need for more robust verifiers in the future.
G.2 Comparison with Baseline Using Verifier
To further validate the effectiveness of our grounding verifier, we integrate it into the AGUVIS bascline by sampling multiple candidate positions during inference and selecting the one with the highest verifier score. Specifically, AGUVIS generates one deterministic output at temperature 0.0 and samples 20 additional candidate points using a temperature of 0.7. While this approach explores a broader range of plausible locations, it incurs substantial computational overhead for these models.
In contrast, our GUI-Actor uses the attention map to propose multiple candidate points within single pass. This leads to a much more efficient process-requiring requiring only about 5% of the computation compared to AGUVIS- -whle achieving considerably higher grounding accuracy on ScreenSpot and ScreenSpot-v2, and significantly outperforming AGUVIS on the more challenging ScreenSpot-Pis benchmark. These results demonstrate both the efficiency of GUI-Actor and the general effectiveness of the verifier in improving action selection, even when applied to other models.
25

---

H Detailed Numbers of Ablation Study
Table 10: Ablation study on ScreenSpot-Pret ScreenSpot, and ScreenSpot-v2.
SereenSpot-Pro:
Dev
Creative
CAD
Scientific
Office
OS
Avg
GUI-Acto-7BB
38.8
40.2
29.5
44.5
56.5
36.2
40.7
Aguvis-7B 7B (bbox sup.)
12.4
17.0
1.5
18.1
21.7
11.7
13.8
Aguvis-7B (point sup.)
15.7
19.4
17.3
24.4
11.7
15.6
ScreenSpot:
Mobile-Text
Mobile-Icon
Desktop-Text
Desktop-Icon
Web-Text
Web-Icon
Avg
GUI-Actor77B
94.9
82.1
91.8
80.0
91.3
85.4
88.3
Aguvis-7B (bbox sup.)
92.3
73.4
76.4
92.6
74.8
84.4
Aguvis-7B (point sup.)
92.3
79.0
93.3
71.4
92.6
75.2
85.1
SereenSpot-r2:
Mobile-Text
Mobile-Icon
Desktop-Tes
Desktop-Icon
Web-Text
Web-Icon
Avg
GUI-Actor-7B
96.5
84.3
91.7
84.1
93.9
89.5
Aguvis-7B (bbox sup.)
92.3
73.4
92.3
76.4
92.6
74.8
84.4
Aguvis-7B (point sup.)
96.1
80.1
96.1
74.6
93.6
74.3
87.0
I Online Benchmark Evaluation on OS-World-W
To evaluate the real-world effectiveness our proposed AUt-Actor, we conduct experiments on OS- World [16] using GUI-Actor-7B for quick validation. OS- oS-World is live benchmark designed to test GUI agents in realistic desktop environments. We focus on curated subset 49 Windows-specific tasks, denoted OSWorld-Ve W, covering variety of multi-step office and multi-application scenarios. Each task is paired with handcrafted verification scripts to ensure reliable automatic evaluation.
Following the standard evaluation pipeline, adopt GPT-4o the planner. At each step, the planner observes the current GUI screenshot and user instruction and generates natural language plan. This plan is then grounded into concrete actions-eithe via coordinate-based or coordinate-free mechanisms- by the underlying grounding model, which plays a critical role in determining the agent's is success.
We compare OUI-Actor with several leading visual grounding baselines: Aguvis-7B 77 NAVI [8 and OmniAgent [59]. As shown in Table [1] GUI-Actor achieves the highest task success rate at 12.2%, outperforming OmniAgent and NAVT (both 10.2%) and substantially surpassing Aguvis- (4.0%). These results highlight the effectiveness and robustness of GUI-Actos in complex, world GUI environments. Despite having no exposure to OSWorld-V tasks during training, GUI-Actor generalizes well to unseen scenarios, delivering more accurate and reliable grounding performance than existing alternatives.
Table 11: Task Success Rate on the OSWorld-W subset (49 live Windows GUI tasks). GUI-Actor significantly outperforms existing grounding models in this real-world setting.
Grounding Model
Success Rate (%)
#Tasks Completed
7 (point sup.)
4,00
2/49
NAVI
10.2
5/49
OmniAgent
10.2
5/49
GUI-Actor- 7B (Ours)
12.2
6/49
Grounding Small UI Elements
J.1 Quantitative Analysis
key limitation of GUI-Actor lies in the dependency on the backbone laaguag model's fixed patch size, which makes it challenging to localize very small UI elements that may occupy less than one visual patch. This limitation can lead to errors when such elements fall entirely within a single patch, preventing accurate grounding.
To quantify this issue, we analyze performance on the ScreenSpot-PPee 48 dataset, where targets are grouped by bounding box area relative the patch size (n 14 x 14 pixels). As summarized in
26

---

Table 12 grounding accuracy for the smallest UI elements [0, n)) is significantly lower than for larger ones, highlighting the difficulty of tinegrained localization. Nevertheless, our sitentioo-weighhed refinement
computes the weighted center of top-K patches consistenty improves accuracy across all size ranges, including the most challenging small-target cases.
Table 12: Accuracy by target UI element size ScreenSpot- measured by area, 14 14. without Verifier.
Range
# of Examples
Top-1 Patch Center
Weighted Sum of Patch Cluster
[0, n)
23
4.35%
13.04% (1 200%)
n, 4n)
349
6.59%
11.46% (1 73.91%)
An, 9n)
367
14.99%
25.07% (1 67.27%)
(9n,)
842
42.04%
55.58% (1 32.20%)
All
1581
27.39%
38.14% (1 39.26%)
J.2 "Zoom-In" Strategy
To further alleviate this limitation, here we study "Zoom-In" strategy, in which GUI-Actor first performs a coarse prediction and then zooms in by factor 2x around the predicted region for a second round of grounding. This hierarchical approach allows finer spatial reasoning and substantially improves robustness in high-resolution scenarios. The results Table 13 demonstrate the effectiveness of this strategy, showing notable gains in overall performance on ScreenSpot-Pro
These analyses collectively demonstrate that small UI elements, though rare in typical desktop envi- ronments, pose meaningful challenges for patch-based grounding models. Our proposed refinement and zoom-in mechanisms effectively mitigate these errors, while future work may further explore multi-scale feature integration or affset-based localization to enhance fine-grained grounding.
Table 13: Effectiveness the "Zoom-In" strategy on ScreenSpot-Pro.
Model for Zoom-In Round
Model for Final Round
Zoom-In Ratio
ScreenSpot-Pro
Aguvis-7B
22.9
GUI-Actor-7B (w/o Verifier)
38.1
GUI-Actor-7B (w/o Verifier) GUI-Actor-7B (w/o Verifier)
Aguvis-7B
2x
43.0
GUI-Actor-7B (w/ Verifier)
2x
50.3
K Error Analysis on GUI Grounding Failures
To better understand the causes of grounding failures, we conducted detailed error analysis across 40 examples from ScreenSpot- -ro and ScreenSpot-v2, respectively. The breakdown in Table [4] categorizes common failure types and highlights the influence of dataset characteristies such as instruction ambiguity, non-unique valid regions, and the size of target UI elements.
Table 14: Error analysis on 40 examples from ScreenSpot-Prr and ScreenSpot-v2, respectively. ScreenSpot-Pro
Error Type
ScreenSpot-v2
The ground-truth valid action region not unique (model cap- tures another valid region not labeled as ground truth).
7.50%
7.50%
The instruction ambiguous (model predicts a reasonable but non-annotated action).
7.50%
20.00%
The ground-truth region is smaller than 14px x 14px.
5.00%
0.00%
The ground-truth region gains attention but fails to produce an accurate final position.
47.50%
27.50%
The ground-trute region gains little attention.
32.50%
45.00%
27

---

This analysis reveals several key observations. First, many grounding errors stem from ambiguous or under-specified instructions. where multiple reasonable actions exist but only one annotated as ground truth. Second, small UI elements (< 14 x 14 px) continue to present challenges for fine-grained localization, even with our weighted refinement strategies. Finally, a substantial portion of errors arises when the model attends to the correct region but fails to precisely generate the final click coordinate.
We believe these findings provide valuable guidance for future dataset design and model refinement, especially regarding annotation granularity and instruction disambiguation.
L Ablation on Multi-Patch Supervision and Threshold Sensitivity
The multi-patch supervision during training is determined objectively. As described in Section 3 and illustrated in Figure 2b] our multi-patch supervision strategy labels all image patches that are partially or fully covered by the ground-truth bounding box as positive (label 1), and all other patches as negative (label = 0). This binary supervision is applied purely based on spatial overlap, without any threshold tuning.
Regarding the threshold used at inference time aggregate multiple clustered patches into a final prediction, we conducted an ablation study by varying the value from 0.1 to 1.0. As shown in Table performance remains stable within a moderate range. However, extreme values either too permissive (0.1) or too restrictive (1.0, equivalent to using only the top-1 patch) lead to noticeable drops in accuracy. These results suggest that while the threshold choice does have an effect on performance, the method remains robust across broad range of reasonable settings. This demonstrates that the attention-based aggregation is resilient to moderate variations patch selection criteria.
Table 15: Ablation on the threshold used to produce the final click point from the attention map.
ScreenSpot-Par ScreenSpot-v2
0.1
91.4
39.0
0.2
91.7
42.7
0.3 (Reported in Table 4
92.1
44.6
0.35
92.2
45.3
0.4
92.2
46.4
0.45
92.0
45.9
0.5
91.7
46.0
0.6
90.6
45.5
0.7
90.3
43.5
0.8
89.3
41.4
0.9
87.1
39.7
1.0 (Top-1 Patch)
85.1
36.9
28


Reviews:
[{"review_id": "uUDvNmSyhl", "reviewer": "Reviewer_LxDz", "summary": "This paper introduces GUI-Actor, a new framework for Vision-Language Model (VLM) based GUI agents that overcomes the limitations of traditional text-based coordinate generation. The authors argue that relying on coordinates creates issues with spatial-semantic alignment, action target ambiguity, and mismatched granularity between vision and screen. GUI-Actor offers a coordinate-free solution. It uses a special <ACTOR> token and a unique \"action head\" to directly generate an attention map over a screenshot, pinpointing the most relevant area for an action. To handle the ambiguity of target locations (e.g., any part of a button), the model is trained using a multi-patch supervision method where any visual patch overlapping with the correct action area is considered a valid target. A lightweight \"grounding verifier\" then refines the selection from high-attention regions. Experiments show GUI-Actor significantly surpasses existing methods, especially in generalizing to new, high-resolution applications.", "questions": "1. The paper identifies the limitation regarding small UI elements due to fixed patch sizes. Could you provide a more quantitative analysis of this issue? For instance, what proportion of errors on ScreenSpot-Pro are attributable to this, and how effective is the described clustering refinement strategy at mitigating them? Have you considered architectural changes to address this more directly, such as incorporating multi-scale features from the vision encoder into the action head?\n\n2. The verifier is shown to be general enough to boost the performance of the AGUVIS baseline. This is a strong result. Given that your coordinate-free method excels at general localization and coordinate-based methods can offer high precision, have you considered a hybrid model? For example, one that uses the attention head to identify the correct UI element and a secondary mechanism to predict a precise click coordinate within that element's bounding box.\n\n3. The training data for the verifier is constructed from the OS-Atlas dataset, while the main model uses a different collection of datasets. Could you discuss the potential impact of this domain shift? Would there be a benefit to training the verifier on the exact same data distribution that the main model is trained on?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\n- The work tackles a well-defined and critical bottleneck in the development of capable GUI agents. Moving away from brittle coordinate generation is a crucial step for improving the robustness and generalization of these systems. \n\n- The paper's methodology and evaluation are of excellent quality. The proposed architecture is well-motivated and technically sound. The experimental setup is comprehensive, featuring multiple standard benchmarks, rigorous comparisons against relevant baselines.\n\n- The paper is exceptionally well-written, structured, and easy to follow. \n\n- The core idea of reformulating GUI grounding as a coordinate-free, attention-based localization task is novel and insightful within this specific domain.\n\nWeaknesses:\n\n- The primary weakness, which the authors acknowledge, is the dependency on the backbone VLM's fixed patch size. This makes it difficult to ground actions on very small UI elements that may be smaller than a single visual patch.", "was_revised": true, "final_justification": "Thank you to the authors for the detailed rebuttal. I appreciate the clear explanations and the additional experimental results. All of my previous concerns have been adequately addressed.", "rebuttal": "Thank you for reviewing our work and providing valuable feedback. We have carefully addressed your concerns below. Please let us know if you have any further questions.\n\n**W:** *The primary weakness, which the authors acknowledge, is the dependency on the backbone VLM's fixed patch size. This makes it difficult to ground actions on very small UI elements that may be smaller than a single visual patch.*\n\nIn our proposed method, the attention-weighted center of top-K patches can help mitigate this issue and localize small UI elements (**_<14×14 pixels_**) that overlap with 3-4 patches. However, current design of GUI-Actor may still struggle when such elements **_fall entirely within a single patch_**. Fortunately, these extreme cases are **_relatively rare_** in typical computer use scenarios, as shown in **_Table A_**.\n\nTo mitigate this limitation, we explored a simple yet effective **_\"Zoom-In\" strategy_**, where GUI-Actor makes an initial coarse prediction, then zooms in 2× around the predicted location and performs a second round of grounding. **_Table B_** shows this significantly boosts performance on ScreenSpot-Pro, demonstrating its effectiveness in mitigating grounding errors in high-resolution settings with small UI elements.\n\n**_Table A._** _Frequency of small UI elements (longest edge < 14 px) fully contained within a single patch._\n| Dataset         | # of Examples | # of with Small Elements | # of Small Element In One Patch |\n|-----------------|----------------------|--------------| --------------|\n| ScreenSpot-v2    | 1272              | 0                    | 0            |\n| ScreenSpot-pro   | 1581              | 52                   | 10            |\n\n**_Table B._** _Effectiveness of the \"Zoom-In\" strategy on ScreenSpot-Pro._\n| Model for Zoom-In Round              | Model for Final Round              | Zoom-in Ratio | Screenspot-Pro |\n|-------------------------------------|------------------------------------|----------------|----------------|\n| /                                   | Aguvis-7B                          | /              | 22.9           |\n| /                                   | GUI-Actor-7B                       | /              | 38.1           |\n| GUI-Actor-7B (w/o Verifier)         | Aguvis-7B                          | 2x             | 43.0           |\n| GUI-Actor-7B (w/o Verifier)         | GUI-Actor-7B (w/o Verifier)        | 2x             | **50.28**      |\n\n\n**Q1:** *The paper identifies the limitation regarding small UI elements due to fixed patch sizes. Could you provide a more quantitative analysis of this issue? For instance, what proportion of errors on ScreenSpot-Pro are attributable to this, and how effective is the described clustering refinement strategy at mitigating them? Have you considered architectural changes to address this more directly, such as incorporating multi-scale features from the vision encoder into the action head?*\n\nThank you for the constructive suggestion. We provide a more detailed quantitative analysis in **_Table C_**, where ScreenSpot-Pro targets are grouped by bounding box area relative to the patch area (**_n = 14×14_**). As shown, accuracy for the smallest elements $[0,n)$ is significantly lower than for larger ones, underscoring the challenge of grounding small UI components. Notably, Table C also shows that our **_proposed refinement strategy  (attention-weighted center) consistently improves accuracy across all size ranges_**, including the most difficult small-target cases.\n\nWe appreciate the suggestion regarding architectural improvements. Integrating multi-scale features,  coarse-to-fine attention, or learning an offset is a promising direction for improving fine-grained localization. We leave this for future work.\n\n**_Table C._** _Accuracy by target UI element size on ScreenSpot-Pro, measured by area, n=14*14, without Verifier._\n| Range      | # of Examples | Accuracy of Top-1 Patch Center | Accuracy of Weighted Sum of Patch Cluster|\n|------------|----------------|-------------------------------|------------------------------------------|\n| [0, n)     | 23             | 4.35%                         | 13.04% (↑ 200%)                          |\n| [n, 4n)    | 349            | 6.59%                         | 11.46% (↑ 73.91%)                        |\n| [4n, 9n)   | 367            | 14.99%                        | 25.07% (↑ 67.27%)                        |\n| [9n, )     | 842            | 42.04%                        | 55.58% (↑ 32.20%)                        |\n| All        | 1581           | 27.39%                        | 38.14% (↑ 39.26%)                        |\n\n\n**Q2:** *The verifier is shown to be general enough to boost the performance of the AGUVIS baseline. This is a strong result. Given that your coordinate-free method excels at general localization and coordinate-based methods can offer high precision, have you considered a hybrid model? For example, one that uses the attention head to identify the correct UI element and a secondary mechanism to predict a precise click coordinate within that element's bounding box.*\n\nThank you for the insightful question. **_Table D_** presents the results for a two-stage strategy in which the original screenshot is **_first cropped to half its size_**  (H x W -> H/2 x W/2), **_centering the first-round prediction_**. Then the cropped image is used as input for a **_second-round prediction_**. This setup can be viewed as a **_hybrid model_** when different models are used across the two stages.\n\nAs shown in **_Table D_**, this two-stage approach consistently improves performance. It demonstrates that **_GUI-Actor not only offers more accurate coarse-grained localization but is also capable of producing finer-grained click coordinates_**. While **_small UI elements_** pose challenges for GUI-Actor in predicting precise coordinates, they **_similarly affect coordinate generation-based methods_**. In cases where UI elements are very small, coordinate-based models may struggle to capture and learn the correct semantics without explicit spatial alignment. In contrast, GUI-Actor benefits from such explicit alignment through its action head, which helps guide more precise localization.\n\n**_Table D_.** _Effectiveness of the hybrid strategy on ScreenSpot-Pro._\n| Model for First Stage (Crop)     | Model For Second Stage              | Crop Ratio | Screenspot-Pro |\n|----------------------------------|-------------------------------------|------------|----------------|\n| Aguvis-7B                        |                                     | /          | 22.9           |\n| GUI-Actor-7B                     |                                     | /          | 38.1           |\n| Aguvis-7B                        | Aguvis-7B                           | 1/2        | 41.6          |\n| GUI-Actor-7B (w/o Verifier)      | Aguvis-7B                           | 1/2        | 44.4          |\n| Aguvis-7B                        | GUI-Actor-7B (w/o Verifier)         | 1/2        | 46.1          |\n| GUI-Actor-7B (w/o Verifier)      | GUI-Actor-7B (w/o Verifier)         | 1/2        | **46.3**      |\n\n\n**Q3**: *The training data for the verifier is constructed from the OS-Atlas dataset, while the main model uses a different collection of datasets. Could you discuss the potential impact of this domain shift? Would there be a benefit to training the verifier on the exact same data distribution that the main model is trained on?*\n\nThank you for the question.  **_Our Verifier was trained during an earlier stage_** of the project using the OS-Atlas dataset, before the UGround bounding-box dataset was released on May 1, 2025 (available on Hugging Face). Generating a full UGround-based verifier dataset requires extra time, so we decided not to create a new verifier dataset and train it again. Thanks to our diverse negative-sampling strategy and the inherent variety in OS-Atlas, **_the Verifier generalizes very well_**—even when paired with a grounding model trained on different datasets. While one could certainly retrain the Verifier on the exact same distribution as the main model, our experiments suggest that the existing setup already delivers strong performance with minimal domain-shift impact."}, {"review_id": "yQ3Ni0WDEb", "reviewer": "Reviewer_FWfD", "summary": "GUI-Actor introduces an attention-based action head that uses a dedicated <ACTOR> token. This token directly focuses on relevant visual patch regions, removing the need for coordinate-based localization.\n\nThe method also handles spatial ambiguity by using multi-patch supervision. It labels all patches inside ground-truth bounding boxes as positives, ensuring better accuracy in GUI interactions.\n\nUnlike traditional methods, GUI-Actor generates multiple candidates in a single forward pass. Older approaches require multiple inference runs with sampling.", "questions": "The attention-based action head design really needs more explanation. How does the self-attention layer actually pull together information from different patches? The paper talks about \"GUI-aware contextual information\" but doesn't say what makes it special for GUIs. Do they use custom positional encodings? Is there some way to model how GUI elements relate to each other spatially?\n\nThe multi-patch supervision approach could use some ablation studies too. When they decide which patches count as \"positive\" based on overlap, how much does performance change if you tweak those thresholds? It seems like this could make a big difference but they don't test it.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nGUI-Actor ditches coordinate generation entirely. Instead, it uses an attention-based action head with a special <ACTOR> token. This creates a direct connection between what the model sees and what it wants to do. No need for text-based coordinate prediction.\n\nThe framework can generate multiple candidate regions in one go. It does this through attention maps from the transformer backbone. This beats baseline methods that have to run inference multiple times just to get candidates.\n\nGUI-Actor can also handles new scenarios. It performs great on out-of-distribution benchmarks like ScreenSpot-Pro. The attention mechanism adapts to different screen resolutions automatically. It deals with varying layouts much better than coordinate-based methods that struggle with resolution changes.\n\nWeaknesses:\n\nThe paper doesn't really explain how the attention-based action head works. Key parts like the \"GUI-aware\" self-attention are mentioned but not detailed. We don't know how many attention heads are used or how they're configured. Same goes for the MLP projection designs. The exact architecture of the action head remains unclear. This makes it hard to understand or reproduce.\n\nThe experimental setup has some issues. The verifier training uses simple binary classification with only correct/incorrect labels. It relies on fake negative examples generated by random coordinate shifts. But real problems are usually messier than that. Users might click slightly off-target or have different interaction patterns.\n\nThere's also a lack of thorough error analysis. The authors don't dig into when and why things go wrong. What types of GUI elements cause failures? The verifier training uses those oversimplified artificial negatives.", "was_revised": true, "final_justification": "I raised concerns about the attention-based action head design and multi-patch supervision approach. The authors provided detailed responses. They explained the architecture specifics including 8 attention heads and two-layer MLPs. They clarified how self-attention aggregates spatial relationships.They described their negative sampling strategy that includes semantically meaningful UI elements. They also provided an error analysis table examining failure modes. These responses reasonably address my questions. I would like to increase my scores for this work.", "rebuttal": "Thank you for reviewing our work and providing valuable feedback. We have carefully addressed your concerns below. Please let us know if you have any further questions.\n\nWe address W1 and Q1 together, as they both raise points about the architecture and design of the attention-based action head.\n\n**W1:** *The paper doesn't really explain how the attention-based action head works. Key parts like the \"GUI-aware\" self-attention are mentioned but not detailed. We don't know how many attention heads are used or how they're configured. Same goes for the MLP projection designs. The exact architecture of the action head remains unclear. This makes it hard to understand or reproduce.*\n\n**Q1:** *The attention-based action head design really needs more explanation. How does the self-attention layer actually pull together information from different patches? The paper talks about \"GUI-aware contextual information\" but doesn't say what makes it special for GUIs. Do they use custom positional encodings? Is there some way to model how GUI elements relate to each other spatially?*\n\nThanks for pointing out the confusion. As shown in Figure 2, Eq (2), Eq (3) and described in Line 150-152, the action head comprises three main components: **_(1) a multi-head self-attention layer, followed by (2) an MLP to extract vision patch features, and (3) a [ACTOR] token-specific MLP to extract semantic features_**.\n\n\n\nThe **_self-attention layer_** operates over image patches, where **_each patch representation is computed as an attention-weighted sum of all other patch representations_**. This mechanism allows the model to **_explicitly aggregate spatial relationships_** among UI elements and layout structures beyond what the original ViT encoder captures, by leveraging multi-patch supervision. \n\nRegarding positional encoding, we do **_not introduce additional positional embeddings in this self-attention layer_**, as the input features from the ViT backbone already incorporate positional information.\n\nFor the **_configuration details_**: The number of attention heads in the self-attention layer is set to 8; Both MLP components are two-layer feedforward networks with a GELU activation in between. As stated in Line 213–214, we use the same dimensionality as the backbone VLM for all dimension configuration within the action head.\n\nWe will revise the manuscripts and open source our code to ensure clarity and reproducibility.\n\n\n**W2:** *The experimental setup has some issues. The verifier training uses simple binary classification with only correct/incorrect labels. It relies on fake negative examples generated by random coordinate shifts. But real problems are usually messier than that. Users might click slightly off-target or have different interaction patterns.*\n\nThank you for the feedback. We believe the comment may stem from a partial interpretation of our setup, and we would like to clarify the rationale and design behind our verifier training. **(1) Use of Constructed Negatives:**  While the negative examples are indeed constructed rather than collected from real user interactions, **_this contrastive strategy is a widely used and effective practice in many learning paradigms_** such as contrastive learning and self-supervised learning. In our case, these constructed negatives help the verifier distinguish correct clicks from plausible but incorrect alternatives. **(2) Diversity of Negative Samples:** Our negative sampling is not limited to simple random coordinate shifts. As detailed in **Line 185-187**, we deliberately introduced varying levels of difficulty_** to better reflect real-world behavior. In addition to **_random offsets (which naturally include slightly off-target cases)_**, we also **_select negative points located within the bounding boxes of other semantically meaningful UI elements_**. This design simulates realistic interaction ambiguities, such as users clicking on relevant but incorrect elements.\n\n\n**W3:** *There's also a lack of thorough error analysis. The authors don't dig into when and why things go wrong. What types of GUI elements cause failures? The verifier training uses those oversimplified artificial negatives.*\n\nThank you for the valuable suggestion. We conducted a **_detailed error analysis_** across 40 examples from SS-Pro and SS-v2, respectively. **_Table E_** shows the details. We hope this breakdown offers insight into when and why failures occur and shows how dataset properties (e.g., ambiguity, element size) affect performance. We will include this analysis in the next version to guide future improvement.\n\n**_Table E._** _Error Analysis on 40 examples from SS-pro and SS-v2, respectively._\n| Error Type                                                                                             | ScreenSpot-Pro  | ScreenSpot-v2   |\n|--------------------------------------------------------------------------------------------------------|---------|---------|\n| The ground-truth valid action region is not unique (model captures the one not labeled as ground-truth). | 7.50%   | 7.50%   |\n| The instruction is ambiguous (models take reasonable but not accurate actions given the non-specified instruction). | 7.50%   | 20.00%  |\n| The ground-truth region is smaller than 14px * 14px.                                                   | 5.00%   | 0.00%   |\n| The ground-truth region gains some attention, but fails to produce a final accurate position.          | 47.50%  | 27.50%  |\n| The ground-truth region gains little, even no attention.                                               | 32.50%  | 45.00%  |\n\nPlease refer to W2 for the concern on verifier training with artificial negatives.\n\n\n**Q2:** *The multi-patch supervision approach could use some ablation studies too. When they decide which patches count as \"positive\" based on overlap, how much does performance change if you tweak those thresholds? It seems like this could make a big difference but they don't test it.*\n\nThe **_multi-patch supervision during training is determined objectively_**. As described in Line163-164 and illustrated in Figure 2b, our multi-patch supervision strategy labels all image patches that are partially or fully covered by the ground-truth bounding box as positive (label = 1), and all other patches as negative (label = 0). This **_binary supervision is applied objectively based on spatial overlap, without threshold tuning_**. While it is possible to apply soft labels to boundary patches (e.g., based on intersection ratios), we conjecture the impact would be limited, as only a narrow band of edge patches would be affected. We will consider this in future work.\n \nRegarding the **_threshold used at inference time_** to aggregate multiple clustered patches into a final prediction, we conducted an ablation by varying the value from 0.1 to 1.0. As shown in **_Table F_**, performance remains stable within a moderate range (0.3 to 0.6). However, extreme values, _i.e.,_ either too permissive (0.1) or too restrictive (1.0, equivalent to using only the top-1 patch), lead to drops in accuracy. These results indicate that while the threshold choice does impact performance, **_the method is robust across a reasonable range_**. We will include this ablation in the final version to clarify this design choice.\n \n**_Table F._** _Ablation study on the attention threshold for final prediction. Numbers from 0.3 is reported in the paper._\n| Dataset         | 0.1  | 0.2  | 0.3 | 0.4  | 0.5  | 0.6  | 0.7  | 0.8  | 0.9  | 1.0 (Top-1 Patch Center) |\n|----------------|------|-----|------|------|------|------|------|------|------|--------------|\n| ScreenSpot-v2  | 87.6 | 88.8 | 88.9 | 88.4 | 88.1 | 86.7 | 86.2 | 84.1 | 83.2 | 81.5         |\n| ScreenSpot-Pro | 34.5 | 36.4 | 38.1 | 37.8 | 38.6 | 38.0 | 35.7 | 32.7 | 30.0 | 27.4         |"}, {"review_id": "CGJFzuPjbl", "reviewer": "Reviewer_8yA9", "summary": "This paper mainly focuses on the GUI-grounding problem in the GUI agent topic. The authors propose an interesting attention-based action head for the candidate action elements prediction and a grounding verifier to select the most plausible action region as the final grounding prediction. Several experimental results have been presented to show the effectiveness of the proposed method.", "questions": "Please refer to the weaknesses mentioned above.", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The presentation and writing of the paper are clear and well organized.\n2. Exploring attention-based grounding—rather than directly predicting coordinates via text—is both interesting and conceptually sound.\n3. Introducing an < ACTOR > token as a grounding anchor for computing attention with image patch embeddings is a good design choice.\n\nWeaknesses:\n1. The action head requires the model to output the < ACTOR > token. For GUI-Actor-7B with LiteTrain, the VLM parameters remain frozen—how is the model reliably induced to produce this new < ACTOR > token, and was this token newly added to the vocabulary?\n\n2. Tables 4 and 5 show that GUI-Actor-7B (no Verifier) outperforms GUI-Actor-7B w/ LiteTrain (no Verifier) by a substantial margin. Could you analyze which component drives this gain? Is it better < ACTOR > formatting, more accurate action-head predictions, or something else?\n\n3. Adding the Verifier (Tables 4 and 5) yields a 3–9% boost in performance. Why does the action head’s direct attention map exhibit such a significant bias? Although the action model (7B) is much larger than the Verifier (2B), the action head’s direct attention maps still show substantial bias. One would expect the stronger model to leverage its internal reasoning capacity without external checks—could this indicate that the action head was not trained well enough to fully elicit the VLM’s inherent reasoning knowledge?\n\n4. In the setup on L215, the Verifier may be invoked up to K = 20 times, introducing extra runtime and compute cost. Please include an inference-cost comparison against baseline models and report the average number of Verifier calls per sample.\n\nMinor Questions:\n1. Is the self-attention applied to image-patch embeddings in the attention-based action head causal or non-causal? Could this choice affect performance?\n\n2. During GUI-Actor-7B’s supervised fine-tuning, which VLM parameters were actually trained?", "was_revised": true, "final_justification": "The authors have addressed my concerns, and I will maintain my positive rating.", "rebuttal": "Thank you for reviewing our work and providing valuable feedback. We have carefully addressed your concerns below. Please let us know if you have any further questions.\n\n**W1:** _The action head requires the model to output the < ACTOR > token. For GUI-Actor-7B with LiteTrain, the VLM parameters remain frozen—how is the model reliably induced to produce this new < ACTOR > token, and was this token newly added to the vocabulary?_\n\n Yes, the `<ACTOR>` token is newly added to the vocabulary. As noted, in the GUI-Actor-7B w/ LiteTrain setting, all original VLM parameters are frozen; only the `<ACTOR>` token embedding and the parameters introduced in the Action Head are trainable.\n\nDuring inference, if the model is **_only performing the GUI grounding_** task, it is **_not necessary for it to generate the `<ACTOR>` token_**. Instead, in the **_LiteTrain_** setting, we **_directly include the `<ACTOR>`token in the input prompt_**, e.g., \"pyautogui.click(`<ACTOR>`)\", and **_extract its final-layer hidden representation_**. This representation is then passed to the action head for grounding.\n\n\n**W2:** _Tables 4 and 5 show that GUI-Actor-7B (no Verifier) outperforms GUI-Actor-7B w/ LiteTrain (no Verifier) by a substantial margin. Could you analyze which component drives this gain? Is it better `<ACTOR>` formatting, more accurate action-head predictions, or something else?_\n\n**_The `<ACTOR>` token formatting is the same in both settings_**, so the performance gap is not due to differences in prompt structure.\nWe believe the **_performance gain_** of GUI-Actor-7B (without Verifier) over GUI-Actor-7B with LiteTrain (also without Verifier) **_mainly comes from the difference in how many parameters of the model are fine-tuned_**. In the full fine-tuning setting, the entire model—including the ViT backbone—is updated, allowing for deeper task-specific adaptation. In contrast, the LiteTrain variant freezes the backbone and only updates the `<ACTOR>` token embedding and the Action Head, which limits its ability to fully align multimodal representations for the grounding task. In summary, while **_LiteTrain offers a more efficient training approach, full fine-tuning leads to stronger task adaptation and better overall performance_**.\n\n\n**W3:** _Adding the Verifier (Tables 4 and 5) yields a 3–9% boost in performance. Why does the action head’s direct attention map exhibit such a significant bias? Although the action model (7B) is much larger than the Verifier (2B), the action head’s direct attention maps still show substantial bias. One would expect the stronger model to leverage its internal reasoning capacity without external checks—could this indicate that the action head was not trained well enough to fully elicit the VLM’s inherent reasoning knowledge?_\n\nThanks for the insightful comments. While it's true that adding a smaller Verifier improves performance, we believe it's **_not sufficient to conclude that the action head was inadequately trained or unable to leverage the VLM’s reasoning capacity_**.\n\nThe Verifier is trained as a binary classifier to assess whether a given candidate position is plausible, whereas the grounding model learns to predict a full attention map over all input image patches. These two tasks, i.e., evaluation versus direct prediction, differ fundamentally in complexity. As noted in Line 176, **_selecting the correct option from a list (as the Verifier does) is generally easier than directly generating a precise location from an unconstrained space_**. \n\nImportantly, the performance boost from the Verifier suggests that the grounding model often identifies the correct region but may not consistently assign it the highest score, especially in cases involving ambiguity or multiple plausible targets. In these situations, the Verifier helps resolve uncertainty by selecting the most semantically appropriate candidate. **_Rather than compensating for a weak grounding model, the Verifier serves as a complementary module that enhances decision-making in challenging scenarios_**. For instance, we observe greater improvement on the more difficult benchmark, ScreenSpot-Pro.\n\n\n**W4:** _In the setup on L215, the Verifier may be invoked up to K = 20 times, introducing extra runtime and compute cost. Please include an inference-cost comparison against baseline models and report the average number of Verifier calls per sample._\n\nThank you for your constructive suggestion. We acknowledge that the verifier introduces additional computational cost. However, we believe this cost is justified, as test-time scaling with a verifier provides valuable insights into the design of more powerful GUI models. Compared to prior grounding models such as UITARS and AGUVIS, **_GUI-Actor offers a key advantage: it can generate multiple candidate regions in a single forward pass, significantly reducing computation at this stage_**.\n\nTo mitigate the cost introduced by the verifier, we **_have introduced an early stopping mechanism_** in our work—terminating verification once the confidence exceeds 0.95. This reduces the number of verifier calls substantially, as shown in **_Table G_**. On average, only 3.3 verifier calls are needed per sample on ScreenSpot-v2. Thanks to the efficient vLLM setup, this adds just 1.3 seconds per sample on average—an acceptable overhead with further room for parallelization and optimization. We also compared this approach with a **fully parallel solution** that scores all candidate areas simultaneously without early stopping. While it achieves similar latency, it incurs higher computational cost.\n\n\n**_Table G._** _Inference-time comparison of different grounding models with and without the Verifier._\n| Methods                                      | ScreenSpot-v2 | Average Verifier Calls (N) per sample | Time Consuming per sample in ScreenSpot-v2 | Estimated computational cost    |\n|---------------------------------------------|----------------|----------------------------------------|---------------------------------------------|----------------------------------|\n| baselines                                   | 86.0           | /                                     | 1.31s                                       | 1 * 7B                           |\n| Baseline w/ Verifier (Seqential Solution)                        | 88.0           | 9.9                                    | 21.22s                                      | 20 * 7B + N * 2B                 |\n| GUI-Actor w/o Verifier                      | 88.9           | /                                     | 1.42s                                       | 1 * 7B                           |\n| GUI-Actor w/ Verifier (Sequential Solution) | 89.5           | 3.3                                    | 2.72s                                       | 1 * 7B + N * 2B                  |\n| GUI-Actor w/ Verifier (Parallel Solution) | 89.7           | 20                                     | 3.03s                                      | 1 * 7B + 20 * 2B                 |\n\n\n**Q1:** _Is the self-attention applied to image-patch embeddings in the attention-based action head causal or non-causal? Could this choice affect performance?_\n\nThe self-attention mechanism in the action head is **_non-causal_**. This design choice is intentional, as the goal is to enhance the model's ability to capture spatial relationships among image-patch embeddings. A non-causal setup **_allows each patch to attend to all others_**, which aligns with the intuition that **_spatial reasoning in GUI layouts requires global context rather than sequential or directional constraints_**. We believe this non-causal design is better suited for aggregating layout and UI structure effectively.\n\n\n**Q2:** _During GUI-Actor-7B’s supervised fine-tuning, which VLM parameters were actually trained?_\n\nDuring supervised fine-tuning of GUI-Actor-7B, we adopt a **_two-stage training_** strategy. First, we **_warm up the model by training only the newly introduced token embeddings and the parameters in the action head_**, while keeping the backbone VLM frozen. After this warm-up phase, we **_fine-tune the entire model_**, including the backbone VLM parameters. We will clarify this more explicitly in Line 221–224 in the revised manuscript."}, {"review_id": "MHnjfQwIgQ", "reviewer": "Reviewer_YmmE", "summary": "This paper introduces GUI-Actor, a vision-language model for GUI grounding tasks that replaces text-based coordinate prediction with an attention-driven, “coordinate-free” grounding mechanism. The authors align a special <ACTOR> token directly with relevant visual patches through an action-head (MLPs), then apply a lightweight verifier to filter the top-K attended regions to get the final action point. Trained on ∼1 M screenshots, GUI-Actor achieves good element-accuracy on ScreenSpot, ScreenSpot-v2, and the challenging ScreenSpot-Pro benchmarks even if the only trained model parameters are the 100M-parameter action-head.", "questions": "1. The proposed method sets a confidence threshold, but it is not clear how the threshold is defined, and what if no candidate is above the threshold. \n2. The author demonstrated fine-tuning the Qwen-2-VL model only. Is it possible to apply the proposed coordinate-free training to a existing GUI model such as UI-TARS-7B?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: The work is methodologically novel: it drops brittle numeric-coordinate generation in favor of an explicit spatial–semantic attention map, naturally allowing dense multi-patch supervision and diverse candidate proposals. The narrative is clear, well-situated in recent GUI-agent literature, and backed by thorough ablations.\n\nWeaknesses: \n1. Comparing GUI-Actor-7B + verifier to baselines that predict actions in a single pass overstates its advantage, because the verifier injects an extra round of visual evidence and decision logic that competing methods never see. A like-for-like comparison should disable the verifier or add an equivalent post-filter to the baselines.\n2. Running the GUI-Actor-7B + verifier increases GPU memory and adds an extra forward pass per click, introducing latency that is hard to hide in real-time GUI agents. \n3. The proposed method makes the “coordinate-free” pointer to land only at patch centers. The 28 × 28 patch grid becomes too large and coarse when the input resolution is small, making fine-grained targets (e.g., tiny icons or narrow scroll bars) unreachable.", "was_revised": true, "final_justification": "I maintain my positive score and recommend acceptance. The authors addressed all major concerns thoroughly:\n\n- Comparison fairness: Ablations show GUI-Actor outperforms baselines even without the verifier.\n- Efficiency: The verifier is lightweight, with early stopping and vLLM inference reducing latency. The design is reasonable for deployment.\n- Small target precision: The proposed “Zoom-In” strategy effectively handles fine-grained elements.\n- Generality: GUI-Actor works well with other VLMs like UI-TARS, showing broad applicability.\n- Conceptual clarity: Follow-up clarified connections to coordinate-free grounding in prior work.", "rebuttal": "Thank you for reviewing our work and providing valuable feedback. We have carefully addressed your concerns below. Please let us know if you have any further questions.\n\n**W1:** _Comparing GUI-Actor-7B + verifier to baselines that predict actions in a single pass overstates its advantage, because the verifier injects an extra round of visual evidence and decision logic that competing methods never see. A like-for-like comparison should disable the verifier or add an equivalent post-filter to the baselines._\n\nThank you for the valuable comment. We reported the performance of GUI-Actor-7B both **with** and **without** the Verifier in our ablation studies (see Tables 4 and 5). To facilitate a more direct comparison, we reorganize key results and extend the evaluation to Qwen2.5-VL based models, as shown in **_Table H_** and **_Table I_** below. We can see that **_even without the Verifier, GUI-Actor consistently outperforms all SFT counterparts_** (OS-Atlas, Aguvis, and UGround). Notably, on the challenging OoD benchmark ScreenSpot-Pro, it also surpasses UI-TARS (trained on a substantially larger dataset and using a more complex 3-stage pipeline). This underscores the inherent strength of the GUI-Actor grounding model and its robustness to distribution shift. We will include this comparative table in the next revision.\n\n**_Table H._** _Performance Comparison across benchmarks for various models using Qwen2-VL as backbone._\n| Method                    | Backbone VLM | ScreenSpot-Pro | ScreenSpot | ScreenSpot-v2 |\n|---------------------------|---------------|----------------|-------------|----------------|\n| OS-Atlas-7B               | Qwen2-VL      | 18.9           | 82.5        | 84.1           |\n| AGUVIS-7B                 | Qwen2-VL      | 22.9           | 84.4        | 86.0          |\n| UGround-V1-7B             | Qwen2-VL      | 31.1           | 86.3        | 87.6          |\n| UI-TARS-7B                | Qwen2-VL      | 35.7           | **89.5**    | **91.6**       |\n| GUI-Actor-7B w/o Verifie             | Qwen2-VL      | **38.1**       | 86.6        | 88.9           |\n| GUI-Actor-7B w/ Verifier   | Qwen2-VL      | **41.6**       | **87.8**        | **89.8**           |\n\n**_Table I._** _Performance Comparison across benchmarks for various models using Qwen2.5-VL as backbone._\n| Method                     | Backbone VLM  | ScreenSpot-Pro | ScreenSpot-v2 |\n|----------------------------|----------------|----------------|----------------|\n| **_7B models:_**             |                |                |                |\n| Qwen2.5-VL-7B              | Qwen2.5-VL     | 27.6           | 88.8           |\n| Jedi-7B                   | Qwen2.5-VL     | 39.5           | 91.7           |\n| GUI-Actor-7B  w/o Verifie             | Qwen2.5-VL     | **44.6**       | **92.1**       |\n| GUI-Actor-7B w/ Verifier   | Qwen2.5-VL     | **47.7**           | **92.5**           |\n| **_3B models:_**             |                |                |                |\n| Qwen2.5-VL-3B              | Qwen2.5-VL     | 25.9           | 80.9           |\n| Jedi-3B                   | Qwen2.5-VL     | 36.1           | 88.6           |\n| GUI-Actor-3B  w/o Verifie              | Qwen2.5-VL     | **42.2**       | **91.0**       |\n| GUI-Actor-3B w/ Verifier   | Qwen2.5-VL     | **45.9**           | **92.4**           |\n\n\n**W2:** _Running the GUI-Actor-7B + verifier increases GPU memory and adds an extra forward pass per click, introducing latency that is hard to hide in real-time GUI agents._\n\nThank you for the insightful comment. We agree that adding the verifier introduces additional computational overhead. Our **_motivation_** for integrating the verifier is to **_explore test-time scaling_** for GUI-Actor. Unlike typical GUI grounding models, **_GUI-Actor can naturally propose multiple diverse candidate regions in a single pass, making it well-suited for pairing with a verifier at test time._**\n\nTo **_reduce inference time_**, we **_adopt three strategies_**: (1) using a lightweight 2B verifier, (2) applying early stopping when the verifier’s confidence score exceeds 0.95, effectively reducing the average number of verifier calls to 3.3 per sample on ScreenSpot-v2, and (3) leveraging vLLM for efficient inference.\n\nConsidering real-time GUI agents, we have some findings below to support this design. For example, **_GUI-Actor-2B combined with a 2B verifier outperforms standalone 7B grounding models on the ScreenSpot-Pro benchmark_**. This means that with proper parallelization, the overall runtime is approximately equivalent to two 2B model inferences—still **_notably more efficient than running a 7B model—while achieving superior performance_**. This makes our approach particularly **_promising for deployment in real-time GUI agents on resource-constrained devices_** such as phones, which may not support 7B models.\n\n**_Table J._** _Performance comparison across different model configurations._\n| Base Model                   | GUI-Actor-2B w/o Verifier | GUI-Actor-2B w/ Verifier | GUI-Actor-7B w/o Verifier | UI-TARS-7B |\n|-----------------------------|---------------------------|---------------------------|----------------------------|------------|\n| Performance on ScreenSpot-Pro | 36.7                      | 41.8                      | 40.7                       | 35.7       |\n\n\n**W3:** _The proposed method makes the “coordinate-free” pointer to land only at patch centers. The 28 × 28 patch grid becomes too large and coarse when the input resolution is small, making fine-grained targets (e.g., tiny icons or narrow scroll bars) unreachable._\n\nThank you for the thoughtful feedback. Theoretically, GUI-Actor is capable of locating elements smaller than 28x28 but larger than 14x14. The challenge of small element localization does exist for those **_smaller than 14x14_**, e.g., when they are located at the corner patch of a 28x28 grid. Fortunately, these extreme cases are **_relatively rare_** in typical desktop scenarios, as shown in **_Table K_**.\n\nTo **_mitigate this limitation_**, we explored a simple yet effective **_\"Zoom-In\" strategy_** for GUI-Actor, where GUI-Actor makes an initial coarse prediction, then zooms in 2× around the predicted location and performs a second round of grounding. **_Table L_** shows this significantly boosts performance on ScreenSpot-Pro, demonstrating its value in high-resolution settings with small UI elements.\n\n**_Table K._** _Frequency of small UI elements (longest edge < 14 px) fully contained within a single patch._\n| Dataset         | # of Examples | # of with Small Elements | # of Small Element In One Patch |\n|-----------------|----------------------|--------------| --------------|\n| ScreenSpot-v2    | 1272              | 0                    | 0            |\n| ScreenSpot-pro   | 1581              | 52                   | 10            |\n\n**_Table L._** _Effectiveness of the \"Zoom-In\" strategy on ScreenSpot-Pro._\n| Model for Zoom-In Round              | Model for Final Round              | Zoom-in Ratio | Screenspot-Pro |\n|-------------------------------------|------------------------------------|----------------|----------------|\n| /                                   | Aguvis-7B                          | /              | 22.9           |\n| /                                   | GUI-Actor-7B                       | /              | 38.1           |\n| GUI-Actor-7B (w/o Verifier)         | Aguvis-7B                          | 2x             | 43.0           |\n| GUI-Actor-7B (w/o Verifier)         | GUI-Actor-7B (w/o Verifier)        | 2x             | 50.3           |\n\n\n**Q1:** _The proposed method sets a confidence threshold, but it is not clear how the threshold is defined, and what if no candidate is above the threshold._\n\nThank you for pointing out the confusion. The verifier evaluates candidates sequentially, and we use an **_intuitive confidence threshold for early stopping_**: the first candidate exceeding this threshold is selected as the final prediction. **_If no candidate meets the threshold, we fall back to the one with the highest verifier score_**. We will clarify this in the revision. This strategy effectively reduces the average number of verifier calls from 20 to just 3.3 per sample on ScreenSpot-v2, with only a negligible impact on performance.\n\n\n**Q2:** _The author demonstrated fine-tuning the Qwen-2-VL model only. Is it possible to apply the proposed coordinate-free training to an existing GUI model such as UI-TARS-7B?_\n\nThank you for the insightful question. Here we further apply our coordinate-free GUI-Actor framework to UI-TARS-7B. As shown in **_Table M_**, when using UI-TARS-7B as the base model, GUI-Actor achieves **_comparable performance on the ScreenSpot-v2_** benchmark and **_higher accuracy on the more challenging ScreenSpot-Pro benchmark_**. We attribute this improvement to UI-TARS-7B’s strong visual encoder, which is initialized from Qwen2VL and further trained on GUI-specific datasets, likely leading to richer and more semantically aligned representations. When combined with our action-head-based grounding approach, these enhanced features enable more precise localization. These findings suggest that **_our method extends beyond general-purpose VLMs and can effectively complement and enhance existing GUI models_**.\n\n**_Table M._** _Evaluation of GUI-Actor applied to two different base VLMs: Qwen2VL-7B and UI-TARS-7B-SFT._\n| Model                                                      | Screenspot-v2 | Screenspot-Pro |\n|------------------------------------------------------------|----------------|-----------------|\n| UI-TARS-7B                                                 | 91.6           | 35.7            |\n| GUI-Actor w/ Qwen2VL-7B (w/o Verifier)                     | 88.9           | 38.1            |\n| GUI-Actor w/ UI-TARS-7B (w/o Verifier)                     | 91.4           | 40.5            |"}]

Author rebuttal:

Meta-review: This paper presents GUI-Actor, a coordinate‑free attention-based grounding framework for GUI agents. The approach is novel within GUI grounding, well motivated, clearly described, and extensively validated across ScreenSpot, ScreenSpot‑v2, ScreenSpot‑Pro and additional backbones, with strong OoD generalization and thorough ablations. Rebuttal and follow‑up clarifications satisfactorily addressed concerns on small targets, architecture details, efficiency, and fairness. All reviewers converge on positive recommendations. The AC agrees and recommends Acceptance.
Decision: Accept (poster)