ReGUIDE: Data Efficient GUI Grounding via Spatial Reasoning and Search

## Abstract

Abstract Recent advances in Multimodal Large Language Models (MLLMs) have enabled autonomous agents to interact with computers via Graphical User Interfaces (GUIs), where accurately localizing the coordinates of interface elements (e.g., buttons) is often required for fine-grained actions.
However, this remains significantly challenging, leading prior works to rely on large-scale web datasets
to improve the grounding accuracy.
In this work, we propose Reasoning Graphical User Interface Grounding for Data Efficiency (ReGUIDE), a novel and effective framework for web grounding that
enables MLLMs to learn data efficiently
through self-generated reasoning and spatial-aware criticism.
More specifically, ReGUIDE learns to (i) self-generate a language reasoning process for the localization via online reinforcement learning, and (ii) criticize the prediction using spatial priors that enforce equivariance under input transformations.
At inference time, ReGUIDE further boosts performance through a test-time scaling strategy, which combines spatial search with coordinate aggregation.
Our experiments demonstrate that ReGUIDE significantly advances web grounding performance across multiple benchmarks, outperforming baselines with substantially fewer training data points (that is, only 0.2% samples compared to the best open-source baselines).

## 1 Introduction

Graphical User Interface (GUI) agents—i.e., Multimodal Large Language Model (MLLM) agents that interpret visual screen contents and generate web actions in natural language to navigate the web environment—have shown promising capabilities in web navigation tasks as MLLMs continue to improve in general decision-making ability [26, 3, 28, 41]. Despite the recent significant efforts, however, a substantial gap remains between the performance of these agents and that of proficient human users [6], particularly in complex or long-horizon tasks [39]. This gap arises mainly from two challenges: limited visual understanding of web pages [33] and insufficient web domain-specific decision-making ability [11]. To address this, prior work typically decomposes the problem into two parts: (i) building an MLLM that can interpret the visual input [41] and (ii) employing a language model that performs decision-making and planning based on the MLLM’s interpretation [28, 14], where the major bottleneck lies in the visual understanding component [11].

Building an MLLM that can predict the exact pixel coordinates of target region on the screen (e.g., buttons) has shown great promise in providing the visual understanding required by another LLM for effective decision-making [11]. However, grounding remains challenging as it requires fine-grained skills such as reasoning and the manifestation of spatial understanding through token-based predictions [24, 40]. Recent approaches have emphasized the importance of large-scale, high-quality image–instruction pairs for training MLLMs through supervised fine-tuning (SFT) [22, 11]. Gou et al. [11] proposes synthesizing diverse referring expressions that can express the same object in several views. However, such SFT models depend on costly data curation to perform well, which poses a major scalability challenge [17].

To this end, we focus on extracting rich information from web image data to achieve data-efficient grounding by learning to explain the coordinate prediction process in natural language and by leveraging spatial priors to maximize the utility of visual input.

Contribution.
We propose Reasoning Graphical User Interface Grounding for Data Efficiency (ReGUIDE), a novel and effective GUI coordinate grounding method for web agents.
Specifically, ReGUIDE is composed of a two-stage training: (i) self-generation of image explanation through its own reasoning, and (ii) criticism of the localization prediction via spatial priors.
First, ReGUIDE learns to generate the language description of the given web image that can guide itself to correctly predict the coordinate, where this prediction accuracy is used as the reward for online RL.
Then, by leveraging a spatial prior—i.e., the fact that augmentations such as cropping lead to equivariant changes in the target coordinates—the model criticizes its predictions by ensuring consistent outputs under the same language description across augmented image–coordinate pairs.

Furthermore, we introduce an inference-time scaling strategy for ReGUIDE, which integrates spatial search with an LLM test-time scaling scheme.
Specifically, the model generates multiple localization predictions and crops the region where the target is likely to exist based on the predictions.
In this region, we generate multiple coordinate candidates,
then aggregate the candidates into a single coordinate
via statistical voting strategy (i.e., Kernel Density Estimation [30] (KDE)-based voting).

We demonstrate the effectiveness of ReGUIDE through evaluations on multiple web-grounding datasets and agent-setting benchmarks. Notably, ReGUIDE enhances web coordinate grounding performance beyond prior methods, achieving the state-of-the-art performance in web grounding. For instance, ReGUIDE improves the grounding accuracy of Qwen-2.5-VL-3B from 55.5% to 87.3% on Screenspot [7] and from 23.9% to 44.3% on Screenspot-Pro [20]. Furthermore, our experimental results show that ReGUIDE indeed guides the GUI agent to improve the overall decision-making ability, showing a significant performance improvement in agentic tasks.

## 2 Related Work

Graphic User Interface (GUI) grounding.
Recent advances in pixel-level GUI grounding have demonstrated mapping natural language instructions to screen coordinates without relying on HTML or DOM structures [31]. Several prior works [11, 37, 35, 8, 22] train on over a million synthesized screenshots and achieve superior performance.  Gou et al. [11] further shows that synthesizing and augmenting text instructions for each image element can lead to further improvements. However, current approaches depend on massive annotated datasets, incurring substantial computation and labeling costs, and neither leverages the reasoning capabilities of large language models to improve localization under data-scarce or out-of-distribution conditions.

Reinforcement learning in MLLM.
Reinforcement learning has emerged as a powerful mechanism to fine-tune multimodal large language models via self-improvement and feedback. Self-Critical Sequence Training [29] pioneered policy-gradient optimization for image captioning. Building on this paradigm, RLHF-V [38] formulates multimodal RLHF under constrained optimization to jointly maximize helpfulness and minimize unsafe outputs. More recently, Vision-R1 [17] generates synthetic chain-of-thought trajectories and applies iterative policy optimization to boost multimodal math reasoning. While UI-R1 [23] and GUI-R1 [36] are concurrent works that solves the GUI grounding problem with reinforcement learning, ReGUIDE further explores the grounding by utilizing the spatial prior and test-time spatial search. As a result, ReGUIDE significantly outperforms previous models in grounding evaluations as shown in Appendix.

Test-time scaling.
Recent works explored that scaling test-time computation, such as best-of-N sampling, can be even better than scaling train-time computation for performance [32]. Specifically, test-time scaling strategies improve LLM performance by generating numerous candidate outputs and selecting the best [32, 19, 15]. To enhance decision-making, external verifiers are often employed to evaluate and refine these outputs [15]. In the localization task, recent work framed the localization as a search problem and suggested a test-time scalable strategy [34].

Figure: (a) Learning to explain web images via reasoning
Refer to caption: extracted/6474733/images/Concept1.png

## 3 Data Efficient GUI Grounding via Spatial Reasoning and Search

In this section, we present Reasoning Graphical User Interface Grounding for Data Efficiency (ReGUIDE), a Graphical User Interface (GUI) grounding (i.e., coordinate prediction) training framework that self-generates the language description of the image through reasoning, and criticizes the prediction with spatial priors.
We present the core training method in Section [3.1](https://arxiv.org/html/2505.15259v2#S3.SS1), and then the test-time search method in Section [3.2](https://arxiv.org/html/2505.15259v2#S3.SS2).
The overview of ReGUIDE is depicted in Figure [6](https://arxiv.org/html/2505.15259v2#A3.F6).

Problem setup.
We describe the problem setup of our interest, namely the GUI grounding: training an MLLM that can predict the coordinate of the interface (e.g., ‘close button’) in the web image based on the given instruction (e.g., ‘close the window’).
Formally, given an input instruction $x_{\mathtt{inst}}$ and image $x_{\mathtt{img}}$, the MLLM $\mathcal{M}$ generates the output which consists of reasoning path $r$ and predicted coordinate $\mathbf{c}=(c_{1},c_{2})$, i.e., $r,\mathbf{c}\sim\mathcal{M}(\cdot|x_{\mathtt{inst}},x_{\mathtt{img}})$.
Here, we denote the ground-truth region in the pixel coordinate space corresponding to $(x_{\mathtt{inst}},x_{\mathtt{img}})$ as $\mathcal{X}_{\mathtt{gt}}\subseteq[0,W]\times[0,H]$, where $W$ and $H$ are the width and height of $x_{\mathtt{img}}$.
Any coordinate $\mathbf{c}\in\mathcal{X}_{\mathtt{gt}}$ is considered a correct prediction, as all such points result in the same GUI behavior (e.g., triggering the intended button or link).

### 3.1 ReGUIDE: Reasoning Graphical User Interface Grounding for Data Efficiency

We describe the core training pipeline of ReGUIDE. The key idea is to extract rich information from web images, enabling the model to generalize even with a relatively small training dataset. To this end, we train the model to self-explain the reasoning behind its predictions, as learning an explicit reasoning process—rather than merely memorizing input-output patterns—is crucial for generalization [25, 18]. Building on this learned reasoning, we further refine the predictions through a criticism step, which assesses the consistency of the output under image augmentations while maintaining the same reasoning trace.

Learning to explain GUI images via reasoning. At the first stage, we train the LLM to reason about the exact coordinate of the instruction on the screen, where we employ online reinforcement learning (RL) to make MLLM self-evolve by using grounding accuracy as a reward. This enables the model to generate its own text reasoning and evolve without relying on externally provided language descriptions, thus saving the language annotation cost.
RL also tends to generalize better than supervised fine-tuning (SFT), which may suffer from memorization [25].

For a given instruction $x_{\mathtt{inst}}$ and an image $x_{\mathtt{img}}$, we define two reward functions to train MLLM with RL.
Specifically, we consider (i) the result of ground truth prediction as the primary reward and (ii) an auxiliary formatting reward that encourages the model to wrap its reasoning trace between ‘<think>’ and ‘</think>’ tags, ensuring that the generated reasoning follows the desired format for downstream usage by following recent work in LLM reasoning [17, 12].
Formally, we define the reward function $R(\cdot,\cdot)$ as follows:

$$ $R(r,\mathbf{c})=\mathbf{1}[\mathbf{c}\in\mathcal{X}_{\mathtt{gt}}]+\lambda \cdot\mathbf{1}[\text{format}(r)=\texttt{<think>}\cdots\texttt{</think>}],$ (1) $$

where $\mathbf{1}[\cdot]$ denotes the indicator function that returns one if the condition holds and zero otherwise, $\text{format}(r)$ extracts the outermost structure of the generated reasoning trace to check whether it is properly enclosed between <think> and </think>, and $\lambda\in\mathbb{R}^{+}$ is a hyperparameter that balances the contribution of the formatting reward, where we use $\lambda=0.1$ throughout the experiment.

The overall RL objective is to maximize the expected reward:

$$ $\max_{\mathcal{M}}\;\mathbb{E}_{x_{\mathtt{inst}},x_{\mathtt{img}}}\,\mathbb{E }_{(r,\mathbf{c})\sim\mathcal{M}(\cdot\mid x_{\mathtt{inst}},x_{\mathtt{img}}) }\left[R(r,\mathbf{c})\right].$ (2) $$

To this end, we adopt Group Relative Policy Optimization (GRPO) [9] as our RL algorithm, as GRPO enhances both efficiency and stability by sampling multiple candidate outputs per input and computing group-relative rewards.This overall process allows the model to explore diverse reasoning strategies and progressively reinforce those that contribute to accurate grounding, without relying on explicit reasoning supervision.

Learning to predict consistent coordinates under transformations. While the RL stage enables the model to predict coordinates within the correct ground truth region, it does not necessarily guide the model toward predicting the most precise or stable point, typically the center of the target element (as the target UI element is annotated with a bounding box, making the center a natural target). Additionally, the model may be sensitive to variations in scale and cropping, which can degrade performance in real-world scenarios where UI elements may appear at different sizes or positions.

To address these limitations, we introduce a further training phase that enforces spatial precision and view consistency. Specifically, we continually train the MLLM to predict the center of the ground-truth region, while jointly enforcing consistent predictions under image transformations through spatial priors. In particular, we focus on constraining consistency between the model’s predictions on the global view (full image) and the local view (a zoomed-in crop covering the ground-truth region). Notably, such global-local view consistency has been identified as a key factor in the success of prior self-supervised vision methods [5, 27, 13].
Furthermore, we make the local view follow the reasoning process of the global view (as it typically contains relative spatial cues), forming a self-distillation of the reasoning process.

Formally, we collect two data points, namely the global view data $\mathbf{d}_{\mathtt{global}}$ and local view data $\mathbf{d}_{\mathtt{local}}$, where each data point consists of image, instruction, reasoning, and target coordinates.
Here, both data points consist of the same instruction and reasoning, but with different images and target coordinates (due to the spatial transformation).
To ensure high-quality reasoning, we only use samples that correctly predict the target region in the global image $x_{\mathtt{img}}$, i.e., $r$ from $(r,\mathbf{c})\sim\mathcal{M}(\cdot\mid x_{\mathtt{inst}},x_{\mathtt{img}})$ such that $\mathbf{c}\in\mathcal{X}_{\mathtt{gt}}$.
For the global view data, we use the non-transformed image $x_{\mathtt{img}}$ and the center of the ground truth region $\mathtt{center}(\mathcal{X}_{\mathtt{gt}})$, while for the local view data, we use a randomly cropped image $\mathtt{crop}(x_{\mathtt{img}})$ and the center of the transformed ground truth region, denoted as $\mathtt{center}(\mathcal{X}_{\mathtt{gt}}^{\mathtt{crop}})$. Namely, the global and local view data are defined as:

$$ $\mathbf{d}_{\mathtt{global}}:=\big{(}x_{\mathtt{inst}},x_{\mathtt{img}},r, \mathtt{center}(\mathcal{X}_{\mathtt{gt}})\big{)},\quad\mathbf{d}_{\mathtt{ local}}:=\big{(}x_{\mathtt{inst}},\mathtt{crop}(x_{\mathtt{img}}),r,\mathtt{ center}(\mathcal{X}_{\mathtt{gt}}^{\mathtt{crop}})\big{)}.$ $$

For training, the model processes the original global view and the newly created local view in the same batch, and is supervised via next-token prediction loss to generate the updated coordinates as text tokens:

$$ $\min_{\mathcal{M}}~{}\mathcal{L}(\mathbf{d}_{\text{global}})+\mathcal{L}( \mathbf{d}_{\text{local}}),\quad\text{where}\quad\mathcal{L}(\mathbf{d})=-\log \mathcal{M}(r,\mathbf{c}\mid x_{\mathtt{inst}},x_{\mathtt{img}}^{\prime}),$ (3) $$

where $\mathbf{d}=(x_{\mathtt{inst}},x_{\mathtt{img}}^{\prime},r,\mathbf{c})$ and $x_{\mathtt{img}}^{\prime}$ denotes the input image used in the given view (i.e., $x_{\mathtt{img}}$ for the global view and $\mathtt{crop}(x_{\mathtt{img}})$ for the local view).

### 3.2 Test-time Scaling with Spatial Search and Kernel Density Estimation Based Aggregation

Despite their strong reasoning capabilities via natural text, LLMs inherently struggle with coordinate prediction due to their limited awareness of ordinal relationships among numeric tokens. To alleviate the issue, we propose a scalable inference method for models trained with ReGUIDE. Specifically, we introduce a two-stage inference method, namely, composed of the (i) cropping stage and the (ii) voting stage. Here, the key idea for each stage is to (i) zoom in on the image where the UI element is likely to exist, and (ii) predict multiple coordinates on the zoomed image, which are aggregated into a single coordinate for a robust and confident prediction. An illustration of our test-time spatial search strategy is provided in Appendix.

Cropping: Zooming into the UI element area. To crop the region where the UI element is likely to exist, we first predict multiple coordinates on the full image. Then, we treat these predictions as samples from a probability distribution indicating the target’s likely location. Here, we use Kernel Density Estimation (KDE) to analyze these initial samples and identify the region of highest prediction density—the area where the model is most consistently expected to be the target. Thus, we choose the highest density point as the center of the cropped Region of Interest (RoI), effectively allowing the model to "zoom in" on the most promising area.

Concretely, given an input image $x_{\mathtt{img}}$ and instruction $x_{\mathtt{inst}}$, we sample $N$ initial predictions: $\mathcal{C}:=\{\mathbf{c}^{(i)}\}_{i=1}^{N}$, where $(r^{(i)},\mathbf{c}^{(i)})\sim\mathcal{M}(\cdot\mid x_{\mathtt{inst}},x_{
\mathtt{img}})$.
Then, we apply KDE to these $N$ predictions to find the ‘center’ $\mathbf{c}_{\mathtt{KDE}}$ of the predictions by summing 2D Gaussian kernels (with a pre-defined variance $\Sigma$) centered at each prediction $\mathbf{c}^{(j)}$ to estimate a density $S(z;\mathcal{C})$:

$$ $\mathbf{c}_{\mathtt{KDE}}=\operatorname*{argmax}_{z\in[0,W]\times[0,H]}S(z; \mathcal{C})\quad\text{where}\quad S(z;\mathcal{C})=\mathbb{E}_{\mathbf{c}\in \mathcal{C}}\Big{[}\exp\bigl{(}-\tfrac{1}{2}(z-\mathbf{c})^{\!\top}\Sigma^{-1} (z-\mathbf{c})\bigr{)}\Big{]}.$ (4) $$

A fixed-size bounding box $\mathcal{X}_{\mathtt{RoI}}$ (with dimensions $W_{\mathtt{RoI}}\times H_{\mathtt{RoI}}$), centered at $\mathbf{c}_{\mathtt{KDE}}$, defines the region of interest. We crop the image deterministically to this region:
$\mathtt{RoI}(x_{\mathtt{img}})=\mathtt{crop}(x_{\mathtt{img}};\,\mathbf{c}_{
\mathtt{KDE}},\,W_{\mathtt{RoI}},\,H_{\mathtt{RoI}})$.

Voting: Aggregating multiple votes within RoI.
In the voting stage, we further refine the coordinate prediction within this RoI. Given the local zoomed-in view, these predictions are expected to be more precise – the answer space is narrowed. We then reapply KDE to these new predictions within the RoI. This second application of KDE acts as a robust voting mechanism, aggregating the multiple refined predictions to determine the single coordinate with the highest probable point (i.e., the peak of the density). This two-stage searching strategy not only allows ReGUIDE to more precisely predict the final answer but also offers scalability: investing more computational resources in generating and evaluating samples for refinement generally leads to a more accurate result.

Formally, the model re-predicts $M$ new coordinates:
$\mathcal{C}_{\mathtt{RoI}}:=\{\mathbf{c}_{\mathtt{RoI}}^{(i)}\}_{i=1}^{N}$, where $(r^{(i)},\mathbf{c}_{\mathtt{RoI}}^{(i)})\sim\mathcal{M}(\cdot\mid x_{\mathtt{
inst}},\mathtt{RoI}(x_{\mathtt{img}}))$.
Then, we use the same process as in [Equation 4](https://arxiv.org/html/2505.15259v2#S3.E4) to predict the most confident center point $\mathbf{c}_{\mathtt{final}}$ within $\mathcal{C}_{\mathtt{RoI}}\cup\mathcal{C}$ by robustly aggregating coordinates with KDE:

$$ $\mathbf{c}_{\mathtt{final}}=\operatorname*{argmax}_{z\in\mathcal{\mathcal{X}}} S(z;\mathcal{C}_{\mathtt{RoI}}\cup\mathcal{C}).$ (5) $$

## 4 Experiments

We provide an empirical evaluation of ReGUIDE by investigating the following questions:

- •
Can ReGUIDE enhance GUI grounding performance? (Table [1](https://arxiv.org/html/2505.15259v2#S4.T1) and Table [2](https://arxiv.org/html/2505.15259v2#S4.T2))
- •
Can ReGUIDE improve overall GUI agent performance? (Table [6](https://arxiv.org/html/2505.15259v2#S4.T6))
- •
How does the MLLM perform reasoning based on the given instruction and image? (Table [3](https://arxiv.org/html/2505.15259v2#S4.T3))
- •
Do the proposed components enhance the grounding performance? (Table [4](https://arxiv.org/html/2505.15259v2#S4.T4) and Table [5](https://arxiv.org/html/2505.15259v2#S4.T5))

Before answering each question, we outline the experimental protocol (more details in Appendix [A](https://arxiv.org/html/2505.15259v2#A1)).

Evaluation setup. In the main results, we mainly report the grounding accuracy (%) as a metric. The prediction is counted as correct when the predicted point lies inside the ground-truth bounding box. During evaluation, we generate a prediction via greedy decoding. For the searching strategy of ReGUIDE we used prediction sampling $N=M=16$ samples with temperature $T=1.0$, crop bounding box size as $W_{\mathtt{RoI}}=H_{\mathtt{RoI}}=840$, and KDE variance $\Sigma=0.01$. We evaluate ReGUIDE and baselines on ScreenSpot [7], ScreenSpot-v2 [35], ScreenSpot-Pro [20], evaluation benchmarks for GUI grounding. Especially, ScreenSpot-Pro is a more challenging construct with a high-resolution image (i.e., up to 3840 $\times$ 2160). The agentic evaluation setting is in Section [4.3](https://arxiv.org/html/2505.15259v2#S4.SS3).

Training setup. For the main experiment, we train ReGUIDE on Qwen-2.5-VL 3B/7B. To demonstrate ReGUIDE’s data efficiency, we utilize only a 20k subset of UGround [11] dataset constituting approximately 0.2% of its full set. Learning to explain web images via reasoning, we use the GRPO [9] algorithm with two rewards, accuracy reward and output-format compliance reward, and $\lambda=0.1$ for formatting reward weight. For learning to predict consistent coordinates under transformations, where the local view is a random crop of up to 30% of the original area while preserving aspect ratio. More setups in Appendix [A.3](https://arxiv.org/html/2505.15259v2#A1.SS3).

Baselines. We compare our method against several baselines, which fall into two categories. First, we consider the model Qwen-VL-2.5 3B/7B supervised fine-tuned (SFT) on the identical dataset with ReGUIDE (i.e., 20k UGround subset). SFT optimises a coordinate-only loss and does not generate reasoning. Additionally, we consider existing proprietary MLLM and GUI grounding models: GPT-4o [26], Claude3.7 [4], SeeClick [8], OS-Atlas-7B [35], AGUVIS-7B [37], and UGround [11].

**Table 1: Accuracy (%) for ReGUIDE (Ours) and other baselines, including GPT-4o [26], Claude 3.7 [4], SeeClick [8], OS-Atlas-7B [35], AGUVIS-7B [37], UGround [11], Qwen-2.5-VL [2], and ReGUIDE (Ours). We evaluate on three web-grounding benchmarks: ScreenSpot [7], ScreenSpot-v2 [35], and ScreenSpot-Pro [20]. The Proprietary Models include proprietary systems, while Open-Sourced cover public research models. Models supervised-finetuned (SFT) with the same architecture (i.e., Qwen-2.5-VL) and dataset (i.e., a 20K subset of UGround) are grouped separately for controlled comparisons. Bold indicates the best result within each group.**
| Methods | Data Size | ScreenSpot [7] | ScreenSpot-v2 [35] | ScreenSpot-Pro [20] |
| --- | --- | --- | --- | --- |
| Proprietary Models |  |  |  |  |
| 00GPT-4o [26] | - | 18.3 | 16.6 | 00.8 |
| 00Claude 3.7 [4] | - | 82.1 | 87.6 | 27.7 |
| Open-Sourced |  |  |  |  |
| 00SeeClick [8] | 01M | 53.4 | 55.1 | 01.1 |
| 00OS-Atlas-7B [35] | 13M | 82.5 | 84.1 | 18.9 |
| 00AGUVIS-7B [37] | 01M | 84.4 | 85.8 | 22.9 |
| 00UGround-2B [11] | 10M | 77.7 | 81.4 | 26.6 |
| 00UGround-7B [11] | 10M | 86.3 | 89.1 | 31.1 |
| Trained with same model and dataset |  |  |  |  |
| 00Qwen-2.5-VL-3B [2] | - | 55.5 | 70.4 | 23.9 |
| 0000+ SFT | 20K | 56.8 | 56.5 | 11.6 |
| 0000+ ReGUIDE (Ours) | 20K | 87.3 | 89.7 | 44.3 |
| 00Qwen-2.5-VL-7B [2] | - | 84.7 | 82.6 | 29.0 |
| 0000+ SFT | 20K | 84.9 | 88.9 | 25.9 |
| 0000+ ReGUIDE (Ours) | 20K | 90.1 | 91.5 | 44.4 |

### 4.1 Main Results

We first present the main result by comparing the GUI grounding performance with other baselines. Here, we mainly compare ReGUIDE with finetuning baselines with the same model and dataset, and other open and closed models.

As shown in Table [1](https://arxiv.org/html/2505.15259v2#S4.T1), we present the GUI grounding performance of ReGUIDE compared to other baselines. We first compare models trained with the same base architecture (Qwen-2.5-VL) and dataset (a 20k subset of UGround). Our proposed method, ReGUIDE, significantly outperforms supervised fine-tuning (SFT) baselines across all evaluated scenarios. For instance, ReGUIDE achieves an accuracy of 87.3% on ScreenSpot with the 3B model. Similar improvements are observed for the more challenging ScreenSpot-Pro benchmark, where ReGUIDE boosts performance from 23.9% to 44.3%. Also in ScreenSpot-v2 and dataset, ReGUIDE outperforms baseline and SFT model with a meaningful margin.

Comparison with open-sourced and proprietary models.
Beyond models trained with the same data and architecture, we compare ReGUIDE to state-of-the-art open-sourced and proprietary models, including SeeClick [8], Os-Atlas-7B [35], AGUVIS-7B [37], UGround [11], GPT-4o [26], and Claude [4].
ReGUIDE, despite being trained on only a 20k subset of UGround, achieves competitive or superior performance.
Specifically on ScreenSpot, our 3B model surpasses UGround-7B (87.3% > 86.3%) and ReGUIDE’s 7B model’s superior results against other baselines. Also, ReGUIDE consistently outperforms other baselines in ScreenSpot-v2, too.
ReGUIDE shows the most remarkable performance on ScreenSpot-Pro. ReGUIDE performs accuracy of 45.1%, significantly outperforming UGround-7B (31.1%).

Training Efficiency of ReGUIDE.
These results demonstrate that ReGUIDE not only surpasses supervised finetuned within the same training dataset but also establishes a new competitive standard against much larger, heavily trained open and closed models.
Importantly, ReGUIDE achieves these improvements *using only a small fraction of the data (a 20k subset of UGround)* compared to other models (i.e., UGround: 10M, AGUVIS: 1M), highlighting its data efficiency and scalability.
By combining reinforcement learning, transform consistency learning, and scalable searching inference, ReGUIDE achieves a strong performance.

### 4.2 Additional Analysis and Ablation

**Table 2: Grounding Accuracy (%) for ReGUIDE (Ours) and UGround baselines[11] on the ScreenSpot [11] domain split. Results are broken down by device class (Mobile, Desktop, Web, and UI element type (Text, Icon). The right-most column reports the overall average. Bold indicates the best result within each column.**
| Model | Data Size | Mobile | Desktop | Web | Average |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | Text | Icon | Text | Icon | Text | Icon |  |
| UGround-2B [11] | 10M | 89.4 | 72.0 | 88.7 | 65.7 | 81.3 | 68.9 | 77.7 |
| UGround-7B [11] | 10M | 93.0 | 79.9 | 93.8 | 76.4 | 90.9 | 84.0 | 86.3 |
| ReGUIDE-3B | 20K | 94.9 | 81.2 | 93.8 | 83.6 | 87.0 | 81.1 | 87.3 |
| ReGUIDE-7B | 20K | 95.2 | 89.5 | 93.8 | 83.6 | 91.3 | 83.5 | 90.1 |

Generalizability over several domains.
We further analyze the robustness and generalizability of ReGUIDE across various GUI domains in multiple device environments (Mobile, Desktop, Web) and UI element types (Text, Icon). While our main results in Table [1](https://arxiv.org/html/2505.15259v2#S4.T1) report the average accuracy on the ScreenSpot, here we provide a more fine-grained domain-specific evaluation. As shown in Table [2](https://arxiv.org/html/2505.15259v2#S4.T2), ReGUIDE consistently achieves superior or competitive performance compared to the state-of-the-art open-sourced base model (i.e., Uground [11]). These results indicate that the training schemes of ReGUIDE effectively enhance its ability to generalize across diverse environments. Notably, ReGUIDE particularly shows strong performance for Icon elements within the Mobile and Desktop domains. We believe that the multi-view consistency training attributes the results, which likely enhances the model’s understanding of the local visual features present in iconic representations.

**Table 3: Example of ReGUIDE’s generated reasoning and predicted coordinate.**
|  | Target Prompt: Choose WeChat.<br>Response: <think> The task is to find the coordinate of the “WeChat” app icon in the image.<br>The image shows a section labeled “Top free apps” with icons and names of apps.<br>The “WeChat” app icon is located in the first row, first column of this section.<br></think> <answer>(110,437)</answer> |
| --- | --- |

Relative spatial cue of reasoning generations.
We analyzed the generated reasoning path of ReGUIDE and the associated spatial predictions. First, we observe that the model naturally decomposes GUI localization tasks into interpretable reasoning steps, referencing relative spatial information. As shown in Table [3](https://arxiv.org/html/2505.15259v2#S4.T3), the model explicitly identifies intermediate visual landmarks (e.g., a labeled section of "Top free apps") before specifying precise coordinates based on relative positions (e.g., "first row, first column"). This structured, relative reasoning pattern supports the models’ ability to generalize across varying UI layouts.

Figure: Figure 2: Likelihood of coordinate tokens spreaded gaussian naturally.
Refer to caption: extracted/6474733/images/img_confidence_heatmap.jpg

Natural Gaussian distribution of ground likelihood.
As an additional qualitative analysis, we investigate the likelihood distributions of coordinates surrounding the initial predicted coordinates. By calculating the likelihood of coordinates with same generated reasoning.
As shown in Figure [2](https://arxiv.org/html/2505.15259v2#S4.F2), the likelihoods of the surrounding coordinates are a surprisingly smooth, Gaussian-shaped distribution.
This Gaussian pattern indicates that the MLLM implicitly captures continuous spatial uncertainty, despite operating entirely in token space.
This observation provides a strong empirical justification for our proposed Gaussian-weighted inference strategy, that naturally emerging confidence distributions to enhance coordinate prediction accuracy and robustness.

**Table 4: Contribution of each proposed component of ReGUIDE on GUI grounding. We tested all three training components trained with Qwen-2.5-VL-3B: learn spatial reasoning (Reason), learn consistency under transformation (Consistency), and test-time spatial searching (Search). We report the GUI grounding accuracy (%) on ScreenSpot and ScreenSpot-Pro benchmarks.**
| Model Size | Reason | Consistency | Search | ScreenSpot [7] | ScreenSpot-Pro [20] |
| --- | --- | --- | --- | --- | --- |
| 3B | ✗ | ✗ | ✗ | 55.5 | 23.9 |
| ✓ | ✗ | ✗ | 83.3 | 27.2 |  |
| ✓ | ✓ | ✗ | 84.9 | 27.9 |  |
| ✓ | ✓ | ✓ | 87.3 | 44.3 |  |
| 7B | ✗ | ✗ | ✗ | 84.7 | 29.0 |
| ✓ | ✗ | ✗ | 85.6 | 27.0 |  |
| ✓ | ✓ | ✗ | 89.1 | 30.1 |  |
| ✓ | ✓ | ✓ | 90.1 | 44.4 |  |

Analysis of training components. We perform an analysis on each component of ReGUIDE, specifically spatial reasoning (Reasoning), learn consistency under transformation (Consistency), and test-time searching strategy (Searching). As shown in Table [4](https://arxiv.org/html/2505.15259v2#S4.T4), each component plays an important role, leading to gradual and significant improvements when applied sequentially. Additional comparisons of alternative RL policy-optimization algorithms are provided in Appendix [D](https://arxiv.org/html/2505.15259v2#A4).

**Table 5: Contribution of each proposed component for test-time scaling, namely, the cropping and voting. We report the grounding accuracy (%) on ScreenSpot and ScreenSpot-Pro benchmarks with ReGUIDE-3B. The bold indicates the best results.**
| Cropping | Voting | ScreenSpot [7] | ScreenSpot-Pro [20] |
| --- | --- | --- | --- |
| ✗ | ✗ | 84.3 | 27.9 |
| ✓ | ✗ | 81.7 | 42.7 |
| ✓ | ✓ | 87.3 | 44.3 |

Analysis of inference components. We perform ablation to prove the effectiveness of each component in the two-stage inference time searching strategy. As shown in Table [5](https://arxiv.org/html/2505.15259v2#S4.T5), both the crop stage and the voting stage contribute to improvement in performance. It is notable that in Screenspot-Pro, the improvement induced by crop stage is huge, which indicates that localization plays a crucial role for the proper understanding of high-resolution images.

Figure: (a) Effect of $W_{\mathtt{RoI}}$ (crop size).
Refer to caption: x1.png

Ablation on inference time searching strategy. We perform ablation for three hyperparameters on inference time scaling strategy, that is, crop size ($W_{\mathtt{RoI}}$), generation samples ($N$), and temperature ($T$). As shown in Figure [3](https://arxiv.org/html/2505.15259v2#S4.F3), increasing $N$ gradually improves the grounding performance, demonstrating that our strategy is scalable. Unless stated otherwise, we adopt the default hyperparameters $N=16$, $L=840$, and $T=1.0$, which balance performance and cost.

**Table 6: Task-success rate (%) for ReGUIDE (Ours) and UGround [11] baselines on the five offline-agent task benchmarks, i.e., Multimodal-Mind2Web (MM2W) [41, 10]’s cross-domain, -task, -web, Android Control (AC)-low and -high [21]. In every experiment, we used GPT-4o as a planner, while the listed models provide the grounding module.**
| Grounding Module | Data Size | MM2W [41, 10] | AC [21] |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| domain | task | web | high | low |  |  |
| UGround-2B [11] | 10M | 47.7 | 48.6 | 47.6 | 65.0 | 50.0 |
| UGround-7B [11] | 10M | 48.5 | 50.7 | 48.1 | 66.2 | 49.8 |
| ReGUIDE-3B (Ours) | 20K | 48.8 | 50.5 | 47.7 | 66.2 | 49.8 |
| ReGUIDE-7B (Ours) | 20K | 49.5 | 52.0 | 48.7 | 67.4 | 50.0 |

### 4.3 Agentic Settings Experiments

To verify whether better grounding transfers to real tasks, we plug each grounding model with ReGUIDE and use a high-level planner (GPT-4o). Here, we report task-success rate (%) on five offline suites: Multimodal-Mind2Web-cross domain, -cross task, and -cross website [10], AndroidControl-Low and -High [21]. Each benchmark has a different domain of agent settings (i.e., desktop web navigation and mobile-app control). For every episode, the planner emits a natural-language action plan; the grounding module converts each step into pixel coordinates, which the environment executes. Success is recorded when the episode reaches the goal state. All prompts, settings, and frameworks are followed by UGround [11]. The detailed settings are described in Appendix [A](https://arxiv.org/html/2505.15259v2#A1).

As shown in Table [6](https://arxiv.org/html/2505.15259v2#S4.T6), improved grounding performance made by ReGUIDE leads to better agent performance. With only 3B parameters, ReGUIDE matches or edges out the larger UGround-7B on Multimodal-Mind2Web and Android Control-High. The 7B variant of ReGUIDE consistently outperforms other baselines (i.e., +1.3% in Multimodal-Mind2Web cross task). These gains, achieved with identical planners and action budgets, substantiate that the grounding improvements provided by self-evolutionary RL and global–local consistency directly enhance full-task completion.

## 5 Discussion and Conclusion

We proposed ReGUIDE, a novel and effective GUI grounding framework that significantly enhances the capabilities of Multimodal Large Language Models by enabling data-efficient learning through self-generated reasoning and spatial-aware criticism. Our key idea is leveraging online reinforcement learning for self-generating language reasoning and employing spatial priors to criticize predictions, and further boosting performance at inference time through a test-time scaling search strategy that integrates spatial search with coordinate aggregation. We demonstrated that ReGUIDE consistently outperforms other open-sourced baselines, even when trained with a tiny fraction of data, such as only 0.2% of the samples used by the best open-sourced baselines. Crucially, these advances in data-efficient grounding translate to improved performance in downstream agentic tasks, highlighting ReGUIDE’s potential to develop more capable and practical GUI agents.

Future works and limitations. We believe it will be an interesting future direction to train MLLM planners that can operate hierarchically with ReGUIDE, fully capitalizing on its precise grounding performance. Additionally, a potential limitation is that while ReGUIDE is a very effective method, it lacks an explicit safety framework to prevent malicious uses, such as hacking or spreading misinformation—a challenge common to many grounding models. However, future investigations could explore the research and development of robust safeguards.

## References

- Ahmadian et al. [2024]
A. Ahmadian, C. Cremer, M. Gallé, M. Fadaee, J. Kreutzer, O. Pietquin, A. Üstün, and S. Hooker.
Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms.
*arXiv preprint arXiv:2402.14740*, 2024.
- Alibaba [2025]
Alibaba.
Qwen2.5-vl technical report.
*arXiv preprint arXiv:abs/2502.13923*, 2025.
- Anthropic. [2024]
T. Anthropic.
Introducing computer use, a new claude 3.5 sonnet, and claude 3.5 haiku.
[https://www.anthropic.com/news/3-5-models-and-computer-use](https://www.anthropic.com/news/3-5-models-and-computer-use), October 2024.
- Anthropic. [2025]
T. Anthropic.
Claude 3.7 sonnet and claude code.
[https://www.anthropic.com/news/claude-3-7-sonnet](https://www.anthropic.com/news/claude-3-7-sonnet), February 2025.
- Caron et al. [2021]
M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin.
Emerging properties in self-supervised vision transformers.
In *Proceedings of the IEEE/CVF international conference on computer vision*, pages 9650–9660, 2021.
- Chen et al. [2025]
C. Chen, Z. Zhang, I. Khalilov, B. Guo, S. A. Gebreegziabher, Y. Ye, Z. Xiao, Y. Yao, T. Li, and T. J.-J. Li.
Toward a human-centered evaluation framework for trustworthy llm-powered gui agents.
*arXiv preprint arXiv:2504.17934*, 2025.
- Cheng et al. [2024a]
K. Cheng, Q. Sun, Y. Chu, F. Xu, Y. Li, J. Zhang, and Z. Wu.
Seeclick: Harnessing gui grounding for advanced visual gui agents.
*arXiv preprint arXiv:2401.10935*, 2024a.
- Cheng et al. [2024b]
K. Cheng, Q. Sun, Y. Chu, F. Xu, Y. Li, J. Zhang, and Z. Wu.
Seeclick: Harnessing gui grounding for advanced visual gui agents.
*arXiv preprint arXiv:2401.10935*, 2024b.
- DeepSeek [2024]
DeepSeek.
Deepseekmath: Pushing the limits of mathematical reasoning in open language models.
*arXiv preprint arXiv:2402.03300*, 2024.
- Deng et al. [2023]
X. Deng, Y. Gu, B. Zheng, S. Chen, S. Stevens, B. Wang, H. Sun, and Y. Su.
Mind2web: Towards a generalist agent for the web.
In *Thirty-seventh Conference on Neural Information Processing Systems*, 2023.
URL [https://openreview.net/forum?id=kiYqbO3wqw](https://openreview.net/forum?id=kiYqbO3wqw).
- Gou et al. [2024]
B. Gou, R. Wang, B. Zheng, Y. Xie, C. Chang, Y. Shu, H. Sun, and Y. Su.
Navigating the digital world as humans do: Universal visual grounding for gui agents.
*arXiv preprint arXiv:2410.05243*, 2024.
- Guo et al. [2025]
D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al.
Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.
*arXiv preprint arXiv:2501.12948*, 2025.
- Hjelm et al. [2018]
R. D. Hjelm, A. Fedorov, S. Lavoie-Marchildon, K. Grewal, P. Bachman, A. Trischler, and Y. Bengio.
Learning deep representations by mutual information estimation and maximization.
*arXiv preprint arXiv:1808.06670*, 2018.
- Hong et al. [2024]
W. Hong, W. Wang, Q. Lv, J. Xu, W. Yu, J. Ji, Y. Wang, Z. Wang, Y. Dong, M. Ding, et al.
Cogagent: A visual language model for gui agents.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 14281–14290, 2024.
- Hosseini et al. [2024]
A. Hosseini, X. Yuan, N. Malkin, A. Courville, A. Sordoni, and R. Agarwal.
V-star: Training verifiers for self-taught reasoners.
*arXiv preprint arXiv:2402.06457*, 2024.
- Hu [2025]
J. Hu.
Reinforce++: A simple and efficient approach for aligning large language models.
*arXiv preprint arXiv:2501.03262*, 2025.
- Huang et al. [2025]
W. Huang, B. Jia, Z. Zhai, S. Cao, Z. Ye, F. Zhao, Z. Xu, Y. Hu, and S. Lin.
Vision-r1: Incentivizing reasoning capability in multimodal large language models.
*arXiv preprint arXiv:2503.06749*, 2025.
- Lampinen et al. [2022]
A. K. Lampinen, I. Dasgupta, S. C. Chan, K. Matthewson, M. H. Tessler, A. Creswell, J. L. McClelland, J. X. Wang, and F. Hill.
Can language models learn from explanations in context?
*arXiv preprint arXiv:2204.02329*, 2022.
- Lee et al. [2025]
H. Lee, S. Oh, J. Kim, J. Shin, and J. Tack.
Revise: Learning to refine at test-time via intrinsic self-verification.
*arXiv preprint arXiv:2502.14565*, 2025.
- Li et al. [2025]
K. Li, Z. Meng, H. Lin, Z. Luo, Y. Tian, J. Ma, Z. Huang, and T.-S. Chua.
Screenspot-pro: Gui grounding for professional high-resolution computer use.
*arXiv preprint arXiv:2504.07981*, 2025.
- Li et al. [2024]
W. Li, W. Bishop, A. Li, C. Rawles, F. Campbell-Ajala, D. Tyamagundlu, and O. Riva.
On the effects of data scale on computer control agents.
*arXiv preprint arXiv:2406.03679*, 2024.
- Lin et al. [2024]
K. Q. Lin, L. Li, D. Gao, Z. Yang, S. Wu, Z. Bai, W. Lei, L. Wang, and M. Z. Shou.
Showui: One vision-language-action model for gui visual agent.
*arXiv preprint arXiv:2411.17465*, 2024.
- Lu et al. [2025]
Z. Lu, Y. Chai, Y. Guo, X. Yin, L. Liu, H. Wang, G. Xiong, and H. Li.
Ui-r1: Enhancing action prediction of gui agents by reinforcement learning.
*arXiv preprint arXiv:2503.21620*, 2025.
- Ma et al. [2025]
X. Ma, Z. Ding, Z. Luo, C. Chen, Z. Guo, D. F. Wong, X. Feng, and M. Sun.
Deepperception: Advancing r1-like cognitive visual perception in mllms for knowledge-intensive visual grounding.
*arXiv preprint arXiv:2503.12797*, 2025.
- Nye et al. [2021]
M. Nye, A. J. Andreassen, G. Gur-Ari, H. Michalewski, J. Austin, D. Bieber, D. Dohan, A. Lewkowycz, M. Bosma, D. Luan, et al.
Show your work: Scratchpads for intermediate computation with language models.
*arXiv preprint arXiv:2112.00114*, 2021.
- OpenAI [2024]
OpenAI.
Gpt-4o system card.
*arXiv preprint arXiv:2410.21276*, 2024.
- Oquab et al. [2023]
M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, et al.
Dinov2: Learning robust visual features without supervision.
*arXiv preprint arXiv:2304.07193*, 2023.
- Qin et al. [2025]
Y. Qin, Y. Ye, J. Fang, H. Wang, S. Liang, S. Tian, J. Zhang, J. Li, Y. Li, S. Huang, et al.
Ui-tars: Pioneering automated gui interaction with native agents.
*arXiv preprint arXiv:2501.12326*, 2025.
- Rennie et al. [2017]
S. J. Rennie, E. Marcheret, Y. Mroueh, J. Ross, and V. Goel.
Self-critical sequence training for image captioning.
In *IEEE Conference on Computer Vision and Pattern Recognition*, pages 7008–7024, 2017.
- Rosenblat [1956]
M. Rosenblat.
Remarks on some nonparametric estimates of a density function.
*Ann. Math. Stat*, 27:832–837, 1956.
- Shaw et al. [2023]
P. Shaw, M. Joshi, J. Cohan, J. Berant, P. Pasupat, H. Hu, U. Khandelwal, K. Lee, and K. N. Toutanova.
From pixels to ui actions: Learning to follow instructions via graphical user interfaces.
*Advances in Neural Information Processing Systems*, 36:34354–34370, 2023.
- Snell et al. [2024]
C. Snell, J. Lee, K. Xu, and A. Kumar.
Scaling llm test-time compute optimally can be more effective than scaling model parameters.
*arXiv preprint arXiv:2408.03314*, 2024.
- Wang et al. [2024]
S. Wang, W. Liu, J. Chen, Y. Zhou, W. Gan, X. Zeng, Y. Che, S. Yu, X. Hao, K. Shao, et al.
Gui agents with foundation models: A comprehensive survey.
*arXiv preprint arXiv:2411.04890*, 2024.
- Wu and Xie [2024]
P. Wu and S. Xie.
V*: Guided visual search as a core mechanism in multimodal llms.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 13084–13094, 2024.
- Wu et al. [2024]
Z. Wu, Z. Wu, F. Xu, Y. Wang, Q. Sun, C. Jia, K. Cheng, Z. Ding, L. Chen, P. P. Liang, et al.
Os-atlas: A foundation action model for generalist gui agents.
*arXiv preprint arXiv:2410.23218*, 2024.
- Xia and Luo [2025]
X. Xia and R. Luo.
Gui-r1: A generalist r1-style vision-language action model for gui agents.
*arXiv preprint arXiv:2504.10458*, 2025.
- Xu et al. [2024]
Y. Xu, Z. Wang, J. Wang, D. Lu, T. Xie, A. Saha, D. Sahoo, T. Yu, and C. Xiong.
Aguvis: Unified pure vision agents for autonomous gui interaction.
*arXiv preprint arXiv:2412.04454*, 2024.
- Yu et al. [2024]
T. Yu, Y. Yao, H. Zhang, T. He, Y. Han, G. Cui, J. Hu, Z. Liu, H.-T. Zheng, M. Sun, et al.
Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 13807–13816, 2024.
- Zhang et al. [2025]
J. Zhang, Z. Ding, C. Ma, Z. Chen, Q. Sun, Z. Lan, and J. He.
Breaking the data barrier–building gui agents through task generalization.
*arXiv preprint arXiv:2504.10127*, 2025.
- Zhao et al. [2025]
H. Zhao, T. Chen, and Z. Wang.
On the robustness of gui grounding models against image attacks.
*arXiv preprint arXiv:2504.04716*, 2025.
- Zheng et al. [2024]
B. Zheng, B. Gou, J. Kil, H. Sun, and Y. Su.
Gpt-4v(ision) is a generalist web agent, if grounded.
In *Forty-first International Conference on Machine Learning*, 2024.
URL [https://openreview.net/forum?id=piecKJ2DlB](https://openreview.net/forum?id=piecKJ2DlB).

## Appendix A Experimental Details

In this section, we describe the experimental details of Section [4](https://arxiv.org/html/2505.15259v2#S4), including ReGUIDE and baselines.

### A.1 Dataset Details

In this section, we describe the dataset we used in training and evaluation.

- •
UGround Uground is a Graphical User Interface (GUI) grounding dataset that consists of 10M triplets of GUI image, element coordinates, and description of the element, which was synthesized by open MLLM, Llava-Next-13 B. Uground [11] models are trained with their own synthesized web GUI dataset, which consists of 10 M image-instruction pairs. For training ReGUIDE, we randomly select a 20K image-instruction set from UGround [11] dataset, which is approximately 0.2% of the original dataset.
- •
ScreenSpot ScreenSpot [7] is an evaluation benchmark dataset which consists of 1,272 (image, description, bounding box) triplets. There are various GUI domains in multiple device environments (Mobile, Desktop, Web) and UI element types (Text, Icon).
- •
ScreenSpot-v2 Wu et al. [35] refined the original ScreenSpot dataset by resolving ambiguous descriptions and aligning annotations more precisely with visual elements. We use both versions for a more accurate grounding evaluation.
- •
ScreenSpot-pro ScreenSpot-Pro [20] is a benchmark designed to evaluate GUI grounding models in professional, high-resolution environments. It spans 23 applications across five professional categories and three operating systems, highlighting the challenges models face when interacting with complex software..
- •
Android Control Android Control [21] is an offline agent benchmark comprising 15000 distinct user tasks sampled from 833 different Android applications. Each task contains a sequence of screenshots, detailed action records, and accessibility trees derived from human demonstrations (the “golden trajectory”). Each action is also annotated with a specific instruction (e.g., “set the hours to 6”), allowing both high-level and low-level task settings. In our experiments, we followed Gou et al. [11] and evaluated models on 500 randomly sampled steps from the test split.
- •
Multimodal-Mind2Web The Multimodal-Mind2Web [10] dataset contains screenshots with a large vertical dimension (e.g., 1280 x 10000 pixels). The test split includes 1,013 realistic user tasks collected from over 100 different websites, each accompanied by a high-level instruction and a sequence of action steps. To make the data manageable we followed [11], and divide the image vertically with certain amount of duplication. During the agent evaluation, when an agent cannot find an actionable element or explicitly chooses to scroll, the next block is processed, simulating user scrolling.

### A.2 Model Details

- •
Qwen-2.5-VL We use Qwen-2.5-VL [2] is a multimodal large language model (mllm) which inputs text and images and outputs text. The model is equipped with vision-language fusion modules and trained on diverse visual instruction datasets. We trained ReGUIDE from Qwen-2.5-VL-3B and 7B models.
- •
UGround UGround [11] is the model trained from Qwen-2-VL [2], there are two versions UGround-2B and UGround-7B. They are trained with a synthesized dataset that has 10M images, instructions, and target coordinates triplets. This amount of dataset size is about 500 times bigger than the ReGUIDE trained. For the fair comparison, we trained UGround on Qwen-2.5-VL [2] with the same subset used to train ReGUIDE. As shown in Table [1](https://arxiv.org/html/2505.15259v2#S4.T1), ReGUIDE excessively outperforms UGround for both.
- •
Proprietary Models Endpoint
–
gpt-4o-2024-05-13
–
claude-3-7-sonnet-20250219

### A.3 Training and Evaluation Details

- •
Training framework For the training framework, we utilize VERL for reinforcement learning and TRL for fine-tuning.
–
TRL (https://github.com/huggingface/trl)
–
VERL (https://github.com/volcengine/verl)
- •
Training hyperparameters. We describe our training hyperparameters, which we used for training, especially for learning to explaining GUI images via reasoning and learning to predict consistent coordinates under transformations.
Table 7: Hyperparameters for ReGUIDE on grpo training
Hyperparmeter
Value
Optimizer
Adam
Algorithm
GRPO
Learning rate
1e-6
Training data size
20k
Batch size
128
Generation per Sample
8
kl_coef
0.01
Table 8: Hyperparameters for ReGUIDE on training consistent coordinated under transformations
Hyperparmeter
Value
Optimizer
Adam
Learning rate
1e-6
Training data size
20k
Epoch
1
Batch size
64
minimum crop ratio
0.3

### A.4 Compute Resource

For the main development, we mainly use Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz and four A100 80GB GPUs. For training ReGUIDE, it took around 20 hours.

## Appendix B Additional Experimental Results

### B.1 Comparison with additional Baselines

**Table 9: Accuracy (%) for ReGUIDE (Ours) and other baselines, including UI-TARS [28], UI-R1 [23], GUI-R1 [36], and ReGUIDE (Ours). We evaluate on three web-grounding benchmarks: ScreenSpot [7], ScreenSpot-v2 [35], and ScreenSpot-Pro [20].**
| Methods | ScreenSpot [7] | ScreenSpot-v2 [35] | ScreenSpot-Pro [20] | Average |
| --- | --- | --- | --- | --- |
| UI-TARS-2B [28] | 82.3 | 84.7 | 27.7 | 64.9 |
| UI-TARS-7B [28] | 89.5 | 91.6 | 35.7 | 72.3 |
| UI-R1-E-3B [23] | 89.2 | 89.5 | 33.5 | 70.7 |
| GUI-R1-7B [36] | - | - | 31.3 | - |
| ReGUIDE-3B (Ours) | 87.3 | 89.7 | 44.3 | 73.8 |
| ReGUIDE-7B (Ours) | 90.1 | 91.5 | 44.4 | 75.3 |

We compare the grounding accuracy of ReGUIDE with other additional baselines, which are closed-source (UI-TARS [28]) and the concurrently proposed reinforcement learning methods for grounding UI-R1 [23] and GUI-R1 [36]. As shown in Table [9](https://arxiv.org/html/2505.15259v2#A2.T9) ReGUIDE achieves the highest average combined accuracy (75.3%) across ScreenSpot [7], ScreenSpot-V2 [35], and ScreenSpot-Pro [20].

Crucially, the compact ReGUIDE-3B already surpasses the best 7B-parameter base model (UI-TARS-7B) by +1.5 pp in average accuracy (72.3 % $\rightarrow$ 73.8 %), demonstrating that our training and search strategy substantially improves the GUI grounding performance. Moreover, both reinforcement fine-tuned baselines (i.e., UI-R1-E-3B, GUI-R1-7B) perform worse than ReGUIDE, underscoring the effectiveness of our further training for consistency under transformation and search algorithm. The full-sized ReGUIDE-7B further extends this lead to +1.5 pp, establishing a best among the given baselines on the combined benchmarks.

### B.2 Effectiveness of Kernel Density Estimation as majority voting strategy

**Table 10: Comparison of voting algorithm with grounding accuracy (%) on ScreenSpot [7] and ScreenSpot-Pro [20]**
| Methods | ScreenSpot [7] | ScreenSpot-Pro [20] |
| --- | --- | --- |
| Center | 80.3 | 20.2 |
| Medoid | 84.6 | 29.4 |
| KDE (Ours) | 87.3 | 44.3 |

To verify the most effective method for aggregating diverse pixel-wise coordinate predictions, we conducted an ablation study comparing our Kernel Density Estimation (KDE) based strategy against "Center" and "Medoid". "Center" is simply getting the mean point of the predicted coordinates. "Medoid" selects an actual predicted point that minimizes the sum of distances to all other predictions. As shown in Table [10](https://arxiv.org/html/2505.15259v2#A2.T10), KDE shows notable results compared with other voting strategies.

The performance gap can be explained by the statistical properties of each strategy.
First, the "Center" is vulnerable to outliers, a few erroneous predictions can shift the center far from the true target. Also, "Medoid" is robust to outliers but ignores local density, so it may choose a point in a sparse region when the prediction distribution is multi-modal.
However, KDE can overcome these limitations. KDE estimating the underlying likelihood surface, naturally capturing local density peaks while down-weighting isolated outliers. This behaviour aligns with our empirical observation that model confidence around the true GUI element is approximately Gaussian (Figure [2](https://arxiv.org/html/2505.15259v2#S4.F2)), making KDE a principled and effective spatial voting rule.

### B.3 Effect of consistency finetuning after reinforcement stage

**Table 11: Effect of consistency-under-transformation finetuning on ReGUIDE-3B, evaluated on the ScreenSpot [7] benchmark.**
| Method | Augmentation usage | Accuracy (%) |
| --- | --- | --- |
| RL w/o Aug. | – | 83.3 |
| RL w/ Aug. | In RL stage | 83.2 |
| ReGUIDE (Ours). | fine-tune | 84.9 |

We assess the benefit of our *consistency-under-transformation* finetuning on the ReGUIDE-3B model using the ScreenSpot benchmark. As shown in Table [11](https://arxiv.org/html/2505.15259v2#A2.T11), training GRPO on the original screenshots only (RL w/o Aug.) achieves 83.3 % accuracy, while mixing the same augmented views directly into the RL phase (RL w/ Aug) yields no improvement (83.2 %). In contrast, our two-stage pipeline—first training GRPO without augmentation and then applying a separate consistency finetune on the augmented views (ReGUIDE)—raises accuracy to 84.9 %, a gain of +1.6 pp over the vanilla RL model and +1.7 pp over RL w/ Aug. This result confirms that the performance boost arises from explicitly learning to align predictions across transformed views, not from data augmentation alone.

### B.4 Effect of consistency finetuning on transformer attention

Figure: (a) GRPO trained
Refer to caption: extracted/6474733/images/attention_grpo.png

To assess how the consistency-under-transformation stage alters internal representations, we visualised the average attention logits of the last transformer layer in ReGUIDE-3B. We first removed turn-delimiter and <eos> tokens whose weights dominate the map, and then averaged across all heads. As shown in Figure [4](https://arxiv.org/html/2505.15259v2#A2.F4), the GRPO-only model weights most of its attention onto the reasoning span, indicating limited contextual grounding. In contrast, the full ReGUIDE model, after consistency finetuning, distributes attention more evenly across reasoning (i.e., text tokens between <think> and </think>) and other text tokens, suggesting richer spatial awareness and reduced attention collapse.

### B.5 Dataset Scalable Experiment

Figure: Figure 5: Grounding accuracy (%) on ScreenSpot [7] as the UGround [11] training set grows from 3.2 K to 20 K samples. We trained from Qwen-2.5-VL-3B.
Refer to caption: x4.png

To measure how our reinforcement-learning stage benefits from more data, we varied the amount of UGround [11] training samples from 3.2K up to 20K in 3.2K increments and evaluated grounding accuracy (%) on the ScreenSpot [7] benchmark (Figure [5](https://arxiv.org/html/2505.15259v2#A2.F5)). Accuracy rises monotonically with data size, and the full 20K subset yields the best performance. These results indicate that the ReGUIDE training pipeline scales effectively with additional data and has not yet saturated at the 20K level.

## Appendix C Illustration of Searching Strategy.

Figure: Figure 6: Overview of ReGUIDE’s searching strategy.. Starting from the full screenshot, the model samples $N$ coordinate predictions, votes using kernel density estimation, and recenters a crop on the density peak. One searching pass is then run inside the crop, and the point with the highest KDE density is returned as the final prediction.
Refer to caption: extracted/6474733/images/Inference_concept.png

## Appendix D Alternative RL policy optimization algorithms

**Table 12: Comparison of RL algorithms. We report accuracy (%) and average output length (Output Len.) on ScreenSpot Benchmark [7].**
| Methods | Accuracy (%) | Output Len. |
| --- | --- | --- |
| REINFORCE++ [16] | 64.8 | 32.3 |
| RLOO [1] | 77.8 | 95.7 |
| GRPO [9] | 83.3 | 80.8 |

While we primarily adopt GRPO [9] for the learning to visual reasoning, alternative RL algorithms, such as RLOO [1], REINFORCE++ [16], could also be considered. To verify the performance difference between RL poly optimization algorithms, we compared grounding accuracy (%) on ScreenSpot. As shown in the Table [12](https://arxiv.org/html/2505.15259v2#A4.T12), GRPO achieves the highest accuracy (83.3%), followed by RLOO (77.8%), while REINFORCE++ significantly underperforms (64.8%). Interestingly, we observed that the REINFORCE++ trained model loses its reasoning ability (i.e., shortest output length), eventually failing to generate meaningful reasoning paths.

## Appendix E Learning Curve

We report the learning curve of ReGUIDE during training GRPO. The base model of the training curve is Qwen-2.5-VL-3B. We can see that the reward is kept increasing during the training. Also, the mean response length has decreased during the training.

Figure: (a) Mean reward score per step
Refer to caption: extracted/6474733/images/critic_rewards_mean.png

## Appendix F Prompts

We describe our generation prompt for ReGUIDE (in Table [13](https://arxiv.org/html/2505.15259v2#A6.T13) and UGround [11] (in Table [14](https://arxiv.org/html/2505.15259v2#A6.T14). We used the ReGUIDE prompt for training and evaluation ReGUIDE, and the UGround prompt for supervised fine-tuning the Qwen-2.5-VL for comparison and evaluating the UGround model.

**Table 13: Prompt of ReGUIDE**
| Element: Target element instruction.<br>size_x: Width of the given image.<br>size_y: Height of the given image. |
| --- |
| What is the coordinate of [{Element}] in the image?<br>The size of image is ({size_x},{size_y}).<br>Output the thinking process in <think> </think> and final answer (coordinate (x,y)) in <answer> </answer> tags. |

**Table 14: Prompt of UGround**
| Element: Target element instruction. |
| --- |
| Your task is to help the user identify the precise coordinates (x, y) of a specific area/element/object on the screen based on a description.<br>- Your response should aim to point to the center or a representative point within the described area/element/object as accurately as possible.<br>- If the description is unclear or ambiguous, infer the most relevant area or element based on its likely context or purpose.<br>- Your answer should be a single string (x, y) corresponding to the point of the interest.<br>Description: {Element}<br>Answer: |

## Appendix G Generation Examples

**Table 15: Examples of ReGUIDE’s generated reasoning and predicted coordinate.**
|  |
| --- |
| Target Prompt: add description.<br>Response: <think>To find the coordinate of [add description], I need to locate the element labeled Ädd descriptionïn the image. The coordinates are given as (x, y) pairs, and I need to identify the position of this element.</think><answer>(300,2221) </answer> |
|  |
| Target Prompt: check weather on tuesday.<br>Response: <think> To determine the coordinate of the n̈lpf̈older, I need to locate it in the image and identify its position relative to the other folders. The n̈lpf̈older is the second folder in the top row, as indicated by its position in the grid layout. </think> <answer>(584,760)</answer> |
|  |
| Target Prompt: sign up for newsletter.<br>Response: <think> The task is to find the coordinate of the S̈ign Up for Newsletterc̈heckbox in the image. The checkbox is located in the P̈ersonal Informations̈ection, and it is a small square with a label next to it. The coordinates are given in a specific size of the image, and we need to identify the position of the checkbox within the image. </think> <answer>(367,85)</answer> |
|  |
| Target Prompt: start mapping.<br>Response: <think> The S̈tart Mappingb̈utton is typically used to begin the process of adding or editing data on the map. In the image, it is located in the lower left section, which is a common placement for such buttons on web interfaces. </think> <answer>(558,805)</answer> |

## Appendix H Societal Impact

This paper presents ReGUIDE, a method that significantly improves GUI-grounding accuracy. We expect that our approach will enhance digital accessibility and productivity by enabling agents to locate and interact with on-screen elements more reliably. However, the same capability could be misused for malicious purposes, such as automated hacking or spreading misinformation, issues that current agent models already face. Mitigation measures include responsible release practices, access controls, and detection tools to discourage adversarial deployments.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: Please see weaknesses.
soundness: 2
strengths: - Data Efficiency. This is the paper's primary contribution and it is highly significant. Achieving nice performance while training on only 20k samples versus 10M is a major step forward for the field, making high-performance grounding more accessible.
- Effective Inference-Time Search. The test-time scaling strategy, which uses KDE for an initial vote, crop, and then vote again, is well-motivated and empirically powerful. The paper shows this is particularly effective for high-resolution images.
summary: This paper addresses the critical bottleneck of data inefficiency in training MLLMs for GUI grounding. The authors propose ReGUIDE, a novel framework that achieves state-of-the-art results using a very small fraction--0.2% of the data required by existing baselines. The method's novelty lies in its two-stage training process, which first uses online reinforcement learning to reward accurate coordinate prediction, compelling the model to self-generate its own language-based reasoning without explicit supervision. This is followed by another stage that enforces spatial consistency between global and cropped views of the image. The framework is further enhanced by an effective test-time scaling strategy that uses spatial search and Kernel Density Estimation to iteratively refine coordinate predictions. Experiments across multiple benchmarks, including high-resolution and agentic tasks, demonstrate significant performance gains, highlighting the method's effectiveness in learning robust spatial priors from minimal data.
weaknesses: - Two very relevant work, Aria-UI (for GUI grounding SFT) and GTA-1 (for GUI RL training with GRPO) is not discussed nor compared in the paper.
- From Tab.5, the proposed two test-time scaling strategies play important role in model's performance with SS and SS-pro. The questions here would be:
1) since we may easily move the two strategies to existing models like UGround, will they benefit from it?
2) since without the two strategies, the proposed model generally performs on par with the baselines, and considering the authors only used 0.2% of the data, can scaling law in terms data still apply to ReGUIDE models? For example, if using 10x or 100x of the data, how would ReGUIDE benefit from it?

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 4
questions: **1. On the SOTA claim and leaderboard comparison:**

The paper reports state-of-the-art results on ScreenSpot-Pro. However, several stronger 7B baselines have already been listed on the public leaderboard (https://gui-agent.github.io/grounding-leaderboard/), such as GUI-ARP-7B (60.8%), Holo1.5-7B (57.9%), GTA1-7B (55.5%), and GUI-Cursor-7B (56.5%).

It would be helpful if the authors could clarify how ReGUIDE compares with these concurrent approaches and discuss their methodological differences to better position the contribution of this work.

**2. On inference latency and deployability:**

The two-stage test-time search appears to increase inference time by about twofold. Could the authors share their perspective on how this additional latency might affect real-world deployment, particularly for GUI agents requiring real-time interaction? 

**3. On isolating training and inference gains:**

The core contribution is argued to be Data Efficient Grounding achieved through the proposed training paradigm. However, the reported SOTA results include the significant gains from the Test-Time Scaling. To ensure a transparent analysis of data efficiency and computational trade-offs, could the authors please report ReGUIDE's performance in the main results before and after applying the Test-Time Scaling? This separation would clearly isolate the performance improvement attributable solely to the data-efficient training methods.

**4. On handling high-resolution inputs:**

The Cropping stage contributes notably to the gains on the high-resolution ScreenSpot-Pro benchmark, suggesting that the base model may still struggle with large input resolutions. Could the authors elaborate on whether the Consistency training helps mitigate this issue by learning a more robust, scale-invariant representation internally, or whether the improvement mainly depends on the external inference-time Cropping process?
soundness: 3
strengths: **1. Novel Integrated Data-Efficient Framework:**

The paper proposes a highly novel framework, *ReGUIDE*, that tackles the GUI grounding problem with exceptional data efficiency. Its originality lies in the synergistic integration of components across both training and inference: reinforcement learning for self-generated language reasoning, a subsequent training stage enforcing spatial consistency under transformations, and a final test-time spatial search with KDE-based aggregation. This comprehensive combination effectively addresses the spatial reasoning and localization precision limitations of prior approaches.

**2. Strong and Data-Efficient Performance:**

ReGUIDE achieves significant performance improvements, setting a new competitive standard across multiple challenging GUI grounding benchmarks, including ScreenSpot and ScreenSpot-Pro. Notably, it demonstrates remarkable data efficiency, reaching state-of-the-art results while using only a small fraction of the data (a 20K subset, or 0.2% of samples) compared to prior methods trained on millions of examples.

**3. Comprehensive Empirical Evaluation and Insights:**

The paper presents extensive and rigorous experiments, including systematic ablation studies that isolate the contribution of each training and inference component. Moreover, it validates practical utility through strong transfer performance on downstream agentic tasks such as Multimodal-Mind2Web and AndroidControl, further supporting the empirical soundness of the proposed approach.

**4. Clear Presentation and Reproducibility:**

The paper is well written and well organized, featuring clear figures and structured technical explanations. It provides valuable insights into the model’s internal mechanisms through attention map visualization and analysis of self-generated reasoning pathways. The inclusion of complete experimental details, hyperparameter settings, and the stated plan to release code enhances the clarity and reproducibility of the work.
summary: This paper presents ReGUIDE, a data-efficient framework for GUI coordinate grounding that enables multimodal language models to accurately localize interface elements. ReGUIDE employs a two-stage training approach: (1) self-generating reasoning through online reinforcement learning using grounding accuracy as reward, and (2) enforcing spatial consistency between global and local image views using transformation-based priors. At inference, a test-time search strategy combines spatial cropping with KDE-based coordinate aggregation. Using only 20K samples (≈0.2% of UGround), ReGUIDE outperforms prior models, improving Qwen-2.5-VL-3B accuracy from 55.5% to 88.0% on ScreenSpot and from 23.9% to 44.5% on ScreenSpot-Pro, demonstrating significant data efficiency gains.
weaknesses: **1. Lack of comparison with other RL-based methods:**  
For instance, *UI-AGILE-7B*, which also employs reinforcement learning with only 9K examples, achieves 48.7% accuracy on ScreenSpot-Pro when initialized from Qwen2.5-VL, which is comparable to or even surpasses ReGUIDE’s results under a smaller data regime. It would be helpful to discuss this comparison to clearly highlight the differences between the two approaches.

**2. Inference latency:**  
ReGUIDE’s performance gain largely depends on its two-stage test-time spatial search. While effective, this procedure increases inference time by approximately 2.0 to 2.2 times (Table 13), introducing a relatively high latency overhead for real-time or interactive GUI agents. This raises concerns about its practicality in latency-sensitive scenarios where fast responses are critical.

**3. Entanglement of test-time scaling and training effects:**  
The main results combine the gains from both the training components (RL + Consistency) and the inference enhancements (test-time scaling), making it difficult to isolate the true contribution of the proposed data-efficient training. Since other baselines are evaluated using single-pass inference, a fair comparison requires reporting ReGUIDE’s performance both with and without test-time scaling to clarify how much of the impro

---

confidence: 5
contribution: 2
initial_rating: 4
presentation: 3
questions: See above
soundness: 3
strengths: 1. The paper provides an extensive and comprehensive evaluation of ReGUIDE across multiple benchmarks.
2. The writing and organization are clear, coherent, and well-structured.
3. The proposed training and testing time scaling strategies are effective, and the combination of global–local search with voting demonstrates strong performance.
summary: This paper presents ReGUIDE, a data-efficient GUI grounding framework for MLLM-based web agents. It combines reasoning-guided self-generation and spatial-aware criticism to improve coordinate localization without large datasets. A test-time spatial scaling further enhances accuracy. Compared with exisiting grounding models, ReGUIDE perfroms well on extensive expermients results.
weaknesses: 1. The main concern with this work lies in whether the proposed model truly achieves state-of-the-art grounding performance. In fact, several prior studies have already reported superior results. However, the authors did not include these stronger baselines (e.g., [1–3]) in their benchmark comparisons.

2. The ensuing concern is that the related work does not encompass the latest developments.

3. The results presented in Table 7 seem somewhat counterintuitive. First, it is unclear why five data benchmarks are listed, as in reality there are only two. Second, regarding AC benchmark, it is puzzling that the High-level model outperforms the Low model. Finally, it remains unclear how the experimental setup allows web agents to achieve such high performance on mobile tasks. If additional fine-tuning was applied, the reported task completion rate still appears lower than that of some existing mobile agents.

4. The relationship between the effect of consistency fine-tuning and attention–grounding modeling remains unclear. The authors should further clarify how these changes contribute to improved grounding performance. 

5. The analysis of the natural Gaussian distribution of coordinate token likelihood appears to yield conclusions similar to those reported in GUI-G2. Therefore, it is strongly recommended that the authors compare their findings directly with GUI-G2, clarifying any differences in methodology, experimental setup, and quantitative outcomes. 

**Reference**

[1] GTA1: GUI Test-time Scaling Agent

[2] GUI-G2: GAUSSIAN REWARD MODELING FOR GUI GROUNDING

[3] Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learning


Decision: Reject