Learning GUI Grounding with Spatial Reasoning from Visual Feedback

## Abstract

Abstract Graphical User Interface (GUI) grounding is commonly framed as a coordinate prediction task – given a natural language instruction, generate on-screen coordinates for actions such as clicks and keystrokes.
However, recent Vision Language Models (VLMs) often fail to predict accurate numeric coordinates when processing high-resolution GUI images with complex layouts.
To address this issue, we reframe GUI grounding as an interactive search task , where the VLM generates actions to move a cursor in the GUI to locate UI elements.
At each step, the model determines the target object, evaluates the spatial relations between the cursor and the target, and moves the cursor closer to the target conditioned on the movement history.
In this interactive process, the rendered cursor provides visual feedback to help the model align its predictions with the corresponding on-screen locations.
We train our GUI grounding model, Gui-Cursor , using multi-step online reinforcement learning with a dense trajectory-based reward function.
Our experimental results show that Gui-Cursor , based on Qwen2.5-VL-7B, improves the GUI grounding accuracy and achieves state-of-the-art results on ScreenSpot-v2 ( 88.8 % → 93.9 % 88.8\%\rightarrow 93.9\% ) and ScreenSpot-Pro ( 26.8 % → 56.5 % 26.8\%\rightarrow 56.5\% ). Moreover, we observe that Gui-Cursor learns to solve the problem within two steps for 95% of instances and can adaptively conduct more steps on more difficult examples.

## 1 Introduction

Graphical User Interface (GUI) agents solve tasks by translating user instructions into multiple GUI interaction steps (Deng et al., 2023; Li et al., 2024; Zhang et al., 2025a).
A core component of these agents is GUI *grounding*, i.e., identifying a specific pixel coordinate to execute an action, like a click or keystroke.
While early approaches rely on extracting coordinates from textual representations of GUIs (e.g., HTML and DOM trees), such data is often redundant and is not consistently available across platforms (Zhang et al., 2025c; b; Wu et al., 2025b).
Thus, recent research emphasises pure vision-driven GUI grounding that perceives the interface directly from rendered screenshots (Xu et al., 2024; Gou et al., 2025; Yuan et al., 2025).

However, accurately predicting the precise numerical coordinates of an element on a high-resolution GUI image is a significant challenge for current vision-language models (VLMs).
This difficulty largely arises from the problem of *spatial semantic alignment* (Wu et al., 2025b), which requires a language model to generate discrete coordinate tokens by implicitly mapping its understanding of a visual element to a specific set of coordinates on the GUI image.
Current methods, whether based on supervised fine-tuning (Gou et al., 2025; Wu et al., 2024; Xu et al., 2024), reinforcement learning (Lu et al., 2025; Yuan et al., 2025; Liu et al., 2025b; Yang et al., 2025a), or architectural modifications (Wu et al., 2025b), commonly frame GUI grounding as a *one-step coordinate prediction problem during training*: given a screenshot, the VLM is tasked with generating the coordinates of the target.
One potential limitation of this one-step learning paradigm is the *absence of a visual feedback loop*: the model is only supervised on the generated numerical coordinates, but does not receive any information on where its prediction actually lands on the GUIs.
As a consequence, the model may fail to develop a robust alignment between its numerical predictions and their corresponding GUI elements.

To address this limitation, we propose Gui-Cursor, which reframes GUI grounding as a search task in an interactive environment ([Section 2.2](https://arxiv.org/html/2509.21552v1#S2.SS2)), as shown in [Fig. 1](https://arxiv.org/html/2509.21552v1#S1.F1).
During training, Gui-Cursor operates by controlling a virtual cursor on a GUI image; at each step, the model performs spatial reasoning to determine if the cursor is positioned over the target GUI element.
If the model determines that the cursor is not positioned over the target, it predicts a new position, and the environment provides an updated image of the GUI with the cursor located in the new position.
This iterative process terminates when the model determines the cursor is correctly positioned on the target UI element, or when a maximum number of cursor moving steps is reached.
This interactive process during training improves GUI grounding in two key ways:
1) the rendered cursor provides *explicit visual feedback* that shows the predicted position on the GUI image, helping the model better align numerical coordinates with their corresponding GUI elements during training; and
2) this interactive environment allows the model to conduct spatial reasoning based on its visual feedback, allowing it to incrementally refine its predictions at inference time.

We train the model using reinforcement learning (RL) in this interactive environment, as it provides a natural framework for this sequential decision-making task.
We design a reward function that combines a position-based reward with several trajectory penalties that discourage wrong or inefficient search processes ([Section 2.3](https://arxiv.org/html/2509.21552v1#S2.SS3)).
Our ablation studies ([Section 3.2](https://arxiv.org/html/2509.21552v1#S3.SS2)) show that these penalties are helpful to prevent degenerate behaviour and significantly improve the downstream accuracy of the model.
Furthermore, to improve the efficiency of this interactive grounding process, we introduce a strategy that balances training cost and inference accuracy ([Section 2.4](https://arxiv.org/html/2509.21552v1#S2.SS4)).

Figure: Figure 1: The workflow of our method Gui-Cursor. The left part presents an example where Gui-Cursor locates the target successfully: at the first step, the model correctly describes the shape of the target and predicts a coordinate; at the second step, the model evaluates the spatial relationship between the cursor and the target, and it finds the cursor is not correctly positioned, then provides a new position; at the third step, the model terminates the interaction. The right part presents the examples of incorrect or inefficient moving trajectories penalised in our reward function.
Refer to caption: x1.png

Experimental results show that Gui-Cursor, based on Qwen2.5-VL-7B (Bai et al., 2025), achieves a new state-of-the-art (SOTA) performance on ScreenSpot-v2 (Cheng et al., 2024) ($88.8\%\rightarrow 93.9\%$) and ScreenSpot-Pro (Li et al., 2025b) ($26.8\%\rightarrow 56.5\%$), surpassing the prior leading model GTA1 (Yang et al., 2025a) by $6.4\%$ on the more challenging ScreenSpot-Pro benchmark.
In terms of computational efficiency, the fine-tuned model learns to solve the problem within two steps for 95% of the instances and adaptively conducts more steps on difficult tasks, such as locating UI elements with small sizes.
Besides, we test the spatial reasoning capability of the model around a cursor by asking it to classify the spatial relationship between a cursor and another object in a clean background.
We observe that even the strong base model struggles in this simple task, which suggests its grounding ability may not be founded on a robust spatial understanding of images.
Gui-Cursor obtains a higher accuracy without being explicitly trained on this task, indicating that training in an interactive environment has the potential to improve a general spatial understanding capability.

Our work makes the following contributions:

- 1.
We reformulate GUI grounding from the static task to an interactive process by moving a cursor, enabling the model to learn coordinate-spatial alignment from visual feedback during training.
- 2.
We introduce Gui-Cursor, a GUI grounding model trained with reinforcement learning in the interactive environment, using a dense, trajectory-based reward function;
- 3.
our experiments show that Gui-Cursor achieves a new SOTA on ScreenSpot-v2 ($93.9\%$) and ScreenSpot-Pro ($56.5\%$), and the analysis further shows that it learns an adaptive strategy to conduct more steps movement on difficult tasks.

## 2 Interactive GUI Grounding with Visual Feedback

We introduce Gui-Cursor, a method that learns GUI grounding by moving a virtual cursor in an interactive environment.
This interactive environment provides explicit visual feedback through the cursor to help the model better align the coordinate and its actual spatial position on the image.
During inference, it also allows the model to refine the prediction according to the spatial relation of the target and the cursor.
We train this iterative behaviour using reinforcement learning, as it provides a natural framework for optimising this sequential decision-making process.
In the following, we introduce the process of GUI grounding by moving a cursor ([Section 2.2](https://arxiv.org/html/2509.21552v1#S2.SS2)), reward function modelling ([Section 2.3](https://arxiv.org/html/2509.21552v1#S2.SS3)), and a strategy to balance training efficiency and inference accuracy ([Section 2.4](https://arxiv.org/html/2509.21552v1#S2.SS4)).

### 2.1 Problem Formulation

The task of GUI grounding is to map a natural language instruction to a target coordinate on a GUI.
Formally, given a GUI screenshot $O$ with $W\times H$ pixels and a natural language instruction $I$ describing a target element, the goal is to predict an integer coordinate pair $(x,y)$, with $x,y\in\mathbb{N}_{0}$, where an action should be performed at this pixel.
The ground truth is the rectangular area defined by the bounding box $B$. Given the top-left corner $(x_{\min},y_{\min})$ and the bottom-right corner $(x_{\max},y_{\max})$, this area is the set of all points $(x,y)$ such that: $B=\{(x,y){\mid}x_{\min}\leq x\leq x_{\max},y_{\min}\leq y\leq y_{\max}\}$. The prediction $(x,y)$ is correct when $(x,y)\in B$.

### 2.2 GUI Grounding by Moving a Cursor

We design an interactive environment that allows the model to locate the GUI element by moving a cursor.
The interaction process consists of a sequence of steps.
At the initial step $t=0$, the screenshot $O_{0}$ displays with a cursor at its centre $(x_{0},y_{0})$; conditioned on a natural language instruction $I$ and the observation $O_{0}$, the model generates the response $A_{0}$ that contains an action to move the cursor to a position $(x_{1},y_{1})$.
At each subsequent step $t$, the new observation $O_{t}$ displays the cursor at the latest position $(x_{t},y_{t})$, and the model generates the response $A_{t}$ conditioned on the interaction history and the new observation: $(I,O_{0},A_{0},O_{1},A_{1}\dots,A_{t-1},O_{t})$.
Each response $A_{t}$ consists of a thinking process $\{w_{i}\}_{i=1}^{n}$ before an action $\{v_{i}\}_{i=1}^{m}$:

$$ $A_{t}=\text{$<$think$>$}w_{1},w_{2},\dots,w_{n}\text{$<$/think$>$}\text{$<$answer$>$}v_{1},v_{2},...,v_{m}\text{$<$/answer$>$}$ (1) $$

#### Thinking

The thinking tokens $\{w_{i}\}_{i=1}^{n}$ contain the analysis about the target UI element, the current cursor’s location, and the spatial relationship between the cursor and the target.

#### Action

The answer tokens $\{v_{i}\}_{i=1}^{m}$ can be either a new coordinate prediction $\{v_{i}\}_{i=1}^{m}=(x_{t},y_{t})$, or $\{v_{i}\}_{i=1}^{m}=\text{STOP}$ when the model judges the cursor is correctly on the target.
If the new coordinate $(x_{t},y_{t})$ is provided, the cursor will be rendered at the position $(x_{t},y_{t})$ for the next turn observation $O_{t+1}$; otherwise, the position of the cursor will not be updated.

The episode terminates when the model outputs STOP (i.e., $\{v_{i}\}_{i=1}^{m}=\text{STOP}$) or a pre-defined maximum number of steps is reached.
Then, the final position $(x_{T},y_{T})$ of the cursor is returned as the grounding prediction, where $T$ is the number of steps.
[Fig. 1](https://arxiv.org/html/2509.21552v1#S1.F1) illustrates how the model locates the target by moving a cursor in this interactive environment.

### 2.3 Trajectory Reward Modelling

Our reward function aims to guide the model to obtain both an accurate final prediction and a rational search behaviour.
To achieve these goals, we introduce 1) a position-based reward to minimise the distance between the final position and the target, and 2) several trajectory penalties to discourage unreasonable search processes.
Existing GUI grounding methods that use RL frameworks (Luo et al., 2025; Yang et al., 2025a; Yuan et al., 2025) mainly apply a position-based reward.
Differently, regulating the search process is important in our multi-step interaction approach, because it helps to prevent degenerate strategies, e.g., immediately stopping without using visual feedback or repeatedly moving to a visited position.
In the following, we introduce the position reward and trajectory penalties used in Gui-Cursor.

#### Position Reward

This reward measures the quality of the final cursor position, $(x_{T},y_{T})$, relative to the target bounding box $B$.
We adopt the dense distance reward used by SE-GUI (Yuan et al., 2025), which considers both the distance to the box and centrality within it:

$$ $r_{p}=\begin{cases}1+\left(1-\frac{d_{\text{centre}}\left((x_{T},y_{T}),B\right)}{d_{\max}(B)}\right)^{2}&\text{if }(x_{T},y_{T})\in B\\ 1-d_{\text{edge}}\left((x_{T},y_{T}),B\right)&\text{otherwise},\end{cases}$ (2) $$

where $d_{\text{edge}}\left((x,y),B\right)$ is the Euclidean distance of $(x,y)$ to the nearest edge of $B$, $d_{\max}(B)$ is the distance of the vertex of $B$ to the centre of $B$, and $d_{\text{centre}}\left((x,y),B\right)$ is the distance to the centre of $B$.
We normalise all distances by the width and height of the image.
In [Eq. 2](https://arxiv.org/html/2509.21552v1#S2.E2), the position reward is $1-d_{\text{edge}}$ if $(x_{T},y_{T})$ is outside $B$, increasing with proximity to the centre when inside $B$.

#### Trajectory Penalties

While the position reward defines a final goal, it provides no guidance on the search process.
To guide the model in learning a rational search strategy, we introduce four trajectory-based penalties that target specific undesirable behaviours.
We present the examples of the penalised trajectories in the right part of [Fig. 1](https://arxiv.org/html/2509.21552v1#S1.F1).

False Stop Penalty ($r_{\text{FS}}$): Checks if the model outputs STOP but the final cursor position $p_{T}$ is outside the target box $B$:

$$ $r_{\text{FS}}=\mathbb{I}[A_{T}=\texttt{STOP}\land(x_{T},y_{T})\notin B].$ (3) $$

False Move Penalty ($r_{\text{FM}}$): Checks if there is a history position inside the target box, but the model does not output STOP and the final position $p_{T}$ is outside of it:

$$ $r_{\text{FM}}=\mathbb{I}[(\exists t<T\text{ s.t. }(x_{t},y_{t})\in B)\land((x_{T},y_{T})\notin B)].$ (4) $$

False Direction Penalty ($r_{\text{FD}}$): Checks if the final position $(x_{T},y_{T})$ is further from the target box than the initial prediction $(x_{1},y_{1})$ was, in which case the movement does not shorten the distance between the prediction and the target:

$$ $r_{\text{FD}}=\mathbb{I}[d_{\text{edge}}((x_{T},y_{T}),B)>d_{\text{edge}}((x_{1},y_{1}),B)].$ (5) $$

Repeated Position Penalty ($r_{\text{RP}}$): Checks if the model predicts the same coordinate more than once:

$$ $r_{\text{RP}}=\mathbb{I}[\exists i,j\in{1,\dots,T}\text{ s.t. }i\neq j\land(x_{i},y_{i})=(x_{j},y_{j})].$ (6) $$

Then, we define the trajectory reward $R_{\text{T}}$ as a weighted combination of the position reward $r_{p}$ and a sum of the trajectory penalties $r_{\text{FS}}$, $r_{\text{FM}}$, $r_{\text{FD}}$, and $r_{\text{RP}}$ weighted by an hyper-parameter $w_{\text{p}}$:

$$ $R_{\text{T}}=r_{p}-w_{\text{p}}\left(r_{\text{FD}}+r_{\text{FS}}+r_{\text{FM}}+r_{\text{RP}}\right).$ (7) $$

#### Training Objective

We use Group Relative Policy Optimisation (GRPO) to optimise the interaction policy guided by the trajectory reward $R_{T}$ and a format reward to ensure the valid output format DeepSeek-AI et al. (2025).
GRPO has been successfully applied in prior RL-based GUI grounding methods (Luo et al., 2025; Yuan et al., 2025; Yang et al., 2025a), and we also use it for its advantages in training stability and efficiency.
More details are presented [Section B.1](https://arxiv.org/html/2509.21552v1#A2.SS1).

### 2.4 Cursor-Centric Focusing

The iterative nature of Gui-Cursor could be computationally demanding, as it processes a growing sequence of interaction history $(I,O_{0},A_{o},\dots,A_{t-1},O_{t})$ to generate an action.
This becomes infeasible when handling native high-resolution screenshots.
To alleviate this issue, we use a two-part strategy that balances computational efficiency with predictive accuracy.

During training, we downscale the large image to a manageable resolution $P$ (e.g., $1920\times 1080$ pixels in our experiments), preserving the original aspect ratio.
This allows the model to learn the iterative grounding task without a heavy computational burden of processing large images.
Although this may increase the grounding difficulty when searching for small UI elements, it effectively trains the model to approximate the target’s location.

During inference, we employ a cursor-centric focusing strategy (ccf) for any image larger than the training resolution.
ccf begins with a single step to get an initial coarse prediction on the full image.
Then, it crops a $P$-size area centred on this initial prediction, thereby positioning the cursor at the centre of this focused view; in the following steps, the model conducts fine-grained movement within the cropped area.
Here, $P$ is the maximum resolution during training, and the following moving steps will not include the original large image in the interaction history.

This combined strategy enables Gui-Cursor to learn a general interaction policy without a heavy computational burden, while performing accurate inference on higher-resolution displays.

## 3 Experiments

### 3.1 Experiment Settings

#### Implementation Details

We implement Gui-Cursor using Qwen2.5-VL-7B (Bai et al., 2025) as the base model. We train Gui-Cursor with a maximum of 250 steps. The learning rate is $10^{-6}$, the batch size is $32$, and $12$ sample moving trajectories for each instruction.
We set the maximum moving steps to 4 during training.
Following Yang et al. (2025a), we use GUI grounding datasets from Aria-UI (Yang et al., 2025b) and OS-Atlas (Wu et al., 2024), employing the same filtering and preprocessing scripts.
We randomly sample data to train the model.
We apply *online filtering* (Cui et al., 2025) to filter out training examples when all sampled trajectories either successfully or unsuccessfully locate the target — a common data selection strategy to remove overly easy or difficult examples for online policy models.
More implementation details are available in [Appendix B](https://arxiv.org/html/2509.21552v1#A2).

#### Evaluation Benchmarks and Baseline Models

We evaluate our method on the widely used GUI grounding benchmarks: ScreenSpot-V2 (Cheng et al., 2024; Wu et al., 2024) and ScreenSpot-Pro (Li et al., 2025b).
We compare Gui-Cursor against GUI grounding models optimised by supervised fine-tuning: SeeClick (Cheng et al., 2024), UGround (Gou et al., 2025), OS-Atlas (Wu et al., 2024), UI-TARS (Qin et al., 2025) and GUI-Actor (Wu et al., 2025b), and reinforcement learning: UI-R1 (Lu et al., 2025), GUI-R1 (Luo et al., 2025), InfiGUI-R1 (Liu et al., 2025b), SE-GUI (Yuan et al., 2025), LPO (Tang et al., 2025b), GTA1 (Yang et al., 2025a), and GUI-G^2 (Tang et al., 2025a).

**Table 1: ScreenSpot-v2 accuracy (%) for text and icon/widget grounding across mobile, desktop, and web interfaces. Gui-Cursor achieves the highest average score and improves text and icon grounding over both supervised and RL-based baselines.**
| Model | Mobile | Desktop | Web | Average |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Text | Icon/Widget | Text | Icon/Widget | Text | Icon/Widget |  |  |
| Supervised Fine-Tuning Methods |  |  |  |  |  |  |  |
| SeeClick (Cheng et al., 2024) | 78.4 | 50.7 | 70.1 | 29.3 | 55.2 | 32.5 | 55.1 |
| OmniParser-v2 (Lu et al., 2024) | 95.5 | 74.6 | 92.3 | 60.9 | 88.0 | 59.6 | 80.7 |
| OS-Altlas-7B (Wu et al., 2024) | 95.2 | 75.8 | 90.7 | 63.6 | 90.6 | 77.3 | 84.1 |
| UGround Gou et al. (2025) | 95.0 | 83.3 | 95.0 | 77.8 | 92.1 | 77.2 | 87.6 |
| UI-TARS-7B (Qin et al., 2025) | 96.9 | 89.1 | 95.4 | 85.0 | 93.6 | 85.2 | 91.6 |
| UI-TARS-72B  (Qin et al., 2025) | 94.8 | 86.3 | 91.2 | 87.9 | 91.5 | 87.7 | 90.3 |
| Jedi-7B (Xie et al., 2025) | 96.9 | 87.2 | 95.9 | 87.9 | 94.4 | 84.2 | 91.7 |
| GUI-Actor-7B (Wu et al., 2025b) | 96.9 | 89.6 | 97.4 | 86.4 | 95.7 | 84.7 | 92.5 |
| Reinforcement Learning Methods |  |  |  |  |  |  |  |
| LPO-8B (Tang et al., 2025b) | 97.9 | 82.9 | 95.9 | 86.4 | 95.6 | 84.2 | 90.5 |
| SE-GUI-7B (Yuan et al., 2025) | - | - | - | - | - | - | 90.3 |
| GUI-G^2-7B (Tang et al., 2025a) | 98.3 | 91.9 | 95.4 | 89.3 | 94.0 | 87.7 | 93.3 |
| GTA1-7B (Yang et al., 2025a) | 99.0 | 88.6 | 94.9 | 89.3 | 92.3 | 86.7 | 92.4 |
| GUI-Cursor-7B | 99.2 | 90.6 | 94.4 | 91.3 | 96.1 | 89.0 | 93.9 |

**Table 2: ScreenSpot-Pro accuracy (%) broken down by six application domains (CAD, Dev, Creative, Scientific, Office, OS) and by text versus icon queries. Gui-Cursor attains the best average and delivers the strongest icon grounding, notably on CAD, Creative, and OS screens, outperforming both supervised and reinforcement learning baselines.**
| Model | CAD | Dev | Creative | Scientific | Office | OS | Avg. |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Text | Icon | Text | Icon | Text | Icon | Text | Icon | Text | Icon | Text | Icon | Text | Icon | Avg. |  |
| Supervised Fine-Tuning Methods |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| SeeClick (Cheng et al., 2024) | 0.6 | 0.0 | 1.0 | 0.0 | 2.5 | 0.0 | 3.5 | 0.0 | 1.1 | 0.0 | 2.8 | 0.0 | 1.8 | 0.0 | 1.1 |
| OS-Altlas-7B (Wu et al., 2024) | 33.1 | 1.4 | 28.8 | 2.8 | 12.2 | 4.7 | 37.5 | 7.3 | 33.9 | 5.7 | 27.1 | 4.5 | 28.1 | 4.0 | 18.9 |
| UGround-7B (Gou et al., 2025) | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 31.1 |
| UI-TARS-7B (Qin et al., 2025) | 58.4 | 12.4 | 50.0 | 9.1 | 20.8 | 9.4 | 63.9 | 31.8 | 63.3 | 20.8 | 30.8 | 16.9 | 47.8 | 16.2 | 35.7 |
| UI-TARS-72B (Qin et al., 2025) | 63.0 | 17.3 | 57.1 | 15.4 | 18.8 | 12.5 | 64.6 | 20.9 | 63.3 | 26.4 | 42.1 | 15.7 | 50.9 | 17.5 | 38.1 |
| Jedi-7B (Xie et al., 2025) | 42.9 | 11.0 | 50.0 | 11.9 | 38.0 | 14.1 | 72.9 | 25.5 | 75.1 | 47.2 | 33.6 | 16.9 | 52.6 | 18.2 | 39.5 |
| GUI-Actor-7B (Wu et al., 2025b) | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 47.7 |
| Reinforcement Learning Methods |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| UI-R1-3B (Lu et al., 2025) | 11.2 | 6.3 | 22.7 | 4.1 | 27.3 | 3.5 | 42.4 | 11.8 | 32.2 | 11.3 | 13.1 | 4.5 | 24.9 | 6.4 | 17.8 |
| UI-R1-E-3B (Lu et al., 2025) | 37.1 | 12.5 | 46.1 | 6.9 | 41.9 | 4.2 | 56.9 | 21.8 | 65.0 | 26.4 | 32.7 | 10.1 | - | - | 33.5 |
| GUI-R1-3B (Luo et al., 2025) | 26.4 | 7.8 | 33.8 | 4.8 | 40.9 | 5.6 | 61.8 | 17.3 | 53.6 | 17.0 | 28.1 | 5.6 | - | - | - |
| InfiGUI-R1-3B (Liu et al., 2025b) | 33.0 | 14.1 | 51.3 | 12.4 | 44.9 | 7.0 | 58.3 | 20.0 | 65.5 | 28.3 | 43.9 | 12.4 | 49.1 | 14.1 | 35.7 |
| GUI-R1-7B (Luo et al., 2025) | 23.9 | 6.3 | 49.4 | 4.8 | 38.9 | 8.4 | 55.6 | 11.8 | 58.7 | 26.4 | 42.1 | 16.9 | - | - | - |
| SE-GUI-7B (Yuan et al., 2025) | 51.3 | 42.2 | 68.2 | 19.3 | 57.6 | 9.1 | 75.0 | 28.2 | 78.5 | 43.4 | 49.5 | 25.8 | 63.5 | 21.0 | 47.3 |
| GUI-G^2-7B (Tang et al., 2025a) | 55.8 | 12.5 | 68.8 | 17.2 | 57.1 | 15.4 | 77.1 | 24.5 | 74.0 | 32.7 | 57.9 | 21.3 | 64.7 | 19.6 | 47.5 |
| GTA1-7B (Yang et al., 2025a) | 66.9 | 20.7 | 62.6 | 18.2 | 53.3 | 17.2 | 76.4 | 31.8 | 82.5 | 50.9 | 48.6 | 25.9 | 65.5 | 25.2 | 50.1 |
| GUI-Cursor-7B | 80.5 | 33.1 | 65.7 | 18.2 | 62.4 | 25.0 | 83.3 | 32.7 | 84.2 | 43.4 | 65.4 | 31.5 | 73.3 | 29.3 | 56.5 |

### 3.2 Main Experimental Results

#### Grounding Evaluation

[Table 1](https://arxiv.org/html/2509.21552v1#S3.T1) and [Table 2](https://arxiv.org/html/2509.21552v1#S3.T2) show the evaluation results on the ScreenSpot-v2 and ScreenSpot-Pro benchmarks, respectively.
Our method Gui-Cursor achieves $93.9\%$ accuracy on ScreenSpot-v2 and $56.5\%$ accuracy on ScreenSpot-Pro, outperforming a set of strong baseline models.
In the more challenging benchmark ScreenSpot-Pro, which evaluates on GUI images with high resolution and complex layouts, Gui-Cursor outperforms the previous leading model GTA1 (Yang et al., 2025a) by $6.4\%$.
These results demonstrate the effectiveness of Gui-Cursor in improving GUI grounding accuracy.
A qualitative case analysis is provided in [Appendix F](https://arxiv.org/html/2509.21552v1#A6).

#### Ablation Studies

Figure: (a) Ablations on reward penalties and thinking
Refer to caption: x2.png

*Trajectory penalties and thinking process improve the grounding accuracy*.
In [Fig. 2(a)](https://arxiv.org/html/2509.21552v1#S3.F2.sf1), we show the accuracy of our model without using each of the penalty terms, and without generating thinking tokens.
The results show that the accuracy without each penalty is lower than the full reward, and it is less effective in improving accuracy with additional movement steps.
Additionally, we find that without thinking ([Eq. 1](https://arxiv.org/html/2509.21552v1#S2.E1)), the accuracy of Gui-Cursor decreases significantly, whereas Tang et al. (2025a) and Yang et al. (2025a) found that thinking is less effective in improving accuracy.
One possible reason might be that Gui-Cursor is trained in an interactive environment – the explicit analysis of the target and the spatial relationship between the target and the cursor, which is helpful for the model to check whether the cursor is correctly located at the target, can help to align the coordinates with their corresponding position on the GUI.
The above results show that both reward penalties ([Section 2.3](https://arxiv.org/html/2509.21552v1#S2.SS3)) and the generation of explicit spatial reasoning tokens ([Eq. 1](https://arxiv.org/html/2509.21552v1#S2.E1)) are essential for the downstream accuracy of the model.

*Cursor-centric focusing improves the grounding accuracy*.
[Fig. 2(b)](https://arxiv.org/html/2509.21552v1#S3.F2.sf2) shows the accuracy without using cursor-centric focusing (w/o ccf).
We find ccf in ScreenSpot-Pro is more effective ($45.3\%\rightarrow 56.5\%$) than ScreenSpot-v2 ($93.6\%\rightarrow 93.9\%$), demonstrating the effectiveness of ccf on large and complex GUI images.
We also present the two-stage inference strategy used in GUI-Actor (Wu et al., 2025b), where the model first predicts a set of candidate positions and an external verifier model is applied to select the final answer.
Though the verifier introduces new parameters and needs additional training, the accuracy improvement ($44.6\%\rightarrow 47.7\%$, red bars) is less effective than ccf.
To further verify the effectiveness of ccf, we apply a similar approach to the SOTA methods GTA1 (Yang et al., 2025a) and GUI-G^2 (Tang et al., 2025a), labelled with ($+$ Focus) in [Fig. 2(b)](https://arxiv.org/html/2509.21552v1#S3.F2.sf2).
Here, the image is cropped around the initial prediction using the same image size as in Gui-Cursor, and the model makes its second step prediction using this cropped image.
Results show that the accuracy of GTA1 is effectively improved ($50.1\%\rightarrow 54.0\%$, yellow bars), but it is still lower than that of Gui-Cursor ($56.5\%$).
However, without ccf, the accuracy of Gui-Cursor is lower than GTA1, which is because GTA1 is trained with a higher resolution setting ($4096\times 2160$ pixels) than Gui-Cursor ($1920\times 1080$ pixels).
The above results show that ccf is an effective method to improve the accuracy, and it also demonstrates that combining low-resolution training and cursor-centric focusing ([Section 2.4](https://arxiv.org/html/2509.21552v1#S2.SS4)) is an effective strategy to balance the training efficiency and inference accuracy.

#### Analysis on the Cursor Movements

Figure: (a) Percentage of examples with more than one step movement.
Refer to caption: x4.png

Figure: Figure 4: The average target size in one-step and multi-step movement examples.
Refer to caption: x6.png

*Gui-Cursor adaptively conducts more cursor movement on more difficult tasks*.
As shown in [Fig. 3(a)](https://arxiv.org/html/2509.21552v1#S3.F3.sf1), Gui-Cursor conducts single-step interactions on most instances after ccf: it executes more than one step movement on $0.5\%$ of examples in ScreenSpot-v2, but this rate increases to $9.4\%$ on the more difficult dataset ScreenSpot-Pro.
In [Fig. 5](https://arxiv.org/html/2509.21552v1#S3.F5), we present the average size of the target in the samples where the model conducts multi-step (red bar) and one-step movement (blue bar).
On average, the target size for multi-step samples is 5024 pixels, compared to 31584 pixels for one-step samples.
This result indicates that the model may conduct more steps when the target is small.
More details and additional analyses are available in [Appendix C](https://arxiv.org/html/2509.21552v1#A3).

*Trajectory penalties influence the number of movement steps and training dynamics*.
Compared to the model without the repeated position penalty, the percentage of cursor moving steps increases: $0.5\%\rightarrow 1.3\%$ in ScreenSpot-v2 and $9.4\%\rightarrow 16.6\%$ in ScreenSpot-Pro, indicating that the repeated position penalty can improve efficiency.
The model without the false stop penalty tends not to perform multiple cursor movements: $0.5\%\rightarrow 0.0\%$ in ScreenSpot-v2, and $9.4\%\rightarrow 0.1\%$ in ScreenSpot-Pro, which shows that the false stop penalty prevents a degenerate behaviour where the model takes only a single step.
[Fig. 3(b)](https://arxiv.org/html/2509.21552v1#S3.F3.sf2) presents the average number of movement steps and the overall reward in each batch during training.
Gui-Cursor initially converges to single-step predictions after approximately 50 training steps; at later steps, the average number of movement steps increases to an average of 1.25 steps.
The model without the false stop penalty quickly converges to take one-step movements around 20 steps and does not learn to move at later steps, which further shows the false stop penalty is necessary for the model to learn multi-step movements.
The model without the repeated position penalty converges to a single-step policy more slowly than Gui-Cursor, and it becomes unstable when trained for more steps, with the number of steps increasing significantly around 220 steps.
A more detailed analysis of training dynamics is available in [Appendix D](https://arxiv.org/html/2509.21552v1#A4).

## 4 Analysis and Discussions

#### Moving the Cursor for GUI Grounding without Fine-Tuning

We investigate whether general-purpose VLMs can conduct GUI grounding by moving a cursor without fine-tuning.
This zero-shot task assesses the model’s spatial reasoning ability to identify the spatial relationship between a cursor and a target and refine its predictions accordingly.
We test two distinct prompting strategies:
1) *relative Move*: the model is prompted to generate a relative offset $(\Delta x,\Delta y)$, and the cursor is then moved from its current position $(x_{t},y_{t})$ to $(x_{t}+\Delta x,y_{t}+\Delta y)$; and
2) *direct Move*: the model is prompted to generate new absolute coordinates $(x,y)$, and the cursor is rendered at that position — this is the strategy used by Gui-Cursor in the main experiments ([Section 3.2](https://arxiv.org/html/2509.21552v1#S3.SS2)).
More implementation details and analysis are presented in [Section B.3](https://arxiv.org/html/2509.21552v1#A2.SS3).

In [Fig. 5](https://arxiv.org/html/2509.21552v1#S3.F5), we present the accuracy for GPT-4o (Hurst et al., 2024) and Qwen2.5-VL-7B on ScreenSpot-v2.
In the standard one-step GUI grounding setting (blue bars), the accuracy of GPT-4o is low ($17.5\%$), and Qwen2.5-VL-7B performs well ($88.8\%$) because it has been fine-tuned on GUI grounding tasks.
With 10 steps of movement, GPT-4o’s performance improves with both direct move and ($17.5\%\rightarrow 21.7\%$) relative move ($17.5\%\rightarrow 25.5\%$), suggesting it possesses the underlying spatial reasoning capability to
benefit from the iterative process.
However, the same strategies cause a significant performance drop for the Qwen2.5-VL-7B (direct move: $36.3\%$, relative move: $1.3\%$).
This result suggests that *the high single-step accuracy of Qwen2.5-VL-7B does not generalise to this iterative process*, and its success on GUI grounding may not be founded on robust spatial understanding of the GUI image.
Based on this result, it is necessary to fine-tune Qwen2.5-VL-7B to gain the ability to use a cursor.
We use the direct move strategy in Gui-Cursor, as its higher zero-shot accuracy provides a better starting point for reinforcement learning.

#### Probing Spatial Reasoning

We designed a *cursor-in-box* test to isolate and evaluate a VLM’s spatial reasoning accuracy over different positions of the image.
The task is straightforward: given a white background containing a red bounding box and a black cursor, the model must answer “Yes or No” to the question: “Is the black cursor inside the red bounding box?”.
We generated a comprehensive dataset by placing the box at different locations across the image and sampling cursor positions both inside and outside the box for each location.
More implementation details and analysis are presented in [Appendix E](https://arxiv.org/html/2509.21552v1#A5).
This minimalist setup eliminates errors that stem from target misidentification in standard GUI grounding by moving a cursor, allowing us to focus on evaluating spatial reasoning around the cursor.

We evaluate Qwen-2.5-VL-7B and measure performance using the F1 score, visualised in [Fig. 6](https://arxiv.org/html/2509.21552v1#S4.F6).
The results reveal a critical flaw in Qwen-2.5-VL-7B.
The left heatmap shows that the model struggles with this simple task and also exhibits a severe positional bias.
It performs well only when the box is near the centre of the image (green areas) and fails towards the edges (red areas).
This failure suggests its grounding capabilities may be brittle and not founded on a robust spatial understanding of GUI images.
The right heatmap shows that Gui-Cursor achieves higher accuracy, though it is not explicitly trained on this classification task.
Instead, we design trajectory penalties to discourage the wrong-moving behaviours, which may implicitly improve this spatial reasoning capability by rewarding the trajectory with a correct spatial thinking process.

Figure: Figure 6: Cursor-in-box spatial reasoning test results. The heatmap presents F1 scores across different positions. Left: before training, Qwen-2.5-VL-7B exhibits a strong centre bias; Right: after RL training, Gui-Cursor achieves higher F1, despite not being explicitly trained on this task.
Refer to caption: x8.png

## 5 Related Work

#### GUI Grounding

A common implementation for these agents involves a planning model that determines the sequence of actions, and a grounding model that predicts the positions to execute them.
Recently, the purely vision-driven approach has shown significant advantages in accuracy and general applicability, as it does not rely on backends of systems and external utilities (Gou et al., 2025; Wu et al., 2025b; Yang et al., 2025a). SFT is a common method for training vision-driven grounding models on large-scale GUI datasets (Hong et al., 2024; Cheng et al., 2024; Gou et al., 2025; Wu et al., 2024; Lin et al., 2025; Xu et al., 2024; Xie et al., 2025).
Recently, RL with rule-based rewards has emerged as a popular technique for enhancing one-step GUI grounding (Luo et al., 2025; Zhou et al., 2025b; Lu et al., 2025; Yuan et al., 2025; Liu et al., 2025b; Yang et al., 2025a; Tang et al., 2025a).
Different from existing works, our main contribution is reformulating the learning of GUI grounding to a multi-step interaction process, enabling better spatial-coordinate alignment by conducting spatial reasoning from explicit visual feedback during interaction.

#### Multimodal Reasoning

Huang et al. (2025); Zhou et al. (2025a); Shen et al. (2025a); Chen et al. (2025a); Su et al. (2025a) show emergent multimodal reasoning capability in VLMs after RL training. Chen et al. (2025b); Liu et al. (2025a); Zhang et al. (2025d) reveal that VLMs struggle with spatial reasoning and often fail in inferring spatial relationships between objects.
(Wu et al., 2025a) improves image-text interleaved reasoning through RL.
Li et al. (2025a) and Chern et al. (2025) propose to generate visual thoughts to improve multimodal reasoning.
Our multi-step interaction learning process is related to recent works on multi-turn interaction paradigms in VLMs (Shen et al., 2025b; Su et al., 2025b), which suggested scaling test-time interaction to improve reasoning, and tool-augmented reasoning frameworks (Yao et al., 2023; Qin et al., 2024; Qu et al., 2025), where agents iteratively refine their predictions through sequential actions.

## 6 Conclusion

We reframe GUI grounding as an interactive, cursor-driven search that leverages explicit visual feedback and stepwise spatial reasoning.
During training, the rendered cursor provides explicit visual feedback that shows the predicted position on the GUI image, helping the model better align numerical coordinates with their corresponding on-screen positions.
During inference, this interactive environment allows the model to conduct spatial reasoning based on its visual feedback, allowing it to incrementally refine its predictions.
Gui-Cursor couples GRPO with a trajectory-aware reward and employs cursor-centric focusing (ccf) for balancing training efficiency and inference accuracy.
Using Qwen-2.5-VL-7B as base model, Gui-Cursor achieves state-of-the-art accuracy on ScreenSpot-v2 (93.9%) and ScreenSpot-Pro (56.5%), and the model learns an adaptive strategy to conduct more steps movement in difficult tasks.

## References

- Bai et al. (2025)
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Ming-Hsuan Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin.
Qwen2.5-vl technical report.
*CoRR*, abs/2502.13923, 2025.
doi: 10.48550/ARXIV.2502.13923.
URL [https://doi.org/10.48550/arXiv.2502.13923](https://doi.org/10.48550/arXiv.2502.13923).
- Chen et al. (2025a)
Hardy Chen, Haoqin Tu, Fali Wang, Hui Liu, Xianfeng Tang, Xinya Du, Yuyin Zhou, and Cihang Xie.
Sft or rl? an early investigation into training r1-like reasoning large vision-language models.
*ArXiv*, abs/2504.11468, 2025a.
URL [https://api.semanticscholar.org/CorpusID:277824294](https://api.semanticscholar.org/CorpusID:277824294).
- Chen et al. (2025b)
Shiqi Chen, Tongyao Zhu, Ruochen Zhou, Jinghan Zhang, Siyang Gao, Juan Carlos Niebles, Mor Geva, Junxian He, Jiajun Wu, and Manling Li.
Why is spatial reasoning hard for vlms? an attention mechanism perspective on focus areas.
*ArXiv*, abs/2503.01773, 2025b.
URL [https://api.semanticscholar.org/CorpusID:276775433](https://api.semanticscholar.org/CorpusID:276775433).
- Cheng et al. (2024)
Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu.
Seeclick: Harnessing GUI grounding for advanced visual GUI agents.
In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), *Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024*, pp. 9313–9332. Association for Computational Linguistics, 2024.
doi: 10.18653/V1/2024.ACL-LONG.505.
URL [https://doi.org/10.18653/v1/2024.acl-long.505](https://doi.org/10.18653/v1/2024.acl-long.505).
- Chern et al. (2025)
Ethan Chern, Zhulin Hu, Steffi Chern, Siqi Kou, Jiadi Su, Yan Ma, Zhijie Deng, and Pengfei Liu.
Thinking with generated images.
*ArXiv*, abs/2505.22525, 2025.
URL [https://api.semanticscholar.org/CorpusID:278959478](https://api.semanticscholar.org/CorpusID:278959478).
- Cui et al. (2025)
Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, Jiarui Yuan, Huayu Chen, Kaiyan Zhang, Xingtai Lv, Shuo Wang, Yuan Yao, Xu Han, Hao Peng, Yu Cheng, Zhiyuan Liu, Maosong Sun, Bowen Zhou, and Ning Ding.
Process reinforcement through implicit rewards.
*CoRR*, abs/2502.01456, 2025.
- DeepSeek-AI et al. (2025)
DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge,
Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, and S. S. Li.
Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.
*CoRR*, abs/2501.12948, 2025.
doi: 10.48550/ARXIV.2501.12948.
URL [https://doi.org/10.48550/arXiv.2501.12948](https://doi.org/10.48550/arXiv.2501.12948).
- Deng et al. (2023)
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samual Stevens, Boshi Wang, Huan Sun, and Yu Su.
Mind2web: Towards a generalist agent for the web.
In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), *Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023*, 2023.
URL [http://papers.nips.cc/paper_files/paper/2023/hash/5950bf290a1570ea401bf98882128160-Abstract-Datasets_and_Benchmarks.html](http://papers.nips.cc/paper_files/paper/2023/hash/5950bf290a1570ea401bf98882128160-Abstract-Datasets_and_Benchmarks.html).
- Gou et al. (2025)
Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su.
Navigating the digital world as humans do: Universal visual grounding for GUI agents.
In *The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025*. OpenReview.net, 2025.
URL [https://openreview.net/forum?id=kxnoqaisCT](https://openreview.net/forum?id=kxnoqaisCT).
- Hong et al. (2024)
Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, and Jie Tang.
Cogagent: A visual language model for GUI agents.
In *IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024*, pp. 14281–14290. IEEE, 2024.
doi: 10.1109/CVPR52733.2024.01354.
URL [https://doi.org/10.1109/CVPR52733.2024.01354](https://doi.org/10.1109/CVPR52733.2024.01354).
- Huang et al. (2025)
Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaoshen Cao, Zheyu Ye, Fei Zhao, Zhe Xu, Yao Hu, and Shaohui Lin.
Vision-r1: Incentivizing reasoning capability in multimodal large language models.
*ArXiv*, abs/2503.06749, 2025.
URL [https://api.semanticscholar.org/CorpusID:276902576](https://api.semanticscholar.org/CorpusID:276902576).
- Hurst et al. (2024)
OpenAI Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Mkadry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alexander Kirillov, Alex Nichol, Alex Paino, Alex Renzin, Alexandre Passos, Alexander Kirillov, Alexi Christakis, Alexis Conneau, Ali Kamali, Allan Jabri, Allison Moyer, Allison Tam, Amadou Crookes, Amin Tootoochian, Amin Tootoonchian, Ananya Kumar, Andrea Vallone, Andrej Karpathy, Andrew Braunstein, Andrew Cann, Andrew Codispoti, Andrew Galu, Andrew Kondrich, Andrew Tulloch, An drey Mishchenko, Angela Baek, Angela Jiang, An toine Pelisse, Antonia Woodford, Anuj Gosalia, Arka Dhar, Ashley Pantuliano, Avi Nayak, Avital Oliver, Barret Zoph, B. Ghorbani, Ben Leimberger, Ben Rossen, Benjamin Sokolowsky, Ben Wang, Benjamin Zweig, Beth Hoover, Blake Samic, Bob McGrew, Bobby Spero, Bogo Giertler, Bowen Cheng, Brad Lightcap, Brandon Walkin, Brendan Quinn, Brian Guarraci, Brian Hsu,
Bright Kellogg, Brydon Eastman, Camillo Lugaresi, Carroll L. Wainwright, Cary Bassin, Cary Hudson, Casey Chu, Chad Nelson, Chak Li, Chan Jun Shern, Channing Conger, Charlotte Barette, Chelsea Voss, Chen Ding, Cheng Lu, Chong Zhang, Chris Beaumont, Chris Hallacy, Chris Koch, Christian Gibson, Christina Kim, Christine Choi, Christine McLeavey, Chris Hesse, Claudia Fischer, Clemens Winter, Coley Czarnecki, Colin Jarvis, Colin Wei, Constantin Koumouzelis, Dane Sherburn, Daniel Kappler, Daniel Levin, Daniel Levy, David Carr, David Farhi, David Mély, David Robinson, David Sasaki, Denny Jin, Dev Valladares, Dimitris Tsipras, Doug Li, Phong Duc Nguyen, Duncan Findlay, Edede Oiwoh, Edmund Wong, Ehsan Asdar, Elizabeth Proehl, Elizabeth Yang, Eric Antonow, Eric Kramer, Eric Peterson, Eric Sigler, Eric Wallace, Eugene Brevdo, Evan Mays, Farzad Khorasani, Felipe Petroski Such, Filippo Raso, Francis Zhang, Fred von Lohmann, Freddie Sulit, Gabriel Goh, Gene Oden, Geoff Salmon, Giulio Starace, Greg Brockman, Hadi
Salman, Hai-Biao Bao, Haitang Hu, Hannah Wong, Haoyu Wang, Heather Schmidt, Heather Whitney, Hee woo Jun, Hendrik Kirchner, Henrique Pondé de Oliveira Pinto, Hongyu Ren, Huiwen Chang, Hyung Won Chung, Ian Kivlichan, Ian O’Connell, Ian Osband, Ian Silber, Ian Sohl, Ibrahim Okuyucu, Ikai Lan, Ilya Kostrikov, Ilya Sutskever, Ingmar Kanitscheider, Ishaan Gulrajani, Jacob Coxon, Jacob Menick, Jakub W. Pachocki, James Aung, James Betker, James Crooks, James Lennon, Jamie Ryan Kiros, Jan Leike, Jane Park, Jason Kwon, Jason Phang, Jason Teplitz, Jason Wei, Jason Wolfe, Jay Chen, Jeff Harris, Jenia Varavva, Jessica Gan Lee, Jessica Shieh, Ji Lin, Jiahui Yu, Jiayi Weng, Jie Tang, Jieqi Yu, Joanne Jang, Joaquin Quiñonero Candela, Joe Beutler, Joe Landers, Joel Parish, Johannes Heidecke, John Schulman, Jonathan Lachman, Jonathan McKay, Jonathan Uesato, Jonathan Ward, Jong Wook Kim, Joost Huizinga, Jordan Sitkin, Jos Kraaijeveld, Joshua Gross, Josh Kaplan, Josh Snyder, Joshua Achiam, Joy Jiao, Joyce Lee,
Juntang Zhuang, Justyn Harriman, Kai Fricke, Kai Hayashi, Karan Singhal, Katy Shi, Kavin Karthik, Kayla Wood, Kendra Rimbach, Kenny Hsu, Kenny Nguyen, Keren Gu-Lemberg, Kevin Button, Kevin Liu, Kiel Howe, Krithika Muthukumar, Kyle Luther, Lama Ahmad, Larry Kai, Lauren Itow, Lauren Workman, Leher Pathak, Leo Chen, Li Jing, Lia Guy, Liam Fedus, Liang Zhou, Lien Mamitsuka, Lilian Weng, Lindsay McCallum, Lindsey Held, Ouyang Long, Louis Feuvrier, Lu Zhang, Lukasz Kondraciuk, Lukasz Kaiser, Luke Hewitt, Luke Metz, Lyric Doshi, Mada Aflak, Maddie Simens, Made laine Boyd, Madeleine Thompson, Marat Dukhan, Mark Chen, Mark Gray, Mark Hudnall, Marvin Zhang, Marwan Aljubeh, Ma teusz Litwin, Matthew Zeng, Max Johnson, Maya Shetty, Mayank Gupta, Meghan Shah, Mehmet Ali Yatbaz, Mengxue Yang, Mengchao Zhong, Mia Glaese, Mianna Chen, Michael Janner, Michael Lampe, Michael Petrov, Michael Wu, Michele Wang, Michelle Fradin, Michelle Pokrass, Miguel Castro, Miguel Castro, Mikhail Pavlov, Miles Brundage, Miles Wang, Mina Khan,
Mira Murati, Mo Bavarian, Molly Lin, Murat Yesildal, Nacho Soto, Natalia Gimelshein, Na talie Cone, Natalie Staudacher, Natalie Summers, Natan LaFontaine, Neil Chowdhury, Nick Ryder, Nick Stathas, Nick Turley, Nikolas A. Tezak, Niko Felix, Nithanth Kudige, Nitish Shirish Keskar, Noah Deutsch, Noel Bundick, Nora Puckett, Ofir Nachum, Ola Okelola, Oleg Boiko, Oleg Murk, Oliver Jaffe, Olivia Watkins, Olivier Godement, Owen Campbell-Moore, Patrick Chao, Paul McMillan, Pavel Belov, Peng Su, Peter Bak, Peter Bakkum, Peter Deng, Peter Dolan, Peter Hoeschele, Peter Welinder, Phil Tillet, Philip Pronin, Phil Tillet, Prafulla Dhariwal, Qim ing Yuan, Rachel Dias, Rachel Lim, Rahul Arora, Rajan Troll, Randall Lin, Raphael Gontijo Lopes, Raul Puri, Reah Miyara, Reimar H. Leike, Renaud Gaubert, Reza Zamani, Ricky Wang, Rob Donnelly, Rob Honsby, Rocky Smith, Rohan Sahai, Rohit Ramchandani, Romain Huet, Rory Carmichael, Rowan Zellers, Roy Chen, Ruby Chen, Ruslan Ramilevich Nigmatullin, Ryan Cheu, Saachi Jain, Sam Altman,
Sam Schoenholz, Sam Toizer, Samuel Miserendino, Sandhini Agarwal, Sara Culver, Scott Ethersmith, Scott Gray, Sean Grove, Sean Metzger, Shamez Hermani, Shantanu Jain, Shengjia Zhao, Sherwin Wu, Shino Jomoto, Shirong Wu, Shuaiqi Xia, Sonia Phene, Spencer Papay, Srinivas Narayanan, Steve Coffey, Steve Lee, Stewart Hall, Suchir Balaji, Tal Broda, Tal Stramer, Tao Xu, Tarun Gogineni, Taya Christianson, Ted Sanders, Tejal Patwardhan, Thomas Cunninghman, Thomas Degry, Thomas Dimson, Thomas Raoux, Thomas Shadwell, Tianhao Zheng, Todd Underwood, Todor Markov, Toki Sherbakov, Tom Rubin, Tom Stasi, Tomer Kaftan, Tristan Heywood, Troy Peterson, Tyce Walters, Tyna Eloundou, Valerie Qi, Veit Moeller, Vinnie Monaco, Vishal Kuo, Vlad Fomenko, Wayne Chang, Weiyi Zheng, Wenda Zhou, Wesam Manassra, Will Sheu, Wojciech Zaremba, Yash Patil, Yilei Qian, Yongjik Kim, Youlong Cheng, Yu Zhang, Yuchen He, Yuchen Zhang, Yujia Jin, Yunxing Dai, and Yury Malkov.
Gpt-4o system card.
*ArXiv*, abs/2410.21276, 2024.
URL [https://api.semanticscholar.org/CorpusID:273662196](https://api.semanticscholar.org/CorpusID:273662196).
- Kwon et al. (2023)
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica.
Efficient memory management for large language model serving with pagedattention.
In Jason Flinn, Margo I. Seltzer, Peter Druschel, Antoine Kaufmann, and Jonathan Mace (eds.), *Proceedings of the 29th Symposium on Operating Systems Principles, SOSP 2023, Koblenz, Germany, October 23-26, 2023*, pp. 611–626. ACM, 2023.
doi: 10.1145/3600006.3613165.
URL [https://doi.org/10.1145/3600006.3613165](https://doi.org/10.1145/3600006.3613165).
- Li et al. (2025a)
Chengzu Li, Wenshan Wu, Huanyu Zhang, Yan Xia, Shaoguang Mao, Li Dong, Ivan Vuli’c, and Furu Wei.
Imagine while reasoning in space: Multimodal visualization-of-thought.
*ArXiv*, abs/2501.07542, 2025a.
URL [https://api.semanticscholar.org/CorpusID:275471612](https://api.semanticscholar.org/CorpusID:275471612).
- Li et al. (2025b)
Kaixin Li, Ziyang Meng, Hongzhan Lin, Ziyang Luo, Yuchen Tian, Jing Ma, Zhiyong Huang, and Tat-Seng Chua.
Screenspot-pro: GUI grounding for professional high-resolution computer use.
*CoRR*, abs/2504.07981, 2025b.
doi: 10.48550/ARXIV.2504.07981.
URL [https://doi.org/10.48550/arXiv.2504.07981](https://doi.org/10.48550/arXiv.2504.07981).
- Li et al. (2024)
Wei Li, William E. Bishop, Alice Li, Christopher Rawles, Folawiyo Campbell-Ajala, Divya Tyamagundlu, and Oriana Riva.
On the effects of data scale on UI control agents.
In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang (eds.), *Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024*, 2024.
URL [http://papers.nips.cc/paper_files/paper/2024/hash/a79f3ef3b445fd4659f44648f7ea8ffd-Abstract-Datasets_and_Benchmarks_Track.html](http://papers.nips.cc/paper_files/paper/2024/hash/a79f3ef3b445fd4659f44648f7ea8ffd-Abstract-Datasets_and_Benchmarks_Track.html).
- Lin et al. (2025)
Kevin Qinghong Lin, Linjie Li, Difei Gao, Zhengyuan Yang, Shiwei Wu, Zechen Bai, Stan Weixian Lei, Lijuan Wang, and Mike Zheng Shou.
Showui: One vision-language-action model for GUI visual agent.
In *IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2025, Nashville, TN, USA, June 11-15, 2025*, pp. 19498–19508. Computer Vision Foundation / IEEE, 2025.
URL [https://openaccess.thecvf.com/content/CVPR2025/html/Lin_ShowUI_One_Vision-Language-Action_Model_for_GUI_Visual_Agent_CVPR_2025_paper.html](https://openaccess.thecvf.com/content/CVPR2025/html/Lin_ShowUI_One_Vision-Language-Action_Model_for_GUI_Visual_Agent_CVPR_2025_paper.html).
- Liu et al. (2025a)
Jingping Liu, Ziyan Liu, Zhedong Cen, Yan Zhou, Yinan Zou, Weiyan Zhang, Haiyun Jiang, and Tong Ruan.
Can multimodal large language models understand spatial relations?
*ArXiv*, abs/2505.19015, 2025a.
URL [https://api.semanticscholar.org/CorpusID:278905162](https://api.semanticscholar.org/CorpusID:278905162).
- Liu et al. (2025b)
Yuhang Liu, Pengxiang Li, Congkai Xie, Xavier Hu, Xiaotian Han, Shengyu Zhang, Hongxia Yang, and Fei Wu.
Infigui-r1: Advancing multimodal GUI agents from reactive actors to deliberative reasoners.
*CoRR*, abs/2504.14239, 2025b.
doi: 10.48550/ARXIV.2504.14239.
URL [https://doi.org/10.48550/arXiv.2504.14239](https://doi.org/10.48550/arXiv.2504.14239).
- Lu et al. (2024)
Yadong Lu, Jianwei Yang, Yelong Shen, and Ahmed Awadallah.
Omniparser for pure vision based GUI agent.
*CoRR*, abs/2408.00203, 2024.
doi: 10.48550/ARXIV.2408.00203.
URL [https://doi.org/10.48550/arXiv.2408.00203](https://doi.org/10.48550/arXiv.2408.00203).
- Lu et al. (2025)
Zhengxi Lu, Yuxiang Chai, Yaxuan Guo, Xi Yin, Liang Liu, Hao Wang, Guanjing Xiong, and Hongsheng Li.
UI-R1: enhancing action prediction of GUI agents by reinforcement learning.
*CoRR*, abs/2503.21620, 2025.
doi: 10.48550/ARXIV.2503.21620.
URL [https://doi.org/10.48550/arXiv.2503.21620](https://doi.org/10.48550/arXiv.2503.21620).
- Luo et al. (2025)
Run Luo, Lu Wang, Wanwei He, and Xiaobo Xia.
GUI-R1 : A generalist r1-style vision-language action model for GUI agents.
*CoRR*, abs/2504.10458, 2025.
doi: 10.48550/ARXIV.2504.10458.
URL [https://doi.org/10.48550/arXiv.2504.10458](https://doi.org/10.48550/arXiv.2504.10458).
- Qin et al. (2024)
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun.
Toolllm: Facilitating large language models to master 16000+ real-world apis.
In *The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024*. OpenReview.net, 2024.
URL [https://openreview.net/forum?id=dHng2O0Jjr](https://openreview.net/forum?id=dHng2O0Jjr).
- Qin et al. (2025)
Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, Wanjun Zhong, Kuanye Li, Jiale Yang, Yu Miao, Woyu Lin, Longxiang Liu, Xu Jiang, Qianli Ma, Jingyu Li, Xiaojun Xiao, Kai Cai, Chuang Li, Yaowei Zheng, Chaolin Jin, Chen Li, Xiao Zhou, Minchao Wang, Haoli Chen, Zhaojian Li, Haihua Yang, Haifeng Liu, Feng Lin, Tao Peng, Xin Liu, and Guang Shi.
UI-TARS: pioneering automated GUI interaction with native agents.
*CoRR*, abs/2501.12326, 2025.
doi: 10.48550/ARXIV.2501.12326.
URL [https://doi.org/10.48550/arXiv.2501.12326](https://doi.org/10.48550/arXiv.2501.12326).
- Qu et al. (2025)
Changle Qu, Sunhao Dai, Xiaochi Wei, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, Jun Xu, and Ji-Rong Wen.
Tool learning with large language models: a survey.
*Frontiers Comput. Sci.*, 19(8):198343, 2025.
doi: 10.1007/S11704-024-40678-2.
URL [https://doi.org/10.1007/s11704-024-40678-2](https://doi.org/10.1007/s11704-024-40678-2).
- Shen et al. (2025a)
Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, Ruochen Xu, and Tiancheng Zhao.
Vlm-r1: A stable and generalizable r1-style large vision-language model.
*ArXiv*, abs/2504.07615, 2025a.
URL [https://api.semanticscholar.org/CorpusID:277667819](https://api.semanticscholar.org/CorpusID:277667819).
- Shen et al. (2025b)
Junhong Shen, Hao Bai, Lunjun Zhang, Yifei Zhou, Amrith Rajagopal Setlur, Shengbang Tong, Diego Caples, Nan Jiang, Tong Zhang, Ameet Talwalkar, and Aviral Kumar.
Thinking vs. doing: Agents that reason by scaling test-time interaction.
*ArXiv*, abs/2506.07976, 2025b.
URL [https://api.semanticscholar.org/CorpusID:279251333](https://api.semanticscholar.org/CorpusID:279251333).
- Su et al. (2025a)
Alex Su, Haozhe Wang, Weiming Ren, Fangzhen Lin, and Wenhu Chen.
Pixel reasoner: Incentivizing pixel-space reasoning with curiosity-driven reinforcement learning.
*ArXiv*, abs/2505.15966, 2025a.
URL [https://api.semanticscholar.org/CorpusID:278789415](https://api.semanticscholar.org/CorpusID:278789415).
- Su et al. (2025b)
Hongjin Su, Ruoxi Sun, Jinsung Yoon, Pengcheng Yin, Tao Yu, and Sercan Ö. Arik.
Learn-by-interact: A data-centric framework for self-adaptive agents in realistic environments.
*ArXiv*, abs/2501.10893, 2025b.
URL [https://api.semanticscholar.org/CorpusID:275757520](https://api.semanticscholar.org/CorpusID:275757520).
- Tang et al. (2025a)
Fei Tang, Zhangxuan Gu, Zhengxi Lu, Xuyang Liu, Shuheng Shen, Changhua Meng, Wen Wang, Wenqi Zhang, Yongliang Shen, Weiming Lu, Jun Xiao, and Yueting Zhuang.
Gui-g^2: Gaussian reward modeling for gui grounding, 2025a.
URL [https://arxiv.org/abs/2507.15846](https://arxiv.org/abs/2507.15846).
- Tang et al. (2025b)
Jiaqi Tang, Yu Xia, Yi-Feng Wu, Yuwei Hu, Yuhui Chen, Qing-Guo Chen, Xiaogang Xu, Xiangyu Wu, Hao Lu, Yanqing Ma, Shiyin Lu, and Qifeng Chen.
LPO: towards accurate GUI agent interaction via location preference optimization.
*CoRR*, abs/2506.09373, 2025b.
doi: 10.48550/ARXIV.2506.09373.
URL [https://doi.org/10.48550/arXiv.2506.09373](https://doi.org/10.48550/arXiv.2506.09373).
- Wu et al. (2025a)
Jun Wu, Jian Guan, Kaituo Feng, Qiang Liu, Shuning Wu, Liang Wang, Wei Wu, and Tieniu Tan.
Reinforcing spatial reasoning in vision-language models with interwoven thinking and visual drawing.
*ArXiv*, abs/2506.09965, 2025a.
URL [https://api.semanticscholar.org/CorpusID:279306073](https://api.semanticscholar.org/CorpusID:279306073).
- Wu et al. (2025b)
Qianhui Wu, Kanzhi Cheng, Rui Yang, Chaoyun Zhang, Jianwei Yang, Huiqiang Jiang, Jian Mu, Baolin Peng, Bo Qiao, Reuben Tan, Si Qin, Lars Liden, Qingwei Lin, Huan Zhang, Tong Zhang, Jianbing Zhang, Dongmei Zhang, and Jianfeng Gao.
Gui-actor: Coordinate-free visual grounding for GUI agents.
*CoRR*, abs/2506.03143, 2025b.
doi: 10.48550/ARXIV.2506.03143.
URL [https://doi.org/10.48550/arXiv.2506.03143](https://doi.org/10.48550/arXiv.2506.03143).
- Wu et al. (2024)
Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, and Yu Qiao.
OS-ATLAS: A foundation action model for generalist GUI agents.
*CoRR*, abs/2410.23218, 2024.
doi: 10.48550/ARXIV.2410.23218.
URL [https://doi.org/10.48550/arXiv.2410.23218](https://doi.org/10.48550/arXiv.2410.23218).
- Xie et al. (2025)
Tianbao Xie, Jiaqi Deng, Xiaochuan Li, Junlin Yang, Haoyuan Wu, Jixuan Chen, Wenjing Hu, Xinyuan Wang, Yuhui Xu, Zekun Wang, Yiheng Xu, Junli Wang, Doyen Sahoo, Tao Yu, and Caiming Xiong.
Scaling computer-use grounding via user interface decomposition and synthesis.
*CoRR*, abs/2505.13227, 2025.
doi: 10.48550/ARXIV.2505.13227.
URL [https://doi.org/10.48550/arXiv.2505.13227](https://doi.org/10.48550/arXiv.2505.13227).
- Xu et al. (2024)
Yiheng Xu, Zekun Wang, Junli Wang, Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, and Caiming Xiong.
Aguvis: Unified pure vision agents for autonomous GUI interaction.
*CoRR*, abs/2412.04454, 2024.
doi: 10.48550/ARXIV.2412.04454.
URL [https://doi.org/10.48550/arXiv.2412.04454](https://doi.org/10.48550/arXiv.2412.04454).
- Yang et al. (2025a)
Yan Yang, Dongxu Li, Yutong Dai, Yuhao Yang, Ziyang Luo, Zirui Zhao, Zhiyuan Hu, Junzhe Huang, Amrita Saha, Zeyuan Chen, et al.
Gta1: Gui test-time scaling agent.
*arXiv preprint arXiv:2507.05791*, 2025a.
- Yang et al. (2025b)
Yuhao Yang, Yue Wang, Dongxu Li, Ziyang Luo, Bei Chen, Chao Huang, and Junnan Li.
Aria-ui: Visual grounding for GUI instructions.
In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), *Findings of the Association for Computational Linguistics, ACL 2025, Vienna, Austria, July 27 - August 1, 2025*, pp. 22418–22433. Association for Computational Linguistics, 2025b.
URL [https://aclanthology.org/2025.findings-acl.1152/](https://aclanthology.org/2025.findings-acl.1152/).
- Yao et al. (2023)
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao.
React: Synergizing reasoning and acting in language models.
In *The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023*. OpenReview.net, 2023.
URL [https://openreview.net/forum?id=WE_vluYUL-X](https://openreview.net/forum?id=WE_vluYUL-X).
- Yuan et al. (2025)
Xinbin Yuan, Jian Zhang, Kaixin Li, Zhuoxuan Cai, Lujian Yao, Jie Chen, Enguang Wang, Qibin Hou, Jinwei Chen, Peng-Tao Jiang, and Bo Li.
Enhancing visual grounding for GUI agents via self-evolutionary reinforcement learning.
*CoRR*, abs/2505.12370, 2025.
doi: 10.48550/ARXIV.2505.12370.
URL [https://doi.org/10.48550/arXiv.2505.12370](https://doi.org/10.48550/arXiv.2505.12370).
- Zhang et al. (2025a)
Chaoyun Zhang, Shilin He, Jiaxu Qian, Bowen Li, Liqun Li, Si Qin, Yu Kang, Minghua Ma, Guyue Liu, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, and Qi Zhang.
Large language model-brained GUI agents: A survey.
*Trans. Mach. Learn. Res.*, 2025, 2025a.
URL [https://openreview.net/forum?id=xChvYjvXTp](https://openreview.net/forum?id=xChvYjvXTp).
- Zhang et al. (2025b)
Chaoyun Zhang, He Huang, Chiming Ni, Jian Mu, Si Qin, Shilin He, Lu Wang, Fangkai Yang, Pu Zhao, Chao Du, Liqun Li, Yu Kang, Zhao Jiang, Suzhen Zheng, Rujia Wang, Jiaxu Qian, Minghua Ma, Jian-Guang Lou, Qingwei Lin, Saravan Rajmohan, and Dongmei Zhang.
UFO2: the desktop agentos.
*CoRR*, abs/2504.14603, 2025b.
doi: 10.48550/ARXIV.2504.14603.
URL [https://doi.org/10.48550/arXiv.2504.14603](https://doi.org/10.48550/arXiv.2504.14603).
- Zhang et al. (2025c)
Chaoyun Zhang, Liqun Li, Shilin He, Xu Zhang, Bo Qiao, Si Qin, Minghua Ma, Yu Kang, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, and Qi Zhang.
UFO: A ui-focused agent for windows OS interaction.
In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), *Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2025 - Volume 1: Long Papers, Albuquerque, New Mexico, USA, April 29 - May 4, 2025*, pp. 597–622. Association for Computational Linguistics, 2025c.
doi: 10.18653/V1/2025.NAACL-LONG.26.
URL [https://doi.org/10.18653/v1/2025.naacl-long.26](https://doi.org/10.18653/v1/2025.naacl-long.26).
- Zhang et al. (2025d)
Wanyue Zhang, Yibin Huang, Yangbin Xu, JingJing Huang, Helu Zhi, Shuo Ren, Wang Xu, and Jiajun Zhang.
Why do mllms struggle with spatial understanding? a systematic analysis from data to architecture.
*arXiv preprint arXiv:2509.02359*, 2025d.
- Zheng et al. (2025)
Yaowei Zheng, Junting Lu, Shenzhi Wang, Zhangchi Feng, Dongdong Kuang, and Yuwen Xiong.
Easyr1: An efficient, scalable, multi-modality rl training framework.
[https://github.com/hiyouga/EasyR1](https://github.com/hiyouga/EasyR1), 2025.
- Zhou et al. (2025a)
Hengguang Zhou, Xirui Li, Ruochen Wang, Minhao Cheng, Tianyi Zhou, and Cho-Jui Hsieh.
R1-zero’s ”aha moment” in visual reasoning on a 2b non-sft model.
*ArXiv*, abs/2503.05132, 2025a.
URL [https://api.semanticscholar.org/CorpusID:276884980](https://api.semanticscholar.org/CorpusID:276884980).
- Zhou et al. (2025b)
Yuqi Zhou, Sunhao Dai, Shuai Wang, Kaiwen Zhou, Qinglin Jia, and Jun Xu.
GUI-G1: understanding r1-zero-like training for visual grounding in GUI agents.
*CoRR*, abs/2505.15810, 2025b.
doi: 10.48550/ARXIV.2505.15810.
URL [https://doi.org/10.48550/arXiv.2505.15810](https://doi.org/10.48550/arXiv.2505.15810).

## Appendix A The Use of Large Language Models

The large language models are used as a general-purpose assist tool to check typos and grammar errors in writing.

## Appendix B Implementation Details

### B.1 GRPO Training Objective

We optimise the policy $\pi_{\theta}$, i.e., a vision-language model, to locate the target UI element with multi-step interaction using GRPO.
For each instruction $I$, we sample a batch of $N$ trajectories $\{\tau_{1},\dots,\tau_{N}\}$ from the current policy, where each trajectory $\tau_{i}$ is a sequence of generated responses at each step: $\tau_{i}=\{A_{0}^{i},A_{1}^{i},\dots,A_{T}^{i}\}$, and each response $A_{t}^{i}$ consists of tokens with a thinking process and an answer, as shown in [Eq. 1](https://arxiv.org/html/2509.21552v1#S2.E1).
We elaborate the rollout process in [Section B.2](https://arxiv.org/html/2509.21552v1#A2.SS2).
We calculate the reward $R$ for each sampled trajectory: $\{R_{\tau_{1}},R_{\tau_{2}},\dots,R_{\tau_{N}}\}$, where $R_{\tau_{i}}$ refers to the overall reward for the trajectory $\tau_{i}$. The reward $R_{\tau_{i}}$ is calculated by the weighted sum of the trajectory reward and a format reward $R_{\tau}=0.9\times R_{T}+0.1\times R_{\text{format}}$, where $R_{\text{format}}$ is $1$ when the output format follows the format in [Eq. 1](https://arxiv.org/html/2509.21552v1#S2.E1), otherwise it is 0.
Then, we calculate the relative advantage $\hat{A}_{\tau_{i}}$ of each sampled trajectory:

$$ $\hat{A}_{\tau_{i}}=\frac{R_{\tau_{i}}-\text{Mean}({\{R_{\tau_{1}},R_{\tau_{2}},\dots,R_{\tau_{N}}\}}}{\text{Std}(\{R_{\tau_{1}},R_{\tau_{2}},\dots,R_{\tau_{N}}\})}.$ (8) $$

The objective function seeks to increase the likelihood of high-reward trajectories:

$$ $\mathcal{J}(\theta)=\mathbb{E}_{\tau\sim\pi_{\theta}}\left[\sum_{i}\min\left(r_{i}(\theta)\hat{A}_{\tau_{i}},\text{clip}(r_{i}(\theta),1-\epsilon,1+\epsilon)\hat{A}_{\tau_{i}}\right)\right],$ (9) $$

where, $r_{i}(\theta)=\frac{\prod_{t=1}^{T}\pi_{\theta}(A_{t}^{i}\mid I,O_{0},A_{0},\dots,O_{t-1})}{\prod_{t=1}^{T}\pi_{\theta_{\text{old}}}(A_{t}^{i}\mid I,O_{0},A_{0},\dots,O_{t-1})}$ is the probability ratio for responses in the trajectory $\tau_{i}$, $\epsilon$ controls the trust region. We omit the KL regularisation term as previous work shows that it does not improve the grounding accuracy (Yang et al., 2025a).

### B.2 Moving Trajectory Rollout

Given an instruction $I$ and a GUI screenshot $O$, we rollout $N$ different trajectories $\{\tau_{i}\}_{i=1}^{N}$ for each GRPO optimisation step by the following process:
At the first step, we sample $N$ different responses $\{A_{0}^{i}\}_{i=1}^{N}$ conditioned on the same initial history $(I,O_{0})$, where each response has a thinking process before the action, and we do not constrain the action to be different.
At the subsequence steps, we sample $1$ response $A_{t}^{i}$ conditioned on the interaction history $(I,O_{0},A_{0}^{i},\dots,A_{t-1}^{i},O_{t}^{i})$.
Each trajectory $\tau_{i}$ stops growing until the action part in $A_{t}^{i}$ is STOP, or the maximum number of steps $T$ is reached.
We use vLLM (Kwon et al., 2023) for efficient inference during rollout.

### B.3 Relative and Direct Moving Strategies

We try two action strategies for moving the cursor: *relative move* and *direct move*.
In the relative move strategy, the model outputs relative offset $(\Delta x,\Delta y)$, and the position of the cursor will be moved from $(x,y)$ to $(x+\Delta x,y+\Delta y)$, where $\Delta x$ and $\Delta y$ are integers, positive and negative $\Delta x$ moves the cursor right and left, and positive and negative $\Delta y$ moves the cursor down and up, respectively.
We evaluate both strategies for GPT-4o and Qwen-2.5-VL-7B.
We present the prompt used by GPT-4o in [Fig. 20](https://arxiv.org/html/2509.21552v1#A6.F20).

We tuned the prompts of Qwen-2.5-VL-7B for both strategies. However, we found that it fails to conduct relative movement with an accuracy close to zero.
It might be because the model has been heavily fine-tuned on grounding data that requires directly outputting the coordinates.
Therefore, in Gui-Cursor, we employ the direct move strategy, which provides better starting accuracy for RL training.
The system prompt used by Gui-Cursor is presented in [Fig. 19](https://arxiv.org/html/2509.21552v1#A6.F19).

### B.4 Hyperparameter Settings

Gui-Cursor moves a cursor to locate the target.
We use a cursor image with a size of $20\times 31$, and we present it in [Fig. 10](https://arxiv.org/html/2509.21552v1#A5.F10).
We set the weight of the trajectory penalty $w_{p}$ to $0.2$ ([Eq. 7](https://arxiv.org/html/2509.21552v1#S2.E7)) in our experiments.
We empirically observe that the false stop penalty is crucial for preventing the model from making only a single prediction without utilising visual feedback, which may be because it results in a higher reward for trajectories with accurate movements.
We find that the accuracy improves slightly when increasing the weight of the false stop penalty $r_{\text{FS}}$ to $0.5$ and keeping other penalties $0.2$.
We train the model using 8 NVIDIA A100 80GB GPUs.
We implement RL training based on EasyR1 (Zheng et al., 2025)
During training, we evaluate the model on the validation set every 20 steps and save a checkpoint every 50 steps.
We measure the success rate on the validation set, where a sample is considered successful if the final predicted position is within the target area and the final action is STOP.
In our main experiments, we use the model at the 200th step, as the success rate does not improve in later steps, as shown in the third figure of [Fig. 8](https://arxiv.org/html/2509.21552v1#A4.F8).
Because sampling multi-step moving trajectories is time-consuming, we set the maximum number of movement steps to $4$ during training.
We find that setting the maximum steps to $3$ decreases around $1.0$ success rate in validation data.
We set the maximum response length to $512$.
We set the temperature to $0.5$ for sampling trajectories during training and use greedy decoding during inference.

### B.5 Cursor-Centric Focusing

During training, we set the maximum resolution $P$ to $1920\times 1080$, and any larger image will be downscaled to this resolution while keeping the original aspect ratio.
Qwen2.5-VL-7B uses a patch size of $14\times 14$ and aggregates each 4 adjacent patch features before forwarding to the transformer, so the maximum number of tokens for each image is around $26$k.
During inference, we apply ccf to obtain better accuracy in the task with a higher resolution than the maximum resolution during training.
It crops the full image based on the initial prediction.
The crop is sized to the maximum training resolution while maintaining the original image’s aspect ratio.
It attempts to centre this crop on the initial prediction; however, if the prediction is near an edge, the crop area is shifted to ensure it remains entirely within the image boundaries.
Although we have shown that ccf can improve accuracy effectively, its accuracy could be impacted by the initial prediction, as the target may be outside the focused area when the initial prediction is far away from the target.
We find that $10.3\%$ of examples in ScreenSpot-Pro have this issue.
Several potential strategies could be used to alleviate this issue, such as increasing the training resolution size to improve the initial prediction precision, keeping the initial image in the interaction history, and training the model to decide which area to focus.
We will explore these strategies in future work.

## Appendix C Number of Movement Steps and Target Sizes

Figure: Figure 7: Analysis of the relationship between moving steps and the size of the target UI element. Above: the average number of pixels of the targets. Below: the average proportion of target size in the GUI image. All results in the last column refer to the average values in both datasets. The red and blue bars refer to the samples that move one step and more than one step, respectively. The comparisons show that the target size is often smaller in the samples that move more than one step.
Refer to caption: x9.png

**Table 3: ScreenSpot-v2 accuracy (%) of VLMs without fine-tuning using the direct and relative moving strategies.**
| Model | Mobile | Desktop | Web | Average |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Text | Icon/Widget | Text | Icon/Widget | Text | Icon/Widget |  |  |
| GPT-4o | 20.5 | 22.7 | 21.1 | 24.3 | 10.0 | 6.3 | 17.5 |
| $+$ Direct Move, 10 Steps | 29.3 | 30.1 | 26.3 | 20.0 | 14.8 | 9.7 | 21.7 |
| $+$ Relative Move, 10 Steps | 46.3 | 27.7 | 36.6 | 15.0 | 17.5 | 9.9 | 25.5 |
| Qwen-2.5-VL-7B | 97.6 | 87.2 | 90.2 | 74.2 | 93.2 | 81.3 | 88.8 |
| $+$ Direct Move, 4 Steps | 50.0 | 28.3 | 57.2 | 19.0 | 42.9 | 20.3 | 36.3 |
| $+$ Relative Move, 4 Steps | 2.7 | 2.1 | 1.7 | 1.6 | 0.0 | 0.0 | 1.3 |

We analyse the relationship between the target UI element size and the moving steps.
In [Fig. 7](https://arxiv.org/html/2509.21552v1#A3.F7), we group the samples into two groups: only conducting one step movement after ccf, and conducting more than one step movement after ccf.
Within each group, we then calculate the average target size and the average proportion of the target in the image, shown in the first and second row of [Fig. 7](https://arxiv.org/html/2509.21552v1#A3.F7), respectively.
The first row shows that the samples with more than one step movement have a smaller target size.
The second row further shows that the target with a smaller relative size in an image may need more movement steps.
These results indicate that smaller targets may increase the difficulty of the tasks, and thus the model may require more movement steps in these samples.
We also analyse the average number of image pixels in each group, and we found that the higher resolution may not always be related to more movement steps, e.g., in ScreenSpot-Pro, the average number of pixels in the moving one-step group is $56\times 10^{5}$, which is larger than the moving more steps group $50\times 10^{5}$.

## Appendix D Training Dynamics

Figure: Figure 8: The training dynamics of Gui-Cursor. The first and second figures show the average number of response tokens and the average trajectory reward in each batch during training, respectively. The values are calculated after online filtering. The third figure shows the success rate every 20 training steps in the validation data. The success rate metric requires the final prediction to be within the target area, and the model explicitly outputs a STOP token at the last step.
Refer to caption: x10.png

We present the average number of response tokens, trajectory reward $R_{\text{T}}$ ([Eq. 7](https://arxiv.org/html/2509.21552v1#S2.E7)), and the success rate in validation data during training in [Fig. 8](https://arxiv.org/html/2509.21552v1#A4.F8), where the token numbers and the trajectory reward are calculated after the online filtering.
According to [Fig. 8](https://arxiv.org/html/2509.21552v1#A4.F8) and [Fig. 3(b)](https://arxiv.org/html/2509.21552v1#S3.F3.sf2), we observe that the learning of Gui-Cursor mainly has three stages.

*Phase one* (cold start) – From 0 to 25 steps, the response length decreases, and the trajectory reward increases. At this stage, the model learns to output the correct format, and the accuracy of prediction improves, resulting in higher trajectory reward values.

*Phase two* (single-step grounding) – From 50 to 150 steps, the response length decreases, and the trajectory reward also decreases. Since we apply the online filtering, the easy training samples will be filtered out. The decrease in the trajectory reward indicates that more difficult samples are used to train the model. As shown in [Fig. 3(b)](https://arxiv.org/html/2509.21552v1#S3.F3.sf2), the average number of movement steps is close to 1 at this stage. These results indicate that at this stage, the model continues to improve one-step grounding accuracy but hasn’t learnt a good cursor moving policy.

*Phase three* (multi-step movements) – From 150 to 200 steps, the trajectory rewards, the response length, and the average number of movement steps increase. This indicates that the model learns to execute multiple movements to refine the cursor position, thereby obtaining better accuracy.
Additionally, the increase in response length may indicate that the model generates more spatial analysis, thereby improving the moving correctness.

Figure: Figure 9: Accuracy of intermediate checkpoints on test benchmarks.
Refer to caption: x11.png

We also present the evaluation results on saved checkpoints (every 50 steps) in [Fig. 9](https://arxiv.org/html/2509.21552v1#A4.F9), which also shows a three-stage learning property.
We observe that on ScreenSpot-v2, the accuracy is highest at the 150th step, which is the end of the second stage.
In contrast, on the more difficult ScreenSpot-Pro benchmark, the accuracy continues to improve after the 150th step.
These results indicate that
1) the model improves one-step grounding accuracy at the first two stages, which is usually sufficient for easier tasks in ScreenSpot-v2;
2) it learns rational multi-step movement at the third stage, which helps to improve accuracy for more difficult tasks in ScreenSpot-Pro; and
3) though it hasn’t learn a moving policy well at the second stage, the visual feedback helps the alignment between coordinates and their on-screen positions, and the accuracy at the 150th steps in SceenSpot-v2 is $94.2$, significantly higher than the baselines presented in [Table 1](https://arxiv.org/html/2509.21552v1#S3.T1).

## Appendix E Cursor-in-Box Test

Figure: Figure 10: Example of a box position in the white background. The black frames are used for edge visualisation and are not a part of the image for testing. For each test example, we only show one box with one cursor in the white background.
Refer to caption: Figures/200.png

We design the ”cursor-in-box” test to evaluate how well a VLM can infer the spatial relationship between a cursor and another object.
We present the model with a white background showing a red bounding box and a black mouse cursor, and the model classifies whether the cursor is inside the box.
We present the prompt in [Fig. 18](https://arxiv.org/html/2509.21552v1#A6.F18).
We generate the test dataset by putting the bounding box across different positions of the white background, and putting the cursor inside or outside each box.
In [Fig. 10](https://arxiv.org/html/2509.21552v1#A5.F10), we present an example showing a box in a white background, and several cursors around it are used to test the classification accuracy at this box position.
We only show one box with one cursor for each test example.
For each box, we select $5$ positions inside it, with $4$ close to the corners and one at the centre; the positions outside the box have a distance shorter than $3$ times the cursor size.
We test the classification F1 scores at different positions of the background, and then obtain the heatmap as shown in [Fig. 6](https://arxiv.org/html/2509.21552v1#S4.F6).

## Appendix F Case Study

We present case studies and visualise the prediction process of Gui-Cursor.
We render the predicted cursor trajectory as a sequence of positions connected by blue lines.
Each cursor position is labelled with a red number at its top-left position to indicate the step number.
The target is highlighted with a red bounding box.
If ccf is applied, the corresponding focused region is shown with an orange bounding box.

We present positive examples with single-step movement ([Fig. 11](https://arxiv.org/html/2509.21552v1#A6.F11)), multi-step movement ([Fig. 17](https://arxiv.org/html/2509.21552v1#A6.F17), [Fig. 12](https://arxiv.org/html/2509.21552v1#A6.F12) and [Fig. 16](https://arxiv.org/html/2509.21552v1#A6.F16)).
We also present negative examples with single-step movement ([Fig. 15](https://arxiv.org/html/2509.21552v1#A6.F15)), and multi-step movement ([Fig. 13](https://arxiv.org/html/2509.21552v1#A6.F13) and [Fig. 14](https://arxiv.org/html/2509.21552v1#A6.F14)).

[Uncaptioned image]: Figures/Cases/1562-single-True.png

Figure: Figure 11: Positive example from ScreenSpot-Pro with single-step movements.

[Uncaptioned image]: Figures/Cases/CCF-1138-multi-True.png

Figure: Figure 12: Positive example from ScreenSpot-Pro with multi-step movements. The model changes targets during interaction.

[Uncaptioned image]: Figures/Cases/CCF-162-multi-False.png

Figure: Figure 13: Negative example from ScreenSpot-Pro with multi-step movements. The task is to locate the position for the given instruction, but the model hallucinates taking a series of actions.

[Uncaptioned image]: Figures/Cases/CCF-1518-multi-True-toomany.png

Figure: Figure 14: The cursor is correctly positioned, but the same position is predicted repeatedly. The model is waiting for the action to be executed.

[Uncaptioned image]: Figures/Cases/1542-misunderstand.png

Figure: Figure 15: Negative example of one-step movement. The model misidentifies the target

[Uncaptioned image]: Figures/Cases/1428-no-crop-pos.png

Figure: Figure 16: Positive example with multi-step movement. The model outputs STOP at the last step without thinking.

[Uncaptioned image]: Figures/Cases/method-case.png

Figure: Figure 17: Positive example with multi-step movement.

Figure: Figure 18: The prompt of ”cursor-in-box” test.

Figure: Figure 19: The system prompt of Gui-Cursor.

Figure: Figure 20: The system prompt of GPT-4o used in the relative move strategy.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: - It is unclear whether reinforcement learning is necessary for this method. Could the proposed approach also function effectively without additional training, for example in a test-time scaling or supervised fine-tuning (SFT) setting?
soundness: 3
strengths: - The draft is well-organized and clearly written.
- The intuition behind the proposed method is solid and straightforward.
- The authors provide well-designed ablation studies that help deepen the understanding of their approach.
summary: The authors tackle the mislocalization problem in existing visual GUI agents and propose an interactive visual reasoning approach that evaluates spatial relations between the cursor and the target. In this framework, the rendered cursor serves as visual feedback for vision-language models. The proposed method is validated on standard GUI grounding benchmarks.
weaknesses: - One concern lies in the novelty of the proposed method. In the broader vision domain — such as object grounding and visual question answering — many prior works have already utilized initial predictions as visual feedback, often through bounding boxes or check marks. This work appears to be a straightforward application of that idea to GUI grounding tasks, and the discussion does not sufficiently clarify how it differs from or advances beyond previous literature in general vision contexts.

- There are also concerns about the trade-off between training cost and performance. The current draft is not convincing regarding the practicality of this method. Please provide a detailed comparison of training efficiency with existing supervised fine-tuning (SFT) approaches such as GUI-Actor-7B or RL approaches GTA1-7B to better understand the improvements. Given the recent trend toward large-scale pretraining in GUI agents, such a Pareto analysis is particularly important.

- Experiments are conducted only on GUI grounding benchmarks. It remains unclear whether the proposed method generalizes to broader GUI agent tasks—especially those involving user instructions that do not exactly match GUI elements. Additional experiments on datasets such as AITW or Multimodal-Mind2Web would strengthen the claim that the proposed interactive reasoning approach can handle more realistic and diverse scenarios.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: **1. Ablation on single-step training:** Could the authors provide results for a single-step version of GUI-Cursor trained under identical settings (same base model, GRPO algorithm, training data, and position reward) but constrained to max_steps = 1? This would help isolate the contribution of multi-step interactive learning from other design factors.
 
**2. Evaluation in interactive agent settings:** While ScreenSpot-v2 and ScreenSpot-Pro are strong static grounding benchmarks, evaluating GUI-Cursor in more realistic interactive agent environments (e.g., *WebArena* and *Multimodal-Mind2Web* for web, *AndroidControl* for mobile, and *OSWorld* for operating system agents) could better demonstrate its applicability to real-world GUI agent tasks. 

**3. Comparison with concurrent work:** According to the latest ScreenSpot-Pro leaderboard ([https://gui-agent.github.io/grounding-leaderboard/](https://gui-agent.github.io/grounding-leaderboard)), more recent results report improved performance. For instance, GTA1-7B has been updated to 55.5%, and newer models such as GUI-ARP-7B (60.8%) and Holo1.5-7B (57.9%) achieve higher scores. Could the authors clarify whether these are concurrent submissions and discuss how GUI-Cursor compares to these latest results? If these methods were developed independently around the same time, a brief discussion of their methodological differences would be valuable for the community. 

**4. Figure presentation:** In Figure 2(b), some text labels overlap with the bars, which slightly affects readability. Improving the layout or adjusting the legend placement could make the comparison clearer.
soundness: 3
strengths: **1. Clear and well-motivated approach:** The reformulation of GUI grounding as an interactive search task with visual feedback is intuitive and addresses a fundamental limitation of existing methods: models receive supervision only on numerical coordinates without seeing where predictions actually land, leading to spatial-semantic misalignment.

**2. Effective inference strategy:** The proposed Cursor-Centric Focusing (CCF) balances computational efficiency and accuracy, particularly on high-resolution, complex GUIs.

**3. Strong empirical results:** GUI-Cursor-7B achieves state-of-the-art performance on both ScreenSpot-v2 (93.9%) and ScreenSpot-Pro (56.5%, +6.4% over prior best).

**4. Comprehensive and insightful analysis:** The paper provides detailed ablations, movement-step analysis, and spatial reasoning diagnostics that clarify how interaction and feedback improve spatial understanding. The cursor-in-box diagnostic test reveals that strong VLMs struggle with basic spatial reasoning and exhibit severe center bias, while interactive training on GUI grounding improves this capability as an emergent property, showing that visual feedback genuinely enhances spatial understanding beyond task-specific performance.

**5. Clear presentation:** The paper is well-written, logically structured, and supported by clear figures and experimental evidence.
summary: This paper proposes GUI-Cursor, a reinforcement learning framework for GUI grounding that reframes one-step coordinate prediction as an multi-step interactive search problem, where the model iteratively moves a virtual cursor to locate UI elements while receiving visual feedback at each step. The key insight is that existing VLMs suffer from spatial-semantic misalignment because they only receive supervision on numerical coordinates without seeing where their predictions actually land on the GUI. 
GUI-Cursor addresses this by training with GRPO reinforcement learning using a trajectory-level reward design (including penalties for false direction, false stop, repeated position, etc.) and a Cursor-Centric Focusing (CCF) inference strategy to balance efficiency and accuracy.
Built upon Qwen2.5-VL-7B, GUI-Cursor achieves state-of-the-art results on ScreenSpot-v2 (93.9%) and ScreenSpot-Pro (56.5%, +6.4% over prior best), with the model learning to adaptively use more steps for difficult cases (e.g., small targets). 
Additionally, cursor-in-box spatial reasoning test show that GUI-Cursor significantly enhances the model’s ability to understand and reason about spatial relationships between the cursor and visual elements, indicating that reinforcement-driven interaction helps build more robust spatial understanding beyond task-specific grounding.
weaknesses: **1. Lack of downstream validation:** The evaluation focuses solely on static grounding benchmarks without demonstrating practical benefits in real GUI agent systems or broader multimodal interaction scenarios.

**2. Title and spatial reasoning scope:**
The paper prominently features “Spatial Reasoning” in its title, but the treatment of this aspect is somewhat limited. The *cursor-in-box* diagnostic test is interesting, but the paper does not provide a deeper analysis of how GUI grounding relates to spatial reasoning or whether this capability can transfer to other spatial reasoning tasks. Additionally, the mechanism by which visual feedback enhances spatial understanding remains underexplored.

**3. Computational cost:** While 95% of examples are solved within 2 steps, the paper does not analyze the computational overhead of processing multiple high-resolution images with growing context windows, or compare inference efficiency against single-step baselines. The trade-off between accuracy gains and computational cost deserves more thorough investigation.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Please see Weaknesses. The reviewer is willing to raise the score if the authors address most, if not all, of the questions above in the Weakness section.
soundness: 3
strengths: S1: Reframing GUI grounding from static coordinate prediction to dynamic cursor-driven search is well-motivated.

S2: The combination of position-based reward with four specific trajectory penalties (false stop, false move, false direction, repeated position) effectively guides learning toward efficient search behaviors. 

S3: CCF balances computational constraints during training with accuracy needs during inference.
summary: This paper proposes a solution to the challenge that vision-language models struggle to accurately predict pixel coordinates for GUI elements by reframing GUI grounding as an interactive task. Instead of directly outputting coordinates, the model iteratively moves a virtual cursor on the screen, using visual feedback at each step to refine its position until reaching the target. The model is trained with reinforcement learning using rewards that encourage accurate positioning and penalize inefficient search patterns. GUI-Cursor achieves substantial improvements on ScreenSpot-Pro.
weaknesses: W1: The evaluation relies solely on two benchmarks, limiting confidence in generalizability. ScreenSpot-v2 is saturated with baselines already achieving >90%. With substantial gains observed only on ScreenSpot-Pro, it remains unclear whether the multi-step approach generalizes beyond the specific challenges of this benchmark. Evaluation on diverse benchmarks, like OSWorld-G[1] and UI-Vision[2], is necessary to validate that the computational overhead of multi-step interaction provides consistent benefits across different GUI grounding scenarios beyond the ScreenSpot family


W2: The interaction history grows: ($I$, $O_0$,$A_0$, ..., $O_t$). Thus, multi-step interaction accumulates context linearly. With CCF, each $O$ is ~26k tokens.  So, for a hard case that requires 3 steps, the model processes ~78k image tokens and text tokens. Does it slow down inference significantly?

W3: Table 3 reveals that Qwen2.5-VL-7B drops from 88.8% to 36.3% accuracy when using direct cursor movement without fine-tuning. This dramatic failure suggests the base model cannot transfer its existing spatial understanding to cursor control. Consequently, GUI-Cursor's success may primarily reflect learning a new interaction interface (cursor mechanics) rather than improved spatial reasoning. The paper claims to improve "spatial semantic alignment" and "spatial reasoning," but the evidence suggests interface adaptation is the dominant factor. Though the cursor-in-box test shows modest improvements, it uses a highly simplified setting that is far from practical usage.

[1] Xie et al. Scaling Computer-Use Grounding via User Interface Decomposition and Synthesis.

[2] Nayak et al. UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interaction.

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 3
questions: Please refer to the weaknesses.
soundness: 2
strengths: - Clear problem reframing. Turning coordinate emission into an interactive search with visual feedback is intuitive and well motivated by spatial–semantic misalignment in patch‑token VLMs. The cursor overlay makes the prediction visible to the model, which is a nice design touch.
- Simple, effective shaping. The four trajectory penalties are targeted at real degeneracies (premature STOP, oscillation, moving the wrong way, repetition). Ablations show each term helps; the gains are especially visible on ScreenSpot‑Pro where large canvases exacerbate these errors.
- A pragmatic efficiency recipe. Training at lower resolution and then zooming near the first guess is a strong “80/20” trick. The authors also apply the same focus idea to baselines, which is thoughtful.
summary: The paper reframes GUI grounding from one-shot coordinate generation to an interactive cursor-moving process. A Qwen2.5‑VL‑7B observes a screenshot with a rendered cursor, reasons about the target and the cursor–target relation, and iteratively issues either a new (x, y) or STOP. Training uses GRPO with a dense distance reward and trajectory penalties. To keep training affordable, the model is trained at 1920×1080 and, at inference, applies cursor‑centric focusing (CCF): one coarse step on the full screen, then crops around the initial click for fine steps. On ScreenSpot‑v2 and ScreenSpot‑Pro, the method reports good performance and shows that most cases finish in 2 steps. The paper also probes spatial reasoning via a simple “cursor‑in‑box” test and argues that interaction plus visual feedback strengthens spatial understanding.
weaknesses: - The core idea of iterative focus via cropping after an initial guess now appears in multiple concurrent works. GUI‑Spotlight iteratively narrows focus with dedicated crop tools and multi‑turn RL; GUI‑ARP performs adaptive multi‑stage inference with GRPO and attention‑guided crops; GUI‑RC/GUI‑RCPO refine grounding at inference via test‑time consensus/RL. The paper should position CCF and multi‑step cursor moves against these, not just older one‑shot baselines. As it stands, CCF feels close to those “zoom‑in” strategies, and the contribution reduces to (i) rendering a cursor as visual feedback and (ii) a particular set of penalties. [1, 2]
- Outdated SOTA claim. Newer papers report higher ScreenSpot‑Pro accuracy (e.g., GUI‑ARP 60.8% with a 7B backbone), which overtakes the reported 56.5%. The paper should update comparisons and discuss where it still wins (e.g., step efficiency) versus absolute accuracy. [2]
- Missing or under‑cited related work. The submission’s Related Work does not cover GUI‑Spotlight (iterative focus with tools) [1], GUI‑ARP (adaptive region perception and stage control) [2], nor test‑time RL via region consistency (GUI‑RC/GUI‑RCPO) [3]. These are directly relevant to the “interactive narrowing + RL” story and should be acknowledged.
- The cursor‑in‑box probe is neat but synthetic. Given substantial evidence that VLMs struggle with spatial relations, evaluation on public spatial benchmarks (e.g., SpatialMQA, SPHERE) would strengthen the generalization claim [4].

## References
[1] GUI-Spotlight: Adaptive Iterative Focus Refinement for Enhanced GUI Visual Grounding.

[2] GUI-ARP: Enhancing Grounding with Adaptive Region Perception for GUI Agents.

[3] Test-Time Reinforcement Learning for GUI Grounding via Region Consistency.

[4] Can Multimodal Large Language Models Understand Spatial Relations?


Decision: Reject