GTA1: GUI Test-time Scaling Agent

## Abstract

Abstract Graphical user interface (GUI) agents autonomously operate across platforms ( e.g. , Linux) to complete tasks by interacting with visual elements. Specifically, a user instruction is decomposed into a sequence of action proposals, each corresponding to an interaction with the GUI. After each action, the agent observes the updated GUI environment to plan the next step. However, two main challenges arise:
i) resolving ambiguity in task planning (i. e., the action proposal sequence), where selecting an appropriate plan is non-trivial, as many valid ones may exist;
ii) accurately grounding actions in complex and high-resolution interfaces, i. e., precisely interacting with visual targets. This paper investigates the two aforementioned challenges with our G UI T est-time Scaling A gent, namely GTA1. First, to select the most appropriate action proposal, we introduce a test-time scaling method. At each step of task execution, we sample multiple candidate action proposals and leverage a judge model to evaluate and select the most suitable one. It trades off computation for better decision quality by concurrent sampling, shortening task execution steps, and improving overall performance.
Second, we propose a model that achieves improved accuracy when grounding the selected action proposal to its corresponding visual elements. Our key insight is that reinforcement learning (RL) facilitates visual grounding through inherent objective alignments, rewarding successful clicks on interface elements. Experimentally, our GUI grounding model establishes state-of-the-art performance across diverse benchmarks.
For example, GTA1-7B achieves 50.1%, 92.4%, and 67.7% accuracies on Screenspot-Pro, Screenspot-V2, and OSWorld-G benchmarks, respectively.
Furthermore, when paired with a planner applying our test-time scaling strategy, it exhibits state-of-the-art agentic performance ( e.g. , 45.2% task success rate on OSWorld benchmark). We open-source our code and models here .

## 1 Introduction

Automating user task execution across diverse platforms through GUI agents represents a significant milestone toward general artificial intelligence, supporting users in tasks ranging from placing online orders to performing complex, expert-level professional workflows [1]. To solve a task, a GUI agent translates user instructions into multi-steps of interactions such as action proposals consisting of clicks or keystrokes [2]. This introduces ambiguity in planning, as multiple valid action proposal sequences may exist for the same user task. The challenge is further intensified by the high-resolution (up to 4K), complex, and hierarchical layouts of GUI [3, 4, 5, 6], requiring accurately identifying the coordinates to interact with the target user interface (UI) elements. This work aims to address both challenges (i. e., planning and grounding) towards a performant GUI agent.

Formally, existing works [1, 7, 8, 6, 9] often pair a GUI grounding model with a planner (*e.g.*, Gemini 2.5 [10], Claude 3.7 [11], or o3 [12]). The planner is responsible for determining the action proposal required at each step, while the grounding model is used to locate the target interface elements when an action must be visually grounded.
However, due to the inherent flexibility of user tasks, multiple feasible action proposal sequences may exist for completing the same task. Some of these plans are more direct and efficient, while others may involve unnecessarily long or complex action sequences.
This makes the agent highly susceptible to cascading failures, i. e., errors in early grounding or planning steps can derail the entire task. This fragility is often reflected in the high performance variance among multiple runs of the same agent [13, 6]. A straightforward idea to avoid it is rollouting the full action proposal sequences in advance. However, unlike domains such as mathematical problem-solving, GUI environments lack “lookahead” capability: agents cannot rollout several multi-step plans and retrospectively choose the best one, because executing actions in GUI environments usually has irreversible state effects.
Therefore, we pose a central question:

How can we design GUI agents that remain robust in planning, considering the lack of “lookahead” and the presence of multiple plausible action proposal sequences?

Figure: Figure 1: Performance comparisons of our method and state-of-the-art methods. Each method is labeled with its name next to its icon; a red dot represents our method. (a) Grounding accuracy (%) on the ScreenSpot-Pro benchmark [3] across various model scales in billions (B) of parameters. (b) Task success rate (%) on the OSWorld benchmark [14], shown as it evolves over time.
Refer to caption: /html/2507.05791/assets/x1.png

Beyond planning, GUI grounding models usually rely on supervised fine-tuning (SFT) [1, 5, 2, 4, 6]. It rigidly trains models to predict the exact center of the target element. While this approach can be effective, it often struggles to generalize in complex GUI environments, particularly in out-of-domain professional interfaces [3]. Moreover, the formulation of SFT introduces a mismatch with the nature of the GUI grounding task: any coordinate within the target element should be considered a valid prediction, yet SFT optimizes only for the center, limiting the model flexibility and robustness, and preventing the model from perceiving appropriate supervision signals.

In the alternative, motivated by the success of DeepSeek-R1-Zero [15], RL, specifically Group Relative Policy Optimization (GRPO) [16], has been explored for the GUI grounding tasks. Following [15], past works [17, 18, 19] formulates by first performing a “thinking” (i. e., textual CoT reasoning), and then deriving a coordinate prediction. In general, the model is trained using a format function that enforces the presence of textual reasoning, and a click reward that verifies whether the predicted coordinate falls within the target element region. Meanwhile, some works [19, 20] also extend this formulation to predict the bounding box of the target UI element. While these approaches have shown performance improvements over SFT, we ponder another fundamental question:

Is explicit “thinking” or auxiliary bounding box reward truly necessary for effective GUI grounding?

We investigate these two questions by proposing two complementary strategies:
i) a test-time scaling strategy for planning, which works in tandem with a grounding model to execute user tasks more robustly;
ii) a RL-based grounding model to predict interaction coordinates.
Specifically, we develop a test-time scaling strategy that addresses planning ambiguity without requiring “lookahead”. For an agent consisting of a grounding model and a planner, our method avoids committing to a single action proposal sequence to complete a user task. Instead, at each step of task execution, we sample multiple candidate action proposals from the planner. These candidates are then evaluated using a multimodal large language model, which act as a judge to select the most contextually appropriate option. Before executing the chosen action proposal in the GUI environment,
we ground the actions to the target visual element by predicting its associated coordinates using the grounding model if the action is coordinate-based, ensuring accurate interaction with the target interface element.
This strategy enables the agent to navigate ambiguous tasks more robustly by exploring short-term alternatives without relying on forward rollout of the action proposal sequence.

Then, we introduce
a simple and straightforward approach that optimizes a GUI grounding model by designing it to directly predict a coordinate, and rewarding the model based on whether the predicted point falls within the target UI element. Due to the objective alignment between the training signal and the grounding task, this method is highly efficient. Despite its simplicity, this approach achieves state-of-the-art performance across diverse GUI grounding benchmarks. Interestingly, we also find that “thinking” in the form of reasoning over the task object, past trajectories, and user instructions can be beneficial in dynamic environments where context evolves over time. However, such reasoning is usually difficult to generalize in broader grounding scenarios, limiting its practicality.
Our contributions are summarized as follows:

- •
we conduct a comprehensive study of GUI agents, focusing on the key challenges of grounding and planning in real-world, high-resolution, and dynamic UI environments;
- •
we propose a simple yet effective GUI grounding model that directly predicts interaction coordinates without requiring explicit reasoning;
- •
we introduce a test-time scaling strategy for the planner that pairs with the grounding model, improving planning robustness, and resolving execution uncertainty.

A performance overview is shown in [Fig. 1](#S1.F1). Our method consistently outperforms previous approaches across diverse GUI grounding benchmarks. Furthermore, using o3 [12] as a planner and a judge for test-time scaling,
GTA1 demonstrates precise interaction in real environments to accomplish user tasks, establishing a lightweight and effective pathway toward agentic GUI behavior.

## 2 Related Work

We review recent achievements of GUI agents, with a primary focus on multimodal large language model-based agents. We exclude studies that do not rely on visual capabilities (*e.g.*, those using HTML or a11y trees). We first reviews the GUI grounding, and then categorize existing agents into two broad types: native GUI agents and two-stage GUI agents.

#### GUI Grounding.

GUI grounding refers to the task of mapping user instructions (usually at a low level) to specific coordinates corresponding to target UI elements. Early works [1, 5, 9, 2] primarily focus on SFT, training models to predict the center point of the intended UI element. However, this approach does not fully align with the nature of the GUI grounding task, where any coordinate within the target element should be considered a success. As a result, SFT-based models often exhibit poor generalization, particularly when evaluating on high-resolution and visually complex user interfaces [3]. With the success of DeepSeek-R1-Zero [15], RL (specifically GRPO [16]) has drawn increased attention in this domain. However, many recent efforts naively replicate techniques from other domains [17, 18, 19], such as prompting the model to generate a “thinking” (i. e., CoT reasoning) before producing an answer, and the answer is rewarded only if the predicted coordinates fall within the target element region. This strategy overlooks an important insight: the “thinking” degrade performance in GUI grounding, where precise spatial perception matters more than multi-step verbal reasoning. The main benefit of RL in the GUI grounding task lies in objective alignment, rewarding any coordinate within the correct region rather than producing a textual “thinking”. A concurrent study [20] makes similar observations, noting that CoT reasoning (i. e., “thinking”) is not required for RL training in GUI grounding and may even hinder grounding accuracy. Our work further distinguishes itself in the following ways:
i) we clarify that “thinking” is not necessary for GUI grounding in static environments;
ii) we demonstrate that “thinking” improves grounding performance in dynamic, real-world environments when provided with past trajectories and task objectives;
iii) we conduct a comprehensive study of RL-based GUI grounding across models of various scales. Going beyond prior work, we also evaluate how the model, when paired with a planner, performs in realistic and dynamic environments, a critical aspect that existing studies largely overlook.

#### Two-stage GUI Agent.

One major challenge in GUI grounding is accurately locating the coordinates of UI elements intended for interaction. To address this, two-stage GUI agents modularize into planning and action, each handled by separate models [2]. They usually leverage advanced reasoning models, such as GPT-4o [21] and Claude 3.7 [11], as planners to generate an action proposal for each step from the user task instruction, using real-time UI screenshot and past trajectories as context. A separate visual grounding module then maps these instructions to specific UI elements, enabling the development of vision-only GUI agents [5, 1, 2]. While most existing work primarily focuses on GUI grounding, more complex components, such as memory management and external knowledge bases, are also being explored to enhance agent performance [8]. This paper follows the two-stage method on establishing a GUI agent.

#### Native GUI Agent.

A native GUI agent completes user tasks in an end-to-end manner.
Four main aspects are studied:
i) perception, interpreting the user interface to understand the current state;
ii) memory, storing knowledge and historical experiences to support making decisions;
ii) planning, analyzing the task and reflecting on progress to generate action proposals;
iv) action, performing atomic operations based on the action proposal to effectively progress toward the task goal. We refer the reader to [13] for more details. Some examples of native GUI agents can be specified to CUA [21, 12] and Claude Computer Use [11]. One of the main challenges for native GUI agents lies in long-context learning. To address this, some approaches employ a sliding window mechanism [13], while others maintain a textual description of past trajectories to manage contextual information [9]. In practice, end-to-end native GUI agents have demonstrated strong performance in completing user tasks, as shown by benchmarks that reflect dynamic and realistic UI environments, such as OSWorld [14]. However, this paper is the first to show that a two-stage GUI agent can achieve competitive performance in such environments, challenging the assumption that end-to-end approaches are inherently superior.

## 3 Method

#### Overview.

Our method adopts a two-stage GUI agent framework composed of a planner and a grounding model, and focuses on improving planning robustness and grounding accuracy through the following key components: i) test-time scaling for planning, which introduces a scaling strategy during inference to effectively handle planning ambiguity in complex GUI environments;
ii) grounding model training, filtering out training samples with annotation errors to improve supervision quality, and optimizing a grounding model using RL (*e.g.*, GRPO) to directly predict coordinates without relying on any intermediate “thinking” (i. e., CoT reasoning) on the derived data.

Figure: Figure 2: Overview of our GUI agent architecture. At each step, the trajectory, current UI screenshot, and user instruction are sent to a planner, which samples multiple action proposals. A multimodal large language model judge is then used to select the best candidate action proposal. When the candidate action proposal is a coordinate-based action (*e.g.*, a click), the grounding model predicts a precise interaction point on the GUI for executing the action. For non-coordinate-based actions (*e.g.*, key presses), the action can be executed directly without grounding.
Refer to caption: /html/2507.05791/assets/x2.png

### 3.1 Test-time Scaling for Planning

At each step of executing user instructions in real-world environments, a planner is provided with the user instruction (i. e., task objective), the trajectory so far, and the current UI screenshot. Based on this context, we sample $K$ candidate action proposals, denoted as $\{\boldsymbol{a}_{k}\}_{k=1}^{K}$, where $\boldsymbol{a}_{k}$ represents a corresponding action (*e.g.*, clicking ‘the blue button’ or performing a keystroke).

Then, a multimodal large language model judge is used to evaluate the $K$ candidates $\{\boldsymbol{a}_{k}\}_{k=1}^{K}$ based on their alignment with the user intent and the current GUI state. The judge, which can be the planner model itself, picks the best candidate action proposal $\boldsymbol{a}_{k*}$ from $\{\boldsymbol{a}_{k}\}_{k=1}^{K}$, allowing the agent to select the most contextually appropriate option. Once the best candidate action proposal $\boldsymbol{a}_{k*}$ is selected, the grounding model $\pi(\cdot,\cdot)$ takes $\boldsymbol{a}_{k*}$ and the current screenshot $\boldsymbol{x}$ as the input. If $\boldsymbol{a}_{k*}$ is a coordinate-based action (*e.g.*, a click), the grounding model predicts a precise interaction point on the GUI, which is then used to execute the action. For non-coordinate-based actions (*e.g.*, key presses or text input), the action can be executed directly without grounding. This process is repeated step by step until the task is completed or the agent reaches a termination condition.

By incorporating sampling, judging, and grounding into each step, our agent avoids overcommitting to suboptimal action proposals and demonstrates improved robustness in complex, ambiguous, and dynamic GUI environments. We show an overview in [Fig. 2](#S3.F2).

Figure: Figure 3: Examples from the Aria-UI dataset collection [1]. The blue bounding box shows the derived annotation $b^{\mathsf{ann}}$, while the red bounding boxes are detected using OmniParser [22]. A large green arrow is used to draw attention to the misaligned blue bounding box. Our lightweight cleaning strategy filters out such cases where the annotation does not match the actual UI element.
Refer to caption: /html/2507.05791/assets/x3.png

### 3.2 Grounding

#### Data Cleaning.

We leverage dataset collections curated by Aria-UI [1] and OS-Atlas [4]. To train with a reward signal that verifies whether the predicted coordinates fall within the target UI element, we require a dataset that provides accurate bounding boxes for annotated interactive elements. For data points from desktop and web domains, the collection usually involves screenshots paired with accessibility tools such as A11y or HTML parsers to extract element structure and bounding box annotations. However, these bounding boxes can sometimes be misaligned with the actual visual rendering due to UI animations, timing inconsistencies, or rendering delays, introducing noise into the training signal.

Therefore, to improve data quality, we apply a lightweight cleaning strategy. Given a UI screenshot $\boldsymbol{s}$, we use OmniParser [22], denoted as $\mathcal{P}(\cdot)$, to detect all UI elements in $\boldsymbol{s}$, resulting in a set of bounding boxes $\{b_{i}\}=\mathcal{P}(\boldsymbol{s})$. Each $b_{i}$ represents the bounding box of a detected UI element.

For a data point $\boldsymbol{s}$ with an annotated bounding box $b^{\mathsf{ann}}$, we discard the sample if the maximum Intersection over Union between $b^{\mathsf{ann}}$ and any $b_{i}\in\{b_{i}\}$ is smaller than a predefined threshold $\tau$,

$$ $\displaystyle\max_{b_{i}\in\mathcal{P}(\boldsymbol{s})}\text{IoU}(b^{\mathsf{ann}},b_{i})<\tau\ ,$ (1) $$

where $\text{IoU}(\cdot,\cdot)$ computes the overlap between two bounding boxes, defined as the area of their intersection divided by the area of their union. This helps ensure that the annotation $b^{\mathsf{ann}}$ in the training data remains consistent with the actual visual target, thereby reducing noise caused by misaligned annotations. We show some samples in [Fig. 3](#S3.F3).

#### Training.

In our RL training, we follow the GRPO framework [15, 16] to sample $N$ responses $\{\boldsymbol{o}_{n}\}_{n=1}^{N}$ from the policy multimodal large language model $\pi(\cdot,\cdot)$, given a screenshot $\boldsymbol{s}$ and an action proposal $\boldsymbol{a}$ as input. Here, each response $o_{n}$ represents a pair of pixel coordinates on the screen, i. e., $\boldsymbol{o}_{n}=(x_{n},y_{n})$, where $x_{n}$ and $y_{n}$ denote the horizontal and vertical positions, respectively.
Unlike prior approaches, we do not prompt the model to generate a “thinking” (i. e., intermediate CoT reasoning) before producing a response. Instead, the model directly outputs the predicted coordinates, aligning more closely with the nature of the GUI grounding task.

Then, each response is evaluated by checking whether the predicted coordinate $(x_{n},y_{n})$ falls within the annotated bounding box $\mathbf{b}^{\mathsf{ann}}=(x_{\text{min}},y_{\text{min}},x_{\text{max}},y_{\text{max}})$. This yields a set of $N$ binary rewards $\{r_{n}\}_{n=1}^{N}$, where each reward is

$$ $\displaystyle r_{n}=\begin{cases}1,&\text{if }x_{\text{min}}\leq x_{n}\leq x_{\text{max}}\text{ and }y_{\text{min}}\leq y_{n}\leq y_{\text{max}}\ ,\\ 0,&\text{otherwise}\ .\end{cases}$ (2) $$

We then normalize the rewards $\{r_{n}\}_{n=1}^{N}$ into advantages using Z-score normalization,

$$ $\displaystyle v_{n}=\frac{r_{n}-\frac{1}{N}\sum_{n=1}^{N}r_{n}}{\sqrt{\frac{1}{N}\sum_{n=1}^{N}(r_{n}-\frac{1}{N}\sum_{n=1}^{N}r_{n})^{2}}}\ .$ (3) $$

Finally, the model is optimized by

$$ $\displaystyle\mathcal{L}=-\frac{1}{N}\sum_{n=1}^{N}\frac{\pi(o_{n}\mid\boldsymbol{x},\boldsymbol{a})}{\pi^{\mathsf{old}}(o_{n}\mid\boldsymbol{x},\boldsymbol{a})}\cdot v_{n}\ ,$ (4) $$

where $\pi^{\mathsf{old}}(\cdot\mid\cdot,\cdot)$ denotes the old policy, and $v_{n}$ is the advantage associated with the prediction $o_{n}$. The advantage serves as a weight to encourage the model to increase the likelihood of high reward predictions while suppressing low-reward ones.

**Table 1: Comparison with state-of-the-art methods on the ScreenSpot-Pro dataset [3]. We report the grounding accuracy (%) across various task domains, categorizing results by grounding target type: Text, Icon, and the overall average (Avg). We use ‘-’ to denote unavailability, and ‘^∗’ to denote the results evaluated by us (which will be updated if improved evaluation scripts become available). The final average scores are highlighted in dark blue.**
| Agent Model | Development | Creative | CAD | Scientific | Office | OS | Avg |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Text | Icon | Avg | Text | Icon | Avg | Text | Icon | Avg | Text | Icon | Avg | Text | Icon | Avg | Text | Icon | Avg | Text | Icon | Avg |  |
| Proprietary Models |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| GPT-4o [21] | 1.3 | 0.0 | 0.7 | 1.0 | 0.0 | 0.6 | 2.0 | 0.0 | 1.5 | 2.1 | 0.0 | 1.2 | 1.1 | 0.0 | 0.9 | 0.0 | 0.0 | 0.0 | 1.3 | 0.0 | 0.8 |
| Claude 3.7 Sonnet [11] | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | 27.7 |
| Operator [23] | 50.0 | 19.3 | 35.1 | 51.5 | 23.1 | 39.6 | 16.8 | 14.1 | 16.1 | 58.3 | 24.5 | 43.7 | 60.5 | 28.3 | 53.0 | 34.6 | 30.3 | 32.7 | 45.0 | 23.0 | 36.6 |
| Seed-1.5-VL [24] | - | - | 53.8 | - | - | 59.2 | - | - | 59.0 | - | - | 61.4 | - | - | 74.8 | - | - | 60.2 | - | - | 60.9 |
| UI-TARS-1.5 [13] | - | - | 63.9 | - | - | 50.4 | - | - | 58.2 | - | - | 69.3 | - | - | 79.6 | - | - | 51.0 | - | - | 61.6 |
| Open-Source Models |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| SeeClick [5] | 0.6 | 0.0 | 0.3 | 1.0 | 0.0 | 0.6 | 2.5 | 0.0 | 1.9 | 3.5 | 0.0 | 2.0 | 1.1 | 0.0 | 0.9 | 2.8 | 0.0 | 1.5 | 1.8 | 0.0 | 1.1 |
| Qwen2-VL-7B [25] | 2.6 | 0.0 | 1.3 | 1.5 | 0.0 | 0.9 | 0.5 | 0.0 | 0.4 | 6.3 | 0.0 | 3.5 | 3.4 | 1.9 | 3.0 | 0.9 | 0.0 | 0.5 | 2.5 | 0.2 | 1.6 |
| ShowUI-2B [26] | 16.9 | 1.4 | 9.4 | 9.1 | 0.0 | 5.3 | 2.5 | 0.0 | 1.9 | 13.2 | 7.3 | 10.6 | 15.3 | 7.5 | 13.5 | 10.3 | 2.2 | 6.6 | 10.8 | 2.6 | 7.7 |
| CogAgent-18B [27] | 14.9 | 0.7 | 8.0 | 9.6 | 0.0 | 5.6 | 7.1 | 3.1 | 6.1 | 22.2 | 1.8 | 13.4 | 13.0 | 0.0 | 10.0 | 5.6 | 0.0 | 3.1 | 12.0 | 0.8 | 7.7 |
| Aria-UI [1] | 16.2 | 0.0 | 8.4 | 23.7 | 2.1 | 14.7 | 7.6 | 1.6 | 6.1 | 27.1 | 6.4 | 18.1 | 20.3 | 1.9 | 16.1 | 4.7 | 0.0 | 2.6 | 17.1 | 2.0 | 11.3 |
| UI-R1-3B [18] | 22.7 | 4.1 | - | 27.3 | 3.5 | - | 11.2 | 6.3 | - | 42.4 | 11.8 | - | 32.2 | 11.3 | - | 13.1 | 4.5 | - | - | - | 17.8 |
| OS-Atlas-7B [4] | 33.1 | 1.4 | 17.7 | 28.8 | 2.8 | 17.9 | 12.2 | 4.7 | 10.3 | 37.5 | 7.3 | 24.4 | 33.9 | 5.7 | 27.4 | 27.1 | 4.5 | 16.8 | 28.1 | 4.0 | 18.9 |
| UI-TARS-2B [28] | 47.4 | 4.1 | 26.4 | 42.9 | 6.3 | 27.6 | 17.8 | 4.7 | 14.6 | 56.9 | 17.3 | 39.8 | 50.3 | 17.0 | 42.6 | 21.5 | 5.6 | 14.3 | 39.6 | 8.4 | 27.7 |
| Qwen2.5-VL-3B [25] | 38.3 | 3.4 | 21.4 | 40.9 | 4.9 | 25.8 | 22.3 | 6.3 | 18.4 | 44.4 | 10.0 | 29.5 | 48.0 | 17.0 | 40.9 | 33.6 | 4.5 | 20.4 | 37.8 | 6.6 | 25.9 |
| Qwen2.5-VL-7B [25] | 51.9 | 4.8 | 29.1 | 36.9 | 8.4 | 24.9 | 17.8 | 1.6 | 13.8 | 48.6 | 8.2 | 31.1 | 53.7 | 18.9 | 45.7 | 34.6 | 7.9 | 22.4 | 39.9 | 7.6 | 27.6 |
| UGround-7B [2] | - | - | 35.5 | - | - | 27.8 | - | - | 13.5 | - | - | 38.8 | - | - | 48.8 | - | - | 26.1 | - | - | 31.1 |
| UGround-72B  [2] | - | - | 31.1 | - | - | 35.8 | - | - | 13.8 | - | - | 50.0 | - | - | 51.3 | - | - | 25.5 | - | - | 34.5 |
| UI-TARS-7B [28] | 58.4 | 12.4 | 36.1 | 50.0 | 9.1 | 32.8 | 20.8 | 9.4 | 18.0 | 63.9 | 31.8 | 50.0 | 63.3 | 20.8 | 53.5 | 30.8 | 16.9 | 24.5 | 47.8 | 16.2 | 35.7 |
| InfiGUI-R1-3B [19] | 51.3 | 12.4 | 32.4 | 44.9 | 7.0 | 29.0 | 33.0 | 14.1 | 28.4 | 58.3 | 20.0 | 41.7 | 65.5 | 28.3 | 57.0 | 43.9 | 12.4 | 29.6 | 49.1 | 14.1 | 35.7 |
| SE-GUI-3B [29] | 55.8 | 7.6 | 35.1 | 47.0 | 4.9 | 29.0 | 38.1 | 12.5 | 31.8 | 61.8 | 16.4 | 43.3 | 59.9 | 24.5 | 50.9 | 40.2 | 12.4 | 25.5 | 50.4 | 11.8 | 35.9 |
| Jedi-3B [6] | 61.0 | 13.8 | 38.1 | 53.5 | 8.4 | 34.6 | 27.4 | 9.4 | 23.0 | 54.2 | 18.2 | 38.6 | 64.4 | 32.1 | 57.0 | 38.3 | 9.0 | 25.0 | 49.8 | 13.7 | 36.1 |
| GUI-G1-3B [20] | 50.7 | 10.3 | 31.1 | 36.6 | 11.9 | 26.6 | 39.6 | 9.4 | 32.2 | 61.8 | 30.0 | 48.0 | 67.2 | 32.1 | 59.1 | 23.5 | 10.6 | 16.1 | 49.5 | 16.8 | 37.1 |
| UI-TARS-72B [28] | 63.0 | 17.3 | 40.8 | 57.1 | 15.4 | 39.6 | 18.8 | 12.5 | 17.2 | 64.6 | 20.9 | 45.7 | 63.3 | 26.4 | 54.8 | 42.1 | 15.7 | 30.1 | 50.9 | 17.5 | 38.1 |
| Jedi-7B [6] | 42.9 | 11.0 | 27.4 | 50.0 | 11.9 | 34.0 | 38.0 | 14.1 | 32.2 | 72.9 | 25.5 | 52.4 | 75.1 | 47.2 | 68.7 | 33.6 | 16.9 | 26.0 | 52.6 | 18.2 | 39.5 |
| UI-TARS-1.5-7B^∗ [28] | 58.4 | 12.4 | 31.8 | 58.1 | 15.4 | 40.2 | 38.6 | 11.0 | 31.8 | 66.7 | 21.9 | 47.2 | 74.6 | 35.9 | 65.6 | 49.5 | 13.5 | 33.2 | 57.5 | 16.9 | 42.0 |
| Qwen2.5-VL-32B [25] | 74.0 | 21.4 | 48.5 | 61.1 | 13.3 | 41.1 | 38.1 | 15.6 | 32.6 | 78.5 | 29.1 | 57.1 | 76.3 | 37.7 | 67.4 | 55.1 | 27.0 | 42.3 | 63.2 | 22.5 | 47.6 |
| SE-GUI-7B [29] | 68.2 | 19.3 | 44.5 | 57.6 | 9.1 | 37.2 | 51.3 | 42.2 | 42.1 | 75.0 | 28.2 | 54.7 | 78.5 | 43.4 | 70.4 | 49.5 | 25.8 | 38.8 | 63.5 | 21.0 | 47.3 |
| Qwen2.5-VL-72B [25] | - | - | 53.5 | - | - | 44.9 | - | - | 44.4 | - | - | 59.1 | - | - | 72.6 | - | - | 49.5 | - | - | 53.3 |
| GTA1-7B | 66.9 | 20.7 | 44.5 | 62.6 | 18.2 | 44.0 | 53.3 | 17.2 | 44.4 | 31.8 | 76.4 | 57.1 | 82.5 | 50.9 | 75.2 | 48.6 | 25.9 | 38.3 | 65.5 | 25.2 | 50.1 |
| GTA1-32B | 82.5 | 28.3 | 56.2 | 69.2 | 14.7 | 46.3 | 43.7 | 23.4 | 38.7 | 79.9 | 31.8 | 59.1 | 80.8 | 43.4 | 72.2 | 70.1 | 32.6 | 53.1 | 69.9 | 27.2 | 53.6 |
| GTA1-72B | 79.9 | 33.1 | 57.2 | 73.2 | 20.3 | 51.0 | 56.9 | 28.1 | 49.8 | 81.9 | 38.2 | 63.0 | 85.3 | 49.1 | 77.0 | 73.8 | 37.1 | 57.1 | 74.5 | 32.5 | 58.4 |

## 4 Experiment

#### Implementation Detail.

We train the grounding models using dataset collections from Aria-UI [1] and OS-Atlas [4], applying a data cleaning threshold of $\tau=0.3$. Our model is initialized from UI-TARS-1.5-7B [13], Qwen2.5-VL-32B-Instruct [25], and Qwen2.5-VL-72B-Instruct [25]. We use a learning rate of $10^{-6}$ and sample $N=8$ responses per input during training. The batch sizes are set to 256, 128, and 128 for the 7B, 32B, and 72B models, respectively. The 7B and 32B models usually converge around 250 iterations, while the 72B model converges around 50 iterations. When testing on the real-world dynamic environment, we use the action space from [8] and o3 as the planner [12].

#### Dataset.

We evaluate our method on two sets of benchmarks: i) GUI Grounding: we use ScreenSpot-V2 [5, 4], ScreenSpotPro [3], and OSWorld-G [14] datasets, evaluating using the metric of accuracy; ii) Agent Task Execution: we use OSWorld [14] benchmark, measuring performance by task success rate.

#### Baseline.

We compare with various state-of-the-art methods: CogAgent [27], OminiParser [22],
Qwen2.5-VL [25], Aria-UI [1], OS-Atlas [4], UGround [2], ShowUI [26], Aguvis [9], Jedi [6], GUI-G1 [20], SE-GUI [29], GUI-R1 [17], UI-R1 [18], InfiGUI-R1 [19], UI-TARS [13], Seed-1.5-VL [24], UI-TARS-1.5 [13], GPT-4o [21], o3 [12], Claude 3.7 Sonnet [11], and Gemini-2.5 [10].

**Table 2: Comparison with state-of-the-art methods on the ScreenSpot-V2 dataset [5, 4] across mobile, desktop, and web domains. We report grounding accuracy (%) categorized by grounding target type: Text, Icon/Widget, and the overall Average (Avg). We use ‘-’ to denote unavailability, and ‘^∗’ to denote the results evaluated by us (which will be updated if improved evaluation scripts become available). The final average scores are highlighted in dark blue.**
| Agent Model | Mobile | Desktop | Web | Avg |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Text | Icon/Widget | Text | Icon/Widget | Text | Icon/Widget |  |  |
| Proprietary Models |  |  |  |  |  |  |  |
| Operator [23] | 47.3 | 41.5 | 90.2 | 80.3 | 92.8 | 84.3 | 70.5 |
| Claude 3.7 Sonnet [11] | - | - | - | - | - | - | 87.6 |
| UI-TARS-1.5 [13] | - | - | - | - | - | - | 94.2 |
| Seed-1.5-VL [24] | - | - | - | - | - | - | 95.2 |
| Open-Source Models |  |  |  |  |  |  |  |
| SeeClick [24] | 78.4 | 50.7 | 70.1 | 29.3 | 55.2 | 32.5 | 55.1 |
| OmniParser-v2 [22] | 95.5 | 74.6 | 92.3 | 60.9 | 88.0 | 59.6 | 80.7 |
| Qwen2.5-VL-3B [25] | 93.4 | 73.5 | 88.1 | 58.6 | 88.0 | 71.4 | 80.9 |
| UI-TARS-2B [13] | 95.2 | 79.1 | 90.7 | 68.6 | 87.2 | 78.3 | 84.7 |
| OS-Atlas-Base-7B [4] | 95.2 | 75.8 | 90.7 | 63.6 | 90.6 | 77.3 | 85.1 |
| OS-Atlas-Base-7B [4] | 96.2 | 83.4 | 89.7 | 69.3 | 94.0 | 79.8 | 87.1 |
| Jedi-3B [6] | 96.6 | 81.5 | 96.9 | 78.6 | 88.5 | 83.7 | 88.6 |
| Qwen2.5-VL-7B [25] | 97.6 | 87.2 | 90.2 | 74.2 | 93.2 | 81.3 | 88.8 |
| UI-TARS-1.5-7B^∗ [13] | 95.9 | 84.8 | 94.9 | 80.7 | 90.6 | 86.2 | 89.7 |
| UI-TARS-72B [13] | 94.8 | 86.3 | 91.2 | 87.9 | 91.5 | 87.7 | 90.3 |
| UI-TARS-7B [13] | 96.9 | 89.1 | 95.4 | 85.0 | 93.6 | 85.2 | 91.6 |
| Jedi-7B [6] | 96.9 | 87.2 | 95.9 | 87.9 | 94.4 | 84.2 | 91.7 |
| Qwen2.5-VL-32B^∗ [25] | 98.3 | 86.7 | 94.3 | 83.6 | 93.6 | 89.7 | 91.9 |
| Qwen2.5-VL-72B^∗ [25] | 99.0 | 90.1 | 96.4 | 87.1 | 96.6 | 90.6 | 94.0 |
| GTA1-7B | 99.0 | 88.6 | 94.9 | 89.3 | 92.3 | 86.7 | 92.4 |
| GTA1-32B | 98.6 | 89.1 | 96.4 | 86.4 | 95.7 | 88.7 | 93.2 |
| GTA1-72B | 99.3 | 92.4 | 97.4 | 89.3 | 95.3 | 91.6 | 94.8 |

**Table 3: Performance comparison of state-of-the-art models on the OSWorld-G [6] dataset. We report grounding accuracy (%) categorized by different capabilities, along with the overall average (Avg). We use ‘^∗’ to denote the results evaluated by us (which will be updated if improved evaluation scripts become available). The final average scores on the benchmark are highlighted in dark blue.**
| Agent Model | Text<br>Matching | Element<br>Recognition | Layout<br>Understanding | Fine-grained<br>Manipulation | Refusal | Avg |
| --- | --- | --- | --- | --- | --- | --- |
| Proprietary Models |  |  |  |  |  |  |
| Operator [23] | 51.3 | 42.4 | 46.6 | 31.5 | 0.0 | 40.6 |
| Gemini-2.5-Pro [30] | 59.8 | 45.5 | 49.0 | 33.6 | 38.9 | 45.2 |
| Seed1.5-VL [24] | 73.9 | 66.7 | 69.6 | 47.0 | 18.5 | 62.9 |
| Open-Source Models |  |  |  |  |  |  |
| Qwen2.5-VL-3B [25] | 41.4 | 28.8 | 34.8 | 13.4 | 0.0 | 27.3 |
| OS-Atlas-7B [4] | 44.1 | 29.4 | 35.2 | 16.8 | 7.4 | 27.7 |
| Qwen2.5-VL-7B [25] | 45.6 | 32.7 | 41.9 | 18.1 | 0.0 | 31.4 |
| UGround-7B [2] | 51.3 | 40.3 | 43.5 | 24.8 | 0.0 | 36.4 |
| Aguvis-7B [9] | 55.9 | 41.2 | 43.9 | 28.2 | 0.0 | 38.7 |
| UI-TARS-7B [13] | 60.2 | 51.8 | 54.9 | 35.6 | 0.0 | 47.5 |
| Qwen2.5-VL-32B^∗ [25] | 57.9 | 70.2 | 73.8 | 49.2 | 0.0 | 59.6 |
| Jedi-3B [6] | 67.4 | 53.0 | 53.8 | 44.3 | 7.4 | 50.9 |
| Jedi-7B [6] | 65.9 | 55.5 | 57.7 | 46.9 | 7.4 | 54.1 |
| UI-TARS-72B [13] | 69.4 | 60.6 | 62.9 | 45.6 | 0.0 | 57.1 |
| Qwen2.5-VL-72B^∗ [25] | 52.6 | 74.6 | 74.7 | 55.3 | 0.0 | 62.2 |
| UI-TARS-1.5-7B^∗ [13] | 52.6 | 75.4 | 72.4 | 66.7 | 0.0 | 64.2 |
| GTA1-7B | 63.2 | 82.1 | 74.2 | 70.5 | 0.0 | 67.7 |
| GTA1-32B | 52.6 | 73.1 | 72.0 | 59.9 | 0.0 | 61.9 |
| GTA1-72B | 57.9 | 76.9 | 77.3 | 66.7 | 0.0 | 66.7 |

**Table 4: Comparison with state-of-the-art methods on the OSWorld [14] benchmark. We show the number of steps used by different agent in the second column. We report success rate (%) as the evaluation metric.**
| Agent Model | Step | Success Rate |
| --- | --- | --- |
| Proprietary Models |  |  |
| Claude 3.7 Sonnet [11] | 100 | 28.0 |
| OpenAI CUA 4o [12] | 200 | 38.1 |
| UI-TARS-1.5 [13] | 100 | 42.5 |
| OpenAI CUA o3 [12] | 200 | 42.9 |
| Open-Source Models |  |  |
| Aria-UI w/ GPT-4o [1] | 15 | 15.2 |
| Aguvis-72B w/ GPT-4o [9] | 15 | 17.0 |
| UI-TARS-72B-SFT [13] | 50 | 18.8 |
| Agent S w/ Claude-3.5-Sonnet  [7] | 15 | 20.5 |
| Agent S w/ GPT-4o [7] | 15 | 20.6 |
| UI-TARS-72B-DPO [13] | 15 | 22.7 |
| UI-TARS-72B-DPO [13] | 50 | 24.6 |
| UI-TARS-1.5-7B [13] | 100 | 26.9 |
| Jedi-7B w/ GPT-4o [6] | 100 | 27.0 |
| Agent S2 w/ Claude-3.7-Sonnet [8] | 50 | 34.5 |
| Agent S2 w/ Gemini-2.5-Pro [8] | 50 | 41.4 |
| GTA1-7B w/ o3 | 100 | 45.2 |
| GTA1-32B w/ o3 | 100 | 42.1 |
| GTA1-72B w/ o3 | 100 | 43.4 |

### 4.1 Grounding Performance

We compare our method with state-of-the-art approaches on the ScreenSpot-Pro [3], ScreenSpot-V2 [15, 4], and OSWorld-G [14] datasets, as shown in [Tab. 1](#S3.T1), [Tab. 2](#S4.T2), and [Tab. 3](#S4.T3), respectively. Among the three benchmarks, the ScreenSpot-Pro benchmark is the most challenging, designed for high-resolution, complex, and professional GUI grounding scenarios. It spans 23 applications across 5 industries and 3 operating systems. The ScreenSpot-V2 benchmark evaluates grounding capability across mobile, desktop, and web domains, while the OSWorld-G benchmark focuses on the Linux environment, providing a comprehensive benchmark for measuring diverse capabilities such as text matching, element recognition, layout understanding, and precise manipulation.

Our method consistently demonstrates the best performance among open-source models, while remaining competitive with the leading proprietary models. On the ScreenSpot-Pro [3] benchmark, our 7B model even outperforms significantly larger alternatives, for example, achieving 50.1% scores compared to 34.5% scores from UGround-72B. On the ScreenSpot-V2 benchmark, our best-performing model, GTA1-72B, achieves performance comparable to the proprietary Seed-1.5-VL [24], with only a 0.3% grounding accuracy difference. Similarly, on the OSWorld-G [6] benchmark, our method surpasses all previous state-of-the-art approaches, setting a new benchmark with a grounding accuracy of 66.7%.

Overall, our method is trained using a simple strategy that determines whether a click correctly lands on the target element. Although this is a straightforward approach, it achieves state-of-the-art performance among open-source models, highlighting its robustness and effectiveness. This demonstrates the potential of our method as a strong foundation for grounding models in agents that interact with complex GUI environments.

### 4.2 Agent Performance

We compare our method with state-of-the-art approaches on the OSWorld [14] benchmark in [Tab. 4](#S4.T4). It consists of 369 tasks distributed across real-world web and desktop applications, providing a diverse and challenging testbed for assessing agent capabilities to complete user tasks in Linux environments. We assess various scales of our grounding model using o3 as the planner and the judge for our test-time scaling strategy, forming the GTA1 agent series. Among the three model scales, GTA1-7B achieves the highest task success rate of 45.2% on the OSWorld benchmark, outperforming all state-of-the-art methods. It is worth highlighting that, even when using the o3 planner, GTA1-7B significantly outperforms its native agent variant, CUA o3, while operating with a shorter execution horizon (i. e., 45.2% from our method with a 100-step horizon vs. 42.9% from CUA o3 with a 200-step horizon [23]).

The strong performance of the GTA1 agent demonstrates its effectiveness in handling complex and real-world user tasks across diverse scenarios. This highlights the robustness and generalization capability of our approach. In the following subsections, we present a more detailed analysis of the GTA1 agent to examine it.

**Table 5: Ablation of optimization rewards. We consider three reward types: click reward (i. e., whether the prediction selects the correct target element), IoU reward (i. e., enforcing predictions of the bounding box of the target element), and format reward (i. e., enforcing “thinking”)**
| Click<br>Reward | IoU<br>Reward | Format<br>Reward | ScreenSpot-Pro | ScreenSpot-V2 | OSWorld-G |
| --- | --- | --- | --- | --- | --- |
| ✓ | ✓ | ✓ | 44.5 | 89.3 | 59.9 |
| ✓ | ✓ |  | 42.2 | 89.2 | 59.2 |
| ✓ |  | ✓ | 46.9 | 93.2 | 67.0 |
| ✓ |  |  | 50.1 | 92.4 | 67.7 |

Figure: Figure 4: Scalability of $K$. Subplots (a) shows the scalability of $K$ using 15 steps on the OSWorld benchmark [14], while (b) use 50 steps, and (c) use 100 steps. We vary $K$ over {1, 8, 16, 32} for test-time scaling, and measure performance using task success rate (%). When $K=1$, the performance degrades to the setting where the test-time scaling strategy is not used. We color the performance of $K=1$ over {15, 50, 100} steps on the OSWorld benchmark with red dotted lines.
Refer to caption: /html/2507.05791/assets/x4.png

### 4.3 Discussion and Ablation

#### Click Reward Works the Best.

We explore the impact of different optimization objectives for GTA1, focusing on widely studied rewards such as the format reward (i. e., enforcing “thinking”) and IoU rewards (i. e., enforcing prediction of bounding box of the target element) in [Tab. 5](#S4.T5) (a). Specifically, we evaluate three settings: i) optimizing the model using the format reward, IoU reward, and click reward ([Eq. 2](#S3.E2));
ii) using the IoU reward and click reward;
ii) applying the format reward and click reward.
The three settings achieve performance of 44.5%/42.2%/46.9%, 89.3%/89.2%/93.2%, and 59.9%/59.2%/67.0% on the ScreenSpot-Pro [3], ScreenSpot-V2 [5, 4], and OSWorld-G [14] benchmarks, respectively. However, all of these combinations generally underperform compared to using the click reward alone (with the exception of the ScreenSpot-V2 benchmark), which yields the accuracies of 50.1%, 92.4%, and 67.7% on the respective benchmarks.

#### “thinking” Benefits Grounding in Dynamic Environment Only.

Across various benchmarks, we observe minimal performance differences between grounding models trained with and without “thinking”. However, they often succeed on different samples, likely due to training instability rather than systematic reasoning gains.
We find that “thinking” can be effective in dynamic environments such as the AndroidWorld benchmark [31], where the model is provided with the task object, past trajectories, and the user instruction. For example, by training an in-domain 7B model based on [25] using the AndroidControl dataset [32], using or not using “thinking” have similar grounding performance on the AndroidControl test fold. However, the task success rate on the AndroidWorld benchmark increased from 39% to 44% when using “thinking”. This improvement is attributed to the increased complexity of the textual prompts (i. e., combination of task object, past trajectories, and the user instruction), which encourages the model to engage in “thinking” when operating under challenging and dynamic conditions.

#### Test-time Scaling Generalizes Well.

We demonstrate the generalization capability of our test-time scaling strategies through two sets of experiments. First, when using $K=8$, increasing the horizon from 50 to 100 steps on the OSWorld benchmark [14] improves the success rate from 43.4% to 45.2%, indicating robustness across varying execution lengths. In comparison, the baseline performance with $K=1$ is 41.3% and 43.4% for the 50-step and 100-step horizons, respectively. Second, we show that our test-time scaling strategies generalize to other agents, as evidenced by the scalability of $K$ on UI-TARS-1.5-7B [13], shown in [Fig. 4](#S4.F4). We assess performance with 15-, 50-, and 100-step horizons in [Fig. 4](#S4.F4) (a), [Fig. 4](#S4.F4) (b), and [Fig. 4](#S4.F4) (c) respectively, by varying $K$ over {1, 8, 16, 32}. The red dotted lines mark the baseline performance of UI-TARS-1.5-7B without any test-time scaling (i. e., $K=1$) at each horizon on the OSWorld benchmark. Our test-time scaling strategy consistently boosts performance, yielding two main insights:
i) with our test-time scaling, UI-TARS-1.5-7B executed for only 15 steps and $K=32$ already outperforms the baseline that executes for 100 steps without scaling. Since the $K$ candidate action proposals are sampled concurrently, this also cuts wall-clock time substantially;
ii) the greatest overall gain occurs with a 50-step horizon. Using 15-step horizon is occasionally insufficient to complete certain tasks, whereas 100-step horizon provide excessive slack, diluting the benefit of additional steps. We present qualitative comparisons of UI-TARS-1.5-7B with and without test-time scaling strategies in [Fig. 5](#S5.F5) and [Fig. 6](#S5.F6). As shown, without the test-time strategy, errors in early grounding or planning stages can propagate and derail the entire task execution, making the agent highly susceptible to cascading failures.

## 5 Conclusion

This paper investigates two key challenges toward building intelligent GUI agents: disambiguation in task planning and precise visual grounding in complex interfaces. We address these challenges with two strategies. First, to improve task planning, we introduce a scalable test-time strategy that concurrent samples multiple action proposals at each step and use a multimodal large language model judge to select the most suitable one. Second, we propose a grounding model, which employs a simple RL-based optimization approach that directly rewards successful clicks on target elements, bypassing the explicit “thinking” required by prior methods. Overall, GTA1 achieves state-of-the-art performance on standard visual grounding benchmarks and demonstrates robust behavior when integrated with a planner and our test-time scaling strategy as part of a complete agent. We hope this work inspires further research on GUI agents.

## References

- [1]
Y. Yang, Y. Wang, D. Li, Z. Luo, B. Chen, C. Huang, and J. Li, “Aria-ui: Visual grounding for gui instructions,” arXiv preprint arXiv:2412.16256, 2024.
- [2]
B. Gou, R. Wang, B. Zheng, Y. Xie, C. Chang, Y. Shu, H. Sun, and Y. Su, “Navigating the digital world as humans do: Universal visual grounding for gui agents,” arXiv preprint arXiv:2410.05243, 2024.
- [3]
K. Li, Z. Meng, H. Lin, Z. Luo, Y. Tian, J. Ma, Z. Huang, and T.-S. Chua, “Screenspot-pro: Gui grounding for professional high-resolution computer use,” arXiv preprint arXiv:2504.07981, 2025.
- [4]
Z. Wu, Z. Wu, F. Xu, Y. Wang, Q. Sun, C. Jia, K. Cheng, Z. Ding, L. Chen, P. P. Liang, et al., “Os-atlas: A foundation action model for generalist gui agents,” arXiv preprint arXiv:2410.23218, 2024.
- [5]
K. Cheng, Q. Sun, Y. Chu, F. Xu, Y. Li, J. Zhang, and Z. Wu, “Seeclick: Harnessing gui grounding for advanced visual gui agents,” arXiv preprint arXiv:2401.10935, 2024.
- [6]
T. Xie, J. Deng, X. Li, J. Yang, H. Wu, J. Chen, W. Hu, X. Wang, Y. Xu, Z. Wang, et al., “Scaling computer-use grounding via user interface decomposition and synthesis,” arXiv preprint arXiv:2505.13227, 2025.
- [7]
S. Agashe, J. Han, S. Gan, J. Yang, A. Li, and X. E. Wang, “Agent s: An open agentic framework that uses computers like a human,” arXiv preprint arXiv:2410.08164, 2024.
- [8]
S. Agashe, K. Wong, V. Tu, J. Yang, A. Li, and X. E. Wang, “Agent s2: A compositional generalist-specialist framework for computer use agents,” arXiv preprint arXiv:2504.00906, 2025.
- [9]
Y. Xu, Z. Wang, J. Wang, D. Lu, T. Xie, A. Saha, D. Sahoo, T. Yu, and C. Xiong, “Aguvis: Unified pure vision agents for autonomous gui interaction,” arXiv preprint arXiv:2412.04454, 2024.
- [10]
Deepmind, “Gemini 2.5: Our most intelligent ai model,” technical report, Deepmind, 2025.
- [11]
Anthropic, “Claude 3.7 sonnet and claude code,” technical report, Anthropic, 2025.
System Card.
- [12]
OpenAI, “Openai o3 and o4-mini system card,” technical report, OpenAI, 2025.
System Card.
- [13]
Y. Qin, Y. Ye, J. Fang, H. Wang, S. Liang, S. Tian, J. Zhang, J. Li, Y. Li, S. Huang, et al., “Ui-tars: Pioneering automated gui interaction with native agents,” arXiv preprint arXiv:2501.12326, 2025.
- [14]
T. Xie, D. Zhang, J. Chen, X. Li, S. Zhao, R. Cao, T. J. Hua, Z. Cheng, D. Shin, F. Lei, et al., “Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments,” Advances in Neural Information Processing Systems, vol. 37, pp. 52040–52094, 2024.
- [15]
D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al., “Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,” arXiv preprint arXiv:2501.12948, 2025.
- [16]
Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al., “Deepseekmath: Pushing the limits of mathematical reasoning in open language models,” arXiv preprint arXiv:2402.03300, 2024.
- [17]
R. Luo, L. Wang, W. He, and X. Xia, “Gui-r1: A generalist r1-style vision-language action model for gui agents,” arXiv preprint arXiv:2504.10458, 2025.
- [18]
Z. Lu, Y. Chai, Y. Guo, X. Yin, L. Liu, H. Wang, H. Xiao, S. Ren, G. Xiong, and H. Li, “Ui-r1: Enhancing action prediction of gui agents by reinforcement learning,” arXiv preprint arXiv:2503.21620, 2025.
- [19]
Y. Liu, P. Li, C. Xie, X. Hu, X. Han, S. Zhang, H. Yang, and F. Wu, “Infigui-r1: Advancing multimodal gui agents from reactive actors to deliberative reasoners,” arXiv preprint arXiv:2504.14239, 2025.
- [20]
Y. Zhou, S. Dai, S. Wang, K. Zhou, Q. Jia, et al., “Gui-g1: Understanding r1-zero-like training for visual grounding in gui agents,” arXiv preprint arXiv:2505.15810, 2025.
- [21]
A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford, et al., “Gpt-4o system card,” arXiv preprint arXiv:2410.21276, 2024.
- [22]
Y. Lu, J. Yang, Y. Shen, and A. Awadallah, “Omniparser for pure vision based gui agent,” arXiv preprint arXiv:2408.00203, 2024.
- [23]
OpenAI, “Computer-using agent: Introducing a universal interface for ai to interact with the digital world,” 2025.
- [24]
D. Guo, F. Wu, F. Zhu, F. Leng, G. Shi, H. Chen, H. Fan, J. Wang, J. Jiang, J. Wang, et al., “Seed1. 5-vl technical report,” arXiv preprint arXiv:2505.07062, 2025.
- [25]
S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, et al., “Qwen2. 5-vl technical report,” arXiv preprint arXiv:2502.13923, 2025.
- [26]
K. Q. Lin, L. Li, D. Gao, Z. Yang, S. Wu, Z. Bai, S. W. Lei, L. Wang, and M. Z. Shou, “Showui: One vision-language-action model for gui visual agent,” in Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 19498–19508, 2025.
- [27]
W. Hong, W. Wang, Q. Lv, J. Xu, W. Yu, J. Ji, Y. Wang, Z. Wang, Y. Dong, M. Ding, et al., “Cogagent: A visual language model for gui agents,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14281–14290, 2024.
- [28]
C. Qin, A. Zhang, Z. Zhang, J. Chen, M. Yasunaga, and D. Yang, “Is chatgpt a general-purpose natural language processing task solver?,” arXiv preprint arXiv:2302.06476, 2023.
- [29]
X. Yuan, J. Zhang, K. Li, Z. Cai, L. Yao, J. Chen, E. Wang, Q. Hou, J. Chen, P.-T. Jiang, et al., “Enhancing visual grounding for gui agents via self-evolutionary reinforcement learning,” arXiv preprint arXiv:2505.12370, 2025.
- [30]
Deepmind, “Introducing gemini 2.0: our new ai model for the agentic era,” technical report, Deepmind, 2025.
- [31]
C. Rawles, S. Clinckemaillie, Y. Chang, J. Waltz, G. Lau, M. Fair, A. Li, W. Bishop, W. Li, F. Campbell-Ajala, et al., “Androidworld: A dynamic benchmarking environment for autonomous agents,” arXiv preprint arXiv:2405.14573, 2024.
- [32]
W. Li, W. E. Bishop, A. Li, C. Rawles, F. Campbell-Ajala, D. Tyamagundlu, and O. Riva, “On the effects of data scale on ui control agents,” Advances in Neural Information Processing Systems, vol. 37, pp. 92130–92154, 2024.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 4
presentation: 2
questions: - Under matched compute, how much does your test-time scaling improve over other test-time strategies?
- Does the judge score candidates or generate a new action, and how sensitive are results to this choice?
- If you increase the number of action proposals beyond 32, do gains continue?
- What is the performance delta on OSWorld with and without the grounding model? Please provide ablations quantifying its impact.
- How do open-source models perform under your test-time scaling, with and without the GTA grounding model? It would help to report accuracy-cost curves and confidence intervals for these settings.
soundness: 3
strengths: - Strong empirical results across both grounding datasets and interactive environments, reaching SOTA on multiple benchmarks.
- The two-stage (planning+grounding) framework makes it straightforward to switch Planner/Judge modules or upgrade the grounding model.
- The light-weight data cleaning pipeline can be integrated with existing data pipelines.
summary: The paper’s main contributions are (i) a test-time scaling strategy that selects the next action and (ii) a GRPO-trained GTA model for coordinate grounding. Specifically, at each step, the planner samples multiple candidate actions, and a multimodal LLM judge selects the best one. For actions requiring coordinates, a GTA model trained with GRPO provides more precise grounding. The GTA grounding model achieves SOTA on grounding benchmarks (ScreenSpot-Pro, ScreenSpot-V2, OSWorld-G). On real OSWorld tasks, o3 with test-time scaling + GTA1-7B grounding attains a 45.2% success rate with a 100-step cap, surpassing CUA o3 at 42.9% with a 200-step cap.
weaknesses: - The description of test-time scaling is under-specified. For example, when the judge "picks the best candidate", is the output a score over candidates, or a rewritten action? Some results (e.g., Table 1) could be simplified to improve readability by moving some baselines to the appendix.
- Evaluations of test-time scaling (TTS) focus on ablations against the same agent without TTS. At least there should be equal-compute comparisons (e.g., self-consistency / majority voting / alternative sampling strategies).
- There are no confidence intervals. Reported gains over an OpenAI o3 baseline are ~2 points (Table 4). Without multiple runs and confidence intervals, statistical significance is unclear.
- The results of larger models are missing. GTA1-7B with GPT-5 and 32B configurations are "unavailable". If this is due to API cost, a proxy study (e.g., replaying 7B trajectories with a 32B grounding model) would strengthen the insights of the experiments. Likewise, results for open-source planners with test-time scaling + GTA grounding would be valuable.
- The paper appears to lack OSWorld ablations isolating the impact of the GTA grounding model.
- The proposed method largely adapts known ideas. Test-time scaling here is similar to best-of-N, and the GTA model is based on GRPO + data cleaning + reward shaping. It is reasonable that these techniques can boost performance, so more analysis and insights are required in the experiments (see suggestions above).

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: How is the training dataset built from the collection mentioned in the paper?
soundness: 2
strengths: - strong empirical results on relevant benchmarks
- simple method
- ablations and discussion on the efficacy of the different reward signals (for thinking, for location)
summary: The paper introduces a GUI agent that works in two stages: planning and grounding. The proposed solution improves each stage differently:
1. the planning stage through test-time scaling: sampling K times at each step and asking a judge to select the "best" one.
2. the grounding stage by fine-tuning a model using GRPO
weaknesses: - the dataset curation step should be part of the "algorithm", otherwise the comparison with other methods is not 
- reduced novelty
- no ablation to understand the improvement brought by each of the two components in the agent task execution scenarios

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: 1. Experiments use o3/GPT-5 as planners and a multimodal LLM judge. Can you test how performance transfers to an open-source stack?

2. Could you report the judge’s pairwise selection accuracy and show how judge errors propagate to end-task failure?

3. Can you share success–vs–cost curves (tokens, latency, $) across K and horizon, and results for adaptive policies (early exit, dynamic K per step)?
soundness: 3
strengths: 1. The paper tackles two concrete pain points—plan selection and precise grounding, and pairs them with two simple fixes: test-time scaling for planning and a minimal RL recipe for click grounding. The framing is clear and the approach maps tightly to the problems.

2. Test-time scaling actually pays off. Sampling K proposals per step and using a judge consistently boosts success while also cutting wall-clock via concurrent sampling.

3. Directly rewarding clicks inside the target element keeps the training signal simple and on-task, yet reaches SOTA on ScreenSpot-Pro, ScreenSpot-V2, and OSWorld-G. The simplicity is a strength.

4. The paper shows that explicit reasoning isn’t needed for static UIs and can even hurt, while helping in dynamic settings when trajectories/objectives are involved. This nuance helps practitioners know when to spend tokens on reasoning.

5. Results cover both grounding and full agent task execution (OSWorld and WindowsAgentArena), and scaling tests generalize across agents, horizons, and K settings, with qualitative analyses of cascading failures. The breadth makes the claims more convincing.
summary: GTA1 is a GUI agent that improves both plan selection and precise clicking on complex screens. At each step, it samples multiple action proposals and uses a judge model to pick the best one, trading extra compute for better decisions. For grounding, it trains a simple RL model that directly predicts click coordinates with a click reward, achieving SOTA results on ScreenSpot-Pro, ScreenSpot-V2, and OSWorld-G, and strong task success on OSWorld and WindowsAgentArena. The key contributions are test-time scaling for robust planning, a minimal RL recipe for grounding, and an analysis showing “thinking” isn’t needed in static UIs but can help in dynamic ones.
weaknesses: 1. Test-time scaling samples K proposals each step and uses a judge, which can raise token and inference costs even if concurrent sampling cuts wall-clock time. The paper shows speedups and success gains but does not report detailed cost curves (tokens, latency) across K and horizons.

2.  The paper observes little average gain from “thinking” and attributes sample-wise differences to training instability. “Thinking helps only in dynamic UIs” needs stronger controls. The authors can test different “thinking” budgets, report variance to separate instability from genuine reasoning effects, and clarify when trajectories/objectives trigger reliable wins.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: 1.I have observed that the training data originates from Aria-UI-Web, OmniACT, UI Vision, Widget Caption, and OSAltas-Desktop. Could you elaborate on the rationale for selecting these particular datasets? Furthermore, what considerations were made in terms of the volume and heterogeneity of the data?

2.it is stated that the training dataset underwent a filtering process based on the quality of bounding boxes. Could you provide the specific quantities of the dataset both prior to and subsequent to this filtering procedure?

3.This manuscript evaluates performance on benchmarks such as OS-World and WindowsAgentArena. What was the reasoning for not incorporating agent data into the training set? Additionally, could you compare the potential efficacy of a unified model that performs both planning and grounding, versus the current methodology which involves test-time scaling for planning followed by a separate grounding step?

4.It is mentioned that 70k data samples were utilized for training. Is it anticipated that performance would continue to enhance with a further increase in the scale of the data? I wonder if there is a scaling law here?
soundness: 3
strengths: 1.GTA1 effectively addresses the planning and localization challenges for GUI agents through a testing-time scaling strategy and a reinforcement learning-based localization model, achieving state-of-the-art performance across multiple benchmarks.

2.Extensive experiments have validated the effectiveness and generalizability of the method.
summary: This paper introduces the GUI Testing-time Scaling Agent (GTA1), which aims to solve challenges in planning and action localization for GUI agents. Through a testing-time scaling strategy, it samples multiple candidate action proposals at each step, from which a judgment model selects the optimal solution. This is combined with a reinforcement learning-based localization model that directly predicts interaction coordinates and is optimized via click rewards, leading to effective performance improvement.
Experimental results show that GTA1 achieves state-of-the-art performance on multiple benchmarks. For instance, on the ScreenSpot-Pro benchmark, the 7B model reached an accuracy of 50.1%, surpassing UGround-72B's 34.5%. On the OSWorld benchmark, GTA1-7B achieved a task success rate of 45.2%, outperforming all compared methods. These results demonstrate the effectiveness and robustness of the proposed method.
weaknesses: It is noted in the manuscript that the foundational model for GTA1-7B is UI-TARS-1.5-7B, while for GTA1-32B it is OpenCUA-32B. This raises the question of how the performance would be affected if qwen2.5-vl or qwen3-vl were used as the base models instead. Further experimentation is required to exhibit the superior generalization capabilities of the proposed methodology.


Decision: Accept (Poster)