GTA1: GUI Test-time Scaling Agent
Authors: Yan Yang
Dongxu Li
Yutong Dai
Yuhao Yang
Ziyang Luo
Zirui Zhao
Zhiyuan Hu
JUNZHE HUANG
Amrita Saha
Zeyuan Chen
Ran Xu
Liyuan Pan
Caiming Xiong
Junnan Li
Venue: ICLR 2026 Poster (2026)
Keywords: GUI Agent; Multimodal Large Language Model
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Graphical user interface (GUI) agents autonomously complete tasks across platforms (\eg, Linux) by sequentially decomposing user instructions into action proposals that iteratively interact with visual elements in the evolving environment. However, two main challenges arise: i) planning (\ie, the action proposal sequence) under expansive action space, where selecting an appropriate plan is non-trivial, as many valid ones may exist; ii) accurately grounding actions in complex and high-resolution interfaces, \ie, precisely interacting with visual targets. This paper investigates the aforementioned challenges with our \textbf{G}UI \textbf{T}est-time Scaling \textbf{A}gent, namely \name. First, we conduct test-time scaling to select the most appropriate action proposal: at each step, multiple candidate proposals are sampled and evaluated and selected by a judge model. It trades off computation for better decision quality by concurrent sampling. Second, we propose a model that improves grounding of the selected action proposals to its corresponding visual elements. Our key insight is that reinforcement learning (RL) facilitates grounding through inherent objective alignments, rewarding successful clicks on interface elements. Experimentally, \name achieves state-of-the-art performance on both grounding and agent task execution benchmarks.

Published as a conference paper at ICLR 2026
GTA1: GUI TEST-TIME SCALING AGENT
Yan Yang'
Dongxu Li Z
Yutong Dai
Yuhao Yang"
Ziyang Luo Zeyuan Chen'
Zirui Zhao!
Zhiyuan Hu'
Junzhe Huang?
Amrita Saha
Ran Xu- Liyuan Pan
Caiming Xiong _ Junnan Li P
Salesforce Al Research
"The Australian National University
9 University of Hong Kong dongxuli i 00 com
Yangtze Delta Region Academy
8 Corresponding Author
cxi cnggsa esforce. com
junnan. 11 @sal @ eeforre....c.. com
ABSTRACT
Graphical user interface (GUI) agents autonomously complete tasks across plat- forms (e.g.. Linux) by sequentially decomposing user instructions into action pro- posals that iteratively interact with visual elements in the evolving environment. However, two main challenges arise: i) planning ei.e., the action proposal se- quence) under expansive action space, where selecting appropriate plan is non- trivial, as many valid ones may exist; ii) accurately grounding actions complex and high-resolution interfaces, e.e precisely interacting with visual targets. This paper investigates the aforementioned challenges with our GUI Test-time Scaling Agent, namely GTA1. First, conduct test-time scaling to select the most appro- priate action proposal: at each step, multiple candidate proposals are sampled and evaluated and selected by judge model. trades off computation for better deci- sion quality by concurrent sampling. Second, we propose model that improves grounding of the selected action proposals to its corresponding visual elements. Our key insight is that reinforcement learning (RL) facilitates grounding through inherent objective alignments, rewarding successful clicks on interface elements. Experimentally, GTAI achieves state-of-the-art performance on both grounding and agent task execution benchmarks.
INTRODUCTION
Automating task completions across diverse platforms through GUI agents represents significant milestone toward general artificial intelligence, supporting activities from online orders to expert workflows (Yang al. 2024). To solve task, GUI agent translates user instructions into multi- step interactions such as action proposals consisting of clicks or keystrokes (Gou et al. (022).. This introduces planning challenge, as multiple valid action proposal sequences may exist for the same user task. The challenge is further amplified by the high-resolution (up to 4K), complex, and hierar- chical layouts of GUI (Li 2025 Wu al. 2024 Cheng et al. 2024 Xie et al. 2025). requiring accurate coordinate identification of the target interface elements. This work aims to address both challenges (i. e., planning and grounding) towards a performant GUI agent.
Formally. existing works (Yang et al., 2024, Agashe et al., 2024; 2025 Xie et al. 2025 Xu et al., 2024] often pair GUI grounding model with planner (e.g., o3 (OpenAl 2025b)). The planner determines an action proposal at each step, while the grounding model locates the target interface elements for interactions (e.g., click areas). However, due to the inherent ficxibility of user tasks, multiple feasible action proposal sequences may exist for completing the same task, some more direct and efficient than others. This makes the agent vulnerable to cascading failures, i. e., errors in early grounding or planning steps can derail the entire task. One way to avoid this is roll out full action sequences in advance, but unlike domains such as math problem-solving, GUI environments lack "lookahead" capability: actions often have irreversible state effects, limiting the practicality in real-world use. This raises a central question: how can GUI agents remain robust in planning despite the lack of "lookahead" and the presence of multiple plausible action proposal sequences?
Beyond planning, GUI grounding models predominantly rely on supervised tine-tuning (SFT) (Yang
al. 2024 Cheng et al. 2024 Gou et al. 2024 Wu et al. 2024 Xie al. 2025), which trains

---

Published as a conference paper at ICLR 2026
OS Verified Success Rate (%)
ScreenSpot-Pre Accuracy (%)
GTAI
60
GTA1-32B
60
Agent S2.5
GTA1-7B
Jedi-7B
50
Quen2.5-VLL-228
Agent S2
BE-GUI-70 7B
Qwen2.5-VL-322B S
Clande Sonnet
UL TARS 1,5-7B
40
Claude 3.7 Sonnet
Sood 1L
40
InñGUI-RI- R-33
Iedi-7B
UH-ARRS 728
X
CLA
CL TARS 1.5. XB
30
Sp UGround-7B
20
2025-02
2025-04
2025-06
2025-08
3B 4B
7B
18B
32B
72B
Parameters
Date
(a)
(b)
Figure 1: Comparison of GTA1 (red dot) with state-of-the methods. (a) Grounding accuracy (%) on the ScreenSpot-Pre benchmark (Li et. al. 2025) across model scales in billions (B) parameters. (b) Success rate (%) on the WSWodd benchmark (Xie et al. 2024 over time.
models to predict the exact center of target interface elements. While effective in simple settings, this approach struggles to generalize to complex and high-resolution scenarios, particularly in pro- fessional interfaces (Li et al.) 2025). Fundamentally, SFT introduces misalignment with the task itself: any coonlinate within the target element constitutes a valid interaction, yet SFT penalizes deviations from the center. It limits model flexibility, reduces robustness, and preventing the model from perceiving appropriate supervision signals.
Alternatively, inspired by DeepSeck-RI-Zen (Guo Policy Optimization (GRPO) (Shao et al., 2024). has been explored in GUI grounding. Following (Guo al. 2025a), prior works (Luo al.] 2025 Lu al. 2025: Liu et al., 2025) formulates by performing textual "thinking" (1. e., CoT reasoning). before predicting interaction coordinates, training with format reward enforce reasoning and a click reward ensure predictions fall within the target element. Some methods further extend this approach by predicting the bounding box of the target UT element (L.uu al. 2025; Zhou et al.) 2025). While improving over SFT, we ask: is explicit "thinking" or auxiliary bounding box reward necessary for effective GUI grounding?
2025a). RL, particularly Group Relative
We investigate the two questions by complementary strategies: i) a test-time scaling strategy for planning that works alongside grounding model for robustly executing user tasks; ii) RL-based grounding model that directly predicts interaction coordinates. Specifically, our test-time scaling method addresses the challenge of effective planning in expansive action space without requiring "lookahead".. Instead of committing to a single action proposal sequence, at each step, multiple candidate proposals are sampled from the planner. Then, multimodal large language model acts as a judge to select the most contextually appropriate option. If the action is moordinate-based the grounding model predicts the target location for accurate execution. This enables the agent to navi- gate complex user tasks by exploring short-term alternatives without rolling out the full sequence.
Then, introduce a simple and straightforward GUI grounding optimization approach: the model directly predicts coordinates and receives reward when the prediction falls within the target UI element. effectively aligns the training objective with the task, making the method highly ef- ficient. Despite its simplicity, this approach achieves state-of-the-art performance across diverse GUI grounding benchmarks. Interestingly, we also observe that "thinking" ei. task object, past trajectories, and user instructions help in dynamic environments where context evolves over time, but such strategies often fail to generalize broadly, limiting their practicality.
e., reasoning) over the
Overall, our contributions are summarized as follows: i) a comprehensive study of grounding and planning for GUI agents; ii) a test-time scaling strategy that improves planning robustness and re- duces execution uncertainty; iií) simple yet effeetive GUI grounding model that directly predicts interaction coordinates without requiring explicit reasoning. A performance overview shown in Fig. 1 Our method consistently outperforms prior approaches both GUI grounding and task completion benchmarks, providing an effective pathway toward agentic GUI behavior.
2

---

Published as a conference paper at ICLR 2026
2 BACKGROUND
We review recent GUI agent advances, focusing on multimodal large language model-based agents, excluding non-visual approaches (e.g., HTML A tree-based methods). We first review the GUI grounding, and then categorize the agents into native GUI or two-stage GUI agents.
GUI grounding refers to the task of mapping user instructions to specific coor- dinates corresponding to target UI elements. Early works (Yang al. 2024 Cheng al. [024 Xu et al. [0224] Gou et al. 2024) primarily focus on SFT, training models to predict the center point of the intended UI element. However, SFT does not fully align with the nature of the GUI grounding task, where any coordinate within the target element should be considered a success. As a result, SFT-based models often exhibit poor generalization (e.g., on high-resolution and visually complex user interfaces (Li et al.] ). With the success of DeepSeek-RI -anSSekk-R1-2 R1-Zero (Guo et al.] 2025a). RL (specifically GRPO (Shao et al. (024)) has drawn increased attention. Many recent eftorts naively replicate techniques from other domains (Luo et al. 2025 Lu et al. 2025 Liu et al.) 2025), such as e., COT reasoning) before producing an answer, and the answer is rewarded only if the predicted coordinates fall within the target element region. This strategy overlooks an important insight: the "thinking" degrades performance in GUI grounding. A concurvent study (Zhou et al. 2025) makes similar observations, noting that CoT reasoning (i. e.. "thinking") is not required for RL training in GUI grounding and may even hinder grounding accu- racy. Our work further distinguishes itself in the following ways: i) clarify that "thinking" is not necessary for GUI grounding in static environments; ii) we demonstrate that "thinking" improves grounding performance in dynamic, real-world environments when provided with past trajectories and task objectives; iii) we conduct a comprehensive study of RL-based GUI grounding across mod- els of various scales. Going beyond prior work, we further evaluate how the model, when paired with planner, performs in realistic and dynamic environments, critical aspect that existing studies largely overlook.
GUI Grounding.
prompting the model to generate "thinking" (i.e.,
Two-stage GUI Agent. One major challenge in GUI grounding is accurately locating the coor- dinates of UI elements intended for interaction. To address this, two- stage GUI agents modularize into planning and action, each handled by separate models (Gou et al. 2024). They usually leverage advanced reasoning models, such as GPT-4o (Hurst et al., 2024) and Claude 3.7 (Anthropic) 2025) as planners to generate an action proposal for each step from the user task instruction, using real- time UI screenshot and past trajectories as context. A separate grounding module then maps these instructions to specific UI elements, enabling the development of vision-only GUI agents (Cheng et al. 2024 Yang et al. 2024 Gou al. (024)). While most existing work primarily focuses GUT grounding, more complex components, such as memory management and external knowledge bases, are also being explored to enhance agent performance (Agashe et al. 2025). This paper follows the two-stage method on establishing a GUI agent.
Native GUT Agent. A native GUT agent completes user tasks in an end-to-end manner. Four main aspects studied (Qin et al., 2025): perception, interpreting the user interface understand the current state; ii) memory, storing knowledge and historical experiences to support making decisions; ii) planning, analyzing the task and reflecting on progress to generate action proposals; iv) action, performing atomic operations based on the action proposal to effectively progress toward the task goal. Examples of native GUI agents can be specified to CUA (Hurst et al., [024] OpenAl 20256) and Claude Computer Use (Anthropic 2025). One of the main challenges for native GUI agents lies in long-context learning. To address this, some approaches employ a sliding window mechanism (Qin et al. 2025), while others maintain textual description of past trajectories to manage con- textual information (Xu et al. 2024). In practice, end-to-end native GUI agents have demonstrated strong performance in completing user tasks, as shown by benchmarks that reflect dynamic and re- alistic UI environments, such as World (Xie et al. 2024). However, this paper is the first to show that two-stage GUI agent can achieve competitive performance in such environments, challenging the assumption that end-to-end approaches are inherently superior.
3 METHOD
Overview. Our method adopts a two-stage GUI agent framework composed of a planner and a grounding model, and focuses improving planning robustness and grounding accuracy through the following key components: i) test-time scaling for planning, which scales inference computation to effectively handle planning selection challenges in complex GUI environments; ii) grounding
3

---

Published as a conference paper at ICLR 2026
Thwugrt Werees
dhoun frue some bicc
Acdiet: becton Reyders
User
Carwe UI
lve sine 12afl evente
Though Ind Eima andry: beat"
Traeetory
Divee Donrel
Coondisata
Prodkct
Action ageis soseector coot. Oparthe saaig
Control
Figure 2: Overview our GUI agent architecture. At each step, the trajectory, current UI screenshot, and user instruction are sent to planner, which samples multiple action proposals. A multimodal large language model judge is then used to select the best candidate action proposal. When the candidate action proposal is coordinate-base based action (e.g., click), the grounding model predicts a precise interaction point on the GUT for executing the action. For non-coordinate- based actions (e.g., key presses), the action can be executed directly without grounding.
model training, filtering out training samples with annotation errors to improve supervision quality, and optimizing a grounding model using RL (e.g., GRPO) to directly predict coordinates without relying on any intermediate "thinking" (i. e., CoT reasoning) on the derived data.
3.1 TEST-TIME SCALING FOR PLANNING
At each step of executing user instructions in real-world environments, a planner is provided with the user instruction (i. c., task objective), the trajectory so far, and the current UI screenshot. Based on this context, we sample K candidate action proposals, denoted as {P& 1.. where pa: represents a corresponding action (e.g., clicking 'the blue button' or performing a keystroke).
Then, a multimodal large language model judge is used to evaluate the K candidates {px 1 based on their alignment with the user intent and the current GUI state. The judge, which can be the planner model itself, picks the best candidate action proposal pk- from (px ) allowing the agent to select the most contextually appropriate option. Once the best candidate action proposal Pk- is selected, the grounding model (T:,)) takes Pi. and the current screenshot action (e.g., a click), the grounding model predicts a precise interaction point the GUI, which is then used execute the action. For non-coordinate-biee based actions (e.g., key presses or text input), the action can be executed directly without grounding. This process is repeated step by step until the task is completed or the agent reaches termination condition.
the input. If
Pk- is a coordinate-bass
By incorporating sampling, judging, and grounding into each step, our agent avoids overcommitting to suboptimal action proposals and demonstrates improved robustness in complex and dynamic GUI environments. We show an overview in Fig. 2
3.2 GROUNDING
Data Cleaning. We leverage curated open- source datasets to train our model, e.g., the Aria-UI collection (Yang et al. (0024). To train with a reward signal that verifies whether the predicted coordinates fall within the target UI element, require a dataset that provides ac- curate bounding boxes for annotated interactive elements. For data points from desktop and web domains, the collection usually involves screenshots paired with accessibility tools such as Ally or HTML parsers to extract element structure and bounding box annotations. How- ever, these bounding boxes can sometimes be misaligned with the actual visual rendering due to UI animations, timing inconsistencies, or render- ing delays, introducing noise into the training signal.
Figure 3: Examples from the Aria-UI dataset Yang et (024). The blue bounding box shows the annotation bann while red bounding boxes are detected by OmniParser (Lu et al., 2024]. The green arrow highlights misaligned annotations, which our cleaning strategy filters out.
4

---

Published as a conference paper at ICLR 2026
Therefore, to improve data quality, we apply lightweight cleaning strategy. Given a UI screenshot 8, we use OmniParser (Lu et al. 2024), denoted as MM-), to detect all UI elements in s, resulting in set of bounding boxes } (s). Each represents the bounding box of detected UI element.
For data point with annotated bounding box Bamn, discard the sample if the maximum Intersection over Union between Jann and any b; E {b; is smaller than a predefined threshold T,
max IoU .b)) <T.
b.EM(()
where Io)(::) computes the overlap between two bounding boxes, defined as the area of their intersection divided by the area of their union. This helps ensure that the annotation bann in the training data remains consistent with the actual visual target, thereby reducing noise caused by misaligned annotations. We show some samples in Fig. 3
Training. In our RL training, we follow the GRPO framework (Guo et al. 2025a Shao et al. 2024) sample N responses {o, from the policy multimodal large language model T(;))) given a screenshot and an action proposal p input. Here, each response on represents pair of pixel coordinates on the screen, e., O, = (I', Yn). where Zn and yn denote the horizontal and vertical positions, respectively. Unlike prior approaches, we do not prompt the model to generate "thinking" (i. e., CoT reasoning) before producing response. Instead, the model directly outputs the predicted coordinates, aligning more closely with the nature of the GUI grounding task.
Then, each response is evaluated by checking whether the predicted coordinate (In; Mn) falls within the annotated bounding box bann (Imin: Vmin: Tmax, Imax)) ). This yields set of N binary rewards {ra)a=1 where each reward
{1,
Tmin I on I Fmax and Ymin S In S Umax
(2)
0, otherwise
We then normalize the rewards
(rn
IN into advantages (An) )N using Z-oc normalization,
Fn N
1 EN
En Fn
A₂
(3)
Dm (rn E r2)
Finally, the model is optimized by
T(On 8, Told n s.
T(on 8, gold (on s.p)"
L N min
An, clip(
1- 1- (c,,1+)).A))
(4)
where ol ..-) denotes the old policy, v, the advantage associated with the prediction On. clip(:, :)) is clip function, and e is a hyperparameter. The advantage serves as weight, encourag- ing high reward predictions while suppressing low reward ones.
EXPERIMENT
Implementation Detail. We train our model using mixture of dataset (Yang et al. 2024 Wu
et al. 2024 Nayak et al. 2025 Kapoor et al.) 2024 Li et al.
2020).
applying a data cleaning
threshold 7 0.3. Our model is initialized from Qin et al.
2025 Wang al. 2025). When
testing on the real-world dynamic environment, we use the action space from (Agashe et al. 2025) and 03 as the default planner (OpenAi 2025b). Refer to App.
B for more details.
Dataset. We evaluate our method on two sets of benchmarks: i) GUT Grounding, where we use ScreenSpot-Pro (I.i et al. 2025). ScreenSpot-V2 (Cheng et 2024 Wu al,
and OSWorld- (Xie et al., 2024) datasets, evaluating by the metric accuracy; ii) Agent Task Exccution, where use OSWorld (Xie et al.) 2024) and WindowsAgentArren (Bonatti et al. 2024) benchmarks, measuring performance by task success rate. For the OSWlrd benchmark, we explore both its original release and OSWorld-Verinee
(i.e., updated variant).
Baseline. We compare with various state-of-the-an methods: CogAgent (Hong et al. 2024)
OminiParser (Lu et al. 2024), Qwen2.5-VL (Bai et al.) 2025), Aria-Ul (Yang et al., 2024). OS- Atlas (Wu et al. 2024) UGround (Gou et al. 2024). ShowUl (L.in et al. 2025), Aguvis (Xu al.
5

---

Published as a conference paper at ICLR 2026
Table I: Comparison with state-of-the-art methods on the ScreenSpot-PPy dataset (Li et al. 2025), We report the grounding accuracy (%) across various task domains, categorizing results by ground- ing target type: Text, Icon, and the overall average (Avg). We use '.' to denote unavailability, and to mark the results evaluated by us (subject to update with improved evaluation scripts), The final average scores are highlighted in dark blue, and the best scores are in bold.
Agent Model
Development Creative
CAD
Scientific
Office
OS
Avg
Icon Avg Text Avg Icon Avg Text Avg Icon Avg Text Avg Icon Avg.
Proprietory Models
Claude 3.7 Sonnet Aniiropic 2025
27.7
Operator OOpenA)
50.0
19.3
35.1
51.5
23,1
39.6
16.8
14.1
16.1
58.3
24.5
43.7
60.5
28.3
53.0
34.6
30.3
32.7
45.0
23.0
36.6
Seed-1.5.VI. G1b
59.2
59.0
61.4
60.2
UI-TARS-13
Qin Ct 2025
63.9
50.4
58.2
69.3
79.6
51.0
61.6
Open-Source
Models
OS-Atlas-7B Wi al. 2024
33.1
1.4
17.7
28.8
17.9
12.2
4.7
10.3
37.5
7.3
24.
5.7
27.4
27.1
4.5
16.8
28.1
4.0
18.9
UI-TARS-2H 2B
el al 20226
47.4
4.1
26.4
42.9
6.3
27.6
17.8
4.7
14.6
56.9
17.3
39.8
50.3
17.0
42.6
21.5
5.6
14.3
39.6
8.4
27.7
Qwea2,5-VLL335
2025
38.3
3.4
21.4
40,9
4,9
25.8
22.3
6.3
18,4
44.4
10.0
29.5
48.0
17.0
40.9
33.6
4,5
20.4
37.8
6.6
25.9
Qwea2.5-VL-77
ClEa 2025
51.9 4.8
29.1
36.9
24.9
17.8
1.6
13.8
48.6
31.1
53.7
18,9
45.7
34.6
22.4
39.9
7.6
27.6
DGmand:7R (Gob 20144
35.5
27.8
13.5
38.8
48.8
26.1
31.1
UGrownd-728 (Gou ot Aiz
31.1
35
138
50.0
513
25.5
34.5
UE-TARS-7B (Qihl
20243
58.4
12.4
36.1
50.0
9.1
328
20.8
9.4
18.0
619
31.8
50.0
63.3
20.8
$3.5
30.8
16.9
24.5
47.8
16.2
35.7
InGGLUI -1:
32.44444444
29.0
14.1
28.4
41.7
65.5
57.0
43.9
29.6
49.1
14.1
SE-GUI-38 Yuui el 21755
55.8 7.6
35.1
47.0
4,9
29.0
38
12.5
31
61.
16.4
43.
59.9
24 50.9
40.
12.4
25
50.4
11.8
35.9
Jodi-BB (Xic
13.8
38.1
53.5
34.6
27.4
9.4
23.0
54.2
38.6
64.4
57.0
25.0
13.7
36.1
GUI-G1-3B Ziou et a. 2025
50.7
10.3
31.1
36.6
11.9
26.6
39.6
9.4
32
61.8
30.0
48.0
67.2 32
59.1
23.5
10.6
16.1
49.5
16.8
$7.1
UI-TARS-72B (Qth
63.0
17.3
57
39.6
12.5
64.6
63.3
26.4
4
30.1
17.5
Jodi-7B (Xie et mi 2025
42.9
11.0
50.0
919
34.0
38.0
14.1
32.2
72.9
25.5
42 75 47.
68
33.6
16.9
26.0
52.6
18.2
39.5
U1-TARS-.:--15
2023
58.4 12.42433
31
1
15.4
40.2
38.6
11.0
31
66
21
47 7
35.9
65
49
13.5
33.2
57.5
16.9
42.0
Qwen2.5.VI_322R (Ra) al 2015
74.0
21
48.5
1
13.3
41.
38
15.6
32.6
78.5
29
ST. 76.3
37.7
67
55
27.0
42.
63.2
22.5
47.6
SE-GUI-7B YYunn
68.2 19.3
44.5
57.6
9.1
37.2
51.3
42.2
42.1
75.0
28.2
54.7
78.5
43.4
70.4
49.5
25.8
38.8
63.5
21.0
47.3
Owen2.5-VL.7222
[Bar al
2025
53.5
44.9
44.4
59.1
72.6
49.5
53.3
OpeaCUA-32B
(Wang 21
55.3
GTA1-78
66.9 20.7
44.5
62.6
18.2
44.0
53.3
17.2
44.4
76.4
31.8
57.1
82.5
50.9
75.2
48.6
25.9
38
3
65.5
25.2
50.1
GTAI-32B
83.1 37.9
61.2
72.2
25.9
52.8
70.1
31.3
60.5
84.7
39.1
65.0 89.3
64.2
83.5
76.6
51.7
65.3
78.9
38.9
63.6
2024
Jedi (Xie et al. 2025). GUI-GI (Zhou et al.
2025 SE- SE-GUI Yuan al. 2025 ). GUI-RI (Luo
et al.
2025). UI-RI Lu et al. 2025), InfiGUI-RI (Liu et al. (025). UI-TARS (Qin et al. 2025)
Seed-1 VL. (Guo et al. 2025b), b UI-TARS-1.5 1.5 (Qin al. 2025). o3 OpenAl 2025b), Claude 3.7
Sonnet (Anthropic 2025). and 2emini-2.5 (Deepmind) 2025b).
4.1
GROUNDING PERFORMANCE
We compare our method with state-of-the-art approaches on the ScreenSpot-Pro LLi et al. 2025
ScreenSpot-V V2 (Guo al. 2025a Wu et al., (2024), and OSWorld-G (Xie et al. 2024) datasets, as shown in Tab. T Tab. and Tab. 3 respectively. Among the three benchmarks, the ScreenSpot-Pie benchmark is the most challenging, designed for high-resolution, complex, and professional GUI grounding scenarios. The ScreenSpot-V2 benchmark evaluates grounding capability across mobile, desktop, and web domains, while the OSWorld-G benchmark focuses on the Linux environment, providing a comprehensive benchmark for measuring diverse capabilities such as text matching, element recognition, layout understanding, and precise manipulation.
Our method consistently demonstrates the best performance. On the ScreenSpot-Pre (Li et al. 2025) benchmark, our 7B model outperforms much larger alternatives, for example, achieving 50.1 % scores compared to 34.5% scores from UGround-72B. On the ScreenSpot-Ve V2 benchmark. our best- performing model, GTA1-32B. achieves the same performance as the proprietary 1.55-L.5-VL (Guo) et al. 2025b). Similarly, on the OSWorld-G (Xie et al. 2025) benchmark, our method surpasses all state-of the-th--rtttttt approaches, setting a new benchmark with grounding accuracy of 72.2%.
Overall, our method achieves of-te-----heaart performance using simple click-based training strat- egy, demonstrating both robustness and effectiveness.. This highlights its potential as a strong foun- dation for grounding models in complex GUI environments.
4.2
AGENT PERFORMANCE
Tab. and Tab. respectively compares our method with the state-of-the approaches on the OSWorld (Xie et al.) 2024) and WindowsAgemtArena (Bonatti et al.] 2024) benchmarks. These

---

Published as a conference paper at ICLR 2026
Table 2: Comparison with state-of-the-ar methods on the ScreenSpot-V2 dataset (Cheng et al, 2024 Wu et al. 2024) across mobile, desktop, and web domains. We report grounding accuracy (6) categorized by target types: Text, Icon/Widget, and the overall average (Avg) highlighted in dark blue. We use '-" to denote unavailability, and to mark the results evaluated by us (subject to update with improved evaluation scripts), The best scores are bold.
Desktop
Agent Model
Avg
Text
Icon/Widget
Text
leoneWidget
Text
lcon/Widget
Proprietary Medels
Operator (OpenAl 2025a
47.3
41.5
90.2
80.3
92.8
84.3
70.5
Clande 3.7 Nonnet Antiropic 2025
87.6
UI-TARS-1, (Qin
94.22
Soed-1.5-VL
G8O 2E31. 20236
95.2
Open-Sowrye Models
SeeClick Guo at 2025h
78.4
50.7
70.1
29.3
55.2
32.5
55.1
OmniParss-v22
2024
95.5
60.9
88.0
Qwen2.5-VI-316 Iiha 20125
93.4
73.5
88
58.6
88.0
71.4
80.9
UI-TARS-22 QQi 2012
95.2
79.1
90.7
68.6
87.2
78.3
84.7
OS-Allas-usse Bases-B [Wu al 2024
95.2
75,8
90.7
63.6
90.6
77.3
85.1
OS-AtlasBass--78 (Wu
96.2
83.4
89.7
69.3
94.0
79.8
87.1
Jodi-SB
2025
96.6
78.6
88.5
88
Qwen2.55VL-7B [Bsa al 2025
97.6
87.2
90.2
74.2
93.2
81.3
88.8
UI-TARS-1.5-78 (omre
95.9
80.7
90.6
UI-TARS-72B QQin ai 2024
94.8
86.3
91.2
87.9
91.5
87.7
90.3
UI-TARS-7B (Qin
96.9
Jedi-7B [Xi otai 20233
96.9
87.2
95.9
87.9
94.4
84.2
91.7
Qwen2.5-VL332B Qwea2.5-VL-725"
96.3
86.7
83.6
Bar al 2025
99.0
90.1
96.4
87.1
96.6
90.6
94.0
OpenCUA-328 Wang
93.4
GTAI-7B
99.0
88.6
94.9
89.3
92.3
86.7
92.4
Table 3: Performance comparison of state-of-the-ar models on the OSWorld-G (Xie 2025) dataset. We report grounding accuracy (%) categorized by different capabilities, along with the overall average (Avg) highlighted in dark blue, We use *_' to denote unavailability, and mark the results evaluated by us (subject to update with improved evaluation scripts). For methods not tagged with 1, the results are based on refined grounding instructions. The best scores are bold.
Layout Understanding
Fine-grained Manipulation
Agent Model
Matching
Recognition
Refusal
Avg
Proprietory Models
Operator OOenAA 2025a
51.3
42.4
46.6
31,5
0.0
40.6
Operator OpenA1 ATASH
57.8
Gemini-2 2 PTo
20258
59.8
45.5
49.0
33.6
38,9
45,2
Gomini-2.5-Prr
DDseptiin
20255
57.5
Sood1.5-VL Guo 20256
73.9
47.0
62.9
Open Source Models
Qwen2.5.V1.33
Bai 2025
41.4
28.8
34.8
13.4
0.0
27.3
OS-Allas7B
Wu
16.8
Qwen2.5-VL-7B
Bai n 2025
45.6
32.7
41.9
18.1
0.0
31.4
UGround-7R/
(Gou ct A002
51.3
40.3
43.5
24.8
0.0
36.4
Aguvis
Xu
38.7
UL-TARS-7B"
(Qin
21255
60.2
51.8
5499
35.6
0.0
47.5
Qwen2.5.VL..221
(15a sl 2025
57.9
70.2
73.8
49.2
0.0
59.6
Jedi-3B" [icc
2025
44.3
Jedi-3B Xie m
61.0
Jedi.7B
Jedi-BB Xie
11.A5
63.8
UI-TARS-72B
QQin 2025
69.4
45.6
0.0
57.1
Qwen2.5-VL-711B
[Bu 2015
52.6
74.6
74.7
55.3
0.0
62.2
UI-TARS-L.5-7B*
etinet 2075
36.8
62.7
62.2
50.8
0.0
52.8
UI-TARS-1.5.77
(Qin
52.6
75.4
72.4
66.7
0.0
64.2
OpenCUAA-32B
Wang et al 2025
59.6
OpenCUA-322B
Wang ala..
62.1
GTA1-7B'
42.1
56.1
0.0
55.1
1-7B
63.2
82.1
74.2
70.5
0.0
67.7
GTA1-32B
65.2
GTA1-32B
63.2
83.6
84.4
70.5
0.0
72.2
benchmarks evaluate agents completing user tasks in real-world desktop applications, such as web browsers and office softwares. We assess various scales of our grounding model using and GPT-5 both the planner and the test-time scaling judge. forming the GTAI agent series. Our method consistently finds the best performance across all benchmarks. For example, GTAI-

---

Published as a conference paper at ICLR 2026
Table 4: Comparison with state-of-the-art methods on the OSWorld and OSWorld- Veriited Xie et al. 2024) benchmarks. A dash ('-) indicates unavailable results, the second column shows the number steps, and success rate success rate (%) is reported. The best scores are in bold.
Agent Model
Step
OSWorld
OSWorld- Veriffed
Proprietary Models
Sonnet (Amhropic
2025
28.0
OpenAI CUA 40 Open
20
200
38.1
UI-TARS-1.5 ef 2025
100
42.5
41.8
OpenAI CUA OpenA 2025h
200
42.9
Open-Source Models
Aria: UI w/ GPT-4u Yang et al
2024
15
15.2
Aguvis-72B w/ GPT-50 Xu
2024
17.0
UI-TARS-72BB SFT
QQin
18.8
Agent S Claude T.5. Agashe et
2024
20.5
Agent S w/ GPT-4o Agashe
20.6
03 (OpenAl 20256
100
20.1
23.0
UI-IAAK-7218-22
Oin
UL.TARS 72B DPO Omn et
2025
50
24.6
UI-IARS--15-775 QQia 20
100
26.9
27.4
Jedi 7B w/ [Xie al 2025
100
51.0
Jedi-7B w/ GPT:4o [Xie ef al. 2025
100
27.0
Agent Claude-3.7 3.-Soonee (Agoshe 2025
34.5
Agent S2 w/ Gemini-2.5-PP
(Acasie al 2023
50
41.4
45.8
Agent
a3 Agashe et
56.0
Agent S2.5 w/ GPPSspgiis pygishi at 2025
100
58.4
CoAct-1 w/o3 odmam & Opennti COA (Song et al. 2025
150
60.8
GTA1-7B w/ o3
100
45.2
53.1
GTA1-7B
GTA1-32DD w/al GTA1-32B w/
100
55.4
Table 5: Comparisons on Win- dowsAgent/rena (Bonatti et al.] 2024) benchmarks. We report suc- cess rate (%) for evaluations. The best scores in bold.
Table 6: Ablation of optimization rewards. We study three types of rewards used to guide model learning: click reward ei. whether the prediction falls within the target element bounding box), IoU reward (i. e., enforcing predictions of the bounding box of the target element), and format reward ei. enforcing "thinking" before predictions).
Agent Model
Step Success Rate
Kimi-VL (Team al
10.4
Click
ToUl
WAA
Bohalli of 2024
19.5
Reward
Reward
Reward
ScreeaSpot-Pro
ScreenSpot-V2 OSWordd-G
Jedi- Xie al 2025
100
33.7
GTAI-7B W7
89.3
$9.9
GTA1-7B w/ GPT:5
100
49.2
42.2
89.2
59.2
67.0
GTAI-328 w/ GPT-5
100
50.6
50.1
92.4
67.7
7B achieves the highest task success rate of 45.2% on the OSWorld benchmark, outperforming all state-of-the-an methods. It is worth highlighting that, even when using the planner, GTA1-7B significantly outperforms its native agent variant, CUA o3, while operating with shorter execution horizon ei. 45.2% from our method with a 100-step horizon vs. 42.9% from CUA o3 with a 200-step horizon (OpenAl 2025a)). The strong performance of the GTAI agent demonstrates its effectiveness in handling complex and real-world user tasks across diverse scenarios.
4.3 DISCUSSION AND ABLATION
Click Reward Works the Best. We explore different optimization objectives for GTAI, focusing on widely studied rewards such as the format reward ei.e., enforcing "thinking") and IoU rewards (i.e., encouraging accurate bounding box predictions for the target element) in Tab. 6 We evaluate three settings: optimizing the model using the format reward, IoU reward, and click reward (Eq, 2
using the IoU reward and click reward; applying the format reward and click reward. The three set- tings achieve performance of 44.5 44.5% 21.4422.2/// /464 9%, 89.3%/89 ,,2 2%/93.29 29.2%. and 59.9%/59.22 /1t/992.%/661 2%%67.00 .0% respectively on the ScreenSpot-PP Pro Li et al. 2025). ScreenSpot V2 (Cheng et al. 2024; Wu et al.] 2024], and OS (Xie et al., 2024) benchmarks. However, all of the settings generally under- perform compared to using the click reward alone (except on the ScreenSpot- V2 benchmark), which yields the accuracies of 50.1
11.1% %, 92.4%, and 67.7% the three benchmarks.
8

---

Published as a conference paper at ICLR 2026
Success Rate (%)
40
Success Rate (%)
Success Rate (%)
40
30
30
30
20
20
20
16 K
32
1
8
16
32
8
16
32
K
K
(a)
(b)
(c)
Figure 4: Scalability of the number K of action proposals in test-time scaling. The task execution steps in (a), (b), and (c) are 15, 50, and 100, respectively. We vary K over {1, 8, 16, 32), and measure performance using task success rate (%). When K = 1, it degrades to the setting where the test-time scaling strategy is not used. We color the performance of K over (15, 50, 100) steps on the OSWorld benchmark with red dotted lines.
"thinking" Benefits Grounding in Dynamic Environment Only. Across various benchmarks, we observe minimal performance differences between grounding models trained with and without "thinking". However, they often succeed on different samples, likely due to training instability rather than systematic reasoning gains. We find that "thinking" can be effective in dynamic environments such as the AndroidWorld benchmark (Rawles et al. 2024), where the model is provided with the task object, past trajectories, and the user instruction. For example, by training model based on (Bai et al., (025)) using the AndroidControl dataset (Li et al. 2024). using or not using "thinking" have similar grounding performance the AndroidControl test fold. However, the task success rate on the AndroidWorld benchmark increased from 39% to 44% when using "thinking". This improvement attributed to the increased complexity of the textual prompts e., combination of task object, past trajectories, and the user instruction), which encourages the model to engage in "thinking" when operating under challenging and dynamic conditions.
in-domain 7B
Test-time Scaling Generalizes Well. We demonstrate the generalization capability of our test- time scaling strategies through two sets of experiments. First, with K = 8 action proposals, in- creasing the horizon from 50 100 steps on the OSWorld benchmark (Xie et al., 2024) improves the success rate from 43.4% to 45.2%, indicating robustness across varying execution lengths. In comparison, the baseline performance with = 1 is 41.3% and 43.4% for the 50-step and 100- step horizons, respectively. Second, we show that our test-time scaling strategies generalize to other agents, as evidenced by the scalability
K on UI-TARS- 1.5-7B (Qin et al., 2025). shown in Fig, 4 while we show its token usage in Fig. 5 We assess performance with 15-, 50-, and 100-step horizons in Fig. 44(a) (a), Fig. (b), and Fig. 4 (c) respectively, by varying K over {1, 8, 16, 32). The red dotted lines mark the baseline performance of UI-TARS- 1.5-7B without any test-time scaling (i. K 1) at each horizon on the OSWorld benchmark. Our test-time scaling strategy consistently boosts per- formance, yielding two main insights: i) with our test-time scaling, UI-TARS-L.5-7B executed for only 15 steps and K = 32 already outperforms the baseline that executes for 100 steps without scaling. Since the K candidate action proposals are sampled concurrently, this also euts wall-clock time substantially; ii) the greatest overall gain occurs with 50-step horizon. Using 15-step horizon occasionally insufficient slack, diluting the benefit of additional steps. Moreover, we present the qualitative comparisons of UI-TARS- 1.5-7B with and without test-time scaling strategies in App. A highlighting the effective- ness of the test-time strategy in making the agent highly susceptible to cascading failures.
complete certain tasks, whereas 100-step horizon provide excessive
5 CONCLUSION AND LIMITATION
This paper investigates two key challenges toward building intelligent GUI agents: selecting ef- fective plans and precise grounding in complex interfaces. We address these challenges with two strategies. First, to improve task planning, we introduce scalable test-time strategy that concurrent samples multiple action proposals at each step and use a multimodal large language model judge to select the most suitable one. Second, we propose grounding model, which employs simple RL- based optimization approach that directly rewards successful clicks on target elements, bypassing the explicit "thinking" required by prior methods. Overall, GTAI achieves state-of-the-art perfor- mance on standard grounding benchmarks and demonstrates robust behavior when integrated with

---

Published as a conference paper at ICLR 2026
planner and our test-time scaling strategy for user task execution in GUI environment. However, approach has limitations. Although achieves the highest accuracy on the challenging ScreenSpot- Pro benchmark, it still struggles in certain scenarios. For example, similar to prior work, ground- ing model has difficulty in selecting custom foregrounds and backgrounds when applied to image editing with GMIP. We hope this work inspires further research GUI agents.
REFERENCES
Saaket Agashe, Jiuzhou Han, Shuyu Gan, Jiachen Yang, Ang Li, and Xin Eric Wang. Agent s: An open agentic framework that uses computers like human. arXiv preprint arXiv:2410.08164 2024.
Saaket Agashe, Kyle Wong, Vincent Tu, Jiachen Yang, Ang Li, and Xin Eric Wang. Agent s2: A compositional generalist-specialis framework for computer use agents. arXiv preprint arXiv:2504.00906 2025.
Anthropic. Claude 3.7 sonnet and claude code. Technical report, Anthropic, 2025. URL. https:
7 7/www. ant.roatiiiii..... c. cwm nwws//wwwweeeee/eeeeee http:///llucee cltcccaaaueeeeeeee 3 System Card.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.1322) 2025.
Rogerio Bonatti, Dan Zhao, Francesco Bonacci, Dillon Dupont, Sara Abdali, Yinheng Li, Yadong Lu, Justin Wagle, Kazuhito Koishida, Arthur Bucker, Lawrence Jang, and Zack Hui. Windows agent arena: Evaluating multi-modal os agents at scale, 2024. URL
Abbs/2409.om 08264
Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiy- ong Wu. Seeclick: Harnessing gui grounding for advanced visual gui agents. arXiv preprint arXiv:2401.10935, 2024.
Deepmind. Introducing gemini 2.0: our new ai model for the agentic era. Technical report, Deepmind, 2025a. URL https: 20tp:/bbeee222///ppoeeeet--t-tte google-gemini -ai wwwpduddddddeeddaeeeeeee dewember-eeeeeeeeeee-e
deepmind/
Deepmind. Gemini 2.5: Our most intelligent ai model. Technical report, Deep-
mind, 2025b. URL httts:: http:/::///booooooggeeeeeeeeee
http:///teeccoooloo/gooeeoeeeeee marchh20025.
gemini model 1 thiinking wriikinng-udatess
Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. Navigating the digital world as humans do: Universal visual grounding for gui agents. arXiv preprint arXhv:2410.05243, 2024.
Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.12945 2025a.
Dong Guo, Faming Wu, Feida Zhu, Fuxing Leng, Guang Shi, Haobin Chen, Haoqi Fan, Jian Wang. Jianyu Jiang, Jiawei Wang, et al. Seedl. 5-v1 technical report. arXiv preprint (tti::2555000662 2025b.
Wenyi Hong, Weihan Wang, Qingsong Lx, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. Cogagent: A visual language model for gui agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14281-14290, 2024.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276 2024.
10

---

Published as a conference paper at ICLR 2026
Raghav Kapoor, Yash Parag Butala, Melisa Russak, Jing Yu Koh, Kiran Kamble, Waseem Alshikh, and Ruslan Salakhutdinov. Omniact: dataset and benchmark for enabling multimodal generalist autonomous agents for desktop and web, 2024. URL
17553
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng. Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention, 2023. URL htt.tttpp///aiiii org/ abs/ 2309. D6180
Kaixin Li, Ziyang Meng. Hongzhan Lin, Ziyang Luo, Yuchen Tian, Jing Ma, Zhiyong Huang, and Tat-Seng Chua. Screenspot-pro: Gui grounding for professional high-resolution computer use. arXiv preprint wrXiv:2504.07981, 2025.
Wei Li, William Bishop, Alice Li, Christopher Rawles, Folawiyo Camphell-Ajala, Divya Tyam- agundlu, and Oriana Riva. On the effects of data scale ui control agents. Advances in Neural Information Processing Systems, 37:92130 30-92114, 2024.
Yang Li, Gang Li, Luheng He, Jingjie Zheng, Hong Li, and Zhiwei Guan. Widget captioning: Generating natural language description for mobile user interface elements, 2020. URL https:
//arxiv.org
was///////a////a/.aa//.///aaae/e abs/ awwaabaaaaa 20000 04295
Kevin Qinghong Lin, Linjie Li, Difei Gao, Zhengyuan Yang, Shiwei Wu, Zechen Bai, Stan Weixian Lei, Lijuan Wang, and Mike Zheng Shou. Showui: One vision-language-actinn model for gui visual agent. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp-
19498-19508, 2025.
Yuhang Liu, Pengxiang Li, Congkai Xie, Xavier Hu, Xiaotian Han, Shengyu Zhang, Hongxia Yang. and Fei Wu. Infigui-rl: Advancing multimodal gui agents from reactive actors to deliberative reasoners. arXiv preprint arXiv:2504. 14239, 2025.
Yadong Lu, Jianwei Yang, Yelong Shen, and Ahmed Awadallah. Omniparser for pure vision based gui agent. arXiv preprint arXiv:2408. 00203, 2024.
Zhengxi Lu, Yuxiang Chai, Yaxuan Guo, Xi Yin, Liang Liu, Hao Wang. Han Xiao, Shuai Ren, Guanjing Xiong, and Hongsheng Li. Vi-r1: Enhancing action prediction of gui agents by rein- forcement learning. arXiv preprint arXiv:2503.21620 2025.
Run Luo, Lu Wang, Wanwei He, and Xiaobo Xia. Gui-rl: A generalist rl-style vision-language action model for gui agents. arXiv preprint arXiv:2504. 10458, 2025.
Shravan Nayak, Xiangru Jian, Kevin Qinghong Lin, Juan A. Rodriguez, Montek Kalsi, Rabiul Awal, Nicolas Chapados, M. Tamer AÜzsu, Aishwarya Agrawal, David Vazquez. Christopher Pal, Per- ouz Taslakian, Spandana Gella, and Sai Rajeswar. Ui-vision: A desktop-centtri gui benchmark for visual perception and interaction, 2025. URL htttps://aaxiiiii
abs /2003. 11155.3..166666 15661
OpenAI. Computer-using agent: Introducing universal interface for ai to interact with the digital world. 2025a. URL httpttttt////paai...... com/ index http:///ottttttutttneeeeeaeeeeee
OpenAl. Openai 03 and 04-mini system card.
Technical report, OpenAl, 2025b. URL
httt::://ccd..oopnaaccoom/////// http:///pp..22222122700000-00110 027--550220-778888 8000 01017779--0000-777777722000
03
pdf System Card.
Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. Is chatgpt general-purpose natural language processing task solver? arXiv preprint arXiv:2302.06476, 2023.
Yujia Qin, Yining Ye. Junjie Fang. Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, et al. Ui-tars: Pioneering automated gui interaction with native agents. arXiv preprint arXiv:2501.12320 2025.
Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Mary- beth Fair, Alice Li, William Bishop. Wei Li, Folawiyo Campbell-Ajala, et al. Androidworld: A dynamic benchmarking environment for autonomous agents. arXiv preprint arXiv:2405.1457 2024.
11

---

Published as a conference paper at ICLR 2026
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:4402.03300 2024.
Linxin Song, Yutong Dai, Viraj Prabhu, Jieyu Zhang, Taiwei Shi, Li, Junnan Li, Silvio Savarese, Zeyuan Chen, Jieyu Zhao, Ran Xu, and Caiming Xiong. Coact-l: Computer-using agents with abs / 2200/550.0000 03923
coding as actions, 2025. URL CRL
Kimi Team, Angang Du, Bohong Yin, Bowei Xing, Bowen Qu, Bowen Wang. Cheng Chen, Chenlin Zhang, Chenzhuang Du, Chu Wei, Congcong Wang, Dehao Zhang, Dikang Du, Dongliang Wang, Enming Yuan, Enzhe Lu, Fang Li, Flood Sung. Guangda Wei, Guokun Lai, Han Zhu, Hao Ding. Hao Hu, Hao Yang, Hao Zhang. Haoning Wu, Haotian Yao, Haoyu Lu, Heng Wang, Hongcheng Gao, Huabin Zheng, Jiaming Li, Jianlin Su, Jianzhou Wang, Jiaqi Deng, Jiezhong Qiu, Jin Xie, Jinhong Wang, Jingyuan Liu, Junjie Yan, Kun Ouyang, Liang Chen, Lin Sui, Longhui Yu, Meng- fan Dong. Mengnan Dong, Nuo Xu, Pengyu Cheng, Qizheng Gu, Runjie Zhou, Shaowei Liu, Sihan Cao, Tao Yu, Tianhui Song, Tongtong Bai, Wei Song. Weiran He, Weixiao Huang, Weixin Xu, Xiaokun Yuan, Xingeheng Yao, Xingzhe Wu, Xinhao Li, Xinxing Zu, Xinyu Zhou, Xinyuan Wang, Y. Charles, Yan Zhong, Yang Li, Yangyang Hu, Yanru Chen, Yejie Wang, Yibo Liu, Yibo Miao, Yidao Qin, Yimin Chen, Yiping Bao, Yigin Wang. Yongsheng Kang, Yuanxin Liu, Yuhao Dong, Yulun Du Yuxin Wu, Yuzhi Wang. Yuzi Yan, Zaida Zhou, Zhaowei Li, Zhejun Jiang, Zheng Zhang, Zhilin Yang. Zhiqi Huang, Zihao Huang, Zijia Zhao, Ziwei Chen, and Zongyu Lin. Kimi-vl technical report, 2025. URL https: / a wwwpaaaiivvv.rro org/ aaaaaabs//2/0/ 2200. 07491
Xinyuan Wang. Bowen Wang, Dunjie Lu, Junlin Yang, Tianbao Xie, Junli Wang. Jiaqi Deng. Xi- aole Guo, Yiheng Xu, Chen Henry Wu, Zhennan Shen, Zhuokai Li, Ryan Li, Xiaochuan Li, Junda Chen, Boyuan Zheng, Peihang Li, Fangyu Lei, Ruisheng Cao, Yeqiao Fu, Dongehan Shin, Martin Shin, Jiarui Hu, Yuyan Wang, Jixuan Chen, Yuxiao Ye, Danyang Zhang, Dikang Du, Hao Hu, Huarong Chen. Zaida Zhou, Haotian Yao, Ziwei Chen, Qizheng Gu, Yipu Wang. Heng Wang, Diyi Yang, Victor Zhong, Flood Sung, Y. Charles, Zhilin Yang, and Tao Yu. Opencua: Open foun- dations for computer-use agents, 2025. URL https: httpp:::/a.....o..////oo
wwx.v.ooog.......... abs//508//2250 09123
Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, et al. Os-atlas: A foundation action model for gener- alist gui agents. arXiv preprint arXiv:2410.2222
2024.
Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh J Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. Advances in Neural Information Processing Systems, 37:52040-52094, 2024.
Tianbao Xie. Jiaqi Deng, Xiaochuan Li, Junlin Yang, Haoyuan Wu, Jixuan Chen, Wenjing Hu, Xinyuan Wang, Yuhui Xu, Zekun Wang, et al. Scaling computer-use grounding via user interface decompasition and synthesis. arXiv preprint arXiv:2505. 0atr:2555..222. /3227, 2025.
Yiheng Xu, Zekun Wang, Junli Wang. Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, and Caiming Xiong. Aguvis: Unified pure vision agents for autonomous gui interaction. arXiv preprint 2rXiv:2412.044555 2024.
Yuhao Yang, Yue Wang, Dongxu Li. Ziyang Luo, Bei Chen, Chao Huang, and Junnan Li. Aria-ui: Visual grounding for gui instructions. arXiv preprint arXiv:2412. 2024.
Xinbin Yuan, Jian Zhang, Kaixin Li, Zhuoxuan Cai, Lujian Yao, Jie Chen, Enguang Wang, Qibin Hou, Jinwei Chen, Peng-Tao Jiang, et al. Enhancing visual grounding for gui agents via self- evolutionary reinforcement learning, arXiv preprint arXiv:2505. 12370, 2025.
Yuqi Zhou, Sunhao Dai, Shuai Wang, Kaiwen Zhou, Qinglin Jia, et al. Gui-g1: Understanding rl rl-zero-like training for visual grounding in gui agents. arXiv preprint arXiv:2505.15810 2025.
12

---

Published as a conference paper at ICLR 2026
Table of Contents in Appendix
Additional Result
14
A.I Qualitative comparisons
14
A.2 Token Usage
14
A.3 Failure Case
14
Model Training Details
14
Evaluation Details
18
ScreenSpot- V2
18
ScreenSpot-Pro
18
OSWorld-G
18
OSWorld
18
C,5
WindowsAgentArenaa
18
Prompt
18
D.I Grounding Prompt
18
D.2 Planning Prompt
19
D.3 Selection Prompt
25
D.4 Additional Information
27
13

---

Published as a conference paper at ICLR 2026
A
ADDITIONAL RESULT
A.1 QUALITATIVE COMPARISONS
We present qualitative comparisons of UI-TARS-1.5-7B (Qin al. 2025) with and without test- time scaling strategies in Fig. and Fig. [ As shown, without the test-time strategy, errors in early grounding or planning stages can propagate and derail the entire task execution, making the agent highly susceptible to cascading failures.
A.2 TOKEN USAGE
We show the performance with respect to the average number of tokens used per task for test-time scaling with UI-TARS-L.5-7B (Qin et al. 2025 in Fig. 5
Success Rate (%
Success Rate (%)
Success Rate (%
40
40
40
K-
K=32
K=
K=8
K=16
Km32
30
K=8 K=16
K=32
30
30
K=
20
20
20
0.1
0.25
0.50
0.30
0.70
1.10
0.50
1.25
2.00
Number of Tokens (M)
Number of Tokens (M)
Number of Tokens (M)
(b)
(c)
Figure 5: Average number of tokens used per task (in millions) versus task success rate across different numbers K of action proposals in test-time scaling. Subfigures (a), (b), and (c) correspond to task execution lengths of 1, 15, 50, and 100 steps, respectively. We vary E (1,8, 16, 32) and report performance using task success (%) together with the average number of tokens consumed (in millions). When K = 1, it degrades to the setting where the test-time scaling strategy used.
A.3 FAILURE CASE
We present several failure cases of test-time scaling in Fig. In these cases, all generated action proposals {p} fail to produce the correct next step toward completing the task, ultimately causing the judge to select that setting the number of the search result display Google infeasible. the bottom row, method lacks the knowledge correctly insert note in LibreOffice Impress and instead proposes an incorrect random action.
incorrect action. Specifically, in the top row, the model fails recognize
B MODEL TRAINING DETAILS
Our models are initialized from UI-TARS-L.5-77 (Qin et al. 2025) and OpenCUA-32B (Wang et al., 2025). We use learning rates of 10-0 and 10-5 respectively. rolling out N = 8 responses per input during training. The batch size is set to 256, and the and 32B models typically converge after approximately 250 iterations. The training parameters are summarized below (Tab. 7
During training, all images are resized to make their dimensions are divisible by 28 (Bai et al. 2025). The predictions are then retrieved, and the ground-truth bounding boxes are scaled by the same ratio for reward calculation. The 7B and 32B models are trained on 16 H100 and 32 H200 GPUs, taking approximately 2 and days, respectively.
When evaluating on real-world dynamic environment benchmarks (i. e., World (Xie et al. 2024 and Window/Agen AgentArenntta (Bonatti et al. 2024)), we use the action space from (Agashe et al. 2025 and employ o3 and GPT-5 as planners OOpenAl 2025b). By default, K action proposals are sampled at each step for selection.
The training data are sourced from Aria-UI, Web (Yang et al. 2024), OmniACT (Kapoor et al.) 2024), UI Vision (Nayak et al. 2025), Widget Caption (Li al. 2020), and OS-Alras-Desktop
14

---

Published as a conference paper at ICLR 2026
User Instruction: Seurch for one way flight from Dublin to Vienna on 10th next month for 2 adults.
FREE A A TRWEL INSURANCE
FREE SEATS & TRAVEL INSURANCE
Key Step 1
Key Step 2
Key Step Clear the text field (After multiple attempts of modifying the text to "Dublin") "
Key Step 1: Click the "London Stansted
& IRWEL WAUNANLE
DON'T RISN A OUR BAGS POLICY EXPLAINED
THE
Key Step 3
Key Step 4
Key Step 3: Scroll down to find the flight
Key Step 4: Scroll up to find the flight (Fail to locate ticket by serolling)
(a) w/o Test-time Scaling
FRATS & TRAEE IEUURNCE
INSURANCE
Key Step 1
Key Step 2
Key Step 1: Click the "London Stansted
Key Step 2: Attempt to modify the "From" field to "Dublin"
8
FEAT & TRWEL IEURANCE
Key Step 3
Key Step 4
Key Step 3: Type "Dublin" the "From" field
Key Step 4: Keep modifying the search information
(b) w/ Test-time Scaling
Figure 6: Example trajectories improved by our test-time scaling strategy. We show key steps of completing the user task using UI-TARS-i.5-7B. (a) Without our test-time scaling strategy, UI- TARS- shifts its action proposal from modifying the search field to scrolling the page to find the ticket. This occurs due early planning and grounding errors in the "From" field. (b) With our strategy, it consistently modifies the search information to complete the task.
15

---

Published as a conference paper at ICLR 2026
User Instruction: Could you please change the number of search results displayed on one page to 50? find that having more results visible at once significuntly enhances my reseurch efficiency, us reduces need to constantly click through multiple pages.
Pi; Previous attempts failed already in the address bar; for change approach. Since it's matler for Google's own search box, why not just go to their website? Typing google.com into the address should the direct way.
desired settings, and
Px: This search function seems bit weak; I can't an option to adjust number resalls displayed. remember Chrome's settings page has "Performance" option, which should involve some interface display settings. I'll check the left-hand menu first; mayhe I find the function need there.
Ps:
User Instruction: page 2, add note same to the title into the slide. Make the font of title bold.
Ps; I noticed "Click to add Text" in the center of the slide, waiting for input. The completinn the task is add the title "Team Members" itself as a note. The cursor is now in the right to directly type this text to maintain consistency with the title.
Team Members
position:
Pz: just elicked the input bex write note, but that doesn't seem right. Looking the har, seems appropriate add notes using the Insert function. Let me click the "Insert" option at the top this operation again. Ps;
Figure 7: Failure cases. In these examples, all action proposals {p} fail to contain the correct step needed to progress toward the task instruction, leading the judge to select an incorrect action since no proposal is valid. In the top row, the model fails to recognize that the task is infeasible. In the bottom row, the model lacks the required knowledge on how to add note in LibreOffice Impress.
Table 7: Training parameters for GTAI models.
Parameter
GTA1-7B
GTA1-32B
Base Model
1TTARS-11.-778 (Qin et al. 2025
OpenCUA-32BB (Wang ct al.
2025
Learning Rate
10
10
Optimizer
AdamW
AdamW
Max Model Length
32768
32768 6 1 250
Model Type
bfloat 16
Max Gradient Clip Norm
1 250
Optimization Iterations
(Rollout per Input)
8 2 x 10 1 32
8 2 x 10 I
Value for clipping
Rollout Temperature
Rollout Max Response Length
128
Image Processor Max Pixels
12356789
12356789 32 H200
Training GPUs
16 H100
(u et al. 2024 70K datasets are sampled from training, and the retained dataset is sampled as summarized in the table below to form our 70K training dataset.
Table 8: Distribution of datasets and domains in our training set.
Domain
Dataset
Percentage
Atlas-Dessstop
57.3
oS
OMniACT
2.6
UI Vision
5.7
Web Mobile
Aria-UI-Wet
14.2
Widget Caption
20.2
16

---

Published as a conference paper at ICLR 2026
User Instruction: Hey. need quick way back to this site. Could you whip shorteut my desktop for me?
Key Step 1
Key Step 2
Key Step 1: Right click
Key Step Click the Chrome setting
Key Step 3
Key Step 4
Key Step 3: Close the Chrome (After multiple grounding and planning errors while finding the save shortcut)
Key Step 4: Open Chrome (Fail by getting stuck authentication)
(a) w/o Test-time Scaling
Key Step 1
Key Step 2
Key Step I: Click the Chrome setting
Key Step 2: Click the "cast" panel
4-m
Key Step
Key Step 4
Key Step Click the "Cast"
Key Step 4: Save the shorteut
(b) Test-time Scaling
Figure 8: Example trajectories improved by our test-time scaling strategy. We show key steps of completing the user task using UI-TARS-i.5-7B. (a) Without our strategy, UI-TARS-i.5-7B tempts to save a shortcut by closing Chrome, encountering authentication, and getting stuck. This results from early planning and grounding errors in locating the shorteut panel. (b) With our strat- egy, it focuses on opening Chrome settings and successfully completes the task.
17

---

Published as a conference paper at ICLR 2026
C EVALUATION DETAILS
We evaluate our method for GUI Grounding on the ScreenSpot-V V2 (Cheng et al. 2024 Wu et al.
2024).
ScreenSpot-Pro (Li al. )025), and OSWorld-G (Xie et al., 2024) benchmarks, and for
Agent Task Execution on the WSWorl (Xie et al. 2024) and WindowsAgentArem (Bonatti et al.,
2024
benchmarks. Detailed descriptions are provided below.
C.1
SCREENSPOT-V V2
The ScreenSpot-V V2 benchmark (Cheng al., 2024 Wu et al. 2024 is extension of ScreenSpot CCeng et al. 2024), correcting and re-annotating 11.32% of incorrect samples (e.g., fixing spelling errors and incorrect bounding boxes). It assesses grounding ability across three domains: mobile, desktop. and web. The native image resolution and paired instructions are fed into our model for grounding, using single GPU with 6 precision.
C.2 SCREENSPOT-PRO
The ScreenSpotPro benchmark (Li et al.) 2023) focuses on professional nigh-resolution (up to 3840 x 2160) computer use and are mainly categorized into Development, Creative, CAD, Sci- entific, Office, and OS domains, spanning 23 applications. The paired instructions are typically concise and instruction-level, further increasing the challenge. Both our 7B and 32B models can perform inference efficiently single 80GB GPU with bfloat16 precision.
C.3 OSWORLD-G
The OSWorld-G benchmark (Xie et al. 2025 is curated with finely annotated samples across five task types: text matching, element recognition, layout understanding, precise manipulation, and refusal. In addition to instruction-level annotations, it provides fine-grained annotations for each example, rephrasing the original instructions explicitly decompose the GUI knowledge required to complete the task. We report performance on both annotation types. Similarly, our 7B and 32B models perform inference efficiently on a single 80GB GPU with bf precision.
C.4 OSWORLD
The OSWorld benchmark (Xie et al., 2024), based on the Ubuntu operating system, originally tains computer tasks spanning both web and desktop applications in open domains. It has been further refined by improving environment stability and evaluation functions, resulting in OSWorld- Verified (Xie et al. 2024h. We evaluate both versions using o3 and GPT-5 (OpenAl 2025b as planners, while serving our 7B and 32B models with the vllm codebase (Kwon et al. (023) for
grounding. All evaluations are performed using 48 Docker instances (i. e., virtual environments) and served models (Kwon et al. 2023)
C.5 WINDOWSAGENTARENNA
The WindowAggentArena benchmark (Bonatti et al. 2024 is built on the Windows operating system and focuses on commonly used applications, tools, and web browsers. Similar to the OSWorld eval- uations, evaluate it using o3 and GPT-5 (OpenAl 2025b) as planners, running under 8 Docker instances (i. e., virtual environments) with 8 served models (Kwon et al. 2023).
D PROMPT
We present the system prompts used to evaluate our models, categorized into grounding and planning prompts.
D. GROUNDING PROMPT
We present the GTAL-7B and GTAI-32B prompt, respectively in Tab. 9 and Tab.
10
18

---

Published as a conference paper at ICLR 2026
Table 9: The system prompt used for GUI grounding with GTA1-7B.
You are an expert UT element locater. Given GUT image and user's s element description, provide the coordinates of the speci fied element as a single (x,y) point. The image resolution is height (height) and width (width). For elements with area, return the center point.
Output the coordinat pair exactly:
(x,
Table 10: The system prompt used for GUI grounding with GTA1-32B.
You are a GUI agent. You are given a task and screenshot the screen. You need to perform a series pyautogui actions to complete the task.
D.2 PLANNING PROMPT
We present the system prompt tuned for o3 and GPT-5 as the planner in Tab. 11 and Tab.
12
Table 11: The system prompt employed for planning task executions with o3.
You are agent which follow my instruction and perform desktop computer tasks as instructed.
You have good knowledge of computer and good internet connection and assume your code will run on a computer for controlling the mouse and keyboard.
You are on Ubuntu operating system and the resolution of the screen is 1920x1080.
For each step, you will get: An observation computer screen and you will predict the action of the computer based on the image.
an image, which is the screenshot of the
Access to the following class and methods to interact with the UI:
class Agent:
def click instruction: str, num clickst int 1, button_type: str = 'left', hold_keys: List = []):
rrclick the element
Args:
instruction:str: decribe the element you want interact with in detail including the visual description and function description. And make it clear and concise. For example you can describe what the element looks like, and what will be the expected result when you interact. with it.
19

---

Published as a conference paper at ICLR 2026
num num_clicks:inn number of times to click the element
batton_type:stee which mouse button to press can be "left", "middle", or "right" hold_keys:List, list of keys to hold while clicking
def done (self, return_value Union [Dict, str, List, Tuple, int, float, bool, NoneType] :one) 'En the current task with a success and the required return value'!!
startinng_escripiionn: str, deniiogddecrip:ion:: str, hold keys: List []): DDra from the starting description to the ending description
def
Args:
starting des.iigig.derrppiinnssss....
very detailed description of where to start the drag action.
This description should be at least a full sentence. And make it clear and concise. ending_deet desting_decrripiinssttt, a very detailed description of where to end the drag action. This description should be at least a full sentence. And make it clear and concise. hold_keys:List list of keys to hold while dragging
def fail (self):
Ennn the current task with failure, and replan the whole task.''r
def highlight_text_ppe (self, starting_phrase: str, ending_phrase: str):
"igghhighh text span between provided starting phrase and ending phrase. Use this to highlight words, lines, and paragraphs. Args:
phatiesstpppasesttt, the phrase that denctes the start of the text span you want to highl iggh. If you only want highlight in that single word. ending._hrase::ta.
word, just pass
the phrase that denotes the end of the text span you want to highlight. If you only want to highlight one word, just pass in that single word.
def hold and press (self, hoys: List, press_keya List):
Hold a list of keys and press a list of keys
Args:
hold_keys:List, list of keys to hold
press_keys:List list of keys press in sequence
20

---

Published as a conference paper at ICLR 2026
def hotkey (self, keys: List): prrrress hotkey combination
Args:
keys:List the keys press in combination in list format (e.g. l'ctrl', 'c'
...
def open (self, " Open any application or file with name app_or_filenane Use this action to open applications or files on the desktop, do not open manually.
str):
Args:
app_or, or filename:str, the name of the application or filename to open
def scroll (self, instruction: str, clicks: int, shift:
bool = False)
"''Scroll the element in the specified direction Args:
instruction:stz.
very detailed description
which element to enter scroll in. This description should be at least full sentence. And make it clear and concise.
clicks:int, the number of clicks to scroll can be positive (up) negative (down)
shift:bool, whether to use shift+scroll for horizontal scrolling
def sa___eel__l_llll values (self,
[ict, Any),
app sap_name: str, _ame: str):
Urus this to set individual cell values in a spreadsheet. For example, setting A2 to "hello" would be done by passing ("A2": "hello") as sell_values. The sheet must be opened before this command can be used. Args:
cell_values: [ict, Anyl, A dictionary of cell values to set in the spreadsheet. The keys are the cell coordinates in the format "A1", "R2", etc.
Supported value types include: float, int, string, bool, formulas.
pop_name: str, The name of the spreadsheet application. For example, "Some "Some_sheet.we sheet. sheet_name: str, The name of the sheet in the spreadsheet. For example, "Sheet1".
def switch applications (self, app_code) Switch to a different application that is already open
Args:
app_code:str the code name of the application to switch to from the provided list of open
applications
21

---

Published as a conference paper at ICLR 2026
def eype(self, plement_descripge optional [str] = None, text: str " overwrite: bool False, enter:
bool = False)
" Type text into a specific element
Args:
element_ description:str, a detailed description of which element to enter text in. This description should be at least full sentence. text:str, the text to type
overwrite:bool, Assign it to True if the text should overwrite the existing text, otherwise assign it to False. Using this argument clears all text in element.
enter:bool, Assign it to True if the enter key should be pressed after typing the text, otherwise assign it to False.
...
def wait (self, time: float):
''Wtait for specified amount of time
Args:
time: float the amount of time to wait in seconds .".
The following rules are IMPORTANT:
If previous actions didn't t achieve the expected result, do not repeat them, especially the last one. Try to adjust either the coordinate or the action based the new screenshot.
not predict multiple clicks at once. Base each action on the current screenshot; do not predict actions for elements or events not yet visible in the screenshot. You cannot complete the task by outputting text content in your response. You must use mouse and keyboard to interact function when you
with the computer. Call "'agent. think the task can not be done.
You must use only the available methods provided above to interact with the UI, do not invent new methods.
You should provide a detailed led observation of the current computer state based on the full screenshot. in detail in the "Observation:" section.
Provide any information that possibly relevant to achieving the task goal and any elements that may affect the task execution, such as pop-ups, netifications, error messages, loading states, etc..
You MUST return the observation before the thought.
You should think step by step and provide a detailed thought process before generating the next action:
Thought:
Step by Step Progress Assessment:
Analyze completed task parts and their contribution to the overall goal
22

---

in a comment and only return ONE line of Published as a conference paper at ICLR 2026
Reflect on potential errors, unexpected results, or obstacles If previous action was incorrect, predict a logical recovery step
Next Action Analysis:
List possible next actions based on current state
Evaluste options considering current state and previous actions
Propose most logical next action
Anticipate consequences the proposed action Your thought should be returned in "Thought:" section. You MUST return the thought before the code.
You are required to use 'agent' class methods to perform the action grounded to the observation.
Return exactly ONE line of python code to perform the action each time. each step (example: "'agent.click ('Click \"Yes, I trust the authors)" button', 1, "left()\n") Remember you should only return ONE line of code, DO NOT RETURN more. You should return the code inside a code block, like this:
'pythhn
agent. C*Click \YYes, trust the authors)" button', , 1, "left")
For your reference, you have maximum of 100 steps, and current step is [current_stepl out of (max_steps) If you are in the last step, you should return "'agent.done "" or faagent.ail(() ". according to the result.
Here are some guidelines for you:
1. Remember generate the corresponding instruction to the code before code.
2. 'agent. clickt click can have multiple clicks. For example, agent.click ('Click \YYes, trust the authors)" button', 2, "left") is double click.
3. Return m'aagen.dooee "* in the code block when you think the task is done (Be careful when evaluating whether the task has been puccessfully completed). Return
in the code block when you think the task (gient.fail())) can not be done.
4. Whenever possible, your grounded action should use hot-keys with the hotkey () action instead of clicking or dragging.
5. Save modified files before returning "'agent.donel () "", When you finish modifying a file, always save it before proceeding using
baagent.hotkey ctrl', 's'])' or
equivalent. Tasks may involve multiple files. Save each after finishing modification.
6. If you meet "Authentication required" prompt, you can continue to click "Cancel" to close it.
My computer's password is (CLIENT_PASSWOR) feel free use it when you need sudo rights.
23

---

Published as a conference paper at ICLR 2026
First give the current screenshot and previous things we did short reflection, then RETURN ME THE CODE ASKED FOR NEVER EVER RETURN ME ANYTHING ELSE.
Table 12: The system prompt employed for planning task executions with GPT-5.
Role and Objective
An agent with strong computer knowledge and a good internet connection, designed to execute desktop computer tasks on Ubuntu precisely as instructed by the user. Assumes tool calls will to control the computer. Has access to all its reasoning and knowledge for use in tasks.
Instructions
Begin each user task with concise checklist (3-7 items) conceptual, non-implementation sub-tasks.
Revise the sub-tasks checklist as the task progresses, based on the latest screenshot and previous actions. Interact solely using the provided tool actions; do not invent or assume any unlisted methods. Use only tools explicitly listed in the available actions for every step. Base every action on observable elements in the latest screenshot; never anticipate present or visible.
assume elements not yet
For each step, you will receive a screenshot, tool execution results, and the remaining number of steps allowed the user task.
If an option or input is not specified in the user task (e.g., creating a new file without specifying a name), use the default settings.
## Action Execution Guidelines
Execute exactly one tool call per interaction.
Prefer the 'hotkey' action (tool call) over 'click' 'drag_and_drop' where possible.
For spreadsheet value formula changes in LibreOffice
Cale, Writer, Impress, always use 'set_cell_values for both single-cell and multi-cell value or formula editing. When highlighting text, use only the ntighlight_tex__ppp or 'hotkey' (tool calls).
Dismiss "Authentication required" prompts by clicking "Cancel".
All tool calls are permitted within the provided action list; do not attempt actions outside this set.
Additional Information
Leave windoss/aplications open at task completion.
Upon fully completing the user's task, briefly summarize results if applicable, then return 'TERMINATE'.
**Feasibility First**: Confirm the task can be completed with available files, applications, and environments before starting.
**Strict Adherence**: Only perform actions the user has explicitly requested; avoid unnecessary steps.
24

---

Published as a conference paper at ICLR 2026
**Completion Criteria*:: Only return "TERMINATE" when all user requirements are met in full.
-xImpossibility Handling**: Return "INFEASIBLE" if completion is blocked by environmental constraints.
**Screenshot Verificationte: Always check the screenshot before proceeding.
Additional Rules
use it if sudo The sudo password is "[CLIENT_PASSSWOS privileges are required.
Leave all windows and applications open after completing the task.
Only use 'TERMINATE' when all user requirements have been fully satisfied; provide a brief summary of results if applicable.
Before proceeding, confirm that the task is feasible with the currently available files, applications, and envi environmen;; if it is impossible to complete due to environmental constraints, return 'INFEASIBLE', Strictly follow user instructions, avoiding unnecessary or extraneous steps.
Always review the latest screenshot before every action.
Execution Procedure
Briefly review prior actions, the current checklist, and the latest screenshot before each tool call.
Before each action, state in one line the purpose and required minimal inputs.
After each action, validate the result in 1-2 lines using the updated screenshot. If the action was unsuccessful, adapt your approach before proceeding.
Only return the selected action (s) do not elaborate or output other information.
Work deliberately and avoid unnecessary or extranecus steps; strictly adhere to user instructions.
Proceed methodically and efficiently, ensuring all user requirements are met before terminating.
D.3 SELECTION PROMPT
The system prompt used with and GPT-5 for selecting action proposals, shown in ?? and ?? respectively.
Table 13: The system prompt for o3 select an action proposal.
You are expert at evaluating the planning and reasoning of UI agents working toward achieving a goal.
My computer's password is COLLEENT_PASSWRR feel free to use it when you need sudo rights or login.
Each time, will provide you with:
The current screenshot of the UI of width (width) and height (height)
25

---

Published as a conference paper at ICLR 2026
The goal of the task
Past histories of planning and actions that have been taken A list of (N_PLANNING) different planning approaches
toward achieving the goal in the current state in this form: <screenshot caption> Thought: <planning and reasoning> Action: <DI action>
Observation:
Your task is to select the single most effective planning approach that best advances toward the goal. Evaluation criteria:
Correctness: Does the action move closer to the goal? Effectiveness: Does it make meaningful progress immediately?
Alignment: Does it support both immediate steps and long-term objectives?
Planning quality: Is the thought process clear, concise, and logical?
Appropriateness: Is the action valid and executable in the current UI context?
Note that some planning approaches may be similar - do not 1et the number similar approaches dominate your decision. Evaluate each planning on its own merits.
Respond **nny*** with valid JSON (no extra keys comments): "'json
((
"explaining": "Your explanation of why this planning is best using the evaluation criteria", "index": The index of the best planning (0, 1, (N_INDEX))
H}
...
Table 14: The system prompt for GPT-5 to select action proposal.
Role and Objective
Assess the planning and reasoning of a agent to determine the most effective action for advancing toward a specified task goal. You may use the computer password [ICHENNTPASSWOR)] during this process if needed.
Workflow Checklist
Begin each assessment by generating a concise checklist (adapt as appropriate for task complexity) of evaluation steps to ensure a systematic and methodical analysis. Inputs
For each assessment, you will receive:
The task goal
The history of planning and actions performed
current screenshot
A list of (N_PLANNING) alternative planning approaches for achieving the goal, in the current context. Each approach will be formatted as:
26

---

Published as a conference paper at ICLR 2026
Thought: <sumnary, goal, screenshot observation>
Action: <proposed UI action>
Action Function Definition
Actions are formatted as function calls. The specification for these calls is provided here: DONNCTION_CALLLDEFINTTII
Assesament Criteria
Correctness: Does the proposed action logically advance the goal?
Effectiveness: Is immediate progress made?
Alignment: Does it support both the step and overall objective?
Planning Quality: Reasoning is clear, concise, and logical. Appropriateness: Action is valid/executable in the current context.
Matchness: Does the action correspond exactly to names/noum in the user task? Avoid generalization or conflation.
Exactness: Does the action relate to the user task?
extra or unnecessary steps are performed. Completeness: terminate, does the action complete the user task?
Be aware that some planning approaches may be similar-vvaluate each on its own merits, and do not allow the frequency of similar approaches to bias your assessment. Carefully assess each approach and select the best one based on the above criteria.
Output Format
Produce a single, strictly valid JSON object with the following fields:
'explaining' (string, required): A concise (1-4 sentences) justification for why the chosen approach optimal in light of the assessment criteria; briefly explain why.
if none are effective,
'index' (integer, required): The 0-based index (0, (N_INDEX)) identifying the best approach. You must choose one of the approaches.
Do not output anything except the required JSON object
*Crrefulll evaluate each approach and select the best one based on the criteria.**
D.4 ADDITIONAL INFORMATION
We show the system prompts applied in ablations (Tab. in Tab. 15 Tab. 16 and Tab. 16 corresponding to three settings: i) click, loU, and format rewards, ii) click and loU rewards, in) click and format rewards. For the AndroidWorld benchmark (Rawles al., (024), follow the planner setting from (Yang et al. 2024) and use the system prompt for grounding shown in Tab. 18
Table 15: The system prompt for applying click, IoU, and format rewards.
27

---

Published as a conference paper at ICLR 2026
You are an expert UI element locater. Given GUT image and user's element description, provide the coordinates of the specified element as a single (x, point. The image resolution is height (height) and width (width). For elements with area, return the center point.
First analyze the reasoning process within <think></think> tags, then provide the bounding ng box of the target element within <bbox></bbox> tags and output the coordinate pair within <answer<</answer> tags.
Output exactly:
<think>your reasoning process</think> <bbox> x1, </bbox> <answer> (x, < answer>
Table 16: The system prompt for applying click and loU rewards.
You are an expert UI element locator. Given a GUI image and user's s element description, provide the coordinates of the specified element as a single (x, point. The image resolution is height (height) and width (width). For elements with area, return the center point.
First provide the bounding box of the target element within <bbox></bbox> tags, then output the coordinate pair within <answer></answer> tags.
Output exactly:
<bbox> [x0, y0, x1,,yl/</bb//l </bbox>
<answer> (x,y) </answer>
Table 17: The system prompt for applying click and format rewards.
You are an expert UI element locator. Given a GUI image and user's element description, provide the coordinates of the specified element as a single (x, point. The image resolution is height (height) and width (width). For elements with area, return the center point.
First analyze the reasoning process within <think></think> tags, then provide only the coordinate pair within <answer></answer. tags.
Output exactly:
<think>your reasoning process</think> <answer> (
Table 18: The system prompt for grounding with action histories.
28

---

Published as a conference paper at ICLR 2026
You are an expert UI element locater specializing in precise coordinate (x,y) prediction of the described element.
Por each request, you'll 11 receive:
A screenshot of a uI interface (resolution:
(width))((eiight) pixels)
Context about the user's objective and previous actions description of the target UI element whose center must be predicted
Output exactly:
(x,
29


Reviews:
[{"review_id": "L6KiDhieoY", "reviewer": "Reviewer_p5q9", "summary": "The paper’s main contributions are (i) a test-time scaling strategy that selects the next action and (ii) a GRPO-trained GTA model for coordinate grounding. Specifically, at each step, the planner samples multiple candidate actions, and a multimodal LLM judge selects the best one. For actions requiring coordinates, a GTA model trained with GRPO provides more precise grounding. The GTA grounding model achieves SOTA on grounding benchmarks (ScreenSpot-Pro, ScreenSpot-V2, OSWorld-G). On real OSWorld tasks, o3 with test-time scaling + GTA1-7B grounding attains a 45.2% success rate with a 100-step cap, surpassing CUA o3 at 42.9% with a 200-step cap.", "questions": "- Under matched compute, how much does your test-time scaling improve over other test-time strategies?\n- Does the judge score candidates or generate a new action, and how sensitive are results to this choice?\n- If you increase the number of action proposals beyond 32, do gains continue?\n- What is the performance delta on OSWorld with and without the grounding model? Please provide ablations quantifying its impact.\n- How do open-source models perform under your test-time scaling, with and without the GTA grounding model? It would help to report accuracy-cost curves and confidence intervals for these settings.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Strong empirical results across both grounding datasets and interactive environments, reaching SOTA on multiple benchmarks.\n- The two-stage (planning+grounding) framework makes it straightforward to switch Planner/Judge modules or upgrade the grounding model.\n- The light-weight data cleaning pipeline can be integrated with existing data pipelines.", "weaknesses": "- The description of test-time scaling is under-specified. For example, when the judge \"picks the best candidate\", is the output a score over candidates, or a rewritten action? Some results (e.g., Table 1) could be simplified to improve readability by moving some baselines to the appendix.\n- Evaluations of test-time scaling (TTS) focus on ablations against the same agent without TTS. At least there should be equal-compute comparisons (e.g., self-consistency / majority voting / alternative sampling strategies).\n- There are no confidence intervals. Reported gains over an OpenAI o3 baseline are ~2 points (Table 4). Without multiple runs and confidence intervals, statistical significance is unclear.\n- The results of larger models are missing. GTA1-7B with GPT-5 and 32B configurations are \"unavailable\". If this is due to API cost, a proxy study (e.g., replaying 7B trajectories with a 32B grounding model) would strengthen the insights of the experiments. Likewise, results for open-source planners with test-time scaling + GTA grounding would be valuable.\n- The paper appears to lack OSWorld ablations isolating the impact of the GTA grounding model.\n- The proposed method largely adapts known ideas. Test-time scaling here is similar to best-of-N, and the GTA model is based on GRPO + data cleaning + reward shaping. It is reasonable that these techniques can boost performance, so more analysis and insights are required in the experiments (see suggestions above).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pChjUFzIXl", "reviewer": "Reviewer_9sM6", "summary": "The paper introduces a GUI agent that works in two stages: planning and grounding. The proposed solution improves each stage differently:\n1. the planning stage through test-time scaling: sampling K times at each step and asking a judge to select the \"best\" one.\n2. the grounding stage by fine-tuning a model using GRPO", "questions": "How is the training dataset built from the collection mentioned in the paper?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- strong empirical results on relevant benchmarks\n- simple method\n- ablations and discussion on the efficacy of the different reward signals (for thinking, for location)", "weaknesses": "- the dataset curation step should be part of the \"algorithm\", otherwise the comparison with other methods is not \n- reduced novelty\n- no ablation to understand the improvement brought by each of the two components in the agent task execution scenarios", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3nEeDINo7U", "reviewer": "Reviewer_88ea", "summary": "GTA1 is a GUI agent that improves both plan selection and precise clicking on complex screens. At each step, it samples multiple action proposals and uses a judge model to pick the best one, trading extra compute for better decisions. For grounding, it trains a simple RL model that directly predicts click coordinates with a click reward, achieving SOTA results on ScreenSpot-Pro, ScreenSpot-V2, and OSWorld-G, and strong task success on OSWorld and WindowsAgentArena. The key contributions are test-time scaling for robust planning, a minimal RL recipe for grounding, and an analysis showing “thinking” isn’t needed in static UIs but can help in dynamic ones.", "questions": "1. Experiments use o3/GPT-5 as planners and a multimodal LLM judge. Can you test how performance transfers to an open-source stack?\n\n2. Could you report the judge’s pairwise selection accuracy and show how judge errors propagate to end-task failure?\n\n3. Can you share success–vs–cost curves (tokens, latency, $) across K and horizon, and results for adaptive policies (early exit, dynamic K per step)?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper tackles two concrete pain points—plan selection and precise grounding, and pairs them with two simple fixes: test-time scaling for planning and a minimal RL recipe for click grounding. The framing is clear and the approach maps tightly to the problems.\n\n2. Test-time scaling actually pays off. Sampling K proposals per step and using a judge consistently boosts success while also cutting wall-clock via concurrent sampling.\n\n3. Directly rewarding clicks inside the target element keeps the training signal simple and on-task, yet reaches SOTA on ScreenSpot-Pro, ScreenSpot-V2, and OSWorld-G. The simplicity is a strength.\n\n4. The paper shows that explicit reasoning isn’t needed for static UIs and can even hurt, while helping in dynamic settings when trajectories/objectives are involved. This nuance helps practitioners know when to spend tokens on reasoning.\n\n5. Results cover both grounding and full agent task execution (OSWorld and WindowsAgentArena), and scaling tests generalize across agents, horizons, and K settings, with qualitative analyses of cascading failures. The breadth makes the claims more convincing.", "weaknesses": "1. Test-time scaling samples K proposals each step and uses a judge, which can raise token and inference costs even if concurrent sampling cuts wall-clock time. The paper shows speedups and success gains but does not report detailed cost curves (tokens, latency) across K and horizons.\n\n2.  The paper observes little average gain from “thinking” and attributes sample-wise differences to training instability. “Thinking helps only in dynamic UIs” needs stronger controls. The authors can test different “thinking” budgets, report variance to separate instability from genuine reasoning effects, and clarify when trajectories/objectives trigger reliable wins.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iKKCOr4ZtM", "reviewer": "Reviewer_UGQ4", "summary": "This paper introduces the GUI Testing-time Scaling Agent (GTA1), which aims to solve challenges in planning and action localization for GUI agents. Through a testing-time scaling strategy, it samples multiple candidate action proposals at each step, from which a judgment model selects the optimal solution. This is combined with a reinforcement learning-based localization model that directly predicts interaction coordinates and is optimized via click rewards, leading to effective performance improvement.\nExperimental results show that GTA1 achieves state-of-the-art performance on multiple benchmarks. For instance, on the ScreenSpot-Pro benchmark, the 7B model reached an accuracy of 50.1%, surpassing UGround-72B's 34.5%. On the OSWorld benchmark, GTA1-7B achieved a task success rate of 45.2%, outperforming all compared methods. These results demonstrate the effectiveness and robustness of the proposed method.", "questions": "1.I have observed that the training data originates from Aria-UI-Web, OmniACT, UI Vision, Widget Caption, and OSAltas-Desktop. Could you elaborate on the rationale for selecting these particular datasets? Furthermore, what considerations were made in terms of the volume and heterogeneity of the data?\n\n2.it is stated that the training dataset underwent a filtering process based on the quality of bounding boxes. Could you provide the specific quantities of the dataset both prior to and subsequent to this filtering procedure?\n\n3.This manuscript evaluates performance on benchmarks such as OS-World and WindowsAgentArena. What was the reasoning for not incorporating agent data into the training set? Additionally, could you compare the potential efficacy of a unified model that performs both planning and grounding, versus the current methodology which involves test-time scaling for planning followed by a separate grounding step?\n\n4.It is mentioned that 70k data samples were utilized for training. Is it anticipated that performance would continue to enhance with a further increase in the scale of the data? I wonder if there is a scaling law here?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.GTA1 effectively addresses the planning and localization challenges for GUI agents through a testing-time scaling strategy and a reinforcement learning-based localization model, achieving state-of-the-art performance across multiple benchmarks.\n\n2.Extensive experiments have validated the effectiveness and generalizability of the method.", "weaknesses": "It is noted in the manuscript that the foundational model for GTA1-7B is UI-TARS-1.5-7B, while for GTA1-32B it is OpenCUA-32B. This raises the question of how the performance would be affected if qwen2.5-vl or qwen3-vl were used as the base models instead. Further experimentation is required to exhibit the superior generalization capabilities of the proposed methodology.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)