UI-Ins: Enhancing GUI Grounding with Multi-Perspective Instruction as Reasoning
Authors: Liangyu Chen
Hanzhang Zhou
chenglin cai
Jianan Zhang
Panrong Tong
Xu Zhang
Quyu Kong
Chen Liu
Yuqi Liu
Wenxuan Wang
Yue Wang
Qin Jin
Steven HOI
Venue: ICLR 2026 Poster (2026)
Keywords: GUI Grounding
GUI Agents
Multimodal Large Language Model
Primary area: foundation or frontier models, including LLMs

Abstract: GUI grounding, which maps natural-language instructions to actionable UI elements, is a core capability of GUI agents. Prior works largely treats instructions as a static proxy for user intent, overlooking the impact of instruction diversity and quality on grounding performance. Through a careful investigation of existing grounding datasets, we find a 23.3% flaw rate in their instructions and show that inference-time exploitation of instruction diversity yields up to a substantial 76% relative performance improvement.
In this paper, we introduce the Instruction-as-Reasoning paradigm, treating instructions as dynamic analytical pathways that offer distinct perspectives and enabling the model to select the most effective pathway during reasoning. To achieve this, we propose a two-stage training framework: supervised fine-tuning on synthesized, diverse instructions to instill multi-perspective reasoning, followed by reinforcement learning to optimize pathway selection and composition.
Our resulting models, UI-Ins-7B and UI-Ins-32B, achieve state-of-the-art results on five challenging grounding benchmarks and exhibit emergent reasoning, selectively composing and synthesizing novel instruction pathways at inference. In particular, UI-Ins-32B attains the best grounding accuracy, scoring 87.3% on UI-I2E-Bench, 57.0% on ScreenSpot-Pro, and 84.9% on MMBench-GUI L2. Furthermore, our model demonstrates strong agentic potential, achieving a 74.1% success rate on AndroidWorld using UI-Ins-7B as the executor. Our in-depth analysis reveals additional insights such as how reasoning can be formulated to enhance rather than hinder grounding performance, and how our method mitigates policy collapse in the SFT+RL framework. All code and models are released.

Published as a conference paper at ICLR 2026
UI-INS: ENHANCING GUI GROUNDING WITH MULTI- PERSPECTIVE INSTRUCTION-AS-REASONNI
Liangyu Chen Hanzhang Zhou², Chenglin Jianan Zhang', Panrong Tong² Xu Zhang'. Chen Liu", Quyu Kong'. Yuqi Liu', Wenxwan Wang' Yue W Qin J HO
Renmmn University of China yue. w@al ieaiibaaiinn com, qjin@ruc edu.en
Tongyi Lab, Alibaba Group CCUH
ABSTRACT
GUI grounding, which maps natural-language instructions to actionable UI el- ements, is core capability of GUI agents. Prior works largely treats instruc- tions as a static proxy for user intent, overlooking the impact of instruction di- versity and quality on grounding performance. Through careful investigation of existing grounding datasets, find 23.3% flaw rate in their instructions and show that inference-tims exploitation of instruction diversity yields up to sub- stantial 76% relative performance improvement. In this paper, we introduce the paradigm, treating instructions as dynamic analyti- cal pathways that offer distinet perspectives and enabling the model select the most effective pathway during reasoning. To achieve this, we propose a two-stage training framework: supervised fine-tuning on synthesized, diverse instructions to instill multi munti-perspeettiee reasoning, followed by reinforcement learning to op- timize pathway selection and composition. Our resulting models, UI-Ins-7B and UI-Ins-32B, achieve state-of-the-ard results on five challenging grounding bench- marks and exhibit emergent reasoning, selectively composing and synthesizing novel instruction pathways at inference. In particular, UI-Ins-32B attains the best grounding accuracy, scoring 87.3% on Ul-12E-Bench, 57.0% on ScreenSpot-Pee Pro. Pro. and 84.9% on MMBench-GUII L2. Furthermore, our model demonstrates strong agentic potential, achieving 74.1% % success rate on AndroidWorld using UI- Ins-7B as the executor. Our in-depth analysis reveals additional insights such how reasoning can be formulated to enhance rather than hinder grounding perfor- mance, and how our method mitigates policy collapse in the SFT+RL framework. All code and models are released.
sostructionassReasommi
INTRODUCTION
Automated agents for graphical user interfaces (GUls) are an important frontier in the pursuit of artificial general intelligence (AGI) (Wang et al. 2024). Their effectiveness is dependent on GUI grounding, i.e., the task of mapping a natural language instruction to the corresponding actionable UI element in a screenshot or live interface.
The natural language instruction is central to GUI grounding: is primary input alongside the GUI screenshot and translates high-level user intent into low-level, executable actions. Consequently, the clarity and precision of instruction directly impact grounding success. However, the impact of grounding instruction has been largely overlooked in prior works. In this paper, we provide a com- prehensive analysis covering instruction diversity, quality, and algorithmic strategies, and establish a concrete basis for more effective GUI grounding.
We focus on instruction diversity and reveal fundamental mismatch: humans flexibly choose the most effective pathway among multiple instructional perspectives, whereas current models are trained in narrow, fixed style. For example, to express single intent such as "close window".
Work was done during internship at Tongyi Lab, Alibaba Group. Corresponding authors.

---

Published as a conference paper at ICLR 2026
MMBench-GUI L2
Har Uniin Uli GTAI Internnii Infouu
83.1
MMIBench-GUI 1.2 Accuarcy
81.5
UI-12 Bencs Accuarcy
20.8
V
UH2E
77.4.
78.5
16
12k
16 Parameter Size
128
S5.Pro
Showdown
Parumeter (B3
Figure 1: Performance comparisons of UI-Ins and other state-of-the-art methods.
humans may describe the corresponding UI element as its appearance ("click the red X"), function ("close the file manager"), spatial location ('the button in the top-right corner"), high-level in- tent ("get rid of this screen"). Humans strategically switch among these perspectives, choosing the most effective description for the task at hand, as illustrated Fig. El Our quantitative analysis in Sec. [2] likewise shows that leveraging instruction diversity is key to improving grounding accuracy. However, prevailing GUT grounding models are typically trained map single instruction style to an action, with limited capacity to reason across different perspectives. This limitation forms key bottleneck to flexible adaptability and robust interpretation of GUI grounding tasks.
Those insights motivate paradigm shift: rather than treating instructions as static inputs, we can regard them as dynamic reasoning pathways. Different instruction types are not merely alternative phrasings; they encode distinct analytical angles for identifying UI element. An intelligent GUI agent should not only understand a command but also actively select the most effective reasoning process to infer the user's intent. We term this new paradigm
eerruction-as-Raasmme
Beyond this conceptual shift, we also find pervasive instruction quality issues in grounding datasets. Specifically, manually inspected 1,909 data entries sampled from some popular datasets, in- cluding OS-Atlas (Wu et al. 20244)), Widget Captioning (Li et al., 2020), and AMEX (Chai et al.) 2025). As shown in Fig. 2b we found that a notable 23.3% samples contained various quality flaws, introducing considerable noise that could adversely affect model training.
Based on these findings, we introduce a simple yet effective framework. We propose data pipeline systematically cleans noisy annotations and, crucially, augments existing data with a rich diversity of instruction styles, creating dataset curated specifically for ing. With this high-quality data as our foundation, we then propose our Instruction-as as Reasoning framework. This novel two-stage training paradigm first uses Supervised Fine- Tne-Tuning (SFT) to explicitly teach the model to use diverse instruction perspectives employs Group Relative Policy Optimization (GRPO) (Guo el al. 2025 Shao et al., 2024) in the Reinforcement Learning (RL) stage, enabling the model to learn how choose the optimal instruc- tion perspective as the reasoning pathway for any given situation. Building on this framework, we introduce the UI-Ins-BB and U1-Ins-32B models. Empirical evaluations conducted across multiple distinct benchmarks validate the strength of our approach, as illustrated in Fig. [
instruction reason- -aaruuttionn--RRasoninnn
reasoning pathways. Then, it
To provide additional insights for grounding, conduct in-depth analysis of our Instruction- as- as-Reasoning from multiple perspectives. First, how can reasoning be formulated to enhance, rather than hinder grounding? Consistent with prior works (Lu et al. 2025 Tang et al.. 2025), we confirm that a free-form reasoning approach often degrades model performance during GRPO.
In contrast, experimental results indicate that our proposed Instruction-as --aa--s-nn- consistently enhances performance by a large margin across various base models, establishing it as a highly ef- fective reasoning paradigm for grounding. Second, how can we mitigate policy collapse in the SFT+RL framework? We identified that models fine-tuned via SFT using only coordinates as ground truths often exhibit highly uniform responses, leading to ineffective exploration and policy collapse in RL. This is also noted by Phi-Ground (Zhang et al. 2025). However, our Instruction- as- Reasoning framework mitigates this issue by instilling diverse exploratory capabilities after SFT, enabling the model to generate diverse rollouts during RL and thereby avoid policy collapse. Finally, is UI-Ins's reasoning capability limited to predefined perspectives seen during training? Inter- estingly, we observed that after training with Cattuuctiionn---eaoooiiii the model not only learns to
2

---

Published as a conference paper at ICLR 2026
Ambiguous match:
Data
Cleaned Data
26-1
Intent
71.0
69:2
70
21.3
Location
MisMatch:
60
18.9
29.8
Functionality
40
35.2
Appearance
50
Combination
Original
40
60
Ul-28 Bench
MMitench-Gll
(a)
(c)
Figure 2: Preliminary analysis of GUI Grounding Instructions. (a) Instruction diversity influences performance significantly. (b) Instruction quality problems in existing open-source datasets. (c)
Low instruction quality undermines training efficacy.
select the optimal reasoning pathway but also develops emergent capabilities to combine different reasoning perspectives and to reason from novel instruction perspectives not seen in training.
In summary, our contributions are as follows:
Systematic Investigation into GUI Grounding Instructions. We conduct a systematic analysis of instructions in GUT grounding. revealing two crucial insights: (1) striking 23.3% % of samples' instructions major datasets are flawed, and (2) there massive potential improvement in leveraging instruction diversity, which can unlock up to substantial 76% relative performance gain even without training.
Instruction-as-Reet Reasonnig Paradigm. Building insights above, propose the Instruction-as- Reasoning paradigm, which reframes instructions from static inputs to dynamic reasoning path- ways. We realize this through SFT+GRPO training framework that first teaches the model to use diverse instruction perspectives as reasoning pathways and then incentivizes it to select the optimal analytical reasoning pathway for any given GUI scenario.
SOTA Performance Across Diverse Benchmarks. Our UI-Ins-BB and UI-Ins-32B establish SOTA performance across five most well-known grounding benchmarks. Notably, UI-Ins-32B achieves 87.3% on U1-12E-Bench, 5 ScreenSpot-Pro, and 84.9% % MMBench-GUT L2, significantly surpassing its strongest counterparts. Moreover, our superior grounding capability leads to strong online agent performance on AndroidWorld when combined with GPT-5 as the planner, yielding 74.1 % success rate.
In-depth Analysis. Our analysis provides additional insights for grounding. how reasoning can be formulated to augment rather than hinder performance and how method mitigates policy collapse in the SFT+RL framework. Furthermore, we reveal that our approach unlocks emergent reasoning capabilities, allowing the model to reason from novel perspectives.
demonstrate
2
How MUCH Do INSTRUCTIONS REALLY MATTER?
The natural language instruction is primary input to grounding tasks, serving as the sole carrier of high-level intent in GUI grounding. But what extent the key aspects an instruction's for- mulation, namely its analytical perspective and its correctness, truly impact model's performance? Prior works have largely treated the instruction as a simple input string, leaving its impact underex- plored, We highlight that the instruction is central, understudied variable in GUI grounding. To probe this view, we conduct a preliminary analysis guided by two foundational research questions:
RQ1: How does the diversity of instructional perspectives affect grounding accuracy?
RQ2: What is the state of instruction quality in GUI grounding datasets, and what is its impact?
2.1 DOES INSTRUCTION DIVERSITY UNLOCK HIGHER PERFORMANCE?
Humans instinctively choose the most effective way to describe an object based on the context like
Fig. Does providing model with similarly diverse, perspective-rich instructions unlock better

---

Published as a conference paper at ICLR 2026
performance? To investigate this, conducted controlled experiment the ScreenSpot- Pro benchmark. We systematically rewrote its original instructions to reflect four distinct perspectives: Appearance, Functionality, Location, and Intent. We then evaluated the zero-shot performance of Qwen2.5-VL-77 on each instruction set.
The results, shown in Fig. reveal two critical insights. First, instruction diversity matters signifi- cantly. Instructions from perspectives of appearance, function, and intent all substantially outperform the original instructions. This demonstrates that even without retraining. simply providing diverse instruc- tion perspectives can unlock significant latent capa- bilities within the model. Second, the ability to select the most appropriate instruction perspective leads to higher performance ceiling. The "Combined" bar, representing the performance ways pick the best- pest-performing perspective for each sample, achieves a relative improvement 76%, far surpassing any single instruction perspective.
*
t
access
sovEv
Apslioatsm
Appearance:
looks
Functionality: Close manager window.
like picture.
model could al-
Location: Edit line after the addFault function.
Intent: Mute all the system sound.
Overall, these results reveal considerable untapped potential in leveraging instruction diversity, both by introducing multiple instruction perspectives and by selecting the optimal perspective per instance. This motivates our algorithm that learns to leverage di- verse instruction perspectives as reasoning and dy- namically chooses the best analytical angle.
Figure 3: Effective instruction perspectives in different GUI scenarios. Samples are from OS-Atlas Dataset and the ground truth bounding box is labeled with a green box be- side yellow star.
2.2 CAN WE TRUST EXISTING DATASETS FOR INSTRUCTION QUALITY?
While utilizing instruction diversity is promising, its effectiveness rests a foundation that the orig- inal instructions are correct. But is this founda- tion valid? To probe the instruction quality of the grounding datasets, we conducted a large-scale man- ual analysis. Specifically, ples from three prominent datasets, OS-Atlas (Wu et al. (2024a), AMEX (Chai et al. 2025). and Wid- get Captioning (Li et al. 2020).
Ambigunus Instruction
Instruction siide
<heck people
interacted with the
talk pload peojary
examined 1, 1,9 sam-
TROKIING
denend berder this sffre
Our analysis reveals pervasive instruction quality is- sues. As shown in Fig. 22b] 23.3% of instructions exhibit substantive flaws, including ambiguity or re- ferring to nothing shown in Fig. 4 To further quan- tify the impact of such flaws, we trained the same model on the original dataset and on a cleaned ver- sion. Experimental results are depicted in Fig. [c models trained on cleaned data achieve substantial and consistent performance gains across multiple benchmarks. In other words, flawed instruction data can significantly degrade downstream performance when used for training.
O
Bloom: Real Life, fee Videors wethh wove
Figure 4: Instruction quality flaws in ground- ing datasets. Left: Ambiguous match, an in- struction maps to multi UI elements. Right: Mismatch, no valid UI element matches the instruction.
These findings indicate that existing datasets suffer from instruction quality problems that actively harm model performance. Consequently, data cleaning is not optional niceties but necessary pre- requisites for meaningful training, especially when our goal is to teach models to leverage diverse instruction perspectives as reasoning.
4

---

Published as a conference paper at ICLR 2026
Input Bbox
Preprocessdd Boox
Generated Instructions
Expand
Appearance
Location
Refinement
Origin
Functionality
Intent
Input Instruction
Omniparser detect
Detected Bboxes
Figure 5: Overview of our data pipeline. The pipeline first preprocesses the ground-truth box, then leverages GPT-4.1 to generate instructions from diverse perspectives, and finally filter results by ensuring a precise alignment between the instruction and the ground truth box in verification stage.
3
METHOD
Our methodology is architected address the two fundamental challenges identified in Sec. 2 the data quality issues and the untapped potential of instruction diversity. We first introduce high- quality data pipeline designed to establish the necessary preconditions for effective model training. With this robust data foundation, as-Reasoning. a two-stage training framework that empowers models to use diverse instructions as reasoning pathways and select the optimal analytical perspective during reasoning.
then present
algorithmic contribution, Instruction-
3.1 TASK DEFINITION
GUT Grounding aims to localize a single UI element corresponding to natural language instruction on a graphical user interface (Wang et al. 2024). Formally, given screenshot and an instruction
I, the model f should predict point (Tp: Up) that indicates the target element's location.
3.2 DATA PIPELINE FOR MULTI-PERSPECTIVE REASONING
Our preliminary analysis (Sec. revealed that data quality is a prerequisite for meaningful training (.2c and that instruction diversity unlocks significant performance gains (Sec. [2.].. To this end, we developed a data processing pipeline focused on two primary objectives: establishing a clean data foundation and then systematically augmenting with diverse, pealliiperseectiee instructions.
Pre-processing. To rectify the pervasive annotation noise found in existing datasets, we first perform a lightweight pre-processing step. We use OmniParser V2 (Lu et al. (024) detect all UI elements on screenshot and apply simple loU-based method to refine or filter the original ground truth bounding box. This ensures each instruction is associated with a reliable spatial anchor, and the flawedinstructions are filtered at the same time. The pre-processing forms the clean foundation necessary for the subsequent augmentation.
Multi-Perspeecee Instruction Augmentation. The core of our pipeline focuses on enriching in- struction diversity. We leverage GPT-4.1 (OpenAl, (2025)) to generate new instructions from four fundamental analytical perspectives identified in analysis: appearance, functionality, lo- cation, and intent. For each data instance, the model receives the screenshot with the highlighted target element and is prompted to create set of high-quality, diverse phrasings. To mitigate L.M hallucinations and ensure a strict one-to-one mapping, each generated instruction undergoes a verifi- cation step where GPT-4.1 confirms it unambiguously refers only to the target element. This process yields a high-quality, multi-perspective corpus specifically curated to teach complex reasoning.
3.3 INSTRUCTION-AS -EAA--SNI--
With such a multi-perspective dataset at hand, we introduce the framework to use it. As discussed in Sec. 2. leveraging diverse instruction perspectives and dynamically choosing the best analytical angle are key to unlock superior grounding performance. As shown in Fig. 6 our Instruction-as- Reasoning framework is two-stage training approach that instills this capability: (i) SFT stage that teaches the model to use multi- mulliipersppectve instructions as explicit reasoning pathways, and (ii) RL stage that trains the model the optimal analytical angle for each sample.
5

---

Published as a conference paper at ICLR 2026
Learning Diverse Reasoning Paths
Learning Select Optimal Perspective
System
System
Point-rr-bo Reward
Yoa scciase 'ismraciic
ad this
Appraceede
Fanctid
Reward
if in Iox. 0 pclllers
Imput Image
Inpul Image
Uierr
User
Vilion
Vaiion
Extraction
Resvardic
Language Model
Language
Rollouts
Advanaages
Objeative
Augmerted
rinebrint eshos the cntrection inom apovenured
Instruction
Cntruction-aa-Reesome
Instrction Rezsoning fornax
coortnate
Center
GT Bbox
Figure 6: Overview of Instruction-as Instruction-asRReasonnng We leverage diverse instructions as explicit rea- soning pathways to teach model multi- celtipperspective reasoning paths in SFT stage, then let model explore unconstrained perspectives to find the optimal ways different scenarios.
3.3.1 SFT STAGE: LEARNING TO GENERATE DIVERSE REASONING
The goal of the SFT stage is to explicitly instill the model with the ability to perform Instruction- as-Reasoning: utilizing diverse instruction perspectives as analytical reasoning before predicting the grounding coordinate point. Concretely, the model first generates an intermediate reasoning text, i.e., a rewritten instruction from one instruction perspective, which serves as an actionable reasoning pathway (Fig. (). Then outputs the final coordinate point.
Given the grounding model with parameters the training objective in SFT stage is maximize the log-likelihood of the target sequence Y across the entire dataset D, formally expressed as:
max
log P(Y, gr (S, I;0), where Y, Rg @ Pyt
(S.LY ED
In this formulation, denotes sequence concatenation. The ground truth reasoning text, R.g., is randomly sampled from one of the valid augmented instruction perspectives, while Pyt represents the ground-truth coordinate point. An example of SFT prompt and answer is in See ??. This unified objective elegantly compels the model to co-optimize two distinct but related skills:
Reasoning Generation: Learning produce reasoning (Rgt) in instruction perspective.
Grounded Prediction: Learning to predict the correct coordinate point (Pys) conditioned on both inputs and its self eefergenerteed reasoning.
By fine-tuning on this objective, the model learns to reason from diverse instruction perspectives, building a foundational for RL stage training.
3.3.2 RL STAGE: LEARNING TO SELECT THE OPTIMAL PERSPECTIVE
The SFT stage equips the model with the ability reason from multiple perspectives. However, it does not teach model which reasoning pathway better. To transcend this limitation and incentivize the model dynamically select the most effective analytical perspective, we introduce an RL stage.
The goal of this stage is to fine-tune the SFT-trained model to discover and select reasoning strategies that maximize grounding accuracy. To achieve this, we employ Group Relative Policy Optimization (GRPO) (Guo et al. 2025). In this phase, we modify the prompt to simply ask the model to before answering, without providing the explicit list of predefined perspectives (appearance, func- tion. etc.). This open peen-ended instruction encourages the model to explore a wider space of reasoning patterns, including synthesizing multiple perspectives or even formulating entirely novel ones. The model then learns to select the optimal analytical perspective from the feedback of RI. rewards.
We calculate rewards by point-in-box bo- function, then, the rewards {r:)g are normalized by:
à Σe
(2)
Li=1 (r: EE ri

---

Published as a conference paper at ICLR 2026
Table I: Overall performance on MMBench-GUl L2 and U1-I2E-Bench benchmarks, We use 1-'
to denote unavailability, and .** to denote the results evaluated by us.
MMBench-GUlI L2
Ul-12E-Bench
Model
Size
Basic
Advanced
Avg.
Explicit
Implicit
Avg.
Qwen2.5 VL
Bai et al.
2025
7B
38.0
29.8
33.9
58.4
51.0
53.8
OS-Atlas (W 2024a
7B
52.8
30.1
41.4
63.2
55.8
58.6
SEGUI(todo) XXu et al. 2025
7B
51.0
40.5
45.7
61.1
53.2
Uground-Vi Gou al.
2025
7B
78.4
53.0
65.7
81.3
63.6
70.3
UI-TARS-1.5 Seed
2025
7B
78.4
50.4
64.3
81.3
68.2
73.2
UI-TARS QQn et 2025
7B
71.4
55.3
61.4
GUI-Actor [w
2025
7B
85.7"
67.5*
76.5"
71.6"
66.1*
68.2"
GUI-G2 Tang el al.
2025
86.8
70.9*
78.8"
82.1
67.5*
73.1"
InfiGUI-GI [Luu al.
2025d)
7B
88.5
73.2
80.8
85.0
72.7
77.4
GTAI Yang el al.
2025
7B
84.4"
72.6*
78.5
87.0"
72.8*
78.2"
GTA1 Yang ct al.
2025
32B
89.0"
77.9
83.4"
91.4"
78.7"
83.5"
Qwen2.5.VL Bai al.
2025
32B
80.6"
63.8"
72.1
73.8"
62.7*
66.9
Qwen2.5-VL Bai ci al. 2025
72B
79.7"
66.7"
73.2"
71.3*
61.2"
65.0"
Uground-V1 Gou et al. 2025
72B
84.5
71.3
76.3
UI-TARS QQn et 2025
72B
80.9
69.4
73.7
UI-Ins-7B
7B
89.0
77.3
833
88.9
76.3
81.1
UI-Ins-32B
32B
90.5
79.4
84.9
92.9
83.9
87.3
where the rollout number. Finally, the model is optimized by minimizing the objective:
T(; 1.
Fotalo, I, AA,.
(3)
where (- denotes the old policy and A.,, is the advantage associated with prediction o,. By iteratively applying this process, the model learns to prioritize reasoning pathways that consis- tently lead to cothe rrect coordinate point, effectively learning an optimal, context-dependent strat- egy for instruction perspective selection. Interestingly, we find that the model also learns to combine multiple perspectives and even formulate entirely novel reasoning perspectives (Sec. 4.5
4
EXPERIMENT AND RESULTS
4.1 EXPERIMENTAL SETTINGS
We source data from several public datasets, including OSAtlas (Wu et al. 2024b), Omniact (Kapoor ct al. 2024), Android Control (Li ct 2024). AMEX (Chai al. and AgentNet (Wang al. 2025b), covering diverse operating systems such as Windows, MacOS, Linux, and Android. All data subsequently processed through our pipeline ensure quality. We employ Owen2.5-VL-77 and Qwen2.5-VL-2 as our backbone architectures. For a comprehensive evaluation, we use multiple grounding benchmarks and online agent benchmarks. More details are in Appendix C.1
4.2 GROUNDING RESULTS
As demonstrated in Tab. UI-Ins-32B achieve SOTA performance on MMBench-GUI L2 and UI-I2E Bench which emphasize complex instruction understanding and outperforms against strong baselines with varying instruction complexity. On MMBench L2, our model shows progres- sively larger gains on more difficult tasks. For instance, UI-Ins-7B surpasses Qwen2.5-VL-7B by 134.2% on the "Basic" subset, and this margin increases to 159.4% on the more challenging 'Ad- vanced' subset. This pattern holds on U1-Ins-32B, where UI-Ins-32B's advantage over Qwen2.5- VL-32B grows from 12.3% ('Basic") observed on the 11-22-BBench When compared to GTAI, UI-Ins-32B's performance gain expands from 1.6% 'explicit' subset to of greater improvement on the 'Advanced' and 'implicit' subsets demonstrates that our approach successfully equips the model with enhanced robustness for difficult scenarios.
much larger 24.5% ('Advanced"). A similar trend is
substantial 6.6% on 'implicit' subset. The consistent trend
Furthermore, to provide broader validation our models' capabilities, we conduct extensive eval- uations the ScreenSpot- V2, ScreenSpot-Pro, and Showdown benchmarks. As detailed in Tab.

---

Published as a conference paper at ICLR 2026
Table 2: Performance comparison on ScreenSpot-Pro, ScreenSpot-Ve and ShowDown.
ScreenSpot-Pro
ScreenSpot-V2
ShowDown
Model
Size
Text
Icon
Avg.
Text
Icon
Avg.
Avg.
Qwen2.5- VL (Bai ct al.
2025
7B
38.9
7.1
26.8
94.2
81.8
88.8
43.6"
OS-Atlas Wu ct al.
2024
7B
28.1
4.0
18.9
92.5
73.3
85.1
41.1
UI-TARS Qun
7B
46.0
16.0
35.7
95.4
86.6
91.6
66.1
UI-TARS-
2025
7B
42.0
92.9
83.3
67.2
GUI-G Tang al.
2025
7B
64.9
18.4
47.5
96.1
89.7
93.3
70.4
UGround-v [Gou ct 2025
7B
45.2
8.1
31.1
88.1
86.8
87.7
57.8
InfiGUI-GI (Liu et al. 2025d)
7B
69.1
24.5
51.9
97.4
88.4
93.5
68.2
GTA1 Yang ct ali (2025)
7B
65.5
25.2
50.1
95.7
88.1
92.4
67.9"
GUI-Actor (Wu el al. 22225
7B
58.8
20.7*
44.2
96.0
87.0
92.1
64.6"
SE-GUI Yuuan al. 20225)
7B
61.8
22.8
43.2
90.3
GTAI Yang
(2025)
32B
65.6
28.1
53.6
97.1
88.3
93.2
71.1"
Qwen2.5 VL Bar
(205
32B
62.6
48.0
88.3
71.1"
GUI-Owl Yang cl al 2025)
32B
65.6
28.1
53.6
97.1
88.3
93.2
71.1"
UGround-va Yang el a. (2025)
72B
49,0
11.1
34.5
97.1
88.3
93.2
71.1"
Qwen2.5-VI, Bar
2025)
72B
69.4
27.3
53.3
97.1
88.3
93.2
62.3"
UI-Tars Yang 2025
72B
65.6
28.1
53.6
97.1
88.3
93.2
71.1"
UI-Ins-7B
7B
70.0
23.5
52.2
98.2
88.6
94.0
73.1
Ul-Ins-32B
32B
73.7
30.0
57.0
98.2
90.6
94.9
73.8
Ul-Ins-32B again achieves SOTA performance, and UI-Ins-BB consistently outperforms other mod- els of a similar scale. We also provide an error analysis and qualitive analysis in Appendix
4.3 ONLINE AGENT RESULTS
To rigorously evaluate the stability and reliabil- ity of grounding models in realistic settings, we employ Ul-Ins-7B the grounding executor under a GPT-5 (OpenAI 2025b) planner in An- droidWorld (Rawles al.) (024)) online bench- mark, where each action must be grounded and executed in dynamic environment.
Table 3: SOTA Performance on AndroidWorld. Our framework achieves this result by using our model as a grounding executor under GPT-5 planner, surpassing strong baselines including UI- TARS-2 and the Gemini 2.5 Computer Use.
Model
S.R.
OpenAl CUA-03 (OpenAl 2025
52.5
As suggested in Tab. 3 despite our simple ar- chitecture without extra knowledge guidance in our designed prompts as shown in Ap- pendix C.2 74.1% task success rate, outperforming strong closed-source models such Gemini 2.5 Com- puter Use (DeepMind, 2025) and UI-TARS- (Wang al. 2025a). This result demon- strates that UI-Ins provides precise and stable visual grounding, maintaining semantic align-
Gemini 2.5 Computer Use DeepMind
2025
69.7
UI-TARS-2 (Zhou et al.
2024
73.3
InfiGUIAgent (Liu
2025b
9.0
framework still achieves
Uground (Gou et al. 2025
44.0
Aria-UI Yang et al. AgentS2 (hou al.
2024b
44.8
2024
54.3
JT-GUIAgentV22 (China Mobile 2025
67.2
Qwen2.5-VL-78 (GPT-5 planner) UI-Ins-7B (GPT-5 as planner)
50.0
74.1
ment and action reliability across diverse app layouts and dynamic interface updates.
Additionally, UI-Ins-BB grounding executor achieves a substantial 24.1% performance improvement over its base model (Qwen2.5-V) --- in same configuration. This demonstrates that enhanced grounding capability can effectively translate into improved performance on online agent tasks.
4.4 ABLATION STUDY
Data Pipeline Ablation Study As shown in Fig. 7a we manually inspected 1,542 samples gener- ated by our data processing pipeline and found an error below 8%. This represents significant reduction from the 23.3% error rate observed in the original data. To further validate the effective- ness of our data pipeline, we conduct an ablation study using SFT on 210k origin samples, which corresponds to 180k cleaned samples. As shown in Fig. 7b our data pipeline provides a consistent performance improvement across multiple benchmarks.
8

---

Published as a conference paper at ICLR 2026
Mismatch: 5.3
Ambiguous
95
Accuracy Benchmarks 1
90
Origin
Cleaned
68.1
90.2
85
75
72.3
74.3
Precise Match 93.5
66.3
65
63.5
60
MMBBnnch-GUI
UI-12E
SS.V2
(a)
(b)
Figure 7: (a) Instruction quality distribution after data processing pipeline. (b) Performance com-
parison between Qwen2.5-VL-78 training with origin data and cleaned data by processing pipeline.
Training Stage Ablation Study To validate the necessity of SFT+RL training stages for method. We com-
Table 4:
Ablation study on training stages. We report accuracy on MMBench-GUI L2 (MM), UI-I2E-Bennd (I2E), Showdown (Show), ScreenSpot-Pee Pro (Pro), and ScreenSpot- V2 (V2).
our Instruction-as-Reasonn pare the UI-Ins-BB against two variants: one trained only with SFT and another trained only with RL. In all settings, the model is prompted generate intermediate reasoning process. The results Tab. 4 indicate that both stages critical for achieving optimal performance.
SFT
RI
MM
12E
Show
Pro
V2
x
X
63.4
56.0
43.6
24.4
86.5
x
V
72.4
69.2
66.6
37.0
88.6
x
76.3
70.1
67.5
37.1
90.6
83.1
81.1
73.1
52.2
94.0
The absence of either stage leads to an accu- racy degradation, highlighting the importance of first teaching the model generate reasoning from diverse perspectives and then allowing it to optimize the selection of the optimal reasoning pathway.
4.5 DEEPER INSIGHTS INTO INSTRUCTION REASOONIN
Having established the strong performance of Un-Ins, we now delve deep into the Instruction- as-Reasoning to understand its effectiveness. We investigate several central questions below:
Table 5: Ablation the intermediate reasoning component. Its removal results in a significant performance degradation across all benchmarks. V represents let the model use Instruction as Rea- soning in the corresponding stage.
an intermediate reasoning step necessary? A fundamental question is whether generating intermediate reasoning is essential. To answer this, we conducted an ablation study by com- pletely removing the reasoning generation from both the SFT and RL stages, training the model to predict coordinates directly. Experimental results depicted in Tab. 5 Compared to method (the 4th row), removing reasoning (the first row) leads to a substantial performance drop across all benchmarks. This result confirms that the intermediate reasoning crucial to the success of our approach.
SFT
RL
MM
12E
Show
Pro
V2
X
X
79.1
70.7
66.1
44.8
91.7
X
V
78.8
71.6
68.4
48.0
92.0
X
81.6
76.2
72.0
47.5
93.1
83.1
81.1
52.2
(IR) vs. Free- Form Reasoning (FFR). Given that reasoning critical, what kind of reasoning is effective? Prior works (Lu al. 2025 Yang et al. 2025) Zhou et al., 2025 Tang et al. 2025) have shown that FFR difficult to optimize and can even degrade performance. We test this hypothesis against IR in Tab, As shown in the top section of the table, applying FER degrades the perfor- mance both UI-Tars- T1-Tars--155-77 1.5-7B and Qwen2.5- VL-7B, confirming prior findings. For instance, causes a 6.4% relative drop in SS. for UI-Tars-1.5-7B. In contrast, the bottom section
Crrrrcctionas-RReasonnn
Table 6: Comparison between FFR and IR in RL. Our instruction-a-RReasonni is the key to un- locking effective reasoning for GUI grounding.
Method
Base Model
SS.Pro
RL (w/o FFR)
UJ-Tars-115577
50.1
RL (w/ FFR)
U1-Tars- 1.5-7B
46.9. (6.4)%
RL (w/o FFR)
Qwen2.5-VL-7B
36.4
RL (w/ FFR)
Qwen2.5-VL--7B
36.44 (0)%
RL (w/o IR)
U1-Tars-1.5-7B
48.7
RL (w/ IR)
Ul-Tars-1.5-7B.
51.21 (5.1%)
(w/o IR)
Qwen2.5-VI-7B
RL (w/ IR)
Qwen2.5-VL-7B
52.21 (9.9%)

---

Published as a conference paper at ICLR 2026
shows that training models with IR yields significant increases in accuracy. We can thus conclude from the experiments that unstructured FFR fails to improve, whereas IR is the key to unlock effec- tive reasoning for GUI grounding.
The Hidden Benefit: Stabilizing SFT+RL. A critical challenge in SFT+RL training for grounding is the policy collapse issue during RL. We compare our SFT+RL framework with a standard one in this ablation. The standard SFT training provides a poor policy initializa- tion, often causing the model's performance to degrade during RI., as evidenced the upper part of Tab. 7 In contrast, our instruction- as-reasoning based SFT acts as a powerful ex- ploratory warm-up. By pre-training the model to generate diverse reasoning pathways, we em- power it with strong exploratory capability, achieving a significant performance increase during RL. This demonstrates that our SFT stage only teaches the reasoning format, but also enables effective and stable policy optimization in the RL. phase.
Table Instruction-an sarruction-as-Reasoninng prevents policy collapse in RI. and achieves significant accuracy gain in RL. This table contrasts our method with standard SFT+RL pipeline. Scores after 100 RL steps are reported.
Method
Base Model
SS.Pro
SFT (w/ IR)
Qwen2.5-VLL
-LLBBB
37.0
RL
Owen2.5-VL-70
34.91 (5.7%)
Zero-Shot
JEDI-7B
39.5
RL
JEDI-7B
34.5. (12.7%)
SFT (w/ IR)
Qwen2.5-VL-78
37.1
+ RL
(wen2.5-VL-78
46.01 (24.0%)
Emergent Capabilities: Reasoning Beyond Predefined Perspectives. Does our framework merely teach the model to use the four predefined perspectives? A qualitative analysis of model responses on the UI-I2E Bench (Appendix D.2) reveals that it learns to reason at much deeper level. We summarize three key emergent capabilities as following:
Strategic Selection: The model learns to strate- gically select different reasoning perspectives for different scenarios after RL. As shown in Fig. 9b and top section in Fig. 8 diverse and accurate instruction perspectives are selected.
Compositional Integration: The model often combines multiple perspectives into single, cohesive reasoning, as shown in middle section in Fig. 8 All 1477 samples of UI-12E Bench contain 5245 reasoning ways in total, as shown in Fig. 9a This synthesis is not explicitly taught but emerges as an effective reasoning strategy during RL.
Single Perspective Reasoning
Click the button to center the text (Functionelity]
Combination Perspectives Reasoning
Click the button with the icon of centered horizontal lines [Appearance] to center the text (Functionality)
Combination Perspectives and Emergent Perspective Reasoning
Emergent Perspective: Most impressively, as shown in Fig. 9b), the model is capable of gen- crating entirely new analytical angles beyond the four trained perspectives, such as reasoning from the perspective of group affiliation UI element state, as demonstrated in Fig.
To the right of the currently highlighted option [Location], , click the inactive button [State), identifiable by its icon of centrally. aligned staggered lines [Appearance), to center the text [functionaaltt]] which will set as the new exclusive active state [Prediction] in alignment control group [Group Affllstioon.
Figure Different reasoning capabilities of UI-Ins.
CONCLUSION
In this work, we investigate GUI grounding instructions, revealing a 23.3% dataset flaw rate and finding that exploiting instruction diversity yields up to a 76% performance boost. We thus propose Instruction as Reasoning, novel SFT and RL framework that treats diverse perspectives as distinct reasoning pathways. Our models, UI-Ins-7B and UI-Ins-32B, set new state-of-the-an performance across five grounding benchmarks. Specifically, UI-Ins-322 scores 87.3% % on UI-I2E Bench, 57.0% on ScreenSpot-Pree Pro, and 84.9% MMBench-GUII L2. Furthermore, UI-Ins-7B demonstrates strong agentic potential, achieving a 74.1% % execution success rate on AndroidWorld Our in-depth analysis further reveals helpful insights for GUI grounding.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Anthropic.
Claude
3.7 sonnet.
https://wy.anthrppic.cooo
claude- 3- 7- s-snnett 2024. Accessed: 2025-08-02.
Anthropic. Our 3.5 models and computer use.
3 5
s -allss-aanncccocoe-eeaooe
sep 2024. Accessed: 2025-09-22.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, al. Qwen2. 5-v1 technical report. arXiv preprint wrXiv:2502.3992. 2025.
Yuxiang Chai, Siyuan Huang, Yazhe Niu, Han Xiao, Liang Liu, Guozhi Wang, Dingyu Zhang, Shuai Ren, and Hongsheng Li. Amex: Android multi-annotation expo dataset for mobile gui agents. In Findings of the Association for Computational Linguistics: ACL 2025, pp. 2138-21 2156. Association for Computational Linguistics, 2025. doi: 10. 10.11835/v1/2055 www.:o25...nninnaaa........ URL
http:///kk..o..orrr/0111116111
findings acl. 110
China Mobile. JT-GUIAgent-Vis V1: A Planner- Agent for Reliable GUI Interaction. Project Website, 2025. URL
quiagent.githie
io/
DeepMind. Developing a computer use model. Google Blog, Oct 2025, URL URtattt///lt google/technol htwpchnnnoooyooooooooooooooooo t /twp///eeeeemmmm///gggeeeeeee/ee http://wweeee.llooooooooooooooor Ac-
cessed: October 22, 2025.
Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Su. Navigating the digital world as humans do: Universal visual grounding for GUI agents. In The Thirleenth International Conference on Learning Representations, 2025. URL https: // //openreview wwvvie.w.... net. forum? www.mmmddkkoaaiisss...
Zhangxuan Gu, Zhengwen Zeng, Zhenyu Xu, Xingran Zhou, Shuheng Shen, Yunfei Liu, Beitong Zhou, Changhua Meng, Tianyu Xia, Weizhi Chen, Yue Wen, Jingya Dou, Fei Tang, Jinzhen Lin, Yulin Liu, Zhenlin Guo, Yichen Gong, Heng Jia, Changlong Gao, Yuan Guo, Yong Deng, Zhenyu Guo, Liang Chen, and Weiqiang Wang. Ui-venus technical report: Building reg--performance ui agents with rft, 2025. kttts:////ax:....ommmm http///wwv..o//b///55/0000
abs 110.508881113333
Daya Guo, Dejian Yang. Haowei Zhang. Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in 1lms via reinforcement learning. arXiv preprint arXiv:2501.12945 2025.
Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxuan Zhang. Juanzi Li, Bin Xu, Yuxiao Dong, Ming Ding, and Jie Tang. Cogagent: A visual language model for gui agents, 2024. URL httth::t//aaat
org/ abs/
08914
ZongHan Hsieh, Tzer-Jen Wei, and ShengJing Yang. Zonui-3b: A lightweight vision-language model for cross-resolution gui grounding. https: / http//avvvv.. attp:::/arxi..oo//abbb..ss/ htt.v/.orr//aab/55.2224444 2025. 234449 [cs.CV], version 2, last revised Jul 2025.
Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Zhe Xu, Yao Hu, and Shaohui Lin. Vision-rl: Incentivizing reasoning capability in multimodal large language models, 2025.
Raghav Kapoor, Yash Parag Butala, Melisa Russak, Jing Yu Koh, Kiran Kamble, Waseem Alshikh, and Ruslan Salakhutdinov. Omniact: dataset and benchmark for enabling multimodal generalist autonomous agents for desktop and web, 2024.
Kaixin Li, Meng Ziyang, Hongzhan Lin, Ziyang Luo, Yuchen Tian, Jing Ma, Zhiyong Huang, and Tat-Seng Chua. Screenspot-pre GUI grounding for professional high-resolution computer use. In Workshop on Reasoning and Planning for Large Language Models, 2025. URL https:
//openreview. net/forum? d=XakND

---

Published as a conference paper at ICLR 2026
Wei Li, William Bishop, Alice Li, Chris Rawles, Folawiyo Campbell-Ajala, Divya Tyamagundlu, and Oriana Riva. On the effects of data scale on control agents, 2024. URL https://
arxiv. org. abs / 2006 03679
Yang Li, Gang Li, Luheng He, Jingjie Zheng, Hong Li, and Zhiwei Guan. Widget captioning: Generating natural language description for mobile user interface elements, 2020. URL https:
//axiii.or////o..aalaaooo..o...a httpaaab//..00.. 04295
Kevin Qinghong Lin, Linjie Li, Difei Gao, Zhengyuan Yang. Shiwei Wu, Zechen Bai, Weixian Lei, Lijuan Wang, and Mike Zheng Shou. Showui: One vision-languagc-action model for gui visual agent, 2024. URL httpss://arxii
org/
Xinyi Liu, Xiaoyi Zhang, Ziyun Zhang, and Yan Lu. Ui-e2i-synth: Advancing gui grounding with large-scale instruction synthesis, 2025a.
http://wwww...g/bb////ba////// 11257
Yuhang Liu, Pengxiang Li, Zishu Wei, Congkai Xie, Xueyu Hu, Xinchen Xu, Shengyu Zhang, Xiaotian Han, Hongxia Yang, and Fei Wu. Infiguiagent: A multimodal generalist gui agent with native reasoning and reflection. arXiv preprint arXiv:2501.0457. 2025b.
Yuhang Liu, Pengxiang Li, Congkai Xie, Xavier Hu, Xiaotian Han, Shengyu Zhang, Hongxia Yang. and Fei Wu. Infigui-rl: Advancing multimodal gui agents from reactive actors to deliberative reasoners, 2025c. URL http:///.aaav......aaoooooo abs aww.aas///55/ 2500 14239
Yuhang Liu, Zeyu Liu, Shuanghe Zhu, Pengxiang Li, Congkai Xie. Jiasheng Wang, Xueyu Hu, Xiaotian Han, Jianbo Yuan, Xinyao Wang, et al. Infigui-gl: Advancing gui grounding with adaptive exploration policy optimization. arXiv preprint ar arXiv:2508.057. 2025d.
Yuqi Liu, Bohao Peng, Zhisheng Zhong, Zihao Yue, Fanbin Lu, Bei Yu, and Jiaya Jia. Seg- zero: Reasoning-chain guided segmentation via cognitive reinforcement. arXiv preprint arXiv:2503.06550, 2025e.
Yuqi Liu, Tianyuan Qu, Zhisheng Zhong, Bohao Peng, Shu Liu, Bei Yu, and Jiay Jia. Vision- reasoner: Unified visual perception and reasoning via reinforcement learning. arXiv preprint arXiv:2505. /2081, 2025f.
Yadong Lu, Jianwei Yang, Yelong Shen, and Ahmed Awadallah. Omniparser for pure vision based gui agent, 2024. URL httpt::t/////ttttt org/ abs/ 2408 00203
Zhengxi Lu, Yuxiang Chai, Yaxuan Guo. Xi Yin, Liang Liu, Hao Wang, Guanjing Xiong, and Hongsheng Li. Ui-rl: Enhancing action prediction of gui agents by reinforcement learning. arXiv preprint 25v:22 1620, 2025.
Run Luo, Lu Wang, Wanwei He, and Xiaobo Xia. Gui-rl: A generalist rl-style vision-language action model for gui agents. arXiv preprint arXiv:250411448,, 2025.
OpenAl. Gpt-4o system card, 2024. BRRL:ttttp///arxvv.ooom http://www..o//aa//1//000200.2
OpenAl. Developing generalist computeruuuuuu agent. OpenAl, 2025, URL httptt:t//////aaaa
cttpdeex/ccoputee--eeee comm/index//commtttt--ul-aa. -agneepgtttttttiiiia-eeeeeeeee// Accessed: October 22, 2025.
OpenAl. Gpt-4.1 announcement. httpps//openaa..com/neeeee//
2025a. Ac-
cessed: 2025-08-03.
OpenAl. Gpt-5 model overview.
http:/aaai..cm/p/t----0 2025b. Internal model re- lease; peer-reviewed technical report available at the time of writing.
Yujia Qin, Yining Ye. Junjie Fang. Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, et al. Ui-tars: Pioneering automated gui interaction with native agents. arXiv preprint arXiv:2501. AXXi:2501.113266 2025.
Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajile Daniel Toyama, Robert Berry, Divya Tyamagundlu, Timothy Lillicrap, and Oriana Riva. Androidworld: A dynamic benchmark- ing environment for autonomous agents, 2024. URL https: arxiv. org/ ab4//455. 14573
12

---

Published as a conference paper at ICLR 2026
ByteDance Seed. Ui-tars-1.5. https:
2025.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL http:://rrxiv org/abs/2402.03300 2(3):5, 2024.
Fei Tang, Zhangxuan Gu, Zhengxi Lu, Xuyang Liu, Shuheng Shen, Changhua Meng. Wen Wang, Wenqi Zhang, Yongliang Shen, Weiming Lu, Jun Xiao, and Yueting Zhuang. Gui-g²: Gaussian reward modeling for gui grounding, 2025. URL BtRL:ttts:/// //axttss//axiv.. org/ abs/ abs/2507.1184
General Agents Team. The showdown computer control evaluation suite, 2025.
URL
https:
//github. wtt htt.a//w..ccoooeeeeeaaaaaggeea/a showdown
Haoming Wang, Haoyang Zou, Huatong Song, Jiazhan Feng, Junjie Fang, Junting Lu, Longxiang Liu, Qinyu Luo, Shihao Liang, Shijue Huang, et al. Ui-tars-2 technical report: Advancing gui agent with multi-turn reinforcement learning. arXiv preprint 210.:::250902544444 2025a.
Xinyuan Wang, Bowen Wang, Dunjie Lu, Junlin Yang. Tianbao Xie, Junli Wang. Jiaqi Deng, Xi- aole Guo, Yiheng Xu, Chen Henry Wu, Zhennan Shen, Zhuokai Li, Ryan Li, Xiaochuan Li, Junda Chen, Boyuan Zheng, Peihang Li, Fangyu Lei, Ruisheng Cao, Yeqiao Fu, Dongehan Shin, Martin Shin, Jiarui Hu, Yuyan Wang, Jixuan Chen, Yuxiao Yc, Danyang Zhang. Dikang Du, Hao Hu, Huarong Chen, Zaida Zhou, Haotian Yao, Ziwei Chen, Qizheng Gu, Yipu Wang, Heng Wang, Diyi Yang. Victor Zhong, Flood Sung, Y. Charles, Zhilin Yang, and Tao Yu. Opencua: Open foun- dations for computer-use agents, 2025b. htRthtts::
axiiv. org/abs/////oooooooogo/g.o/o.aaa abs 2508 ..300..000 09123
Ye Wang, Ziheng Wang, Boshen Xu, Yang Du, Kejun Lin, Zihan Xiao, Zihao Yue, Jianzhong Ju, Liang Zhang. Dingyi Yang. Xiangnan Fang. Zewen He, Zhenbo Luo, Wenxuan Wang, Junqi Lin, Jian Luan, and Qin Jin. Time-rl: Post-training large vision language model for temporal video grounding, 2025c. URL htttttts: /etp://axxiv... http://axxvv.gg//aaae /abs/ 2503 13377
Yiqin Wang. Haoji Zhang, Jingqi Tian, and Yansong Tang. Ponder & press: Advancing visual gui agent towards general computer control, 2024. URLhttts::/aaxiii wwwp:/s/g/a/////e/. 01268
Qianhui Wu, Kanzhi Cheng, Rui Yang, Chaoyun Zhang, Jianwei Yang, Huiqiang Jiang, Jian Mu, Baolin Peng, Bo Qiao, Reuben Tan, et al. Gui-actor: Coordinate-free visual grounding for gui agents. arXiv preprint vxXiv22506.0314:3 2025.
Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, and Yu Qiao. Os-atlas: A foundation action model for generalist gui agents, 2024a. URL https:/ /ttx/axvvv....///// org/ abs/2410 23218
Zhiyong Wu, Zhenyu WI, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng. Zichen Ding, Liheng Chen, Paul Pu Liang, et al. Os-atlas: A foundation action model for gener- alist gui agents. arXiv preprint arXiv:2410.23218, 2024b.
Tianbao Xie, Jigqi Deng. Xiaochuan Li, Junlin Yang. Haoyuan Wu, Jixuan Chen, Wenjing Hu, Xinyuan Wang. Yuhui Xu, Zekun Wang, Yiheng Xu, Junli Wang, Doyen Sahoo, Tao Yu, and Caiming Xiong. Scaling computer-use grounding via user interface decomposition and synthesis, 2025. URL UtLtttts://aartt
Yiheng Xu, Zekun Wang, Junli Wang. Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, and Caiming Xiong. Aguvis: Unified pure vision agents for autonomous gui interaction, 2025.
URL wttpt:tt:////..... /ww.i//aavvvv///////a org abs/ /alas/241 221/24412222 04454
JingJing Xie Xuehui Wang, Zhenyu Wu et al. Mmbench- gui: Hierarchical multi platform evaluation framework for gui agents. arXiv preprint wxXiv:2507.1947. 2025.
An Yang, Baosong Yang. Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Ma. Jianxin Yang, Jin Xu, Jin- gren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao,
13

---

Published as a conference paper at ICLR 2026
Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wen- bin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu. Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yugiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 technical report, 2024a. URL
https://arxiss org/ abs 2
Yan Yang, Dongxu Li, Yutong Dai, Yuhao Yang, Ziyang Luo, Zirui Zhao, Zhiyuan Hu, Junzhe Huang, Amrita Saha, Zeyuan Chen, Ran Xu, Liyuan Pan, Caiming Xiong. and Junnan Li. Gtal: Gui test-time scaling agent, 2025. URL (tRlsttts:://// //tp://aaxvvv. org/ abs 2507 05791
Yuhao Yang, Yue Wang, Dongxu Li, Ziyang Luo, Bei Chen, Chao Huang, and Junnan Li. Aria- ui: Visual grounding for gui instructions, 2024b. URL httpss//arxiv..rr/ wtt.oorrg/abbs//ggoo 16256
Xinbin Yuan, Jian Zhang, Kaixin Li, Zhuoxuan Cai, Lujian Yao, Jie Chen, Enguang Wang, Qibin Hou, Jinwei Chen, Peng-Tao Jiang, al. Enhancing visual grounding for gui agents via self- evolutionary reinforcement learning. arXiv preprint 1.Xiv:2505.12770 2025.
Miaosen Zhang, Ziqiang Xu, Jialiang Zhu, Qi Dai, Kai Qiu, Yifan Yang, Chong Luo, Tianyi Chen, Justin Wagle, Tim Franklin, et al. Phi-ground tech report: Advancing perception gui grounding. 2025.
arXiv preprint arXiv:2507.23777
Wangchunshu Zhou, Yixin Ou, Shengwei Ding, Long Li, Jialong Wu, Tiannan Wang, Jiamin Chen, Shuai Wang, Xiaohua Xu, Ningyu Zhang, Huajun Chen, and Yuchen Eleanor Jiang. Sym- bolic learning enables self-evolving agents. 2024.
18332
Yuqi Zhou, Sunhao Dai, Shuai Wang, Kaiwen Zhou, Qinglin Jia, and Jun Xu. Gui-gl: Un- derstanding r1-zero-like training for visual grounding in gui agents, 2025. URL https:
// //aaxiv.ooo abs 2205 15810
Jinguo Zhu, Weiyun Wang, et al. Internv13: Exploring advanced training and test-time recipes for / axxi.. www.vorrg/aaoo.. abs/ /2044 10479
open-source multimodal models, 2025.
14

---

Published as a conference paper at ICLR 2026
A RELATED WORK
A.1 REASONING IN GUI GROUNDING
Reasoning is critical capability for MILMs. However, for GUT grounding task, enabling the model to perform Free-Fore Form reasoning (FFR) during the RL stage does not improve performance and may even degrade it, as demonstrated by GUI-GI (Zhou et al.] 2025), GTAI (Yang et al. (025), GUI- 2025 and our own experiments in Sec. 4.5 Although some
G2 (Tang al. 2025), UI-RI (Lu et al. works, such as InfiGUI-GI (Liu et al. 2025d). InfiGUI-RI (Liu et al.] 2025c). and GUI-RI (Luo) al. 2025), have utilized the Free-Form Reasoning, they did not provide ablation experiments to investigate its effectiveness. Furthermore, GUI-RI found that the model performance is improved as the reward weight for the "thinking" format was decreased. Nevertheless, the failure of free- form reasoning does not imply that reasoning is ineffective for GUI grounding. Our Instruction- as-Reasoning method instills strong exploratory capabilities by training the model with diverse and effective reasoning pathways during the SFT stage. Consequently, the model generates more diverse rollouts during RL, effectively mitigating policy collapse.
A.2 INSTRUCTION IN GUI GROUNDING
Comprehending the user instruction is critical for achieving success in GUI grounding. Prior works, such as Aria-UI (Yang et al. 2024b) and Phi- Phi-Ground (Zhang et al.) (025), have primarily focused on augmenting instructions the input level, using advanced MLLMs to paraphrase them into varying styles. Yet, this methodology suffers from critical limitations: (1) it treats instructions merely as static inputs rather than dynamic reasoning pathways; (2) it lacks deep analysis of the impact of instruction on grounding; (3) it fails to demonstrate significant, consistent performance gains. Dif- fering from these approaches, our work provides an in-depth investigation of how the diversity and quality of instructions can affect model performance. Moreover, our Instruction-as-Reett -ea------- not only enhances the diversity of instructions but also innovatively repurposes these diverse instructions as learnable reasoning pathways for the model, leading to substantial performance improvements.
A.3 TRAINING PARADIGM IN GROUNDING
Prior GUI grounding methods mainly focus training in Supervised Fine-Tuning (SFT) paradigm, such as JEDI (Xie et al. 22225, OS-Atlas (Wu et al. 2024b). Aguvis (Xu et al. 2025),
Uground (Gou et al. (2025) and Aria- UI Yang et al., (2024)) Reinforcement learning methods, par- ticularly GRPO (Guo et al. 2025) have demonstrated remarkable sucess on various visual-language tasks, including Semantic Segmentation (Liu al. (025e)) Visual Question-Answering (Liu al.) 2025B Huang et al. 2025) and Temporal Video Grounding (Wang et al. 2025c). Consequently, re- cent efforts have increasingly focused on adapting RL for GUI grounding. GUI Grounding methods like GUI-RI (Luo et al., 2025). GUI-Actor (Wu et al. 2025) and GTAI (Yang et al. 2025) play as an pioneer role in pure RL paradigm and surpass SFT-based methods by large margin. However, a key limitation of pure RL paradigm is that it overlooks the substantial benefit offered by initial SFT stage. While InfiGUI-RI (Liu et al. 2025c) achieved success with an SFT+RL framework by reframing GUI grounding as trajectory- level task that encourages model reflection, the SFT+RL paradigm remains notoriously difficult to implement in practice, which is also demonstrated by Phi- Ground (Zhang et al.) 2025) and our experimental findings in Sec. 4.5 SFT+RL framework is prone to policy collapse issues. Our Instruction- -ssoonn----ie as-Reasoning method addresses this gap by leveraging SFT to teach model with broader world knowledge and reasoning format demonstrations, and then utilize the RL stage to further incentivize the model to select the best reasoning pathway, establishing a successful example for the SFT+RL training paradigm.
B DATA PIPELINE DETAILS
B.1 INSTRUCTION DIVERSITY AUGMENTATION
To enhance instructional diversity, expanded the instruction set based on frequently occurring scenarios, categorizing them into four types: appearance-based function-based apatial-based, and intent-based. When leveraging GPT-4.1 augment instructions from open-source datasets,
15

---

Published as a conference paper at ICLR 2026
mitigated potential hallucinations arising from poor-quality original instructions. To achieve this, we visually grounded the process by overlaying the ground-truth point or bounding box as distinct circular rectangular marker the input image. We provide our detailed prompt Prompt
Prompt 1: Prompt for Instruction Generation
## Task:
Generate and Translate Unambiguous Grounding Instructions
## Input:
GUI Screenshot: An image of user interface.
Original Instruction: An initial English instruction.
Highlighted Element: A visual marker e.g., red <annotation, htppe> the screenshot pointing to the target UI element.
CORE OBJECTIVE
Your primary task to first translate the Original Instruction into high-quality Chinese, and then generate four new, distinct types of grounding instructions. For all generated instruc- tions, you must adhere to this critical rule: the instruction must correspond to one and only one element on the entire screen the one highlighted. Clarity and uniqueness are the top priorities.
IMPORTANT SAFEGUARD
The <annotation is ground-truth annotation provided only for your reference. Your instructions must never refer to the annotation itself.
It is noticeable that the original instruction may not align with the ground-trute annota- tion, you should follow the ground-truth annotation first.
## Instructions Generation Requirements: Generate one new, clear, and unambiguous instruction for each of the following four cate- gories.
Appearance-Basse
direct and literal description of the element's 's visual characteristics (e.g.. its text, icon, color, shape). Combine features as needed to ensure the description is completely unique. Function-Basee
A clear description of the element's 's purpose or the immediate outcome of interacting with it (e.g.. "the button used to confirm and save your profile changes").
Spatial-Based:
An instruction that identifies the element based on its position relative to other prominent, easily identifiable UI elements (landmarks). The described spatial relationship must lead to a unique location.
Goal-Based:
A concise phrase that describes the user's ultimate goal or intent. The user must infer which single UI element on the screen fulfills this goal.
## Output Format:
The final output must be a single, well-formed JSON object. The JSON structure should begin with the original instruction and its translation, followed by the newly generated in- structions.
Now, please process the following inputs and generate the instructions in the specified JSON format.
Original Instruction:
<instruction_han here>>
B.2 INSTRUCTION QUALITY REFINEMENT
To verify and filter the quality both the original and the newly generated diverse instructions, we prompted GPT-4.1 to assess whether each instruction uniquely corresponded to single element in the GUI screenshot. To mitigate potential model hallucinations during this verification process, we visually grounded the task by overlaying the ground-truth annotation directly onto the input image. We provide our detailed prompt as Prompt
16

---

Published as a conference paper at ICLR 2026
Prompt 2: Prompt for Verification
## Task:
Quality Evaluation of GUI Grounding Datum ## Role:
You are meticulous Data Quality Analyst specializing in interface datasets. Your task to critically evaluate a given data sample for its quality and correctness in a structured, two-step process.
## Input:
GUI Screenshot: An image of user interface.
Grounding Instruction: An English command intended to guide user to specific element. Ground-Truth Bounding Box: A red box drawn on the screenshot, highlighting the target Ul element.
IMPORTANT
Ground-Truth Point: A blue hollow circle drawn the center of the Ground-Truth Bound- ing Box, which the key to help you locate the target UI element, because screenshots usually have other red bboxes which may cause distribution. ## Output Process (Two Steps):
### Step 1: Chain-of-Thouge
Reasoning
First, you must articulate your reasoning process in plain text. Analyze the input and think step-by-step. Your reasoning should cover the following points:
Instruction Analysis:
What specific element does the instruction describe? Identify its key features (text, function, location, etc.), it is important you should locate the target UI element according to the blue hollow circle and the red bbox.
Scan the entire screenshot. Are there any other elements that could match this description, even partially?
Conclude whether the instruction is unique or ambiguous based on this scan. Bounding Box Analysis:
What is the target element identified by the instruction? Does it have the blue hollow circle in the center the box?
Does the red box tightly enclose this entire target element?
Does the box cut off any part of the element?
Does the box include significant empty space or other unrelated elements?
Conclude whether the bounding box is appropriately sized, too large, or too small. ### Step 2: Final JSON Output
After you have completed your reasoning, provide the final answer as single, well-formed JSON object. This ISON should be the very last part of your response. Do not add any text after the JSON object.
"instruction.c ietiiiitinvllaaiionn:.::
"reasoning": "<A concise summary of your reasoning from Step about the instruc- tion's uniqueness.>"
"s unique": <true or false>>
bbox_ evaaluatat:::""
"reasoning": "<A concise summary of your reasoning from Step I about the bounding box size.>".
"is appooppppyy sized": sized"" <true or false>
Now, please perform this two-step evaluation for the following data. Grounding Instruction:
<instruction here>
17

---

Published as a conference paper at ICLR 2026
C EXPERIMENT DETAILS
C.I EXPERIMENTAL SETTINGS
Data and Implementation Details. We collect data from several public datasets, including OS- Atlas (Wu et al. 2024a), Omniact (Kapoor et al. 2024), Android Control (Li et al., 2024), AMEX CChai et al. 2025), and AgentNet (Wang et al. 2025b), covering diverse operating systems such as Windows, MacOS, Linux, and Android. All samples are processed through our pipeline to ensure quality. We employ Qwen2.5-VL-77 and Qwe32.5-VL-32B
backbone architectures. The
training procedure consists of two stages:
SFT Stage We fine-tune the models on approximately 283k instances for one epoch. Each instance is organized Prompt + show Instruction-ass Instructtion-ss-Reeaannn -aaa--snnnn with red. To teach the model to reason from diverse instruction perspectives, each training instance constructed by randomly selecting two distinct instruction perspectives from and goal) that we defined. One is designated as the instruction perspective, and the other as the reasoning perspective. We use global batch size of 256 and a learning rate of 5e-6.
of four (appearance, spatial, function
Prompt 3: SFT Training Data Example
You are GUI agent. You are given task and your action history, with screenshots. You need to perform the next action to complete the task. ## Output Format
# System Prompt:
Return json object with function name and arguments within <tool. calll></tooo_caaa
XML tags:
<think>>
</think>>
<tool call>
("name": "grounding", "arguments": <arge-json-objett>>
</tool_call>
<args-json-object>> represents the following item of the action space:
repressnns
## Action Space
{"action": "click", "coordinate": [x. y]}
Your task is to accurately locate UI element based on the user's instructions. The screen- shot resolution is height [height} and width {width].
First, you should carefully examine the screenshot and analyze the user's instructions from one of the following different perspectives:
Appearance Perspective: You should prioritize visually scanning the user interface to find an element that precisely matches the label, text color, background, shape, and its relative position to other elements described in the appearance-based instruction. Your decision-making will be driven by direct pattern matching of these visual attributes.
Function Perspective: You should identify which UI element is responsible for achieving the specified system state change or executing the described action, such as switching the media display mode. Your decision-making will be guided by understanding the intended effect or utility of interacting with the element.
Spatial Perspective: You should first locate any specified reference points (e.g.. another UI element screen edge) and then precisely pinpoint the target element based its exact directional and positional relationship within the UI layout. Your ciccision-making will be based geometric and relational positioning within the interface.
Goal Perspective: You should identify the element or sequence of actions that directly leads to the fulfillment of the user's stated ultimate objective. Your decision-makiin will be driven by finding the most direct and efficient path to achieve the desired end-state for the user.
You should translate the user's instruction into the corresponding instruction, and then
provide the final action.
18

---

Published as a conference paper at ICLR 2026
# User Instruction:
Click on the CSDN bookmark in the bookmarks bar to access the CSDN website.
# Assistant:
<think>
will analyze this instruction from Appearance- Based perspective, the user's instruction can be represented as Click the bookmark with the red "C' icon and the label 'CSDN' in the bookmarks bar.
</think>
<tool call>>
""roo""ding"""""""""""""""" "araummenss"""act"""""""""""" ("action":"ellckk ""cooriintt""" (1888,671)
</tool_call>
RL Stage The GRPO training utilizes 33k instances, expanded to approximately 100k train- ing samples by generating sample per instruction perspective, each instance organized as Prompt We leave the analytical perspective unspecified in the prompt to encourage exploration. We adopt a learning rate of le-6 and 8 rollouts. The batch size set to 256 for the 7B model and 128 for the 32B model.
Prompt 4: RL Training Data Example
You are GUI agent. You given task and your action history, with screenshots. You need to perform the next action to complete the task. ## Output Format
# System Prompt:
Return json object with function name and arguments within
XML tags:
<think>
<think>>
<tool call>>
""name": "grounding", "arguments": <args-json-object>>
<tool_call>
<args-json-objet>> represents the following item the action space:
## Action Space
("action": "click", "coordinate": [x. y]}
Your task is to accurately locate a UI element based on the user's instructions. The screen- shot resolution is height (height) and width {width). First, you should carefully examine the screenshot and analyze the user's instructions in <think>..<thhnn
tags and then output the coordinate.
# User Instruction:
Click on the CSDN bookmark in the bookmarks bar access the CSDN website.
# Assistant:
<think>
</think> <tool call>
</tool_cal>>
Baselines and Metrics. We compare our method's grounding performance against extensive SOTA baselines. These include models that are primarily trained using supervised fine-tuning, such as Jedi (Xie et al. 2025) and Aguvis (Xu et al. 22025), methods that using RL paradigm, such GUI- Actor (Wu et al. 2025) and InfiGUI-GI (Liu et al. 2025d), and influential grounding models such UI-Tars- 1.5-s-1..77B- (Seed 2025) and GTAI-7B Yang et al. (2025).. Besides, also compare UI-Ins with some agentic frameworks such as AgentS2 (Zhou et al. 2004) and InfiGUlAgent (Liu et al. 20255) on the online benchmark.
19

---

Published as a conference paper at ICLR 2026
Following prior works (Yang et al. 2025 Liu et al. 2025d Tang al. 2025).
evaluate GUI
Grounding performance using the point-in-box. accuracy. A prediction is considered correct the predicted coordinate point p (It, Ut, Fr, Wr), where the (Ii, 3/) denotes the top-left corner and (Ir, Dr) represents the bottom-right corner. The accuracy over test set of size N is formally defined as: Accuracy N Ei [(p; E 6;) where II(.) is the indicator function, which equals the condition is true and Ó otherwise.
(x Up) falls within the ground-trute counddtruth bounding box b
Evaluation Benchmarks. We evaluate our method on five widely-used grounding benchmarks and a challenging online agent environment.
Grounding Benchmarks: MMBench-GUU 1.2 (uuehui Wang et al. 2025 tests performance on hicrarchical instructions, while UI-I2E-Bench (Liu et al., 2025a focuses on explicit instruc- tions and deeper semantic reasoning for implicit instructions. Showdown (Team 2025) evaluates instruction- -follwiin-folloiing and low-level control capabilities. ScreenSpot-Pro Li et al. (2025) exam- ines semantic understanding high-resolution professional softwares. ScreenSpot- V2 (Wu et al.) 2024a) is widely adopted benchmark that evaluates model across different operating systems.
It is notable that, for a comprehensive evaluation, we show the subset scores of all benchmarks, especially for MMBencchGUUI L2 and UI-12E Bench, which emphasize instruction understand- ing and difficulty. MMBench-GUI L2 is divided into 'Basic" and 'Advanced' subsets, which are distinguished by their instruction style for the same UI element. 'Basic' instructions provide comprehensive visual features, such as "A rectangular button with a dark purple background," whereas 'Advanced' instructions describe the element's inferred purpose, like "Upgrade your cur- rent workspace." Similarly, U1-1EE-Bench contains 'explicit' and 'implicit" subsets, where an explicit instruction might be "Enter your email in the subscription field," while an implicit one requires inference, such as "Click to dispatch the email."
Online Agent Benchmark: To evaluate our model's 's practical utility in dynamic setting, we report performance on AndroidWorld (Rawles et al. 2024). Unlike simulated or replayed settings, this benchmark introduces realistic challenges, including UI drift, variable rendering latency, asyn- chronous state transitions, and stochastic user feedback, which collectively pose a strong challenge to the temporal and spatial consistency of grounding.
C.2 ONLINE BENCHMARK EVALUATION
For the evaluation of the AndroidWorld benchmark, we develop simple yet effective agent frame- work to evaluate the grounding capability of our model in the online environment. Our framework consists of two main agents, a planner (i.e., GPT-5), which serves as the high-level controller to decide the executed action in each step, and executor (i.e. UI-Ins 7B) that identifies the pre- cise coordinates based on each instruction from the planner. Specifically, during each step. the planner receives the task goal, historical records of thinking and actions in previous steps, and the current screenshot, and produces the next reasoning trace and corresponding structured JSON ac- tion that conforms to our pre-defined action space. When the predicted action type "clis cclick " or " "long-press" ", the ISON instruction is forwarded to our grounding executor. Then our executor interprets the textual description of the target element (e.g., "blue circle button at top-right") and out- puts precise screen coordinates, which then performs the "cl ickk " or " "long loongppesss press" operation on the Android device. The resulting screen update and exccution feedback are sent back to the plan- ner, enabling it to iteratively refine its decisions and complete the task through perception-action loop. We provide our detailed system prompt as Prompt 5
Prompt 5: Prompt for Online Agent Evaluation
## Task. Control Android phone answer queries and execute tasks with precise, verifiable actions.
## Role. Android Phone Operator AI. Responsibilities:
Retrieve information from the device answer user questions.
Perform tasks by executing precise UI actions.
## Action Framework. Respond with EXACT JSON for one of the following actions:
20

---

Published as a conference paper at ICLR 2026
Action
Description
JSON Example
open app
Open app from <Ava 1 lable Apps>
fact fan "ype":"open app", *app name "Chrone" }
slick
Tap visible element
{ "ascionayye" :"lllck"""""""" "Lazget"; Mbaue "Balu
batton at right"
long press
Long-prest poeg--pess visãble element
{ faction, beriionn.oppeee... t"pee: "rensepresss" "cargut": (oessage
Inpot tex:
Type into a field
fast fon "ipput text", "cext":""lll"""" "carget" "nescage ingut bex"
Respond
"aon ion fenswer", "text""""t's 25 degrees today, }
navigate.hore
Return to home sereen
"astionatype" :nnnatt"""""""ttiiii"" ""aarigttt.too=" }
navi gate back
Navigate back
ion type prariigate back"
scroll
Scroll upddow//leltiight Mark task status
rast type **ccolll", "disectloa": "down"" "action.tyye" ""sttuss", "otates": "ccmplote" "accion.type" : "waie" }
status
update
## Execution Principles.
Communication Rule
Always use answer to reply to users; do not assume on-screen text sufficient.
Follow the user instruction strictly (e.g., single number, True/False, comma-separated items).
Do not answer for waiting loading; wa t.
2. Efficiency First
Choose the simplest valid path.
If an action fails twice, try an alternative (e.g., leng.press instead of click)
3. Smart Navigation
Prefer open_app with the available app list over manual navigation.
Gather information when needed (e.g., open Calendar to check schedule).
For scrolling: direction is inverse to swipe; if scroll fails, try the opposite direction.
4. Text Operations
Activate the input box before typing.
Prefer input_text over manual typing.
For manipulation: long-pre to select, use selection bar (Copy/Paste/Select All), delete
by selecting then cutting.
## Current Context.
User Goal: {goal}
Previous Actions: (history]
["Camera", "Chrome", "Clock", "Contacts", "Dialer", "Markor", "Tasks", "Simple Draw Pro", "Simple Gallery Pro", "Simple SMS Messenger", "Audio Recorder", "Pro Expense", "Broccoli APP", "OSMand", "VLC", "Joplin" "Retro Music", "OpenTracks", "Simple Calendar Pro", "Files", "Settings", ]
Available Apps:
## Decision Process.
1. Analyze goal, history, and current screen.
2. Determine the task is complete; output atatus if true.
3. If not complete, choose the most appropriate single action.
4. Output in the exact format below, ensuring the action is valid ISON.
## Output Format.
Thought:
Carone
intosmation....
Autions yoed "opennappp "appuname" "Chcoms"
21

---

Published as a conference paper at ICLR 2026
Table 8: Performance comparison on the MMBench-GUI L2 benchmark.
WWndows MacOS
Linux
iOS
Android
Web
Model
Avg.
Bas. Adv. Bas.
Adv.
Bas.
Adv.
Bas.
Adv. Bas.
Adv.
Bas,
Adv.
GPT-4o OpenAl 2024
1.5
4.3
1.1
5.1
3.3
2.5
1.4
3.2
2.9
2.9
Claude-3.7 (Atthropic 2024)
1.5
0.7
12.5
7.5
1.1
0.0
13.7
10.6
1.4
1.4
3.2
2.3
4.7
Qwen-Max- VI. Yang et al.
2024a
43.9
36.8
58.8
56.1
53.9
30.1
77.4
59.1
79.5
70.1
74.8
58.8
58.0
Show (Lin et al. 2024
9.2
4.4
24.1
10.4
25.1
11.7
29.0
19.7
17.4
8.7
22.9
12.7
16.0
Qwen2.5-VL-77 (Bar et al. 2025
31.4 16.5
31.3
22.0
21.5
12.2
66.6
55.2
35.1
35.2
40.3
32.5
33.9
OS-Atlas-7B [Wu
(2024a
36.9 18.8 44.4
21.7
31.4
13.3
74.8 48.8 69.6 46.8
61.3
35.4
41.4
Aguvis-7B (Xu
(0055)
37.3 21.7 48.1 33.3
33.5
25.0 67.5 65.2 61.0 51.0 61.6
45.5
45.7
UI-TARS-1.5 1. [Seed] 2025)
68.3 39.0 69.0 44.5 64.4
37.8 88.5 69.4 90.5 69.3 81.0 56.5
64.3
UGround-VI-7B (Gou el al.
2025
66.8 39.0 71.3 48.6 56.5
31.1 92.7 70.9 93.5 71.0 88.7 64.6
65.7
GUI-Actor-7B" (Wu et al.
2025
80.8 55.1 81.4 60.4
64.9 41.8 94.3 82.7 93.5 79.7 89.7 72.1
76.5
SE-GUI-7B" Yuuan
2025
77.5 57.7 77.1 60.7
68.6
44.9 95.5 80.0
95.5 83.7 89.7 68.8
76.6
GTAI-7B" Yang
76.8 57.4
63.9
68.6 53.6 93.9 83.3 96.3 84.5 90.3
74.7
78.5
GUI-G2-7B (Tagg al.
2025
79.7 55,1 79.7 64.7 69.6 50.0
95.2
82.7 96.6
85.4 91.9
75.6
78.8
InfiGUI-G1-7B (Liu 2025d
82.7 61.8 83.8
63.9
72.3 52.0
94.9
89.4
95.2
85.6
93.5
76.3
80.8
Qwen2.5-VL-722 (Bai et al.
2025
55.7 33.8 49,9 30.1 40.3
20,9
56.1 28.2
55.6
25.4
68.4 45.8
41.8
Qwen2.5-VL-32BB [Bai et 2025
73.4 49.3 76.2 57.8 61.3 33.2 91.1 80.6 90.4 80.6 81.6 65.6
72.1
IntemV113-782 (Zhu
20255
70.1 42.6 75.7 52.3 59.2 41.3 93.6 80.6 92.7 78.6 90.7 65.9
72.2
UI-TARS-DPO-22H (Qin 2025
78.6 51.8 80.3 62.7 68.6 51.5 90.8 81.2 93.0 80.0
88.1 68.5
74.3
GTA ITA1-32 Yang ct 2025
82.3 66.9 89.0 74.0 73.3 52.0 96.2 88.2 95.8 88.5 95.2 79.9
83.4
Un-Ins-B
82.7 64.7 87.2 75.1 71.7 51.5 94.9 89.7 95.8 89.0
93.2 80.8
83.1
UI-Ins-32B
84.9 68.4 88.4 73.4
68.6 56.1 96.5 91.2 97.2 92.4
94.8 85.1
84.9
Table 9: Performance comparison on ScreenSpot- V2 and ShowDown
ScreenSpot-V V2
ShowDown
Model
Mobile
Desktop
Web
Avg.
Avg.
Text
Icon.
Icon.
Text
Icon.
Phi-ground-7B (Zhang al. 2025
90.2
76.4
93.6
75.9
96.5
62.0
83.8
62.5
OS-Atlas-7B Wu el al.) 2024a
95.2
75.8
63.6
90.6
77.3
85.1
41.1
UGround-v1 111BB (Gou et al. 2025
83.6
90.5
85.8
86.3
95.5
83.2
87.7
57.8
Owen2.5-VL-77 [Ba al. 2025
97.6
87.2
90.2
74.2
93.2
81.3
88.8
43.6*
UI-Tars- 100s-1155777 1.5-7B [Sced] 2025
92.2
81.5
91.0
84.2
95.5
84.5
89.0
67.2
SE-GUI-7B (Yuan
2025
99.3*
89.1*
96.4"
78.6*
92.7*
81.3*
90.8*
63.6*
UI-TARS-7B QQ ct al. 2025
96.9
89.1
95.4
85.0
93.6
85.2
91.6
66.1
GUI-Actor-7B (Wu ct al 2025
97.6
88.2
96.9
85.7
93.2
86.7
92.1
64.6"
OpenCUA-7B (Wang et 2025b
92.3
GTA1-7B Yang
2025
99.0
88.6
94.9
89.3
92.3
86.7
92.4
67.9*
GUI-GP-7B [Tagg et 2025
98.3
91.9
95.4
89.3
94.0
87.7
93.3
70.4*
InfiGU--G1-/B (Liu et al 2025d
99.0
91.9
94.3
82.1
97.9
89.2
93.5
68.2"
U1- Ul-Venus-7B (Gu et al. (2025)
99.0
90.0
96.9
90.7
96.2
88.7
94.1
Qwen2.5-VL-722 [Bar
202.5
95.5"
84.4"
93.8*
88.0"
88.5*
81.8*
88,2'
62.3*
Qwen2.5-VL-32B (Bai all
2025
97.9"
88.2"
98.5'
79.3*
91.2"
86.2*
91.3*
58.2*
GTA1-32B Yang al. 2125
98.6
89.1
96.4
86.4
95.7
88.7
93.2
71.1"
OpenCUA-322 (Wang
2025b)
93.4
UI-Ins-BB
99.0
90.5
97.9
81.4
97.4
91.6
94.0
73.1
UI-Ins-32B
98.6
90.0
99.0
87.9
97.0
93.1
94.9
73.8
D RESULTS DETAILS
D. GROUNDING BENCHMARK RESULTS
In this section, provide performance details of all five benchmarks. We report all subset scores for MMBennchGUII L2 (Tab. 3). ScreenSpot-Pre (Tab. IT), UI-I2E Bench (Tab 10). ScreenSpot-Ve V2 (Tab. 9 and Show Down benchmark (Tab.
22

---

Published as a conference paper at ICLR 2026
Table 10: Performance comparison on the UI-IZE-Bench benchmark.
Grouped by Platform
Grouped by Implicitness
Model
Overall
Web
Desktop
Mobile
Explicit
Implicit
OS-Atlas-4B
2024a)
54.6
19.9
58.6
51.5
39.9
44.3
U1-I2E-VLM-4P Liu of al
2025a
60.9
38,9
61.4
61.9
48.3
53.4
Uground-V 1-2B Gou ct al 2025
66.4
49.5
59.9
72.9
47.9
57.4
UI-TARS22B QQin et 2025
62.2
54.0
66.7
74.1
54.5
62.0
Aguvis-7B (Xu
22055
45.1
47.6
60.3
61.1
48.4
53.2
Qwen2.5-VL-71B (Bai
2025
56.9
41.6
61.7
58.4
51.0
53.8
OS-Atlas-7B Wu al.
2024a
$2.2
48.9
68.1
63.2
55.8
58.6
UI-TARS-7B Qun el 2025)
56.5
58.0
65.7
71,4
55.3
61.4
GUI-Actor-7B' [Wu
2025
65.2
63.2
72.9
71.6
66.1
68.2
U1-12E- VLM-7B (Liu
2025a
62.1
64.0
76.2
72.0
67.9
69.5
Uground- V1-7B (Gou cl al 2025
70.8
65.7
73.5
81.3
63.6
70.3
SE-GUI-7B" Yuan et al 2025
68.4
66.3
77.4
77.5
68.6
72.0
GUI-G"-7B" Tang al. 202.
53.4
67.8
84.0
82.1
67.5
73.1
UI-TARS-I.5-7B Seed
2025
79.5
68.8
81.3
InfiGUI-G1-7B (Liu
2025d)
84.6
66.3
83.0
85.0
72.7
77.4
GTA1-7B' Yang et al. 2025
77.5
71.3
83.5
87.0
72.8
78.2
Qwen2.5-VL-726 [Bar
2025
49.0
47.2
55.3
49.6
52.5
51.4
Owen2.5-VL--22B
76.7
61.2
67.5
73.8
62.7
66.9
UI-TARS-72B (Qin ct 2025
77.1
69.8
75.5
80.9
73.7
Uground-V1-72B (Gou ct al. 2025
74.7
74.6
78.2
84.5
71.3
76.3
GTA1-32B" Yang 2025
93.3
77.6
84.4
91.4
78.7
83.5
UI-Ins-7B
90.5
72.8
83.8
88.9
76.3
81.1
Ul-Ins-32B
81.9
88.2
92.9
83.9
87.3
Table 11: Performance comparison on the ScreenSpot-Pre benchmark.
CAD
Dev.
Creative Scientifie
Office
OS
Model
Avg.
Text
Icon
Text
Icon
Text
Icon
Text Icon
Text
Icon
Text
Icon
GPT-4o OpenAl
2024)
2.0
0.0
1.3
0.0
1.0
0.0
2.1
0.0
1.1
0.0
0.0
0.0
0.8
Claude C.
Anthropic 2024
14.5
3.7
22.0
3.9
25.9
3.4
33.9
15.8
30.1
16.3
11.0
4.5
17.1
UI-R1-3B Lu al.
2025
11.2
6.3
22.7
4.1
27.3
3.5
42.4
11.8
32.2
11.3
13.1
4.5
17.8
ZonUI-3B Hstch
2025)
31.9
15.6
24.6
6.2
40.9
7.6
54.8
18.1
57.0
26.4
19.6
7.8
28.7
Qwen2.5- V (Bal
(2025
16.8
46.8
49.3
52.5
37.4
3.8
26.8
GUI-R1-7B [uo et all (2025)
23.9 6.3 49.4
4.8
38.9
8.4
55.6
11.8
58.7
26.4
42.1
16.9
31.0
UI-TARS-7B (Qin et 2025
20.8 9.4 58.4
12.4
50.0
9.1
63.9
31.8
63.3
20.8
30.8
16.9
35.7
GUI-Actor-7B Wu et 2025
47.7 9.4
59.1
15.9
59.6
16.1
70.1
25.5
69.5
41.5
55.1
19.1
44.6
SE-GUI-7B Yuan
2125
51.3 14.1
68.2
19.3
57.6
9.1
75.0
28.2
78.5
43.4
49.5
25.8
47.2
GUI-G²-7B (Tagg el al.
2025
55.8 12.5
68.8
17.2
57.1
15.4
77.1
24.5
74.0
32.7
57.9
21.3
47.5
OpenCUA-7B Wang el al. 2025b
50.0
GTA1-7B (Yng
(2025)
53.3
17.2
66.9
20.7
62.6
18.9
76.4
31.8
82.5
50.9
48.6
25.9
50.1
UI-Venus-7H (Gu
(202))
60.4
21.9
74.7
24.1
63.1
14.7
76.4
31.8
75.7
41.5
49.5
22.5
50.8
InfGUU-GG-7B [Liu ct all 2025d
57.4
23.4
74.7
24.1
64.6
18.2 80.6
31.8
75.7
39.6
57.0
29.2
51.9
CogAgent-188B (oong ct al.
2024
7.1
3.1
14.9
0.7
9.6
0.0
22.2
1.8
13.0
0.0
5.6
0.0
7.7
UGround-v1-728 (Gou ct
2025
16.8
4.7
55.8
4.8
54.0
10.5 70.8
22.7
61.0
18.9
40.2
7.9
34.5
UI-Tars-72B (Qin
2025
18.8
12.5
63.0
17.2
$7.0
15.4 64.6
20.9
63.3
26.4
42.1
15.7
38.1
Qwen2.5-VLL3225 (Bar et al.
2025
34.5
20.3
74.0
22.1
16.1
75.0
30.0
74.6
30.2
64.5
33.7
50.5
DWv-n2.5-LL-72B [Bar
2025
54.3
14.1
78.6
26.9
62.6
20.3
77.8
34.5
80,2
47.2
67.3
28.1
53.3
GTA1-32B Yang et all (2025
43.7
23.4
82.5
28.3
69.2
14.7
79.9
3188
80.8
43.4
70.1
32.6
53.6
OpenCUA-328 (Wang 2025b
55.3
UI-Ins-7B
60.9
20.3
75.3
18.6
65.2
18.9
81.3
29.1
79.7
37.7
57.0
25.8
52.2
UI-Ins-32B
51.8 29.7
83.1
26.9
69.7
18.9
83.3
34.5
88.7
50.9
70.1
34.8
57.0
D.2 REASONING PERSPECTIVE ANALYSIS
We performed detailed classification of the model's reasoning process by first manually defining ten distinct analytical perspectives. We then utilized GPT-4, to examine 1477 responses generated by UI-Ins-7h Ins-7B on the whole UI-12F benchmark based the taxonomy following:
23

---

Published as a conference paper at ICLR 2026
Taxonomy: Reasoning perspectives
Appearance
Abbreviation: app
Definition: Describes the static visual properties of UI element, including its color, shape, icon, style, and the literal text it displays.
2. Functionality
Abbreviation: fune
Definition: Describes the element's purpose, its action, or what happens when a user inter- acts with it.
3. Location
Abbreviation: loc
Definition: Describes the element's spatial position on the screen or in the viewport, which can be absolute (e.g.. "top-left") or relative other elements (e.g.. "below the title").
4. Intent
Abbreviation: intent
Definition: Describes the high-level user goal or plan that motivates the entire action. It is often the starting point of a reasoning chain.
5. Structural Relationship
Abbreviation: struct
Definition: Deseribes the element's position within the UI's layout hierarchy (like DOM tree), emphasizing its parent, child, or sibling relationship to other elements or containers.
6. State
Abbreviation: state
Definition: Describes the current dynamic condition of an element, such as whether it is interactive, active, selected, disabled, or checked.
Component Type
Abbreviation: ctype
Definition: Identifies the clement as a standard, reusable design pattern or component, rather than just describing its appearance.
Sequential Position
Abbreviation: seq
Definition: Describes the element's order or temporal place within multi-ster user task.
Salience
Abbreviation: salience
Definition: Describes the element's degree visual prominence, which often determined by its size, contrast, unique styling, or animation.
10. Accessibility
Abbreviation: ally
Definition: Describes non-visual properties provided for assistive technologies, such as screen readers. This includes ARIA labels, roles, and other accessibility attributes.
Based on this analysis, we show the statistics details in Fig. as demonstraed in FigPa Instruciton- as- Re-assoning help model combines different reasoning perspective during inference, furthermore, ans Fig 9b shows UI-Ins is capable generate entirely new analytival angles beyond the four trained perspectives.
D.3 ERROR ANALYSIS
We conducted an error analysis and identified three primary types of failures in the GUI grounding performance of UI-Ins:
Lack of Domain-Specific Knowledge: As shown in Fig. 10 (a). The model's erroneous selection of "Jazwares" demonstrates failure in knowledge required to associate the abstract description "company known for building block toys"' with the correct brand entity, "MEGA".
knowledge grounding, it lacks the external
24

---

Published as a conference paper at ICLR 2026
Learned in SFT
Others
815
698
600
Explored RL
Structural Relationship
440
instance InstanceNumiinn Numbbr
500
Component Type
749
400
Intent
200
300
Location
1110
2188
Functionality
100
56
Appearance
1461
2000
1600
1200
800
400
Perspective Number in Each Response
Perspective Number
Figure 9: (a) UI-Ins combine multiple reasoning pathways in each response. (b) UI-Ins can select different reasoning paths and can explore emergent reasoning perspectives after RL.
Refine your search to view Charizard products from the company known for building block toys.
An empty time slot at AM the calendar event creation interface.
Bandal
Hasbro (44)
Jazwares (440))
MEGA
Nintendo (504)
(a)
(b)
button grid/table located in the status bar the bottom of the excel window.
A green microphone icon located at the bottom of the Spotify player interface.
play Settings
2514
0160LMGB00A 1EAA
Faitta REEAN
(c)
Ground Truth
Ul-ins Prediction
Zoom in for visualization
Figure 10: Error analysis of UI-Ins. (a): Lack of domain specific knowledge. (b): Lack of layout understanding ability. (c) and (d): Hallucination of MLLMs.
Lack of layout understanding ability: Illustrated Fig. (b), the model unable to discern correct clickable area required to fulfill the instruction, demonstrating a weakness in understanding the structural layout of the user interface.
Visual Ambiguity and Hallucination: As seen in Fig. 10 (c) and (d), when visually similar distractor icon is present alongside the ground-trute target, the model struggles to disambiguate between them and may select the incorrect one.
D.4 QUALITATIVE ANALYSIS
To provide a comprehensive qualitative analysis, we compare our UI-Ins-7B and GTA1-7B on am- biguous grounding tasks which need to understand the implicit instructions, shown in Fig [I this comparison indicates our Instruction-as -R-assonngg approach is key to resolving challenging cases where other models fail. Furthermore, provide the grounding results of UI-Ins-32B across var-
25

---

Published as a conference paper at ICLR 2026
Make important points stand out your issue by applying emphasis.
Access the search Tunctionality to find files or text within V5Code workspace.
Create shareable link to this specific map location showing Portland.
Cao
GTAT: (2394,804)
(1496,164)
GTA1: (639,156)
Ul-Ins: <thinks will click the "B' button
Ul-ins: <thinko will click the magnifying glass left
[U--ins: <thinks will click the button
formattinn toolbar
icon pointing right up, located on the right sidebar, think
description </tinnhhn kctol_ call> [1590,801] s/toolllll
sidebar.</thinle
<tool_call call [962,463] </tool_call
calls [2506,581) </tool_call calls
Ground Truth
GTAT Prediction
Ul-Ins Prediction
Zoom visualization
Figure 11:
Reasoning from diverse instruction perspectives enables UI-Ins-7B succeed
ambiguous grounding tasks. This qualitative comparison with GTA1-7B showcases how our aariiiin-------oninnngo process is key resolving challenging cases where other models fail.
ious platforms and software applications. As shown in Fig. 12 UI-Ins-32B demonstrates robust performance on diverse platforms.
26

---

Published as a conference paper at ICLR 2026
A menu option labeled 'Deselect All' with gray background
Get help or additional information about the editing process.
IUDOE Steetid
Deselct ABl
Soow Atoraak
Deeply All legs in This lirrry
Deliio Petoreeli Terp Uleario
Refine your search view Charizard products from company known for building block toys.
Refinee Charizard products from the company known for building block toys.
262%
Outliner
Find similar videos featuring cats by selecting topic tags.
Access your personal account settings and information in the health app.
7:50
18:34
TKB/s Ø
Q
:
Health Every step
m
010000
Description
X
full options car
#shorts #eat afunny
PK
Competition
775K
12, 99.991889
Viowe
2025
#shorts #cat #funny
12 1aller
B0 8PM Auril
Herlh
Wertoct
Device
Ground Truth
Ul-Ins Prediction
Zoom in for visualization
Figure 12: Success Examples of UI-Ins-32B
27


Reviews:
[{"review_id": "jzn7f2HTK8", "reviewer": "Reviewer_zWAk", "summary": "This paper first identifies two types of flaws in current GUI grounding datasets: insufficient prompt diversity and poor prompt quality. The authors conduct preliminary experiments showing that pairing a model with the instruction perspective most suitable for a given task can boost grounding performance even without retraining. They then develop a data-processing framework to filter out unclear instructions and increase perspective diversity, and introduce \"Instruction-as-Reasoning,\" a two-stage training pipeline to further improve grounding. Experiments demonstrate new state-of-the-art results on five major grounding benchmarks.", "questions": "1. Do all baselines use the same data cleaning and preprocessing? The paper points out that the original data has a 23.3% defect rate. If the baseline model is trained on uncleaned data, while this method uses cleaned and augmented data, the performance improvement may come from the difference in data quality rather than algorithm improvement.\n2. When related to \"ambiguous & mismatch\", did the authors measure human–AI agreement?\n3. In Figure 6, if the author could choose a clearer screenshot of the input image to explain, that would be better.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. New SOTA performance across five grounding benchmarks\n2. Propose improving grounding ability by matching the most suitable instruction type to each task.", "weaknesses": "1. I am doubtful about the contribution of the \"Instruction as Reasoning\" paradigm. I think this paradigm is essentially just a combination of \"instruction augmentation + chain-of-thought generation\". Compared to existing multi-perspective instruction augmentations (such as Jedi, UI-E2I-Synth, etc.), does this work possess genuine novelty at the algorithmic level, or is it merely an innovation in framework naming?\n2. Also, the paper claims \"instructions as dynamic reasoning pathways,\" but in practice, it primarily employs a two-stage training mechanism of SFT+GRPO. What is the fundamental difference between this mechanism and the typical RLHF/GRPO structure? Is it simply a renaming of \"prompt diversity\" to \"reasoning diversity\"?\n3. The figure and the table in this paper are not perfect and lack inspiration for the readers. For example, Figure 6 is unclear and should be either condensed or supplemented with more useful information. I am interested in how the model selects optimal strategies across different perspectives during RL, which the figure does not show. Researchers in the field are likely already familiar with the \"Advantages\" and \"Objectives\" sections, making them redundant.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "u7khQU1cCZ", "reviewer": "Reviewer_7NYR", "summary": "This paper aims to address the lack of diversity in reasoning for GUI grounding. It proposes a data synthesis pipeline and a two-stage training framework. The resulting model achieves strong performance on grounding tasks and improves the accuracy of existing methods in real-world navigation scenarios.", "questions": "1. In Line 918, is the Action Space in the SFT training prompt fixed to only one action?\n\n2. The paper states that during the SFT stage, the diversified instruction serves as the reasoning. In the actual SFT training data, is the reasoning directly the diversified instruction? If so, does the instruction then need to be replaced with a standard instruction? The construction process of the data samples is not sufficiently clear.\n\n3. In the data synthesis pipeline, the diversified instruction is used as reasoning. Is the instruction used for training also augmented accordingly?\n\n4. AgentNet is a multi-step trajectory dataset that includes many non-grounding actions. How is this type of dataset converted into the Grounding format?\n\n5. Sampling of SFT data: According to Equation (1), the final reasoning used for training is sampled from four perspectives. Why not use the full dataset directly for training?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This paper offers a novel perspective: instruction and reasoning diversity.\n2. The overall presentation of the paper is fairly clear.\n3. The experimental section is comprehensive.", "weaknesses": "1. One of the core contributions of the paper is the construction of a data synthesis pipeline. However, the description in Section 3 remains unclear:\n\n    1.1 There is a lack of description of the raw dataset formats. Although Tables 9 and 10 in the Appendix describe the datasets used for training, they do not provide the original dataset details. For example, what is the original size of OSAtlas, and which subset of OSAtlas is used? In addition, the dataset meta information though full pipeline is unclear. For example, how to process the multi-step dataset AgentNet. How much data is remained at each stage.\n\n\n    1.2 The formalization of the training data synthesis pipeline is missing. Since most grounding datasets consist of a single image paired with multiple element annotations, the statement on Line 215, “For each data instance, the model receives the screenshot with the highlight,” is ambiguous. Does “data instance” refer to one image together with its set of element annotations, or to a single element?\n\n2. The analysis of the online evaluation experiments is relatively weak. While the overall performance improves, it is unclear which specific capabilities are enhanced; more detailed analysis is needed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qCoN1OUy9m", "reviewer": "Reviewer_iJUH", "summary": "This paper is well-written. It addresses the task of GUI grounding. The authors make two primary observations:\n\n1. Existing GUI grounding datasets suffer from a high flaw rate (a manually verified 23.3%).\n2. Current models fail to leverage \"instruction diversity,\" the concept that a single intent can be expressed from multiple perspectives (e.g., by appearance, function, or location). The authors demonstrate that simply exploiting this diversity at inference time can yield up to a 76% relative performance improvement.\n\nBased on these insights, the paper introduces the \"Instruction as Reasoning\" (IR) paradigm, which treats instructions not as static inputs but as dynamic, selectable \"analytical pathways\". To implement this, the authors propose a high-fidelity data pipeline to clean existing datasets and augment them with multi-perspective instructions. They then introduce a two-stage training framework. The resulting models, UI-Ins-7B and UI-Ins-32B, achieve new SOTA performance on five challenging benchmarks.", "questions": "- Have you explored any other online agentic benchmark and applied your model on different platforms (e.g., Web/Desktop, etc.)?\n- Could you share more details of the rationale behind the design of the four components (appearance, functionality, location, and intent) during the data augmentation? Have you done any ablation studies on different components and evaluated their significance?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper presents solid empirical foundation. The preliminary analyses in Section 2 are data-driven conclusions from well-designed controlled experiments. The solid investigation provides an irrefutable \"why\" for the rest of the paper.\n- The \"Instruction as Reasoning\" (IR) paradigm is a key conceptual contribution. It moves beyond generic \"think step-by-step\" reasoning and proposes a *structured* form of reasoning that is directly derived from the problem's multimodal context.\n- The SFT+RL two-stage framework is an elegant solution. It correctly identifies that the model must first learn *how* to perform a new skill (SFT for reasoning generation) before it can learn *when* to apply it (RL for reasoning selection) .\n- The method is not just elegant; it is highly effective. It achieves SOTA on five distinct grounding benchmarks, with particularly strong gains on \"advanced\" and \"implicit\" instructions, indicating a deeper semantic understanding. The SOTA on AndroidWorld proves that these offline grounding gains translate directly to improved online agent performance.\n- A significant finding is the model's emergent ability to synthesize multiple perspectives and generate entirely new analytical angles not seen in the SFT data. This suggests the SFT+RL framework doesn't just teach the model to memorize four perspectives but encourages a more general and flexible reasoning capability.", "weaknesses": "- In 3.2 Multi-Perspective Instruction Augmentation, you mention that you leverage GPT-4.1 (OpenAI, 2025) to generate new instructions from the four fundamental analytical perspectives identified: appearance, functionality, location, and intent. However, you haven’t presented further analysis on the crafted dataset. The paper lacks a systematic comparison between the original dataset and the augmented one to prove the effectiveness.\n- This paper only conducts experiments on one online agent benchmark, especially in the context of Android. This might be insufficient to prove the performance of the UI-Ins in dynamic GUI environments", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Y6SBrPtbEE", "reviewer": "Reviewer_UPmG", "summary": "This paper addresses the GUI grounding task by proposing a novel paradigm called \"Instruction as Reasoning\" (IR). The authors argue that existing methods treat instructions as static user intent, overlooking the value of instruction diversity. The core contribution is a two-stage training framework: first, using Supervised Fine-Tuning (SFT) on a cleaned and augmented dataset (containing multi-perspective instructions) to teach the model to generate intermediate reasoning paths; second, employing a Reinforcement Learning (RL) stage to optimize the model to autonomously select the best reasoning path for a given task. The authors introduce the UI-Ins-7B and UI-Ins-32B models, achieving SOTA performance on five major benchmarks and demonstrating an emergent ability to combine and generate novel reasoning paths in new scenarios. I think this is a well-motivated and experimentally sound paper.", "questions": "1. **Rationale for Selecting GRPO**  \n   Provide more justification for using GRPO in the RL stage. Include comparisons or discussions with other popular RL algorithms to clarify the advantages or unique suitability of GRPO for this task.\n\n2. **Qualitative Analysis of Emergent Reasoning**  \n   Expand the qualitative analysis to better illustrate the phenomenon of \"emergent reasoning.\" More detailed case studies are needed to show when and why the model synthesizes multiple perspectives or formulates entirely novel instructional perspectives.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The models achieve SOTA performance on five benchmarks, including MMBench-GUI L2 and UI-I2E-Bench.\n2. The \"Instruction as Reasoning\" (IR) paradigm is novel. A key insight of the paper is that traditional \"Free-Form Reasoning\" (FFR) may degrade grounding performance, while the proposed structured IR significantly improves it.\n3. The proposed two-stage SFT+RL (GRPO) framework is methodologically sound. The SFT stage effectively teaches the model to understand and generate diverse reasoning paths, while the RL stage optimizes the path selection strategy.\n4. The experimental evidence supporting the method is convincing. The ablation studies are strong, validating the necessity of both the SFT and RL stages, the importance of the intermediate reasoning step, and the role of IR-SFT in stabilizing RL training.", "weaknesses": "1. **Choice of GRPO**  \n   The paper employs GRPO for the RL stage but does not sufficiently justify why GRPO was chosen over other reinforcement learning algorithms. A comparative discussion or empirical evidence supporting this choice would strengthen the methodological soundness.\n\n2. **Definition of \"Optimal\" Selection**  \n   The paper claims that the RL stage learns to select the \"optimal\" analysis perspective. However, there is insufficient analysis explaining:\n   - Under which contexts the model selects specific perspectives  \n   - Why these selections are considered \"optimal\"  \n   \n   A deeper examination of selection patterns and their correlation with performance would help substantiate this claim.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)