GuirlVG: Incentivize GUI Visual Grounding via Empirical Exploration on Reinforcement Learning
Authors: Weitai Kang
Bin Lei
Gaowen Liu
Caiwen Ding
Yan Yan
Venue: ICLR 2026 Poster (2026)
Keywords: GUI Visual Grounding
Reinforcement Fine-Tuning
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Graphical user interface visual grounding (GUI-VG)—a core capability for GUI agents—has primarily relied on supervised fine-tuning (SFT) of multimodal large language models (MLLMs), demanding extensive data curation and significant training costs. However, as MLLMs continue to advance and even cover GUI domains during pretraining, the necessity of exhaustive SFT post-training becomes increasingly questionable. Meanwhile, the recent successes of rule-based reinforcement fine-tuning (RFT) suggest a more efficient alternative. However, despite its promise, the optimal manner of RFT for GUI-VG remains unexplored. To bridge this gap, we introduce GuirlVG, a reinforcement learning–based GUI-VG method built on a systematic empirical study and a novel stabilization technique. Preliminarily, we find that naive application of RFT underperforms the SFT baseline, motivating a deeper exploration of RFT. First, we decompose RFT into its core components and analyze the optimal formulation of each. Second, as part of this exploration, we propose a novel Adversarial KL Factor that dynamically stabilizes training to mitigate reward over-optimization. Third, we further explore the training configurations of RFT to enhance the effectiveness. Extensive experiments show that GuirlVG, with only 5.2K training samples, outperforms SFT methods trained on over 10M samples, achieving a +7.7% improvement on ScreenSpot, a +17.2% improvement on ScreenSpotPro and 91.9% accuracy on ScreenSpotV2.

Published as a conference paper at ICLR 2026
GUIRLVG: INCENTIVIZE GUI VISUAL GROUNDING VIA EMPIRICAL EXPLORATION ON REINFORCEMENT LEARNING
Weitai Kang' Bin Lei Gaowen Liu Caiwen Ding Yan Yan University of Illinois Chicago Uniiveriity of Minnesota "Cisco Research
ABSTRACT
Graphical user interface visual grounding (GUI-VG) core capability for GUI agents-han has primarily relied on supervised fine-tuning (SFT) of multimodal large language models (MLLMs), demanding extensive data curation and significant training costs. However, as MLLMS continue to advance and even cover GUI domains during pretraiming, the necessity of exhaustive SFT post-training becomes increasingly questionable. Mcanwhile, the recent successes of rule- based reinforce- ment fine-tuning (RFT) suggest more efficient alternative. However, despite its promise, the optimal manner of RFT for GUI-VG remains unexplored. To bridge this gap, we introduce GuirlVG, reinforcement learning-based GUI-VG method built systematic empirical study and novel stabilization technique. Prelim- inarily, we find that naive application of RFT underperforms the SFT baseline, motivating deeper exploration of RFT. First, decompose RFT into its core components and analyze the optimal formulation of each. Second, as part of this exploration, we propose novel Adversarial KI. Factor that dynamically stabilizes training to mitigate reward over-optimization. Third, we further explore the training configurations of RFT to enhance the effectiveness. Extensive experiments show that GuirlVG, with only 5.2K training samples, outperforms SFT methods trained on over 10M samples, achieving a +7.7% improvement on ScreenSpot, a +17.2% improvement on ScreenSpotPro and 91.9% accuracy on ScreenSpotV2.
Accuracy (ScreenSpot)
88.0%
88.7%
87.4%
85.6%
5.2K Data
81.0%
83.4%
13.58M Dota
73.3%
79.9%
79.2%
1.3M Data
53.4%
2K Data
364K Data
RFT (Trivial)
SeeClick
UGround
OS-Atlas
SRF
Point &
Tune
Adv.
Img.
Train
In-Bbox
KL
Res.
Steps
Previous SFT Methods
Exploration Trajectory GuirIVG
Figure 1: Step-by-step exploration of GuirlVG. Starting from trivial RFT, we progressively add Soft Reward Function, In- In-Bbox reward with point prediction, / tuning, our Adversarial KI. Factor, image resolution prompting. and extended training. With only 5.2K data, GuirlVG surpasses SFT methods trained on up to 13.58M data. Circle size reflects data scale used by each method.
INTRODUCTION
Graphical user interface (GUI) agents [Gou et al., 2024, Lin et al., 2024, Cheng et al., 2024, Qin et al., 2025, Xu et al., 2024, Huang et al., 2025b, Lei al., Wu al., 2024, Hong et al., 2024), empowered

---

Published as a conference paper at ICLR 2026
by the rapid advancement of foundation models or multimodal large language models (MLLMs) [Liu et al., 2024b, Wang et al., 2024, Bai et al., 2025a]. are increasingly capable of perceiving and acting within digital environments via screenshots. A core capability underpinning such agents is GUI visual grounding
the task of localizing actionable elements in a screenshot conditioned on a textual instruction [Gou et al., 2024, Cheng et al., 2024, Lei et a..]. Recent efforts have primarily approached GUI-VG through post-training of MLLMs via supervised fine-tuning (SFT), a paradigm that demands large-scale domain-specific data curation and significant training resources [Wu et al., 2024, Cheng et al., 2024, Qin et al., 2025, Gou et al., 2024, Lei et al.]. These advancements co-evolve with MLLM's 's capabilities, tailoring each generation of MLLMs to GUI-centric benchmarks.
However, SFT raises critical concerns regarding efficiency. As MLLMs continue to improve in general perception and reasoning- wwith some already ingesting GUI-related data during pretraining [Wang et al., 2024, Bai et al., 2025a]- the necessity of extensive post-training becomes increasingly ques- tionable. Given the persistent training cost incurred with each new MILLM generation, a fundamental question arises: Does exhaustive SFT remain the most effective post-training strategy?
Meanwhile, the success of rule-based reinforcement fine-tuning (RFT) by Group Relative Policy Optimization (GRPO) [Shao et al., 2024a] in DeepSeek-R1 [Guo et al., 2025] inspires new directions. Recent methods transfer GRPO to visual grounding domains [Yuan et al., 2025, Luo et al., 2025, Shen al., 2025, Bai al., 2025b] with notable improvements. Despite these advances, prior work has systematically studied RFT for GUI-VG. In fact, our results even reveal that naive application of RFT to GUI-VG under fair experimental settings underperforms the SFT baseline, prompting critical question: What is the optimal formulation of RFT objectives for GUI visual grounding?
In this paper, instead of purely pursuing the best performance, we focus on step-by-step and fair ablation to obtain rigorous findings that provide insighls into how to design RFT for GUI-VG. We do not compare with other RFT- GUI-VG methods, since differences in data, training, and models would yield limited rigorous conclusions in systematic experiments. We elaborate on this point in section 2.1. We introduce GuirlVG, RFT-based GUI-VG design built upon our comprehensive em- pirical study and a novel stabilization technique toward GRPO. 1 We begin by deconstructing GRPO into its core components- format reward, accuracy reward, and KL penalty- and systematically ablate each component to derive an optimal configuration. @ To further address over-optimization caused by reward functions, we introduce a novel Adversarial KL Factor, which dynamically scales explore
the KL penalty based rewards to stabilize the learning process. ① Additionally, wide range of training setups, including hyperparameter tuning, LoRA enablement, and prompt engineering, to uncover best practices for effective RFT on GUI-VG. O Finally, we conduct extensive experiments on SpreenSpot [Cheng et al., 2024], ScreenSpot [Wu et al., 2024], and ScreenSpot- Pro [Li et al., 2025], demonstrating that GuirlVG achieves state-of-the-art results using few as 2K~~5.2K 2K training examples. Compared to prior SFT baselines trained on hundreds of thousands to over ten million data, our method achieves superior accuracy with up to +17.2% % absolute gains on ScreenSpotPro, highlighting the data efficiency and strong effectiveness of GuirlVG.
2 BACKGROUND
2.1 RELATED WORK
Why do we need empirical studies? While prior works [Yuan et al., 2025, Luo et al., 2025, Shen et al., 2025, Bai al., 2025b] have proposed various modeling choices for RFT-based GUI-VG, these advances often emphasize reward function novelty or performance improvements without a systematic examination of underlying design factors. As GUI-VG continues to evolve rapidly, such one-off comparisons offer limited guidance for practitioners, since conclusions are often confounded by differences in data, training setups, and model structure. Empirical studies fill this gap by providing controlled and transparent analyses that isolate the effect of specific design choices. This type of investigation is essential for moving beyond ad-hoc innovation toward principled understanding, enabling the community to identify robust practices and avoid misleading interpretations of perfor- mance gains. There are some pioneer works for empirical studies in Multimodal Large Language Model, e.g. 1.5 [Liu et al., 2024a], Prismatic [Karamcheti et al., 2024], Eagle [Shi et al., 2024] and Idefics2 [Laurençon et al., 2024]. However, there remains a lack of empirical studies investigating RFT in the context of GUI-VG.
2

---

Published as a conference paper at ICLR 2026
GUI Visual Grounding. Visual grounding ability in graphical user interfaces [Cheng et al., 2024, Gou et al., 2024, Lei et al.] has become one of the main bottlenecks for AI agents [Gou et al., 2024, Lin et al., 2024, Cheng et al., 2024, Qin et al., 2025, Xu et al., 2024, Huang et al., 2025b, Lei et al., Wu et al., 2024, Hong et al., 2024]. To address this, SeeClick [Cheng et al., 2024] introduces a large-scale pretraining pipeline for GUI-VG and proposes automated method to generate training data. Similarly. UGround [Gou et al., 2024] utilizes synthesized web-based data to support grounding training, and AGG [Lei et al.] builds a dedicated engine collect extensive GUI images with annotations. OS-Atlas [Wu et al., 2024] further expands grounding data across multiple operating systems. TARS [Qin al., 2025] combines GUI-centric pretraining with task condiiondt fine- tuning to improve alignment between perception and reasoning. Despite the variety in their data construction, these methods commonly adopt the supervised fine-tuning (SFT) paradigm, which relies heavily on large volumes of high-quality labeled training data.
Reinforcement Fini-Tuning. Rule-based Reinforcement Fine-Tuning (RFT) with Group Relative Policy Optimization (GRPO) [Shao al., 2024a] has recently demonstrated effectiveness across different domains [Shao et al., 2024b, Liu & Zhang. 2025, Wang* et al., 2025, Guo et al., 2025]. Unlike supervised fine-tuning (SFT), which enforces token-level supervision strictly corresponding to the answer, RFT encourages models to freely explore their reasoning process and provides supervision only the level of the final output. This more flexible objective incentivizes stronger reasoning capabilities [Guo et al., 2025]. Furthermore, in the RFT algorithm- GRPO, task-specific rule- based reward functions are designed to provide supervision signals that are automatically verifiable. This eliminates the need for training separate critic model [Schulman et al., 2017, Ouyang et al., 2022] or relying on human feedback [Kaufmann et al., 2023], thereby mitigating the risk of reward hacking [Weng, 2024] and making RFT an effective alternative to SFT.
2.2 PRELIMINARIES
Group Relative Policy Optimization (GRPO). Given a task input which additionally specifies a particular response format in the prompt, i.e. instructing the model reason within think >< think > tags and answer within < answer >< /answer > tags, the model generates group of N candidate responses {01, .z...., ... ON J. Each candidate evaluated using a rule-based reward function, yielding rewards (21,72,.... .NN) }. For each response O2, this rule-based reward function scores two rewards: format reward, r/ which assesses whether the response adheres to the instructed tag structure, and an accuracy reward, rf. which evaluates the correctness of the response, such as classification accuracy [Chen al., 2025] or intersection-over--nion (IoU) in detection tasks [Huang et al., 2025a, Liu et al., 2025b]. The total reward for is computed as
r..
(2.1)
The relative reward (also referred to as the advantage Ad) of the i-th candidate is computed by normalizing the rewards within the group of candidate responses:
A; - P2.
(2.2)
Std((ri,
where Mean(-) and Std(-) denote the mean and standard deviation, respectively. To stabilize training, GRPO additionally constrains model update by minimizing the KL divergence between the current model and reference model (typically the original model). Thus, the objective J, to maximize for each candidate becomes
A, - -- DxL (o; || o.
(2.3)
where hyperparameter controlling the KL penalty strength, and o; is the corresponding response from the reference model. We omit details, such as clipping, averaging, etc.
Implementation. Unless specified otherwise, we fine-tune Qwen2.5-Ve VL [Bai et al., 2025a] using LoRA [Hu et al., 2022] with rank of 64 and an alpha of 128, while keeping the vision module frozen. Training data are randomly sampled from ShowUI [Lin et al., 2024], which crawls visually rich website data and augments desktop data from OmniAct [Kapoor et al., 2024] using GPT-40 [Hurst et al., 2024]. The group size of candidate responses, N, is set to 6, and the batch size is set to 4. The KI. divergence coefficient (13) is set 0.04 by default. The learning rate is set to 1 x 10-5, with two
3

---

Published as a conference paper at ICLR 2026
training epochs, AdamW optimizer, and a linear decay schedule. We 6XNVIDIA A100-80G GPUs for training. For the SFT baseline, we adopt LLaMA Factory [Zheng et al.. 2024] with the same training configurations for a fair comparison. For the efficiency and fairness of experiments, we report performances at step 500 for both RFT and SFT, where convergence is typically observed. Training beyond 500 steps yields only marginal improvements, with our final version reaching peak performance around step 1,300. Accordingly, our final version is only trained on 5,200 samples.
Evaluation Suite. We evaluate on three widely-used GUI-VG benchmarks across different plat- forms: ScreenSpot [Cheng et al., 2024]. ScreenSpot v2 [Wu et al., 2024]. and ScreenSpot-PPe [Li al., 2025]. ScreenSpot evaluates GUT grounding capabilities across mobile, desktop, and web environments, while ScreenSpot v2 improves evaluation reliability by correcting annotation errors. ScreenSpot-Pe focuses on high-resolution professional scenarios, featuring expert-annotated tasks spanning 23 applications, five industries, and three operating systems. All benchmarks report the accuracy of whether the predicted point coordinate falls inside the ground truth bounding box of the corresponding element in the screenshot
METHODOLOOGY
3.1 CAN TRIVIAL ADOPTION OF RFT BEATS SFT?
We begin by comparing the SFT baseline with a trivial adoption of RFT for GUI-VG. Specifically, adopt the commonly used implementation from HuggingFace [2025], Shen et al. [2025], using the following prompt for a given description of the target element:
Please provide the bounding box coordi- nates [xl, yl, x2, y2] of specific element based on this sentence: <description>, First, think through the reasoning process within <think> < /think> tags. Then, out- put the bounding box coordinates in JSON format within <answer>> < /answer>> tags.
Table 1:
Comparison of zero-shot, SFT, and trivial RFT on ScrcenSpot (Qwen2.5-VL, 500 training steps).
Method
Backbone
Step
Acc (%)
Zero-Shot
Qwen2.5-VL
500
72.6
SFT
Qwen2.5-VE VL
500
82.6
RFT (trivial)
Qwen2.5-VL
500
79.2
For the format reward, value of is assigned if the output exactly matches the pattern "<ttink>>.. //answer>", and otherwise. The accuracy reward assigns a bounding box (bbox) array enclosed in square bracket is detected and the IoU between the predicted and ground-truth bboxes exceeds 0.5, and O otherwise. During inference, the center of the predicted bbox is used as final prediction. Due to space limitations, RFT (trivial), along with the implementation details the SFT baseline and the zero-shot setup for Qwen2.5-VI. in section A. 1. As shown in table 1, both SFT and trivial RFT lead to improvements over the zero-shot baseline, but RFT (trivial) does not outperform SFT.
/think>...<aswerr.
provide the detailed pseudo-code of
Finding 1. Careful design of rule-based reinforcement fine-tuning, beyond common practice, is necessary for effectively improving GUI visual grounding performance.
3.2 HOw TO DESIGN REWARD FUNCTIONS IN GRPO?
As defined earlier, the default format reward enforces exact tag matching, while the accuracy reward relies strict JSON-style output consistent with the model's pretraining. The model is sharply penalized (rewarded 0) if any part of the expected structure missing such as an omitted < answer>>> tag-or or it has minor style deviation, e.g., outputting coordinates as a tuple instead of a JSON list. This rigid design introduces training noise and instability, even when the model successfully performs reasoning and answering.
Table 2: Compare the default reward function and our SRF on ScreenSpot (Qwen2.5-V6 VL, 500 training steps).
To address this, we propose the Soft Reward Function (SRF), which pro- vides partial credit to the presence of each tag and relaxes output style. Specifically, SRF removes the JSON requirement from the prompt. For the
Method
Backbone
Step
Acc (%)
Default
Qwen2.5-VL
500
79.2
SRF (Ours)
Qwen2.5-VL
500
79.9
4

---

Published as a conference paper at ICLR 2026
Cormatrewart SRF assigns +0.5 for each of <think> and </think>, +1/3 for each of <answer> and </answer>, and +1/3 if the content inside the answer tags contains the correct number of coordinates. The total score is normalized [0, I]. For the accuracy reward, SRF ignores style and simply extracts numeric values present in the output. Detailed prompts and pseudo-code are provided in section A.2 due space constraints. As shown in table 2, SRF provides +0.7% improvement the default reward function.
Finding 2. Looser reward functions in RFT with fractional reward better support stable RFT training, and strict adherence to pretraining-styy output is not necessary.
3.3 How TO DESIGN MODEL PREDICTION FORMAT ALONG WITH ITS ACCURACY REWARD FUNCTION?
The goal of GUI visual grounding is to predict point that falls within the target element to enable the downstream action. To support this functionality, the most direct design is to predict a point and assign binary reward based on whether it lies within the ground-trute bounding box [Shen et al., 2025] (In-Bbox). Alternatively, one can define reward based on a distance threshold & [Liu et al., 2025a], where the point prediction is rewarded with I if it falls within & pixels of the target center (denoted as Distance @ k.). Another option is to output a bounding box and derive point prediction from its center, evaluating via IoU with the ground truth. This be used continuous reward or a thresholded one (e.g., I.5@0.5 gives a reward of I if IoU > 0.5, and otherwise, as in our default format).
Table 3: Comparison of different prediction formats and accuracy reward functions under SRF on ScreenSpot (Qwen2.5-VL,500 training steps).
Prediction
Reward
Backbone
Step
Acc (%)
Bbox
loU@0.5
Qwen2.5-VI. VL
500
79.9
Bbox
loU
Qwen2.5-VI.
500
81.6
Point
Distance @ 80
Qwen2.5-VI.
500
82.7
Point
In-Bbox
Qwen2.5-VL
500
83.4
Building on our Soft Reward Function, we evaluate four configurations. The threshold of 80 for Distance @K & is empirically selected for best performance. As shown in table Point prediction with In-Bbox performs best.
Finding The most effective RFT design is one that aligns directly with the task's functional goal- specifically, point prediction with in-bounding box reward for GUI-VG.
3.4 HOW TO BALANCE THE KL PENALTY IN GRPO?
In GRPO, the KL penalty term enforces the current model to stay close to the original model, mitigating swward-driven over-optimization [Shao et al., 2024a, Guo et al., 2025]. The hyperparameter plays a critical role in determining the strength of this regularization. In our experiments, we observed that model performance highly sensitive this parameter.
We first empirically explore the effect of different values for 3, then introduce a novel strategy we call Adversarial KI. Factor, which dynamically scales the KI. penalty based reward strength. The intuition is that high-reward responses are more likely to cause over-optimization in GRPO. However, the KL penalty with the original model does not necessarily increase proportionally, especially when the original model itself assigns high probability to such responses. Therefore, static KL term may fail to counterbalance the effect of reward. To address this, we define the Adversarial KL. Factor as the ratio of the reward to its theoretical maximum and use it as multiplicative modifier to B to scale the KL penalty proportionally. This dynamic formulation ensures that as reward increases, the regularization also strengthens adaptively. The modified GRPO objective is:
Jj = A; - DkL( orig) A,
9; Mean(
Pt (3.1)
0; =
(2..,,
m
where 2 is the maximum possible reward under our setup.
5

---

Published as a conference paper at ICLR 2026
Table Comparison of different KL settings under SRF, point prediction, and In In-Bbox reward on ScreenSpot -0wen2.5-VL,, 500 training steps).
Adversarial
3
Backbone
Step
Acc (%)
X
4e-2
Qwen2.5-Vi VL
500
83.4
0
Qwen2.5-VL VI
500
84.7
/
le-4
Qw
500
85.6
le-4
Qwen2.5-VL
500
87.4
/
le-6
Qwen2.5-VL
500
77.5
Results shown in table Simply tuning provides clear performance improvements, demonstrat- ing the importance of empirically calibrating the KL penalty. Notably, our Adversarial KL Factor strategy (row 4) achieves a substantial +1.8% gain over the best 3 baseline (row 3), validating the advantage of dynamically adjusting KL strength in response to reward magnitude. Row 5 further indicates that setting 3 too small results degraded performance.
Finding 4. GRPO is sensitive the strength of the KL penalty, which requires empirical exploration. Our Adversarial KL Factor dynamically balances this penalty, leading to optimal performance.
3.5 SHOULD WE FULLY FINE-TUNE THE MODEL OR USE LORA?
We further investigate the impact of tinettunigg strategies by comparing full model fine-tuning (Full- FT) with LoRA [Hu et al., 2021] fine-tuning (LoRA-FT) applied to the LLM component. In practice, we observe that full fine-tuning tends to destabilize training unless a much smaller learning rate is used. Therefore, we reduce the learning rate for full fine-tuning 1 x 10-6, while keeping other hyperparameters consistent. We also report the training time per iteration using 6x A6000 GPUs.
Table 5: Comparison of Full-FT and LoRA-FT under SRF, point prediction, In- Bbox reward, B 1 x 10-4, and Adversarial KL Factor ScreenSpot (Qwen2.5-VL, 500 training steps). Training time is reported per iteration over 6x A6000 GPUs.
Config
Backbone
Step
Time
Acc (%)
Full-FT
Qwen2.5-VL
500
749.4 s
87.5
LoRA-FT
Qwen2.5-VL
500
28.4 s
87.4
As shown in table 5, Full-FT requires over 25 times more training time per iteration compared to LoRA-FT, while yielding only a marginal improvement of +0.1%. Given this modest performance gain relative to the substantial increase in computational cost, we adopt LoRA-FT as more efficient strategy for GUI-VG reinforcement fine-tuning in our study.
Finding 5. LoRA offers comparable performance to full fine-tuning while being significantly more efficient, making it practical choice for GUI-VG with reinforcement fine-tuning.
3.6 How TO DECIDE THE GROUP SIZE AND BATCH SIZE IN GRPO?
The hyperparameters group size and batch size play critical roles in GRPO [Shao et al., 2024a]. Specifically, group size affects the normalization of advantage estimates, while batch size determines how each sample contributes to the final objective function. Therefore, it necessary to empirically examine how different configurations of these two hyperparameters impact the final performance.
As shown table 6, the configuration with group size and batch size 4 achieves the highest accuracy, which is our default setting. Interestingly, increasing the group size from 6 to 8 leads substantial performance drop, even though larger groups theoretically provide better baseline estimates for advantage in GRPO to serve
stable substitute for the critic model in PPO [Schulman et al.,

---

Published as a conference paper at ICLR 2026
2017]. This counterintuitive result suggests that RFT is sensitive to seemingly minor changes in implementation details and highlights the need for systematic validation of hyperparameter choices.
Table 6: Effect of group size and batch size under SRF, point prediction, In In-Bbox reward,
1 x 10 Adversarial KL Factor and LoRA on ScreenSpot (Qwen2.5-VL, 500 training steps).
Group
Batch
Backbone
Step
Acc (%)
6
1
Qwen2.5-VL VL
500
86.5
6
4
Qwen2.5-VL
500
87.4
8
4
Qwen2.5-VL.
500
83.9
Finding 6. GRPO performance varies significantly with group and batch size configurations,
highlighting the importance of empirical hyperparameter tuning.
3.7 How TO INVOLVE IMAGE RESOLUTION INFORMATION IN THE PROMPT?
Prompting image resolution may provide additionally helpful context, especially for high-resolution GUT screenshots. We explore when such information should be incorporated into the prompt. Specifically, compare three strategies: (1) never provide resolution; (2) provide resolution during both training and testing; (3) provide resolution only at test time. When resolution is included, we prepend the prompt with "The screenshot resolution is {width) (height)."
Table 7: Effect of image resolution in the prompt under SRE, point prediction, In-Bbox reward, LoRA, groupsize = 6, and batchsize 4 on ScreenSpot (Qwen2.5-VL, 500 training steps).
Train
Test
Backbone
Step
Acc (%)
/
Qwen2.5-VV VL
500
83.7
x
Qwen2.5-VL
500
87.4
X
V
Qwen2.5-VL
500
88.0
As shown in table 7, the highest accuracy achieved when resolution information is excluded during training but added at test time. We hypothesize that withholding resolution during training may challenge the model learn better spatial reasoning ability. At test time, the additional resolution context then serves as a useful signal to refine predictions.
Finding 7. Withholding the cue of image resolution during training fosters better learning, while providing at test time proves beneficial.
3.8 FInAL DESIGN CHOICES FOR RFT ON GUI-VG
Based on the studies above, finalize set of design choices for effective and efficient RFT pipeline for GUI visual grounding under GRPO. We propose the Soft Reward Function (SRF) to provide partial credit for format compliance while relaxing output constraints. For the prediction format, we use direct point prediction with the In-Bbox binary reward. To stabilize training, we introduce the Adversarial KL Factor with coefficient of 1 10-4. We employ LoRA for efficient fine-tuning and set the group size to 6 and batch size to 4. Image resolution information is withheld during training and added only at inference. We train 1,300 steps for final version.
COMPARISON WITH PREIVOUS METHODS
We compare final RFT method against prior approaches across three GUI-VG benchmarks introduced in section 2.2: ScreenSpot [Cheng et al., 2024], ScreenSpot v2 [Wu et al., 2024], and ScreenSpot-Pro [Li al., 2025].
7

---

Published as a conference paper at ICLR 2026
Table 8: Comparison of various models on ScreenSpot. The optimal result is bolded. "Size" refers model size. "#Train" refers to training samples.
Mobile Text
Desktop
Web
Method
Size
#Train
Avg.
Icon
Text
Icon
Text
Icon
Fuyu [Bavishi al., 2023]
8B
41.0
1.3
33.0
3.6
33,9
4.4
19.5
CogAgent [Hong et al., 2023]
18B
400K
67.0
24.0
74.2
20.0
70.4
28.6
47.4
SeeClick [Cheng ul., 2024]
9.6B
364K
78.0
52.0
72.2
30.0
55.7
32.5
53.4
AGG ct.]
0.4B
35M
86.1
62.8
81.8
46.2
74.2
48.4
66.6
OmniParser [Lu et al., 2024]
93.9
57.0
91.3
63.6
81.3
51.0
73.0
U/Ground [Gou et al,, 2024]
7B
1.3M
82.8
60.3
82.5
63.6
801.4
70.4
73.3
ShowUl-G [I.in et al., 2024]
2B
119K
91.6
69.0
81.8
59.0
83.0
65.5
74.9
ShowUl [Lin al., 2024|
2B
256K
92.3
75.5
76.3
61.1
81.7
63.6
75.1
OS-Atlas [Wu et al., 2024]
4B
13.58M
85.7
58.5
72.2
45.7
82.6
63.1
68.0
OS-Atlas [Wu al., 2024]
7B
13.58M
93.0
72.9
91.8
62.9
90.9
74.3
81.0
GuirlVG GuirVG
7B
2K
96.3
86.0
93.3
77.1
91.7
83.5
88.0
7B
5.2K
96.0
84.7
92.8
80.0
92.6
85.9
88.7
Results the ScreenSpot benchmark are shown in table 8. Our method substantially outperforms previous methods that rely on supervised fine-tuning (SFT), despite using significantly fewer training samples. Specifically, while prior SFT methods trained on hundreds of thousands million examples, our RFT method achieves superior performance with just 2K training samples. For example, we outperform OS-Atlas-which uses 6. 6.79K times more data-by +7.0% % in accuracy, highlighting the efficiency and effectiveness of RFT as post-training strategy. When increasing training to 1300 steps using 5.2K training samples, our method achieves further improvements. outperforming OS-Atlas by +7.7%. Notably, on the Mobile-Icon subset, our method exceeds OS- Atlas by +11.8% despite our training data containing no mobile-specific samples. This suggests that RFT enhances out-of-domain reasoning capabilities, aligning with the claim from Chu et al. [2025] that "SFT memorizes, RL generalizes."
ten
Table 9: Comparison of various models ScreenSpot v2. The optimal result is bolded. "Size" refers to model size. "#Train" refers to training samples.
Mobile
Desktop Text
Web
Method
Size
#Train
Avg.
Text
Icon
Icon
Text
Icon
SeeClick [Cheng et al., 2024]
9.6B
364K
78.4
50.7
70.1
29.3
55.2
32.5
55.1
OS-Atlas [Wu et al., 2024]
4B
13.58M
87.2
59.7
72.7
46.4
85,9
63.1
71.9
OS-Atlas [Wu et al., 2024]
7B
13.58M
95.2
75.8
90.7
63.6
90.6
77.3
84.1
GuirVVG
7B
2K
99.3
89.6
94.8
72.9
95.7
83.3
90.9
GuirlVG
7B
5.2K
98.3
89.6
94.3
80.7
95.7
86.2
91.9
Results ScreenSpot v2 (table 9) mirror the trends observed ScrcenSpot. With only 2K training examples, our method surpasses all previous methods, and with 5.2K examples, it reaches new state-of-the-art of 91.9% % average accuracy-+7.89 % higher than OS-Atlas (7B). Performance gains are consistent across all subdomains, reaffirming the generalization strength of our RFT pipeline.
Table 10: Comparison of various models on ScreenSpot-Pe Pro. The optimal result is bolded.
Development
Creative
CAD
Scientifie
Office
OS
Model
Avg
Text
Icon
Text
Text
Icon
SeeClick [Cheng al., 2024]
0.6
0.0
1.0
0.0
2.5
0.0
3.5
0.0
1.1
0.9
2.8
0.0
1.1
OS-Atlas-4B [Wu al., 2024]
7.1
0.0
3.0
1.4
2.0
9.0
5.1
3.8
5.6
0.0
3.7
ShowU/1-B i1 et al., 2024]
16.9
1.4
0.0
2.5
13.2
15.3
10.3
2.2
7.7
CogAgent- 18B [Hong et al., 2023]
14.9
0.7
9.6
0.0
7.1
3.1
22.2
1.8
13.0
0.0
5.6
0.0
7.7
Aria GUI (Yagg al., 2024]
0.0
23.7
UGround-7B [Gou et al., 2024]
26.6
2.1
27.3
2.8
14.2
1.6
31.9
2.7
31.6
11.3
17.8
0.0
16.5
OS-Atlas-7B [Wu et al., 2024]
33.1
1.1
28.8
12.2
37.5
33.9
27.1
GuirlV-22K-7B GuirVVGSS.2K-B
57.8
9.0
38.9
10.5
26.9
7.8
44.4
14.5
57.
22.6
39.3
14.6
31.6
64.9
7.6
42.9
11.2
28.9
9.4
63.9
16.4
63.8
26.4
43.9
13.5
36.1
Finally, results on ScreenSpot-Pree (table 10) demonstrate the strong generalization of our method to high-resolution, professional GUIs.
With only 2K training examples, our approach already
8

---

Published as a conference paper at ICLR 2026
outperforms all prior methods by large margin, achieving 31.6% % average uecuracy-SSurpasiii surpassing the best SFT baseline OS-Atlas (7B) by +12.7%. Scaling up to 5.2K examples further boosts performance to 36.1% %, an absolute gain of +17.2% % over OS-Atlas. This trend is consistent across all domains, including particularly challenging ones like Creative, CAD and OS, confirming the robustness of our RFT pipeline in complex real-world scenarios.
Input
(Zoom-in around
Model Output
Processed Grounding Result
Ground Truth]
"Intel on Instagram"
<think
To find the point on Instagram, need to identify the sacial media icons typically associated with Instagram, which often the camera icon. On the webpage, there's camera Icon typically representing the Instagram photo sharing feature.
in
X
intel
</thinke
<answer> The point 525, 829 </answer>
"The Coolest And
<think
Weirdest Gadgets We
To find the element labeled "The Coolest And Weirdest Gadgets We Saw At Ifa," need to locate where this text appears on the screen. It typically would be in the titles or descriptions of content sections. Given the structure of the web page, the description might associated contextual hint like image subheading. Possible locations could the bottom of section or list articles. </thinke
15:27
Saw Ifa"
Great
visual
5:1
The Coglest and Weirdest
The
Fist iPhanos, AbPads
Gadgets We Saw at IFA
Godgets We Saw at IBA
<answer> The point 689, 940) </onswers
Figure 2: Qualitative Result of GuirlVG.
5 QUALITATIVE RESULTS
In this section, we present qualitative results to illustrate the reasoning capabilities of GuirlVG in GUI visual grounding tasks. fig. 2 shows two representative examples, each consisting of the input (left), model output with intermediate thinking steps (middle), and the final grounding result (right). The thinking process is highlighted with green color. the first example, the task is to locate icon on a webpage. GuirlVG begins by reasoning that it needs to identify social media icons, often represented by a camera icon. Recognizing the webpage context, the model correctly identifies the Instagram icon and grounds the instruction to the coordinates, as shown in the red dot in the grounding result. In the second example, the instruction is to find a text. GuirlVG reasons that the target is a text and is likely appear as a contextual hint like subheading. By analyzing the structure of the webpage, the model further reasons that the target is at the bottom of a section. These qualitative results underscore GuirlVG's textual understanding and advanced reasoning abilities, enabled by our reinforcement yearning-basee approach. By explicitly modeling the thinking process, GuirlVG not only achieves high accuracy but also provides interpretable steps, making it reliable solution for GUI-VG tasks.
6 CONCLUSION
In this work. we revisit the paradigm of post-training for GUI visual grounding and present the first comprehensive empirical study rule-based reinforcement fine-tuning (RFT) this domain. Through systematic analysis and series of targeted innovations- iiniludinniinniinn the decomposition of GRPO components, introduction the Adversarial KL Factor, and extensive tuning of training configurations- demonstrate that RFT, when properly optimized, decisively outperforms super- vised fine-tuning (SFT). Using as few as 2K training examples, our method surpasses strong SFT baselines trained orders of magnitude more data across three challenging benchmarks, achieving new state-of-te sfate-of-the-an and highlight RFT more data-efficient and generalizable alternative for GUI-VG.
the-art art performance. These findings challenge the prevailing reliance on large-scale SFT

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENT
This research is supported by NSF IIS-2525840, CNS-2432534, ECCS-2514574, NIH IRFIMH13376444 and Cisco Research unrestricted gift. This article solely reflects opinions and conclusions of authors and not funding agencies.
REFERENCES
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502. 13923, 2025a.
Sule Bai, Mingxing Li, Yong Liu, Jing Tang, Haoji Zhang, Lei Sun, Xiangxiang Chu, and Yansong Tang. Univg-rl: Reasoning guided universal visual grounding with reinforcement learning. arXiv preprint 142222,505.1231,, 2025b.
Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, and Sagnak Taşırlar. Introducing our multimodal models, 2023. URL https: //www//www///../ adept. ai/
blog/Euyu-B8l 8b.
Liang Chen, Lei Li, Haozhe Zhao, Yifan Song, and Vinci. RI-v; Reinforcing super generalization ability in vision-language models with less than $3, attpp::/giiii com R1-V, 2025. Accessed: 2025-02-02.
www/meep-eeee.eep... Deep- / Agent/
Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiy- ong Wu. Seeclick: Harnessing gui grounding for advanced visual gui agents. arXiv preprint arXiv:2401.10935, 2024.
Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc Le, Sergey Levine, and Yi Ma. Sft memorizes, r1 generalizes: A comparative study of foundation model post-training. arXiv preprint 1rXiv:2501.17161, 2025.
Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. Navigating the digital world as humans do: Universal visual grounding for gui agents. arXiv preprint arXiv:2410.05244 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et al. Deepseck-rl: Incentivizing reasoning capability in llms via reinforcement learning, arXiv preprint arXiv:2501. 2025.
Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. Cogagent: A visual language model for gui agents. arXiv preprint arXiv:2312.0888 2023.
Wenyi Hong Weihan Wang. Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. Cogagent: A visual language model for gui agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
14281, 2024.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint 1rXiv:2/06.0968: 2021.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen Yuanzhi Li, Shean Wang, Lu Wang. Weizhu Chen, al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao. Zheyu Ye, Fei Zhao, Yao Hu, and Shaohui Lin. Vision-rl: Incentivizing reasoning capability in multimodal large language models. arXiv preprint arxiv:2503. 06749, 2025a.
Zhiyuan Huang, Ziming Cheng. Junting Pan. Zhaohui Hou, and Mingjie Zhan. Spiritsight agent: Advanced gui agent with one look. arXiv preprint arXiv:2503.03/ /96, 2025b.
10

---

Published as a conference paper at ICLR 2026
HuggingFace. Open rl: A fully open reproduction of deepseek-rl, January 2025. URL https: // //github...
/ open-ri.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card, arXiv preprint arXiv:2410.22 2226,, 2024.
Raghav Kapoor, Yash Parag Butala, Melisa Russak, Jing Yu Koh, Kiran Kamble, Waseem Alshikh, and Ruslan Salakhutdinov. Omniact: A dataset and benchmark for enabling multimodal generalist autonomous agents for desktop and web, 2024.
Siddharth Karamcheti, Suraj Nair, Ashwin Balakrishna, Percy Liang, Thomas Kollar, and Dorsa Sadigh. Prismatic vlms: Investigating the design space of visually-conditioned language models. arXiv preprint arXiv:2402.07866 2024.
Timo Kaufmann, Paul Weng, Viktor Bengs, and Eyke Hüllermeier. A survey of reinforcement learning from human feedback. arXiv preprint arXiv:2312. 14925, 10, 2023.
Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision- laiionlanggage models? arXiv preprint arXiv:2405. 02246, 2024.
Weixian Lei, Difei Gao, and Mike Zheng Shou. Grounding multimodal large language model in gui world. In The Thirteenth International Conference on Learning Representations.
Kaixin Li, Ziyang Meng. Hongzhan Lin, Ziyang Luo, Yuchen Tian, Jing Ma, Zhiyong Huang, and Tat-Seng Chua. Screenspot-pro: Gui grounding for professional high-resolution computer use. arXiv preprint arXiv:2500.07988, 2025.
Kevin Qinghong Lin, Linjie Li, Difei Gao, Zhengyuan Yang. Shiwei Wu, Zechen Bai, Weixian Lei, Lijuan Wang, and Mike Zheng Shou. Showui: One Assionlangguage-attn model for gui visual agent. arXiv preprint arXiv:2411.17460 2024.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee, Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16296-26306 2024a.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 26296-26306, 2024b.
Jiawei Liu and Lingming Zhang. Code-rl: Reproducing rl for code with reliable rewards. 2025.
Yuqi Liu, Bohao Peng, Zhisheng Zhong, Zihao Yue, Fanbin Lu, Bei Yu, and Jiaya Jia. Seg- zero: Reasoning-chain guided segmentation via cognitive reinforcement. arXiv preprint 2xXiv:2503.06520. 2025a.
Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinforcement fine-tuning. arXiv preprint arXiv:2503.07785, 2025b.
Yadong Lu, Jianwei Yang, Yelong Shen, and Ahmed Awadallah. Omniparser for pure vision based gui agent. arXiv preprint arXiv:2408.00203. 2024.
Run Luo, Lu Wang, Wanwei He, and Xiaobo Xia. Gui-rl: generalist rl-style vision-language action model for gui agents. arXiv preprint arXiv:2504. 10458, 2025.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Yujia Qin, Yining Ye, Junjie Fang. Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, et al. Ui-tars: Pioneering automated gui interaction with native agents. arXiv preprint arXiv:2501.12326, 2025.
11

---

Published as a conference paper at ICLR 2026
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 2024a.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.038: 03300, 2024b.
Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, et al. Vlm-rl: A stable and generalizable rl r1-style large vision-language model. arXiv preprint arXiv:2504.076155 2025.
Min Shi, Fuxiao Liu, Shihao Wang, Shijia Liao, Subhashree Radhakrishnan, De-An Huang. Hongxu Yin, Karan Sapra, Yaser Yacoob, Humphrey Shi, et al. Eagle: Exploring the design space for multimodal llms with mixture of encoders. arXiv preprint arXiv:2408. /5998, 2024.
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing I Liu, Jialin Wang. Wenbin Ge, et al. Qwen2-vl: Enhancing lision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.1219) 2024.
Zihan Wang*, Kangrui Wang*, Qineng Wang*, Pingyue Zhang* Linjie Li* Zhengyuan Yang. Kefan Yu, Minh Nhat Nguyen, Licheng Liu, Eli Gottlieb, Monica Lam, Yiping Lu, Kyunghyun Cho, Jiajun Wu, Li Fei-Fei, Lijuan Wang, Yejin Choi, and Manling Li. Training agents by reinforcing reasoning, 2025, URL https: / github.........gg com/ ZihanWang314/ /aggen..
Lilian Weng. Reward hacking in reinforcement learnnggg httpps://lit hwwp///ggiaaawng...ommmmmmo posts/ 2024-12222 11 -ewa--eewrrd-aaee www.rrdhhhckggg. 2024.
Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang. Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, et al. Os-atlas: foundation action model for generalist gui 2024.
agents. arXiv preprint 200.v:2410.2:3
Yiheng Xu, Zekun Wang, Junli Wang. Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, and Caiming Xiong. Aguvis: Unified pure vision agents for autonomous gui interaction. arXiv preprint 211:::2111.044544 2024.
Yuhao Yang, Yue Wang, Dongxu Li, Ziyang Luo, Bei Chen, Chao Huang, and Junnan Li. Aria-ui: Visual grounding for gui instructions. arXiv preprint arXiv:2412.166556 2024.
Xinbin Yuan, Jian Zhang, Kaixin Li, Zhuoxuan Cai, Lujian Yao, Jie Chen, Enguang Wang, Qibin Hou, Jinwei Chen, Peng-Tao Jiang, et al. Enhancing visual grounding for gui agents via self-evolutionary reinforcement learning. arXiv preprint arXiv:2505. 12370, 2025.
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangehi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Pro- ceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume System Demonstrations), Bangkok, Thailand, 2024. Association for Computational Linguistics. URL htttt:p//aaxii org/ abs/2403 13372.
12

---

Published as a conference paper at ICLR 2026
A
APPENDIX
A.I ADDITIONAL DETAILS FOR SECTION 3.1
We provide additional details of the trivial adoption of RFT (RFT-trivial) and the implementation of
SFT, which contributes to the reproducibility of the results of this paper.
Algorithm Format Reward Calculation
1: function FORMATREWARDDOompeccio
2: pattern k- regex "<think> +?</think> (atatsssssaaassssaeeaa. * answer>"
3: return 1.0 if completion matches pattern else 0.0
4: end function
Algorithm 2 Accuracy Reward Calculation
1: function AcCURACYYEEAAADDcomppin
GT _oo)
2: answer answer.pattern + regex <answer> (.+?) </answer>
3: bbox toor-pattern regex (Ad+), \s* (Ad+), \s* (Ad+), As* (Xd+))
4:
reward + 0.0
5:
if completion matches answer pallern then pred bbox t- find the match in completion if length of bbod_bbox is 4 and bob, GT_ > 0.5 then reward t 1.0
6:
7:
8:
9: end if
10: end if
return reward
12:
end function
The format reward function of RFT-trivial is shown in algorithm and the corresponding accuracy reward function is shown in algorithm 2.
For the SFT baseline, we use the following prompt:
Please provide the bounding box coordinates of the region described by this sentence: <description>. The answer format is:
json \n [bbox 2d: <ground-truth bounding box'>, label: <description>> \n.
We adopt the official evaluation code of Qwen2.5-VL! obtain the zero-shot baseline performance.
A.2 ADDITIONAL. DETAILS FOR SECTION 3.2
We provide details our Soft Format Reward in algorithm 3 to help readers better understand The prompt use in section 3.2 is follow:
Please provide the bounding box coordinates [xl, yl, x2. y2] of a specific element based on this sentence: <description>. First, think about the reasoning process in the mind within <think> /think> tags. Then, output the bounding box coordinates within <answer>> < /answer'> tags.
A.3 MATHEMATICAM FORMULATION OF REWARD FUNCTIONS
We provide the formal mathematical definitions for the reward functions and design choices discussed in the main methodology sections.
https: / / thub. com /Qwen MM/ wtw.en2...11.///.... w-w..//..bbb/////.o ob/mai i in ht//coooook//cooo/ hwtpo//ooooooooottrrrroeooo 1pynb
13

---

Published as a conference paper at ICLR 2026
Algorithm 3 Soft Format Reward Calculation
1: function SOrTHoRMAATReAAADiccmmliiiin
2: score 6-
3: if "f""hhnn>>" in completion then
4:
score f- score
5: end if
6: if< /think>" in completion then
7:
score score 0.5
8:
end
9:
if full "<answer>..<< /aanswr>"" block detected then score + score 2/3
10:
if exactly two numbers found inside the block then score + score + 1/3
12:
end if
14:
"< else if "<answer>" or "< /answer>" detected then
score & score 1/3
16:
end if
17:
return score/2
normalized by the maximum possible reward
18:
end function
"Intel on Instagram" "
Lora
Multiple Responses
Ethinth fincf the point on Intoprom, need identily the social medio icons typically associated with Inttoprom, which oftee the comaro iton. On the welpoge, there's comera foo rysicaly representing the instopram
Soft Reward Function
[Format Reward)
Model
Chrck:
Provibe Image resolution? Train Cies
<thint>
</hink>>
Adversarial KL Factor
cosswer-
Train Config
c/enswe>
rumbers inside
Groups
photo shoving frature. krinins emqwoers
Point & In-Bbox (Accuracy
intel
Reference Model
Botches
stops
polrt 1525. A/nnnemn
Grypaner
5.28 data
point boxg:
Figure 3: The overview of GuirlVG.
A.3.1 FORMULATION FOR SECTION 3.1 (TRIVIAL RFT)
In the trivial RFT setting, rewards are strictly coupled to exact string matching. Let [1] denote the indicator function.
Rifomat match
Race
|match (insswer..../aamsvv) (match bbox array with 4 numbers]
1 [oU boxgt) > 0.5]
A.3.2 FORMULATIOO FOR SECTION 3.2 (SOFT REWARD FUNCTION)
To mitigate the sparse reward signal caused by strict syntax constraints, the Soft Reward Function (SRF) decomposes the format reward into partial credits, relaxing output style:
(1
1
Riomal
|match (think)) + 2
1l match (/think)) + (match (answer))
1
+ 3
[match (/answer)] |match numbers inside answer)
A.3.3 FORMULATION FOR SECTION 3.3 (ACCURACY REWARD VARIANTS)
We empirically compare different designs for model prediction combined with the corresponding accuracy reward function. The formulations for the variants are defined follows:
14

---

Published as a conference paper at ICLR 2026
R
(box boxgi) > 0.5]
RioU
lo UboX box
Rriisance@e
I| prod
pointzell2 point 2112 S 80
Rin Bhox
point pred
boxgi
A.3.4 FORMULATIOO FOR OTHER SECTIONS
For Section 3.4, the formulation for the Adversarial KL Factor is provided Eq. 3.1 of the main paper. For Sections 3.5, 3.6, and 3.7, the underlying optimization equation remains the GRPO, consistent with Eqs. 2.1, 2.2, and 2.3 provided in the Preliminaries section.
A.4 OVERVIEW OF GUIRLVG
As shown in fig. 3, we provide the overview of GuirIVG. Based empirical results, finalize a set of design choices for GUI visual grounding under GRPO. We propose the Soft Reward Function (SRF) provide partial credit for format compliance while relaxing output constraints. For the prediction format, we use direct point prediction with the In-Bbox binary reward. To stabilize training, we introduce the Adversarial KL Factor with coefficient of ß = 10 4 We employ LoRA for efficient fine-tunan tue-tuning and set the group size to 6 and batch size to 4. Image resolution information is withheld during training and added only at inference. We train 1,300 steps for our final version.
15


Reviews:
[{"review_id": "AIuRl3dhNd", "reviewer": "Reviewer_PTho", "summary": "GUIRLVG addresses GUI visual grounding (GUI-VG) by proposing a rule-based RFT framework, challenging the dominant supervised fine-tuning (SFT) paradigm that requires massive labeled data. The paper first finds that naive RFT underperforms SFT, then conducts a systematic empirical study to optimize RFT’s core components. \n\nKey innovations include:\n1. Replaces rigid binary rewards with fractional credit for partial format compliance, reducing training noise.\n2. Aligns directly with GUI-VG’s goal (locating actionable points) for better performance than bounding box/IoU-based rewards.\n3. Adopts LoRA for fine-tuning (25x faster than full fine-tuning with negligible performance loss), optimal group/batch sizes (6/4), and resolution prompting (withheld during training, added at test time).\n\nTrained on only 2K–5.2K samples, GUIRLVG outperforms SFT baselines trained on 1M–13.58M samples: achieving 88.7% accuracy on ScreenSpot, 91.9% on ScreenSpotV2, and 36.1% on ScreenSpotPro.", "questions": "Overall, this is a strong, readable paper with clear motivation and well-structured proposed solutions. Its core strengths lie in a coherent research narrative, impactful empirical findings on RFT for GUI-VG, and compelling results demonstrating superior data efficiency over SFT methods.\n\nHowever, its theoretical novelty is somewhat weakened by two key omissions: the lack of mathematical details for proposed modules (e.g., the Soft Reward Function/SRF) and insufficient theoretical analysis, evaluation, and comparison with other RFT-based models. The authors should clarify these aspects to strengthen the work’s rigor.\n\nAdditionally, including a visual overview of the overall GuirlVG pipeline would greatly benefit readers, enabling them to grasp the end-to-end design more straightforwardly.\n\nI would like to raise my evaluation score if the authors supplement the paper with more theoretical details (or designs) and thorough comparisons with existing RFT-based models.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. This paper introduces a novel framework that incorporates Rule-based Reinforcement Fine-Tuning (RFT) into GUI visual grounding for the first time. Its results outperform other Supervised Fine-Tuning (SFT)-based methods, providing a valuable indication for future research directions in visual grounding.\n\n2. The results presented in Section 4 are highly impressive: using only 2K or 5.2K training samples, the framework achieves superior performance compared to previous SFT-based methods that rely on massive datasets and larger model sizes. If these findings can be reproduced, they will undoubtedly reshape the future direction of GUI visual grounding (GUI-VG) research.\n\n3. The research motivation is fully clarified by addressing key questions posed by the authors, while the literature review and preliminary analysis further elaborate on this motivation. The paper follows a clear and logical storyline, with the problem formulation and proposed pipeline progressing in a coherent manner.", "weaknesses": "1. Although the empirical research approach and writing style are acceptable, the theoretical details of the design and calculation processes need more explicit elaboration. \n\nFor instance, in Section 3.2, when proposing the “Soft Reward Function,” a specific mathematical formulation would be preferable to purely natural language descriptions. This issue persists in other methodology subsections. Otherwise, this presentation reads more like an application report, which weakens the theoretical novelty of the proposed pipeline.\n\n2. A critical oversight is the lack of comparison with existing RFT-based GUI-VG methods. \n\nAs mentioned in section 2.1, ”While prior works have proposed various modeling choices for RFT-based GUI-VG, these advances often emphasize reward function novelty or performance improvements without systematic examination of underlying design factors.”  Although,  the authors claim that\"differences in data, training, and models would yield limited rigorous conclusions in systematic experiments.\"  Given that several RFT-based GUI-VG methods (e.g., GUI-R1, VLM-R1, UnivG-R1) share core design foundations (e.g., GRPO-based reinforcement fine-tuning), I think it is not impossible to align those models with GuirlVG’s experimental setup for fair comparison.\n\nOmitting these comparisons undermines the paper’s claim of advancing RFT for GUI-VG, as readers cannot determine whether GuirlVG’s performance gains stem from novel technical contributions or only better hyperparameter tuning or data selection. Additionally, the paper fails to distinguish GuirlVG from these prior RFT methods theoretically: it does not explicitly address how its design choices differ from or improve upon the reward function novelty or modeling choices of existing RFT-based approaches—an essential detail to validate its theoretical novelty.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8bLzR8wwG8", "reviewer": "Reviewer_qjDC", "summary": "The authors applied RFT to the GUI-agent which is underexplored. the study step-by-step analyze the intermediate resutls and propose an efficient way that outperforms SFT on the tasks of interest.", "questions": "See weaknesses", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. It is a well-motivated study\n2. The experimental results are sufficient to convince its effectiveness \n3. The methodology is efficient, clear, and easy to follow", "weaknesses": "1. What about the performances of other steps? Any indications from those?\n2. Could be more ablations on hyperparameters of the config.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DnkqnMai04", "reviewer": "Reviewer_R3mo", "summary": "This paper presents GuirlVG, a RL-based framework for graphical user interface visual grounding (GUI-VG). Through a systematic empirical study, the authors identify critical factors influencing RFT success in GUI-VG and introduce key improvements: a Soft Reward Function (SRF), In-Bbox point reward, and a novel Adversarial KL Factor to stabilize training. Using as few as 2K–5.2K samples, GuirlVG outperforms strong SFT baselines, achieving impressive results.", "questions": "Please see weaknesses.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- Comprehensive empirical study\n\nThe paper systematically dissects RFT components, including reward design, KL penalty, fine-tuning method, and prompt structure, offering rare empirical clarity in a field often driven by ad hoc innovation.\n\n- Novel stabilization mechanism\n\nThe Adversarial KL Factor dynamically scales the KL penalty, effectively mitigating reward over-optimization—a notable technical contribution to GRPO-style RL for multimodal models.\n\n- Strong empirical results\n\nGuirlVG achieves state-of-the-art accuracy on multiple GUI grounding benchmarks with orders-of-magnitude less data, convincingly demonstrating the data efficiency of RFT.\n\n- Clarity and transparency\n\nEach ablation is well-motivated. \n\n- Practical insight for the community\n\nThe findings (e.g., point-based rewards outperforming bbox IoU, LoRA matching full FT, resolution prompting effects) are valuable practical takeaways likely to influence future GUI agent training.", "weaknesses": "The main reason I gave a score of 4 is that the scale of the empirical experiments are not sufficient to make a strong conclusion:\n\n- The experiments are only done on Qwen2.5-VL. While I understand the Qwen-VL series is probably the only modern model architecture choice in the field, more experiments are required to find out if the findings in the paper are universal or model-specific. For example, Finding 5 says \"LoRA offers comparable performance to full fine-tuning\", but is it the case with Qwen2-VL which is not extensively trained on the task? Even on the Qwen2.5-VL architecture, there are some choices to experiment with, such as UI-TARS.\n\n- Some experiments are too small to draw a conclusion. For example, in Table 6, only group size = {6, 8} and batch size = {1, 4} are tested, with a total of only three runs. The conclusions from such experiments can be very brittle.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ROWQW5DBHB", "reviewer": "Reviewer_9VFd", "summary": "this manuscript introduces GuirlVG, a rule-based reinforcement fine-tuning (RFT) method designed to address the high costs and extensive data requirements of traditional supervised fine-tuning (SFT) for graphical user interface visual grounding (GUI-VG) tasks. The core of GuirlVG lies in the systematic empirical study and improvement of Group Relative Policy Optimization (GRPO) to achieve efficient training. Experimental results demonstrate that GuirlVG, using only 5.2K training samples, surpasses SFT methods that use over 10M samples across three major benchmarks. It showed a +7.7% improvement on ScreenSpot, a +17.2% improvement on ScreenSpot-Pro, and achieved a 91.9% accuracy rate on ScreenSpot-V2. These findings confirm that RFT is a more data-efficient and generalizable post-training solution for GUI-VG.", "questions": "please see weakness.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.Achieves performance far exceeding SFT with significantly fewer training samples.\n\n2.Provides a relatively comprehensive analysis of the impact of various parameters in the GRPO training process on the final performance.", "weaknesses": "1.The paper states its aim is to provide guidance for the design of Reinforcement Fine-Tuning (RFT) in GUI-Visual Grounding (GUI-VG) tasks. However, the final performance of its 7B model shows a significant gap compared to other 7B models on the current screenspot-pro leaderboard(such as SE-GUI-7B, GTA1-7B). This makes it difficult to be convinced of the validity of the paper's conclusions.\n\n2.Although the work explores various parameters within the GRPO training process, it does not introduce any fundamental innovations to the algorithm itself. This significantly diminishes the work's originality.\n\n3.Furthermore, the parameter settings are likely strongly dependent on the specific dataset, model architecture, and model scale used. Therefore, the conclusions drawn in the paper may not be generalizable. More experimental results are needed to verify the generalizability of these findings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)