BTL-UI: Blink-Think-Link Reasoning Model for GUI Agent
Authors: Shaojie Zhang
Ruoceng Zhang
Pei Fu
Shaokang Wang
Jiahui Yang
Xin Du
ShiqiCui
Bin Qin
Ying Huang
Zhenbo Luo
Jian Luan
Venue: NeurIPS 2025 poster ()
Keywords: Large Vision-Language Model
GUI Agent
Reinforcement Fine-Tuning
Primary area: applications

Abstract: In the field of AI-driven human-GUI interaction automation, while rapid advances in multimodal large language models and reinforcement fine-tuning techniques have yielded remarkable progress, a fundamental challenge persists: their interaction logic significantly deviates from natural human-GUI communication patterns. To address this gap, we propose Blink–Think–Link (BTL), a brain-inspired framework for human-GUI interaction that mimics the human cognitive process between users and graphical interfaces. The system decomposes interactions into three biologically plausible phases: (1) \textbf{Blink} - rapid detection and attention to relevant screen areas, analogous to saccadic eye movements; (2) \textbf{Think} - higher-level reasoning and decision-making, mirroring cognitive planning; and (3) \textbf{Link} - generation of executable commands for precise motor control, emulating human action selection mechanisms. Additionally, we introduce two key technical innovations for BTL framework:
(1) Blink Data Generation - an automated annotation pipeline specifically optimized for blink data, and
(2) {BTL Reward – the first rule-based reward mechanism that enables reinforcement learning driven by both process and outcome.}
Building upon this framework, we develop a GUI agent model named BTL-UI, which demonstrates competitive performance across both static GUI understanding and dynamic interaction tasks in comprehensive benchmarks. These results provide conclusive empirical validation of the framework's efficacy in developing advanced GUI agents.

BTL-UI: Blink-Think--ae Reasoning Model for GUI Agent
Shaojie Zhang" Ruoceng Zhang' Pei Fu" Shaokang Wang Jiahui Yang Xin Du Shiqi Cui Bin Qin Ying Huang Zhenbo Luo Jian Luan
MiLM Plus, Xiaomi Inc
(zhangshaojie5, zhangruocengl, fupeil, luozhenbo, luanjian) @xiaomi.com
Abstract
In the field of Al-driven human-GUT interaction automation, while rapid advances in multimodal large language models and reinforcement fine-tuning techniques have yielded remarkable progress, fundamental challenge persists: their interac- tion logic significantly deviates from natural human-GUl communication patterns. To address this gap, we propose Blink-ThinkkLiin work for numan-GUl interaction that mimics the human cognitive process between users and graphical interfaces. The system decomposes interactions into three biologically plausible phases: (1) Blink rapid detection and attention to rele- vant screen areas, analogous saccadic eye movements; (2) Think - higher-level reasoning and decision- mmaking, mirroring cognitive planning; and (3) Link gen- eration of executable commands for precise motor control, emulating human action selection mechanisms. Additionally, we introduce two key technical innovations for BTL framework: (I) Blink Data Generation
(BTL), brain-inspired frame-
automated annotation pipeline specifically optimized for blink data, and (2) BTL Reward the first rule-based reward mechanism that enables reinforcement learning driven by both process and outcome. Building upon this framework, we develop GUI agent model named BTL-UI, which demonstrates competitive performance across both static GUI understanding and dynamic interaction tasks comprehensive benchmarks. These results provide conclusive empirical validation of the framework's efficacy in developing advanced GUT agents.
Introduction
Automation of graphical user interface (GUI) interactions constitutes a pivotal milestone in developing genuinely intelligent digital assistants [1, 2, 3]. Recent breakthroughs in large vision-language models (VLMs) [4, 5] and reinforcement learning fine-tuning techniques have substantially improved agents' capabilities in natural language command interpretation, visual element perception, and multi-step task execution through human reasoning [6. 7].
However, current mainstream systems adopt mainly two approaches. The first relies on supervised fine-tuning (SFT) to align model behavior with task objectives, but this method faces major limitations: a strong dependence on large-scale expert-labeled data and limited generalization capability when faced with nu-oofddistrbuutin scenarios. The second approach involves rule-based reinforcement fine-tuning (RFT) [8], as shown in Figure 1 (a), which enhances generalization in complex tasks by using a structured reasoning format: intermediate cognitive steps are encapsulated within <think> tags, and final decisions are expressed through <answer> tags. Although effective in improving task performance, these methods [9, 10, 11] exhibit two critical shorteomings: (1)
Equal contribution; Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Sottr Trugge
ceaiii Analats and
Tomer AA firecam fent can on derer the anteree
de Be sarve scandd paecss samnir ane clod
Fot inos Rock rouss aomm BASEDON
de with strveity
lado supectioly du
Hor
Uver tndractee
Action librkry
Pemptti
Sug
Recse wing preents
Pl prosus:
adl culliy disphot incis vil magli
Lond deown desar
1igh Trproaly Roadi vide
mare cheppe qn riihh Thirr poccuinn waen reprrmmnnt here ded
Delorniia Varwee
6 Thekk Aravver
[c) Bleee Lank
Figure I: Framework comparison of previous Think-Answer and RFT. Specifically, colorful text is supervised by rule-based reinforcement learning. And different colors of text indicate different reward rules. The previous "Think-Answer framework is optimized by format reward, action type reward, and corresponding args reward. And our framework is optimized by dual format reward, blink reward, and link reward.
in GUT tasks for
significant deviation from natural human-GUl interaction patterns, and (2) excessive focus on interaction outcomes while lacking effective process-oriented reward mechanisms.
Cognitive studies [12, 13, 14] demonstrate that human-GUI interaction achieves remarkable effi- ciency through three sequential processes: (a) Blink Phase. Rapid target location during saccadic intervals; (b) Think Phase. Multimodal information integration with intentional reasoning; (c) Link Phase. Generation of precise motor execution commands. Building upon this cognitive finding, we innovatively propose biologically inspired interaction paradigm the Blink-Think- Link (BTL) paradigm- for GUI agents, and computationally simulate this paradigm through a structured output mechanism (as shown in Figure I (b)):
<blink>: Where relevant areas of the screen are rapidly located, analogous to saccadic eye movements. The visual attention-related aegion-of-intereee information is encapsulated within <blink></blink> tags.
<think> Where the system engages in high-level reasoning and decision-making, mirroring
cognitive task planning. The reasoning processes are recorded in <think></hhin>> tags.
<link>: Where actionable commands are generated for precise execution, reflecting human action selection mechanisms. The action commands are output in <link></links tags.
Specifically, to model human visual localization capabilities during blink intervals, we developed an innovative blink data generation pipeline to automatically produce several region-of-interes (ROI) annotations for training samples. Furthermore, to address the limitations of current reward models in rule-based RFT algorithms that over-rely on outcome- rewards while neglecting guidance for intermediate interaction processes, We propose the innovative BTL Reward, Process-Outcome Integrated Reward Mechanism, which comprises three core components: (1) the Dual Format Reward for template and content matching, (2) the Blink Reward for fine-grained guidance of interaction processes, and (3) the Link Reward for action outcome evaluation. By combining the Blink Reward's granular process supervision with the Link Reward's 's precise outcome feedback, this mechanism pioneers the organic integration of process-oriented and outcome-driven approaches. Compared to conventional reward schemes focusing solely on final outcomes, the BTL reward mechanism delivers more sophisticated and multi-dimensional training guidance. Finally, building upon this framework, we develop BTL-UI, GUI Agent that demonstrates the framework's effectiveness across multiple GUI tasks.
In general, the main contributions are summarized as follows:
1. We propose BTL (Blink-Think-Liin an innovative framework that simulates the human cognitive process in the human-GUT interaction by explicitly modeling how users perceive, process, and act upon interface elements.

---

GUT Input
Completions Generation
Systen prempt
Completion
Policy Model
User instruction
Prompt
becal doe had ofl the aed Tirs toe The oage tor to caher
Update
Dulkore
Wendyy
Riatl Camdralyy M'eche" "porter 2000
Policy Optimization
Completion
aud wwd.rraeee prur rdeerr uoeggr tarssnaeee.
deor 201 (000) cagii comtte light sesert
KL Calculation
on
Advontage Calculation
Rule-Based Reward Calculation
Dual Format Reward
Blink Reward
Link Reward
Temples Format Chock
IOU Deduplication
Complete Astion Analyia
Reference
Screenshat
Content Formor Check
Honganan Masching
Model
Figure 2: Overall framework of BTL. We adopt Group Relative Policy Optimization (GRPO) to optimize the proposed BTL. Firstly, the base model generates N completions for given GUT task sample. Furthermore, GRPO computes the relative advantages within group of completions, eliminating the need for manually annotated data. Finally, the policy model updates parameters under the guidance of relative advantages and the KL divergence constraint.
2. We propose two key innovations to jointly advance the learning of GUT agents within this framework: (1) Blink Data Generation-an efficient data annotation pipeline that automatically generates multi-region Regions of Interest (ROIs) for training samples; (2) BTL Reward the first rule-based Process
Process-Outcome Outcome Integrated Reward Mechanism.
3. We develop BTL-UI, a GUI agent trained via the BTL framework, and extensive experiment results demonstrate that the model achieves competitive performance across multiple GUI benchmarks.
2 Related Work
2.1 GUI Agents
Autonomous agents powered by large language models (LLMs) and VLMs have recently garnered considerable scholarly interest due to their interactive functionalities. For GUI tasks, earlier systems relied on LLMs read and interpret structured representations such HTML and accessibility trees. However, since icons, images, diagrams, and their spatial relationships are difficult to express in such structured languages, agents based on LLM often perform poorly [6. 15, 16]. Therefore, VLM- based agents have been introduced to perceive visual GUI signals directly with better performance [2, 17, 18, 19, 20, 21]. For example, UGround [22] developed specialized GUI grounding model for GUI element localization. OS-Atlas [23] proposed foundational model for GUI agents by interpreting human intentions and predicting actions in the form of function calls. Aguvis [24] integrated explicit planning and reasoning within the model, enhancing its ability to navigate and interact with complex digital environments autonomously. UI-TARS [9] combines GUI-related pretraining with task-level reasoning fine-tuning to better capture the complexity of GUI interactions. Although research on VLM-based GUI agents has made impressive progress, they mainly follow the SFT training paradigm, which directly mimics the ground -ound-truth actions provided in the curated data.
2.2 Reinforcement Fine- TaeeTTunnng
With the advent of rule-based reinforcement learning approaches such as OpenAT-ol [25] and DeepSeek-R1 [8], recent studies have demonstrated that RFT improves the reasoning abilities of the
model and provides greater generalizability [26]. Subsequent approaches [26, 27, 28) have introduced
this paradigm to VLMs. For example, Vision-RI [27] combined vision criterion-driven reward
3

---

function and a progressive rule refinement strategy to enhance VLM's object localization capabilities. Visual-RFT [26] adopted the reinforcement learning strategy to enhance visual perception and grounding ability of VLMSs. VIMM-RI [29] demonstrated that RFT with small amounts of high-quality data can enable VLMSs to solve complex vision-language tasks.
For GUI tasks, UI-RI [30] and GUI-RI [10] introduced rulc-based reinforcement learning frameworks that require minimal expert supervision, demonstrating competitive performance. InfiGUI-RI [11] further advanced the field by bridging reactive execution and deliberative reasoning through the Actor2Reasoner architecture. However, existing RFT-based GUI agents predominantly adopt rule- based reinforcement learning, which focus on final outcomes and lack intermediate process guidance, often overlooking key aspects of human cognition and interaction.
3 Method
In this section, we introduce BTL, new framework grounded in cognitive science theory, with its core concept derived from the Blink-Think- Think-Link paradigm observed in human-GUl interactions. The framework is shown in Figure We detail the implementation details of this framework through the following components: Preliminaries, Blink Data Generation, BTL Reward, and Policy Optimization.
3.1 Preliminaries
The interaction between a GUI agent and its environment can be naturally formulated as a Markov Decision Process (MDP), defined by the tuple (S, Z, T, O). Here, 8 denotes the state space representing possible screen states; A is the action space that encompasses interaction types such as clicking, typing, and scrolling: Z is the observation space, including screenshots or structured UI representations; T S x A x S + [0, 1] defines the probability of transitioning from one state to another given action; and O : S A + Z specifies the probability of receiving particular observation given a state and an action.
During task execution, at each discrete time step I, the agent receives input tuple (z+, u, h), where 2, E 2 the current state of the sereen, u refers to the global task instruction and is its interaction history. The BTI. process can then be formalized as structured policy function F:
F({2,, u. h)) + o, [b,, d,, az).
where 0 denotes the BTL output at time f, consisting of: bi-visual attention regions, reasoninng and decision trace, a,-the final action to be executed. Each action o, = (0),6t) E A composed of an action type (e.g., click) and its corresponding parameters 6, (e.g., coordinates, text input).
Upon execution of a, the environment transitions to a new state 1 and the process repeats until the task is completed or terminal condition is met.
3.2 Blink Data Generation
One of the core innovations of the BTL framework is its ability to simulate the human mechanism of rapidly locating ROIs in visual scene during the blink phase. To achieve this, we propose an automated Blink data generation pipeline that annotates ROIs on the screenshot corresponding to the user instruction in the MDP. As illustrated in Figure 3, the pipeline consists of two main stages. A parsing model [31] first processes the raw screenshot to extract semantic UI elements. Then, an analysis model [5] is used to evaluate the visual importance and contextual relevance of these elements, allowing filtering and prioritization to produce the final ROI annotations.
Specifically, in the first stage, we extract individual UI elements such buttons, icons, and text fields, annotating each with bounding box coordinates, type, and semantic captions. These annotations form structured representation of the state of the screen, enabling bottom-up human-like interpretation of the GUI. The output of this stage a comprehensive list of elements, denoted È = {e1, 62, en where each eg = {idg, bboxg, typeg, caption,, interactivitys! represents the attributes related to the element. This foundational representation serves as the input for subsequent filtering and prioritization steps that model instruction-directed visual attention.
In the second stage, we employ Qwen2.5-V1.-32B [5] to simulate top-down attention by filtering and ranking elements based on visual saliency and task relevance. Oriented by task instruction u and

---

Screenchet Imput
Element Parser Output
Filter Prompp
Syster promp"
10
S
User
Exrants
communsees & Nevi Vaaneer
Schanahho
9
Ecrest Purker
Parsec Lit
Filte sd Ebeaants Screwsther
Figure 3: Two-stage data construction pipeline. In the first stage, the basic properties of UI elements are obtained by a parsing model. To eliminate the redundancy of the number and attributes of elements, the analysis model in the second stage simplifies the list to À elements with their positions (<bbox>), while the reserved <caption> attribute indicates whether the element interactive, In the example shown in the figure, the instruction for the current step is "Use the GPS to locate nearby museum and then book ride with Lyft." Accordingly, the most relevant element in the Blink output the "Maps & Navigation" app with </D>10<<1I>>
interaction history h, the model dynamically selects subset of elements from the filtered list E. This selection process can be formulated as:
RII f(E, u, h):
where EROI E denotes the resulting set of X candidate elements that are most relevant to the current task step during blink phases, encapsulated within <blink> </blink> tags. The choice of the number of elements after filtering, A, achieves a trade-off between BTL performance and efficiency. This attention-guide annotation not only mimics human visual focus during blink phases but also provides a high-quality reference for optimizing the agent policy.
3.3 BTL Reward
Effective GUI agents must excel at both interface grounding and long-horizon planning. To this end, we design a three-component rule-based reward scheme, denoted as RET. that mirrors the human Blink-Think-Linn cognitive cycle:
RATL Rioemat +
Polink
Rünk-
Each term provides targeted supervision a different phase of interaction, detailed below.
Dual Format Reward. Following previous work [8, 26) that leverages format rewards to encourage predefined templates for easy answer extraction, we introduce dual format reward to evaluate whether the generated output adheres to both the expected structural template and content. Specifically, the template check function fiemplate is used check whether the generated completions meet the Blink-Ti Soin-Think-Linn three-stage grammatical structure. Furthermore, the content check functioo feontem adopted evaluate whether the blink content complies with the XML format and the link content complies with the ISON format, which facilitates the parsing of trajectory planning and actions with corresponding arguments. We adopt binary reward scheme, assigning reward of only when the prediction o, fully satisfies both format and content criteria as follows:
Riormar (0i) [1. if fiemplate 1A 1A feontent otherwise
Blink Reward. This component incentivizes the rapid and accurate localization of the interface elements relevant to the instruction From the agent's prediction Os, we extract set of ROIs P: = (rf) and compare them to ground-truth annotations G, = {9.) (see $3.2). We adopt the Hungarian matcher [32] M (s, r), classical assignment algorithm used to compute the optimal
5

---

one-to-one matching between predicted and ground ground-truth truth bounding boxes based on IoU scores, under a given threshold
indeTmaach f 1 G., Fp7 E Ps S.t. M(r.,
It is worth noting that in the planning task, the elements related to the instruction may not be explicitly present in the current screenshot. And the corresponding operation should to other pages through scrolling going back. Thus, in the predicted results, P. = Ø is allowed. Consequently, the blink reward can be defined as follows:
L, max (s(y) y indermachh. elif P. P #22A, #2 otherwise
if 0 @A VVa
Phinnk(0))
(6)
0.
where A" denotes the non-interactive action spaces and s(.) refers to the reward allocation function, which is determined based on the priorities of elements the annotations.
Link Reward. The link phase assesses the agent's ability to generate fully coherent executable command. Recent RFT-hased GUI agents [10, 11, 30] always split the reward of the predicted action into a reward for the action type and a reward for the action args (e.g. click coordinates or input text). However, this kind of reward will split action into independent contents, which is not in line with human cognition. At the same time, this staged action reward will cause reward hacking, which prevents the agent from understanding the designed action space. Thus, we employ strict binary criterion: the agent receives reward only if both the action type and its associated arguments are exactly correct. Formally, the link reward is defined
(1, Sype = IA Sargs =
Rrin (o.) =
0. otherwise
This all-or-nothing scheme ensures that the final command is internally consistent and accurately reflects the intended GUT operation.
3.4 Advantage Computation and Parameter Update
As shown in Figure adopt Group Relative Policy Optimization (GRPO) optimize the proposed BTL. Since its supervision is based solely on the final outcome, GRPO is particularly suited for tasks with explicit, objective answers. Furthermore, GRPO significantly reduces memory overhead for VLMs by removing the reward models or value models in other performance optimization methods [33].
Given a base model to be optimized, GRPO starts by initializing a trainable policy model Ty and frozen reference model Tet. For a given GUI task sample { h). the policy model Ty first generates group of completions {01, oN.. Then, the reward function computes the whole group's rewards (R1, R2,... Rv). }, which further used to calculate the advantage A; of each completion within the group by:
R₂ - mean meean(fRg (fR 1=1))=1)) A
std( (
After the reference model computes the logits to output each completion given the task. the policy model Ty is optimized by maximizing the following objective:
1 N
N
To(oil2s, h.h) A, 3 ICC(TT(O)) h)l u, ai)))
(GRPO(0)
(9)
Tea (o: 224, U. h)
where N is the number of completions in one group and / is the hyperparameter to control the KL. divergence constraints. This objective motivates the model to tend to produce the completion with higher advantage within a group. but not to stray too far from the initial model.

---

Experiment
4.1 Implementation Details
Experimental Setup. We develop the BTL--UI-3B/7 model based on Qwen2.5-VL-33B77 and adopt the ms-swift framework [34] for RL training. As shown in Table I, we train BTL-UI UT in a mix of grounding and planning data.
Evaluation. To conduct thorough eval- uation of BTL-UI, we employ range of critical benchmarks that focus on specific aspects of the GUI agent's s grounding and planning capabilities:
Table 1: RFT data for BTL-UI.
Category
Source
Size
Show -l [35]
1K
Grounding
Grounding: Screenspot series benchmarks assess fundamental GUI understanding and element grounding accuracy across di- verse platforms (Mobile, Desktop, Web), ScreenSpot [7] evaluates the single-step GUI grounding performance across multiple plat- forms. ScreenSpot- V2 [7], e--anotated
ShowUl-Desktop [35]
1K
AndroidControl [36]
500
Low-Level
GUI-Odyssey [37]
500
AndroidControl [36]
500
High-Level
GUI-Odyssey [37]
500
version, addresses annotation errors present in the original ScreenSpot. ScreenSpot-Pre [38] specifi-
cally increases the difficulty with complex desktop applications and high-resolution screens.
Table 2: GUI grounding accuracy ScreenSpot [7]. Bold means the best results, and underline means the second best results. Avg. denotes the average performance on mobile, desktop and web subtasks.
Mobile
Desktop
Web
Model
Method
Model Size
Avg.
Text
Icon
Text
Icon
Icon
GPT-4o [39]
30.5
20.6
19.4
7.8
18.8
Qwen2-VI
ZS
7B
75.5
60.7
76.3
54.3
35.2
25.7
55.3
OS-Atlas-Bat [23]
ZS
7B
93.0
72.9
91.8
62.9
90.9
74.3
82.5
Qwen2.5-VL. [5]
7S
3B
90.5
61.1
60.0
43.2
80.9
40.0
65.0
Qwen2.5-VL [5]
ZS
7B
86.3
83.8
85.6
67.1
87.4
78.6
84.8
Inter VL3 [40]
8B
79.5
CogAgent [6]
SFT
18B
67.0
24.0
74.2
20.0
70.4
28.6
47.4
Aria-UI [41]
SET
3.9B
92.3
73.8
93.3
64.3
86.5
76.2
82.4
SceClick [7]
SET
9.6B
78.0
52.0
72.2
30.0
55.7
32.5
53.4
ShowUT [35]
SFT
2B
92.3
75.5
76.3
61.1
81.7
63.6
75.1
Aguvis [24]
SET
7B
95.6
77.7
93.8
67.1
88.3
75.2
84.4
UGround [22]
SET
7B
$2.8
82.5
63.6
80.4
70.4
73.3
UGround-V] [22]
SET
2B
89.4
88.7
65.7
68.9
77.7
UGround-V] [22]
SFT
7B
94.1
79.9
93.8
76.4
90.9
84.0
86.3
UI-TARS [9]
SFT
2B
93.0
75.5
90.7
68.6
84.3
74.8
82.3
UI-TARS [9]
SET
7B
94.5
85.2
95.9
85.7
90.0
83.5
89.5
UI-RI [30]
RFT
3B
90.2
59.3
85.2
73.3
GUI-RI [10]
RFT
3B
93.8
64.8
89.6
72.1
GUI-R1 [10]
RFT
7B
91.8
73.6
75.7
BTI-UI
RFT
3B
96.3
77.3
88.2
57.9
80.0
68.9
80.0
BTI-UI
RFT
7B
97.1
83.8
90.2
70.7
88.7
84.5
87.2
Planning: AndroidControl [36] and GUl-Odyssey [37] evaluate the agent's grounding and planning ability to execute multi-step tasks within realistic Android environments. These benchmarks provide agents with task instruction. current screenshot, and previous interaction history, aimed at enabling accurate prediction of the next action. Furthermore, according to the input, the settings on AndroidControl can be divided into low-level tasks and high-level tasks. High-level tasks only input the global instruction to the agent, while low-level tasks will additionally input the single-step action plan. And GUI-Odyssey only adopts the high-level experimental setups.
Evaluation Metries. For grounding tasks, we use click point prediction accuracy as our evaluation metric. For planning tasks, according to OS-Atlas [23], we report three standard metrics for GUI agents: action type prediction accuracy (Type), click point prediction accuracy (GR) and step success rate (SR). Specifically: Type measures the exact-match accuracy between predicted and ground-truth

---

Table 3: GUI grounding accuracy ScreenSpot-Ve [7]. Bold means the best results, and underline means the second best results. Avg. denotes the average performance on mobile, desktop and web subtasks.
Mobile
Desktop
Web
Model
Method
Model Size
Avg.
Text
Icon
Text
Icon
Text
Icon
GPT-4o [39]
7$
30.5
23.2
20.6
19.4
11.1
7.8
18.8
OS-Atlas-Base [23]
2S
4B
85.7
58.5
72.2
45.7
82.6
63.1
70.1
OS-Atlas Aast [23]
ZS
7B
93.0
72.9
91.8
62.9
90.9
74.3
82.5
Qwen2.5-VL [5]
ZS
3B
92.1
66.8
72.6
46.8
83.0
44.3
70.4
Qwen2.5-VL [5]
ZS
7B
97.9
86.7
87.6
68.6
91.5
79.3
87.1
In3 [40]
zs
8B
81.4
SeeClick [7]
SFT
9.6B
78.4
50.7
70.1
29.3
55.2
32.5
55.1
Aguvis [24]
SET
7B
95.6
77.7
93.8
67.1
88.3
75.2
84.4
UI-TARS
SFT
2B
95.2
90.7
68.6
87.2
78.3
84.7
UI-TARS [9]
SFT
7B
96,9
89.1
95.4
85.0
93.6
85.2
91.6
BTL-UI
RFT
3B
97.9
83.4
88.7
62.1
83.3
69.0
82.9
BTL-UI
RFT
7B
98.6
89.6
92.3
70.7
92.3
80.3
89.1
action types (e.g., "click" vs. "swipe"). GR evaluates grounding performance via click point prediction accuracy in specific action types (e.g. "click" and "long press"). SR is the step-wise success rate: step is counted successful only if both the predicted action and its associated arguments (e.g., click coordinates or input text) match the ground truth.
4.2 Experimental Results
We evaluate BTL-UI across three key capabilities: grounding, low-level planning, and high-level reasoning. The results demonstrate consistent and significant improvements over existing baselines, validating the effectiveness of the Blink-Thinn Think-Lint framework.
Grounding Capability. To assess how well the model can localize UI elements, report grounding accuracy on the ScreenSpot benchmark series in Table 2, 3, and 4. the original ScreenSpot dataset, BTL-UI-7B achieves average accuracy of 87.2% outperforming the baseline Qwen2.5-Ve VL (84.8%) and surpassing the supervised fine-tuned Aria-UI (82.4%). On the corrected ScreenSpot- V2, performance further improves to 89.1% 1.. In the ScreenSpot benchmark, the BTL-UI-3B consistently outperforms other RFT-based models, achieving an average accuracy of 27. 27.1%. which substantially higher than UI-RI (17.8% on the same scale. This suggests that the Blink Phase, which encourages carly-stage attention to semantically relevant regions through ROI supervision, enables more precise perception and grounding even under diverse visual layouts. Although the overall grounding performance of BTL-UI remains slightly lower than that of UI-TARS [9], which is strong GUI Agent developed based on Qwen2- VL [4] with training on 50B tokens, the proposed BTL-UI shows certain advantages in the mobile subtasks.
Planning Capability. As shown in Table 5, BTL-UI exhibits strong generalization and reasoning ability on both low-level and high-level GUI planning benchmarks. In AndrridCoomtrol-Lovw BTL- UI-7B achieves SR of 88.0%, surpassing the previous best model OS-Atlas-Prat Pro--s (85.2%) and GUI-R1-7B (66.5%) while the 3B variant attains a comparable 84.8%, confirming the efficiency of the BTL reinforcement paradigm. For long-horizon tasks in AndroidControl -High, which require multi-step reasoning and contextual grounding, BTL-UI-7B achieves an SR of 69.2%. outperforming GUI-RI-7B (51.7%) and approaching the SFT-based GUI foundation model. OS-Atlas-Pro-77 (71.2%). This improvement reflects the synergy between Blink-phase attention and Link phase symbolic reward, which jointly stabilize execution and reduce accumulated errors in extended interaction sequences. In GUI-Odyssey, benchmark that emphasizes hierarchical decision-makin and interface switching, BTL-UI-7B reaches an SR 65.4%, significantly surpassing GUI-RI-7B (38.8%). Although there is still performance gap compared to UI-TARS, the proposed BTL-UU is comparable large-scale SFT models such as AS-Atlas, while requiring significantly less training data.

---

Table 4: GUI grounding accuracy Screenspot-Pro [38]. Bold means the best results, and underline means the second best results. Avg. denotes the average performance on all subtasks.
Model
Method
Model Size
Dev
Creative
CAD
Sclentifie
Office
OS
Avg.
Text
Icon
Text
Icon
Text
Icon
Text
Icon
Text
Icon
Text
Icon
GPT-4o [39]
zS
1.3
0.0
1.0
0.0
2.0
0.0
2.1
0.0
0.0
0.0
0.0
Qwen2-VL. [[] [4]
ZS
7B
0.5
0.0
2.6
0.0
1,5
0.0
6.3
0.0
3,4
1.9
0.9
0.0
1.6
Qwen2.5-VL
3B
Qwen2.5-VL. [5]
ZS
7B
29.0
OS-Atlas Base [23)
zs
7B
33.1
144
28.8
2.8
12.2
4.7
37,5
7.3
33.9
5.7
27.1
4.5
18.9
ShowUl [35]
SFT
2B
16.9
14
9.1
0.0
2.5
0.0
13.2
7.3
15.3
7.5
10.3
2.2
UGround [23]
SFT
7B
26.6
2.1
27.3
2.8
14.2
1.6
31,9
2.7
31.6
11.3
17.8
0.0
16.5
U'Gruud-VV
7B
SceClick [7]
SFT
9.68
0.6
0.0
1.0
0.0
2.5
0.0
3.5
0.0
1.1
0.0
2.8
0.0
1.1
CogAgent [6]
SET
18B
14.9
0.7
9.6
0.0
7.1
3.1
22.2
1.8
13.0
0.0
0.0
UI-TARS [9]
SFT
2B
47.4
4.1
42.9
6.3
17.8
4.7
56.9
17.3
50.3
17.0
21.5
5.6
27.7
UL-TARS [9]
SET
7B
58.4
12.4
50.0
9.1
20.8
9.4
63.9
31.8
63.3
20.8
30.8
16.9
38.7
[30]
11.3
GUI-RI [10]
RFT
3B
26.4
7.8
33.8
4.8
40.9
5.6
61.8
17.3
53.6
17.0
28.1
5.6
GUIRI
RET
7B
23.9
6.3
49.4
4.8
38.9
8.4
55.6
11.8
58.7
26.4
42.
16.9
BTL-UI
RFT
3B
47.4
4.8
29.8
11.9
28.9
7.8
44.4
14.5
48.6
11.3
32.7
4,4
27.1
BTL.-UI
RFT
7B
53.9
7.3
26.7
15.9
35.9
14.6
47.2
13.0
62.7
24.7
$5.7
19.7
33.7
Table 5: GUI planning accuracy on AndroidControl [36] and GUI-Odyssey [37]. Bold means the best results, and underline means the second best results.
Model
Method
Model Size
AndrokdControl/
AndroidControl-En
GUI-Odyssey
Typc
GR
SR
Type
GR
SR
Typo
GR
SR
GPT-4o [39]
ZS
74.3
38.7
30.9
21.2
14.2
5,4
0S-Arlas-Base [23]
ZS
7B
73.0
73.4
50.9
57.4
54.9
29.8
60.4
39.7
27.0
Qwen2.5.VL. [5]
46.5
26.5
Qwen2.5.VL. VL
ZS
7B
83.4
87.1
62.5
68.7
59.7
47.1
55.6
37.8
34.4
[7]
93.0
62.9
A41)
SFT
3,98
87.7
67.3
43.2
10.2
86,8
36.5
Aguvis [24]
SFT
80.5
61.5
OS-Atlas Pro (23)
SIT
91.9
83.8
80.6
84.7
73.8
67.5
83.5
61.4
56.4
OS-Atlas-Pro [23]
SET
93.6
88.0
85.2
85.2
78.5
71.2
84.5
67.8
62.0
UJ-TARS [9]
SFT
98.1
87.3
89.3
81.2
78.4
68.9
93.9
86.8
83.4
UI-TARS [9]
SFT
7B
98.0
89.3
90.8
83.7
80,5
72.5
94.6
90.1
87.0
[30]
RFT
79.2
57.9
55.7
45.4
52.2
34.5
GULRI [10]
RFT
3B
83.7
$1.6
64.4
58.0
56.2
46.6
$4.8
41.5
41.3
GUL-RI [10]
BTL-UI
RFT
3B
95.6
86.1
84.8
84.0
71.4
63.4
84.4
77.2
64.0
BTL-UI
RFT
96.8
88.5
88.2
76.9
78.4
65.4
4.3
Ablation Study
As shown Table to clarify the contributions each component our BTL. framework, we conduct an ablation study on the AndroidControl benchmark. When trained only with SFT, BTL-UI achieves baseline performance with an SR of 60.6%. While further using the generated Blink data, SFT obtains 5% improvement. This proves that Blink data is not only suitable for RFT,
Table 6: Ablation study of BTL-UL. All ablation experiments evaluated on the AndroidControl- High benchmark by evaluating the grounding and planning capabilities of the agent.
(a) Ablation study of training method and BTL.. Blink Data refers the data contribution in $3.2. BTL. Reward denotes the reward
(b) Ablation study Blink Phase ROIs.
design in $3.3.
AndroidControl-HEn
A
Type
GR
SR
Blink
BTL
AndroidControl-HHne
SET
RFT
Reward
Type
1 87.0
72.1
66.6
2
87.6
72.8
67.4
68.7
79,4
63.9
60.6
3
88.0
74.2
68. 1
86.4
69.9
65.6
4
86.8
75.6
68.4
86.2
65.4
5
88.2
76.9
69.2
88.2
76.9
69.2
6
89.4
73.1
69.2

---

but also for SFT. Furthermore, RFT without Blink data achieves an SR of 65.6%. After adopting Blink data and BTL reward, SR is improved by 3.6%.
Moreover, we examine the effect of varying the number of Blink ROIs (A): increasing from 1 to 6 steadily improves success rates from 66.6% to 69.2%, after which gains plateau, suggesting an optimal trade-off between annotation complexity and attention coverage. It is observed that from Table as À increases, the performance is saturated, so the final À is selected as 5.
Overall, the ablation results confirm that each element the BTL framework- Phase for tar- geted attention, Think Phase for structured reasoning, and the Link Phase for precise validation- -layss a crucial role in achieving competitive performance in GUI interaction tasks.
4.4 Visualization
We present the visualization results and qualitative analysis in the appendix.
5 Conclusion and Limitations
We propose the BTL framework, an innovative GUT interaction architecture inspired by the biological cognitive paradigm of Blink-Think-Linn. This framework simulates the human closed-loop system of visual perception, cognitive decision-making, and action execution during GUI operations, over- coming the limitations of traditional outcome-driven RFT approaches. Experimental results show that the BTL-UI agent, developed under this framework, achieves significant performance improvements across a variety of GUI interaction tasks.
We believe that the BTL framework proposed in this study establishes a promising and generalizable paradigm for developing digital assistants that are more natural, efficient, and aligned with human cognition. It not only benefits human-GUT interaction but can also be extended to other human- computer interaction tasks, such as embodied intelligence.
The proposed BTL framework introduces <blink> tag outputs compared to conven- tional Think-Answer structured outputs. Although the gink-generrtte ROI regions are adaptive and can be empty (zero-length), they typically increase the output sequence length in most cases. While demonstrating performance improvements across various GUI task metries, this design incurs additional computational processing overhead.
Limitations.
References
[1] Shuai Wang, Weiwen Liu, Jingxuan Chen, Yuqi Zhou, Weinan Gan, Xingshan Zeng, Yuhan Che, Shuai Yu, Xinlong Hao, Kun Shao, et al. Gui agents with foundation models: A comprehensive survey, arXiv preprint arXiv:2411.00 244.0999, 2024.
[2] Xueyu Hu, Tao Xiong, Biao Yi, Zishu Wei, Ruixuan Xiao, Yurun Chen, Jiasheng Ye, Meiling Tao, Xiangxin Zhou, Ziyu Zhao, et al. Os agents: A survey on milm-based agents for general computing devices use, 2024.
[3] Dang Nguyen, Jian Chen, Yu Wang, Gang Wu, Namyong Park, Zhengmian Hu, Hanjia Lyu, Junda Wu, Ryan Aponte, Yu Xia, et al. Gui agents: A survey, arXiv preprint arXiv:2412.1883 13501, 2024.
[4] Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint 24Xi:2400911199, 2024.
[5] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, al. Qwen2.5-vl technical report. arXiv preprint 139233 2025.
[6] Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. Cogagent: A visual language model for gui agents. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 14281-14299 2024.
10

---

[7] Kanzhi Cheng. Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. Seeclick: Harnessing gui grounding for advanced visual gui agents. arXiv preprint arXiv:2401. 10935, 2024.
[8] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.12948 2025.
[9] Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang. Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, et al. Ui-tars: Pioneering automated gui interaction with native agents. arXiv preprint arXiv:2501.12322 2025.
[10] Xiaobo Xia and Run Luo. Gui-rl: A generalist rl rl-style Usiio-languagee action model for gui agents. arXiv preprint arXiv:2504.10455 2025.
[11] Yuhang Liu, Pengxiang Li, Congkai Xie, Xavier Hu, Xiaotian Han, Shengyu Zhang, Hongxia Yang, and Fei Wu. Infigui-rl: Advancing multimodal gui agents from reactive actors to deliberative reasoners. arXiv preprini arXiv:2504. /4239, 2025.
[12] Simon P Liversedge and John M Findlay. Saccadic eye movements and cognition. Trends in cognitive sciences, 4(1):6-14, 2000.
[13] Alejandro Jaimes and Nicu Sebe. Multimodal human-computer interaction: A survey, Computer vision and image understanding, 108(1-2):1 16-134, 2007.
[14] Robert JK Jacob. The use of eye movements in buman-computee interaction techniques: what you look at is what you get. ACM Transactions Information Systems (TOIS), 92):1522-66, 1991.
[15] Songqin Nong, Jiali Zhu, Rui Wu, Jiongchao Jin, Shuo Shan, Xiutian Huang, and Wenhao Xu. Mobileflow: multimodal IIm for mobile gui agent. arXiv preprint arXiv:2407.04344 240..04346666 2024.
[16] Yunpeng Song, Yiheng Bian, Yongtao Tang, Guiyu Ma, and Zhongmin Cai. Visiontasker: Mo- bile task automation using vision based ui understanding and Ilm task planning. In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology, pages 1-17, 2024.
[17] Xiao Liu, Bo Qin, Dongzhu Liang, Guang Dong, Hanyu Lai, Hanchen Zhang, Hanlin Zhao, Iat Long Iong, Jiadai Sun, Jiaqi Wang. et al. Autoglm: Autonomous foundation agents for guis. arXiv preprint arXiv:2411.0082 2024.
[18] Huawen Shen, Chang Liu, Gengluo Li, Xinlong Wang, Yu Zhou, Can Ma, and Xiangyang Ji. Falcon-ui: Understanding gui before following user instructions. arXiv preprint arXiv:2412.09362 2024.
[19] Fei Tang, Yongliang Shen, Hang Zhang, Siqi Chen, Guiyang Hou, Wenqi Zhang, Wenqiao Zhang, Kaitao Song, Weiming Lu, and Yueting Zhuang. Think twice, click once: Enhancing gui grounding via fast and slow systems. arXiv preprint arXiv:2503. 06t.:2503.06470, 2025.
[20] Filippos Christianos, Georgios Papoudakis, Thomas Coste, Jianye Hao, Jun Wang, and Kun Shao. Lightweight neural app control. arXiv preprint arXiv:2410.17882 2024.
[21] Jiani Zheng. Lu Wang, Fangkai Yang, Chaoyun Zhang, Lingrui Mei, Wenjie Yin, Qingwei Lin, Dongmei Zhang, Saravan Rajmohan, and Qi Zhang. Vem: Environment-fie free exploration for training gui agent with value environment model. arXiv preprint arXiv:2502.18966. 2025.
[22] Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. Navigating the digital world humans do: Universal visual grounding for gui agents. arXiv preprint 20.:24/00002243 2024.
[23] Zhiyong Wa, Zhenyu Wo, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, et al. Os-atlas: A foundation action model for generalist gui agents. In International Conference Learning Representations, 2025.

---

[24] Yiheng Xu, Zekun Wang, Junli Wang, Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, and Caiming Xiong. Aguvis: Unified pure vision agents for autonomous gui interaction. arXiv preprini arXiv:24/2.044555 2024.
[25] Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Camey, et al. Openai ol system card. arXiv preprint arXiv:2412.1677 2024.
[26] Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinforcement fine-tuning. arXiv preprint #rXiv:2503.0178. 2025.
[27] Yufei Zhan, Yousong Zhu, Shurong Zheng, Hongyin Zhao, Fan Yang, Ming Tang, and Jinqiao Wang. Vision-rl: Evolving human-freee alignment in large vision-language models via vision- guided reinforcement learning. arXiv preprint arXiv:2503. 18013, 2025.
[28] Huajie Tan, Yuheng Ji, Xiaoshuai Hao, Minglan Lin, Pengwei Wang. Zhongyuan Wang, and Shanghang Zhang. Reason-rft: Reinforcement fine-tuning for visual reasoning. arXiv preprint arXiv:2503.20752 2025.
[29] Haozhan Shen, Peng Liu, Jingcheng Li, Chumxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, al. VIm-rl: A stable and generalizable r1-style large vision-language model. arXiv preprini arXiv:2504.07611
2025.
[30] Zhengxi Lu, Yuxiang Chai, Yaxuan Guo, Xi Yin, Liang Liu, Hao Wang, Guanjing Xiong, and Hongsheng Li. Ui-rl: Enhancing action prediction of gui agents by reinforcement learning. arXiv preprint arXiv:2503.21620 2025.
[31] Yadong Lu, Jianwei Yang. Yelong Shen, and Ahmed Awadallah. Omniparser for pure vision based gui agent, 2024.
[32] Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1 ((12)):33997, 1955.
[33] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal 2017.
policy optimization algorithms. arXiv preprint
[34] Yuze Zhao, Jintao Huang, Jinghan Hu, Xingjun Wang, Yunlin Mao. Daoze Zhang, Zeyinzi Jiang, Zhikai Wu, Baole Ai, Ang Wang, Wenmeng Zhou, and Yingda Chen. Swiftra scalable lightweight infrastructure for fine-tuning, 2024.
[35] Kevin Qinghong Lin, Linjie Li, Difei Gao, Zhengyuan Yang, Zechen Bai, Weixian Lei, Lijuan Wang, and Mike Zheng Shou. Showui: One sasiio-lanuuage-cttin model for generalist gui agent. In NeurIPS 2024 Workshop on Open-World World Agents, 2024.
[36] Wei Li, William Bishop, Alice Li, Chris Rawles, Folawiyo Campbell-Ajala Divya Tyamagundlu, and Oriana Riva. On the effects of data scale on computer control agents, arXiv e-prints, pages arXiv-2406, 2024.
[37] Quanfeng Lu, Wenqi Shao, Zitao Liu, Fanqing Meng, Boxuan Li, Botong Chen, Siyuan Huang, Kaipeng Zhang, Yu Qiao, and Ping Luo. Gui odyssey: A comprehensive dataset for cross-app gui navigation on mobile devices. arXiv preprint arXiv:2406.0884
2024.
[38] Kaixin Li, Ziyang Meng, Hongzhan Lin, Ziyang Luo, Yuchen Tian, Jing Ma, Zhiyong Huang, and Tat-Seng Chua. Screenspot-pro: Gui grounding for professional high-resolution computer use. arXiv preprint arXiv:2504.079881 2025.
[39] OpenAl. 4o, 2024. Accessed: 2025-01-03.
[40] Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504. 204.994444,7, 2025.
[41] Yuhao Yang, Yue Wang, Dongxu Li, Ziyang Luo, Bei Chen, Chao Huang, and Junnan Li. Aria-ui: Visual grounding for gui instructions. arXiv preprint 4rX2v:2412.16256, 2024.
12

---

A Prompt
System Prompt
You are GUI Agent capable reasoning based on user instructions, action history, and the current screenshot. You should first observe the layout of the screenshot and extract N elements RELATED TO the user instruction, where <= N <=5. Next, think about the reasoning process BASED ON the observations and instructions your mind, and then provide the with the answer.
The observation process (can be None if N 0), reasoning process and answer are enclosed within <blink><//link>, <think></think> and <link></link> tags, respectively, i.e.. <blink>
celmmentxxidbl////ccbbbosssssee x0, y0, xl, httpA///we.ttooooooooooo//////on celementx<db222//bbcchhoooo y2, x3, httpa///a.a...aaaaaaaaaaccoocaaa
cdementxxid44//00.....eemmm
</blink>
<think> reasoning process here </think>
<link> answer(""Plan": ... "Action": ".unction"::....... </link>.
where captions must be one of [dynamic, static]. "dynamic" refers to the interactive area, and "static" refers to the con-interactive areas, such text and diagrams in the screenshot. And the observation can be <blink> None </blink>, if N 0.
User Instruction Prompt
You are given a task and your action history, with screenshots. You need to perform the next action complete the task. You MUST CHOÓSE the next action from the following defined action space.
## Action Space
Action 1: Back
format: ('function': 'Back')
purpose: back to the previous screen.
Action 2: Home
format: ('function': 'Home')
purpose: navigate to the home page.
Action 3: Tap
format: ('function': 'Tap', 'position': [x, y])
purpose: tap the specified position.
Action 4: Type
format: ('function': "Type', 'text': 'str')
purpose: enter specified text at the designated location.
Action 5: Swipe
format: ('function': 'Swipe', 'direction': 'str')
purpose: swipe on the screen in the specified direction.
Action 6: LongPress
format: ('function': 'LongPress', 'position': [x, y])
purpose: long press the specified position
## User Instruction
High Level Instroction
## Action History Step :......
Step 2: .....
## Screenshots
<image>
Table 7: The prompt for the BTL-UI.
The prompt of BTL-UI is shown in Table 7. The system prompt is used to format the output of the model according to the three-phase paradigm Blink-Think-Link, Moreover, the model outputs according the format of the system prompt, which is convenient for the calculation of the BTL
13

---

reward to adjust the model distribution. As shown in equation 6, because the output of Blink Phase can be 2, we emphasize that the Blink Phase can be output None in the system prompt.
OS-Atlas [23] has found that blindly mixing data from different sources for multitask fine-tuning can significantly harm performance due to action space conflicts. We unify the action space of GUI-Odyssey [37] and AndroidControl [36]. And we prompt the model to select the corresponding action from the defined action space, Furthermore, we declare the format and purpose in the user instruction prompt, so that the model can better understand each action type. For the grounding and high-level tasks, only the high-level instruction will be fed into the model. For the low-level tasks, both the high-level and the low-level instructions will be given to the model. In the end, the interaction history will be added to the user instruction prompt.
B Visualization
edenentingg 2iidd 270, 770,799,2000 115.2200///eeeeeeeeeeee /bbsseess om condnmmiiii loaph umvi clement
ieleentiggg 3wiidd/baaii 232 2209222200002 240)- e00 um condhnmnieooae hospe honv/ clemento
stether sondmmaao////g capt
k/binko
etrunto
The Lear wants lette has to Rede GupShop and snarch for other nodio etctione Currently, as on the N'Nea
Playing! suncen opp search redie stutiors, need to the search icon which lecoteed fop right thrsugh optiond
IPPlitt: the icon dearch for other redio statiens", "Mchion": (functionn "Tap", "pusition": [1010, 20731]]
3070 http:///oeoopitioooooeoooo https//dooooccooooeoeeeeeeeeeeee
eebementssi wttp:///ww.////ooooooo000 424, wt4p://baspoeepppoooooo www.yyomooooomoooooooooo home////heeee
www.eetttiddaattteeeeeeeoooooeee 424 314 (tt444999///ooooo.
dernentugeed/@@gbeeerrr 424 506 498
eleesentiii wttp:////w////ooooooo23 424.757 (17/488)///ooo
thiss
The user wants pearch for other rade stutions besides the current one. The poarch bar is aircody octive, po read to typc the nome snother station to find The noxt step is to type the nome of the desened radio etectin into the enarch bar. sknko
Vion" "Type the nome enother rado station", "Achion: ('function": "Type", "hex"" FM
Aindo
http///ppp///eeeeeeeeeeeeeeeeeee relorpentwithhl /tdwetasnn 25 4244 228, 504) (044)/10 hww.:///moooooooo coptiss-oocittteeeoe httpp///////pp///////o srpents
relonentooos /twbchooonn 424, 314 408 /bbosooosption ton mynoooo loclogh onw iilement:
eeleneebiitdd 404 404 -34 458 /55ss//bssosc form copp eleneen
weleeeettiii www.bbuu0024444400000000 424.767 767 (98 bbbooooa for www.yynmiis/oapepoooo wnn- clement>
/bbleoo
ttinko
The user wants to searck For other rodio stations bes des the current one. The search bor is stroody active do nead to type come onether station to It. next step is type nome the desired redio shation into soorch Hinlo
['iann "Select the search neadt for Radio SupShup 94.3 FM". "Action" ('function! "Top", ppottion" (546, 1342g0]
Figure 4: Visualization of the interaction trajectory of the proposed BTL-UI on AndroidControl High. The corresponding ID of this random case is 19477. And the high-level instruction is 'Listen live to Radio GupShup 94.3 FM and search for other radio stations." The tap icon in black is the prediction of BTL-UI, and the other is the ground-truth.
The visualization of the interaction trajectory of our BTL-UI is shown in Figure 4. The high-level instruction is "Listen live to Radio GupShup 94.3 FM and search for other radio stations." The Blink Phase can locate the ROIs related to the instruction. And the thinking Phase can reason based on the instruction, interaction history, and candidate area. As shown in step 2 of the interaction trajectory in
14

---

Figure 4, in the Blink Phase, BTL-UI not only locates the input box to complete the task, but also analyzes the historical search records in the screenshots.
However, since AndroidControl an offline interaction benchmark, there are some unreasonable labeling data. For instance, step 2 needs input the text of "94.3 FM" according the task instruction. But the search box in the screenshot after interaction shows "93.5 FM", which may affect subsequent interactions. In step 3, the labeled action click the search icon. And the search icon is also located in the Blink Phase. Due to the interaction errors in step 2 caused by data noise, the Think Phase of BTL-UI believes that clicking on the "94.3 FM" in historical search records in the screenshot is more reasonable. Therefore, we suppose our BTL-UI has stronger reasoning and error correction abilities.
C Experiment Statistical Significance
In this section, we report the experiment's statistical significance. The random factor that affects our results the sampling of the training process. As shown in Table I, the training data of our BTL-UI is sampled from various datasets. In the data sampling process, fix the random seed to 2025 to maintain reproducibility. And the sampled data is further adopted to generate Blink Data, following the pipeline in $3.2. Moreover, BTL-UI adopts the ms-swift [34] framework for RL training. During the training process, we also fix the random seed to 2025 to maintain reproducibility.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have polished our abstract and introduction to accurately reflect our main contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are discussed in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
16

---

Justification:
The paper does not propose a theory and does not have any theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Experiments are described in detail, including the dataset we use, the hyperparameters, and the model training details.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

reproducible, they Answer: [Yes]
Justification: The source code will be released upon acceptance with detailed instructions including data access and preparation, training and inference code, and the environment needed.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (kttps://nips.co
public/ Soadee//CodSuubmise
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //nips. http:/uulcggguuie het////oooSuSisseeeeee ssionPolicy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
All the detailed information is provided in the experiment setup.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The random factor that affects our results is the sampling of the training process. We show quantitative error metrics in the supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
18

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We show the computer resources in experimental details (NVIDIA H100 GPUs) in experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: : htt////urriscccoooo
Answer: [Yes]
Justification:
All the researches in this paper conform to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Our goal is as simple as to improve agent comprehension. This is not security. safety, or privacy-related research direction and related any potential harmful or malicious usage in the future.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
19

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All assets we used are properly cited.
Guidelines:
The answer NA means that the paper does use existing assets. The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi twwoee..ooo..... com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
20

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release assets. The codes and model checkpoints will be released upon acceptance.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
21

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: This paper just takes LLM for editing.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non non-standard components.
Please refer to our LLM policy (https / htt.://///c/oeeeene////222/20 for what should or should not be described.
22


Reviews:
[{"review_id": "FqvsZkpPxs", "reviewer": "Reviewer_gfqS", "summary": "1. This paper proposes a new structured output mechanism for RFT-training of GUI agents which uses \\<blink\\> tags for proposing likely ROIs in images, \\<think\\> tags for reasoning, and \\<link\\> tag for grounded action generation.\n    \n2. The Blink data generation pipeline produces ROI annotations for training using Omniparser for proposals and Qwen-32B model for ranking.\n    \n3. The authors propose a three stage BTL Reward\n    \n    1. Dual format reward for Template and content matching\n        \n    2. A novel blink reward that rewards the agent based on the overlap with generated ROIs with ROIs generated from blink data annotations. They use the Hungarian algorithm for allocation of matches.\n        \n    3. Link reward for action outcome evaluation which exactly matches the full action as well as ground truth coordinate", "questions": "1. Is this method’s overall accuracy upper bounded by the proposals that are generated by omniparser? What if the true ROIs do not get generated by Omniparser? Can this method still generate such ROIs as it's trained to favor (reward) the ones generated by Omniparser?\n    \n2. Please add the missing baselines mentioned and compare BTL-UI on Screenspot Pro.\n    \n3. The exact reward at the link stage might overfit to the exact actions found in the offline datasets, and the agent might not adapt and generalize to real-world use cases. Could you provide a justification and, preferably, experiments in online GUI environments?\n    \n4. Please explicitly note and clarify what x, y, and i are in the section on Blink reward; this will make it easier to follow your explanations. Also, include a brief description of what the Hungarian matcher is for better clarity.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. The structured output template is well motivated and generates GUI actions in a principled manner.\n    \n2. The writing and method are clear and easy to understand.\n    \n3. Clear ablation studies demonstrating the impact of Blink Data and BTL reward.\n    \n\n**Weaknesses**\n\n1. The authors claim SOTA results on multiple benchmarks, but some necessary, high-performing baselines are missing:\n    \n    1. No comparison with baselines like UI-Tars [1], UGround-v1 [2] on Screenspot\n        \n    2. Similarly, missing comparison with UI-TARS on the AndroidControl benchmark.\n        \n2. Missing evaluation on Screenspot Pro [3], which is a more challenging and recent test for GUI agents featuring diverse applications, high-res screenshots, densely packed realistic UI elements, and more complex scenes. This is a standard evaluation benchmark for GUI agents focusing on better grounding.  \n    \n3. Lack of evaluation in online environments, to demonstrate that the agent does not overfit to offline actions from the training datasets.\n\n**References**\n\n1. Qin, Y., Ye, Y., Fang, J., Wang, H., Liang, S., Tian, S., … Shi, G. (2025). UI-TARS: Pioneering Automated GUI Interaction with Native Agents (arXiv:2501.12326). arXiv.\n2. Gou, Boyu, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents. arXiv preprint arXiv:2410.05243 (2025).\n3. Li, Kaixin, Ziyang Meng, Hongzhan Lin, Ziyang Luo, Yuchen Tian, Jing Ma, Zhiyong Huang, and Tat-Seng Chua. ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use. arXiv preprint arXiv:2504.07981 (2025).", "was_revised": true, "final_justification": "The authors added the requested baselines and addressed my key questions. The authors have added the lack of tests on online environments in the limitations section. However, I think explicitly testing in online environments is necessary, which is why I would like to maintain my score of 4 and not raise it further.", "rebuttal": "Dear Reviewer gfqS,\n\nThank you for your valuable feedback. We sincerely appreciate your insights and are fully committed to addressing these points to improve the quality of our work.\n\n## Response to W1\nThank you for pointing out the need for a more comprehensive baseline comparison.  Our initial experimental comparison did not include UI-TARS due to oversight. We will add a comparison and detailed analysis with high-performing baselines, including UI-TARS and UGround-v.  We address each of the concerns below:\n- Screenspot: As disclosed in their technical report, UI-TARS is a strong GUI Agent developed based on Qwen2-VL with training on 50B tokens. While our BTL-UI is developed based on Qwen2.5-VL, with RFT on 4K samples.  Although the overall grounding performance of BTL-UI is still lower than that of UI-TARS, it shows certain advantages on the subtasks of Mobile and Web.\n\n|Method|Size|MT|MI|DT|DI|WT|WI|Avg|\n|-|-|-|-|-|-|-|-|-|\n|UI-TARS|2B|95.2|79.1|90.7|68.6|87.2|78.3|84.7|\n|UGround-v1|2B|89.4|72.0|88.7|65.7|81.3|68.9|77.7|\n|BTL-UI|3B|96.3|77.3|88.2|57.9|80.0|68.9|80.0|\n|UI-TARS|7B|94.5|**85.2**|**95.9**|**85.7**|90.0|83.5|**89.5**|\n|UGround-v1|7B|93.0|79.9|93.8|76.4|**90.9**|84.0|86.3|\n|BTL-UI|7B|**97.1**|83.8|90.2|70.7|88.7|**84.5**|87.2|\n\n- AndroidControl: As disclosed in their official repository, UI-TARS adopts the recent five images as input on multi-step interaction tasks for both training and inference, which is computationally expensive. However, in recent methods, a single image training and inference are adopted. Therefore, to ensure comparability, we reproduce the performance of UI-TARS based on the released checkpoint and prompt (*). Although the overall performance of BTL-UI is still lower than that of UI-TARS, under the same experimental settings (single image), BTL-UI is superior to UI-TARS.\n\n|Method|Size|Images|High|\n|---|---|---|---|\n|UI-TARS|2B|5|68.9|\n|UI-TARS*|3B|1|53.6|\n|BTL-UI|3B|1|63.4|\n|UI-TARS|7B|5|**72.5**|\n|UI-TARS*|7B|1|58.2|\n|BTL-UI|7B|1|69.2|\n- Why don't we use UI-TARS as the baseline? When we conducted experiments on UI-TARS, we found that it has a weak ability to follow instructions, as it cannot output in the specified format according to the designed prompt, but instead adheres to its original output format. Furthermore, BTL-UI requires deep reasoning in the Think Phase. We suppose the space for deep reasoning of UI-TARS with large-scale data SFT and DPO is relatively limited. Therefore, we finally developed BTL-UI based on a foundational model Qwen2.5-VL.\n\n## Response to W2\nThanks for highlighting the importance of the Screenspot-Pro benchmark. We would like to clarify that we have already evaluated our BTL-UI model on ScreenSpot-Pro, and the corresponding results are provided in Table 6 of the Appendix (Section A). Specifically, compared to existing baselines, our BTL-UI achieves SOTA performance on this benchmark across both 3B and 7B model scales. Furthermore, a brief comparison with UI-TARS is shown in the following:\n\n|Method|Model|Acc|\n|-|-|-|\n|UI-TARS|2B|27.7|\n|BTL-UI|3B|27.1|\n|UI-TARS|7B|35.7|\n|BTL-UI|7B|33.7|\n\nWe appreciate the opportunity to emphasize this result in the main paper.  In the revised version, we will highlight the ScreenSpot-Pro results more clearly in the main body (e.g., by merging them into Table 2 or Table 3) and discuss our model’s performance and robustness in this more challenging setting.\n\n## Response to  W3\nWe appreciate the reviewer’s concern regarding the agent's generalization capability beyond offline datasets. Indeed, evaluating GUI agents in online interactive environments is critical for demonstrating robustness and avoiding overfitting to dataset-specific trajectories.\nWhile our current evaluation focuses on standardized offline benchmarks (e.g, Screenspot series, AndroidControl, and GUI-Odyssey) adopted by recent methods, we emphasize that several components of the BTL-UI framework were explicitly designed to mitigate overfitting:\n- Our Blink Phase guides attention toward semantically relevant UI regions rather than relying on memorized element positions.\n- The BTL Reward introduces the Process Reward Model (PRM), which encourages generalizable decision-making rather than imitating fixed trajectories.\n- The proposed BTL-UI is optimized by Group Reward Policy Optimization (GRPO), a relative advantage-based policy optimization method that does not directly depend on supervision of ground-truth actions, further reducing the risk of overfitting to offline data, and demonstrating stronger generalization.\nWe acknowledge, however, that online evaluation is an important next step. In our future work, we will integrate BTL-UI into online testing platforms, such as OSWorld [1] and AndroidWorld [2], to assess real-time adaptability. We will include a discussion of this direction in the revised manuscript’s limitations section.\n\n## Response to Q1\nThank you for your insightful question regarding the potential dependency of our model on the proposals generated by OmniParser during the Blink phase.\nWe clarify as follows:\n1. The accuracy of BTL-UI  is not strictly upper bounded by OmniParser-generated proposals. The two-stage Blink Data Generation pipeline (Section 3.2) decouples raw element extraction from task-relevant filtering. While OmniParser extracts initial UI elements, the second stage leverages Qwen2.5-VL-32B to dynamically analyze, filter, and prioritize these elements based on task instructions and interaction history, effectively refining OmniParser’s outputs. Only data samples that can pass the blink calibration ('*Response to W1*' to Reviewer 96rE) are selected as high-quality data for RFT.\n2. At inference time, BTL-UI is free to predict any region it deems relevant. In particular, the Blink Reward (Eq. 6) awards a non‑zero reward even when the model outputs an empty blink set ($P_i=\\varnothing$), provided that either no ground‑truth ROI exists on‑screen or the chosen action belongs to the non‑interactive action set $\\mathcal{A}^*$. This design explicitly prevents the model from being forced to conform to OmniParser proposals when they are incomplete.\n3. Importantly, since the BTL-UI has the ability to understand screenshots and ground elements, the model operates independently, guided by its learned policy from BTL Reward over screenshots and instructions. Furthermore, the visualization (Appendix C) demonstrates that the BTL-UI is capable of discovering novel ROIs and reasoning beyond annotated ground-truth.\nWe agree this limitation should be more explicitly discussed, and we will add a clarification in the revision to ensure readers are aware that the OmniParser is used to extract UI elements from data structures but does not bound the model’s capability to discover or reason about novel UI regions.\n\n## Response to Q2\nPlease kindly refer to '*Response to W1*' and '*Response to W2*'.\n\n## Response to Q3\nWe appreciate the reviewer’s concern about the Link Reward’s potential overfitting to offline datasets and its generalization to real-world online environments. Here, we justify the design and provide supporting evidence, along with plans for online validation:\n1. Justification of the Link Reward Design: The Link Reward’s strict \"all-or-nothing\" criterion (reward=1 only if both action type and arguments are exactly correct) is intentionally designed to avoid reward hacking and ensure the semantic consistency between the agent’s intent and execution, which is critical for GUI interactions where even minor errors can derail tasks. Due to its focus on structural correctness rather than memorizing dataset-specific patterns, we suppose BTL-UI does not inherently overfit to offline data. In addition, since RFT is not supervised by the ground-truth directly during training like SFT, RFT has a lower risk of overfitting.\n2. Planned Online Environment Experiments: Our current experiments on offline benchmarks adopted by recent methods, including AndroidControl and GUI-Odyeesy, that mimic online variability. Moreover, BTL-UI inherently supports GUI multi-trajectory prediction through its GRPO-based optimization framework, enabling it to explore and evaluate multiple potential interaction paths for complex tasks. We suppose these suggest the BTL framework generalizes beyond static offline data, as it handles dynamic state transitions analogous to real-world online interactions. However, due to resource limitations, we are unable to conduct experiments in an online environment further. We will add the absence of online evaluation to the \"Limitations\" section of the revision and list it as further work.\n\n## Respone to Q4\nWe appreciate the reviewer's suggestion to improve the clarity of our Blink Reward formulation.\nIn response, we will revise Section 3.3 to define all variables involved in the reward computation explicitly. Specifically:\n- $i$indexes a given training sample.\n- $x$indexes the predicted region-of-interest (ROI) bounding boxes $p_i^x \\in P_i$\n- $y$indexes the ground-truth ROI boxes $g_i^y \\in G_i$ in Section 3.2\nWe will also revise the definition of index_math in Eq. (5) to clearly reflect these index roles.\nMoreover, we will include a brief inline explanation of the Hungarian matcher:\n```\n...we adopt the Hungarian matcher, a classical assignment algorithm used to compute the optimal one-to-one matching between predicted and ground-truth bounding boxes based on IoU scores, under a given threshold τ.\n```\n[1] Xie T, Zhang D, Chen J, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments[J]. Advances in Neural Information Processing Systems, 2024, 37: 52040-52094.\n\n[2] Rawles C, Clinckemaillie S, Chang Y, et al. Androidworld: A dynamic benchmarking environment for autonomous agents[J]. arXiv preprint arXiv:2405.14573, 2024."}, {"review_id": "g0DTClkrH4", "reviewer": "Reviewer_NRM3", "summary": "This paper introduces BTL-UI, a GUI agent framework based on the \"Blink-Think-Link\" (BTL) paradigm that models human cognitive processes during graphical user interface interactions by decomposing tasks into three sequential stages - Blink (visual attention and element localization using <blink> tags), Think (reasoning with <think> tags), and Link (action execution with <link> tags).\n\nThe framework includes two novel contributions: 1) an automated Blink data generation pipeline that creates region-of-interest annotations using a two-stage parsing and filtering process, and 2) a BTL reward mechanism that provides process-outcome integrated supervision through dual format rewards, blink rewards for ROI accuracy, and link rewards for action correctness.\n\nBTL-UI demonstrates state-of-the-art performance across multiple GUI benchmarks including ScreenSpot, AndroidControl, and GUI-Odyssey.", "questions": "Apart from the questions naturally raised in the weaknesses part, I would love to discuss more on the fundamental innovation of the paper.\nI might consider the core \"innovation\" of stage-wise processing can be more accurately characterized as a regularization technique designed to improve data fitting and behavioral control. If so, the imposed stage structure may fundamentally limit the framework's extensibility:\n- Information Transmission Loss: Inter-stage dependencies create bottlenecks where critical information may be lost or inefficiently transmitted between processing phases.\n- Local Optimization Constraints: Stage-wise optimization may prevent the system from achieving global optima that end-to-end learning approaches could discover.\n- Task Decomposition Rigidity: The fixed Blink-Think-Link structure may prove inadequate for tasks requiring different decomposition strategies or parallel processing of multiple information streams.\n\nHow do you consider the proposed solution useful in the long run, especially if the benchmark/application becomes more complex?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength\n- The cognitive-inspired \"Blink-Think-Link\" paradigm provides an interpretable framework that aligns GUI interaction with human cognitive processes, offering valuable insights into structured task decomposition.\n- Strong empirical performance across multiple benchmarks (ScreenSpot, AndroidControl, GUI-Odyssey) with consistent SOTA results and thorough ablation studies validating each framework component.\n- Well-designed reward mechanism integrating process and outcome supervision, automated ROI generation pipeline, and efficient GRPO optimization reducing memory overhead compared to traditional RL approaches.\n\nWeakness\n- As the authors acknowledge, the three-stage processing pipeline introduces significant latency that may compromise real-time GUI interaction requirements, potentially limiting practical deployment scenarios. However, modern GUI applications require sub-millisecond response times, and the additional processing stages could fundamentally compromise user experience in real-world deployments. Therefore, some missing points from the current paper could be considered as weaknesses:\n    - Latency Benchmarks: No metrics or experiments analyzing inference or interaction latency in practical real-world settings.\n    - Real-Time Optimization Strategies: No discussion of methods to mitigate computational demands (e.g., employing lightweight models, approximate inference techniques, or moving toward end-to-end systems that avoid intermediate steps).\n    - Scalability Testing: The absence of deployment experiments on consumer-grade hardware or devices limits insight into the GUI agent’s performance outside high-performance computing environments.\n\n- The experiments focus on structured tasks such as ScreenSpot and AndroidControl, primarily involving short-term interactive behaviors (clicking, positioning, swiping and other static tasks). For complex nested tasks (such as multi-step planning and non-linear task objective adjustments), it is not clear whether the reward mechanism or policy model can effectively handle critical intermediate steps. The reward design in this paper emphasizes \"process reward accuracy\" (such as Blink relative correct regions), but these rewards still lean toward explicit short-term task objectives and are not suitable for complex tasks that require continuous re-derivation of long-term goals.\n\n- The paper lacks crucial implementation details that compromise reproducibility: Missing hyperparameter specifications (learning rates, reward function weights, optimization schedules). Not sure these details would be released.", "was_revised": true, "final_justification": "The authors have addressed key concerns regarding latency, real-time optimization, and scalability, providing evidence of the framework's potential for practical deployment. While the framework shows promising results, further testing in real-world environments and more flexible task decomposition strategies are needed, but current version is enough for a good academic paper. Overall, the paper’s technical contribution outweighs the limitations, justifying a borderline accept rating.", "rebuttal": "Dear Reviewer NRM3,\n\nThank you for your valuable suggestions. We sincerely appreciate your insights. We arrange our responses in the order your questions are raised, and hope they clarify your concerns.\n\n## Response to W1\nWe sincerely thank you for raising this important point regarding latency, practical real-time optimization, and scalability, especially in the context of real-world GUI applications requiring fast and reliable response times. We respond to each sub-point in detail below:\n1. Latency Benchmarks: As discussed in the Conclusion and Limitations (Section 5), while our current work focuses on performance metrics like grounding accuracy and task success rate, we recognize the need for explicit latency analysis. We will supplement this by quantifying end-to-end latency across the BTL pipeline. A brief comparison is shown in the following:\n|Model|Size|Format|Tokens|Latency/ms| |\n|-|-|-|-|-|-|\n| | | | | bf16 (H100)|fp8 + medusa (4090)|\n|BTL|7B|Link|58.4|1231|412|\n|BTL|7B|Think + Link|127.7|2425|864|\n|BTL|7B|Blink + Think + Link|185.1|4247|1462|\n\nAs shown in the table, after quantizing the model and using Medusa [1] to accelerate it, the BTL can achieve real-time response. In future work, we will aim to decrease the number of output tokens to achieve real-time optimization.\n\n2. Real-Time Optimization: As discussed in the first point, model quantization and Medusa, which is a simple LLM Inference acceleration framework with multiple decoding heads, can achieve real-time optimization. Moreover, as shown in our ablation study (Table 5-b), the BTL framework supports adaptive $\\lambda$ scaling to reduce the number of ROIs in the Blink Phase, which can balance performance and efficiency.\n\n3. Scalability Testing: Beyond H100 experiments, we have tested on consumer hardware, specifically the RTX 4090, with quantized 7B or real-time interaction, ensuring the framework adapts to constrained resources.\n\n## Response to W2\nThanks for this important observation. We would like to clarify that while our experiments include structured benchmarks, they explicitly evaluate multi-step planning and long-term goal adaptation, with design choices in the BTL framework tailored to address such scenarios.\n1. The BTL Reward mechanism is designed to support continuous re-derivation of long-term goals through its process-oriented supervision. The Blink Reward provides fine-grained guidance at each step by evaluating ROI relevance to the current sub-goal, while the Link Reward ensures alignment with the global objective. And our ablation study (Table 5-a) confirms that the integration of process and outcome rewards is critical for complex tasks. Removing Blink Data or BTL Reward leads to significant drops in AndroidControl-High SR (from 69.2% to 65.6% or 65.4%), indicating that process supervision (e.g., Blink’s ROI guidance) is essential for maintaining alignment with long-term goals during multi-step execution.\n2. BTL-UI inherently supports GUI multi-trajectory prediction through its GRPO-based optimization framework, enabling it to explore and evaluate multiple potential interaction paths for complex tasks. Specifically, during policy optimization, the model generates a group of completions ($\\{o_1,o_2,...,o_n\\}$) for each GUI task instance,  each representing a distinct interaction trajectory spanning Blink, Think, and Link Phases. This multi-trajectory generation allows the model to consider diverse strategies—such as alternative ROI selections in the Blink phase or varying reasoning steps in the Think phase—before determining the optimal path.\nIn summary, while our current benchmarks include structured tasks, their design incorporates complex nested scenarios, and the BTL framework, via its process-outcome reward integration, demonstrates effectiveness in handling intermediate steps and adapting to long-term objectives. \n\n## Response to W3\nThanks for pointing out this important issue regarding reproducibility. We acknowledge that the initial submission omitted several key implementation details due to space constraints. In the revised version, we will include a comprehensive appendix section detailing the following:\n|Hyperparameter|Value|\n|-|-|\n|learning_rate|from 1e-6 to 0|\n|reward function weights|[0.1, 0.3, 0.6]|\n|temperature|0.9|\n|num_generations|8|\n|num_epochs|3|\n|per_device_train_batch_size|1|\n|gradient_accumulation_steps|2|\n|$\\beta$(KL coefficient)|0.04|\n|$\\epsilon$(clipping parameter)|0.2|\n\nWe will clearly state this in the revised Implementation Details(Section 4.1). Moreover, we will release the training code, data, and corresponding checkpoints to ensure reproducibility.\n\n## Response to Q1\nThank you for this profound observation, which prompts us to clarify the fundamental nature of the BTL framework and its distinction from mere regularization. We contend that the Blink-Think-Link structure is not merely a regularization technique but a biologically inspired cognitive paradigm that aligns AI-driven GUI interaction with human cognitive mechanisms, thereby addressing the limitations of existing approaches. We address your concerns as follows:\n1. Information Transmission Loss: The BTL framework is designed to minimize inter-stage information bottlenecks through deliberate architectural choices.  First, the Blink Phase's ROI predictions (`<blink>` tags) are explicitly fed into the Think Phase as structured inputs, ensuring that visual attention signals directly guide reasoning. Second, the BTL Reward mechanism ensures consistency across stages: the Dual Format Reward verifies structural coherence, while Blink and Link Rewards jointly monitor visual grounding and action execution, preventing information loss through multi-stage alignment. Our ablation study (Table 5) shows that the absence of Blink Data or BTL Reward leads to significant performance drops (e.g., SR decreases from 69.2% to 65.6% without Blink Data), confirming that inter-stage information transmission is both efficient and critical.\n2. Local Optimization Constraints: BTL-UI avoids local optima by globally optimizing the entire pipeline. The GRPO-based policy update (Equation 9) maximizes a joint objective that integrates rewards from all stages, ensuring Blink, Think, and Link Phases are co-optimized rather than tuned independently. The BTL Reward’s process-outcome integration (combining Blink’s granular reward and Link’s outcome feedback) further guides the model toward global Optimization. For instance, in the high-level task of AndroidControl, only the full BTL framework (with all stages and rewards) achieves the highest SR (69.2%), outperforming ablated versions that optimize stages in isolation. This demonstrates that the framework escapes local optima by leveraging cross-stage dependencies.\n3. Task Decomposition Rigidity: We appreciate the reviewer's insightful comments. Indeed, the fixed Blink-Think-Link decomposition may not be suitable for all GUI tasks (e.g., highly reactive or parallel-perception settings). However, the current structure is parameterized: for example, when no relevant ROI exists, the Blink Phase can output `<blink> None </blink>`, effectively collapsing to Think-Link. Moreover, the BTL framework is inherently extensible and can support adaptive or task-specific decompositions. Specifically, the Blink Phase can be conditioned on the instruction context to modulate its granularity of attention, focusing broadly or narrowly depending on whether the query is high-level or low-level.\nIn summary, BTL’s innovation lies in its cognitive alignment with human-GUI interaction, using structured phases to mimic biological processes rather than imposing arbitrary regularization. This design not only improves performance but also enhances interpretability and alignment with human behavior, addressing critical gaps in existing GUI agents.\n\n## Response to Q2\nWe appreciate the reviewer’s forward-looking question regarding the long-term applicability of our BTL-UI framework.\nWe believe our approach is particularly well-suited for scaling to more complex benchmarks and real-world applications for the following reasons:\n1. The core Blink-Think-Link paradigm mirrors the human cognitive process of visual attention, reasoning, and action, an architecture that naturally adapts to more complex GUI environments. As interfaces grow more intricate, such as nested menus and multi-app workflows, the Blink Phase can dynamically locate task-relevant ROIs. This ensures the framework does not rely on fixed element patterns but instead generalizes to novel, complex layouts.\n2. The GRPO-based optimization supports multi-trajectory exploration, enabling the model to evaluate diverse interaction paths (e.g., alternative ROI selections or reasoning steps) for complex tasks. This capability—generating and comparing multiple completions to maximize long-term advantage—allows BTL-UI to adapt to unexpected changes in complex applications.\nTherefore, we suppose the BTL framework remains useful as benchmarks and applications grow more complex, with empirical results validating its scalability across diverse task types.\n\n[1] Cai T, Li Y, Geng Z, et al. Medusa: Simple llm inference acceleration framework with multiple decoding heads, 2024[J]. arXiv preprint arXiv: 2401.10774, 2024."}, {"review_id": "CTHVw3EmvL", "reviewer": "Reviewer_96rE", "summary": "This paper introduces BTL-UI, a new GUI agent framework inspired by human cognitive processes during graphical interface interactions. The authors propose a three-stage model called Blink-Think-Link (BTL): (1) Blink for visually identifying relevant UI regions, (2) Think for reasoning about the task, and (3) Link for producing executable actions.\n\nTo support this framework, they design two key components: (1) A Blink Data Generation pipeline that automatically annotates regions of interest using a large vision-language model. and A BTL Reward, which integrates feedback across the three stages—rewarding structure, attention, and action correctness.\n\nThe resulting agent, BTL-UI, is trained using a GRPO-based reinforcement fine-tuning method and evaluated on multiple GUI benchmarks for grounding and planning. It achieves state-of-the-art performance across several tasks, surpassing both supervised and prior RFT-based agents.", "questions": "Please refer to weakness section. The confusion I had is around the dataset used for RFT, it would be great if the authors can provide more detailed information on the data.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- The paper proposes a new framework, Blink-Think-Link (BTL), inspired by how humans interact with GUIs. This adds an explicit visual attention step (<blink>) before reasoning and acting, which is not investigated in prior GUI agent work.\n- The authors design a full pipeline with automated ROI annotation (blink data), a structured output format, and a reward scheme that gives feedback across all three phases. The reward mechanism avoids issues like partial-credit “reward hacking” found in previous RFT works.\n- BTL-UI achieves state-of-the-art performance on several GUI benchmarks, showing improvements over strong supervised and RFT-based baselines. Ablations show that each part of the framework helps.\n\n\n**Weakness**\n- It is not clear how the final dataset samples used for RFT are selected from their original dataset. Are they randomly sampled from the original dataset and then go through the Blink data generation pipeline? Or are there criteria to select specific samples from the original dataset which would better fit the Blink data generation pipeline? \n- I am not very sure how the first stage of the data generation pipeline is done. are the bounding boxes extracted by an off-the-shelf detection model, or provided by the original dataset?\n- Since Qwen2.5-VL-32B is used for the second stage, how good is the data quality after second stage? I imagine the data quality will be bounded by the capability of Qwen2.5-VL-32B. It should be ok when we are finetuning smaller models I guess, but I am curious whether this is a general recipe to improve performance of large models, e.g., whether we can use this data to RFT Qwen2.5-VL-32B itself and have similar performance improvement.\n\n**Minor Weakness**\n- Typos\n  - L268/269: `BLink data` -> `Blink data`; sometime it is `blink data` or `Blink Data` throughout the text\n  - L269\" `archives and SR of xxx` -> `achieves an SR of`\n  - L268: `Blikn data` -> `Blink data`\n- Other presentation suggestions\n  - Define the acronyms when they are first introduced. e.g., GRPO", "was_revised": true, "final_justification": "The rebuttal has addressed my concerns. It would be great if the authors can include the experiments with RFT on larger models to further strengthen the paper, but I don't see it as a reason to reject based on my discussion with the authors. Therefore, I increase my score to accept.", "rebuttal": "Dear Reviewer 96rE,\n\nThank you for your valuable feedback. We truly value your insights and are fully committed to addressing these points to enhance the quality of our work.\n\n## Response to W1\nWe appreciate the reviewer's question regarding the selection of RFT dataset samples. As found in previous work [1], RFT requires higher data quality than SFT. We adopt instruction action check and blink calibration to select high-quality data for Blink data.\n1. Instruction Action Check: We found that there are samples in AndroidControl and GUI-Odyssey, where the labeled action doesn't match the instruction. To filter out high-quality samples, we use regular matching to extract keywords (e.g, click, scroll, and swipe) from the annotated actions and instructions, and verify their consistency.\n2. Blink Calibration: Each sample will generate several candidate elements with bboxes during the Blink data generation process. At the same time, the coordinates of the click and long-press actions are annotated in the original dataset. Then, we check if the coordinates of the annotated click are within the bboxes. Only data samples that can be matched are selected as high-quality data.\n\nFinally, we sample training data from source datasets after instruction action check and blink calibration, following a fixed random seed (2025) to ensure reproducibility. Moreover, we will release the training data to promote further study.\n\n## Response to W2\nThanks for seeking clarification regarding how bounding boxes are obtained in the first stage of our Blink data generation pipeline.\n\nTo clarify, the bounding boxes used in the Blink Phase are **not provided by the original datasets**, such as AndroidControl and GUI-Odyssey, as most of these datasets lack detailed UI element annotations. Moreover, as we discussed in '*Response to W1*', these datasets only provide the coordinates of the click, but not the bounding boxes. Instead, we extract candidate UI elements and their bounding boxes using an **off-the-shelf rule-based UI parser**, which we refer to as OmniParser.\n\nWe will clarify this more explicitly in Section 3.2 and Figure 3 of the revised manuscript. Thank you again for highlighting the need for a clearer explanation of this step.\n\n## Response to W3\nWe appreciate the reviewer’s concern about the data quality from the second stage of the pipeline and its applicability to larger models.\n1. Regarding data quality: The second stage, which uses Qwen2.5-VL-32B as the analysis model, filters and prioritizes UI elements based on task instructions and interaction history to generate ROI annotations. As we discussed in '*Response to W1*', the data samples that pass the blink calibration process are selected as high-quality data. Moreover, recent studies [1] suggest that RFT only requires a small amount of high-quality data to stimulate the model's capabilities. Therefore, we suppose the capability of Qwen2.5-VL-32B has less influence on the final data quality.\n2. Regarding applicability to larger models: The mainstream view of recent approaches [2,3,4] is that RFT can further stimulate the model's ability rather than injecting new knowledge into it. From this perspective, the capacity of base models and data quality are the key to determining the upper bound. As discussed in point 1, Qwen2.5-VL-32B is adopted as the analysis model to produce high-quality data, but not the knowledge distillation similar to a teacher-student network. Moreover, GUI-R1 adopted Qwen2.5-VL-7B to rollout high-quality data to RFT Qwen2.5-VL-3B and Qwen2.5-VL-7B, and both the performance improvements are solid. In summary, we believe that using the Blink data to RFT Qwen2.5-VL-32B, we can gain similar performance improvement.\n\n## Response to W4\nThanks for your careful reading and helpful feedback on writing clarity and presentation.\n1. We will correct all spelling errors and inconsistencies, including:\n  - Replacing “BLink data”, “Blink Data”, and “Blikn data” with the consistent term “Blink data” throughout the paper.\n  - Fixing “archives an SR” to the correct phrasing “achieves an SR”.\n2. We will also ensure that all acronyms (e.g., GRPO: Group Relative Policy Optimization) are properly defined when first introduced in the main text (Section 3.4) and included in the glossary of symbols in the appendix.\n\nThese revisions will be reflected in the updated version. We appreciate the reviewer’s attention to detail in helping improve the clarity and professionalism of the paper.\n\n## Response to Q1\nWe appreciate the reviewer’s follow-up request for more detailed information on the dataset used for RFT. We fully agree that a clear understanding of the data construction pipeline is essential for reproducibility and transparency. Regarding the confusion you mentioned, we have already responded point by point. We will clarify this more explicitly in Section 3.2 and Figure 3 of the revision. \n\n[1] Liu Z, Sun Z, Zang Y, et al. Visual-rft: Visual reinforcement fine-tuning[J]. arXiv preprint arXiv:2503.01785, 2025.\n\n[2] Gandhi K, Chakravarthy A, Singh A, et al. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars[J]. arXiv preprint arXiv:2503.01307, 2025.\n\n[3] Yue Y, Chen Z, Lu R, et al. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?, 2025[J].  arXiv preprint arXiv:2504.13837, 2025.\n\n[4] Shah D J, Rushton P, Singla S, et al. Rethinking reflection in pre-training[J]. arXiv preprint arXiv:2504.04022, 2025."}, {"review_id": "vIyyGvdhYN", "reviewer": "Reviewer_RGD1", "summary": "This paper proposes a new training framework for GUI agents, inspired by biological cognition. The framework simulates the human recognition process through three phases: blink, think, and link. In the blink phase, the agent quickly localizes relevant objects or regions. In the think phase, it formulates a plan to solve the given task. Finally, in the link phase, it generates a precise execution command. In addition to introducing this framework, the authors propose a method for generating training samples specifically for the blink phase. Using both the constructed dataset and existing training data, the authors fine-tune Qwen2.5-VL-3B/7B models with GRPO. The resulting models outperform several baselines on grounding and planning tasks.", "questions": "- Could you clarify how the blink module differs from standard attention mechanisms? It seems to perform a similar filtering function. A more detailed discussion may help to better justify the biological inspiration claim.\n\n- Since recent VLMs may already possess strong filtering abilities as the authors utilize Qwen2.5-VL-32B to build the dataset, what additional benefits does your biologically-inspired blink phase provide?\n\n- UI-TARS appears to be a relevant and strong baseline in this domain. Including such results could strengthen the paper. Also, I am wondering why there are gaps in performance scores between the two papers.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Quality and Significance\n- Leveraging concepts from biological cognition is a reasonable and promising direction for enhancing model performance.\n\n- Although this paper claims to be inspired by biological cognition, it is unclear whether the proposed method meaningfully embodies such principles. First, in the \"blink\" phase, the method aims to extract more relevant objects from an input image. However, this function appears analogous to the attention mechanism already present in existing VLMs. If the authors equate biological cognition with this kind of filtering process, it may overlap with what attention mechanisms already accomplish. Second, while the authors utilize Qwen2.5-VL 32B to collect the training dataset for the blink, we may argue that such ability is already equipped in recent models if Qwen2.5-VL 32B extract relevant elements correctly; if it is not, then the reliability of the collected dataset is questionable. While the method does achieve improved performance, the gains might stem primarily from the new training set—which appears to use the same domain as in the test set—rather than from any principled use of a biologically-inspired cognitive framework.\n\n- The performance comparisons should be strengthened. UI-TARS is a strong and relevant baseline, as it provides a medium-sized model (7B), pretrained weights, and demonstrates strong performance on grounding and GUI-related tasks. However, this paper does not compare against UI-TARS. Furthermore, UI-TARS uses several of the same baselines (e.g., SeeClick, OS-Atlas, Aguvis) and benchmarks (ScreenSpot, AndroidControl) as this paper. Notably, I observed discrepancies in performance scores between the two—for example, OS-Atlas-Base achieves 50.9 SR on AndroidControl-Low in this paper, compared to 85.2 SR in UI-TARS. It would be helpful to clarify the cause of this gap (apologies if I have misunderstood something).\n\nClarity\n- The paper is generally easy to follow.\n\n- Given the performance discrepancies noted above, it would be beneficial to provide more detailed descriptions of the experimental settings and evaluation procedures.\n\n\nOriginality\n\n- The level of originality appears limited, as many prior studies have drawn inspiration from biological systems (albeit not specifically in the context of GUI agents). Furthermore, as discussed above, it is questionable whether the proposed method genuinely reflects the principles of biological cognition.", "was_revised": true, "final_justification": "After reading the authors' rebuttal, I now have a better understanding of the paper's strengths and its contributions to the community, and I have accordingly raised my score.\n\nHowever, I strongly recommend that the authors include a comparison with UI-TARS in the paper, as it is a strong baseline and readers would likely be interested in such a comparison. Although the proposed method's overall performance is inferior to that of UI-TARS, its contribution remains valuable given its lower training cost. Additionally, releasing the training dataset would further enhance the utility and reproducibility of the work.", "rebuttal": "Dear Reviewer RGD1,\n\nThank you for your valuable suggestions. We sincerely appreciate your insights. We arrange our responses in the order your questions are raised, and hope they clarify your concerns.\n\n## Response to W1\nWe appreciate the reviewer's question regarding the selection of RFT dataset samples. As found in previous work [1], RFT requires higher data quality than SFT. We employ instruction action checks and blink calibration to select high-quality data for Blink data.\n\n### 1. Is the Blink Phase Redundant with Attention in VLMs?\nWhile existing VLMs use attention mechanisms to weight visual features of each patch, the Blink Phase explicitly models the biological dynamics of human visual attention. Specifically, saccadic eye movements rapidly locate task-relevant regions in discrete intervals. Unlike implicit attention weights in VLMs, Blink generates structured ROI annotations (`<blink>` tags) that mimic the \"spotlight\" of human visual focus. Moreover, previous approaches [2,3] have proven that Chain-of-Thought (CoT) can elicit reasoning in LLM. CoT demonstrates a strong ability to solve complex problems by explicitly outputting the reasoning process step by step, rather than directly answering. In dynamic GUI interaction tasks, BTL-UI mirrors how human visual attention combines stimulus salience and goal relevance by locating several key elements related to the user instruction and outputs them to promote further deep reasoning in the Think Phase. As shown in the ablation study (Table 5-a), if the model output is in the format of `<think>...</think>/n<link>...</link>` without Blink Phase, the success rate (SR) decreased by 3.8%. Therefore, we argue that the Blink Phase serves a distinct role that is not simply equivalent to built-in attention in existing VLMs.\n### 2. If Qwen2.5-VL-32B Can Extract Elements, Why Not Use It Directly?\nThis is a fair and insightful question. We clarify our rationale below:\n- We don't adopt Qwen2.5-VL-32B extract elements. Instead,  in the first stage of Blink data generation, we extract candidate UI elements and their bounding boxes and types using OmniParser, which is an off-the-shelf rule-based UI parser. \n- In the second stage, we use Qwen2.5-VL-32B as the analysis model, filtering and prioritizing UI elements from the first stage, based on task instructions and interaction history, to generate ROI annotations. Then, only data samples that can pass the blink calibration are selected as high-quality data. Specifically, each sample will generate several candidate elements with bboxes during the Blink data generation process. At the same time, the coordinates of the click and long-press actions are annotated in the original dataset. We check if the coordinates of the annotated click are within the bboxes. \n\nIn summary, we don't adopt Qwen2.5-VL-32B extract elements directly; instead, we produce high-quality data through a pipeline of elemental extraction, analysis, and verification. We will clarify this more explicitly in Section 3.2 and Figure 3 of the revision.\n\n### 3. Do Performance Gains Come Merely from In-Domain Data?\nWe suppose the performance gains stem from the BTL framework, not just domain overlap. Recent studies [1] suggest that RFT only requires a small amount of high-quality data to stimulate the model's capabilities. And the goal of RFT is to stimulate the model's capabilities rather than inject new knowledge [4,5,6]. Therefore, the quality of training data is more important than the domain. Moreover, as we discussed in the first point and the comparison with the recent RFT-based GUI Agent on various benchmarks, BTL-UI achieves more competitive performance, which is due to the BTL framework, inspired by cognitive studies.\n\n## Response to W2\nWe thank the reviewer for pointing out the absence of direct comparison with UI-TARS and for highlighting the potential discrepancies in baseline numbers.\n\n### 1. Comparison with UI-TARS\nWe acknowledge that UI-TARS is a relevant baseline, as it focuses on GUI interaction and shares benchmarks like ScreenSpot and AndroidControl. Our initial experimental comparison did not include UI-TARS due to oversight. We will add a comparison and detailed analysis with high-performing baselines, including UI-TARS and UGround-v1. A brief comparison is shown below:\n\n- Screenspot: As disclosed in their technical report, UI-TARS is a strong GUI Agent developed based on Qwen2-VL with training on 50B tokens. While our BTL-UI is developed based on Qwen2.5-VL, with RFT on 4K samples.  Although the overall grounding performance of BTL-UI is still lower than that of UI-TARS, it shows certain advantages on the subtasks of Mobile and Web.\n|Method|Size|MT|MI|DT|DI|WT|WI|Avg|\n|-|-|-|-|-|-|-|-|-|\n|UI-TARS|2B|95.2|79.1|90.7|68.6|87.2|78.3|84.7|\n|BTL-UI|3B|96.3|77.3|88.2|57.9|80.0|68.9|80.0|\n|UI-TARS|7B|94.5|**85.2**|**95.9**|**85.7**|90.0|83.5|**89.5**|\n|BTL-UI|7B|**97.1**|83.8|90.2|70.7|88.7|**84.5**|87.2|\n\n- AndroidControl: As disclosed in their official repository, UI-TARS adopts the recent five images as input on multi-step interaction tasks for both training and inference, which is computationally expensive. However, in recent methods, a single image training and inference are adopted. Therefore, to ensure comparability, we reproduce the performance of UI-TARS based on the released checkpoint and prompt (*). Although the overall performance of BTL-UI is still lower than that of UI-TARS, under the same experimental settings (single image), BTL-UI is superior to UI-TARS.\n|Method|Size|Images|High|\n|---|---|---|---|\n|UI-TARS|2B|5|68.9|\n|UI-TARS*|3B|1|53.6|\n|BTL-UI|3B|1|63.4|\n|UI-TARS|7B|5|**72.5**|\n|UI-TARS*|7B|1|58.2|\n|BTL-UI|7B|1|69.2|\n\n### 2. Clarification on OS-Atlas Score\nThanks for giving us the opportunity to clarify the performance issues regarding OS-Atlas.\n- OS-Atlas is a GUI foundational model developed based on InternVL-2 and Qwen2-VL with Continuous Pre-Training (CPT) on GUI-specific tasks. OS-Atlas-Base (50.9 on AndroidControl-Low) is the model with CPT and without additional Supervised Fine-Tuning (SFT). Thus, as shown in 'Method' in Table 4, the experimental results of OS-Atlas-Base are zero-shot (ZS).\n- Moreover, the results of OS-Atlas (85.2) in UI-TARS have been fine-tuned on the training set of AndroidControl.\n\nFor a more comprehensive comparison and to enhance the readability of the article, we will add experimental results of OS-Atlas-SFT to reduce misunderstandings. We hope our response clears things up for you.\n\n## Response to W3\nThanks for your thoughtful observation. While biological inspiration is not new in AI, our BTL-UI distinguishes itself through two unique dimensions:\n1. Novelty in Domain-Specific Adaptation: Most prior biologically inspired methods [7] target general vision/language tasks. In contrast, we are the first to adapt biological visual cognition to GUI interaction, a domain with distinct challenges: dynamic interfaces, multi-modal elements including text, icons, and buttons, and task-dependent relevance. This domain shift is non-trivial, as GUI agents require modeling human-like interaction logic, which demands reinterpreting biological principles for goal-driven, interactive scenarios.\n2. Mechanistic Depth Beyond Analogy: Our framework does not stop at superficial analogy to attention. Instead, it operationalizes specific biological processes:\n  - Blink Phase mirrors how human visual attention combines stimulus salience and goal relevance by locating several key elements related to the user instruction and outputs them to promote further deep reasoning in the Think Phase.\n  - BTL Reward Integrates process-outcome reward to simulate the “action-evaluation” loop in human cognition, where aided actions trigger re-focusing (Blink) on new ROIs. This mirrors how humans adjust visual attention based on interaction feedback, a mechanism absent in SFT of VLM.\n  - As shown in the ablation study (Table 5-a), if the model output is in the format of `<think>...</think>/n<link>...</link>` without Blink Phase, the success rate (SR) decreased by 3.8%.\n\nIn summary, our goal is not to replicate human cognition, but to abstract useful behavioral priors in GUI interaction tasks to improve the agent's visual perception and reasoning capabilities to enhance generalization.\n## Response to Q1\nThanks for raising this important question again. We have discussed this '*Response to W1*'. Please kindly refer to it.\n## Response to Q2\nThanks for raising this insightful question again. We have discussed this '*Response to W1*' and '*Response to W3*'. Please kindly refer to them.\n\n## Response to Q3\nThanks for highlighting the relevance of UI-TARS as a strong baseline in the GUI agent domain again. We have discussed this '*Response to W2*'. Please kindly refer to it.\n\n[1] Liu Z, Sun Z, Zang Y, et al. Visual-rft: Visual reinforcement fine-tuning[J]. arXiv preprint arXiv:2503.01785, 2025.\n\n[2] Wei J, Wang X, Schuurmans D, et al. Chain-of-thought prompting elicits reasoning in large language models[C]. Advances in Neural Information Processing Systems, 2022, 35: 24824-24837.\n\n[3] Guo D, Yang D, Zhang H, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning[J]. arXiv preprint arXiv:2501.12948, 2025.\n\n[4] Gandhi K, Chakravarthy A, Singh A, et al. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars[J]. arXiv preprint arXiv:2503.01307, 2025.\n\n[5] Yue Y, Chen Z, Lu R, et al. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?, 2025[J].  arXiv preprint arXiv:2504.13837, 2025.\n\n[6] Shah D J, Rushton P, Singla S, et al. Rethinking reflection in pre-training[J]. arXiv preprint arXiv:2504.04022, 2025.\n\n[7]  Fukushima K. Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position[J]. Biological cybernetics, 1980, 36(4): 193-202."}]

Author rebuttal:

Meta-review: This paper proposes BTL-UI, a GUI agent framework built on the “Blink-Think-Link” paradigm, which is inspired by human cognitive processes in graphical interface interactions. The framework decomposes interaction into three phases: Blink for localizing relevant regions, Think for reasoning, and Link for executing actions. To support this design, the authors introduce an automated Blink data generation pipeline and a novel BTL reward scheme that integrates process- and outcome-based supervision. The system demonstrates strong empirical performance across multiple GUI benchmarks, consistently achieving state-of-the-art results.

In general, the paper presents a well-motivated and novel framework that is both biologically inspired and technically sound. The reviewers appreciated its interpretability, clear methodology, well-designed reward system, and strong empirical validation through ablations and benchmark comparisons. The paper is clearly written, easy to follow, and makes a valuable contribution to the development of advanced GUI agents.

However, as pointed out by the reviewers, there are some concerns about the paper. These include limited discussion of latency and real-time deployment constraints, lack of scalability testing on consumer hardware, reliance on Qwen2.5-VL-32B for data quality in Blink annotation, and questions about whether the Blink module truly provides benefits beyond existing attention mechanisms. Additionally, some reviewers noted missing details in experimental setup and implementation, as well as the need for more online environment evaluation. The authors have addressed many of these concerns in the rebuttal, adding baseline comparisons, clarifications on dataset construction, and discussions on scalability and future work.

Overall, the strengths of the paper outweigh its limitations. The work provides a significant and original contribution to the area of GUI agents, backed by thorough experiments and thoughtful design. I recommend accepting the paper.
Decision: Accept (poster)