GUI-Reflection: Empowering Multimodal GUI Models with Self-Reflection Behavior
Authors: Penghao Wu
Shengnan Ma
Bo Wang
Jiaheng Yu
Lewei Lu
Ziwei Liu
Venue: NeurIPS 2025 poster ()
Keywords: GUI Agent
Multimodal Large Languge Model
Vision Language Model
Reflection
Primary area: applications

Abstract: Multimodal Large Language Models (MLLMs) have shown great potential in revolutionizing Graphical User Interface (GUI) automation. However, existing GUI models mostly rely on learning from nearly error-free offline trajectories, thus lacking reflection and error recovery capabilities. To bridge this gap, we propose GUI-Reflection, a novel framework that explicitly integrates self-reflection and error correction capabilities into end-to-end multimodal GUI models throughout dedicated training stages: GUI-specific pre-training, offline supervised fine-tuning (SFT), and online reflection tuning. GUI-reflection enables self-reflection behavior emergence with fully automated data generation and learning processes without requiring any human annotation. Specifically, 1) we first propose scalable data pipelines to automatically construct reflection and error correction data from existing successful trajectories. While existing GUI models mainly focus on grounding and UI understanding ability, we propose the GUI-Reflection Task Suite to learn and evaluate reflection-oriented abilities explicitly. 2) Furthermore, we built a diverse and efficient environment for online training and data collection of GUI models on mobile devices. 3) We also present an iterative online reflection tuning algorithm leveraging the proposed environment, enabling the model to continuously enhance its reflection and error correction abilities. Our framework equips GUI agents with self-reflection and correction capabilities, paving the way for more robust, adaptable, and intelligent GUI automation, with all data, models, environments, and tools to be released publicly.

GUI-Reflection: Empowering Multimodal GUI Models with Self-Reflection Behavior
Penghao Wu
Shengnan Ma % Bo Wang", Jiaheng Yu Lewei 9 Ziwei Liu
S-Lab, Nanyang Technological University Sense Time Research
Find of https::/penghaoott wu. w.. github..hh...... ii/iii///u /O//UI_ Reflection/
Project Page:
Might find the size setting
on connect Next Fant
Might find the setting
on wrong! No size
File setting wrong, so probably should
1. Recognize the mistake.
Bytes!
X X
2. Undo the incorrect oction and get bock track.
3. Summarize the mistake and make onother
on cornect
Current GUI Models
GUI Models with Self-Reflection Behaviors
Figure 1: Illustrative comparison of typical GUT models versus our proposed GUI model with self-reflection behaviors. While current models fail to recognize and recover from errors (left), our model (right) demonstrates the ability to: 1. Recognize its mistake; 2. Undo the incorrect action and get back on track; Summarize the mistake and make another try, ultimately succeeding.
Abstract
Multimodal Large Language Models (MLLMs) have shown great potential revolutionizing Graphical User Interface (GUI) automation. However, existing GUI models mostly rely on learning from nearly error-free offline trajectories, thus lacking reflection and error recovery capabilities. To bridge this gap, we propose GUI-Reflection, a novel framework that explicitly integrates self-reflection and error correction capabilities into end-to-end multimodal GUI models throughout dedicated training stages: GUI-specific pre-training, offline supervised fine-tuning (SFT), and online reflection tuning. GUI-reflection enables self-reflection behavior emergence with fully automated data generation and learning processes without requiring any human annotation. Specifically, 1) first propose scalable data pipelines to automatically construct reflection and error correction data from exist- ing successful trajectories. While existing GUI models mainly focus on grounding and UI understanding ability, we propose the GUI-Reflection Task Suite to learn and evaluate reflection-oriemt oriented abilities explicitly. diverse and efficient environment for online training and data collection of GUI models on mobile devices. We also present an iterative online reflection tuning algorithm leveraging the proposed environment, enabling the model to continu- ously enhance its reflection and error correction abilities. Our framework equips GUI agents with self-reflection and correction capabilities, paving the way for more robust, adaptable, and intelligent GUI automation, with all data, models, environments, and tools to be released publicly.
Furthermore, we built a
"Corresponding authors: ziwei. liu@ntu.edii sg
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Graphical User Interface (GUI) automation stands as critical frontier for enhancing productivity and accessibility across the vast landscape of digital applications and devices. The advent of large language models (LLMs) and multimodal large language models (MLLMs) has catalyzed significant progress in this area. Typical GUI agents can be mainly categorized into two groups: agent-based
frameworks and end-to-end GUI models. Agent-based frameworks 49
52
40
39 I usually leverage
the reasoning and generalization capabilities of some foundation models (e.g. GPT-4o (20) with agentic modules and external tools to complete tasks. While representing leap forward, such agent-based modular frameworks often rely on intricate prompt engincering and complex workflows, suffering from high computation and cascaded errors, potentially limiting their adaptability in real- world scenarios. As for end-to-end multimodal GUI models [14 43
44 300 which interact with GUIS more like humans do, the perception, reasoning, and action grounding are integrated within a single model. Such models not only promise more adaptable and scalable GUI agents but also provide potentially valuable avenue for studying broader aspects of artificial general intelligence P.
The current paradigm for training end-to-end multimodal GUI models includes a GUI pre-training stage to inject GUI-related knowledge into the base MLLM, followed by supervised fine-tumine (SFT) stage with demonstration trajectories. However, one big issue of this paradigm is that it relies heavily offline datasets composed of pre-collected successful interaction trajectories. While this approach teaches models to mimic expert demonstrations for specific tasks, it inherently limits their ability handle the complexities and unpredictability of real-world interactions. When encountering unfamiliar UI interfaces, incorrect attempts, or execution failures, these models lack the capability to recognize the error, understand its cause, or formulate corrective plan based failed attempts. Crucially, even though the base models (the base MLLM before GUI-specific training) they trained on might originally contain certain reflection and reasoning abilities, this offline SFT process, focused solely on successful examples, can inadvertenlly diminish such capabilities or behaviors.
Recent LLM research has shown that through online training like reinforcement learning (RL), the reasoning and reflection abilities of the base models can be greatly enhanced. Morcover, recent studies [3 48 [5] have shown that the verification and reflection behaviors of the base model crucial for the success of RL training, and such capabilities in the base models largely affect the performance upper bound in the RL stage. However, for end-to-end multimodal GUI models, the current paradigm makes it difficult to sample or explore potentially corrective or reflective behaviors after the offline SFT stage, and further RL training cannot effectively activate or enhance such reflection abilities. UI-TARS (30] explores the incorporation of reflection and correction behaviors through an online bootstrapping mechanism. However, its design primarily focuses on the final online feedback to guide learning.
stage and depends on
To address these fundamental limitations, we introduce automatic framework designed to explicitly integrate self-reflection and error correction capabilities into end-to-end multimodal GUI models throughout different training stages. We first decompose the reflection and error correction ability for GUI agents into three core capabilities: (1) verifying the correctness of previous actions and recognizing errors deviations, (2) backtracking when deviating from the correct trajectory, and (3) reflecting on erroneous attempts to learn from mistakes and inform subsequent actions. Our framework strategically embeds the learning of these abilities across distinct training phases, including GUI-specific pre-training, offline supervised fine-tuning, and online training, aiming to cultivate GUI models capable of robust error handling and adaptive recovery as illustrated in Fig [
Specifically. during the GUI pre- pre-training phase, while current efforts primarily focus on GUI visual grounding and general UI understanding, we identify critical gap: the lack of explicit training signals for reflection and correction related abilities, which leads to the degradation of reflection behaviors in the base MILM. To address this, we propose the GUI-Reflection Task Suite with Action Verification, Action Reversal, and Mistake-Informed Reattempt tasks, specifically designed to evaluate and cultivate the reflection-oriented capabilities for GUI models. Besides, we have designed a scalable automatic data pipeline to construct realistic reflection and error correction scenarios derived from existing offline successful trajectories and inject such data into the offline SFT stage. This allows the model to learn the behaviors of reflection and correction.
"See Appendix for detailed discussion of related work.
2

---

GUI Pre-training
Action
GUI Offline SFT
GUI Online Learning
Periecton-orieneed Skil Cultvation
Verification
Rellection behavior leerning from poror-frea-oo data
Learning interactions with rellection turiin
Action
Reversal
Should've
Mistake- informed
Reattempt
Figure 2: The GUI-Reflection framework includes (I) Learning basic reflection-oriented skills from GUI-Reflection Task Suite in the GUI pre-training stage; (2) Learning reflection and correction behaviours from automatically generated error scenarios in the offline SFT stage; (3) Continuously enhancing reflection and correction capabilities via reflection tuning in the online learning stage.
Furthermore, to further improve the GUI models through real interactions, we have built a robust and extensible environment for Android- based tasks. The environment includes 215 programmatic task templates in distributed framework. Within this learning environment, we design a simple iterative online reflection tuning algorithm. This algorithm allows the model to interact with the environment and receive automatically generated pre-error correction and post-error reflection supervisions to continuously enhance its general capability and reflection and error correction capabilities iteratively.
Our main contributions are: 1) We propose EUI-Reflection Task Suite, designed to explicitly train and evaluate the crucial reflection-oriented abilities of GUI models during GUI pre-training. 2) We introduce a scalable, automatic data pipeline to construct reflection and error correction scenarios from existing successful trajectories, enabling the injection of these behaviors during offline SFT without manual annotation. 3) We develop an online learning environment for mobile GUI agents and iterative online reflection tuning algorithm, allowing models to continuously enhance their reflection and error correction capabilities through online interaction and learning from mistakes.
2 GUI-Reflection Framework
In this section, we first describe the core architecture of our GUI model and then elaborate on our GUI-Reflection framework (illustrated in Fig 2. which injects the self-reflection and correction behaviors into GUI models through pre-training, offline SFT, and online training stages.
2.1
End-to-End Multimodal GUI Agent Model
2.1.1 Action Space
As we mainly focus on mobile tasks, the action space includes the following common atomic actions for mobile device interactions: Click, Long Press, Scroll, Type, Press Enter, Press Back, Press Home, Open App, Task Impossible, Task Complete. Besides the above common actions, we also include two actions that are often ignored in current datasets or methods. First, the GUI model often needs to retrieve and integrate information or answer certain questions, therefore, an Answer action added into the action space for the agent to provide certain information or answers. Second, many tasks often require dozens of steps to complete, and it would be implausible and inefficient to contain complete history information like past screenshots in the context. Therefore, it is necessary extract and keep certain information obtained in the process of the task execution as a reference for the usage in future steps (e.g. the model needs to search for different information online and send the information to someone via email). To achieve this, additionally define Memorize action which stores certain important information at certain steps into memory bank for future references. The detailed descriptions of actions provided in Appendix
2.1.2 Model Structure
We adopt Intern VL2.5-8B 13 the base MILM of our GUI model. The input of the model includes the following parts: 1) The overall instruction of the task; 2) The sereenshots of the past n steps; 3) The screenshot of the current step; 4) The memory bank; 5) The complete action history.
The model output consists of three parts: 1) action thought, 2) action description, and 3) atomic action. The action thought is the thinking process behind the action decided to take. And it may
3

---

contain aspects including analysis of the current screen state, assessment of the prior step's outcome, reflection about previous steps' actions, consideration of overall task progress, and the rationale for the chosen action. These elements are included dynamically and cohesively, only when pertinent to the decision process, without rigid structure or explicit labels. The action description describes the action in natural language, while the action atomic is certain action type with the corresponding parameters. Detailed input and output formats and examples can be found in Appendix E
Based on the input and output definitions, a training sample at step t in a trajectory can then be defined as (G, Mf It- 00:t-1; (2 thought ageee, at),, where G denotes the overall task goal, M, denotes the current memory content at step i, Jt-nct represents the screenshots of the past n steps and the current thought represents the action thought of the current
step, ao:--1 represents the complete past actions, at step, adesc represents the action description of the current step, and de the grounded atomic action. We use a, = (a thought aqese, az) to represent the action outputs.
2.2 GUI-Reffection Task Suite: Reflection-oriented Abilities in Pre-training
While GUI grounding and understanding are crucial for basic GUI interactions, we argue that it also important to maintain or enhance the model's nascent abilities for self-reflection and error recognition within the GUI context. In this pre-training stage, we do not directly incorporate the complete GUT-related reflection and correction behaviors, instead, we further decompose such reflection and correction behaviors into smaller ceflection-orienter atomic capabilities and design the GUI-Reflection Task Suite to evaluate and learn such capabilities.
Action Verification A GUI agent with reflection ability might execute an incorrect action due to limited knowledge or unfamiliarity with the task, but it would recognize the mistake by observing the outcome of the action. Recognizing the error or mistake is the very first and crucial step in the reflection and correction process. To address this foundational capability, our first introduced pre-training task is Action Verification. The core idea is to test the model's ability to determine if an implicit action, executed on a previous GUI state, accomplished a specific purpose, based on observing the resulting GUI state outcome.
In this task, the model is presented with screenshots of two consecutive steps together with a textual action purpose describing potential goals or outcomes that the action performed on the first screen aimed achieve. The model's 's objective is meticulously inspect the visual differences between the screenshots and judge whether that specific purpose was successfully fulfilled by the implicit action. To construct data for this task, we randomly sample paired screenshots together with the corresponding action from GUI trajectory datasets. We adopt an MLLM to annotate the true action purpose for this action to a positive sample and annotate a negative purpose, which corresponds to a different action and is not accomplished in the second outcome screenshot.
Action Reversal Our second task is termed Action Reversal. This task addresses the scenario where an undesired or incorrect action has been recognized, and the objective is to determine the subsequent action required to revert the GUI to its state immediately preceding the execution of the original action. In essence, the model learns how to effectively undo a given action and eliminate its consequences. This capability crucial for enabling more sophisticated error correction and exploration strategies.
We define this task as multiple-choie questions. The step-wise action, paired with the screenshot before and after the action execution, is presented to the model, and the model needs to choose the correct undo action. We construct the data from GUI trajectory datasets and use MLLMs to annotate the undo action and interference options.
Mistake-informed Reattempt After recognizing an error and potentially reverting the state, a critical reflective capability is to make informed new attempt based on the known mistakes. To evaluate this ability, we introduce the Mistake-informed Reattempt task. In this task, the model is first asked to ground GUI elements based on given instruction. We then identify the samples that are incorrectly grounded. The model is informed of the prior mistake and is asked to make a new prediction. This process can also be repeated multiple times with multiple failed attempts.
We construct both training and evaluation data for these three tasks. For the Action Verification and Action Reversal tasks, the training and evaluation data are constructed from the training and test splits of AndroidControl 21 and GUI-Odyssey (27). We have 1206 and 420 samples for the evaluation of these two tasks, respectively, where the evaluation data has been filtered by human annotators to

---

Questien: Fiperhes vieedders case
Questinn
FizzAlznggt
esscuion
unds CIK
Eactraction:
Parpase 3:
Mixing
Plesages
Options:
Segpane::
Sozi Bovel
Second Aptept
Folka the tarten spe. uoder
Ciit Acae [.kin)
Dechruction:
Face musk [ bepet The 1004
Pris Step
Cavent Stop
Answer:
Sariel Stop
Answeet:
soe)
First Altrepp Social Afhineg
Action Verification
Action Reversal
Wistabeinnfommed Resttempt
Figure 3: Examples of Action Verification (left), Action Reversal (middle), and Mistake-informed Reattempt (right) tasks from the -UI-Reflection Task Suite.
ensure correctness. For the Mistake-informed reattempt task, the training data is constructed from Wave-UI 22. AMEX [I] and OS-ATLASS
43 For this task, we evaluate directly on ScreenSpot [44 and ScreenSpotv2 [4 Examples of these three tasks are provided Fig 3 The
detailed data construction process and statistics are provided in Appendix
2.3 Automatic Grounded Action Annotation
Before introducing the reflection data generation for the offline SFT stage. we first discuss how to generate grounded action annotations for GUI models without human annotations. End-to-end GUI models need provide the final grounded action (e.g. the start and end position of scroll action or the clicked point of click action). Therefore, one difficulty of automatic action annotation for end-to-end GUI models to generate the grounded action together with the paired action thought consistently. In order to solve this problem, we utilize the strong generalization ability of the current general MLLMs and the specific action grounding ability of GUI models. More precisely, to annotate the action outputs (including the action thought, action description, and the grounded atomic action), we first utilize general MILMS to generate the desired action thought and action description. Next, we concatenate the generated action thought and description together with the input information for the GUI model, and let the GUI model output the grounded atomic action correspondingly. Due to the rugo-regressive nature of LLM. the GUI model outputs the atomic action conditioned on the provided action thought and action description. To ensure the generated grounded atomic action is consistent with the action thought and action description, we sample multiple atomic actions from the GUI model and utilize the MLLMs again for filtering.
2.4 Reflection Behavior in Offline SFT
During the SFT stage, the GUT model is trained on offline GUI interaction trajectories that are mostly error-free. The ability to recognize possible mistakes based on execution results and the ability to recover or learn from mistakes are greatly limited in such a training approach. Therefore, design scalable automatic data pipeline to create realistic reflection and correction data from the existing successful trajectories.
The difficulty of creating reflection and correction data from existing error-free trajectories is how to get an incorrect action and its corresponding outcome screenshot. We design two approaches to address this problem. For the first approach, we adopt an MLLM to modify the original goal G to G such that an action a, at a certain step becomes an incorrect one. The modified goal is constructed to make the now-incorrect action appear as an easy or natural mistake that a user unfamiliar with the app, button functions, or certain operations might make. Based on G and a,, we now construct the new action outputs step + 1 after the mistake. At step t + the agents should recognize the previous mistake made in the last step and make reflections in a,41 ttoought with ā and ā,+1 generated accordingly. Note that ē.+1 could be some rollback action, such press back, if the previous incorrect action āj leads to an off-track state or could also be some correction action where the agents can directly continue with the correct action towards the modified goal. A reflection data thouught sample (G, M It- (n- 0p:4; (1 att1 ā(+1) )) is then constructed.
Furthermore, for cases where ā press back, we assume the screenshot ie after execution of ăe+1 is the same as 7, Then we can further generate action output for step ( 2, in which we summarize the previous error step I, make reflections, and try a new correct action āe+2- This simulates the scenario where the agent learss from its mistake after backtracking and attempts an
5

---

Iterative Online Reflection Tuning
Step-dise fitering
Og.
Distributed Environment
Priorstape
Weighted Sampling
emorstep
Preverar Pepre correction
Training Pool
Programmate N.L'b-based Verffer
Task Pool
step
0x+1 port
Post
commctionn
Figure 4: The Iterative Online Reflection Tuning algorithm. It features weighted task sampling, verification. Successful
interaction in distributed environment, and Degraamatic/MLLLMMbaeeeee trajectories are further filtered step-wise, while unsuccessful ones are mined for correct prior steps and undergo automated pre-error and post-error (reflection) correction annotation.
alternative action to achieve the goal from the restored state, And this data sample can represented thought adese , 0x+2).
as (G, M (4 Le], [co:r, d++1]; 2e42
For the second approach, we keep the original task instruction goal. For some step t in the original successful trajectory, we construct an ineffective incorrect action à which should not change the screenshot I, (e.g. scroll down when it is already the bottom or click on some non-interactive element). Then assert this action before the actual ag, and modify the original d, thought d, athought by adding reflection content about the added ineffective incorrect action. The data sample created in this approach is represented as (G, M,, (4- 1):1, Ae], [4o a|; de
thouugh desc az).
2.5 Iterative Online Reflection Tuning
2.5.1 Environment
Effective online training necessitates diverse, efficient, and scalable environment. However, current public online environments for training mobile GUI agents 41 only include overly simple or repetitive tasks, which lack the complexity and diversity agents likely to encounter in real-world scenarios. To overcome these limitations, we developed a specialized environment for efficient online learning, testing, and data collection of mobile GUI agents.
Specifically, our environment includes 215 task templates across 11 Apps. Each task template can be instantiated randomly with dynamic parameters. Based on the complexity of the tasks, we split the tasks into two levels, where 135 level-1 tasks include relatively easier ones, and 80 level-2 tasks have higher complexity. Our platform distributed host-worker system. The workers only run CPU-intensive Android Emulators. and all GPU-related inference or training tasks are running on the host machine. For trajectory collection, the agent model deployed on the host machine receives environment observations from the worker and sends the predicted action back to the worker to interact with the environment.
For task evaluation, we support both programmatic and MLLM-based verifiers. Programmatic verifier directly evaluates the success of the task by accessing the device's system states and databases, providing accurate reward signals. For the MLLM-based verifier, the task information, action history, and the corresponding screenshots will be provided an MLLM, which determines whether the task is successful. This is helpful for tasks where some critical information or intermediate results needed for the evaluation are not accessible from the device states. Besides, the MLLM-based evaluation is also able to check the step-wise correctness of the trajectory, providing dense process reward. To improve the accuracy of the MLLM-based evaluations, we also provide task guidance for each task template, describing the general procedures and important points of the task. Details about the environment and verifiers are provided in Appendix H
2.5.2 Algorithm
We design iterative reflection tuning algorithm for the GUI model trained with offline SFT to further improve the general and reflection capabilities through interacting with our online environment.
To elaborate, in each iteration, the current GUI model collects multiple rollouts by interacting with the environment. Different from the regular filtered behavior cloning algorithm, which directly takes all successful trajectories for training and discards the unsuccessful ones, for successful trajectories,

---

Table Evaluation results on the three tasks in GUI-Reflection Task Suite
(a) Evaluation results on Action Verification and Action Re- (h) Evaluation results the distake-informan versal tasks. Ace denotes the overall accuracy. TN and TP Reattempt task. The average scores of the mobile, denote the accuracy for negative samples (action purpose desktop, and website subsets the benchmarks not fulfilled) and positive samples (action purpose fulfilled), are reported. The best numbers are in bold. respectively. All numbers are in %. The best numbers among open-souree models are in bold.
ScreenSpot
ScreenSpotv2
InteraVL3-88
71.59
72.02
Models
Action Verification
Action Reversal
2nd uttempt
73.84 2.25)
74,42 (* 2.40)
Acc
TN
TP
Acc
3rd attempt
75.13 3.54)
75.73 (- 3.7L)
pass 3
77.90 (1 6.31)
80.09 ( 8.07)
Semini-2.5-Flaa
87.85
92.02
83.69 $3.69
95.24
Gemini-2.5-Poo
88.22
88.04
88.40
95.71
GUI- GUI-Pretrain
83.58
84.84
Claude-3.7 Sonnntt
71.53
57.48
85.58
90.00
2nd attempi
84,50(1 0.92)
85.65 (" 0.81)
GPT-40
86.68
93.11
80.25
86.19
3rd attempt
84.75(1 1.17)
85.85 (? 1.01)
Qwen2.5 VL-BBB
76.36
83.41
76.90
pass
87.24 (3.66)
88.39 ( 3.45)
Qwe-2.5-V1.-721 72B
86.48
90.38
82.59
91.90
DU1-Pretrain-ed
85.12
86.88 89.27 ( 2.61)
IntemMLL2.5588
62.76
51.07
74.46
48.33
2nd attempt
88.00 2.88)
VL3-BB
60.11
26.86
93.36
63.80
3rd attempt
89.61 (1 4.49)
90.50 (* 3.62)
IntemWL13-78B
68.24
52.07
84.41
82.38
pass 3
87.35 (1 2.23)
88.84 (* 1.96)
GUI-Pretrain-88
57.95
21.55
94.36
40.71
BUll-PrttainnRel
87,56
93.53
81.59
93.81
we additionally check the step-wise correctness of each step and only keep the correct steps. For unsuccessful trajectories, we find the first step / where the model performs incorrect action. And all the steps before are kept for training. Then, for the incorrect action de at step I, a pre-error correction action pre is annotated to be the actual correct action for step t. For the step t + 1 after the execution of the incorrect action ae, post-crror correction action d.++1.post is annotated, in which the model recognizes the previous mistake and makes reflections. Furthermore, when the action astt,post is Press Back, assume the screenshot I1+2 is the same as Iç and assign u,+2 to be the same action as à, with additional reflection added to the action thought summarizing the previous mistakes and making new try accordingly. All the action annotations are automatically constructed utilizing a general MLLM and the current GUI model. The training data collected in this iteration is used fine-tune the current GUT agent model. This process is illustrated in Fig 4
After each iteration, the sampling weights for different types of tasks are dynamically adjusted based on the corresponding success rates in this iteration, such that more difficult tasks are sampled more in the next iteration. We also adopt the curriculum learning strategy, where in the first & iterations, only level-1 tasks are included. After the first k iterations, level-1 tasks with success rates lower than certain threshold are kept and combined with level-2 tasks for subsequent iterations.
3 Experiments
3.1 Training Data
For the GUI pre-training stage, besides the reflection-related data constructed, we also include GUI- related grounding, captioning, OCR, and VQA data. For the GUI offline SFT stage, use public mobile device GUI interaction datasets including AITW (33), AITZ (50), AMEX [I], GUI-Odyssey 27), and AndroidControl (2I). We unify the action annotation of these datasets and annotate the corresponding action thought and action description via Gemini-2.00Flan (38). The detailed statistics of the training data are provided in Appendix D
3.2 Evaluations on GUI-Reflection Task Suite
In this section, we evaluate tasks in our GUI-reflection Task Suite. For the Action Verification task and Action Reversal task, we include 1) closed-source models: Gemini-2.5- 2.5-ssh/Pro (38) Claude- 3.7-Sonnet (4) and GPT-4o
20): 2) open-source models: Qwen2.5-VLL-7777
[9 Intern L22.5-8B 53 3) GUI baseline: GUI-Pretrain, which is our base MILM pre-
13. and VL.-8/V1.3-8777 trained with regular GUT pre-training data, and GUT-Pretrain-Reee which is pre-trained with additional GUI-Reflection Task Suite training data.
7

---

As shown in Table la powerful closed souree models perform strongly on these two tasks. The 72B scale open-source models perform comparatively well, while 7B scale models have lower performance. especially for the capability to recognize the failed action (TN in Action Verification) that is critical for recognizing mistakes in the reflection process. Note that after the regular GUT pre-training, the GUI-Pretrain model performs much worse than the previous general MILM, indicating the loss of such reflection-related abilities after GUI-specific pre-training. After adding the corresponding data in the GUI pre-training phase, CUL-Pretrain-Re -ee retains and even greatly improves such capabilities, performing on par with the best closed-source models.
For the Mistake-informed Reattempt task, we evaluate models' ability to reattempt based on known mistakes on the instruction grounding benchmarks ScreenSpot [14] and ScreenSpotv2 [43]. For samples that are incorrectly grounded, we provide the incorrect predictions and ask the model to make another attempt accordingly, and repeat this process for rounds. We also provide the pass @3 (temperature=1.0) 1.0) results for comparison. As shown in Table Tbj we observe that the general MILM InternV1.3-8 and the GUI pre-trained model GUI-Pretrain pose limited ability to utilize the known mistakes for more informed attempts (the 3rd attempt performance is lower than pass @ass@)). After adding our reflection-related training data, the GUI-Pretrain- Ref baseline can more effectively utilize the mistakes make better predictions (the 3rd attempt performance is higher than pass@3). 3).
These experimental results show that large-scale general-purpose purpose MILMs possess some inherent reflection capabilities in the GUI context, while such capabilities are still very limited in smaller-scale models, and the standard GUI pre-training tends to further diminish these abilities. However, by incorporating training data from our reflection-oriented tasks during the pre-training phase, such essential capabilities can effectively improved.
3.3 Effectiveness of Reflection for GUI Agents
In this part, we continue from the GUI- Pretrain- Ref model and conduct experi- ments to validate the effectiveness of re- flection data in the SFT and online stages. First, we verify the effect of augmenting of- fline GUI SFT data with reflection data and conducting iterative reflection tuning in the online environment. We conduct experi- ments in our GUI environment by training models with the level-1 tasks for itera- tions and evaluating the performance on the level-2 tasks. As shown in Table the baseline model trained without reflection data in offline SFT and using only filtered BC achieves a success rate of 14.58% on level-2 tasks. Incorporating reflection data during the offline SFT stage significantly boosts this to 23.61% % with the same filtered BC online training. Critically, when our online reflection tuning algorithm is applied online, the success rate further improves to 34.75 34.72%, demonstrating the benefits of explicitly training for reflection at multiple stages.
Table 2: Ablation study reflection data in SFT and online training.
Reflection SFT
Online Algo.
Success (%)
Filtered BC
14.58
Filtered BC
23.61
+ Reflection Tuning
34.72
Table 3: Comparison of our model against other base- lines on AndroidWorld, showing Success Rates (SR).
Success (%)
Acc. Tree denotes Accessibility Tree. The best number in 8B-scale end-to-end models is marked in bold.
20
Baseline
Input
SR
Level-1
Agent-Based Based
GPT-4o UGround
Image Acc. Tree
32.8
Reflection Tuning Iterations
GPT-lo + Aria-Ul 45
Image Acc. Tree
44.8
GPT-4o Aguvis-7B 7B
Image
37.1
Figure 5: Success Rate (%) on Level-1 and Level-2 tasks across iterative reflection tun- ing iterations. Our iterative reflection tun- ing with curriculum learning strategy pro- gressively improves model performance.
End-to-End
Aguvis-72B A4
Image
26.1
UI-TARS-72B
Image
46.6
OS-Gensis-88 36
Image Acc. Tree
16.9
UI-TARS-7B 30
Image
33.0
GUI-Rellction-88 (Ours)
Image
34.5

---

We further conduct the full reflection tuning process in the online environment with all tasks. In the first 3 iterations. only level-1 tasks are included. and the tasks with success rates lower than 80% are combined with level-2 tasks for another 3 iterations. As shown in Fig 5] the model rapidly improves on level-1 tasks, eventually reaching around 90% and maintaining this high performance. For the more complex level-2 tasks, the agent starts at lower success rate and shows steady improvement, reaching 29.36% by iteration 3. When more challenging level-1 tasks are combined with all level-2 tasks, the learning on level-2 tasks continues robustly. This illustrates the effectiveness of the online reflection tuning algorithm, enabling the model to effectively enhance its general ability and rapidly learn to master previously unfamiliar and complex tasks.
Furthermore, to evaluate our model on more general and comprehensive tasks, we combine the training data collected in the online training stage with a similar-sized subset of the original offline data and fine-tune the offline SFT model to inject valuable reflection experiences while maintaining the generalization ability. We evaluate our model on the AndroidWorld [32] benchmark. As shown in Table 3 our model achieves a competitive success rate of 34.5% among end-to-end models, demonstrating the effectiveness of our proposed framework.
Task Instruction
Task Instruction
Toggle all alarms that are in AM in the Fossify Clock
In the LibreO Viewer open the njrh7 .doc under the Meetings sdk, gahone64 x86 storage area. Then export PDF with name er00r.pdf in the same director.
App.
Action Thought
Action Thought
200
200-
currently the settings dialag for the 10:30 am alarm. My goal is to tum this alarm the previous step, mista kenly clicked the alarm time instead of the toggle switth. correct this, I need to close this settings Cancel button to return to the alarm list.
that clicking 'All files' in the previous step did not change the screen, indicating it was incorrect action. mistakenly thought clicking the section header would reveal browsing options, but seems it's just proceed with opening file, I need to init'tee the file selection process. The correct way do this click the SELECT FILE TO OPEN button.
3:30
gas
$30.
10:30
930
specified
1:30
400
4.00
dialog
Task Instruction
In Simple Calendar Pro, create a calendar event in two weeks from today 17h with tite 'Meeting with the Team' and the description We will prepare for annual budget.". The event should last for 30 mins.
Action Thought Step2
intended open the iiplle Calendar Pro app, but mistakknlly clicked Googl Calendar pp icon. Now I am in the Google Calendar app, which is not correct app for this task. need go back the app drawer find and open correct Simple Calendar app.
5D
Action Thought Step3
am currently in the app drower previously clicked the wrong Calendar app (the blue one with '15", realizing wasn't the required Simple Calendar Pro. need to find the correct app to proceed with creating the calendar event. see two calendar apps here; the correct one is the orange Icon with the text Calendar" below It. click this icon now.
Figure 6: Illustrative examples of our GUI model's self-reflection and correction behaviours. The model demonstrates its ability to: (Top-left) identify and undo mis-elick; (Top-right) recognize an unproductive action and then find the correct interactive element; and (Bottom) recover from opening an incorrect app by navigating back and selecting the correct based learned cues.
Reflection Behavior Examples We also provide examples of reflection and correction behaviors from the GUI models trained with our framework in Fig We observe that, without our explicit reflection training, GUT models typically struggle to recognize or recover from errors, often leading to task failure when encountering unexpected situations. Conversely, our framework enables models to analyze mistakes and execute corrective steps, effectively navigating towards the task goal despite initial incorrect attempts.
4 Conclusion
This paper introduced GUI-Reflection, comprehensive framework designed to equip multimodal GUI models with essential self-reflection and error correction capabilities. By systematically integrat- ing reflection-relate
learning across pre-training, offline SFT, and online tuning, Guf-Reflection enables agents to recognize their mistakes, undo incorrect actions, and learn from these errors make better subsequent decisions.

---

Acknowledgments
This study supported by the Ministry of Education, Singapore, under its MOE AcRF Tier 2 (MOE- T2EP20221-001 MOE-T2EP20223-0000) and under the RIE2020 Industry Alignment Fund- Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contribution from the industry partner(s).
References
[1] Saaket Agashe, Jiuzhou Han, Shuyu Gan, Jiachen Yang, Ang Li, and Xin Eric Wang. Agent s: An open agentic framework that uses computers like human. arXiv preprint arXiv:2410.08164, 2024.
[2] AgentSea Team. Wave-UI Dataset.
wave-ui 2023.
[3] AgentSea Team. Wave-UI-251 Dataset.
wave ui - 25k 2024.
[4] Anthropic.
Claude 3.5
sonnet.
-laude-3-5-somnee
[5] Gilles Bacchler, Srinivas Sunkara, Maria Wang, Fedir Zubach, Hassan Mansoor, Vincent Etter, Victor Cärbune, Jason Lin, Jindong Chen, and Abhanshu Sharma. Screenai: A vision-language model for ui and infographics understanding. In IJCAI, 2024.
[6] Chongyang Bai, Xiaoxue Zang, Ying Xu, Srinivas Sunkara, Abhinav Rastogi, Jindong Chen, et al. Uibert: Learning generic multimodal representations for ui understanding. In IJCAI, 2021.
[7] Hao Bai, Yifei Zhou, Mert Cemri, Jiayi Pan, Alane Suhr, Sergey Levine, and Aviral Kumar. Digirl: Training in-the-wild device- device-control control agents with autonomous reinforcement learning. arXiv preprint arXiv:2406. 1196, 2024.
[8] Hao Bai, Yifei Zhou, Li Erran Li, Sergey Levine, and Aviral Kumar. Digi-q: Learning q-value functions for training device-control agents. arXiv preprint 2tdi::502.117660 2025.
[9] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502. 13923, 2025.
[10] Andrea Burns, Kate Saenko, and Bryan A. Plummer. Tell me what's next: Textual foresight for generic ui representations, In ACL Findings, 2024.
[11] Yuxiang Chai, Siyuan Huang, Yazhe Niu, Han Xiao, Liang Liu, Dingyu Zhang, Peng Gao, Shuai Ren, and Hongsheng Li. Amex: Android multi-annotation expo dataset for mobile gui agents. arXiv preprint arXiv:2407. 2tti:2247.114900 17490, 2024.
[12] Wentong Chen, Junbo Cui, Jinyi Hu, Yujia Qin, Junjie Fang, Yue Zhao, Chongyi Wang, Jun Liu, Guirong Chen, Yupeng Huo, Yuan Yao, Yankai Lin, Zhiyuan Liu, and Maosong Sun. Guicourse: From general vision language models to versatile gui agents. arXiv preprint 13317 2024.
[13) Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, al. Expanding performance boundaries of sounesource multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.052711 2024.
[14] Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Li YanTTa, Jianbing Zhang, and Zhiyong Wu. SeeClick: Harnessing GUI grounding for advanced visual GUI agents. In ACL, 2024.
[15] Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D Goodman. Cognitive behaviors that enable self-impres improving reasoners, or, four habits of highly effective stars. arXiv preprint arXiv:2503.01 3037,, 2025.
[16] Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. Navigating the digital world as humans do: Universal visual grounding for GUI agents. In ICLR, 2025.
10

---

[17] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et Deepseek-rl: Incentivizing reasoning capability in IIms via reinforcement learning. arXiv preprint M...i:220111294:.. 2025.
[18] Wenyi Hong, Weihan Wang. Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang. Yuxiao Dong, Ming Ding, et al. Cogagent: A visual language model for gui agents. In CVPR, 2024.
[19] Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Zhe Xu, Yao Hu, and Shaohui Lin. Vision-rl: Incentivizing reasoning capability in multimodal large language models. arXiv preprint arXiv:2503.06749) 2025.
[20] Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
[21] Wei Li, William E Bishop, Alice Li, Christopher Rawles, Folawiyo Campbell-Ajala, Divya Tyamagundlu, and Oriana Riva. On the effects of data scale on ui control agents. In NeurIPS, 2024.
[22] Yanda Li, Chi Zhang, Wanqi Yang, Bin Fu, Pei Cheng, Xin Chen, Ling Chen, and Yunchao Wei. Appagent v2: Advanced agent for flexible mobile interactions. arXiv preprint arXiv:2408. 11824 2024.
[23] Yang Li, Gang Li, Luheng He, Jingjie Zheng, Hong Li, and Zhiwei Guan. Widget captioning: Generating natural language description for mobile user interface elements. In EMNLP, 2020.
[24] Yuhang Liu, Pengxiang Li, Zishu Wei, Congkai Xie, Xueyu Hu, Xinchen Xu, Shengyu Zhang, Xiaotian Han, Hongxia Yang, and Fei Wu. Infiguiagent: A multimodal generalist gui agent with native reasoning and reflection. arXiv preprini S05ii::2501.055777 2025.
[25] Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinforcement fine-tuning, arXiv preprint wxXiv:2503.0178: 2025.
[26] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2C19,
[27] Quanfeng Lu, Wengi Shao, Zitao Liu, Fanqing Meng. Boxuan Li, Botong Chen, Siyuan Huang, Kaipeng Zhang, Yu Qiao, and Ping Luo. Gui odyssey: A comprehensive dataset for cross-app gui navigation on mobile devices. arXiv preprint arXiv:2406.0844 2024.
[28] Yadong Lu, Jianwei Yang, Yelong Shen, and Ahmed Awadallah. Omniparser for pure vision based gui agent, 2024.
[29] Fanqing Meng, Lingxiao Du, Zongkai Liu, Zhixiang Zhou, Quanfeng Lu, Daocheng Fu, Tiancheng Han, Botian Shi, Wenhai Wang, Junjun Hc, et al. Mm-eureka: Exploring the frontiers of multimodal reasoning with rule-based reinforcement learning. arXiv preprint arXiv:2503.07366 2025.
[30] Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, al. Ui-tars: Pioneering automated gui interaction with native agents. arXiv preprint arXiv:2501. 12326, 2025.
[31] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. NeurlPS, 2023.
[32] Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Mary- beth Fair, Alice Li, William Bishop. Wei Li, Folawiyo Campbell-Ajala et al. Androidworld: A dynamic benchmarking environment for autonomous agents. arXiv preprint 44545:2405.14573. 2024.
[33] Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. An- droidinthewild: A large-scale dataset for android device control. In NeurlPS, 2023.
[34] Darsh J Shah, Peter Rushton, Somanshu Singla, Mohit Parmar, Kurt Smith, Yash Vanjani, Ashish Vaswani, Adarsh Chaluvaraju, Andrew Hojel, Andrew Ma, et al. Rethinking reflection pre-training. arXiv preprint arXiv:2504.0400
2025.
[35] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. NeurIPS, 2020.
11

---

[36]
Qiushi Sun, Kanzhi Cheng, Zichen Ding, Chuanyang Jin, Yian Wang, Fangzhi Xu, Zhenyu Wu, Chengyou Jia, Liheng Chen, Zhoumianze Liu, et al. Os-genesis: Automating gui agent trajectory construction via reverse task synthesis. arXiv preprint arXiv:2412. 19723, 2024.
[37]
Srinivas Sunkara, Maria Wang, Lijuan Liu, Gilles Baechler, Yu-Chung Hsiao, Jindong Chen, Abhanshu Sharma, and James Stout. Towards better semantic understanding of mobile interfaces. arXiv preprint arXiv:2210.0266 2022.
[38]
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Saan-Batistte Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.1 /805, 2023.
[39] Junyang Wang, Haiyang Xu, Haitao Jia, Xi Zhang, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. Mobile-agent-v2: Mobile device operation assistant with effective navigation via multi-agent collaboration. In NeurIPS, 2024.
[40] Junyang Wang, Haiyang Xu, Jiabo Ye, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. Mobilc-agent: Autonomous multi-modal mobile device agent with visual perception. arXiv preprint arXiv:2401.16158, 2024.
[41] Taiyi Wang. Zhihao Wu, Jianheng Liu, Jianye HAO, Jun Wang, and Kun Shao. Distrl: An asynchronous distributed reinforcement learning framework for on-device control agent. In ICLR, 2025.
[42] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. NeurlPS, 2022.
[43] Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, and Yu Qiao. OS-ATLAS: Foundation action model for generalist GUI agents. In ICLR, 2025.
[44] Yiheng Xu, Zekun Wang, Junli Wang, Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, and Caiming Xiong. Aguvis: Unified pure vision agents for autonomous gui interaction. arXiv preprint arXiv:2412.044
2024.
[45] Yuhao Yang. Yue Wang, Dongxu Li, Ziyang Luo, Bei Chen, Chao Huang, and Junnan Li. Aria-ui: Visual grounding for gui instructions. arXiv preprint arXiv:2412.16256. 2024.
[46] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. NeurIPS, 2023.
[47] Yixin Ye, Zhen Huang. Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. Limo: Less is more for reasoning. arXiv preprint arXiv:2502.03387, 2025.
[48] Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? arXiv preprini arXiv:2504. J 13337, 2025.
[49] Chi Zhang, Zhao Yang, Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. Appagent: Multimodal agents as smartphone users. arXiv preprint arXiv:2312.13771, 2023.
[50] Jiwen Zhang, Jihao Wu, Teng Yihua, Minghui Liao, Nuo Xu, Xiao Xiao, Zhongyu Wei. and Duyu Tang. Android in the zoo: Chain-of-actions -ectiontthought for GUI agents. In EMNLP Findings, 2024.
[51] Junlei Zhang, Zichen Ding, Chang Ma, Zijie Chen, Qiushi Sun, Zhenzhong Lan, and Junxian He. Breaking the data b gui agents through task generalization. arXiv preprint arXiv:2504.1 10127, 2025.
[52] Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-4v(ision) is a generalist web agent, if grounded. In ICML, 2024.
[53] Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504. 10479, 2025.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction have clearly state the claims and the contributions the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: The limitations are discussed in Sec B in the appendix.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: There is no theoretical result in this paper.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have clearly described the details of our model structure, training data, training process, and implementation details in the paper.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification:
We do not provide access to the code or data during the submission phase.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper provide the training details in Appendix Sec D
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We do not report error bars as would be too computationally expensive.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: The paper provide the experiments resources in Appendix See D
9. Code of ethics
13

---

Question:
Does the research conducted in the paper conform, in every respect, with the
NeurIPS Code of Ethics
Answer: [Yes]
Justification: The research conforms with the NeurIPS Code of Ethics in every respect.
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper discusses both potential positive societal impacts and negative societal impacts in Appendix Sec
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
12. Licenses for existing assets
Question:
Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The creators or original owners of data and models used in this paper are properly cited, and the licenses for the data are provided the Appendix See D
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects. 15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [Yes]
Justification: The paper has describe how LLMs are used in the data construction pipeline and how LLMs are used as verifiers in the environment.
14

---

A Related Works
A.1 Mobile GUI Agents
Driven by the success of large language models and multimodal large language models, research in mobile GUI automation has seen significant advancements. Current approaches to developing mobile GUI agents can be broadly categorized into agent-based frameworks and end-to-end models, each with distinct characteristics and tradc-off'.
Agent-based By leveraging the advanced reasoning, planning, generalization capabilities, and broad knowledge of foundation LLMs (e.g. GPT-4o),, agent-based frameworks structure sophisticated agentic workftows. One approach uses foundation models to directly engage with GUI interfaces 44 22 39. Such methods usually depend on accessible device information, particularly accessibility trees, allow the model to ground actions at the element level. These systems primarily consume textual information from accessibility trees, often enriched with screenshots featuring Set-of-Mark (SoM) augmentations for improved visual understanding. To enhance their operational efficacy, these workflows frequently incorporate components such as memory [39], reflection [22] [9], knowledge documents [49 (22), task decomposition [39], and visual tool integration (28), thereby improving task
completion and overall agent robustness.
Another agent-based approach [16 45 combines powerful foundation model with specialized GUI grounding model. In this setup, the LLM handles high-level planning and reasoming, while the dedicated GUI grounding model is responsible for accurately identifying and interacting with GUI elements based on a low-level instruction or element description from the L.M.. UGround [16] trains universal GUI grounding model and combines it with a planning model. Aria-Ul [45] improves the grounding model by providing more task context, like overall task instruction and history information.
30 aim to directly map raw GUI inputs (task information and screenshots) to grounded actions within a single model. The common paradigm for these models involves two-stage training process: I) GUI-specific pre-training where the model learns fundamental GUI understanding and accurate grounding: ; 2) GUI offline SFT, where the pre-trained model is fine-tuned on demonstration trajectories to learn task sssciipecfiic behaviors Some methods [7] 41] 8 also apply reinforcement learning on the fine-tuned model, but experiments only conducted in relatively simple and repetitive tasks. Beyond only predicting the atomic action, recent methods adopt the CoT' idea to train the model to output additional components like the thinking process [50] 44 30 and low-level action description (50] 44) InfiGUIAgent (24) adds
End-to-end End-to-end GUI models [8 14
43 51
expectation--effectioon components in the training data, but they only use the existing offline data, with most steps being error-free, and the reflection component primarily learns confirm success rather than to actively diagnose and recover from a wide array of potential failures. UI-TARS (30) introduces an online bootstrapping process learn the self-reflection and correction behaviors. However, unlike our proposed framework, which employs a fully automated pipeline for generating reflection data and integrates reflection capability enhancement across pre-training, offline SFT, and online stages, UI-TARS's approach requires considerable human annotation efforts and focuses the learning of these behaviors only in their final online bootstrapping process.
A.2 LLM and MLLM Reasoning and Reflection
The pursuit of enhanced reasoning in Large Language Models (LLMs) has evolved from structured towards leveraging Reinforcement Learning. While initial RL.HF methods [35 31 showed promise, recent paradigms focusing on outcome-based rewards have demonstrated great potential to intrinsically cultivate complex reasoning and even emergent self-reflection [17]. For the multimodal domains, current research is actively exploring how to adapt similar RL techniques to improve multimodal reasoning 19 25 29 involving visual information like images and videos, though challenges related to data and effective training signals remain. Furthermore, recent studies underscore the critical importance of the inherent capabilities and behaviors present in the base models before task-specific fine-tuning or reinforcement learning begins. Research indicates that foundational abilities for verification and reflection are not merely helpful but often prerequisites for successful online learning and significantly influence the ultimate performance ceiling achievable through RI. current end-to-end GUI model training pipelines, which often rely heavily offline SFT with near
prompting and SFT data construction 22
46 47
48 [5]. This highlights a potential vulnerability in
15

---

crror-free data. Such approaches may inadvertently suppress or fail to cultivate these vital reflective capabilities present in the base MLLM.
B Limitation
In this work, the constructed reflection-related data focuses primarily on visual and action-grounded errors direct element functioning misunderstanding, potentially neglecting deeper and more complex errors, such as errors in high-level planning or complex task decomposition. Besides, our framework currently mainly focuses mobile environments. While the underlying principles generalizable, adapting GUI-Reflection to other platforms such as desktop systems or web-based interfaces may require domain-specific dataset construction and engineering adjustments.
C
Societal Impacts
GUI-Reflection has the potential improve digital accessibility and productivity by enabling more robust and error-tolerant GUT agents. However, these capabilities could also be misused for automated manipulation in malicious contexts. Furthermore, the reliance on synthetic data may introduce biases if not carefully curated, potentially leading unintended behaviors in sensitive applications. Responsible deployment, transparency usage, and alignment with human intentions are critical for maximizing societal benefit while minimizing risks.
D Training Details
Besides the reflection-related data construct in the GUI-Reflection pipeline, the statisties of other data and the corresponding license information we used in the GUT pre-training and offline SFT stages are provided in Table 4 and Table
Table 4: The detailed training data information for the GUI-Pretrainin Stage. The total number reported is at the element level, while in implementation, the elements on the same image are grouped single training sample in the multi-turn conversation format.
Data Source
Platform
Task Type
Total Samples
License
UI RefExp
Mobile
Grounding 16,660
CC BY 4.0
Widget Captioning
23
Mobile
Grounding
96,648
CC BY 4.0
SeeClick- Rico
Mobile
Grounding
173,275
CC BY 4.0
RICO Semantics
37
Mobile
Grounding
31,560
CC BY-SA 4.0
OpenApp 10
Mobile
Grounding
142,810
BSD-3-Clause license
AMEX
Mobile
Grounding
1,360,595
CC BY 4.0
OS-Altas
4
Mobile
Grounding
89,860
Apache-2.0
Wave-UI
Web
Grounding 79,412
MIT
Wave-U1-25K
Web
Grounding
24,978
MIT Apache-2.0
SeeClick-Wed 14]
Web
Grounding
2,968,695
GUTEnv 212
Web
Grounding
340,477
CC BY 4.0
ScreenQA 5
Mobile
VQA
62,401
CC BY 4.0
Table 5: The detailed training data information for the offline SFT stage.
Data Source
Platform
Total Steps
License
AITW
Mobile
19,831
CC BY 4.0
AITZ
50
Mobile
14,686
CC BY 4.0
AMEX
[
Mobile
39,023
CC BY 4.0
AndroidControl
21
Mobile
89,603
Apache-2.0
GUI-Odyssey
Mobile
102,202
CC BY 4.0
For the GUI pre-training, we train the model for epoch with learning rate of 10-5, For the SFT stage, train the pre-trained model for epoch with learning rate of 3 10-5 In each
16

---

reflection tuning iteration, we train the model on the collected data in this iteration for epochs with learning rate x 10-5 For our final model, we randomly sample 51694 samples from the offline SFT data and combine them with 63353 samples collected in the online iterations and finetune the model after offline SFT for I epoch with a learning rate 10-5, We use Adam W [26] optimizer for all the training. All the training is conducted 11.5 hours, and the SFT stage takes about 8.5 hours. Each training iteration during the online training stage needs about 2 hours, and the final training stage takes 4 hours.
32 H100 GPUs. The pre-training stage takes about
E
Implementation Details
F.1 Model Details
The detailed descriptions of the valid actions for our GUI model are provided below.
Action Space
CLICK [[x, y]]. Click the screen at position [x,y].
LONG_PRESS [[x, y]]. Long press the screen at position [x, y]. [ y1, x2, y2]]. Scroll from the position [x1, yil to [x2, y2] TYPE [tex]] Type in the text; content: text]. Store information into the memory. ANSWER [text] Answer with the text.
XEMORI2E
PRESS _HOSS_ Go back to the home screen. PRESS_ BACK. Go back the previous OPEN_APP (app__ name]. Open the app named app_name. PRESS_ENTER. Press the enter key. WAIT. Wait for device response. Indicate the task is completed.
TASK IMPOSSIBLE. Indicate the task is impossible.
The input and output formats of our GUI agent model shown below.
Input Format of the GUI Agent
<image>
<image>
The images are the screenshots from the past steps.
<image>
The image the current screenshot.
<INSTRUCTION> (user instruction): {goal}
<MEMORY> (stored memory content): (current memory)
<PAST ACTIONS> (past actions): faction history)
Based on the above information, your task is to reason about the next action and provide your thinking process and the action. Your output should follow the following format:
<THOUGHT>: the thinking process
<ACTION DESC>: the description about the next action
<ACTION>: the next action
Output Format of the GUI Agent
<THOUGHT>; {action thought) <ACTION DESC>: faction description)
<ACTION>: (action)
define the action descriptions to follow fixed formats, and the formats for different action types shown below.
17

---

Action Description Format
CLICK: click {element) to (purpose)
LONG_PRESS: long press the (element) to {purpose)
SCROLL: scroll {direction} to {purpose)
TYPE: type in the content '(content)'
MEMORIZE: memorize {memory_ (ummoryy ANSWER: answer with the text '0'
PRESS _HME. Go back to the home screen PRESS_HOME.
BACCK Go back to the previous screen OPEN :PP: open the "[app_name)' app PRESS PRESS_ENTER: press enter
WAIT: wait
task complete MASKIMMPOSSTBLS task impossible
In our model implementation, the screenshot history length n is All past sereenshots except the one in the last step are downsampled to 448 x 448, We also visualized the click point on the past screenshot using a red dot the corresponding action click or long press. The coordinates in the action representations are normalized to integers in the range 0 to 999.
E.2 Evaluation Details
The original scroll and swipe implementation in AndroidWorld 332 always so we modify it to make the scroll action trajectory follow the start and end points predicted by the model. The original type action implementation includes clicking the target element and typing, so we modify it to only include typing the text, and the model needs two actions (click + type) to complete the original type action. We find that in some defined are impossible for agents that do not use the UI element information, so we increase the maximum step by 5 for all test cases.
fixed trajectory,
of AndroidWorld, the maximum steps
F
Details of GUI-Reflection Task Suite
F1
Action Verification
For the action verification task, randomly sample step-wise data from AndroidControl [2]] and Odyssey [27] datasets. Each data sample consists of a ground truth action, the screenshot before the action, and the screenshot after the action. We only consider action types including CLICK, LONG PRESS, and SCROLL as the purposes of other actions are relatively fixed. Then, we extract the purpose from the action descriptions of the ground truth actions to be the positive purpose. To construct the negative purpose, we use Gemini-2.55Prro to annotate the corresponding negative purpose for this sample. The prompt for this annotation is shown in Table
For the evaluation data the action verification task, we have 603 positive samples and 603 corresponding negative samples from the Odyssey test split. For the training data, construct 16220 paired samples from AndroidControl and 15616 paired samples from Odyssey. The evaluation and training template for this task is shown in Table 9
F.2
Action Reversal
The data for this task is constructed in two steps. First, we sample step-wise action data paired with the screenshot before and after the action execution from existing datasets. Gemini-2.5-Pro instructed to generate the appropriate undo action for this data pair as the ground truth. Our model is supposed to learn from this action reversal process. The annotation MLLM is instructed to prioritize app-internal revert actions instead overly relying on the general Press Back action during this undo action generation process. When multiple revert actions are available, we select most straightforward and efficient option. After obtaining the correct undo action, we instruct Gemini-2.0-PP F.-Faahh to generate interference options. The Press Back action is excluded from the interference generation action space, as the functionality of the back button can vary significantly across different applications. For this task, when the current action is CLICK, semi-transpan red
18

---

Original Task Goal
New Tadk Goal
Open the Sky 100 and caneer foon search for octs using ao image hoves
Incoreet Action
Action Thought
Ineed
sonrc hor whih west camp mape bud mace keetly
pardicu
Sortin where to ven kon vooie Therefre should press Look bulton.
Action Thought R.v
openedile Etay app. indalll clicedd the man asacct bux but realind tuat eas isceeet da even sanonn :an one the onerth the top a the cameroo to con neatss which #sage
Stspt
Step 1+p
0kpp+
Figure 7: Example of the reflection data generated with the first approach in the offline SFT stage. The click point of the incorrect action is highlighted with a red circle on the first screenshot, and the click point of the action at step + 2 highlighted with a green circle in the third screenshot.
circle is painted at the click location on the first screenshot, serving as a visual cue. We construct the evaluation and training data from AndroidControl and Odyssey. We have 420 samples in total for the evaluation part and 8642 samples for training. The task template for this task is shown Table 10
F.3
Mistake-informed Reattempt
For this task, we construct training data from the Wave-UI (2), AMEX [. and OS-ATLAS ATLAS-Desktop datasets and directly evaluate on the grounding benehmark ScreenSpot [14] and ScreenSpot
43
To obtain the failed attempts in training data, we first train a GUI-Pretrain model with the three target datasets for this task excluded. Then we conduct inference on these three datasets and select samples with failed predictions. We also use the bounding boxes of other elements annotated on the same image as mistake candidates. To construct innfake-informeed training data, for each sample, we randomly choose to 5 failed attempts and provide these mistakes in the prompt. The bounding boxes of the mistakes are also drawn using red rectangles on the image. We have 31836 samples in total for the training of this task. The task template is shown in Table 11
G Details of Reflection Data in Offline SFT
We use two approaches to construct reflection data in the offline SFT stage. For the first approach, we first adopt Gemini-2, Pro (38) (prompt shown in Table to modify the original goal to make the action incorrect. With the modified goal G. we further generate the reflection action at t + with the prompt shown in Table 13 If the reflection action ā++1 is Press Back, we assume Is+2 = 4, Then step t 2, the agent needs summarize the previous mistake and make an informed new attempt. To obtain such action annotation, we first generate the correct action å at step t after the goal is modified to G using the prompt shown in Table 14 The action ag does not contain the reflection part about the mistake, we further modify the action thought de action description and grounded action by adding reflection content using the prompt provided in Table 15
tthought while keeping the ,thought to Ø(+2
For the second approach, we first create an ineffective incorrect action using the prompt in Table I6 And then we modify the original a, thought by adding reflection content about additional inserted ineffective action using the prompt in Table
We build reflection data from the AndroidControl dataset and obtain 17557 samples with the first approach and 1 15394 samples with the second approach in total. Examples of the reflection data generated via theses two approaches are provided in Fig 7 and Fig
H Details of Online Iterative Reflection Tuning
The Apps in our online environment with the task statistics and examples are shown in Table 21
For the MILLM-based verifier, we adopt Gemini-2.0-Flase as the MILMM and provide the complete sequence of screenshots, task goal, task guidance, and action sequence to it for judgment. The prompt for this process is provided in Table 18
19

---

Task Goal
0.O
cmm-Sppport Warsed Legginge
Ineffeclive Action
Wormnn' duppprt High-hheesedd regpings complete
Action Thought Reflection
lead deeaa Thes becassse soi 'Vlamee' Erapport ligh Sth-WWattee ated Leppingent with Possets is state lext and not a ciikkabl element, clikig had to effest am curmenty product about product, noed sarol down information
Figure 8: Example reflection data generated with the second approach the offline SFT stage. The click point of the ineffective action is highlighted with red circle the first screenshot.
In our online reflection tuning algorithm, we further check the step-wise correctness for those successful trajectories using Gemini-2.0-Fian with the prompt shown in Table For unsuccessful trajectories, we use GPT-4o and the prompt in Table 20 to identify the first error step. The pre- correction and post-reflection annotations are similar to the process used in Sec with additional task guidance provided for more accurate annotation.
I Detailed Experiment Results
provide detailed evaluation results on ScreenSpot and ScreenSpotv2 in Table 6 and Table
Table 6: Detailed evaluation results on ScreenSpot.
Mobile
Desktop
Web
Model
Avg
Text
Icon/ Coon/tidget
Text
Icon/Wir Cron/Widget
Text
Icon/W Coon/Widget
Intern
81.32
52.40
46.39
30.00
43.48
25.24
46.47
InternVL3-88
91.57
75.11
76.80
52.86
77.39
55.82
71.59
GUI-Pretrain
92.67
74.67
91.75
72.86
90.87
78.64
83.58
HU1-Pretrain-Ree Ref
95.24
79.48
92.78
69.29
90.43
83.50
85.12
Table 7: Detailed evaluation results on ScreenSpotv2.
Mobile
Desktop
Web
Model
Avg
Text
Icon/Widger
Text
Icon/ Coon/Widget
Text
Icon/ Icon/Widger Widget
InternVL2.5-BB
80.69
57.34
45.36
29.28
35.47
27.58
45,95
VL3-8B
92.76
79.15
75.26
52.14
75.21
57.63
72.02
GUI-Pretrain
95,86
78.67
93.30
71.43
91.45
78.33
84.84
GUI-Pretrain-Red
97.24
83.41
94.33
70.71
92.31
83.25
86.88
20

---

Table 8: Prompt for negative purpose annotation
## System Role
You are an expert understanding GUI operations.
Information Provided
You will receive the following information:
I. Two screenshots: the first screenshot corresponds to the current step, and the outcome
screenshot corresponds to the next step.
2. Current Action: action executed at the current step
3. Current Action Purpose: The purpose of executing the current action
The current action successfully achieved the current purpose (can be seen from the outcome of this action, which is the outcome screenshot). Your task is to come up with new purpose, such that it is not satisfied according the outcome screenshot. Note that your new purpose should be somewhat difficult and confusing, such that someone who is unfamiliar with the GUI operations or functions might think it is satisfied by the current action executed. But make sure your new purpose has no ambiguity so an expert can determine that it is not satisfied based the screenshots.
Note that the new purpose should be verb phrase (starting with a verb) describing the overall purpose and it should **not** be some direct low-level action instructions like "click the xxx'.
Make sure your new purpose is NOT satisfied after the execution of the current step. Important
Make sure your new purpose can actually be satisfied by doing some **single action** different from the current action on the **first screen**
You also need to provide a simple explanation about what single action on the current step can satisfy your new purpose and why an expert can from the outcome screen that your new purpose is not satisfied.
## Input:
Current Action: (action description)
Current Action Purpose: (action purpose)
## Output Format:
Directly output your results in json format as follows:
"new purpose": "your new purpose".
"explanation": "your explanation"
Table 9: Task template for Action Verification
<image> <image>
Given an action purpose and two screenshots (the first screenshot corresponds to the step before certain action while the second one is the outcome screenshot after the execution of the action). You need to judge whether the action purpose has been satisfied by the action executed between these screenshots based on the screenshots content. Directly answer Yes or No.
## Input
action purpose: (action purpose)
## Output:
21

---

Table 10: Task template for Action Reversal
<image>
<image>
You are expert in evaluating the hehavior of GUI agents that interact with Android phone interfaces. Your task to assist in training intelligent agents by identifying the correct #eeeeert operation*** to undo previously executed incorrect operation.
You will be given:
A **current action** * that was performed
A pair of screenshots:
**Screen A **; the UI before the action
**Screen *** the UI after the action
A set of "*six revert operation action choices (A-Fyes
The agent must choose **the one correct revert operation** that best reverts Screen B back to Screen A.
## Valid Action Space:
*penn app[app]** Open the specified app.
** *Click** Tap on specific UI element.
*ooogg Press** Long press on specific UI element.
**rroll*** Perform a scroll gesture on the screen.
Typee[text]*** Input the specified text into a text field.
PPers **;e**; Return the home screen.
Press Back**: Return to the previous screen.
** PPess Enter**; Confirm input using the enter/return key.
Evaluation Criteria:
The **correct revert operation** ** must be the most effective and reasonable way to return the UI from **Screen B** to #*Screen A * based on the change caused by the current action.
-Only **one option** is correct. The remaining five should be plausible but incorrect.
Evaluate the options based on:
Whether the action targets the correct UI element
Whether reverses the effect of the current action
## Input:
** Current *ction**
(action_desss
** *Choice*** A-F options (undo_options
Output Format:
Directly output the option letter only
Table 11: Task template for Mistake-informed Reattempt
<image>
You are given screenshot of a mobile phone screen, a question, and some incorrect answers have already been excluded for you.
You need to give a correct answer based on the screenshot. Notice that the correct answer should be different from the incorrect answers.
The question is: (grounding instruction]
The incorrect answers (also annotated using red bbox in the image) are: (incorrect answer) The correct answer is:
22

---

Table 12: Prompt for instruction modification.
## System Role
You are an expert in understanding the operations from GUI agent. The agent's task to help human users operate an Android phone by completing specific instructions provided by the user. Your goal to modify the original instruction such that the action taken by the agent becomes incorrect in the context of your modified instruction.
## Valid Action Space
The actions that the agent may take to complete the task are as follows:
Open app[app]: Open the 'app" APP
Click: Click on the current screen.
Long Press: Long press the current screen.
Scroll: Scroll on the screen.
Type[text]: Type the 'text' into the input field.
Press Home: Return to the home page.
Press Back: Go back to the previous page.
Press Enter: Click the enter button.
Wait: Wait for the device respond load something
Task Complete/Tase Impossible: The agent indicates the task is completed or impossible. ## Information Provided
You will receive the following information:
1. Task Instruction: The original task instruction for the GUI agent.
2. Past Actions: The past actions taken by the agent to complete this task before this step.
Empty if the first step.
3. Future Actions: The future actions taken by the agent to complete this task after this
step. Empty if it the last step.
4. Current Action: The action taken by the agent the current step.
5. Screenshots: The screenshot at the current step and the screenshot at the next step
after the current action is executed. The click point is highlighted using red dot in the current screenshot and the scroll positions are visualized using a red arrow in the current screenshot.
Your job is to analyze the provided information to provide modified instruction. With the modified instruction, the current action becomes incorrect, while the past actions are still correct (compatible with the modified instruction). Incorrect action means that the current action taken by the agent is clearly wrong or unnecessary for completing the modified instruction deviates from the correct way.
Important
Your modified instruction should be natural, reasonable, and also realistic. With your modificd instruction, the now-incorrect action (**current action**) becomes an easy or natural mistake a user who is unfamiliar with the App, button functions, or certain operations might make. The user will probably realize the mistake when seeing the execution of this action (the second screenshot).
## Useful Guidelines
make the evaluation process more accurate, please follow these guidelines:
1. Based on the information, judge the feasibility of having a reasonable instruction meeting the mentioned requirements. Ignore the following if not.
2. Provide your modified instruction.
3. Explain why the modified instruction is reasonable and realistic, and does not change the correctness of the past actions.
4. Explain why the current action** becomes incorrect with your modified action, and explain why it is easy natural mistake a user might make.
Note that the red dot and red arrows are just for visualizing the actions; do not mention them in your response.
Note that the future actions are provided to better understand the overall task and context; you do not need consider them when creating modified instruction.
23

---

Note that your modified instruction should clearly be possible for the agent to complete based on the provided information.
Input:
Task Instruction: jonsskinttuuctio
Past Actions: {past_actions
Future Actions: [future_actions actions)
Current Action: {action]
## Output Format: Directly output your results in json format follows:
[[
"Task Feasibility": "Yes No", if no, the following entries should be empty. "Modified Instruction": "Your modified instruction". "Explanation 1": "Explain why the modified instruction is good and compatible with action history.".
"Explanation 2": "Explain why the current action becomes natural mistake for user not familiar with the app and certain operations."
11
Table 13: Prompt for reflection action annotation 0.+1 of the first approach in offline SFT.
System Role
You are an expert in correcting the step-wise operation of GUI agent. The agent's task is to help human users operate an Android phone by completing specific instructions provided by the user but it performs an incorrect action at a certain step. Your task is to make a reflection about the previous incorrect step and reason about the next correct atomic action after this mistake, provide the action thought, and the action type.
## Definition of Action Thought
The action thought is the rationale behind the actual action taken at a certain step. The action thought should be compact paragraph consisting of 3-4 sentences. It could include the following aspects if they are helpful and important for the action reasoning process: observation: concise description of the current screenshot, focusing the task related content and progress.
reflection: simple analysis of the unexpected situation caused by the incorrect action in the previous step, admit and analyze the mistakes.
action rationale: a brief reasoning process to integrate the above information and provide a natural thinking process leading to the actual action from the perspective of the operator The action thought should be natural and logically fluent. The action thought should *not* be clearly separated into the above aspects and does not need to mention the keywords like observation and reflection.
Information Provided
You will receive the following information:
1. Task Instruction: The overall task instruction for the GUI agent.
2. Past Actions:
A list of past actions prior to the previous incorrect step. Empty if it is the
first step.
The incorrect action taken at the previous step. 4. Error Action Analysis: An analysis from an expert explaining why the previous action is incorrect.
3. Previous Incorrect Action:
5. Screenshots: The first screenshot corresponds to the step before the incorrect action is taken while the second screenshot corresponds to the current step after executing the incorrect action. The click point is highlighted using a red dot in the first screenshot.
Your job is to analyze the provided information and provide the action thought leading the agent should take at the current step after the execution of the previous incorrect action.
24

---

## Valid Action Space
The actions that the agent may take complete the task are as follows:
Click: Click at certain position on the current screen.
Long Press: Long press a certain position on the current screen.
Scroll: Scroll on the screen, scroll down/up/leftrrigtt where the direction is the opposite
direction the figure movement.
Type[text]: Type the "text" into the input field.
Press Home: Return to the home page.
Press Back: Go back to the previous page.
Press Enter: Click the enter button.
Wait: Wait for the device to respond load something.
Task Complete/Tast Impossible: Indicate the task is completed or impossible.
## Useful Guidelines
To make your reasoning and response accurate, please follow these guidelines:
1. Analyze the provided information
2. Provide your action thought. In the action thought, use first-person description, that is, using 'I' instead of "the user' or "the agent'
3.
Provide your action type which must be consistent with your action thought.
The correct action type should be one of the following: [Click, Long Press, Scroll, Type, Press Home, Press Back, Press Enter, Wait, Task Complete, Task Impossible]
Important Notes:
Note that the red dots for click and long press only for better understanding of the actions; do not mention them your action thought.
Note that some sub-task might need multiple actions (e.g. typing something needs clicking the text input field and then typing) and you should only give the very first atomic action for the current step.
Note that the previous step type in some incorrect content, you have to first find ways to clear it before typing the correct one.
Note that when you want to click or press the backspace button, the correct action type is click instead of Press Back (which means going back the previous page). And in such cases, try to use the word 'click' instead of 'press' in your action thought.
## Input:
Task Instruction: (task_instruckee
Past Actions: ppast_actions:
Previous Incorrect Action: [incorrect_action
Error Action Analysis: [error )
## Output Format:
Directly output your results in json format as follows:
{{
"Action Thought": "The action thought"
"Action Type": "The action type"
})
Table 14: Prompt for correct action annotation ā the first approach in offline SFT.
## System Role
You are expert in correcting the step-wise operation of GUI agent. The agent's 's task is to help human users operate an Android phone by completing specific instructions provided by the user but it performs incorrect action certain step. Your goal is to reason about the correct action and provide the reasoning process leading to the correct action at this step.
## Definition of Action Thought
25

---

The action thought is the rationale behind the actual action taken at a certain step. The action thought should be compact paragraph consisting of 3-4 sentences. It could include the following aspects if they are helpful and important for the action reasoning process: observation: a concise description of the current screenshot, focusing on the task related content
reflection: a simple analysis of whether the previous step's action is successful and as expected
progress analysis: a brief summary the progress towards the overall goal before taking the action at the current step and plan about the sub-tasks that still need to be done in order to complete the final goal action rationale: a brief reasoning process to integrate the above information and provide a natural thinking process leading to the actual action from the perspective of the operator The action thought should be logically fluent, and it does not necessarily include all the above aspects. The action thought should *not* be clearly separated into the above aspects and does not need to mention the keywords like observation and reflection.
## Information Provided
As an evaluator, you will receive the following information: 1. Task Instruction: The overall task instruction for the GUI agent.
2. Past Actions: A list of past actions prior to this step. Empty if it is the first step.
3. Screenshots: The first screenshot corresponds to the previous step while the second screenshot corresponds to the current step. Only the current screenshot is provided if it is the first step. The click point is highlighted using red dot in the first screenshot.
Your job to analyze the provided information and provide the correct action thought leading to the correct action that the agent should take.
## Valid Action Space
The actions that the agent may take to complete the task are as follows:
Click: Click at certain position the current screen.
Long Press: Long press at a certain position on the current screen.
Scroll: Scroll on the sereen, scroll down/up//eft/right,, note that the direction is the opposite direction the figure movement.
Type[text]: Type the 'text' into the input field.
Open Applapp]: Open the 'app' App.
Press Home: Return to the home page.
Press Back: Go back to the previous page.
Press Enter: Click the enter button.
Wait: Wait for the device to respond or load something.
Task Complete: Indicate the task is completed
Task Impossible: Indicate the task is impossible.
## Useful Guidelines
To make your reasoning and response more accurate, please follow these guidelines:
1. Based on the provided information, provide the correct action thought. In the action thought, use first-person description, that is, using instead of 'the user' or 'the agent' 2. Provide the correct action type which must be consistent with your action thought, The correct action type should be one of the following: [Click, Long Press, Open App. Scroll, Type, Press Home, Press Back, Press Enter, Wait, Task Complete, Task Impossible]
## Important Notes
Note that when you want to click or press the backspace button, the correct action type is click instead of Press Back (which means going back to the previous page). And in such cases, try to use the word 'click" instead of 'press' in your action thought.
## Input:
Task Instruction: joinkiinstuuctio]
Past Actions: [past_actions)
26

---

## Output Format:
Directly output your results in json format as follows:
"Action Thought": "The correct action thought"
"Action Type": "TThe correct action type"
11
Table 15:
Prompt for adding reflection action thought a;
athought of the first approach in offiie SFT.
## System Role
You are an expert in correcting the step-wise operation of a GUI agent. The agent's task is to help human users operate an Android phone by completing specific instructions provided by the user. The agent performs an incorrect action at a certain step, and it needs to execute the press back action go back the previous normal state. Your task is to add some reflection content about the previous incorrect action to the correct action thought.
## Definition of Action Thought
The action thought is the rationale behind the actual action taken at a certain step. The action thought should be compact paragraph consisting of 3-4 sentences. It could include the following aspects if they are helpful and important for the action reasoning process: observation: concise description of the current screenshot, focusing on the task related content
reflection: a simple analysis of whether the previous step's action is successful and expected
progress analysis: brief summary of the progress towards the overall goal before taking the action at the current step and plan about the sub-tasks that still need to be done in order to complete the final goal action rationale: a brief reasoning process to integrate the above information and provide natural thinking process leading to the actual action from the perspective of the operator The action thought should be logically fluent, and it does not necessarily include all the above aspects. The action thought should "not* be clearly separated into the above aspects and does not need to mention the keywords like observation and reflection.
## Information Provided You will receive the following information:
1. Task Instruction: The overall task instruction for the GUI agent.
2. Incorrect Action: The previous incorrect action
3. Error Action Analysis: An analysis from an expert explaining why the previous action is incorrect.
4. Correct Action Thought: The actual correct action thought leading to the correct action
5. Screenshot: The screenshot after performing the press back action to go back to the normal status
Your job is to analyze the provided information and modify the correct action thought by adding some reflection and lessons learned about the incorrect action narrated in the first person as if you have performed the previous incorrect action, realize the mistake, and press back to go to the current status.
## Important Notes: Your modified action thought should still be consistent with the provided correct action thought.
## Input:
Task Instruction: Incorrect Action: { _actionncto
Error Action Analysis: aerror_anllyse
Correct Action Thought: (correct_actie action ____attttooouohhhtt_
27

---

## Output Format:
Directly output your results in json format as follows:
{{
"Updated Action Thought": "TThe updated action thought with reflection added"
Table 16: Prompt for creating ineffective incorrect action of the second approach in offline SFT.
### System Role
You expert in understanding the operations from GUI agent. The agent's 's task is to help human users operate an Android phone by completing specific instructions provided by the user. Your goal is to come up with an incorrect ineffective Action that is different from the correct action and would not change the current screen.
## Candidate Actions
You may consider the following actions
click: click on the current screen.
The complete format of this action is "click <element> <purpose>", where <element> is noun phrase starting with 'the' indicating the clicked element, and <purpose> is verb phrase indicating the direct purpose and expected outcome of clicking this UI element.
long press: long press on the current screen.
The complete format of this action is "long press <element> to <purpose>", where <ele- ment> is noun phrase starting with 'the' indicating the pressed element, and <purpose> is verb phrase indicating the direct purpose and expected outcome of long pressing this UI element.
scroll: scroll on the screen.
The complete format of this action is "scroll <direction> <purpose>", where <direction> direct purpose and expected outcome of this scrolling action.
is one of the following directions: ['up', 'down', 'left', 'right') and the direction is the opposite direction of the movement of the finger. <purpose> is verb phrase indicating the
type: type some 'text' into the input field. The complete format of this action is "type in the <content>", where <content> is the typed text.
## Information Provided
You will receive the following information: 1. Task Instruction: The original task instruction for the GUI agent.
2. Past Actions: The past actions taken by the agent to complete this task before this step.
Empty if it is the first step.
3. Future Actions: The future actions taken by the agent complete this task after this
step. Empty if it is the last step.
4. Current Action: The action taken by the agent the current step.
5. Screenshot: The screenshot at the current step. The click point is highlighted using red
dot in the current screenshot.
Your job is to analyze the provided information and decide whether is possible to come up with an incorrect ineffective action.
The requirement
the incorrect ineffective action:
1. The action is incorrect; it is clearly wrong unnecessary for completing the task deviates from the correct way.
2. It is easy natural mistake user who is unfamiliar with the App, button functions, or certain operations might make. The user will probably realize the mistake when seeing the execution of this action.
28

---

3. This action has no effect on the current screen, which means the current screen will remain exactly the same after executing the incorrect action.
Some examples: scroll down while it is already at the bottom, click the entry name instead of the actual text field for entering information, type in something without activating the input field yet.
## Useful Guidelines
To make the evaluation process accurate, please follow these guidelines:
1. Based on the information, judge the feasibility of having an incorrect ineffective action
meeting the mentioned requirements. Ignore the following if not.
2. Provide your incorrect ineffective action if possible.
3. Explain why the incorrect ineffective action incorrect, and explain why it is easy natural mistake a user might make.
Note that you have to strictly follow the complete format for your action.
## Input:
Task Instruction: [task instruction
Past Actions: |past_ actions]
Future Actions: [future actions)
Current Action: (action)
## Output Format:
Directly output your results in json format as follows:
11
"Task Feasibility": "Yes or No", if no, the following entries should be empty.
"Incorrect Ineffective Action": "Your incorrect ineffective action", "Explanation": "Explain why the action is incorrect and why it is natural mistake for a user not familiar with the app and certain operations."
}}
Table 17: Prompt for adding reflection about the ineffective incorrect action of the second approach in offline SFT.
## System Role
You are an expert in correcting the step-wise operation of GUI agent. The agent's task is to help human users operate Android phone by completing specific instructions provided by the user. The agent performs an incorrect action at a certain step, and needs to realize the mistake and perform the correct action. Your task is add some reflection content about the previous incorrect action to the correct action thought.
## Definition of Action Thought The action thought is the rationale behind the actual action taken at a certain step. The action thought should be compact paragraph consisting of 3-4 sentences. It could include the following aspects if they are helpful and important for the action reasoning process: observation: concise description of the current screenshot, focusing on the task related content
reflection: a simple analysis of whether the previous step's action is successful and as expected
progress analysis: a brief summary of the progress towards the overall goal before taking the action at the current step and plan about the sub-tasks that still need to be done in order to complete the final goal action rationale: a brief reasoning process to integrate the above information and provide natural thinking process leading to the actual action from the perspective of the operator The action thought should be logically fluent, and it does not necessarily include all the above aspects. The action thought should *not* be clearly separated into the above aspects and does not need to mention the keywords like observation and reflection.
29

---

## Information Provided
You will receive the following information:
1. Task Instruction: The overall task instruction for the GUI agent.
2. Incorrect Action: The previous incorrect action 3. Error Action Analysis: An analysis
from an expert explaining why the previous action incorrect and why it might happen.
4. Correct Action Thought: The actual correct action thought leading to the correct action
5, Screenshot: The screenshot of the current step after executing the incorrect action
Your job is to analyze the provided information and modify the correct action thought by adding some observation and reflection content realizing and acknowledging the incorrect action narrated in the first person as you have performed the previous incorrect action.
## Important Notes:
The previous incorrect action has no effect on the screenshot; that is, the screenshots before and after the incorrect action are the same. Your modified action thought should still be consistent with the provided correct action thought.
!!! Note that the correct action thought does not include the previous incorrect action, so the 'previous step' in the correct action thought actually corresponds to the step before the previous incorrect action. So you have **remove** the part describing the success of the previous step or description about the progress in the original correction action thought! For example, you should remove parts like 'I have successfully xxx' or 'The previous step has successfully xxx'.
In the first sentence of your thought, you should directly mention that the previous step is unsuccessful or incorrect by observation, and then do reflection explaining why you made that mistake and the fact learned from this failure.
## Input:
Task Instruction: [task instruction Incorrect Action: {incorrect_ _cctiinttttnntt
Error Action Analysis: (error_analysee
Correct Action Thought: [correct_action action _
## Output Format:
Directly output your results in json format follows:
"Updated Action Thought": "The updated action thought with reflection added"
Table 18: Prompt for the MLLL--asse verifier.
## System Role
You are an expert in evaluating the performance of GUI operation agent. The agent's
task is to help human users operate an Android phone by completing specific instructions provided by the user. Your goal evaluate whether the agent successfully completed the task or not.
## Information Provided
As an evaluator, you will receive the following information:
1. Task Instruction: The original task instruction for the GUI agent.
2: Task Guidance: reference.
An overall description about how to **correctly** ** complete this task as
The ground truth answer for this task. Empty the answer is not available the task does not require answer.
3: Correct Answer:
4. Operation History: A list of actions (in the form of images and corresponding actions)
taken by the agent execute the Task Instruction.
30

---

5. Agent Answer:
The answer provided by the agent for the task. Empty if there is no answer action.
6. Final Status: The final task status is indicated by the agent. The status is either Task Complete Task Impossible.
Your job to analyze the provided information to determine whether the agent successfully completed the task, based the alignment between the Task Instruction, the actions performed (shown screenshots and operation history), and the expected outcome.
## Effective Action Space
The actions that the agent may take complete the task are as follows: Click[[x, y]]: Click the location [x, y] on the current screen, marked with a red circle in the screenshot.
Long Press[[x. y]]: Click the location [x, y] on the current screen, marked with a red circle in the screenshot.
Scroll[[x1, yl, x2, y2]]: Scroll from position [xl, yI] to [x2, y2], as shown by the arrows in the screenshot, access information that is not currently visible. Type[text]: Type the 'text' into the input field.
Memorize[text]: Store some 'text' into an intermediate memory for future reference.
Answer[text]: The agent provides the 'text' as the answer.
Press Home: Return to the home page.
Press Back: Go back to the previous page.
Press Enter: Click the enter button.
Wait: Wait for the device to respond or load something
Task Complete/Tast Impossible: The agent indicates the task completed or impossible.
Task Evaluation Criteria:
SUCCESS: The agent successfully performs all the necessary actions to meet the Task Instruction.
The agent failed perform least one necessary action, the task could not be completed correctly, based on the screenshots and operation history.
NOT SUCCESS:
## Useful Guidelines
To make the evaluation process accurate, please follow these guidelines:
1. Analyze the Task Instruction: You should first analyze what critical milestones have to
be completed and what necessary outcomes are expected for this task.
2. Analyze the agent's operations, provide a definitive verdict on whether the task has been successfully completed together your reasoning process.
3. Provide your final answer: "SÚCCESS" or 'NOT SUCCESS'
## Important Notes When the task instruction asks certain question or seeks certain information, the agent has to provide the **correct** answer before completing the task, otherwise, the task should be NOT SUCCESS.
When the agent indicates the task is impossible, you should judge whether this task is indeed impossible to complete (e.g. due to network issues). You should output SUCCESS only when both you and the agent indicate the task is impossible.
You should carefully examine the screenshots to double check whether the operations taken by the agents achieve the desired and expected outcome. Any minor input or format issues should be judged as NOT SUCCESS.
Note that the agent might make some wrong attempts during the process, and it should be judged as SUCCESS as long as the agent has successfully completed the task at the end.
## Input:
Task Instruction: joask_isttructio]
Task Guidance: (task_ coissk_guidncee
Correct Answer: (correct_answer
Agent Answer: (eeen__asswer
Final Status: (final status)
31

---

## Output Format:
Directly output your results in json format as follows:
[[
"Task Analysis": "An analysis of the task",
"Reasoning": "Reason about whether the agent has successfully completed the task",
"Final Result": "SUCCESS" or "NOT SUCCESS"
}}
Table 19: Prompt for checking the step-wise correctness of successful trajectories.
## System Role
You are an expert in evaluating the step-wise operation correctness of GUI agent. The agent's 's task is to help human users operate an Android phone by completing specific instructions provided by the user. Your goal evaluate whether the current action taken by the agent is correct in terms of completing the overall task instruction.
## Information Provided
an evaluator, you will receive the following information:
1. Task Instruction:
The overall task instruction for the GUI agent.
2. Task Guidance:
An overall description of how to correctly complete this task as a
reference.
3: GT Answer: The ground truth answer for this task. Empty the answer is not available the task does not require answer.
4: Stored Memory: Information the agent has stored in the intermediate memory from previous steps. Empty if none.
5. Task Progress: A summary describing the current progress of the overall task before
taking the action at the current step. The progress is empty if it is the first step.
6. Current Action: The action taken by the agent at the current step.
7. Screenshots: The screenshot the current step and the screenshot at the next step after the current action is executed. The click point is highlighted using a red dot in the current screenshot and the scroll positions visualized using red arrow in the current screenshot.
Your job is to analyze the provided information to determine whether the current action taken by the agent is correct for achieving the overall task goal.
Valid Action Space
The actions that the agent may take to complete the task are follows: Click: Click on the current screen, the click point is marked with red circle in the screenshot. Long Press: Long press on the current screen, the press point is marked with red circle in the screenshot.
Scroll: Scroll on the screen, the touch point and lift point of the scroll are marked with red arrow in the sereenshot.
Type[text]: Type the 'text" into the input field.
Mcmorize[texi]: Store some 'text' into an intermediate memory for future reference
Answer[text]: The agent provides the 'text' as the answer.
Press Home: Return to the home page.
Press Back: Go back to the previous page.
Press Enter: Click the enter button.
Wait: Wait for the device respond load something
Task Complete/Task Impossible: The agent indicates the task completed or impossible.
## Task Evaluation Criteria:
CORRECT: The current action taken by the agent reasonable without skipping any necessary steps and makes correct progress towards completing the overall task instruction.
32

---

INCORRECT:
The current action taken by the agent is clearly wrong or unnecessary for completing the task, or deviates from the correct way. You should consider whether the outcome of the current action (from the second screenshot) is as expected or not when making the judgment.
## Useful Guidelines
To make the evaluation process more accurate, please follow these guidelines:
1.
Based on the provided information, analyze the agent's action, provide a definitive
verdict on whether the current action is correct, together with your reasoning process.
2. Provide your conclusion: 'CORRECT" or 'INCORRECT'
3. Based on the current progress and the screenshot for the next step, update the task progress summarizing the overall progress after taking this current action. One or two sentences.
Note that some sub-tasks expected outcomes might need multiple actions (e.g. typing something needs to click the text input field and then type), and you only need to assess the correctness of the current action.
Note that the 'Current Action' represents the purpose of the agent, which might be incon- sistent with the actuation of the action due its unfamiliarity with certain operations or elements when the action is to click/long press/scroll. In such cases, the 'Current Action' itself may seem correct, but the grounded action is wrong. For example, the agent's action (purpose) is to click App A, but it actually clicks App B as it does not know which icon is App So you should also make the judgment based on the actual action and its outcome. !!! Note that an action should be considered correct if it attempts to recover from previous mistake redirect the process back toward the final goal, even if earlier steps were incorrect unnecessary.
## Input:
Task Instruction: [task instruction
Task Guidance: [task guidance)
GT Answer: fgt_ answer)
Stored Memory: (memory)
Task Progress: p
Current Action: (action)
## Output Format:
Directly output your results in json format as follows:
"Reasoning": "Reason about whether the current action is correct",
"Conclusion": "CORRECT" "INCORRECT",
"Updated Progress": "The updated task progress summary after taking this action." "
})
Table 20: Prompt for identifying the first error step of the unsuccessful trajectories.
## System Role
You are expert in evaluating the step-wise operation correctness of GUI agent. The agent's task to help human users operate an Android phone by completing specific instructions provided by the user. Your goal is to evaluate whether the current action taken by the agent is correct in terms of completing the overall task instruction.
## Information Provided
As an evaluator, you will receive the following information:
1. Task Instruction: The overall task instruction for the GUI agent.
2. Task Guidance: reference.
An overall description of how to correctly complete this task as a
33

---

3: GT Answer: The ground truth answer for this task. Empty if the answer is not available or the task does not require an answer.
4: Stored Memory: Information the agent has stored in the intermediate memory from
previous steps. Empty none.
5. Task Progress: A summary describing the current progress of the overall task before
taking the action at the current step. The progress empty if it the first step.
6. Current Action: The action taken by the agent the current step.
7. Screenshots: The screenshot at the current step and the screenshot at the next step after the current action is executed. The click point is highlighted using red dot in the current screenshot and the scroll positions are visualized using a red arrow in the current screenshot.
Your job is analyze the provided information determine whether the current action taken by the agent is correct for achieving the overall task goal.
## Valid Action Space
The actions that the agent may take complete the task are as follows:
Click: Click on the current screen, the click point is marked with red circle in the screenshot.
Long Press: Long press on the current screen, the press point is marked with red circle in the screenshot.
Scroll: Scroll on the screen, the touch point and lift point of the scroll are marked with red arrow in the sereenshot.
Type[text]: Type the 'text' into the input field.
Memorize[text]: Store some 'text' into an intermediate memory for future reference.
Answer[text]: The agent provides the 'text" as the answer.
Press Home: Return to the home page.
Press Back: Go back to the previous page.
Press Enter: Click the enter button.
Wait: Wait for the device to respond load something.
Task Complete/Tast Impossible: The agent indicates the task completed or impossible.
## Task Evaluation Criteria:
CORRECT:
The current action taken by the agent is reasonable without skipping any necessary steps and makes correct progress towards completing the overall task instruction. INCORRECT: The current action taken by the agent is clearly wrong unnecessary for completing the task, or deviates from the correct way. You should consider whether the outcome of the current action (from the second screenshot) as expected not when making a judgment.
## Useful Guidelines
make the evaluation process more accurate, please follow these guidelines:
1. Based on the provided information, analyze the agent's action, provide a definitive
verdict on whether the current action is correct, together with your reasoning process.
2. Provide your conclusion: 'CORRECT' or 'INCORRECT'.
3. Based on the current progress and the screenshot for the next step, update the task progress summarizing the overall progress after taking this current action. One or two sentences.
Note that some sub-tasks or expected outcomes might need multiple actions (c.g. typing something needs to click the text input field and then type), and you only need to assess the correctness the current action.
Note that if the task requires question answering (GT Answer is not empty), the agent must provide the answer before marking the task as complete.
## Input:
Task Instruction: faaskinstruction]
Task Guidance: (task_gutt cuiss_ggidancee
GT Answer: {gt_ansswr answer)
34

---

Stored Memory: {memory
Task Progress: f progress]
Current Action: [action)
## Output Format:
Directly output your results in json format as follows:
11
"Reasoning": "Reason about whether the current action is correct",
"Conclusion": "CORRECT" "INCORRECT",
"Updated Progress": "The updated task progress summary after taking this action." "
Table 21:
Task examples of online learning environment.
Apps
Difficulty Level
Number
Example Task
Simple Calendar Pro
Level-1
13
Change the view (view) in Sim- ple Calendar Pro.
Simple Calendar Pro
Level-2
7
Change the location of event {eventl] | to be the same as event (event2) in Simple Calendar Pro.
Contacts
Level-1
12
Delete the contact named [name) in the Contacts App.
Contacts
Level-2
9
Find [name] in the Contacts App, change the phone number to (new number) and email to (new email)
Dice
Level-1
10
In the Dice App, roll the dice {n times] times with the default setting.
Dice
Level-2
4
In the Dice App, set Dice to 1 and Sides to [n sides), ). and roll the dice twice with this setting. Record the results and answer the sum of num- bers.
FitBook
Level-1
Open the FitBook App, delete the entry with the highest calories in the diary tab.
FitBook
Level-2
8
Open the FitBook App. find and open the (entry) entry in the Diary tab, then change its date to (n days) days before the original date.
Fossify Clock
Level-1
15
Add clock (timezone) timezone in the Fossify Clock App.
Fossify Clock
Level-2
9
Start all timers and then pause them in the Fossify Clock App.
Fossify Messages
Level-1
8
Delete the conversation from (num- ber) in the Fossify Messages App-
Fossify Messages
Level-2
7
Send the same message {n times) times to (number) in the Fossify Messages App with the following content: (message)
35

---

LibreOffice
Level-1
10
In the LibreOffice Viewer App, go to the [folder) folder in the _ak_gphone64_x660 64 storage area. How many does are there? Answer with the number only.
LibreOffice
Level-2
4 Find the numbers logged in {docfilel] and (docfile2) in the cdk_gphone64_x66 64 storage area of the LibreOffice Viewer App. Answer the of the numbers.
Markor
Level-1
13
Find all markdown notes Markor: Answer with their names separated with comma.
Markor
Level-2
6
Check the file sizes of the note [namel] and [name2] in Markor. Then answer the larger file size in Bytes (number only without units), for example: 500.
ProExpense
Level-1
11
What expenses are logged in the Pro Expense App? Answer the names separated by comma.
ProExpense
Level-2
Delete all expenses in the Pro Ex- pense App that are higher than [amount].
Broccoli
Level-1
13 Add the following recipe in the Broccoli APP: (recipe).
Broccoli
Level-2
9
In the Broccoli APP, check the preparation times of recipes ( [title]] and [title2). and answer the longer preparation time. Please directly an- swer the time in the following for- mat: XX mins/XX hrs.
Chrome (WehShopping)
Level-1
9
Go to (website) using Chrome, search for (product), what is the overall rating of the first search sult? Answer with the number only.
Chrome (WebShopping)
Level-2
9
Search for the following items (product list) on (website) and add them to my cart. What is the total price of all the items in my cart?
36


Reviews:
[{"review_id": "r611cVKao6", "reviewer": "Reviewer_DG2g", "summary": "The paper introduces GUI-Reflection, a framework that adds explicit self-reflection and error-correction behaviours to end-to-end multimodal GUI agents. During GUI-specific pre-training, the authors inject three synthetic tasks—Action Verification, Action Reversal and Mistake-Informed Re-attempt—built from existing successful trajectories. In offline SFT, they automatically augment clean demonstrations with reflection episodes that contain (i) the original wrong action, (ii) a natural-language action-thought explaining the error, and (iii) pre- and post-correction actions. An online reflection-tuning loop runs in a mobile-device sandbox: after every rollout, mistakes are detected, labelled and verified by an MLLM-based checker, then fed back for further fine-tuning. On the authors’ suite of 18 mobile tasks, success on level-2 tasks jumps from 14.6 % → 34.7 % after three reflection-tuning iterations, and the final 8 B-parameter model reaches 34.5 % success on the public AndroidWorld benchmark—outperforming all prior end-to-end 8 B models and rivaling much larger systems (e.g. UI-TARS-72 B).", "questions": "What fraction of synthetic corrections are rejected by the programmatic / MLLM checker, and what is the observed false-accept rate? A small human audit (e.g. 100~200 samples) would ground confidence in the pipeline.", "limitations": "See weaknesses", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Pros:\n\n• First GUI agent to demonstrate emergent self-reflection without human feedback, a capability likely needed for reliable automation.\n\n• End-to-end data pipeline—no human labour after initial success logs.\n\n• Competitive results on a public benchmark.\t\n\nCons:\n\n• Only 8b model is tested, no evidence of the method effectiveness on various scales, especially larger ones where true performance bottleneck emerges.\n\n• Reflection gains measured on synthetic tasks; real app generalisation unclear.\n\n• Scope limited to Android mobile; no evidence on web/desktop/iOS GUIs.\n\n• Eval relies on MLLM judge, and verification precision/recall not quantified, leading to risk of error accumulation.", "was_revised": true, "final_justification": "Thank the author for the responses, I have raised my rating to 4 since authors address many of my concerns.\nThat being said, I am still concerned about several points that need further experiments:\n1. method robustness to model scale and base performance.\n2. benchmark varieties. As of today, MLLM-as-a-judge can not be fully trusted on complex real-world task, and the nearly 100% judge success rate on the relatively simple AndroidWorld task largely contributes to the effectiveness of the model. I doubt if MLLM-as-a-judge can still work in this method for more difficult tasks.  \n\nAddressing about concerns by empirical evidences could greatly strengthen the paper.", "rebuttal": "We thank the reviewer for the detailed feedback and constructive suggestions. Below, we address each point raised.\n\n---\n\n> **Q1: Only 8B model is tested; no scaling analysis is provided.**\n\n**A1:**  \nWe have additionally trained InternVL3-78B with our proposed GUI-Reflection framework. The model achieves a success rate of 43.1% on the AndroidWorld benchmark, which is comparable to UI-TARS-72B (46.6%), despite UI-TARS being trained with a significantly larger and more diverse set of trajectories. We also train an InternVL3-78B baseline without the reflection-related designs and it only achieves 31.5%. This result highlights the effectiveness and scalability of our reflection-based training pipeline. We acknowledge that the overall performance of GUI agents is jointly influenced by different factors like **capacity of the base model** and the **scale and diversity of the training data**. Investigating these factors in isolation and exploring more systematic scaling trends will be an important direction for future work.\n\n---\n\n> **Q2: Reflection gains measured on synthetic tasks**\n\n**A2:**  \nWe would like to clarify that the apps used in both our environment and the AndroidWorld benchmark are **real, commonly used Android apps**, and the tasks are carefully designed to mirror practical, real-life usage scenarios such as calendar scheduling, file management, and document editing. This ensures that our evaluations reflect the model’s effectiveness in realistic contexts, rather than synthetic toy settings. Additionally, for research reproducibility, safety, and better controllability, we choose open-source apps where we have full access to low-level APIs, system states, and internal databases. This enables accurate evaluation and flexible task designs, which are difficult to achieve with closed-source or proprietary applications.  \nWe fully agree that evaluating GUI agents in more diverse and naturalistic settings is important. Extending our environment to include a broader set of real-world apps is an active direction and will be prioritized in future work.\n\n---\n\n> **Q3: Scope limited to Android mobile; no web/desktop/iOS evidence**\n\n**A3:**  \nWe agree that extending to other platforms is valuable. Our proposed GUI-Reflection framework is **platform-agnostic by design**, and the core methodology — including the reflection task suite, automatic correction data generation, and iterative online tuning — is applicable to a broad range of GUI environments, including desktop and web interfaces. Due to resource and engineering constraints, our current experiments focus on the Android mobile platform, which offers a practical starting point with a rich set of apps and well-supported emulation tools. Nevertheless, we view this as a first step toward a more general and cross-platform framework, and we plan to extend our method to other platforms such as desktop and web GUI environments in future work.\n\n---\n\n> **Q4: Eval relies on MLLM judge; fraction of synthetic corrections rejected & observed false-accept rate**\n\n**A4:**  \nThank you for your valuable suggestions. In our online reflection tuning process, the MLLM-based judgment and annotation are primarily used in the following parts:  \n**1)** ***An MLLM-based verifier judges whether a task is completed by the agent.***  \nWe have 215 task templates in our proposed environment, and most of them (202 out of 215) are evaluated by the **programmatic verifier**, ensuring correctness in judgment. Only 13 tasks require evaluation by the **MLLM-based verifier** (Gemini-2.0 Flash) because they involve intermediate results, which are not feasible for a programmatic verifier. For each task template, we provide detailed task guidance (including task descriptions and instructions) to the MLLM verifier to ensure accurate judgment. We randomly sampled 100 MLLM judgment samples and manually checked their correctness, achieving **100% accuracy** in this part.  \n**2)** ***An MLLM-based verifier checks the step-wise correctness of the successful trajectories.***  \nFor trajectories deemed successful by the programmatic or MLLM-based verifier, we use another MLLM-based verifier (Gemini-2.0 Flash) to assess the correctness of each step. Only steps verified as correct are retained for training. We randomly sampled 200 step-wise samples and manually checked their correctness. The results are summarized below (Note: we denote **correct** steps as positive here):\n| Metric           | Value |\n|------------------|-------|\n| True Positive (TP)  | 187   |\n| True Negative (TN)  | 6   |\n| False Positive (FP) | 1   |\n| False Negative (FN) | 6   |\n| Precision            | 99.5%  |\n| Recall               | 96.9%  |  \n\nThe verifier achieves **near-perfect Precision (99.5%)**, ensuring that the quality of the step-wise samples for training is maintained. Note that we can be **overly strict** here (a lower Recall is acceptable) since the number of discarded samples remains relatively small.\n\n**3)** ***An MLLM-based checker identifies the first incorrect step of unsuccessful trajectories***  \nFor unsuccessful trajectories, we use an MLLM-based checker (GPT-4o) to identify the first incorrect step. We randomly sampled 200 step-wise samples and manually checked the correctness of the judgment from the MLLM-based checker. The results are summarized below (Note: we denote **correct** steps as positive here), and we find that the MLLM-based checker ensures high quality in this process:\n| Metric           | Value |\n|------------------|-------|\n| True Positive (TP)  | 162   |\n| True Negative (TN)  | 31   |\n| False Positive (FP) | 3   |\n| False Negative (FN) | 4   |\n| Precision            | 98.2%  |\n| Recall               | 97.6%  |\n\n**4)** ***An MLLM annotates the pre- and post-error corrections for incorrect steps.***  \nFor incorrect steps in unsuccessful trajectories, we use an MLLM (Gemini-2.5-Pro-Exp) to annotate both pre-error and post-error corrections. We also provide carefully designed task guidance and prompts to help the MLLM generate high-quality annotations. We randomly sampled 200 MLLM-annotated samples and manually checked their correctness. The accuracy of the MLLM annotation is **97.5%**. Note that we **do not use an additional MLLM-based checker** to double-check the annotation, as the MLLM annotator already achieves high accuracy in this task."}, {"review_id": "p53Gy9GCv9", "reviewer": "Reviewer_phbt", "summary": "Noting that current UI control models lack strong abilities to recognize and correct mistakes in executing instructions. To this end, the authors propose 5 new methods for generating training data, spread across 3 proposed training phases:\n\n* A \"GUI pre-training\" phase along with methods for generating \"action verification\", \"action reversal\", and \"mistake-informed reattempt\" SFT data.\n* A \"GUI offline SFT\" phase, with methods for augmenting existing trajectory data to construct reflection and error correction scenarios.\n* A \"GUI online learning\" phase, an interative process for collecting additional data for error correction via environment interaction.\n\nThe authors show that this recipe enable constructing a competitive 8B model on the AndroidWorld datasets. Some qualitative anecdotes of error correction behavior are observed.", "questions": "See weaknesses above.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "__Strengths__\n\n* The limited ability of many current UI control models to recognize and recover from incorrect actions is an important problem to study.\n* The authors' proposed methods for generating additional training data focused on this capabilitiy may provide useful, reusable approaches for improving performance.\n* The authors demonstrate that their proposed recipe enables competitive performance on the AndroidWorld benchmark.\n\n__Weaknesses__\n\nWhile the paper addresses an important problem, and some of the proposed methods may in fact be quite useful for addressing it, I did have some issues with the current manuscript and set of results that I think would benefit from revisions and resubmission. The proposed training pipeline is quite complex, and the paper proposes many new methods for data augmentation. However, the experiments don't seem to sufficiently justify the proposed complexity and diversity of methods relative to prior work. The paper reads as a collection of ad-hoc methods, without clear take-aways for which aspects of the overall recipe are most useful for future work to adopt, and why.\n\nMore specifically:\n\n* The pipeline involves 3 distinct training phases in addition to the base model's pre-training. However, there is only one end-to-end result in the main paper on a standard end-to-end benchmark: the AndroidWorld result in Table 3. To justify the complexity of the proposed approach, it would have been useful to show clearer ablations for how the various components affect end-task performance. The single competitive-but-not-SOTA result on AndroidWorld is not particularly useful for understanding this. How does each of the proposed datasets and training phases affect end-task performance? Can the recipe be simplified or the number of training phases reduced? These are questions that would have been helpful to address in the paper.\n* In general, the paper would perhaps read better if it was more focused on a smaller set of methods and contributions. If there are some aspects of the overall recipe that do not significantly impact end-task performance, perhaps these could be relegated to the appendix.\n* Similarly, the number of different proposed training sets and training phases also made the exposition of the overall recipe, and the presentation of the various intermediate results in Tables 1 and 2, a bit difficult to follow at times.\n* Key questions related to the effectiveness of the proposed method for \"Iterative Online Reflection Tuning\" relative to prior work don't appear to be addressed. For example, how important is it that this method uses on-policy sampling from the student model vs. sampling trajectories from a more powerful MLLM (more akin to a standard distillation procedure)? How does the proposed method compare to more standard rejection sampling? Ablations comparing end-to-end task performance would be useful to answer these questions. Is simply scaling up this training stage sufficient towards improving error correction abilities? Focusing on answering these questions alone would have been an interesting and useful paper.\n\nMinor nits:\n\n* The details in section 2.3 of how existing models were used for data annotation were a bit sparse, perhaps there is an Appendix section to reference?\n* \"pre-training stage\" -> This is seemingly an unconventional and confusing name for a stage that occurs after the base model has been pre-trained (InternVL2.5-8B)\n* The terminology \"agent-based\" vs. \"end-to-end\" seemed a bit strange, given that all approaches for UI control are \"agents\" in some sense. Would \"prompted\" vs. \"task-specific tuning\" be accurate?\n* Line 54 in Introduction - \"further RL training cannot effectively activate or enhance such reflection abilities\" - citation needed? This claim seems questionable in general.\n* While the examples in Figure 6 are useful, it would be useful, if possible,  to quantify the improvement in error-correction behavior over a relevant baseline on AndroidWorld.\n* (Not a weakness, just a note) Looking at the AndroidWorld leaderboard, there do appear to be 8B models that outperform the proposed model (e.g. ScaleTrack, V-Droid), but these appear to be released within the last couple months, and therefore should be considered concurrent work.", "was_revised": true, "final_justification": "I am a bit split on acceptance. I had significant issues with the presentation and lack of key ablations in the original draft. However, the authors in rebuttal added some of these key ablations and pledged to revise some of the presentation issues. I think the paper could be improved above the acceptance threshold with these changes, but it is difficult to re-evaluate the entire paper in this new context.\n\nI am raising my score to 4, but also think the paper could potentially benefit from re-submission and new round of reviews with the ablations and presentation changes discussed during rebuttal.", "rebuttal": "We thank the reviewer for the detailed and thoughtful feedback. We appreciate the recognition of our goal to improve error correction in GUI agents and the suggestions to improve the clarity and focus of the paper. We address each point raised below.\n\n---\n\n> **Q1: Only one end-to-end benchmark result is shown; lack of ablations makes it hard to assess the impact of individual stages or components.**\n\n**A1:**  \nWe appreciate this insightful suggestion. We agree that providing clear ablations is important to demonstrate the necessity and effectiveness of each proposed training stage. In addition to AndroidWorld, we emphasize that our proposed mobile GUI environment (introduced in Section 2.5.1) also serves as a practical and diverse end-to-end evaluation suite. In Section 3.3 and Table 2, we present an end-to-end evaluation on our environment that verifies the effectiveness of adding reflection-related components at different stages. \n\nTo further address the reviewer's suggestion, we have also conducted additional ablations on AndroidWorld, where we systematically compare models with and without reflection-related designs in each stage of our overall pipeline. As shown in the table below, for the three stages of our training pipeline (GUI pre-training, GUI offline SFT, and GUI online training), removing reflection-related design in any stage harms final end-to-end performance, and the reflection-related design in the offline SFT and online stages has larger importance.\n\n|Exp|SR on AndroidWorld (%)|\n|------------------|-------|\n|No Reflection in GUI Pre-training|31.9|\n|No Reflection in offline SFT|26.1|\n|No Reflection in online training|25.9|\n|No Reflection in any stage|20.7|\n|**GUI-Reflection**|**34.5**|\n\nThese results collectively validate the **effectiveness of our GUI-Reflection framework as an integrated whole**, and reaffirm our core motivation that reflection and error-correction capabilities should be explicitly injected across all stages of the GUI training pipeline.\n\n---\n\n> **Q2: Paper may benefit from focusing on fewer methods; some parts could be relegated to the appendix; intermediate results in Tables 1 and 2 make the paper hard to follow.**\n\n**A2:**  \nThank you for the suggestion. We will improve the organization and presentation of our paper to enhance clarity. We will move less important content into the appendix and streamline the exposition of our pipeline to highlight the core insights and contributions.   \nOur framework is a **progressive learning process** instead of a set of isolated data augmentations. Specifically: 1) In the GUI pre-training stage, the model learns **basic reflection-oriented atomic skills**. 2) In the GUI offline SFT stage, the model learns to exhibit **reflection behaviors**. 3) In the online training stage, the model learns to **actively reflect and self-correct through real-time interaction**, improving generalization and robustness under open-ended execution. \n\nFor the experiments on the GUI-Reflection Task Suite in Tables 1 & 2, our goal is to show that regular GUI pre-training significantly weakens reflection-related capabilities. We introduce this task suite to both evaluate and recover such abilities at the pre-training stage. Beyond this, the GUI-Reflection Task Suite serves as a **complementary benchmark for general-purpose GUI models**. While most existing GUI benchmarks focus on grounding and UI understanding, our tasks emphasize higher-level reasoning skills such as verifying action correctness and reversing errors. These reflection-oriented abilities are crucial for building self-correctable GUI agents and are also useful in broader scenarios—for instance, as a verifier or reward model to supervise intermediate steps.\n\n---\n\n> **Q3: Effectiveness of the proposed method for \"Iterative Online Reflection Tuning\"**\n\n**A3:**  \nThanks for your constructive suggestions. We additionally include two baselines: 1) standard rejection sampling, and 2) sampling trajectories from a more powerful MLLM (distillation), and compare them with our proposed method in the online stage and evaluate the end-to-end performance on AndroidWorld. We evaluate these alternatives under a controlled setting where the total number of online training samples is kept similar across methods. For the distillation baseline, we use **Seed1.5-VL** (achieve 62.1 on AndroidWorld) to generate trajectories. The final results are shown in the table below.  \n|Baseline| SR on AndroidWorld (%)|\n|------------------|-------|\n|Standard Rejection Sampling|25.9|\n|Distillation|22.4|\n|Ours|**34.5**|  \n\nWe observe that these baselines exhibit the following limitations:  \n- **Standard rejection sampling** is inefficient for complex tasks involving one or several key steps in which the model might get stuck. The model may repeatedly fail to discover successful actions on its own. In contrast, our method enables the model to receive direct feedback on critical error steps, including pre-error correction and post-error reflection guidance from an MLLM, resulting in more efficient and focused improvement.\n- For **distillation**, we observe several challenges:  \n  1. If using prompted agentic methods to produce high-quality trajectories, the cost is prohibitively high, making it infeasible to scale.  \n  2. Even with strong end-to-end models like Seed1.5-VL, we found that their trajectories **lack explicit reflection behavior**. Through manual inspection and keyword analysis (e.g., \"mistake\", \"incorrect\", \"unsuccessful\"), we observe almost no signs of self-correction or reflective reasoning, even though Seed1.5-VL is trained on data sources similar to UI-TARS, which include some human-annotated annotations.\n  3. The student model design has to be aligned with the teacher model. For example, in our original design, the agent model predicts the thought, action description, and action, but the Seed1.5-VL only predicts the thought and final action. Therefore, we have to redesign the output format and action space of our model to be aligned with the teacher model. However, there still exist some issues where the style of the reasoning thought might be inconsistent with the one in the offline stage.\n  4. As a result, the effect of distillation in this case is similar to **simply adding more offline SFT data**, without the benefits of online reflective feedback. This leads to subpar improvements compared to our iterative reflection tuning.\n\nOverall, these comparisons further support the necessity of our design, where targeted correction and reflection behaviors are injected into the learning loop in a way that neither rejection sampling nor distillation can easily replicate. We will continue to expand experiments in this crucial stage to provide more comprehensive insights in the revised version.\n\n---\n\n### Minor Comments\n\n- **Q4:** Section 2.3 details are sparse  \n  **A4:** Thanks for pointing this out. We will improve this section and add a pointer to the appendix where the detailed annotation pipelines are described.\n\n- **Q5:** \"pre-training stage\" phrasing is confusing  \n  **A5:** Thank you for the feedback. In the GUI agent literature, the term \"pre-training\" typically refers to the GUI-specific pre-training stage applied on top of a base MLLM, and this usage is common in prior work. To avoid confusion, we will explicitly revise the phrasing to \"GUI pre-training\" throughout the paper to make this distinction clearer to readers.\n\n- **Q6:** Terminology \"agent-based\" vs. \"end-to-end\" seems unclear  \n  **A6:** Thank you for the suggestion — we agree that the terminology can be improved. In the revised version, we will adopt the clearer phrasing of \"prompted agents\" versus \"task-specifically tuned agents\", which more accurately reflects the distinction we intend to draw.\n\n- **Q7:** Line 54 claim about RL not helping reflection lacks citation  \n  **A7:** Our statement was specifically intended to describe the behavior we observed **within the end-to-end GUI training paradigm**. In our experiments, we found that GUI models trained with standard offline SFT on successful trajectories always fail to exhibit reflection or self-correction behaviors (please also see quantitative results in **A8**); therefore, we state that subsequent RL training alone does not reliably recover these capabilities, as such behaviors can not be obtained by exploration (sampling at a high temperature). This empirical observation was a key motivation for our GUI-Reflection framework.  \nWe will revise the sentence to make this scope and statement clearer, and we will supplement it with supporting results or qualitative examples to avoid misunderstanding and overgeneralization.\n\n- **Q8:** Quantify the improvement in error-correction behavior over a relevant baseline on AndroidWorld.  \n  **A8:** Thank you for the suggestion. To provide quantification results, we analyze the **occurrence of reflection-related behaviors** in the model's action thoughts on AndroidWorld. Specifically, we measure how often the model generates thoughts containing keywords indicative of self-reflection or error awareness (e.g., \"mistakenly\", \"unsuccessful\", \"incorrect\", etc.) While this does not directly assess the correctness of reflection, it serves as a proxy signal for the emergence of such behaviors. The ratio of the occurrence of reflection-related behaviors in our GUI-Reflection model is **14.2%**, while the baseline without reflection-related content in all stages is **0%**.\n\n- **Q9:** Clarify whether some 8B baselines released recently outperform your model  \n  **A9:** Thank you — we will include the discussion of these related concurrent methods in the next version."}, {"review_id": "FmCoqCNNW9", "reviewer": "Reviewer_pU3E", "summary": "The paper introduces GUI-Reflection , a framework that enhances end-to-end multimodal GUI models with self-reflection and error correction capabilities. Unlike traditional GUI models that rely on error-free training data and lack recovery mechanisms, GUI-Reflection integrates reflection behaviors through pre-training, offline supervised fine-tuning, and online reflection tuning. The framework includes a GUI-Reflection Task Suite to explicitly train and evaluate action verification, reversal, and mistake-informed reattempts. It also features an automated data pipeline and a distributed mobile environment for iterative online learning. Experimental results show that GUI-Reflection significantly improves the model’s ability to recognize and recover from errors, achieving strong performance on Android-based tasks and benchmarks like AndroidWorld.", "questions": "1.\tWhy does Action Verification in Figure 3 have two Purposes?\n\n2.\tIn Section 2.3, since MLLMs can be used to filter the coordinates generated by the GUI model, why not directly use MLLMs to generate the coordinates?\n\n3.\tI appreciate the authors’ approach of constructing error data in the first manner during the Offline SFT phase. Nevertheless, I would like to ask whether alternative strategies could be considered for generating erroneous actions—particularly those that are more representative of real-world failure cases. Focusing optimization efforts on actions with very low occurrence probabilities may not lead to meaningful improvements in practical scenarios.\n\n4.\tDoes Table 1b include results for Qwen series models? I suggest the authors add these for comparison.\n\n5.\tIn Table 2, \"Filtered BC\" should be expanded as \"Filtered Behavior Cloning\" upon its first appearance.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strength:\n1.\tThe paper addresses a highly important and timely problem in GUI agents—building reflection capabilities, which is a critical issue needing to be solved.\n\n2.\tThe proposed solution by the authors is reasonable and demonstrates a progressive enhancement of the model's reflective abilities.\n\n3.\tThe experimental results on the dataset validate that the proposed method effectively improves the reflection capabilities of GUI agents.\n\n### Weaknesses：\n\n1.\tMany details are missing, and no code is provided, even though the authors include supplementary material in the appendix (e.g., specifics about training loss functions).\n\n2.\tThe experimental baselines are somewhat limited. I recommend the authors include experiments based on the Qwen series models for more comprehensive evaluation.", "was_revised": true, "final_justification": "The authors have addressed most of my concerns, and I appreciate their clarifications. However, two key issues remain unresolved: (1) The generalization of the method is not convincingly demonstrated. Full results on the Qwen2.5 series are still missing, and the method was only tested on InternVL, limiting its demonstrated applicability. (2) The strategy for generating incorrect examples is unconvincing. Simply replacing correct answers to create negative cases may not generalize to real-world settings. Given these concerns, I maintain my score of 4.", "rebuttal": "We thank the reviewer for the thoughtful and encouraging feedback. We appreciate the recognition of our work’s importance and are grateful for the detailed suggestions, which we address below point by point.\n\n---\n\n> **Q1: Many details are missing (e.g., specifics about training loss functions), and no code is provided.**\n\n**A1:**  \nThanks for your suggestion. We use the standard cross-entropy loss as the loss function for training the MLLM, and we will add more information about training and model details in our next revision. We make sure the code, model, and data are open-sourced and publicly available.\n\n---\n\n> **Q2: Experimental baselines are somewhat limited. Suggest evaluating the Qwen series models.**\n\n**A2:**  \nWe agree that broader comparisons help assess generalization across architectures. We additionally conduct the GUI Pre-training with our reflection-related data on Qwen-2.5-VL-7B, and the results are shown below. We can see that, similar to the InternVL-8B baseline, after GUI Pre-training with reflection data, the model's reflection-oriented abilities are effectively improved. We will complete the full training pipeline in our framework (including the offline SFT and online training) with Qwen-2.5-VL-7B and add the corresponding results to our paper.\n\n| Model           | Action Verification | Action Reversal |\n|----------------|------------|--------------|\n| Qwen2.5-VL-7B   | 76.36      | 76.90        |\n| + GUI Ref Pre-training | 90.87      | 89.28        |\n\n\n| Model           | ScreenSpot | ScreenSpotv2 |\n|----------------|------------|--------------|\n| Qwen2.5-VL-7B   | 74.94      | 77.50        |\n| - 2nd attempt  | 79.63 | 82.39 |\n| - 3rd attempt  | 80.44 | 83.22 |\n| - pass@3       | 80.61 | 83.37 |\n| Qwen2.5-VL-7B + GUI Ref Pre-training   | 84.95      | 86.76        |\n| - 2nd attempt  | 91.67 | 92.35 |\n| **- 3rd attempt**  | **93.79** | **94.19**  |\n| - pass@3       | 89.22 | 90.30 |\n\n---\n\n> **Q3: Why does Action Verification in Figure 3 have two Purposes?**\n\n**A3:**  \nAs described in Section 2.2 and shown in Figure 3, Purpose 1 corresponds to the actual action goal (positive), while Purpose 2 is a plausible but incorrect goal (negative). In this task, the model is only provided with **one of these two purposes** and needs to determine whether the purpose has been achieved. We will add clarifications about this to avoid misunderstanding.\n\n---\n\n> **Q4: Why not directly use MLLMs to generate the coordinates?**\n\n**A4:**  \nWe use MLLMs to filter the coordinates generated by the GUI model by **visualizing the predicted coordinates with some red dots**. However, the MLLMs (e.g., Gemini or GPT-4o) can not directly generate or precisely understand the numerical coordinates, so we use our GUI agent to output the coordinates based on the MLLM-generated action thought as described in Section 2.3.\n\n---\n\n> **Q5: Could alternative strategies be used to generate more representative failure cases during error data construction?**\n\n**A5:**  \nThank you for this thoughtful suggestion. We believe the core concern lies in the representativeness of error cases, particularly regarding (1) real-world failure scenarios and (2) low-probability actions.\n\nFirst, collecting real-world failure for GUI cases is inherently challenging and costly, as it requires a lot of human supervision to avoid undesirable consequences or safety risks that erroneous actions on real apps could result in. Thus, we adopt a safer and more efficient strategy to utilize existing offline dataset to generate synthetic but plausible errors.  \nIndeed, one of the main challenges in constructing error cases from existing offline datasets is that they only contain successful trajectories, so we **lack the post-state screenshots that would result from actual erroneous actions**. This makes realistic error simulation non-trivial. To address this, we prompt the MLLM not only to modify the original goal but also to design the error cases to resemble natural and plausible human mistakes — such as misclicking visually similar elements or misinterpreting UI semantics. While these are not exhaustively representative, they capture common failure patterns and provide useful supervision for learning reflection behaviors. \n\nSecond, we clarify that our goal in the Offline SFT stage is **not to fix model's common mistakes**, but only to proactively activate the model's basic ability to anticipate and detect its own errors. The main goal of this stage is to endow the model with the behaviour to detect and correct errors, and these capabilities are then further refined and expanded in the online reflection tuning stage, where real interaction feedback allows more complex behaviors to emerge. \n\nLooking ahead, we plan to explore the use of **generative models or learned world models** to simulate a wider range of realistic erroneous actions and resulting states, enabling richer error distributions and more robust error-recovery learning.\n\n---\n\n> **Q6: Add results for Qwen series models in Table 1b.**\n\n**A6:**  \nThanks for your suggestion. We have added the corresponding results. Please see the table in A2.\n\n\n---\n\n> **Q7: Table 2 should expand “Filtered BC” as “Filtered Behavior Cloning” at first appearance.**\n\n**A7:**  \nThank you for pointing this out — we will revise the text to expand “Filtered BC” as “Filtered Behavior Cloning” upon its first appearance for clarity."}]

Author rebuttal:

Meta-review: This paper presents GUI-Reflection, a framework engineered to empower multimodal GUI agents with self-reflection and error-correction functionalities, thereby addressing their prevalent inability to recover from errors. In contrast to conventional GUI models that rely solely on error-free training datasets and lack recovery mechanisms, GUI-Reflection integrates reflection behaviors through pre-training, offline supervised fine-tuning, and online reflection tuning. This comprehensive approach substantially enhances performance, enabling an 8B model to surpass prior models of its size and compete with significantly larger systems on the AndroidWorld benchmark.

The reviewers concur that the paper addresses a pertinent and timely issue: equipping GUI agents with the capability to recognize and recover from errors. The proposed method is demonstrated to yield robust and competitive outcomes on the public AndroidWorld benchmark. The major concern arises from the restricted experimental scope, which employs a restricted set of baseline models for comparison and testing primarily confined model scale and platform. Consequently, the reviewers suggest that the paper can benefit from a revision incorporating these modifications to comprehensively evaluate the paper within this revised context.

The authors are also engaged in the rebuttal phase, proactively answering questions and addressing concerns raised in the initial reviews.
Decision: Accept (poster)