Title: GUI-Reflection: Empowering Multimodal GUI Models with Self-Reflection Behavior
Venue: NeurIPS 2025
Accepted: yes
Keywords: GUI Agent

Multimodal Large Languge Model

Vision Language Model

Reflection
Abstract: Multimodal Large Language Models (MLLMs) have shown great potential in revolutionizing Graphical User Interface (GUI) automation. However, existing GUI models mostly rely on learning from nearly error-free offline trajectories, thus lacking reflection and error recovery capabilities. To bridge this gap, we propose GUI-Reflection, a novel framework that explicitly integrates self-reflection and error correction capabilities into end-to-end multimodal GUI models throughout dedicated training stages: GUI-specific pre-training, offline supervised fine-tuning (SFT), and online reflection tuning. GUI-reflection enables self-reflection behavior emergence with fully automated data generation and learning processes without requiring any human annotation. Specifically, 1) we first propose scalable data pipelines to automatically construct reflection and error correction data from existing successful trajectories. While existing GUI models mainly focus on grounding and UI understanding ability, we propose the GUI-Reflection Task Suite to learn and evaluate reflection-oriented abilities explicitly. 2) Furthermore, we built a diverse and efficient environment for online training and data collection of GUI models on mobile devices. 3) We also present an iterative online reflection tuning algorithm leveraging the proposed environment, enabling the model to continuously enhance its reflection and error correction abilities. Our framework equips GUI agents with self-reflection and correction capabilities, paving the way for more robust, adaptable, and intelligent GUI automation, with all data, models, environments, and tools to be released publicly.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces GUI-Reflection , a framework that enhances end-to-end multimodal GUI models with self-reflection and error correction capabilities. Unlike traditional GUI models that rely on error-free training data and lack recovery mechanisms, GUI-Reflection integrates reflection behaviors through pre-training, offline supervised fine-tuning, and online reflection tuning. The framework includes a GUI-Reflection Task Suite to explicitly train and evaluate action verification, reversal, and mistake-informed reattempts. It also features an automated data pipeline and a distributed mobile environment for iterative online learning. Experimental results show that GUI-Reflection significantly improves the model’s ability to recognize and recover from errors, achieving strong performance on Android-based tasks and benchmarks like AndroidWorld.\n\nSTRENGTHS AND WEAKNESSES:\n### Strength:\n1.\tThe paper addresses a highly important and timely problem in GUI agents—building reflection capabilities, which is a critical issue needing to be solved.\n\n2.\tThe proposed solution by the authors is reasonable and demonstrates a progressive enhancement of the model's reflective abilities.\n\n3.\tThe experimental results on the dataset validate that the proposed method effectively improves the reflection capabilities of GUI agents.\n\n### Weaknesses：\n\n1.\tMany details are missing, and no code is provided, even though the authors include supplementary material in the appendix (e.g., specifics about training loss functions).\n\n2.\tThe experimental baselines are somewhat limited. I recommend the authors include experiments based on the Qwen series models for more comprehensive evaluation.\n\nQUESTIONS:\n1.\tWhy does Action Verification in Figure 3 have two Purposes?\n\n2.\tIn Section 2.3, since MLLMs can be used to filter the coordinates generated by the GUI model, why not directly use MLLMs to generate the coordinates?\n\n3.\tI appreciate the authors’ approach of constructing error data in the first manner during the Offline SFT phase. Nevertheless, I would like to ask whether alternative strategies could be considered for generating erroneous actions—particularly those that are more representative of real-world failure cases. Focusing optimization efforts on actions with very low occurrence probabilities may not lead to meaningful improvements in practical scenarios.\n\n4.\tDoes Table 1b include results for Qwen series models? I suggest the authors add these for comparison.\n\n5.\tIn Table 2, \"Filtered BC\" should be expanded as \"Filtered Behavior Cloning\" upon its first appearance.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nNoting that current UI control models lack strong abilities to recognize and correct mistakes in executing instructions. To this end, the authors propose 5 new methods for generating training data, spread across 3 proposed training phases:\n\n* A \"GUI pre-training\" phase along with methods for generating \"action verification\", \"action reversal\", and \"mistake-informed reattempt\" SFT data.\n* A \"GUI offline SFT\" phase, with methods for augmenting existing trajectory data to construct reflection and error correction scenarios.\n* A \"GUI online learning\" phase, an interative process for collecting additional data for error correction via environment interaction.\n\nThe authors show that this recipe enable constructing a competitive 8B model on the AndroidWorld datasets. Some qualitative anecdotes of error correction behavior are observed.\n\nSTRENGTHS AND WEAKNESSES:\n__Strengths__\n\n* The limited ability of many current UI control models to recognize and recover from incorrect actions is an important problem to study.\n* The authors' proposed methods for generating additional training data focused on this capabilitiy may provide useful, reusable approaches for improving performance.\n* The authors demonstrate that their proposed recipe enables competitive performance on the AndroidWorld benchmark.\n\n__Weaknesses__\n\nWhile the paper addresses an important problem, and some of the proposed methods may in fact be quite useful for addressing it, I did have some issues with the current manuscript and set of results that I think would benefit from revisions and resubmission. The proposed training pipeline is quite complex, and the paper proposes many new methods for data augmentation. However, the experiments don't seem to sufficiently justify the proposed complexity and diversity of methods relative to prior work. The paper reads as a collection of ad-hoc methods, without clear take-aways for which aspects of the overall recipe are most useful for future work to adopt, and why.\n\nMore specifically:\n\n* The pipeline involves 3 distinct training phases in addition to the base model's pre-training. However, there is only one end-to-end result in the main paper on a standard end-to-end benchmark: the AndroidWorld result in Table 3. To justify the complexity of the proposed approach, it would have been useful to show clearer ablations for how the various components affect end-task performance. The single competitive-but-not-SOTA result on AndroidWorld is not particularly useful for understanding this. How does each of the proposed datasets and training phases affect end-task performance? Can the recipe be simplified or the number of training phases reduced? These are questions that would have been helpful to address in the paper.\n* In general, the paper would perhaps read better if it was more focused on a smaller set of methods and contributions. If there are some aspects of the overall recipe that do not significantly impact end-task performance, perhaps these could be relegated to the appendix.\n* Similarly, the number of different proposed training sets and training phases also made the exposition of the overall recipe, and the presentation of the various intermediate results in Tables 1 and 2, a bit difficult to follow at times.\n* Key questions related to the effectiveness of the proposed method for \"Iterative Online Reflection Tuning\" relative to prior work don't appear to be addressed. For example, how important is it that this method uses on-policy sampling from the student model vs. sampling trajectories from a more powerful MLLM (more akin to a standard distillation procedure)? How does the proposed method compare to more standard rejection sampling? Ablations comparing end-to-end task performance would be useful to answer these questions. Is simply scaling up this training stage sufficient towards improving error correction abilities? Focusing on answering these questions alone would have been an interesting and useful paper.\n\nMinor nits:\n\n* The details in section 2.3 of how existing models were used for data annotation were a bit sparse, perhaps there is an Appendix section to reference?\n* \"pre-training stage\" -> This is seemingly an unconventional and confusing name for a stage that occurs after the base model has been pre-trained (InternVL2.5-8B)\n* The terminology \"agent-based\" vs. \"end-to-end\" seemed a bit strange, given that all approaches for UI control are \"agents\" in some sense. Would \"prompted\" vs. \"task-specific tuning\" be accurate?\n* Line 54 in Introduction - \"further RL training cannot effectively activate or enhance such reflection abilities\" - citation needed? This claim seems questionable in general.\n* While the examples in Figure 6 are useful, it would be useful, if possible,  to quantify the improvement in error-correction behavior over a relevant baseline on AndroidWorld.\n* (Not a weakness, just a note) Looking at the AndroidWorld leaderboard, there do appear to be 8B models that outperform the proposed model (e.g. ScaleTrack, V-Droid), but these appear to be released within the last couple months, and therefore should be considered concurrent work.\n\nQUESTIONS:\nSee weaknesses above.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces GUI-Reflection, a framework that adds explicit self-reflection and error-correction behaviours to end-to-end multimodal GUI agents. During GUI-specific pre-training, the authors inject three synthetic tasks—Action Verification, Action Reversal and Mistake-Informed Re-attempt—built from existing successful trajectories. In offline SFT, they automatically augment clean demonstrations with reflection episodes that contain (i) the original wrong action, (ii) a natural-language action-thought explaining the error, and (iii) pre- and post-correction actions. An online reflection-tuning loop runs in a mobile-device sandbox: after every rollout, mistakes are detected, labelled and verified by an MLLM-based checker, then fed back for further fine-tuning. On the authors’ suite of 18 mobile tasks, success on level-2 tasks jumps from 14.6 % → 34.7 % after three reflection-tuning iterations, and the final 8 B-parameter model reaches 34.5 % success on the public AndroidWorld benchmark—outperforming all prior end-to-end 8 B models and rivaling much larger systems (e.g. UI-TARS-72 B).\n\nSTRENGTHS AND WEAKNESSES:\nPros:\n\n• First GUI agent to demonstrate emergent self-reflection without human feedback, a capability likely needed for reliable automation.\n\n• End-to-end data pipeline—no human labour after initial success logs.\n\n• Competitive results on a public benchmark.\t\n\nCons:\n\n• Only 8b model is tested, no evidence of the method effectiveness on various scales, especially larger ones where true performance bottleneck emerges.\n\n• Reflection gains measured on synthetic tasks; real app generalisation unclear.\n\n• Scope limited to Android mobile; no evidence on web/desktop/iOS GUIs.\n\n• Eval relies on MLLM judge, and verification precision/recall not quantified, leading to risk of error accumulation.\n\nQUESTIONS:\nWhat fraction of synthetic corrections are rejected by the programmatic / MLLM checker, and what is the observed false-accept rate? A small human audit (e.g. 100~200 samples) would ground confidence in the pipeline.\n\nLIMITATIONS:\nSee weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-05"}