Title: Enhancing GUI Agent with Uncertainty-Aware Self-Trained Evaluator
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multimodal Large Language Model

GUI

Agent

Automatic Evaluation
Abstract: Benefiting from the availability of extensive navigation trajectories, both manually and automatically annotated, current graphical user interface (GUI) agents have achieved remarkable advancements in performance. However, these annotated datasets often contain substantial noise, which impedes effective agent training and underscores the necessity for rigorous trajectory quality assessment. In contrast to existing prompting-based evaluators that rely on proprietary multimodal large language models (MLLMs), we propose an Uncertainty-aware Reinforced Self-Training (URST) framework to train lightweight MLLMs for efficient and reliable trajectory evaluation. URST iteratively fine-tunes MLLMs using their own generated thoughts and judgments to enable self-improvement, while its uncertainty-aware sampling strategy ensures the selection of the most informative training examples. To further enhance reasoning and judgment capabilities, we propose a simplified group policy optimization approach that effectively leverages diverse positive and negative samples for evaluator learning. Our evaluator demonstrates superior judgment performance across both in-domain and out-of-domain datasets. When used to filter navigation datasets, it consistently leads to performance improvements in training GUI agents.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a framework, Uncertainty-aware Reinforced Self-Training, for training lightweight Multimodal Large Language Models to serve as efficient and reliable evaluators for GUI agent navigation trajectories. Addressing the limitations of costly and domain-agnostic prompting-based methods that rely on proprietary MLLMs, URST enables an evaluator to improve itself iteratively by learning from its own generated thoughts and judgments. The framework introduces two key technical contributions: a novel uncertainty-aware sampling strategy that uses prediction entropy to select the most informative training examples, and a Simplified Group Policy Optimization algorithm that effectively leverages both positive and negative samples to enhance the model's reasoning. The authors demonstrate that their URST-trained evaluator not only surpasses the performance of prompting-based and other fine-tuning methods on both in-domain and out-of-domain datasets but also leads to consistent performance improvements in downstream GUI agents when used as a data filter.\n\nSTRENGTHS AND WEAKNESSES:\n__Strengths__:\n- The writing of the paper is clear, making it easy to follow.\n\n- The problem formulation is interesting and novel.\n\n\n__Weaknesses__:\n\n- The biggest concern of this work is insufficient baseline comparisons. In Table 1, the paper should also report the performance of non-finetuned models, (namely, Qwen2VL-2B, Qwen-VL-Max, gpt-4o etc) for better understand the effectiveness of their proposed training methods and advantages over proprietary models. Additionally, since the proposed approach also performs some forms of distillation of larger models (line 167), the author should also compare with other simpler distillation-based methods so that readers can better appreciate the proposed method.\n\n\n- Also critically, performance enhancement is quite minimal in both in-domain and cross-domain when it comes to actually enhancing GUI agent performance. In Section 4.4, the improvement is < 2% on in-domain and <1% on out-of-domain. Based on the results, it is hard to convince that the trajectory evaluator is a must/effective in enhancing GUI agent performance. \n\n- For section 4.4, the author should also compare their approach with others. In fact, the comparison with baselines here is more important than Table 1 as they reflect how different approaches are able to improve GUI agent performance, which is the ultimate goal of building a trajectory evaluator.\n\nQUESTIONS:\n- Line 123: why is this approach/formulation restricted to “offline” dataset online?\n\n- Equation 1: why does the evaluator need to take in the chain-of-thought c as input?\n\n- Can you elaborate on how the “Prompting-based Methods” are implemented? For example, for AgentTrek, did you just use gpt-4o-mini and fasttext classifier to filter trajectories and then train the model on the filtered dataset?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-13"}

comments: SUMMARY:
This paper proposes a framework for developing evaluators for GUI agents. Unlike API--based evaluators, which are expensive or small LM based evaluators which may not be as good, it seeks to develop better SLM-based judges. The key innovation is an iterative self-training based approach to select quality examples for training MLLMs. It uses uncertaining-aware reinforced training to select quality examples that add more value. The results are reported on AITW-ID, AITW-OOD, and a benchmark based on AndroidWorld created by authors.

STRENGTHS AND WEAKNESSES:
Strengths:
1. Alleviates the issues of existing evaluators such as high cost of API-based evaluators
2. Makes use of negative samples as well in reinforced self-training
3. Improvement in GT agreement compared to existing baselines

Weaknesses:
1. Entropy is an indicator of confidence of evaluator. Although the entropy decreases (Fig. 4), it remains high. 
2. The performance on standard benchmarks (AITW-ID, AITW-OOD) is 2-3%, which is marginal.

QUESTIONS:
1. Is this approach applicable just to GUI agents or LLM agents in general? I see no design considerations specific to GUI agents. It would be great if the authors can evaluate this on LLM agent benchmarks in general e.g. JudgeBench [1].

[1] Tan et al., JudgeBench: A Benchmark for Evaluating LLM-based Judges, ICLR 2025.

LIMITATIONS:
yes, limitations in appendix.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-06-21

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces URST, a framework for training MLLMs to evaluate GUI navigation trajectories more reliably. URST enables self-improvement through iterative self-generated judgments and uses an uncertainty-aware sampling strategy to select informative training examples. A simplified group policy optimization (SGPO) method helps the evaluator learn from diverse feedback. The resulting evaluator outperforms existing approaches on both in-domain and out-of-domain data and improves GUI agent training.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\nThe model achieves strong performance across multiple benchmarks. The writing is clear and the method is presented in a straightforward yet effective way. The motivation for self-improvement is particularly compelling in the GUI domain, where the cost of human annotation is high and scalable data generation is challenging.\n\n## Weaknesses\n\n**Reliance on Evaluator Uncertainty**: The method relies heavily on the uncertainty of the evaluator to guide trajectory selection. This implicitly assumes that the evaluator is sufficiently accurate, and when it expresses high confidence, the corresponding trajectory is likely of high quality. However, since the evaluator is not trained on any ground-truth data, its overall reliability is questionable. This casts doubt on whether its uncertainty is a trustworthy signal for trajectory assessment. A more thorough evaluation either automatic or human on how well the evaluator’s uncertainty correlates with actual trajectory quality would greatly strengthen the claim.\n\n**Evaluator Evolution**: It would be valuable to track and report the performance trajectory of the evaluator itself over the course of training. Showing how its evaluation quality changes particularly in early iterations through either automatic metrics or human judgments would provide insights into the robustness and stability of the evaluator as it evolves.\n\n**Alternative Sampling Metrics**: While evaluator uncertainty is used to select trajectories, it is not the only potential signal for quality. For instance, policy uncertainty could also serve as an indicator. If the policy model is uncertain about its decisions, the trajectory may be of lower quality. Comparing the effectiveness of evaluator uncertainty against alternative metrics (e.g., policy entropy) would offer a more comprehensive justification for the current choice and potentially reveal better alternatives.\n\n**Baseline RL Methods**: Although the authors mention GRPO when introducing SGPO, there is no ablation or comparison with GRPO, PPO, or other standard reinforcement learning methods. This makes it unclear whether SGPO is tailored for the self-improvement framework or simply an arbitrary choice. Including these comparisons would help clarify the advantages of SGPO and whether it meaningfully outperforms existing baselines.\n\nQUESTIONS:\n**Generalization Beyond Mobile GUI Tasks**: The evaluation currently focuses only on mobile GUI tasks. To demonstrate broader applicability, it would be helpful to test the pipeline on other GUI domains, such as desktop or web interfaces. This would provide stronger evidence of the method's generalizability across varied interaction modalities.\n\nLIMITATIONS:\nSee Weaknesses part.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel framework called Uncertainty-aware Reinforced Self-Training (URST) for training lightweight multimodal large language models (MLLMs) to evaluate the quality of GUI navigation trajectories. Unlike existing methods that rely heavily on prompting proprietary models, URST trains evaluators using self-generated data, enhanced by uncertainty-aware sampling and simplified group policy optimization (SGPO). Extensive experiments demonstrate that URST outperforms state-of-the-art prompting-based and fine-tuning methods, particularly in generalization to out-of-domain data. The trained evaluator also enhances downstream GUI agents by filtering noisy training data.\n\nSTRENGTHS AND WEAKNESSES:\n***Strengths***\n\nFirst, the paper addresses a critical bottleneck in GUI agent training—noisy trajectory data—by proposing an efficient, cost-effective, and domain-adaptive evaluator. The use of lightweight open-source MLLMs and self-training avoids the high latency and cost of proprietary models. The URST framework incorporates two novel components: (1) uncertainty-aware sampling to prioritize informative, high-entropy samples and (2) SGPO to leverage both correct and incorrect samples for contrastive learning. These innovations are validated through rigorous ablation studies, showing measurable performance gains, especially in out-of-domain scenarios.\n\nSecond, the experiments are comprehensive and reproducible, with comparisons across multiple benchmarks, including in-domain (AITW-ID) and out-of-domain (AITW-OOD, AW-OOD) datasets. URST shows consistent improvements in F1 score and downstream agent performance. The authors clearly document their method, training configuration, datasets, and open-source code/data commitments, which greatly increases the impact and credibility of the work. Additionally, the qualitative examples illustrate robust reasoning capabilities and explain how URST corrects common failure modes in baseline methods.\n\n***Weaknesses***\n\nDespite its strengths, the paper relies on a proprietary MLLM to generate the initial training data, creating a partial dependency on the very systems it aims to replace. Although the authors argue this is limited to initialization, it could limit true independence in practice. Moreover, statistical significance and variability are not fully explored—the paper omits error bars due to compute limitations, which may raise questions about result stability.\n\nAnother minor weakness is that URST’s scalability to larger models or real-time deployment scenarios is not discussed in depth. While lightweight models are used for efficiency, more details on training cost, iteration limits, and potential bottlenecks (e.g., entropy computation) would strengthen the paper’s practical relevance.\n\nOverall, this paper presents a timely and technically significant contribution to GUI agent training and trajectory evaluation by enabling efficient, domain-adaptive, and self-improving evaluators. The proposed URST method introduces meaningful algorithmic innovations and demonstrates solid empirical gains over existing approaches. Its potential for real-world impact is high, particularly for practitioners seeking to train robust GUI agents without relying heavily on commercial LLM APIs.\n\nQUESTIONS:\n1. The paper reports performance metrics without error bars or statistical significance due to compute constraints. The authors could run multiple trials with different random seeds and report standard deviation or confidence intervals.\n\n2. The reasoning performance is described qualitatively but not measured explicitly. It is suggested to quantify the quality of generated chain-of-thought (CoT) using human or model-based annotation. And add visualizations or examples of failure cases to show when/why URST succeeds or fails.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}