Enhancing GUI Agent with Uncertainty-Aware Self-Trained Evaluator
Authors: Gongwei Chen
Lirong Jie
Lexiao Zou
Weili Guan
Miao Zhang
Liqiang Nie
Venue: NeurIPS 2025 poster ()
Keywords: Multimodal Large Language Model
GUI
Agent
Automatic Evaluation
Primary area: evaluation

Abstract: Benefiting from the availability of extensive navigation trajectories, both manually and automatically annotated, current graphical user interface (GUI) agents have achieved remarkable advancements in performance. However, these annotated datasets often contain substantial noise, which impedes effective agent training and underscores the necessity for rigorous trajectory quality assessment. In contrast to existing prompting-based evaluators that rely on proprietary multimodal large language models (MLLMs), we propose an Uncertainty-aware Reinforced Self-Training (URST) framework to train lightweight MLLMs for efficient and reliable trajectory evaluation. URST iteratively fine-tunes MLLMs using their own generated thoughts and judgments to enable self-improvement, while its uncertainty-aware sampling strategy ensures the selection of the most informative training examples. To further enhance reasoning and judgment capabilities, we propose a simplified group policy optimization approach that effectively leverages diverse positive and negative samples for evaluator learning. Our evaluator demonstrates superior judgment performance across both in-domain and out-of-domain datasets. When used to filter navigation datasets, it consistently leads to performance improvements in training GUI agents.

Enhancing GUI Agent with Uncertainty-Aware Self-Trained Evaluator
Gongwei Chen, Lirong Jie, Lexiao Zou, Weili Guan; Miao Zhang, Liqiang Nie
Harbin Institute of Technology, Shenzhen
{chengongwei, zhangmi ao, g https:// thub. com Com/JLL81888/URST7
edu, on
Abstract
Benefiting from the availability of extensive navigation trajectories, both manually and automatically annotated, current graphical user interface (GUT) agents have achieved remarkable advancements in performance. However, these annotated datasets often contain substantial noise, which impedes effective agent training and underscores the necessity for rigorous trajectory quality assessment. In con- trast to existing prompting-band based evaluators that rely on proprietary multimodal large language models (MLLMs), propose Self- Trainingg (URST) framework to train lightweight MILMs for efficient and reliable trajectory evaluation. URST iteratively fine-tunes MLLMs using their own generated thoughts and judgments to enable self-improvement, while its uncertainty-aware sampling strategy ensures the selection of the most informative training examples. To further enhance reasoning and judgment capabilities, we propose simplified group policy optimization approach that effectively lever- ages diverse positive and negative samples for evaluator learning. Our evaluator demonstrates superior judgment performance across both in-domain and out-of- domain datasets. When used to filter navigation datasets, it consistently leads to performance improvements training GUI agents.
pnncetaiinty-wwrr Reinforced
Introduction
With the powerful language under- standing and generation abilities, (multimodal) large language model- based agents [53
Navigation Trajectory
(a) Prompting-based Methods
What's the weather like in Chicago?
Telbreed Prompt
Judgmannt
38, 45 can perform more effectively across diverse and complex tasks, demon- strating greater generality and adapt- ability. In graphical user inter- face (GUT) scenario, the paradigm for building agents has shifted from to na- In the 57
62
7
28
Derght
MILLM
inegu
Unceriarity Awtrre Sampling
Tojectriee
SGPO
N
Judgment
MLLM
framework-basee 42
57
Uncertainty-Awee Reinforeed Self-Training
tive agents 6
20
carly stage, many works
422
16] leverage the advanced founda- tion models by designing task-specific workflows and optimizing prompts to construct framework based agents.
Figure Comparison uncertainty-awarr reinforced self- training (URST) method with prompting-basee methods. Our URST iteratively fine-tunes lightweight MLLMs with self- generated thoughts via reinforcement learning.
"Corresponding authors
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

The human-defined workflows make the agents hard to adapt to continuously evolving environ- ments (3. promising paradigm to build native agents by learning GUI navigation tasks in an end-to-end manner, ensuring the scalability and adaptabilityy of agents.
The main challenge of building native agents is the requirement for large-scale GUT navigation data. Recently, growing number of navigation trajectory datasets [34, 59 S 24 have emerged, collected through various schemes ranging from manual semi-automated and fully automated approaches. Although these datasets have driven significant progress in native agent development, their inherent noise and quality issues continue to hinder further improvement [59 27 These data
deficiencies demand automated evaluation methods that can act as quality filter. Current autonomous trajectory evaluation approaches [27 [50 [3] mainly rely on advanced proprietary multimodal large language models (such as GPT-4V. Qwen VL-Max, Gemini, etc.) with tailored prompting and continuous API calls. These prompting-her
evaluators suffer from two main limitations: (1) the significant domain gap between general and GUI scenarios; (2) the substantial computational cost. and delay associated with proprietary MILMM API calling.
To address these challenges, we devise an uncertainty-aware reinforced self-training (URST) method to train an open-sourced lightweight MLLM as an efficient trajectory evaluator. In contrast to supervised fine-tuning on a well-constructed dataset, we explore a self-training strategy that eliminates the need for thought annotations while still enhancing the model's reasoning and judgment capabilities. Our URST method iteratively fine-tunes the model on self self-generated data, achieving model self- improvement. each iteration, our method first performs sampling-bast based data collection, then fine- tunes the model on newly generated data. In contrast to existing self-training methods [56] 12 (9),
we propose a novel uncertainty-awes -aware sampling method to prevent the inclusion of easy samples in the generated dataset. To facilitate efficient learning with diverse positive and negative samples, we devise a simplified group policy optimization method to replace the rejection sampling fine-tuning. To mitigate the difficulty bias, our SGPO eliminates the standard deviation normalization and equally treats each training input, enabling an accurate measurement of sample advantage.
The experiments reveal superior performances of URST in both in in-ommiin (an average gain of 2. 2.45% and oft-of-domain (an average gain of 2. 2.89%) datasets, compared to the of prompting- based methods. This comparison suggests that domain-specific adaptation through self-training can outweigh the general reasoning power of large foundation models. Notably, evaluators (84.13%) trained with self-generated synthetic data exhibit significantly larger performance than those (79.01% %) trained on data from proprietary MLLMs, highlighting the potential of self seff-geneeated data. We also implement some self-training works and compare them with URST. The results show that URST outperforms these methods by a margin of 6%, thanks to advanced sampling and policy optimization techniques. We also perform GUI agent training with the self-trained evaluator and obtain consistent performance improvements on two navigation datasets. Overall, our findings suggest reinforced self-training promising approach to training powerful GUI evaluators.
The key contributions of this work are:
We introduce URST that enables learning from self-generated data for training efficient MLLM-based evaluators, which is the first fine-tuning work for GUI trajectory evaluation.
We devise a novel uncertainty-aware sampling method for generating more valuable samples for effective fine-tuning. Mcanwhile, simplified group policy optimization is proposed to leverage positive and negative samples for reasoning and judgment enhancement.
Extensive experiments demonstrate the effectiveness of our URST on two automatically manually collected datasets. The evaluator trained with URST achieves superior perfor- mances in judgment accuracy and shows a positive effect on navigation performance.
2
Related Works
I7 have significantly advanced agent research by enabling richer perception and reasoning across various domains such as gaming environments [45] [8] 19,, and GUI interaction 31 48). GUI Agents can be categorized into Agent Frameworks and Native Agent Models based on whether they fine-tune the leverage the general understanding and reasoning capabilities of advanced MLLMs. These works enhance the flexibility
GUI Agents.
Recently, multimodal large language models (MLLMs)
base model on the GUI domain [3I. Agent Framework systems (60

---

of task execution by designing task modules such as planning, execution, memory, reflection, etc.. and optimizing prompts for each component, However, agent frameworks depend on manually encoding GUI domain knowledge through custom prompts, external scripts, or tool-usage heuristics, which inherently limit their adaptability to evolving GUI scenarios without human expert involvement.
Recent advancements [47, 4 [1 [20 have shifted towards constructing large-scale GUI operating trajectories for line-tuning MLLMs, thereby enabling end-to-end perception, planning, and execution natively within the GUI domain. These native agent models reduce the need for human-engineered workflows by utilizing either manually annotated or automatically collected trajectories, providing the potential for self-evolution as environments change. In contrast, work tries to build evaluator to measure trajectory quality, remove ineffective samples automatically, and ultimately enhance the performance of GUI agents.
Autonomous Evaluation by LLMs. Recently, LLM-as-a-judge approaches have been proposed to evaluate the correctness of the trajectory. AutoEval 27 prompts a general advanced MLLM to evaluate task completion with the last screenshot. WebVoyager [3] further inputs all screenshots for judgment. Some works [50] [0] also incorporate a general LLM to evaluate sample quality during dataset construction. Webjudge [51] proposes evaluation pipeline with key screenshots identification to reduce token consumption. These approaches rely on general proprietary LLMs, which lack domain-specific knowledge tailored to GUI environments. In contrast, we advocate transitioning from prompting-basee to fine-tuned evaluators, facilitating domain-speciff evaluation via model self-improvement.
Reinforcement Learning with LLMs. Reinforcement Learning (RL) has been demonstrated to significantly enhance model performance during the post-training phase of LLMs Early works utilized RL to align LLM outputs with human preferences (55] through algorithms such Proximal
Policy Optimization [35] and Direct Preference Optimization [32]. More recently, Deepseek [66
10 proposes group relative policy optimization (GRPO) by introducing group-level advantage estimation, and substantially augments the reasoning capabilities of LLMs. Following this pioneering have applied GRPO across various domains, consistently reporting notable performance gains. Our work leverages group-level policy optimization for enhancing the trajectory evaluator's 's reasoning ability in self-evolved training. To the best of our knowledge, this is the first study to employ reinforcement learning for evaluating GUI trajectories.
work, a growing body of studies 226 58,
22 29
3 Method
3.1 Problem Definition
GUI navigation task requires an agent to predict a sequence of actions and interact with an environment based on task instructions. Here, we focus on vision-based GUI navigation offline datasets, where each sample is composed of task instruction I, a sequence of state-action pairs S = {su, ....... ST, or }. In this work, 8; represents a screenshot observation instead of structural text description (like A Ally tree). Given a navigation sample, the trajectory evaluator & produces chain-of-thought (CoT) c, and a judgment y, which can be formulated as follows:
(c, v) = E(x), r= (I.
(1)
Due to the introduction of CoT reasoning, we adopt a language modeling objective to train the evalu- ator instead of a binary classification objective. The CoT reasoning provides abundant information about task intention, state content, and task progress to make a solid judgment.
The dominant paradigm of constructing automatic evaluators for GUI Agent is adapting a proprictary MLLM with tailored prompts, termed prompting based based evaluator. This paradigm is quite expensive and relies on API calls to proprietary models. Another direction is to fine-tune an spuresourced MLLM with the training samples annotated by advanced proprietary MLLMs for low-cost local de- ployment. However, this supervised fine-tuning method may be constrained by the limited capability of proprietary MILMs in domain-specific tasks. In this work, we devise novel uncertainty- reinforced self-training method fine-tune open-sourced MILMSs as trajectory evaluators, enabling model self-improvemment through their own outputs. The whole procedure is shown in Algorithm
3

---

Initial Training
N 1=11
Uncertainty-Awar Sampling Top-I. selection
Simplified Group Policy Optimization
Trajectories
D
D
1) CoT Reasoning c
{
2) Trajectory Judgment
Entropy-bused Ranking argsort argsortylcct., (e(()))E))))
Advantage
A
Policy Grad. Optimization
Dinit
{(2,, Cia V/ 1
Nimn
m
[las, cJt 1
m
Initial Evaluator: & tnstt
(1) Initialization
Next lteration Model
(2) Uncertainty-AAare Reinforced Self-training
Overview of our uncertainty- aware reinforced self- self-training (URST) framework. We begin by using an initial training set to adapt a general MILM to the trajectory evaluation domain. Subsequently, URST iteratively fine-tune the MLLMM with its own thoughts and judgments, guided by uncertainty-aware sampling and simplified group policy optimization.
Figure 2:
3.2 Uncertainty-awarr Reinforced Self-Training
Our approach first assumes access to an initial MLLM-base evaluator ginit set of training inputs D. and reward model R judge the correctness of the outputs. Given a training input æ: () the MLLM-based evaluator is expected to generate (1) a CoT reasoning c, followed by (2) a final judgment y to the given input. We require reward model verify the correctness of the final judgments, rather than assessing the internal CoT reasoning process. Specifically, a proprietary MLLM adapted to verify the judgment as the reward model. Our method iteratively samples multiple training inputs from D and outputs from the current evaluator &, then constructs training datasets fine-tune the evaluator via simplified group policy optimization.
3.2.1 Initialization
As the trajectory evaluation is not a common task in MI.LM pre-training or instruction-tuning current open-sourced MI.LMs, especially small ones, struggle with understanding high-level navigation instructions and complex state-action pairs. We construct and collect small amount of trajectory evaluation data Dinit { Ci, y:) Nam fine-tune an open-sourced MILM the initial evalua- tor. To collect such data, we directly prompt an advanced proprietary MILMM to generate trajectory judgments with detailed CoT reasoning.
Given the initial trajectory evaluation dataset Dinit, we fine-tune an Iiltuuction-tunee MLLM TB as the initial evaluator by minimizing the negative log likelihood loss:
LSFT -B. [log ginit (Ci, ye [r:)]
After the initialization, our MLLM-bases evaluator acquires a basic ability to assess unseen trajectory data. We start from this initial evaluator and devise uncertainty- aware sampling method to expand the training data and use iterative reinforced self seff-training to develop more powerful evaluator.
3.2.2 Uncertainty-Aware Sampling
Given set of training inputs D, ML.L--baee evaluator m" sample subset of training inputs { 1 in the m-th iteration, and generate K output sequences {(ef, vf } for each training
input zy based on the current evaluator. Specifically, by setting the temperature to a value greater than O, the MLLM-based evaluator repeated inference. Based on the generated dataset mm = {(I,. cf vf | E [N' & E [K]] a reward model will score these samples by verifying the correctness of their judgments y. In experiments, the reward score simply corresponds to p if / = Bg. and otherwise, Y, is the ground truth annotated by the reward model.
produce different outputs for the same input through K
The self gelf-generated dataset mm with the rewards can be directly used to fine-tune the current evaluator to yicld a new evaluator for the next iteration. However, we find the random sampling of inputs is

---

Algorithm Uncertainty-awaae Reinforced Self- Training
Input: an instruction-tuned MLLM B. a set of training inputs D, a reward model. Initialization
Generate the initial dataset D Dinit by using proprictary MLLM: Dimit = {(2,, C₁ ys );= N
Fine-tune R to obtain the initial evaluator gindt
Cold start by supervised fine-tuning
8d = gini
for m M do
// Uncertainty- aware Sampling
Generate dataset Dm by uncertainty-awant sampling: Dm {(z), ct. 15) 1 [L],
[K]} s.t. I. ~ D, (c), uf)) E
Annotate D with the reward r/.
Simplified Group Policy Optimization
Obtain g by minimizing the SGPO objective in equation
end for
Output: The final evaluator MM
inefficient due to the inclusion of less informative samples. To address this issue, we propose an uncertainty-aware sampling method choose the hard samples in each iteration. Our method ranks the training imput based on the prediction uncertainty of the evaluator. We use the entropy of multiple judgments over the same imput measure the prediction uncertainty. Given the generated outputs (u5 j=1h. the entropy ej of the training input j is computed as follows:
Cj Pyes log Pyes
Pyes) log(1 Pyes),
K I(u Yes).
Pyes
The value range of judgment y is (Yes, No). T(.) is an indicator function. We select Top-L L samples with the highest entropy values construct the generated dataset. Higher entropy means that the evaluator can't make a solid decision on the training input, suggesting its value to learn. Compared to random sampling in existing self-training methods, our uncertainty-aware sampling focuses the most valuable samples in each iteration, and efficiently expands the training dataset to ensure the diversity and richness of the final dataset.
3.2.3 Iterative Reinforced Self-Training
In the m-th iteration,
the newly generated dataset Dmm from the uncertainty- aware sampling
phase to fine-tune the current evaluator, yielding a new evaluator E gm++, Once the evaluator is improved, new dataset of high-quality samples can be generated once again. The data generation and training steps can be iteratively performed M times, resulting final evaluator gM
Existing reinforced self-training methods [56 15 iteratively train the model. This fine-tuning scheme uses reward values to filter the generated samples and keep positive outputs for training. It has one main limitation: the exclusion of negative outputs. Incorrect outputs can also contain valuable information that helps the model identify error patterns in generations. To address this issue, we introduce Simplified Group Policy Optimization (SGPO) in our iterative reinforced self-training framework.
12
39]
use rejection sampling fine-tuning to
3.2.4 Simplified Group Policy Optimization
The recently proposed group relative policy optimization (GRPO) [36]
compares groups of candidate
responses directly, mining discriminative positive and negative outputs within the same input. The group-level optimization introduces comprehensive contrastive signals for efficient learning. Inspired we design simplified group policy optimization (SGPO) for better group-level contrastive learning. We first design two types of rewards, format reward and judgment reward.
by GRPO and some recent variants 8
23 54
5

---

Format rewards f: We employ format reward model that enforces the model to put its CoT reasoning process between "<think>" and "</think>" tags.
Judgment rewards r: The judgment reward model will score the samples verifying the correctness of their judgments y. We use the rewards in the uncertainty- sampling phase judgment rewards.
SGPO computes the relative quality of K different outputs ((e), vy )= based on the summation of rewards f + We can directly measure the relative quality Aÿ as follows:
A = mean (r)
Note that our SGPO eliminates the standard variance normalization in the original GRPO. This normalization can help to mitigate the influence of extreme reward values [8], but may introduce input-level difficulty bias [23] After our uncertainty-aware sampling, the standard deviations of all output groups fall within a relatively small range (00.23, 0.47] the first iteration), indicating the absence of extreme reward values. In this case, the normalization term assigns greater weight to samples with smaller standard deviations, thereby overemphasizing these less informative samples (low standard deviation means low entropy). Thus, removing normalization can yield positive gains by treating each training input equally.
Then SGPO optimizes the evaluator & by minimizing the following objective:
1 K
(min (ht clip(h. + - (DXx)
LSGPO -
cm (ct (mg)
where and B are hyper-parameters, and hy model, preventing excessive deviation from the initial evaluator, which is estimated as in 10
(c [ Der is adopted to regularize the
4 Experiments
4.1 Datasets
Table Comparison of URST with prompting- and fine-tuned evaluators three test sets, ArTW-ID-trajj, ArTW-OOD-traj and AW-OOD-traj. "Iter." denotes iterative training process. We adopt QwenVL-Max and Qwen2.5VL-3B as the base MLLM for prompting- -basdd and fine-tuning- based methods, respectively. "URST*" use Qwen2VL-2B as the base MLLM. "AutoEval*" uses the last two screenshots input, which is line with the fine-tuning methods.
AITW-ID-traj
AITW-OOD-tre Acc. F1
AW-OOD-traj Acc.
Overall Acc.
Method
Iter.
Acc.
Fl
F1
Fl
Prompting-basee based Methods
AgentTrek [0
84.17
87.58
90.00
70.00
73.54
69.74 79.74
80.56
76.80
Web VeyVoyager 13
85.83
88.74
88.33
65.00
81.61
77.10
84.45
80.54
WebJudge
[52
82.50
85.91
85.71
66.67
77.58
70.59
79.43
77.64
AutoEval
27
X
83.33
86.30
94.16
80.00
80.72
73.94
84.88
79.77
AutoEval
85.00
88.00
94.16
80.00
82.06
75.90
85.95
81.48
Fine-tuning-bet based Methods
SFT
84.17
87.42
94.17
78.79
77.58
71.91
83.59
79.01
STaR
56
81.67
84.51
95.00
81.25
78.03
68.39
83.37
76.60
ReST EM
39
84.17
87.25
91.67
72.22
79.82
70.59
84.02
78.11
*
82.50
86.79
89.17
71.11
81.61
78.53
83.80
81.01
URST
87.50
90.45
94.16
82.05
81.61
79.60
86.39
84.13
We collect and construct training set and three test sets for GUI trajectory evaluation. The training set is built on a subset of Android-in-the- Wild (AITW) datasets. analyzed in (27], about 36%
of the human demonstrations in this dataset are actually incorrect. We randomly sample 1500

---

trajectories from AITW training set, and use Qwen- VL-Max to generate the thoughts and judgments for supervised fine-tuning. In the self-training setting. we only sample 300 trajectories with the thoughts and judgments generated from Qwen-VI.-Man as the initial training set.
AITW-ID-traj and ATTW-OOD-traj are in-domain and out-of-domain test sets built on ATTW dataset. Each of these two test sets contains 120 tasks and was manually annotated in (27]. AITW-ID- traj and
share the task goals, but have different trajectory distributions. Following OS-genesis [I, we also collect some agent-executed trajectories from an online environment, AndroidWorld. After manually annotation and filtering. we keep 223 trajectories and obtain new out-of-domain test set, AW-OOD-traj. The implementation details can be found in Appendix.
4.2 Main Results
To comprehensively assess the advantages of our method, we compare with prompting- -based based works, AgentTrek [50]. WebYoyager [3], Webjudge [52], and AutoEval 277, following the settings in the original papers. Meanwhile, we also implement some representative self-training methods as fine-tuning-based baselines (i.e., STaR (56). ReSTEM (39]), which are originally designed for LLM
post-training instead of trajectory evaluator training. The implementation details of these self-training methods can be found in Appendix.
The results in Table show that our URST consistently achieves the highest Fl score with an average of 84.13%. Among the prompting-basse methods, Web Voyager and AutoEval demonstrate superior performances across all datasets. WebVoyager uses all screenshots of each trajectory as input, achieving better performance than methods with incomplete screenshots, but it suffers from significant token overload. Following the setting of fine-tuning methods, we augment AutoEval with the last two screenshots, resulting in an advanced AutoEval with the best performance among the prompting-basee works. Despite the success of these methods, URST still outperforms them by leveraging -enhanceg-e self-training strategy.
To fully showease its advantages. we compare our URST with supervised fine-tuning (SFT) and various self-training methods. SFT uses the thoughts and judgments generated by an advanced proprietary MLLM (Qwen- VL-Maxx,, which can treated as Distillation. By inheriting the reasoning patterns of a larger base MLLM, the smaller model Qwen2.5VL-333 evaluation capability (81.48% vs. 79.01%). telf-training methods try to use self generated thoughts and judgments to achieve model self-improvement. We find current self-training approaches, STaR and ReSTEM indicate inferior results to SFT with externally generated thoughts, which can be attributed to the less valuable training examples and limited reasoning enhancement ability of rejection fine-tuning. In contrast, URST achieves the highest results across all metrics, highlighting the value of uncertainty-awaa sampling and group-level policy optimization.
demonstrates comparable
4.3 Ablation Studies
We conduct the ablation studics for our URST with Qwen2.5VL-3B to examine the effectiveness of uncertainty- sampling and group relative policy optimization. As shown in Table 2 elim- inating uncertainty-aware sampling and SGPO decreases the overall performance by 6.00% and 9.49%, demonstrating the significant value of these two key components. Without uncertainty- sampling, our method cannot identify the most valuable training data, and is prone to overfitting due to excessive fine-tuning (-3. 73%). SGPO can mitigate this overfitting problem and maintain its generalization on OOD evaluation datasets (0.0% on AITW-OOD-tra and +1.46% on AW-OOD-traj) compared to SFT. On the other hand, without SGPO, uncertainty- aware sampling can maintain the out- of-domain performance (from 78.95% to 78.7797799 79.9 on NUT--OOD-trajj under the same environment, but impair the out-of-domain capability (from 71.91% to 63.22% on AW-OOD-traj) across different environments, compared to SFT. Incorporating advantage normalization consistently decreases the performance, underscoring the importance of accurately measuring sample advantages.
Impact of multiple iterations. As shown in Figure 3 our results show that multiple iterations can consistently enhance the performance of the evaluator, especially on Fl scores. The performance gains across three iterations are 0.65%/1.3 55.5%%.% 1.50%/3.42/ 0.44% /1.2%% in terms of accuracy and Fl score, respectively. Iteration leads to the biggest performance increase, while performing
7

---

Table 2: Ablation studies for different components the proposed URST. We report the Fl score.
Method
AITW-ID-traj AITW-OOD-tra
AW-OOD-traj
Overall
URST
90.45
82.05
79.60
84.13
w/o uncertainty-aware sampling
83.69
78.79
73.37
78.13
w/o SGPO
85.33
78.95
63.22
74.64
w/o normlization removing
89.61
78.95
77.95
82.69
SFT
87.42
78.79
71.91
79.01
more iterations slightly reduces the gain effect due to the relatively low quality of the newly sampled training data, shown in Figure
Accuracy
Random Sampling
F1
0.8
Uncertaint--aware Sampling
85.95
Performance
85
84.45
84.13
Entropy
0.6
82.85
0.4
80
79:43
0.2
77.88
0.0
Initial
Iter.
Iter. 1
Iter.
Iter.
Figure 3: Analysis of the iteration number M. Our URST consistently boosts the performance of the evaluator.
Figure 4: The change of entropy distributions over various iterations. We present entropy dis- tributions of different sampling methods, random and ancertainty-awar sampling.
Impact of the number K of generated outputs per trajectory. Table 3 reveals how training with different numbers of K affects evaluator performance. Generally, increasing the sampling number consistently enhances the evaluator's 's performance across in-domain and out-of out-of-domain datasets. When increasing K from 4 8, F1 score of ArTW-OOD-traij has the largest performance gain of 12.5%, indicating the significant OOD generalization effect from comprehensive group-level contrastive learning. However, further enlarging K 16 led a performance decrease. We speculate that a large sampling number forces the model to generate overly homogeneous responses, leading to overfitting. Overall, K 8 shows the best Fl scores across all trajectory evaluation datasets.
Table 3: Analysis of the sampling number K. We perform the comparison experiments under the setting of one iteration.
Sampling Number K
AITW-ID-traj
AITW-OOD-traj
AW-OOD-traj
Overall
84.51
75.00
72.19
77.55
86.11
87.50
72.41
79.43
16
84.72
87.50
71.76
78.61
Impact the entropy. Our uncertainty-aware sampling uses sampling-bass entropy to measure the uncertainty of the evaluator, which is inspired by self-consistency methods in LLM 63 Besides the proposed metric, we also devise an alternative metric, prediction-basee entropy, which uses greedy decoding to generate single judgment of each training input, and computes the probability entropy of the judgment prediction. In the Table 4 we compare these two metrics with our URST. Sampling-bass based entropy consistently outperforms prediction-bar based entropy across all test sets. Our sampling based entropy uses temperature sampling and entropy of multiple outputs to measure the uncertainty of the evaluator, which could explore various reasoning paths for answer generation and provide a reliable assessment of the model's is confidence. However, prediction-basee greedy decoding to generate a single output for entropy computation, and is prone to suffer from the over-confidence issue of LLM [49], thus making less reliable measurement.
entropy uses

---

Table 4: The comparison of entropy variants in our uncertainty- sampling.
URST
AITW-ID-traj
AITW-OOD-tra)
AW-OOD-traj
Overall
w/ preddition--aast entropy
89.47
73.68
75.68
81.07
sampling-basee entropy
90.45
82.05
79.60
84.13
4.4 The enhancement of GUI Agent
By using an evaluator filter the incorrect trajectories, can improve the quality and reliability of the dataset. We choose two GUI navigation datasets to conduct filtered Behavior Cloning training with trained evaluator. One is AITW dataset [34], which is constructed by human experts. Another is automatically collected OS- Ge-eeeis dataset [41] We first filter the original navigation dataset D by discarding the trajectories identified failures. This filtering process will produce new dataset D by using the trained evaluator. The same number of trajectories from the original and filtered datasets randomly sampled train GUT Agents.
Table Comparisons of GUI navigation performances with original and filtered datasets. The base MLLM is Qwen2.5VL-3B We choose AITW and OS-Genesis as the representatives of manually and automatically generated noisy datasets and evaluate them under the settings of standard and high noise. We report the step success rate, average time cost of filtering samples, and cost per 1M tokens.
AITW
OS-Genesis
Datasets
Time (s)
Cost (S)
Standard
High
Standard
High
Original Doo DO by A
0
63.50
20.88
46.83
46.43
5.99
0.41
64.98
38.93
47.26
47.65
De by SFT
2.17
0
64.32
32.59
47.18
47.22
D by URST
2.17
0
65.02
40.41
47.66
48.26
We conducted some comparison experiments by using two settings, Standard- and High-Noise.
Standard-Noise:
Discarding all incorrect trajectories, and using all steps each trajectory for training GUI agents.
Discarding all incorrect trajectories, and using the last 40% steps in each trajectory for training GUI agents. The underlying rationale is that the majority of incorrect trajectories tend to fail in the last few steps. When tasks are highly challenging or involve unseen environments, the data automatically collected by the agent often contains high levels of noise, meaning that the entire trajectory is wrong from the very beginning.
High-Noise:
Based on the Table 5 can conclude that our URST can consistently outperform other baselines (including AutoEval & and SFT) in filtering noisy datasets. 2) When the noisy level rises, all filters (based on AutoEval SFT, and URST) can achieve greater impact on agent performance, with URST consistently achieving the best result by significant margin. 3) URST demonstrates clear overall advantage, delivering optimal outcomes time cost, expenditure, and performance.
In an online environment, the trajectory evaluator can serve a reward signal to enhance existing GUI agent at inference time, using the Reflexion technique (38). Specif- ically, a GUI agent first attempts a task, and an external evaluator is used to judge whether its attempt was success- ful or not. The agent will be prompted reflect on the fail- ure and retry unsuccessful tasks. We conduct experiments on the most popular online benchmark, AndroidWorld, with M3A agents [33] The best prompting-based method, AutoEval* and our URST are used as a reward signal for eliciting reflection. Table 6 shows that URST achieves superior performance, with 1.8% improvement over AutoEval underscoring its clear advantage in enhancing GUI agents.
Table 6: Comparison of online naviga- tion performance of M3A 3 using the
Reflexion technique 38 with different evaluators. We report the success rate.
Methods
AndoridWorld
M3A
39.7
M3A + AutoEval*
44.8
M3A + -URST
46.6

---

4.5 Further Analysis
The change of entropy distribution over various iterations. Figure 4 we illustrate the entropy distributions of different sampling methods across various iterations. The training data sampled by our URST exhibits significantly higher entropy, indicating its high quality. As the number of iterations increases, the entropy of the training examples gradually decreases, due to the improved capability of the evaluator.
Case Study. To further illustrate the reasoning capabilitiies of the proposed method, we present qualitative case study comparing URST with SFT baseline in trajectory evaluation tasks, as shown in Figure [ Unlike models that passively imitate annotated thoughts, often leading to shallow reasoning or hallucination issues, our URST maintains high reasoning accuracy by generating its own intermediate thoughts and conducting group-level contrastive training. In out-of-domain scenarios, URST demonstrates greater robustness by accurately leveraging contextual cues and interface semantics.
Task Goal: Search "wacbosk
Task Goal:
the
Task Gooli cleck, Tack
by
16:00
16:00
16 00
23:00
ethinis, However, thane indication that the
SFT: ethinko, Howwoerr the lost state stowe the sorted by created dete, whrch the sorting been changed 'Tifee /..//iiioo HureverNNO//rreeee
shate of the that the dlann Sunday, Alanday and "iansda,, which natches the user'sietenn sitteet....//iiie s./tiiiioo <rreee>\E///nseee
cthinko, cFTio,
beer addind the car,. Therafors, the not been completed per the user's intent//ttrlo Hensker-DD-/ceew /crewerr
URS T: itrinko, odded it to the cart as indicated by the quentity selecter showing "1" and the shopping cart lon with a number 3 next to it.//thiko -eraeer/ES//nnieen
URST Ttitioo.. last stute carliees that the rotes
URST Thhiie.... Howexer, the list state the screen shons that only Sunday selected nepeat day, not Monday and Tuesday. </thhii
new sorted by Title" as indicated by the list of notes
arranged alphabetically by their titles <//hunko -eraxer//S//aateee
HeravenNOV/anne
Figure 5: Illustration of some trajectory evaluation tasks with the model-generated reasoning. The left one is an in-domain task, and the others are out-of-domain tasks. URST produces correct judgments through precise reasoning, whereas SFT fails.
5 Conclusion
In this work, we propose an uncertainty-aware reinforced self-training (URST) framework to train lightweight MLLMs as efficient GUI trajectory evaluators. URST integrates uncertainty-awee sampling and simplified group policy optimization (SGPO) to effectively identify informative samples and enhance reasoning capabilities. The approach achieves superior performance on both in-domain and out-of-domail evaluation datasets and enhances downstream GUI agent performance by filtering noisy training data. This work offers a scalable and promising approach for building robust GUI trajectory evaluators, paving the way for more adaptive and capable GUI navigation agents.
Acknowledgements
We thank anonymous reviewers for their constructive feedback. This study is supported by Shen- zhen Science and Technology Program (KQTID202407211222700) National Natural Science Foundation of China (Grant No. U24A20328, 62476071, 62306084, and U23B2051), 1), Shenzhen College Stability Support Plan under Grant GXWD20231118112224222 and Technology Program under Grant ZDSY ZDDSYS20220626092200008 and KJZD20230923 11113399931111111000 China Postdoctoral Science Foundation (Grant No. 2024M764192).
and Shenzhen Science
10

---

References
[1] Saaket Agashe, Jiuzhou Han, Shuyu Gan, Jiachen Yang, Ang Li, and Xin Fric Wang. Agent An open agentic framework that uses computers like a human. In The Thirteenth International Conference on Learning Representations, 2025.
[2] Yuxiang Chai, Siyuan Huang, Yazhe Niu, Han Xiao, Liang Liu, Dingyu Zhang, Peng Gao, Shuai Ren, and Hongsheng Li. AMEX: Android July 2024, arXiv:2407. 17499 [es].
Expo Dataset for Mobile GUI Agents, arXiv preprint,
[3] Gongwei Chen, Leyang Shen, Rui Shao, Xiang Deng, and Liqiang Nie. Lion; Empowering multimodal large language model with dual-level visual knowledge. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 26540-26550, 2024.
[4] Gongwei Chen, Xurui Zhou, Rui Shao, Yibo Lyu, Kaiwen Zhou, Shuai Wang. Wentao Li, Yinchuan Li, Zhongang Qi, and Liqiang Nie. Less more: Empowering gui agent with context-aware simplification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, October 2025.
[5] Wentong Chen, Junbo Cui, Jinyi Hu, Yujia Qin, Junjie Fang, Yue Zhao, Chongyi Wang, Jun Liu, Guirong Chen, Yupeng Huo, Yuan Yao, Yankai Lin, Zhiyuan Liu, and Maosong Sun. GUICourse: From General Vision Language Models Versatile GUT Agents, June 2024, arXiv:2406. 1317 [es].
[6] Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. SeeClick: Hamesssing GUI Grounding for Advanced Visual GUI Agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics. arviv, February 2024. arXiv:2401. 10035 [cs].
[7] Filippos Christianos, Georgios Papoudakis, Matthieu Zimmer. Thomas Coste, Zhihao Wu, Jingxuan Chen, Khyati Khandelwal, James Doran, Xidong Feng. Jiacheng Liu, Zheng Xiong, Yicheng Luo, Jianye Hao, Kun Shao, Haitham Bou-Ammar, and Jun Wang. Pangu-Agee Agent: A Fine- Tunable Generalist Agent with Structured Reasoning. arXiv preprint, December 2023. N..i:22312.188777 [cs].
[8] Xiangxiang Chu, Hailang Huang, Xiao Zhang, Fei Wei, and Yong Wang. GPG: Simple and Strong Reinforcement Learning Baseline for Model Reasoning, April 2025. arXiv:2504.02446 [es].
[9] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang. Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP: Towards General-purpont Vision-Language Models with Instruction Tuning. arXiv preprint, June 2023. arXiv:2305.0653. [cs].
[10] AL., Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang. Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuashu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjic Ji. Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang. Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao. Kang Guan, Kexin Huang, Kuai Yu, Lean Wang. Lecong Zhang. Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang. Meng Li. Miaojun Wang. Mingming Li, Ning Tian, Panpan Huang, Peng Zhang. Qiancheng Wang. Qinyu Chen, Qiushi Du Ruiqi Ge, Ruisong Zhang. Ruizhe Pan, Runji Wang. R. J. Chen, R. I.. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shanqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao. Wenqin Yu. Wentao Zhang. W. L. Xiao, Wei An. Xiaodong Liu, Xiaohan Wang. Xiaokang Chen, Xiaotao Nie Xin Cheng. Xin Liu, Xin Xie. Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q Li. Xiangyue Jin, Xiaojin Shen, Xiaosha Chen. Xiaowen Sun, Xiaoxiang Wang. Xinnan Song. Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang. Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong. Ying Hc, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang. Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang. Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang. Zhewen Hao, Zhicheng Ma. Zhigang Yan. Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song. Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. DeepSeek-RI R1: Incentivizing Reasoning Capability in LLMS via Reinforcement Learning, January 2025. XXiv:2501.12944 [cs].
11

---

[11] Boyu Gou, Ruohan Wang, Boyuan Zheng. Yanan Xie, Cheng Chang. Yiheng Shu, Huan Sun, and Yu Su. Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents. arXiv preprint. October 2024, arXiv:2410.05243 [es].
[12] Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahem, Miaosen Wang. Chenjie Gu. Wolfgang Macherey, Amaud Doucet, Orhan Firat, and Nando de Freitas. Reinforced Self-Training (ReST) for Language Modeling, August arXiv:2308.00 008380888.88888 [cs].
2023,
[13] Hongliang Hc, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models, June 2024. arXiv:2401.13919 [es].
[14] Wenyi Hong, Weihan Wang. Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxuan Zhang, Juanzi Li, Bin Xu, Yuxiao Dong, Ming Ding, and Jie Tang. CogAgent: Visual Language Model for GUI Agents. In IEEE/CVF Conference on Computer Vision and Pottern Recognition. arXiv, December 2023, arXiv:23120089 [es).
[15] Arian Hosseini, Xingdi Yuan, Nikolay Malkin, Aaron Courville, Alessandro Sordoni, and Rishabh Agarwal. V-STaR: Training Verifiers for Self Taught Reasoners, August 2024, MxXiv2442.006457 [es].
[16] Wenjia Jiang, Yangyang Zhuang, Chenxi Song, Xu Yang. and Chi Zhang. AppAgenIX: Evolving GUI Agents as Proficient Smartphone Users. arXiv preprint, March 2025, arXiv:2503.0223 [es].
[17] Wei Li, Bing Hu, Rui Shao, Leyang Shen, and Liqiang Nie. Lion-fs: Fast & slow video-language thinker as online video assistant. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages
2025.
[18] Zaijing Li, Yuquan Xie, Rui Shao, Gongwei Chen, Dongmei Jiang, and Liqiang Nie. Optimus-1: 1: Hybrid multimodal memory empowered agents excel in long-horizon tasks, In Advances in Neural Information Processing Systems, volume 37, pages 49881 1-91199911125 2024.
[19] Zaijing Li, Yuquan Xie, Rui Shao, Gongwei Chen, Dongmei Jiang, and Liqiang Nie. Optimus-2: Multi- modal minecraft agent with goal-observation-action conditioned policy. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 9039-9049, 2025.
[20] Kevin Qinghong Lin, Linjie Li, Difei Gao, Zhengyuan Yang, Shiwei Wu, Zechen Bai, Weixian Lei, Lijuan Wang, and Mike Zheng Shou. ShowUT: One Vision-I V anguuge-AAtton IEEÉ/CVF Conference Computer Vision and Pattern Recognition, November 2024.
Model for GUI Visual Agent. In
[21] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lee. Visual Instruction Tuning, April 2023. 7Xiv:2304.08448 [cs].
[22] Yuqi Liu. Bohao Peng, Zhisheng Zhong, Zihao Yue, Fanbin Lu, Bei Ya, and Jiaya Jia. Seg-zero: Reasoning- chain guided segmentation via cognitive reinforvement. arXiv preprint arXiv:2503.06522 2025.
[23] Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding Ri-Zero-Like Training: A Critical Perspective, March 2025. arXiv:2503.200 207883 [cs].
[24] Quanfeng Lu. Wenqi Shao, Zitao Liu, Fanqing Meng. Boxuan Li. Botong Chen, Siyuan Huang, Kaipeng Zhang, Yu Qiao, and Ping Luo. GUI Odysscy: Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devices. arXiv preprint, June 2024, arXiv:2406.0845 [cs].
[25] Xing Han Lù Amirhossein Kazemnejad, Nicholas Meade, Arkil Patel, Dongchan Shin, Alejandra Zam- brano, Karolina Staúczak, Peter Shaw, Christopher Pal, and Siva Reddy. Agentrewardbench: Evaluating automatic evaluations of web agent trajectories. arXiv preprint arXiv:2504.08944 2025.
[26] Zhengxi Lu, Yuxiang Chai, Yaxuan Guo, Xi Yin, Liang Liu, Hao Wang, Guanjing Xiong, and Hongsheng Li. UI-R1: Enhancing Action Prediction of GUI Agents by Reinforcement Learning, March 2025. arXiv:2503.21620 [cs] version: 1.
[27] Jiayi Pan, Yichi Zhang, Nicholas Tomlin, Yifei Zhou, Sergey Levine, and Alane Suhr. Autonomous Evaluation and Refinement of Digital Agents, In First Conference on Language Modeling, arXiv, April 2024. arXiv:2404.06477. [cs].
[28] Ajay Patel, Markus Hofmarcher, Claudiu Leoveanu- Condrei, Marius Constantin Dinu, Chris Callison- Burch, and Sepp Hochreiter. Large Language Models Can Self- Improve At Web Agent Tasks. arXiv preprint, May 2024. arXiv:2405.20309 [cs].
12

---

[29] Yingzhe Peng, Gongrui Zhang, Miaosen Zhang, Zhiyuan You. Jie Liu, Qipeng Zhu, Kai Yang. Xingzhong Xu, Xin Geng, and Xu Yang. Lmm-rl: Empowering 3b Imms with strong reasoning abilities through two-stage rule-based rl. arXiv preprint arXiv:2503.0755 2025.
[30] Pranav Putta, Edmund Mills, Naman Garg, Sumeet Motwani, Elan Markowitz, Chelsea Finn, and Rafael Rafailov. Agent Q: Advanced Reasoning and Learning for Autonomous Al Agents. arXiv preprint.
[31] Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, Wanjun Zhong, Kuanye Li, Jiale Yang. Yu Miao, Woyu Lin, Longxiang Liu, Xu Jiang. Qianli Ma, Jingyu Li, Xiaojun Xiao, Kai Cai, Chuang Li, Yaowei Zheng, Chaolin Jin, Chen Li, Xiao Zhou, Minchao Wang, Haoli Chen, Zhaojian Li, Haihua Yang. Haifeng Liu, Feng Lin, Tao Peng. Xin Liu, and Guang Shi. UI-TARS: Pioneering Automated GUT Interaction with Native Agents. arXiv preprint, January 2025. Xiv::2501 122366 [cs].
[32] Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning. and Chelsea Finn. Direct Preference Optimization: Your Language Model is Secretly Reward Model, July 2024. arXiv:2305. NDw::305.182900 [es].
[33] Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajabe Daniel Toyama, Robert Berry, Divya Tyamagundlu, Timothy Lillicrap, and Oriana Riva. AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents. arXiv preprint, June 2024, arXiv:2405.1 14573 [cs].
[34] Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. Android in the Wild: A Large-Scale Dataset for Android Device Control. In Advances in Neural Information Processing Systems, 2023, July 2023.
[35] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. pages 1-12. 2017, arXiv: 1707.06347.
[36] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang. Y. K. Li, Y. Wu, and Daya Guo. MeetSeekMath; Pushing the Limits of Mathematical Reasoning in Open Language Models, April 2024, arXiv:2402.03300 [cs].
[37] Leyang Shen, Gongwei Chen, Rui Shao, Weili Guan, and I Liqiang Nie. Mome: Mixture of multimodal experts for generalist multimodal large language models. In Advances in Neural Information Processing Systems, volume 37, pages 42048-48 420400 2024.
[38] Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language Agents with Verbal Reinforcement Learning, October 2023. arXiv:2303.1 1366 [cs].
[39] Avi Singh, John D. Co-Reyes. Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Peter Liu, James Harrison, Jaehoon Lee, Kelvin Xu, Aaron Parisi, Abhishek Kumar, Alex Alemi, Alex Rizkowsky, Azade Nova, Ben Adlam, Bernd Bohnet, Gamalcldin Elsayed, Hanic Sedghi, Igor Mordatch, Isabelle Simpson, Izzeddin Gur, Jasper Snoek, Jeffrey Pennington, Jiri Hron, Kathleen Kencaly, Kevin Swersky, Kshiteej Mahajan, Laura Culp, Lechao Xiao, Maxwell L. Bileschi, Noah Constant, Roman Novak, Rosanne Liu, Tris Warkentin, Yundi Qian, Yamini Bansal, Ethan Dyer. Behnam Neyshabur, Jascha Sohl-Dickstein, and Noah Fiedel. Beyond Human Data: Scaling Trainirain for Problem-Solving with Language Models, April 2024. urXiv:2312.063 [es].
[40] Hongjin SU, Ruoxi Sun. Jinsung Yoon, Pengcheng Yin. Tao Yu, and Sercan O Arik. Learn-by-interact, A data-centric framework for self-adaptive agents in realistic environments. In The Thirteenth International Conference on Learning Representations. 2025.
[41] Qiushi Sun, Kanzhi Cheng, Zichen Ding, Chuanyang Jin, Yian Wang, Fangzhi Xu, Zhenyu Wu, Chengyou Jia, Liheng Chen, Zhoumianze Liu, Ben Kao, Guohao Li, Junxian Hc, Yu Qiao, and Zhiyong Wu. OS- Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesis. arXiv preprint, December 2024, 2X1v:2412.19923 [es].
[42] Junyang Wang, Haiyang Xu, Jiabo Ye, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. Mobile-Agent: Autonomous April 2024. arXiv:2401.16155 [es].
Mobile Device Agent with Visual Perception. arXiv preprint.
[43] Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge. Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2- WW:: Enhancing UssionLLaguuag Model's Perception of the World at Any Resolution. arXiv preprint. September 2024.
13

---

[44] Xiaoqiang Wang and Bang Liu. OSCAR: Operating system control via state-aware reasoning and re- planning. In The Thirleenth International Conference on Learning Representations, 2025.
[45] Zihao Wang, Shaofei Cai, Anji Liu, Yonggang Jin, Jinbing Hou, Bowei Zhang. Haowei Lin, Zhaofeng He, Zilong Zheng, Yaodong Yang. Xiaojian Ma. and Yitao Liang. JARVIS-I: Open-World Multi-task Agents with Memory-Augmentedd Multimodal Language Models, November 2023. arXiv:231 [es].
[46] Zora Zhiruo Wang, Jiayuan Mao, Daniel Fried, and Graham Neubig. Agent Workflow Memory. arXiv preprint, September 2024. arXiv:2409.074 .07229 [cs].
[47] Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang. Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, and Yu Qiao. OS-ATLAS: A Foundation Action Model for Generalist GUI Agents. arXiv preprint, October 2024. arXiv:2410.23218 [cs].
[48] Bin Xie, Rui Shao, Gongwei Chen, Kaiwen Zhou, Yinchuan Li, Jie Liu, Min Zhang, and Liqiang Nie. Gui-explorer: Autonomous exploration and mining of transition-aware knowledge for gui agent. In Anmual Meeting of the Association for Computational Linguistics (ACL), 2025.
[49] Miao Xiong. Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. Can Ilms express their uncertainty? an empirical evaluation of confidence elicitation in llms. arXiv preprint urXiv:2306.13066
2023.
[50] Yiheng Xu, Dunjie Lu. Zhennan Shen, Junli Wang, Zekun Wang, Yuchen Mao, Caiming Xiong, and Tao Yu. AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorials. In International Conference on Learning Representations. arXiv. March 2025. arXiv:2412.09605 [cs].
[51] Tianci Xue, Weijian Qi. Tianneng Shi. Chan Hee Song, Boyu Gou, Dawn Song, Huan Sun, and Yu Su. An illusion of progress? assessing the current state of web agents. arXiv preprint arXiv:2504.01. 2025.
[52] Tianci Xue, Weijian Qi, Tianneng Shi, Chan Hee Song, Boyu Gou, Dawn Song, Huan Sun, and Yu Su. An Illusion of Progress? Assessing the Current State of Web Agents, April 2025. arXiv:2504.0135 [es].
[53] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing Reasoning and Acting in Language Models, October 2022.
[54] Qiying Yu, Zheng Zhang, Ruofci Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin. Zhiqi Lin, Bole Ma, Guangming Sheng. Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu. Weinan Dai, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu. Wei- Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. DAPO; An Open-Source LLM Reinforcement Learning System at Scale, March 2025. arXiv:2503. 1120003.1447 14476 [cs].
[55] Tianyu Yu. Yuan Yao. Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu. Hai- Tao Zheng, Maosong Sun, and Tat-Seng Chua. RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback, March 2024. arXiv:2312.0084. [cs].
[56] Eric Zelikman. Yuhuai Wu. Jesse Mu, and Noah D. Goodman. STaR: Bootstrapping Reasoning With Reasoning, May 2022. arXiv:2203. 14465 [cs].
[57] Chi Zhang, Zhao Yang. Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang. Bin Fu, and Gang Yu. AppAgent: Multimodal Agents as Smartphone Users. arXiv preprint, December 2023. txXiv:2312.13777 [cs].
[58] Jingyi Zhang, Jiaxing Huang, Huanjin Yao, Shunyu Liu, Xikun Zhang. Shijian Lu, and Dacheng Tao. RI-VL: Learning Reason with Multimodal Large Language Models via Stee-wise Group Relative Policy Optimization, March 2025. arXiv:2503. 129337333119 [cs].
[59] Jiwen Zhang, Jihao Wu, Yihua Teng, Minghui Liao, Nuo Xu, Xiao Xiao, Zhongyu Wei, and Duyu Tang. Android in the Zoo: Ch Action-Thoughe for GUI Agents. Findings of the Association for Computational Linguistics: EMNLP 2024. arXiv, July 2024. arXiv:2403.022 02713 [cs].
[60] Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gis-oV((sion) is a Generalist Web Agent, if Grounded. arXiv preprint, March 2024. arXiv:2401.0166 [es].
[61] Longtao Zheng, Rundong Wang. Xinrun Wang. and Bo An. Synapse: Trajectory-as-Exeee Prompting with Memory for Computer Control, January 2024. mxXiv:2366078666 [cs].
[62] Ruiwen Zhou, Yingxuan Yang, Muning Wen, Ying Wen, Wenhao Wang. Chunling Xi, Guoqiang Xu, Yong Yu, and Weinan Zhang. TRAD: Enhancing LLM Agents with Step-Wise Thought Retrieval and Aligned Decision, March 2024. XXXi:244330022) [cs].
14

---

[63] Xin Zhou, Yiwen Guo. Ruotian Ma, Tao Gui, Qi Zhang, and Xuanjing Huang, Self-Consistency of the Internal Reward Models Improves Self-Rewardin Language Models, February 2025. arXiv:2502.08922 [es].
15

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes].
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
16

---

Justification:
The formulas are numbered and refes-eeferenced correctly in this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We fully and clearly describe the training framework in the main body, and provide the training details and model configurations in the Appendix.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer: [Yes]
Justification: We use publicly available datasets for training and a combination of publicly available and self-collected datasets for evaluation. We add the code and data supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https for more details.
/ //nips.c//pppp
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide detailed training and test configuration details in the Appendix,
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Due the high computational cost of large models, we perform only three independent runs with different random seeds for SFT and URST. The comparison is presented Table I0
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
18

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
This information is provided in Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpp:://nerripp..coo /ttp://sihrr/////pppppooooooa/aa
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeulPS code of Etics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The societal impacts are discussed in Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
19

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The capability of the model that will be releasing is limited simple tasks, and therefore does not have a high risk for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have properly cited the assets that we are using.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
20

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com/dataseta
datasets.papersstt
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [Yes]
Justification: Our code and relevant assets including self-collected data are provided in the supplemental material.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This research does not involve crowdsourcing human subjects. Annotations trajectories in AW-OOD-traj are carried out by authors alone.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification:
This research does not involve crowdsourcing or human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
21

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: We describe the methods built on LLMs in the main paper, such as sections method, experiments.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https://neurd) auttep//eeerrrppcccc///oeerrrerp /c/////ooeeeeeeee//22////.. for what should or should not be described.
22

---

A Trajectory Evaluation Datasets
The AITW-ID-tra) and AITW-OOD-traj evaluation datasets are directly adopted from the setup intro- duced in AvtoEval 27. Both datasets are based on the mndroid-in-the-Wild (AITW) 34 benchmark, which consists of human-executed trajectories collected in Android emulator environments.
The AITW-ID-tra traj test set includes 120 trajectories randomly sampled from the General, WebShop- ping, and GoogleApps subsets of the AITW training set (40 episodes from each). These trajectories were originally collected via human demonstrations on an Android emulator, where annotators manu- ally executed task flows. However, the data contains significant noise. Common failure cases include early termination, completing the wrong task, or making mistakes in task parameters. As result, the correctness of these episodes is not guaranteed. After manual verification, only approximately 64.2% of the selected episodes in AITW-ID-traj were found to be successful.
The AITW-OOD-traj test shares the same task goals as AITW-ID-traj but differs in the source of trajectories. In this case, the trajectories were generated by CogAgent (14), an autonomous agent interacting with an Android emulator. The emulator was configured using Android Studio's built-in environment to simulate a Pixel 4 device with API level 33. Manual annotation revealed that only 14.2% of the generated trajectories successfully completed the assigned tasks.
Following OS-Genesis [41, we adopt a fully automated data collection process to construct the AW-OOD-traj test set. This set is built using automatically generated task goals and agent-executed trajectories within the AndroidWorld (33) environment. We employ reverse task synthesis approach to create a diverse range of high-level task goals that align with dynamic environments. The agent from Agent Q 30] is then used to interact with the environment based on these task goals, generating approximately 500 new trajectories. We filter out low-quality trajectories that are too short or based on invalid task goals. After filtering, each remaining trajectory is manually annotated using major voting scheme, resulting in the final AW-OOD-tra test set containing 223 trajectories. 43.0% of trajectories in this test set are annotated as successful,
Table 7: The statistics of three trajectory evaluation datasets.
Datasets
#Success
#Total
Success Rate
AITW-ID-traj
77
120
64. 14%
AITW-OOD-traj
17
120
14.17%
AW-OOD-traj
96
223
43.05%
B Experimental Details
B.1 Implementation Details
All experiments were conducted on 4 NVIDIA A100 40GB GPUs The key hyperparameters used in our experiments are summarized in Table The model is trained for epochs using 300 samples annotated with Qwen-VL-Max VL-Max during the initialization SFT stage. For SRPO training, each iteration involves 4 epochs of training on 400 samples sampled via URST. Consequently, after 3 iterations, the total number of training samples amounts to 1,500. Both Initialization and subsequent SGPO training are conducted with full-parameter fine-tuning. each Iteration, the learning rate is warmed up linearly from O to le-6 across 5 global steps and then reduced to minimum of 0 using cosine decay. We adopt a 3 value of 0.001, which balances the reward signal and divergence constraint in the policy update. To manage training efficiency and computational cost, the maximum pixel limit for each visual input was set at 802,816. an input image exceeds this limit, it is cropped and resized while preserving the original aspect ratio. To further enhance memory efficiency and scalability, all experiments are conducted using DeepSpeed's 's Zero-3 optimization stage, and flash attention is employed to accelerate training. During both the uncertainty apply the temperature set to 1.0, top-k sampling with i = 50, and nucleus (top-p) sampling with p 0.9 ensure the diversity of the outputs.
sampling and SGPO training,
An example of the prompt format used during training and inference illustrated in Figure 9 Specifically, we incorporate the last two states of the trajectory into the input of the MLLM. This
23

---

Table 8: Hyperparameter settings used in the experiments.
Hyperparameter
Value
SFT training epoch at initialization stage
SRPO training epoch
sample size per iteration
400
B
0.001
learning rate
1e-6
warmup ratio
0.05
max pixels
802,816
device train batch size DeepSpeed optimization stage
Zero-3
temperature
top-k top-p
50
0.9
design enriches the prompt with critical state transition information while avoiding excessive context length and token overhead. Furthermore, the action history is refined to explicitly include both click coordinates and scroll directions, offering more fine-grained representation of user interactions. These enhancements provide the model with richer transition cues, thereby improving its ability to make informed and accurate predictions.
During the data filtering process for Behavior Cloning training, we leverage our trained evaluator to assess and filter the original navigation dataset D resulting cleaned dataset D The retention rate of the trajectories is reported in Table 9 the subsequent training the GUI navigation agent, employ SFT with Qwen2.5-VE V---3B for 2 epochs. The prompt templates used during training and inference for the AITW and OS-Genesis datasets are illustrated in Figure 15 and Figure T6
Table 9: The retention rate of trajectories in two GUI navigation datasets after being filtered by trained evaluator.
AITW
OS-Genesis
General
WebShop
GoogleApps
Total
81.25%
40.20%
81.85%
67.77%
45.14%
B,2 The compared methods
To comprehensively assess the advantages of our method, we compare it with prompting-bai based works, AgentTrek [50], Web Voyager [33 Webjudge [51], and AutoEval (27), following the settings in the original papers. The prompt templates used for these prompting- baasd based evaluation methods can be found in Figure 10 an advanced version of AutoEval by providing the last prompt template used for A illustrated in Figure [4 Meanwhile, we also implement some representative self-training methods as fine-tunigg- based basclines, which are originally designed for LLM post-trainis training instead of trajectory evaluator training. Here, we briefly introduce the compared self-training methods.
22
13 Following the fine-tuning methods, we also implement screenshots, termed AutoEval The
Self Self-Taught Reasoner (STaR) [56] employed greedy decoding instead of temperature sampling for data generation, which is restricted to one model- -eeerr-eee response per input during data collection. Here, adopt temperature sampling generate responses and choose one of the correct responses as training data.
EM) 39 is a simplified and advanced version of ReST [2], which decouples data collection and policy optimization in the reinforcement learning pipeline. ReST iteratively conducts self-generated data collec- tion and swardd-eighted fine-tuning to achieve self-improvement of the policy. However, it has one main restriction: requiring human-generated outputs, which are unavailable in
Reinforced Self-Training with

---

our experiments. The main differences of ReSTEM from ReST are the exclusion of human- generated outputs and the iteratively fine-tuning on the base model instead of the model from the previous iteration. Here, we set the cut-off threshold for the maximum number of responses per input to 4.
To ensure fair comparison, implement all the above methods dataset matching the size of the final dataset (1500 training inputs) in our URST method, which includes data from the initial phase and all iterative sampling steps.
C
More Analyses
Iter.
1.9
Dler.
0.4
Iter. 2
188
reward
roward std
0.3
1.7
1.6
0.2
1.5
0.1
20
40
60
80
20
40
80
step
102.5
0.7
100.0
0.6
completion length
97.5
grd grdd nom
95.0
0.5
0.4
90.0
87.5
85.0
40
60
step
Figure 6: Visualization of the training process various iterations. We present the changes in key training metrics, including total reward, reward standard deviation, response length, and gradient norm with respect training steps.
C.1 The training dynamics of URST
To provide more details, we report key metrics from our training process across multiple iterations. as shown in Figure 6
Across all three training iterations (Iteration 0, 1, and 2), total reward shows gradual upward trend as training progresses. The improvement from Iteration 0 to Iteration is particularly notable, while subsequent iterations display more modest and convergent changes. This pattern suggests that the model adapts and learns more effectively during the earlier stages of training, and as training progresses, the rate of improvement stabilizes.
Both the reward standard deviation and gradient norm exhibit clear downward trend over training steps and across iterations. This suggests that the model becomes more stable and less volatile as
25

---

training proceeds. As the training continues, these metrics converge, reflecting increased confidence and consistency in the model's behavior.
Response length remains relatively stable across all iterations, without any significant increasing or setup employs cold-start initialization. Consequently, the
decreasing trend. This is expected, reasoning patterns, especially regarding output length and format, exhibit relatively limited variation during the training process.
C.2 Statistical significance and stability
In the Table 10 we present the statistical results of SFT and URST with three different random seeds. URST significantly outperforms SFT across all three test sets. These improvements are statistically meaningful given the non-overlapping confidence intervals, particularly for the score, where URST shows substantial gain (e.g., +4.82 on AITW-OOD-tra). Moreover, the smaller standard deviations in most metrics indicate that URST produces more stable results across different runs. Overall, these findings demonstrate that URST not only improves performance but also enhances result consistency.
Table 10: The statistical results SFT and URST with three different random seeds
SFT
URST
Methods
Acc.
F1
Acc.
F1
AITW-ID-traj
83.61
87.5
87.78
90.64
AITW-OOD-traj
9..61#1.42
78.71#3.00
94.7211077
83.53+2.09
AW-OOD-tt
1..1
71.30#1.17
83.40+1.227
82.14
Overall
83.08:10.71
78.62::.771
87.4740.81
85.59=1.05
C.3
The evaluation of evaluator
As shown in Figure 3 we present the overall performance across multiple iterations. It can be observed that all iterations consistently enhance the evaluator's performance. However, performing additional iterations yields diminishing gains due to the relatively low quality of the newly sampled training data. the Table IT we also provide the detailed results of multiple iterations. The performance of URST on AITW-ID and AW-OOD-HE However, its performance on AITW-OOD-traj peaks iteration and exhibits fluctuations (primarily in the F1 score) in subsequent iterations. We attribute this behavior to the severe class imbalance in AITW-OOD O-OD-trraa (success: 14%, fail: 86%), as shown in Table The F1 score is highly sensitive to even small performance variations in the minority class.
gradually increases with iterative training.
Table 11: The detailed results of URST across different iterations.
AITW-ID
tr-jj
AITW-OOD traj
AW-OOD-tre -raj
Overall
Phase
Acc.
F1
Acc.
F1
Acc.
F1
Acc.
F1
Initial
80.00
82.85
95.00
81.25
79.82
73.05
83.80
77.88
Iter. 0
83.33
86.11
96.67
87.50
78.48
72.41
84.45
79.43
Iter.
86.67
89.47
93.33
78.95
81.61
78.31
85.95
82.85
Iter. 2
87.50
90.45
94.16
82.05
81.62
79.60
86.39
84.13
C.4 Generalization beyond mobile GUI tasks
Recently proposed AgentRewardBench 25 is the first benchmark to assess the effectiveness of MLLM-based evaluators for evaluating web agents. It contains 1302 web trajectories across benchmarks and 4 agents. Due the lack of training data, we split AgentRewardBendd into two sets, 1047 trajectories for the training set and 255 for the test set, and ensure a similar class distribution, as shown in Table [2
We evaluate the prompting-based (WehJudge, AutoEval*) and fine-tuning- (SFT, URST) methods on this web benchmark, and report the results in the Table [3 URST is trained under the
26

---

Table 12: The statistics trajectory evalua- tion datasets constructed from AgentReward-
Table 13: Comparison of our URST with prompting-base based and fine-tuned evaluators on AgentRewardBench 25].
Bench [25].
Datasets
#Success
#Total
Success Rate
AgenteewardBench
Methods
Acc.
F1
Training
1047
293
27.99%
Test
255
63
24.71 %
WebJudge
83.07
68.15
AutoEval*
85.10
74.67
SFT
84.31
74.36
URST
87.06
77.85
setting of initial-300 samples, iter. 0-200 samples, iter. 1-200samples, and SFT is trained with the same total number of samples. Our URST achieves the highest performance on AgentReward reaching 87.06% accuracy and 77.85% Fl score, substantially outperforming the other baselines. These comparisons underscore the effectiveness and generalization of URST.
C.5
Case study
Here we present more qualitative cases comparing URST with the SFT baseline in trajectory evalua- tion tasks, as shown in Figure 7
Example (left). The SFT model incorrectly concludes that the task is completed, reasoning that the appearance of protection plan pop-up "typically appears after adding an item to the cart." However, this might be a case of over-reliance on common patterns without a comprehensive check. In contrast, URST correctly identifies that the product has not actually been added to the cart, noting that the last screen only shows the protection plan dialog "instead of the actual product being added the cart." This shows URST's ability to discern subtle differences in interface feedback.
Example (middle). The SFT model falsely claims that "the option to display folders first is not selected," resulting in a wrong NO judgment. While URST provides a more accurate analysis. grounds reasoning directly the UI evidence, avoiding unsupported assumptions. This case highlights URST's advantage in mitigating hallucinations.
Example 2 (right). The SFT model incorrectly concludes that the task is incomplete due to the "lack of explicit confirmation" in the last two screen states. This reflects a weakness in tracking fine-grand finengraieed transitions and recognizing subtle but semantically meaningful changes the GUI elements. In contrast, URST successfully identifies this transition by integrating an action history that includes the precise click location, demonstrating a strong sensitivity to GUI elements and their associated semantic cues.
To further analyze the limitations of our proposed method, present several representative failure cases observed during trajectory evaluation, as illustrated in Figure 8
Example (left): Failure Due Insufficient Observation. The evaluator is unable to determine whether the correct item has been deleted, as the last two visual states and action history do not provide sufficient evidence verify the deletion of the specific entry "New Jacket". While deletion action is present, there is no direct indication that it affected the correct item in the last two states. This case exposes the trade-off between limited observation and reasoning accuracy: relying on only the last two states can reduce computational overhead, but sometimes omits essential information for task validation. Finding a balance between context efficiency and task-specific observability remains an open challenge.
Example 2 (middle): Failure Due to GUI elements Misinterpretation. Although the evaluator correctly identifies the selection of the paint bucket tool and color, it falsely concludes the task complete. The confusion arises from an orange cursor widget visible at the bottom of the screen, which the model incorrectly interprets as the filled canvas area. This case highlights a common failure mode where the model lacks domain-specific visual understanding, confusing GUI indicators (like cursors or tool previews) with actual content
27

---

Search "razer rege" on neangg com, select the First entry, and it the cort.
Tark Gooll Clear the shapping cort on noecgg.com
Task Gool: Use Manier to sont the filer in the /astorege/een httpa//eeeeeeee/////eeeeet optiona to dipply folcirs first and includee doffiles
by name and arable the
Tesk Goal: Work the Avecedo Toust with Egg inccipe as favernt the Brnccoll opo.
SF T. -hhroo. The les't screer shows pop-up for aiding other the car This suppestt that the sjent has aucoesefully odded ../thani onperrotE///rreee
SFT: strinlo.. However, The option disploy folders First (FFlldrr first in selected, which meats not beer enabled yet./thwio
SFT: <thitir., However, there confimection the
protection shich typically
of the showing the heart Serg filled any irdiceriar... ../thhiio <ssseenDD//rrneee
URS T: <hiio.... However, the last soneen showe
protection plan offer insteod the actuol product baing
URST T. athilo.. corfiers that the files are sorted by and that the for displyyin folders first und including dottiles are exabred. Therefore...
ethisko., URST..bnsk.
The action histery shows that whrch Typkally indicates marking recipe es fovorite... <rtrinko
odded the cart. Therefore, the tesk has not been
completed os interded, </triiio oeneeenDD//ssseer
seccos/fuly///iiie
Figure 7: Illustration of some trajectory evaluation tasks with the produces correct judgments through precise reasoning, whereas SFT fails.
reasoning. URST
(like painted canvas). This failure case underscores the importance of equipping the model with more comprehensive domain-specific knowledge about apps and a better understanding of GUI elements.
Example 3 (right): Failure Due to Instruction Misunderstandinng While the evaluator correctly identifies that the default search engine has been changed to "Yahoo!", it overlooks the second part of the instruction, "and then return to the main settings menu." The final screen still shows the search engine selection page, indicating that the task only partially complete. This failure case suggests that the evaluator sometimes lacks robust multi-step instruction comprehension. It underscores the need for improved task decomposition and intent modeling in trajectory evaluation settings.
Took Gonl Pee Experts, the ertry iitled NNov Jocket with anamcunt SLB USD
Tosk Gool: Sieple Ornw the buckett hool Gooli opp to Fill an carval with the colar seftinge manu
URS kistery shon dence "Now Zocket" being
URST ettijo. scrice hastory shows ircluding opp indeed dooss there
URST mmiio. action hastrry shows... Holy
correspond opering
The stare the confires sear ck engine sad been succeesfully chenged to Vihool oppaara to be completed os meseeii//tteee wwwker
delited Therefane //iiile
corvis. which mutickes the oster specifed the user. </theko
Figure 8: Illustration of failure cases observed in trajectory evaluation.
D Limitations and Potential Social Impacts
There are multiple potential limitations in this work. First, we only fine-tuned one family of models, Qwen2-VL./ Qwen2.5-VIL VIL; however, while the absolute performance values would change, we expect our relative findings to be consistent across model families. Additionally, while our results suggest
28

---

that CoT reasoning is crucial to the performance improvement of evaluator training with RL, we have extensively explored the effects of different prompting techniques in this work, which will be an interesting future direction. Further, due to computational limitations, our evaluator is trained only with tasks from AITW instead of all possible tasks on the mobile or computer device. Our design of the URST algorithm aims for maximal implementation efficiency and effectiveness, so we hope that our approach to serve as a base algorithm for future research to build on, including algorithmic research well as expanding the space of tasks.
The URST framework enhances data quality through rigorous trajectory filtering, leading to agents trained on cleaner, nigher-quality, examples, thus minimizing erroneous or harmful behaviors during deployment. The framework relies on lightweight MLLMs and avoids dependence on proprietary models, potentially democratizing access to powerful trajectory evaluation tools for institutions or developers with limited resources. However, we also acknowledge that this work has some potential negative impacts. If the initial training data contains biases, the self-improvement loop could reinforce these issues, potentially resulting unfair incorrect judgments, especially in edge cases unfamiliar interface designs. Advanced GUI agents trained with URST could potentially be used to manipulate user interfaces in unethical ways deployed without proper safeguards.
You expert evaluating perforeance an navigation agent. agent designed help complete task. screan, your goal is whether agent successfully completed task not.
navigate
intent, agent history,
two states
thinks enclused within ethinks </thinko and </answer> respectively, i.c., <chinioreasoning
reasoning
herec/ www.tioorsseerraaaaaaaaa YES NOc/answer>
Prompt
Intent: Access details
Casera app explore the available settings adjusting photo
Action History:
1: Open app: 'Carera'
2: Tap point((y, x]): [0.8025, 0.9116]
3: scroll down
up
5: Top point((y, x]): [0.0796, 0.1148]
Tap point(ly. x]]: [0.8029,
point(ly, x1): (0.1468, 6.2893] Status: "successful"
The second last state and the last state of the screen are shown in the Images.
Figure 9: Prompt example for fine-tuning-bast based methods.
29

---

expert in evaluating the performance of android navigation agent. agent is decigned goal, the agent's trajectory, your decide shather Systen Proupt
navigate device agent's execution is successful
"Evaluation Criteria"
shothen the agent's trajectory is effective and conresponding to the goal
"Instructions" 1. Revieu the agent's actions step by step.
2. if the agent is stack in the very first login stage, which reans it fails to log into target website at the beginning, that's -Fallure.
Dutornine agent has achieved the task goal based on the trajectory. task can be considered successful most trafectory effective.
agent
can't
Finishing
doing repeated
actions may offective
fai
task should reganded
almost
is sturk in the toop at the early stage of the task, which means they don't even get clase to the goal before they get stuck in the loop, that's failure. for exaeple, the agent begin to stuck before third step.
5. If the
6. when the task Is to change the google account password, it can't be regarded as successful when agent finish at trying to
click "ranage your account".
7. if there are over 8 correct action in the trajectory, it can be regard as a successful agent.
8. final saving action is not must. the task is successful if the agent does nost things right and just forget to save the
change at last.
9, if the original task has subtasks, the agent only complete one of then, that's still success, e.g- the task is to update
nase and birtheay, but agent only update nare, tsat's fine.
18. the task is to post a review, the agent can be considered successful when it finish writing the review and neach the step post it, don't have to click the post button,
11. we
have printer, some printing related task can considered successful the agent reach the step
print butzon.
12. task is finished the initial state agent nothing because it, it should regarded as
"IMPORTANT"
response should be contain:
cyour thoughts and reasoning process> Status: "success" or "failure"
Thoughts:
User Prompt
The goal of the task: (task_pool)
Trajectory: (history, actians)
The last state of the screen is shown in the inage.
Figure 10: Prompt template for AgentTrek.
Systen Prorpt
As an evaluator, you xill be presented with three prinary cosponents to assist you in your role:
1. Task Instruction: This is a clean and specific directive provided in natural language, detailing the online activity to be corried out.
visual representation the screen shouing result or intersediate state perforning task. It serves as visual proof the actions taken in response the instruction.
Screenshots:
Result Respanse: This textual response obtained after the execution instruction.
textual result in respance
You D0 NOT NEED to interact with screens perfor actions.
inforsation presented the scroonshet comparing the instructions. assunctions conduct thorough assecsment of the task instruction against the outcoes depicted in the Your primary responsibility
SHOULD
screenshot and in the response, evaluating whether the actions taken align with the given instructions.
KOTE that the instruction ray Involve sone than one task, for exarple, locating the garage and surearizing the reviow. Failing to cosplete either task, such not providing sumsary, should be considered unsuccessful.
NOTE that the screenshat is authentic, but the response proviced by LiM is generated at the end of state, and there ray be discrepancies les between the test and the scrweshots.
-- Note the difrrncc::: 1) besutt response say contradict the ccrvenchat, then the content of the screenshot prevails, 2) The content in the Result response is not nentioned on the screenshot, choose to believe the content.
You should elaborate on how you arrived at your final evaluation and then provide definitive verdict on whether the task has "FAILURE".
boen successfully accomplisted, either
User Prompt TASK: [task_goal)
Result Response: (last_action)
(ful_oo__ll_statee)) screenshots
Figure 11: Prompt template for Web Voyager
30

---

Key Point Identification Proupt
You are expert tasked with analyring given task to identify the key points explicitly stated in the task description.
-sdrjectivens; Carafully analyse the task description and extract the critical eleeents explicitly sentioned in the task for schieving its goal.
**Instrcction=*:
1. Read the task descristion carefully.
2. Identify and extract *Key pointsre directly stated in the task description.
*lkey paint is critical elesent candition, or step explicitly iondd task description.
Do not infer or add any unstated elenents.
Words such "best," "highest," "cheapest," "latest," "sost recent," "lowest, "closest," "Highest-rated," "langast, sont. function(e-go) key point
"newest" go through
"Filter by highest").
**feepaadd withee;
**Kyy Points*:; numbered list of the explicit points for completing this task, line, without explanatiios additional details.
Task: goal)
Key Screenshot Identification Prompt
You are an expert evalustor tasked kith detensining whether image contains inforsation about the necessary steps to corplete a task.
""bjjective*:; Analyze the provided image and decide if it shows essential steps or evidence required for completing the task. Use your reasoning to explain your decision before assigning a score.
*Instructions""
Provide detailed description of irage, including its contents, visible elerents, text (IF any), and any notable features.
Carefully exanine the irage and evaluste whether contains neceszary steps or evidence crucial to task completion: completion,
could filters, step-by-stes instructions.
actions, progress indicators,
usage, applied
Does information indispensbbee for understanding ensuring task success?
show actions, progress indicators, critical informstion directly related completing the task?
IIf contains partial
relevant information, consider its usefulness rather than disnissing it outnigght
3. Provide your response in the folloding format:
**Reasoning**; Explain your thought process and observations. Mention specific clements in the image that indicate necessary steps, evidence, or lack thereof.
*SScong""; Assign a score based on the reasoning, using the folloxing scale:
-eage;; The irage does not contain any necessary steps or relevant informtion.
<<ge*; The irage contains ninisal or aubiguous inforration, unlikely to be essential.
imape inc ludes, soee relevant steps hints hut' lacks clarity or completeness.
-nge:: irage contains irportant steps or evidence
highly relevant but not fully comprchensive.
-"g""; image clearly displays recessary steps evidence crucial completing the task.
Respand
"*Roasnning" (Your explanation]
2. **Srore**: [1-5]
Task**; (task_
"Key Points for Task completion*:: (key points)
The snapshot of the screen is shown in the ivage.
Outcome Judgement Prompt
You are on expert in evaluating the perforrance of an android navigation agent. The agent is designed to help huran user navigate the device to complete a task. Given the user's is task, the agent's action history, key points for task completion, some potentially ixportant states in the agent's trafectory and their reasons, your goal is to determine whether the agant has completed the task and achieved all requiresents.
Your response must strictly fallow the folloxing evalustion critecial "Irportant Evalustion Criteria": 1: filtered results confimatioon, or visible displayed correctly. filters were properly applied (L.e., nissing selection, missing must carefully check whether results), "bast,"
not considered successful.
snapshots "latest, "sost "newest" correctly applied using the filber function(e.g. sort function).
history neet these key points. that specific filter "lowest," "clesest," "aaghest-rated," "largest," and
conditions,
Cortain points requirements should be applied the filter. Otherwise, search with requirements inout will be deweed fallure since it cannot guarantse that all results seet the requiresntsi specific 4: If the task requiess flltering by a range of roney, years, or the nambrr of beds and bashroons, the applied filter aust exactly satch the given requtresent. any deviation results in failure. To ensure the task is successful, the applied filter aust precisely natch the specifled range without being too broad or too narros. Exaeples of Failure tases:
If the requirement is less than $58, bet the applied filter is less than $25, it is a failure. If the requirment is $1588-43500. but the applled filter is $3808-$2500, it is failure. If the requirement i5 $25-$290, but the applied filter is $3-5268, it is failure. If the required years are 2894-2012, but the filter applied is 2881-2012, it 15 failure. 2815,
requiedd years are task recuires exactly beds, Seac tasks resuire subeission action or display results be considered successful. epty(e.g.,
applied filter apolied 2814, beds, it failure.
fallure.
retrieved
found),
agent correctly perforred the required
If should still concidered available stens, applying -illerr not necessary. page itees that reet requirements
displays
the agent selects cheapest lowest price), task still considered successful.
*IYPORTANT*
Foreat your response into twa Tines as shown helow:
Thoughts: syour thoughts and reasoning process based an double-chocking each koy points and the evaluation criterlas status: "success" or "failure"
User Task: (task goal)
key Points: (koy_points)
Action History: (history_actions)
potestially ieportant snapshots of agent's trajectory their reasons: (thoughts)
Figure 12: Prompt template for WebJudge.
31

---

System Prorpt
evalusting the performance of andreid navi navigate the device to complete tasi, diven the user intent, and the Final state of the serwen, goal Ss to decidde whether the agent has successuully complated the task on not.
You
"IMPORTANT
Foreat your response into two lines as shown below:
Thoughts:
syour thoughts and reasoning procesey'
Status: succees" or "fallure"
User Prompt
User Intent: (task_pool)
Action History: (history_actions
The last snapshot of the screen shown in the Irage.
Figure 13: Prompt template for AutoEval.
export vallsting formance an android navigation agent. agent designed help navigate complete task. user's intent, the agent history, two states screen, your goal shathen agent successfully cospleted task
"Output
your response
Thoughts: syour thoughts reasoning pracess> oYES NO:
User Prompt
(task_poal)
Action History: (histony, actions)
The second last and the last state of shown the
Figure 14: Prompt template for AutoEval which constructed by enhancing AutoEval with the last two screenshots.
System Prorpt
You
and helpful visual assistant that is well trained manipulate mobile Your task Is to navigate the current screen to complete the user request. mobile You provided with history actions trying to accomplish the user request. You required to decide the next single-step valid action conduct on the current screen.
provided with screenshot of the
#4 Valid action types on the screen
DUAL POINT
TYPE
PRESS BACK
HOPE
PRESS,
STATUS, TASK COMPLETE
Dutput
response
strictly
format, consisting following keys corresponding
"action type": "estring,
decision action type?",
"touth_point": "clist, coordinates
point the
when action_type DUAL_POINT, it should
[-1, -1l>" -1l>">
"Iift poine": "clist, coordinates for the l1ft point the screen, when actioon type not DUAL POINT, it should be
[-1, -1}>
"type text": "estring, the text to type on the screen, only for TYPE action type, otheruise it should be an string?"
#4 Output Exaeplal
("action_typer type": "DUAL POINT", "touch point": [0.8949, 0.2041], "1SFt point": (8.8549, 0.2941], "typo text": "")
Butput Exaeple2
("action_type "OINI_ "touch_point": (9.8, 8.5], "lift_ooint": [0.2, 0.5], "type_test"; "")
Output Exarpla3
("action_type"
"touch_point": [-1.0, -1.8], "lift_pointts [-1.,, -1.0], "type_text": "Livein")
('action.type "PRESS BACK", "touch_point": (-1.8, -1.01, pdint" [-1.0, -1.0), texe": "")
Exaeples
Factian_type" "PRESS HONE", "touch_pontt"s [-1.0, -1.0), "lift_point"; [-1.8, -1.6), "tyee texe": **
#t Output Examples
("action_ type": "PRESS, ENTER", "touch_poine": [-1.0, -1.0), "ointt [-1.3, -1.0), "oypottex":: "*)
#4 Output Exaepla7
("action_ type": "STATUS TASC COMPLETE', point"' [-1.8, -1.0], "1ift point"' [-1.8, <1.0), "type text": "*
User Prompt
cirage>
History
(history actions)
Generate next actions
Figure 15: Prompt template for GUI navigation agent training with AITW dataset.
32

---

Prompt cinagas
You are a au task expert, will provide you with high-level instruction, an action history, screanshet with its corresponding accossbilitty tree.
High-level Instruction: (high_]evel instrrcintt
history: (action history:
Accessibility tree: (accessiollity_tomm tree))
Please generate the Tow-leve] thought and action for the next
Figure 16: Prompt template for GUI navigation agent training with OS- SS-Genesis dataset.
33


Reviews:
[{"review_id": "D93Dpxfd5u", "reviewer": "Reviewer_3VNW", "summary": "This paper proposes a novel framework called Uncertainty-aware Reinforced Self-Training (URST) for training lightweight multimodal large language models (MLLMs) to evaluate the quality of GUI navigation trajectories. Unlike existing methods that rely heavily on prompting proprietary models, URST trains evaluators using self-generated data, enhanced by uncertainty-aware sampling and simplified group policy optimization (SGPO). Extensive experiments demonstrate that URST outperforms state-of-the-art prompting-based and fine-tuning methods, particularly in generalization to out-of-domain data. The trained evaluator also enhances downstream GUI agents by filtering noisy training data.", "questions": "1. The paper reports performance metrics without error bars or statistical significance due to compute constraints. The authors could run multiple trials with different random seeds and report standard deviation or confidence intervals.\n\n2. The reasoning performance is described qualitatively but not measured explicitly. It is suggested to quantify the quality of generated chain-of-thought (CoT) using human or model-based annotation. And add visualizations or examples of failure cases to show when/why URST succeeds or fails.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "***Strengths***\n\nFirst, the paper addresses a critical bottleneck in GUI agent training—noisy trajectory data—by proposing an efficient, cost-effective, and domain-adaptive evaluator. The use of lightweight open-source MLLMs and self-training avoids the high latency and cost of proprietary models. The URST framework incorporates two novel components: (1) uncertainty-aware sampling to prioritize informative, high-entropy samples and (2) SGPO to leverage both correct and incorrect samples for contrastive learning. These innovations are validated through rigorous ablation studies, showing measurable performance gains, especially in out-of-domain scenarios.\n\nSecond, the experiments are comprehensive and reproducible, with comparisons across multiple benchmarks, including in-domain (AITW-ID) and out-of-domain (AITW-OOD, AW-OOD) datasets. URST shows consistent improvements in F1 score and downstream agent performance. The authors clearly document their method, training configuration, datasets, and open-source code/data commitments, which greatly increases the impact and credibility of the work. Additionally, the qualitative examples illustrate robust reasoning capabilities and explain how URST corrects common failure modes in baseline methods.\n\n***Weaknesses***\n\nDespite its strengths, the paper relies on a proprietary MLLM to generate the initial training data, creating a partial dependency on the very systems it aims to replace. Although the authors argue this is limited to initialization, it could limit true independence in practice. Moreover, statistical significance and variability are not fully explored—the paper omits error bars due to compute limitations, which may raise questions about result stability.\n\nAnother minor weakness is that URST’s scalability to larger models or real-time deployment scenarios is not discussed in depth. While lightweight models are used for efficiency, more details on training cost, iteration limits, and potential bottlenecks (e.g., entropy computation) would strengthen the paper’s practical relevance.\n\nOverall, this paper presents a timely and technically significant contribution to GUI agent training and trajectory evaluation by enabling efficient, domain-adaptive, and self-improving evaluators. The proposed URST method introduces meaningful algorithmic innovations and demonstrates solid empirical gains over existing approaches. Its potential for real-world impact is high, particularly for practitioners seeking to train robust GUI agents without relying heavily on commercial LLM APIs.", "was_revised": true, "final_justification": "The author's respnse addressed the concerns/suggestions in my review. I have no other question.", "rebuttal": "We appreciate the reviewer’s insightful comments and helpful suggestions. Below are our responses and explanations:\n\n---\n\n> **Q1.** The paper relies on a proprietary MLLM to generate the initial training data, which could limit true independence in practice. \n\n**A1.** We thank the reviewer for raising this important point about dependency on proprietary MLLMs. While our method does leverage a proprietary MLLM for both initial data generation *and* as a reward model during iterative self-training, this design reflects a pragmatic trade-off to address the absence of ground-truth trajectory annotations. Crucially, the role of the proprietary model diminishes over time: it is only used to score the judgment predictions, and the lightweight MLLM’s parameters are progressively refined using its own diverse CoTs. This mimics successful iterative approaches in other domains (e.g., robotics [1] or NLP [2,3]), where advanced models guide early-stage learning but are phased out as the student model converges.\n\nThat said, we fully agree that reducing reliance on proprietary systems is desirable. However, our experiments suggest that **URST still advances independence compared to end-to-end reliance on proprietary MLLMs (like prompting-based methods, Webjudgment, AutoEval\\*, and etc.)**, as the final evaluator operates without external API calls during deployment. We will add this limitation and direction to the revised version.\n\n[1] Self-Improving Robots: End-to-End Autonomous Visuomotor Reinforcement Learning, CoRL 2023.\n\n[2] Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models, Google DeepMind, TMLR 2024.\n\n[3] CREAM: Consistency Regularized Self-Rewarding Language Models, ICLR 2025\n\n---\n\n> **Q2.** URST’s scalability or real-time deployment. More details on training cost, iteration limits.\n\n**A2.** Thanks for your constructive suggestions. In Table 1 of the main paper, we have presented URST with two different MLLMs, Qwen2VL-2B and Qwen2.5VL-3B. Compared to Qwen2Vl-2B, Qwen2.5VL-3B achieves a superior performance across all three test sets, indicating the scalability of URST.\n\nFor real-time deployment, we analyze the inference time of prompting-based methods (WebJudge, AutoEval*) and fine-tuning-based methods (URST) in the following table. In terms of inference efficiency, URST demonstrates a significant advantage over both WebJudge and AutoEval. **URST achieves an average inference time of only 2.17 seconds per sample, which is approximately 17.6× faster than WebJudge (38.1 s) and 2.8× faster than AutoEval\\* (5.99 s).** This substantial reduction in computational cost highlights the lightweight design of URST and its suitability for large-scale trajectory evaluation. We will include these results and analyses in the revised version. \n\n|  | WebJudge | AutoEval* | URST |\n| --- | --- | --- | --- |\n| Inference Time (s / per sample) | 38.1 | 5.99 | 2.17 |\n\nWe also investigate the training cost of URST and provide detailed results on the training time for all iterations. All experiments were conducted on a server equipped with 4 A100 40GB GPUs. As shown in the table below, the initial phase with SFT requires only a small amount of time, whereas the subsequent iterative training with sampling and SGPO accounts for the majority of the time consumption. We will include these results and analyses in the revised version. \n\n|  | Initial | Iter. 0  | Iter. 1  | Iter. 2 |\n| --- | --- | --- | --- | --- |\n|  |  | Sampling / SGPO | Sampling / SGPO | Sampling / SGPO |\n| Training Time | 13min | 1h43min / 5h28min | 1h33min / 5h31min | 1h21min / 5h32min |\n| F1 | 77.88 | 79.43 | 82.85 | 84.13 |\n\nThe table above also presents the performance across all iterations. It can be observed that each iteration consistently improves the evaluator’s performance. However, performing additional iterations (> Iter. 1) yields diminishing gains due to the relatively low quality of the newly sampled training data. Considering the computational cost, we limited our experiments to three iterations, which were sufficient to achieve the best performance observed in our study. Therefore, we did not further investigate the upper bound on the number of iterations. **We agree that the choice of iteration limits is an important factor, and we believe it can be determined by balancing training cost and performance requirements to reach an optimal trade-off under given computational constraints.** We will include these results and analyses in the revised version.\n\n---\n\n> **Q3.** The authors could run multiple trials with different random seeds and report standard deviation or confidence intervals.\n\n**A3.** Thanks for your constructive suggestions. In the following Table, we present the statistical results of SFT and URST with three different random seeds. URST significantly outperforms SFT across all three test sets. These improvements are statistically meaningful given the non-overlapping confidence intervals, particularly for the F1 score, where URST shows a substantial gain (e.g., +4.82 on AITW-OOD-traj). Moreover, the smaller standard deviations in most metrics indicate that URST produces more stable results across different runs. **Overall, these findings demonstrate that URST not only improves performance but also enhances result consistency.**\n\n| Methods | AITW-ID-traj |  | AITW-OOD-traj |  | AW-OOD-traj |  | Overall |  |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\n|  | acc | f1 | acc | f1 | acc | f1 | acc | f1 |\n| SFT | 83.61±0.79 | 87.15±0.50 | 93.61±1.42 | 78.71±3.00 | 77.13±0.97 | 71.30±1.17 | 83.08±0.71 | 78.62±0.71 |\n| URST | 87.78±0.39 | 90.64±0.27 | 94.72±0.78 | 83.53±2.09 | 83.40±1.27 | 82.14±1.80 | 87.47±0.81 | 85.59±1.05 |\n\n---\n\n> **Q4.** It is suggested to quantify the quality of generated chain-of-thought (CoT). And add visualizations or examples of failure cases.\n\n**A4.** Thanks for your constructive suggestions. To evaluate the reasoning quality of different models, we conducted pairwise comparisons of their Chain-of-Thought (CoT) outputs by using GPT-4.1. Specifically, URST was compared separately with SFT and AutoEval*, and the winning rates of the latter models were recorded. As shown in the table, SFT outperformed URST in only **32%** of the paired cases, while AutoEval* surpassed URST in only **36%** of the comparisons. These results indicate that **URST consistently maintains a clear advantage over both baselines**, as it was judged superior in the majority of cases across both pairwise evaluations. This suggests that **URST’s reasoning process is more reliable and better aligned with human preferences than that of the other models.**\n\n|  | SFT is winner | AutoEval* is winner |\n| --- | --- | --- |\n| URST | 32% | 36% |\n\nIn section 4.5 and C.2, we have presented the case study of examples where URST succeeds or fails. Here, we would like to restate interesting and insightful findings:\n\n1. When inspecting the success cases of URST, we found that URST has advantages of discerning subtle differences, mitigating hallucinations, and tracking fine-grained transitions (as shown in Figure 5 of the main paper and Figure 2 of the Appendix).\n2. When inspecting the failure cases of URST, we found that URST has limitations of insufficient observation, GUI elements misinterpretation, and instruction misunderstanding (as shown in Figure 3 of the Appendix)."}, {"review_id": "RyChLLwS7a", "reviewer": "Reviewer_ktZw", "summary": "This paper introduces URST, a framework for training MLLMs to evaluate GUI navigation trajectories more reliably. URST enables self-improvement through iterative self-generated judgments and uses an uncertainty-aware sampling strategy to select informative training examples. A simplified group policy optimization (SGPO) method helps the evaluator learn from diverse feedback. The resulting evaluator outperforms existing approaches on both in-domain and out-of-domain data and improves GUI agent training.", "questions": "**Generalization Beyond Mobile GUI Tasks**: The evaluation currently focuses only on mobile GUI tasks. To demonstrate broader applicability, it would be helpful to test the pipeline on other GUI domains, such as desktop or web interfaces. This would provide stronger evidence of the method's generalizability across varied interaction modalities.", "limitations": "See Weaknesses part.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "## Strengths\n\nThe model achieves strong performance across multiple benchmarks. The writing is clear and the method is presented in a straightforward yet effective way. The motivation for self-improvement is particularly compelling in the GUI domain, where the cost of human annotation is high and scalable data generation is challenging.\n\n## Weaknesses\n\n**Reliance on Evaluator Uncertainty**: The method relies heavily on the uncertainty of the evaluator to guide trajectory selection. This implicitly assumes that the evaluator is sufficiently accurate, and when it expresses high confidence, the corresponding trajectory is likely of high quality. However, since the evaluator is not trained on any ground-truth data, its overall reliability is questionable. This casts doubt on whether its uncertainty is a trustworthy signal for trajectory assessment. A more thorough evaluation either automatic or human on how well the evaluator’s uncertainty correlates with actual trajectory quality would greatly strengthen the claim.\n\n**Evaluator Evolution**: It would be valuable to track and report the performance trajectory of the evaluator itself over the course of training. Showing how its evaluation quality changes particularly in early iterations through either automatic metrics or human judgments would provide insights into the robustness and stability of the evaluator as it evolves.\n\n**Alternative Sampling Metrics**: While evaluator uncertainty is used to select trajectories, it is not the only potential signal for quality. For instance, policy uncertainty could also serve as an indicator. If the policy model is uncertain about its decisions, the trajectory may be of lower quality. Comparing the effectiveness of evaluator uncertainty against alternative metrics (e.g., policy entropy) would offer a more comprehensive justification for the current choice and potentially reveal better alternatives.\n\n**Baseline RL Methods**: Although the authors mention GRPO when introducing SGPO, there is no ablation or comparison with GRPO, PPO, or other standard reinforcement learning methods. This makes it unclear whether SGPO is tailored for the self-improvement framework or simply an arbitrary choice. Including these comparisons would help clarify the advantages of SGPO and whether it meaningfully outperforms existing baselines.", "was_revised": true, "final_justification": "The response resolves most of my concerns and I’m impressed by the consistent improvements across iterations.", "rebuttal": "We appreciate the reviewer’s insightful comments and helpful suggestions. Below are our responses and explanations:\n\n---\n\n> **Q1.** **Reliance on Evaluator Uncertainty**: The method relies heavily on the uncertainty of the evaluator to guide trajectory selection. This casts doubt on whether its uncertainty is a trustworthy signal for trajectory assessment.\n\n**A1.** Thanks for your insightful comments. We would like to provide a detailed clarification. The uncertainty measurement of the evaluator aims to find the most valuable trajectories for the model, as trajectories with high uncertainty values reflect the limitations or knowledge gaps of the evaluator. The measurement of uncertainty does not involve the trajectory annotations and is irrelevant to the trajectory annotation quality. \n\nAlthough the evaluator is not trained on any ground-truth data, the initialization phase generates a strong base evaluator by distilling from a proprietary MLLM (generally has around an 80% F1 score). In subsequent iterations, the evaluator can leverage a reward model for judgment assessment, and gradually achieve better performance. As shown in the table of A3, we present a new metric for measuring uncertainty and also achieve a superior performance, demonstrating the effectiveness of our uncertainty-aware sampling. \n\nIn summary, **the uncertainty is a useful signal for trajectory value assessment, and irrelevant to the trajectory annotation quality. The evaluator can achieve self-improvement with the help of distillation from a proprietary MLLM and guidance from the reward model despite the lack of ground-truth data.**\n\n---\n\n> **Q2.** **Evaluator Evolution**: It would be valuable to track and report the performance trajectory of the evaluator itself over the course of training.\n\n**A2.** Thanks for your constructive suggestions. As shown in Figure 3 of the main paper, we present the overall performance across multiple iterations. **It can be observed that all iterations consistently enhance the evaluator’s performance. However, performing additional iterations yields diminishing gains due to the relatively low quality of the newly sampled training data.** In the following table, we also provide the detailed results of multiple iterations. The performance of URST on AITW-ID-traj and AW-OOD-traj gradually increases with iterative training. However, its performance on AITW-OOD-traj peaks at iteration 0 and exhibits fluctuations (primarily in the F1 score) in subsequent iterations. We attribute this behavior to the severe class imbalance in AITW-OOD-traj (success: 14%, fail: 86%), as shown in Table 1 of the Appendix. The F1 score is highly sensitive to even small performance variations in the minority class. We will include these detailed analyses in the revised version.\n\n|  | Total |  | AITW-ID-traj | (success: 64%, fail: 36%) | AITW-OOD-traj | (success: 14%, fail: 86%) | AW-OOD-traj | (success: 43%, fail: 57%) |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\n|  | Acc | F1 | Acc | F1 | Acc | F1 | Acc | F1 |\n| Initial | 83.80 | 77.88 | 80.00 | 82.85 | 95.00 | 81.25 | 79.82 | 73.05 |\n| Iter. 0 | 84.45 | 79.43 | 83.33 | 86.11 | 96.67 | 87.50 | 78.48 | 72.41 |\n| Iter. 1 | 85.95 | 82.85 | 86.67 | 89.47 | 93.33 | 78.95 | 81.61 | 78.31 |\n| Iter. 2 | 86.39 | 84.13 | 87.50 | 90.45 | 94.16 | 82.05 | 81.62 | 79.60 |\n\n---\n\n> **Q3.** **Alternative Sampling Metrics**: Comparing the effectiveness of evaluator uncertainty against alternative metrics would offer a more comprehensive justification.\n\n**A3.** Thanks for your constructive suggestions. In this work, our uncertainty-aware sampling uses **sampling-based entropy** to measure the uncertainty of the evaluator, which is inspired by self-consistency methods in LLM [1]. Besides the proposed metric, we also devise an alternative metric, **prediction-based entropy**, which uses greedy decoding to generate a single judgment of each training input, and computes the probability entropy of the judgment prediction. In the following table, we compare these two metrics with our URST. Sampling-based entropy consistently outperforms prediction-based entropy across all test sets. \n\n**Our sampling-based entropy uses temperature sampling and entropy of multiple outputs to measure the uncertainty of the evaluator, which could explore various reasoning paths for answer generation and provide a reliable assessment of the model’s confidence.** However, prediction-based entropy uses greedy decoding to generate a single output for entropy computation, and is prone to suffer from the over-confidence issue of LLM [2], thus making less reliable measurement.\n\n| URST | Total |  | AITW-ID-traj |  | AITW-OOD-traj |  | AW-OOD-traj |  |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\n|  | Acc | F1 | Acc | F1 | Acc | F1 | Acc | F1 |\n| w/ prediction-based entropy | 84.67 | 81.07 | 86.67 | 89.47 | 91.67 | 73.68 | 79.82 | 75.68 |\n| w/ sampling-based entropy | 86.39 | 84.13 | 87.50 | 90.45 | 94.16 | 82.05 | 81.62 | 79.60 |\n\n[1] Self-Consistency Improves Chain of Thought Reasoning in Language Models\n\n[2] Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs\n\n---\n\n> **Q4.** **Baseline RL Methods**: There is no ablation or comparison with GRPO, PPO, or other standard reinforcement learning methods.\n\n**A4.** Thanks for your constructive suggestions. We would like to provide clarification regarding the comparison with GRPO. SGPO in URST is exactly GRPO without normalization in advantage computing. The experimental comparison of SGPO and GRPO is presented in row “URST” and “w/o normalization removing” of Table 2. **Compared to GRPO, SGPO can achieve an improvement of 1.44% in the overall F1 score.** We also show the detailed comparison in the following table.\n\n| URST | AITW-ID-traj | AITW-OOD-traj | AW-OOD-traj | Total |\n| --- | --- | --- | --- | --- |\n|  w/ GRPO | 89.61 | 78.95 | 77.95 | 82.69 |\n|  w/ SGPO | 90.45 | 82.05 | 79.60 | 84.13 |\n\nPPO needs an additional critic model for value computing, which requires a more complex training setting and a large dataset [1]. Recent studies [1,2,3] show that GRPO with verifiable rewards is computationally efficient and shows advantages in the CoT-based reasoning scenario, compared to PPO. Thus, we refer to GRPO as a strong RL baseline.\n\n[1] What’s Behind PPO’s Collapse in Long-CoT? Value Optimization Holds the Secret\n\n[2] VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks\n\n[3] DAPO: An Open-Source LLM Reinforcement Learning System at Scale\n\n---\n\n> **Q5.** **Generalization Beyond Mobile GUI Tasks**: It would be helpful to test the pipeline on other GUI domains, such as desktop or web interfaces.\n\n**A5.** Thanks for your constructive suggestions. Recently proposed **AgentRewardBench** [1] is the first benchmark to **assess the effectiveness of MLLM-based evaluators for evaluating web agents.** It contains 1302 web trajectories across 5 benchmarks and 4 agents. Due to the lack of training data, we split AgentRewardBench into two sets, 1047 trajectories for the training set and 255 for the test set, and ensure a similar class distribution.\n\n|  | #Trajectory | #success | Success Rate |\n| --- | --- | --- | --- |\n| Training | 1047 | 293 | 27.99% |\n| Test | 255 | 63 | 24.71% |\n\nWe evaluate the prompting-based (WebJudge, AutoEval*) and fine-tuning-based (SFT, URST) methods on this web benchmark, and report the results in the following Table. URST is trained under the setting of initial-300 samples, iter. 0-200 samples, iter. 1-200samples, and SFT is trained with the same total number of samples. Our URST achieves the highest performance on AgentRewardBench, reaching 87.06% accuracy and 77.85% F1 score, substantially outperforming the other baselines. **These comparisons underscore the effectiveness and generalization of URST.**\n\n| Methods | AgentRewardBench |  |\n| --- | --- | --- |\n|  | Acc. | F1 |\n| WebJudge | 83.07 | 68.15 |\n| AutoEval* | 85.10 | 74.67 |\n| SFT | 84.31 | 74.36 |\n| **URST** | **87.06** | **77.85** |\n\n[1] AGENTREWARDBENCH: Evaluating Automatic Evaluations of Web Agent Trajectories, arxiv 2025."}, {"review_id": "eZOQB9sHrA", "reviewer": "Reviewer_oZh1", "summary": "This paper proposes a framework for developing evaluators for GUI agents. Unlike API--based evaluators, which are expensive or small LM based evaluators which may not be as good, it seeks to develop better SLM-based judges. The key innovation is an iterative self-training based approach to select quality examples for training MLLMs. It uses uncertaining-aware reinforced training to select quality examples that add more value. The results are reported on AITW-ID, AITW-OOD, and a benchmark based on AndroidWorld created by authors.", "questions": "1. Is this approach applicable just to GUI agents or LLM agents in general? I see no design considerations specific to GUI agents. It would be great if the authors can evaluate this on LLM agent benchmarks in general e.g. JudgeBench [1].\n\n[1] Tan et al., JudgeBench: A Benchmark for Evaluating LLM-based Judges, ICLR 2025.", "limitations": "yes, limitations in appendix.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Alleviates the issues of existing evaluators such as high cost of API-based evaluators\n2. Makes use of negative samples as well in reinforced self-training\n3. Improvement in GT agreement compared to existing baselines\n\nWeaknesses:\n1. Entropy is an indicator of confidence of evaluator. Although the entropy decreases (Fig. 4), it remains high. \n2. The performance on standard benchmarks (AITW-ID, AITW-OOD) is 2-3%, which is marginal.", "was_revised": true, "final_justification": "I read the author's response. However, I would like to see the results on the JudgeBench benchmark, which is very relevant for this problem. Hence, I would like to keep my score (borderline accept).", "rebuttal": "We appreciate the reviewer’s insightful comments and helpful suggestions. Below are our responses and explanations:\n\n---\n\n> **Q1.** Entropy is an indicator of confidence of evaluator. Although the entropy decreases (Fig. 4), it remains high.\n\n**A1.** Thanks for your insightful comments. As shown in Figure 4 of the main paper, the entropy of the training data sampled by URST is relatively high (a mean value of 0.4), which can be attributed to the goal of uncertainty-aware sampling: selecting uncertainty data with high entropy. If we measure the confidence of the evaluator, we should refer to the entropy of randomly sampled training data (a mean value of 0.02), which is very low and shows a strong confidence of the trained evaluator.\n\n---\n\n> **Q2.** The performance on standard benchmarks (AITW-ID, AITW-OOD) is 2-3%, which is marginal.\n\n**A2.** Thanks for your insightful comments. In the following table, we present the results and inference time of prompting-based and fine-tuning-based methods. Compared to SFT, URST achieves significant performance improvements, especially on F1 scores (increases 3.49% - 4.82%). Compared to AutoEval*, URST obtains a slightly smaller performance improvement ( increases 2.64% - 3.53% on F1 score), but it delivers a 2.8x inference speedup. Overall, our URST achieves a well-balanced yet remarkable improvement in both performance and efficiency.\n\n| Methods | Inference Time (s / per sample) | AITW-ID-traj | AITW-OOD-traj |\n| --- | --- | --- | --- |\n| WebJudge | 38.1 | 85.91 | 66.67 |\n| AutoEval* | 5.99 | 88.00 | 80.00 |\n| SFT | 2.17 | 87.15 | 78.71 |\n| **URST** | **2.17** | **90.64** | **83.53** |\n\n---\n\n> **Q3.** It would be great if the authors can evaluate this on LLM agent benchmarks in general e.g. JudgeBench.\n\n**A3.** We appreciate the reviewer’s insightful comment regarding the general applicability of our approach beyond GUI agents. While we fully agree that evaluating on broader LLM agent benchmarks such as JudgeBench would further strengthen our claims, conducting such extensive experiments is unfortunately beyond our current time and resource constraints. The evaluation setting and training data collection process of JudgeBench differ from those of our MobileGUIBench.\n\nNevertheless, to address the reviewer’s concern about generalizability, we have extended URST to a **web-based benchmark** and conducted additional experiments. Recently proposed **AgentRewardBench** [1] is the first benchmark to **assess the effectiveness of MLLM-based evaluators for evaluating web agents**. It contains 1302 web trajectories across 5 benchmarks and 4 agents. Due to the lack of training data, we split AgentRewardBench into two sets, 1047 trajectories for the training set and 255 for the test set, and ensure a similar class distribution.\n\nWe evaluate the prompting-based (WebJudge, AutoEval*) and fine-tuning-based (SFT, URST) methods on this web benchmark, and report the results in the following Table. URST is trained under the setting of initial-300 samples, iter. 0-200 samples, iter. 1-200samples, while SFT is trained with the same total number of samples. Our URST achieves the highest performance on AgentRewardBench, reaching 87.06% accuracy and 77.85 F1 score, substantially outperforming the other baselines. **The results demonstrate that URST maintains its effectiveness and exhibits strong generalization capabilities in a broader setting.** \n\n| Methods | AgentRewardBench |  |\n| --- | --- | --- |\n|  | Acc. | F1 |\n| WebJudge | 83.07 | 68.15 |\n| AutoEval* | 85.10 | 74.67 |\n| SFT | 84.31 | 74.36 |\n| **URST** | **87.06** | **77.85** |\n\n[1] AGENTREWARDBENCH: Evaluating Automatic Evaluations of Web Agent Trajectories, arxiv 2025."}, {"review_id": "8v5ddUB4wv", "reviewer": "Reviewer_nyzi", "summary": "This paper presents a framework, Uncertainty-aware Reinforced Self-Training, for training lightweight Multimodal Large Language Models to serve as efficient and reliable evaluators for GUI agent navigation trajectories. Addressing the limitations of costly and domain-agnostic prompting-based methods that rely on proprietary MLLMs, URST enables an evaluator to improve itself iteratively by learning from its own generated thoughts and judgments. The framework introduces two key technical contributions: a novel uncertainty-aware sampling strategy that uses prediction entropy to select the most informative training examples, and a Simplified Group Policy Optimization algorithm that effectively leverages both positive and negative samples to enhance the model's reasoning. The authors demonstrate that their URST-trained evaluator not only surpasses the performance of prompting-based and other fine-tuning methods on both in-domain and out-of-domain datasets but also leads to consistent performance improvements in downstream GUI agents when used as a data filter.", "questions": "- Line 123: why is this approach/formulation restricted to “offline” dataset online?\n\n- Equation 1: why does the evaluator need to take in the chain-of-thought c as input?\n\n- Can you elaborate on how the “Prompting-based Methods” are implemented? For example, for AgentTrek, did you just use gpt-4o-mini and fasttext classifier to filter trajectories and then train the model on the filtered dataset?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "__Strengths__:\n- The writing of the paper is clear, making it easy to follow.\n\n- The problem formulation is interesting and novel.\n\n\n__Weaknesses__:\n\n- The biggest concern of this work is insufficient baseline comparisons. In Table 1, the paper should also report the performance of non-finetuned models, (namely, Qwen2VL-2B, Qwen-VL-Max, gpt-4o etc) for better understand the effectiveness of their proposed training methods and advantages over proprietary models. Additionally, since the proposed approach also performs some forms of distillation of larger models (line 167), the author should also compare with other simpler distillation-based methods so that readers can better appreciate the proposed method.\n\n\n- Also critically, performance enhancement is quite minimal in both in-domain and cross-domain when it comes to actually enhancing GUI agent performance. In Section 4.4, the improvement is < 2% on in-domain and <1% on out-of-domain. Based on the results, it is hard to convince that the trajectory evaluator is a must/effective in enhancing GUI agent performance. \n\n- For section 4.4, the author should also compare their approach with others. In fact, the comparison with baselines here is more important than Table 1 as they reflect how different approaches are able to improve GUI agent performance, which is the ultimate goal of building a trajectory evaluator.", "was_revised": true, "final_justification": "The reviewer has addressed all my concerns and thus I have increased my ratings.", "rebuttal": "We appreciate the reviewer’s insightful comments and helpful suggestions. Below are our responses and explanations:\n\n---\n\n> **Q1.** The biggest concern of this work is insufficient baseline comparisons.\n\n**A1.** Thanks for your comments. We would like to clarify this point about insufficient baseline comparisons. **The prompting-based methods (AgentTrek, WebVoyager, WebJudge, and AutoEval) are non-finetuned models** because they use proprietary MLLMs that cannot be trained (as described in Lines 131-133). These prompting-based methods are based on Qwen-VL-Max, and show inferior performances to our URST, demonstrating the effectiveness of URST. \n\n**“SFT” adopted in our work is a typical distillation-based method** (as described in Lines 248-249). It uses the predictions (CoTs and judgments) from a larger model, Qwen-VL-Max to train Qwen2.5VL-3B for knowledge distillation. “SFT” also achieves inferior results, highlighting the value of our URST. For clarification, we will revise our manuscript and add more detailed descriptions.\n\n---\n\n> **Q2&3.** It is hard to convince that the trajectory evaluator is a must/effective in enhancing GUI agent performance. For section 4.4, the author should also compare their approach with others. In fact, the comparison with baselines here reflects how different approaches are able to improve GUI agent performance, which is the ultimate goal of building a trajectory evaluator.\n\n**A2&3.** We thank the reviewer for raising this important point about the goal of building a trajectory evaluator. Drawing on prior studies [1-3] and our practical insights, we identify **three key roles of the trajectory evaluator for enhancing GUI agents**.\n\n### **1. An autonomous evaluator for online environments**\n\nIt eliminates the need for manually written verification scripts and has greater scalability and adaptability. The primary contribution of our work is to build this kind of evaluator for assessing the agent-executed trajectories (such as samples in AITW-OOD-traj and AW-OOD-traj), showing the effectiveness and superiority of our URST.\n\n### **2. A filter for cleaning the noisy dataset**\n\nBy filtering the incorrect trajectories, we can improve the quality and reliability of the dataset. The impact of the filter increases as the noise level in the data rises. To prove this point, we conducted some comparison experiments by leveraging two settings, Standard-Noise and High-Noise.\n\n- **Standard-Noise**: Discarding all incorrect trajectories, and using all steps in each trajectory for training GUI agents.\n\n- **High-Noise**: Discarding all incorrect trajectories, and using the last 40% of steps in each trajectory for training GUI agents. **The underlying rationale is that the majority of incorrect trajectories tend to fail in the last few steps.** When tasks are highly challenging or involve unseen environments, the data automatically collected by the agent often contains high levels of noise, meaning that the entire trajectory is wrong from the very beginning.\n    \nBased on the following Table, we can conclude that 1) our URST can consistently outperform other baselines (including AutoEval* and SFT) in filtering noisy datasets. 2) When the noisy level rises, all filters (based on AutoEval*, SFT, and URST) can achieve a greater impact on agent performance, with URST consistently achieving the best result by a significant margin. 3) URST demonstrates a clear overall advantage, delivering optimal outcomes in time cost, expenditure, and performance.\n\n| Methods | Time Cost (s / per sample) | Cost ($) (per 1M Tokens) | AITW |  | OS-Genesis |  |\n| --- | --- | --- | --- | --- | --- | --- |\n|  |  |  | Standard-Noise | **High-Noise** | Standard-Noise | **High-Noise** |\n| Original | 0 | 0 | 63.50 | 20.88 | 46.83 | 46.43 |\n| Filtered by AutoEval* | 5.99 | 0.41 | 64.98 | 38.93 | 47.26 | 47.65 |\n| Filtered by SFT | 2.17 | 0 | 64.32 | 32.59 | 47.18 | 47.22 |\n| Filtered by URST | 2.17 | 0 | 65.02 | 40.41 | 47.66 | 48.26 |\n\n### **3. A reward signal for agent inference enhancement**\n\nIn an online environment, the trajectory evaluator can serve as a reward signal to enhance an existing GUI agent at inference time, using the Reflexion technique [4]. Specifically, a GUI agent first attempts a task, and an external evaluator is used to judge whether its attempt was successful or not. The agent will be prompted to reflect on the failure and retry unsuccessful tasks. We conduct experiments on the most popular online benchmark, AndroidWorld, with M3A agents. The best prompting-based method, AutoEval*, and our URST are used as a reward signal for eliciting reflection. URST achieves superior performance, with a 1.8% improvement over AutoEval*, underscoring its clear advantage in enhancing GUI agents.\n\n|  | AndoridWorld successful rate |\n| --- | --- |\n| M3A | 39.7% |\n| M3A + AutoEval* | 44.8% |\n| **M3A + URST** | **46.6%** |\n\n[1] Autonomous Evaluation and Refinement of Digital Agents, COLM 2024.\n\n[2] AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorials, ICLR 2025.\n\n[3] An Illusion of Progress? Assessing the Current State of Web Agents, COLM 2025.\n\n[4] Reflexion: language agents with verbal reinforcement learning, NeurIPS 2023.\n\n---\n\n> **Q4.** Line 123: Why is this approach/formulation restricted to “offline” dataset online?\n\n**A4.** For training a trajectory evaluator, we need a collected dataset containing GUI navigation trajectories. This navigation dataset can be collected in an online environment. When using this collected dataset, we do not need online interaction, thus it can be treated as an “offline” dataset.\n\n---\n\n> **Q5.** Equation 1: Why does the evaluator need to take in the chain-of-thought c as input?\n\n**A5.** Thanks for pointing it out. This is a typo, we will revise equation (1) by removing chain-of-thought c from the inputs.\n\n---\n\n> **Q6.** Can you elaborate on how the “Prompting-based Methods” are implemented?\n\n**A6.** The implementation details of prompting-based methods are described in Figures 5-9 in the Appendix. We also provide the codes for reproduction. For AgentTrek, we follow the same setting in the original paper for evaluating the GUI navigation trajectory. The GPT-4o-mini and fasttext classifiers adopted in AgentTrek are used to filter textual tutorials (for better data collection), instead of GUI navigation trajectories. Thus, we exclude these steps."}]

Author rebuttal:

Meta-review: The paper received all positive reviews, leading to a final acceptance recommendation.
Decision: Accept (poster)