ProRe: A Proactive Reward System for GUI Agents via Reasoner–Actor Collaboration
Authors: Gaole Dai
Shiqi Jiang
Ting Cao
Yuqing Yang
Yuanchun Li
Rui Tan
Mo Li
Lili Qiu
Venue: ICLR 2026 Poster (2026)
Keywords: LLM
GUI Agent
Reward System
Primary area: applications to robotics, autonomy, planning

Abstract: Reward is critical to the evaluation and training of large language models (LLMs). However, existing rule-based or model-based reward methods struggle to generalize to GUI agents, where access to ground-truth trajectories or application databases is often unavailable, and static trajectory-based LLM-as-a-Judge approaches suffer from limited accuracy. To address these challenges, we propose ProRe, a proactive reward system that leverages a general-purpose reasoner and domain-specific evaluator agents (actors). The reasoner schedules targeted state probing tasks, which the evaluator agents then execute by actively interacting with the environment to collect additional observations. This enables the reasoner to assign more accurate and verifiable rewards to GUI agents. Empirical results on over 3K trajectories demonstrate that ProRe improves reward accuracy and F1 score by up to 5.3\% and 19.4\%, respectively. Furthermore, integrating ProRe with state-of-the-art policy agents yields a success rate improvement of up to 22.4\%. The source code is available at https://github.com/V-Droid-Agent/ProRe.

Published as a conference paper at ICLR 2026
PRORE: A PROACTIVE REWARD SYSTEM FOR GUI AGENTS VIA REASONER-ACTOR COLLABORATION
Gaole Dai NTU
Shiqi Jiang? Microsoft Research
Ting Cao Tsinghua University
Yuging Yang
Microsoft Research
Yuanchun Li Tsinghua University
Rui Tan NTU
Mo Li
Lili Qiu
HKUST
Microsoft Research
ABSTRACT
Reward is critical the evaluation and training of large language models (LLMs). However, existing rule-based or model-based reward methods struggle to gen- eralize to GUI agents, where access to ground-truth trajectories or application databases is often unavailable, and static trajectory-based LLM-as-a-Judge ap- proaches suffer from limited accuracy. PRORE, proactive reward system that leverages general-purpose reasoner and domain-specific evaluator agents (actors). The reasoner schedules targeted state probing tasks, which the evaluator agents then execute by actively interacting with the environment to collect additional observations. This enables the reasoner to as- sign accurate and verifiable rewards to GUI agents. Empirical results over 3K trajectories demonstrate that PRORE improves reward accuracy and Fl score by up to 5.3% and 19.4%, respectively. Furthermore, integrating PRORE with state-of-the-arr policy agents yields a success rate improvement of up to 22.4%. The source code is available at htltttts:///iieee wtw.ttttuu.ooooDDoottoo com/VV-rrodd--ggett/ Proke
address these challenges, we propose
INTRODUCTION
Verifiable rewards are pivotal for enabling the continual evolution of large language model (LLM)- based agents Wang et al. (2024b); Guo et al.] (2025); Silver & Sutton (2025). Within this paradigm, L.MSs operate as policy networks, undertaking user requests to generate reasoning. invoke tools and functions, and manipulate graphical user interfaces (GUIs) Qi et al. (2024). Rewards function as quantitative feedback signals that steer the agent's learning process Gao et al. (2024), promoting optimal behaviors while discouraging suboptimal actions.
Reinforcement learning with verifiable rewards (RLVR) has the potential to significantly advance GUI agents Wang et al. (2024c); Xu et al. (2025), Wang et al. (2025). A simple yet effective bi- nary reward for GUI automation is to assess whether the specified task has been successfully com- pleted. To obtain such reward signal, existing methodologies could be generally categorized into rule-based and LLM--ased, as illustrated Figure [ In the rule-based paradigm, human experts manually construct verification code snippets to ascertain the realization of the intended state for each task. For instance, Android Rawles et al. (2024) and WindowsAgenlArent Bonatti et al. (2024) datasets contain more than 116 and 150 manually engineered unit testing code, respectively, to provide grounded signals of task accomplishment for individual GUI automation tasks. While this approach offers high accuracy, it is inherently limited in scalability, as the manual creation of unit testing scripts demands substantial human effort and resources. thereby preventing its use as reward mechanism for large-scale GUT agent training.
LLM-as-a-judge is thus proposed to enable scalable agentic rewards Gu et al. (224)) Bai et al. 2024). Leveraging the capabilities of advanced LLMS such as GPT-4o, this approach evaluates GUI task trajectories, often represented as screenshots, by prompting the model with queries such
WWrk done during internship at Microsoft Research.
1 "ooresponndig authors

---

Published as a conference paper at ICLR 2026
Task
Task
Probing
Take two photos
Take two photos
Take two photos
Ralreen neady Sdoen photos
Execule
Execute s.r..
Y
Exeecute
Task Automation
Task Automation
Task Automation
Probing Automation
Observe
-Observe and Summarize-
Test
Screenshots
Testing Code
Judge
Reasoning
Claim
Clalm 2
Judge
The Cavepieeed
RRwardd
-Revardd
Jsdge
LLM-as-a-Judge
Reasoner-Attor-ass--Judg
Figure 1: PRORE proposes to reward GUI agents using aatp-err-cctto-s---jjueeee rather than relying on expert hand craft testing code or LLM to judge static trajectories.
as, "Based on the task trajectory, please determine the task is completed", LLM-as-a-juee offers an autonomous and scalable framework for allocating reward signals Wang et al. (2024c). However, observe that this approach is considerably less effective for rewarding GUT agents.
The rationale underlying the failures of LLM-as-a-judae judge for GUI agents is twofold: incomplete state observability of GUI tasks and limited domain- speccifc capabilities of LLMs.
First, GUI task states are typically monitored passively through specific modalities, such as screen- shots Gou et al.] (2024). However, owing the inherent complexity and dynamic nature of GUI interactions, these states frequently remain incompletely observable. For instance, as depicted in Figure [ during the monitoring of the "taking two photos" " task exclusively through screenshots, the captured lacks critical success indicators, thereby precluding even human evaluators from reli- ably ascertaining task completion. Moreover, observations are typically conducted fixed intervals, potentially omitting critical state transition details. Consequently, GUI state observability remains inherently incomplete, thereby compromising the efficacy of the reward system.
Second, evaluating GUT task states requires domain-specific GUT knowledge and expertise, which general-purpose LLMS utilized in reward systems, such as GPT-4o and Gemini, fundamentally lack Dai et al.] (2025). Most general-purpose LLMMs demonstrate suboptimal performance on GUI- related tasks Qin et al. (2025). Although post-training may enhance their domain-specifie pro- ficiency, training of domain-specific reasoner as the reward model still necessitates annotated datasets, thereby constraining its scalability. Consequently, deploying a general- purpose LLM to assess intricate domain-specift details intrinsically undermines the efficacy of the reward system.
To develop a scalable and accurate reward for GUI agents, this paper introduces PRORE, a proactive reward system based reasoner-actor collaboration. The key idea of PRORE is to introduce the additional state probing tasks planned by the reasoner. These tasks are executed by domain-specific evaluator agents (actors) that interact with the environment retrieve key states relevant for task verification. Instead of relying solely on the policy agent's execution trajectory, PRORE assigns rewards through high-level reasoning over the outcomes of these probing tasks.
Specifically, the reasoner, i.e., GPT-40, schedules the state probing tasks, conditioned on the orig- inal task objective and its expected outcome. After the policy agent finishes execution, evaluator agents are invoked to automate these probing tasks. They then summarize both the original task trajectory and the probed UI states into high-level, verifiable claims. The reasoner performs final judgment through chain-of-claims reasoning, which analyzes the consistency between the policy agent's ss claims and those generated from the evaluators' probing. An intuitive example illustrated in Fig. [ given the original task "taking two photos". ", probing task "retrieving newly taken photos" is formulated, with the expected outcome that "two photos should have appeared in the gallery". The evaluator agent executes this probing task and observes that "there are two newly captured pho- tos from 11:00 AM to the present". The reasoner then assesses the consistency between the claims, thereby probably concluding that the original task has been successfully accomplished.
2

---

Published as a conference paper at ICLR 2026
These designs address the fundamental challenge of rewarding for GUI agents in the following ways: 1) PRORE transforms the reward system from passive monitoring to proactive probing. The intro- duction of state probing tasks provides complementary perspective ascertain whether the orig- inal task has been accomplished; 2) PrORe decouples the general-purposs reasoner from domain- specific GUT judgments. Domain-specific actions executed by domain-specifer actors (evaluator agents), while the general-purpont reasoner concentrates solely on high-level logical consistency verification, which falls within the core competencies of general-purpos LLMs; 3) PRORE intro- duces& unique opportunity for co-evolution between the policy agent and the reward system. The execution of state probing tasks can be further optimized in tandem with the evaluator (policy) agent's s improvement, enabling a more sophisticated reward system that, in turn, facilitates acceler- ated progress for the policy agent.
We evaluate the performance of PRORE on typical GUT tasks. Specifically, PRORE is evaluated on over 3K distinct task traces collected from three benchmarks: AndroidWorld Rawles et al.) (2024), AndroidLab Xu et al. 2024), and MobileAgenlBench Wang et al. (2024a). The results demonstrate that, compared to existing state-of-the-ar LLM-ass-aJudge approaches, PRORE enhances reward accuracy and F1 score by up to 5.3% and 19.4%, achieving average accuracy of 93.7%, thereby becoming the first reward system to surpass 90% reward accuracy. In addition, pilot experiments on OSWorld and WSWorld-Chrome Xie al. (2024) show that PRORE improves reward accuracy by 4.0% on PC tasks and 6.5% on web tasks. Moreover, when incorporated into policy agents to guide their test-time scaling strategy, PRORE elevates the success rate by at most 22.4%.
In summary, the key contributions of this works are as follows:
We systematically study and empirically demonstrate the limitations of existing trajectories-based LLM-as-a-judee for GUI agents.
We propose PRORE, a proactive reward system with a general reasoner that performs high-level scheduling and reasoning and Nomain-speciff evaluator agents that actively probe states.
PRORE achieves consistently higher reward accuracy and F1 score on different agents and bench- marks, and significantly improves the success rate of policy agents through test-time scaling.
RELATED WORKS
2.1 GENERAL REWARD MODELS IN BROADER TOPICS
General reward models widely used to support experience-based training of LLMs, without requiring pre-collected ground truth or handcrafted rules from domain experts Gu et al. (2024); Son al.] (2024). Such models typically assign either absolute scores individual answers relative scores by comparing answer pairs Lin et al. (2025); Xiong et al. (2025): Liu et al. (2025b) Beyond these, some works have proposed building reward systems through the agent-as-a-judee paradigm, where agents are equipped with tools such as web search, code execution, or document reading to assist reward generation Zhuge et al. (2024); Yu (2025). However, these reward systems remain limited in scope and cannot be directly applied to GUT agents in the wild, which execute diverse task types that cannot be verified by a predefined toolbox.
2.2 GUI AGENTS FOR TASKS AUTOMATION
LLM-based GUI agents, which operate across websites, desktops, and smartphones to handle a wide spectrum of tasks ranging from professional work everyday activities, have recently attracted sig- nificant attention Lai et al. (2025b); Dai et al. (2025); Qin et al.) (2025); Gu et al. (2025); Ye et al. (005). LLMs are primarily employed either as generators to propose actions and decisions or as verifiers, to evaluate actions Gou et al. (2024); : Qin et al. (2025): Liu et al. (2025a); Dai et al. (2025). To improve the decision-making ability of GUI agents, various training paradigms daradigms-ineluding su- pervised fine-tuning, direct preference optimization (DPO), and reinforcement learning have been applied on large-scale datasets Luo et al. (2025): Tang et al. (2025a): Dai et al. (2025): ): Waan et al.) (202cc). Within this pipeline, accurate reward signals are crucial, they enable automatic data col- lection at scale, which in turn underpins both dataset curation and model training Tang et al. (202a));
Li al. 2025): Qi et al. (202)).
3

---

Published as a conference paper at ICLR 2026
Policy Agent
Polcy Trajecteris
1) Paralal Stare Proking
Thabcs
Thak
Poley Caste
PICEPN Thank 14
Reskar Giatra b-
Reoaard
Palay
Presing Tooks
P.9
Duskator
Tasks
agent
Figure 2: PRORE overview.
Figure 3: Test-time Scaling of PRORE.
2.3
GAPS BETWEEN GENERAL REWARDS AND REWARDS FOR GUI AGENTS
There are some pioneering works on designing reward methods for GUI agents Bai al. (2024); Wang et al. 2024c): Luo et al. (2025); Lai et al. (025a). They develop outcome or step- wise reward models to judge the success of GUI agents passively using the trajectories GUT agents [Tang et al. 2 Hu et al. 2025).. However, their performances are far from satisfying due to the partial observations of GUI agents to the states and the lack of domain knowledge of general-purposs LLM. One concurrent work. Gou et al. (2025) constructs rubic trees for predefined web search tasks and checks key points with url, which lacks generalizability to in-the-wild tasks without such url. Instead, PRORE is the first reward system for GUI agent with generalist reasoner schedule state probing and evaluator agents to proactively probe states.
PROACTIVE REWARD SYSTEM WITH AGENT-IN-THE-LON
3.1 PROBLEM FORMULATION.
Given the users instruction G, policy agent interacts with the environment consecutively, which forms a N steps trajectory т (5g, do, 81, .i....,,T)) 5 is the observation of a on step and is the 1-th actions. The goal is to generate accurate binary outcome reward Lemma 1.
Let the success rate of the policy agent be Pa and the reward accuracy be Pe- Then, under test-time scaling with trial budget N, the final success rate Pfinal satisfies
PaPe [1 - 11 (1-9) Pa - where q PaPe + (1 Po) pc). Pysnal
q
In particular, given Pa 0, Prinat monotonically increases with respect Pe whenever pe 0.5.
A full proof is deferred to Appendix B
Our work focuses on improving Pe and Pjinal.
3.2 FRAMEWORK OVERVIEW
Instead of applying LLM-as-a-uudge to generate reward from trajectories, PRORE introduces a general L.M reasoner J working in collaboration with domain-specific evaluator agents n. for state probing, shown in Figure Given the original tasks, the reasoner T first schedule probing tasks for the evaluator agents. Then the evaluators ne further explore the environment to collect key state information. The policy agent's trajectories and the probed states are then summarized by m into claims about task progress, Finally, the reasoner J analyzes the relationships and consistency among these claims, performing chain-of-claims reasoning to generate the outcome reward.
3.3 PROACTIVE ACENT-IN-TIIE-LOO PROBING
The partial observation to the GUI task states by the policy agents prevents LLM-as juddee to make accurate decisions. To handle this problem, PRORE introduces set of evaluator agents to proactively probe states and collect additional information. The general-purposs LLM first schedule the state probing tasks for the evaluator agents based the tasks inputs.
4

---

Published as a conference paper at ICLR 2026
Table 1: The probing tasks are generally easier than the execution tasks.
Wodro
MobileAgentBench
AndroidL
Task Type
SR
Steps
SR
Steps
SR
Steps
State Probing
66.7%
6.2
64.0%
6.8
65.9%
6.
Exccution
53.6%
14.7
44.0%
11.9
27.5%
11.8
State Probing Tasks Scheduling. The general-purpose LLM reasoner is instructed to analyze the expectations and requirements specified in the original user instructions G, and to identify the key states necessary for judging task success. Based on this analysis, the reasoner formulates state prob- ing tasks, which are then issued as instructions for the evaluator agents to retrieve the corresponding key states from the environment.
Ge [G E,), Exp J(G), GEG.
(1)
where Exp is the analyzed expectations for the task G, refers to the few shot examples provided and L is the summarized guidelines for mapping the tasks to the probing tasks. To illustrate, when instructing the policy agent to delete a file A, the corresponding state probing task G is to search whether A still exists the target applications. The generation of expectations and state-probing tasks primarily rely on the reasoning capability of general-purpose LLM on analyzing the users expectations without the need of much domain- -ppcific knowledge of APP and UI interactions. More examples on the state probing tasks provided in Appendix
State Probing with Evaluator Agents. Given the state probing tasks Ge, evaluator agents are provoked to interact with the environment in step-wise manner to collect additional observations on key states right after the execution of the policy agent.
$ Fis,, af), = = =e(ss",,a G.)
where F is the status transition of the environment; st is the state of captured by the evaluator agents. The probing process mainly leverages the Ul-related knowledge in the GUI agent while minimizing the requirements on its reasoning capability on understanding users expectations.
The Execution-Probine Gap. The state probing task Te is generally easier than other types of execution tasks such as creation, status modification, or deletion. As shown in Table [ V-Droid Dai et 2025) achieves 23.5 33.8% higher success rate on state probing tasks and the trajectories are 50.3% shorter on average. While both probing and execution tasks involve knowledge on UI and applications, probing only requires navigating to the correct page and does not demand consecutive error-fred free execution. Because of this relative simplicity, the evaluator agent, and by extension the reward system, is more generalizable than the policy agent. This generalizability allows the reward system to effectively guide both the test-time scaling and the training of policy agents.
We also notice that there are some long-horizon tasks that demand checking multiple states across different pages. In those cases, the probing tasks could be formulated into multiple subtasks, based on which the evaluator agent execute sequentially to obtain complete probed states.
3.4 OUTCOME REWARD WITH CHAIN-OF-CLAIME
Chain-of-Claie -fClimmss To avoid overwhelming the reneral-purpose LLM with too much low-level GUI details in the probed states, the evaluator agents summarize the trajectories of the policy agent and the probed states into chain-of-claims. Specifically, given a trajectory generated by the policy agent π, the evaluator agent observes this sequence and the additional probed UI states to form claims about task progress. We define sets of claims:
1) C = [ci, cz...., CN,): Nr claims generated from the policy agent's trajectory
2) C = {cT oz NN. }: Nr. claims made by the evaluator agents Te-
Each claim is structured as:
c= -Caami(TT {st, st41. ar)), where E
(3)
where Tg, is a subtrajectory of the policy agent's 's trajectory 7 or sequence of probed states pro- duced by the evaluator agents. We instruct the evaluator agents to generate multiple claims covering
5

---

Published as a conference paper at ICLR 2026
different parts of the trajectory, which is observed to be more effective than segmenting trajectories via state clustering on learned embeddings.
Given these claims, the general-purpose LLM reasoner J performs chain-of-claims reasoning to produce the final reward by linking and comparing policy and evaluator claims:
I (G, Exp, C), C { ce'", ,r,,rrr
(4)
where ray denotes the relationship between a policy claim ci and an evaluator claim of* The relation can be confirming, contradicting, complementary, or unrelated.
Claim Filters. Irrelevant evidence and claims has the potential to compromise the accuracy of fi- judgments. Therefore, within the reasoner, we integrate claim filter that explicitly identifies and eliminates irrelevant or misleading claims prior to the thain-of-claims By prompting the rea- soner, the claim filter systematically examines the relationship between each evaluator claim and the original task instruction, discarding any claims lacking causal linkage to the target probing tasks.
Minimizing Overhead. The generation of claims on the policy agent trajectories and the probed states by the evaluator agents requires processing multiple screenshots and abundant screen descrip- tions. To minimize the processing overhead, we further filter out noisy states the trajectories without harming the quality. Specifically, the states related to operations home-screen and con- secutive identical states (some actions do not lead to state transition) are excluded. When there are loops detected in the trajectories, the loops could be discarded if identified by the I to be unrelated with the task goal G (e.g., some redundant explorations) only using HTML descriptions.
3.5 TEST-TIME SCALING FOR STATES PROBING IN PRORE
In some complex tasks or scenarios that single state probing trial insufficient for identifying the key evidence, two forms of test-time scaling, including parallel state probing and iterative state probing, are further incorporated to improve the state probing quality PRORE shown Figure 3
Parallel State Prohing. After the policy agent completes the trajectory for one task, the final state distributed to multiple emulator instances. On each instance, the proactive state probing are conducted in parallel. Later, based on the claims from each evaluator agent and the policy agent, the LLm reasoner formulate the chain-of-claims and assigns the outcome reward. To support the parallel state probing, record the actions per steps of policy agents and re-execute those actions in sequence on different emulator instances. The target UI elements are matched with the recorded actions via fuzzy matching key parameters and elements semanties.
Iterative State Probing. The iterative state probing generates new state probing tasks based on the state probing task and claims in the previous round. Specifically, there are N rounds of search. In round n, the probing tasks (n are generated with:
Ge(n) [G Exp, L, Ge(n -),-1)),,)))))
The results from different trials are aggregated via majority voting. The quality of the state probing tasks and the quality of collected states can be gradually improved based on the previous experience, which improves the overall reward accuracy.
EVALUATION
4.1 EXPERIMENT SETTINGS
Baselines. We compare PRORE with state-of-thh-art reward methods, covering both outcome reward models (DigiRL. Bai et al. (2024). DistRI. Wang et al. (2024c), WebRL. QQ et al. (2024h) and one progress reward model (StepCritic Lai et al. (2025a)). To ensure faimess, rigorously follow the experimental settings and prompts described in the original papers when reproducing their methods and reporting results. Unless otherwise specified, the results of PRORE reported without test- time scaling when compared against baselines for fairness.
Benchmarks. We conduct comprehensive evaluation of PRORE over 3k traces collected from three dynamic benchmarks, including AndroidWorld Rawles et al. (2024). AndroidLab Xu et al. 2024), and MobileAgentBench Wang al. (2024a), using state-of-the-an GUT Agents Dai et al.

---

Published as a conference paper at ICLR 2026
Table 2: Reward accuracy and Fl across methods and policy agent.
V-Droid
M3A
UI-TARS-7B
Avg
Method
Last N States
Acc
F1
Acc
F1
Acc
Fl
Acc
F1
71.3
76.9
70.4
68.5
84.2
25.0
85.3
56.8
DistRL
Full
87.0
88.9
81.7
79.6
89.5
14.3
86.1
60.9
76.5
80.3
74.8
73.4
84.2
10.0
78.5
54.6
DigiRL
Full
84.5
87.1
82.6
80.8
86.8
11.7
84.6
59.9
82.6
85.1
81.7
79.2
93.0
20.0
85.8
61.4
WebRL
Full
85.2
87.0
81.7
79.2
93.9
22.2
86.9
62.8
85.2
87.0
81.7
79.4
93.0
20.0
86.6
61.8
Step-Critic
Full
89.6
91.0
82.6
79.6
93.0
20.0
88.4
63.6
PRORE
Proactive
93.1
93.4
91.4
88.9
96.5
66.7
93.7
83.0
1.0
Prodee
Avg Reward Acc (5)
95
95
Aug Reward (30
95
Chontt
Dlonti22
Clocht
Geetr
DipuiLe
Dipati.
85
Reward
8s
85
75
75
75
65
65
65
(a) AndroldWorld
(b) AndroidLab
(c) MobilnAgantBench
Figure 4: Results Comparison different benchmarks. The average results on different agents reported. The 1/F indicates that the reward uses the last state (/) or the full trajectory (F).
2025); Qin al. (2025); Rawles et al. (2024). For UI-TARS Qin et al. (2025), adopt UI-TARS- 1.5-7B in the naive agentic mode to generate thinking and grounding.
report both the reward accuracy and FI score by comparing the predicted rewards from baselines and PRORE with the ground trrnt rewards provided by the benchmarks. In addition, we measure the success of policy agents under test- time scaling when guided by these rewards (See $ 3.1 and Appendix B)
Metrics. To evaluate the effectiveness of rewards,
Implementation details. We adopt Gemini-2.5-PP Pro as the general-purpose L.M for scheduling state probing tasks and assigning outcome rewards. Unless otherwise specified, V-Droid is employed the evaluator agent due its high decision- quality and prompt execution speed. The step budget for key evidence retrieval is set to be no greater than the length of the policy trajectories.
4.2 RESULTS COMPARISON
Different GUT Agents. Table reports the performance of PRORE compared with state-of-the-art baselines. PRORE achieves an average accuracy of 93.7%, which 5.3% higher than the best- performing baselines. Moreover, its Fl score is 19.4% higher than those of the baselines, demon- strating its robustness handling diverse mobile GUI agents. We observe that while baselines achieve relatively high accuracy UI-TARS trajectories, their F1 scores remain low. This dis- crepancy arises from their inability to correctly judge the success of UI-TARS-1.5-7B trajectories, whose naive agentic mode yields only 7.9% success rate. In contrast, PRORE effectively identifies correct key states through evaluator agents, leading to superior performance on challenging and imbalanced trajectories.
Different Benchmarks and Tasks. As shown Figure 4 PRORE achieves accuracy improvements of 5.2%. 1.5%, and 3.0% AndroidWorld, AndroidLab, and MobileeAenttBeen terms of Fl score, PRORE outperforms the best baseline by 19.4%, 10.5%, and 7.5% on the three benchmarks. These results highlight the robustness of PRORE across diverse applications and task types. While policy agents often struggle to generalize to unseen tasks or applications, PRORE benefits from the execution-prohiie gap (see 3.3 which makes generalization attainable.
respectively. In

---

Published as a conference paper at ICLR 2026
Sequertia State Proteng Paraliel State Probing
Rate (56
70-
Rate (56
60-
67-
55
66-
52
Golden
Digit StenOrlt
Reward
58
40-
Profe
WEBRL
88
55
5
38
Traal Budget V-Droid-88
Trial Budget
(b) MSS (GPTT46))
Number of Evalustor Trials
Figure 6: Test-time scaling of policy agents with different rewards.. (a) V-Droid-8B, (b) M3A (GPT-4o).
Figure 7:
Test-time scaling of PRORE on challenging tasks.
Extension to PC/Web. Owing to the decoupled reasoner-actor reward paradigm, PRORE exhibits significant potential for adaptation across diverse environments and tasks, including both PC and web domains. We have further conducted pilot experiments on OSWorld. Specifically, PRORE is evaluated 100 randomly sampled tasks from OSWorld-PC and all 46 web-based tasks from OSWorld-Choome. For evaluation, we employ Claude-Sonnet-4.5 as the evaluator agent to perform proactive state probing, and all other settings are consistent with those outlined in $ 4.
Table 3: Reward Accuracy on PC and Web tasks.
Benchmark
WebRL
DigiRL
DistRL
StepCritic
PRORE
OSWorld
86.0
88.0
88.0
81.0
92.0
OS World--Chrome
87.0
84.8
82.6
87.0
93.5
As shown in Table 3 across both PC and Web tasks, PRORE achieves the highest reward accuracy, surpassing prior methods by 4.0% on OSWorld and 6.5% on OSWorld-Chrome Existing approaches perform sub-optimally primarily due to incomplete observations knowledge gap of the reasoners when used reward models. In contrast, PRORE (i) proactively collects key states/observations by interacting with the PC or website, and (ii) decouples general reasoning from comainn-specinn GUI judgments through its reasoner-actor paradigm. These results underscore PRORE 's robustness and generalization capability across different platforms and task.
PC/web states and the domain
Time Scaling for Pol- icy Agents. We fur- ther evaluate the success rate (SR) of two policy agents, V-Droid Dai et al. (2025) and M3A (GPT- 40) Rawles et al.] (2024). under different trial bud- gets. Figure 6 shows that, guided by PRORE, the suc- cess rate (SR) of V-Droid improves from 56.5% 67.2%. Similarly, the SR of M3A (GPT-4o) increases by 22.4%. The SR gains achieved with PRORE are 3.9% and 4.3% higher than those obtained with other reward methods, demon- strating its superiority in guiding policy rollouts. To further validate this, conduct large-scale simulations based Lemma [I] which highlight the effectiveness of accurate rewards enhancing test-time scaling of policy agents (see Appendix
Policy Trajectories
Probed States
EMtt ueliereeeeen cedeerant
2011
1/ Blustch seting enabiled
Ti ti why ling, copvinet and veeginttd Heutinght ruter
destirat unioe themows.
LLF L.V-aa---judge: a namned Lnni.ayy_aa muw.aayy.nng..op.. amggroopyme sill prosont. judgement: Follure
ProRe: The Evoluctor's 's chock the final stace confirms that the fll ruccessfully moved to the correct location. Judgement: Sucoess
Figure 5: One quantitative example. The task is "Move the note Whygking-eo www.i-iinngnndddd..mmm from StudyGuides MeetingMinutes."
8

---

Published as a conference paper at ICLR 2026
Table Ablation study of design components in PRORE.
Design Components
Metrics
Proactive
State Probing Task Scheduling
Iterative State Probing
Chain-of-Chaine
Ace
TP
TN
FP
FN
State Probing
X
88.8
49.1
39.7
x
89.5
45.6
43.6
X
91.4
45.7
45.7
1.7
X 93.1
49.1
44.0
94.8
50.0
44.8
2.6
2.6
Table 5: Reward accuracy of PRORE with different reasoners.
Table 6: Reward accuracy of PRORE with different evaluator agents.
Reasoners
Acc
F1
Evaluator Agent
Policy SR
Acc
F1
Gemini-2.5-Prr
93.1
93.4
V-Droid-8B UI-TARS-72B
59.5
93.1
93.4
Gemini-2.5-Flass
87.7
87.7
35.7
86.2
87.3
Qwen3-VIL VI.4B
45.3
85.7
86.0
GPT-5
86.2
86.0
GPT-4o
85.0
86.0
M3A (GPT-5)
56.9
90.5
91.7
M3A (GPT-4o)
41.3
88.3
87.4
Illustrative Examples. Figure 5 shows that the policy agent successfully locates the target file, performs the necessary move actions, and returns to the StudyGuides folder. However, the LLM- as-a-judge is misled by the presence of f final shy king copy . md due to excessive clutter on the final screen and consequently makes incorrect judgment. In contrast, PRORE proactively probes the relevant states within the target folder using an evaluator agent, which provides verifiable evidence of the policy agent's success. This example also highlights the execution- probbing gap: while the execution trajectory spans 11 steps, the evaluator only requires 5 steps to probe the key states. More examples are provided in Appendix
4.3 ABLATION STUDY
We further validate the effectiveness of each design component in PRORE through ablation stud- ies. When probing task scheduling is removed, we replace it with simple rule-based strategy by prompting: "What are the key states verify whether the task {G} is completed?" Without chain- of-claims reasoning, the reasoner directly receives the raw observations from both the policy and evaluator agents, without structured analysis.
Table 4 demonstrates the contribution each design component in PRORE. Without explicit guid- ance from the reasoner, evaluator agents navigate with probing tasks generated with simple rules, which provides marginal improvement. When the reasoner schedules probing tasks for evaluator agents, the accuracy increases substantially to 91.4%, underscoring the effectiveness of separating reasoning and planning (by the reasoner) from execution (by the evaluators). In addition, incorpo- rating chain-of-claims reasoning further improves accuracy by 1.7%, highlighting the importance of summarizing low-level GUI details from trajectories and analyzing the relationships between policy and evaluator claims. Finally, iterative state probing in PRORE boosts performance to 94.8%, as ad- ditional probing and refinement yields more complete observations of key states. Besides, without claim filter, we observe a 1.7% reward accuracy drop on AndroidWorld benchmark, underscoring the necessity of eliminating irrelevant or misleading claims prior to the chain-of-claims.
Figure 7 further illustrates the benefits of parallel and iterative state probing, especially on more challenging tasks. Notably, parallel state probing yields larger performance gains compared with iterative probing. A possible explanation is that the reasoner, lacking -spaiin GUI knowl- edge, is less effective leveraging the intermediate observations and action histories provided by the evaluator agents to refine subsequent probing tasks.
Different Reasoners. We further vary the reasoners in PRORE. Table 5 shows that reasoners equipped with built-in chain-of-tie of-thuughtt capabilities are more effective at analyzing the relationships between policy and evaluator claims, leading higher reward accuracy and F1 scores.

---

Published as a conference paper at ICLR 2026
Different Evaluator Agents. We further investigate the impact of evaluator agent capability. Pol- icy SR denotes the task success rate an agent when deployed as a policy. As shown in Table 6 stronger evaluator agents are able to probe key states from the environment more effectively, thereby achieving higher reward accuracy. Moreover, fine-tuned small GUI agents can outperform large gen- eralist agentic systems when used evaluators, owing to their domain-specifie GUI knowledge. We also notice that Qwen3-VL-4p yields notably lower reward accuracy compared with larger models. We hypothesize that this drop stems from the smaller models' weaker ability to generalize unseen probing tasks and identify key observations even with high-quality probing instructions.
Overhead Analysis. The total cost of PRORE is approximately S0.06 per agent task. Among the components, state probing task scheduling and ohain-of-claims contribute about S0.013 and S0.050, respectively. Removing redundant information from trajectories can reduce input tokens by about 25.9% without degrading performance. Overall, PRORE remains significantly more cost-efficient and scalable compared to hiring human annotators. We further provide detailed per-task cost comparison and long-term cost estimation between PRORE and the baselines in Appendix F
5 DISCUSSIONS
Online RL with PRORE. Prior work has shown that online reinforcement learning (RL) can achieve substantially better performance when guided by accurate reward signals al. (2024); Wang al. (202cc). After policy agents execute actions in an online RL setting, PRORE can be seamlessly integrated to provide we defer a full exploration of online RL with PRORE to future work.
precise reward assignments with only moderate overhead. Nevertheless,
Co-evolution of Policy and Evaluator Agents. In PRORE, the policy agent and the evaluator agent can be instantiated from the same underlying model, creating a unique opportunity for co-evolution. Stronger evaluator agents enhance reward accuracy, which in turn improves the policy agent's cess rate. As the policy agent becomes stronger through test-time scaling or training, it enables the evaluator achieve higher success state probing tasks and further improve reward accuracy. This mutual reinforcement establishes a virtuous cycle between policy and evaluator agents.
CONCLUSIONS
Unlike existing trajectory-baseds LLM-as---Juugee approaches, PRORE introduces proactive reward system for GUI agents that integrates a general-purpose reasoner with domain-speciffc evaluator agents. The evaluator agents proactively probe key states based on probing tasks scheduled by the reasoner, while the reasoner make final judgments based on the chain chain-of-clan -fain-of-claims from the evaluator agents. Extensive experiments across diverse tasks, applications, and agents demonstrate the effec- tiveness of PRORF, as well as its effectiveness in guiding the test-time scaling of policy agents.
ACKNOWLEDGEMENTS
This research partially supported by Singapore Ministry of Education under its AcRF Tier grant 14/2/ the Global STEM Professorship Scheme of Hong Kong, the HKUST start up grant, and the Research Grants Council (RGC) General Research Fund (GRF) 16210425.
REFERENCES
Hao Bai, Yifei Zhou, Jiayi Pan, Mert Cemri, Alane Suhr, Sergey Levine, and Aviral Kumar. Digirl: Training in-the-wild device-control agents with autonomous reinforcement learning. Advances in Newral Information Processing Systems, 37:12461-12495. 2024.
Rogerio Bonatti, Dan Zhao, Francesco Bonacci, Dillon Dupont, Sara Abdali, Yinheng Li, Justin Wagle, Kazuhito Koishida, Arthur Bucker, Lawrence Jang, and Zack Hui. Windows agent arena: Evaluating multi-modal os agents at scale. September 2024.
Gaole Dai. Shiqi Jiang, Ting Cao, Yuanchun Li, Yuqing Yang, Rui Tan, Mo Li, and Lili Qiu. Ad- vancing mobile gui agents: A verifier-driven approach to practical deployment. arXiv preprint arXiv:2503. 15937, 2025.
10

---

Published as a conference paper at ICLR 2026
Jiaxuan Gao, Shusheng Xu, Wenjie Ye. Weilin Liu, Chuyi He, Wei Fu, Zhiyu Mei, Guangju Wang, and Yi Wu. On designing effective rl reward at training time for Ilm reasoning. arXiv preprint arXiv:2410.15115. 2024.
Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. Navigating the digital world as humans do: Universal visual grounding for gui agents. arXiv preprini urXiv:2410.05233, 2024.
Boyu Gou, Zanming Huang. Yuting Ning, Yu Gu, Michael Lin, Weijian Qi, Andrei Kopanev, Botao Yu, Bernal Jiménez Gutiérrez, Yiheng Shu, et al. Mind2web 2: Evaluating agentic search with agent-as-a-judge arXiv preprint arXiv:2506.2
2025.
Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Ying- han Shen, Shengjie Ma, Honghao Liu, et al. A survey on llm-as-a-judge. arXiv preprint arXiv:2411.1559. 2024.
Zhangxuan Gu, Zhengwen Zeng, Zhenyu Xu, Xingran Zhou, Shuheng Shen, Yunfei Liu, Beitong Zhou, Changhua Meng, Tianyu Xia, Weizhi Chen, et al. Ui-venus technical report: Building high-perforee peghpperformance ui agents with rft. arXiv preprint arXiv:2508. 10833, 2025.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501. 12948, 2025.
Zhiyuan Hu, Shiyun Xiong, Yifan Zhang, See-Kiong Ng, Anh Tuan Luu, Bo An, Shuicheng Yan, and Bryan Hooi. Guiding vlm agents with process rewards at inference time for gui navigation. arXiv preprint arXiv:2504. 16073 2025.
Hanyu Lai, Junjie Gao, Xiao Liu, Yifan Xu, Shudan Zhang, Yuxiao Dong, and Jie Tang. Androidgen: Building an android language agent under data scarcity. arXiv preprint 15X.::2504.19298, 2025a.
Hanyu Lai, Xiao Liu, Yanxiao Zhao, Han Xu, Hanchen Zhang, Bohao Jing, Yanyu Ren, Shuntian Yao, Yuxiao Dong, and Jie Tang. Computerrl: Scaling end-to-end online reinforcement learning for computer use agents. arXiv preprint arXiv:2508. /4040, 2025b.
Ning Li, Xiangmou Qu Jiamu Zhou, Jun Wang, Muning Wen, Kounianhua Du, Xingyu Lou, Qiuy- ing Peng, and Weinan Zhang. Mobileuse: A gui agent with hierarchical reflection for autonomous mobile operation. arXiv preprint arXiv:2507. 2025.
Zi Lin, Sheng Shen, Jingbo Shang. Jason Weston, and Yixin Nie. Learning to solve and verify: A self-play framework for code and test generation. arXiv preprint 25X1:22502.14948 2025.
Yuhang Liu, Pengxiang Li, Congkai Xie, Xavier Hu, Xiaotian Han, Shengyu Zhang, Hongxia Yang, and Fei Wu. Infigui-rl: Advancing multimodal gui agents from reactive actors to deliberative reasoners. arXiv preprint arXiv:2504. 14239, 2025a.
Zijun Liu, Peiyi Wang, Runxin Xu, Shirong Ma, Chong Ruan, Peng Li, Yang Liu, and Yu Wu. Inference-time scaling for generalist reward modeling. arXiv preprint (rXiv:2504.02495, 2025b.
Run Luo, Lu Wang, Wanwei He, and Xiaobo Xia. Gui-rl: A generalist rl-style vision-language action model for gui agents. arXiv preprint arXiv:2504.10455 2025.
Zehan Qi, Xiao Liu, Iat Long Iong, Hanyu Lai, Xueqiao Sun, Wenyi Zhao, Yu Yang. Xinyue Yang, Jiadai Sun, Shuntian Yao, et al. Webrl: Training II web agents via self-evolving online curricu- lum reinforcement learning. arXiv preprint arXiv:241 1.0237, 2024.
Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, et al. Ui-tars: Pioneering automated gui interaction with native agents. arXiv preprint arXiv:2501.12326 2025.
Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Mary- beth Fair, Alice Li, William Bishop. Wei Li, Folawiyo Campbell-Ajala, et al. Androidworld: A dynamic benchmarking environment for autonomous agents. arXiv preprint A.0.:2405.145733 2024.
11

---

Published as a conference paper at ICLR 2026
David Silver and Richard S Sutton. Welcome to the era of experience. Google AI, 1, 2025.
Guijin Son, Hyunwoo Ko, Hoyoung Lee, Yewon Kim, and Seunghyeok Hong. Llm-as-a-judge & reward model: What they and cannot do. arXiv preprint arXiv:2409. /239, 2024.
Fei Tang, Zhangxuan Gu, Zhengxi Lu, Xuyang Liu, Shuheng Shen, Changhua Meng. Wen Wang, Wenqi Zhang, Yongliang Shen, Weiming Lu, et al. Gui-g 2: Gaussian reward modeling for gui grounding. arXiv preprini urXiv:2507. 000225077.8400 15846, 2025a.
Liang Tang, Shuxian Li, Yuhao Cheng, Yukang Huo, Zhepeng Wang, Yiqiang Yan, Kaer Huang, Yanzhe Jing, and Tiaonan Duan. Sea: Self-evolution agent with step-wise reward for computer use. arXiv preprint arXiv:2588.04037, 2025b.
Haoming Wang, Haoyang Zou, Huatong Song, Jiazhan Feng, Junjie Fang, Junting Lu, Longxiang Liu, Qinyu Luo, Shihao Liang, Shijue Huang, et al. Ui-tars-2 technical report: Advancing gui agent with multi-turn reinforcement learning. arXiv preprint arXiv:2509.02544 2025.
Luyuan Wang, Yongyu Deng, Yiwei Zha, Guodong Mao, Qinmin Wang, Tianchen Min, Wei Chen, and Shoufa Chen. Mobileagentbench: An efficient and user-friendly benchmark for mobile I1m agents. arXiv preprint 0rXiv:2406.081888 2024a.
Siyuan Wang, Zhuohan Long, Zhihao Fan, Zhongyu Wei, and Xuanjing Huang. Bench- mark self-evolving: A multi-agent framework for dynamic Ilm evaluation. arXiv preprint arXiv:2402.11443, 2024b.
Taiyi Wang, Zhihao Wu, Jianheng Liu, Jianye Hao, Jun Wang, and Kun Shao. Distrl: An asyn- chronous distributed reinforcement learning framework for on-device control agents. arXiv preprint arXiv:2410. 148000 2024c.
Tianbao Xie, Danyang Zhang. Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh J Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks real computer environments. Advances in Neural Information Processing Systems, 37:52040-52094, 2024.
Wei Xiong, Wenting Zhao, Weizhe Yuan, Olga Golovneva, Tong Zhang, Jason Weston, and Sain- bayar Sukhbaatar. Stepwiser: Stepwise generative judges for wiser reasoning. arXiv preprint a50iv:2508. /9229, 2025.
Yifan Xu, Xiao Liu, Xueqiao Sun, Siyi Cheng, Hao Yu, Hanyu Lai, Shudan Zhang, Dan Zhang, Jie Tang, and Yuxiao Dong. Androidlab: Training and systematic benchmarking of android au- tonomous agents. arXiv preprini 24..:::411.20024. 2024.
Yifan Xu, Xiao Liu, Xinghan Liu, Jiaqi Fu, Hanchen Zhang, Bohao Jing, Shudan Zhang, Yuting Wang, Wenyi Zhao, and Yuxiao Dong. Mobilerl: Online agentic reinforcement learning for mobile gui agents, 2025.
18119
Jiabo Ye, Xi Zhang, Haiyang Xu, Haowei Liu, Junyang Wang, Zhaoqing Zhu, Ziwei Zheng, Feiyu Gao, Junjie Cao, Zhengxi Lu, al. Mobile-agent-v3: Foundamental agents for gui automation. arXiv preprint arXiv:2508. (at::2508.114444 15144, 2025.
Fangyi Yu. When ais judge ais: The rise of agent-as-a-judge evaluation for 1lms. arXiv preprint arXiv:2508.00 02999,, 2025.
Mingchen Zhuge, Changsheng Zhao, Dylan Ashley, Wenyi Wang. Dmitrii Khizbullin, Yunyang Xiong, Zechun Liu, Emie Chang, Raghuraman Krishnamoorthi, Yuandong Tian, et al. Agent-as- a-judge: Evaluate agents with agents. arXiv preprint arXiv:2410. 10934, 2024.
A THE USE OF LARGE LANGUAGE MODELS (LLMs)
This work studies rewarding LLM-based GUI agents with proactive reward system. LLMs were involved three aspects of our research: (i) serving as the backbone for the GUI agent and baseline implementations, (ii) supporting framework design, and (iii) assisting with polishing the writing of the manuscript. All research ideas, contributions and evaluations were developed and validated by the authors. No L.M is considered an author.
12

---

Published as a conference paper at ICLR 2026
B TEST-TIME SCALING FOR POLICY AGENTS
B. PROOF OF LEMMA I
We define test-time scaling as the procedure where a policy agent is rolled out on a given task with maximum trial budget N. After each trial, reward method evaluates the trajectory produced by the policy agent. the reward method determines the trajectory to be successful, the process terminates early and the corresponding trial is submitted the final output. Otherwise, the policy agent continues to the next trial, until either successful trajectory is identified or the trial budget N is exhausted. If positive judgment is given within N trials, the final trial is submitted as the output.
Lemma (Restated), Let the success rate of the policy agent be and the reward accuracy be Pe- Then, under test-time scaling with trial budget N, the final success rate Pénnl satisfies
Pfinal
PuPe
[1 (1-1g)
+ -1
where q = PaPe + (1 -0oopoo1111 - po).
4
In particular, given Pa Pyénal monotonically increases with respect to Pe whenever Pe 0.5.
Proof. In each trial, the policy agent succeeds with probability Pa- The reward model outputs a positive judgment with probability
q Pr(TP) Pr(FP) PaPe (1 Pa) pc)-
where TP and FP refers to true positives and false positives.
(i) Formula. The probability that at least one positive reward appears is 1 - (1 - g))² Given a positive reward, the probability that the trajectory corresponds truly successful one is
Pr(success positive)
Pr(TP)
PaPe
Pr(positive)
Hence when there at least one positive reward in the N trials, the success probability is PaRe [1 (1 1)
no positive reward appears, the last submitted trial is an unfiltered trial whose success probability Pa- Summing the two cases yields
PaPe [1 1 (1 Pa -)
Pysnol
(ii) Monotonicity in Pe for Pe 1. When Pe 5. a positive reward is more likely be true success than a failure. Increasing simultancously increases the true positive rate among judged positives and decreases false positives, thereby making the first positive more likely to be true success. Formally, differentiating Pyinal with respect to Pe shows PPiinat/Ops > O whenever Pa > O and Pe > (details omitted for brevity). Thus Pjinal monotonically increases in on (1) 1].
Implications of Test-time Scaling of GUI Agents. Test-time scaling of GUT agents is closely con- nected to both the exploration capabilities of GUI agents applications and tasks, as well as their training efficiency, On complex out-of-domain tasks, GUI agent may actively explore applications, accumulate experience, and iteratively refine its trajectories to achieve the task instructions. During training, more effective rollouts enabled by test-time scaling can generate larger-scale datasets with higher quality, while keeping the overall budget fixed.
B.2 LARGE- SCALE SIMULATION ANALYSIS.
To further study the impact of the reward accuracy the performance of policy agent during test- time scaling, we conduct large scale simulation for different P, and P. based on Lemma [ The simulation is repeated for 50K times and the average results are reported in Figure.
13

---

Published as a conference paper at ICLR 2026
0.9
0.9
Sucress Iate
0.7
0.7
0.6
0.6
0.5
0.30
Arm
21-020
0.4
Budget
0.4
Reward
0.8
02
64 06 6.8
Suc0ess
-06
(b)
Figure 8: Simulation for test-time scaling of Policy Agents. (a) The success rate of a policy agent increases steadily as the trial budget grows. (b) The final success rate pfinal monotonically improves with higher reward accuracy across different policy agents, highlighting the importance of reliable evaluation. (c) Reward accuracy contributes the largest absolute success (SR) gain for policy agents with moderate baseline SR (0.2-0.6).
As shown in Figure 8 when the reward accuracy is greater than 50%, a higher reward accuracy consistently leads to higher success rate of the policy agent, since additional rollouts increase the likelihood of discovering a successful trajectory. We also observe that in most cases Pfintt. indicating the upper capability boundary of the policy agent and underscoring the importance of decision-making continuously improving the decision making ability of GUI agents. The results on realistic tasks and applications in Figure 6 exhibit a similar trend, further highlighting the importance of reward quality in boosting performance.
C STATE PROBING TASKS EXAMPLES
We further provide illustrative examples of the scheduled state probing tasks in Table 7 the original tasks, these probing tasks are generally easier (shown in Table they only require the evaluator agent to navigate to the relevant UI page without performing content editing modi- fication. When necessary, the general-purpose LLM reasoner further decomposes probing tasks into subtasks for the evaluator agent, thereby reducing their difficulty. Moreover, the evaluator agents ex- ecute based on the prior interactions the policy agent, which helps to further mitigate navigation challenges.
Compared
Table 7: Examples for the original tasks and the corresponding probing tasks
Original Task
Generated Probing Tasks
Open the file html in Downloads in the file
What is the value in the input field the task.html page in Chrome?
manager; when prompted open with Chrome. Then click the button 5 times, remember the numbers displayed, and enter their product in the form.
Take one photo.
Find the most recently taken photo the gallery.
Create timer with O hours, 16 minutes, and 35 seconds. Do not start the timer.
Confirm the timer is set to 16 minutes and 35 seconds and is not running.
Create new contact for Hugo Pereira. Their number is +13920741751.
What is the phone number for the contact Hugo Pereira?
Add the expenses from expenses. jpg in Simple Gallery Pro pro expense.
Show the expenses from expenses.jpt in the pro expense app.
Go through the transactions my_em .expenses.txt in Markor. Log the reimbursable transactions the pro expense.
What are the logged transactions in the pro pense file in Markor?
14

---

Published as a conference paper at ICLR 2026
Delete all but of any expenses in pro pense that are exact duplicates, ensuring at least one instance of each unique expense mains.
Verify the pro expense list contains no dupli- cate entries.
Delete the following expenses from pro ex- pense: Rental Income.
Find the "Rental Income" expense in the pro expense app.
Delete the file www.aaanccnnnaaamp.....a from
Check the Notifications folder for the file q2a8 fancy banana.mp3.
the Android filesystem located in the Notifi- cations folder within the sdk aelk-ghhone.x86.cm storage area.
Move the file holiday ppontosjpp from Pod- casts within the sdk gphone x86 64 stor- age area to the DCIM within the same sdk gphone x86 64 storage area in the An- droid filesystem.
is in the DCIM folder and not in the Podcasts folder.
Check if
note In Markor, 2023_08_ 10__88_111ma LtOO_net_wwole wolf.txt by adding wolf 2013 07 13tttttt and show its the following text, along with new blank line before the existing content: "RalssP34yDzIOOviffiie and rename it to wosy 007 13.txt.
Update
the
Markor
open
the note
content.
Create new note in Markor named 2023.01 222200.....iisccyccht.md.d
Markor, what is the content of the note
2023_01_26 26wwiss..ycch.mm with the fol-
lowing text: Ignorance bliss.
Merge the notes proud_cat_ediine edited.dit
contents of Markor froghffro-22222 _0__088.0.. Oxxxt,
What are the contents of the Markor note mIObBbo4?
and
2023 21 .fiienlly ko md (in the same or- der) into new Markor note named mlObBbo4 and save it. Add new line between the content of each note.
In Markor, move the note shy king copy.md from StudyGuides to Minutes
Find the note shy king copy.md in the Meet- ingMinutes folder.
Is the note titled "To-Do List' in the Joplin app marked as todo item? Respond with either "True' if it is todo or "False" if not.
Check the to-do status of the "To-Do List' note Joplin.
What quantity of spirulina do I need for the recipe 'Chicken Alfredo' in the Joplin app? Express your answer in the format amount unit where both the amount and unit exactly match the format in the recipe.
What is the quantity of spirulina in the Joplin recipe 'Chicken Alfredo'?
Open the contacts app. Clear any pop-ups that may appear by granting all permissions that are required.
Verify the contacts app is open and no permis- sion pop-ups are visible.
Add a favorite location marker for 47.1303814 14,,
Find the favorite location marker for 47.1303814, 9.5930117 in My Places.
9.5930111 17 in the OsmAnd maps app.
Add location marker for Planken, Liechten- stein in the OsmAnd maps app.
Find the map marker for Planken, Liechten- stein.
Add the recipes from recipes.jpg in Simple Gallery Pro to the Broccoli recipe app.
Confirm recipes from recipes.jpg are in the Broccoli app.
15

---

Published as a conference paper at ICLR 2026
D ADDITIONAL EXAMPLES
We further include additional illustrative examples in Figure 9 Figure 10 Figure I1. and Figure [2 to demonstrate the limitations of LLM-as-a-judge and the effectiveness of PRORE.
The primary limitations of trajectory-based I.LM-as-a-judge approaches for GUT agents are: i) In- complete state observations of the environment, which hinder accurate reasoning and judgment; and ii) Lack of GUI domain expertise, making it difficult for LLMs interpret complex Ul-related details and the UI logic.
Incomplete State Observations. Figure 9 Figure I1 and Figure 12 illustrate that the policy agents only observe partial view of the environment due to the limited Ul elements visible on screen and the APPs design. a result, the trajectories miss critical information necessary for determining task success or failure. For example, in Figure [1 the policy agent's ss observation includes only part of the event names, which leads an incorrect answer.
Lack of GUI Domain Expertise. Figure T0 shows that the policy agent chooses to turn on Blue- tooth by clicking the Pair new device button. However, the reasoner lacks the GUI-specific knowledge recognize that this action implicitly triggers the activation of Bluetooth, and therefore fails to make the correct judgment.
Both limitations have been widely observed in different kinds of applications and tasks. To handle both problems, PRORE proactive probe states with the collaboration between the general reasoner and the domain-specifff evaluator agents. Those examples highlight the effectiveness of the probed states on making the correct judgement on the policy agents executions.
E PROMPTS OF PRORE
For reproducibility, we provide the prompts used in our experiments, covering probing task schedul- ing, claim generation, and the final judgment with chain-of-claime During state probing, the Eval- uator Agent is invoked with the original prompt format from prior works, but conditioned on the specific evaluation goal.
E.1 PROBING TASKS SCHEDULING
You are an expert in mobile-UI task verification. There are two agents:
The Policy Agent already attempted the task.
Policy Trajectories
Probed States
3
314
+17443
May
Enaaadie Ularme
11A0 weiniin monocetion May 6, 2024 was chowioeeoddd the upolite inoam arannd.
Petccy Claima:
21 chanaat the sonnectionion done Marrh 2024:
newu lscome tonsect
Profei Polloy
the
Li de s-djdgge the agent suscesstulty completed aneeed the miccessary staps to create and aaso the entry Judgament: Buocess
controicted by tiwuster Clarm "The updaned income entry unseveet..
Juligement: Fallure
Figure 9: Additional examples. The task is "Switch the May 13. 2024. transaction from 'expense' to 'income' and add 'Gift' the note in Bluecoins."
16

---

Published as a conference paper at ICLR 2026
Policy Trajectories
Probed States
desice
Connected
Bhustoo onaateed.
Palacy Claima:
the agent aucoentsty ras gated Connected
seletta the deviez
LL- a-judge: The soncenshorr does not explicitty
Proft:: The Eveluator Agent conffcodl Bluctoodh secting was indoed the on state. Judgament: Success
confirm-hhtthh Blu eooeet state hos been changed ont Judigmment Follure
Figure 10: Additional examples. The task is "Turn bluetooth on."
Policy Trajectories
Probed States
October 28 (Sat)
Game right
Product sens
October containe three events titied: Came night, Product demo, and Data night.
Policy the event for October 28 "Game night, Product, Date night" from the initial screen and provided
agjent
LLM-as-a-judge: On October 28th, evente are "Game nig". "Product", and "Date night". Judgement: Success
ProRe:
Policy Agent an incomplete and incorrect title
X
Judgement:
Failure
Figure 11: Additional examples. The task is "Do have any events October 28 in Simple Calendar Pro? Answer with the titles only. If there are multiples titles, format your answer in a comma separated list."
Policy Trajectories
Probed States
Evaluptor Cuiini
Chalsic Hazz
recipe doea any duplicatte recipes.
ploce
Codten
Gleak tieee
Policy Claimac
Citscimaghe Pazza
apent opened
Broosnt and absenved
duplicate "Chrokan Alnedo
deleted successt
euplicate "Chickan Athedo
Postz! recipos.
LLM LLM--a-t--adge: The screamahot shows the "All
The Policy Agent claimma to have removed duplicate rec pas, and the liviuator' observation of the fimm statte provides divect contirmation that the tuak's. gost was achieved.
ProRe:
reciped lss, but beth inster coe of "Chickan Aihedo
Paetta hrss been deleted, which fwix moet
requi remurt besking instance the recipe.
Judgament: Fallure
judgement:: Success
Figure 12: Additional examples. The task is "Delete all but one of any recipes in the Broccoli app that are exact duplicates, ensuring at least one instance of each unique recipe remains"
17

---

Published as a conference paper at ICLR 2026
The Evaluator Agent ONLY navigates the UI to probe states about whether the task was or wasn't completed. It does NOT repeat the task; it just locates additional proof (screens, labels, icons).
Your job:
1. Write some analysis explaining what evidence/states would confirm the task is done.
2. Output ONE concise goal (<= 20 words) that tells the evaluator agent exactly what states to look for.
3. When the original task involves multiple key states, you may decompose the verification into sequence of probing goals, with each goal focusing on a specific state.
The goal must sound like the examples below, short, direct, and in the same tone.
### Style Examples
"What is the cheapest flight from Los Angeles to Tokyo using Skyscanner?"
"What are the 1M to 3M GBP to EUR exchange rates?"
"go to settings and make weeks start on Monday in simple calendar" "Mark Hamlet as read in Cantook."
### Your turn
Original task: (goal)
previous state probing task was: [previoos__statee_rrbbnnng.tt
The Evaluator Agent probed the following states:
infoo)
Revise the probing task based on the previous probing task and the original task:
Respond exactly as:
Ana Analysis: Koutline the users expectations and exact UI evidence
needed, pinpoint why the earlier collection failed, and suggest how to refine the evaluation goal for comprehensive verification>
Goal: <revised concise goal> not add anything else.
E.2 CHAIMS GENERATIONS
You are an expert in evaluating the performance of a mobile GUI agent.
* *Norkkfoww overview:**
1. +*User** provides a task intent.
2. The **Policy *gent** executes UI actions to fulfil that task; its steps are recorded as *Action History*.
3. The **Evaluator Agent** runs after the Policy Agent has finished, and proactively interact with the environment to gather additional observations.
4. **You** will now produce concise **laamms** for the Agent** only.
*(roie.capitalizee
You must follow step-by-step analysis:
18

---

Published as a conference paper at ICLR 2026
1. Read the **Task Goal** and the capla.capitalize ()) Agent's action history (if available).
2. Examine the provided (role. screenshots are attached in order).
() screens (HTML
3. Synthesize related observations into claims. Each claim must:
List the supporting step indices.
Give brief, evidence-grounder rationale.
State concise, goal-relevant claim.
4. Include any details critical to the final judgment directly in the claims (e.g., specific titles, timestamps, targeta, confirmations error messages)
6. Do **not** judge final success/ fai lure here; only produce claims.
INPUTS
TASK GOAL: (intent)
ACTION HISTORY (Irole. capitalizttt ()) Agent):
(action (action_history if action history else [No action history provided]")
HTML STATES (TRACE of capltaazaie ))) Agent) (btml_text_blook]
OUTPUT GUIDELINES [guidelines)
OUTPUT SCHEMA
" "(role_key]
I1
"steps": [<list of step numbers>],
"reasoning": "<brief explanation of why this claim is justified>",
"claim" "sconcise, goal-relevant claim>" 1,
]
Return only the JSON under **Clains:**
Guidelines for Policy Agent write claims:
**Gudeelines for writing claims (Policy Agent): +*+
### Core Mandate: The Actor's Report Think of yourself as the agent actively performing the task.
Your claims are direct report your own actions and their immediate results.
Your goal is to narrate your journey through the task, focusing only on the steps you took and the UI states you directly
observed or caused.
Be concise, factual, and strictly focused the task goal. Avoid speculation or opinions about why something happened.
4## Claim Generation Rules:
Aim for (min_claims] clamms) claims total.
19

---

Published as a conference paper at ICLR 2026
=xFor Tasks Involving Information Sources (e.g., "from
image, "using the details in the file");**
++1. Access the Source:** Generate claim confirming you **accessed and viewed the specified information sourcex*. *Example:* "The agent opened "expenses. in the gallery to view the expense details."
**2 Confirm Data Match:** In the claim about entering the data, explicitly state that the **data entered matches the data from the source**.
*Example;* "The two expenses entered, 'Office Supplies for 150' and 'Travel Expenses for 200', match the content of pppersss.jpg'...
**For Editing, Modifying, or Deletion Tasks:**
*+1. Capture the 'Before' State:** First, generate claim that **describes the initial state of the item BEFORE the modificcatinn*.
*Example:* "Before editing, the contact's phone number was '555-123-4567'.
**2. Report the 'After' State:** Then, generate a separate claim describing the **successful modification or deletion+*.
*Example:* "The contact's phone number was successfully updated to '555-987-6543'."
**Report All Critical Actions:**
Describe your key actions and their direct consequences using state/action phrasing (e.g., "Recording saved and appears in list").
Highlight any mismatches, errors, or unintended actions you performed (e.g., "Opened the wrong menu," " "A PPermission Denied' error appeared").
**Be Efficient and Relevant:**
Merge duplicate claims that describe the same state. Ignore trivial system indicators (battery, clock, signal), home/launcher screens, and redundant repeated actions unless they are evidence of an error or loop.
Output must be valid JSON following the schema below.
Guidelines for the Evaluator Agent to write claims.
AGuidelines for writing claims (Evaluator Agent)
### Core Mandate: The Detective Analogy
Think of yourself as a detective arriving at a scene *after* the suspect (the Policy Agent) has left.
The action history and screenshots you see are your own investigation, using your mmagnffinng glass' and 'tools' to inspect the scene.
Your goal is to make claims about the state of the scene **as the Policy Agent left it**.
**You must NEVER create claim about your own investigative actions.** For example, if you tap 'aave' or 'Delete' to check a confirmation dialog, you must not clain "The agent saved the file" or "The agent deleted the item." Your actions are not part of the evaluated task.
4# Clain Generation Rules: Aim for (min_claims)
claims total.
20

---

Published as a conference paper at ICLR 2026
Focus the evidence:** All claims must describe the final state resulting from the Policy Agent's s work, using your observations as proof.
**Be factual and concise:** Merge duplicated icates and report on what is present or missing. Avoid speculation.
IIIentiff mismatches:** If your investigation reveals that the final state contradicts the task goal (e.g., wrong file type, incorrect note name, content not saved, settings not changed), these are critical claims *IIgnore trivial states:** Do not report on system indicators (battery, clock), home screens, or app launchers unless directly relevant to the task goal.
include.
**Phrase claims effectively:** Prefer state/action summaries (e.g., "Recording saved and appears in list") over simple lists of uI elements.
Output must be valid JSON following the schema below.
E.3 JUDGE WITH CHAIN-OF-CLAIMS
You are an expert judge evaluating whether a mobile CUI agent (Policy Agent) has completed the user's task.
*Norkflow overview:**
1. **User** provides a task intent.
2. The **Policy *gent** executes UI actions to fulfil that task; its steps are recorded as +Action History+.
3. The **Evaluator Agent** runs after the Policy Agent finished, and proactively probes the resulting states to gather additional observations.
4. Your job is to analyze these claims together, identify their relationships, and determine whether the Policy Agent successfully completed the task.
You must follow two-stage analysis:
### Stage Filter Evaluator Claims
Carefully review the evaluator claims.
**Discard any claim that describes actions or outcomes caused by the Evaluator Agent itself** (e.g., accidental saves, unintended edits, stray taps/scrolls).
-Kepp only evaluator claims that serve *xevidence about the Policy Agent's actual outcome*x.
If in doubt, prefer to exclude rather than include.
### Stage Compare Policy vs. Evaluator Claims
1. **Read the Task Goal** carefully to understand what success means.
2. Comparre Policy Claims and (filtered) Evaluator Claimsx::
Mark as **confirmed** policy claim.
an evaluator claim supports
Mark as **contradictedrx
an evaluator claim directly disproves a policy claim.
Mark as *xcomplementary.* if the evaluator provides
additional relevant evidence.
Mark as *xunsupported** if no evaluator claim addresses a policy claim.
3. Highlight any **critical confirmations or contradictionss. that rectll determine success.
21

---

Published as a conference paper at ICLR 2026
4. Decide the outcome reward: did the Policy Agent achieve the user's s task goal?
**Guiellinee in
Refore labeling contradiction, check if the agents simply observing different aspects of the same content (e.g., Policy saw page 1, Evaluator scrolled to page 2).
If so, their claims are *xcomplementary.. Your job is to **synthesiz*** them into a single, more complete
understanding.
When claims are in direct conflict, act as a critical arbiter rather than a passive matcher. Evaluate reliability and consistency; do not assume both sides are equally valid. Consider the correctness of the **target** (e.g., the right file, event, app)
For question-answer tasks, the Policy Agent must include explicit claim stating the answer it provided, expressed
exactly as required by the task.
Ignore evaluator stray/accidental claims unrelated to the goal. If claims indicate progress but also critical issues (wrong extension, malicious steps), treat as compensated or failure depending on severity.
INPUTS
TASK GOAL: (intent)
POLICY CLAIMS: (policy_claims)
EVALUATOR CLAIMS: (evaluator_claims)
OUTPUT INSTRUCTIONS
Write your reasoning in two sections:
Analysis:
Stage 1: List which evaluator claims you discarded and why.
Stage 2: Compare the remaining evaluator claims against the policy claims, showing relations (confirmed, contradicted, complementary, unsupported).
Explain how these relations support your judgment.
Status: success or failure
Return only these two sections, exactly in this format.
F COST COMPARISON
We further analyze and compare the cost between PRORE and the baselines from per-task cost and the long-term cost perspective. The per-cost cost comparison is detailed in Table
Table 8: Per-task cost comparison across baselines.
Methods
DistRL
DigiRL
WebRL
StepCritic
PRORE
Cost (S)
0.010
0.014
0.013
0.017
0.063
22

---

Published as a conference paper at ICLR 2026
While PRORE incurs additional computational overhead, primarily due to proactive probing and the chain-of-claims mechanism, it achieves significantly higher reward accuracy and F1 scores, as demonstrated in Table 2 and Figure 4 We believe that enhanced reward accuracy ultimately trans- lates to greater efficiency when deploying such a reward system. Therefore, we conducted the additional measurements and analysis.
Firstly, the reward system can be employed to guide test-time scaling of policy agents. As illustrated in Fig. 6 PRORE facilitates markedly more efficient test-time scaling: the policy agent attains 63% success rate with only 2 trials under PRORE, whereas the baseline approaches require at least 5 trials, resulting in 2.5x speedup.
Secondly, the reward system can be employed during training to guide the roll-outs of policy agents. In this setting, the total cost consists of both the rollout cost and the reward evaluation cost. To quantify this, we estimate the cost of collecting 1,000 correctly identified successful trajectories.
correctly identified successful trajectory requires both (i) a successful rollout by the policy agent, and (ii) the evaluator correctly recognizing it as success. Assuming a 60% policy success rate and using the average accuracies T93.e for PRORE and 88.4% for StepCritic), the probability of obtaining one useful trajectory is 0.60 x Acc. Consequently, PRORE requires approximately 1,780 rollouts, whereas StepCritic requires 1, 1,885 rollouts-i.e. StepCritic needs 1 additional rollouts to achieve the same amount of useful data. The corresponding evaluator costs are $112.1 for PRORE and $32.1 for StepCritic.
Table 9: Long-term cost comparison (1,000 useful trajectories)
Method
Avg Acc (%)
Rollouts
Rollout Cost
Reward Cost
Total Cost
PRORE
93.7
1778.7
1636.4
112.1
1748.5
StepCritic
88.4
1885.4
1734.5
32.1
1766.6
WebRL
86.9
1917.9
1764.5
24.9
1789.4
DistRL
86.1
1935.7
1780.9
19.4
1800.2
DigiRL
84.6
1970.1
1812.5
27.6
1840.0
This leads to the following cost difference:
Costsupprriic CostpROrE 110 x CostRoliuu 85.6
PRORE becomes more economical once the rollout cost exceeds $0.78. Using Azure A100 pric- ing (+z
typical 72B GUI agent rollout with 30 steps (30 s/step) costs roughly $0.92, already above this threshold. Thus, under realistic deployment conditions where rollout cost domi- nates (GPU hosting, LLM inference, environment rendering). PRORE becomes more cost-effective for scale training and evolution, despite its higher per-task evaluation overhead.
Similar deductions apply to all other basclines. Table summarizes the total cost of collecting 1,000 useful trajectories for each method.
23


Reviews:
[{"review_id": "OqsgtkGJco", "reviewer": "Reviewer_a4X7", "summary": "ProRe introduces a proactive reward framework that lets a general-purpose LLM schedule state-probing tasks and delegates their execution to domain-specific evaluator agents, shifting GUI reward design from passive trajectory scoring to active evidence collection.", "questions": "1. Can you provide a small-scale pilot on Mind2Web or OS-World to demonstrate ProRe’s ability to transfer to web/pc environments?\n2. How does the system detect and prevent an incorrect reward when the evaluator fails to reach the requested UI state?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Novel reasoner-actor paradigm free from hand-crafted rules or static screenshots\n2. Probing tasks are easier than original tasks, so evaluators succeed more often; system is generic and cheap ($0.06/task)", "weaknesses": "1. Evaluations limited to mobile apps; generalization to web/desktop still open\n2. Failure of evaluator cascades to reward; no failure-detection or fallback strategy offered\n3. ProRe’s evaluator runs only a few probing tasks after the entire trajectory, yielding sparse signals that cannot offer per-step fine-grained evaluation or dense rewards for the GUI agent", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "doFFHi8fAy", "reviewer": "Reviewer_CGdL", "summary": "ProRe is a proactive reward system that combines a general-purpose reasoner with domain-specific evaluator agents to actively probe environments and assign more accurate rewards for GUI agents.\nIt achieves notably higher success rate when integrated with state-of-the-art policy agents.", "questions": "How much extra cost does ProRe introduce?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper is the first to propose a Reasoner–Actor reward mechanism, which evaluates beyond static trajectories and thus provides more accurate and reasonable judgments.\n2. The method shows significant accuracy improvements on three public benchmarks: AndroidWorld, AndroidLab, and MobileAgentBench.\n3. Multiple ablation studies are conducted to verify the effectiveness of each module.", "weaknesses": "1. The computational cost of the Reasoner–Actor reward mechanism may be high, and there is no comparison with other algorithms in this regard.\n2. Experiments are conducted only on Android mobile platforms; the generalization ability to other platforms (e.g., Web/Desktop) remains unverified, which may limit applicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hXyszNE7VN", "reviewer": "Reviewer_1uzD", "summary": "The paper proposes a proactive reward system for GUI agents, aims to address the high evaluation cost problem and the inaccuracies in evaluation under the LLM-as-a-Judge approach. Specifically, by introducing Probing Tasks that enable the evaluation agent to interact with the environment and trace the complete workflow of the execution agent, all key observations gathered during the search are incorporated into the LLM-as-a-Judge assessment.", "questions": "1. Why don't save the results of each step during execution and then use them for evaluation?\n2. With only binary success rates, we can actually obtain more granular scores through Probing Tasks, such as which subtasks were completed, to guide the optimization of the policy model. Why didn't the paper do that?\n3. Whether the evaluated agent will modify some contents during the search process, resulting in inaccurate detection.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written, with a clear and straightforward structure.  \n  2. The motivation is clear: to address the issue of insufficient observational evidence in LLM-as-a-Judge methods.  \n  3. The method is easy to implement, exhibits strong generalizability, and is straightforward to reproduce.", "weaknesses": "1. The generation of Probing Tasks appears to rely solely on the LLM's own ability, without a mechanism to validate their effectiveness. For example, in Table 6: the original task—\"Add a favorite location marker for 47.1303814, 9.5930117 in the OsmAnd maps app\"—and the generated probing task—\"Find the favorite location marker for 47.1303814, 9.5930117 in My Places\"—do not seem significantly different. The task is not broken down into finer-grained observation points.  \n  2. The method has limited innovation, as it essentially breaks down the original problem into multiple verification subtasks and uses prompts to guide the large model in interacting with the environment to complete these subtasks.  \n  3. The application scenario is limited, as this evaluation method seems only applicable to situations where the environment can actually be accessed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CkOML07ng5", "reviewer": "Reviewer_9Ljn", "summary": "The paper introduces ProRe — Proactive Reward Model, a new framework designed to enhance both LLM-based evaluation and data generation by integrating reward modeling with preference-guided generation. Unlike existing LLM-based evaluators (which passively score outputs), ProRe actively interacts with candidate responses by generating proactive feedback and counterfactuals. This proactive behavior enables the model not only to evaluate, but also to synthesize new, higher-quality training examples in a closed-loop fashion. The authors benchmark ProRe across several scenarios, including LLM-as-a-Judge (automatic evaluation) and reward-guided data synthesis.\nExperiments on MT-Bench, AlpacaEval, and internal instruction datasets show that ProRe consistently surpasses baselines such as RMHF, LLaMA-RM, and GPT-4-judge — achieving higher correlation with human judgments and generating data that improves downstream SFT and alignment quality.", "questions": "1. How does ProRe ensure that proactive feedback remains constructive rather than simply contradictory? Is there a filtering or quality-control step for generated counterfactuals?\n2. Are proactive and passive reward signals jointly optimized, or does the model alternate between them?\n3. What is the computational overhead of the proactive generator compared to a vanilla reward model?\n4. Can ProRe be extended to reinforcement-style fine-tuning, i.e., using its own reward outputs to guide policy gradients?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Innovative Proactive Paradigm. The main novelty lies in making the reward model proactive — not only scoring responses but actively generating contrastive examples and rationales.\n2. Unification of Evaluation and Data Generation. Traditional reward models are confined to evaluation; ProRe bridges the gap between reward estimation and data synthesis.\n3. Empirical effectiveness and generality. On evaluation benchmarks (e.g., MT-Bench, AlpacaEval), ProRe achieves strong correlation with human judgments (up to +4.2 Kendall’s τ improvement over LLaMA-RM). On data generation tasks, SFT models trained with ProRe-generated data outperform baselines by 1.5–3.0 points on MMLU and GSM8K.", "weaknesses": "1. Limited analysis of reward misalignment risk. While ProRe aims to proactively improve data, it also introduces the risk of self-reinforcing biases—if the reward model’s early feedback is flawed, later generations may amplify errors.\n2. Dependence on backbone LLM quality. ProRe relies on the underlying LLM (e.g., LLaMA-3-8B or GPT-4) for generating feedback and counterfactuals. The generalization to weaker models (e.g., <7B) is not well studied, raising questions about scalability and accessibility.\n3. Experimental variance and reproducibility. The proactive feedback generator’s decoding parameters are not detailed (temperature, beam width), which affects reproducibility.\n4. The paper hints that proactive reward models may “self-improve” through interaction, but does not analyze stability (could the model overfit to its own generated rewards?).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)