WebArbiter: A Generative Reasoning Process Reward Model for Web Agents
Authors: Yao Zhang
Shijie Tang
Zeyu Li
Zhen Han
Volker Tresp
Venue: ICLR 2026 Poster (2026)
Keywords: Process Reward Model
Web Agent
Reasoning Reward Model
Inference-Time Scaling
Benchmark
Primary area: reinforcement learning
TL;DR: WebArbiter frames web process reward modeling as principle-guided text generation, outperforming GPT-5 on the new WebPRMBench and delivering up to 7.2-point gains in reward-guided trajectory search on WebArena-Lite.

Abstract: Web agents hold great potential for automating complex computer tasks, yet their interactions involve long-horizon, sequential decision-making with irreversible actions. In such settings, outcome-based supervision is sparse and delayed, often rewarding incorrect trajectories and failing to support inference-time scaling. This motivates the use of Process Reward Models (WebPRMs) for web navigation, but existing approaches remain limited: scalar WebPRMs collapse progress into coarse, weakly grounded signals, while checklist-based WebPRMs rely on brittle template matching that fails under layout or semantic changes and often mislabels superficially correct actions as successful, providing little insight or interpretability. To address these challenges, we introduce WebArbiter, a reasoning-first, principle-inducing WebPRM that formulates reward modeling as text generation, producing structured justifications that conclude with a preference verdict and identify the action most conducive to task completion under the current context. Training follows a two-stage pipeline: reasoning distillation equips the model with coherent principle-guided reasoning, and reinforcement learning corrects teacher biases by directly aligning verdicts with correctness, enabling stronger generalization. To support systematic evaluation, we release WebPRMBench, a comprehensive benchmark spanning four diverse web environments with rich tasks and high-quality preference annotations. On WebPRMBench, WebArbiter-7B outperforms the strongest baseline, GPT-5, by 9.1 points. In reward-guided trajectory search on WebArena-Lite, it surpasses the best prior WebPRM by up to 7.2 points, underscoring its robustness and practical value in complex web tasks.

Published as a conference paper at ICLR 2026
WEBARBITER: A PRINCIPLE-GUIDED REASONING PROCESS REWARD MODEL FOR WEB AGENTS
Yao Zhang!
Zeyu Li?, LMU Munich Technical University of Munich
Shijie Tang",
Zhen Han"
Volker Tresp
'Munich Center for Machine Learning (MCML)
Project Page: MebArbiter
ABSTRACT
Web agents hold great potential for automating complex computer tasks, yet their interactions involve nong-horizon, sequential decision-making with irreversible actions. In such settings, outcome-basse -based supervision is sparse and delayed, often rewarding incorrect trajectories and failing to support inference-time scaling. This motivates the use of Process Reward Models (WebPRMs) for web navigation, but existing approaches remain limited: sealar WebPRMs collapse progress into coarse, weakly grounded signals, while checklist-based WebPRMs rely brittle template matching that fails under layout or semantic changes and often mislabels superficially correct actions as successful, providing little insight interpretabil- ity. To address these challenges, we introduce WebArbiter, a reasoning first, principle-inducing WebPRM that formulates reward modeling as generation, producing structured justifications that conclude with a preference verdict and identify the action most conducive task completion under the current context. Training follows a two-stage pipeline: reasoning distillation equips the model with coherent principle-guided reasoning. and reinforcement learning corrects teacher biases by directly aligning verdicts with correctness, enabling stronger generalization. To support systematic evaluation, we release WEBPRMBENCE a comprehensive benchmark spanning four diverse web environments with rich tasks and high-quality preference annotations. On WEBPRMBENCII, WebArbiter-P7 7B outperforms the strongest baseline, GPT-5, by 9.1 points. In swward-guided trajec- tory search on WebArena-Lite, surpasses the best prior WebPRM by up to 7.2 points, underscoring its robustness and practical value in complex web tasks.
Denpbede-t
Wridteppendre
Average BoN Accuracy (9))
80-
Average BoN Accuracy (96)
100
GPSS
WedeAr
70
GPT-
60
* Wibbrititer (BB)
3P TT
Damin Flash
Claude
Spnnet
60
67T-4o-min
LLamsi-T0--nstuutto
50
40-
20
30
6 OOwnnt 5-3f--teerncct
20
3B
7888
70B Unknown
Mind2Web
WebArena AssistantBench WorkArena
Number Parameters
Domains
Figure 1: Performance comparison WEBPRMBENCH Left: Average Best-of-N Acc model size, showing superior efficiency despite smaller scale. Right: Domain-wise Avg BoN Acc, where WebArbiter achieves the best results across all environments, confirming robustness and scalability.

---

Published as a conference paper at ICLR 2026
INTRODUCTION
Large Language Models (LLMs) (Achiam et al. 2023 Guo et al. 2025a have demonstrated impres- sive capabilities in planning (Huang et al. 2024 Zhang et al. 2025a), decision-maiee making (Li al.] 2024), and complex task execution (Xi et al. 2024 Zhang et al. (025b). Extending these abilities with browser access enables LLM agents to perform complex web tasks similar to humans (OpenAl 2025a Anthropic 2024a Adept 2022). However, web interactions involve long horizons, multi-step decisions, and actions that can be irreversible. For example, submitting an incorrect form may not be recoverable. This requires agents to make reliable decisions throughout the interaction process, rather than relying solely on final outcomes. Traditional Outcome Reward Models (ORMs) are ill-suited: they provide only sparse and delayed feedback, may misclassify incorrect trajectories successes, and cannot guide inference-time strategies, such as reward-guided search.
To this end, we propose WehArbiter, a reasoning-first principle-inducing WebPRM. It formulates process reward modeling as text generation: given task context and candidate actions with their reasoning traces, the model produces a structured justification that concludes with a preference verdict, identifying the action most conducive to task completion. Unlike scalar scores or checklist- based methods tied fixed templates, WebArbiter dynamically derives principles from user intent and the current state, incorporates them into reasoning chains that verify whether an action advances task completion. Training follows two-stage pipeline: reasoning distillation equips the model with coherent principle-guided reasoning, and reinforcement learning (RL) corrects teacher biases and aligns verdicts with correctness. This design transforms reward signals from shallow correlations into auditable analyses, making judgments robust to environment and page variations, resistant to spurious cues, and accurate in credit assignment.
To advance the evaluation of WcbPRMs, we introduce WEBPRMBENCH, the first comprehensive evaluation benchmark spanning diverse environments dedicated to WebPRMs. It provides 1,150 step-level preference instances, each consisting of one correct action and four rejected alternatives, collected across 4 web environments: AssistantBench (Yoran et al. 2024), Mind2Web (Deng et al.) 2023). WorkArena (Drouin al., 2024 Boiivvrtt et al. 2025). ) and WebArena (Zhou al. 2023). The tasks span everyday activities such as online shopping and forum posting, as well as enterprise scenarios like updating schedules in management platforms. By combining scale, diversity, and fine-grained supervision, WEEPR.MBENCH establishes unified standard for systematic evaluation of WebPRMs, with Pairwise and Best-of-N (BoN) Accuracy as the primary metrics.
As shown in Fig. experiments on WEBPRMBENC show that WebArbiter achieves the highest Avg. BoN Ace among all evaluated models, outperforming the strongest proprietary LLM baseline, GPT-5, by 9.1 points, and consistently surpassing the previous SOTA WebPRM, WebShepherd, across all environments. Beyond static evaluation, WebArbiter also proves effective practice: in reward-guided trajectory search on WebArena-Lite (Liu et al., 2024b), it delivers substantial gains, surpassing WebShepherd by up to 7.2 points, further demonstrating robustness in realistic interaction settings.
The key contributions of this work are:
1. We propose WebArbiter, a reasoning- first, principle-inuccing PRM trained with reasoning distillation and RL, providing auditable reasoning chains and correctness-alinnee signals.
2. We release WEBPRRBENCI, the first comprehensive evaluation benchmark to provide systematic WebPRM evaluation across 4 web environments, using Pairwise and Best-of-N (BoN) Accuracy standard metrics.
3. We show that WebArbiter achieves SOTA performance on WEBPRMBENCH, surpassing both proprietary L.MMs and the previous SOTA WebPRM. WehArbiter delivers up to 7.2% gains in reward-guided trajectory search on WebArena-Litte
4. We analyze the effects of different training components through systematic ablations, show- ing that cold-start RL alone is unstable across environments, whereas reasoning distillation and explicit principles are essential for stable and transferable progress-awee judgments, with RL primarily acting as an amplifier.
Equal contribution.
Corresponding authors.
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORK
2.1 LLM-BASED AUTONOMOUS WEB AGENTS
LLM advances have enabled browser-operating agents (Kim et al.
2024
Sun et al.
2024 Prasad
et al. 2023 Fu al. 2024 Ma et al. 2023 Zheng al. 2023 Tao al.
2023). One line distills davironment-specite state-action pairs from demonstrations, strong on seen states yet brittle on novel ones, with SteP a leading example on WebArena (Sodhi et al. 2024 Zhou al., 2023). A second
line pursues open- eedde exploration via reflexive evaluation and search (Pan et al. 22244] (Shinn et al. 2024 Koh et al. 2024| Zhang et al.) (0255b). third direction applies RL (Qi et al. [202]] Wei et al., 2025). yet real sites provide sparse and delayed signals, which makes value learning unstable without
dense step feedback. Therefore, WebAgents require a process-level judge that assesses progress step
by step and supplies auditable signals for search and planning.
2.2 REWARD MODELS IN REASONING AND WEB TASKS
RMs fall into two families. Scalar RMs attach a single numeric score to a response and use either
Ouyang al. 2022 Liu et al. 2024a 2025 Park et al.] 2024 Wang et al.] 2024a 2023b 2024b). Generative RMs instead
absolute discriminative schemes for evaluation (Uesato et al. 2022
produce natural language feedback from which rewards are extracted, aligning with LLM-as-Judge
and supporting both single-instance evaluation and multi-response comparison; they show promising
scalability but raise reliability concerns due to bias and hallucination (Lightman et al.. 2023 Wang
eet al. 2023a; IZhang et al.) 2025d] Wu et al.) 2024; Ye et al.) [225 Zhang et al. 2024 2025c).
Building on these, Reasoning RMs cast judging as a deliberate process: they first generate an explicit, context-grounded chain of principle and analysis, then issue single preference verdict, yielding adaptive test-time compute, stronger grounding, and interpretable feedhack (Chen et al.,
2025 Guo et al. 2025b; Mahan et al. (2024). In web agents, action rewards have been derived by the following methods: LLM-as-Judge. (Zhang et al. 2025b Koh et al., 2025), slow and unstable during search; scalar scoring (Miao et al., 2025 which collapses progress into coarse values with little interpretability and weak grounding, and checklist-driven generative feedback (Chae et al. (025), whose external templates are brittle under layout and semantic drift and prone surface correlations. These limitations motivate a reasoning-firs' approach that turns rewards from shallow correlations into auditable amalyses. WebArbiter produces structured justifications with a single preference verdict, induces principles from the current instruction and state, and is trained by reasoning distillation followed by RL, so that judgments remain robust to environment variations, resist spurious cues, and provide accurate credit assignment while supporting inference-time scaling.
METHODOLOGY
In this section, we present the design of WebArbiter. Fig, provides overview of the WebArbiter framework, including the two-stage training pipeline and the inference-time principle-guides decision process. We begin by framing web navigation Partially Observable Markov Decision Process (POMDP) in $3.T. then describe how we construct a pairwise-preferenc dataset for training in $3.2 We introduce the training pipeline of WehArbiter model in $3.3 For clarity, we summarize all notations in Appendix A
3.1 BACKGROUND
We formalize web navigation as a POMDP. The environment is defined by a state space S, an action space A, and an observation space O. T S A S denotes the state transition function. At step the agent receives a partial observation o, E O, executes a, AAA, and transitions to 8p+1 = T(sp ap) with new observation 0p+1. Following WebArena (Zhou et al. 2024), we represent observations using accessibility trees, i.e., text-only encodings of visible interactive elements and their attributes. Given task instruction I and the initial state 8o S, the agent aims generate a trajectory T (a.....,ap ap) that completes the task. Here P is the trajectory length and ap E AA denotes the action at step p. The task evaluator determines whether the task is completed based the final state.
3

---

Published as a conference paper at ICLR 2026
X
T
Og
Candidate (aju cj)
Inference Phase
Onvoge office mayer pace qo the page
Thought
pou weie
[13]] lnn M.C home:
V'd Menscel' which libaly futther
(SV) Lutton loggle ua guage fur datia f
(Gep.Cog Ccp)
Actionc smil) 100)
1,0g acpe Seps (a sp). (4)
Action Candidate
(og, cf)
Thought
Action (36))335)
Thought riseed of tous solly
Thaught The ster hos
dredt municatinn explating
Action soutiit 4001
Adtion c1660390)
Principles
Training Phase
Stage
Didilacioo
Princpte clarity and reffurees Priecpple 2: CoCrreetnene ressoning Princppie tack proprees
Stage
Reesent Ancurcyy Action Cohernecn II
2esporte
Resporne
X
WebArbiter
Sinutiooal
Roscorse
Prinsiple 1. clarity and sdtfidress
Saeponse 1 provides dlear navigation retruct ono whike response
algement ressoring Reipoorr
Respanee
Answer Rmppnnel
Reinforcement Learning
Figure 2: Overview of WebArbiter. Given instruction I, current observation Op. and history (A.cp, Cep), the model compares candidate actions (a). and (a e.) In Stage 1, principle- guided reasoning traces are distilled from a stronger teacher LLM. In Stage 2, WebArbiter is trained with RL using verifiable rewards R E {-1,+ +1}, producing structured justifications and final verdict. During inference, the model induces principles (e.g., clarity, correctness, progress) from (I, Op: acpp Cepp (a,, c.),, (a%. c.)). applies them to candidate actions, and outputs auditable judgment identifying the action that best advances task completion.
3.2 TRAINING DATASET CONSTRUCTION
We build on the WEBPRM COLLECTION (Chae al. 2025
for training WebArbiter. Each instance
consists of an instruction Z, sequence of observations O = -o1..... 0p), and expert-annotate trajectories. Specifically, the dataset provides a set of positive actions A = (at "ut taken from expert demonstrations and negative actions A (oi,.... ...., ap)
44))
..) obtained from rejected trajectories. We convert these into pairwise preference samples where each candidate action paired with its
reasoning trace, yielding the preference dataset Dirain used for WebArbiter training.
3.3 WEBARBITER: PRINCIPLE- INDUCING REASONING PROCESS REWARD MODEL
WebArbiter is built on Transformer-decodee backbone and formulates process reward modeling as a text generation task. At each state, it evaluates candidate actions {(aß, eg) of)] where each action ag is paired with reasoning trace cg explaining why the agent generated this action. Given task instruction I, observation Op. and history (a<p; Cep), the model autoregressively generates structured justification j (j1..... .i) of length L that concludes with preference verdict y selecting the most appropriate action among the candidates. The historical traces are Ccp (cI...., i.e., the per-action reasoning traces for previously executed actions. A concrete training example is provided in Appendix B While our experiments instantiate this framework in the standard pairwise preference setting, the design is general and extends naturally to multi-candidate settings.
Unlike the scalar WebPRM (Miao et al. 2025) that collapses progress into opaque scores or the checklist-based WebPRM (Chae et al. 2025), WebArbiter is a reasoning-fiss: principleinnducing WebPRM: it dynamically derives principles from user intent and the current state, integrates them into reasoning chains that explicitly assess whether each candidate action truly advances task completion. This design moves reward signals beyond shallow correlations toward auditable analyses, yielding judgments that robust to environment changes, resistant spurious cues, and precise in credit assignment.
Formally, the preference dataset is defined as
Drrain ((I(()
.0p <pp Cpp (a ( c
(1) (6) (1) 1(0) 2(0)), (1) -0022100. ),0 (0) M
(1)
4

---

Published as a conference paper at ICLR 2026
where y {ap. ) denotes the preferred action. For notational simplicity, let
z (I, Ops acpps Ceps (a cpp,, (a c.)).
(2)
WebArbiter Te is parameterized by 0 and models the justification j autoregressivell:
Toli x)
molsi 2,ja))
(3)
3.3.1 TRAINING OVERVIEW
The overall training objective to maximize the likelihood that the predicted preference matches the ground truth:
max 11
(( ( [1 v)
Training proceeds in two stages. First, reasoning distillation, described in $3..2 equips the model to generate coherent, principle-guides justifications, promoting judgments grounded in explicit reasoning rather than surface correlations, property later validated by ablation studies in $5.1.3 Concretely, we sample K examples from Drrain to construct Dsrr for supervised distillation, while the remaining data form DRI. for RI. Second, RI., detailed in $3.3.3] aligns the model's verdicts with correctness signals and yields interpretable step-level rewards for long-horizon decision-making. Together, these stages enable WebArbiter to provide robust, interpretable, and scalable supervision for web agents.
3.3.2 STAGE 1: REASONING DISTILLATION
Directly prompting an instruction-ture LLM as reward model often yields superficial, inconsistent chains that do not justify why an action advances the task. We therefore distill principle-guides reasoning from stronger teacher. Concretely, o3 synthesizes structured justifications that first derive task-specific principles from the instruction and state, then ground these principles in the page, compare candidate actions against them, and finally output the preferred action. This equips WebArbiter with principles rather than surface heuristics. Ablations in $5.1.3 show that removing explicit principles and relying solely reasoning based justifications notably degrades performance, highlighting the role of principle induction in stabilizing step-level judgments. Given (a:(0), ()) E
DEFT, the teacher generates justification j(8) (j(() ,I.. The distillation dataset is then:
:(0)).
DSFT (z) )(0))
Objective. Reasoning distillation adjusts 0 to maximize the likelihood of generating the teacher justification j that concludes with the preferred action y given æ. We minimize the standard negative log-likelihood:
L
1
log To (j()) xx00 10.32)
(SF
(5)
K
3.3.3 STAGE 2: REINFORCEMENT LEARNING
While distillation provides initial reasoning ability, it inherits teacher biases and may overfit to superficial patterns, limiting generalization to unseen environments. To further enhance judgment accuracy, stability, and generalization, we introduce a RL stage. WebArbiter πą is treated as judgment policy that outputs justification that concludes with final verdict ÿ. During rollout, Ty generates the full justification and verdict, after which a correctness reward R(x, ÿ) E (-1, is assigned solely based on whether matches the ground-trus preference y. The distilled model from 3.3.2 serves the reference policy Tref ensuring stable optimization.
Objective. RL adjusts 0 to maximize the expected reward while stabilizing reasoning style via KI. regularization. The optimization objective is defined as:
(8) max E (a.))) (Da, ( R(x, 8) DKI. (TT)
(6)
5

---

Published as a conference paper at ICLR 2026
Table 1: Data distribution of WEEPRMBENCH, the first comprehensive evaluation benchmark spanning diverse environments for WebPRMs.
Mind2Web
Environment
WebArena
AssistantBench
WorkArena
Total
Cross-Tass Task
Cross-Wehsite
Cross-Domain
Count
142
148
417
201
30
212
1150
In practice, we adopt Group Relative Policy Optimization (GRPO) (Shao et al. 2024) to optimize this objective, which enables stable updates under binary verifiable rewards. Through this RI stage, WebArbiter directly aligns its verdicts with correctness signals and converts structured justifications into reliable, interpretable step-level reward signals.
WEBPRMBENCH
This section introduces WESPRMBENCH, a comprehensive multi-envirrnment benchmark for evaluating WebPRMs.
4.1 BENCHMARK CONSTRUCTION
WERPRMBENCH constructed from sucessful trajectories in AGENTREWARDBENCH
Lu
et al. 2025) expanding beyond WERREWARDBENCH (Chae et al., 2025), which only provides
Mind2Web (Deng al. 2023) and limited WebArena data (Zhou et al. 2023 We enrich
WebArena with additional trajectories and incorporate AssistantBench (Yran et al. 2024) and WorkArena (Drouin al. 2024 Boisvert al. 2025), resulting in broader coverage real-world
tasks across four environments. MindWWeb emphasizes cross-task generalization across hetero- geneous websites. WebArena provides controlled environments such as shopping, CMS, Reddit, and GitLab. AssistantBench introduces peen-world tasks on real websites. WorkArena focuses on enterprise workflows, including IT and HR. This diversity enables systematic evaluation across both consumer-facing and enterprise scenarios, covering a broad range of task complexities.
For each state, the action from the successful trajectory is retained as the positive label, and four rejected alternatives with associated reasoning traces synthesized to form preference pairs. To ensure data quality, we sample negatives from diverse policy models to broaden coverage, apply rule-based filters remove invalid or mismatched actions, discard inconsistent cases, and conduct expert verification to ensure reliability. We also conduct targeted auditing to eliminate false negatives. To avoid positional bias, the positive action is not fixed specific side and may appear either side of the preference pair. Reasoning traces are truncated to fixed length to minimize formatting noise. The resulting benchmark spans 1,150 step-level preference instances across four environments, as shown in Tab. Full construction details and benchmark statistics are provided in Appendix
4.2 EVALUATION PROTOCOL
Evaluating WebPRMs requires metrics that capture both local preference fidelity and global decision reliability under realistic multi-candidate settings. Inspired by RMB (Zhou al. 2025), we adopt two complementary metrics: Pairwise Accuracy, which measures correctness on individual preference pairs, and Best-of-N (BoN) Accuracy, which evaluates robustness when ranking among multiple distractors. Compared with Pairwise Acc, BoN Acc applies a stricter criterion by requiring the correct action to outrank all distractors simultaneously, providing stronger discriminative power and better alignment with downstream agent performance. A deeper analysis of BoN vs. Pairwise Ace is in Appendix j
Pairwise Ace. Given preference pair (at,a" ,a)), where at is the correct action and a is rejected one, the WebPRM is correct it assigns a higher preference to at. Formally:
Acc Pairuiise
Σ
[To(aat== x To(a =)]a
Deenchh
(at.a 1A--)ED De

---

Published as a conference paper at ICLR 2026
Table 2: Results on WEBPRMBENCH with Pairwise and BoN Acc. * denotes our models. Bold numbers indicate the best results, while underlined numbers denote the second best. Our WebArbiter- 7B achieves the highest Avg BoN Acc, outperforming the second-best baseline, i.e., GPT-5, by 9.1.
Mind2Web
WebArena
AssistantBench
WorkArena
Avg.
Models
Painwise
BoN
Pairwise
BoN
Pairwise
BoN
Pairwise
BoN
Pairwise
BoN
LLM-as-judge Proprictary Language Models
GPT--4o-miii
81.74
50.92
78.23
56.72
89.17
73.33
81.43
46.70
82.64
$6.92
GPT-4o
79.99
52.62
84.58
66.67
85.83
66.67
84.33
55.19
83.68
60.29
GPT-5
80.86
62.39
84.83
71.64
81.67
63.33
81.14
64.62
82.13
65.50
Claude-3.7-Sonnee
80.20
57.90
82.80
64.10
81.50
611.3
82.10
60.60
81.65
60.98
Gemini-2.5-
81.30
57.01
82.71
62.19
80.00
63,33
83,30
56.13
81.83
59.67
DeepSeek-RI
81.62
57.37
82.04
60,21
78.49
56.18
84.12
63.89
81,57
59.41
LLM-as--uuke judgee, Open-source Language Modela
Qwen2.5-3B t
76.46
36.93
60.32
15.42
75.83
33.33
64.45
19.34
69.27
26.76
Qwen2.5-7B-Ime
77.79
39.18
74.88
42.79
84.17
53.33
77.58
35.85
77.61
42.78
Hama-3-70B-IInsiie
80.55
49.36
77.36
50.75
85.83
70.00
79.08
80.71
52.55
WebPRMs (38)
WebShepherd-31B
87.50
65.21
68.16
41.29
66.67
46.67
50.00
21.23
68.08
43.60
WebArbiter-3B
93.32
78.42
81.97
56.22
78.33
46.67
81.01
54.81
83.65
59.06
WehPRMs (7B+)
WebShepherd-8B WehArbite--7B
86.66
73.69
68.33
43.88
$5.92
30.00
54.56
25.53
64.34
43.28
97.07
89.53
88.43
68.66
89.17
70.00
82.09
70.19
89.19
74.60
BoN Acc. For each instance (a+, a "...., ....... ()) E DBench. the WebPRM considered correct only when I is consistently ranked above all Q distractors, with Q 4 in our benchmark, BoN Acc is:
[Daessal
Q
AcCBoN
Il [ Te(a; q=1
(8)
[Deecch
==1
EXPERIMENTS
We conduct comprehensive experiments to evaluate WebArbiter on the reward modeling benchmark WEBPRMBENCH in 55.1 and on practical applications in 5.2
5.1
WEBPRMBENCH
5.1.1
EXPERIMENTAL SETUP
Baselines. We compare WebArbiter against three categories of baselines. (1) Proprietary LLM-as-
judge models, including GPT-4o-mini (OpenAl 2024a), GPT-4o (OpenAl 2024b), GPT-5 ApenAA 2025b), Saude-3.7-Sonmee (Anthropic) 2025). Gemini-2.5-Flash (Pichai & Hassabis 2025), 3, and
DeepSeek-RI (Guo al. 2025a), which are prompted act as judges by selecting the preferred
action given task context. (2) Open-source LLM-as-judde models, represented by Qwen2.5-22
Instruct and Qwen2.5-7B Biee2.5-7B-Inntruu InInnrutt (Qwen al. 2025). and Llama-3-70B. 3---3---BB-nnnntee Instruct (Meta 2024).
providing accessible yet competitive instruction-tuned baselines. (3) WebPRMs, where we include WebShepherd (Chae et al. (025)).
We train WebArbiter on WEBPRM Collection (Chae et al., 2025 which comprises 30k step-level preference pairs drawn from the Mind2Web environment. We use 10k pairs for stage-1 reasoning distillation and the remainder for stage-2 RL. Models are initialized from 1ren2.5-3B-Instruue and Sivn2.5-7B-Innstruee (Qwen et al. 2025) and fine- tuned with LoRA (Hu ct al. 2022). Further implementation details are provided in the Appendix C and all prompts provided in Appendix
Implementation Details.
Evaluation Metries. We report results using two complementary metries: Pairwise Accuracy, which measures correctness on individual preference pairs, and Best-of-N (BoN) Accuracy, which evaluates robustness under multi-candidate settings. Detailed definitions provided in 4.2

---

Published as a conference paper at ICLR 2026
Table Ablation results on WEBPRMBENCH with Qwen2.5-7B-Instrrv as backbone. We report Pairwise and BoN Acc across web environments. WebArbiter, combining principle-guided reasoning distillation with RL, achieves the highest overall performance.
Mind2Web
WebArena
AssistantBench
WorkArena
Avg.
Method
Pairwise
BoN
Pairwise
BoN
Pairwise
BoN
Pairwise
Bol
Pairwise
BoN
Instruct (Original)
77.79
39.18
74.88
42.79
84.17
53.33
77.58
35.85
77.61
42.78
Instruct Cold Start RL
96.18
86.00
71.10
35.80
72.40
33.60
74.90
37.90
78.15
48.33
Instruct Cold Start RI. Principles
96.18
88.00
77.80
46.30
80.10
48.90
82.40
51.80
84.12
58.75
Instruct W Prisipls RL.
98.48
94.34
74.60
41.50
77.20
40.20
79.10
44.60
82.35
55.16
* WehArbrite-75 7B
97.07
89.53
88.43
68.66
89.17
70.00
82.09
70.19
89.19
74.60
5.1.2 MAIN RESULTS
WebArbiter Significantly Outperforms Baselines. As shown in Tab. 2 WebArbiter achieves the highest Avg. Pairwise Acc and Avg. BoN Acc, surpassing both proprietary and open-source LLMs. While LLM-as-judge methods often maintain moderate Pairwise Acc, their performance drops sharply on BoN Acc, revealing poor robustness to hard negatives. In contrast, WehArbiter sustains strong results on both metrics, establishing its reliability under realistic multi-candidate settings. We further analyze inference-time scaling behavior in Appendix G
Advantage over the SOTA WebPRM. WebShepperd (Chae al. 2025) represents the previous SOTA WebPRMs. Trained on the same WEBPRM Collection, which was drawn from the Mind2Web environment, WebArbiter-7B achieves an Avg. BoN Ace of 74.60%, surpassing WebShepherd--8 by an absolute gain of 31%. Unlike WebShepherd, which relies on fragile checklists, WebArbiter employs griccple-guided reasoning, yielding judgments robust to environment and page variations. Case studies illustrating these differences are provided in Appendix H
Robust Generalization Across Environments. As shown in Tab. attains SOTA BoN Acc on Mind2 and WorkArena, and remains competitive with strong proprietary LLMs on WebArena and AssistantBench. These results indicate that principle-guided reasoning enables both effective in- domain learning and robust performance in heterogeneous, noisy, and enterprise-scale environments.
5.1.3 ANALYSIS OF TRAINING DESIGN
Training Recipes We compare four training variants to disentangle the effects of RL, principle guidance, and justification style. Instruct (Original) denotes a purely instruction-tunec model without additional optimization. Instruct Cold Start RL directly applies RL top of the instruction model. Instruct + Cold Start RL. + Principles augments RL with principle prompting during training, enabling explicit principle induction before judgment. Instruct SFTwio Principles RL performs reasoning distillation without principles, followed by RL, thereby testing whether narrative-style justifications alone are sufficient. As shown in Tab. WebArbiter achieves the best performance. Explicit principles anchor judgments to progress, producing stable supervision under multi-candidate web settings.
RL Alone is Unstable Across Web Environments. Cold Start RL performs well on in-domain Mind2Web but collapses on oo-main benchmarks. This highlights that reward optimization without reasoning distillation struggles in noisy and complex environments.
Principles Enable Enviromnviron Generalization. Augmenting RL with principles improves both Avg. Pairwise and BoN Acc, especially on AssistantBench and WorkArena, where real-world tasks require context- and state-dependent judgments beyond surface layout cues. AssistantBench features open-worlh websites with high structural variability, while WorkArena involves enterprise workflows governed by state-dependent constraints. Principle-guided reasoning provides transfcrable criteria for assessing true task progress in both cases, improving robustness and generalization.
Reasoning Without Principles is Insufficient. SFT Principies + RL, which relies solely on narrative- style justifications, improves linguistic fluency and coherence of the generated explanations but consistently underperforms principle-aware settings. Without explicit principles to anchor judgment, the model tends to rationalize actions post hoc based surface plausibility, making it vulnerable to
8

---

Published as a conference paper at ICLR 2026
Table Results on WEEPRMBENCH under full-data and limited-data (10K) training regimes. We report Pairwise and BoN Ace across web environments. Reasoning distillation improves over answer- only SFT, while WebArbiter, i.e., reasoning distillation + RL, achieves the best overall performance.
Method
Mind2Web
WebArena
AssistuntBench
Avg.
Pairwise
BoN
Pairwise
BuN
Pairwise
BuN
Pairwise
BuN
Pairwise
BoN
Truin Fuil
Instruct + SFT
85.14
60.91
80.85
$2.73
82.50
56.67
79.57
52.88
82.02
55.80
Instruct Disrilled& 4 SFT
87.42
61.18
81.59
52.73
83.33
63.33
81.13
56,73
83,37
58.49
WebArbiiet7B (Instruct Distilled RL)
97.07
89.19
Train on 10K (Sugge1 Reasowing Distillatoon Dala
Instruct SPT
84,53
60.82
82.21
58.71
82.50
56.67
80.58
39.62
82,46
53.95
Instruct Distilee
85.20
63.40
83.10
61 10.80
83.00
60.20
81.40
55.60
83.18
60.25
spurious correlations and context-specific cues. As result, narrative reasoning alone is insufficient to reliably track genuine task progress in complex, long-horizon real-world web navigation.
Reasoning Supervision We analyze the role of reasoning supervision by comparing answer- only SFT, distilled reasoning, and RL under both full-data and limited-data settings. Instruct + SFT optimizes the instruction-un exposing any intermediate reasoning or justification during training. Instruct + Distilled + SFT runs an answer-only SFT stage on top of the distilled checkpoint, fine-tuning the model directly toward the final decision and serving controlled comparison to RL-baased training. WebArbiter (Instruct Distilled + RI.) further builds upon distilled reasoning by applying RI. with verifiable rewards, encouraging principle-guided judgments that better reflect true task progress. Results on WEBPRMBENCH are reported Tab.
model to directly output the final preference decision, without
Reasoning Distillation Improves Judgment Stability, with RL an Amplifier. Comparing Instruct Distilled + SFT with Instruct + SFT, we find that reasoning supervision leads to more reliable reward judgments, particularly in multi-candidate settings measured by BoN Acc. Under the full-data setting, applying answer-only SFT after distillation yields environment-dependen gains, final-answer optimization can reintroduce shorteut correlations specific to individual web environments. Nevertheless, reasoning distillation induces a more stable discrimination among competing trajectories by grounding judgments in true task progress rather than surface-level cues. Building upon this reasoning distillation phase, WebArbiter further applies RL to enlarge the margin between truly progress progress-making and spurious trajectories, achieving the highest overall performance.
Reasoning Supervision Is Especially Effective Under Limited Data. Under the 10K (Stage-l Reasoning Distillation) setting, Instruct Distilled consistently outperforms Instruct + SFT across all environments, yielding clear improvements in both Pairwise and BoN Acc. Since both models are trained with identical data budgets, these gains cannot be attributed to data scale, but instead reflect a training objective that explicitly biases the model toward progress-aware reward judgments.
5.2
REWARD-GUIDED TRAJECTORY SEARCH
5.2.1
EXPERIMENTAL SETUP AND IMPLEMENTATIONS
Reward-guidee trajectory search represents one of the most practical applications of PRMs, as it directly leverages fine-grained step-level supervision improve decision quality during agent execution. To evaluate WebArbiter in this setting, we conduct experiments jet al. 2024b), which contains diverse, long-horizon tasks such as online shopping and content management, closely reflecting real-world web activities. Performance is measured with Success Rate. Following WebShepherd (Chac et al.] 2025). we adopt a Best-of-N sampling strategy: the policy model generates N 5 candidate actions for each step, and WebArbiter selects the most promising one through a Knockout Tournament mechanism (Guo et al. 2025b). We evaluate two policies, GPT-4o-mini (OpenAl 2024a) and GPT-40 (OpenAl 2024b).
WebArena-LLite L LLuu
We did not have access the MAP domain during this work and therefore excluded it from our experiments.

---

Published as a conference paper at ICLR 2026
Table 5: Success Rates (%) of trajectory search with GPT-4o-mini and GPT-4o as policy on WebArena- Lite. Results reported from the WebShepherd (Chae et al. 2025). A is relative to the w/o Trajectory Search baseline. Our WebArbiter consistently achieves the highest gains across both policy models.
Policy
WehPRM
Shopping
CMS
Reddit
GitLaab
Avg.
A
w/o Trajectory Search"
21.74
22.86
19.05
34.38
24.51
GPT-4o-mini
GPT-4o-mini
24.44
22.86
26.32
33.33
26.74
+2.23
WebShepherd-8PP
26.09
45.71
23.81
40.62
34,06
+9.55
* WebArbiter-7B
37.78
42.86
36.84
46.67
41.04
+19.13
w/o Trajectory Search"
23.91
31.43
28.57
56.25
35.04
GPT-4o
GPT-44-mini
26.67
37.14
42.11
40.00
36.48
+1.44
WcbShepherd-8BB
30.43
42.86
47.62
46.88
41.95
+6,91
WebArbiter-7B
44.44
42.86
52.63
56.67
49.15
+14.11
5.2.2
DOWNSTREAM ANALYSIS ACROSS DOMAINS
As shown in Tab. WebArbiter achieves substantial average improvements under both policy models, significantly outperforming all baselines. These gains stem from two main factors. First, reasoning mitigates spurious correlations that often mislead WebPRMs in domains such Shopping and Reddit. The improvements on Shopping are particularly pronounced, as these tasks require dense semantic retrieval and inference: stronger policies can propose more promising candidate actions, and WehArbiter's structured reward modeling further amplifies these advantages. Second, in Gitl Lab, tasks frequently admit multiple equivalent paths. WebShepherd is brittle under such variability, whereas WebArbiter reasons over historical trajectories and the current state to evaluate action validity, enabling stronger generalization dynamic workflows. We provide qualitative case studies in Appendix to further illustrate these failure modes of checklist-bare -based supervision. By contrast, CMS exhibits more cemplate-driven structure, where actions closely follow standardized patterns. In such settings, checklist chasel supervision remains comparatively effective. which narrows the relative performance gap. Overall, WebArbiter's reasoning- design consistently provides robust, interpretable, and scalable supervision across diverse domains.
CONCLUSION
We presented WebArbiter, a reasoning-first, principle-inducing process reward model that frames reward modeling as structured text generation and produces auditable step-level judgments with ratio- nales. Through reasoning distillation and RL, WebArbiter converts superficial correlations into robust, progress-aware signals that verify task advancement, yield consistent step-level judgments across trajectories, and generalize across dynamic web environments. To support systematic evaluation, we released WERPRMBENTE ronments for WebPRMs web navigation, covering four domains with diverse tasks and step-level preference annotations. Experiments demonstrate SOTA performance WEBPRMB ENCH and improvements in reward-guided trajectory search on WebArena-Lite, establishing principle-guided reasoning WebPRMs as robust and interpretable foundation for scalable web agents.
the first comprehensive evaluation benchmark spanning diverse envi-
LIMITATIONS AND FUTURE WORK
WebArbiter relies on text-based accessibility tree representations without visual observations; in environments where layout, spatial arrangement, or visual cues carry task-relevant information, this text-only formulation may miss critical signals for accurate progress assessment. Incorporating multimodal observations into the reward modeling process is a natural extension. Training and evaluation are also conducted exclusively in English-language web environments, where multilingual settings may introduce additional challenges such as divergent page structures and interaction conventions. Finally, current of WebPRMs focuses inference-time trajectory search; investigating whether based step-level supervision can serve as a dense reward signal during agent post-training, thereby improving the policy itself, is important open direction.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, 1lge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et Gpt-4 technical report. arXiv preprint arXiv:2303.08774. 2023.
Adept. Act-1: Transformer for actions. addet..a//bi adept.aa//loogactt-1 2022.
Anthropic. Introducing computer use, a new claude 3.5 sonnet, and claude 3.5 haiku. 2024a.
com 5-5 -mode wtw.oeeessssaa-oooooooeeeeeee
Anthropic. Introducing computer use, new claude 3.5 sonnet, and claude 3.5 haiku. https 7/ wttpa//www.........ooooeeaeaaaaa
October 2024b.
Anthropic. Claude 3.7 sonnet and claude code. 2025.
Léo Boisvert, Megh Thakkar, Maxime Gasse, Massimo Caccia, Thibault Le Sellier De Chezelles, Quentin Cappart, Nicolas Chapados, Alexandre Lacoste, and Alexandre Drouin. Workarema++:: Towards compositional planning and reasoning- common knowledge work tasks, 2025. URL
Mkttps//arxiv.orxt www./orr///saoooooooooo.e/ss..st 240 05291
Hyungjoo Chae, Sunghwan Kim, Junhee Cho, Seungone Kim, Seungjun Moon, Gyeom Hwangbo, Dongha Lim, Minjin Kim, Yeonjun Hwang, Minju Gwak, Dongwook Choi, Minseok Kang. Gwanhoon Im, ByeongUng Cho, Hyojun Kim, Jun Hee Han, Taeyoon Kwon, Minju Kim, Beong woo Kwak, Dongjin Kang, and Jinyoung Yeo. Web-shepherd: Advancing prms for reinforcing web agents, 2025. URI. htttps:///rxi..orr//absss org/abs 250.00.5775551777777
Xiusi Chen, Gaotang Li, Ziqi Wang, Bowen Jin, Cheng Qian, Yu Wang, Hongru Wang, Yu Zhang, Denghui Zhang. Tong Zhang, Hanghang Tong, and Heng Ji. Rm-rl: Reward modeling as reasoning, 2025. URL https: /tt////axiii org/ abs absaabas//555// 2200 02387
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang. Evan Rosen, al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261, 2025.
Xiang Deng, Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards generalist agent for the web. Advances in Neural Information Processing Systems, 11-28091-28111 2023.
Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H. Laradji, Manuel Del Verme, Tom Marty, Léo Boisvert, Megh Thakkar, Quentin Cappart, David Vazquez, Nicolas Chapados, and Alexandre I Lacoste. Workarena: How capable are web agents at solving common knowledge work tasks?, 2024.
http://aa.....ogga///////// 22400 047.11718
Yao Fu, Dong-Ki Ki Kim, Jaekyeom Kim, Sungryull Sohn, Lajanugen Logeswaran, Kyunghoon Bae, and Honglak Lee. Autoguide: Automated generation and selection of state-aware guidelines for large language model agents. arXiv preprint arXiv:2403.08978 2024.
Daya Guo, Dejian Yang, Haowei Zhang. Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12949 2025a.
Jiaxin Guo, Zewen Chi, Li Dong, Qingxiu Dong, Xun Wu, Shaohan Huang, and Furu Wei. Reward 14674
reasoning model, 2025b. URL htt.ttss//axxxv..rog/e
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu Yuanzhi Li, Shean Wang. Lu Wang. Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
Xu Huang, Weiwen Liu, Xiaolong Chen, Xingmei Wang, Hao Wang, Defu Lian, Yasheng Wang, Ruiming Tang, and Enhong Chen. Understanding the planning of Ilm agents: A survey, 2024.
URL htt/tt aaxiiii.o v. org/ abs. 02716
11

---

Published as a conference paper at ICLR 2026
Geunwoo Kim, Pierre Baldi, and Stephen McAleer. Language models can solve computer tasks. Advances in Neural Information Processing Systems, 36, 2024.
Jing Yu Koh, Stephen McAleer, Daniel Fried, and Ruslan Salakhutdinov. Tree search for language model agents, arXiv preprint arXiv:2407.01476, 2024.
Jing Yu Koh, Stephen McAleer, Daniel Fried, and Ruslan Salakhutdinov. Tree search for language model agents, 2025, URL htt////pp.//aaaa....o/////////.0
Manling Li, Shiyu Zhao, Qineng Wang, Kangrui Wang, Yu Zhou, Sanjana Srivastava, Cem Gokmen, Tony Lee, Erran Li Li, Ruohan Zhang, et al. Embodied agent interface: Benchmarking Ilms for embodied decision making. Advances in Neural Information Processing Systems, 37:100428- 100534, 2024.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference Learning Representations, 2023.
Chris Yuhao Liu, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He, Chaojie Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou. Skework-reward: Bag of tricks for reward modeling Ilms, 2024a. URL
wtt. httpo:///aaxv.orrr//////1111. 18451
Xiao Liu, Tianjie Zhang. Yu Gu, lat Long long, Yifan Xu, Xixuan Song, Shudan Zhang, Hanyu Lai, Xinyi Liu, Hanlin Zhao, Jiadai Sun, Xinyue Yang. Yu Yang, Zehan Qi, Shuntian Yao, Xueqiao Sun, Siyi Cheng. Qinkai Zheng, Hao Yu, Hanchen Zhang, Wenyi Hong, Ming Ding, Lihang Pan, Xiaotao Gu Aohan Zeng, Zhengxiao Du, Chan Hee Song. Yu Su, Yuxiao Dong, and Jie Tang. Visualagentbeneh: Towards large multimodal models visual foundation agents, 2024b. URL https::/arxiv. o abs / 240 06327
Yantao Liu, Zijun Yao, Rui Min, Yixin Cao, Lei Hou, and Juanzi Li. Pairjudge rm: Perform best-of-n sampling with knockout tournament, 2025. URL httph: arxiv. wrtp:oo////bb////// 13007
Xing Han LU, Amirhossein Kazemnejad, Nicholas Meade, Arkil Patel, Dongchan Shin, Alejandra Zambrano, Karolina Stańczak, Peter Shaw, Christopher J. Pal, and Siva Reddy, Agentrewarddenchh Evaluating automatic evaluations of web agent trajectories, 2025. URL httl::tts////a http:/..////xxvvoo/////
0tss/2504.8899
Kaixin Ma, Hongming Zhang, Hongwei Wang. Xiaoman Pan, and Dong Yu. Laser: Lim agent with state-space exploration for web navigation. arXiv preprint 230.:22009.0112,, 2023.
Dakota Mahan, Duy Van Phung, Rafael Rafailov, Chase Blagden, Nathan Lile, Louis Castricato, Jan-Philipp Fränken, Chelsea Finn, and Alon Albalak. Generative reward models, 2024. URL
www./og//////ooooo///ooo//oo.oee abs/ / 2410. 12832
Meta. Introducing meta laaaa 3: The most capable openly available Ilm to date. ai. meta. wtt.c//oo//l..cc.eeeeo
ama -3/ 2024.
Bingchen Miao, Yang Wu, Minghe Gao, Qifan Yu, Wendong Bu, Wenqiao Zhang, Yunfei Li, Siliang Tang, Tat-Seng Chua, and Juncheng Li. Boosting virtual agent learning and reasoning: step-wise, multi-dimensional, and generalist reward model with benchmark, 2025. URL
https://arxiz: /tt
OpenAI. Gpt-4o mini: advancing cost-efficient intelligence. 2024a.
OpenAI. Gpt-4o. p 2024b.
OpenAl. Introducing operator. hpwppeapei.oommtttttroocoonnnnnn 2025a.
OpenAl. Gpt-5 is here. openai wtt.ccom//g/g www./cptt.tt.
gpt-5
2025b.
12

---

Published as a conference paper at ICLR 2026
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022. URL
htttp:://rrxvv.. http//xxiv...rg/aa.. www.gg/bbs/2200 02155
Jiayi Pan, Yichi Zhang, Nicholas Tomlin, Yifei Zhou, Sergey Levine, and Alane Suhr. Autonomous evaluation and refinement of digital agents. arXiv preprint arXiv:2404.06477, 2024.
Junsoo Park, Seungyeon Jwa, Meiying Ren, Daeyoung Kim, and Sanghyuk Choi. Offsetbias:
Leveraging debiased data for tuning evaluators, 2024. URL https://axxiv.org abs/
2407. 06551
Sundar
Pichai and Demis Hassabis.
Gemini
2.5
flash.
deepminds nd. www.gooooogoooooooooooooooo proogle/modelie wwwpeelssgemmiiiiieaaaeee 2025.
Archiki Prasad, Alexander Koller, Mareike Hartmann, Peter Clark, Ashish Sabharwal, Mohit Bansal, and Tushar Khot. Adapt: As-needed decomposition and planning with language models. arXiv preprint arXiv:231 1.1.1.0.5 1.5772, 2023.
Zehan Qi, Xiao Liu, Iat Long Iong, Hanyu Lai, Xueqiao Sun, Wenyi Zhao, Yu Yang, Xinyue Yang. Jiadai Sun, Shuntian Yao, Tianjie Zhang, Wei Xu, Jie Tang, and Yuxiao Dong. Webrl: Training Ilm web agents via self-evolving online curriculum reinforcement learning, 2025. URL https: ftttss://ariv..oo wwtp:///orr/oo/oo/o//ooooooos/// 2tw1.//////1...2....... 02337
Qwen, An Yang. Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang. Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang. Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang. Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. URL https::/ http://.wrrr/ab///2221133
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang. Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. /ttppsss////artttt.oom abb/
34C2.03300
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rihf framework. arXiv preprint arXiv: 2409. 19256, 2024.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances Neural Information Processing Systems, 36, 2024.
Paloma Sodhi, S. R. K. Branavan, Yoav Artzi, and Ryan McDonald. Step: Stacked I1m policies for web actions, 2024.
Haotian Sun, Yuchen Zhuang, Lingkai Kong, Bo Dai, and Chao Zhang. Adaplanner: Adaptive planning from feedback with language models. Advances in Neural Information Processing Systems, 36, 2024.
Heyi Tao, Sethuraman TV, Michal Shlapentokh-Rothhnn Derek Hoiem, and Heng Ji. Webwise: Web interface control and sequential exploration with large language models. arXiv preprint arXiv:2310. 16042, 2023.
Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211.14277 2022.
Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce llms step-by-step without human annotations. arXiv preprint 1.Xiv:33/2.08935.. 2023a.
13

---

Published as a conference paper at ICLR 2026
Tianlu Wang, Ilia Kulikov, Olga Golovneva, Ping Yu, Weizhe Yuan, Jane Dwivedi- Yu, Richard Yuanzhe Pang, Maryam Fazel-Zarandi,, Jason Weston, and Xian Li. Self-taught evaluators, 2024a. /ttts:::: http://////xxxv...////a/a
02666
Zhilin Wang, Yi Dong, Jiaqi Zeng, Virginia Adams, Makesh Narsimhan Sreedhar, Daniel Egert, Olivier Delalleau, Jane Polak Scowcroft, Neel Kant, Aidan Swope, and Oleksii Kuchaiev. Helpsteer: Multi-attribute helpfulness dataset for steerlm, 2023b. URL https: org.abbb.orr/ab.......
2311.09522
Zhilin Wang. Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J. Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. Helpsteer2: Open-source dataset for training lop-performing reward models, 2024b. URL httpt::s/saai....tttt
2406. 08673
Zhepei Wei, Wenlin Yao, Yao Liu, Weizhi Zhang, Qin Lu, Liang Qiu, Changlong Yu, Puyang Xu, Chao Zhang, Bing Yin, Hyokun Yun, and Lihong Li. Webagent-rl: Training web agents via end-to-end multi-turn reinforcement learning, 2025. httttt:::/aaa/ /ttp://aaivvvvvvv orq hbwpg//bb///5// 16421
Tianhao Wu, Weizhe Yuan, Olga Golovneva, Jing Xu, Yuandong Tian, Jiantao Jiao, Jason Weston, and Sainbayar Sukhbaatar. Meta-rewarding language models: Self-improving alignment with
Ilm-as as-a--mta--uugee, 2024. /tLtatts::// //axii..aaivv.vaa org/ abs/ 19594
Zhiheng Xi, Yiwen Ding, Wenxiang Chen, Boyang Hong, Honglin Guo, Junzhe Wang, Dingwen Yang, Chenyang Liao, Xin Guo, Wei He, et al. Agentgym: Evolving large language model-based agents across diverse environments. arXiv preprini urXiv:2406.0415/, 2024.
Ziyi Ye, Xiangsheng Li, Qiuchi Li, Qingyao Ai, Yujia Zhou, Wei Shen, Dong Yan, and Yiqun Liu. Learning llm-as-a-judge for preference alignment. In The Thirteenth International Conference on Learning Representations, 2025.
Ori Yoran, Samuel Joseph Amouyal, Chaitanya Malaviya, Ben Bogin, Ofir Press, and Jonathan Berant. Assistantbench: Can web agents solve realistic and time-consuming tasks?, 2024. URL
https: wttp://wwwww..rra/////00..00
Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction. arXiv preprint arXiv:2408.1524 2024.
Yao Zhang, Chenyang Lin, Shijie Tang, Haokun Chen, Shijie Zhou, Yunpu Ma, and Volker Tresp. Swarmagentic: Towards fully automated agentic system generation via swarm intelligence. arXiv preprint ar arXiv:2506.1 2025a.
Yao Zhang, Zijian Ma, Yunpu Ma, Zhen Han, Yu Wu, and Volker Tresp. Webpilot: A versatile and autonomous multi-agent system for web task execution with strategic exploration. Proceddings of the AAAI Conference on Artificial Intelligence, volume 39, pp.
2025b.
Yao Zhang, Yu Wu, Haowei Zhang, Weiguo Li, Haokun Chen, Jingpei Wu, Guohao Li, Zhen Han, and Volker Tresp. Groundedprm: Tree-guided and fidelity-aware process reward modeling for step-level reasoning. 2025c. URL
14942
Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. The lessons of developing process reward models in mathematical reasoning. arXiv preprint arXiv:2501.07311 2025d.
Longtao Zheng, Rundong Wang, Xinrun Wang, and Bo An. Synapse: senjectory-as-exene prompting with memory for computer control. In The Twelfth International Conference on Learning Representations, 2023.
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Pro- ceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Bangkok, Thailand, 2024. Association for Computational Linguistics. www.oogggabbb///// 13372
14

---

Published as a conference paper at ICLR 2026
Enyu Zhou, Guodong Zheng, Binghai Wang, Zhiheng Xi, Shihan Dou, Rong Bao, Wei Shen, Limao Xiong, Jessica Fan, Yurong Mou, Rui Zheng. Tao Gui, Qi Zhang, and Xuanjing Huang. Rmb: Comprchensively benchmarking reward models in Ilm alignment, 2025. URL https:
//arxiv.org/ abs/ 2210.44100000000 09893
Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo. Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: realistic web environment for building autonomous agents. arXiv preprint 11Xh:
Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Webarena: A realistic web environment for building autonomous agents, 2024. URL httpstt:://// /ttppii.:///xi..../ org.orgggo...... abs/ 2307 13854
15

---

Contents
Introduction
2
2
Related Work
2.1
LLM-Based Autonomous Web Agents
2.2
Reward Models in Reasoning and Web Tasks
3
Methodology
3.1
Background
3.2
Training Dataset Construction
3.3
WebArbiter: Principle-Inducing Reasoning Process Reward Model
3.3.1
Training Overview
3.3.2
Stage I: Reasoning Distillation
3.3.3
Stage 2: Reinforcement Learning
WEEPRCBENCH
4.1
Benchmark Construction
Evaluation Protoco]
5
Experiments
5.1
WEBPRMBENCH
5.1.1
Experimental Setup
5.1.2
Main Results
5.1.3
Analysis of Training Design
5.2
Heward-Guider Trajectory Search
5.2.1
Experimental Setup and Implementatio
5.2.2
Downstream Analysis across Domains)
10
6
Conclusion
10
Notation Summary
17
Example of Preference Dataset
17
Training Details
18
Prompt Repository
18
Benchmark Construction
19
Preference Pair Construction
19
Dataset Composition and Statistics
20
Analysis of BoN Ace vs. Pairwise Ace Evaluation
21
Inference- Time Scaling
22
H
Case Study: WebArbiter WebShepherd
22
H.1
Milestone Creation under Multiple Equivalent Paths
22
H.2 Merge Request Identification under Ambiguous Context
23
16

---

Published as a conference paper at ICLR 2026
A
NOTATION SUMMARY
For clarity, we summarize the main notations used throughout this paper:
web environment, defined by state space S, action space A, and observation space O.
state transition function T: SS A S.
I: task instruction.
Sp Op, up: state, observation, and action at step
Cpi reasoning trace associated with action Up-
Eepi reasoning traces of previously executed actions.
T ( ap): trajectory of length P.
.(,,...,EE)) jz): structured justification of length L, consisting of explicit reasoning and
final verdict.
Zg: WebArbiter model parameterized by 8.
ÿ: predicted preference verdict.
Drain; DSFT, DRL: training datasets for supervised distillation and RI..
Deechh evaluation dataset for WEBARBITER-BBAE
at: action from a successful trajectory (positive label).
or a a:; rejected action (q ......,)) Q).
Q: number of distractors in WEBARBITER-BENT BENCI (set to 4)..
M: dataset size (training or evaluation).
R(x, ú): verifiable reward signal.
B EXAMPLE OF PREFERENCE DATASET
To illustrate the construction preference pair, consider a task instruction:
Instruction I: "Find the 2026 conference submission page the ICLR website."
Current observation Op: [356] banner 'header', role=='banne: [359] link 'Home' [380]
button 'Select Year (2026)' [386] button 'Dates' [391] button 'Calls' [396] button 'Guides" [401] button 'Organization" [403] heading 'ICLR 2026'
History (a<p, Cep)::
a1: Search "ICLR", c1: "I need to find the official ICLR website first. Let me search for 'ICLR' to locate it."
the ICLR official website link in the search results. Clicking on will take me to the ICLR homepage where I can find the conference submission information."
a2: Click link "ICLR homepage", o2: "I
Candidate actions and reasoning traces:
(a) (a)::)):: Click link "Call for Papers": c/: "I can see 'Call for Papers' link on the ICLR homepage. This link would likely lead the submission details page, which should contain information about the 2026 conference submission process that I'm looking for."
-aZ, c)): Click "About" link; ef: "I can see an 'About' link on the ICLR homepage. Since I need find the 2026 conference submission page, the 'About' section might contain conference overview information including links to submission details or important dates for the 2026 conference."
Label y:: af is preferred.
This example represented in the dataset as:
(I, Ops Ocpp Ceps (a)) (a). y = at).
17

---

Published as a conference paper at ICLR 2026
C TRAINING DETAILS
All training conducted on NVIDIA A100-80GB GPUs with fixed random seeds. Our training framework is based on LLama-Factory (Zheng al. 2024) and VERL (Sheng al. 2024
Distillation Stage. We train the model for epochs with a learning rate of 8e-4, using LoRA with rank of 128. We apply a cosine learning rate scheduler with warmup ratio of 0.1. We the batch size 256 and the maximum sequence length 8,192 tokens.
RLVR Stage. We employ the VERI. framework for GRPO training. The learning rate is set to 7e-6 for the 7B model, and 9e-6 for the 3B variant. The training uses fixed batch size of 512 with mini-batch size of 128, and adopts Fully Sharded Data Parallel (FSDP) for enhanced memory efficiency. For rollout generation,
deploy vLLM with tensor parallelism of 4 and GPU memory utilization limited to 0.4. Response sampling uses standard parameters (temperature= I.O, top-p=1.0), generating 7 candidate responses per prompt. We apply KL regularization with a coefficient of 1.0 10- and clip ratio of 0.2. The maximum input sequence length is 8, 192 tokens, and the maximum response length is 4,096 tokens.
D PROMPT REPOSITORY
WebArbiter
You are skilled expert at evaluating assistant responses. You should evaluate given responses based the given judging criteria, Given the context of the conversation and two responses from the Assistant, you need to determine the better response. Provide an overall comprehensive comparison upon them. HH Intent #ddd
(intent)
HNN AXTREE 4bd
Note: [bid] is the unique alpha-numeric identifier at the beginning of lines for each element in the AXCree. Always use bid to refer to elements in your actions. [observation]
### Trajectory ###
Note: The trajectory contains the sequence of previous actiens and their corresponding thoughts. Each entry reflects the agent's internal reasoning ('thought and the concrete operation performed ('action'). (trajectory)
#### start url ####
[start_ istart_url
#### current url ####
The URL provides clues about the user's position in the application flow, Use both the path and query parameters to infer page type (e.g., homepage, search results, product detail, cart, checkout).
(current_url)
#### Assistant Responses #### [The Begin of Response 1]
THOUGHT:
[thought1]
ACTION:
(action1]
[The End of Response 1]
[Tee Begin of Response 21
ThoUGHT:
(thought2)
ACTION;
(action2)
18

---

Published as a conference paper at ICLR 2026
[Tee End of Response ## Output Instructions ###
Format your output strictly using the following xMi-style tags: <State>Summarias the current state based the JRb, AXTree, and previous actions. Include what page the user is currently on, and what relevant uI elements or information are visible.</Stater
<Criteria>Other potential criteria specific to the query and the context, and the weights of each critoria.</rritezl.. <Analyeis>Comparo Response and Response in detail accerding the <state> and <Criteria> fatiaa..///naayy/ </analysis> <hnswer>Response 1 or Response 2</Answe>>
Rules for <Answer>;
If Response is better, output exactly: Renswereee 1</Answer>
If Response 2 is better, output exactly: <hnswerRRespcnse
2</Answer>
Important Notes:
Be objective and base your evaluation strictly on centent the responses.
Do not let the response order, length bias your judgment.
E BENCHMARK CONSTRUCTION
E.1 PREFERENCE PAIR CONSTRUCTION
Positive samples. We construct WEBPRMBENCH using the successful trajectories from AGEN- TREWARDBENCH, human-verified evaluation suite that aggregates over thousand trajectories generated by multiple LLM-based web agents across diverse real-world environments. Each trajec- tory in BGENTREWARDDENCH annotated for success and execution quality by expert annotators, providing reliable source of environment -rrounddd optimal behavior. From this dataset, we select only those trajectories that complete each task with the minimum number of steps. Each trajectory independently reviewed by annotators to ensure monotonic progress and to verify that no redundant or detour actions are present. When deviations are identified, annotators revise the trajectory to recover the shortest valid execution path consistent with successful task completion. For consistency, missing reasoning traces are completed to ensure that every state-action pair paired with a coherent rationale. The resulting actions from these validated minimal-she trajectories as positive labels, reflecting actions empirically verified to succeed in the real web environment.
Negative samples. For each state, sample four alternative actions and their associated reasoning from a diverse ensemble of policy models, covering both open-source and proprietary LLMs. The pool includes high-capacity instruction-tun tuned models such as Qwen2.5-7B 72B-Instruct (Qwen
ct al. 2025), Llama-3.3-8B 70B-Instruct (Meta) 2024), as well as frontier commercial models
including GPT-4o 4o-mini (OpenAl 2024a Claude-3.5-Haiku Claude-3.7-Sonnee (Anthropic
20246 2025), and Gemini-2.5-Flasl Gemini-2.5-Pre (Comanici et al., 2025). This ensures that
alternative actions exhibit broad stylistic and policy diversity rather than reflecting any single model's reasoning behavior. Since alternative actions may still succeed under certain web interfaces, we apply rule-based filtering procedure remove actions that remain potentially valid. We retain only actions that are clearly invalid or non-progressing, ensuring that negative samples correspond to failures under the actual environment dynamics rather than differences in reasoning style. To ensure consistency and avoid false negatives, the filtered actions are manually reviewed, and any remaining actions that appear potentially valid are discarded. If more than four valid rejected actions remain after filtering, randomly sample subset to maintain consistent number of action pairs per instance. All rationales are truncated to a fixed length to reduce formatting noise while preserving semantic content.
19

---

Published as a conference paper at ICLR 2026
Table 6: WEBPRMBENCH Website Visit Counts
Domain
Domain
#
Domain
service-now.com
212
w
48
wa-forum-xl-2
38
wa-gitlab-xl-1
23
wn-shopping--dmiin-te
21
google.com
17
wa-openssrrettaa-x122
wa-shopping-admma
ryanair.com
wa-forum-xl-1
12
wa-shopping-xI-2
11
last.fm
10
delta.com
duckduckgo.com
8
wa-gitlab-xl-2
8
redbox.monster
target.com
7
united.com
wa-shopping- xI-1
kohls.com
soundcloud.cm
spothero.com
yellowpages.com
6
ametheatres.com
5
exploretock.com
qatarairways.com
5
aa.com
foxsports.com
4
ikea.com
4
kayak.com
4
marriott.com
rentalcars.com
4
sixflags.com
travelzoo.com
yelp.com
discogs.com
gamestop.com
koacom
3
mta.info
3
tesla.com
cabelas.com
2
rottenlomatoes.com
2
extremoweethherwacch.com
Chosen Rejected
3735
80
(77.5%)
(70.
Percentage (%)
60
40
20
(1.7%)
click
An
scroll
select option
others
Figure Action-type distribution in WEBPRMBENCH.
E.2 DATASET COMPOSITION AND STATISTICS
The final benchmark consists of 1,150 step-level preference instances across four environments, each containing one environment-verte verified positive action and four negative alternatives.
Website distribution. Tab. 6 summarizes the distribution of visited websites in WEBPRMBENCH, highlighting the diversity and long-tailed nature real-world web environments covered by the benchmark.
Action-type distribution. Fig. 3 reports the action-type distributions of environment-verife positive actions and negative actions WEBPRMBENCH. In both sets, click and fill constitute the majority of actions, consistent with common interaction primitives in real-world web navigation. The distribution of rejected actions closely mirrors that of chosen actions, with only minor shifts in relative proportions, indicating that negative actions not dominated by rare structurally distinct types but arise from the same gigh-freuuency operations as successful actions. As a result, action-type identity alone provides reliable signal for correctness. Effective discrimination, therefore, requires
20

---

Published as a conference paper at ICLR 2026
Table 7: Standard deviation of model scores under BoN and Pairwise evaluation across web environ- ments on WEBPRMBENCH.
Std. deviation
Mind2Web-BoN
0.149
Mind2Web-pairwisc
0.060
WebArema-BoN
0.153
WebArrena-pairwisee
0.081
AssistantBench-BooN
0.139
AssistantBench-ppirriiss
0.093
WorkArena-BotY.
0.173
BarkArrea-pairwiis
0.116
Mind2Web (r 0.936)
WebArena (r=0.956)
100
80
70
60
Painwise Accuracy
50
AssistantBench (r=0.844)
WorkArena (r= (r=0.843)
90
80
60
50
20 40
60
80
20
40
60
80
BoN Accuracy (%)
Figure 4: Correlation between BoN and Pairwise Acc across web benchmarks. Each scatter point corresponds to a PRM. We report the correlation coefficient r for each environment. While the two metrics are strongly correlated across all environments, BoN exhibits higher variance and provides finer-grained discrimination among models, particularly in complex web environments.
assessing whether an action advances task progress under the current state, rather than relying on action-type assumptions or global frequency-basee heuristics.
F ANALYSIS OF BoN Acc vS. Pairwise Ace EVALUATION
We analyze how BoN Ace and Pairwise Ace behave as evaluation metrics for WebPRMs WEBPRM- BENCH. This comparison is practically important because WebPRMs are commonly used to rank multiple candidate actions during agent execution, whereas Pairwise Acc only measures correctness on isolated preference pairs. In our benchmark. BoN Ace imposes stricter evaluation criterion by requiring the correct action outperform all distractors simultaneously, making it more representative of realistic multi-candidate decision- maniinaaannngggn scenarios.
BoN Ace Provides Stronger Discriminative Power Across Environments. Tab. reports the standard deviation of model scores under BoN and Pairwise Acc. Across all four environments, BoN
21

---

Published as a conference paper at ICLR 2026
95
95-
Roerage Patreive
75-
75-
20-
70-
WobArbiiir-388
WrbArbilen-388
WobArbier-78
Wrb-4rbiti-88
@sampled llggswll)
llngscale)
(a) Pairwise Accuracy
(b) BoN Accuracy
Figure 5: Inference-time scaling of WebArbiter. Left: Pairwise and Right: BoN Ace the number of sampled reward evaluations K increases.
Acc consistently exhibits higher variance than Pairwise Acc, indicating substantially less score com- pression and larger separation among models. This effect is particularly pronounced in Work where complex interaction dynamics and harder distractors amplify small weaknesses into measurable performance gaps. These results confirm that BoN Ace offers finer-grained discrimination among WebPRMs, especially in settings where robust multi-candidate judgment is required.
BoN Acc and Pairwise Acc Are Consistent but Not Equivalent. Fig. shows that BoN Acc and Pairwise Acc are strongly positively correlated across all environments. This indicates that the two metrics capture broadly aligned notions of WebPRM quality and induce similar overall ordering of models. However, the correlation strength varies across environments, reflecting differences in interaction structure and distractor difficulty.
G INFERENCE-TINE SCALING
We further analyze how WebArbiter benefits from increased inference-tinee time compute by varying the number of sampled reward cvaluations. As shown in Fig. 5 both Pairwise and BoN Acc improve consistently the sampling budget increases for WebArbiter-31 and WebArbiter-7B. confirming that the proposed reasoning-basee WebPRM supports inference-time scaling. The improvements are moderate for Pairwise Acc but substantially more pronounced under the stricter BoN Acc, highlighting the advantage of additional inference computation in multi-distractor ranking scenarios.
H CASE STUDY: WEBARBITER VS. WEBSHEPHERD
This section presents two GitLab-based case studies that concretely illustrate the failure modes of checklist-driven WebPRMs. These cases highlight how checklist-style supervision can become brittle under structural variability, and how WebArbiter's reasoning-baut based evaluation yields more reliable action preferences.
H.1 MILESTONE CREATION UNDER MULTIPLE EQUIVALENT PATHS
The task to create milestone for an upcoming merge operation. At the current step, the agent on the GitLab project homepage, where the left navigation directly supports milestone management, alongside other entries such as "Project information" that lead to altermative but non-essential paths. Two candidate actions considered: navigating through "Project information" or directly entering "Issues", as shown Fig. 6
exposes an "Issues" entry that
WebShepherd evaluates these candidates using checklist-style criteria that emphasize procedurally typical navigation patterns. In GitLab, however, multiple interface paths may lead to the same functionality, and conventionally expected steps not always necessary in the current context.
22

---

Published as a conference paper at ICLR 2026
As result, WebShepherd may favor navigating through "Project information" despite the fact that milestone creation already accessible via "Issues", introducing an avoidable detour. In contrast, WebArbiter reasons the current state and task objective assess whether an action directly contributes to task progress. Observing that the required functionality is already available, assigns higher preference entering "Issues" and deprioritizes redundant navigation steps. This example reflects a common characteristic of GitLab workflows: path multiplicity with varying informational value, under which checklist-driven supervision struggles generalize consistently.
H.2 MERGE REQUEST IDENTIFICATION UNDER AMBIGUOUS CONTEXT
The second task requires locating specific merge request referenced by 404 link, checking for reply, and responding accordingly. The agent is initially presented with merge request overview page listing multiple candidates, none of which explicitly linked to the given URL, while global search function is available to resolve this ambiguity, as shown Fig. 7 The agent can either open one of the visible merge requests or initiate search identify the correct target.
Checklist-basee based supervision tends to favor actions that satisfy immediate procedural milestones, such entering merge request page, without explicitly verifying whether the selected entity matches the task specification. Consequently, opening an arbitrary merge request may be preferred even though the task's 's referent has not yet been identified. WebArbiter, by contrast, evaluates action validity by reasoning about task preconditions and required evidence. Since identifying the correct merge request a prerequisite for any subsequent review or response, actions that do not support disambiguation are penalized. WebArbiter therefore prefers initiating search and defers content- interaction until the task context is correctly grounded. This case further illustrates how checklist-bans rewards can conflate interaction progress with task progress in dynamic settings, whereas reasoning-bann based evaluation maintains alignment between actions and task intent.
23

---

Published as a conference paper at ICLR 2026
I Instruction
(A.cps Cep)
Trajectory
Create milestone for the upcoming task of merging all branches storting on 2044 and ending on March 30, 2044
Thought: On the GitLab project homepage,
Action: click [2826]
Thought: The current Page is on
(aps cp) Action Candidate
22) (ap: Cp)
Action Candidate 2
Thought: Start from "Project information" locate managed.
Thought:
The milestone created from the "Issues" section.
Action: click [281]
Action: [331]
D
dotfiles
You can't push pull repositories using SSH until you add SSH key your profi
Project information
SSH key
Don't again
Repasitory
Issues
The Auto DeuQps pipaline has been anabled and and-wil be used no alternative CI con
Container registry not enabled on this GitLab instance. Ask administrator to enz
11
Merge requests
CI/CD
Settings
More information
Security & Compliance
Byte Blaze
dotfiles
13 Deployments
Packages and registries
dotfiles t
Infrastructure
Project ID: 193
Monitor
553 Commits 1 Branches
WebShepherd Analysis
WebArbiter Analysis
Action Candidate
Induced Principles
Checklist 1: Navigate to Repository
Correct element reference (25%) the used present and appropriate?
....Thhouhht omitted): Yes
Checklist 2: Locate Milestone Creation Option in Project Information
Goal alignment (45%)
chosen action
move toward creating milestone for the specified task?
Clicking "Project information" likely to lead to the project overview area where milestone creation option are located, satisfying Checklist 2's goal, .In Progress Checklist 3: Set Milestone Dates
Efficiency relevance (20%) Does avoid
unnecessary detours?
Structured Justification:
...oohhogght omitted): No
Action Candidate 1
Conrect element: [281] exists and is clickable.
Checklist 4: Confirm Milestone Creation
Goal alignment: Weak, Clicking" "Project information" is
....TThought omitted):
untikely to reveal milletonee controls.
Efficiency: Adds an extra, possibly irrelevant step.
Action Candidate
Action Candidate 2
Checklist 1: Navigate to Repository
Correct element: [331] exists and is clickable.
...(Thought omitted]: Yes
Goal alignment: Good - the Issues section is indeed
Checklist 2: Locate Milestone Creation Option in
milestones
Project Information
Efficiency: Slihtlly better than Action candidate 1; directly pursues the needed functionality without intermediate detour.
this action unlikely to reach the project information area where milestone creation options are located.; No Checklist 3: Set Milestone Dates
Preference Verdict:
.... (Toogght omitted): No
Given the weights, Action candidate 2 scores higher on Goal alignment and Efficiency, while both are correct on element reference.
Checklist 4: Confirm Milessone Creation
...(Thought omitted): No
Answer: Action Candidate 1
Answer: Action Candidate
Figure 6: Milestone creation under multiple equivalent paths in GitLab, Checklist-based WebShep- herd prefers procedurally typical but non-essential navigation step under path multiplicity, while WebArbiter reasons over the current state and correctly selects the action that directly advances milestone creation.
24

---

Published as a conference paper at ICLR 2026
I Instruction
(a Czp)
Trajectory
Go to the merge request on 404 link I have to review, find if the outhor of the merge request responded the end, and reply "Thank you" he did. Otherwise remind him simple @
Thought: To achieve the goal of reviewing..
Action: click (157)
Thought: The goal
(a op)) Action Candidate 1
(az, Cp)) Action Candidate
Thought: Use the search bar find the merge request related to the 404 link. Action: click [114]
Thought: Click merge request, "WIP- Post", to review relevant request. Action: click (574)
Search GitLab
Merge requests
Open
Merged
Closed
All
Recent searches
Reviewer
Byte Blaze
WIP Pitfalls accessible components
al Typroject/al hymp://c//a1yyprooecc.oom//// 1490 created years ago Erik Kroes
accessible dropdowns
al Typroject/al Typroject.com!147 created years ago by James Bateson
WebShepherd Analysis
WebArbiter Analysis
Action Candidate 1
Induced Principles
Checklist Access Merge Request Details
Correctness (60%) Action valid bid and is executable
....Thhughht omitted); No
Checklist 2: Review Merge Request Content
2. Intent alignment (25%) Moves toward finding the target merge request
This action does not open any merge request page: No
Reasoning quality (15%) Thought logically supports the action
Checklist 3: Check for Author Response
....Thoughh omitted); No
Checklist Send Reminder No Response
Structured Justification:
....(Toought omitted): No
Action Candidate 1
Checklist Confirm Task Completion
Correctness: Clicks [114], valid search textbox.
...(Thouugh omitted):
Alignment: Focuses the search box but does not actually
search; limited progress.
Action Candidate
Reasoning: Intends search, reasonable
Checklist Access Merge Request Details
Action Candidate 2
....Thoughh omitted):
Correctness: Clicks [574], which exists but opens an unrelated merge request.
Checklist Review Merge Request Content
Alignment: Opening an unrelated request does not help
Clicking '574' navigates to the merge request's page.
locate the 404-link merge request.
aligning with the goal for Checklist 2.; Yes
Reasoning: Thought implies clicking the target request, but chosen element unrelated, showing clear mismatch.
Checklist 3: Check for Author Response
....ghhuught omitted): No
Checklist 4: Send Reminder No Response
Preference Verdict:
....Thoughht omitted]: No
Action candidatee closer the user's intent, while Action candidatt opens unrelated merge request the goal.
Checklist Confirm Task Completion
...(Thoughh omitted); No
Answer: Action Candidate 2
Answer: Action Candidate
Figure 7: Merge request identification under ambiguous context. When the target merge request is not yet identified, WebShepherd prematurely commits to an arbitrary request, whereas WebArbiter reasons about task preconditions and prioritizes disambiguation via search.
25


Reviews:
[{"review_id": "DY5P4xBKW7", "reviewer": "Reviewer_xtyG", "summary": "The paper presents a novel approach to enhancing the performance of web agents in complex, multi-step tasks. It identifies the limitations of existing reward models, which often rely on coarse scoring or checklist-based methods that lack interpretability and robustness. To address these challenges, the authors introduce WebArbiter, a reasoning-first model that formulates reward modeling as a text generation task. The results demonstrate that WebArbiter significantly outperforms existing models, showcasing its robustness and practical value in real-world web tasks.", "questions": "See above", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces a fresh perspective on reward modeling by framing it as a text generation task. This innovative approach allows for the generation of structured justifications for actions, which is a significant departure from traditional scalar scoring and checklist-based methods. By integrating reasoning distillation and reinforcement learning, the authors create a model that not only enhances decision-making but also provides a more interpretable framework for understanding agent behavior.\n\n2. By addressing the limitations of existing reward models and providing a more reliable framework for web agents, the paper has the potential to impact various applications in automated web tasks. The advancements presented in WebArbiter could lead to more effective and interpretable web agents, ultimately enhancing their utility in real-world scenarios.", "weaknesses": "1. The description of the WEBPRMBENCH construction is somewhat abstract. It would be helpful to include concrete examples or illustrations from the benchmark to make the process easier for readers to understand.\n\n2. The main paper references an appendix (e.g., “Appendix XXX”), but no appendix is actually included.\n\n3. The figures in the paper are low-resolution. Please replace them with higher-quality versions to improve readability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0NgqMJDXqW", "reviewer": "Reviewer_EzxW", "summary": "This paper proposes WebArbiter, a principle-guided reasoning process reward model (PRM) for web agents. It aims to improve the interpretability, reliability, and robustness of reward modeling for long-horizon web interaction tasks. Unlike scalar WebPRMs (which output coarse numeric rewards) or checklist-based generative WebPRMs (which rely on fragile templates), WebArbiter formulates reward modeling as structured text generation—producing reasoning chains that derive principles, compare candidate actions, and conclude with a preference verdict. The model is trained via a two-stage pipeline:\n(1) Reasoning distillation from a stronger teacher LLM to teach principle-based justification, and\n(2) Reinforcement learning with verifiable correctness rewards to align reasoning with factual outcomes.\nExperiments on the newly released WEBPRMBENCH benchmark and WebArena-Lite show that WebArbiter achieves significant performance improvements over previous WebPRMs and even surpasses strong LLM-as-judge baselines like Gemini Flash and GPT-5.", "questions": "1.\tWhy are DeepSeek-R1 and Rubrics-based reward models not included in the baseline comparison, given their conceptual proximity?\n\t2.\tHave the authors compared downstream supervised agent performance (e.g., post-training agents) when using WebArbiter versus other PRMs?\n\t3.\tHow does WebArbiter’s inference latency or search overhead compare to baseline self-reflective reasoning LLMs like GPT-5?\n\t4.\tCan the principle-guided reasoning process generalize to non-web reasoning tasks, or is it specifically tailored to the web environment?\n\t5.\tHow is the benchmark annotation quality ensured? Were multiple annotators or verifiers used to avoid label bias?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.\tSolid empirical improvement: WebArbiter-7B achieves +10.9% BoN accuracy over Gemini Flash and +48% over WebShepherd, showing consistent advantages across diverse benchmarks.\n\t2.\tInterpretability: The model generates explicit, auditable reasoning chains for each action decision.\n\t3.\tBenchmark release: The introduction of WEBPRMBENCH enriches the evaluation ecosystem for process reward models in web agents.\n\t4.\tThorough ablation: The authors analyze the roles of principle guidance, reasoning distillation, and RL alignment in model performance.", "weaknesses": "1.\tLimited conceptual novelty —\nThe paper’s design (reasoning-based PRM + two-stage pipeline) closely mirrors prior reasoning reward models like RM-R1, RewardReasoningModel, and Generative Verifier. The only difference lies in applying it to the web navigation domain. The overall framework lacks a strong theoretical or methodological innovation beyond scene adaptation.\n2.\tMissing key baselines —\nTable 2 compares WebArbiter only against WebShepherd and a few LLM-as-judge models. However, it omits recent strong generative reward models (e.g., Rubric-RM, Generative Verifier from DeepSeek), which are more directly comparable. This omission weakens the evidence for claimed superiority.\n3.\tNo evaluation of downstream supervision quality —\nIn Table 4, the paper reports WebArbiter’s gains in reward-guided trajectory search but does not compare against the downstream effects of other PRM or generative models listed in Table 2. Without showing whether the higher WEBPRMBENCH score translates into better supervised model performance, it’s hard to confirm the practical utility of the learned reward model.\n4.\tLack of comparison with reasoning-capable LLMs —\nTable 4 focuses on reward-guided trajectory search with GPT-4o-mini and GPT-4o, but omits comparisons with stronger reasoning models such as GPT-5 or DeepSeek-R1, which already exhibit built-in search and reflection abilities. These models might achieve comparable or better results without explicit PRM supervision, calling into question whether WebArbiter’s explicit reward modeling still offers clear advantages.\n5.\tBenchmark dependency —\nWEBPRMBENCH is designed and used solely by the authors, which may limit reproducibility and objectivity. Its annotation criteria and preference construction may bias evaluation toward the proposed model’s reasoning style.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "85xqLtusuM", "reviewer": "Reviewer_Pe8d", "summary": "This paper introduces **WebArbiter**, a novel Process Reward Model (WebPRM) designed to provide step-level supervision for web navigation agents. Unlike traditional outcome-based reward models that offer sparse and delayed feedback, or existing scalar and checklist-based WebPRMs that are brittle and lack interpretability, WebArbiter formulates reward modeling as a text generation task. It produces structured justifications that conclude with a preference verdict, identifying the action most conducive to task completion. The model is trained in two stages: first, reasoning distillation from a stronger teacher LLM to instill principle-guided reasoning; second, reinforcement learning to align verdicts with correctness and improve generalization. The authors also released **WEBPRMBENCH**, a comprehensive benchmark across four diverse web environments (Mind2Web, WebArena, AssistantBench, WorkArena).", "questions": "1. **Multilingual Generalization:** Does WebArbiter support non-English web environments? If not, are there plans to extend its reasoning and principle induction to multilingual contexts?\n2. **Reward Hacking Mitigation:** How does the model avoid reward hacking during RL training? Are there mechanisms to ensure that the generated justifications genuinely reflect task progress rather than exploiting shortcuts?\n3. **Scalability and Efficiency:** What are the inference latency and computational requirements of WebArbiter compared to simpler WebPRMs? How feasible is it for real-time web agent control?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- **Interpretability and Robustness:** WebArbiter’s reasoning-first approach provides auditable justifications, making it more interpretable and less reliant on superficial cues compared to scalar or checklist-based methods.\n- **Effective Training Pipeline:** The two-stage training (reasoning distillation + RL) effectively combines the benefits of principled reasoning and correctness alignment.", "weaknesses": "- **Limited Multilingual Support:** The paper does not discuss multilingual capabilities, which limits its applicability in global web environments.\n- **Potential for Reward Hacking:** While the RL stage aligns verdicts with correctness, the risk of reward hacking (e.g., over-optimizing for superficial patterns in justifications) is not addressed.\n- **Narrow Benchmark Scope:** Although diverse, WEBPRMBENCH is limited to four environments, and its real-world coverage (e.g., multilingual or highly dynamic sites) is unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c5RZZChmWg", "reviewer": "Reviewer_wacB", "summary": "WebArbiter introduces a principle-grounded, generative process reward model designed to evaluate reasoning trajectories in web agents. The model first undergoes reasoning distillation via SFT to induce explicit, interpretable principles from a teacher LLM, and is subsequently optimized through GRPO using binary verifiable rewards. It employs LoRA adapters on top of Qwen2.5-Instruct (3B/7B) for efficient RL training. The authors also propose WEBPRMBench, a new benchmark covering multiple web-agent environments (Mind2Web, WebArena, AssistantBench, WorkArena).", "questions": "1. Why was Qwen2.5 chosen instead of Qwen3-8B, given that baselines (WebShepherd) use newer backbones? \nWould results hold under an architecture-matched setting?\n\n2. Does the WebArbiter generalize to challenging QA tasks that involve multi-round web search, like BrowseComp and HLE?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The PRM is not just a classifier but a generative verifier. It produces explicit reasoning chains before outputting a verdict token. This design enables process-level supervision.\n2. Rewards are automatically derived from web-environment logs, whether the predicted “correct” or “incorrect” verdict matches the actual success of an action. This eliminates extra human annotator costs.\n3. WebArbiter achieves strong accuracy not just on Mind2Web (its training base) but also on unseen environments like AssistantBench and WorkArena, showing transfer capability. This indicates its principle-based reasoning generalizes beyond template matching.", "weaknesses": "1. The binary verifiable reward assumes one correct action per state, which is unrealistic. In tasks in web environment, several actions could succeed, but only one is labeled “Correct.” This introduces false negatives and reward noise,  a core limitation in deterministic PRM setups.\n2. The evaluation omits new RL-based WebAgent baselines such as WebAgent-R1 [1] and WebSailor [2] that achieve strong results on WebArena and QA tasks. \n3. Using Qwen2.5-7B instead of Qwen3-8B (as in WebShepherd) introduces an architecture-generation gap. Qwen3 includes better instruction alignment, longer context, and multi-turn reasoning improvements. Reported performance gains in WebArbiter may partly reflect architectural differences, not purely training-method superiority.\n\n[1] Wei, Zhepei, et al. \"WebAgent-R1: Training Web Agents via End-to-End Multi-Turn Reinforcement Learning.\" ICML 2025 Workshop on Computer Use Agents.\n\n[2] Li, Kuan, et al. \"WebSailor: Navigating Super-human Reasoning for Web Agent.\" arXiv preprint arXiv:2507.02592 (2025).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)