Web-Shepherd: Advancing PRMs for Reinforcing Web Agents
Authors: Hyungjoo Chae
Sunghwan Kim
Junhee Cho
Seungone Kim
Seungjun Moon
Gyeom Hwangbo
Dongha Lim
Minjin Kim
Yeonjun Hwang
Minju Gwak
Dongwook Choi
Minseok Kang
Gwanhoon Im
ByeongUng Cho
Hyojun Kim
Jun Hee Han
Taeyoon Kwon
Minju Kim
Beong-woo Kwak
Dongjin Kang
Jinyoung Yeo
Venue: NeurIPS 2025 spotlight ()
Keywords: Web Agent
Reward Model
LLM
Primary area: applications

Abstract: Web navigation is a unique domain that can automate many repetitive real-life tasks and is challenging as it requires long-horizon sequential decision making beyond typical multimodal large language model (MLLM) tasks.
Yet, specialized reward models for web navigation that can be utilized during both training and test-time have been absent until now. Despite the importance of speed and cost-effectiveness, prior works have utilized MLLMs as reward models, which poses significant constraints for real-world deployment. To address this, in this work, we propose the first process reward model (PRM) called Web-Shepherd which could assess web navigation trajectories in a step-level. To achieve this, we first construct the WebPRM Collection, a large-scale dataset with 40K step-level preference pairs and annotated checklists spanning diverse domains and difficulty levels. Next, we also introduce the WebRewardBench, the first meta-evaluation benchmark for evaluating PRMs. In our experiments, we observe that our Web-Shepherd achieves about 30 points better accuracy compared to using GPT-4o on WebRewardBench. 
Furthermore, when testing on WebArena-lite by using GPT-4o-mini as the policy and Web-Shepherd as the verifier, we achieve 10.9 points better performance, in 10x less cost compared to using GPT-4o-mini as the verifier. 
Our model, dataset, and code are publicly available at https://github.com/kyle8581/Web-Shepherd.

WEB-SHEPHERD Advancing PRMs for Reinforcing Web Agents
Hyungjoo Chac
Sunghwan Kim
Junhee Cho
Seungone Kim?
Seungjun Moon
Gyeom Hwangbo²
Dongha Lim?
Minjin Kim
Yeonjun Hwang²
Minju Gwak?
Dongwook Choi
Minseok Kang²
Gwanhoon Im?
BncongUng Cho²
Hyojun Kim²
Jun Hee Han
Taeyoon Kwon
Minju Kim²
Beong-woo Kwak? Dongjin Kang²
Jinyoung Yeo2t
Georgia Institute of Technology
Departmenn Artificial Intelligence, Yonsei University CCarnegie Mellon University
Abstract
Web navigation is unique domain that automate many repetitive real-life tasks and is challenging as it requires beyond typical multimodal large language model (MLLM) tasks. Yet, specialized reward models for web navigation that can be utilized during both training and test-time have been absent until effectiveness, prior works have utilized MLLMs as reward models, which poses significant constraints for real-world deployment. To address this, in this work, we propose the first process reward model (PRM) called WEB-SHHPPHERR which could assess web navigation trajectories in step-level. To achieve this, we first construct the WEBPRM COLLECTION, a large-scale dataset with 40K step-level preference pairs and annotated checklists spanning diverse domains and difficulty levels. Next, we also introduce the WEBREWARDBENCH the first meta-evaluation benchmark for evaluating PRMs. In our experiments, we observe that our WEB- on
sequential decision making
Despite the importance of speed and cost-
SHEPHERD achieves about 30 points better accuracy compared to using GPT-4o
Furthermore, when testing on WebArena-lite by using GPT--4omiin as the policy and WEB-SHePHER as the verifier, we achieve 10.9 points better performance, 10x less cost compared to using GPT-4o-mini as the verifier. Our model, dataset, and code are publicly available at LINK.
100
Gemtni-2.55
Gene- 2.5-M-2-333
Web- Steppardd
1001
cost-efficient
WebRewerdDench Score
WeaRewarrBeeeen (MAR)
60-
Shepherd
80
60
GPT
Cludde337-Some
60
Ocmini-2
Owen- Owon-2.5-VL.11 2.5VVL-88
20
Jstept
(ilep)
(Un))
40
16
Cost (USD) 107
Figure 1: Perforraance and cost- cost-efficiency of WYBB-SHEHEER (3B). WEB- BEB-SHEPHERD achieves the state-of-the-an performance while requiring significantly lower cost compared to existing baselines.
"Equal contribution. This work done while Hyungjoo Chac Yonsei University. Conresponding author. Contact: hchae360gatech. edu and jinyeodyonsei.cc ac. kr.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Web browsers serve as a common interface for countless digital tasks, making automation in this space a natural focus for recent advances in intelligent agents. Recent advances in multimodal large language models (MLLMs) have enabled agents to handle basic web interactions, such as retrieve addresses from map services navigating simple webpages [1. 2]. However, current agents remain highly unreliable, often exhibiting brittle behaviors such as repeatedly entering the same query when encountering minor issues, eventually failing the task [3-5]. This unreliability primarily stems from the long-horizon nature of web navigation, requiring agents to reason across multiple steps and maintain goal-directed planning, which MILMS often find challenging [6]. Hence, create better performing web agent, there is a need for better learning methods and inference-time algorithms.
One effective method that allowed large language models (LLMS) to perform well across various tasks is using a reward model to perform search at test-time (e.g., Best-of-n), or using for Reinforcement Learning (e.g., RLHF). However, specially trained reward models have been under-explored in the web navigation domain. Prior works such as Pan et al. [7] and Koh et al. [8] do not train separate reward models, but instead employ MLLMs as evaluators in inference- time algorithms,, which has fundamental problems. First, using the evaluation from prompted MLLMS becomes a significant constraint in web navigation where speed and cost are crucially important. For example, using only GPT-40 for tree search on WebArena (consisting of 812 queries) requires approximately $1 $14,000, and running inference on one A100 takes 40 hours, which is major obstacle to deploying MLLMS as web navigation agents in real-world scenarios. Additionally, throughout our experiments, we confirm that prompting MLLMS performs worse than trained reward models. In summary, considering speed, cost, and performance, specially designed reward models for web navigation are absolutely necessary.
To address these challenges, we present WEB-SHEPHERRD which is, to the best of our knowledge, the first reward model trained specifically for evaluating trajectories of web navigation. In particular,
is designed as a process reward model (PRM) rather than an outcome reward model (ORM), because unlike other domains, ORM cannot be integrated into test-time algorithms in web navigation. For example, in mathematics, and the ORM can choose one, but in web navigation, if an LLM makes eight attempts to book a plane ticket, the airplane ticket cannot be refunded, so decisions about which action take must be made at the process level. Furthermore, even during training-time timee PRM can provide more fine-graing -neiirai reward signals, making more reliable than ORM [9, 10]. DEB-SHEPEERD employs a structured checklist that explicitly decomposes high-level user instructions into clear, interpretable subgoals. By referencing this checklist as evaluation criteria, WEB-SHEPHERY accurately assesses step-level progress, enabling precise and robust guidance throughout agent trajectories.
L.LM can write multiple solutions
The key contribution of this paper is that also provide suite of training data and benchmark to test PRMs for web navigation. First, we release the WEBPRM COLLECTION, which contains human-crafted instructions that covers diverse tasks across multiple difficulty levels. The notable feature of the WEBPRM COLLECTION is that it contains 40K step-level annotations for which action an agent should take and that each instruction contains an annotated checklist- of subgoals that enable WEB-SHEPPHEE to make accurate judgments. Second, we release the WEBREWARDBENCH the first meta-evaluation benchmark to assess PRMs in web navigation. The WEBREWARD allows practitioners to test newly proposed PRMs without running resource- intensive web navigation agents, enabling efficient testing of different design choices and conducting ablation experiments. WEB-SHEPHERD achieves 85. 500%% performance on the W WEBREWARDBENCY (WebArena set), significantly outperforming GPT-4o-mini with prompting at 5.0%. Furthermore. when using SHEPHRD'''''S 's reward as guidance in tree search on GPT--4-mini policy, it achieves 34.55% success rate on WebArena-lite [1, 11], confirming it 10.9 points better in performance, and 10x more cost-effective than using GPT-44- mini as the evaluator.
sequences
2 Related Work
MLLM-based web agents. Multimodal large language models (MLLMs) have emerged as power- ful foundation models for web agents due to their strong generalization capabilities and adaptability to diverse interface. Previous work has leveraged MI.LMs to complete web tasks via carefully designed instructions, often augmented with external tools (e.g., grounding module or verification) [12-15] or workflow [4]. Moreover, other approaches have trained MLLM-based agents to imitate expert trajec-

---

tories using next-token prediction objective [16-18]. While these models perform well in nn-iisribution, they often fail generalize to unseen environments. To overcome these challenges, recent research has increasingly focused on inference-time scaling [7, 19] or reinforcement learning (RI.) [20-22], which enables agents to improve decision-making through reward feedback.
Inference-time scaling for web agents. Inference-time scaling has emerged as a crucial approach for multi-turn interactions in web environment. Recent studies have explored techniques such as tree search [23, 19], long chain-of-thoupg
(Co')) [24, 25), and incorporating verifiers or judges to enhance agent performance with natural language feedback [26, 27]. For example, Pan al. evaluator to assess whether a trajectory is successful; if not, they apply Reflexion [26] to retry based on the generated feedback. Extending this direction, other work [8. 5] investigates an interesting direction that tries to search the optimal browsing path with prompted value function and A * l-ke algorithm and world model.
[27] use prompting-basee
Rewards for web navigation. Prior works rely on binary rewards (success or failure) [21, 22] from rule-based evaluations that require human annotation and lacks scalability in dynamic web environ- ments [1. 3]. To address these, recent studies have explored leveraging LLMs via prompting [14, 7] or training outcome reward models (ORMs) [20]. However, binary reward offers limited guidance for credit assignment, especially in long-bou long-horizon tasks. To enable more informative feedback, the reason- ing literature has introduced process reward models (PRMs), which assign step-level reward [9, 10]. Building on this idea, recent work has explored using LLMs to estimate state-action values by prompting [19, 28, 29]. Nevertheless, the reliability and efficiency of MILMSs process-level reward models remain underexplored. In this work, we aim to develop a PRM for web agents to support effective learning and cost-efficient inference-time guidance.
3 Preliminaries
We formulate the web navigation problem as a partially observ- able Markov decision process (POMDP) defined by the tuple (S, A, O, T, R), where S the set of environment states, A is the set of agent actions, O is the set of observations, T(s' s, a) the transition function, R(s, a) is the reward function. At each time step t, the agent receives vation ot O that only partially reflects the true underlying state E S. In the context of web environments, or consists of two modalities: (1) an accessibility tree of text sequence of intractable elements that captures the hierarchical and se- mantic structure of the webpage elements [1, 30], and (2) a
User Instruction (I) Buy WIt 100000
Observation (c
rrodeee-rendered obser-
LEVERAGE: REDEMPTION
[43] Inc. Amnanoni art /ertoe (howncamssnn amaston conveet-nwwoog
[254] starchoos Secreh mancel aunsom pietewlerr
[345) ink "Chossc kinguage shopping Amazznn United....
rendered screenshot image imn depicting the visual appearance of the browser [3]. Given these observations, the agent selects an action ae E A from a discrete set of browser-level com- mands, including operations such as cl click(i), scroll(d), and type("tcxt"), where is the index of DOM or accessibility node, and d denotes a scroll direction or offset. The agent's goal is to select actions that maximize the expected reward over a trajectory (0), or,
(503)
bution 'Spaad to Charge ge Langaage or
Action (a) f234 "Sony Wil-1 (000094")
Figure 2: Example of web naviga- tion under POMDP.
4 WEBPRM COLLECTION
The major challenge of building PRM in web navigation is the lack of training dataset. To address this, we collect WERPRM COLLECTION, the first dataset for training PRMs for web agents. Our goal is to collect a dataset DD that contains (I,O, C, A+, A-), where A is a sequence of chosen actions (aT azz ...., a;) i.e., an expert trajectory, and A- is a sequence of rejected actions (aj 4g ..... 0, along with the checklist C, observations (01, 02, ..., 0n), and user instruction I.
4.1 Collecting User Instruction and Expert Trajectory
From human experts, we collect user instructions 7 and the chosen actions A- We select websites that permit access via playwright from the pool of sites used in Mind2Web [16]. Prior to annotation,
3

---

Reai-world Websites (70+)
Easy
Modium
Y Iigudd
sourddlouu)
Oranuction
11:00
expaote
apps
rraa
N...
Stsgia
Importan stoce
Chooldat
Collection doW Ditinet
rep autrocke
Ci Wicsoter
Antohats chosen trajectories from expert and rejected action from MLLM
Foodbook
Human Annotator
Construct
feedback
Instruction
Checkfist
Chosen Action
C: Rejected Action
2004) 4.2"
Chaceite I: Search Product
Goaat Lete the procuct name the product lsiig.
Deltaabet goste
AZTOON Mac
ACTION diaat119
cirilly
Observation
Foodback Judgement
len cask repuaayy wckeile
Checcillr Acconss Product Page spooils
Choiliee Seaech Prodect
Chaustal Bearch for thokket
cie fommu desoccel
sroduct lasing
buttr degen 2 deacee
intormation abour the Apple Macilcck Pro
Checklie Accoss Page
Chocidas Accsss Product Page
infermation
Warranty informatio The aperr caver
Goal Scrall through the prod detal
Ths agerts surnet banawa sae
velty
the "WARSOVI' SU "OOt!" seccion
20 Chouliir
Checki
Figure 3: Overview of the dataset collection process of WEBPRM COLLECTION (top) and an example instance of our dataset (bottom).
all annotators participated in three-hour training session designed to familiarize them with our annotation toolkit and clarify the differences between human and agent browsing behaviors. Following annotation, all collected data were reviewed by panel of 10 human evaluators to ensure quality and consistency. During this process, we filtered out invalid trajectories that could not be reproduced, as well as vague instructions prone to misinterpretation. Annotators were instructed to craft instructions spanning three difficulty levels: easy, medium, and hard.
4.2 Annotating Checklist and Rejected Action
Checklist. To mitigate bias toward specific websites and reduce sensitivity to action orderings, construct coarse-grained checklists that emphasize meaningful task progress over exact execution steps. For example, fine- aineigranedd actions such as filter A and filter B are abstracted into higher- subgoal like filtering. This abstraction enables the model to generalize across semantically equivalent strategies. Given instruction I and expert trajectory A+, we use GPT-4o to generate subgoal analysis and corresponding checklists.
Rejected actions. To collect rejected actions a; we sample 5 candidate actions from diverse poli- cies and select those that differ from the expert action a," However, some of these alternatives may correspond to valid but different actions to- ward task completion (e.g., fill (423, "Sony Camera") vs. click (search_ ), rather than being truly suboptimal or incorrect. To minimize such cases, we apply rule-based filtering and col- lect up to five rejected actions a, per expert action at" More details about dataset construction are provided in Appendix B.
buttibution Instructions Websites
comple:
seafth corpple
4b
Action
1200k
Travel
Shapping
Entertainment
Tracrl
Shopplng
Entertatrmeett
Arthess
Other
Pute
Moxi GaYA
Tusrt
Trajectory Dificutty Level
Checkfiss Oifhcuty
40
Trabottry Length
of Chardiiet
4.3 Dataset Statistics
wnthMiinppyy..... oour
As shown in Figure 4, we analyze two key as- pects across difficulty levels: the length agent trajectories and the number of checklist subgoals. The left violin plot illustrates that trajectory length increases with difficulty. Easy tasks generally re- quire fewer steps (median 5), whereas medium tasks show more variability (median 9), and hard tasks involve significantly longer trajectories
Easry
Medium
Hard
Medium
Figure 4: Statistics of WEBPRM COLLECTION.

---

Step 1. Checklist Generation
Stap Rewardinn
Reward--wided Search
Refine with Reward Model feedback
User instrrctiiion
OLt of C
Were Agant
REA
Feedback F
Chacod al Yea 2122 Progress Cheedlat No
RL with Reward Model
Desencatin
Checkfal
C
Avarage
0.609
0.531
0-472
Dwiveren
Reward
Agent
WEB-SHEPHERD
Process Rewaad Medel for Web Navigation
Succesful
Figure 5: Overview of PEBSSHEHEERD (left) and its diverse use cases (right).
(median = 20), with some exceeding 40 steps. This indicates that our difficulty annotation effectively reflects the complexity and required interaction depth. The right violin plot shows that the number of checklist items also grows with task diffficlly,, though the range is more concentrated. Easy tasks typically involve 3-4 checklist items, while medium and hard tasks consistently require 4-5 subgoals.
5 WEB-SHEPHERD
In this section, we introduce WEB-SHEPHERE a process reward model designed to provide dense and reliable supervision web agents and enable more informative credit assignment. We train WEB-SHEPHIERD on the WEBPRM COLLECTION to support two key functionalities: (1) generating task-specific checklists, and (2) assigning rewards based on checklist completion.
5.1 Step 1: Checklist Generation
As illustrated Figure 5, FEB-SHEPHERD first generates a checklist that outlines key intermediate milestones for achieving the user's goal. Given an instruction I, it produces checklist C' comprising sequence of natural language subgoals (g), 92, ... Is). This checklist then serves as the foundation for reward prediction, enabling WEB-SHEPHERD to track progress toward the goal. We further investigate the impact of checklist quality in Section 7.1.
5.2 Step 2: Reward Modeling with Checklist
Reward modeling as next-token prediction. To leverage the internal reasoning capabilities of MLLMs,, choose next-token prediction modeling loss over targets formed by concatenating the feedback and the judgment .7, treating the full sequence an observation o, and an answer a, the model is trained to generate the corresponding feedback and judgment an auto-regressive manner. The loss is defined as:
learning objective [31]. We optimize the language
coherent response. For example, given input consisting of checklist C,
CNTP =
Σ log Po yee, C,o,a).
(I)
where y = [F; J] denotes the concatenated feedback and judgment tokens. This objective encourages the model to learn to evaluate the trajectories based on the checklist with reasoning and provide valuable feedback that explains the evaluation.
Scoring process reward. Since the reward is predicted via token generation, the output resides in discrete space. To obtain continuous reward signal, several mapping strategies can be employed. One approach is to sample multiple output sequences and compute the average reward. Alternatively, we employ verbalizer [32] to estimate soft probabilities over label tokens (e.g.. "Yes", "No"", and "In Progress")) using the logits from the LM head. At inference time, WER-SHEPHERD generates the feedback F P(- C, a) and compute the reward for each checklist item using the probabilities
5

---

of "Yes" and "In Progress" tokens follow:
& 1 P("Yes P.
ak(o,a)=
P(Inn "I,
(2)
where I. denotes the number of checklist and rı is the score assigned to the k response. The final reward is computed as the average: r(o, a) EE (x( We provide an empirical comparison of different scoring strategies in Appendix E.3.
6 Experiments
To evaluate the effectiveness of PRMs for web navigation, conduct comprehensive experiments in assigning process-level reward for web agents, focusing on both the accuracy of reward assignment and the utility of those rewards in improving agent performance.
6.1 WERREWARDBENCH
In developing PRMs, a reliable benchmark (c.g., RewardBench [33]) is essential for evaluating their performance. However, there does not yet exist benchmark specifically designed to evaluate how accurately models assign process rewards to web agents' trajectories. To address this, we introduce WEBREWARDBENCH, benchmark that directly measures the accuracy of predicted rewards.
6.1.1 Setup
use two data sources, Mind2Web and WebArena, to obtain user instructions for web navigation tasks. For Mind2Web, utilize the expert demonstrations provided in the dataset. In contrast, since expert trajectories are unavailable in WebArena, we manually annotate them. As a result, construct reliable benchmark for evaluating PRMs, we follow the setup of Kim et al. [34] and collect preference pairs (ot, at ( )=1)). }), where each observation oe paired with chosen action and four rejected actions. Additionally, we provide reference checklists for each tasks to ensure fair and consistent evaluation. Further details on benchmark construction are provided in Appendix D.1.
Benchmark construction.
obtain 69 instances from WebArena and 707 instances from Mind2Web. To
Metrics. We evaluate process reward prediction using the following three metrics: (1) Mean Reciprocal Rank (MRR): The average of the reciprocal ranks of the preferred action in the list of all candidate actions sorted by predicted reward. A higher MRR indicates that the model consistently ranks the preferred action closer to the top. (2) Step Accuracy (Acc. step): The proportion of steps where the model assigns the highest predicted reward the preferred action at among the five candidates. (3) Trajectory Accuracy (Ace. traj): The proportion of full trajectories where the model ranks a highest at every step among the candidate actions.
Baselines. Prior work has leveraged prompted MILMMs to obtain process-level rewards by exploiting their reasoning and image understanding capabilities [5, 8). Following this approach, construct baselines using representative MLLMs from both open-source and closed-source categories. For open-source models, we use GPT-4o-mini and GPT-4o; for closed-source models, adopt Qwen- 2.5-VL-72B, which are widely used in recent literature.
Implementation of WER-SHEPHERD. We train SHR-HHEDEERD dataset using the follow- ing base models: for text-only settings, we use Qwen2.5-3B (35) and Qwen3-8B [36]; for multimodal settings, Qwen2.5-VL-3B [37]. All models trained for 3 epochs using LoRA [38).
6.1.2 Results
MILMS struggle with assigning correct process rewards. We evaluate the ability of models to accurately assign process rewards WEBREWARDBENCH under different input types (text only vs. text and image) and with or without using the checklist. As shown in Table 1, state-of-the--at MLLMs struggle to provide reliable rewards for web navigation tasks. This limitation is particularly
Step accuracy omitted due the limited space. We provide the full results in Appendix

---

Table 1: Evaluation results WEBREWARDBENCH. T: text observation, I: image observation.
Mind2Web
WebArena
Model
Inputs
Checklist
Craas Task
CmassTask
Cines Website
Cross Domain
Teur
MRR
(traj)
MRR
(Inaj)
MRR
(Iraj)
(traj)
T
47.5
0.0
47.6
13.5
45.4
0.8
34.4
5.0
GPT-4omini
T
63.9
5.0
66.1
12.8
63.3
12.4
60.0
15.0
T+1
48.2
0.0
49.3
0.0
49.5
0.8
38.7
0.0
T+1
58.8
2.5
64.4
5.1
63.0
4.1
53.8
5.0
T T
56.9
5.0
55.8
2.6
59.8
3.3
59.2
15.0
GPT-4o
67.4
7.5
70.3
5.1
70.2
11.6
69.7
15.0
T+
52.5
5.0
52.2
0.0
52.8
1.7
49.7
5.0
T+1
62.4
5.0
68.1
15.4
65.1
6.6
59.7
10.0
T
54.2
54.6
5.0
Qwon-255VLL-22
T
39.4
0.0
62.4
0.0
57.9
1.7
52.3
5.0
T+I
50.1
2.5
47.6
0.0
49.8
0.8
43.1
0.0
T+1
52.9
2.5
53.5
2.6
52.0
2.5
47,3
0.0
WEB-SHEPHERD (3B)
T
87.6
55.0
88.0
43.6
87.2
47.1
91.1
60.0
T+1
85.0
42.5
87.3
41.0
84.4
37.2
92.5
65.0
WEB SHEPHERD (8B)
T
88.3
57.5
87.9
51.3
91.3
61.2
97.8
85.0
evident in the trajectory accuracy metric. this measure, models frequently fail to assign correct rewards consistently at each time step within single task. In contrast, WEB-SHEPHERD significantly outperforms all baselines, demonstrating a substantial performance gap across all benchmark settings.
Checklist allows reliable reward assignment. Table demonstrates that both baseline and our models benefit significantly from the checklist in assigning rewards. Checklists lead to more accurate and consistent reward assignments, tory accuracy across all baselines. These results suggests that checklists serve as valuable guidance, helping models maintain coherence in predicting the process reward. Furthermore, as shown Figure 6, when we conduct ablation studies with models that are trained to either assign rewards without checklists or checklists without feedback, we observe a substantial performance drop. These findings underscore the importance of both checklists and feedback for assigning reliable rewards.
evidenced by improvements in trajec-
100
feedback wio cheokiss feedback Shopherd rours)
WesRewardBeen
Multimodal input does not always improve perfor- mance. Contrary to our expectations, incorporating mul- timodal input does always lead performance gains; in some cases, using multimodal input even degrades the performance. For example, when using GPT-4o the reward model, we observe notable improvement in tra- jectory accuracy only the cross-website Mind2Web subset. This observation is consistent with the findings of Xue et al. [6]. which suggest that processing inputs from multiple modalities can introduce ambiguity and act as a source of noise, ultimately hindering the model performance.
80
20
MRR (step)
ACC Istep)
ACC (traj)
Figure 6: Results ablation study with WEB-SSEPHERR (3B).
6.2 Reward-Guided Trajectory Search
Reward-guides search using Best-of-n (BoN) sampling offers practical proxy for evaluating the capability of reward model to guide policies [10, 39, 40J. Notably, it allows us assess the potential for reward overoptimization without relying reinforcement learning. In addition, it provides an effective approach to adapting an MLLM policy without fine-tuning [8. 5, 41).
Setup. We evaluate our approach on WebArena-lite and WorkArena [30] in an online setting. WebArema-lite [11] subset of WebArena [1], comprising 165 instructions with error-corrected judge code from the earlier version. WorArena is remote-hosted benchmark of 33 tasks based on the widely-used ServiceNow platform. Among 5 action candidates sampled from the policy, the action that is assigned the highest reward is executed. For the policy,
GPT-4o-mini. and compare

---

Table 2: Success rates of trajectory search with GPT-4o-mini and GPT-4o policy WebArena-lin
Policy
PRM
Checklist
Shopping
CMS
Reddit
GitL.ab
Map
Total
A
wo Trajectory Search
NIA
21.74
22.86
19.05
34,38
19.35
23.64
GPT-4o-mini
GPT-4o-miii
13.04
9.52
15.75
-7.80
21.74
31.43
14.29
34.38
16.13
24.24
+0.60
WEB- SHEPHERD (3B)
32.61
19.05
32.26
32.12
+8.088
WLO-SHEPHERE (8B)
26.09
45.71
23.81
40.62
35.48
34.55
+10.90
w/o Trajectory
23.91
28.57
56.25
19.35
31.52
GPT-4o
GPT-4o-mini
21.74
31.43
28.57
40.62
12.90
26.67
-4.85
SHEPHERD (3B)
25.81
+-5.45
WEB SHEPHERD (SB)
30.43
42.86
47.62
46.88
35.48
39.39
+7.87
Table 3: Success rates of trajectory search with GPT-4o-miin as policy on WorkArena.
Checklist
Knowledge
List-filter
Service Catalog
w/o Trajectory Search
N/A
50.00
0.00
10.00
0.00
5.00
25.00
2.22
9.39
GPT-4o-mini
10.00
10.00
0.00
6.67
20.00
5.56
12.42
+-3.3
Wab-Shepherd (3B)
57.50
14.00
10.00
0.00
10.00
10.00
11.11
14.85
+5.46
Web Shepherd (SD)
65.00
14.00
20.00
0.00
10.00
20.00
7.78
15.76
+6.37
performance when guided by our proposed PRM versus a prompt-based PRMs. We report the success rate (SR), which measures the proportion of tasks which the final state satisfies the condition.
Main results. We present the results in Table 2. Interestingly, when using GPT-4o-mini as the reward model, we observe a slight improvement in the GPT-4o-mini policy. However, overall performance degrades when GPT-4o is used as the policy model, dropping from 31.52 26.67. In contrast, applying WEB-SHEPIEERD leads to substantial performance gains for both the GPT-4o-mini and GPT-4o policies across nearly all domains. Notably, WEB-SHEEPHERD boosts the GPT-4o-mini's browsing performance from 23.64 to 34.55, which is about points higher than GPT-4o without trajectory search. These results suggest that BEB-SHEPHERD remains effective in the online setting, even when paired with a stronger policy model.
Results on WorkArena. To assess the robustness across domains, we also evaluate our models on WorkArena, benchmark completely out-of-domain for WEB-SHEPIIRRE As shown in Table 3, trajectory search guided by the PRM improves the success rate WorkArena, where the Total score increases from 9.39 to 12.42 when comparing the baseline without trajectory search. Moreover, model consistently outperforms GPT-4o-mini across all domains except for Menu. We attribute the relatively low performance in the Menu domain to the complexity of its multi-level dropdowns and embedded search boxes, which cause the policy model to produce unreliable action candidates.
Can WEB-SHEPHERD provide useful feedback? To evaluate the effectiveness of the feedback generated by WEB-SHEPHERD, we conduct experiments in which the agent performs step-wise refinement using our feedback, similar to the Self-Refine [42]. Specifically, the agent refine current action with the feedback when its current reward is lower than the previous reward assigned by WEB- SHEPHERD. Interestingly, contrary to previous findings by Chae et al. [5] suggesting that step-wise feedback from models is not helpful and may even be detrimental, we observe notable improvements when incorpo- rating model feedback during refinement. A possible explanation that WER-SHEPHERD not only learss the impact of actions but also identifies patterns that characterize suboptimal behavior.
Table 4: Results of refinement with feed- back from WEB- TEB--SheHeRRD using GPT- 4o-mini as the policy on WebArena- lite.
Models
SR
A
w/o refine
23.64
WYED-SHPHEER (3B)
26.67
+3.03
WEB SHEPHERD (8B)
27.88
+4.24
7 Discussion
7.1 The Impact of Checklist Quality in Reward Prediction
We assess the quality of checklists generated by both baseline models and SHEPHERDD using G-Eval [43], with GPT-4o as the evaluator. To ensure reliable evaluation, provide the reference

---

WoskkwordBooh scceed
60-
Chocline from Whe deeeheee DMN Di Grouns Trush Chocdat
Overll Saln
4.09
4.14
4.16
4.0
Ieb-Sraptert Wwo-Grephed
OPf-loetii
GFf45
yanle de CINcE uç
Figure 7: Evaluation of checklist quality (left) and its relationship to reward accuracy (right).
Table 5: Effect of instruction proportion and the number of rejected actions on model performance.
(a) Effect of instruction proportion.
(b) Ablation on the number of rejected actions.
Proportion of instruction
Mind2Web
WebAreea
Mind2Web
MRR (step)
ACC (traj)
MRR (slep)
ACC (traj)
(taj)
rejected actions
MRR (step)
ACC (trag)
MRR (step)
ACC
0.25
68.38
11:34
65.97
10,00
71.42
12.79
63.04
10.00
73.57
77.66
17.55
76.47
20.00
0.75
83.64
33.63
88.09
55.00
79.70
24.91
77.46
20.00
Ours (3B)
87.62
48.57
91.06
60.00
checklist the evaluator alongside each generated checklist. The details of G-Eval are provided in Appendix E.7. As shown in Figure 7 (left), all models, except WEB-SHEPHERD (3B), generate high quality checklists. Notably, model, which is trained solely for checklist generation, achieves the highest score. Motivated by this result, we also release a standalone version of the checklist generation model. To better understand the role of checklist quality, we analyze reward prediction performance using checklists from various sources: an early version of our model (A), our final models (B and C) and ground-truth checklists (D). In Figure 7 (right), we observe that high-quality checklists lead to more reliable reward assignments. However, the results also suggest that model's 's capability imposes natural ceiling on reward prediction performance, regardless of checklist quality.
7.2 Training Objective: Bradley-Terry Modeling vs. Generative Reward Modeling
The Bradley-Terry (BT) loss has been widely adopted as a training objective for learning reward models based human preferences [44]. However, its suitability for building PRMs in web navigation remains an open question. To investigate this, we compare WEB-SHEPHERD (3B) with variant trained using the BT loss, with the identical training data. As shown in Figure 8, the BT-based model underperforms than ours, particularly in WebArena subset ofut-of- We find that the BT loss fails to effectively leverage the checklist for reward assignment, resulting in weaker sensitivity to task progress. These findings suggest that BT modeling's 's key limitation- poor generalization observed across domains also manifests in PRMs for web navigation.
100
Modtwing
Webh Shepherd (ours)
WesRowardBench Score
80
78
60
60.8
7.3 Cost Efficiency of WEE-SHEPHEER
We assess the cost efficiency of SHEPHERDD by comparing to API-based models. For WEB- SHEPHERD, costs are estimated using the hourly rate of an A100 80GB GPU instance (S1.19/hour), com- bined with throughput measured via vLLM [45]. Each
MRR Istegl
ACC (stes) AGC (raa)) MRR (stegi ACC (teal ACC Itra])
Mind2Web
WebArena
Figure 8: Analysis the training objective.
instance averages 81 31,287 input and 1,953 output tokens and we compute cost API-based models using publicly available prices. As shown in Figure (right), performance at the lowest cost per 1,000 instances roughlyy 10x cheaper than GPT-4o-mini and 100x cheaper than GPT-4o.
delivers the best

---

7.4 Data Scaling Law for PRM Training
We conduct analysis on the effect of the (1) number of instructions, and (2) number of rejected actions in the dataset on the performance of the PRM. Specifically, construct datasets using the subset of WehPRMCollection 0.25, 0.5, and 0.75 percent of instruction and its corresponding chosen-rejected pairs and 1,2, and 3 number of max rejected actions. We trained variants of See-Shepherd with these datasets using the same model (i.e., 2owen22.533B-IInsmmae and hyperparameters. The results are shown in Table 5a and Table 5b.
Overall, if we use about half of the original dataset (in terms of both the number of instructions and the number of rejected actions), there is a drastic decrease in ACC (traj) both of the benchmarks. Especially, in the out-of -f--ma- benchmark, WebArena, instruction ablation results in ACC (traj) decreases from 60.0 to 15.0, which suggests it failed to generalize to unseen domains. In rejected ablation, only decreasing by one rejected action is critical, resulting 60.0 + 20.0 ACC (traj) score in WebArena. These results highlight that both the number of instructions and the number of rejected actions are critical for training an effective PRM; reducing either significantly impairs generalization, particularly in out-of-domain settings such WebArena.
Home Kitchen
Task: Find the Mintendo Switch game card storage
groggmmm strenge
Roward Curree
Theectes
Plagst: Busion cowepe?
Thaughs SAarch yoor DRN CUNE
Thought Sotora mon catgorter...
Thrught: strrgg colio....
opicay
Attler 132015900)
the polcyy generate incarnec action sember state, the tnapecloyy to deviade from the intended path, it becomes deficca asain an appropprate reward
Shap--h navigation wio etviorments
Figure 9: Trends of reward score for successful and failed cases of neward-guided trajectory search.
7.5
Case Study
Figure 9 presents qualitative analysis of WEB-SHIEPHERD. We sample 30 success and 30 failure cases and plot the reward score trends as a function of the normalized step index over the trajectory length. While failure cases exhibit relatively flat reward curves, successful cases show a smooth and consistent increase reward over time. In addition, we identify the three most frequent sources of error: (1) incorrect reasoning about the effects of actions, where the model fails to anticipate future rewards appropriately-for for example, assigning a low reward to scroll action that would have revealed the desired information in the next step; (2) misinterpretation of the observed state, often due to not properly accounting for the impact of previous actions, leading the model to repeat actions unnecessarily; and (3) hallucinations in the generated checklist, such as assuming the presence of filtering functionality on a website when no such feature exists.
8 Conclusion
This paper studies process reward modeling for web navigation and introduces "'BB-SHEPHERD, the first PRM designed specifically for evaluating web agent trajectories. We also release two key resources to support the development of PRMs: (1) WEBPRM COLLECTION, a dataset consisting of human- annotated instructions and expert trajectories, and (2) WEBREWARDBENCH, a reliable benchmark designed evaluate the capabilities of PRMs. Our experiments demonstrate that process- level rewards improve inference-ting -irrenece-iieeeeeee search, achieving 34.55% success rate on WebArena-lite compared to 23.64% for baselines. The checklist-basel approach offers a generalizable framework that could extend beyond web navigation to other sequential decision-makinn domains where sparse rewards and partial observability remain challenging. We believe WEB-Ste SER-SHEPHERD establishes a foundation for developing more reliable web agents through interpretable reward modeling.
10

---

Acknowledgement
This work was supported by Institute of Information & communications Technology Planning & Evaluation (ITTP) grant funded by the Korea government (MSIT) (No. RSS2024-004558882
National AI Research Lab Project). This research was supported by the MSIT (Ministry of Science, ICT), Korea, under the Global Research Support Program in the Digital Field program (RS-2024- 00436680) supervised by the IITP (Institute for Information & Communications Technology Planning & Evaluation). This project is supported by Microsoft Research Asia. This research was supported by a grant of Korean ARPA-H Project through the Korea Health Industry Development Institute (KHIDI), funded by the Ministry of Health & Welfare, Republic of Korea (grant number ASS2002--005123744 Jinyoung Yeo is the corresponding author.
References
[1] Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, al. Webarena: A realistic web environment for build- ing autonomous agents. In The Twelfth International Conference on Learning Representations, 2024.
[2] Yichen Pan, Dehan Kong, Sida Zhou, Cheng Cui, Yifei Leng, Bing Jiang, Hangyu Liu, Yanyi Shang, Shuyan Zhou, Tongshuang Wu, et al. Webcanvas: Benchmarking web agents in online environments. arXiv preprini arXiv:2406. 12373, 2024.
[3] Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Russ Salakhutdinov, and Daniel Fried. Visualwebarena: Evaluating multimodal agents realistic visual web tasks. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 881-905, 2024.
[4] Zora Zhiruo Wang, Jiayuan Mao, Daniel Fried, and Graham Neubig, Agent workflow memory. arXiv preprint arXiv:2409.07429, 2024.
[5] Hyungjoo Chae, Namyoung Kim, Kai Tzu iunn Ong, Minju Gwak, Gwanwoo Song, Jihoon Kim, Sunghwan Kim, Dongha Lee, and Jinyoung Yeo. Web agents with world models: Learning and leveraging environment dynamics in web navigation. In The Thirteenth International Conference on Learning Representations, 2025. URL http://///peeeee..eeeeeee id=moWiY JuSGFF
[6] Tianci Xue, Weijian Qi, Tianneng Shi, Chan Hee Song, Boyu Gou, Dawn Song, Huan Sun, and Yu Su. An illusion of progress? assessing the current state of web agents. arXiv preprint 2xXi::5504.03822 2025.
[7] Jiayi Pan, Yichi Zhang, Nicholas Tomlin, Yifei Zhou, Sergey Levine, and Alane Suhr. Au- tonomous evaluation and refinement of digital agents. arXiv preprint w:Xiv2244.006474 2024.
[8] Jing Yu Koh, Stephen McAleer, Daniel Fried, and Ruslan Salakhutdinov. Tree search for language model agents. arXiv preprint arXiv:2407.01476, 2024.
(9] Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike. John Schulman, llya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2023.
[10] Peiyi Wang. Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce lims step-by-step without human annotations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1426-9439, 2024.
[11] Xiao Liu, Tianjie Zhang, Yu Gu, lat Long long, Song XiXuan, Yifan Xu, Shudan Zhang, Hanyu Lai, Jiadai Sun, Xinyue Yang, Yu Yang. Zehan Qi, Shuntian Yao, Xueqiao Sun, Siyi Cheng, Qinkai Zheng, Hao Yu, Hanchen Zhang, Wenyi Hong. Ming Ding, Lihang Pan, Xiaotao Gu, Aohan Zeng, Zhengxiao Du, Chan Hee Song, Yu Su, Yuxiao Dong, and Jie Tang. Visualagentbench: Towards large multimodal models visual foundation agents. In The Thirteenth International Conference on Learning Representations, 2025. URI. https:/ httpa:://pppeeeeee http///peeeeeeeee//ee/eeeeeeeeee httee//ormmmmddddnnccess
11

---

[12] Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-4v (ision) is a generalist web agent, if grounded. International Conference on Machine Learning, pages 61385. PMLR, 2024.
[13] Longtao Zheng, Rundong Wang. Xinrun Wang, and Bo An. Synapse: Trajectory-as-exemppe prompting with memory for computer control. In The Twelfth International Conference Learning Representations, 2023.
[14] Hongliang He, Wenlin Yao, Kaixin Ma. Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. Webvoyager: Building an end-to-end web agent with large multimodal models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6864 6890, 2024.
[15] Heyi Tao, Sethuraman T V, Michal Shlapentokh-Rothne and Derek Hoiem. Webwise: Web interface control and sequential exploration with large language models, 2023. URL pctps://arxxe org/ aaaaab////111 16042.
[16] Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: towards a generalist agent for the web. In Proceedings of the 37th International Conference on Neural Information Processing Systems, pages 28091-28114, 2023.
[17] Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. Seeclick: Harnessing gui grounding for advanced visual gui agents, 2024. URL https:/ //tp:///aaxvv///aa www.iii.rrg//o org/ abs/2401...... 10935.
[18] Jihyung Kil, Chan Hee Song, Boyuan Zheng, Xiang Deng, Yu Su, and Wei-Lun Chao. Dual- view visual contextualization for web navigation, 2024. URL 2402. 2402.04476.
prg/abs/
[19] Jing Yu Koh, Stephen McAleer, Daniel Fried, and Ruslan Salakhutdinov. Tree search for language model agents, 2024. URL https://aa/a http://araivv..og////////// org/abs/2407.
014766.
[20] Zehan Qi, Xiao Liu, lat Long long. Hanyu Lai, Xueqiao Sun, Wenyi Zhao, Yu Yang, Xinyue Yang, Jiadai Sun, Shuntian Yao, Tianjie Zhang, Wei Xu, Jie Tang, and Yuxiao Dong. Webrl: Training Ilm web agents via self-evolving online curriculum reinforcement learning, 2025. URL //axxiv
abs/a///////// 1.111.00 02337.
[21] Hao Bai, Yifei Zhou, Jiayi Pan, Mert Cemri, Alane Suhr, Sergey Levine, and Aviral Kumar. Digirl: Training in-the-wild device-control agents with autonomous reinforcement learning. In The eighth Annual Conference on Neural Information Processing Systems, 2024.
[22] Hao Bai, Yifei Zhou, Li Erran Li, Sergey Levine, and Aviral Kumar. Digi-q: Learning value functions for training device-control agents, 2025. URL http:::://axxv.or///// argg/bs////2/ 15760.
[23] Xidong Feng, Ziyu Wan, Muning Wen, Stephen Marcus McAleer, Ying Wen, Weinan Zhang, and Jun Wang. Aiphazero-like tree-search can guide large language model decoding and training. arXiv preprint arXiv:2309.177 17179, 2023.
[24] Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.167200 2024.
[25] Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501. 2025.
[26] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
12

---

[27] Jiayi Pan, Yichi Zhang, Nicholas Tomlin, Yifei Zhou, Sergey Levine, and Alane Suhr. Au- tonomous evaluation and refinement of digital agents. In First Conference on Language Modeling, 2024.
[28) Pranav Putta, Edmund Mills, Naman Garg, Sumeet Motwani, Chelsea Finn, Divyansh Garg, and Rafael Rafailov. Agent q: Advanced reasoning and learning for autonomous ai agents, 2024. URL https://aaxixi org/ //////2280001999
[29] Gaole Dai, Shiqi Jiang. Ting Cao, Yuanchun Li, Yuqing Yang, Rui Tan, Mo Li, and Lili Qiu. Advancing mobile gui agents: A verifier-driven approach to practical deployment, 2025. URL
https://arxiv.. /rrg.//rr///o////////////oo org/ abs/2503. 15937.
[30] Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H Laradji, Manuel Del Verme, Tom Marty, Léo Boisvert, Megh Thakkar, Quentin Cappart, David Vazquez, al. Workarena: How capable are web agents at solving common knowledge work tasks? arXiv preprint arXiv:2403.07718. 2024.
[31] Dakota Mahan, Duy Van Phung. Rafael Rafailov, Chase Blagden, Nathan Lile, Louis Castricato, Jan- Jan-Philipp Fränken, Chelsea Finn, and Alon Albalak. Generative reward models. arXiv preprint arXiv:2410. /282,, 2024.
[32] Shengding Hu, Ning Ding, Huadong Wang, Zhiyuan Liu, Jingang Wang, Juanzi Li, Wei Wu, and Maosong Sun. Knowledgeable prompt-tuning: Incorporating knowledge into prompt verbalizer for text classification. In Proceedings of the 60th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2225-2240, 2022.
[33] Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403. 13787, 2024.
[34] Sunghwan Kim, Dongjin Kang, Taeyoon Kwon, Hyungjoo Chae, Jungsoo Won, Dongha Lee, and Jinyoung Yeo. Evaluating robustness of reward models for mathematical reasoning. arXiv preprint arXiv:2410.01729, 2024.
[35] An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang. Jianhong Tu. Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv preprint arXiv:2412.15 15115, 2024.
[36] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang. Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu. Shuang Luo. Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang. Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL httttttss///aaatta org abs / 09388.
[37] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923. 2025.
[38] Edward Hu, Yclong Shen, Phillip Wallis, Zeyuan Allen-Zhu Yuanzhi Li. Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, (2):3, 2022.
[39] Yantao Liu, Zijun Yao, Rui Min, Yixin Cao, Lei Hou, and Juanzi Li. Pairwise rm: Perform best-of-n sampling with knockout tournament. arXiv preprint arXiv:2501.13007 2025.
13

---

[40] Zijun Liu, Peiyi Wang, Runxin Xu, Shirong Ma, Chong Ruan, Peng Li, Yang Liu, and Yu Wu. Inference-ting time scaling for generalist reward modeling. arXiv preprint arXiv:2504.00 02490, 2025.
[41] Yu Gu, Kai Zhang, Yuting Ning, Boyuan Zheng, Boyu Gou, Tianci Xue, Cheng Chang, Sanjari Srivastava, Yanan Xie, Peng Qi, et al. Is your IIm secretly world model of the internet? model-based planning for web agents. arXiv preprint arXiv:2411.06550 2024.
[42] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36, 2024.
[43] Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: Nlg evaluation using gpt-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511-2522, 2023.
[44] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
[45] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large lan- guage model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
[46] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023.
[47] Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhanghhi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Bangkok, Thailand, 2024. Association for Computational 13372.
Linguistics. URL http: /ttp:://axxivi http://aaaxvvv..oaabbbsssess
[48] Pin-Lun Hsu, Yun Dai, Vignesh Kothapalli, Qingquan Song, Shao Tang, and Siyu Zhu. Liger- kemel: Efficient triton kernels for Ilm training, 2024. URL https: http///gttttttt/////// thub. com/ / linkedin/ Liger-Kernel.
[49] Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction. arXiv preprint 2.X.::2400..124000 2024.
[50] De Chezelles, Thibault Le Sellier, Maxime Gasse, Alexandre Lacoste, Alexandre Drouin, Massimo Caccia, Léo Boisvert, Megh Thakkar, Tom Marty, Rim Assouel, al. The browsergym ecosystem for web agent research. arXiv preprint 22..22472.05667, 2024.
14

---

A Limitations and Societal Impacts
A.1 Limitations
Expansion to coordinate-based actions. Recently, coordinate- oetions-where agents inter- act with digital environments using direct coordinate inputs without requiring additional backend programs to convert actions- gained attention due to their adaptability across diverse interfaces. have also collected a dataset to extend WEB-SSIEPIERI to support coordinate- action for- mats. However, this direction falls outside the primary scope of this work, we leave its exploration for future research.
Application to reinforcement learning. An interesting direction for future work use WEB- SHEEPHERD as a reward signal in reinforcement learning. While we plan to explore this setting, it requires significant computational resources and therefore left for future work. In particular, aim to investigate whether reward signals from PRMs can improve learning efficiency-iie how quickly rewards increase during training as well as final performance on existing benchmarks.
Selection of the base model for WEB-SHEPHERD. While our current implementation of WEB- SHEPHERD uses relatively lightweight base models (3B-8B), the approach model-agnostic and can be extended to larger scales. In principle, WEB-SHEPPERDD can be scaled up to stronger foundation models in the 32B-72B range, which may further improve performance in complex web environments. We leave the exploration of such scaling future work, particularly in resource-rich settings.
Multimodal instructions. While most instructions in existing web agent benchmarks are purely textual, some tasks-such as those in VesualWebArena in]-oncoeeorate both text and image modal- ities. Extending WEB-SHEPHERD to handle multimodal instructions is a promising direction for future work, as would enable the agent to operate in more complex and realistic web environments that require visual understanding in addition to text comprehension.
A.2 Societal Impacts
Positive impacts. Web agents have the potential to perform wide range of tasks typically carried out through a web browser, which serves as a universal interface for information access, online services, and task execution. However, current agents are often restricted to simple tasks, such as retrieving an address or clicking through static pages. We believe that WEB-SHEPHERD can broaden the capabilities of web agents, enabling them to tackle more complex, goal-oriented tasks in dynamic environments. This advancement could benefit users with accessibility needs, support automated workflows in professional domains, and improve the scalability of digital assistance.
Negatíve impacts. Despite their potential benefits, web agents also pose several risks. Without proper safeguards, agents with the ability to autonomously interact with websites could unintentionally or maliciously perform harmful actions-suce as submitting unauthorized forms, modifying user data, or accessing sensitive information. Moreover, if reward models are misaligned or insufficiently robust, agents may exploit unintended shortcuts to maximize rewards without accomplishing the intended task. To mitigate these risks, is crucial to incorporate safety mechanisms, including strict execution constraints, permission controls, soman-in-the-lowe oversight, and careful auditing of model outputs in deployment scenarios.
B WEBPRM COLLECTION
B.1
Data Annotation Toolkit
To reduce the burden of human annotation, we developed a specialized toolkit for collecting web agent trajectories. It is designed to streamline the annotation process while ensuring the collection of high-quality data. Figure 10 shows screenshot of the toolkit interface. This tool helps the annotators to interact with the browser by taking the user's inputs and showing the execution output (e.g., observation) within the graphical interface. For example, they select an action type from predefined set (c.g., fill) and provide the corresponding argument (e.g.. "sony headphone") via the
15

---

Step Recording
Zooni
Zoem
to Frame
Raload
62%
Coordratta:: (834,
Cats.com
CAT FOOD
CAT LITTER
LITTER BOX
CAT HEALTH
Low Priority!
Overview
Autian
Scroll
Pill
Select Optinn
Ciis
2
Dudfte Click
Hover
Press
Faeus
Coordiaate Navigatonn Teb
Show Exaippies
nodifarr
B
M0 4M
Pancule
Figure 10: Our annotation toolkit used for collecting WEBPRM COLLECTION.
action panel. Selecting the action type via button clicks, rather than manually typing the entire action sequence, significantly reduces errors. In addition, the sidebar displays snapshots of previous user interactions, allowing annotators to easily track progress, review past actions, and undo the most recent one necessary.
B.2 Details of Human Annotation
Our data collection process follows the steps below:
Step 1: Website list selection. We begin by selecting candidate websites from those used in the Mind2Web training dataset. Since web navigation requires browser automation, we manually filter out websites that incompatible with annotation process specifically, those that block Playwright by requiring CAPTCHA verification or by rejecting HTTP requests entirely. After applying this filtering process, retain 50 websites that are technically accessible and semantically appropriate for annotation.
Step Annotator recruiting and education. We recruit groups of human annotators to construct the dataset, with overall supervision and quality control handled by designated project managers. All annotators completed three-hour education session conducted by the project managers prior to annotation. This training covers detailed explanation about data annotation interface, guidelines for writing quality task instructions, examples of good and bad trajectories, and principles for designing judge codes. Upon completing the training, each annotator is assigned to 3-4 websites from the filtered website list.
Step Data annotation. The annotation process structured into three distinct phases. In the first phase, we ask each annotator to create 20 task instructions for each of their assigned websites. These tasks are distributed across three difficulty levels: 5 easy, 10 medium, and hard tasks. Annotators are instructed to design tasks that reflect realistic user goals, such as booking a reservation, retrieving specific information, modifying a user profile. In the second phase, annotators execute the tasks they created and record expert trajectories interacting with our annotation toolkit (Figure 10). These expert trajectories have complete sequence observation-action pairs needed to complete the task successfully. Lastly, annotators write judge code that can automatically assess the trajectory towards the user's goal. To ensure compatibility with existing benchmarks and code bases, we follow the format of judge code of WebArena [1].
16

---

Step 4: Verification. To ensure the quality of WEBPRM COLLECTION, we introduce two safe-
guards throughout the annotation process.
To verify judge codes, we conduct programmatic verification that checks whether each judge code correctly evaluates the corresponding annotated trajectory "success'. If mismatch is detected, the annotator is instructed revise the judge code.
Automatic Verification:
Manual Verification:
Project managers manually review all annotated trajectories and their associated judge codes, filtering out erroncous or low-quality data. As a result, 15% of the annotated data was discarded during this step.
B.3 Annotating Dataset with MLLMs
Reasoning for chosen action. Recent web agents widely adopt ReAct (i.e., Reason + Act) [46] framework, in which the agent first produces a rationale (thought) to explain its current understanding or intent, and then selects an action based on that reasoning. However, our datasets lacks this intermediate reasoning step-it does not capture what the agent was thinking when choosing each action. To enrich our dataset with such reasoning traces, we leverage Qwen-2.5- VL-72B, prompting with the current observation (URL,, accessibility tree representation, image screenshot), the selected action and a screenshot obtained after executing the action. The model is then asked to generate corresponding rationale that explains the decision behind the chosen action.
Checklist with human trajectory. To effectively extract key subgoals (i.e., checklist) that are essential for achieving the user's instruction, we provide GPT-4o with both user instruction and human trajectories, which include the intermediate thoughts. The model is prompted to generated a reasoning process that analyzes the give task, and then to produce a checklist grounded in that reasoning. This approach significantly enhances checklist quality. In contrast, the low- low-quality checklist shown in Figure 7 were generated by a model trained without the reasoning component, highlighting the importance of the task-specific reasoning in generating reliable checklists.
Annotating additional checklist. The number of checklists obtained from human trajectories is 851 instances, which is relatively small for training. To address this, we first augmented the dataset using IK user instructions provided in the Mind2Web training set. For each instruction, we used GPT-4o to generate a corresponding checklist. Subsequently, we further expanded the dataset by prompting the model to generate new instructions based on existing examples, and then constructing corresponding checklists for each, Through this augmentation process, collected total of 3.6K checklist instances.
Collecting rejected actions from various policy models. To construct robust reward model, we collect diverse candidate actions from multiple policy models. These include 2wen-2.5-VLL-77 and Qwen-2.5-VL-72B (both in text-only and multimodal settings), GPT-4o-mini (used specifically for generating negative actions that differ from the given chosen action), and Qwen-2.5-3B model fine-tuned with human trajectories from WEBPRM COLLECTION. For each chosen action, we collect up to five rejected actions sampled from these policies. However, an action that differs from the chosen one is not necessarily incorrect. For example, directly filling a search box versus clicking before typing can both be functionally valid. To eliminate such cases, we apply rule-based filtering that retains only clearly invalid (i.e., rejected) actions. Each action consists of a keyboard or mouse operation (e.g., cli ck and fill) and its corresponding argument, such as a unique element ID or text string. We apply different filtering rules depending on the type of the chosen action, as detailed below:
send_mse ford__sg_tte _o_user, scroll, goto: If the operation type differs, the candidate is consid- ered negative action. In particular, if the operation is bend_msg_to_ _uuee we verify its correctness using GPT-4o.
drag_and
If the candidate action's 's operation is not one of wrrgg_and_dre scroll, hover, it is classified as negative action.
click, dclick: If the argument (e.g., element ID) does not match the chosen action's argument and the candidate action is not semantically equivalent (e.g., clicking an unrelated element), it is considered incorrect.
17

---

click, fill: If both actions target the same element but differ only in order, the candidate is not considered negative. Otherwise, mismatches in target elements or unrelated inputs are marked as negative.
Others: Actions with unmatched operation types or arguments that do not lead to equivalent outcomes are treated as negative.
While this rule-based filtering substantially improves the quality of negative samples, it cannot guarantee correctness all cases. We leave further improvement of this filtering process for future work. Finally, if more than five valid rejected actions remain after filtering, we randomly sample subset maintain consistent number of action pairs per instance.
B.4 Statistics of WEBPRM COLLECTION
Human annotated data. Figure 11 shows the statistics of human annotated data, collected a total of 851 tasks through ad annotation process, as detailed in Appendix B.2. These tasks are categorized into 244 easy, 426 medium, and 181 hard tasks, covering wide range of real-world scenarios with varying levels of complexity. Our annotated data spans a diverse set of websites, as illustrated in Figure Ila which shows the distribution of verified tasks actoss different domains. A portion of annotated data- aatuntnn of 15%-was was discarded during a manual verification step conducted by project managers ensure data quality.
Figures 1 11b provides a linguistic overview of the instructions in our dataset. This sunburst chart visualizes root verbs and their most common direct objects, revealing frequent combinations such as visit webpage and find restaurant. These patterns reflect realistic user intents and highlight the diversity of task formulations in the dataset. In addition, Figure 1 lc presents the distribution of action types observed during annotation, with click, scroll, and fill appearing most frequently.
Rejected actions. After the rejected action generation step, we obtained 30,960 rejected actions from 9,473 chosen actions. Figure 12 presents an overview of the rejection statisties. The generation flow-iee., how rejected actions are derived from specific chosen actions-is shown in Figure 12a. Also, Figure 12b compares the distributions chosen and rejected actions. As in the statistics, the distribution of rejected actions differs slightly from that of the chosen actions. For example, the proportion of click actions increased, while the proportion scroll actions decreased. We leave the development of more refined methods to reduce this distributional difference to future work.
C WEB-SHEPHERD
C.1
Training
We train the model for epochs with a learning rate of Ie-4, using LoRA with rank of 16. Training is conducted using DeepSpeed ZeRO Stage 2 on an RTX A6000 (48GB) server with 8 GPUs, totaling approximately 16 GPU-hours. We leverage the LLaMA-Factory [47] framework and apply the Liger kernel [48] optimization during training.
C.2 Inference
We use vLLM [45] to perform inference with SHEPHERD. The decoding is configured with temperature of 1.0, and nucleus sampling is applied to generate five output sequences per prompt.
To compute the probability of each label, we apply mapping from semantic labels to token-level logits. Specifically, we aggregate the logits of the following token variants corresponding to each label:
Yes: ["dYes", "Yes", "ČYes", "dyes", "yes", "Ćyes", "ĆYES", "YES",
"ČyEs", "ÉDone", "Done", "CDone", "ÉCompleted", "Completed", "ÉCompleted", "GCorrect", "Correct", "[Correct"]
No: ["GNo", "No", "ČNo", "áND", "NO", "ČND", "cNot", "Not", "ČNot", "GNone", "None", "ČNone", "GNope", "Nope", "ČNope", "GUn", "Un", "CUn", "GWrong", "Wrong", "cWrong"]
18

---

Number Tasks by Website
20
Hard
Mediuu
Number of lotal Taasks
Easy
15
10
you
(a) Number of tasks by website.
walg walgreen
teaalt
page
webpage
Hard
apple
Medium
gamespot
studio
Easy
w rebsite
show
visit
go
page
Counts
3k
game
find
restauurnt
fighht
tell
websiie
Lake
price
lumbee
page
200r
(b) Instructions.
(c) Actions.
dataset: (a) Number of tasks per website, grouped by difficulty (Easy, Medium, and Hard). (b) The distribution root verbs and direct objects in instructions. (c) Action type distribution, broken down by difficulty.
Figure 11: Statistics of the
Generation chosen
Rejected
Distribution Chosen Rejected Actions
rejected
Reaeted Actions (hrm action
clic
select_option
click
Proportton
others
goLback
63
fi1
gacback
scroll
option
mont.mms..cu user
03
aend_msg_to_user
others
press
sandl
36,054
Trover
prica
op0ion
pr.beck
others
Action Typp
Generation flow of rejected actions.
(b) Proportions of chosen and rejected actions.
Figure 12: Statistics of the rejected actions: (a) Generation flow of chosen to rejected actions. Green bands indicate that the rejected actions share the same action type as their originating chosen action. (b) Proportions of chosen and rejected actions. showing how the distribution of action types shifts during the rejection generation process.
19

---

In Progress: ["ÉIn", "In", "ČIn", "GPending", "Pending", "CPending", "GPart", "Part", "CPart", "GPartial", "Partial", "CPartial", "CInProgress", "InProgress", "CInProgress"]
Logits corresponding to these token variants are summed for each label to compute the final label probabilities.
D WEBREWARDBENCH
D.1
Data Construction
Chosen action. For WebArena, we manually annotate the expert trajectories, since it is not provided the benchmark. On the other hand, Mind2Web provide them, so we use them as the chosen actions. One important change we make on Mind2Wel is converting the HTML observation space to bid-based observation. In the HTMI there exist DOM backend ids so we utilize them for the conversion. Lastly, since it is increasingly hard to assure the quality of human-annotated rationales, we incorporate LLMs to annotate Chain-of-Thoughee (CoT) a post-hoc manner.
Rejected actions. Following the setup of Kim et al. [34], we construct reliable benchmark by collecting multiple rejected samples from various models. In this work, we use three MILMs-GPT and -.-e-2..-
4o-mini, Qwen-2.5-VI.
policy models. For each chosen action, we sample four rejected actions from these policies. To ensure that the rejected actions are truly incorrect, we apply rule-based filtering (as described in Appendix B.3) and additional human filtering performed by the authors. Finally, we collect 776 step-level data instances derived from 220 task, each associated with one chosen and four rejected actions, resulting in total of 3,880 test instances.
D.2 Analysis of WEBREWARDBENCH
Figure 13a shows the distribution of chosen and rejected actions categorized by action type and source model across both WebArena and Mind2Web. In both datasets, click and fill actions dominate among the chosen actions, which is consistent with the typical interaction patterns required in web navigation environments. Notably, the rejected actions across all source models exhibit similar distributions, with click actions being the most frequently rejected. This suggests that despite differences in model architecture and scale, the failure modes of MLLMs often concentrate similar types of actions. Additionally, the inclusion of multiple source models GoT--o-mini, Qwen2.5- 7B, and SW --rrthrr further contributes to the diversity of rejected actions. This model-level heterogeneity ensures that the benchmark captures a broad range suboptimal behaviors, enhancing its generality and diagnostic value.
Figure 13b visualizes the joint distribution of trajectory length and the number of checklist items associated with each task instance. The majority of trajectories fall within the 2-8 step range, while checklist items typically range from to be accompanied by a greater number of checklist items, indicating that tasks with longer horizons are generally more complex and goal-rich. However, we also observe several short trajectories with multiple checklist items, suggesting that brevity in execution does not necessarily imply low task complexity. This variability further highlights the importance of step-level evaluation addition to trajectory-level metrics.
The plot reveals general trend that longer trajectories tend
E
Additional Results
E.1
Evaluating MLLMs as Process Reward Models for Web Navigation
We conduct experiments to investigate the most suitable format for reward prediction when using MLLMs as preference reward models (PRMs). Specifically, we evaluate how helpful a generated action is in progressing toward the goal from the current state. We consider two formats: a Likert scale rating (1-5) and a 3-class classification with labels helpful, neutral, not helpful. To reduce variance, each instance is sampled five times and the scores are averaged. Table 8 shows that the Likert scale consistently outperforms the 3-class classification, indicating that fine- fine-grained evaluation provides a more informative learning signals.
20

---

Table 6: Evaluation results on WEDREWARDBENCI without using checklist. T denotes text observation, and I denotes image observation. Acc. (s) refers to step accuracy, while Acc. (1) refers trajectory accuracy.
Mind2Web
WehArema
Model
Inputs
Chots-Task Acc. (5)
Crass-Domnti Acc. (9)
MRR
Acc. (O
MRR
Acc. (4)
Acc. (
MRR
Acc. (1
MRR
Acc. (6)
Acc. (i)
Reward Assigument with Likert Soale
GPT-Mo-mini
47.5
15.5
0.0
47.6
13.5
0.0
45.4
11.8
34.4
5.0
T+1
44.7
12.7
2.5
4288
8.8
0.0
43.1
10.1
34.6
GPT-fo
T
$6.9
28.8
3.0
55.8
2644
2.6
59.8
33.6
3.3
59.2
37.7
15.0
T-1
52.5
21.8
5.0
52.2
21.0
0.0
52.8
23.3
1.7
50.0
24.6
5.0
Oww.-2.5-VL-2288
26.1
51.8
20.3
0.0
47.6
19.4
Reward Axxignment with Closs
GPT-to-mini
44.7
12.7
2,5
42.8
8.8
0.0
43.1
10.1
34,6
12.
Qwen-2 V.--7-2
50.6
25.4
3.3
53.
29.1
5.1
34.4
30.5
233
48.0
24.6
0.0
Table 7:
Evaluation results on WEDREWARDBENCI with using checklist. Results are averaged over four test types, and reflect performance under different setting, including whether the "In Progress" label is used during prediction.
WEDREWARDBENCH
Model
Use 'In Progress'
MRR
Acc. (step)
Acc. (traj)
GPT-4o-mini
59.3
33.5
63.3
40.7
11.3
Table 8: Evaluation results on WEBREWARDBENCIWVl using checklist. denotes text observation, and denotes image observation. Acc. (s) refers to step accuracy, while Acc. (1) refers to trajectory accuracy.
WebArena
Model
Iuputs
Cross Tark Acc. (5)
Cruss Webnite Act. (s)
Cross Domaie
Text Acc. (s)
MRR
Acc. (U
MRR
Auc. (0
MRR
Acc. (5)
Aoc. (U
MRR
Acc. (0
Reward Assignment with Reference Checklist
T
40.1
5.0
600
39.1
5.1
53.8
GPT-4o
I
T
62.4
67.4
46.5
39.4
5.0
7.5
68.1
52.0
50.0
15.4
5.1
65.1
70.2
51.3
43.2
11.6
6.6
09.7
60.0
53.6
37.7
15.0
10.0
Q.e--2.5-VL--2BB
T
59.4
35.2
0.0
62.4
40.5
0.0
57.9
32.9
1.7
52.3
30.4
5.0
Clamle-3.7-amme
60.7
41.6
7.5
34.5
10.3
60.3
40.3
55.2
57.7
Gemini-2.S.aam Bash
T
53.4
27.5
5.0
59.7
7.7
57.2
32.1
4.1
57.2
36.2
WEB SEa-SHEPHERD (3B)
T+1
T
87.6
85.0
76.8
80.3
42.5
55.0
87.3
88.0
79.1
79.7
43.6
41.0
87.2
84.4
79.1
74,1
37.2
47.1
92.5
90.1
87.0
85.5
60.0
65.0
(SB)
Reword Axsigument with Cleekfist Generation
GPT-4o-mini
+
T
59.9
55.3
30.3
36.6
2.5
2.5
55.4
57.7
27.0
29.7
0.0
5.1
$7.0
56.6
32.6
31.7
4.1
5.0
57.8
51.3
37.7
30.4
15.0
5.0
GPT-40
39.4
2.6
55.4
34.8
Qwen: 2.-VV-2B
WES-SuRrIENT (3B)
1+1
T
$1.1
85.3
69.7
75.4
50.0
25.0
78.6
83.8
64.9
74.3
23.1
33.3
84.8
77.9
61.3
75.3
39.7
22.3
94.6
85.9
75.4
89.9
40.0
70.0
WES SHEPHERD (8B)
T
87.3
80.3
50.0
84.3
76.4
38.5
86.0
76.7
43.8
96.5
94,2
80.0
21

---

Distribution of Chosen vs. Rejected Actions in WebArena and Mind2Web source
0.8
Chosen
0.7
Rojocted (owen25 5 781
Rojected (SPT4o min)
Rejacted (2wan22 728)
0.4
0.3
dlic
others
Inflass biu
ostion
WebArena
Mind2Wab
(a) Distribution of chosen and rejected actions in WEBREWARDBENCH.
Trajectory Checklist
Checklla
250
Wajectory Length
Count
(b) Trajectory length and checklist item count.
Figure 13: Statistics WEBREWARDBENCH: (a) Proportions of each chosen versus rejected action.
(b) Distribution of trajectory lengths and number of checklist items.
Table 9: Detailed results of refinement with feedback from WEB- WEB-SHEPHERD in Table 4
Policy
Model
Shopping
CMS
Reddit
GitLab
Map
Total
w/o refine
21.74
22.86
19.05
34.38
19.35
23.64
GPT-4o-mini
SEE-SHEEHERD (3B)
23,91
31.43
34.38
22.58
26.67
+3.03
WER-SHEPHERD (8B)
23.91
34.29
33.33
34.38
16.13
27.88
+4.24
Furthermore, we examine how reward prediction changes when a reference checklist is provided. We compare evaluation schemes: one that uses binary labels ('Yes' or 'No') for each checklist item, and another that introduces an additional label ('In Progress') to indicate when an action partially completes a checklist item. As shown Table 7, incorporating the Progress label leads more reliable reward assignments when using checklists. Based on this finding, we adopt the In Progress setting for checklist- based reward prediction in both the WERREWARDBENCE evaluation and the training of WEB-SHEPHEEE
E.2 Detailed Results of Refinement
We conduct experiments for refinement with feedback from reward models (Table 4). We show the detailed experimental results in Table 9.
E.3
Scoring Strategy: Probability vs. Token
When using generative models for reward prediction, on can either directly interpret the model's natural language output (e.g., "Yes") as reward signal compute the probability of specific response [49] to derive reward. To investigate which approach more effective, we compare the following strategies:
22

---

Table 10: The impact of scoring strategies reward assignment. Results evaluated WEBRE- WARDBENCHand represent the average score across four types test sets.
WEBREWARDBENCH
Model
Strategy
MRR
Acc. (step)
Acc. (traj)
res
72.7
60.7
12.2
prob
86.3
77.0
43.1
WEBB-SHEHEERD (3B)
5 avg
83.7
75.2
29.5
prob
88.5
81.2
51.4
res
77.7
67.4
14.8
prob
86.9
79.2
48.2
WEB-SHEPHERD (8B)
5 avg 88.8
82.9
51.6
prob
91.3
86.1
63.7
Table 11: Impact of the ratio between chosen and rejected samples on WEB-SHEPHEEE SHEPHERD'' perfor- mance.
Sample ratio (chosen: rejected)
WehArena- [11]
Model
Total
Shopping
CMS
Reddit
Gitlab
Map
1:1
24.85
23.91
25.71
19.05
34.38
19.35
WEB-SHEPHERD (8B)
1: (Ours)
32.12
32.61
37.14
19.05
34.38
32.26
1 res: Sample single response at temperature and use the output directly for reward assignment.
1 prob:
Compute the probability of a specific word (e.g., "Yes") at temperature 0.
5 avg: average.
Sample five responses at temperature 1, convert each to a reward directly and compute
Sample five responses at temperature 1 and compute the average probability of the targe word.
5 prob:
This setup allows us to analyze the trade-offs between deterministic and stochastic decoding, as well as between output-based and probability-basee reward estimation. As illustrated in Table 10, we observe that sampling multiple responses (e.g., 5 samples) leads to more effective reward estimation overall. When using only single sample, computing the probability or the target tokens yeilds significantly better results than relying on the raw token output-especialle at the treajectory level, where the performance gap is more pronounced.
E.4
Relationship between Reward and Task Success
potential issue in using signal from reward model RL is reward over-optimization, where policy overfitted to the imperfect reward signal [34]. In such cases, the model may receive high reward signals despite failing the actual task, resulting degraded performance. To mitigate this, the reward model must be well-aligned with actual task success and progression. Therefore, examine the alignment between WEB-SHEPHERS and task success. Figure 14 presents the correlation between final-step rewards and task success, based on the reward-guided trajectory search results described in Section 6.2. To compare rewards across trajectories, we normalize the final-step reward by subtracting the average reward of preceding steps within the same trajectory. For WEB-SHEPHEER. we observe that higher normalized final-step rewards are associated with higher success rates, while GPT-4o-ming shows no clear correlation between normalized rewards and task success. This suggests that WEB- TEB-SHEPHERD is better aligned with actual task success and thus less susceptible to reward over-optimization compared to GPT-4o-mini.
F.5 The Impact the Ratio between Chosen and Rejected Actions in Training Dataset
To better understand the effect of learning to criticize rejected actions, we construct training datasets with two different ratios of positive to negative examples: 1:1 and 1:4. Using GPT-44-mini as the policy model, we conduct trajectory search experiments WebArena-Lite. As shown in Table 11,
23

---

WebShepherd (3B) (ours)
GPT-4o-mini
1.0
1.0
succeds rates
0.8
0.8
0.6
0.6
0.4
0.4
-0.4
-0.6
normalized finalistep rewards
Figure 14: Task success rates binned by normalized final-step reward for WED-SHEPHERR (3B) (left) and GPT-4o-mini (right).
models trained with the 1:4 ratio provide more effective guidance at inference-time. This finding suggests that learning predict rewards across diverse set of rejected actions is more beneficial, even wen the ratio of positives highly imbalanced.
E.6 Cost Efficiency
Table 12: Cost per 1,000 instances (USD) across different models.
We provide the full cost breakdown for TEB-SHEPHERD and the baseline models Table 12. The cost of WEB- SHEPHERD is estimated as follows: first, we compute the number of input and output tokens per instance by running the model on the evaluation set. Then, we measure the biroughput-defing
Model
Cost (USD)
GPT-40
435,7 74
GPT-4o-mini
43.57
as the number of tokens (input output) processed per minute-on server equipped with a single A100 80GB GPU. Finally, using the hourly cost of the hardware ($1. 19/hour), we derive the cost per 1,000 instances. This allows for a fair comparison with API-based models, whose costs are based on pricing information from OpenRouter', OpenAI", Anthropic", and Google?.
Qwen-2.5-VL-722B
$3.69
Claude Sonnet 3.7
273. 16
Gemini 1.5 (Pro)
13.37
WEB-SHEPHERD 3B
4.67
E.7 Evaluating the Quality of the Generated Checklist
We use LLM-as-a-Jude method [43] evaluate the quality of the generated checklists. However, since LLMs are trained on large-scale web data but do not possess complete or up-to-date knowledge of all websites, their evaluations can be unreliable in this context. To address this limitation, provide a reference checklist during evaluation, allowing the LLM to assess the generated output relative to a known, task-specific ground truth. We evaluate the quality of checklist along three key dimensions: (1) Validity -ahehity-whethhr whether any incorrect or irrelevant checklist items are generated; (2)
Subgoal Granularity whether the steps are overly fine- mee-grineed or unnecessarily detailed; and (3) Goal Coverage whethee the checklist includes all key steps necessary to complete the final goal. Specifically, the L.M is prompted to assign quality score on Likert scale (i.e., from to 5), along with rationale explaining the evaluation. To reduce evaluation variance, each instance is rated three times, and we report the average score.
Table 13 presents the checklist quality across different checklist sources, evaluated along three dimensions and an overall score. We observe that, with the exception of our initial model version (trained only on checklist generation without reasoning) and produce checklists of comparable quality. Notably, the model trained solely for checklist generation (i.e., without multi-task learning like WEB-SH (EB--SHEHERD), suggesting the benefit of task-specific supervision. Based on this observation, we also release the checklist-only model support broader use cases.
(3B), most models
'https://openai.
http:////ooeaa..cpppciiici com/pricing
"https://ai.oons dww/geeini--eeeeeeeeeeeeewe

---

Table 13: Results of G-Eval checklist quality evaluation.
Checklist Source
Overall
Validity
Subgoal Granularity
Coverage
GPT-4o-mini
4.14
4.39
4.12
3.92
GPT-4o
4.16
4.15
3,90
Owen-2.5.V1.-222
4.18
4.36
4.21
3,97
Early version of ours (low quality)
2.97
3.08
3.10
2.74
ESEESHEPEERD (3B)
3,86
4,00
3.97
3.60
WER-SHEPHERRD (8B)
3.91
4.07
3.97
3.68
Checklist only model (ours, 8B)
4,21
4.50
4.17
3.97
F Details of Experiments
F.1 WEBREWARDBENCH
Evaluation. We evaluate model performance under the following default setting: five sampled outputs generated using temperature of 1.0. In the baseline setting without checklist, outputs are assessed using a Likert-scale. For the checklist-based reward prediction setup, we evaluate the completion status of each checklist item using three labels: Yes, In Progress, and No. The prompts used to evaluate PRMs on WEBREWARDBENCH are presented below:
Reward prediction w/o checklist: Figure
Checklist generation: Figure 19 (baseline), Figure 21 (ours)
Reward prediction based on checklist: Figure 20 (baseline), Figure 22 (ours)
F.2 Reward-guided Trajectory Search
Environment. We BrowserGym [50], unified framework for evaluating web agents in line environments. BrowserGym standardizes the action space across different implementations, improving reproducibility. also processes both textual and visual observations through an overlay of set-of-marks, enabling richer interaction signals. Additionally. it supports automatic Docker-basse website resets and identifies task dependencies prevent unintended side effects between tasks.
Policy and action selection. We use GPT-4o-mini and GPT-4o as the policy model. To obtain action candidates, we sample 20 output sequences using nucleus sampling with a temperature of 1.0. The top-n most frequent actions across these samples are selected as candidates.
We then score each candidate action using the reward model and select the with the highest predicted reward. In cases where multiple actions receive the same score, we execute the action that sampled more frequently.
Refinment experiments. We use reward model's thought and checklist evaluation re- sponses-excluding the actual reward score-as feedback for refinement. The refinement is repeated up to two times, as long as it leads to a higher reward score than the previous step. At the end of the refinement step, we obtain up to three action candidates and select the with the highest reward score as the final action.
G Case Study
25

---

Goal: recently moved, my address
Magnolia Path, Atlanta, GA, 30303, update my information SneppopSihopping accordingly
ARR
ARB
PRR
Jone Ston Mocke
Beauty Personal Care
Sports Outdoors
Clvtting Shoes Jenddry
Home & Nitches
Office Products
Toels Home improveeent
Health & Flausshold
Pati Lown Garder
Dectroric
Phonen Accessories
Video Corms
Grocery's Courmet Food
Edit Address
Fallure to account for future rewards led to suboptimal action.
Froz Nome
Name
Company
Phone humber
Compare Products
butl, Alana
Action Candidate scroll(0,200))
Action Candidate 2: click("1422)
Think:
Think:
The previous action failed because the save button was not clickable. Therefore, will scroil to find visible
There is button that needs be clicked to save the changes made the address. will search for the approprinte button submit changes.
buttons trigger submission address update.
Reward Thought:
Reward Thought: Account
Checklist
(thooughh omitted): Yes
- ....thhughh omitted): Yes
Chocklist 2: Open Address Book
Chocklist 2: Open Address Book
...oohogght omitted): Yes
...(thoughh omitted): Yes
Checklist Locate
Existing Address
Checidiist Locaat
Existing Address
(thoughh omitted): Yes
.... (thoughtt omitted): Yes
Checklist 4: Imput New Address Details (ttoought omitted):
Checkliit 4: Input New Address Details ...tthought omitted):
Checklist 5: Save Changes (thought omitted): No
Checklist S: Save Changes
...ohhouggt omitted): No
Reward: 0.588
Reward: 0.754 (Chosen)
Figure 15: Erroneous Example. Failed to properly anticipate future rewards, the agents clicked sub-optimal bid instead of scrolling find the save button.
26

---

Goal: What is the price range for products from EYZUTAK?
Previous Action: ciick('242')
Misunderstanding observations and previous
Stop Market
repeated
Becury Personal Core
Sports Outdcoors
Clothing Shoes jawelry
Kltchen
Office Products
Tools Home Ingrovement
Health Household Patio, Leaen Gardin Elctronks
Call Phomes Accossorio
Video Gurees
Grocery & Gournet Food
One Stop Market
Product Showcases
SO
Cooss
A63 Cangente sfnerge feeetty
BOG Dery
Action Candidate click('242)
Action Candidate 2: click((539)
Think:
Think:
webpage provides product includes
page displays product EYZUTAK with Beauty EYZUTAK offerings.
My
may
be to navigate baok the home page.
Reward Thought: Access
Reward Thought:
Checklist 1: Access Product Search
... (thought omitted): Yes
-...(houught omitted); Yes
Checklist 2: Enter Brand Name
Checklist
Brand Name
(..hhouggtt omitted): Yes
... (thought omitted); Yes
Checklist 3: Identify Relevant Products (tthhuuggh omitted): Yes
Checklist 3: Identify Relevant Products (thought omitted);
Checklist 4: Price
Chocklist 4: Analyze Price Range (thought omitted)): No
...ohhouggh omitted):
Reward: 0.655 (Chosen)
Reward: 0.582
Figure 16: Erroneous Example. By misinterpreting the current observation and ignoring previous actions, the agent performs repetitive actions.
27

---

me customers have expressed dissatisfaction with Chloe
IM
Hallucinations in generated checklist led to inaccurate reward calculations.
Add New Qustomer
377
Delich
Courriss
Silkoer Sock
117
11.
1:4
Amandatim
Sared
01130
Un
RMC
Action Candidate click("173)
Action Candidate (23377; tank)
Think:
Think:
need to find customer feedback regarding the Chloe tank.
The current webpage is fecused on the Admin Dashboard under the Customers section. will in the search box with "Chioe tank look
should back "Dashboard" section since this is the next logiccal action.
Reward Thought:
Reward Thought:
Checkliat Access Customer Reviows Section
Checkliist Access Customer Reviews Section
(thought omitted); No
...(thought omitted]: No
Checklist 2: Locate Product
Checklist 2: Locate Product
(thought omitted):
(tthought omitted]:
Checklist 3: Identify Dissatisfied Customers
Checkfist 3: Identify Dissatisfied Customers ((thhughh omitted]: No
(thought omitted); No
Communicate Dissatisfied
Communicate
.... (thought omiitted);
... (thought omitted):
Reward: (Chosen)
Reward:
Figure 17:
Erroneous Example. When generating the checklist, hallucinations such navigating to non-existent sections (Customer Reviews Section) or referencing pages that don't t exist (Product Page) lead to incorrect reward calculations.
28

---

H
Prompts Used in Our Works
Prompts
You expert evaluator of web agent.
Your task is to assess how helpful a given agent's THOUGHT and ACTION is making progress toward the user's goal, hased on the current state of webpage.
Action space:
Description of Action space]
Task Description
Evaluute how helpful given thought and action achieving goal.
Use the following scale:
**Sooring Criteria (1 to 5y**
**55 (Very Helpfuly**:; The directly effectively moves fulfilling key part of the goal.
*** (Helpful)**: The action contributes meaningfully to progress, though it may require follow-up actions.
(Somewhat Helpfuly**: The action partially relevant or preparatory step, but docsn't make immediate progress.
#*2 (Slightly Helppfll)**; The action is weakly related to the goal might only indirectly help.
(Not Helpfuly**; The action unrelated, redundant, or distracts from the goal.
Given Information
## User Instruction
(intent)
## Trajectory (trajectory)
# Current State
Current URL
(current_uri]
вбй AXTREE
Note: [bid] the unique alpha-numeric identifier beginning lines for each element
the AXTree. Always bid to refer elements your actions.
SOM Image Screenshot
Here current image screenshot the page, annotated with bounding boxes and corresponding hids:
## Agent's Response
THOUGHT: (thought)
ACTION: (action)
Output Format Please return your response following format:
REASON:
[Your explanation for the score]
SCORE:
[1-5]
Figure 18: Prompt used to assign reward with Likert-scall for bascline models.
29

---

Prompis
You are an Al assistant tasked with generating structured checklists that highlight key subgoals necessary to complete task.
##f Task Description
User Instruction (Goal); {intent)
Start Website URL: (start_url)
Guidelines for Checklist Generation
1. Identify Essential High-Level Subgoals:
subgoal should represent a significant step involving user interaction that leads noticeable page transitions or meaningful changes in system state.
Consolidate closely related user actions (such as applying multiple filters or select- ing several options) into a single subgoal, rather than separate checklist items for each action.
Prioritize only the most critical interactions necessary for meaningful progression, avoiding the inclusion of minor or unnecessary steps (e.g.. scroll, hover).
2. Provide Concise Subgoal Analysis: Before creating the checklist, offer brief paragraph summarizing the main subgoals, emphasizing significant transitions page-level interactions.
3. Ensure Clear Goal:
multiple related interactions occur (e.g., setting filters 2, and 3), combine them into one subgoal with clear criteria verifying all required conditions.
The checklist should contain only essential steps, explicitly excluding unnecessary actions, and should not exceed five critical subgoals. It is not necessary to use all five checklist itemss if fewer steps adequately represent the essential subgoals.
#ifll Output Format
Before generating the checklist, first produce concise subgoal analysis in single paragraph summarizing the required interactions. Then, based on this, generate the checklist following the format below:
[SURCOAL ANALYSIS]
[Une-po paragrapph summary explaining the key subgoals and their logical sequence in task completion.]
[CHECKLISTS]
Checklist X:
[Short title of the action/goal]
Goal:
[Brief description of the subgoal at this stage, emphasizing the purpose of the action.]
Figure 19: Prompt used to generate checklist for baseline models.
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
30

---

Prompis
You are an expert evaluator web agent.
Your task is to assess how helpful given agent's THOUGHT and ACTION in making progress toward the user's goal, based on the current state of the wehpage.
# Action space: [Description of Action space]
# Task Description
Your lask is to evaluate how well the agent's THOUGHT and ACTION satisly each item in the checklist.
Use the task instruction, trajectory (including previously completed steps from history), current webpage state, and the agent's current response evidence your evaluation. For each checklist item:
Mark it as 'Yes' it is clearly and fully satisfied either in the current response or already completed the history.
Mark as Progress' if the agent has made partial but meaningful progress toward completing the item.
Mark it as 'No' if there is ambiguity, insufficient evidence, or the step is incomplete or not yet started.
# Given Information
## User Instruction (intent)
#H Trajectory (trajectory)
Current State
### Current URL
{current_urll
#НН AXTREE
Note: [bid] is the umique alpha-numeric identifier at the beginning of lines for each element
in the AXTree. Always use bid to refer to elements in your actions.
meoxt_observation.
#### SOM Image Screenshot
Here is a current image screenshot of the page, it is annotated with bounding boxes and corresponding hids:
<IMAGE_ SMAAGEPLACCHHOLDERR
## Agent's Response
THOUGHT:
{thought)
ACTION:
(action)
Output Format
Please return your response in the following format:
REASON:
[Write a single, coherent paragraph explaining how well the agent's response satisfies the checklist overall. Use both the history and the agent's current thought/action as evidence. Xention specific strengthe missing elements that influence your decision.]
CHECKLIST EVALUATION:
Checklist X:
[Yea / In Progreas No]
Figure 20: Prompt used assign rewards with checklist for baseline models.
Justification:
The abstract and the introduction well reflect our contributions. Especially, the third and the last paragraph covers scope and contributions, respectively.
31

---

Prompis
You are an Al assistant tasked with generating structured checklists that highlight key subgoals necessary to complete task.
# Task Description
Generate checklist which are key milestones for achieving the given instruction. Frist, provide a concise subgoal analysis in single paragraph summarizing the required interactions.
Then, based this, generate the checklist with brief description.
# Given Information
User Instruction (intent)
Current URI. [start_url]
Figure 21: Prompt used generate checklist for WEB-SHEPHERD.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitation of our work in Appendix A.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they seale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
32

---

Prompis
You are an expert evaluator of web agent. Your task is to assess how helpful given agent's THOUGHT and ACTION in making progress toward the user's goal, based on the current state of the webpage.
# Task Description
Evaluate how well the agent's THOUGHT and ACTION satisfy each item in the checklist using the task instruction, trajectory (including previously completed steps), current webpage state, and the agent's latest response.
Start writing concise paragruph summarizing the agent's overall performance.
Refer the reasoning provided in trajectory, discuss whether THOUGHT appropriate the ACTION moves the task forward.
Then, assess each checklist item individually using the following labels:
Yes: The item is fully and clearly satisfied, either in the current response or previously completed.
In Progress: There is meaningful partial progress toward completing the item. No: The item is not satisfied due to ambiguity, insufficient evidence, or lack of progress.
# Given Information
## User Instruction (intent)
#H Trajectory (trajectory)
Current State
### Current URL
{current_uri]
#НН AXTREE
Note: [bid] is the unique alpha-numeric identifier at the beginning of lines for each element
in the AXTree.Alway use bid to refer to elements in your actions.
[text_obssevvatin?
#i#l SOM Image Screenshot
Here current image screenshot of the page, it is annotated with bounding boxes and corresponding bids:
[MAGE_PLACEHON
Checklist
{checklist)
Agent's Response
THOUGHT:
(thought)
ACTION: (action)
Figure 22: Prompt used assign rewards for WEB-SHEPHEER..
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
33

---

Prompts
Instructions
You are given a draft thought and action for the current step. This draft has not been executed yet. was evaluated by a reward model using checklist based the task goals. Your is reflect
the thccklist-based feedback improve the proposed action. Based the page state and the provided feedback, revise the thought if needed and produce better action that will be executed.
Your answer will be interpreted and executed by program, so be precise and follow the formatting instructions.
Goal: (intent)
Current State HU Current URL: {current_ur1)
AXTrec:
Note: [bid] the unique alpha-numeric identifier the beginning lines for element
AXTree. Always use bid to refer elements your actions.
Note: only elements that visible the viewport presented. You might need to scroll the page,
labs menus
more.
Note: You can only interact with visible elements. If the "visihle" tag is not present, the element is not
visible on the page.
History interaction with task: (trajectory)
Action space: [Description of Action space]
Thought and Action:
Thought: (thought)
Action: {action)
Reward Model Feedback:
The reward model evaluates actions using checklist derived from task-specific goals. Each chocklist item represents a key subgoal or intermediate step.
Feedback:
<leedbacks>
(feedback)
</feedback>
Concrete Examples
Here concrete example how to format your answer. Make follow the template with proper tags:
<EXAMPLE PLACEHOLDER>
Your Response:
<think
To move toward Checklist 1-accessing the #*Shoverthoughts** forum-1 should first make it easier locate that forum in the long list. Clicking *"Alphabetical link
alphabetically, can then quickly scroll "Shoverthoughta" and open This directly progreases to the first checklist item.
</think <action>
click(?117')
</action>
Figure 23: Prompt used to generate a refined action for Refinement.
Answer: [NA]
Justification:
We do not have any theoretical results.
Guidelines:
34

---

Prompis
You are an expert evaluator web agent. Your task is to assess how helpful given agent's THOUGHT and ACTION is making progress toward the user's goal, based on the current state of the webpage.
# Action space: [Deccription of Action space]
# Given Information
## User Instruction {intent)
Trajectory (trajectory)
Current State
#### Current URL
(current_url]
### AXTREE
Note: [bid] is the unique alpha- numeric identifier at the beginning of lines for each element in the AXTree. Always use bid to refer to elements in your actions.
(text_observation]
#if Agent's Response
THOUGHT: (thought)
ACTION: (action)
Output Format:
Please return your response in the following format:
REASON: [Your explanation for the score]
SCORE: [1-5]
Figure 24: Prompt used to assign rewards with Likert-scale for baseline models in trajectory search.
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We fully provide the experimental details in Appendix D. addition, we release the code and data to allow easy reproduction of our results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
35

---

Prompis
You are an expert evaluator web agent. Your task is to assess how helpful given agent's THOUGHT and ACTION is making progress toward the user's goal, hased on the current state of the webpage.
# Action space:
[Action Space Description]
# Task Description
Your task is to evaluate how well the agent's THOUGHT and ACTION satisfy each item in the checklist.
Use the lask instruction, trajectury (including previously completed steps from history), current webpage state, and the agent's current response as evidence for your evaluation. Clearly consider any items already successfully completed currently in progress according to the provided trajectory.
For each checklist item: Mark it as 'Yes' it is clearly and fully satisfied either in the current response or already completed the history.
Maak as 'In Progress' if the agent has made partial but meaningful progress toward completing the item.
Mark it as 'No' if there is ambiguity, insufficient evidence, or the step is incomplete or not yet started.
# Given Information
## User Instruction (intent)
Trajectory (trajectory)
Current State
### Current URI.
[current.url]
НАН AXTREE
Note: [bid] is the unique alphu-numeric identifier at the beginning of lines for each element in the AXTree. Always use bid to refer to elements in your actions.
seext_observatin)
## Checklist {checklist.)
Agent's Response
THOUGHT: {thought)
ACTION: (action)
Output Format
Please returm your response following format:
REASON: [Write a single, coherent paragraph explaining how well the agent's 's response satisfies the checklist overall. Use both the history and the agent's current thought/action as evidence. Mention specific strengths or miasing elementa that influence your decision. CHECKLIST EVALÚATION:
Checklist X: [Yes / In Progress / No]
Figure 25: Prompt used to assign rewards with checklists for baseline models in trajectory search.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (c.g., in the case
36

---

Prompis
#if Instruction
You are an expert evaluator tasked with assessing checklists for goal-directed weh navigation. These checklists are designed to guide an agent through multi-step tasks on website. Each checklist consists of subgoals, presented step-by hep-by-step. with brief descriptions explaining the purpose of each step. Using the provided intent, start URL, and the reference checklist, evaluate the quality of the checklist according to the following criterion.
Criteria: Checklist Validity
Does the checklist contain only valid, relevant, and logically consistent steps that align with the intent and the reference checklist, without introducing incorrect misleading actions?
Using the rubric below, provide brief justification and assign score from to 5 (number only, where very poor and excellent).
Rubric:
-I: Very poor: Checklist contains multiple invalid, irrelevant, or misleading steps that conflict with the intent or contradict the reference checklist.
-2: Poor: Checklist includes some valid steps hut also contains serious logical errors
clearly irrelevant actions that compromise task validity.
-3: Fair: Most steps reasonable and aligned with the task, but there are one two questionable weakly justified steps that reduce overall reliability.
4:: Good: Checklist is mostly valid and logically sound, with only minor issues such as slight ambiguities bordertine-relevant steps.
Excellent: All steps are valid, relevant, and logically consistent with the intent and reference checklist, with no incorrect or mislcading content.
Respond the following format:
Justification:
Score:
## Input
Intent:
(intent)
start_url: (start_ur]]
reference checklist: cnfffrrnne_ccheckliie
generuted checklist:
[generated_checkllis
## Output
Figure 26: Prompt used to evaluate the quality of the generated checklist based on checklist validity.
of large language model), releasing of model checkpoint, other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
37

---

Prompis
#if Instruction
You are an expert evaluator tasked with assessing checklists for goal-directed weh navigation. These checklists are designed to guide an agent through multi-step tasks on website. Each checklist consists of subgoals, presented hep-by-step. with brief descriptions explaining the purpose of each step. Using the provided intent, start URL, and the reference checklist, evaluate the quality of the checklist according to the following criterion.
Criteria: Subgoal Granularity
aligned with the level of detail found in the reference checklist? Are the checklist steps appropriately scoped, neither fine-grained coarse, and
Using the rubric below, provide brief justification and assign score from to 5 (number only, where very poor and excellent).
Rubric:
-I: Very poor: Checklist is extremely unbalanced granularity, with most steps being ei- ther overly line-grained or excessively coarse, making the structure difficult to interpret or use.
-2: Poor: There are several steps with inappropriate granularity- detailed or too
broad and the overall checklist lacks consistency in how actions are broken down.
-3: Fair: The checklist has mix of well-scoped and pourly scoped steps, with
few instances of overly fine coarse granularity that cause mild disruption in flow.
4: Good: Most steps appropriately scoped, with only minor inconsistencies in granularity density that do not significantly hinder readability execution.
Excellent: The generated checklist strikes an appropriate level of granularity neither too coarse nor too closelysely resembling the reference checklist. addition, progression through the checklist items is relatively uniform density.
Respond in the following format: Justification:
Score:
##f Input
Intent:
(intent)
start_url: {start_url)
reference checklist: (reference_checkklist
generuted checklist: (generated_chedeee
## Output
Figure 27: Prompt used to evaluate the quality of the generated checklist based on subgoal granularity.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers have some path reproducing verifying the results.
38

---

Prompis
#if Instruction
You are an expert evaluator tasked with assessing checklists for goal-directed weh navigation. These checklists are designed to guide an agent through multi-step tasks on website. Each checklist consists of subgoals, presented hep-by-step. with brief descriptions explaining the purpose of each step. Using the provided intent, start URL, and the reference checklist, evaluate the quality of the checklist according to the following criterion.
Criteria: Goal Coverage
Does the checklist comprehensively reflect the key steps necessary to achieve the final goal, captured the reference checklist?
Using the rubric below, provide brief justification and assign score from to 5 (number only, where very poor and excellent).
Rubric:
-I: Very poor: Checklist omits most key steps found the reference checklist and contains vague, irrelevant, or misleading content.
-2: Poor: Checklist includes a few relevant steps, but misses many essential ones
from the reference checklist, resulting in structure that does not support goal completion.
-3: Fair: Checklist reflects most major sleps from the reference checklist but misses
two key actions or includes loosely related steps.
4: Good: Checklist includes nearly all essential steps from the reference checklist,
with only minor omissions slight ambiguities in otherwise coherent structure.
Excellent: Checklist fully captures all key steps covered the reference check- list, with clear subgoals that directly support achieving goal.
Respond the following format:
Justification:
Score:
## Input
Intent:
(intent)
start_url: (start_ur]]
reference checklist: cnnffrrene__checkliie
generuted checklist:
(generated_checkliis
## Output
Figure 28: Prompt used to evaluate the quality of the generated checklist based on goal coverage.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
The code for our experiments and example data from WEBPRM COLLECTION and WERREWARDBENT are included in the supplemental material.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
39

---

Please see the NeurIPS code and data submission guidelines (https://nips.. cc/ htte//uuiides/ooeSutaissssssss
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ /nips. htcc/pullc/gguuiiiee hutbbiii/guuddeeeeeeuuuuiiii http///eoeeeesiiioeooooooooooo ssionpolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide the training and inference hyperparameters in Appendix C.2.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: In the experiment in Section 6.2 we sample 20 outputs from the policy allow reliable experiment.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
40

---

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The compute resources used for running the experiments are described in Appendix C.2.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http///eerrrps.eeee http://tttp..ppppptte/e/eeeeeeet
Answer: [Yes]
Justification:
We have carefully read the code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss them in Appendix A.2.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinfornation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
41

---

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification:
We describe the verification process for constructing our dataset in Appendix
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We include the citations and the explanations for the models, benchmarks, and code used in our work.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcon thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re- e-packaaed, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
42

---

Justification: We provide the details of WEBPRM COLLECTION and WEB- SEB-HHEEDERD in Appendix and Appendix C.2, respectively.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification:
We detail the human annotation process in Appendix B.2.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: All annotation is made by the authors.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
43


Reviews:
[{"review_id": "S6cFhb870T", "reviewer": "Reviewer_MZUs", "summary": "This paper introduces Web-Shepherd, a PRM specifically designed to evaluate and guide web navigation agents at a step-by-step level. Prior work has relied on prompting, general-purpose MLLMs as evaluators, which is sometimes slow and expensive for practical use. This paper first constructs \"WebPRM Collection\", a large-scale dataset of 40K step-level preference pairs, where each task instruction is paired with a human-annotated checklist of subgoals. Second, it introduces WebRewardBench, a evaluation benchmark for accurately assessing the performance of web navigation PRMs without requiring costly full trajectory rollouts. Web-Shepherd works in two stages: it first generates a task-specific checklist from the user's instruction and then uses the checklist to assign dense, step-level rewards to an agent's actions. Experiments demonstrate that Web-Shepherd significantly outperforms strong MLLM baselines like GPT-4o on WebRewardBench. Furthermore, when used to guide a GPT-4o-mini policy in a trajectory search setting on WebArena-lite, it achieves a substantial performance boost of over 10 points, while being 10x more cost-efficient than using the same MLLM as a verifier.", "questions": "Please see Weaknesses above.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n- [s1] This paper is well written and easy to follow.\n- [s2] WebPRM Collection (40K data) and WebRewardBench (accompanying with WebArena and Mind2Web) can be good resources for the research community (and potentially industry).\n- [s3] Constructing checklist seems to be the good way to evaluate the intermediate process in web navigation.\n- [s4] The proposed Web-Shepherd-8B achieved the best performance among other MLLMs on WebRewardBench (in Table 1). Also, GPT-4o-mini/GPT-4o policy with Web-Shepherd-8B achieved the best performance on WebArena-lite.\n\n### Weaknesses\n- [w1] WebRewardBench seems to be already saturated in Table 1. There are a bit doubt about its sustainability (or effective duration) as a benchmark.\n- [w2] Evaluating intermediate state of web navigation would be ill-defined tasks (no signle solution, the key state and action are very sparse in some tasks). I feel it is fine for now because Web-Shepherd-8B actually improved the trajectory search performance. But PRM approach may not the optimal approach to guide web navigation performance.\n- [w3] There is no comparison against ORM in trajectory search. For instance, https://arxiv.org/abs/2411.02337 has built ORM for web navigation.\n- [w4] The dataset and benchmark might be too connected with WebArena and Mind2Web. It may less help with other web navigation tasks or situations.\n- [w5] In Table 1, any models cannot leverage any image informations at all.", "was_revised": true, "final_justification": "The reviewer provided the detailed follow-up experiments in the rebuttal. My concerns are [w1] benchmark saturation and sustainability, [w2]  optimality of PRM approach to guide web navigation performance, and [w4] \"overfitting\" to WebArena and Mind2Web. I understood those would face mixed opinions, and I feel the authors provided some positive signs to deal with them. As in the initial review, I'm leaning toward acceptance.", "rebuttal": "Dear Reviewer MZUs,\n\nThank you for your deep understanding of our work, and appreciate the interesting question you raised. We would like to address the concerns in the comments below:\n\n---\n\n## W1\n\nWe notice that Web-Shepherd has already achieved pretty high performance on WebRewardBench, especially in terms of MRR. However, when examining step-level accuracy (Acc step) and trajectory-level accuracy (Acc traj), there is room for improvement.   For example, in Table 6 (Appendix E), Acc traj of Web-Shepherd (8B) is around 50-60 in Mind2Web.\n\nIn addition, considering recent findings that reasoning models (e.g., OpenAI’s o-series) can be used as PRM \\[1\\] and its superiority of evaluating web agents \\[2\\], we tested o4-mini’s performance on WebRewardBench. Even though it benefits from a long sequence of reasoning, we find o4-mini struggles in WebRewardBench.\n\n|  | Mind2Web |  |  | WebArena |  |  |\n| :---- | :---- | :---- | :---- | :---- | :---- | :---- |\n|  | MRR (step) | ACC (step) | ACC (traj) | MRR (step) | ACC (step) | ACC (traj) |\n| GPT-4o | 69.30 | 49.93 | 8.07 | 69.73 | 53.62 | 15.00 |\n| o4-mini (text) | 73.64 | 58.11 | 22.61 | 70.00 | 55.07 | 15.00 |\n\n---\n\n## W2\n\nWe fully agree with the reviewer’s point. We are still actively exploring the broader question of how to better guide web agents. As the reviewer noted, it remains unclear whether step-level rewards are inherently superior to outcome-based rewards, as each approach has distinct advantages and limitations.\n\n- **ORM:** Outcome-based reward modeling assesses only the final state to determine task success or failure. This simplifies the task objective and yields a more straightforward evaluation criterion compared to PRM. However, as noted in WebJudge \\[2\\], evaluating a web agent’s performance solely based on the final state overlooks the quality of intermediate actions and introduces a significant gap compared to human evaluation.\n\n- **PRM:** Process-based reward modeling evaluates agent behavior at a finer granularity by assessing step-level progress. While this provides a more detailed signal, it can introduce noise, especially when preferences over individual steps are ambiguous. In designing Web-Shepherd, we aimed to mitigate this by focusing on whether the agent achieves sub-goals, rather than modeling step-level preferences directly. As shown in Section 7.2, our approach (Checklist-based Generative Reward Modeling) effectively addresses key limitations of conventional PRM (e.g., Bradley-Terry modeling), demonstrating strong generalization to the WebArena benchmark.\n\nFor future work, a promising direction is to use both signals from ORM and PRM, leveraging the strengths of both to better capture task success and agent behavior.\n\n---\n\n## W3\n\nTo compare PRM with ORM, we use the ORM provided in WebRL which is based on LLaMA-3.1-8B-Instruct and trained on the expanded version of WebArena-Lite (which is in-domain for this experiment). For fair comparison, we trained Web-Shepherd using the same backbone LLM, LLaMA-3.1-8B-Instruct. We show the results in the table below.\n\n|  | Shopping  | CMS | Reddit | Gitlab | Map | Total |\n| :---- | :---- | :---- | :---- | :---- | :---- | :---- |\n| w/o Trajectory Search | 21.74 | 22.86 | 19.05 | 34.28 | 19.35 | 23.64 |\n| ORM | 32.61 | 17.14 | 14.29 | 31.25 | 16.13 | 23.64 |\n| PRM | 28.26 | 37.14 | 19.05 | 28.12 | 29.03 | 29.09 |\n\nIn terms of Total score, which is an average across all domains, ORM’s performance is par on w/o Trajectory search. There is a significant increase in Shopping domain (21.74 \\-\\> 32.61), but underperforms w/o Trajectory Search in other domains.  \nWhen comparing our PRMs with the ORM, our method outperforms in overall, showcasing the effectiveness of process-level guidance. Since this version is based on LLaMA-3.1-8B, which is usually weaker than Qwen-3, it results lower performance than our original Web-Shepherd. Lastly, while PRM chooses a single action to be executed without executing all action candidates, ORM needs to run 5 trajectories independently, which takes 5x more environment-agent interaction costs. \n\n---\n\n## W4\n\nWe agree that testing on more benchmarks to cover diverse domains would help increase the robustness of our experiments. Thus, we expanded the trajectory exploration experiment to WorkArena [3], which is completely out-of-domain for Web-Shepherd. (We also carefully inspect other benchmarks, such as AssistantBench, we frequently face CAPTCHA issues that ban web agents from further exploring the websites. We are still exploring other benchmarks that can possibly run our experiments.)\n\n|  | Dashboard | Form | Knowledge | List-filter | List-sort | Menu | Service Catalog | Total |\n| :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- |\n| w/o Trajectory Search | 50.00 | 0.00 | 10.00 | 0.00 | 5.00 | 25.00 | 2.22 | 9.39 |\n| GPT-4o-mini | 55.00 | 10.00 | 10.00 | 0.00 | 6.67 | 20.00 | 5.56 | 12.42 |\n| Web-Shepherd (3B) | 57.50 | 14.00 | 10.00 | 0.00 | 10.00 | 10.00 | 11.11 | 14.85 |\n| Web-Shepherd (8B) | 65.00 | 14.00 | 20.00 | 0.00 | 10.00 | 20.0 | 7.78 | 15.76 |\n\nWe find that trajectory search with PRM also improves the success rate in WorkArena when we compare w/o Trajectory Search and using GPT-4o-mini as PRM (i.e., the Total success rate increased from 9.39 to 12.42). In addition, Web-Shepherd outperforms GPT-4o-mini over all domains except Menu. Furthermore, Web-Shepherd outperforms GPT-4o-mini across all domains except for Menu. A potential explanation for the relatively low performance in the Menu domain is the complexity of the menu bar in WorkArena, which includes multi-level dropdown structures and search boxes embedded within the dropdowns. In such cases, the policy model often generates highly unreliable action candidates, and Web-Shepherd appears to be undertrained for handling these complex interaction patterns.\n\n---\n\n## W5\n\nWhen we first observed the results of multimodal input in Table 1, they seemed counterintuitive, and we initially suspected an error in our implementation. After extensive verification and repeated trials, however, we found no mistakes. We later realized that similar patterns of low performance with multimodal input have also been reported in prior work. For instance, AgentRewardBench \\[4\\], which introduces a benchmark for evaluating ORMs for web agents, similarly observes that incorporating both text and image observations can underperform compared to using text-only observation. This degradation is due to the increased complexity of multimodal input, which may distract the model and hinder learning.\n\nOn the other hand, we observe subtle improvements with visual information. In Table 6 (Appendix E.1), we provide results on WebRewardBench with the generated checklist — in the main table, we provide results with reference checklist. In the table below, we observe gentle improvements over all metrics without ACC (traj) in Mind2Web. It seems that self-generated checklists require much less reasoning cost compared to reference checklists that are unfamiliar to the model.\n\n|  |  | Mind2Web |  |  | WebArena |  |  |\n| :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- |\n| PRM | Modality | MRR | ACC (step) | ACC (traj) | MRR | ACC (step) | ACC (traj) |\n| GPT-4o-mini | Text | 56.5 | 30.6 | 3.9 | 51.3 | 30.4 | 5.0 |\n| GPT-4o-mini | Text+Image | 57.4 | 32.1 | 2.5 | 57.8 | 37.7 | 15.0 |\n\n---\n\n**References**  \n\\[1\\] Scaling Evaluation-time Compute with Reasoning Models as Process Evaluators (Kim et al., 2025)\n\n\\[2\\] An Illusion of Progress? Assessing the Current State of Web Agents (Xue et al., 2025\\)\n\n\\[3\\] Workarena: How capable are web agents at solving common knowledge work tasks? (Drouin et al., 2024)\n\n\\[4\\] AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectories (Lù et al., 2025\\)"}, {"review_id": "WwwFkQyEBZ", "reviewer": "Reviewer_nLse", "summary": "The authors propose to use process-based reward predicted by a relatively small specialized model  to provide fine-grained feedback for long horizon web navigation tasks carried out by web agents. This is in contrast with works that ask heavyweight off the shelf MLLMs to verify whether each step is successful, and promises to be more efficient and effective. Furthermore, the authors create and share the WebPRM Collection dataset with human per-step annotations for 40k trajectories, along with the WebRewardBench that measures the performance of a candidate model on this dataset. With this dataset, the authors train a new model (Web-Shepherd) to provide step-wise process-based reward assignment. The authors show that this model greatly outperforms current out of the box large proprietary and open source models for this task.", "questions": "It would be great to see the authors’ response to the concerns listed in the weaknesses section. \n\nAs well, as with dataset papers, it would be nice to have some insight into data scaling laws: how much impact on the WebShepherd model would there be if the training dataset size was halved vs current vs doubled? This could give the reader an understanding of whether there is merit in further scaling up such data collection.", "limitations": "It would be great to see the authors’ response to the concerns listed in the weaknesses section. \n\nAs well, as with dataset papers, it would be nice to have some insight into data scaling laws: how much impact on the WebShepherd model would there be if the training dataset size was halved vs current vs doubled? This could give the reader an understanding of whether there is merit in further scaling up such data collection.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n\nThe work represents a significant contribution to the community to advance agentic AI for web tasks by providing the large and useful WebPRM Collection dataset, which is based on human collected trajectories of interactions that have undergone a rigorous level of review by evaluators. \n\nThe fine-tuning of the WebShepherd on the proposed dataset is well conceived and executed. The experiment to produce a lightweight Web-Shepherd model shows convincing performance improvements over state of the art proprietary and open source models, and is shown to translate to significantly higher success rates when used to guide trajectory search for GPT-4o and GPT-4o-mini. \n\nThe paper is overall very well written and structured, and very thorough in providing a complete view of the dataset’s characteristics in the supplementary material. For example, the data collection diagram in figure 3 provides ample information to understand the process and its strengths / limitations. The authors undertake an extensive amount of quantitative analysis of the impact of the WebShepherd model, for example by looking into exactly how much improvements in checklist quality translates to the accuracy of rewards. \n\nWeaknesses\n\nThe originality of the work is not exceptionally strong. Process based rewards have been well understood to yield better performance for the final model when used as a much denser supervision source. It is also not entirely surprising that a small model fine-tuned on the PRM dataset can outperform off the shelf larger models. \n\nAs well, the work is missing some important citations. While it is true that to my knowledge the work is the first to propose a human annotated dataset for process based rewards in the context of web agents, the similar concept has been used for other domains. For example, ReST-MCTS (Zhang et al, NeurIPS2024) proposed per step rewards for math and scientific reasoning. VisualPRM (Wang et al, arxiv 2025 - not yet published in peer reviewed medium) provided a PRM dataset for visual reasoning. It would be good to contrast the work with these and explain how the authors’ proposal differs / improves on the methodologies.", "was_revised": true, "final_justification": "The new experimental results showing the performance scaling relative to amount of data available is illuminating, and it seems that future work to further increase the dataset size can still result in additional performance gain.", "rebuttal": "Dear Reviewer nLse,\n\nWe appreciate your comments and positive feedback on our work. We will address the concerns and the questions raised by the reviewer in the comments below:\n\n---\n\n## W1\n\nWe agree that process-based rewards have been previously explored and are known to provide denser supervision, often leading to improved downstream performance. Our work does not claim to introduce an entirely novel concept, but rather focuses on adapting and operationalizing PRM in the context of complex, multi-turn web environments—an application domain where reward modeling poses unique challenges. While it may not be surprising that a small model fine-tuned on the PRM dataset can outperform larger off-the-shelf models, we believe our contributions lie in the careful design of the reward modeling, extensive analysis, and the empirical insights obtained from adapting PRM to this setting.\n\n---\n\n## W2\n\nThank you for highlighting the importance of prior work such as Rest-MCTS and VisualPRM. As the reviewer noted, these studies have effectively demonstrated the benefits of process-level supervision, and our work builds upon these insights. The key distinction between our approach and theirs lies in how the process-level reward data is constructed. While Rest-MCTS and VisualPRM rely on MCTS or Monte Carlo sampling to annotate reward scores, such methods are heavily dependent on the performance of the underlying policy model. This dependency poses a significant limitation in the context of web agents, where the action and observation spaces are extremely large, resulting in a very low likelihood of obtaining positive rewards through rollout-based methods. \n\nOur work provides a path toward modeling PRMs for web agents where the process reward is hard to estimate from the outcome reward due to relatively low success rate and significant cost of agent-environment interaction time during rollout. We leverage synthetic and human annotation in a hybrid manner and improve generalization capability. We will make sure that this discussion is included in the next version of the draft.\n\n---\n\n## Q1\n\nFollowing the reviewer’s suggestion, we conducted analysis on the effect of the (1) number of instructions, and (2) number of rejected actions in the dataset on the performance of the PRM. Specifically, we construct datasets using the subset of WebPRMCollection 0.25, 0.5, and 0.75 percent of instruction and its corresponding chosen-rejected pairs and 1,2, and 3 number of max rejected actions. We trained variants of Web-Shepherd with these datasets using the same model (i.e., Qwen-2.5-3B-Instruct) and hyperparameters. The results are shown in the tables below.\n\nOverall, if we use about half of the original dataset (in terms of both the number of instructions and the number of rejected actions), there is a drastic decrease in ACC (traj) on both of the benchmarks. Especially, in the out-of-domain benchmark, WebArena, instruction ablation results in ACC (traj) decreases from 60.0 to 15.0, which suggests it failed to generalize to unseen domains. In rejected ablation, only decreasing by one rejected action is critical, resulting 60.0-\\> 20.0 ACC (traj) score in WebArena. These results highlight that both the number of instructions and the number of rejected actions are critical for training an effective PRM; reducing either significantly impairs generalization, particularly in out-of-domain settings such as WebArena.\n\n**\\[Ablation on the number of instructions\\]**\n\n|  | Mind2Web |  |  | WebArena |  |  |\n| :---- | :---- | :---- | :---- | :---- | :---- | :---- |\n| Proportion of instruction | MRR (step) | ACC (step) | ACC (traj) | MRR (step) | ACC (step) | ACC (traj) |\n| 0.25 | 68.38 | 49.75 | 11.34 | 65.97 | 47.83 | 10.00 |\n| 0.5 | 77.46 | 62.56 | 21.54 | 73.57 | 56.52 | 15.00 |\n| 0.75 | 83.64 | 72.00 | 33.63 | 88.09 | 79.71 | 55.00 |\n| Ours (3B) | 87.62 | 79.72 | 48.57 | 91.06 | 85.51 | 60.00 |\n\n**\\[Ablation on the number of rejected actions\\]**\n\n|  | Mind2Web |  |  | WebArena |  |  |\n| :---- | :---- | :---- | :---- | :---- | :---- | :---- |\n| Number of max rejected action | MRR (step) | ACC (step) | ACC (traj) | MRR (step) | ACC (step) | ACC (traj) |\n| 1 | 71.42 | 52.77 | 12.79 | 63.04 | 43.48 | 10.00 |\n| 2 | 77.66 | 62.72 | 17.55 | 76.47 | 63.77 | 20.00 |\n| 3 | 79.70 | 66.04 | 24.91 | 77.46 | 63.77 | 20.00 |\n| 4 (Ours, 3B) | 87.62 | 79.72 | 48.57 | 91.06 | 85.51 | 60.00 |"}, {"review_id": "zSFqk5qHtS", "reviewer": "Reviewer_88aJ", "summary": "This paper proposes a checklist-based process reward model (PRM) for web agents. The PRM is trained on a large dataset, the WebPRM collection, of step-level preference pairs on web navigation tasks collected by the authors. To evaluate the PRM better, they also proposed a WebReward Bench and showed that Web-Shepherd performs better than GPT-4o-mini. They also evaluated it by using a best-of-N search on WebArena and proved that Web-Shepherd can serve as a good reward model, outperforming GPT-4o-mini.", "questions": "1. Could you provide some concrete examples? Maybe pick one or two tasks to briefly summarize in which step and how the reward model helps guide the agent? This will make it more intuitive.\n2. Some missing experimental details: was the main experiment conducted with the verbalizer scoring strategy?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n1. PRM is an important topic for building better web agents and can be applied in multiple different scenarios: inference-time search, RL, etc.\n2. This work produces a reward model, along with the dataset to train on and a benchmark to evaluate reward models for web agents. Overall, it is solid and impressive\n3. The WebPRM collection is based on Mind2Web, but the reward model trained on it, i.e. Web-Shepherd, performs well on WebArena, which is quite out-of-domain.\n\nWeaknesses:\n1. Is Web-Shepherd trained on both checklist generation and reward generation, or only trained on reward generation? If it is also trained on checklist generation, one potential problem is that the accuracy of the generated checklist is bounded by GPT-4o. If it is co-trained on both tasks, will it result in better performance to train two distinguished models to do these two tasks rather than a single one, given that the nature of the two tasks does not align perfectly?\n2. I noticed a performance drop in GitLab split when using GPT-4o as the base policy and WebShepherd as PRM. I think one possible reason why this happens could be the low quality of the checklist generated on this website. It would be interesting to conduct an experiment comparing with a ground-truth evaluator (human) in a later version to see how well WebShepherd generates reward given the oracle checklist, but considering the large amount of work required for this, this is not required during this period.", "was_revised": true, "final_justification": "Most of my issues are solved, except for part of W2, which is about the weakness of the checklist-based reward model. However, I do not think it is a reason to reject the paper. I think the main part of the paper is solid enough to be accepted.", "rebuttal": "Dear Reviewer 88aJ,\n\nWe appreciate your thoughtful review and recognition of our efforts. We hope that the following responses help clarify the reviewer’s questions:\n\n---\n\n## W1\n\nYes, Web-Shepherd is trained on both checklist generation and reward modeling. To ensure the checklist quality is not biased or limited by GPT-4o’s outputs, we provide human-annotated trajectories to GPT-4o when generating a checklist dataset. This ensures that the model learns from expert reasoning grounded in successful demonstrations.\n\nWe fine-tuned Qwen2.5-3B under two settings: (a) only reward generation dataset, and (b) joint training with both checklist and reward generation dataset. We evaluated both settings on WebRewardBench using ground-truth checklists. As shown in our results, this joint training does not lead to significant performance degradation. In fact, we observe gentle improvements in certain out-of-domain tasks, suggesting that multi-task learning may enhance generalization.\n\n|  | Mind2Web |  |  | WebArena |  |  |\n| :---- | :---- | :---- | :---- | :---- | :---- | :---- |\n|  | MRR (step) | ACC (step) | ACC (traj) | MRR (step) | ACC (step) | ACC (traj) |\n| (a) only reward generation | 88.84 | 80.31 | 48.35 | 89.06 | 82.61 | 45.00 |\n| (b) checklist \\+ reward generation | 87.62 | 79.72 | 48.57 | 91.06 | 85.51 | 60.00 |\n\n---\n\n## W2\n\nThank you for the insightful suggestion. To better understand the performance drop in the GitLab, we manually annotated checklists and conducted experiments using these human-annotated checklists. We observed performance improvements compared to using self-generated checklists. We found that the checklist quality for GitLab was slightly lower than for other websites, which may also have contributed to the observed performance drop.\n\nHowever, the performance gain was still limited compared to the baseline (i.e., w/o trajectory search). We identified two main bottlenecks:  \n(1) Lack of correct action candidates due to limitations in the policy model, and  \n(2) Reward prediction errors that occur even when high-quality checklists are provided.\n\nThis is likely because the remaining failures after GPT-4o’s greedy search tend to be more challenging cases, which are less amenable to recovery through reward-guided search alone. \n\n| Policy: GPT-4o | GitLab |\n| :---- | :---- |\n| w/o trajectory search | 56.25 |\n| w/ self-generated checklist (Ours 8B) | 46.88 |\n| w/ human checklist (Ours 8B) | 60.00 |\n\n---\n\n## Q1\n\nHere’s an example for clarification.  \nTask: “File upload” on a CMS\n\nFirst, WebShepherd generates the checklist:  \n(1) navigate to the correct page,  \n(2) locate the upload button, and  \n(3) select a file.\n\nThen, WebShepherd provides natural language feedback and a scalar reward for the agent’s actions. For example:\n\nAction: “Clicked on the Settings’ tab”\n\n- Feedback: “The agent clicked on the Settings tab, which does not directly lead to the upload interface…”  \n- Reward: 0.09 (checklist 1 is incomplete)\n\nAction: “Clicked the 'Media' page”\n\n- Feedback: “The agent navigated to the Media page, which is the correct destination for uploading files...”  \n- Reward: 0.31  (checklist 1 is completed)\n\nThis fine-grained, step-level supervision provides informative signals that help the agent make progress toward the final goal. Moreover, the natural language feedback can serve as an additional source of guidance when the agent needs to refine.\n\n---\n\n\n## Q2\n\nFollowing the previous work \\[1\\], we use the verbalizer to calculate the score. This allows the reward model to produce more discriminative scores by computing probabilities over semantically meaningful tokens (e.g., “Yes”, “No”) rather than relying solely on scalar outputs.\n\n\n---\n\n**References**  \n\\[1\\] Generative verifiers: Reward modeling as next-token prediction. (Zhang et al., 2024\\)"}, {"review_id": "qP7Og1EpGr", "reviewer": "Reviewer_THVf", "summary": "This paper introduces Web-Shepherd, the first process reward model (PRM) designed for assessing web navigation tasks step-by-step. To train and evaluate this model, the authors created two new resources: the WebPRM Collection, a large dataset with 40,000 step-level preference pairs, and WebRewardBench, the first benchmark for evaluating web navigation PRMs. Web-Shepherd utilizes a unique checklist-based system to decompose high-level instructions into clear subgoals, enabling more accurate progress assessment. Experiments show that Web-Shepherd significantly outperforms existing models, achieving about 30 points higher accuracy than GPT-4o on WebRewardBench and improving agent success rates on the WebArena-lite benchmark by 10.9 points, all while being 10 times more cost-effective than using a prompted GPT-4o-mini as a verifier", "questions": "- The author could elaborate more on the experimental setting for checklist generation. For example, which model they used? Is the checklist generation model the same architecture as the reward model introduced in section 5.2? Did you train a single model on both checklist generation and reward modeling? Is your checklist generation data basically a distilled gpt-4o but with guidance from expert trajectory during the distillation process?\n\n- In Table 1, for WEB-SHEPHERD (3B), did you train two separate models for T and T+I settings? Why didn't you report performance for the T + I setting for a 8B model as well?\n\n- In Table 2, we can have separate experiments with a fixed checklist for every PRM to better understand the reward modeling capabilities by isolating checklist quality. \n\n- How can your approach address situations where a “change-of-plan” is needed? It is possible that the agent may encounter an unseen website and the checklist generated is not applicable to the websites. In these scenarios, the agent would likely need to modify the checklist (therefore leading to increase/decrease in checklist length, etc). Can your approach still provide useful feedback for refining actions (Table 3) in these cases?\n\n- Line 189, the equation should be divided by K since it’s average?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "__Strengths__:\n- The introduction of a PRM benchmark and checklist-based reward modeling is interesting and impactful.\n\n- The paper has conducted rigorous experiments with insightful discussions.\n\n- The proposed approach is effective not only on the proposed reward modeling benchmark but also on “downstream” web agent tasks.\n\n\n__Weaknesses__:\n\n- The paper only conducts web agent evaluation on one benchmark. Showing effectiveness on additional benchmarks like VisualWebArena can help us better appreciate the performance and understand the usefulness of process rewards based on multimodal inputs.\n\n- The authors could better demonstrate the values of their approach by showing that equipping with such reward-guided search improves efficiency (e.g. less searched states, takes shorter steps to reach goals).", "was_revised": true, "final_justification": "The authors have addressed all my concerns and I would recommend this paper to be accepted.", "rebuttal": "Dear Reviewer THVf,\n\nFirst of all, we sincerely appreciate the time and effort you took to review our work, and we are grateful for your recognition of its contributions.\n\n---\n\n## W1\n\nWe also acknowledge the reviewer’s suggestion that evaluating on additional benchmarks would strengthen the paper. In response, we conducted supplementary experiments on the **WorkArena** benchmark \\[1\\], which differs from WebArena in task structure and domain.\n\nSpecifically, we used GPT-4o-mini as the policy model and compared the performance of greedy search and reward-guided search with GPT-4o-mini and Web-Shepherd (3B and 8B). As shown in the results below, reward-guided search with Web-Shepherd continues to show clear advantages, validating the transferability of our method to unseen websites.\n\n|  | Dashboard | Form | Knowledge | List-filter | List-sort | Menu | Service Catalog | Total |\n| :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- |\n| w/o Trajectory Search | 50.00 | 0.00 | 10.00 | 0.00 | 5.00 | 25.00 | 2.22 | 9.39 |\n| GPT-4o-mini | 55.00 | 10.00 | 10.00 | 0.00 | 6.67 | 20.00 | 5.56 | 12.42 |\n| Web-Shepherd (3B) | 57.50 | 14.00 | 10.00 | 0.00 | 10.00 | 10.00 | 11.11 | 14.85 |\n| Web-Shepherd (8B) | 65.00 | 14.00 | 20.00 | 0.00 | 10.00 | 20.00 | 7.78 | 15.76 |\n\nAlthough we additionally experimented with benchmarks that include both textual and visual observations, our current model is mainly trained on textual inputs. We made this design choice deliberately, as we found that naively incorporating screenshots often introduces ambiguity and noise that can hinder performance. Nonetheless, we recognize that certain tasks (e.g., “find the red shirt”) require understanding visual context. This highlights the need for future work to develop more effective multimodal reward models that can jointly leverage textual and visual inputs for web agent training. Furthermore, we agree with the reviewer’s suggestion and plan to extend our experiments to VisualWebArena to better assess the model’s capabilities in multimodal scenarios.\n\n---\n\n\n## W2\n\nThank you for raising this important point regarding efficiency. To evaluate whether our reward-guided search improves not only accuracy but also sample efficiency, we conducted a comparison against greedy search. As shown in the table below, our reward-guided approach significantly improves both success rate (Acc) and efficiency (Avg. Step to reach the goal) across different policies.\n\nIn particular, we observe that using a strong reward model (Ours-8B) leads to more efficient behavior, achieving higher accuracy with fewer steps, compared to using GPT-4o-mini as the reward model. This underscores the critical role of a robust reward model in guiding agents effectively during search.\n\n| Policy | Search Method | Acc | Avg. Step |\n| :---- | :---- | :---- | :---- |\n| GPT-4o-mini | w/o trajectory search | 23.64 | 19.07 |\n|  | GPT-4o-mini | 24.24 | 19.06 |\n|  | Ours (8B) | 34.55 | 14.73 |\n| GPT-4o | w/o trajectory search | 31.52 | 16.16 |\n|  | GPT-4o-mini | 26.67 | 13.30 |\n|  | Ours (8B) | 39.39 | 12.51 |\n\n\n---\n\n\n## Q1\n\nAs illustrated in Figure 5, WebShepherd is not a framework but a single unified model that performs two key roles: (1) generating a checklist for a given task, and (2) providing step-level rewards and natural language feedback for the agent’s actions based on the checklist.  \nTo train the model, we use high-quality checklists derived from human-annotated trajectories with GPT-4o. Checklist generation is not distilled from GPT-4o, but learned directly from expert supervision, enabling strong generalization.  \n\n---\n\n## Q2\n\nWhen we compare text-only input and multimodal input in the 3B setting, we do not observe a notable improvement from using the multimodal input. Thus, we do not further train 8B models for multimodal input. Furthermore, our GPU resource is not enough to run finetuning 8B VLMs. \n  \n---\n\n## Q3\n\nThank you for this insightful suggestion. To isolate the effect of checklist quality on reward modeling, we conducted an experiment using manually annotated checklists. Specifically, we annotated checklists for 50 tasks and compared reward-guided search performance using self-generated versus human-annotated checklists.\n\nThe results (see table below) indicate that there is no significant performance gap between the two settings. This suggests that once the checklist quality reaches a certain threshold, the performance is more influenced by the reward prediction or the policy model. However, since the evaluation was done on a limited set of 50 examples, further large-scale experiments are necessary to draw definitive conclusions.\n\nAdditionally, we observed that while our model generally produces useful and well-structured checklists, there are notable failure cases:\n\n- **Redundant Checklist**: The model occasionally adds unnecessary subgoals, which can confuse the reward assignment or mislead the agent.  \n- **Missing Subgoals**: Some key steps are omitted, resulting in coarse-grained checklists that fail to reflect the full task structure.\n\nAlthough these issues did not result in notable performance drops in this experiment, they highlight opportunities for improvement in checklist generation quality.\n\n| PRM | Method | Accuracy |\n| :---- | :---- | :---- |\n| \\- | w/o trajectory search | 0.18 |\n| GPT-4o-mini | w/ self-generated checklist | 0.20 |\n|  | w/ human checklist | 0.24 |\n| Ours (3B) | w/ self-generated checklist | 0.24 |\n|  | w/ human checklist | 0.24 |\n\n---\n\n## Q4\n\nWe appreciate the reviewer’s interesting question. When the checklist is completely wrong, we can assume that there is an error in the checklist generation given that the reward score does not increase after several attempts. In these cases, a practical strategy is to sample a new checklist from Web-Shepherd, possibly refining it based on the agent’s past trajectory.\n\nImportantly, our model is explicitly trained to generate coarse-grained checklists rather than overly detailed ones. This design choice helps avoid overfitting to specific website structures and allows for better generalization to unseen websites. As shown in our main experiments on WebArena and additional experiments on WorkArena, Web-Shepherd performs effectively even in out-of-distribution environments.\n\nFurthermore, we observed that even if part of the checklist is incorrect, the remaining steps still provide effective reward. This allows the reward model to continue offering valuable feedback, helping the agent refine its actions and move toward the goal, even when a change-of-plan is required.  \n\n---\n\n## Q5\n\nThank you for pointing this out. We will correct the equation by dividing by $K$ in the next version of the draft.\n\n---\n\n**References**  \n\\[1\\] Workarena: How capable are web agents at solving common knowledge work tasks? (Drouin et al., 2024\\)"}]

Author rebuttal:

Meta-review: This paper proposes Web-Shepherd, a process reward model for web navigation, offering step-level evaluation that outperforms GPT-4o in terms of accuracy by a large margin. The work also contributes with the WebPRM Collection, a large-scale dataset with 40K step-level preference pairs and introduces WebRewardBench, a meta-evaluation benchmark for evaluating PRMs.

Summary of Strengths:
- The paper is clearly written and well-structured.
- The work contributes with new resources and a benchmark: the WebPRM Collection and WebRewardBench can benefit both research and industry.
- Web-Shepherd outperforms larger models on WebRewardBench and improves trajectory search performance in WebArena-lite while being more cost-efficient.
- Even when the model is trained with Mind2Web data, Web-Shepherd generalizes well to WebArena, showing robustness to different data.
- PRMs are useful for advancing web agents during inference-time search and reinforcement learning.

Summary of Weaknesses:
- The paper doesn't compare to ORMs. There were discussions on this topic during the rebuttals, and authors included additional experimentation to clarify.
- The dataset and the benchmark are tightly tied to WebArena and Mind2Web, and hence the work may have limited applicability  to broader web navigation settings. Showing results on additional benchmarks could be useful, which is mentioned as future work during the rebuttals.

The discussions and clarifications that the authors provided resulted in score increases by some reviewers. Given the resources and the novel contributions with good results, the paper would be a good contribution to NeurIPS.
-
Decision: Accept (spotlight)