Web-Shepherd: Advancing PRMs for Reinforcing Web Agents
Authors: name: Hyungjoo Chae; name: Sunghwan Kim; name: Junhee Cho; name: Seungone Kim; name: Seungjun Moon; name: Gyeom Hwangbo; name: Dongha Lim; name: Minjin Kim; name: Yeonjun Hwang; name: Minju Gwak; name: Dongwook Choi; name: Minseok Kang; name: Gwanhoon Im; name: ByeongUng Cho; name: Hyojun Kim; name: Jun Hee Han; name: Taeyoon Kwon; name: Minju Kim; name: Beong-woo Kwak; name: Dongjin Kang; name: Jinyoung Yeo
Published: 2025-05-21 08:56:55

Abstract: Web navigation is a unique domain that can automate many repetitive real-life
tasks and is challenging as it requires long-horizon sequential decision making
beyond typical multimodal large language model (MLLM) tasks. Yet, specialized
reward models for web navigation that can be utilized during both training and
test-time have been absent until now. Despite the importance of speed and
cost-effectiveness, prior works have utilized MLLMs as reward models, which
poses significant constraints for real-world deployment. To address this, in
this work, we propose the first process reward model (PRM) called Web-Shepherd
which could assess web navigation trajectories in a step-level. To achieve
this, we first construct the WebPRM Collection, a large-scale dataset with 40K
step-level preference pairs and annotated checklists spanning diverse domains
and difficulty levels. Next, we also introduce the WebRewardBench, the first
meta-evaluation benchmark for evaluating PRMs. In our experiments, we observe
that our Web-Shepherd achieves about 30 points better accuracy compared to
using GPT-4o on WebRewardBench. Furthermore, when testing on WebArena-lite by
using GPT-4o-mini as the policy and Web-Shepherd as the verifier, we achieve
10.9 points better performance, in 10 less cost compared to using GPT-4o-mini
as the verifier. Our model, dataset, and code are publicly available at LINK.