Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge
Authors: name: Boyu Gou; name: Zanming Huang; name: Yuting Ning; name: Yu Gu; name: Michael Lin; name: Weijian Qi; name: Andrei Kopanev; name: Botao Yu; name: Bernal Jiménez Gutiérrez; name: Yiheng Shu; name: Chan Hee Song; name: Jiaman Wu; name: Shijie Chen; name: Hanane Nour Moussa; name: Tianshu Zhang; name: Jian Xie; name: Yifei Li; name: Tianci Xue; name: Zeyi Liao; name: Kai Zhang; name: Boyuan Zheng; name: Zhaowei Cai; name: Viktor Rozgic; name: Morteza Ziyadi; name: Huan Sun; name: Yu Su
Published: 2025-06-26 17:32:50

Abstract: Agentic search such as Deep Research systems, where large language models
autonomously browse the web, synthesize information, and return comprehensive
citation-backed answers, represents a major shift in how users interact with
web-scale information. While promising greater efficiency and cognitive
offloading, the growing complexity and open-endedness of agentic search have
outpaced existing evaluation benchmarks and methodologies, which largely assume
short search horizons and static answers. In this paper, we introduce Mind2Web
2, a benchmark of 130 realistic, high-quality, and long-horizon tasks that
require real-time web browsing and extensive information synthesis, constructed
with over 1,000 hours of human labor. To address the challenge of evaluating
time-varying and complex answers, we propose a novel Agent-as-a-Judge
framework. Our method constructs task-specific judge agents based on a
tree-structured rubric design to automatically assess both answer correctness
and source attribution. We conduct a comprehensive evaluation of nine frontier
agentic search systems and human performance, along with a detailed error
analysis to draw insights for future development. The best-performing system,
OpenAI Deep Research, can already achieve 50-70% of human performance while
spending half the time, showing a great potential. Altogether, Mind2Web 2
provides a rigorous foundation for developing and benchmarking the next
generation of agentic search systems.