BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of
  Deep-Research Agent
Authors: name: Zijian Chen; name: Xueguang Ma; name: Shengyao Zhuang; name: Ping Nie; name: Kai Zou; name: Andrew Liu; name: Joshua Green; name: Kshama Patel; name: Ruoxi Meng; name: Mingyi Su; name: Sahel Sharifymoghaddam; name: Yanxi Li; name: Haoran Hong; name: Xinyu Shi; name: Xuye Liu; name: Nandan Thakur; name: Crystina Zhang; name: Luyu Gao; name: Wenhu Chen; name: Jimmy Lin
Published: 2025-08-08 17:55:11

Abstract: Deep-Research agents, which integrate large language models (LLMs) with
search tools, have shown success in improving the effectiveness of handling
complex queries that require iterative search planning and reasoning over
search results. Evaluations on current benchmarks like BrowseComp relies on
black-box live web search APIs, have notable limitations in (1) fairness:
dynamic and opaque web APIs hinder fair comparisons and reproducibility of deep
research methods; (2) transparency: lack of control over the document corpus
makes it difficult to isolate retriever contributions. In other words, the
current evaluations may compare a complete deep research system at a given
time, but they do not foster well-controlled experiments to provide insights
into the capability of underlying deep research LLMs. To address these
challenges, we introduce BrowseComp-Plus, a benchmark derived from BrowseComp,
employing a fixed, carefully curated corpus. Each query in BrowseComp-Plus
includes human-verified supporting documents and mined challenging negatives,
enabling controlled experimentation. The benchmark is shown to be effective in
distinguishing the performance of deep research systems. For instance, the
open-source model Search-R1, when paired with the BM25 retriever, achieves
3.86% accuracy, whereas the GPT-5 achieves 55.9%. Integrating the GPT-5 with
the Qwen3-Embedding-8B retriever further enhances its accuracy to 70.1% with
fewer search calls. This benchmark allows comprehensive evaluation and
disentangled analysis of deep research agents and retrieval methods, fostering
insights into retrieval effectiveness, citation accuracy, and context
engineering in Deep-Research system.