DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents
Authors: Mingxuan Du
Benfeng Xu
Chiwei Zhu
Licheng Zhang
Xiaorui Wang
Zhendong Mao
Venue: ICLR 2026 Poster (2026)
Keywords: LLM based Agent
Evaluation
Deep Research
Primary area: datasets and benchmarks

Abstract: Deep Research Agents (DRAs) are emerging as one of the most practical classes of LLM-based agents. Given an open-ended research task, they find, analyze, and synthesize large numbers of online sources to produce a comprehensive report at the level of a research analyst. This can compress hours of manual desk research into minutes. However, a comprehensive benchmark for systematically evaluating the capabilities of these agents remains absent. To bridge this gap, we introduce DeepResearch Bench, a benchmark consisting of 100 PhD-level research tasks, each meticulously crafted by domain experts across 22 distinct fields. To evaluate DRAs comprehensively, we propose two complementary and fully automated methodologies. The first is a reference-based method with adaptive criteria to assess the quality of generated research reports. The second evaluates a DRA’s information‑retrieval and collection capabilities by assessing its effective citation count and overall citation accuracy. By conducting extensive human consistency experiments, we demonstrate that our evaluation methods are highly aligned with expert judges and faithfully reflect human judgments of quality differences among DRA-generated content. We are open-sourcing DeepResearch Bench and key components of these frameworks to accelerate the development of practical LLM-based agents.

Published as a conference paper at ICLR 2026
DEEPRESEARCH BENCH: A COMPREHENSIVE BENCHMARK FOR DEEP RESEARCH AGENTS
Mingxuan Du Benfeng Xu Chiwei Zhu', Licheng Zhang', Xiaorui Wang', Zhendong Mao ! University of Science and Technology of China, "MetastoneTechnology Beijing, China dumingxuan, benfeng) ) @mail. uste. edu. en
ABSTRACT
Deep Research Agents (DRAs) emerging one of the most practical classes of LLM-based agents, Given open-ended research task, they find, analyze, and synthesize large numbers of online sources to produce a comprehensive report the level of a research analyst. This can compress hours of manual desk research into minutes. However, comprehensive benchmark for systematically evaluating the capabilities of these agents remains absent. To bridge this gap, we introduce DeepResearch Bench, a benchmark consisting of 100 PhD-level research tasks, each meticulously crafted by domain experts across 22 distinct fields. To evalu- ate DRAs comprehensively, we propose two complementary and fully automated methodologies. The first is reference-basee method with adaptive criteria assess the quality of generated research reports. The second evaluates DRA's information--ettiewwe and collection capabilities by assessing its effective cita- tion count and overall citation accuracy. By conducting extensive human consis- tency experiments, we demonstrate that our evaluation methods are highly aligned with expert judges and faithfully reflect human judgments of quality differences among DRA gennrreee content. We have open-sourced DeepResearch Bench and key components of these frameworks at https://github.com wwt.://cao//iiiiiiimee Ayanamaann00000 30ddepp research benchheeehhghe to accelerate the development of practical LLM--ased agents.
Comrni-22 5 Doep Research
Opensi Deep Research
Purpissity Doop Research
Grok Desper Search
61.2
Efective Chatton (Count)
FACT Effective Citatin
Overall
Camprehandveneess
Desth
Instruction Following
Readabilty
Citation Accurscy
Figure 1: Overview of four Deep Research Agents' performance on DeepResearch Bench. The left figure shows the report quality scores achieved by the DRAs. The right figure shows each agent's citation accuracy and the average number of effective citations.
"Work done during the internship at Metastone.
I Project lead.
I "oorrsppnniing author.

---

Published as a conference paper at ICLR 2026
INTRODUCTION
We are now entering a new phase Al Yao 2025), a period marked by comprehensive advances in Large Language Model (LL.M) capabilities (DeepSeek-Al et al. 2025 OpenAl 2024). These advancements enable the construction of LLM-based Agent systems designed to tackle increasingly
complex tasks (Masterman et al. 2024 Hong et al. 2024 Yang et al. 2024). In this evolving landscape, defining tasks that genuinely reflect world demands and designing robust evaluation methodologies to measure the progress of these Agent systems are becoming critically important. Deep research represents one such well-defined task domain, with Deep Research Agents (DRAs) (Li et al. 2025 Zheng et al., 2025a Schmidgall et al., 2025) emerging the most widely utilized LLM-based agents today.
However, comprehensively evaluating DRAs is challenging. Because their internal reasoning and retrieval are opaque, the final report is the primary observable. Moreover, for complex research tasks, establishing definitive ground truth often infeasible.
These demanding evaluation requirements for DRAs pose a significant challenge for existing eval- uation frameworks, which often fall short of offering dedicated assessment of the multifaceted capabilities of such agents (Liu et al. 2023). Current benchmarks typically focus on assessing iso-
lated capabilities--Sti suuch as web browsing and information retrieval (Wei al. 2025 Zhou et al,, 2025
2024), or generative abilities disconnected from real-time information acquisition (Que et al.
2024
Bai et al. 2024, Wu et al., 2025b).
To bridge this gap, we introduce DeepResearch Bench, 100-task benchmark across 22 domains, with each task crafted and iteratively refined by domain experts. To reflect real research needs, we allocate per-domain task counts via a statistical analysis of over 96,000 user queries, following the pipeline in Figure aa.)
Building on this dataset, we introduce two novel, highly human-aligned evaluation methodologies. The first one is Reference- based and Adaptive Criteria-driven Evaluation framework with Dynamic Weighting (denoted RACE for ease of subsequent reference), which targets the assessment of report generation quality. And the other one a framework for Factual Abundance and Citation Trustworthiness (denoted FACT), which focuses evaluating information retrieval and citation accuracy. Overview results are shown in figure I Furthermore, we believe these methodologies are not confined to deep research scenarios; see Appendix for broader discussion.
Our primary contributions are as follows:
We present DeepResearch Bench, the first specialized benchmark for evaluating Deep Research Agents, built via large-scale analysis of real user queries and close collaboration with domain experts, balancing challenge while faithfully reflecting authentic needs.
We further propose RACE and FACT, two novel evaluation frameworks that respectively assess the report generation quality and the information retrieval abilities of Deep Research Agents.
We conduct comprehensive human studies validate the reliability of our frameworks, and will publicly release the benchmark and evaluation protocols upon acceptance to foster future research.
DEEPRESEARCH BENCH CONSTRUCTION
2.1 TOPIC DISTRIBUTION ANALYSIS
Deep Research Agents (DRAs) are intended to serve actual human research needs. Therefore, to effectively test their capabilities, the design of DeepResearch Bench is grounded in the real-world distribution human research task demands. To obtain this distribution, we collected an in-house dataset of 96, 147 raw user queries from interactions with web search-enabled Chatbots. To ensure user privacy, all raw query logs underwent rigorous anonymization. Further details of the in-house data provided in Appendix B
Following the pipeline shown in Figure a[),, we then defined the concept of Deep Research tasks as problems requiring agents to conduct multiple rounds of web searches, gather information, perform
2

---

Published as a conference paper at ICLR 2026
96ke Real- World Search Query Pool
44k Filtered deep nesearch tasks
Task Court of Tapics
Finance Business-
Technology
Business
Sclennc Tech tasks 3.Softwrre tasks
Deep Research Tesk Recanding og the
Refine
Topic Distribttion
Whi le the market
fastinnrr therde
Topic Demoins
problen UAVA, rest ojen saroe
Iestte ID0- berch Experts Froduce
etrehegld like matti-
Software Health
Hedft
Total 44ke
comennlyy inylorent
coanoded
fezter ligh-
locks cangle
A.
&
By Faoncil sevvity
Tesk Dahoset
and Cento Dng weuting
(e) Distribution Analysis Data Collection
Deep Research Agents
Task Dataset
Deep Research Report
Reference Report
c
Ø
frevide of sarnent glibal eriledee All techesoll rested componis, eg.
Adaptive Critenia
Judge LLM
C1 Idden-iichha Ficeton and befinitonn
2c
Judge LLM
Ascessor whether the report deerry idernniees cpp nocches or schools thought camerity preseet
Approsctes
C1 (Atellydi Torger ideniiied red techrooo approaches whik Reference iden prinary approoches.
Weighting
Waight82
5/10 Reference: 2/10
Comprehenstvonesse
2 Counrig of Regnecsrhetive Compernce per appresch
(2 [Arolysi] Torger Report focuded textual ondlyii
Iniight/DepthD222
Ascessss ahether The repert sdeett e cuffisiont
Instruction Follewiny
Weight
Score Rafemmres Rapami
Store of Torget Report: ST:3 (halei 2Pa
Readiabiltt:0.15
SF-ST/15T-5R)
(B) RACE Framework
Deep Research Report
Statement-URR List
Judge LLM
Stetenont One: The brender
URL One: lettae /veve sook- woos/hetafic dago
Judge
Judge LLM
Prtppe
Extract
Pai Twee wn Not Support
Statement-URL List
Pair N
Statevent Ni looos,, Aooox
URL, httpgt///mm..oxx
Poir Support
FACT Framework
Page
Figure 2: Overview of DeepResearch Bench. (a) Distribution analysis and dataset construction pipeline. (b) RACE framework. (c) FACT framework.
analysis, and produce high-quality reports. Guided by this definition, we employed DeepSeek- V3-0324 (DeepSeek-A] (025) to conduct filtering, identifying queries that aligned with our deep research requirements. This process ultimately yielded dataset of 44,019 queries conforming to our deep research task definition.
To categorize the deep research queries, we adopted the topic taxonomy proposed by WebOrganizer (Wettig et al. 2025), selecting 22 distinct topic domains for this classification. We then employed DeepSeck-V3-0323 to classify these 44,019 queries into the 22 selected topic domains. By statis- tically aggregating the LLM's classification results, we obtained the distribution of these queries across the various topics. This distribution, visualized in Figure g indicates the real-world user demand for deep research within these domains.
2.2 BENCHMARK TASK COLLECTION
Guided by the observed user-demand distribution, we set the target number of tasks per domain and proportionally compressed it to final set of 100 tasks (50 Chinese, 50 English), preserv- ing the topical balance. The bilingual design reflects the multilingual nature of real-world deep research demands. The dataset size was limited because running a single deep-research task is resource-intensive: moreover, many frontier benchmarks, such as xbench-DeepSearch (Chen et al.) 2025a) and Mind2Web 2 (Gou et al. 2025), contain roughly one hundred tasks, indicating that this scale represents a practical trade-off between quality and stability.
Once the target task count for each topic domain was determined, our focus shifted to constructing research tasks that are both highly challenging and firmly grounded in authentic research demands. We invited PhD holders and senior practitioners with over five years of relevant domain experience to propose candidate tasks. It is important to clarify that these 100 benchmark tasks were indepen- dently created by domain experts based on the target allocations, rather than being selected from the
3

---

Published as a conference paper at ICLR 2026
Rewer
17.80%
16.05%
Sofware Develepennt
2019
5 1.5%
Crtea
4,22'
11.5%%
5.0%%
633
8.10%
8.0%%
Food Diingg
Fadiin leauty
550
Hileoy
Cearss
Figure 3: Topic distribution of the filtered deep-research tasks. share. Right: bar chart of absolute task counts 22 domains.
Left: donut chart showing topic
44,019 filtered queries. The 44K samples served solely inform the topic distribution, ensuring our benchmark reflects real-world demand patterns. Examples are shown in Figure All submissions underwent manual screening by our team to verify their quality, clarity, complexity, and alignment with our definition of deep research. This rigorous vetting process resulted in the 100 high-quality benchmark tasks that constitute DeepResearch Bench.
Science & Technology
Finance & Business
Deep Research Task
Deep Research
Regording the attitude control problem for UAVS, most open- source flight contreliers curmently implement coscaded PTD control alganithms, However, single parameters typically per forms well only conditionns In practical applications, UAVs operate ocross flight states. What methods can be employed to enhance the actual control performance of PID algorithms, and how should PID porometers be optimally selected?
While the morket features diverse quantitative strotegies like multi factor and frequency trading, lacks single, stondardized benchmark assessing multiple dimensions such as returns, risk, adeptobility to market conditions. Could we develop general yet nigorous evaluation framework to enable accurate comparison and analysis of verious advanced strategies?
FID controller
performance across
specific flight
Autemation Brginaaring
PDD Pinaciaa Enginearing
Figure 4: Two example tasks from DeepResearch Bench.
3 EVALUATION METHODOLOGY
Our evaluation methodology focuses on two critical aspects: their capabilities in information re- trieval and collection, and the quality of the final reports they generate. To assess these respective dimensions, we developed two complementary frameworks, RACE and FACT.
3.1 RACE: A FRAMEWORK FOR REPORT QUALITY EVALUATION
Evaluating long-form research reports presents significant challenges. Existing approaches using fixed checklists Que et al. (2024) or static rubries Shao et al. (2024); Bai al. (2024) struggle to adapt to diverse tasks, specialized domains, and nuanced quality aspects of deep research tasks. To address this, we introduce our Reference-basee Adaptive Criteria-driven Evaluation framework with Dynamic Weighting (RACE). leveraging the LLM-as-a-Judge method Zheng et al. (2023). RACE offers a more adaptive and robust evaluation by first dynamically generating task specific weights and criteria. It then employs reference-basee -ased scoring approach, comparing the target report to high-quality reference. Finally, a relative score computed to assess the target report's quality.
Dynamic Weight & Adaptive Criteria Generation. Directly prompting LLMs to generate task- specific criteria from scratch can lead to results that deviate significantly from the intended assess- ment goals. Following Google's Gemini Deep Research guidance (Google 2025), adopt similar

---

Published as a conference paper at ICLR 2026
high-level design principles and establish four orthogonal top-level dimensions based on domain expertise: Comprehensiveness (COMP), Insight/Depth (DEPTH), InstructionnFoolloiing (INST), and Readability (REAd). Detailed definitions are provided in Appendix
As illustrated in Figure 2),, for each task t. the Judge LLM is prompted to produce task task-specific weights Wa for the four dimensions d E {COMP, DEPTH, INST, READ]. These weights ensure the evaluation aligns with the task's 's intent. Subsequently, for each dimension d, the Judge LLM
generates a set of tailored criteria {ca,*} with corresponding weights [wa..} (where Σk=1 Ka Wd, = 1), which are clear and actionable for evaluating the report within that dimension. Importantly, once generated for a task, these criteria and weights remain fixed across all DRA evaluations for that task, ensuring consistent and fair comparison. This query-based approach also enables future work to easily scale up the testset by generating criteria for new tasks.
Reference- Based Scoring. Preliminary experiments indicated that scoring reports in isolation often yields insufficiently discriminative results; models tend to assign uniformly high scores, masking genuine quality variations. To mitigate this, RACE adopts a reference- based scoring strategy. For each task f, high-quality report Riet is selected as reference. All generated criteria {ca,a} across all dimensions are aggregated into a comprehensive list C.- The Judge LLM then analyzes the target report Rege and the reference report Reef against each criterion E C.. This yields lists of scores for both reports for each criterion, which are then used for final score calculation:
(fsugt,cheEC., (srer,etecc) = JudgeLLM(t, Regt. Ryet, Ci)
(I)
Overall Score Calculation. Finally, we compute the overall quality score of the target report. First, dimension-level scores Sa(It) are calculated by weighting criterion-level scores SR.ca, with criterion weights Wd,k- Second, these Sa(R) scores are combined using the task-specific dimen- sion weights Wa to produce intermediate overall scores Smn (R). Finally, the target report's ' score Stmal (Regt) is determined relative the reference report's score:
Sint (Regs)
Sünal (Regt) =
(2)
Sint (Rigt) ) + Sine (Bret)
3.2 FACT: FRAMEWORK FOR WEB RETRIEVAL EVALUATION
To assess the factual grounding of report content and the agent's effectiveness in retrieving and utilizing web-based information, introduce framework for Factual Abundance and Citation Trustworthiness(AAA
As illustrated in Figure (c), let I denote the set of all tasks in the bench- mark. For each task E T, FACT evaluates DRAs through the following automated steps:
Statement-URL Pair Extraction and Deduplication. We employ a Judge LLM to extract discrete statements from the DRA-generated report for task along with their corresponding cited source URLs. Let Pt denote the initial set of all extracted statement- URL pairs for task t. The Judge LLM then examines Pt to identify cases where multiple statements associated with the same URL describe exactly the same fact. In such cases, only one representative pair is retained. We denote the deduplicated set as U., with Nu,t = [U]] representing the total number of unique statement-URL. pairs for task t.
Support Judgment. Each unique statement-URL pair in U, undergoes a support evaluation. We retrieve the textual content of the cited webpage using the Jina Reader API, then the Judge LLM assesses whether this content provides sufficient evidence for the statement. This yields a binary judgment: 'support' or 'not support'. Let N., denote the number of pairs in U, that are judged as 'support" for task
Calculation of Citation Metrics. Based on these support judgments across all tasks, we calculate two primary metrics:
Citation Accuracy (C. Acc.) measures the precision of an agent's 's citations, reflecting its ability to correctly ground statements with appropriate sources. For each task f, we first compute the per-task accuracy as Acc; Nsa/Na, (with Acc; if Nu,, = 0). The overall Citation Accuracy is then
5

---

Published as a conference paper at ICLR 2026
the average across all tasks:
C. Acc.
Σ Acci
(3)
[
Average Effective Citations per Task (E. Cit.) quantifies the average number of verifiably sup- ported statements an agent generates per task. sums the supported pairs across all tasks and divides by the total number of tasks:
Exer No.t
E. Cit.
(4)
(T]
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
When employing the RACE framework, pre-processing step involves cleaning citation formatting from the generated reports, as overly lengthy or complex citation styles can adversely affect the Judge LLM's scoring process. For RACE evaluation tasks, we utilize Gemini-2.5-pro as the Judge LLM. As for the FACT framework, Gemini-2.5-flan both Statement- URL pair extraction and support judgment, which sufficient capabilities while more economic for the token-consuming citation verification task. The reference reports used in RACE's scoring methodology were selected from deep research articles generated by the Gemini- 2.5-pro-based Deep Research, as available in April 2025. All main results reported in Table [ are evaluated on the complete set of 100 tasks from DeepResearch Bench.
Implementation Details
is employed for
Evaluated Models In work. we broadly evaluate leading commercial Deep Research Agents, including OpenAl Deep Research, Gemmin--.--pro-baseed Deep Research and so on. Due to the lack of transparency regarding the iteration cycles of these commercial products, specify the data collection timeframes in Appendix [E] In the open-source domain, we evaluate two representative RL-based systems with open-source frameworks: (1) DeepResearcher ZZeng al. (2025b), trained via reinforcement learning in real-world web environments, and (2) Tongyi DeepResearch Fu et al. 2025), leveraging RL training with Qwen3 the base model. Both are locally deployed following their official open-source implementations. Additionally, we evaluate LangChain's 's Open Deep Re- search (ODR) (LangChain al. 2025), an open-source framework with frontier LLM as backbone. We test it with both GPT-4.1 and GPT-5 as the backbone LLMs, with configurations detailed in Appendix [F] We also evaluate strong LLMs with built-in search tools under standardized conditions by setting the
size to high; see Appendix G.2
4.2
MAIN RESULTS
4.2.1
EVALUATION ON RACE FRAMEWORK
As shown in Table I within the DRA category, Gemini-2.5-PPr Deep Research achieved the highest overall performance, while OpenAl Deep Research also demonstrated strong capabilities. Notably, the open-source LangChain Open Deep Research (ODR) further exhibited competitive results, sur- passing several proprictary DRAs in our evaluation. When upgraded from GPT-4.1 to GPT-5 as the backbone, LangChain ODR achieves an overall score of 50.60, surpassing even Gemini-2.5-Pro Deep Research, demonstrating that open-source frameworks can achieve competitive results with state-of-the--a proprietary models when powered by advanced LLMs.
The scores for these top agents are relatively close, which is characteristic of the reference-based relative scoring employed by RACE. However, this should not be concerning, as subsequent exper- iments 4.3 revealed strong linear correlation between these RACE scores and human judgments, suggesting the framework effectively captures meaningful performance differences between models. In fact, these scores are highly linearly correlated with human evaluations, just mapped to a different reference frame (similar to how scores of 45 and 50 versus human scores of 90 and 100). Therefore, we should focus on rankings and proportional differences between scores rather than absolute score values.
For fully open-source DR systems, we observe significant performance gaps. Both DeepRe- searcher Zheng al. (2025b) and Tongyi DeepResearch Fu al.) (2025) trained via RI. with

---

Published as a conference paper at ICLR 2026
Table 1: Overall evaluation results of DeepResearch Bench. Bold denotes the highest score in each column for Deep Research Agents (and for LLM with Search Tools within their respective section). Underlined denotes the second highest.
RACE
FACT
Model
Overall Comp. Depth Inst.
Read. C. Acc. E. Cit.
Open-Source LLM + Open-Source Agent Framework
DeepResearcher
10.77
8.66
5.96
15.57
15.65
Tongyi DeepRescarch
40.46
39.46
34.44
46.22
44.27
Proprietary LLM Open- Agent Framework
LangChain Open DeepResearch (GPT-5)
50.60
50.06
50.76
51.31
49.72
32.94
21.06
LangChain Open DeepResearch (GPT-4.1)
43.44
42.97
39.17
48.09
45,22
49.10
29.49
Proprietary Deep Research Agent
Claude Rescarch
45.00
45,34
42.79
47.58
44.66
Grok Deeper Search
38.22
36.08
30.89
46.59
42.17
73.08
8.58
Perplexity Deep Research
40.46
39.10
35.65
46.11
43.08
82.63
31.20
Doubao Deep Research
44.34
44.84
40.56
47.95
44.69
52.86
52.62
Gemini-2.5-PP Deep Research
49.71
49.51
49.45
50.12
50.00
78.30
165.34
Kimi Researcher
44.64
44.96
41.97
47.14
45.59
OpenAl Deep Research
46.45
46.46
43.73
49.39
47.22
75.01
39.79
Proprietary LLM Built-in Search Tools
Claude-3.7 3.7--onnet
Sonnet w/Search
40.67
38.99
37.66
45.77
41 .6
93.68
32.48
Hlude-3.5-Somme w/Search
28.45
24.82
22.82
35.12
35.08
94.04
9.78
Saaeexity-Sona--Reaaan
40.22
37.38
36.11
45.66
44.74
39.36
8.35
Perplexity-Sone SeresittySSonn--eee Sonar--onnnn
40.18
37.14
36.73
45.15
44.35
48.67
11.34
Perplexity-Sonn Ponar-Pro
38.93
36.38
34.26
44.70
43.35
78.66
14.74
Perplexity- -Sonar
34.54
30.95
27.51
42.33
41.60
74.42
8.67
35.12
34.06
29.79
41.67
37.16
81.81
32.88
32.39
31.63
26.73
38.82
34.48
81.92
31.08
GPT-40-Search-Previae
35.10
31.99
27.57
43.17
41.23
88.41
4.79
GPT-4o-Mini-Scarde
Sww.--ccrarhc-ceeeee Preview
31.55
27.38
22.64
40.67
39.91
84,98
4.95
GPT-4.1 w/Search
33.46
29.42
25.38
42.33
40.77
87.83
4.42
GPT-4.1-miml w/Search
30.26
26.05
20.75
39.65
39.33
84.58
4.35
open-source frameworks that can be locally deployed. DeepResearcher achieves overall score of 10.77, significantly lower than other systems. Through manual inspection, we found that it fre- quently fails to generate complete, well-structured reports, which substantially impacts its RACE scores. In contrast, Tongyi DeepResearch achieves an overall score of 40.46, comparable to Per- plexity Deep Research among proprietary DRAs. This demonstrates that with appropriate RI train- ing, open-source systems can approach the performance of commercial products, though challenges remain in generating high-quality long-form reports consistently.
Moreover, traditional LLMs with built-in search (often limited to single-round or a few simple search turns) now struggle compete with modern DRAs under identical evaluation settings.
4.2.2 EVALUATION FACT FRAMEWORK
Viewing evaluation results by FACT Table Deep Research Agents (except Grok) tend to include more Effective Citations than LLMs with Search Tools. Notably, Gemini-2.5-Pro Deep Research achieved an average of 165.34 effective citations in its final reports, significantly outperforming other models. This suggests that can retrieve and integrate more information from a larger amount of evidence, potentially enabled by longer context window and stronger context understanding. However, its citation accuracy is lower than that of Perplexity Deep Research and far below Claude-

---

Published as a conference paper at ICLR 2026
3.7 w/Search, indicating trade-off between citation aceuracy and effective citation counts some extent.
We also note that Claude Research and Kimi Research do not have FACT scores because were unable to parse citation links from their official UI. For open-source RL-based systems (DeepRe- searcher and Tongyi DeepResearch), FACT scores unavailable because these systems failed to follow instructions to generate citations in the required format, making citation extraction and ver- ification infeasible. Meanwhile, LangChain Open Deep Research and Doubao show relatively low FACT scores. For LangChain Open Deep Research, the low citation accuracy can be attributed to its multi-agent architecture with dedicated summarization sub-agent, which causes misalignment between citations and the retrieved content during the summarization process. For Doubao, based on our inspection, many webpages reachable by built-in browse tools were inaccessible to our Jina- based crawling pipeline, and even for accessible pages, the fetched contents sometimes differed, which likely affected the measured metrics.
Overall, the FACT framework is designed to offer complementary observation to RACE; given the lack of transparency and variability built-in search/browse tools across DRAs, we use FACT as an observational dimension for analysis, while our benchmark's s overall rankings rely on RACE scores.
4.3 HUMAN CONSISTENCY
Evaluating the quality of deep research reports remains open-ended task. Therefore, to validate the effectiveness of our proposed RACE framework, we must rely on assessing its human consis- tency. We conducted experiments using 50 Chinese tasks from DeepResearch Bench, with reports generated by four distinct agents. For each task, three domain-expert annotators scored these reports. Further details are provided in Appendix G.1
4.3.1 HUMAN DATA COLLECTION
To gather human judgments, we recruited 70+ annotators with Master's degrees and relevant do- main expertise. Using custom interface, they evaluated reports across four dimensions and overall performance, guided only by basic scoring criteria minimize bias. Each annotator was limited to three queries maximum to ensure diverse perspectives.
4.3.2 EVALUATION METRICS
To validate the consistency between evaluation methods and human judgment, designed four metrics that quantify different aspects of alignment with human evaluations. The detailed calculation processes for all metrics provided in Appendix H
Pairwise Agreement Rate (PAR) This metric measures how often our evaluation method's prefer- ences match human experts' preferences when comparing pairs of reports.
Overall Pearson Correlation (OPC) This metric quantifies the linear relationship between average model scores from evaluation method and those from human experts.
Filtered Average Pearson & Spearman Correlation To mitigate per-task noise, we filter out tasks with low inter-rater agreement (ICC < 0) and then compute two metries on the remaining subset: the Filtered Average Pearson Correlation (FAP) and the Filtered Average Spearman Correlation (FAS). Detailed definitions and formulas are provided in Appendix H
4.3.3 COMPARISON OF DIFFERENT EVALUATION METHODS
Given that existing evaluation methods are generally unsuitable for assessing DRAs, we compare RACE(Full) and several ablation variants against Vanilla Prompt baseline (direct scoring by the Judge LLM). As shown in Table 3 RACE(Full) achieves the best overall performance, significantly exceeding the baseline and other variants. Notably. its Pairwise Agreement Rate also surpasses hu- man agtemm,, indicating reliable and efficient aligned evaluation. We further include robustness experiments reference selection, article length, and judge model choice in Appendix!]
8

---

Published as a conference paper at ICLR 2026
Table 2: Comparison of human consistency
different evaluation methods. Prefixed with '-', indicating removal of specific components from the full framework. Best scores for each
metric among automated methods are in bold.
Evaluation Method
PAR
OPC
FAP
FAS
Overall Score
Vanilla Prompt
58.89
98.89
40,30
43.75
60.46
RACE(Full)
71.33
99.54
60.24
59.12
72.56
No Criteria Weights
70.67
99.62
59.83
56.27
71.60
- No Dim Weights
70.89
99.54
60.11
57.22
71.94
-No Weights
71.11
99.69
59,46
58.17
72.11
Reference
66.56
97.46
57.51
51.23
68.19
Reverse Position
69.56
97.20
56.75
55.49
69.75
Static Criteria
68.33
98.73
57.86
57.70
70.65
Human Antee-mgremment
68.44
Table 3: Comparison of human consistency scores, and average per task using different Judge
LLMs within the RACE(Full) framework. The best for each metric are bold
Judge LLM
License
PAR
OPC
FAP
FAS
Overall
Cost (S)
Gemini 2.5 Pro o3
Proprietary
71.33
99.54
60.24
59.12
72.56
0.13
Proprietary
68.11
96.22
57.64
$2.36
68.58
0.37
04-mini
Proprictary
70.89
97.06
59.54
59.02
71.63
0.04
Claude 3.7 Sonnet
Proprictary
70.78
96.53
58.22
63.61
72.28
0.47
Qwen3-2355 2.5B--hnn-
Apache 2.0
70.78
84.47
56.80
56.94
67.25
4.3.4 COMPARISON OF DIFFERENT JUDGE LLM
Leveraging the RACE framework, further compare the performance and cost of several leading LLMs when used as the Judge LLM. As detailed in Table 3 Gemini 2.5 Pro achieves the best overall performance and maintains competitive average cost ($0. per query), only higher than that of o4- 04-mini In addition, we experimented with open-source alternative, Qwen3-235B-Ae B-22BB Thinking-2507. as the Judge LLM. While its human consistency lags behind closed-source models, the gap is not large, suggesting it can serve as feasible open-source substitute. Moreover, with Qwen3 as the Judge, the relative ranking across the four DRAs remains consistent with the Gemini- based results, indicating that RACE is robust to the choice of Judge backbone. Further details on using open-source judges are provided Appendix L.3 To balance performance and cost in our main results, selected Gemini 2.5 Pro as the Judge LLM in our final framework.
5 RELATED WORK
LLM-based Agent Evaluation With the comprehensive advancement of LLM capabilities, LLM- based Agents are increasingly being applied to real-world scenarios Mon- Williams et al. (2025):
Wang et al. (2025), promising to alter many aspects of daily life and professional work significantly.
Yao's blog Yao (2025) highlights that defining more realistic problems and designing novel evalua- tion methods critical for constructing more practical AI Agent systems. Numerous evaluations have already been designed specifically for Agents, targeting diverse capabilities. These include evaluations for agents scientific domains CChn al.] (2025): Laurent et al. (2024); Mitchener et al. (2025); Chen et al. (2025b), creative writing Wu et al. (2025b): Bai et al. (2024): Que et al. (2024), code generation and software engineering Jimenez et al. (2024); Zhuo al. (2025) Quan et al. 2023): Jain et al.) (2024); Xiao et al. (2025), and in their roles as human assistants, often en- hanced by capabilities such as web browsing and tool-use Wei et al. (2025); Zhou et al. )025); Yan al. (2024); Deng et al. (2024); Wang et al. 2024). Closest to setting, Xu et al. (2025) focuses on a single scientific domain and does not reflect real-world user demand, while Bosse et al. (2025) adopts offline RetroSearch setting and reports process- oriented metrics (c.g., hallucinationn/tol usage) rather than evaluating report quality or citation fidelity. This perspective underscores our belief that constructing benchmarks specifically designed for Deep Research Agent, grounded in

---

Published as a conference paper at ICLR 2026
real-world scenarios, alongside developing human-alignen evaluation methods, is urgently needed to guide the development of Al agent systems.
Deep Research Agent (2025) and Google's s Gemini Google (2025), such agents have attracted significant attention and have become of the most widely deployed LLM-based agent categories. Subsequently, related works LangChain et al. (2025); Li et al. (2025): Zheng et al. (2025a) quickly followed up, also introducing their own designed DRA frameworks. However, the field still lacks a standardized evaluation methodology for these DRAs, preventing meaningful comparative analysis of their ca- pabilities. Among these works, some use QA datasets Phan al.) (2025); Mialon al. (2023); Wu et al. (2025a) as evaluation metrics, but this approach neither aligns with real-worke DRA applica- tions nor comprehensively assesses their broader capabilities. Others employ the -uue- methodology Zhenng et al. (2023), yet these efforts lack both comprehensive framework design and verification of human consistency. In contrast, our DeepResearch Bench addresses this gap by providing systematic, unified evaluation method with strong human consistency, supporting subsequent DRA development and assessment.
After the release of Deep Research Agents (DRAs) by OpenAl OpenAl
CONCLUSION
this work, we introduce DeepResearch Bench, the first comprehensive benchmark for evaluating the report generation and web retrieval capabilities of Deep Research Agents. Comprising 100 high-quality research tasks across 22 distinct domains, this benchmark meticulously curated to reflect authentic needs. Our key evaluation frameworks, RACE and FACT, have demonstrated high consistency with human judgments, affirming their reliability. We hope DeepResearch Bench will guide developers and researchers in constructing more powerful and human-centric AI agent systems, truly addressing genuine user requirements.
ETHICS STATEMENT
All authors affirm adherence to the ICLR Code of Ethics. This work does not involve personally identifiable information or sensitive attributes. Our in-house query logs were rigorously anonymized prior to processing (removing user identifiers, IPs, and session metadata). Human studies were conducted with experienced annotators, who were compensated fairly; no protected vulnerable populations were involved. We report results transparently and avoid claims beyond empirical ev- idence. Potential risks include misuse of agent outputs and propagation of web inaccuracies; our FACT framework and standardized citation formatting are designed to encourage faithful grounding and make verification easier. We disclose no conflicts interest or external sponsorship that would bias the findings.
REPRODUCIBILITY STATEMENT
We provide all essential details to reproduce our results. The benchmark construction pipeline (topic taxonomy, filtering and classification steps), evaluation prompts, and hyperparameters are documented in the main text and Appendix (e.g.. Sections 3 and implementation details in Sec- tion D. Exact Judge L.M configurations for R RACE/FACT. including dynamic weighting and criteria generation prompts, are included in the Appendix. All scripts and data used in this paper included in the supplementary materials, which also contain screenshots of the annotation interface and the annotator instructions/guidelinss facilitate faithful reproduction.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Yushi Bai, Jiajie Zhang, Xin Lv, Linzhi Zheng. Siqi Zhu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMss, August 2024.
org/abs /200.240...2 c7055 arXiv:2408.0700 [es].
Nikos I. Bosse, Jon Evans, Robert G. Gambee, Daniel Hnyk. Peter Muuhlbache,, Lawrence Phillips, Dan Schwarz, and Jack Wildman. Deep research bench: Evaluating ai web research agents. arXiv preprint ar0iv:2506. 250.0287000 2025. URL.
http://aaxxvv.../aab.a// org/abs./////oooooo 2506. 06287
Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherbum, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Lilian Weng, and Aleksander Madry. Mle-bench: Evaluating machine learning agents on machine learning engineering, 2025, URL https://arxi>> org/ abs /oabs/24100 ..110000099
Kaiyuan Chen, Yixin Ren, Yang Liu, Xiaobo Hu, Haotong Tian, Tianbao Xie, Fangfu Liu, Haoye Zhang, Hongzhang Liu, Yuan Gong, Chen Sun, Han Hou, Hui Yang. James Pan, Jianan Lou, Jiayi Mao, Jizheng Liu, Jinpeng Li, Kangyi Liu, Kenkun Liu, Rui Wang. Run Li, Tong Niu, Wen- long Zhang, Wenqi Yan, Xuanzheng Wang. Yuchen Zhang, Yi-Hsin Hung. Yuan Jiang, Zexuan Liu, Zihan Yin, Zijian Ma, and Zhiwen Mo. xbench: Tracking agents productivity scaling with profession-aligna real-world evaluations, 2025a. URL arxiv. org/ abs 2506. 13651
Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang, Botao Yu, Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, Vishal Dey, Mingyi Xue, Frazier N. Baker, Benjamin Burns, Daniel Adu- Ampratwum, Xuhui Huang, Xia Ning, Song Gao, Yu Su, and Huan Sun. ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery, March 2025b. URL /tLL:/tcpp///xxxvooo/////// 000..//0//220000000..00 arXiv:2410.05080 [cs].
DeepSeck-AL depssct-aiiDeeppees-3333.3 Hugging atwpeeppseee--a/Deeeeee
Hugging Face, March 2025. URL https://
DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao. Hui Qu, Hui Li. Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang. R. J. Chen, R. I.. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang. Shuiping Yu. Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun. Tian Pei. Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao. Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xi- aosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang. Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo. Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie. Zhengyan Zhang, Zhewen Hao, Zhicheng Ma. Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. DeepSeek-RI. Incentivizing Reasoning Capability in LLMs via Reinforcement Learning, January 2025. RRLhttp: //arxii////xv arXiv:2501. 12948 [cs].
arxiv. org/ abs/2501..2948

---

Published as a conference paper at ICLR 2026
Shihan Deng. Weikai Xu, Hongda Sun, Wei Liu, Tao Tan, Jianfeng Liu, Ang Li, Jian Luan, Bin Wang, Rui Yan, and Shuo Shang. Mobile-Bench: An Evaluation Benchmark for LLM-based Mo- bile Agents, July 2024. URLhttp: htap:::///xivv..i hww./org//bbb. hwwggg/bb//240... 00993 arXiv:2407.00993 [es].
Dayuan Fu et al. Qwen3 technical report. arXiv preprint arXiv:2510.2470) 2025, URL https: wttp:///aab./.....22./
Gemini Google. Deep Research is now available on Gemini 2.5 Pro Experi-
mental., April 2025.
httpe///bboogggggee///eeeoooo gemiin w-r------po---ee-r-----e
URL
Geep-researcio
Boyu Gou, Zanming Huang. Yuting Ning. Yu Gu, Michael Lin, Weijian Qi. Andrei Kopanev, Botao Yu, Bernal Jiménez Gutiérrez, Yiheng Shu, Chan Hee Song, Jiaman Wu, Shijie Chen, Hanane Nour Moussa, Tianshu Zhang, Jian Xie, Yifei Li, Tianci Xue, Zeyi Liao, Kai Zhang, Boyuan Zheng, Zhaowei Cai, Viktor Rozgic, Morteza Ziyadi, Huan Sun, and Yu Su. Mind2web 2: Evaluating agentic search with agent-as-a-judee 2025. https /ttpaaaaivv....///..//....a.wav. org/ /
2506. 21506
Sirui Hong. Mingchen Zhuge, Jiaqi Chen, Xiawu Zheng, Yuheng Cheng, Ceyao Zhang, Jinlin Wang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. MetaGPT: Meta Programming for A Multi-Agent Col- laborative Framework, November 2024. URL http:://ara wttpp//wwwwr///////..oooo 00352 arXiv:2308.003: 00352 [cs].
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code, June 2024. URL hhtt:://arxxt org/ abss/
2403. 07974] arXiv:2403.07974 [cs].
Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?, Novem- ber 2024.
2210. 06770. arXiv:2310.06770 [cs].
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. Re- wardbench: Evaluating reward models for language modeling. arXiv preprint 1xXiv:2403.13787 2024. URL
LangChain et al. Open deep research. GitHub repository, 2025. URL. httpp:::::/// com/ LangChain AI, Accessed: 2025-05-14.
langchain aatpnnaaapppeedeeeee wawpeeeneeeeee__eeeeee L
Jon M. Laurent, Joseph D. Janizek, Michael Ruzo, Michaela M. Hinks, Michael J. Hammerling, Siddharth Narayanan, Manvitha Ponnapati, Andrew D. White, and Samuel G. Rodriques. LAB- Bench: Measuring Capabilities of Language Models for Biology Research, July 2024. URL 10362 Ttt1:2207.10036 [cs].
Xiaoxi Li, Jiajie Jin, Guanting Dong, Hongjin Qian, Yutao Zhu, Yongkang Wu, Ji-Rong Wen, and Zhicheng Dou. WebThinker: Empowering Large Reasoning Models with Deep Research Capa- bility, April 2025. URL httt::///aaaiiiie org/ abs/ 2544. 21177 arXiv:2504.21776 [cs].
Xiao Liu, Hao Yu. Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. AgentBench: Evaluating LLMs as Agents, October 2023. URL
org. abs/
2308. 03688 arXiv:2308.0 000688008888888 [cs].
Saumya Malik, Valentina Pyatkin, Sander Land, Jacob Morrison, Noah A. Smith, Hannaneh Ha- jishirzi, and Nathan Lambert. Rewardbench 2: Advancing reward model evaluation. arXiv preprint arXiv:2506.072 01937, 2025. htts:tss:: /tte:s///aavvvv. wwwxii..or//aa... aww///0555 01937
Tula Masterman, Sandi Besen, Mason Sawtell, and Alex Chao. The Landscape of Emerging AI Agent Architectures for Reasoning, Planning, and Tool Calling: A Survey, April 2024. URL wtwp://wwwww...////////oaa/ 11584 arXiv:2404.111 11584 [cs].
http::/arxii org.
12

---

Published as a conference paper at ICLR 2026
Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. GAIA: a benchmark for General AI Assistants, November 2023. URL http:
//arxiv.org/aaa. httpg//ab//33111 12983 arXiv:2311.12983 [cs].
Ludovico Mitchener, Jon M. Laurent, Benjamin Tenmann, Siddharth Narayanan, Geemi P. Wellawatte, Andrew White, Lorenzo Sani, and Samuel G. Rodriques. BixBench: a Com- prehensive Benchmark for LLM-based Agents in Computational Biology, March 2025, URL http://arxivy: org/ absgg///bs// /200 00096 arXiv:2503.0009 [q-bio].
Ruaridh Mon-Williams, Gen Li, Ran Long, Wenqian Du, and Christopher G. Lucas. Em- bodied large language models enable robots to complete complex tasks in unpredictable environments. Nature Machine Intelligence, (04), April 2025. ISSN 2522- 5839. doi: 00000/444226--22-00005555
BtLhhtps:/// http::://www.atattee wtt.:/.coorrt...
142256-025-01005-
Publisher: Nature Publishing Group.
OpenAL. OpenAl ol System Card, December 2024. URL http:::/arriiv.org/bb///////
16720 arXiv:2412.16720 [cs].
OpenAl. Introducing deep research OpenAI,
Lntroddccno--eepp-reeeeee. Putm
Long Phan et al. Humanity's Last Exam, April 2025, URL http://axxxw.or///bb/////////
14249 1rXiv:2501.14249 [cs].
Shanghaoran Quan, Jiaxi Yang, Bowen Yu, Bo Zheng, Dayiheng Liu, An Yang, Xuancheng Ren, Bofei Gao, Yibo Miao, Yunlong Feng, Zekun Wang. Jian Yang, Zeyu Cui, Yang Fan, Yichang Zhang, Binyuan Hui, and Junyang Lin. CodeElo: Benchmarking Competition-level Code Generation of LLMs with Human-comparable Elo Ratings, January 2025. URL http:
//arxiv.org/art
01257 arXiv:2501.01257 [cs] version: 2.
Haoran Que, Feiyu Duan, Liqun He, Yutao Mou, Wangchunshu Zhou, Jiaheng Liu, Wenge Rong, Zekun Moore Wang, Jian Yang, Ge Zhang, Junran Peng, Zhaoxiang Zhang, Songyang Zhang, and Kai Chen. HelloBench: Evaluating Long Text Generation Capabilities of Large Language Mod- els, September 2024. URL http://aaaxi....a///////eee 16191 arXiv:2409.16199 [es].
Samuel Schmidgall, Yusheng Su, Ze Wang, Ximeng Sun, Jialian Wu, Xiaodong Yu, Jiang Liu, Zicheng Liu, and Emad Barsoum. Agent Laboratory: Using LLM Agents as Research Assistants, January 2025.
/ttpps:://aaxx.oorttr /2500 04227v1
Yijia Shao, Yucheng Jiang. Theodore A. Kanell, Peter Xu, Omar Khattab, and Monica S. Lam. Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Models, April 2024.
14207 arXiv:2402.14200 [es].
Luyuan Wang, Yongyu Deng, Yiwei Zha, Guodong Mao, Qinmin Wang, Tianchen Min, Wei Chen, and Shoufa Chen. MobileAgentBench: An Efficient and Friendri Benchmark for Mobile LLM Agents, June 2024. Uttp:t:: wwtpxivvvogg/// abs /lbbs/2405 2406 08184 arXiv:2406.0818. [cs].
Wenxuan Wang, Zizhan Ma, Zheng Wang, Chenghan Wu, Wenting Chen, Xiang Li, and Yixuan Yuan. A Survey of LLM--ased Agents in Medicine: How far are we from Baymax?, February 2025. URLLhttp://axxiiv.oom wttp://or/r/bb//.....11 2211 12Xiv:2502.12211 [cs] version: 1.
abs
Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung. Alex Tachard Passos, William Fedus, and Amelia Glaese. BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents, April 2025. URL http:://axxi....c//// http://aaxxvv..o//bbb.as/ org/abs/ arXiv:2504. 12516 [es].
2504. 12516
Alexander Wettig, Kyle Lo, Sewon Min, Hannanch Hajishirzi, Dangi Chen, and Luca Soldaini. Organize the Web: Constructing Domains Enhances Pre-Training Data Curation, February 2025. URL htll.htt:///art v. org. abs////////22 111552.113330 10341] arXiv:2502.108: 10341 [es].
13

---

Published as a conference paper at ICLR 2026
Jialong Wu, Wenbiao Yin, Yong Jiang. Zhenglin Wang. Zekun Xi. Runnan Fang, Linhai Zhang, Yulan He, Deyu Zhou, Pengjun Xie, and Fei Huang. WebWalker: Benchmarking LLMs in Web Traversal, January 2025a. URL http:///axivv.../aa..ooo XXiv:2501.07577 [es].
Yuning Wu, Jiahao Mei, Ming Yan, Chenliang Li, Shaopeng Lai, Yuran Ren, Zijia Wang, Zhang, Mengyue Wu, Qin Jin, and Fei Huang. WritingBench: Comprehensive Bench- mark for Generative Writing, March 2025b. URL http://wxwvv.rr///////5/..0.00 arXiv:2503.05244 [es].
Yijia Xiao, Runhui Wang, Luyang Kong, Davor Golac, and Wei Wang. CSR-Bench: Bench- marking LLM Agents in Deployment of Computer Science Research Repositories. In Lais Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Na- tions of the Americas Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies (Volume I: Long Papers), pp. 12705-12723, Albuquerque, New Mex- ico, April 2025. Association for Computational Linguistics. ISBN 979-8-99176-18966. URL www. /tas://accaane httpss//eelanttoooooo.or////// naacl naacl-ong. 633/
Tianze Xu, Pengrui Lu, Lyumanshan Ye, Xiangkun Hu, and Pengfei Liu. Researcherbench: Evaluating deep ai research systems on the frontiers of scientific inquiry. arXiv:2507.16280, 2025. URLhtttps:/aaxxi. http://s///aa.l.oooor/aa/////// 16880
arXiv preprint
Fanjia Yan, Huanzhi Mao, Charlie Cheng-Jie Ji, Tianjun Zhang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. Berkeley Function Calling Leaderboard, February 2024. URL https:
ca cal__callinn__.... leaderboard html biblex[howpublishee]]]] = ('url htttts/t////grrlla.ss..ettt wer.eeeeeeeeeeeeebeeeeeeeeee
//gorilla. wtw.laa....be.eee..eee.
ferptyyffuntttoo_ncee (ealrrooglladrrroorrhh.tml) citationkey: leaderboard.
John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineer-
ing, November 2024. URL http:///rxiy: org/ abs/ 2405 15793 arXiv:2405.1599 [cs].
Shunyu Yao. The Second Half, April 2025,
URL httpp::/ysyytth..iitit io/
The The-Second-HalF) Blog post.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing. Hao Zhang, Joseph Gonzalez, and Ion Stoica. Judging LLM LLM-as-a-Judge with MT-Bench and Chatbot Arena, December 2023. URL http: arXiv:2306.05685 [cs].
//axxiv.or// abs///b////// 2306 05685
Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu. DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Envi- ronments, April 2025a. URL http:: http:://wwwx.....////////////// 03100 arXiv:2504.03160 [es].
Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cal, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu. Deepresearcher: Scaling deep research via reinforcement learning in real-world environ- ments. arXiv preprint trXiv:2504.03160 2025b. URL wttttttt//// //t:::///xxiii/ wwt./oggg//aaaaeee
03160
Peilin Zhou, Bruce Leon, Xiang Ying, Can Zhang, Yifan Shao, Qichen Ye, Dading Chong, Zhiling Jin, Chenxuan Xie, Meng Cao, Yuxin Gu, Sixin Hong. Jing Ren, Jian Chen, Chao Liu, and Yining Hua. BrowseComp-ZIH Benchmarking Web Browsing Ability of Large Language Models in Chinese, May 2025. URL hollhttp:/// hattp://axxiv.... org/ abs 2504. 19314 arXiv:2504.193 19314 [cs].
Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. WebArena: A Realistic Web Environment for Building Autonomous Agents, April 2024. URL http:/ttp//aa org/ abs/ 2307. 13854 arXiv:2307. 13854 [cs].
14

---

Published as a conference paper at ICLR 2026
Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brumner, Chen Gong, Thong Hoang, Armel Randy Zebaze, Xiaoheng Hong, Wen-Ding Li, Jean Kaddour, Ming Xu, Zhihan Zhang, Prateek Yadav, Naman Jain, Alex Gu, Zhoujun Cheng, Jiawei Liu, Qian Liu, Zijian Wang, Binyuan Hui, Niklas Muennighoff, David Lo, Daniel Fried, Xiaoning Du, Harm de Vries, and Leandro Von Werra. BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions, April 2025. arXiv:2406.1877 15877 [es].
15

---

Published as a conference paper at ICLR 2026
A LIMITATIONS
While DeepResearch Bench and the RACE/FACT frameworks offer a comprehensive evaluation of Deep Research Agents, several limitations remain. (1) Benchmark scale: Curating expert-level, distribution-grounder tasks is labor-intensive; our 100-task set prioritizes quality and topic balance over scale. (2) Domain coverage: Despite multi-domain design, residual coverage bias may ex- ist; future versions will broaden reviewer pools and domains. (3) Human evaluation throughput: Expert judging costly, limiting sample sizes; we plan larger studies further tighten confidence intervals. (4) Tooling opacity: FACT depends each system's built-in search/browse stack and external fetchers; differences reachability and page variants can affect effective-citaiinn measure- ment. (5) Judge dependence: RACE relies a Judge LLM via proprietary APIs. In the paper and Appendix, we report results with open-source judges (c.g., Qwen3) and find they can serve as practical substitutes to certain extent, offering more open alternative.
B IN-HOUSE DATA DETAILS
At the time of constructing DeepResearch Bench, there open-source dataset that directly captured real user interactions with production DRAs. Consequently, to support the analysis real- istic topic distributions, chatbot. We then applied multi-stage post-processing pipeline, including anonymization, filtering extract deep-reseearhh style tasks, and topic categorization, approximate the domain distribution of real-world DRA tasks.
resorted in-house log of user interactions with search-augmented
The collected queries cover wide range of real information needs, for example: "analyze the time- series trend of used-car prices for specific brand in my city," and "investigate product's R&D team, development timeline, and estimate ARR." This diversity, together with careful identifica- tion of research tasks, leads us to believe that the resulting distribution is a close proxy to real user demand.
As the closest public reference, we also examined the recently released Search Arena 24K dataset. Using exactly the same filtering and categorization pipeline as the main paper, we derived deep-research domain distribution from Search Arena 24K and compared it against our in-house distribution (Table 4
As we can see, the two distributions share the same top-5 domains and are overall similar across major categories, which supports the reasonableness using our in-house data as a surrogate for real-world DRA usage.
Table 4: Major topic distribution comparison (%) Search Arena 24K dataset.
between in-house data and the open-source
Category
Sci & Tech
Fin Biz
Soft Dev
Edu & Jobs
Health
Literature
Our In-house Data Search Arena 24K
17.80 13.68
14.45
11.59 15.81
8.38 4.97
8.02 10.47
5.01
17.74
4.54
Category
History
Hardware
Industrial
Art & Design
Games
Our In-house Data
4.23 4.42
3.68
3.49 4.10
3.24 1.54
3.22
Search Arena 24K
3.19
2.57
C DIMENSION DEFINITIONS
The RACE framework evaluates research reports based on four top-level dimensions. Their defini- tions are provided in Table
16

---

Published as a conference paper at ICLR 2026
Table 5: Definitions of Core Evaluation Dimensions for Report Quality
Dimension
Description
Comprehensiveness (COMP)
Article covers key areas of the industry, ensures overall under- standing. and docs not omit important parts.
Insight/Depth (DEPTи)
Article deeply analyzes causes, impacts, and trends, providing valuable insights.
Instruction- Satpuntioon-olllwinnn/eeeeeaee (INST)
Article closely follows the research topic and directly answers questions.
Readability (READ)
Article has clear structure, fluent language, and is easy to un- derstand.
D JUDGE LLM SELECTION FOR THE FACT FRAMEWORK
The FACT framework employs a Judge LLM for crucial automated steps: Statement-URL Pair Extraction and Deduplication, followed by Support Judgment. The selection of this Judge LLM is pivotal, aiming to balance evaluation accuracy with operational costs, especially given the significant token consumption inherent in these processes. To determine an optimal model for these tasks, we specifically evaluated Gemánt ni -2 5-FF5 Its judgments were compared against human evalu- ations on a randomly sampled set of 100 statement-URE pairs derived from our benchmark tasks. This comparison demonstrated strong agreement with human annotators: Gemini-2 FFlash's judgment aligned with human 'support' determinations in 96% cases and with 'not support' de- terminations in 92% of cases.
We further find out that the accuracy of Cemini -2. 5-1.2..-.1asi in these FACT-specific evaluation steps is very close that of femini-2. 5-Pro. The operations within the FACT framework (such as extracting statements from full reports and analyzing webpage content for support) are known to be token-intensive, making cost-effectiveness critical consideration. Since femini-2.. 5-Flass demonstrated comparable accuracy to Semini-22 5--Pro 5-Pro for these specific tasks but at a more advantageous cost, we select Gemini- 5-Flash as the Judge LLM for the FACT framework. This choice enables us to maintain high evaluation reliability while managing operational costs effectively.
DATA COLLECTION TIMEFRAMES
The data for the commercial models evaluated in this paper were collected during specific time- frames in 2025, as detailed in Table 6 These dates indicate when the model outputs used in our experiments were generated.
Table 6: Data Collection Timeframes for Evaluated Models (2025)
Model Category Provider Group
Data Collection Date Range
Deep Research Agents (DRAs)
OpenAl Deep Research
April 1
May
Gemini 2.5 Pro Deep Research
April 27 April 29
Perplexity Deep Research
April - April 29
Grok Deeper Search
April 27 April 29
Claude Research
June 23
June 25
Doubao Deep Research
June 29
July
Kimi Researcher
June 29
-July
LangChain Open Deep Research
June 29
-July
LLM with Search Tools (Grouped by Provider)
Claude Models (w/Search)
May 12
May 13
Perplexity Models
May
-May 12
GPT Models (w/Search)
May 11
May 12
Gemini Models (Grounding/w/Search))
May 12
May 13
17

---

Published as a conference paper at ICLR 2026
F OPEN DEEP RESEARCH REPRODUCTION DETAILS
We evaluate the open-source LangChain Open Deep Research (ODR) implementation using its de- fault settings. Specifically, the research model is set to GPT-4.1 and the summarization model is set to 1PTT-4.1mint following the project's recommended defaults. Unless otherwise stated, no additional modifications applied to the configuration.
G EXPERIMENT DETAIL
G.1 HUMAN EVALUATION EFFORT
The human evaluation process required considerable time investment to ensure thorough assessment. On average, each expert annotator spent approximately 1.5 hours per query to evaluate the reports from the four agents. This meticulous process resulted in total of 225 person-hours of human evaluation across all tasks and annotators. This substantial effort provided a robust and reliable dataset of human judgments, forming the basis for our human consistency analysis.
G.2 CONFIGURATION OF LLMS WITH WEB SEARCH TOOLS
To ensure a standardized and comparable evaluation environment for Large Language Models (LLMs) equipped with built-in web search tools, the following configurations were uniformly ap- plied:
Thinking Budget/Compuuaaiionl Resources: For models that support configurable "thinking budget" or a similar computational resource limit for generation, this was uni- formly set to a high value, equivalent to 16,000 tokens where applicable. This allowed models ample processing capacity for complex queries.
Search Context Size: In cases where models offered a parameter to control the amount of information retrieved and utilized from web searches (e.g.. the search context s i ze option as found in the Perplexity AI API), this was consistently set to "high". This configuration aimed to maximize the contextual information available to the L.LM from its search actívities.
Maximum Search Iterations: The maximum number of web search queries, "search turns," " permitted during the generation process was standardized to five for all LLMS that provided such configurable limit. This ensured comparable depth of web exploration across these models.
To accommodate potentially comprehensive responses while maintaining consistency, maximum output token limit for all LLMSs was set 36,000 tokens. If model's inherent maximum output capacity was less than this 36,000-token threshold, its specific native maximum limit was adhered to.
Output Length:
Citation Formatting and Standardization: A critical aspect of our methodology was the standardization of citation presentation facilitate consistent downstream evaluation, particularly when employing frameworks like FACT for factual assessment. Citations as provided by each LLM were parsed accordance with their respective official API docu- mentation. Subsequently, the generated reports were systematically restructured: citation markers were inserted in the format '[1][2]* at the end of the relevant sentences, and consolidated "References" list, compiling all unique cited sources, was appended to the conclusion of each article. This uniform approach to citation structure was essential for equitable and rigorous factual verification.
These standardized settings implemented to minimize variability arising from differing default configurations and to enable a more direct comparison of the models' capabilities the context of deep research tasks.
18

---

Published as a conference paper at ICLR 2026
H DETAILED CALCULATION OF HUMAN CONSISTENCY METRICS
This appendix provides the detailed calculation methods for the four metrics used to validate the consistency between our RACE framework and human judgment, as introduced in Section 4.3.2
H.1 PAIRWISE AGREEMENT RATE
The Pairwise Agreement Rate measures the proportion of report pairs (across all tasks) where the evaluation method's preference matches the human preference.
For each of the N, = 50 tasks in our study, four generated deep research reports result in Np ()= = 6 unique pairs per task. Human preference for each pair (e.g., Report A is better than Report B, or they are tied) established from the average overall scores assigned to each report by three domain experts.
Let p) be an indicator function for task and pair p:
{
if the method's preference matches the human preference for pair p of task otherwise.
[(t, p)
(5)
The Pairwise Agreement Rate is then calculated as:
EE Ep I(t.p)
No N₂ Np
Pairwise Agreement Rate
(6)
This metric reflects the evaluation method's reliability in replicating human comparative judgments.
H.2 OVERALL PEARSON CORRELATION
This metric quantifies the linear relationship between average model scores from the evaluation method and those from human experts, aggregated across all N, = 50 tasks.
Let X be vector of average scores per model (e.g., for the different DRAs evaluated) obtained from our method, aggregated across tasks. Let Y be the corresponding vector average scores per model obtained from human experts, also aggregated across all tasks. The Overall Pearson Correlation is the standard Pearson correlation coefficient r(X, calculated between these two vectors. This reflects the overall score correlation between the method and human experts for the evaluated DRAs.
H.3 FILTERED AVERAGE PRARSON CORRELATION
This metric calculates the average of per-task Pearson correlations (r:) between the method's scores and mean human scores, specifically on tasks where human judgment is more consistent.
Given that human scores from limited number of experts (k 3) for the expert inconsistencies can affect task-level metric stability. To mitigate this, tasks are filtered based on inter-rater reliability using the Intraclass Correlation Coefficient (ICC). For each task, ICC(1,1) (a one-way random effects model) is computed from the l: 3 human experts scores for the reports:
reports per task,
MSB - MSW
ICC 1) = MSB + (k 1)MSW'
where MSB is the mean square between reports and MSW is the mean square within reports. Tasks with poor inter-rater reliability (e.g., 1C)( < 0) are excluded. This yields a filtered subset of Nittered tasks, denoted as Tritered (37 our experiments).
The Filtered Average Pearson correlation is then the average of per-task Pearson correlations (r,) between the method's scores and mean human scores over Tritened:
Filtered Avg Pearson
Tt-
(8)
Nittered
This procedure provides more robust assessment of absolute-score correlation.
19

---

Published as a conference paper at ICLR 2026
H.4 FILTERED AVERAGE SPEARMAN CORRELATION
Using the same filtering method and the subset Tittered this metric evaluates model ranking consis- tency.
For each task 1. E Tiltered: the Spearman rank correlation coefficient is calculated between model rankings derived from our evaluation method and those from average human scores. The Filtered Average Spearman Correlation is then the average of these Pt values:
Filtered Avg Spearman
Σ Pt.
fittred
This reflects how well the method preserves relative model ordering compared to humans, specifi- cally on tasks with more consistent human judgments.
ANALYSIS OF RACE ROBUSTNESS
1.1 ROBUSTNESS REFERENCE SELECTION
To assess sensitivity to the choice of reference article, replace the default Gemini-2.5-PPr Deep Research reference with reports from Resedeeh and Limi-Rescarcher across the full task set. The ranking remains identical (Gemini OpenAl > Perplexity Grok), indicating that RACE is robust to reference selection.
Table 7: RACE results with
as reference.
Overall
Comp.
Insight
Inst.
Read.
Gemini-2.5-Prt Deep Research
55,30 (Ist)
55.56
56.22
52.89
56.55
OpenAl Deep Research
52.27 (2nd)
52.70
50.73
$2.80
54.41
Perplexity Deep Research
44.11 (3rd)
43.16
39.71
48.47
47.80
Grok Deeper Search
42.53 (4th)
41.29
35.96
48.89
47.29
Table 8: RACE results with Kimi- Simi-Reseaarchr as reference.
Overall
Comp.
Insight
Inst.
Read.
Gemini-2.5-Pro Deep Research
55.22 (Ist)
55.27
57.08
52.71
55.04
OpenAl Deep Research
52.38 (2nd)
52.48
52.21
52.49
52.91
Perplexity Deep Research Grok Deeper Search
44.24 (3rd)
43.28
40.49
48.53
46.66
41.15 (4th)
39.31
34.64
48.13
45.72
I.2 ROBUSTNESS TO LENGTII INFLATION
To evaluate robustness to length-based bias, start from Gemini-2.5-Pro Deep Research articles and segment them into paragraphs. For each paragraph, we prompt Gemini 2. 5 Pro to rewrite by expanding length while preserving the original information and maintaining logical coherence with the surrounding context. We perform this process iteratively, yielding average article lengths of approximately x x1.47 (expand) and x2.19 (expand?) the original. Subsequent RACE evaluations show that scores do not increase with length; under higher expansion they decline, indicating that RACE is resilient to simple length-inflatio attacks.
Table 9: Length bias analysis under controlled expansions.
Target
Avg. Length
Overall
Comp.
Insight
Inst.
Read.
Gemini-Dese Denniii-DeepRescarchh
33.4k
48.92
48.45
48,30
49,29
49.77
(expand)
49.0k (x 1.47)
48.68
48.78
48.49
49.12
47.57
Gemini-Ded Reeiin-DDeeResearchh (expand?)
73.2k (x2.19)
47.07
48.32
47.19
48.30
40.49
20

---

Published as a conference paper at ICLR 2026
1.3 OPEN-SOURCE JUDGE LLM
We additionally evaluated an open-source Judge LLM, Qwen3 101-33-23BB-2220 -eaa--Tiiiiin---- -5077, substitute for proprietary models in the RACE framework. The Judge LLM performed the same steps as in our main setup: dynamic dimension weighting, task-specific criteria generation, and reference-based scoring. Hyperparameters and prompts were kept aligned with the Gemini-based setup where applicable.
Using Qwen3 the Judge, the relative ranking across the four DRAs remains identical to the Gemini-based setup, demonstrating that RACE is robust to the choice of Judge backbone. Com- pared with closed-source judges, Qwen3 yields lower commannnnnsstteeyy metries on the correlation scale; however, the gap is moderate, and Qwen3 remains practical open-source alternative when access or cost constraints arise.
Table 10: Overall RACE scores of four DRAs under different Judge LLMs.
Judge
Gemini-2. P.5-Pro DR
OpenAl DR
Perplexity DR
Grok DS
Gemini-2.5- 2.5-pro
48.88
46.98
42.25 41.25
40.24
Qwen3-2333 (omn--23bb-thinknn
50.17
46.88
38.04
J ANALYSIS OF RACE GENERALIZABILITY
RewardBench evaluate reward/judge models in generative settings with accuracy-based, multi- skill preference judgments across Factuality, Precise Instruciion-Folloiingg Math, Safety, Focus, and handling Ties (Lambert et al. 2024 Malik et al. 2025).
We apply RACE general reward- modeling method by using DeepSeek-R1 the judge and replacing vanilla direct scoring with RACE's s pipeline: (1) dynamically generate evaluation dimen- sions weights, (2) generate task-specific, executable criteria, and (3) perform reference-based com- parative scoring. Other settings follow the RewardBench protocol. With RACE, DeepSeek- shows substantial gains and approaches leading proprietary judges, indicating that RACE strength- ens reward modeling and transfers beyond deep research, consistent with our rebuttal analysis.
Table 11: RewardBench-v2 (Generative) results with and without RACE.
Model
Score
Factuality
Precise IF
Math
Safety
Focus
Ties
LMunit (SOTA)
82.1
87.2
90.1
Claude- Claude-Opus-4
76.5
82.7
41.9
74.9
89.5
86.2
83.7
Gemini-2.5-pra
74,8
71.2
52.1
68.3
88.7
79.4
81.2
DeepSeck-RR RI w/o RACE
51.5
44.4
19.9
46.2
70.1
55.5
47.7
DeepSeck-R1 w/ RACE
74.4
72.9
45.6
74.3
90.9
76.8
47.6
K STATISTICAL ANALYSIS OF PERFORMANCE DIFFERENCES
To provide rigorous evidence for the robustness of evaluation results, conducted comprehen- sive statistical analyses examining both the significance of performance differences between agents and the sensitivity of rankings task selection.
K. PAIRWISE STATISTICAL SIGNIFICANCE
We performed paired f-tests between all top- oo--erformmng DRAs on their common task set. Since one agent (Doubao DR) did not respond to one task, we report results based on 99 tasks common to all agents. Table 12 presents the results.
Key findings:
Top-tier agents show clear separation: Gemini-2.5-Pro DR significantly outperforms all others (p 0.001, Cohen's d ranging from 0.65 to 1.19, medium to large effects).
21

---

Published as a conference paper at ICLR 2026
OpenAI DR forms a distinct second tier: It significantly outperforms Claude, Kimi, and Doubao (p I 0.006, Cohen's 's d from 0.28 to 0.44), though the effect sizes are smaller than Gemini's s advantage.
Mid-tier agents statistically indistinguishable: Claude Research, Kimi Researcher, and Doubao DR show no significant pairwise differences (p > 0.18, Cohen's d < 0.15), indicating they perform comparable level.
Table 12: Pairwise Statistical Significance Tests (Top-5 DRAs). * * * < 0.001; ** < 0.01; p 0.05; n.s. not significant. A: mean difference in percentage points. CI: 95% confidence interval for the difference. Paired t-tests based on common tasks.
Comparison
A (%)
95% CI
1-stat
p-value
Significance
Gemini-2.5-Prrt
OpenAI DR
100
3.26
[2.27, 4.25]
6.52
<0.001
Gemini-2.5-Prrt DR vs Claude Research
100
4.72
[3.82, 5.62]
10.42
< <0.001
Gemini-2.5-Prru DR Kimi Researcher
100
5.07
[4.09, 6.06]
10.24
< <0.001
Gemini-2.5.4 P.o DR Doubao DR
99
5.36
[4.46, 6.26]
11.81
<00001
OpenAl DR Claude Research
100
1.46
[0.43, 2.48]
2.81
0.006
OpenAl DR Kimi Researcher
100
1.81
[0.91, 2.711
4,00
0.001
OpenAl Doubao
99
2.13
[1.17, 3.09]
4.39
0.001
Claude Research vs Kimi Researcher
100
0.35
[-0,54, 1,251
0.79
0.432
D.S.
Claude Research vs Doubao DR Kimi Researcher vs Doubao DR
99
0.66
[-0.33, 1.64]
1.32
0.188
n.s.
99
0.28
[ 0.69. 1,26]
0.58
0.564
11S.
These results confirm that the observed performance gaps in Table are statistically robust, not ar- tifacts of random variation. The presence of both significant and non-significant differences demon- strates that our evaluation can discriminate meaningful performance gaps while correctly identifying agents of similar capability.
K.2 TASK SAMPLING SENSITIVITY
To assess whether rankings depend on the specific 99-100 tasks selected, we performed leave-one- out (LOO) analysis: for each task, we removed and recomputed rankings on the remaining tasks.
Table 13 shows that all agents maintained their ranks in 100% of LOO iterations. This perfect stability indicates that rankings are not driven by any single critical task, performance differences are consistent across task subsets, and the benchmark size (99-100 tasks) is sufficient to produce stable rankings.
Table 13: Leave-One-Out Ranking Stability Analysis. Based on 99 tasks common to all evaluated agents. Consistency: percentage of LOO iterations where the agent maintained its rank.
Position
Agent
Consistency (%)
Seminii-25--Pr Deep Research
100.0
2
OpenAl Deep Research
100.0
3
Claude Research
100.0
4
Kimi Researcher
100.0
Doubao Deep Research
100.0
K.3
DISCUSSION
The statistical analyses presented above establish the robustness of our evaluation framework from two complementary perspectives:
Performance differentiation: Pairwise comparisons reveal a clear performance hierarchy with statistically significant gaps between top performers and others. Crucially, we also observe significant differences among mid-tier agents, demonstrating that our framework appropriately iden- tifies both meaningful differences and statistical ties.
22

---

Published as a conference paper at ICLR 2026
Ranking stability: Perfect LOO stability shows that rankings are highly robust to task selection. The 99-100 task scale provides sufficient statistical power to produce reliable, generalizable results.
Together, these analyses validate that DeepResearch Bench produces statistically sound and stable evaluations, suitable for drawing meaningful conclusions about DRA capabilities.
23

---

Published as a conference paper at ICLR 2026
L ADDITIONAL DEEP RESEARCH TASK EXAMPLES
"Investigate how, under chronic antigen stimulation (e.g., the tumor micro-environment or latent HIV infection), mitochondrial dynamics (fusion-fission balance) in CD&+ T cells drive bifurcation into terminally exhausted and tissue-resident memory (Trm) fates via epi- genetic reprogramming (e.g.. m6A modification, pectate-mediate histone lactylation). De- velop quantitative models based on metabolic metabolic-epigennttie interaction networks."
"Analyze liability allocation in accidents involving vehicles with advanced driver- assistance systems (ADAS) operating in shared human-machine driving context. In- tegrate technical principles of ADAS, existing legal frameworks, and relevant case law to systematically examine the boundaries of responsibility between the driver and the system."
"How conduct comprehensive and accurate situational awareness of space targets in cislunar space, and support the effectiveness of short-term cislunar tracking and mon- itoring tasks? Compare existing sensing architectures, data-fusion pipelines, and control strategies."
24

---

Published as a conference paper at ICLR 2026
M
PROMPT TEMPLATES
Clean Article Prompt
<system_role>
You are a professional article editor who good at cleaning and refining article content.
</system_role role>
<user [esser_ppopp>>>>
Please help clean the following research article, removing all citation links, citation marks (such as [1]. [2], 1, 2, etc. or other complex citation formats), reference lists, foot- notes, and ensuring the content is coherent and smooth. Keep all other original content of the article, removing only the citations. If the content of the citation mark is used as part of sentence in the article, keep the text content and remove other marks.
Article content: "(article)"
Please return the cleaned article in full, without adding any additional comments or expla- nations.
</user_Pan ppospr>rompt>>>>
Generate Dynamic Dimension Weight Prompt
<system_ro _oree>_rr>>___
You are experienced research article evaluation expert. You excel at deeply understanding objectives, challenges, and core value points of specific research tasks, and based on this, setting dynamic, reasonable, and well-supported dimension weights for subsequent article quality assessment.
</system_ role>
<user_s perer___rrppt>t
There is deep research task follows:
<task>
(ttask_rroppt]
</task>
<instruction>
Background: The research team will conduct in-depth and comprehensive research based on the '<task>' above and ultimately produce high-quality research article. Your Task: As evaluation expert, you need to set the evaluation criteria weights for this specific '<task>' for our assessment team. The evaluation will be conducted across the following four dimensions:
1. Comprehensiveness: The breadth, depth, and relevance of information coverage.
2. Insight: The depth, originality, logic, and value of the analysis and conclusions.
3. Instruction Following: Whether the report accurately and completely responds to all requirements and constraints of the task.
4. Readability: Clarity of structure, fluency of language, effectiveness of data pre- sentation, and overall ease of understanding.
Evaluation Formula: Total Score = Comprehensiveness! Comprehensiveness Weight + In- sight Insight Weight + Instruction Following * Instruction Following Weight + Readability RRedabilitt Weight.
(Note: The sum of all weights must be exactly 1.0)
Core Requirements:
1. In-depth Task Analysis: Carefully study the specific content of the '<task>', its implicit goals, potential difficulties, and the core value of its outcomes.
2. Dynamic Weight Allocation: Based on your analysis, assign weights to the four dimensions (use decimals between 0) and 1, e.g. 0.3).
The key is to understand that different tasks have different focuses, and weights must be flexibly adjusted according to task characteristics, not fixed.
25

---

Published as a conference paper at ICLR 2026
3. Justify Allocation Reasons: Your analysis (<<analysis>) must clearly and specifically explain why each dimension is given a particular weight, and di- rectly link the reasons the requirements and characteristics of the <task>.
This crucial for evaluating the quality of your work.
4. Standard Format Output: Strietly follow the format of the example below, first outputing the "<analysis>" text with detailed reasons, and then immediately providing the
with the weight allocation results.
</instructions
<examples rationale>
The following two examples are provided to demonstrate how adjust evaluation dimen- sion weights and explain the reasons based on changes in task nature. Please focus on learning the thinking logic and analytical methods in these examples, rather than simply imitating their content weight values.
[eexamples_aa ionale>
<example_1>
<task>
"Analyze the feasibility of investing in electric vehicle (EV) charging infrastructure in sub- urban areas."
</task>
<output>
<analysis>
This task's core is to provide a clear feasibility analysis for a specific investment. The value lies in the thoroughness of the assessment and the practicality of its conclusions. Therefore, evaluation emphasizes insight and comprehensiveness.
Insight (0.35): The task requires deep analysis of feasibility. The quality of the strategic recommendations derived from this analysis key.
Comprehensiveness (0.30): A thorough investigation of all relevant factors (tech- nical, economic, social, environmental) crucial for a reliable feasibility study.
Instruction Following (0.20): The report must specifically address EV charging infrastructure in suburban areas and focus investment feasibility.
Readability (0.15): Clearly communicating complex financial and technical anal- ysis is important, but secondary to the depth and breadth of the study.
</analysis>
<json_output>
[( ""comprchensiveness"" 0.30, "insight": 0.35, "instruction "instuuctionfoollownngg" 0.20, "readability":
0.15 }}
frison_outputs
</output>
</example_1>
Please strictly follow the above instructions and methods. Now, begin your work on the
following specific task:
<task>
"(task painsp""rmmp)"
</task>
Please output your '<ana '<analysiss yysis* and '<json_outputs </user_prompt>
26

---

Published as a conference paper at ICLR 2026
Generate Comprehensiveness Criteria Prompt
<system role>
You are an experienced research article evaluation expert. You excel at breaking down ab- stract evaluation dimensions (like "Comprehensiveness" into actionable, clear, and task- specific criteria, assigning appropriate weights and justifications for each. //system_role
<user_prompe
Background: We are evaluating deep research article written for the following task across Comprehensiveness, Insight, Instruction Following, and Readability. 1. Comprehensiveness: The breadth, depth, and relevance of information coverage.
four dimensions:
2. Insight: The depth, originality, logic, and value of the analysis and conclusions.
3. Instruction Following:
Whether the report accurately and completely responds to all requirements and constraints of the task.
4. Readability: Clarity of structure, fluency of language, effectiveness of data pre- sentation, and overall ease of understanding.
<task> "(task_prompt)"
</task>
<instruction>
Your Goal: For the Comprehensiveness dimension of this research article, develop a set of detailed, specific, and highly task-relevant evaluation criteria. You need to:
1. Analyze Task: Deeply analyze the "<task>' to identify key information areas, perspectives, and depths that must be covered to achieve "comprchensiveness"
2. Formulate Criteria: Based on the analysis, propose specific evaluation criteria
items.
3. Explain Rationale: Provide brief explanation ('explanation") for each criterion, stating why it is important for assessing the comprehensiveness of this '<tas>>'
4.
Core Requirements:
Task-Centric: Analysis, criteria, explanations, and weights must directly relate to the core requirements and characteristics of the '<task>'
2. Well-Justified: The '<analysis> section must clearly articulate the overall thinking behind setting these criteria and weights, linking it to the '<task>' The 'explanation' for each criterion must justify its specific relevance.
3.
</instruction>
The following example demonstrates how to formulate comprehensiveness criteria based on task requirements. Focus on learning the thinking logic and analytic analytical methods from this example, not just imitating its content or weight values.
e/example_raae ieteerrtiinaall
<example_1>
<task>
"Analyze the impact of remote work trends on commercial real estate in major US cities and recommend investment strategies."
</task>
<output>
<analysis>
To comprehensively evaluate a research article on "the impact of remote work on commercial real estate in major US cities and recommended investment strategies," considerations must span multiple dimensions.
Specifically, evaluation criteria need cover:
27

---

Published as a conference paper at ICLR 2026
1. Remote Work Trends & Adoption Data: Coverage of current and projected re- mote/hybrid work models, adoption rates across industries and demographics.
2. Impact on Commercial Real Estate Sectors: Analysis effects on office, retail, and industrial spaces, including vacancy rates, leasing trends, and property valua- tions major US cities.
3. Geographical Variations: Examination of how impacts differ across various major US cities (c.g., tech hubs vs. financial centers, downtown suburban).
4.
Weight allocation should be balanced between the impact analysis... </ </analysis> <json_output>
[[
"criterion": "Analysis of Remote Work Trends and Adoption", "explanation": "Assesses if the article thoroughly examines current and projected remote/hy- brid work models...".
"weight": 0.15
H. {(
"criterion": "Comprehensive Coverage CRE Sector Impacts",
"explanation": "...".
"weight": 0.20
l1. [{
"criterion": "Examination of Geographical Variations and Nuances", "explanation":"...
"weight": 0.15
}}.
[[
"criterion": "Discussion of Broader Economic and Social Consequences",
"explanation": "...". "...".
"weight": 0.10
}}
]
//json_output>
</output>
</example_i>
Please strictly follow the above instructions and methods. Now, begin your work on the following specific task:
<task>
"(task "rmnsp]_rmpp"
</task>
Please output your '<anal lysiayssss' and '<json_outputx> </user_prompt>
28

---

Published as a conference paper at ICLR 2026
Score Prompt In RACE(Full)
<system role>
You are strict, meticulous, and objective research article evaluation expert. You excel at using specific assessment criteria deeply compare two articles on the same task, providing precise scores and clear justifications.
+/system_role
<user_promps
Task Background
There is deep research task, and you need to evaluate two research articles written for this task. We will assess the articles across four dimensions: Comprchensiveness, Insight, Instruction Following, and Readability. The content is as follows:
<task>
</task>
Articles to Evaluate
<article_i>
"(article 11"
</article_1>
<article_2>
"(article_2)"
</article_2
Evaluation Criteria Now, you need to evaluate and compare these two articles based the following evaluation criteria list, providing comparative analysis and scoring each on scale of 0-10. Each criterion includes explanation, please understand carefully. scriteria_list (criteria list)
</criteria_lice list>
<Instruction>
Your Task
Please strictly evaluate and compare '<article_ 1>' and '<article_22 based on each criterion in the '<criters perrieerralist>'' l
You need to: Analyze Each Criterion: Consider how each article fulfills the requirements of each criterion.
2. Comparative Evaluation: Analyze how the two articles perform on each criterion, referencing the content and criterion explanation.
3. Score Separately: Based your comparative analysis, score each article on each criterion (0-10 points).
Scoring Rules
For each criterion, score both articles on scale 0-10 (continuous values). The score should reflect the quality of performance that criterion:
Very poor performance. Almost completely fails to meet the criterion requirements.
0-2 points:
Poor performance. Minimally meets the criterion requirements with significant deficiencies.
2-4 points:
4-6 points: Average performance. Basically meets the criterion requirements, nei- ther good nor bad.
Good performance. Largely meets the criterion requirements with no- table strengths.
6-8 points:
8-10 points: rion requirements.
Excellent/outstaaddng performance. Fully meets or exceeds the crite-
Output Format Requirements
Please strictly follow the ><output_ format> below for each criterion evaluation. Do not include any other unrelated content, introduction, or summary. Start with "Standard I" and proceed sequentially through all criteria:
29

---

Published as a conference paper at ICLR 2026
</Instruction> <outputtformatz>
{{
"comprehensiveness... [
"criterion": [Text content of the first comprehensiveness evaluation criterion],
"analysis": [Comparative analysis],
"article_ 1 score": [Continuous score 0- 10].
"article score" [Continuous score D-]
}}.
{{
"criterion": [Text content of the second comprchensiveness evaluation criterion], "analysis": [Comparative analysis].
"article :_score:: [Continuous score 0- 0-10]. "article 2 score": [Continuous score 0-10]
"insight":
{{
"criterion": [Text content of the first insight evaluation criterion), "analysis": [Comparative analysis].
"article_ L_se______srr__.__ [Continuous score 0-10],
"article 2coree: [Continuous score 0-10]
11.
e/output_format
Now, please evaluate the articles based on the research task and criteria, providing de- tailed comparative analysis and scores according to the requirements above. Ensure your output follows the specified "<output_ format>' and that the JSON format is parsable, with all characters that might cause JSON parsing errors properly escaped. </user_prompt>
30

---

Published as a conference paper at ICLR 2026
Static Score Prompt
<system role>
You are a strict, meticulous, and objective research article evaluation expert. You excel using specific assessment criteria deeply compare two articles on the same task, providing precise scores and clear justifications.
</system_roke mole>>>>e>eeeee
<user_prompt>
Task Background
There is deep research task, and you need evaluate two research articles written for this task.
We will assess the articles across four dimensions: Comprchensiveness, Insight, Instruction Following, and Readability.
The content is as follows:
<task>
"(task_prompt)"
</task>
Articles to Evaluate
<article_1>
"(article_ 1)" </article_1>
<article_2>
"(article_2) _
</article_
Evaluation Criteria
Now, you need to evaluate and compare these two articles based on the following fixed evaluation criteria list, providing comparative analysis and scoring each scale of 0-10. Each criterion includes explanation, please understand carefully.
<criteria_ list>
# Comprehensiveness
[[[ "criterion": "Information Coverage Breadth", "explanation": "Evaluates whether the article covers all key areas and aspects related to the topic without omitting important information.", "weight": 0.25
H.
[I
"criterion": "Information Depth and Detail",
"explanation": "..."
"weight": 0.25
}).
"crittrrion"": "Data and Factual Support",
"explanation": "....
"weight": 0.25
[(
"criterion": "Multiple Perspectives and Balance", "explanation".".
"weight": 0.25
}]]
Insight
[[[ ("criterron": "Analysis Depth and Originality". "explanation":
"weight": 0.25
ll.
</criteria_list>
<Instruction>
31

---

Published as a conference paper at ICLR 2026
Your Task
Please strictly evaluate and compare '<article_ 1>" and '<article__ 2> based on each criterion the "<criteria list>'. You need to:
1. Analyze Each Criterion: Consider how each article fulfills the requirements of each criterion.
2. Comparative Evaluation: Analyze how the two articles perform each criterion, referencing the content and criterion explanation.
3. Score Separately: Based on your comparative analysis, score each article on each criterion (0-10 points).
Scoring Rules
For each criterion, score both articles on a scale of 0-10 (continuous values). The score should reflect the quality of performance on that criterion:
0-2 points: Very poor performance. Almost completely fails to meet the criterion requirements.
8-10 points: rion requirements.
Excellent/outstanding performance. Fully meets exceeds the crite-
Output Format Requirements
Please strictly follow the
forputttforttt>>/o below for each criterion evaluation.
Do include any other unrelated content, introduction, or summary.
Start with "Standard 1" and proceed sequentially through all criteria: </Instruction
koutput_fooratt___
{{
"comprchensiveness::
{(
"criterion": [Text content of the first comprehensiveness evaluation criterion],
"analysis": [Comparative analysis].
"article I score": [Continuous score 0-10],
"article score": [Continuous score 0-10]
l1.
[(
}1.
"i"nssight": [
{(.
"criterion": [Text content of the first insight evaluation criterion],
"analysis": [Comparative analysis].
"article :_score:: [Continuous score 0-10],
"article :core": [Continuous score 0-10]
l1.
1)
</output_format>
Now, please evaluate the two articles based on the research task and criteria, providing de- tailed comparative analysis and scores according to the requirements above. Ensure your output follows the specified "cuupput__ ftrppput__ooatt>>_aa and that the JSON format is parsable, with all characters that might cause JSON parsing properly escaped. </ </user_prompt:
32

---

Published as a conference paper at ICLR 2026
Point-wise Score Prompt
<system role>
You are a strict, meticulous, and objective research article evaluation expert. You excel using specific assessment criteria to thoroughly evaluate research articles, pro- viding precise scores and clear justifications. </system_roo eole
<user_prompt>
Task Background
There is deep research task. and you need to evaluate research article written for this task. We will assess the article across four dimensions: Comprehensiveness, Insight, Instruction Following, and Readability.
The content as follows:
<task>
"(task_prompt)
</task>
Article to Evaluate
starget_articc
"(article))
</target_article>
Evaluation Criteria
Now, you need to evaluate this article based on the following evaluation criteria list, pro- viding analysis and scoring each scale of 0-10. Each criterion includes an explanation, please understand carefully.
<criteria_ ccriteria_list
</criteria_list
<Instructio>>
Your Task
Please strictly evaluate 'starget_artclee" based on each criterion in the Scriterrialist..
You need to:
1. Analyze Each Criterion: Consider how the article fulfills the requirements of each criterion.
2. Analysis and Evaluation: Analyze the article's performance each criterion, referencing the content and criterion explanation, noting strengths and weaknesses.
3. Score: Based on your analysis, score the article on each criterion (0-10 points).
Scoring Rules
For each criterion, score the article on scale of 0-10 (continuous values). The score should reflect the quality of performance on that criterion:
Very poor performance. Almost completely fails to the criterion requirements.
0-2 points:
Poor performance. Minimally meets the criterion requirements with significant deficiencies.
2-4 points:
4-6 points: Average performance. Basically meets the criterion requirements, nei- ther good nor bad.
Good performance. Largely meets the criterion requirements with no- table strengths.
6-8 points:
8-10 points: rion requirements.
Excellen//outstanding performance. Fully meets or exceeds the crite-
Output Format Requirements
Please strictly follow the <ootppttformat>>> format>' below for each criterion evaluation. Do not include any other unrelated content, introduction, summary.
Start with "Standard 1" and proceed sequentially through all criteria:
33

---

Published as a conference paper at ICLR 2026
</Instruction> <outputtformat>
{{ "comprehensiveness... [
"criterion": [Text content of the first comprehensiveness evaluationn criterion),
"analysis": [Analysis].
"target_ score": sargeet [Continuous score 0-10] }).
[(
"criterion": [Text content of the second comprchensiveness evaluation criterion], "analysis": [Analysis].
score:" [Continuous score 0-10)
11.
"insight":
{f
"criterion": [Text content of the first insight evaluation criterion], "analysis": [Analysis].
farget_scoree, [Continuous score 0-10]
l1. ...
J.
}}
V/output_format:
Now, please evaluate the article based on the research task and criteria, providing detailed analysis and scores according the requirements above. Ensure your output follows the specified "<output" format>> and that the JSON format parsable, with all characters that might cause JSON parsing errors properly escaped.
</ //user_prompp>
34

---

Published as a conference paper at ICLR 2026
Vanilla Prompt
<system role> You are a strict, meticulous, and objective research article evaluation expert. You excel using specific assessment criteria to thoroughly evaluate research articles, pro- viding precise scores and clear justifications. </system_roo eoreee>
<user_prompt>
Task Background
There is deep research task. and you need to evaluate research article written for this task.
<task>
"(task_ prompt)"
</task>
Article to Evaluate
<target_article
"(article)"
</target_artaieeec _arrgeetarttcle>>>>
<Instruction>
Your Task
Please evaluate the overall quality of the above "starget_article>> as a response to '<tas>>.
Please provide an overall score between and 10. Also, provide brief justification for your score.
Output Format Requirements
Please strictly follow the 'coutput format>' below for your evaluation result.
Do not include any other unrelated content, introduction, or summary. </Instruction <output format>
[[ "["overall_soone score": [Continuous score 0-10], "justification": "[Scoring justification]" } s/output_formee format>
Now, please evaluate the article based on the task and provide your score and justification according to the specified format.
Ensure your output is valid JSON format and escape any special characters as needed. </user_prompt>
35


Reviews:
[{"review_id": "1iFo3Egkwo", "reviewer": "Reviewer_Yinh", "summary": "The paper introduces **DeepResearch Bench**, a new benchmark consisting of 100 carefully curated, PhD‑level research tasks spanning 22 domains. It proposes two automated evaluation frameworks:  \n\n* **RACE** – a reference‑based, dynamically weighted “LLM‑as‑judge” protocol that generates task‑specific criteria and dimension weights (Comprehensiveness, Insight, Instruction‑following, Readability) and scores reports relative to a high‑quality reference.  \n* **FACT** – an automated pipeline that extracts statement‑URL pairs from agent reports, deduplicates them, and judges factual support using a separate judge LLM, yielding Citation Accuracy and Average Effective Citations per task.  \n\nThe authors validate RACE and FACT by (i) extensive human consistency experiments (pairwise agreement, Pearson/Spearman correlations), (ii) robustness analyses (reference selection, length inflation, judge model choice), and (iii) comparisons across a wide range of commercial Deep Research Agents (DRAs) and LLMs with search tools. All data, prompts, and code are released. The paper makes a solid engineering contribution by releasing a well‑designed benchmark and novel, human‑aligned evaluation frameworks for Deep Research Agents. The methodology is sound, the experiments are extensive, and the analyses address many potential pitfalls. However, the reliance on proprietary judges, limited benchmark scale, and lack of statistical significance reporting temper the impact. Overall, the work is a valuable resource for the community and merits inclusion in the conference, albeit not as a flagship oral contribution.", "questions": "1. **Task Sampling Variability** – How sensitive are RACE scores to the specific set of 100 tasks? Have you performed a leave‑one‑out or bootstrapped analysis to estimate variance across different task subsets?  \n2. **Open‑Source Judge Calibration** – When using Qwen3 as the judge, did you fine‑tune prompts or temperature settings to match the proprietary judge’s behavior? Could the performance gap be reduced with prompt engineering?  \n3. **Citation Format Normalization** – Could you elaborate on the preprocessing pipeline that normalizes citations across agents? Is there a risk that aggressive cleaning removes legitimate citation cues, thereby affecting FACT scores?  \n4. **Statistical Significance** – Did you compute confidence intervals for the pairwise agent comparisons (e.g., via bootstrap of human scores)? If not, could you add these to substantiate claims of superiority?  \n5. **Ethical Review of Web Scraping** – FACT retrieves full web pages via Jina Reader. Were any of the source pages subject to robots.txt restrictions or copyright considerations, and how were these handled?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. **Clear Gap Identification** – The paper convincingly argues that existing benchmarks either target isolated capabilities (browsing, code) or single‑domain research, leaving a need for a comprehensive DRA evaluation suite.  \n2. **Benchmark Construction Pipeline** – Leveraging a large in‑house query log (≈96 k queries) and a taxonomy‑driven filtering/classification pipeline yields a realistic distribution of tasks. Expert‑authored tasks and a bilingual (50 EN/50 ZH) set improve ecological validity.  \n3. **Innovative Evaluation Design** – RACE’s dynamic weighting and reference‑based scoring address known pitfalls of static rubrics and absolute LLM scoring (e.g., length bias, generic high scores). FACT provides a concrete, interpretable measure of citation grounding, which is highly relevant for research‑assistant agents.  \n4. **Human‑Alignment Validation** – The human consistency study is thorough: 70+ annotators, pairwise agreement (PAR = 71 % for RACE vs. 59 % for a vanilla prompt), and high Pearson correlations (OPC ≈ 0.99). The filtered‑task analysis further strengthens confidence.  \n5. **Robustness Analyses** – Sensitivity to reference choice, report length inflation, and judge LLM (closed‑source vs. open‑source) is explored, showing that rankings are stable.  \n6. **Reproducibility Commitment** – All prompts, hyper‑parameters, and scripts are released; the paper details cost per evaluation and provides open‑source judge alternatives.  \n7. **Ethical Considerations** – The authors appropriately describe anonymization of query logs, fair compensation of annotators, and discuss potential misuse of generated reports.", "weaknesses": "1. **Scale and Diversity Limitations** – 100 tasks, while high quality, may still be insufficient to capture the full variance of real‑world research queries, especially for emerging sub‑domains (e.g., AI‑ethics, climate modeling). This limits statistical power for fine‑grained comparative analysis.  \n2. **Reliance on Proprietary Judge LLMs** – The primary RACE and FACT evaluations depend on Gemini‑2.5‑Pro/Flash APIs. Although the authors provide open‑source alternatives (Qwen3) and show similar rankings, the absolute scores and some correlation metrics dip noticeably, raising concerns about reproducibility for researchers without access to the same commercial models.  \n3. **Citation Extraction Fragility** – FACT assumes that agents emit citations in a parsable format; the paper notes that several DRAs (Claude Research, Kimi) could not be evaluated because of UI‑specific citation styles. This suggests that FACT may penalize agents for UI design rather than factual grounding.  \n4. **Baseline Coverage** – The experimental section focuses on commercial DRAs and a few open‑source agents (LangChain ODR). There is no systematic evaluation of baseline retrieval‑only pipelines (e.g., traditional IR + summarizer) that could contextualize the performance gap.  \n5. **Statistical Reporting** – While human‑consistency numbers are presented, confidence intervals or statistical significance tests for the differences between agents are absent. It is unclear whether the observed gaps (e.g., Gemini‑2.5‑Pro vs. OpenAI DRA) are robust.  \n6. **Potential Conflict of Interest** – Many evaluated agents are from the same ecosystem (Google, OpenAI, Perplexity) that also provide the judge LLMs. The paper does not discuss mitigation of possible bias introduced by using a judge model that may be tuned toward its own family’s output style.  \n7. **Limited Discussion of Failure Cases** – The analysis of low citation accuracy for high‑citation agents (e.g., Gemini‑2.5‑Pro) is brief; deeper error analysis (hallucinated citations, URL unreachable, mis‑matched statements) would improve understanding of trade‑offs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9X5FF2ePXY", "reviewer": "Reviewer_xj7Q", "summary": "This paper introduces DeepResearch Bench, a benchmark designed to evaluate deep research systems. The benchmark consists of 100 expert-authored PhD-level tasks across 22 domains. For evaluation, the authors propose two methods i.e.,\n(1) RACE which uses LLM-as-a-Judge across four dimensions (comprehensiveness, depth, instruction-following, and readability),\n(2) FACT, which assesses citation accuracy and the number of verifiable evidence sources.\nThey ran experiments comparing both deep research agents and llm + web tool calls. they also conducted human eval to study alignment between llm as a judge and expert human annotations.", "questions": "- How were the final 100 benchmark tasks selected from the 40k+ samples?\n- Are the task difficulty levels annotated or estimated in any way? How is the distribution of difficulty across the 40k samples and 100 selected questions?\n- Are RACE’s adaptive criteria regenerated for each query instance or fixed once per domain?\n- Was the Judge LLM calibrated  before correlation tests?\n- Why were Gemini-generated reports used as reference as mentioned in 4.1.1 ? could this bias results toward Gemini agents?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The problem is timely and there is not many standard benchmarks nor evaluation strategies  designed for deep research systems.\n- The problems encompass a large set of topics.\n- they evaluate from several different perspectives i.e., both quality of the report as well as citation quality.\n- Conducted human eval to validate llm based evaluation\n- covered deep research systems as well as llm + search tools", "weaknesses": "- Some formatting problems in prompt template Appendix M\n- The baseline set omits several relevant open-source and hybrid systems such as Deep Researcher, OpenScholar, or opensrouced LLM + tool-use/web-agent frameworks. T\n- It is not explained how the 100 benchmark tasks were chosen from 44 K filtered queries. Did annotators manually reviewed all queries to come up with 100 queries?\n-  The benchmark lacks any quantitative or qualitative analysis of task difficulty. \n- In Section 3.1.1, it is unclear whether the RACE criteria are fixed per task or dynamically generated per query. \n- The FACT framework measures accuracy and count but lacks a notion of coverage i.e., how many claims in the report remain unsupported. Please see https://arxiv.org/abs/2411.17375 for claim coverage eval.\n- Section 4.1.1 implies that Gemini-generated reports serve as the reference. This design choice risks biasing evaluation in favor of Gemini-based agents.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "f7E622orcW", "reviewer": "Reviewer_oeAc", "summary": "In order to evaluate the performance of Deep Research Agents (DRAs) that gather and summarize information for research tasks, this paper presents a benchmark named DeepResearch Bench, with 100 PhD-level research tasks. This work also proposes two automatic methods for evaluation with Judge LLM, including a reference-based approach to measure report quality using adaptive criteria and reference reports, and a citation-based approach assessing the accuracy and effectiveness of retrieved sources. This paper further shows its consistency with human evaluation to highlight its potential.", "questions": "1. How is the variance across the domain or across tasks within a domain?\n\n2. As the final set includes English and Chinese tasks, if both are evaluated for the main results, is there any insight into the DRA performance across languages?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper contributes the first benchmark to evaluate Deep Research Agents. The design is also grounded in the real-world human research tasks, which are collected based on user interaction with Chatbots.\n\n2. The work conducted comprehensive experiments on popular DRAs, including one open-sourced one, and also compared the DRAs with search-enabled LLMs.\n\n3. The work also analyzed the consistency of the proposed evaluation with human evaluation.", "weaknesses": "1. The major concern comes from the validation of the consistency with human evaluation. It's unclear why the final set of tasks is designed to have 50 tasks for Chinese and English, respectively, and which split the experiments are conducted on. It's noted in line 364 that the human evaluation was conducted on the Chinese tasks, which might limit the claim of consistency with human evaluation if the main results are obtained on English tasks.\n\n2. There are 100 tasks in total, with 50 Chinese and 50 English tasks, across 22 domains. The number of tasks per domain and language might be limited.\n\n3. The authors conducted a comparison of human evaluation consistency using various Judge LLMs to select the candidate for Judge LLM. However, the performance of the proposed metrics is highly dependent on the Judge LLM, and the RACE evaluation also relies on the reference reports. While Gemini models are used as the Judge LLM, and Gemini-generated reports are used as the reference reports, this might introduce model bias and favor Gemini-based DRAs, which is also shown in the experiment results.\n\n4. The introduction of the evaluation frameworks in Section 3 is fragmented and eliminates many details, such as the computation of the FACT framework. More complete descriptions of the methods should be included in the main text to increase clarity, potentially moving space from Figures 1, 3, and 4.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "McYVdzhZCx", "reviewer": "Reviewer_8pm9", "summary": "The paper presents DeepResearch Bench, a benchmark for evaluating Deep Research Agents (DRAs) that perform open-ended web-based research. It comprises 100 expert-curated tasks across 22 domains and introduces two automated evaluation frameworks: RACE for report quality and FACT for retrieval accuracy. Experiments and human studies show strong alignment between these frameworks and expert judgments. Overall, the work provides a solid foundation for systematic and scalable evaluation of LLM-based research agents.", "questions": "n/a", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1.\tThe paper introduces a well-motivated and comprehensive benchmark for Deep Research Agents, grounded in real-world user queries and expert-curated tasks across 22 domains, resulting in a high-quality and realistic evaluation suite.\n2.\tThe proposed RACE and FACT frameworks jointly assess report quality and retrieval reliability, effectively addressing a key gap in existing evaluation methods — making this a timely and impactful contribution.\n3.\tThe experiments are extensive and insightful, covering both commercial and open-source DRAs, and offering clear comparative results and valuable guidance for future research and development.", "weaknesses": "There is no major weakness in this paper. The only minor concern lies in the size of the benchmark — with 100 tasks, its scale may still limit robustness for fine-grained statistical analysis or long-tail domain evaluation. However, given the high quality and expert-level design of the tasks, this limitation is acceptable and does not detract from the overall contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)