Characterizing Deep Research: A Benchmark and Formal Definition
Authors: Abhinav Java
Ashmit Khandelwal
Sukruta Prakash Midigeshi
Aaron Halfaker
Amit Deshpande
Navin Goyal
Ankur Gupta
Nagarajan Natarajan
Amit Sharma
Venue: ICLR 2026 Poster (2026)
Keywords: Benchmark
Evaluation
Deep Research
Primary area: datasets and benchmarks
TL;DR: We formally define Deep Research and introduce a benchmark to evaluate it.

Abstract: Information tasks such as writing surveys or analytical reports require complex search and reasoning, and have recently been grouped under the umbrella of _deep research_ --- a term also adopted by recent models targeting these capabilities. Despite growing interest, the scope of the deep research task remains underdefined and its distinction from other reasoning-intensive problems is poorly understood. In this paper, we propose a formal characterization of the deep research (DR) task and introduce a benchmark to evaluate the performance of DR systems. We argue that the core defining feature of deep research is not the production of lengthy report-style outputs, but rather the high fan-out over concepts required during the search process, i.e., broad and reasoning-intensive exploration. To enable objective evaluation, we define DR using an intermediate output representation that encodes key claims uncovered during search—separating the reasoning challenge from surface-level report generation. Based on this formulation, we propose a benchmark LiveDRBench with 100 challenging tasks over scientific topics (e.g., datasets, materials discovery, prior art search) and public interest events (e.g., flight incidents, movie awards). Across state-of-the-art DR systems, F1 score ranges between 0.02 and 0.72 for any sub-category. OpenAI's model performs the best with an overall F1 score of 0.55. Analysis of the reasoning traces reveals that systems cover only about half of the necessary search queries, with proprietary models issuing broader and and deeper queries than open source models, highlighting gaps in both coverage and reasoning depth. The benchmark is available at [this https URL](https://github.com/microsoft/LiveDRBench).

CHARACTERIZING DEEP RESEARCH: A BENCHMARK
AND FORMAL DEFINITION
Abhinav Java'
Ashmit Khandelwal'
Sukruta Midigeshi"
Microsoft Research Bengaluru, India
Microsoft Research
Microsoft Research
Bengaluru, India
Bengaluru, India
Aaron Halfaker
Amit Deshpande
Navin Goyal
Microsoft Redmond, USA
Microsoft Research
Microsoft Research Bengaluru, India
Bengaluru, India
Ankur Gupta
Nagarajan Natarajan
Amit Sharma'
Microsoft Redmond, USA
Microsoft Research Bengaluru, India
Microsoft Research
Bengaluru, India
ABSTRACT
Information tasks such as writing surveys or analytical reports require complex search and reasoning, and have recently been grouped under the umbrella of deep research - a term also adopted by recent models targeting these capabilities. Despite growing interest, the scope of the deep research task remains underdefined and its distinction from other reasoning-intensin In this paper, propose formal characterization of the deep research (DR) task and introduce a benchmark to evaluate the performance of DR systems. We argue that the core defining feature of deep research not the production of lengthy report-style outputs, but rather the high fan-out over concepts required during the search process, i.e., broad and bessoning-intensive exploration. To enable objective evaluation, we define DR using an intermediate output representation that encodes key claims uncovered during search seercct-e the reasoning challenge from surface-level report generation. Based on this formulation, propose benchmark LIVEDRBENT with 100 challenging tasks over scientific topics (e.g., datasets, materials discovery, prior art search) and public interest events (e.g., flight incidents, movie awards). Across state-of-the-art DR systems, Fl score ranges between 0.02 and 0.72 for any sub-category. OpenAl's model performs the best with overall F1 score of 0.55. Analysis of the reasoning traces reveals that systems cover only about half of the necessary search queries, with proprictary models issuing broader and and deeper queries than open source models, highlighting gaps in both coverage and reasoming depth. The benchmark available at thtps: //github. cwtp:///cccrroofoe/eeeeBeeecccc
problems is poorly understood.
INTRODUCTION
Deep Research (DR) has emerged popular paradigm for applying the reasoning capabilities of AI systems. DR systems have been shown to be useful for writing research reports (Li et al. 2025). business reports (Microsoft and Spataro 2025), and answering needle-in-a-haye hhysshay queries (OpenAl 2025). These tasks involve non-trivial reasoning that may take a human expert hours to complete and DR systems are expected significantly reduce that time (FutureSearch et al.) 2025 OpenAl 2025).
Beyond demonstrative examples, however, no characterization of the problem exists. Typically, DR tasks are understood to be report-writing tasks that take non-trivial time for a human to complete. They are considered similar to reasoning-intensio search tasks, but expected to be much harder than
Joint first authors, ordered alphabetically.
Joint advising. Correspondence nagarajn@mi crosoft. com, weachar@mirrsoft.ccomm

---

"angeertee of SUsTEAW Curn CUAD
"ONT AOO
Deep Beoearchh
REASONING INTENSITT
Claimn HaTeENNte
Tcleral weticica..
ScholargABeench
Prence"
fort US prasltvee
NunuaiQuessiora IivBIQ
HotPorQA 2WIN
FRAMES
LieeRBonnh
"pore 2n Pair artorn have trid...
SEARCH INTENSITY
"pora Sckukomannsian
ISPORT
Figure 1: Characterizing the deep research task. Left (a) shows the landscape of various multi-hop reasoning tasks. Compared existing tasks, the deep research task involves both high search and high reasoning intensity. Right (b) shows a stylized process of generating an answer to a DR query: DR query > Claims >> Long form report. LIVEHRBENCH focuses on the precision and completeness of the intermediate but crucial step of claim generation.
Openr) ER
Pappicnly
Gemini
0.287
0.5
9.
0.158
8,001
0.079
0.3
0.4
0.
Openst DR
Porplesity of
Gomini DR
OpenAi akmin)
Geminl Pro
Openit GPF4.
Gemini 2.5 Fleeli
Sonar Pro
Soner Reasoning
0.1
Natoriats
Oco
NovalDS
NovelDS
NevelDS
Prior
Entities
Flights
Extract
Figure 2: Average FI score for deep research Figure 3: Average Fl scores of deep research models and search-enables LLMs on LIVEDR- models split by the eight categories of LIVEDR- BENCH.
BENCH. OpenAl DR obtains the highest FI.
those studied in the literature such as multi-hop QA (Yang et al.) 2018 Hongjin et al.).. Interestingly. the hardness of task also depends on the available document corpus. For example, "share list of all Oscar-winning movies adapted from books with women authors" seems to be a hard, research- oriented task, but may not qualify as DR if there's 's webpage providing exactly this information. Overall, the lack of a formal definition makes it hard to evaluate DR models and measure progress.
Defining DR. We characterize the deep research task and provide benchmark that can evaluate progress objectively. Given a document corpus, we propose that the DR task can be split into two subtasks: 1) synthesizing claims picces of information relevant by subclaims from retrieved evidence- and 2) writing a report based on these claims. Rather than the common expectation of long report-like output, we posit that the defining challenge of DR the first sub-task: synthesis of relevant information from corpus. We conceptualize this as a directed acyclic graph (DAG), where the nodes represent information such as original query, retrieved documents and synthesized information, and edges represent actions such as issuing search queries and reasoning over retrieved documents (see Figure I(b). We call a query a DR query if its information synthesis process is both search-intensive and reasoning- intensive. That is, given a document corpus. 1) the number of information units atooiic pieces of information such as paragraph or a chunk be processed for obtaining the final answer is high (search intensity); and 2) at least one of the tasks tasks-findin these information units, processing them or combining them to form the final claims non-trivial reasoning from human expert (reasoning intensity). Practically, we posit that any query that takes more than 10 minutes for a human expert qualifies DR. Assuming the entire web as the retrieval corpus, examples include "all oscar-winning movies adapted from books by women authors with runtime 80-95 minutes" or "find material having the following properties", In contrast, examples of non-DR queries include "properties of the material used in this paper" (low search intensity) and "list of papers accepted at NeurlPS 2024" (low reasoning intensity).
user's 's query, often supported
2

---

Objective Scoring
Multiple Claims
Open Web Search
Easy to Update
Publicly Available
Benchmark
in Response
Deep Reseurch Bench (FutureSeurch et all)
x
x
x
BrowseComp (Wei all)
DeepResearch Bench (Du at.)
DeepResearhhym (Coelho ot al.)
X
LiveDRBench
Table I:
Comparison of DR benchmarks. By employing claim-based evaluation, LIVEDR BENCH can provide objective scoring for report-based DR answers that involve multiple claims. We focus on the open web search setup to allow direct measurement of the gap between proprietary and open DR systems, which necessitates mechanisms to easily update benchmark questions as the Web is updated.
The above definition distinguishes DR from commonly studied caasonig-inntensvve information tasks (Krishna et al.) 2024; Yang al.) 2018 [H et al.] 2020;; Hongjin et al.e Figure [(a).
Multi-hop QA datasets like HotpotQA Yang et al., 2018) are low in both search and reasoning intensity; scientific or legal QA tasks (e.g., CURIE CCui et al.),, CUAD (Hendrycks
al. 2021
demand reasoning but little search; and explanation tasks such as ELI5 (Fan et al. (019) involve reasoning but limited retrieval focused on a specific concept. In contrast, most DR tasks exhibit both high search and reasoning requirements, such as GAIA (Mialon et al. 2023) or Humanity's 's Last Exam (Phan et al. 2025). Beyond these task completion DR benchmarks that are evaluated on single final answer, we focus on typical information-synttenn text output, such as writing a policy proposal, an academic survey, or an analytical report.
DR applications that involve a complex
Evaluating DR. Given the complex output, assessing the quality of a detailed report is an inherently subjective task, involving evaluations of style and substance, as done by Du et al. (2025). Instead, for objective evaluation, evaluate DR output only on the correctness and completeness of its substantive claims. Formally, we define the solution to a DR problem in terms of an intermediate output representation that consists of nested list of claims: (query, list of claims) where ideal solution gets all the claims and their (recursive) subclaims correct. Since users may typically evaluate both the claim and its supporting subclaims for assessing validity of the DR report, we define modified Precision and Recall metrics that assign zero credit if the claim or its subclaims are incorrect.
Based on this formulation, we provide a benchmark, LIVEDR spanning science and world events. In Table T we show the key benefits compared to other DR benchmarks (FutureSearch et al.) 2025 Du et al. 2025): problem inversion technique (Wei et al. 2025a for constructing DR queries that allows using the entire web as the retrieval corpus (easy comparison of both proprietary and open systems) and allows easy addition of new queries; objective evaluation based on our intermediate claim- representation, and queries that capture utility of DR across society, including scientists, information professionals, and the general public. Specifically, the scientific domain contains queries to identify datasets or material given certain properties and to identify whether given idea is novel. The world events domain includes queries about flight crashes and entities such as books and movies. These queries span various types of DR queries, from needle-in haystact to queries that require enumerating a list of entities.
consisting of 100 DR queries
We use LIVEDR IEEDRBENCC BENCH to evaluate sfate-of-the-arr DR systems from OpenAl, Perplexity AI. and Google, as well as the opensource Deep Researcher (Zheng et al. 2025) agent. LIVEH presents challenging
queries for these systems; the Fl score per category in these models ranges from 0.02-0.72 (see Figures 2 and Overall, OpenAl's DR system obtains the highest (average) F1 of 0.55. Analysis of the reasoning traces reveals that systems cover only about half of the necessary search queries, with proprietary models issuing broader and and deeper queries than open source models. Our results motivate future directions on improving the search mechanisms for DR models.
ieformationnsyytheeen
BACKGROUND & RELATED WORK
We discuss recent deep research (DR) systems and their evaluation, as well as earlier efforts in long-form report generation which served as precursors. Our focus is on information synthesis tasks. While synthesis is core to DR, DR tasks may also involve computer use (Wei et al. 2025a). coding (Wang al. 2025), calling external tools (Mialon et al. 2023).
3

---

Tisk
Description Shortened Example
Inst.
SCIFACTS
Find revearch papers that all datasets specified in the query.
19
Gen
Fiud the paperts) that use all of the dotasets montioned: American Community Survey Flooding Data from FEMA,
SCIFACTS Materials
Find materials match specified measured properties. Find the materiall)) that swiiffy every one the measured propertier: Direct band gop: 3.37 eV. Exelton binding energy: 60 me ...
NOVELDS
Find a dataset matching unique characteristics and return its metadata.
6
ddeniffiction
Identify a dataset of long-form in-the-wild videos that are segmented into scenes with and user engugement signals. The dataset should... and support andlyiis at the video and scene level.
NOVELDS
Find a dataset matching unique characteristics and extract specitic findings from its paper. I'm looking for GPT generated corpus decisions rosed in quotidian lift ... How does this corpus reveal GPT-4's implicit generation bins ocrost various value dimensions ...? Retrieve peet dataset papers in the sume problem space based on high-tevel description. want to compure existing 3D segmentation datasets ... that represent urhan environ- ments as richdy amnotate point clouds. How do they compare in terms of data acquisition methods such as .
Identifiction and Extrac- tion
NOVELDS
3
Peer Retrieval
PRIORART
Find whether a proposed research idea has already been fully or partially explored in existing papers.
17
Can you help identify if the following idea has abraad been done-.. in other paperx? We develop comprehensivv evaluction framework for eliciting reasoning mistakes in L.Mss..
explore espects of mistake correction in LLMs and explicit mistake detection in reasoning chains ... We focus on distinguishing among self-generated responses...
ENTITIES
Find an exhaustive list of seal-world cataties matching detasled criteria.
20
Provide a comprehensive fist films that meet the following criteria: ... category of animation bw not anime. 2. Tomatometer rating of 95% or higher. 3, runtimme must be between 50 and 85 minutes inclusive.
FLIGHTS
Find a real-world event that fits the described conditions. In which flight incident did commercial airliner perform unusually high number of gu arounds before safely landing? For each attempi, detail the time, runway, landing aidss..
Total
100
Table 2: Tasks in LIVEDRBENCH, spanning eight categories.
Deep Research task. In late 2024, Google launched a product for deep research queries Gooole (b), followed by OpenAl's model in February 2025 (OpenAl 2025), which was later integrated with Microsoft Copilot (Mierosoft and Spataro| 2025). Perplexity (Perplexity aa) and Grok (XAI) DR systems soon followed, though details on design and evaluation of proprietary systems re- main limited. Academic work on knowledge synthesis and long-form generation can be viewed as precursors to DR. Examples include scientific literature search (Asai et al., 2024), planning- based prompting (Godbole et al., 2024), and conversational prompting (Shao et al.] 2024). ). After commercial launches, open-source implementations such as Huggingface Deep Research (Hugging. Face: Roucher et al.] (025), WebThinker (Li et al.) 2025). Open Deep Research (Zhang) 2025), and DeerFlow (Bytedance] (2225) were proposed. These typically use either (i) agentic orchestration of
LLM calls (Huggingl Face: Fagg: Roucher et al. 2025 Zhang 2025 Bytedance 2025) or (ii) reinforcement- learning finetuning (Shi et al. 2025 Li et al.
2025 Zheng et al.) (025).. Although surveys on DR Xu and Peng 2025). ), a formal problem definition is lacking, which we
exist (Huang et al. 2025
provide in Section
Evaluating Deep Research, Since DR output is typically a report, most evaluation focuses on long- form text quality (Du et al., 2025 Xu and Peng 2025 Chandrahasan et al. 2025). DeepResearch Bench (Du et al., (225)) scores reports on comprehensiveness, insight, readability. and instruction following, using LLM judges such GPT-4. ResearcherBench (Xu et al. 2025) uses human experts to create critetria for Al research queries, but is difficult to scale. In contrast, we provide an objective evaluation via claims, drawing on prior work in long-form form evaluation (He et al. 2025) and claim
verification (Metropolitansky and Larson 2025 Dmonte et al.) 2024). As for the query types, bench- marks either focus on long reports (Du et al. 2025 Xu et al. 2025). FutureSearch (FutureSearch et al. 2025) evaluates on economically valuable tasks (e.g., numeric answers, evidence gathering, claim validation), but its dataset is not public. Our approach differs by providing public benchmark spanning both needle-in saaleeiin---hhaaee --ahaaatta and broader information informattion-gatheiinn tasks (Table 2 with mechanisms to refresh questions to reduce contamination. Using the entire web as the corpus allows exact comparison of proprictary and open DR systems, unlike benchmarking efforts that build a static corpus (Kang and Xiong 2024
queries Wei et al. (2025b) or focus on queries requiring
Coelho et al. 2025). Furthermore, we formally define DR (Definition 1) capture
4

---

both search intensity and reasoning intensity, and introduce claims-based benchmark that measures both breadth (search coverage) and depth (claim correctness). These dimensions are not explicitly modeled in prior works like Mind2Web (Deng et al. 2023).
3 DEFINING DEEP RESEARCH
As discussed in Section | report. Writing a report given a set of claims is a long-form text generation problem (He ct al. 2025) that can be independently studied.
we focus on the DR subtask of generating the required claims to write
Defining an information unit. The definition of deep research (and multi-hop RAG) depends on the retrieval corpus. Some queries may appear multi multi-hop (e.g., "movie adapted from the book that won the 40th Booker Prize") but are not because they can be answered from single Wikipedia page (e.g,, paragraph in page about the prize-winning book that also mentions the movie name). Other cases may involve combining information from multiple sections of a single large document (e.g., a 200-page financial report)
single document but can be considered multi-hop due multiple retrievals to fetch each section. Hence, we define an "information unit" to correspond to an atomic piece of information such as a paragraph, typically implemented
chunk in retrieval systems (Liu 2024). Consequently, single document may include multiple information units.
et al.
3.1
DEEP RESEARCH: TASKS INVOLVING BOTH SEARCH AND REASONING INTENSITY
The deep research task can be considered as an extreme version of the multi-hop hop RAG task. Given a document corpus, answering multi-hop RAG question requires retrieving and combining information from more than one document. There are three aspects in which the DR task more complex than multi-hop RAG: how many pieces of information need to be processed; how easily those pieces of information are found, processed or combined; and how complex the final output is.
1. Number of information units that need to be processed. Unlike multi-hop tasks that are based on 3-5 information units (Yang et al. 2018 Ho et al. 2020 Krishna et al., (024), many DR tasks require combining information from large number of input sources, such writing survey based on tens of academic papers and content within them, or listing all items that meet certain criteria. In terms of the search process of an ideal human expert, this corresponds roughly to the minimum number of searches issued by the expert.
Complexity of finding, processing, or combining the required information units. In many DR tasks, the search queries to issue are not obvious from the user's 's question and require tens of reasoning- intensive iterations. In other DR tasks, the search queries may easy to enumerate, but processing or combining the retrieved information units to develop output claims is reasoning-intensive. Thus, DR task includes at least one of the following reasoning components.
For needle-in-a-haye -nasssaat questions (e.g.. "find material(s) with the following properties"). identifying the right search queries based previous queries' results is the key subproblem.
Finding information units.
Processing information units. For other DR questions, the searches may be easy to enumerate, but processing each result may require reasoning. For example, consider the question, "English language films since 2024 meeting the following criteria...", where finding the listing of films is trivial but processing each film's properties needs reasoning.
Combining information units. In other DR questions, the most difficult part is combining the retrieved information units to answer the user's question. For example, consider the question, "write survey of open-weights language models released by American companies"- most reasoning-intensivee part is comparing information (e.g., latency) across results of multiple searches.
3. Complexity of the Output. Finally, a typical DR task requires a detailed report, involving multiple claims and their justification. We abstract out the content of the report as a nested list of claims, AA Assuming independent top-level claims, the structure of AA can be considered as list of dictionaries, where each claim corresponds to dictionary (see [p). Each top-level claim can have nested subclaims, which are the keys of the dictionary. We have the following definition for DR.
Definition 1. Given a document corpus C, query set Q, and an ideal human expert who does not already know the answer any query g E Q.
5

---

Answer
Anawer Possiibia
Q. Which material band gap-0.69 ev, distoc. de cm Lottice Const. a=0 31 88mm?
Bend gep: 0.89 eV, Distocation density: 4x10" cm?
band gapeo E8 dislocation dansity- 10"
Answer The materisl is GaN
materielet
Properties
InN
the material
this paper
STEP Find reasoning problern
problemn
STEP
unique
Figure 4: Problem inversion for LIVEDRBENCH. The first step is to find a long- or document reasoning problem. In the second step. this problem is inverted to create new question asking for an event or entity consistent with the properties mentioned in the document. In the third step, the question is refined (e.g., more properties are added) such that it admits unique answer; and in case there are additional valid documents (A1, A2), the ground-truth set of documents is updated.
1. A query g is said be deep research query if answering it requires processing a large number of information units (search intensity) and at least one of the following subtasks- finding the required information units, processing them, or combining requires non-trivial reasoning (reasoning intensity).
2. The deep research task can be formulated as a tuple (user query 4, answer list A, corpus C) where A consists of independent claims that make up the final answer. Each claim in A may recursively include sub-claims that need to be derived and presented to arrive at the claim.
In practice, reasoning- intensive for a human expert may translate to a solving time of >> 10 minutes using tools such as web search, and search intensity to 20 information units (through >= 10 queries).
3.2 CLAIMS-BASED EVALUATION: PRECISION AND RECALL OVER GROUND-TRUTH CLAIMS
key benefit of defining the DR output terms of claims that it unites the different kinds of DR tasks. Whether asking for list of entities, set of datasets or a particular material, each query's s answer can be broken down into claims that are necessary to provide the answer, and in turn, sub- claims that necessary to establish each claim. For developing that the accuracy of subclaims critical for the accuracy a claim: 1) users may distrust claims that have incorrect subclaims, and 2) we want to reward DR systems that search corpus and find the answer rather than those that rely on memorized knowledge.
evaluation metric, we assume
From Definition] 1 assume that the ground truth can be written as A [A] ....,,..AA,, where each A is a dictionary containing subclaims for the ith claim. Similarly, the DR output can be parsed into list of claims, A (for an example prompt, score between any claim A; and the ground truth claim A Typically, this will be a binary score {0, 13. For each independent claim, its contribution the Precision metric is the product of its own agreement score and the precision of its subclaims (which would be an average over agreement scores for all the subclaims). For the recall metric, we follow the same approach.
Appendix (F). Let 8,4 be the agreement
Pree(A) Es. (A,(A.) (ri) Rec(A) EA, (A.) Rec(A.)
(I)
Σs.
Ex;
where Prec(A,) -R)))))))RRAAA) = 1 whenever subb((A)) = O. Given a claim A,, subo(A,) represents its subclaims; su,)(A,) = Φ for an atomic claim. w; is an optional weight that can be assigned to each claim, e.g., to weight the most important claims higher. Note that we can also have a strict version of metrics that provide score of zero if any of the subclaims is incorrect (see Appendix B).
4 LIVEDRBENCH: A CHALLENGING BENCHMARK FOR OPEN WEB DR
We start by discussing the desiderata for a DR benchmark, present the problem inversion technique that helps us create LIVEDRBENCE satisfying the desiderata, and then give details on the tasks.
Criteria for tasks. Formulating Deep Research (DR) evaluation tasks is tricky because the models typically have access to the entire Web at inference time. We list the following desiderata for DR

---

evaluation benchmark:
(I) The tasks should not be answerable using any single Web source. This is especially challenging, as the Web is constantly updated and new article specific to a query may collapse DR task into single retrieval one.
(2) The tasks should require accessing wide variety of sources the Web, force the models to find the right grounding rather than relying on parametric memory.
(3) Evaluation of each task should be objective and reproducible, and metrics should help benchmark the progress in the DR field.
(4) The tasks should be amenable to periodic updates. This helps keep pace with fast-moving Web as well as fast-changing models.
(5) The tasks should capture a wide variety of DR domains and users (e.g., scientists, researchers and analysts, common Web users).
Striking a balance between all these constraints is non-trivial. For instance, constructing a benchmark by asking domain experts to provide the questions, as by Du et al. (2025), satisfies the second property, but it may not be easy to update. Instead, we leverage a strategy based on inversion of standard NLP and reasoning problems (see Figure 4 described next.
Problem inversion technique. We build on the insight used in the construction of OpenAl's recent -rafft questions for which the answers are hard to find but easy to verify. However, their procedure requires asking human raters to think of a concept or fact with many unique characteristics and invert it ask question find such a concept. To craft DR questions in a scalable way, we templatize the procedure and apply it to different kinds of information sources such as research publications and publicly available reports or lists. As publications or public reports keep updating. we can easily update our benchmark questions. More importantly, our construction is flexible and allows creation of questions whose answers include multiple entities and claims.
benchmark (Wei et al.
As shown in Figure 4 the proposed inversion process involves three steps: 1) identify documents- typically long documents-that include detailed information; 2) extract entities or concept classes described therein with unique characteristics; and 3) devise a DR question to find such entities having the said characteristics, without knowledge of the input documents. In particular, long-context reasoning problems are especially suitable for inversion, since they already include questions that can be used to identify the entities. The standard long-context reasoning problem has the following flavor, as in the Curie benchmark for reasoning over scientific papers (Cui et al.):
[Long-contex: Reasoningl Given a long source document on an entity or a subject and a list of keys, identify values for the keys consistent with the document.
For instance, given a research article about the effects of a certain treatment, answering questions about the subjects of study, statistical significance, etc. would fall under long-context reasoning. The inverted problem becomes a DR task:
[DR] Given keys and values that conform a certain (unique) entity, identify the entity and locate the sources that corroborate the identification.
In the above example, the inverted DR task is to locate the research study article(s) given the parameters of user study, participant details, symptoms, and chemicals used in the treatment. Note that although we started with a single research article, it is possible that the dictionary of (keys, values) may yield multiple articles as the correct answer. In practice, we manually verify that DR question yields a unique answer and if not, we update the extracted dictionary (or in some cases, extend our ground-truth manually by adding the additional correct answers). In other cases, multiple entities form the answer by design. For example, consider the (hypothesized) long document combining all Oscar award announcements till date; we can ask inverted questions about any subset of awardees that satisfy certain relevant criteria.
LIVEDRBENCE tasks. Based on the aforementioned desiderata (1-5), our tasks cover the domains of science (facts and datasets), innovation (prior art search), and global culture and events (entities and flights). As discussed in the Introduction, as output (see Figure [T and ensure that these tasks have a range of reasoning and search requirements. majority of tasks in BENCHI are based on inverting the long-context reasoning problem (as discussed above). The eight task categories in LIVEDR BENCH are listed in Table 2 We give details of task construction in Appendix and examples in Appendix D Below, we give brief description. (1-2) SCIFACTS. We create 31 questions, derived from scientific papers and their questions from the
formulate tasks that expect claims with right grounding
7

---

CURIE dataset (Cui et al),, to cater to scientists and researchers: (1) the Materials subset consists of 17 questions emulating scenarios where a scientist might want to look for a specific material that perfectly fits the given criteria. The ground-truth answer for each question consists of the material name (claim) and the source paper title (subclaim); (2) the Geo subset consists of 19 questions emulating a researcher attempting to search for papers that precisely utilize given set of geospatial analysis datasets. The ground-truth answer is a list of paper titles (claims) which use given datasets. (3-5) NOVELDS. We create 20 questions based on dataset papers. drawing from both our domain knowledge and the dataset papers accepted at ICLR 2025: (3) the Dataset Identification questions are designed to emulate scenarios where researcher looking for details of a dataset satisfying specific characteristics. The ground-truth answers consist of dataset name (claim) and meta-data such as publication yer, venue, and source link (subclaims); (4) Dataset Identification and Extraction questions also require extracting specific results (e.g., interpreting figure) from the dataset paper (subclaims); (5) the Peer Dataset Retrieval questions elicit articulating the key features that define and distinguish multiple related datasets pertaining to given high-level description. The ground ground-truth answers include dataset details (claims) and supporting details such as performance metrics or dataset attributes (subclaims).
(6) PRIORART. We devise 17 questions motivated by the laborious problem of validating (or invali- dating) claims in patent applications (Churnet 2012). Using recent ML papers from ICLR 2025 and ICML 2025, we manually combine ideas and contributions from 2-3 papers to write a new paper abstract, and the DR question is to identify the key ideas and the source papers that demonstrate those ideas. The ground truth answer consists of list of ideas or contributions in the abstract (claims), paper titles and links of the source papers and rationales (subclaims) for why the references demonstrate the claims.
(7) FLIGHTS. We construct 7 questions based on official flight investigation reports published by national aviation authorities. Each question involves inverting the report: identifying the correct flight incident and specific details, given a high-level description elaborating the causes or timeline of the incident. The ground-truth answer is identification details such as timestamp (claims) and supporting details such as sensor readings (subclaims).
(8) ENTITIES. We devise 20 questions pertaining global culture (e.g., sports, books) that require consulting a broad range of sources on the Web. These questions pertain to lists of entities connected to an event (e.g.. a yearly award like Oscars). They are constructed by inverting the set of documents corresponding to global event or entity: developing a certain criteria and asking a question to produce an exhaustive list of entities satisfying the specified criteria. The ground- -ooun answer is a list of entity names (claims).
Updating the benchmark. As remarked above, our templatized inversion process makes it easy to update the benchmark as new scientific papers or global events happen. SCIFACTS can be augmented with other long-context reasoning benchmarks such as DocFinQA Reddy et al. NOVEL.D with new dataset papers. FLIGHTS with new flight incidents, PRIORART with new ML papers, and ENTITIES with new editions of the global events such as awards.
USING LIVEDRBENCH TO EVALUATE DEEP RESEARCH SYSTEMS
Setup. We evaluate three proprietary DR models: OpenAl DR OPeenAl 2025). ). Perplexity DR (PerplexityAI] a)), and Gemini DR (with 2.5 Pro) (Google, b). Among open DR models, we consider DeepResearcher (Zheng et al., 2025); others such as HuggingFace DR (HuggingFacee Roucher et al., 2025) and LangChain Open DR (Lang 2025) did not run successfully due to infinite looping or system errors. In addition, we also evaluate three reasoning-enabled baselines: OpenAl's 04-mini (OpenAl b). Perplexity's 's Sonar Reasoning (PerplexityAl c) and Gemini 2.5 Pro (Google a) and non-reasoning basclines: OpenAl's GPT-4. [OpeAAl a). Perplexity's 's Sonar Pro (PerplexityAl (b.
and Gemini 2.5 Flash (Google a). All baselines have web search enabled. Additional setup details discussed in Appendix A and additional evaluation results are discussed in Appendix! []
5.1 RESULTS
Overall. Table B reports precision, recall, and F1 scores for all Deep Research models across all task categories. The claim agreement scores are computed using GPT-4o (see Appendix [C.3 for details) and the metrics computed using Equation T Across all categories, the OpenAl DR model
8

---

OpenAl Deep
Perplexity Deep
Gemini Deep
DeepRexearcher
Subset
Research
Research
Research (2.5 Pro)
DS Qwen 32B
Prec
Rec
F1
Prec
Rec
F1
Prec
Rec
Fl
Prec
Rec
F1
SCIFACTS
Materials
0.313
0.316
0.314
0.264
0.105
0.150
0,072
0.013
0.022
0.000
0.000
0.000
SCIPACTS Geo
0,715
0.728
0.721
0,263
0.144
0.186
0,405
0.259
0.316
0.000
0.000
0.000
NOVELDS
0.667
0.667
0.633
0.633
0.633
0.400
0.167
0.167
0.167
Memiiicatiin
NOVEL
Idantifictioo
0.526
0.448
0.470
0.325
0.349
0.333
0.406
0.329
0.345
0.045
0.015
0.023
Extraction
NOVEL DS Poor
Retrioall
0.795
0.480
0.585
0.494
0.248
0.311
0.551
0.249
0.338
0.083
0.028
0.042
PRIORART
0.694
0.463
0.539
0.689
0.339
0.419
0.106
0.096
0.082
0.525
0.136
0.199
ENTITIES
0.813
0.534
0.603
0.673
0.373
0.447
0.494
0.3
0.338
0.307
0.046
0.076
FLIGHTS
0.542
0.546
0.540
0.389
0.347
0.362
0.249
0.276
0.261
0.120
0.073
0.090
Average
0.633
0,523
0.555
0.466
0.317
0.355
0.335
0.240
0.263
0.156
0.058
0.075
Table 3: Precision, recall, and Fl scores LIVEDRBENCH for Deep Research models.
Model
Precision
Recall
Fl
OpenAl
0.526
0.448
0.470
Perplexity DR
0.325
0.349
0.333
DR Model
% Nec.
# Dep.
# Indep. Queries
DR
0.406
Coverage
Queries
DeepResearcher
+ Qwen 32B
0.045
0.015
0.023
OpenAA DR
Perplexity
66.0
34
64
DeepResearcher
0.167
Gemini DR
46.8
39
24
+ GPP.4.1
DeepResearchr
53.4
5
OpenAl o4-mini
0.203
0.146
0.168
DS Qwen 32B
Sonar Reasoning
Gemini 2.5 Pro
0.186
0.015
0.003
0.130
0.142
0.005
GPT-4.1
DeepResearcher
49.4
6
OpenAl GPT-4.1
0.126
0.078
0.088
Souar Pro
0.027
0.020
0.023
Table 5: Comparison of DR models by Neces- sary Query Coverage and Dependen//Indepedent queries.
2.5
0.211
Table 4: Comparison of DR models and baselines on the NOVELDS Identification and Extraction.
achieves the best performance, often significantly better than Perplexity and Gemini DR models, with an overall average Fl of 0.55. While Gemini outperforms Perplexity on SCIFACTS Geo, NOVELDS Identification and Extraction, and Peer Retreival, Perplexity exhibits stronger average performance compared to Gemini. Across models, the Fl score per category ranges from 0.0 (DeepResearcher on SCIFACTS Materials) to 0.72 (OpenAI DR SCIFACTS Geo). The open-source ReepReserrcher model obtains significantly lower scores (except for PRIORART where it outperforms Gemini DR), highlighting big gap proprietary and open DR systems. These results also highlight the difficulty of LiveDRBench compared to other benchmarks. In particular, non-reasoning models such as GPT-4. achieve low Fl scores (0.0-0.33 across categories, e.g., see Table 4). Category- wiie results that include non-DR DR models are in Tables 7-13 in Appendix.
Accuracy of claim and its subclaim. The DR models perform worst on SCIFACTS Materials, NOVELDS Identification and Extraction, and FLIGHTS, likely because these tasks require reasoning across multiple sections of papers and flight reports to extract both main and subelaims. The models perform best on SCIFACTS Geo, which does not involve any subclaim extraction. Similarly, the performance NOVELD: DS Identification is higher due to its relative simplicity paper, venue, and its link are closely related. For SCIFACTS materials, models often correctly extract either the paper title or the material name, but not both, resulting in low overall F1 scores. OpenAI DR achieves an F1 of 0.735 for paper titles and 0.504 for materials, but only 0.314 overall. Similarly, PPL DR reaches 0.348 (title). 0.32 (material), and 0.158 (overall), while Gemini DR performs worst with 0.17, 0.305, and 0.023, respectively. This highlights the difficulty of accurate extraction of claims.

---

5.2 ANALYSIS OF DEEP RESEARCH MODELS
We analyze the search queries present in the DR models' reasoning traces to quantify (1) the coverage over necessary search queries (as a proxy for breadth of search), (2) their dependence on preceding queries (as a proxy for depth); (3) amount of branching and backtracking employed. GPT-4o is used to analyze the reasoning traces (prompts in Appendix G).
Coverage over necessary search queries. To evaluate whether the DR models invoke the complete set of required queries for a given DR question, we generate search queries without which it not possible to fully answer the question. Specifically, we generate characteristie which are queries that need to be issued to locate relevant source papers or reports, and extractive queries, which target the fields to be retrieved from those sources. Coverage is then measured as percentage of these necessary queries that are present in the DR models' traces. Results in Table [ show that DR systems can generate nearly half of the required queries, demonstrating a big potential for improvement. Qualitative examples are included in Appendix H Dependence on preceding queries. To approximately assess the depth of the DR models' traces, analyze the dependence of each query on preceding queries. Using an LLM prompt, each query in the reasoning trace classified as either Independent, if it initiates new line of inquiry, or Dependent, if it builds upon earlier queries. Results highlight the difference in capabilities between open DR and proprietary models. Table shows that proprietary models issue significantly more queries (24-64 on average) than DeepResearcher (5-6), which also leads to more dependent queries (15-39 on average), going deeper over many topics, than DeepResearcher (5).
queries,
Branching and Backtracking. Both these operations are measured using GPT-4o (see Fig. 5 Number of branches can be considered as another aspect of breadth (beyond necessary queries); while backtracking denotes robustness of reasoning to incorrect reasoning paths. OpenAl DR has the highest number of branches, supporting its high necessary queries' coverage score; and suggesting the key role of breadth in DR answer accuracy. We do not significant difference in backtracking.
6 DISCUSSION AND CONCLUSION
Our evaluation points to few directions for improving DR. DR models do not perform well even when the algorithm is straightforward but the search may laborious (e.g.. ENTITIES); interleaving programmatic and model control may be helpful for such tasks. For other problems such as SCIFACTS, DR models would often yield the correct answer but wrong grounding, which may be addressed by better training. Finally, our trace analysis help guide tradeoffs of breadth and depth in DR.
REFERENCES
Akari Asai, Jacqueline He, Rulin Shao, Weijia Shi, Amanpreet Singh, Joseph Chee Chang, Kyle Lo, Luca Soldaini, Sergey Feldman, Mike D' arcyy et al. Openscholar: Synthesizing scientific literature with retrieval-augmented lms. arXiv preprint 21Xiv2241.11411 2024.
Henry Li Bytedance, Daniel Walnut. Deerflow: Deep exploration and efficient research flow. 2025.
Prahaladh Chandrahasan, Jiahe Jin, Zhihan Zhang, Tevin Wang, Andy Tang, Lucy Mo, Morteza Ziyadi, Leonardo FR Ribeiro, Zimeng Qiu, Markus Dreyer, et al. Deep research comparator: A plat- form for one-grained human annotations of deep research agents. arXiv preprint arXiv:2507.05495,
2025.
Dargaye Churnet. Patent claims revisited. Nw. J. Tech. & Intell. Prop., 11:501, 2012.
João Coelho, Jingjie Ning, Jingyuan He, Kangrui Mao, Abhijay Paladugu, Pranav Setlur, Jiahe Jin, Jamie Callan, João Magalhães, Bruno Martins, and Chenyan Xiong. Deepresearchgym: A free, transparent, and reproducible evaluation sandbox for deep research, 2025. URL https
//arxiv.
Hao Cui, Zahra Shamsi, Gowoon Cheon, Xuejian Ma, Shutong Li, Maria Tikhanovskaya, Pe- ter Christian Norgaard, Nayantara Mudur, Martyna Beata Plomecka, Paul Raccuglia, al. Curie: Evaluating Imms on multitask scientific long-context understanding and reasoning. In The Thirteenth International Conference Learning Representations.
10

---

Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36:28091-281 14, 2023.
Alphaeus Dmonte, Roland Oruche, Marcos Zampieri, Prasad Calyam, and Isabelle Augenstein. Claim verification in the age of large language models: A survey, arXiv preprint arXiv:2408. (000:2400.143177.. 14317, 2024.
Mingxuan Du, Benfeng Xu, Chiwei Zhu, Xiaorui Wang, and Zhendong Mao. Deepresearch bench: A comprehensive benchmark for deep research agents, 2025. URL httttsss//aarvv..or///////////
11763
Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. Eli5: Long form question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3558-3577,1 2019.
FutureSearch,
Nikos I. Bosse, Jon Evans, Robert G. Gambee, Daniel Hnyk, Peter Mühlbacher, Lawrence Phillips, Dan Schwarz, and Jack Wildman. Deep research bench: Evaluating ai web research agents, 2025. URL /
06287
Ameya Godbole, Nicholas Monath, Seungyeon Kim, Ankit Singh Rawat, Andrew McCallum, and Manzil Zaheer. Analysis plan-based retrieval for grounded text generation. arXiv preprint DrXiv:2408.10490 2024.
Google. We're expanding gemini 2.5 family of models. 1aaiiiiiggemii-----mmee lereell-fflllyeeepppeee 1x-elydds//// a. Accessed: 2025-08-04.
http://ooo..goggle/rrooouts/
Google. Deep research now available on gemini 2.5 pro experimental, 04.
b. Accessed: 2025-08-
Jacqueline He, Howard Yen, Margaret Li, Shuyue Stella Li, Zhiyuan Zeng, Weijia Shi, Yulia Tsvetkov, Danqi Chen, Pang Wei Koh, and Luke Zettlemoyer. Precise information control in long-form text generation. arXiv preprint arXiv:2506.06::
2025.
Dan Hendrycks, Collin Burns, Anya Chen, and Spencer Ball. Cuad: An expert-annotated nlp dataset for legal contract review, arXiv preprint arXiv:2 100.:060300668 2021.
Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6609-6625, 2020.
SU Hongjin, Howard Yen, Mengzhou Xia, Weijia Shi, Niklas Muennighoff, Han-yu Wang, Liu Haisu, Quan Shi, Zachary Siegel. Michael Tang, et al. Bright: A realistic and challenging benchmark for reasoning-intensive retrieval. In The Thirleenth International Conference on Learning Representations.
Yuxuan Huang, Yihang Chen, Haozheng Zhang, Kang Li, Meng Fang, Linyi Yang, Xiaoguang Li, Lifeng Shang, Songcen Xu, Jianye Hao, Kun Shao, and Jun Wang. Deep research agents: A systematic examination and roadmap, 2025. URL https://aax: horg/abs/2en 18096
Aymeri HuggingFace: Roucher, Albert Villanova del Moral, merve, Thomas Wolf, and Clémentine Fourrier. Open-source deepresearch freeing our search agents. htoo:/ppendeep--eeeeae 2025.
co/
Aaron Hurst, Adam Lerer, Adam Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
Hao Kang and Chenyan Xiong. Researcharena: Benchmarking l1ms' ability to collect and organize information as research agents. arXiv e-prints, pages arXiv-2406, 2024.
Satyapriya Krishna, Kalpesh Krishna, Anhad Mohananey, Steven Schwarez, Adam Stambler, Shyam Upadhyay, and Manaal Faruqui. Fact, fetch, and reason: A unified evaluation retrieval- augmented generation, 2024.
12941
11

---

LangChain. Open deep research.
2025.
Xiaoxi Li, Jiajie Jin, Guanting Dong, Hongjin Qian, Yutao Zhu, Yongkang Wu, Ji-Rong Wen, and Zhicheng Dou. Webthinker: Empowering large reasoning models with deep research capability. arXiv preprint arXiv:2504.2177, 2025.
Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Perey Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157-173, 2024.
Dasha Metropolitansky and Jonathan Larson. Veritrail: Closed-domard hallucination detection with traceability. arXiv preprini arXiv:2505.217886 2025.
Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: benchmark for general ai assistants. In The Twelfth International Conference Learning Representations, 2023.
Microsoft and Jared Spataro. Introducing researcher and analyst in microsoft 365 copilot. aatrruuuinrreeeaeeeraaaaaaaaaaan 2025-07-31.
2025. Accessed:
OpenAl. Introducing gpt-4.1 in the api. https///opeaa..cmm/dddd////tt-1 Accessed:
2025-08-04.
OpenAl. Introducing openai o3 and o4-mini.
https:://opena..com/inexx
darroducing-o--add-----iiiii b. Accessed: 2025-08-04.
OpenAl. Introducing deep research.
https://openaa..cm/mnnexx
introddccin--deep-rsseeerrr 2025.
PerplexityAL Perplexity deep research. httos://ww.prrplxxty..//////b/bu
a. Accessed: 2025-08-04.
PerplexityAI. Sonar pro. http:///ossppppaaaayy//tteeeeeer
sonar-pro b. Accessed: 2025-08-04.
PerplexityAL. Sonar reasoning.
models/sonar-proo c. Accessed: 2025-08-04.
Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, et al. Humanity's 's last exam. arXiv preprint arXiv:2501. 14249. 2025.
Varshini Reddy, Rik
Viet Dac Lai, Michael Krumdick, Charles Lovering, and Chris Tanner. DocFinQA: long-context financial reasoning dataset. Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 445-458, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 10116633w1/222444clsshtttt.. URL http:p/://aclaantoooogoooor///// wwwp//22.....ssoor..... acl-short. 42/
Yijia Shao, Yucheng Jiang, Theodore A Kanell, Peter Xu, Omar Khattab, and Monica S Lam. Assisting in writing wikipedia-like articles from scratch with large language models. arXiv preprint 2wXii:2402.14207. 2024.
Wenxuan Shi, Haochen Tan, Chuqiao Kuang, Xiaoguang Li, Xiaozhe Ren, Chen Zhang, Hanting Chen, Yasheng Wang, Lifeng Shang, Fisher Yu, et al. Pangu deepdiver: Adaptive search intensity scaling via oeen-web reinforcement learning. arXiv preprint arXiv:2505.24332. 2025.
Jianghui Wang, Vinay Joshi, Saptarshi Majumder, Xu Chao, Bin Ding, Ziqiong Liu, Pratik Prabhanjan Brahma, Dong Li, Zicheng Liu, and Emad Barsoum. Geak: Introducing triton kernel ai agent & evaluation benchmarks. arXiv preprint 2rXiv:2507.2311 2025.
12

---

Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. Browsecomp: simple yet challenging benchmark for browsing agents. arXiv preprint arXiv:2504.1253 2025a.
Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. Browsecomp: A simple yet challenging benchmark for browsing agents. arXiv preprint arXiv:2504112516, 2025b.
XAI. Grok 3 beta the age of reasoning agents. https://x.ainnews/ggr 08-04.
Accessed: 2025-
Renjun Xu and Jingwen Peng. A comprehensive survey of deep research: Systems, methodologies, and applications, 2025. URL http:ppp://arrk..ogg/////////// 12594
Tianze Xu, Pengrui Lu, Lyumanshan Ye, Xiangkun Hu, and Pengfei Liu. Researcherbench: Evaluating deep ai research systems on the frontiers of scientific inquiry. arXiv preprint arXiv:2507.16280, 2025.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Conference Empirical Methods in Natural Language Processing (EMNLP), 2018.
David Zhang. Open deep research. http::///atttbbb.cccc///////eeee
2025.
Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu. Deepresearcher: Scaling deep research via reinforcement learning in real-world environments. arXiv preprint arXiv:2504.03166, 2025.
13

---

A DETAILED SETUP
At the time of writing this paper, the proprietary DR models did not have API access. So evaluate these models in their chat interface by manually feeding in the DR tasks our benchmark. Typically, DR models in these chat applications respond with clarification questions in the beginning. We let the models use their best judgment with a fixed response to the effect of "Go Ahead".
For the baselines, we rely on the APIs provided by each model's provider. All hyperparameters kept at their default settings, except for the search context size and reasoning effort/budget parameters. For search context size, we chose "medium" for OpenAl and Perplexity baselines, with the Gemini API not providing an equivalent parameter. For reasoning effort/uuget,, we chose "low" for o4-mini, "medi um" for Sonar Research, and 128 tokens for Gemini 2.5 Pro each corresponding =330 seconds of thinking time by the respective models.
B EVALUATION: STRICT PRECISION AND RECALL
The strict version of the score takes the minimum instead of the mean the constituent claims' scores. This formulation is motivated by trust in the final report: user is unlikely to trust (even a correct) claim if any of its derivatory subclaims are incorrect.
Prec(A)= = min (c,(A:)) Prec(AA); Rec(AA) = min w;s(Ai) (ec(A:) iE(1.2...|AA)
ie :E(1.....1AA1)
where Prec(A,) Ree((A)) when subc(A;)] = 0 as before. Effectively, such score will be I only if the DR model's output exactly matches the list of dictionaries A: no claim is incorrect and no claim missing. Such an evaluation may be useful when asking for an enumeration of a certain class of entities, e.g., all movies that satisfy a constraint. One may consider a DR system's output failure if fails retrieve exactly the number of movies that satisfy the constraint.
C TASK CATEGORIES, UPDATION AND EVALUATION
C.I TASK CATEGORIES
C.1.1 SCIFACTS: SOURCING SCIENTIFIC FACTS
We create 31 queries designed to cater to scientists and researchers. This dataset has two distinct subsets (a) material identification, and (b) geospatial paper identification. Below we discuss both these categories in detail.
Materials. We create 17 questions to emulate scenarios where scientist might want to look for specific compound or material that perfectly fits a given set of criteria. Each question comprises a list of measured properties that maps perfectly to compound; and the task of deep research agent to successfully identify this compound, along with the source of information (academic paper). We leverage the mpve subset from the CURIE (Cui al.) dataset construct valid set of ground truth answers. The CURIE (Cui et al.) dataset consists of long-form comprehension and reasoning questions. Here, subset of questions require the model to enumerate all the measured properties of given material from a target paper. To convert this to a deep research query we leverage the ground truth measured properties and design the task such that the job of the model reference paper title, inference basis, and extract the relevant tables and passages from the reference paper. We compute final metrics based on the material name and paper title.
identify the material,
Geo. We create 19 questions for paper identification to mirror the scenario of researcher attempting to search for citations that precisely utilize given set of geospatial analysis datasets. The task of the deep research agent is to find papers that utilize all the datasets listed in the question. Similar to "Material Identification" (above), we utilize CURIE (Cui et al.) to construct our ground truth corpus from their geo subset consisting of papers from the geospatial analysis field. The geo task in CURIE involves listing all datasets used in given research paper, and we invert this task to construct DR question find the paper.
14

---

Ground-Truth Claims. In the SCIFACTS Geo task, the ground truth is provided list of paper titles which use the datasets mentioned in the original query. Similarly, in SCIFACTS Materials task, provide a list of material names (claims) and associated supporting evidence such as paper titles (subclaims) as the ground truth and compute the metrics according to Eq. [ Since the material identification task consists of both the main claim and the sub-claim, it is significantly harder than the geo task.
C.1.2 NOVELDS: IDENTIFYING DATASET PAPERS
We create 20 questions based on dataset papers, drawing from both our own domain knowledge and the dataset papers accepted at ICLR 2025. This category contains three types of tasks, (a) Dataset Identification, (b) Dataset Identification and Extraction, and (c) Dataset Peer Retrieval. Unlike the Geo category from SCIFACTS, here the goal is output dataset(s) and their associated properties.
Dataset Identification. These questions are designed to emulate scenarios involving researcher looking for a dataset for specific use-case. For each dataset paper, we identify its core contributions and the aspects that distinguish it from related work, to create open-ended queries that human might realistically pose when unaware of the dataset's existence but seeking something with those specific characteristics. The goal is to reflect genuine information need, rather than merely paraphrasing the abstract. The task requires the model not only to identify the appropriate dataset, but also to extract basic metadata such as its publication year, venue, and link to the dataset. Notably, failing to account for even one of the dataset's 's unique attributes often results in the retrieval of a similar yet ill-fitting alternative.
Dataset Identification and Extraction. These questions build upon the Dataset Identification category by requiring not only the identification of the correct dataset but also the extraction of specific findings from the dataset's paper. These questions mimic those researcher might pose when seeking results or findings for specific use case, without being aware of the dataset's existence. Extracting these findings may, for example, involve reasoning over results in a table, reasoning over multiple tables, or interpreting a figure in a paper.
Peer Dataset Retrieval. Some dataset papers include literature surveys or comparison tables highlighting peer datasets with similar goals, and slight variations in task settings. We use these sections to construct questions that reflect shared research directions across multiple datasets, without referencing any dataset by name. These questions emulate real-world exploratory scenarios which researcher is familiar with the broader problem space but not with specific datasets. The task requires the model to search and reason given high-level description to find set candidate datasets, and then to articulate the key features that define and distinguish each one.
Ground Truth Claims, For the NOVELDS Identification task, we provide the ground-trutt -tuut paper title, publication year, venue, and dataset link required by the queries. Here, paper title serves as the main claim, while the year, venue, and link form the subclaims. For NOVELDS Identification and Extraction and NOVELDS Peer Retrieval, we provide list of ground-truth findings peer datasets required by each query. Each item in the list includes a set of main claims (e.g., "task", "model-name". "dataset-name"), with supporting details such performance metrics or dataset attributes serving as subclaims. Precision, recall, and Fl scores are computed based on the overlap of the model outputs with that of the ground-truth.
C.1.3 PRIORART: DECONSTRUCTING PAPER ABSTRACTS
We devise 17 novel tasks motivated by the laborious problem of validating (or invalidating) claims in patent applications Churnet (2012). Tyypccal response to patent application consists of list of claims that are accepted and those that are rejected. For each rejected claim, the response also includes citations that are prior art that already demonstrate the claims in whole in parts, thereby challenging the grounds for patenting. To mirror this real-world scenario, we create DR tasks that involve deep reasoning over synthetic research paper abstracts.
The task is to reason over the given paper abstracts, manually written by us to mimic the paper abstracts in standard ML venues. Each abstract combines ideas and contributions primarily from recent ML papers in ICLR 2025 and ICML 2025 conferences (including accepted and rejected papers). The goal is to understand the key ideas in given abstract and identify papers that were the
15

---

source of those ideas, i.., that directly or indirectly provide evidence to support the ideas. Each task requires the model to list the claims in the abstract, references (paper titles and links), and rationales for why the references demonstrate the claims.
Dataset creation. We rely on OpenReview discussions to create the tasks. We start by identifying papers in ICML 2025 and ICLR 2025 venues OpenReview where there is sufficient engagement from the reviewers and the authors during the rebuttal phase. For each of the identified papers, we start with the paraphrasing of the contributions from the meta-reviewer, reviewers, and in some cases, the authors themselves. This becomes the seed for the synthetic abstract in the task. We find that this seed already challenging to identify, as the paraphrasing is often nuanced, highlighting aspects beyond the corresponding published paper abstract (in particular, lexical matching of the seed text with the original paper abstract wouldn't be helpful). We then go through the discussions to identify closely related papers pointed out by the reviewers. We use the discussions as well as the contributions stated in the text of the related papers to augment the synthetic abstract for the task.
For instance, if the seed text is about method that works under the setting of Scenario A, and the reviewers point out a closely related paper that solves the same problem Scenario B but using a modified method, then we write the synthetic abstract to subsume both the scenarios as well as the modifications of the method as purported contributions.
In most cases, the related work pointed out by the reviewers papers that are not already cited by the paper at hand. Furthermore, in some cases, we found a few other papers that also are closely related to some the claims in the abstract. In those cases, we manually added those references the ground-truth Overall, this forms a challenging DR problem- - the models cannot simply follow the references in the related work of one paper to get all the source papers for the task. In this way, our tasks that require deconstructing paper abstracts are more nuanced than standard DR tasks that elicit literature review or drafting related work given a topic.
Ground- Truth Claims. The ground-truth consists of top-level claims, where each claim corresponds to idea or a contribution discussed in the abstract. Each claim is supported by the sub-claims consisting of ground-trutt paper title, link, and a connection field that describes how the claim is demonstrated in the referenced paper. The DR model expected to produce the same format in its output. We compute precision, recall, and F1 scores based on the overlap of the model-reported references with that of the ground-truth.
C.1.4 FLIGHTS: IDENTIFYING FLIGHT INCIDENTS
We construct questions based on official flight investigation reports published by national aviation authorities (e.g., the NTSB, AAIB, and BEA). Each question is derived by identifying a flight incident with unique features such as an unusual failure mode, rare technical error, or a non-standard response procedure. We then invert the task: the DR question involves identifying the correct flight incident from a high-level description, elaborating on its causes or timeline.
Dataset Creation:
These questions emulate scenarios in which members of the general public, aviation enthusiasts, or journalists seek information about notable incidents without knowing the specific flight number. As official incident reports often span hundreds of pages, the task requires processing long documents and synthesizing information across multiple sections.
Ground-Truth Claims. We provide a ground-truft list of the incidents' timelines or causes, as required by each query. Each item in the list includes set of main claims (c.g., "timestamp", "flight (aahht--rammttrr") "), with supporting details such as sensor readings, flight configurations, anomaly descriptions serving as subclaims. Precision, recall, and F1 scores computed based on the overlap of the model outputs with that of the ground-truth.
1.5.5 ENTITIES: ENUMERATING ENTITIES WITH CONSTRAINTS
We devise 20 questions pertaining to global culture that require consulting a broad range of sources and cannot be accurately answered using small set of sources the Web. These questions focus on search intensity. They ask for an exhaustive list entities within a given category that satisfy all specified criteria. All questions in this category have fixed reasoning depth, ranging from 2 to 4.
16

---

They centered around real-worth events such award shows, the Olympics, book publications, and similar topics.
Dataset creation. We select world events that happen at regular cadence and design questions that cannot be fully answered using any single webpage or a small set of webpages. Each question in this category requires examining wide range of items, typically between 80 and 140, and filtering them based on the provided criteria. The final answers consist of 8 to 30 items that meet all specified requirements.
Ground-Trune Claims. Each question requires a list of entity names (claims) as output. We construct the ground truth using scraping scripts tailored each world event, followed by manual verification of agent-gennraae answers, particularly when they are not present in the initial ground truth. After reviewing all agent responses and adding any valid missing answers, we evaluate performance using precision, recall, and Fl scores against the finalized ground truth list.
C.2 LIVEDRBENCH: ENABLING BENCHMARK UPDATION WITH NOVEL QUESTIONS
We created SCIFACTS queries using Curie Cui et al. by extracting key information from scientific papers and posing queries to use this key information to retrieve the corresponding source paper. Future queries can be similarly generated by applying the same pipeline to newly published papers.
The NOVELDS Peer Retrieval questions can be extended by identifying literature survey tables in newly published papers, and extracting the listed datasets along with their distinguishing features.
For ENTITIES, we include four categories of queries based on distinct world events - movics, book publications, international math Olympiads, and the Olympics. More queries can be generated by selecting a similar event-based category, defining a set criteria, and iteratively scraping all items within that category satisfying the defined criteria.
C.3 EVALUATION
We structure each task so that it expects a list of dictionaries as the output. Each top-level dictionary corresponds to claim and its keys correspond to sub-claims. For instance, for the material identifica- tion task in SCIFACTS, identifying the material name is the claim while identifying the source paper correctly is the sub-claim that supports it. For each key in dictionary, we define the agreement score s E [0, 1] as a measure of the accuracy of the claim represented by the key's value.
For convenience in evaluation, we include an instruction to output JSON object in every DR prompt This avoids the need to parse the output report. However, some DR systems such as Gemini were unable to follow the instruction and still output report. Therefore, our evaluation pipeline includes an additional step of parsing the keys from a given report. While this can be done by prompting L.M such as GPT-4, for the purposes of benchmark evaluation, decided do it manually to avoid any possibility of error.
Metrics. For each benchmark category, we evaluate the correctness (precision) of the stated claims and completeness (recall) of those claims. We first compute the agreement score s of a generated claim with its ground-trut truth counterpart (discussed in Section 3.2) using GPT-4o (prompts included in Appendix E). Then, we compute precision and recall metrics as defined in Equation (.. We also report the Fl score combining these two metrics.
However, there is limitation of the above strategy. While we considered the uniqueness of the answer while creating each DR query, it is possible that multiple correct answers exist. As a result, given only the ground-truth claims, the recall metric may miss correct claims that were not part of the task's evaluation rubric. Similarly, the precision metric may falsely consider a claim incorrect since it was not present in the task's ground-truth answer. Therefore, we follow a hybrid strategy where we manually evaluate all claims generated by all DR systems under evaluation. If any of the generated claims correct, we add them to the ground-trutt set of claims. After this expansion of the ground-truth, we do an automated evaluation against the expanded ground-truth. Our final benchmark data includes this expanded ground-truth for each question.
This format instruction can he easily omitted allow evaluation without the JSON format requirement.
17

---

D EXAMPLE TASKS FROM LIVEDRBENCH
In this section, we provide examples from each category (Table the LIVEDR
SCIFACTS Materials Prompt
Find the material(s) that satisfy every of the listed measured properties.
Instructions
1. In your response, provide a list of JSON object(s) in the following format:
"material": "smaterial name>", " innference_basiss: "<brief explanation of how the listed properties match this material>", "paper_title": "<title of the paper>", sroperry_sourceettalee "<source table from which material name, property and property descriptor were identified>", "property_ supperty_ourceepaasse "<source passage from which material name, property and property descriptor were identified?"
Your inference basis should reference the key properties (e.g., band gap, absorption coefficient, crystal structure) that led you that material.
3. You must provide a valid json file even if you don't have the answer.
Measured properties:
Band gap Eg (room temperature): 3.35 eV
binding energy: 60 meV
Lattice constant (X-ray diffraction XRD): 0.325 nm
Lattice constant с (X-ray diffraction XRD): 0.521 nm
Lattice fringe (High-resolution transmission electron microscopy HRTEM); 0.52 nm
SCIFACTS Geo Prompt
Find the paper(s) that use all of the datasets mentioned below:
Google Earth Engine, Shuttle Radar Topography Mission data, Ground Truth Data, Moderate Resolution Imaging Spectroradiometer (MODIS) Data
Instructions:
You must respond with list JSON object(s) in this format:
"paper_title"" "<full paper titles"
2. You must provide valid json file with empty fields even if you don't have the answer.
18

---

NOVELDS Identification Prompt
I'm looking for diffusion based approaches that enable precise control over illumination in images, particularly those that move beyond simple prompt engineering. I'm especially interested a method that uses a diverse image collection, and incorporates physically grounded constraints-
consistency under blended lighting conditions- training, to allow models to respect the structural coherence light transport. I'm especially curious about methods where the consistency is strong enough infer structural cues like surface normals purely from illumination variation across synthetic views, without being specifically trained for this property. Find paper, ideally published one, that does this. Once you've found a paper that meets my needs, please provide the information in the following format:
titlee <title?,
venuee: <venue>,
"year": <year>,
NOVELDS Identification and Extraction
I'm looking for GPT-4 generated corpus of decisions rooted in quotidian life think commuting, family squabbles or career decisions each tagged against broad socio- psychological dimensions. I'm specifically interested in scenarios where the resolution hinges not objective correctness, but instead on personal values. How does this corpus reveal GPT-4's implicit generation bias across the various value dimensions in each of the pecioopsycooloiical frameworks explored in the corpus?
Provide the overview of GPT-4's generation bias in the following json format:
"framework": <framework_name miost_biasedd.dmmnssl <most sustt_baasee www.eddd.ieeeeeeee...eo..
19

---

NOVELDS Peer Retrieval
I want to compare existing 3D urban segmentation datasets based on real-world scenes that go beyond static 2D maps or flat segmentation overlays, and instead represent urban environments as richly annotated point clouds. How do they compare in terms of data acquisition methods-such as mobile laser scanning. terrestrial laser scanning, aerial laser scanning, and photogrammetry?
For the datasets you find, provide the following information in json format:
name": : <name of the resource?, data intta_aqiisitio_.ee
<data acquisition method>, arrea: <area of the dataset>, "scenes": <number of scenes>, moiitts_iillio"". <number of points?,
}.
ENTITIES Prompt
List all movies released between **2005 and 2011** (inclusive) that have won Oscar in any category**, and are **adattations of works authored (or co-authored) by women** Please provide the answer in **JSON f where each entry includes the following three fields:
"Movie Name" - the title of the film
"Original Work" the name the source material the movie based on
"Author Name" - the name(s) of the female author(s) of the original work
FLIGHTS Prompt
In which high-profile flight investigation military radar data later considered unreliable? Identify the types of radar data that was recorded, what anomalies were found, and why those anomalies raised doubts. Focus reconstructing timeline of anomalous readings and critically examine the underlying causes. For each data type and anomaly, follow this format:
"timestamp_ute" "<timestamp>",
parameter": "xparameter_names
ficcodde__valee "<value>",
"anomaly "=description_of_nee _nnomalyy"
20

---

PRIORART Prompt
have the following ideas for research paper. Can you help identify if this has already been done or implied in full or in parts in other papers? Give your answer as **JSON with three fields: Paper title, link, and connection a field** that quotes exact sentences from the paper and parts of my ideas below to make the case.
Here is the format:
"title": <paper title>, "link"; <link to the paper>, "connection": <connection field>
The objective our paper to understand optimized prompts (e.g. using black-box white-box prompt optimization techniques) better in terms of how they differ from human-written forms, and how they help LMs do better predictions, sometimes therefore also enabling jailbreaking, etc. We formulate adversarial attack form of controllable text generation. Our method injects attack information during the decoding steps, enabling successful attacks.
The key idea is to automatically generate "suffix" (a sequence tokens) to concatenate to the user imput (user prompt, task description, etc.). The tokens will be generated one by one left-to-right by maximising mixed search objective, that uses gradient also account for readability of the prompt. Readability in the objective is important especially because a common complaint against optimized prompts is that they often contains illegible text, punctuations, etc.
vocabulary and
Once we have such an optimized readable prompt, we argue that it consists of influential, specific,
tokens that help better elicit the user behavior, in some cases malicious behavior case of a jailbreaking attempt. Further, we show that the optimized prompts have distinct internal embeddings (using sparse probing techniques), not just lexical behaviors. Notably, we show results on two new attack scenarios (inducing system prompt leakage and addressing over-censored tasks in LLMs) that have not been explored before.
E EVALUATION PROMPTS
Here, we provide the exact prompts used for evaluating the generated outputs. Across all our experiments, leverage GPT-4o Hurst et al. (2024) the judge model and compare the generated answer with the ground truth. To ensure the ground truth is comprehensive, we conduct additional human checks and augment with valid answers that were initially not present. Our benchmark is built on this enriched ground truth set.
For SCIFACTS Materials task evaluate each key (paper title and material name) present in the generated list of dictionaries against the prompt given below and report the combined metric according Eq. I We use the same evaluation prompt to evaluate the paper title in SCIFACTS Geo task.
Prompt for evaluating each key SciFacts
You are an evaluator. Given Golden Answer and a Predicted Answer, check whether the Predicted Answer correct. The prediction is correct if it fully aligns with and contains all the key information present in the Golden Answer. Respond with True the prediction is correct and False otherwise.
Golden Answer: ggolden-answrd
Predicted Answer: (predicted-
21

---

For the NOVELDS and FLIGHTS queries in which both the prediction and ground truth consist of lists of dictionaries, we begin by identifying the corresponding ground truth dictionary for each predicted dictionary. This is done using set of primary keys to be used for matching, using the following prompt:
NO and FLIGHTS prompt to match predicted dictionary against list of ground truth dictionaries using a set of primary keys
You are given a list of dictionaries (the ground truth) and a single predicted dictionary (the model output). Your task is to determine which dictionary in the ground truth list corresponds with the predicted dictionary, using set of primary keys to identify the best match.
You will be given list of primary keys that can be used to compare the dictionaries. If primary key is not present in two dictionaries, make sure to use the other primary keys to determine the match.
Do not consider ANY other fields in the dictionaries except for the primary keys. the value is string, consider them equivalent if they similar in meaning, or if one is subset of the other.
Match based on semantic similarity if one subset of the other, not exact string equality.
Acronyms, shortened names, or partial names are equivalent their expanded forms. "BERT" and "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding" ~ equivalent
"ICLR" and "International Conference on Learning Representations" equivalent A prediction may include extra info (c.g., a year or qualifier). If the ground truth is a subset and meaning is preserved, treat them equivalent. If the ground truth includes qualifier, the prediction's 's qualifier must match present.
You may also be provided with set of extra evaluation notes that provide more specific evaluation criteria. These extra rules may override the general rules above. If after all this match exists, return empty dictionary.
Now compare the following:
Ground Truth List: {list-gt}
Model Output: (dict-pred}
Primary Keys: (primary-keys)
Extra Evaluation Notes: {note}
Your output should be the dictionary from the ground truth list that best matches the model output, an empty dictionary no match is found.
IMPORTANT:
Do NOT explain your reasoning. Do NOT include any extra text. Only output the final dictionary JSON object. Do not wrap it in markdown say anything else.
With corresponding ground-truth dictionary obtained for each predicted dictionary in the NOVELDS and FLIGHTS queries, we evaluate the entire predicted dictionary against the ground truth dictionary, rather than key by key. This is because related sub-claims provide important context, and evaluating keys individually can lead to under-scoring by the LLM judge. The exact prompt used for evaluation provided below:
22

---

NOVELDS and FLIGHTS prompt for evaluating predicted dictionary against a ground truth dictionary
You are evaluating whether two dictionaries refer to the same information. Your task is to determine whether each corresponding value is equivalent.
For each key-value pair in the ground truth dictionary:
If the key is absent in the model output dictionary, it is not equivalent.
If the value is a number (integer or float), they are equivalent within 1 % margin of error.
If the value is string, consider them equivalent they similar in meaning, or if one is subset of the other.
Acronyms or shortened names are equivalent to their expanded forms.
"BERT" and "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding" ~ equivalent
"ICLLR" and "International Conference on Learning Representations" + equivalent
"CNNN" and "ResNet" - equivalent (both are convolutional neural networks) -"Adam" and "SGD" not equivalent (different optimizers)
"0.8" and "0.80" + equivalent numerically
If the value is URL, they must be exactly the same, shortened version of the same URL.
If the value is list, check if every element in the ground truth list is present in the model output list, regardless of order. Follow the same rules for strings as above. If the value present in the model output dictionary, regardless of order. Again, follow the same rules for strings as above.
dictionary, check if every key-value pair in the ground truth dictionary is
Some values may depend on information from other keys in the dictionary. Use such context determine whether two values are equivalent, even they are identical on their own. Ignore trivial formatting differences such as casing, whitespace, common abbreviations if the meaning is preserved.
Now compare the following dictionaries:
Ground Truth: {dict-gt)
Model Output: [dict-pred)
Your output should be JSON object with the same keys as the input dictionaries. For each key, the value should be:
3: The predicted value closely matches the ground-truth, preserving almost all important information with minimal or no loss meaning.
2: The predicted value captures the main idea of the ground-trute but omits key details has minor inaccuracies.
1: The predicted value shows some overlap with the ground-truie but misses most of the essential meaning.
0: The predicted value is largely incorrect, unrelated, does not correspond meaningfully to the ground-truth value.
IMPORTANT:
Do NOT explain your reasoning.
-Do NOT include any extra text.
-Only output the final ISON object. Do not wrap in markdown or say anything else.
Finally, evaluate PRIORART and ENTITIES using the prompt below. Here, the LLM judge is required to check if the predicted string is present in the list of ground truths (paper titles and entities like names of people movies). We ensure the minor differences in formatting ignored.
23

---

PRIORART and ENTITIES prompt to match predicted string against a list of ground truth strings
You are evaluating whether a predicted string is present in ground-truth list of strings.
Allow for minor differences in formatting, such as casing, whitespace, or common abbreviations.
If found, return the matching string from the ground truth list in JSON format:
"match": matched-strinn
If not, return empty dictionary.
Now compare the following:
Ground Truth List: {gt-list}
Predicted String: (pred-str)
IMPORTANT:
Do NOT explain your reasoning.
Doo NOT include any extra text.
Only output the JSON object. Do not wrap it in markdown say anything else.
F JSON EXTRACTION PROMPT
Prompt for extracting ISON fields from a DR report
You are JSON extraction assistant.
You are given:
1. A JSON structure which can be either a list of values, or list of dictionaries with keys.
A textual report that contains information corresponding to the JSON structure.
Your task:
Extract the value(s) from the report for each element key in the JSON.
Fill in the ISON with the extracted values.
If value cannot be found in the report, it null.
Respond only+* with the completed JSON, no extra text or explanation.
Process the following:
Report:
(report)
JSON structure:
(json_schema)
24

---

G
TRACE ANALYSIS PROMPTS
Prompt to get nescessary queries given a DR question
Given question, extract the important search queries necessary to answer it using web search. Focus queries without which the question cannot be answered fully.
Output as JSON with two categories: cearaattrrstic-saarhhee
Search queries related the unique aspects of the question. These will be used to find source paper/report that contains the answers to the question. srractive-searchess You will see that the question asks for specific fields be extracted into JSON object. Come up with search queries aimed at extracting specific this information from the source paper/report. Think of searches for each JSON field and sub-field that the question asks for, and nothing else.
Example
Question: Find a paper which introduces an image dataset designed to promote the develop- ment of autonomous deep learning systems for detecting and classifying waste in complex, real-world, in-situ environments.
What are the top 10 classes/categories by number of annotations in the dataset? Provide them in the format:
{ "class-category": <clas--ategory--aaee, )
"class-cateegry": <class-category--aaee,
]
What baselines used by the dataset? Provide the baselines used by the dataset in the format:
:dttase': <dataset-name>
'model': <model-name>>
<test-1>; <test-1-van -vauu>,,
},
]
Output:
{
characteristiic-sarreess
"image dataset for waste classification", "autonomous deep learning system for waste detection", "real-world, in-situ waste data collection",
},
{
"extractive-sserchhe"
"top classes by number of annotations", "baseline models and benchmark results", "evaluation metrics used"
}
Question {question]
IMPORTANT
Ensure the search terms comprehensive and explicit as possible. Avoid vague generic terms that don't 't add concrete search value. Avoid redundancy; each search term should cover distinct aspect.
25

---

Prompt for checking if a query is present in a DR model's trace
Given a search and reasoning trace from a language model, figure out a particular query was invoked by the model during its reasoning process. It doesn't matter if the model got the answer wrong, just figure out it was invoked or not. It also doesn't need to an exact match, but every important aspect of the query should be present in the invoked query. Output Format:
"excerpt": <the verbatim excerpt from the trace that indicates
the query was invoked, or if not found>, "invoked"; true/false,
}
Trace:
[trace]
Query: {query)
IMPORTANT:
OOnly output the final dictionary as JSON object.
Do NOT explain your reasoning.
26

---

Prompt to get dependent and independent queries from model's trace
You are subquery analysis assistant.
You are given:
A trace: the reasoning steps prior context that has already been carried out.
A paragraph: a passage containing one or more possible queries.
Your task is:
Enumerate all subqueries in the paragraph. subquery is any distinct request for information or lookup implied by the text. There may be multiple subqueries, or none at all. Dependent:
For each subquery. decide whether is Independent
Dependent: The subquery builds on, refers to, or modifies something in the trace (e.g., comparison, continuation, or follow-up based on what the trace already covered). Independent: The subquery initiates a new line of inquiry that does not rely on the trace.
Step 1: Enumerate Subqueries Extract distinct subqueries from the paragraph. Each subquery should standalone question request for information.
Step 2:
Analyse with respect the trace
For each subquery, analyse its dependency on the trace. If trace is empty, then subqueries "Independent".
Step Classify Subqueries Finally, for each subquery, assign a dependency label: "Dependent"
"Independent"
Output Format
"subquery": "[text the subquery]",
dddeendency_.aallyyeee "[analysis]",
"dependency": MonependenttDepeedeee
},
"subquery": "[text of the subquery]",
eeependeenc_anallyiiie "[analysis]",
"dependency": gadepenneentDDepeneeed
Trace: [trace}
Paragraph:
{paragraph}
27

---

Prompt to count the number of branches a given DR trace
You are an expert analyzing research investigation traces. Your task is to read the provided trace and identify how many distinct research *branches* lines of inquiry the author explores. A new branch is a distinct hypothesis, search direction, or subgoal. Count how many times the researcher changes focus, searches for different types of information, or explores parallel possibilities.
Output ONLY the number of branches as an integer.
Prompt to count the number of backtracking events in given DR trace
You are an expert analyzing research investigation traces. Your task is read the provided trace and count how many times the researcher "backtracks* - that is, revisits or reconsiders previous ideas, changes direction after realizing approach is suboptimal, or discards line of inquiry.
Look for transitions like:
'I first thought... but then...'
"Initially considered... later found...'
'Switched from... to...
There may or may not be any transition phrases, just count the number of ideas or directions changed.
Count only actual *changes in direction reconsiderations** Output ONLY the number of backtracking steps as an integer.
H QUALITATIVE ANALYSIS
We present few qualitative examples showing the necessary queries that the DR models failed to invoke.
Example Question I
I want paper that introduces a dataset and method to test image editing models. In my setting, don't have explicit instructions edit the images, but have similar edits done on another image and I want the edit to carry over. It would be great if the method is efficient, and does not require any optimization whatsoever.
Once you've found a relevant paper, please provide the following information in JSON format:
"paper_title"; title": "<title of _ateoof__hhhhhaeeeeeeeeeeeee the__haper"""" "is_published": <true/false>,
"venue": <venue_name:
"year": <publication "dataset_link".. "<link_to. to_ thine_to_ttheae the "ataastee_tt"" }
For this question, only Gemini DR invoked a query for "method for transferring edits between images". while OpenAl DR, Perplexity DR, and both variants of DeepResearcher failed to invoke it.
28

---

Example Question 2
Identify a publicly available dataset of long-form in-the-wild videos that are segmented into scenes and richly annotated with both content descriptions and user engagement signals. The dataset should include multimodal information and support analysis at the video level as well the scene level.
Once you'l "oe've found dataset paper, provide its information in the following JSON format:
"title": <title of the dataset paper>, "is_published": <true/false>, "venue": <venue>, // eg: CVPR "year": <year of publication>, "dataset_url"; <link to the dataset>, // if available, else null
For this question, only OpenAl DR and Perplexity DR invoked query for "in-the-wild video dataset with scene segmentation" ", while Gemini DR, and both variants of DeepResearcher failed to invoke
I ADDITIONAL RESULTS
OpenAl Deep Research
Perplexity Pro
Gemini Deep Research (2.5 Pro)
Research
Pree
SCIFACTS
Geo
0.715
0.728
0,721
0.263
0.144
0.186
0.405
0.259
0.316
SCIFACTS
0,313
0.316
0,314
0.264
0.105
0.150
0.072
0.013
0.022
Materials
NOVELDS
Identificatin
0.667
0.667
0.667
0.625
0.625
0.625
0.375
0.375
0.375
NOVELDS
Idenrification and
0.559
0.428
0,462
0.411
0.351
0.358
0.390
0.284
0.319
Extraction
NOVELDS Peer
0,849
0.498
0.609
0.535
0.248
0.322
0.558
0.241
0.334
Retrieval
0.813
0.534
0.673
0.373
0.494
FLIGHTS
0.525
0.530
0,523
0.385
0.358
0.369
0.243
0.264
0.252
Average
0.644
0.522
0.556
0,482
0.319
0.361
0.339
0.235
0.261
Table 6: Precision, recall, and Fl scores on LIVEDRBENCH for Deep Research models. Claim agreement score is evaluated by the authors.
1.1 CATEGORY- WISE RESULTS
We present category-wise results for the DR models and baselines in Tables 7 Among the baseline non-DR models, reasoning models struggle when their reasoning limit to =30 seconds, underscoring the complexity of LiveDRBench queries requiring intensive compute and planning capabilities. In particular, all reasoning models obtain FI score of zero for the SCIFACTS Materials dataset. The non-reasoning models perform worse, due to their limited ability to plan, conduct iterative searches, and backtrack. Interestingly, OpenAl's 04-mini performs comparably or even better than Perplexity DR and Gemini DR in many of the task categories, yet significantly behind OpenAl DR.
1.2 HUMAN EVALUATION
To check for errors due to GPT-4o based agreement scores, we also present numan--vvlluse precision, recall, and F1 scores across all categories in Table Overall, we find that the metrics computed using
29

---

Model
Precision
Recall
F1
Model
Precision
Recall
F1
OpenAl DR
0.715
0.728
0.721
OpenAI DR
0.313
0.316
0.314
Perplexity DR
0.263
0.144
0.186
Perplexity DR
0.264
0.105
0.150
Gemini DR
0.405
0.259
0.316
Gemini DR
0.072
0.013
0.022
DeepResearcher
DeepResearcher
0.000
0.000
0.000
0.000
0.000
0.000
+ DS Qwen 32B
DS Qwen 32B DeepResearcher GPT-4.1
DeepResearcher
0.000
0.000
0.000
0.000
0.000
0.000
+ GPT-4.1
OpenAl o4-mini
0.138
0.104
0.114
OpenAl o4-mini
0.000
0.000
0.000
Sonar Reasoning
0.000
0.000
0.000
Sonar Reasoning
0.000
0.000
0.000
Gemini 2.5 Pro
0.273
0.200
0.201
Gemini 2.5 Pro
0.000
0.000
0.000
OpenAl GPT-4.1
0.017
0.021
0.019
OpenAl GPT-4.1
0.000
0.000
0.000
Sonar Pro Gemini 2.5 Flash
0.000
0.000
0.000
Sonar Pro
0.000
0.000
0.000
0.059
0.059
0.059
Gemini 2.5 Flash
0.000
0.000
0.000
Table Comparison of DR models on the Scr- FACTS Geo tasks.
Table Comparison of DR models on the ScI- FACTS Materials tasks.
the claim agreement scores determined through manual verification by the authors closely match that of the results in Table This indicates that our benchmark can be used to reliably evaluate any new model using the evaluation prompts in Appendix E
Model
Precision
Recall
F1
Model
Precision
Recall
Fl
OpenAl DR
0.542
0.546
0.540
OpenAl DR
0.813
0.534
0.603
Perplexity DR
0.389
0.347
0.362
Perplexity DR
0.673
0.373
0.447
Gemini DR
0.249
0,276
0,261
Gemini DR
0,494
0.3
0.338
DeepRescarcher
DecpRescarcher
0.120
0.073
0.090
0.307
0.046
0.076
DS Qwen 32B
+ DS Qwen 32B DeepResearcher
DeepRescarcher
0.044
0.022
0.027
0.303
0.035
0.058
+ GPT-4.1
+ GPT-4.1
OpenAl 04-mini
0.340
0.282
0.304
OpenAl o4-mini
0.265
0.082
0.115
Sonar Reasoning
0.237
0.168
0.183
Sonar Reasoning
0.199
0.04
0.064
Gemini 2.5 Pro
0.228
0.210
0.215
Gemini 2.5 Pro
0,254
0.120
0.151
OpenAl GPT-4.1
0.200
0.138
0.160
OpenAl GPT-4.1
0,217
0.0725
0.074
Sonar Pro Gemini 2.5 Flash
0.216
0.187
0.194
Sonar Pro
0.075
0.035
0.042
0.182
0.105
0.117
Gemini 2.5 Flash
0.153
0.044
0.064
Table 9: Comparison of DR models and baselines on the FLIGITTS tasks.
Table 10: Comparison of DR models on the EN-
TITIES tasks.
1.3 SENSITIVITY ANALYSIS
To assess evaluation robustness, repeat the LLM-as-judge evaluation five times with temperature 0.3 and report the standard deviation across runs. Tables [4-19 19 summarize results using GPT-4o and Qwen3-32B (non-thinking) judges. Across experiments, variance remains low (typically < 0.02 absolute), with overall standard deviation ranging from 0.004 to 0.012. This stability is expected because the judge task consists of verification against ground-truth claims. Replacing GPT-4o with Qwen3-32B yields similarly low variance, indicating that the evaluation framework is robust judge model choice.
I.4 IMPACT OF REASONING STRENGTH
We evaluate no models with reasoning set to high, with and without improved prompts (Table Our primary objective is to demonstrate that reasoning intensity is a key driver of performance on LiveDRBench. To further substantiate this claim, we evaluate non-DR models with reasoning explicitly set to high, both with default prompting and with an improved (efficient) prompt designed encourage structured claim extraction and verification. Results are shown in Table 20
We observe
30

---

Model
Precision
Recall
F1
OpenAl DR
0.667
0.667
0.667
Perplexity DR
0.633
0.633
0.633
Gemini DR
0.400
0.400
0,400
DeepResearcher
0.167
0.167
0.167
DS Qwen 32B
DeepResearcher
0.233
0.233
0.233
GPT-4.1
OpenAl o4-mini
0.467
0.467
0.467
Sonar Reasoning
0.200
0.200
0.200
Gemini 2.5 Pro
0.444
0.444
0.444
OpenAl GPT-4.1
0.333
0,333
0.333
Sonar Pro Gemini 2.5 Flash
0.233
0.233
0.233
0.378
0.378
0.378
Table 11: Comparison of DR models and base- lines on the NOVELDS Identification tasks.
Model
Precision
Recall
F1
Model
Precision
Recall
OpenAl DR
0.795
0.480
0.585
OpenAl DR
0.694
0.463
0.539
Perplexity DR
0.494
0.248
0.311
Perplexity DR
0.689
0.339
0.419
Gemini
0.551
0,249
0.338
Gemini DR
0.106
0.096
0.082
DeepResearcher
DeepResearcher
0.083
0.028
0.042
0.525
0.136
0.199
+ DS Qwen 32B
DS Qwen 32B DeepRescrrcher GPT-4.1
DeepRescarcher
0.067
0.019
0.029
0,412
0.135
0.180
+ GPT-4.1
OpenAl o4-mini
0.632
0.254
0.345
OpenAl o4-mini
0.529
0.179
0.254
Sonar Reasoning
0.083
0.009
0.017
Sonar Reasoning
0,230
0.149
0.166
Gemini 2.5 Pro
0.334
0.140
0.196
Gemini 2.5 Pro
0.309
0.322
0.307
OpenAl GPT-4.1
0.465
0.196
0.276
OpenAI GPT-4.1
0,362
0.297
0.311
Sonar Pro Gemini 2.5 Flash
0.231
0.083
0.122
Sonar Pro
0.180
0.100
0.113
0.229
0.163
0.190
Gemini 2.5 Flash
0.272
0.228
0.227
Table 12: Comparison of DR models and base- lines on the NOVELDS Peer Retrieval tasks.
Table 13: Comparison of DR models and base- lines the PRIORART tasks.
substantial improvements when stronger reasoning and improved prompting are applied. For example, o3 improves from 0.539 0.666 F1 with an efficient prompt. These results support our hypothesis that higher reasoning intensity significantly improves performance on claims-based deep research evaluation.
I.5 VARIANCE AND STATISTICAL TESTS
To address concerns regarding robustness and statistical validity, we conduct additional analyses covering (i) alignment with human evaluation, (ii) per-example variability (iii) confidence intervals, and (iv) variance across multiple DR runs.
We evaluate agreement between GPT-4o and human evaluation using (1) Cohen's (following DeepResearchhyy (Coelho et al.) 2025) and (2) Mean Absolute Error (MAE). Human scores (Table(6) are compared with GPT-4o scores (Table 3). Three authors independently performed the human evaluation. Cohen's ranges from 0 (chance agreement) to (perfect agreement). Table 21 shows strong agreement (high k) and low deviation (low MAE) between GPT-4o and human evaluators establishing the efficacy of our model based evaluation. We also compute the Mean Absolute Error (MAE) between Human and GPT-4o Judge and present the results in Table 22 demonstrating low average MAE across the board.
(i) A Alignment with Human Evaluation.
31

---

Opent
Perpaociy
Sources
Number Backtracks
Number of Branches
20
60
40
100
1.0
20
0.15
Spurcs
Bacttracts
0.4
Brandds
0.10
0.3
0.2
0.2
0.05
0.1
0.00
Gemini DR
OpenAI DR
Perplexity DR
Gemini DR
OpenAl DR
Perplexity DR
Gemini DR
OpenAl DR
Perplexity DR
Figure 5: (Top) Box Plots illustrating the number of sources referred, branches considered, and backtracking events occurred against Fl score on the NOVELDS subset using three DR models: Gemini DR, OpenAl DR (OAI DR), Perplexity DR. (Bottom) Fl Efficiency per event.
Table 14: Std. dev. across 5 GPT-4o evaluation runs for OpenAI DR (temp=0.3)
Subset
Precision
Recall
SciFacts Materials
0.312 1 0.000
0.316 + 0.000
0.314 1 0.000
SciFacts Geo
0.715 0.000
0.707 = 0.026
0.711 0.026
NovelDS Identification
0.667 = 0.000
0.667 = 0.000
0.667 = 0.000
NovelDS Identi-Exxe Extrritittation
0.546 = 0.022
0.466 0.019
0.487 + 0.020
NovelDS Peer
0.791 = 0.012
+.000
0.581 + 0.009
PriorArt
0.694 + 0.000
0.463 0.000
0.539 0.000
Entities
0.791 0.012
0.517 0.008
0.584 0.010
Flights
0.560 + 0.029
0.556 + 0.022
0
Overall Avg
0.634 + 0.009
0.521 + 0.010
0.554 1 0.011
(ii) Per-Example Variability. We compute per-example standard errors across all DR models evaluated on LiveDRBench (Table 23). Standard errors remain low across models and metrics, indicating stable aggregate performance despite task difficulty.
(iii) Confidence Intervals. We further report 95% confidence intervals under (a) Gaussian assump- tions and (b) bootstrap resampling (Tables 25 Both approaches yield consistent intervals, confirming that our inferences are not sensitive to distributional assumptions.
(iv) Variance Across DR Runs. After API availability, we performed multiple runs of O3-Deep- Research on LivcDRBench subset (Entities). We follow recommended prompts, default reasoning settings, and set max tool calls=500. Results are shown in Table 26
I.6 METRIC DESIGN ABLATION
We consider the strict version of precision and recall metric for evaluation in this section. Specifically, in this setting, we compute min over all sub-claims predicted subclaims. In SaiFacts Geo, PriorArt, and Entities, there are only main claims with subclaims. In SciFacts Materials, there is single subclaim. In either case, computing a min over the subclaims has no effect on the evaluation. We present these alternative evaluation results in Tables DR models remains consistent under the strict min metric as well.
29
28
The trend among
32

---

Table 15: Std. dev. across 5 GPT-4o evaluation runs for Perplexity DR (temp=0.3).
Subset
Precision
Recall
F1
SciFacts Materials
0,211 + 0.012
0.105 + 0.000
0.141 + 0.000
SciFacts Geo
0.263 +: 0.000
0.145 1 0,000
0.187 + 0.000
NoveIDS Identification
0.633 +: 0.000
0.633 0.000
0.633 + 0.000
NoveIDS
0.369 0.017
0.015
0.378 0.017
NoveIDS Peer
0,553 0.033
0.261 0.008
+ 1:32:00001
PriorArt
0,689 + 0.000
0.339
0.419 0.000
Entities
0.612 1 0.000
1.000
0.419 0.000
Flights
0.415 + 0.026
0.388 1 0.024
Overall Avg
0.468 0.011
0.324 + 0.006
0.362 + 0.007
Table 16: Std. dev. across GPT-4o evaluation runs for Gemini DR (temp=0.3).
Subset
Precision
Recall
F1
SciFactts Materials
0.072 1 0.000
0.013 1 0.000
0.021 +: 0.000
SciFacts Geo
0,405 1 0.000
0.259 1 0.000
0.316 1 0.000
NovelDS Identification
0.400 = 0.000
0.400 0.000
0.400 + 0.000
NoveIDS Identi Extraction
0.430 0.017
0.339 0.016
0.362 0.015
NoveIDS Peer
0,526 = 0.020
0.241 0.007
0.010
PriorAnt
0.106 + 0.000
0.096 0.000
0.082 +
Entities
0.439 + 0.000
0.304
0.337 0.000
Flights
0.277 + 0.011
0.313
0.293 0.013
Overall Avg
0.332 0.006
0.246 0.005
0.267 0.005
Table 17: Std. dev, across 5 Qwen3-32B (non-thinking) evaluation runs for OpenAl DR (temp=0.3).
Subset
Precision
Recall
F1
SciFacts Materials
0,332 + 0.028
0.355 + 0.055
0.343 + 0.000
SciFacts Geo
0.715 = 0.000
0.728 + 0.000
0.721 + 0.000
NovelDS Identification
0.667 = 0.000
0.667 = 0.000
0.667 = 0.000
NovelDS dlenti-Extraction
0.506 0.009
+
0.438 + 0.009
NovelDS Peer
0.789 + 0.008
+ 0.476+0000
0.580 0.005
PriorAnt
0.714 + 0.000
0.473 0.000
0.552 + 0.000
Entities
0,811 + 0.000
0.000
0,600 + 0.000
Flights
0.589 1 0.023
0.572
0.574 0.019
Overall Avg
0.640 0.008
0.527 0.010
0.559
Table 18: Std. dev. across 5 Qwen3-32B (non-thinking) evaluation runs for Perplexity DR (temp=0.3).
Subset
Precision
Recall
F1
SciFacts Materials
0.264 + 0.000
0.164 + 0.000
0.202 : 0.000
SciFacts Geo
0.363 + 0.011
0.233 : 0.036
0.284 + 0.036
NovelDS Identification
0.633 + 0.000
0.633 + 0.000
0.633 + 0.000
NovelDS
0.358 0.016
0.018
0.369 0.016
NovelDS Peer
0.565 + 0.008
+ 0.003
0.337 + 0.004
PriorArt
0.704 1 0.006
0.351 1 0.004
0.432 1 0.005
Entities
0.666 1: 0.000
0.351 1 0.000
0.431 1 0.000
Flights
0.447 1: 0.042
0.381 0.036
0.405 1 0.038
Overall Avg
0.500 0.010
0.345 0.012
0.387 0.012
33

---

Table 19: Std. dev. across 5 Qwen3-32B (non-thinking) evaluation runs for Gemini DR (temp=0.3).
Subset
Precision
Recall
Fl
SciFactt Materials
0.075 1 0.003
0.013 1 0.000
0.022 1 0.001
SciFacts Geo
0,405 1 0.000
0.272 0.005
0.326 + 0.005
NovelDS Identification
0,400 1 0.000
0.400 0.000
0.400 0.000
NovelDS adenti-Extraciin
0.397 0.016
0.306 0.019
NovelDS Peer
0,552 = 0.006
02200
0.339
PriorArt
0,181 = 0.000
0
Entities
0,451 = 0.001
0.307
Flights
0.280 + 0.023
0.314 + 0.030
0.295 0.026
Overall Avg
0.343 0.006
0.251 + 0.007
0.275 0.007
Table 20: Non-DR models with high reasoning.
Model
Precision
Recall
F1
o3 (efficient prompt)
0.800
0.616
0.666
o3
0.645
0.492
0.539
mini (efficient prompt)
0.720
0.341
0.402
Table 21: Cohen's between Human and GPT-4o.
Comparison
Precision
Recall
F1
OpenAI vs. Perplexity DR
0.98
0.98
0.96
Perplexity vs. Gemini DR
0.86
0.92
0.86
Gemini vs. OpenAl DR
0.98
0.96
0.96
Table 22: MAE between Human and GPT-4o.
System
Precision
Recall
OpenAl DR
0.020
0.012
0.014
Perplexity DR
0.022
0.022
0.021
Gemini DR
0.035
0.027
0.028
Table 23: Per-example variability on LiveDRBench.
Model
Precision +:
Recall SE
OpenAl DR
0.633 + 0.031
0.523 0.034
0.555 + 0.032
Perplexity DR
0.466 + 0.037
0.317 0.029
0.355 + 0.030
Gemini DR
0.335 + 0.031
0.240 + 0.025
0.263 + 0.025
DeepResearcher + DS Qwen 32B
0.166 + 0.024
0.069 + 0.013
0,086 = 0.015
DeepResearcher GPT4.1
0.163 + 0.025
0.076 + 0.014
0.088 =\t 0.015
Table 24: 95% Confidence Intervals assuming Gaussian distribution.
Model
Precision Cl
Recall C1
Fl Score Cl
OpenAl DR
0.571 0.695
0.456 0.588
0.484 00.100
Perplexity DR
0,393 0.538
0.253 0.368
0.283 0.401
Gemini DR
0.277 0.399
0.191
-0.288
0.208 0.306
DeepResearcher DS Qwen 32B
0.118 0.2114
0.044
-0.955
0.057 -0.115
DecpRescarcher GPT4.1
0.212 0.115
0.104 0.048
0.058 0.118
34

---

Table 25: Bootstrap 95% Confidence Intervals.
Model
Precision C1
Recall CI
Fl Score CI
OpenAl DR
0.552 0.701
0,442 0.587
0,465 0.607
Perplexity DR
0.364 0.533
0.222 0.354
0.257 - 0.391
Gemini DR
0.229 0.378
0.162 0.281
0.171 -02292
DeepRescarcher + DS Qwen 32B
0.123 0.257
0.037 0.105
0.052 0.126
DecpRescarcher GPT4.1
0.123 0.260
0.040
0.119
0,053
0.135
Table 26: Variance across runs (Entities subset).
Trial
Precision
Recall
1
0.746
0.4559
0.547
0.633
0.349
0.433
3
0.690
0.407
0.472
Mean SE
0.6897 + 0.0565
0,4050 + 0.0550
0.4840 + 0.0579
Table 27: Strict Precision/Recall/FF scores for OpenAl DR, evaluated with GPT-4o. Std. dev. is across 5 GPT-4o runs (temperature=00.).
Subset
Precision
Recall
Fl Score
NovelDS Identification
0.667 + 0.000
0.667 + 0.000
0.667 + 0.000
NovelDS Denti-Extraction
0.450 = 0.019
0.388 0.019
0.406 + 0.019
NovelDS Peer
0.485 0.000
0.334 0.000
0.389 0.000
Flights
0.400 0.051
0.420 1 0.040
0.407 0.045
Table 28: Strict Precision/Recal//F1 scores for Perplexity DR, evaluated with GPT-4o. Std. dev. is
across 5 GPT-4o runs (temperature=0.3..
Subset
Precision
Recall
F1 Score
NovelDS Identification
0,500 + 0.000
0.500 + 0.000
0,500 + 0.000
NovelDS Identi- Extraiixttttionn
0.332 + 0.015
0.359 = 0.017
0.342 + 0.016
NovelDS Peer
0.318 = 0.041
0.174 0.126 0.014
Flights
0.173 + 0.010
0.176 + 0.010
0.174 + 0.010
Table 29: Striet Precision/Recal/F11 scores for Gemini DR, evaluated with GPT-4o.. Std. dev. is across 5 GPT-4o runs (femperature=0.3).
Subset
Precision
Recall
Fl Score
NovelDS Identification
0.333 0.000
0.333 + 0.000
0.333 0.000
NovelDS Identi-Extraction
0.297 + 0.037
0.250 + 0.023
0.255 + 0.027
NovelDS Peer
0.295 1 0.019
0.148 1 0.007
0.196 1 0.011
Flights
0.154 1 0.014
0.157 1 0.017
0.156 1156100 0.016
35


Reviews:
[{"review_id": "MfXQwIOahk", "reviewer": "Reviewer_vd99", "summary": "The authors define Deep Research (DR) as tasks that require both broad search and deep reasoning.\n\nThey formalize it using a claim, subclaim graph that can capture information synthesis quality rather than just report writing quality. \n\nThey propose LiveDRBench, an open-web benchmark that evaluates DR systems with claim-based precision, and recall instead of just report quality evaluation. \n\nThey evaluate several open and commercial models, like OpenAI’s Deep Research, Gemini, and DeepResearcher and they show performance gaps between them across different categories and challenges.", "questions": "How do the authors make sure that the LLM-as-a-judge scores match human judgments, since there is no human evaluation in the paper?\n\nHow is the claim and subclaim evaluation different from the one used in Mind2Web 2 [1] and DeepResearchGym [2]?\n\nWhy do the authors say that DeepResearchGym has no objective scoring when it already uses RAG-based validation and human evaluation to make sure it is objective?\n\nWhat happens if the agent just copies and pastes all the content from the search results into the report, would it still get a high score?\n\n [1] Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge\n [2] DeepResearchGym: A Free, Transparent, and Reproducible Evaluation Sandbox for Deep Research", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The authors investigate the important task of Deep Research, which has broad applications across in both academia and industry.\n\nThey propose a new benchmark, livedrbench, that has 100 tasks and is used to evaluate how well DR agents, both open source and commercial, can identify the right claims and subclaims to produce comprehensive, well-grounded report.\n\nThey include in the benchmark DR tasks across multiple categories, including Materials, SciFacts, and Geo.", "weaknesses": "The authors use LLM-as-a-judge, but there is no human evaluation to ensure alignment with human scores.\nFor example, DeepResearchGym has a nice human evaluation process that validates its LLM-as-a-judge scores against human judgments [2].\n\nThe claim/subclaim idea is very similar to the one in Mind2Web 2 [1], where the evaluation agent checks whether specific claims are present in the report itself in a hierarchical manner. In fact, Mind2Web2 seems to be more precise at identifying facts as it breaks down the claims/subclaims in a nice yes and no tree.\n\nThe precision scoring mechanism proposed by the authors is more of a factuality check (is the claim supported by the citation), it does not seem to capture whether a claim is actually relevant to the DR Query.  For this, the authors should collect hard negative claims as part of the groundtruth, and if the agent puts them in the report they should be counted as false positives (which relates to precision)\n\nThe authors argue that most existing works do not check for claim correctness and focus on report quality, however, DeepResearchGym [2[ and other existing works use strong RAG-based validation to test factuality and recall of the groundtruth claims, so the distinction between this paper and that prior work is unclear.\n\nIt is also unclear what the contributions are of this paper in comparison to existing benchmarks. In Table 1,  the authors claim that DeepResearchGym lacks objective scoring, but that is false in my view since DeepResearchGym uses RAG to assess claim recall and most importantly includes human evaluation to ensure that their llm-as-judge align with human evaluation (which is missing in this paper).\n\nThere really needs to be a human  evaluation to make sure that the llm-as-a-judge used is aligned with human feedback. Also, there is no sensitivity analysis of how the results differ if the evaluation is ran multiple times. LLM-as-a-judge can have high variance in its scores.\n\n Furthermore, the \"easy to update\" claim in Table is weak, as all existing benchmarks seem to be easy to update. In fact, DeepResearchGym has 96,000 tasks, which is far more than the 100 tasks in LiveDRBench, which suggests that the former is actually easier to update and scale.\n\n\n [1] Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge\n [2] DeepResearchGym: A Free, Transparent, and Reproducible Evaluation Sandbox for Deep Research", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XWZn1TtI9A", "reviewer": "Reviewer_5YJ8", "summary": "The paper proposes a formal definition of *deep research* (DR) as the sub‑task of claim synthesis over a large number of information units, and introduces **LiveDRBench**, a publicly released benchmark of 100 web‑search‑driven queries spanning scientific, dataset‑finding, prior‑art, entity‑listing and flight‑incident domains.  The benchmark is built by “problem inversion” of existing long‑context QA items and is evaluated through a claim‑based precision/recall metric that is computed by an LLM judge (GPT‑4o).  Using this infrastructure the authors compare three proprietary DR agents (OpenAI, Perplexity, Gemini) and two open‑source agents, reporting average F1 scores ranging from 0.02 to 0.72 and analysing trace‑level search coverage, depth, branching and back‑tracking.\n\nThe paper tackles an important, under‑explored problem and provides a promising benchmark.  However, the **evaluation methodology relies heavily on an opaque LLM judge**, and suffers from reproducibility issues (especially for proprietary agents).  These methodological shortcomings significantly limit confidence in the reported claims and hinder the benchmark’s immediate usefulness to the community. Please see my comments below.", "questions": "1. Fairness/budget control: How many tokens, queries, and total wall-clock time did each system consume per task? Can you re-run proprietary DR systems with matched query/time budgets to non-DR baselines (and to each other) via APIs?\n2. Variance and significance: Can you report per-task distributions, bootstrap CIs, and statistical tests for pairwise comparisons? How stable are rankings across multiple runs?\n3. LLM-judge reliability: What is the agreement between GPT-4o and multiple independent human annotators (e.g., κ or Spearman/Pearson correlations per category)? Any cross-judge validation (e.g., Claude/LLama judges)?\n4. Ground-truth expansion: How many items were added via model outputs per category? Did any system preferentially contribute to expansions? Can you freeze a versioned ground-truth independent of the measured systems to avoid circularity?\n5. Replicability: Will you release time-stamped queries, full model traces, and archival snapshots of cited sources (e.g., WARC) so results can be reproduced?\n6. Metrics sensitivity: How robust are system rankings to alternative claim aggregation strategies (product vs mean vs strict-min) and claim weighting? Please include rank correlations across metric variants.\n7. Human evaluation details: How many human annotators participated in the “human‑evaluated” verification, what were their expertise levels, and what were the agreement statistics? \n8. Open‑source agent failures: Could you provide logs or error analyses for the DeepResearcher runs that “infinite‑looped” or “failed”?  Might alternative retrieval back‑ends or hyper‑parameter tuning change their performance?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- This is a timely and relevant problem: the framing also separates information synthesis (claims) from long-form report generation, improving construct clarity for DR evaluation.\n- Claim-based evaluation design directly targets correctness/completeness of substantive content rather than stylistic quality; strict metrics are well-motivated for enumeration tasks.\n- LiveDRBench spans multiple realistic DR use cases, including science and public-interest scenarios; explicit effort to make benchmark updatable is valuable.\n- Transparent reporting of category-wise weakness (e.g., SCIFACTS Materials) and plausible sources of difficulty (subclaim extraction across long documents).\n- Provides detailed prompts and evaluation pipeline in appendices; attempts to validate LLM-judging by a human pass that yields “similar” aggregates.\n- Trace-analysis perspective (breadth/depth/coverage) is practically insightful and aligns with qualitative failure modes of DR agents.", "weaknesses": "- Proprietary DR systems were evaluated via chat UIs with unspecified/variable budgets; non-DR baselines used APIs with fixed “≈30s” reasoning but DR systems seemingly ran with much larger, uncontrolled search/time budgets. This undermines comparability and may inflate proprietary DR performance relative to baselines and open-source agents.\n- Single-run reporting without variance: no per-task variance, confidence intervals, or statistical tests. Given high run-to-run variability in browsing agents and changing web results, this is a major limitation for drawing strong comparative claims.\n- Ephemeral open-web setup without timestamped corpora or cached sources; no normalization for geographic or temporal search variability. The “Live” design is appealing but compromises replicability without web snapshots or logged artifacts.\n- LLM-as-judge dependence: GPT-4o is used for matching, scoring, and even for trace analyses (necessary queries, dependency classification, branching/backtracking). Although authors include a “human” table with similar means, there is no inter-annotator agreement, correlation coefficients, or systematic error analysis. Heavy reliance on a single proprietary judge risks evaluation bias.\n- Ground-truth expansion procedure: the rubric is augmented with any valid answers produced by the evaluated systems. While pragmatic, this can introduce dependence of the label space on model outputs, advantaging models that are included (and potentially those aligned with the judge). It also does not guarantee completeness as unseen correct answers remain out-of-scope, distorting recall.\n- Metric design choices not stress-tested: the multiplicative/product formulation across subclaims can collapse scores when one subclaim is wrong; no ablations comparing product vs min (strict) vs additive/averaged schemes and their impact on ranking stability. Weighting scheme w_i is suggested but not analyzed.\n- Claims that OpenAI DR “performs best” are plausible but not sufficiently supported for a benchmark paper because (a) budgets and interaction settings are not matched, (b) no statistical testing, (c) potential judge/model coupling (OpenAI DR and GPT-4o judge), and (d) evaluation performed on live web without controls.\n- Trace-analysis conclusions (e.g., “coverage ~ half”) rely on GPT-4o-generated “necessary queries” and GPT-4o matching in traces, with no human validation or inter-rater reliability. As such, these are best viewed as exploratory rather than definitive evidence of breadth/depth gaps.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qsx5ArZ2FB", "reviewer": "Reviewer_fjWB", "summary": "The paper proposes a formalization of “deep research” (DR) as tasks that are both search-intensive (high fan-out over information units) and reasoning-intensive (non-trivial effort to find/process/combine evidence). It argues that the core of DR is claim synthesis rather than long-form prose generation, and introduces LIVEDRBENCH, a 100-task open-web benchmark with a claims-based evaluation (precision/recall over nested claims, with strict scoring that zeros out a claim when any subclaim is wrong). The benchmark spans eight categories (SCIFACTS–Materials/Geo, NOVELDS—three types, PRIORART, ENTITIES, FLIGHTS), and evaluates proprietary DR systems and open-source baselines, reporting best overall performance for OpenAI’s DR model (avg. F1 ≈ 0.55) and analyzing search traces for breadth/depth coverage.", "questions": "How robust are scores to judge model choice (GPT-4o vs. alternatives) and to noise in claim parsing? Please include a cross-judge concordance table.\n\nFor “necessary query coverage,” what fraction was human-validated, and what is agreement between annotators vs. LLM labeling?\n\nHow do results change if you relax the strict subclaim rule (e.g., soft weights), and if you weight claims by importance?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Clear problem decomposition. Separating “claims synthesis” from “report generation” is crisp and useful; the DAG framing of queries→evidence→claims aligns with how DR agents operate.\n\nObjective evaluation for multi-claim outputs. The nested claim/subclaim metric operationalizes “grounded completeness” better than stylistic LLM-judge scores used in other DR evaluations.\n\nUseful positioning vs. prior benchmarks. The paper contrasts LIVEDRBENCH with report-quality benchmarks (e.g., DeepResearch Bench) and browsing-centric datasets (e.g., BrowseComp) and motivates claim-level scoring for open-web DR.", "weaknesses": "Metric dependence on LLM judging and design choices. Claim agreement and “necessary query” identification rely on LLMs (GPT-4o) and bespoke prompts; the paper needs stronger validation (e.g., human adjudication on a stratified sample, inter-rater agreement, and sensitivity analyses for the “zero credit if any subclaim is wrong” rule).\n\nReproducibility of proprietary model comparisons. Evaluations for commercial DR systems are done via chat UIs without API parity; differences in browsing tools, guardrails, or hidden budgets could confound scores. More details (and repeated trials) are needed.\n\nScope overlap with related work could be made sharper. While the paper positions against BrowseComp (short, verifiable answers) and DeepResearchGym (static corpora like ClueWeb22/FineWeb), the narrative could more explicitly articulate complementary use—e.g., LIVEDRBENCH for open-web, claim-rich synthesis; DeepResearchGym for reproducible search sandboxes; DeepResearch Bench for report-quality LLM-judge eval.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hh1zJwfbwC", "reviewer": "Reviewer_AaxG", "summary": "This paper introduces a benchmark for evaluating deep research systems. The authors first formalize deep research type of questions and focused on how it is different from other reasoning tasks. In addition, they evaluates DeepResearch agents on their proposed liveDRbench with 100 questions. Their evaluation is based on accuracy and completeness of claims from ground truth.", "questions": "- How practical is the claim that LIVEDRBENCH can be “easily updated”? Given the need for verifying unique answers and refreshing ground truth as the web evolves, what is the actual human effort involved in maintaining the benchmark?\n- Could the claim-level F1 metric unfairly penalize systems for surface-level differences in entity names (e.g., “UCLA” vs. “University of California, Los Angeles”)? \n- Who validated the 100 benchmark questions? If these questions are specialized and domain-heavy, how were they verified for correctness by non-experts? \n-Was there any human validation of the evaluation results to understand to what extend  the automatic scoring is trustworthy?\n- Since the benchmark construction involves automatic inversion and possibly LLM-generated ground truth, how do you ensure that no hallucinated or incomplete claims made it into the final dataset?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper makes  make a  formal definition of what constitutes a Deep Research task. I think it is important to take a step to clarify these types of tasks and to distinguish it from multihopQA or long form reasoning.\n- The ablation studies and experiments are relatively comprehensive.\n- Their use of open web search and “problem inversion” makes the benchmark naturally extensible", "weaknesses": "- While the paper claims the benchmark can be easily updated via inversion of long-context tasks, in practice this may be labor-intensive which might require human verification of unique answers and updated ground truth each time the web changes.\n- Some important DR-related systems as baselines are missing, such as OpenScholar or LLM+Tool-use agents that combine retrieval with API or browser actions. Including these would make the evaluation more representative of current “agentic” DR methods.\n- Measuring coverage over issued queries rather than retrieved gold documents may be misleading. what if different query phrasings can retrieve equivalent document sets. Evaluating retrieval coverage (recall of relevant documents) would be a fairer proxy compared to query coverage.\n- If I understood correctly, the claim-level F1 may penalize systems unfairly if entities are represented differently (e.g., “UCLA” vs. “University of California, Los Angeles”). Maybe using some form of soft matching could yield a more realistic measure of factual alignment.\n- First, who were the human validators for the 100 questions? If the questions are truly expert-level and niche, it’s not easy for non-experts to validate them. Second, there should be some level of human validation on the evaluation itself to make sure the results are reliable.\n- I think there is a chance that because the benchmark construction process uses automated inversion and possibly LLM assistance, some ground-truth claims or links might be hallucinated or incomplete. Human audits could help mitigate this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)