DRBench: A Realistic Benchmark for Enterprise Deep Research
Authors: Amirhossein Abaskohi
Tianyi Chen
Miguel Muñoz-Mármol
Curtis Fox
Amrutha Varshini Ramesh
Étienne Marcotte
Xing Han Lù
Nicolas Chapados
Spandana Gella
Christopher Pal
Alexandre Drouin
Issam H. Laradji
Venue: ICLR 2026 Poster (2026)
Keywords: Benchmark
deep research
reasoning
enterprise
insight recall
factuality
heterogeneous data
persona-grounded tasks
multi-domain evaluation
scalable data synthesis
Docker
AI agent
LLM
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: DRBench benchmarks AI agents on enterprise deep research with 100 persona-grounded tasks across diverse domains and 1093 files. Agents are evaluated on insight recall, factuality, and report quality, with DRBA as a strong baseline.

Abstract: We introduce DRBench, a benchmark for evaluating AI agents on complex, open-ended deep research tasks in enterprise settings. Unlike prior benchmarks that focus on simple questions or web-only queries, DRBench evaluates agents on multi-step queries (for example, "What changes should we make to our product roadmap to ensure compliance with this standard?") that require identifying supporting facts from both the public web and private company knowledge base. Each task is grounded in realistic user personas and enterprise context, spanning a heterogeneous search space that includes productivity software, cloud file systems, emails, chat conversations, and the open web. Tasks are generated through a carefully designed synthesis pipeline with human-in-the-loop verification, and agents are evaluated on their ability to recall relevant insights, maintain factual accuracy, and produce coherent, well-structured reports. We release 100 deep research tasks across 10 domains, such as Sales, Cybersecurity, and Compliance. We demonstrate the effectiveness of DRBench by evaluating diverse DR agents across open- and closed-source models (such as GPT, Llama, and Qwen) and DR strategies, highlighting their strengths, weaknesses, and the critical path for advancing enterprise deep research. Code and data are available at https://github.com/ServiceNow/drbench.

Published as a conference paper at ICLR 2026
DRBench: A REALISTIC BENCHMARK FOR ENTERPRISE DEEP RESEARCH
Amirhossein Abaskohi
Tianyi Chen'
Miguel Muñoz-Mármol
Curtis Fox
Amrutha Varshini Ramesh 1.2
Étienne Marcotte'
Xing Han 3 Nicolas Chapados
Spandana Gella 1,3,4
Christopher Pal -3,56 Alexandre Drouin 1.3
Issam H. Laradji 1.2
ServiceNow Research
2 University of British Columbia Mila - Quebec AI Institute
*McGill University
$ Polytechnique Montreal
CCnada CIFAR AI Chair
ABSTRACT
We introduce DRBench, a benchmark for evaluating AI agents on complex, open-ended ended deep research tasks in enterprise settings. Unlike prior benchmarks that focus on simple questions or web-only queries, DRBench evaluates agents on multi-step queries (for example, "What changes should we make to our product roadmap to ensure compliance with this standard?"" that require identifying supporting facts from both the public web and private company knowledge base. Each task is grounded in realistic user personas and enterprise context, spanning a hetcrogeneous search space that includes productivity software, cloud file systems, emails, chat conversations, and the open web. Tasks gen- erated through a carefully designed synthesis pipeline with numan-in-the-lour tion, and agents evaluated their ability recall relevant insights, maintain factual accuracy, and produce coherent, well-structured reports. We release 100 deep research tasks across 10 domains, such as Sales, Cybersecurity, and Compliance. We demonstrate the effectiveness of DRBench by evaluating diverse DR agents across open- and closed- source models (such as GPT, Llama, and Qwen) and DR strategies, highlighting their strengths, weaknesses, and the critical path for advancing enterprise deep research. Code and data are available at https: / gi thub. com/ ServiceNov drbench.
verifica-
INTRODUCTION
Organizations today face a strong need to find useful insights in world full of overwhelming information. Valuable insights are often hidden in noisy data, which can contain many distracting or irrelevant details that obscure the insights that really matter. This challenge is present in enterprise settings, where data is spread across many applications and stored in different formats (e.g.. PDFs, spreadsheets, emails, and internal tools) making extracting relevant information difficult. To uncover these hidden, valuable insights, one must con- duct what is known as deep research. This task involves asking high-level strategic questions (e.g, "What changes should we make to our roadmap to remain compliant?"), planning sub-questions, retrieving and evaluating relevant materials, and producing a clear, actionable summary grounded in data sources (Zheng et al., 2025; Xu & Peng, 2025; Du et al., 2025). These tasks are typically performed by domain experts using mix of search engines, communication platforms, and business applications in iterative, high-effort workflows (Mialon et al., 2024), which unfortunately require a significant amount of human effort.
One promising solution reducing this human effort is agent-based deep research, which uses autonomous software agents to search, extract, and synthesize information across fragmented sources into an insightful report. Recently, LLM- agents have emerged as promising assistants for deep research. Systems such as Locul Deep Researcher (I (learningCircui,, 2025), Deep-Searcher (Tech, 2024), and DeepResearcher (Zheng al., 2025) propose modular agent pipelines that combine retrieval, reasoning, and summarization over documents and web sources. Architectures like OpenHands (All-HandsAI, 2024), OpenManus (FoundationAgentl 2024), and smolagents (HuggingFacce Face, 2024) extend these capabilities to include collaboration, multi-moda search, and complex tool use in enterprise workflows (Xu & Peng, 2025). Despite these advances, evaluating such systems remains an open challenge.
Most existing benchmarks evaluate narrow aspects such report factuality (Coelho al., 2025), web-only synthesis (Bosse et al., 2025), or tabular analytics (Sahu et al., 2025), but they do not assess whether agents identify the most salient insights, remain faithful to retrieved evidence, adapt to enterprise contexts.

---

Published as a conference paper at ICLR 2026
Accessing Websites
Syrth bnd with Destattr Cortanteed
How can Lee't Martee leveraee food safety and customer trusct
Task Data
Task Context
Questtoo
Connact
o
Nestina
E-nalx
Leers Market regional Askan supermeriett chain
Fil jaine
Presen PPTy
Company
Entorpriae
6
Matioraal
Loading
BoeesCate
B
Tabudar bara
Partona
Manager
Coneration
Bonn, F15 Markfrinn
Trasking for codechapee gronnee redure frod vorce and keep groeeyytice com)
Factuality
cherpe
Load Sources
Structured Roport
Lee's Mar cuado 2nd ok productio as
Report Quality A in
Injected Inaight
2124, offecing aane of eneanirrg wilite coafiityy
compl unce parg
lush redaciion toot vasta 02,3044 snving
Load Ground
PSMA 204 compl 15% of mai food 2008 270 sendors. [the houudaabe smei]]
Insight
Truch Snsights
It Susau righ producis
Figure 1: DRBench pipeline. ① The Task Context defines the deep research question grounded by the company and persona given to the agent. 2 Task Data, including both distractor and injected groundtruth insights in different formats (PDFs, DOCX, PPTX, XLSX, chats, etc.) are loaded into the enterprise environment's 's applications. 3 The DRBenchAgent accesses both public web sources and local enterprise data to extract relevant insights for the research question. ④ It produces a structured research report, which is (5) evaluated for Insight Recall (detecting injected groundtruth insights), Factuality (verifying claims are correctly cited). and Report Quality.
To address these limitations, we introduce DRBench, a benchmark designed to evaluate LLM agents on open-ended, multi-step and long-horizon deep research tasks grounded in realistic enterprise contexts. As Figure 1 illustrates, DRBench includes a suite of queries grounded in user personas and organizational scenarios, requiring agents to search across real applications such as cloud file storage (Nextcloud), enterprise chat (Mattermost), and user file systems, and to reason over formats like spreadsheets, slide decks, and PDFs. Our evaluation framework introduces three scoring axes using LLM-ae-a-judge methods inspired by G-Eval (Liu et al., 2023): (1) Insight Recall and Distractor Avoidance, which together evaluate whether the agent surfaces the most salient injected insights while avoiding distractor content; (2) Factuality, which uses TREC-RAG pipeline (Wang et al., 2024) to verify whether claims correctly grounded in their cited sources; and (3) Report Quality, which measures the coherence, completeness, and overall readability of the synthesized report. We conduct a comparative study of agent architectures inspired by recent work (Zheng et al., 2025; Xu & Peng, 2025; LearningCircui,, 2025; Zheng et al., 2025), analyzing how well they perform on DRBench across planning, insight identification, and grounding on facts. Our results show that while agents are competent at document retrieval and summarization, they often miss high-value insights, cite irrelevant evidence, or produce incoherent explanations, highlighting the limitations of current architectures and the need for more targeted innovation.
Our contributions are as follows: (I) We introduce DRBench, the first benchmark for evaluating LLM agents complex enterprise deep research tasks combining public web sources with private organizational data; (2) We provide suite of 100 high-level research tasks with 1093 sub-questions spanning 10 domains, including Sales, Cybersecurity, and Compliance, each grounded in realistic company contexts and personas; (3) We design a reproducible enterprise environment integrating realistic enterprise applications like chat, cloud storage, emails, and documents; (4) We propose a scalable pipeline that generates realistic research questions and insights by combining web facts with synthesized internal data; and (5) We develop an evaluation framework that scores agent reports on insight recall and distractor avoidance, factuality, and overall report quality.
2 RELATED WORK
Deep Research Benchmarks. With the growing capabilities of LLMs in research and reasoning tasks, several benchmarks have emerged, including Deep Research Bench (Bosse et al., 2025), RseaRheearch Bench (Du et al., 2025), DeepResearchGymm (Coelho et al., 2025), ResearcherBench (Xu et al., 2025b), Mind2Web2 (Gou et al., 2025), and GAIA (Mialon et al., 2024). As summarized in Table 1, these efforts primarily evaluate web-only retrieval or synthesis in controlled settings. A recent benchmark by Choubey et al. (2025) further emphasizes closed-form fact retrieval within engineering artifacts, In contrast,
2

---

Published as a conference paper at ICLR 2026
Table I: Comparison of deep research benchmarks (top) and Al agent benchmarks with computer environ- ment (middle). Columns report dataset size, whether both public and local data are required, the provided environment type, task domains, task description, and evaluation method. Unlike prior work, DRBench com- bines public web retrieval with local enterprise data in realistic enterprise applications and evaluates both insight recall, distractor avoidance and report quality. Task Description: types of tasks covered by the benchmark: WR for Web Research, DR for Deep Research with both public and local data, CU for Com- puter Use and/or Mobile Use. DRBench has 1093 total # groundtruth insights that need to be extracted to address the 100 DR Questions. Example groundtruth insights can be found at Table 8 in Appendix A.
Public & Local Data
Provides Euv
Task Domain
Task Description
Benchmark
groundtruth
Main Evaluation Method
Deep Research Bench (Bosse et al., 2025)
89
X
Generic
WR & CU
Answer Accurncy
DeepReeearch Bench (Du et al., 2025)
100
X
Generic
WR
Insight Recall
DeepReseanhhiymm (Coelho et al., 2025)
1,000
Generic
WR
Document Retrieval
ResearchetBench (Xu
2025b)
WR
Insight Recall, Factuality
LivcDRBench (Java al., 2025)
100
Generic
WR & CU
Insight Precision. Recall
BrowseComp Plus (Chen et 2025)
1,005
Generic
WR
Answer Accuracy URL Recall
Mind2Web 2 (Gou et al., 2025)
130
Generic
WR
Purtial Completion
GAIA (Mialon et all,, 2024)
466
X
Generic
WR
Answer Accuracy
GAIA2 (Andrews ct al., 2025)
963
Generic
CU
Action Accuracy
TheAgenCoomanny (Xu et 2025a)
175
Enterprise
CU
Task Completion. Efficiency
OSWorld (Xie al., 2024)
369
Generic
CU
Task Completion
DRBench
1093 (100 tasks)
Enterprise
DR
Insight Recall
DRBench is the first to combine web retrieval with local enterprise data, requiring multi-step deep research grounded in persona- and domain-specific contexts.
Enterprise Environments. Realistic enterprise environments have become an important testbed for evaluating agents in complex multi multi-application workflows. CRMArena-Pro (Huang et al., 2025a;b) targets sales and CPQ pipelines through persona-grounded dialogues, but is limited to conversational sales workflows. OSWorld (Xie et al., 2024) and OSWorld-Gold (Abhyankar et al., 2025) benchmark agents in general-purpose desktop environments, using applications such as Microsoft Word and Excel, yet their focus remains on computer task execution rather than enterprise deep research. TheAgentonpparyy (Xu et al., 2025a) evaluates collaboration among autonomous agents for programming, browsing, and communication, though the tasks are computer-use focused and do not assess deep research capabilities. WorkArena (Drouin et al., 2024; Boisvert et al., 2024) offers a realistic enterprise environment with knowledge work tasks for web agents, though it does not support evaluation of deep research capabilities. In contrast, DRBench offers domain-grounded enterprise environment with applications that would realistically be encountered in organizations. Tasks are tied to concrete personas and roles, requiring agents to search, reason, and synthesize insights across diverse formats, including spreadsheets, PDFs, wikis, emails, and presentations, reflecting realistic enterprise deep research.
Deep Research Agents. A growing line of work explores agents for multi-step search and synthesis across diverse information sources. LangChain's 's Local Deep Researcher (LeamiinCCrruii, 2025) and Zilliz's Searche provide modular pipelines for iterative querying and summarization, while DeepRe- searcher (Zheng et al., 2025) uses RL to enable planning, cross-validation, and self-reflection. Commercial systems such as Gemini Deep Research and Manus.ai synthesize web-based reports with citations, and open-source frameworks like OpenHands (All-HandsAL, 2024), OpenManus (FoundationAgemn 2024), and smolagents (HuggingFace, 2024) offer altemative architectures. Recent work also introduces task-agnostic frameworks for long-form synthesis and evaluation paradigms such as Mind2Web 2 (Gou et al., 2025), which treat agents as judges of browsing trajectories. Building on these efforts, DRBench an- alyzes their strengths and limitations in enterprise contexts, showing that current agents still fall short in consistently extracting and grounding critical insights within complex, heterogeneous environments.
3 DRBench AN ENTERPRISE DEEP RESEARCH BENCHMARK
To evaluate agents complex, open-ended enterprise deep research tasks, we designed DRBench with three guiding principles: it requires agents to integrate both public web data and local enterprise documents, it involves both web search and enterprise application use, and it hosted in an interactive and reproducible
3

---

Published as a conference paper at ICLR 2026
i
COMPANY INFORMATION
WEBSEARCH
EXTERNAL INSIGHT
INTERNAL INSIGHT
FILE GENERATOR
Ried oriine Decument the Company Damale
Brsight
PDF
Target Sott-.
OUTPUT
and Rlely's you FSM4 chence to They lockeve
Repionall
2 OUTPRTT
Lea'' Marhet traska products of C 033884 affecting 20% awentory.
CUTTUN
FPIX
DOCK
Engiages Courte
Grevens salk hoots 20s comploaaee
repply
invendiiity bems narorre trast could extenthe fead saleey
PERSONAS
Q DR QUESTION
DISTRACTOR
Chaose The File Type
DOMAINS
Generation Sections with Insight Section the File
Compllance
Duep Revearch Quaatiion Ceneration
Distrester Generattr
Research
+ OUTPUIT
G OUTPUT
Maricer Anelysis
can Matrrt lewarage 234
Emptiyyee taroorer 19t8
Generation of Sections without Insight in the file
custoreer thart?
tursie 03-3224
Figure 2: DRBench Task Generation Pipeline. The pipeline comprises five main stages during each LLMs generate candidate data such company context, insights, and research questions, while human annotators verify quality and select the final version. Stages $1-S5 denote the five generation steps. enterprise environment. These principles ensure that the benchmark reflects realistic enterprise workflows and provides a controlled yet challenging setting for research agents.
Benchmark Scope. DRBench evaluates document-centric deep research in enterprise settings, where agents must synthesize evidence across heterogeneous applications and sources, including both public web content and private enterprise documents. Tasks are cross cross-applins application by design and reflect realistic research workflows such as policy analysis, compliance assessment, market research, and strategic decision support that rely on reading, retrieving, and reasoning over unstructured enterprise data.
The Enterprise Search Environment. A unique aspect of DRBench is its realistic enterprise search environment. When addressing DR Questions like "What changes should we make to our product roadmap to ensure compliance with this standard?", the DR agents would need to navigate such environment and search across both public and private data sources to uncover relevant insights.
Public insights include information available on the open web or otherwise accessible to general users. Local insights, on the other hand, come from an enterprise's ' private systems. These insights are embedded within vast search space that spans multiple data types (emails, slide decks, chat conversations, and Excel sheets) which reflect the complexity of real enterprise data ecosystems. This environment is populated with data from different applications, accessible to both web-based agents and API-calling agents. For example, an app like Mattermost can be used to host chat conversations (see Appendix E for examples of the applica- tions). The goal of the DR Agent is effectively navigate these public and private data sources address complex, high-level DR questions. For the environment implementation details, please see Appendix D.
Task Definition. Each task is associated with deep research question Q: and Task Context C which includes company information and the user persona. Each task also has a corresponding set of groundtruth insights I consisting of relevant private insights 4 (we also refer this as internal insights), distractor private insights Ia. and public insights Ip. Each private insight, whether relevant or a distractor, is embedded into file f/ which could take the form of PDF, Excel sheet, slide deck, chat log, and on. The agent's s task is to generate report by extracting the public insights I, from accessible sources such as the web, while also extracting the private insights / from the files hosted the enterprise environment, At the same time, the agent must avoid extracting the distractor insights Ja. which are not relevant to the DR question.
DRBench provides 100 realistic deep research tasks explicitly framed around enterprise environments. Each task is associated with public insights extracted form quality, time-invariant URLs and local insights embedded within synthetic enterprise data, typically spanning 2-4 applications and 3-16 supporting files (see Appendix B). Tasks are distributed across 10 enterprise domains (such as Sales and Compliance the full list is in Appendix B) and divided between easy, medium, and hard categories that indicates the difficulty of addressing the DR Question. Finally, DRBench is fully self-hosted, with dated URLs and reproducible evaluation scripts to ensure stability and fair comparison across agent methods.
3.1 DATA GENERATION
To create realistic and reproducible deep research tasks, DRBench employs a five-stage pipeline (Figure 2) that combines large-scale LIM generation with human-in-the-loop verification. The pipeline helps us
4

---

Published as a conference paper at ICLR 2026
generate candidate company contexts, personas, questions, insights, and supporting files using LLM 1-40BB (Dubey et al., 2024). Three human annotators then validate the generated content to ensure that they are realistic and plausible.
Models such as Llama-3.1 1eeee...-----Imme -ns--nstt Llama-3, 3.ama-3.1-405B
The pipeline has been used to generate 100 tasks with 1093 groundtruth insights across 10 enterprise domains, each grounded in realistic personas and company profiles. We control the difficulty of each task by setting the number of insights, file types and application types. The complete list of tasks is provided in Appendix B. Refer to Appendix I for details on the cost of using data generation.
Stage I: Company and Persona Generation. This stage produces the synthetic company profile and user persona that form the Task Context C. L.LMs were used to generate company descriptions detailing the industry vertical, key products, market position, and competitive landscape. In parallel, they were used to create realistic personas across departments (e.g., a Regulatory Affairs Manager or a Market Research Analyst) that serve as the role grounding for the final deep research question. They were then refined by human experts. The prompts used for this stage are provided in Appendix P.I and the list of companies are given in Appendix B.
Stage 2: Public Source and Insight Collection. Given the company and persona context from Stage 1, we have retrieved candidate URLs relevant to the specified domain and company background. To ensure quality, time-invariant insights, the search is restricted to dated, journal-based or industry-report websites that provide authoritative information. Thus, the collected URLs and their contents are expected to be stable in time. Human annotators then review the candidate URLs and select one that is both topically aligned and provides insights into the topic. The selected page becomes the Task URL included in C. Its HTML content is parsed, and LLMs are prompted to extract business- relevvan insights, which are subsequently filtered and validated by human reviewers for accuracy and contextual fit. The public insights I, derived from the Task URL are included in C and serves as a required piece of insight for the agent to retrieve during report generation. Prompts used for this stage and the list of urls are provided in Appendix P.3.
Stage 3: Question Generation. Given the Task Context, we generate the deep research question Q. The prompt (see Appendix P.2) is instantiated with the company profile and persona, the selected domain, the Task URL, and the public insight 4p. The LLM proposes several open-ended candidate questions grounded in this context. Human annotators then review these candidate DR Questions, selecting and refining one to align with the persona and company. They also ensure that the insights available in the provided URL can at least partially support answering the deep research question. For example, if the question concerns compliance with specific regulation, the URL might include relevant insights, such as " "groceries must have a traceability plan." While this doesn't fully resolve the question, it provides foundation.Tha question should be high-level enough allow to synthesize additional supporting private/internal insights T (such an insight could be "the cost of implementing such a plan is X amount") which are needed to strengthen the report generated for the question. This requirement ensures that new internal insights can be generated, as discussed in Stage 4.
Stage 4: Internal Insight Generation. In this stage generate the injected insights set G CI. Using the public insight I, and the deep research question Q, LLMs are used to create company-specific insights aligned with the organization's industry, priorities, customer segments, and business goals. These insights are designed to provide additional supporting facts that need to be extracted to create report that better addresses the DR questions. Human annotators review and refine these insights for accuracy and alignment with the questions. In addition to relevant insights, we also produce distractor insights la. which are plausible but irrelevant statements that do not support resolving the DR Question. Prompt details are provided in Appendix P.4 and example internal insights are provided in Appendix B.
Stage 5: File Mapping and Generation. This stage produces the the set of files {ʃi} containing both the relevant and distractor private insights. First, each insight is assigned to a modality such as email, chat, pdf, docx, and so on. Then the file generation module follows the following three-step "needle-in-a" -haasss-a-aaaatckk"" process: (1) create an outline of the file based on its modality(e.g. document structure or chat configuration), (2) insert the distractor or relevant insight into an appropriate section of the file, and (3) fill the remaining content with realistic but irrelevant information. Human annotators spot-check the generated files to ensure fidelity, coherence and no contradicting information. Prompts for file generation are provided in Appendix P.5 and screenshots of such generated files are in Appendix C.
5

---

Published as a conference paper at ICLR 2026
Tools
Drtlench Enterpatee Environment Smarch Space
Company Employee
Web Search
Internet Search
Enterprine Smarch:
Analyoor
UNL Potch
Local Files
roundcube Mattermost
Space
CONTINT
ACTION PLANWING
TOOL SELECTION
PROCESSING
plis pad ped 2iee
Deep Research Question
Research Loop
REGEARCH
ADAFTIHE ACTION
Findligs
Company Context
RAW REPORT
Seport
REPORT
Strucured tist diedd insight)
ifree nanasiive
GEELLM perkored
Store
Figure 3: DRBench Agent architecture showing the enterprise research workflow from question submission through iterative research cycles final report generation, using both enterprise and web search capabilities. Reports are generated in two formats: raw report, consisting of free-form narrative text, and a structured report that lists the main insights with their corresponding citation(s).
DRBench AGENT
The DRBench baseline agent (DRBA) is the first agent built specifically for deep research in enterprise settings, designed to operate directly within the DRBench environment. Its multi-stage architecture systematically investigates research questions by iteratively retrieving, processing, and synthesizing knowledge from enterprise services and the web until completion or maximum iteration limit is reached (Figure 3). The agent has access to app-specific API calling tools to access diverse information sources and diverse toolset for analyzing retrieved results (Table 9, Appendix F.3). DRBA's architecture is organized into four main components: research planning, action planning, an adaptive research loop, and report writing. See Appendix Appendix F for more details on the implementation details. Refer Appendix I for details on the cost of using DRBA.
Research Planning. The agent decomposes research questions into structured research investigation areas to guide subsequent action generation. This initial decompasition lays out a strategy to systematically cover the research space while maintaining focus on the initial deep research question. The agent supports two research planning modes: (I) Complex Research Planning (CRP), which generates a structured research plan with detailed investigation areas, expected information sources, and success criteria; and (2) Simple Research Planning (SRP), which produces lightweight decomposition of the main question into a small set of self-contained subqueries. See Appendix F.2 for detailed examples of both modes.
Research Loop with Adaptive Action Planning (AAP). The system iterates through (I) tool selection and exccution based on action priorities, (2) content processing and storage in a vector store, (3) adaptive action generation to cover research gaps, and (4) iteration findings storage for traceability and assessment.
Report Writing, The report writing subsystem queries the vector store synthesize the research findings and relevant retrieved content. This component generates a comprehensive report and uses its own citation tracking system to ensure proper attribution of claims. For our evaluation, we use the structured report format, a list of insights with their citations, rather than a free-form narrative response (raw report).
5 EXPERIMENTS AND RESULTS
In this section, we evaluate the DRBench Agent (DRBA) on both the full DRBench benchmark and a reduced subset for ablations. We consider (1) the FullBenchmark, covering all 100 tasks across 10 domains (see Table 7 for details of the tasks), and (2) the MinEval subset, restricted to 15 tasks for efficient ablation studies. Results are reported across four metrics: Insight Recall, Distractor Avoidance, Factuality, and Report Quality, explained in the next section. Implementation details and hyperparameters are in Appendix H. while the impact of the number of research loop iterations on the performance of DRBA is analyzed in Appendix O.

---

Published as a conference paper at ICLR 2026
5.1 EVALUATION METRICS
Our evaluation intentionally operates at the level of atomic insights rather than treating the report single monolithic output. This design is diagnostic rather than purely outcome-oriented, as it enables partial credit when only subset of key findings is recovered, allows localization of specific failure modes such as missing, unsupported, or irrelevant insights, and supports multi-axis analysis across recall, precision, and factual grounding. Compared to end-to-end correctness metrics, insight-level evaluation reveals how and where an agent succeeds or fails, which is especially important for long, compositional research reports.
Insight Recall. Our benchmark directly evaluates the of atomic insights provided by the agent, which was the case for all our experimental results. If an agent provides the full report, it decomposes the report into atomic insights using an LLM (see Prompt 14). Each insight then compared against the groundtruth set using an LLM Judge with Prompt 15, If a match is found, the insight is marked as detected and contributes to the insight recall score; otherwise, it ignored. This metric thus measures recall rather than precision, since judging whether an unmatched insight is nonetheless useful for answering the deep research question is inherently subjective and difficult to automate. In practice, the computed insight recall functions an accuracy measure in our setting, since it reflects the proportion of groundtruth insights tied to the research question that the agent successfully identifies. To prevent agents from trivially achieving 100% recall by copying all content into the generated report, the LLM Judge evaluates only the first k insights, where l equals the number of ground- -ouuth insights plus five. This buffer ensures that reports not penalized for including seemingly relevant insights that are not part of the groundtruth insight set. While this cutoff may seem arbitrary, we found it essential for preventing agents from gaming the metric by copying large portions of the source files. The +5 buffer allows space for few reasonable additional insights without rewarding unrestricted copying. We acknowledge that this design limits our ability to measure the full breadth of useful discoveries, and we discuss its implications and alternatives in Appendix T. Our evaluation relies on short, atomic claims, which keeps LLM-judge variance minimal regardless of whether the judge is a closed or open model; refer to Table 20 in Appendix L for more details.
Distractor Avoidance. To measure precision, we track whether the agent's report includes distractor insights that are irrelevant to the research question. We compute distractor recall analogously to insight recall, and define distractor avoidance as 1 1- distractor recall.
Factuality. Using the same set of insights (that we used for Insight Recall), we follow the methodology of FactScore (Min et al., 2023). If an insight lacks citation or references a non-existent source, it is labeled unfactual. Otherwise, we apply a serrival-auugmentrr system based on text -emeeddin-g-----ee 3-09--large 1-large (OpenAL, 2024) to fetch the top-5 most relevant chunks from the cited document (Appendix H). The LLM Judge with Prompt 16 then determines whether the cited evidence supports the claim. We also store justifications and model confidence scores for interpretability.
Report Quality, Inspired by prior work (Coelho et al., 2025; Abaskohi et al., 2025), we query the LLM Judge with Prompt 17 to assign a 1-10 rating across six dimensions: (1) depth and quality of analysis, (2) relevance to the research question, (3) persona consistency, (4) coherence and conciseness, (5) absence of contradictions, and (6) completeness and coverage. The final report quality score is obtained by averaging these six ratings.
5.2
MAIN RESULTS
We first evaluate our DRBA agent using GPT-4o as the backbone model, a maximum of 15 research loop iterations, and different combinations of planning modules: Simple Research Planning (SRP), Complex Research Planning (CRP), and Adaptive Action Planning (AAP) on the full benchmark. The results are reported in Table 2. Overall, the agent demonstrates moderate ability to ground its answers in factual evidence but struggles to consistently surface the main injected insights necessary for answering the deep research questions. many cases, the agent relies on prior knowledge or external web content rather than integrating the crucial interprise-specifi information available in the files. By contrast, it is consistently strong in avoiding distractors, showing that the agent is robust against misleading or irrelevant information but less effective at prioritizing decision-criticas insights. Note that our LLM Judge backbone is Go. should be also mentioned that Insight Recall is robust to paraphrasing, with negligible variance across reformuaations (see Appendix S).
As the results illustrates, SRP tends to produce more factually grounded answers, while CRP excels at filtering out distractors through structured decomposition. AAP, the other hand, provides the largest
7

---

Published as conference paper at ICLR 2026
Table 2: DRBA performance with different planning configurations on DRBenchhFlllBeeehnnat)) We compare the base agent with variants using Simple Research Planning (SRP), Complex Research Planning (CRP), Adaptive Action Planning (AAP), and their combinations. See Appendix K for the standard error across 3 runs on MinEval. Note that higher numbers correspond to better scores, and the best result on each metric bolded.
Configuration
Insight Recall
Factuality
Distractor Avoidance
Report Quality
Harmonic Mean
Base DRBA
13.18
58,04
95.76
88.23
34.82
+ SRP
13.42
62.11
96.62
89.74
35.68
+ CRP
13.31
59.53
97.14
87.92
35.21
+ AAP
15.97
60.37
96.48
90.08
39.74
+ SRP + AAP
14,83
55,29
96.55
88.96
37.34
+ CRP+ AAP
14.19
$2.08
96.47
87.54
35.89
Table 3: Performance of DRBA the FullBenchmark subset using different backhone language models and planning strategies. Note that higher numbers correspond to better scores, and the best result on each metric is bolded. The full table with more models is given in Appendix M.
DRBA Backhone Model
Planning
Insight Recall
Factuality
Distractor Avoidance
Report Quality
Harmonic Mean
GPT-5
36.52
72.11
93.22
93.41
63.81
GPT-5
SRP
35.41
69.42
94.67
93.88
62.64
GPT-5
CRP
37.48
62,33
91.71
92.03
62.02
Jamms 3.1.:-3.--0000 405B Instruct
16.1
69.3
95.2
88.6
40.68
Llama-3.1-400 I-10SB-Instruct
SRP
15.7
89.7
40.15
Llama-3. 1aae:3.1-405B-nnttteee
18.33
DeepSeck-V3.. V3.1
22.6
49.20
DeepSeek V3.1
DeepSeck-VS. V3.1
CRP
28.21
67.09
93.96
85.57
55.03
Qwen- 2.m-222---B-nttnee
22.8
63.2
95.4
86.9
48.98
Qwen 2.5 72B Instruct
SRP
20.9
61.1
95.1
85.2
46.26
1oonn225-72B-nnsttee
CRP
24.39
55,74
95.12
87.51
49.46
improvements in both insight recall and report quality, suggesting that dynamically adapting the plan during execution helps the agent recover missed evidence and refine its use sources. However, combining CRP or SRP with AAP does not yield clear gains, and in some cases reduces factuality, likely because overlapping strategies create redundant or unstable planning behavior. These findings indicate that adaptive mechanisms are key for improving coverage of injected insights, while lightweight planning is more effective for maintaining factual grounding, and that carefully balancing the two remains an open challenge. In particular, for stronger backbones such GPT-5, complex planning promotes stepwise evidence aggregation across multiple files, which improves recall for insights that require combining numeric values with specific time periods or business details. See Appendix N for detailed results for each task.
ABLATION: EFFECT OF BACKBONE LANGUAGE MODEL ON DRBA
We evaluate the impact of backbone language models on DRBA using the FullBenchmark subset for controlled comparison. As the results in Table 3 shows, GPT-5 achieves the best balance of factual grounding, insight recall, and report quality. Open-source models show mixed results: Llama-3.1-405B excels in factuality but lags in recall, DeepSeck-V33 V3.1 delivers balanced performance through targeted fine-tuning, and Swen-225-722 is reliable but trails GPT-5. These results underline the importance of backbone choice; larger and more advanced models generally yield stronger overall performance, though some open-source options are competitive in specific metrics. addition, our experiments reveal significant limitation in agents' ability to retrieve critical insights from the open web. As shown in Table 22 in Appendix M, no agent successfully sourced external knowledge, highlighting the difficulty extracting relevant information for deep research applications within an unboundedly large search space.
5.4 QUALITATIVE ANALYSIS
In Table 4 we show sample of three groundtruth insights well the predicted insights from using both Llama 3.1 405B and GPT-5. We see that for the first insight, both models are able to effectively recover the groundtruth insight. For the second insight GPT-5 can extract the relevant time of year, where
8

---

Published as a conference paper at ICLR 2026
as Llama-3.1 405B fails to do This possibly suggests that GPT-5 may be better at extracting fine details. These examples highlight that successful insight recall requires more than identifying salient numbers; it also requires correctly binding those values to their associated temporal business context. Models that fail to perform this binding often produce superficially plausible but incomplete insights, which are scored unsuccessful despite containing correct numeric fragments.
We observe systematic qualitative differences in how models recover injected insights. As illustrated in Table 4, larger backbones such as GPT-5 more reliably recover both numeric values and their associated contextual qualifiers (e.g., time period or business condition). In contrast, smaller or weaker models often restate isolated numeric facts without correctly attaching the relevant temporal or semantic context, resulting in lower insight recall despite surface-level correctness. Across all backbones, agents consistently fail identify when required information is missing from private files and should instead be sourced from the web. For example, for the question "How can Lee's Market leverage FSMA 204 regulations to enhance food safety and customer trust?", FSMA 204 does not appear in the enterprise documents. However, agents generate broad queries such as "grocery store customer trust" or "food safety best practices", rather than targeted searches for FSMA 204 regulations. As a result, no agent successfully retrieves the relevant regulatory content, highlighting limitation in detecting and acting upon missing- domain signals. This failure reflects limitations in problem scoping and query formulation rather than search execution itself, suggesting that robust missing- -iksllg-knowledge detection is prerequisite for effective use of open-web retrieval in deep research settings.
Table 4: Insights Recall Improvement Areas (Task DR0002). We highlight in bold where each model was able to accurately find details relevant to the groundtruth insight. We also show the corresponding score where 1.0 is considered a successful recall and 0.0 an unsuccessful recall. The full table with all groundtruth insights and predicted insights is given in Appendix G.
Groundtruth Insight
Insight Predicted by Ilama 3.1 405B
Insight Predicted GPT:5
online customers have inter- acted with personalized product recommen- dations, resulting in a 25% incresse in aver- age order valuee.
45 % of Lee's Market ouline customers engage with personalized product reommendations, resulting in 25% Increase in average order value. (Scor= 1.0)
of online customers engaged with per- sonalized product recommedaatins,, and among those engagers average order value increased by 25% (Score = 1.0)
85% of Loc's Market transactions are linked customer loyalty accounts of 2024.
85% of transactions are linked to loyalty accounts at Lee's Market, providing selid foundutin persoral- ized marketing and improving customer engagementt (Score 0.0)
As of Q2 2024, 85 %5% of transactions were linked to loyalty accounts, leaving 15% unlinked identity gap.
(Score = 1.0)
5.5 PERFORMANCE OF WEB AGENTS ON DRBench
We evaluated Generic WebAgents from AgentLab in a browser-only setting (without API access). The 1-powerd agent achieved only 1.11 % insight recall, 6.67% factuality, and 33.07% % report quality. While the reports appeared well-structured, they lacked grounded insights, with most trajectories degenerating into repetitive clicks irrelevant files or windows. This shows that browser-only agents are currently far from effective for deep research tasks. Further trajectory examples are shown in Appendix J.
5.6 APP-BASED ENVIRONMENT VS LOCAL ENVIRONMENT
In Table 5, we compare results across two settings in DRBench: (1) local, where all the task files (e.g.. PDFs, PPTX, DOCX, XLSX, chats) are directly passed to the agent, and (2) app-based, where the same files must be retrieved through our standard enterprise environment and its apps, introducing additional interaction complexity. We find that OpenAl's Is Deep Research (GPT-5) achieves the highest scores across all metrics. Our agent with GPT-5 and DeepSeek backbones achieves similar performance to Perplexity in the local-only setting, but lags behind OpenAl and Gemini. In the app-based setting, performance declines across both backbones, highlighting the added difficulty of navigating multi-ppllication environments. This gap underscores that the environment in DRBench is intentionally challenging, enabling more realistic evaluation of model capabilities in enterprise research scenarios.
We observe that weaker backbones experience substantially larger performance drops when transitioning from the local to the app-based environment, particularly in insight recall and factuality. This degradation reflects the increased difficulty of multi-step navigation and cross-application context switching. In

---

Published as a conference paper at ICLR 2026
Table 5: Model Performance Comparison Across Local App-based Environments on the FullBenchmark. Note that higher numbers correspond to better scores, and the best result on each metric is bolded.
Enw
Insight
Factuality
Avoidance
Report Quality
(GPT:5)
App
36.52
72.11
DRBA (GPT-5) CRP
App
37.48
62.33
91.71
92.03
62.02
DRBA (DekpSeck V3.1)
App
22.6
68.4
94,9
84.1
49.20
DRBA (DegpSck-V3.I)> CRP
App
28.21
67.09
93,96
85,57
55.03
DRBA (GPT:S)
Local
38.91
75.84
95.46
92.18
66.43
DRBA (GPT-5) CRP
Local
39,74
77.02
95.21
93.37
67.39
DRBA (DeepSeck
Local
31.02
72.31
95.64
86.94
58.80
DRBA (DepSeck-V3.J) CRP
Local
31.88
73:42
95.27
87.86
59.82
Perplexity
Local
34.21
76.08
OpenAl Deep Research (GPT-5)
Local 41.%6
83.64
96.88
93.02
70.35
Gemini
Local
40.88
81.32
96.41
91.54
68.90
contrast, stronger models such as GPT-5 exhibit more stable performance, indicating greater robustness interaction complexity.
HUMAN EVALUATION
Quality of Deep Research Questions. We evaluated the quality of the deep research questions in DRBench through a human study with five expert annotators across the first 15 tasks. Each task was judged on three criteria: (1) grounding in the external website, (2) relevance to the domain and company context, and (3) alignment with associated insights. Annotators provided binary ratings plus optional feedback. Results show strong quality: 12 tasks received unanimous approval, while only three (tasks DRI, DRII, and DR13) received a single negative vote due to minor issues with specificity distractor difficulty, This corresponds to a 96% approval rate (72/75 votes).
Correlation of Used Metrics with Human Preference. We collected human preference subset of tasks'. Each annotator was shown a groundtruth insight with aligning insights from two models" and asked to choose which they preferred, label both as good/bad. Missing alignments as empty strings. We compared agents with AAP no RP against GPT-5 and Llama-3.1 400B-Instruct. The Fleiss x (Fleiss, 1971) across five annotators was 0.67. Most outputs were judged both bad due to missing alignments, but when preferences were expressed, GPT-5 was favored 61.1% % over Llama-405B-Instreee consistent with metric-based findings Section 5.3. Additional analyses
shown
Appendix
Human Validation of the LLM--a-a-Judge Evaluation To validate the reliability of our LLM-as-a-judge evaluation protocol, we conducted an additional human study. Specifically, we recruited four human evaluators and asked them to determine, for each predicted insight, whether it appeared in the corresponding ground truth insight list. We then compared the human judgments against the LLMM-----judg- decisions used in our benchmark. Across 264 evaluated insights, we observed over 91.3% % agreement between the human annotators and the LLM-based judge.
further measured inter-rater alignment using Cohen's is x (Cohen, 1960), obtaining score of 0.683, which indicates substantial agreement (Gwet, 2001). These results confirm that our LLM-assa-judde setup closely aligns with human judgment and provides reliable and scalable evaluation mechanism for insight recall.
7 CONCLUSION
In this work, introduced DRBench, benchmark for evaluating AI agents complex enterprise deep research tasks that require reasoning over both public and private data. DRBench provides 100 persona-grounded tasks situated in realistic enterprise environments, integrating heterogeneous data formats and real- world applications. We also presented DRBench Agent (DRBA) as a strong baseline and analyzed its behavior across planning strategies and backbone models. Our results show that while agents effectively avoid distractors and produce structured reports, they still struggle to reliably extract decision-critical insights. Adaptive planning improves insight recall, whereas simpler strategies better preserve factual accuracy, highlighting a fundamental trade-off between exploration and reliability.
Wee selected tasks with fewer than 8 insights for a reasonable amount of manual work.
"The gold-prediction alignment provided by the insight recall metric.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This work raises important considerations around data privacy, fairnsss, and potential misuse. Although DRBench simulates enterprise research environments with private data, all datasets are synthetically generated or drawn from public, time-invariant web sources. No personal or sensitive user data is included. The synthetic personas and companies are fictional, designed to prevent any risk of harm or re-identification. We highlight that agents evaluated on DRBench must handle sensitive-like contexts (e.g., healthcare, compliance, cybersecurity), which underscores importance of designing systems that prioritize data protection and avoid exposing private enterprise content. Human annotators were involved validating task quality; they were compensated fair rates and gave informed consent.
Large Language Models (LLMs) were used solely to assist with polishing the writing of this paper, such as improving readability and clarity of exposition. All ideas, experimental designs, implementations, analyses, and conclusions are original contributions of the authors.
REPRODUCIBILITY STATEMENT
We have taken multiple steps ensure reproducibility. The DRBench benchmark, including all generated tasks, data generation scripts, supporting files, and evaluation scripts, will be released under a permissive license. Each task is fully self-contained with dated URLs for public insights and synthetic enterprise files for private insights, ensuring stability over time. Detailed descriptions of the task generation pipeline, environment implementation, evaluation prompts, and cost considerations are included in the supplementary materials. We provide open-source code for running agents in the DRBench environment and for reproducing all reported results. Hyperparameter settings, backbone models, and planning strategies are documented. Together, these design choices make our benchmark transparent, reproducible, and extensible for future research.
LIMITATIONS
While DRBench captures realistic enterprise deep research workflows, has several limitations that reflect deliberate design trade-offs. First, the benchmark currently covers finite set of enterprise task families and domains, prioritizing depth and realism over exhaustive breadth; expanding to additional task archetypes (e.g., cross-team decision making, longitudinal audits, or incident response) remains an important direction for future work. Second, our evaluation operates at the level of atomic insights rather than span- or token-level grounding, which limits fine-graine, attribution analysis but enables scalable, robust assessment of long-horizon research outputs; incorporating more granular grounding signals a promising extension. Finally, despite strong human validation results, our evaluation pipeline retains residual dependence on LLM- judge methods, which may introduce subtle biases; future iterations will explore hybrid automaic evaluation and alternative grounding- aware metries to further strengthen reliability.

---

Published as a conference paper at ICLR 2026
REFERENCES
Amirhossein Abaskohi, Amrutha Varshini Ramesh, Shailesh Nanisetty, Chirag Goel, David Vazquez, Christopher Pal. Spandana Gella, Giuseppe Carenini, and Issam H. Laradji. AgentAda: Skill-adaptive data analytics for tailored insight discovery, 2025. URL
hetpps//arxiv.ort aaaaab/////// /wwp///4............ 07422..
Reyna Abhyankar, Qi Qi, and Yiying Zhang. OSWorld-Gold: Benchmarking the efficiency of computer-use agents. In ICML 2025 Workshop Computer Use Agents, 2025. URL
All-HandsAL. OpenHands. https://githinn htt.thhh..cooooecoooo MAll--aand-AA/OpenHHaaa 2024.
Accessed: 2024-06-01.
Pierre Andrews, Amine Benhalloum, Gerard Moreno-Torres Bertran, Matteo Bettini, Amar Budhiraja, Ricardo Silveira Cabral, Virginie Do. Romain Froger, Emilien Garreau, Jean-Baptisee Gaya, Hugo Laurençon, Maxime Lecanu, Kunal Malkan, Dheeraj Mekala, Pierre Ménard, Grégoire Mialon, Ulyana Piterbarg, Mikhail Plekhanov, Mathicu Rita, Andrey Rusakov, Thomas Scialom, Vladislav Vorotilov, Mengjue Wang, and Ian Yu. ARE: Scaling up agent environments and evaluations, 2025. URL patps://arxiv..r org/ alos 2509. 17158.
Léo Boisvert, Megh Thakkar, Maxime Gasse, Massimo Caccia, Thibault Le Sellier de Chezelles, Quentin Cappart, Nicolas Chapados, Alexandre Lacoste, and Alexandre Drouin. WorkArena++: Towards compositional planning and reasoning basnggg--bbdddd- common knowledge work tasks. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL httts:://penreet
net/ wtt.:/formmiiim JKerrummmddPCCKCddppaC
Nikos Bosse, Jon Evans, Robert G Gambee, Daniel Hnyk, Peter Mühlbacher, Lawrence Phillips, Dan Schwarz, Jack Wildman, et al. Deep Research Bench: Evaluating AI Web Research Agents. arXiv preprint arXiv:2506. 062877 2025.
Zijian Chen, Xueguang Ma, Shengyao Zhuang, Ping Nie, Kai Zou, Andrew Liu, Joshua Green, Kshama Patel, Ruoxi Meng, Mingyi Su, Sahel Sharifymoghaldam, Yanxi Li, Haoran Hong, Xinyu Shi, Xuye Liu, Nandan Thakur, Crystina Zhang, Luyu Gao, Wenhu Chen, and Jimmy Lin. BrowseComp-Plus: A more fair and transparent evaluation benchmark of Ceep-Research agent, 2025. URL https: / arxiv.. org/ abs/ 2508. 0660C.
Thibault Le Sellier De Chezelles, Maxime Gasse, Alexandre Drouin, Massimo Caccia, Léo Boisvert, Megh Thakkar, Tom Marty, Rim Assouel, Sahar Omidi Shayegan, Lawrence Keunho Jang, Xing Han Lù, Ori Yoran, Dehan Kong, Frank F. Xu, Siva Reddy, Quentin Cappart, Graham Ncubig, Ruslan Salakhutdinov, Nicolas Chapados, and Alexandre Lacoste. The BrowserGym ecosystem for web agent research, 2025. URL https::///.......... / mrxii.i.. org/ abs/ / 2 422.2..4 054
Prafulla Kumar Choubey, Xiangyu Peng, Shilpa Bhagavath, Hsiig Huang, Caiming Xiong, and Chien-Sheng Wu. Benchmarking deep search over heterogeneous enterprise data. In Saloni Potdar, Lina -ojaa-Barahona, and Sebastien Montella (eds.), Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track, pp. 501-517, Suzhou (China), November 2025. Association for Computational Linguistics. ISBN 979-8-89 1119988891776 19176-333-3 doi: 10.18653/v/2225. ennlp-indussry.34 URL https: /a//a/aannnntoooooooooooooo org / 2025. www.eemlleeeeee inw.ssrndutyy... 34/
João Coelho, Jingjie Ning, Jingyuan He, Kangrui Mao, Abhijay Paladugu, Pranav Setlur, Jiahe Jin, Jamie Callan, JoNo Magalhães, Bruno Martins, al. Deepresearchgym: A free, transparent, and reproducible evaluation sandbox for deep research. arXiv preprint arXiv:2505.19255 2025.
Jacob Cohen. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37-46, 46, 1960.
Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H. Laradji, Manuel Del Verme, Tom Marty, David Vazquez, Nicolas Chapados, and Alexandre Lacoste. WorkArena: How capable are web agents at solving common knowledge work tasks? 2024.
Mingxuan Du, Benfeng Xu, Chiwei Zhu, Xiaorui Wang, and Zhendong Mao. DeepResearch Bench: A comprehensive benchmark for deep research agents. arXiv preprint arXiv:2506. /763, 2025.
12

---

Published as a conference paper at ICLR 2026
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Ilama herd models. arXiv preprint arXf::240.21178,, 2024.
Joseph L Fleiss. Measuring nominal scale agreement among many raters. Psychological bulletin, 76 (5):378, 1971.
FoundationAgents. OpenManus. 2024. Accessed: 2024-06-01.
Boyu Gou, Zanming Huang, Yuting Ning, Yu Gu, Michael Lin, Botao Yu, Andrei Kopanev, Weijian Qi, Yiheng Shu, Jiaman Wu, Chan Hee Song, Bemal Jimenez Gutiemrez, Yifei Li, Zeyi Liao, Hanane Nour Moussa, TIANSHU ZHANG, Jian Xie, Tianci Xue, Shijie Chen, Boyuan Zheng, Kai Zhang, Zhaowei Cai, Viktor Rozgic, Morteza Ziyadi, Huan Sun, and Yu Su. Mind2web Evaluating agentic search with agent-as-a-judge In The Thirty-ninth Annual Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2025. URL httpp:://ppenreiiww. wwtp:ewwwwwwweeeeeeee.e..eeeee
2:
Kilem Gwet. Handbook of inter-rater reliability. Gaithershurg, MD: STATAXIS Publishing Company, pp. 223-246, 2001.
Kung-Hsiang Hsiang Huang. Akshara Prabhakar, Sidharth Dhawan, Yixin Mao, Huan Wang, Silvio Savarese, Caiming Xiong, Philippe Laban, and Chien-Sheng Wu. CRMArena: Understanding the capacity of LLM agents to perform professional CRM tasks in realistic environments. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1; Long Papers), 2025a.
Kung-Hsiang Huang, Akshara Prabhakar, Onkar Thorat, Divyansh Agarwal, Prafulla Kumar Choubey, Yixin Mao, Silvio Savarese, Caiming Xiong, and Chien-Sheng Wu. CRMArena-Pro: Holistic assessment of LLM agents across diverse business scenarios and interactions. arXiv preprint arXiv:2505.18877 2025b.
HuggingFace. smolagents. Accessed: 2024-06-01.
2024.
Abhinav Java, Ashmit Khandelwal, Sukruta Midigeshi, Aaron Halfaker, Amit Deshpande, Navin Goyal, Ankur Gupta, Nagarajan Natarajan, and Amit Sharma. Characterizing deep research: benchmark and formal definition, 2025. URL httpp/:///aiii..o//aa/////////// 04183.
LeamingCireuit. Local deep research. httpps//ggituuub.cmmrrrrrngggggi llocllddep-reseee 2025.
Yao Liu, Deming Ye, Shuohang Wang, Furu Wei, Yujia Ma, and Minlie Huang. G-Eval: NLG evaluation using GPT-4 with better human alignment. EMNLP, 2023.
Xing Han Là, Zdenčk Kasner, and Siva Reddy. WebLINX: Real-word world website navigation with multi-ture turm-itum dialogue, 2024. URL httts:t/ //arxi/aaiiviia/ii org/ haap///b///// 2ttp:///22..5.99
Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. GAIA: a benchmark for general Al assistants. ICLR, 2024.
Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. FActScore: Fininnr atomic evaluation of factual precision in long form text generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 12076-12100, 2023.
OpenAl. text-embedding- 3--ggr.
https: httt:::/pplaffottt openai wttp:cccccccc//////ee//eoc embeddings, 2024. [Large language model]. Accessed September 2025.
Gaurav Sahu, Abhay Puri, Juan A Rodriguez, Amirhossein Abaskohi, Mohammad Chegini, Alexan- dre Drouin, Perouz Taslakian, Valentina Zantedeschi, Alexandre Lacoste, David Vazquez, et al. InsightBench: Evaluating insight extraction for business analytics agents. ICI.R, 2025.
13

---

Published as a conference paper at ICLR 2026
Zilliz Tech. Deep-Searcher. Accessed: 2024-06-01.
2024.
Yuhao Wang, Ruiyang Ren, Junyi Li, Xin Zhao, Jing Liu, and Ji-Rong Wen. REAR: A relevance-aware retrieval-augmentted framework for open-domain question answering. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 5613-5626, 2024.
Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh J Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. NeurIPS, 2024.
Frank F. Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, Zora Z. Wang, Xuhui Zhou, Zhitong Guo, Murong Cao, Mingyang Yang, Hao Yang Lu, Amaad Martin, Zhe Su, Leander Maben, Raj Mehta, Wayne Chi, Lawrence Jang, Yiqing Xie, Shuyan Zhou, and Graham Neubig. Benchmarking LLM agents on consequential real world tasks, 2025a. URL
TheAgentCompany:
Renjun Xu and Jingwen Peng. comprehensive survey of deep research: Systems, methodologies, and applications. arXiv preprint htti::2506..25594 2025.
Tianze Xu, Pengnui Lu, Lyumanshan Ye, Xiangkun Hu, and Pengfei Liu. ResearcherBench: Evaluating deep research systems on the frontiers scientific inquiry. 2025b. URL
https://arxiv wwweriii..orraaaaa alss//b/////////// 2507 0.507.16200 1628C.
Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu. DeepResearcher: Scaling deep research via reinforcement learning in real-world environments. arXiv, 2025.
Shuyan Zhou, Frank Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. WebArena: A Realistic Web Environment for Building Autonomous Agents, April 2024. URL http:/// http///axvvv..//// org/ /b//////////// 2307 13854. arXiv:2307.13854 [cs].
14

---

Published as a conference paper at ICLR 2026
Table 6: Comparison of Deep Research Tasks of Different Benchmarks.
Benchmark
Sample Question
DeepResearchGym (Coelho et al., 2025)
Is the COVID vaccine dangerous
Deep Research Bench (Bosse et al., 2025)
Find a reliable, known number on the internet. The total number of FDA Class II Product Recalls of medical devices.
DeepResearch Bench (Du et al., 2025)
While the market features diverse quantitative strategies like multi-factor and high-frequun frequueyy trading, it lacks a single, standard- ized henehmark for assessing their performance across multiple dimensions such as returs, risk, and adaptability to market conditions. Could develop general yet rigorous evaluation framework to enable accurate comparison and analysis of various advanced quant strategies?
ResearcherBench (Xu et al., 2025b)
Compare the Transformer and Marnba model architectures, analyz- ing their performance and technical characteristics in different appli- cation scenarios. Based on latest research, discuss the advantages and disadvantages of both models and their applicable scenarios. For complex reasoning tasks (e.g., lasks involving multiple citations extended reasoning chains), what are the strengths of current agent technologies, and what are their limitations? Please analyze this in the context of research since June 2024.
LiveDRBench (Java et al., 2025)
BrowseComp-Piut (Chen et al., 2025)
Identify the title of research publication published before June 2023, that mentions Cultural traditions, scientific processes, and culinary innovations. is co-authored by three individuals: one of them was an assistant professor in West Bengal and another one holds a Ph.D. Update all my contacts aged 24 or younger to he one year older than they currenty.
GATA2 (Andrews al., 2025)
DRBench
How can Lee's Market leverage FSMA 204 regulations to enhance food safety and customer trust?
A COMPARISON OF DEEP RESEARCH BENCHMARKS AND AI AGENT BENCHMARKS WITH A COMPUTER ENVIRONMENT
In Table 1, we compare existing deep research benchmarks and AI agent benchmarks that provide computer environment with DRBench. While the questions in existing benchmarks focus on public interest topics and require generic web search and computer use, DRBench provides realistic questions that real personas in organizations need to resolve.
B DRBench TASKS
As shown in Tables 29, 30, 31, 32, 33, and 34 DRBench contains 100 tasks in total, covering industries (retail, healthcare and electric vehicles), 10 task domains (compliance, sales, customer relationship management, market analysis, customer service management, IT service management, cyber security, marketing, quality assurance, and research), and 3 difficulty levels (easy, medium, hard). In addition, we generate the following 3 companies (one for each industry type): (I) a supermarket chain called Lee's Market, (2) a virtual healthcare company called MediConn Solutions, and (3) an electric vehicle company called Elexion Automotive.
Table presents deep research question from DRBench and its supporting groundtruth insights. We also visualize the DR Question and all QA pairs by embedding them with OpenAl's text rext-embedding 3-large model and projecting into 2D using t-SNE in Figure 4. The plot shows that injected supporting insights lie closer to the DR Question, while distractors appear farther away, confirming that our injected insights are semantically aligned with the research objective.
C DRBench EXAMPLES OF INJECTED INSIGHTS
As shown in Figure 2, supporting documents are generated with enterprise insights injected. In Figure 5, show two examples of generated files (PPTX and Mattermost chat) with their embedded insights.
15

---

Published as a conference paper at ICLR 2026
How can Lee's Market leverage chatbots to cahance the consumeer experience for cestennial shoppers?
Toal de saperiag- didinake
bc Acceenng mat lig buen gni caploon wir in of 2004
*
DR Quoste Ceppeeaggssefoooo Domawee13))
Anauna Mltieiiannn on yele oy becuee sereeng 1981
Khe dny benooroooooee hel cmplerees of Iominyy 2004
Smglign kutinotoe ancaad D2s
Eireell peuuuce doval
odaa dagae mocdy by Sid.
Poceetttttab wntees 2e Officcet
O poreoreeeee sererreeee Lan's Marke museem le cosrrena oa Alle 30e
Mif
2004
*
2014
D111 2001
Cin aup cai matice 2J Luity Saaua OI 3007
Varte
A Averge cand maiaamaen nes S2/0 per mank wrthe Sli
huillyy wuth sland
[50 Supurciag caaaaat Sccrral
H&NE Draussion
Figure 4: t-SNE visualization of QA pairs for the DR Question in Task DR0005. The plot shows the distribution of annotated pairs across Supporting Insights (green), Distractors (red), and the central Deep Research (DR) Question (gold star). Out of 49 pairs, 16 correspond to supporting insights and 33 are distractors. The visualization illustrates how relevant insights cluster separately from distractors, highlighting the challenge of retrieving salient information in a distractor- heavy environment.
Enhancing Food Safety Through Regulatory Compliance
Introduction
U.S.
with regplestyy requiama Altion The Sublyy Moose reution poon costoree iruit cord requtemeet la tive busreas moodorioo po volliiiey addnosinng roy Aibry char eus no can producta
Risk-Based Preventive Controls for Supply Chain Management
consa qustarly perfomcanoo proorduras, touutng suiget sdins vesdeo howo ieplomeeen qualit bravi alanpie 19% splort. lo we aie sadiiinel copppets neer guilly and powedh releesaly heaking bhy ho and ei CSS 2000, we nowe aue sigiinn ang lod in the expactation..
Leveraging Technology for Enhanced Traceability and Transparency
Our IT dei stroee bean king on imagrota oe debearies resturce pil eningg (3R12) syetare madi The prijec socf caposhadt io soing und ridd 2000, will sroekdo roalt ti engins int This
Menoadinng
tnewase sa riveue as rass of ths inlutte..
(a) Example supporting file named food-safery- regulatory regulatoryccmppionc... with an injected insight
(b) Example supporting email in Sent mailbox with an injected insight "85% of high-risk food suppliers are FSMA 204 compliant, toraling 270 vendors."
Lee'' 's Market reduced food waste by 8% in Q2 2024, saving S1.2M."
Figure 5: Example files with injected insights in DRBench.
16

---

Published as a conference paper at ICLR 2026
Table DRBench Questions and Statistics. Industry: target industry of the deep research question. Domain: domain of the deep research task. DR Question: question of the deep research task. Difficulty: difficulty of the task defined based on the rubric mentioned in Section # Applications: the number of total applications in the task environment. # Insights: the number of relevant insights to the deep research question. # Distractors: the number non-supportings documents that do not contain relevant insights. Please refer to Tables 29, 30, 31, 32, 33and 34 for details on the details on the remaining 60 tasks.
Indusiry
Quesiion
Dilliculty casy
Applications
Insights
Disraators
Ketail
Complance
How can Lee's Market leverage PSMA 204 reg-
2
ulations entience food safery and customer tnue?
Resal
can pesomaliation drive sales
retini
etsy
2
industry and what stratgics can be used for Lee's
Market in action?
Retail
CRM
How leveniee data drvvn loyslty programs
medium
4
enhense customer enpagement?
Retni
Market Analyis
What are the new wends in the grocery retinl
medium
3
14
murket what stratgicss Lee's Market adop: remsin compesiove?
Resail
CSM
Flo can Lee's Market levenger chafbots to onhunce the consumeer experience for coniennial shoppers?
hudl
4
16
33
Healthcre
Complanee
What are the key factoes inftuencing MecaCoon Conn Solutions decision accept insarance for Iele- health provider, considering (IPPA compliance and specitic dasa privacy regulation?
easy
3
Healtheae
ISM
How can we lovenge ISTM and Al-driven zan- lytis minimiar IT service desk worklad and improve repponar times Mr Solations?
ensy
3
12
Healtheaee
Cyheraccuitt
What is the impact of thind-patty data brenches
medimn
4
MediConn's i vinual bealthowe platfomms and patient dats and what regulations can be implemented defond against these bessches?
Healthaare
CRM
How can MediConn Soluton lovenge tradyy new CRM solutions improve patient CRM solutonss are
madium
12
24
expectad 2025?
Healtheaee
Marketing
What wre the most critical claments of robust
hard
4
30
digital presence for telchealth provider such
MediConn Solutions,
how can we optimize
our website and costert marketing stitegy to omrct digitl-fiss patiems?
Lecine Velicle
Complance
we balance the need durakility and warmany guaramtees for batteries with evohing regulatory rquinements, especially ACC regulations LACC ID, while staing trac with our production timelines through 2035?
easy
Flectric Vehicle
Quality Assrrance
How can Flexion Auterrstvve's quality assurnnce procesess he optimized to addies the unique challenges electri vahicle production, such software and user experience isses, comprred gasolinc cars?
ensy
Elecie Vehicle
Cybeersccurty
Elexson Automotre effectively im- plament cybersccuityy stratgy its electric vehicles, considering the ridks and challenges
mediutt
3
posed by connected and autonemous technnlogies?
Electric Vehicle
Research
Can
Al-eshinced hattery maragement to inprowe EV buturry lifespm by 1587
madium
Elecine Velicle
CSM
How can Liction Automotive incresse customer
hand
15
30
tnst thouug after vales spport while buluning
need esceptional customer care
efficient and cost effective service?
D DRBENCH ENTERPRISE ENVIRONMENT
The DRBench Enterprise Environment provides containerized simulation of realistic enterprise research settings where employees access confidential company information, personal files, and internal communications for comprehensive report generation. The environment simulates both user's local machine filesystem and provides password-protectee access to enterprise services.
To emulate realistic enterprise research settings, DRBench provides a self-contained Docker environment that integrates commonly used applications: Nextcloud for shared documents, Mattermost for internal chat, an IMAP server and Roundcube open- client for emails, and Filebrowser to emulate local files. Each task is initialized by distributing its data across these services, enabling agents to retrieve, analyze, and cite information through enterprise-like like interfaces rather than static files. This design ensures realistic interaction while maintaining reproducibility and controlled evaluation.
17

---

Published as a conference paper at ICLR 2026
Table 8: Example Deep Research Question and Supporting Groundtruth Insights
Deep Research Question
Supporting groundtruth insight
Insight Category External
U.S. grocers are working moet the FDA's s FSMA 204 iracaiilityy rules by Jamary 2026, which roquire tracking lot codes and key data for high-ris
How can Lec's Market leverage FSMA 204 regulations to enhance lond salety and customer trusf?
foods to expediite recalls. This complannee is viewed as an "evolutionary step" modernize grocery operations and enhance food safety.
By capturing detailed traccability data, such as for codes, at every step. retailers
External
regulations and gain inventory henefits, This allows grocens to know exact expintion dates by lot, enabling them to discount items before they expire, thus reducing food waste and keeping products fresher.
Regional groces like Lunds & Byerlys and Raley's see FSMA 204 chance to enhumce their systems and supply chain tramqurrncy. They helieve improwed traceability will boost customer and could signal the start of more extensive future food safety regulstions.
Exteral
tacks high-nik food products Q3 2024, affecting 30% of inventory.
Internal
Loo's Miket reduced food waste by 8% in Q2 saving $1 $1.2M.
Insemall
85% of high-risk lood suppliers FSMA 204 compliani, lotaling 270
Internal
vendors.
D.I. ARCHITECTURE AND SERVICES
The environment implements multi-service architecture within single Docker container. This design prioritizes deployment simplicity and cross- ross-platform compatibility while maintaining service isolation. The container orchestrates the following enterprise services:
Nextcloud: Open-source file sharing and collaboration platform analogous to Microsoft
SharePoint Google Drive, providing secure document storage with authentication.
Mattermost: Open-source team communication platform simulating internal company commu- nications similar to Microsoft Teams or Slack, with teams, channels, and persistent chat history.
FileBrowser: Web-based file manager providing access to the container's local filesystem, simulating employee desktop environments and local document access.
Email System: Roundcube webmail interface with integrated SMTP (postfix) and IMAP (dovecot) services for enterprise email communication simulation.
VNC/NVNCC Desktop: Protocol and browser-based VNC access providing full desktop environment interaction within the container for comprehensive enterprise workflow simulation.
D.2 TASK LOADING AND DATA DISTRIBUTION
At initialization, the environment processes task configuration files (env. json) and distributes data across services through automated Python scripts and it makes sure that this source data is only accessible through the intended applications:
File Distribution:
Documents are placed in appropriate Nexteloud user folders and FileBrowser directories based on task specifications
Communication Import:
Chat histories and team conversations are imported into Mattermost
channels with proper user attribution
Email Integration: Email conversations are loaded into the mail system with realistic threading and metadata
User Provisioning: Enterprise users are automatically created across all services with consistent authentication credentials
18

---

Published as a conference paper at ICLR 2026
from drbench import drbenci_enteeprriee.eeo fask__oadee
Load task confiquration task setp_/aaaeeeeettsak_____e_
Iaitialize environment with automatic pozt allocation
drbench wawphheetteeppeeepeeeee env
wtwp:ppppw..eeccccceeeeeeeeeeeee start_container-Irte auto_ports-True Prevents port conflicts parallel execution
task-task task-task.get__ptt path
Envi ronment provi des service di discovery
available @applabee_apps env.get_avaf wawelaalle_pppa a) ()
Returna: ('nextcloud's portt" 8081, 'credentials'; (..t ad
Pass relevant Information to the agent
Cleanup when research complete
env. env.delete()
Listing DrBench Environment Usage
D.3. PYTHON INTEGRATION
The
class provides programmatic container management with the following capabilities: container lifecycle management, service access information, task-specific data loading, and automatic cleanup. The typical usage pattern shown in Listing demonstrates these integrated capabilities.
D.4 ENTERPRISE SERVICE APIS
Each service exposes both web interfaces for human and web-agent interaction, and programmatic APIs for agent access:
Nextcloud: WebDAV API for file operations, sharing, and metadata retrieval
Mattermost: REST API for message history, channel management, and user interactions
Email: IMAP/SMTP protocols for message retricval and sending
FileBrowser: HTTP API for filesystem operations and file management
This dual-access model enables both agent-driven research and human verification of enterprise scenarios, supporting comprehensive evaluation of research capabilities across realistic enterprise information architectures.
E DRBench EXAMPLES OF APPLICATION SCREENSHOTS
Figures and show the applications provided in DRBench environment: File Browser, Mattermost, Roundcube, and Nexteloud.
DRBench AGENT IMPLEMENTATION DETAILS
F.1 DETAILED WORKFLOW
As depicted in Figure 3, the workflow begins with Company Employee submitting enterprise Deep Research Question along with Company Context. The DRBench agent processes this input through several key stages:
19

---

Published as a conference paper at ICLR 2026
Jeauity 18n
(a) Screenshot of the File Browser interface, displaying organized files and folders within the system.
(h) Screenshot of the Mattermost communication platform, showing a discussion channel and user interface elements.
Figure 6: Screenshots of Applications in DRBench environment (Part 1).
(a) Screenshot of the Nextcloud file management system, illustrating the file list view with various document types.
(b) Screenshot of Roundeube, an email client, it shows an open email in the user's inbox.
Figure 7: Screenshots of Applications in DRBench environment (Part 2).
Stage 1: Research Planning. The agent decomposes the research question into structured research investigation areas to guide subsequent action generation. This initial decomposition lays out a strategy systematically cover the research space while maintaining focus on the initial deep research question.
Stage 2: Action Planning. The Action Planning stage translates the research objectives into executable actions through planning subsystem. This component uses an LLM to create a prioritized sequence of actions. Each action is parameterized with specific tool selection and execution parameters, its dependencies to other actions in the plan, and a priority score.
Stage 3: Research Loop with Adaptive Execution. The Research Loop iterates over the following sub-stages until completion: (1) Tool Selection and Execution: The tool selection and execution subsystem implements a sophisticated priority-based selection of actions from the plan at each research iteration step and proceeds to execute it with the current research context. (2) Content Processing: If necessary, the action will make use of the content processing subsystem extract, synthesize, and store retrieved documents and websites into a vector store to form a -sec-fpec knowledge base that will grow on each iteration. (3) Adap- tive Action Planning: After each execution round, the agent analyzes the most recent findings; if coverage gaps are detected, new actions are created and added to the plan at this point. This ensures that newly discov- ered knowledge is taken into account to answer the research question. (4) Iteration Findings Storage: Results from each iteration are stored in the vector store with rich metadata for traceability and later assessment.
Stage 4: Convergence and Completion. The research loop continues until all actions in the plan have been completed or the maximum iteration limit is reached.
20

---

Published as a conference paper at ICLR 2026
Stage 5: Report Writing. The report writing subsystem queries the vector store synthesize the research findings and relevant retrieved content. This component generates a comprehensive report and uses its citation tracking system to ensure proper attribution of claims within the report.
The vector store serves as the main knowledge integration component, maintaining embeddings of all processed content and enabling semantic retrieval at the report generation stage. This component is crucial a deep research setting to prevent information loss from early research stages in arbitrarily long research sessions.
F.2 RESEARCH PLANNING IMPLEMENTATION
The Research Planning subsystem offers three operational modes to evaluate the impact of structured planning on research effectiveness:
Generates a comprehensive research plan with detailed investigation areas. These areas contain details about the specific research focus, expected information sources, and success criteria, among others. Each area includes an importance level and specific business intelligence objectives (see Listing 2).
Complex Mode:
Simple Mode: Creates focused question decompositions with 4-10 self-contained subqueries derived directly from the main research question. Uses straightforward decomposition without the complex enterprise research structure of complex mode. See examples in Listing 4 and Listing 3 for comparison of different planning modes.
None: Bypasses structured planning entirely, proceeding directly to action generation based the original question. This mode serves baseline measure the added-value of explicit planning stages.
The planning process begins with the enriched query (Prompt 1) and uses the research planning prompt (Prompt 2) to generate structured outputs. In Complex Mode, the system creates detailed investigation areas with enterprise- ficcrprise-focused focused metadata, while Simple Mode produces straightforward question decompositions similar to existing multi-step reasoning approaches. The resulting plan structure directly feeds into the Action Planning System (Appendix F3) for executable action generation.
" "area_id"; "rseearch_fouu "Core strategic
domain, market segment, or business hypothesis to investigate", Airformation_needde needs": "What specific intelligence is required for strategic decisions"
},
"knowlccag_ssuuross"" ["internal", "external", "both"), "research_approen"
": "competitive__nalyp"" market_research strategic_assessment
trend_analysis risk_analysis "key_ see__ccncepss; ["concepti", "concept2"),
serfoornancebbencharrtie
"business wtw.inesss_ratiinael...... "Why this investigation
area is critical for enterprise strategy and decision-makinn exxpected_insiihtts.. "What strategic
understanding or conpetitive intelligence this area should provide", beakkhholderiimpacee "Which
business units or decision-makers will berefit from these insights", importance__evelll "critical | important | supplementary"
Listing 2: Investigation Area Structure for Full Planning Mode
"query": "What are the new trends in the grocery retail market and what strategies can Lea's a Market adopt to zemain competitive?", "plan":
21

---

Published as a conference paper at ICLR 2026
arrea id":
reseearch_focus; "Current trends grocery retail market", information_needa
"Latest consumer preferences",
"Emerging technologies influencing grocery shopping", "Sustainability practices in grocery retail", "Changes supply chain dynamics"
Sewnlleddd__sucees"ss. ["externai"), "knowledge_
aanpearchh_aproacce "trend anzayssss,
key_concepts
(Me-conmerce growth", "sustainability in supply chains"), "Understanding consumer trends and technological advancements that shape shopper betavior is critical for adapting offerings and enhancing customer engagement.", "expected_ sadd_iinsihts" "Identify specific trends affecting customer buying decisions, including the rise of online grocory shopping and preferences for sustainable, local, or organic products.", "stakcholder_impa"
"bssiness_rationaas
"Marketing, Product Development, Supply Chain Management"", iimportance_eevel1 "critical"
"area id": 2,
"research focus"; "Competitive analysis of grocery retsilers", "Informatioo_needds"
"Market share analysis",
Competitor strengths and weaknesses",
" "Inovvative strategies adepted competitors"
1,
stnowledge_sourees ["external"),
"escoarch_approacch:: "compctitive_analyee""
key key_cencepts"; ["market positioning", "competitive differentiation") "businness_rationaaee "A comprehensive understanding of compet ttrr allows for strategic positioning and the ident fication of innevative practices that
be adopted or improved upon.", "expected_insights"" "Detailed profiles of key
competitors, including strategic moves they aze making to capture market share, which can inform Lee's s Market's competitive strategy.", estakeholder_impert impact
"Executive Leadership, Strategic Planning, Marketing", "importance_level" "critical"
Listing 3: Complex Mode Research Plan Example
" "query": "How can leverage
data-driven loyalty programs to enhance customer engagement? "plan":
"mode": "simple",
"subqueries":
"What the key features
of successful data-driven loyalty programs in the retail industry?",
"How can data analytics be used to personalize rewards and incentives in loyalty prograns to increase customer engagement?",
"what types of customer data should be collected and ana mnalyzed optimize loyalty programa for company like Lee's Market?",
22

---

Published as a conference paper at ICLR 2026
"Which
technology platforns and tools are most effective for implementing and managing data-driven loyalty programs in the retail sector?", "How can Lee's s Market measure the success of their
loyalty program in terma of customer engagement and sales growth?", "What the best practices for integrating loyalty progran with existing marketing strategies to enhance customer experience?", "How can Lee's s Market ensure customer data
privacy and security while Leveraging data-driven Loyalty programs?", "What are the potential
challenges and limitations of impl Implementing a data-driven loyalty program for a retail comрany with Lee's s Market's a size and resources?"
,,
"research_ : {
"overall_appres "Query decomposition into focused subqueries"
Listing 4: Simple Mode Research Plan Example
F.3
ACTION PLANNING SYSTEM
The Action Planning System translates research objectives into executable actions through an intelligent planning subsystem that manages tools, prioritization, and dependencies.
Available Tools
Table 9 summarizes the available tools organized by category and their primary purposes.
Table 9: DrBench Agent Tool Categories and Purposes
Category
Tool
Purpose
Internet Search
Externll market research, competitive intel- ligence, and public data analysis. Ideal for market trends, competitor analysis, industry reports, news articles.
Information Retrieval
Enterprise API
Access to proprietary internal data through extensible adapters (Nextcloud, Mattermost, email, FileBrowser). Ideal for internal metrics, communications, confidential documents.
URL Fetch
Direct content extraction from specific URLs. Ideal for deep analysis of reports, whitepapers, case studies, competitor websites.
Analysis
Analyzer
Al-powered synthesis and analysis using vector search. Ideal for cross-referencing find- ings, identifying patterns, generating insights.
Local Processing
Local Search
Document
Semantic search within locally ingested documents. Ideal for targeted retrieval from local files with source references.
Priority Scoring and Dependencies Actions are assigned priority scores (0.0-1.0 1.0 scale) based on strategic importance and expected information value. The priority assignment follows enterprise research principles:
Source Type Prioritization:
Enterprise and local sources receive higher priority than external sources, reflecting the strategic value of proprietary information in competitive analysis.
Query Specificity: Targeted queries addressing specific investigation areas score higher than broad exploratory searches, ensuring focused research execution.
23

---

Published as a conference paper at ICLR 2026
Dependency Management: Actions can specify prerequisite relationships where certain information gathering must precede analysis or synthesis tasks. The scheduler respects these dependencies while maximizing parallel execution within each iteration.
F.4 ENTERPRISE INTEGRATION ARCHITECTURE
Service Adapters
The system implements extensible adapters for enterprise services including Nextcloud file server, Mattermost chat, IMAP email systems, and FileBrowser interfaces. Each adapter handles service-specific authentication, data retrieval, and metadata preservation for proper citation attribution.
Source Prioritization Strategy Enterprise and local sources receive priority multipliers in action scoring, reflecting the strategic value of proprietary information. The system maintains source type classification throughout the pipeline to ensure internal intelligence drives analytical conclusions while external sources provide context and validation.
F.5
TOOL SELECTION AND EXECUTION
The Tool Selection stage implements priority- based action selection and tool invocation within each research iteration to execute the action plan.
Action Selection Process At each iteration, the system selects executable actions based on priority scores and dependency satisfaction:
Priority-Based Scheduling:
Actions are ranked by their priority scores, with enterprise and local sources prioritized over external sources to maximize the value of specific private information.
Dependency Validation: The scheduler checks that all prerequisite actions have completed before making an action available for execution.
Sequential Execution: Actions execute one at a time in priority order, maintaining research coherence and enabling each action build upon previous findings.
Once selected, actions execute through a standardized interface and results are integrated into the research context, informing the following stage of adaptive action planning.
F.6 ADAPTIVE PLANNING
The Adaptive Planning system enables dynamic evolution of the action plan by analyzing results after each iteration to generate extra actions addressing information gaps.
It starts by analyzing the most recently completed actions and performs these two substages:
Source analysis and gap elassification. The system evaluates the possible imbalances in the action completion if information came from internal or external sources and identifies possible scenarios to cover.
Dynamic action generation. After analyzing the sources and results from the previous actions, the system makes LLM call to generate 1-5 extra candidate actions with specific prioritization. After candidate actions are generated, they go through a deduplication process to make sure the plan didn't t cover them already and incorporates the final subset into the priority action plan so they can be considered by the scheduler in the following iteration.
F.7 CONTENT PROCESSING AND VECTOR STORE
The Content Processing system implements a pipeline for unified ingestion of documents in multiple formats (PDF, docx, HTMLL JSON, plain text formats, etc.) that normalizes and cleans text inside the documents and websites retrieved during the research. Content is then deduplicated and chunkized, and embeddings computed for each of these chunks.
The Vector Store implements the storage and retrieval of the content via JSON metadata and NumPy embedding metrices, enabling semantic similarity and keyword based searches
24

---

Published as a conference paper at ICLR 2026
F.8 REPORT GENERATION
Multi-Stage Synthesis Pipeline The Report Generation stage implements a four-stage synthesis approach: (1) thematic content clustering via vector searches, (2) source prioritization and deduplication, (3) LLM Synthesis with source tracking, and (4) Final report writing and citation resolution.
The system generates targeted search queries based on the research plan, including specific to ensure that a predefined set of themes or sections (background, analysis, implementation, trends) are retrieved and written and prevents redundant analyses.
Unified Citation System The citation system implements deferred resolution to keep consistency of citations from section to section by referencing document IDs the Vector Store and carrying them over for each piece of synthesized text. A final citation resolution stage will assign the correct numbering to each document in the final report.
F.9 DRBA PROMPTS
The DRBench agent relies carefully designed prompts orchestrate enterprise research workflows. These prompts implement the core architectural principles: enterprise context enrichment, structured planning decomposition, priority-based action generation, quantitative synthesis requirements, and adaptive research capabilities. The following five prompts represent the critical LLM interactions that enable systematic enterprise research with proper source prioritization and citation tracking:
Enriched Query Generation (Prompt I): Transforms basic research questions into enterprise- contextualized queries by incorporating company information, stakeholder personas, and business context to guide subsequent research activities.
Research Planning (Prompt 2): Decomposes complex research questions into structured investigation areas with defined information needs, knowledge sources, and business rationales, enabling systematic coverage of the research space.
Action Generation (Prompt 3): Converts research objectives into prioritized executable
actions with tool specifications and dependency relationships, emphasizing enterprise source prioritization over external sources.
Adaptive Action Generation (Prompt 4): Analyzes research progress to identify coverage gaps and source imbalances, generating complementary actions that enhance research depth and cross-validate critical findings.
Report Synthesis (Prompt 5): Orchestrates quantitative-fii content synthesis with strict citation requirements, ensuring numerical data leads analytical paragraphs and all claims are properly attributed source documents.
4 Query Generation
Research Question:
fdr question]
Company Context: (company.name) is (company.desc).
This analysis is requested by {name), frole} in |department
Persona Context:
Their responsibilities include: (responsibilitiiss)
Prompt 1: Query Enrichment with Enterprise Context.
G QUALITATIVE RESULTS
Shown below are some illustrative examples of how metrics are computed for different scenarios on given test task.
25

---

Published as a conference paper at ICLR 2026
4
Research Planning (Complex Mode)
Design comprehensive enterprise zesearch strategy for: "(question)"
(tools_section
As senior enterprise researcher with deep business intelligence expertise, create a thorough investigation plan that conbines zigorous research methodology with strategic business analysis. Your goal is to provide insights that drive informed decision-making in complex enterprise ronnenttt
Enterprise Research lessign Principles:
Leverage proprietary internal data competitive advantage while ensuring external narket context
Design investigations that directly inform strategic decisions and business outcomes
Pricritize research areas that maximize ROI and strategic value to the organization
Balance comprehensive analysis with fecused insights relevant to enterprise objectives.
Generate JSON object with strategic research investigation areas:
(output.structure)
Prompt 2: Enterprise Research Planning with Investigation Areas. See example of the output structure in Listing 2
G.1 INSIGHTS RECALL
We showcase the insights recall metric using Task DR0002, whose DR Question is "How can personalization drive sales in the retail industry and what strategies be used for Lee's Market in action?" (also shown in Table 7), which evaluates sales challenges and competitive landscape analysis.
Table 10 shows the overall performance comparison between using Llama 3.1 405B and GPT-5 in our agent. Using GPT-5 in our agent results in increasing the insights recall score from 0.14 to 0.43, successfully answering 3 out of questions compared to Ilama's out 7.
Table 10: Insights Recall Performance Comparison: Llama 3.1 405B GPT-5 (Task DR0002). We summarize the number of questions answered successfully and unsuccessfully as well as the overall insights recall score for the given task.
Metric
Llama 3.1 405B
GPT-5
Improvement
Insights Recall Score
0.14
0.43
+0.29
Questions Answered
3/7
+2
Successfully
Questions Failed
6/7
4/7
-2
The question-by bisioon-b-qqusttion breakdown in Table 11 reveals the specific questions where each approach succeeded or failed. Both models successfully identified the insight related to online customer engagement, but only GPT-5 was able to identify the number of loyalty program members and the customer data collection rate. Neither model successfully answered the remaining questions, indicating these insights may not have been readily available in the source materials that agents struggled to find the right insights
26

---

Published as a conference paper at ICLR 2026
Action Generation
Generate specific executable actions for this research invest investigation. gation area with SOURCE PRIORITIZATION:
Research Focus: {research.focus}
Information Needs:
(information.need)
Knowledge Sources:
Vknowledge.souree
Research Approach:
(research.approcct)
Available Tools: favailable tool names)
Tool Selection Guidelines: (tooleguiellines]
Return JSON array of actions with:
"type": Acticn type (web.search, enterprise.apl, url.fetch, analyzer, localsscarch)
"description"; clear descripticn of what this action will aecomplish
"parameters": Teol-specific parameters including query, search.type,
"priority": Float 0.0-1.0 (enterprise sources: 0. 0.7-1.0, external:
0 4-400.7)
"expecteccoutputt": What information this action should provide
"preferred=too1": Specieic tool class
Prompt 3: Priority-Bassee -aseed Action Generation with Source Awareness
Table 11: Question-by- -uu--sto Insights Recall Analysis (Task DR0002). We breakdown the results question by question for the given task, highlighting specifically which question is answered correctly incorrectly for each model.
Question
Llama 3.1 405B
GPT-5
A
Online Customer Engagement with Personalized Recommendations
1.0
1.0
0.0
Number of Loyalty Program Members
0.0
1.0
+1.0
Customer Data Collection Rate
0.0
1.0
+1.0
Online Sales Growth
0.0
0.0
0.0
Effectiveness of Personalized Marketing
0.0
0.0
0.0
Personalized Promotions vs Mass Promotions
0.0
0.0
0.0
Retail Media Growth
0.0
0.0
0.0
Total Insights Recall Score
1.0/7.0
3.0/7.0
+2.0
In Table 12 we extend Table and show all the groundtruth insights well each of the predicted insights from using both Llama 3.1 405B and GPT-5. As before, highlight in bold where the model was able to accurately find details relevant to the expected insight, and show all the corresponding scores as given in Table 11.
G.2 FACTUALITY
The factuality metric evaluation uses the same Task DR0002 to assess the accuracy and reliability of generated content. Table 13 presents the factuality performance comparison, showing that while using Llama 3.1 405B achieved 0.41 factuality (7 factual claims out 17 total claims), where as using GPT-5 reached 0.65 factuality (13 factual claim out of 20 total claims). This represents significant improvement
27

---

Published as a conference paper at ICLR 2026
4 Adaptive Action Generation
Based the research progress far, suggest actions with INTELLIGENT SOURCE COMPLEMENTARITY.
Original Research Query: fresearch.query,
Completed Actions Summary: {completed.actions sarrooossssuurrrr
Recent Findings Analysis: (findings.json)
Source Composition Analysis: (Lnternal
Available Tools: lavailablee.tol..naa _names]
Generate 1-5 new actions that:
1. **Address gaps** in current research coverage
2. **Balance source types** if findings are heavily external,
prioritize internal sources and vice versa
3. **Build on discoveries** leverage new information to explore
related areas
4. **Enhacce depth*. dive deeper into promising
findings
5. s*Cross-validates verify critical findings through alternative sources
Each action should have:
Strategic rationale for why this action is needed now
Clear connection to research gaps promising
leads
Appropriate pricrity based on strategic value and source
balance
Specific parameters that build on existing knowledge
Prompt 4: Gap-Driven Adaptive Action Generation
in content reliability. This also highlights that GPT-5 is much better prepared to make accurate claims that are sustained by evidence.
Table 14 provides detailed breakdown of factual versus unfactual claims. The agent using GPT-5 generated additional factual claims while producing fewer unfactual claims, resulting in net improvement in accuracy percentage. This demonstrates that GPT-5 may generate higher proportion of factual information than Llama 3.1 405B.
The impact of these factuality improvements of using GPT-5 Llama 3.1 405B is summarized in Table 15. The 24.0 percentage point improvement factuality represents enhanced content quality and research reliability. The increase in total claims shows that GPT-5 can generate more content overall.
H EXPERIMENTAL SETTINGS
All experiments conducted on cluster of NVIDIA A100 GPUs (80GB each). For file generation and task construction, we primarily used the Llama 1amma-3.. 1-405B model, with decoding performed using nucleus sampling at temperature of 0.7 unless otherwise specified. For larger-scale evaluations of the DRBench Agent, we also used closed-source models such as GPT-4o and GPT-5, alongside DeepSeek models, to enable comparison across open- and closed-source backbones.
To ensure reproducibility, the DRBench environment was deployed as a self- ooffacont Docker container with all supporting applications (Nextcloud, Mattermost, and Filebrowser) pre-configured. Fach task was
28

---

Published as a conference paper at ICLR 2026
Table 12: Insights Recall Improvement Areas (Task DR0002). We highlight in bold where each model was able to accurately find details relevant to the groundtruth insight. We also show the corresponding score where 1.0 considered a successful recall and 0.0 an unsuccessful recall.
Groundtruth Insight
Insight Predicted by Liama 3.1 405B
Insigtt Predicted by GPT:5
45% of our ooline customers have interacted with personalized product recommentations, resulting 25% increase average onder valuee.
45% of Lee's Market online customers engage with personalized product recom- mendations, resulting in 25% increuse in average order value.
45% of online customers engaged with personalized product reommeenaatios, and amony those engagers iverage order value increased by 25%
(Soore 1.0)
(Scure
As Q3 2024, Lee's 's Market has 1.2 million koyally program members.
85% of transactions are linked to loyalty at Lee's Market, providing solid foundation for personalized marketing and improving customer engagement. (Score = 0.0)
From Q2 2024 to Q3 2024, loyalty members increused from 1,050,000 to 1,200,000 (+ (+150,000; +14.29%), avenge spend per member rose from 24
17%), and total member spend increased from 20,0000000 to 30,000,0000 (+19.05%).
(Score = 1.0)
85% of Lee's Market transactions linked to customer loyalty accousss as of Q2 2024.
85% of transactions linked loyalty accounts at Lee's Market, providing a solid foundation personalized marketing ingrroving customer engagemeent. (Score 0.0)
As of Q2 2024, 85% of transactions linked to loyalty accounts. leaving a 15% unlinked identity gap.
(Score 1.0)
Lee's Market online sales grew 12% Q2 2024 compared to Q2 2023.
45% of Lee's Market online customers engage with personalized product recommendations, resulting in a 25% increase in average order value.
A naive blended online AOV upside of approximately I 11.25% derived from 45% eogagement multiplied by a 25% AOV lift among engagers.
(Score 0.0)
(Score =.0)
Retailers excelling in personalized marketing are growing revenues about 10 percentage points faster their peets, according to BCG. By ellectively using first party customer leaders could unlock an estimated S570 billion in additional sales, highlighting the importance of duta driven sales stralegies for growth.
Retailers have consistent 25% increase in revenue due to advanced personalization capabilities.
Gmnees nunning datn-driven loyalty paigns have realized an average 3.8% like-for-like sales uplift. (Scure 0.0)
(Scure = 0.0)
Persomalized promotions can deliver returms three times higher than mass prometions, yet many retailers allocase under 5% their budgets to pensoralization. One major chain incressed its pensonalized promo spend from 1% to 10% hy establishing a "customer investanent council"" resulting in $250 million incremental
have seen consistent increase in revenue due to advanced personalization capabilities. (Score 0.0)
Extemaal sources indicate ena-lea personalization can lift revenue 556-15% and advocate personalized e-receipts with relevant offers and coupons to extend purc-ue engagement. (Score 0.0)
Retail media networks are expanding rapidly, with retail media growing approxinately 25% anrually, offering retailers a profitable revenue stream to reinvest technology, data, and personnel. By integrating loyalty data, retailers like Sephora, which links 95% of transactions to loyally accounts, enlance preci- sion in product recommenddation and provide seamless omnichannel experience, boosting comersion rates and customer lifetime value.
85% of transctions are linked to loyally accounts I ce's Market, providing solid foundation for personalized marketing and improving customer engagement. (Scure = 0.0)
As of Q2 2024, 85% of tramsactions were linked loyalty accounts, leaving 15% unlinked identity gap. (Score 0.00
Table 13: Factuality Performance Comparison: Llama 3.1 405B vs GPT-5 (Task DR0002). We show the number of factual and unfactual claims made by each model, as well as the overall factuality score for the given task.
Metric
Llama 3.1 405B
GPT-5
Improvement
Factuality Score
0.41
0.65
+0.24
Factual Claims
7
13
+6 claims
Unfactual Claims
7
-3 claims
29

---

Published as a conference paper at ICLR 2026
Report Synthesis
As an expert research analyst, synthesize the following content into coherent, insightful, and well-supported analysis for the theme: "[theme)" directly related to the overarching research question: "foriginal.questionn
Source Priority Guidelines:
Highest priority (internal company documents, proprictary files, confidential reports, enterprise chat messages, local documents, CRM data, internal APTs, project management tools) Insight ghtt from these sources should form the primary foundation of the analysis.
1. =="internal"*<:
2. **"xxtrraa***:: Medium priority (public ic web sources, academic papers, industry reports, articles). Use these to provide broader context, external validation, or contrasting perspectives.
*yynthesis Requirements:**
**QAANTITAIVVE PRIORITY:** Lead with numerical data, calculations, and aggregations
* Extract ALL percentages, costs, metrics, and performance data
Perform mathenatical operations: aggregate percentages, calculate increases, sum totals
*
Example: "Finance customers (35%) combined with healthcare (408) represent 758 of regulated industry concerns [DOC:doc.l] [DOC:6oc.2]
**FACT VERIFICATION (CRITICAL) :**
* ONLY state what documents explicitly contain no inference or extrapolation
Use exact quotes for key numerical claims: "As stated in the document:
[exact quote]' [DOC:doc.id]"
**Citation Usage (Critical):**
**Format:** Reference sources by their document ID: "Internal review
shows 15a increase [DOC:dos 079c2e0f 1752503636]*
**NEVER HALLUCINATE CITATIONS:** Only use providea doc
values
++iite every numerical claim and calculation with source documents**
Generate 2-4 paragraphs of synthesized analysis with proper inline citations.
Prompt 5: Report Section Synthesis with Citation Requirements
executed by instantiating a fresh container to avoid state leakage across runs. We capped the number of agent iterations according to the settings described in Section with each iteration limited by fixed computational budget.
For model outputs, standardized all prompts and evaluation pipelines across backbones, using identical research questions, company contexts, and injected insight sets. To avoid stochastic variability, we repeated generation three times per task and reported averaged scores.
Finally, all supporting scripts, environment configurations, and evaluation code are fully containerized, enabling consistent replication of reported results across hardware setups.
30

---

Published as a conference paper at ICLR 2026
Table 14: Factuality Content Analysis (Task DR0002). We show the number of factual and unfactual claims made by each model, highlighting the factuality accuracy of each model for the given task.
Agent
Factual
Unfactual
Accuracy
Llama-3.J 3.1-B05B
claims
10 claims
41.0%
GPT-5
13 claim
7 claims
65.0%
Change
+6
-3
+24.0% %
Table 15: Factuality Summary (Task DR0002). We summarize the factuality result improvements made by using GPT-5 over Llama 3.1 405B.
Impact Category
Value
Factuality Improvement
+24.0 percentage points
Claims Added
total claims
Accuracy Enhancement
From 41.0% to 65.0%
Content Quality
More grounded information
Task Domain
Sales
Task Industry
Retail
I DATA SYNTHESIS AND DRBA COST DETAILS
To generate the DRBench tasks, we combined external insight extraction with internal file synthesis. Each task included an average of 10 supporting files spanning heterogeneous formats (PDF, DOCX, PPTX, XISXX, and ISONL.), with each file containing roughly 5 paragraphs of content. Files were designed to embed injected insights while mixing in distractor material, ensuring realistic enterprise complexity without exceeding practical runtime or storage budgets.
Data synthesis was primarily powered by GPT-4o. During task construction, GPT-4o was responsible for (1) extracting structured insights from public web sources, (2) adapting these insights into enterprise enterprise-rrounded grounded interpretations, (3) generating persona-specifh deep research questions, and (4) producing file-level content with a balanced mix of insights and distractors. For evaluation, the DRBench Agent (DRBA) used GPT-40 its backbone, with each task typically requiring 15 iterations and approximately 120-150 model calls.
In terms of cost, GPTT-4-based synthesis of a single task (10 files, paragraphs each, plus metadata) consumed about 30k-00k tokens, while DRBA execution required additional 50k-70k tokens per task. At current GPT-4o API pricing ($5 per million input tokens and $15 per million output tokens), this corresoonds a per-task cost of approximately $1.5-$3.5 depending the mix of input/output tokens and the iteration budget. This makes large-scale benchmarking feasible at moderate cost, while still being significantly cheaper than manual authoring or annotation.
We also note that smaller open-source models such as Llama-3. 1-eaa...--B--Isttttee perform well for file generation. Unlike GPT-4o, which requires API usage, Llama-3.1-88 can be hosted locally and runs efficiently on a single NVIDÍA A100 40GB GPU. This provides a cost-effective alternative for generating large numbers of supporting documents, especially when full closed-source quality is not required.
J WEB AGENTS FOR DEEP RESEARCH TASKS
Since each of the environments can be access directly through a web user interface (UT), we also exper- imented with an agent that can directly interact with the webpages through common browser actions like click, input and scroll. 1, which are executed through playwright'. We implement our web agent using the AgentLab and BrowserGym frameworks (Chezelles et al.. 2025) with a GPT-4.1 backbone. Our agent is implemented from AgentLab's GenericAgent, which achieves respectable performance when used
'https: http::////paawrghhttt *httpa: /ttp///ppaaa..////////// com/ nnde/ (ttex//ppt41111
31

---

Published as a conference paper at ICLR 2026
with GPT-40" backbone; it completes 45.5% of the tasks in WorkArena (Drouin et al., 2024), 31.4% in WebArena (Zhou et al., 2024) and achieves a step-level reward of 13.7% on WebLINX (Lù et al., 2024).
Hyperparameters and Prompt We present the hyperparameters for the agent in tables 16 and 17, which are in majority set to the default hyperparameters, except for the maximum number of input tokens (bound reasonable maximum length) and higher maximum number of steps (to allow the agent perform more actions required to write the report). We further update the agent's s action space on the last step to only allow it to reply to the user with a report, ensuring that each trajectory terminates with report. To ensure that the agent is aware of the tools it can use, we modify the default system prompt (see prompt 6). Additionally, each task intent is provided alongside information about the user and company (see prompt 7).
Results We find that the GPT-4.1 -powered web agent achieves an insights recall and factuality of 1.11% and 6.67% respectively and a report quality score of 33.07%. Although the high report quality indicates that the agent can properly formulate report, the insights quality is severely limited, with none of the claims being backed by useful sources. For example, a DRBench Agent powered by GPT-5 may answer the question What is Lee's Market's current food waste reduction rate of Q2 2024? with An 8% reduction in food waste in Q2 2024 saved Lee's Market $1.2 million, indicating that better inventory control can yield both safety and financial benefits., which achieves score of 1.0 for the question. On the other hand, a GPT-4.1 1.TT-.11-pove -powered agent will provide an unsatisfactory answer, thus achieving an insights recall of 0.0. The most likely cause of this poor performance is the model's limited capability to properly interact with web interfaces when encountering unfamiliar tools. For instance, the agent may be unfamiliar with the VNC and file browser applications, making it harder for it to correctly select the file needs to use. Moreover, whenever the agent ends up performing an ineffective action (e.g. click on an element that does not trigger any change to the page), it tends to persist by reitcrating the same action (see Table 18), or the same sequence of ineffective actions, despite not achieving anything in the previous steps. As result, despite a large number of steps, most of the agent's 's actions are not helpful towards solving the task.
Table 16: Web Agents Boolean Hyperparameters
Value
Flags
True
vision dision.support, use ax tree, use_tabs, wew.ouueeeeeeeeeete...... use.error.logs, use use.history, use.action.history.. use.screenshot, use_som, extract.visible.tag, extract.cllckaabeettgg use.thinking, use.concrete.exam use hints, be cautious, add add messages
use.abstractt.exmmp..
False use ase.past.error..oog
use.think.historr.
use,diff, fillerrvissbbeeeeeeeennts..... filter.with.bid.onlly htlplyy.ffffttrtrrrrrr filter.som.only. multiaction, strict, Long.description individuulLexxmples, use plan, use criticise, use memory, enable chat
K STANDARD ERROR
Restricting to the MinEval subset, we average the results on each task across different runs Table 19. We give both the means and standard errors for the insight recall, factuality, distractor avoidance, and report quality.
L EVALUATING LLM JUDGE SENSITIVITY ACROSS MODEL TYPES
Across models, replacing the GPT-4o judge with JJama-3.1-4053 yields only minor differences in evaluation outcomes (Table 20). This stability arises from the design of our evaluation protocol: almost all metrics, insight recall, factuality, and distractor avoidance- are computed through a blaidd marking strategy after breaking the model output into short, atomic statements. Large language models are highly
Shttpa: //ta://///ppaaaae com/ ndde 1 o-gpt-44/
32

---

Published as a conference paper at ICLR 2026
4 Web Agents System Prompt
You are an agent trying to solve web task based page and user instructions You can interact with the page and explore, and send nessages to the user, Each time you submit an action it will be sent to the browser and you will receive new page. You will be solving tasks that involve Deep Research, by navigating websites and web apps with infornation useful solving the task. You will need to gather insights from data contained in the services provided and the internet to complete your report. You have maximum of 50 steps to complete the task. Refore the end, you must use the action (send_mag.to.user),, which should contain a final Deep Research report detail ing everything ng the user needs to know.
content
You must sustain your claims files, chats, or emails from the enterprise environment or in websites you searched. You must provide a citation (or an inline citation, that works too) with the source of those claims (c.g.
citation examples from >>) Do not make up citations if you haven't retrieved its content.
docunentation
Here expected agent behavior:
* Enterprise Environment Interaction:**
Access Nextcloud files, Mattermost chats, emails, VNC desktop, etc.
Extract relevant information from multiple sources
Navigate complex enterprise data landscapes
*>Report Requirements:**
Synthesize findings into comprchensive research report
Include proper citations for all claims (flexible format aitoonormalized
Draw meaningful insights and conclusions Ground all statements available evidence
**Citation Format (Flexible Auto-Normalized)) ***
Files: 'guarterrly.eppor..pdee
'Analysis document treports/analysi. (anllysss.doo))
URLs:
Direct links or '[rticle Title] (https: / pettps:/exxmpleeeee /exmmle..ommm
Emails: 'Email from alice@company. com on Jan 20, 2025'
Chat: "aaterroott message from john.dce doe in Compliance team, General
channel*
If you need navigate the internet (outaide of the des' gnated webs websites), you the browser inside novNC Desktop.
Prompt 6: Extended instructions given the Decp Research web agent.
33

---

Published as a conference paper at ICLR 2026
Table 17: Web Agents Non-Boolean Hyperparameters
Parameter
Value
chat model model name
gpt-4.
32768
chat.modeel.max..nnuuttteeeen chat.model..nxx.new.tooo.
29672
4096
chat.mocdel..teepprrtuuuuee
0
actiion.aettioonsee...ubbttt
webarena
actioo..actioon.eet..etty force
true
flagss.ma..rromtt.tokke... flags.max.trunc.itr
28672
20
env max_steps
50
Web Agents Task Intent Prompt
How Lee's Market leverage FSMA 204 regulations to enhance food safety and eustomer trust?
Here is some informati on about the user: ID: MNG0003
First name: John
Last name: Doe
Justification: As regulatory affairs manager, John
Here is some information about the company:
Name: Lee's Market
Annual revenue: $500M $600M
Target markets: Asian communities the U.S. and
Prompt 7: Web Agents Task Intent Prompt
Table 18: Web Agents tends to get stuck on cycles of actions, and are unable to backtrack or restart with different application.
9. have been repeatedly clicking "shared" folder (bid 194") To proceed with the research pensonaization strategies
10. have clicked on the "shared" folder multiple times will ty clicking on the "shard"" folder again
11. have boen repeatedly clicking the "shared" folder (bid '194") To proceed. should try clicking the "shured" folder again attempt to open and view any files
have navigated to the "shared" opened
folder, but have it it.. The next logical step to open "stured" folder look for any relevant documents...
opens displays its contents.
Lee's Market, I should open the "shared"
for... click '194')
click ('190")
click ( '194")
click (194')
consistent such simple yes/no judgments, leading to minimal variance across judges. The only metric with larger fluctuations is report quality, which the only non-binary, holistic score. Overall, these results indicate that our findings robust the choice of L.LM judge.
34

---

Published as conference paper at ICLR 2026
Table 19: DRBA performance with different planning configurations MinEval. We compare the base agent with variants using Simple Research Planning (SRP), Complex Research Planning (CRP), Adaptive Action Planning (AAP), and their combinations. Scores are reported for insight recall, factuality, distractor avoidance, report quality, and the overall harmonic mean.
Configuration
Insight Recall (%)
Factuality (%)
Distractor Avoidance (%)
Report Quality (%)
Harmonic Mean (%)
Base DREA
18.8
2.2
66.5 :: 5.2
98.1 1 0.6
91.2 :: 0,2
44.7
SRP
13.7
2.3
62.2 1: 6.3
100.0 0.0
90.1 0.3
36.3
CRP
15.4
70.5 2.9
100.0
91.7
40.0
19.7
4.6
-RRP
14.2
=.2
50.4 = 5.8
99.0 = 2.3
90.6
CRP AAP
18.8 2.9
69.1 + 1.7
100.0 0.0
92.3 0.3
45.2
Table 20: Comparison of MinEval results when switching the LLM judge from GPT-4o Llama-3. Metrics show minimal variance in factuality and distractor avoidance, moderate variance in insight recall, and largest variance in report quality.
Model
Judge
Insight Recall
Factuality
Distractor Avoidance
Report Quality
GPT-4o
39.63
65.17
92.86
93.42
GPT-5
Llama-3. 1.inaa-3.1-a0 1-00BB
38.92
65.11
92.40
91.83
GPT-44
30.26
70.27
96.67
86.88
DeepSeck Chat 3.1
Llama-3,1 - 1-405B
29.54
69.98
96.22
85,10
GPT-4o
26.82
58.35
97.65
Qwen 2.5 72B
J.ama-3.14400 1-40BB
26.11
58.12
97.20
87.45
GPT-4o
17.31
60.84
98.33
91.62
GPT-4o
Llama-3, 3.1-BOSB
16.85
60.31
98.05
89.82
GPT-4o
20.16
69.75
97.90
91.26
Llama 3.1 405B
Llama-3. 1-45BB
19.72
69.42
97.55
89.51
M COMPLEX MODEL ABLATION RESULTS
Extending our discussion in Section 5.3 to more GPT and Llama models Table 21, we see that the smaller GPT-5-mini model lags behind but still outperforms earlier closed-source backbones such as GPT-44 and GPT-4o-mini, particularly in terms of harmonic mean. In addition, smaller variants Llama degrade metric results further. This further substantiates the claim that larger and more advanced models tend to offer a better balance between recall, factuality, and overall report quality.
N QUANTITATIVE RESULTS PER TASK
We show detailed breakdown of the insights recall in Table 23, factuality in Table 24, distractor avoidance in Table 25 and report quality in Table 26 on the MinEval subset for variety of models.
EFFECT NUMBER OF ITERATIONS
We next analyze the effect of varying the number of research loop iterations when using DRBA with GPT-5 as the backbone language model. Results for both the baseline configuration without explicit planning and the complex planning setup are shown in Table 27. Overall, increasing the iteration budget does not guarantee consistent improvements. With no planning, performance initially drops when the agent executes more iterations, as additional exploration often introduces noise and distracts from key insights. However, with larger budget the agent partially recovers, suggesting that small number of additional iterations can help refine factual grounding, while excessive exploration reduces focus.
For the complex planning setting, higher iterations improve certain metrics such factuality, but this comes at the cost of lower insight recall and reduced overall balance. This indicates that while more steps allow the agent verify citations more carefully, they also lead fragmented reasoning and overfitting to peripheral evidence. The best overall performance emerges at moderate iteration counts, highlighting the importance of carefully tuning the iteration budget rather than simply scaling up the number of steps.
35

---

Published as a conference paper at ICLR 2026
Table 21: Performance of DRBA on the FullBenchmark subset using different backbone language models and planning strategies, Scores are reported for insight recall, factuality, distractor avoidance, report quality, and harmonic mean. Note that higher numbers corresponds better scores, and the best result on each metric is bolded.
Model
Planning
Insight Recall
Factuality
Distractor Avoidance
Report Quality
Harmonic Mean
GPT-5
None
36,52
72.11
93,22
93.41
63.81
GPT-5
SRP
35.41
69.42
94.67
93.88
62.64
GPT:5
CRP
37.48
62..3
91.71
GPT-S-mini
None
22.37
54.06
94.93
81.24
46.49
GPT-S-mini
SRP
24.58
54.31
94.18
82.43
48.87
GiPT-5-mini
CRP
22.96
47.19
93.16
82.71
45.67
GPT-4o
None
16.23
61.44
95.86
89.94
40.22
GPT-4o
SRP
18.81
58.74
95.37
90.58
43.61
58.61
95.63
GPT-lo-mini
Nore
42.24
95.79
81.08
30.59
GPT-4o-miii
SRP
11.78
50.13
94,92
81.97
31.35
GPT-4o-mini mini
CRP
11.31
43.97
94.56
82.16
29.87
GPT-OSS- 120B
None
19.26
26.14
94.78
80.93
35.37
GFT-OSS-120B
SRP
15.17
24.54
94.66
80.98
30.87
GPT:0SS-120R
None
16.1
69.3
95.2
88.6
40.68
Hama-3.11440
SRP
15.7
70.4
94.6
89.7
40.15
CRP
18.33
65.72
95.04
89.01
43.70
1amm-3.1-700-Inste
None
16.03
55.04
95.64
81.96
38.76
3.-ma=311770--nntte
SRP
58.66
95.93
81.57
36.35
-lamma3.1-700-Insttel
CRP
14.71
49.91
95.22
$4.38
36.24
DeepSeek-VE. V3.1
None
22.6
68.4
94.9
84.1
49.20
DeepSeck-V3.J
SRP
23.1
69.2
94.1
84.9
49.91
DeepSeck V3.1
CRP
28.21
67.09
93.96
85.57
55.03
Qwen
63.2
95.4
86.9
48.98
Qwen 2,5 72R Instrnct
SRP
20.9
61.1
95.1
85.2
46.26
Qwenn225--22-Innstnn
CRP
24.39
55.74
95.12
87.51
49.46
Table 22:
Total average Insight Recall scores per model and insight source type computed on all the results available for each model running in Complex Research Plan mode on the FullBenchmark. Insights embedded in enterprise sources are more easily retrieved by DRBA all the models.
Model
Enterprise Fact
External Fact
GPT-5
0.597
0.0
DeepSeek-V3. V3.1
0.472
0.0
GPT-5-mini
0.444
0.0
Qwen-2.5-72B-Innnmn
0.417
0.0
0.347
0.0
GPT-OSS-120B
0.333
0.0
GPT-4o-mini
0.194
0.0
1aae-33.-700-Insttuue
0.194
0.0
GPT-4o
0.182
0.0
P DATA GENERATION PROMPTS
P.I. COMPANY AND PERSONA DATA GENERATION
In this section we give the prompts used for company generation 8 and persona generation 9.
P.2 QUESTION GENERATION
In this section give the prompt used to generate deep research questions 10.
36

---

Published as a conference paper at ICLR 2026
Table 23: Mean and standard enror of the insight recall metric for the first five tasks, obtained from three runs of on DRBench our agent (DRBA) using 15 iterations across different backbone models.
Configuration
Plan
DR0001
DR0002
DR0003
DR0004
DR0005
GPT-5
.22 .056
.429 .000
.467 = .033
.519 .098
.281 .035
GPT-5
SRP
.22 .056
.381 .048
= .033
.370 + .098
.386 + .046
GPT-5
CRP
.278
.381 .26
.567 = .067
.370 +
.386 + .046
GPT-5-mini
.111 .056
.286
.367 = .033
.222 +
.298 + .018
GPT-5-mini
SRP
.222 .056
.286 = .082
.333 + .033
.296 + .074
.281 + .063
GPT-5-mini
CRP
.000 = .000
.381 = .126
.367 = .067
.370 + .098
.211 + .053
GPT-4o
.111 = .056
.238 + .048
.167 + .033
.185 + .074
.175 + .018
GPT-4o
SRP
.167 = .000
.333 + .048
.300 + .000
.148 + .037
.070 + .018
GPT-4o
CRP
.111 = .056
.238 + .095
.300 + 100
.111 = .000
.105 + .000
GPT-4o-mini
.000 = .000
.095 = .048
.267 = .033
.185 + .037
.140 + .035
GPT-4o-mini
SRP
.056 = .056
.190
.167 = .067
.148 + .074
.123 +018
GPT-4o-mini
CRP
.000 = ,000
.190
.267 = .033
.074 = .037
.123 2.18
GPT-OSS-120B
.111 = .111
.433 + .033
.222 + .000
.211 + .030
GPT-OSS-120B
SRP
.056 .056
.143
.367 + .033
.148 + .098
.158 + .061
GPT-OSS-120B
CRP
.000 = .000
.990
.333 .067
.111 .064
.281 - .063
Mimm--31-405B-Inmse
.67 .000
.133
.233 .088
.185 .074
.140 .035
1ama-3.1-405B-Im
SRP
222 .056
.190 .048
.167 = .033
.111 .064
.158 .053
CRP
.111 .056
.38 - .048
.300 +
+
.21 + .030
1ama-3.1-70B-Inss.
.17 .000
.381 .048
.20 .000
.074 .037
.105 + .030
llama-.1-77B--nnsiie
SRP
.056
.266 - 0082
.200 +
.185 +
.088 +
llam-3.1-700-Instmee
CRP
.17 = .000
.238 = .048
.267 + .033
.074 + .074
.105 + ,000
DeepSeek-V3.11
.167 = .000
.286 + .082
.300 + .058
.259 + .037
.246 + .046
DeepSeck-V3.
SRP
.78 = .06
.238 + .048
.333 + .033
.148 + .074
.281 + .046
DeepSeek-V2 V3.1
CRP
.167 = 000
.095 = .048
.600 + .058
.370 + .037
.281 + .035
.22 .056
.38 = 048
.400 + .058
.259 + .037
.158 + .061
Instructt
SRP
.111 .566
333 044
.333 .088
.259 .037
.123 + .018
Qwen-2.5-72B-Instee
CRP
.167 .966
143
.400 = .058
.333
.298 + .076
4 Company Generation Prompt
Generate a realistic company structure for {company.name] in the findustry) industry.
Company size: {size) (femployee.range) employees)
The company should focus on this domain {domain}
EXTERNAL INSIGHTS: fexternal
Return ONLY a valid JSON object with this structure: (output.structure)
Make realistic for the (industry) industry.
Prompt Company Generation Prompt Template.
37

---

Published as a conference paper at ICLR 2026
Table 24: Mean and standard error of the factuality metric for the first five tasks, obtained from our agent (DRBA) using 15 iterations across different backbone models.
Configuration
Plan
DR0001
DR0002
DR0003
DR0004
DR0005
GPT-5
.761 .072
.504 .075
.866 .007
.833 + .019
.762 + .077
GPT-5
SRP
.714 - .050
.384 = .076
.848 .034
.812 .070
.846 .029
GPT-5
CRP
.730 =
.291 .094
.782 = .012
+ .064
.674
GPT-5-mini
.585 .045
.297 .119
.705 + .039
.704 .067
.647 = .098
GPT-5-mini
SRP
.647 = .1200
.299 = .056
.624 + .041
.694 + .028
.683 + .020
GPT-5-mini
CRP
.309 = .126
.381 = 116
.699 + .047
.692 - .111
.472 - .114
GPT-4o
.792 + .15
.490 .110
.87 = .056
.570 + .058
.593 + .204
GPT-4o
SRP
.855 = .262
.132
.133 + .041
.693 .139
.614
GPT-4o-mini
SRP
.455 = .166
.653 + .997
.704 + .037
.542
.406 + .020
GPT-4o
CRP
828 = .043
.265 1133
.00 + .000
.690 + .128
.459 + .235
GPT-4o-mini
.611 = .056
.429 = 009
.622 + .062
.481 + .209
.191 + .046
GPT-4o-mini
CRP
.557 = .030
.324 - 1169
.580 + .075
.642 .119
.344 .144
GPT-OSS-120B
.144 = .099
+.00.005
.386 + .040
.337 1117
.455 .051
GPT-OSS-120B
SRP
.074 = .074
118
.410 +190
.31 1155
.451 .080
GPT-OSS-120E
CRP
368 .661
.178 .078
.564 .064
400
Muma--.1-405B-mmm
.82 = .087
.726 .158
803 .028
820
.745 + .022
Gama-3.1-405B-Ime
SRP
802 .125
.638 202
.00 .074
.922
.822 + .083
1ama-3.1-405B-Imm
CRP
.789
.392 .154
.792 -
.771 +.73
.745 +
hama-3.1-70B-Inaa
.618 .109
.431 .160
.684 .104
.12 .021
.687 - .073
Ilama-.1-770--nniii
SRP
.608
.681 .069
800 +
826 .067
.557 +
llama-3.1-70--Imss
CRP
.888 = .082
.866 + .108
.686 + .011
.522 + .270
.559 + .240
DeepSeek-V3.11
860 = .014
.518 + .085
818 + .041
.679 + .095
.757 + .057
DeepSeck-V3.
SRP
.96 # .441
.531 # .886
922 + .056
.769 + .035
.754 + .082
DeepSeek-V2 V3.1
CRP
.881 = .042
.657 = .024
.38 + .550
+74..005
.662 + .098
.674 .077
.493 1939.109 .109
866 + .002
.741 .060
.696 + .060
2.5--2. Instructt
SRP
806 .049
.40
.741 = .074
.724
.550 + .148
Qwen-225-72B-Insttuu
CRP
626 + 114
.396 :96 .056
.723 + .053
+ +139
.586 + .055
Persona Generation Prompt
Generate {persona count) diverse employee personas for (company name) the (industry) industry.
The personas should focus this domain: {domain)
Create diverse roles across seniority levels: Junior, Mid, Senior, Executive
Return ONLY valid JSON array with this exact format: (output.structure)
Make personas realistic with appropriate responsibilltice for their roles.
Prompt 9: Persona Generation Prompt.
P.3 PUBLIC SOURCE AND INSIGHT COLLECTION
In this section we give the prompt used to generate external insights 11. The URLs used for externa insight extraction and deep research question creation be found in Table 28.
38

---

Published as a conference paper at ICLR 2026
Table 25: Mean and standard error of the distractor avoidance metric for the first five tasks, obtained from three runs of on DRBench our agent (DRBA) using 15 iterations across different backbone models.
Configuration
Plan
DR0001
DR0002
DR0003
DR0004
DR0005
GPT-5
.857 .000
.000 = .058
1.00 .000
1.00 + .000
1.00 + .000
GPT-5
SRP
.857 .00
1.00 .000
1.00 + .000
1.00 + .000
1.00 .000
GPT-5
CRP
.05
900
.93 .000
.905 +
1.00 +
GPT-5-mini
.05 .048
.977 .033
.988 = .022
.96
1.00 + .000
GPT-5-mini
SRP
.857 .000
.933 = .033
1.00 = .000
1.00 + .000
1.00 +
GPT-5-mini
CRP
.857 = .000
.86 = .133
1.00 = .000
1.00 - .000
1.00 + .000
GPT-4o
.952 = .048
1.00 = .000
1.00 + .000
1.00 + .000
1.00 + .000
GPT-4o
SRP
.922 = .048
1.00 = .000
1.00 + .000
.976 + .024
1.00 + .000
GPT-4o
CRP
.922 = .048
1.00 = .000
1.00 = .000
.644 + -036
1.00 + .000
GPT-4o-mini
.952 = .048
1.00 = ,000
1.00 + .000
1.00
1.00 + .000
GPT-4o-mini
SRP
.857 = .000
1.00 = .000
1.00 = .000
1.00
1.00 .000
GPT-4o-mini
CRP
.857 .000
1.00 = .000
1.00 + .000
1.00 +000
1.00 + .000
GPT-OSS-120B
.857 = .000
1.00 = .000
1.00 + .000
+.00
1.00 + .000
GPT-OSS-120B
SRP
8857 .000
1.00 = .000
1.00 + .000
1.00 .00
1.00 .000
GPT-OSS-120B
CRP
.05 .048
1.00 = .000
1.00 .000
1.00
1.00 + .000
Muma-3.1-405B-Ima
1.00 ,000
1.00 .000
1.00 .000
1.00
1.00 .000
1ama-3.1-405B-Inse
SRP
905 .048
1.00 .000
1.00 =
1.00
1.00 .000
hama-3.1-405B-Im
CRP
.52
.967 .033
1.00
.76 .024
1.00 + .000
1ama-3.1-70B-Inss.
.05 .048
1.00 = .000
.978 .022
.952 - .048
1.00 + .000
llama-.1-77B--nnsiie
SRP
.055
1.00 = .000
1.00 =
.766
+
llam-3.1-700-Innmiee
CRP
905 = .048
1.00 = .000
1.00 + ,000
1.00 + .000
1.00 + .000
DeepSeek-V3.11
905 # .048
.967 = .033
1.00 + .000
1.00 + .000
1.00 + .000
DeepSeck-V3.
SRP
857 = .000
1.00 = .000
1.00 + .000
.76 + .024
1.00 + .000
DeepSeek-Va. V3.1
CRP
905 = .048
1.00 = .000
1.00 + .000
.999 .000
1.00 + .000
1.00 = .000
1.00 .000
1.00 = .000
1.00 .000
2w
SRP
.952 .048
1.00 = .000
1.00 .000
.92 .024
1.00 + .000
Qwen-2.5-72B-Instte
CRP
952 .048
1,00 = .000
.78 + .022
952
1.00 + .000
P.4 INTERNAL INSIGHT GENERATION
In this section we give the prompts to generate both internal insights (Prompt 12) and internal distractors (Prompt 13).
P.5 FILE GENERATION
In this section we give the prompts used for generating each of the file types used our tasks, which we list as follows:
PDF: Prompts 18, 19, and 20
Excel: Prompts 21, 22, and 23
Powerpoint: Prompts 24, 25, and 26
Email: Prompts 27, 28, and
Chat: Prompts 30, 31, 29, and 33
39

---

Published as a conference paper at ICLR 2026
Table 26: Mean and standard eπror of the report quality metric for the first five tasks, obtained from three runs of DRBench with 15 iterations across different backbone models.
Configuration
Plan
DR0001
DR0002
DR0003
DR0004
DR0005
GPT-5
.36 .088
.918 0004
.924 .005
.909 .001
.927 + .009
GPT-5
SRP
.422 .00
927 .008
.966 .006
.915 - .002
.933 - .007
GPT-5
CRP
.48 .007
921 .001
.40
.22 - .009
.929 + .008
GPT-5-mini
.92 = .002
.84 = .007
.879 = .004
.91 = .000
.886 + .005
GPT-5-mini
SRP
9011 = .009
.899 = .002
.887 = .001
.95 + .008
.892 + .003
GPT-5-mini
CRP
.86 = .001
.82 = .006
.884 = .003
.89 + .009
.890 + .001
GPT-4o
.977 = .000
9911 = .003
.903 = .001
.988 = .009
.909 + .002
GPT-4o
SRP
934 = .008
919 = .000
911 = .009
.23 + .001
.916 + .000
GPT-4o
CRP
.29 .009
914 = .002
.905 + .000
.220 + .008
.913 + .009
GPT-44-mini
.866 .X14
.874 .008
.861 .007
.872 + .002
.879 + .006
GPT-4o-mini
SRP
.933 .002
.811 .005
.66 .004
.78 + .001
.844 + .003
GPT-40-mini
CRP
.889 .003
.877 .006
.864 .005
.855 + .000
.822 + .004
GPT-OSS-120B
.872 .007
.611 .001
.849
.858 -
866 + .008
GPT-OSS-120B
SRP
.878 .006
.867 .009
.854 + .008
863 + .004
.872 + .007
GPT-OSS-120P
CRP
.84 = .007
863 = .000
.851 = .008
860 + .005
.699 + .006
Ilama-3.1 -0000.33---0
914 = .000
.903 = .003
.877 + .001
.999 + .008
.922 + .002
Llama-3, 3,1 1.1aa.33--0BB-Isttuue
SRP
.21 = .008
.10 = .001
.944 = .000
915 + .009
.988 + .000
CRP
.17 = .009
.966 = .002
.899 + .001
9911 + .008
.905 + .001
Llama-3.1770BIIm
.89 .003
.877 = .007
.669 + .005
881 + .001
.833 + .004
Llama-3.1-70B-Inttuu
SRP
895 .002
.883 .005
.874 = .004
886 + .000
.878 + .003
Llama-3.1700-Insttre
CRP
891 .003
.879 .006
.871 .005
883 + .001
.85 + .004
DeepSeck-V3.J
.844 .004
.872 .008
.64 .006
876 + .002
.869 .005
DeepSeek-V3., V3.11
SRP
.890 .003
.878 .006
.870 = .005
.881 - .001
.874 + .004
DeepSeek-VS. V3.1
CRP
.866 .004
.874
.866
.878 +
.871 + .005
Qwen-2.5-722
901 = .001
.899 = .005
.881 = .002
.933 + .009
.855 + .003
Qwen-2.5-72B-Inssn.
SRP
.08 = .000
896 + .003
.88 + .001
.899 + .008
.81 + .002
Qwem-2.5-72BB-Instnuu
CRP
904 = .001
892 = .004
884 + .002
895 + .009
.88 + .003
Table 27: Effect of the number of research loop iterations on the performance of DRBA with GPT-5 as the backbone model, on MinEval.
Planning Method
Iterations
Insight Recall
Factuality
Distractor Avoidunce
Report Quality
Hurmonic Mean
No Planning
15
39:15
72.65
93.14
94.56
66.20
No Planning
30
28.80
69.03
98.57
96.12
57.34
No Planning
50
37.10
78.84
100,00
93.48
66.30
Complex Research Planning (CRP)
15
44.44
62.51
90.95
93.12
66.41
Complex Research Planning (CRP)
30
31.61
73.94
94,38
94.76
60.32
Complex Research Planning (CRP)
50
38.16
66.05
94.38
92.64
63.76
Q EVALUATION PROMPTS
In this section give the prompts for decomposing reports into atomic insights 14, computing insight recall 15, computing factuality 16, and computing report quality 17. These prompts are discussed in detail in Section 5.1.
R HUMAN PREFERENCE EVALUATION
We calculate human score for model task as:
40

---

Published as a conference paper at ICLR 2026
4
Deep Research Question Generation Prompt
Generate 3 Deep Research (DR) questions for the following business
context:
Persona:
(persona.name) - (persona.role)
Department:
Responaibilities: (persona_ respssnirrssossssliitieeeee
(company-nama) (ccompany.iduustio Domain: {domain}
Company:
External Insights: (external.insighet
Generate 3 Deep Research questions that:
1. Are appropriate ate for the persona's role and department
2. Require analysis of the provided internal insights
3. Consider the external market context
Each question should be 1 sentence of 15 words max, in plain english, and end with a question mark. Do not include any preamble or explanation return only the JSON array.
Return ONLY a valid JSON array with this structure: (output.structure)
Prompt 10: Deep Research Question Generation Prompt Template. Subquestions are generated to help human annotators select good DR questions.
External Insight Extraction Prompt
You will be given a report (with url, and date). Based
on the report, generate 3 external insights that summarize
important findings, trends, or takeaways from the report.
Output Format foutput.structure)
Ur1: {ur1} industry: (industry)
Domain: {domain)
Company Information: Crompany
Important notes
Focus only on insights that are external and grounded the report, Insights should be concise, factual, and directly tied to the retail industry context.
Prompt 11: Extemal Insight Extraction Prompt.
1,
if human choice is or "both good"
Sa,= = Sa.b where Sn.1
0. otherwise
where n is the number groundtruth insights task f, s,, is the human score of model a on insight Figure8 shows that the insight recall metric par with human decision.
41

---

Published as a conference paper at ICLR 2026
Table 28: Public URLs For Deep Research Task Creation.
Industry
Domain
Reference
Retail
Compliance
Grocers on FSMA-204 Compliance (Grooerryiive) E
Retail
CRM
Grocery Loyalty & Inflation (EagleEye) L
Retail
Market Analysis
Grocery Trends Outlook 2025 (GroceryDive) L
Retail
ITSM
Retail IT Optimization (Thirdera) L
Retail
CSM
Chatbots & Grocery Interactions (GroceryDoppio)
Retail
Knowledge Mgmt
Retail Knowledge Management (Knowmax) E
Retail
Sales
Personalization in Action (BCG) L
Retail
Cybersecurity
Retail Cybersecurity Threats (VikingCloud) C
Retail
Public Relations
Walmart CSR Strategy (SunriseGeck) E
Healtheare
Compliance
Telehealth Regulations (HealthcareDive)
C
Healthcare
CRM
Future of Healthcare CRM (WTT Solutions)
C
Healthcare
Market Analysis
Future of Telehealth (CHG Healthcare) L2
Healtheare
ITSM
Healthcare ITSM (Topdesk) L
Healthcare
CSM
Patient Engagement Tech (TechTarget) L
Hcalthcare
Knowledge Mgmt
Knowledge Mgmt in Healthcare (C8Hcalth) L
Healthcare
Sales
Sales for Digital Health (Medium)
Healthcare
Marketing
Marketing Telchealth Services (MarketingInsider) E Healthcare Cybersecurity 2024 (AHA) C
Healtheare
Cybersecurity
Automobiles
Compliance
Evolving EV Regulations (WardsAuto)
L
Automobiles
CRM
Salesforce Automotive Cloud Target) E
Automobiles
CSM
EV Aftersales Support (EVReport)
L
Automobiles
Sales
Tesla vs Dealerships WTee L
Automobiles
Research
Al for EV Optimization (Here.com)
L
Automobiles
Cybersecurity
Cybersecurity Risks in Cars (HelpNet(eccrrtt) L
Automobiles
Quality Assurance
EV Quality Issues (GreenCars) L
Automobiles
Asset Mgmt
Digital Twins Autos (RTInsight))
C
Automobiles
Market Analysis
Global EV Outlook 2024 (IEA) L
Human Scores vs Insights Recall
thnights
0.8
Score Value
X
0.4
0.0
muee
Dasea
DResse
DRuteer
010011
Deesia
Deo014
Task ID
Figure 8: Comparison of Human Scores and Insight Recall Scores. As can be seen the human evaluation results are aligned with our automated evaluation.
42

---

Published as a conference paper at ICLR 2026
Internal Supporting Insight Generation Prompt
Based the Deep Research (DR) Question, external market insights, company context, and previous internal insights, generate 3 specific QA pair that an expert data scientist would need to get in order to address the DR Question.
Company: (company.nama) (company.escrritiin
Industry: (industry)
Company Size: {company.sied) (femployee.comu count))
Annual Revenue: (evenu])
Persona Context: (persona_context)
DR Question: (dr.questicn)
External Market Context (for inspiration): fexternal context) QA History: {qa list)
Insight Requirements:
Use the DR Question as the central theme for the answer.
Draw inspiration and supporting details from the other internal insights and external insights provided. Include QUANTITATIVE DATA in the answer: metrics, percentages, dollar amounts, timeframes, KPIs.
Specific Question Instructions the specitic questien should be a question that would be a step towards resolving the DR Question fadditional.ce iadditional.qusiionn instestto....nstruttiiii
Answer Instruct
the answer should 12 words max and minimum words faddit.ooml.nssee...stttttt.
Justification Instructions
the justification should directly explain how the specific.question,
answer pair help address the DR Question in 15 words max
Misc Instructions
the filename should be 3 words max with dashes in between, do not mention the file type in the filename
use the example belcw as inspiration but do not directly
Return ONLY valid JSON object with this exact structure: (output.structure)
Prompt 12: Internal Supporting Insight Generation Prompt Template. specific questions and justification help human annotators to select good insights.
ROBUSTNESS OF INSIGHT RECALL TO PARAPHRASING
potential concern is whether our Insight Recall metric is overly sensitive to wording differences and fails to recognize paraphrased partially reworded insights. Our evaluation protocol, however, is designed to focus on the presence of key factual elements rather than exact lexical similarity. The LLM-ass--judge prompt ex- plicitly assesses whether the core facts of an insight appear in the predicted report, independent of phrasing.
43

---

Published as a conference paper at ICLR 2026
Internal Distractor Insight Generation Prompt
Based the Deep Research (DR) Question, external market insights, company context, and previous internal insights, generate 3 specific QA pairs that are DISTRACTOR questions
these should be questions that an expert data scientist might ask about the company but NOT relevant to addressing the DR Question.
Company: (company - (company.deseripe
Industry: (industry)
Company Size: (company.size) ((employee.countt)
Annual Revenue: (annual_revenue)
Persona Context: [persona context]
DR Question: fdr question)
External Market Context (for inspiration): (external QA History: (qa.list)
DISTRACTOR Requirements:
Cenerate questions that plausible for this company and industry but DO help address the DR Question.
The questions should be about the company's operations, metrics, or business but tangential to the DR Question.
Include QUANTITAIIVE DAIA in the amounts, timeframes, KPIs.
metrics, percentages, dollar
Focus on different business areas that are NOT central to the DR Question (e.g. if DR Question is about pricing, ask about HR metrics)
Specific Question Instructions
the specific question should be a plausible business question for this company but NOT related to the DR Question
the ppccisiicqeustion should lead to quantitative answer and should be dated such as Q3 2025, the question should contain date like Q2 of 2024
the speci fic.question should be 10 words max make sure to be different from any questien in the qa list
choose business areas like: HR metries, facility costs, IT infrastructure, compliance, training, etc. that are UNRELATED to the
DR Question
make sure to be different from any questien in the QA History
Answer Instructions (answer.instruction)
Justification Instructions
the justification should explain why this specific wwe.fiic.uuu fic.questionn
is NOT relevant to addressing the DR Question in 15 words max
Misc Instructions imisc_instructions
Return ONLY a valid JSON object with this exact structure: foutput structure)
Prompt 13: Internal Distractor Insight Generation Prompt specific_questiinn and justification help human annotators to select good insights.
44

---

Published as a conference paper at ICLR 2026
4 Breaking Report into Insights Prompt
Please break down the following report text into insight claims. Each insight claim should be:
1. A single insight, that might include multiple statements and
claims
2. Independent and self-contained
3. Each clain can have more than one sentence, but should be focused on a singlc insight
4. Support each insight with citations from the report text following these specific rules: {rules)
5. Citations should be in one of these formats (various formats will be automatical cally normalized):
fer icitation formats) 6. Do not inglude general aummaries, opinions, or claims that lack citation, just the sentences that are facts.
7. Each clain should be a concise but complete sentence.
Report text: {report.text)
Output format: Please return the insight claims as a JSON array. For example: (output.structure)
Return only valid JSON, additional text. Just the report between <START OF REPORT> and <END REPORT> tags to generate insights. If no insights found, return an empty JSON array: []
Do not use the example outputs as report content.
Prompt 14: Insight Extraction Prompt.
To evaluate robustness, we selected the ground truth insight and then generated three paraphrased versions. For example for the following insight:
"Lee's Market tracks 250 high-riskfood products of Q3 2024, affecting 30 percent of inventory." We generated the following three paraphrased versions:
"As of Q3 2024, Lee's Market is monitoring 250 high-risk food items, which account for about 30 percent of its inventory."
"Lee's Market tracks 250 high-risk foods in Q3 2024, representing roughly 30 percent of what stocks."
"In Q3 2024, Lee's Market has 250 high-risk products under tracking, making up 30 percent of its inventory.
All paraphrased forms produced the same recall outcome, demonstrating that the judge consistently recognizes equivalent factual content.
We further tested paraphrase robustness across reports, generating multiple paraphrase variants per insight and evaluating Insight Recall over 4 runs. The observed variance was 0.0017, indicating extremely low sensitivity to rewording. This confirms that Insight Recall reliably captures factual equivalence rather than surface-level phrasing differences.
INSIGHT LIMIT DESIGN AND LIMITATIONS
We chose the "groundtruth + five" limit because existing works, including Mind2Web and DeepRe- searcher, do not provide mechanisms to prevent agents from achieving perfect recall by copying large
45

---

Published as a conference paper at ICLR 2026
4 Insight Recall Prompt
Your goal is to check if the Predicted Insights extracted from report is groundtruth insight. You must be STRICT and pay attention to every small detail.
Instructions:
Evaluate if the Predicted Insights contain sufficient information to derive a groundtruth insight.
Select the insight that most closely matches the groundtruth insight. insight.
Select and only where:
of yes
yes: Selected insight contains comprehensive information to fully derive the expected insight
no: Selected insight lacks the necessary information, misses key details, or has significant gaps
Be STRICT do not answer yes for partial matches, vague similarities, or general information.
However, exact wording is required and paraphrasing is acceptable.
+ IMPORTANT: Only consider details given in the groundtruth insight when answering yes or no. Don't expect anything more than what is given in the groundtruth insight.
Focus on factual accuracy, completeness, and specificity.
Predicted Insights: (claims.text]
Groundtruth Insight: (gold
Return a valid ison dictionary with the following structure: (output.structure)
Ensure only json dictionary is returned, and return nothing else.
Prompt 15: Insight Recall Scoring Prompt.
sections of the source files. Our early experiments showed that without this constraint, several agents achieved 100% recall simply by extracting entire documents.
The +5 buffer allows agents to report a small number of additional insights that may reasonably arise during deep research, since is difficult guarantee that the groundtruth dataset contains every relevant insight. However, this design also limits our ability to reward legitimate novel insight discovery. As the community develops more robust metrics for insight coverage and novelty, DRBench readily incorporate them.
46

---

Published as a conference paper at ICLR 2026
Factuality Prompt
Given the following relevant source context from multiple sources and an insight, determine if the insight is factually supported by the sources.
Relevant: Source Materials (from multiple sources) {context] Atomic Claim: [insight)
EVALURTION CRITERIA:
The claim is factual if the core factual content is supported by the sources. You should be strict about important details but flexible about exact wording:
REQUIRED for TRUE:
1. All key factual details (numbers, dates, names, percentages, specific facts) must be present in at least one
2. The main substance and meaning the claim must be supported by the source contexts
3. No part of the clain should contradict the information in of the sources
ACCEPTABLE variations:
(acceptable variations] (acreptable.vrriiiiooss)
Mark as FALSE if: Important factual details are missing, incorrect, or unsupperted across all sources
The claim contradicts information any of the sources
The core meaning cannot be verified from any of the source contexts
EXAMPIES:
(examples)
Focus on the substantive factual accuracy rather than exact word-for-word matching. You MUST respond with either true or false under the <factual> tag. Then provide brief explanation under the <explanation> tag explaining which parts are supported or not supported and from which sources.
Format your response EXACTLY as: (output.structure)
Prompt 16: Factuality Scoring Prompt.
47

---

Published as a conference paper at ICLR 2026
Report Quality Prompt
You Deep Research Evaluator. You given:
1. research report.
2. A deep research (DR) question that the report attemptia to answer.
3. A persona that represents the intended audience for the report.
{persona]
(dr.question)
freport.text)
## Instructions:
**ANALYZE THOROUGHLY**: Examine the report in detail and identify subtle problems, minor inconsistencies, areas that could be improved, or any shortcomings that might aftect the quality. Evaluate the report according to the five criteria listed below. For +*each criterion**, provide:
issues, even small ones. Look for
**score between and 10** (must be an integer) using the scale defined below.
A **detailed justificatione+ (2-3 sentences) in **sinple plain English** explaining why you gave that score, including any specific issues strengths you identified.
### Scoring Scale (1-10, integers only): (scoring_scale}
### Criteria:
1. Depth & Quality of Analysis
2. Relevance To DR Question
3. Persona Consistency Coherence Conciseness
5. Degree of radictioon
G. Completeness & Coverage
(output.structure)
Prompt 17: Report Quality Scoring Prompt.
48

---

Published as a conference paper at ICLR 2026
PDF Outline Generation Prompt
You expert business document designer creating realistic enterprise PDF reports. Given Deep Research (DR) Question and company context, generate an outline for a professional business docunent that employce would create based their persona and role.
Company: Company-names) - {company description)
Industry: industry
Company Size: (company.size)
(emppooyee.count))
Annual Revenue: (annual_revenue]
Persona Context (persona_context
DR Question: {dr.questicn]
Document
Structure Requizements:
Create professional PDF document outline with exactly (n.subsections) subsections
The document should be something this persona would realistically
create in their role
Include a concise, professional file title appropriate for enterprise documentation
$ubsection ion Heading Requi (subsect on requi rements)
Introduction Requirements:
Write a professional 4-sentence maximum introdaction paragraph - Should set context for the document and its purpose
Must align with the persona's role and company's s business needs
Should sound like something this employee would write for internal stakeholders
Conclusion Requirements:
Write a professional 4-sentence maximum conclusion
paragraph
Should summarize key takeaways and next steps
Must align with the persona's perspect ive and recommendations
Should provide actionable insights for the intended audience
Return ONLY valid Python dictionary with this exact structure: foutput.structurc)
IMPORTANT:
Ensure the document feels authentic for this personas role and company context
Prompt 18: PDF Outline Generation Prompt. This the first step of embedding an insight into a PDF document. The LLM is asked to generate outline of the document that the insight be injected.
49

---

Published as a conference paper at ICLR 2026
4 PDF Insight Injection Prompt
You are an expert business document writer creating realistic enterprise PDF content. Given a Deep Research (DR) Question, company context, and specific insight, generate professional content that naturally incorporates the insight information to help answer the DR Question.
Company: (company-name) (compnyy.deseription)
Industry: (industry)
Company Size: (company.size) ((employee.count))
Annual Revenue: (annual revenue]
Persona Context: [persona context)
DR Question: (dr questicn]
External Market Context (for reference): (externaL.context]
Target Insight:
Specific Question: (spesiffic.uusttion
Answer: {answer)
Justification: (justification)
Subsection Heading: (subsection.heattee headingg
Content Generation Requizements:
Cenerate realistic business content for the given subsection heading
The content must contain exactly ONE paragraph of 4-5
sentences
Content should professional and sound like scmething this persona would write
The paragraph must naturally incorporate the insight answer information but NOT copy it ford-for-word
Content Strategy:
Present the insight information as business findings, analysis results, or operational data
Embed the key metrics within broader business context implications
Use natural business language to discuss the same information as in the answer
cecditiional.conteent.regg rements]
Justification Requinements:
Explain specifically how this content helps answer the DR Question
Reference the key information that would be useful for dccision-making
Keep justification concise but clear (20 words maximum)
Return ONLY a valid JSON object with this exact structure: foutput structure)
IMPORTANT: (important.cetails)
Prompt 19: PDF Insight Injection Prompt. This is the second step of embedding an insight into a PDF document. The LLM is fed with subheading in the outline from 18, and tasked to write the subsection with the insight embedded.
50

---

Published as a conference paper at ICLR 2026
4 PDF Irrelevant Section Generation Prompt
You are an expert busi nesi document writer creating realistic enterprise PDF content. Given a Deep Research (DR) Question, company context, and subsection headings, generate distractor content for each subsection that is thematically related but does NOT help answer the DR Question.
Company: (company-name) [compayy.description))
Industry: industry
Company Size: (company.size) ((employee.count))
Annual Revenue: (annual revenue]
Persona Context: (persona context)
DR Question: (dr questien]
External Market Context (for reference) (external.context]
Subsection Headings: (subscttion.headingg)
Content Generation Requizements:
Generate realistic business content each subsection
heading
Zach subsection must contain exactly ONE paragraph of 3-4 sentences maximum
Content should be professional and sound like semething this persona would write
Content must be thematically related to the DR Question's s domain but NO? provide information to answer it
(additional generation requi rements]
Distractor Strategy:
Focus adjacent business areas that don't directly impact the DR
Question
Discuss historical context, general industry trends, or procedural information
Include operaticnal details that are realistic but tangential Reference related but non-essential business metrics or activities
Avoid any content that would help someone answer the DR
Question
Justificati on Requí rements:
Explain speci specifically why each paragraph' content doean't help answer the DR Question
Identify what type of distractor strategy was used
(e.g., "Eocuses historical data vs current decision factors")
Keep justificationn concise but clear (15 words maximum)
Return ONLY valid JSON array with this exact structure: (output.structure)
IMPORTANT: (important instructions]
Prompt 20: PDF Inrelevant Section Generation Prompt. This is the third step of PDF document generation. The LLM is asked to out the outline from with irrelevant information.
51

---

Published as a conference paper at ICLR 2026
Excel Schema Generation Prompt
Generate the JSON schema and formatting of a table where the following insight can be presented as row in the table: {insight}
The schema and formatting should contain:
table name: a string of the table name
columns:
list of columns with the follewing
fields:
name: column name
columnttype: one of
STRING| INTEGER FLOAT BOOLEAN PERCENTAR CURRENCY
description: detailed description of what this column represents and how it relates the insight
formatting: dictionary with the following fields: header.style: background color of the header, default is cocoac
column.xidths: width of each column, e.g. {(A: 15, B: 20, 12)}
number_formatee formats: format of each column, e.g. ((A "0. "0.008", "$4, #4".00", C: "YYYY-MM-DD"]
Return only json dictionary with the table name, columns, and formatting.
Requirements:
The schema should be designed so that the insight can be represented row in the table
The schema should make it easy generate more data peints expanding on the subject, theme and scope the insight to populate the table.
Use real realistic column names that would be found in business spreadsheet
not incluae the insight, specific question, er justification as columns in the table
Company Context: {company.info)
Please keep this company context in mind when creating the schema.
Persona :ontext:
[persona)
Please keep this persona context in mind when creating the schema.
Prompt 21: Excel Schema Generation Prompt. This is the first step of Excel file generation. The LLM asked to generate the schema of the Excel file so that the insight can injected.
52

---

Published as a conference paper at ICLR 2026
Excel Data Generation Prompt
Given the following schema: {schema.ard..ormatttuu ting] Generate one row that enbeds the following insight: {insight}
Then generate 5-10 rows of data that populates the table. Make that with the data added, the original insight can still be extracted from the table. Return all the rows in a ison dictionary with the following
ficlds:
insighttrow: list of values each corresponding schema
column in the
irrelevant.rows: a list of rows that are used to populate the table, each row is a list of values each corresponding to column in the schema
Ensure only a json dictionary is returned, and zeturn nothing else.
Requirements:
Make sure the insight row stands out from the irrelevant rows by e.g.
Having the largest value
Covering the most recent timeframe
Excel Data Generation Prompt. This is the second step of Excel file generation. The LLM is asked to generate the data for an Excel file that the insight will be injected into.
Prompt 22:
4 Excel Filename Generation Prompt
Generate professional tilenane for an Excel file that contains the following sheets: (sheet_names)
The filename should:
descriptive and professional
2. Reflect the main thene or purpose the data
3. Be suitable for a business environment
4. Not exceed 50 characters
5. Use only alphanumeric characters, spaces, hyphens, and
underscores
6. Not include file extensions (like xxsxx)
Return only the filename, no additional text or quotes.
Company name: (company name]
Please keep this company name in mind when creating the filename.
Prompt 23: Excel Filename Generation Prompt. This is the third step of Excel file generation. The LLM asked to generate the filename for the Excel file that the insight will be injected into.
53

---

Published as a conference paper at ICLR 2026
Powerpoint Outline Generation Prompt
You expert business presentation designer creating realistic enterprise PowerPoint presentations. Given Deep Research (DR) Question and company context, generate an outline for a professional business presentation that an employee would create based on their persona and role.
Company: (company.name) (company.description)
Industry: (industry)
Company Size: {company size) ((employee count))
Annual Revenue: (annual revenue)
Persona Context: (persona.context]
DR Question: (dr.question]
Presentation Structure Requi
Create professional PowerPoint presentation cutline with exactly (p_subsections) slides
The presentation should be scmething this persona would realistically create in their role
Include concise, professional presentation title appropriate for enterprise presentations
$lide Heading Requ' rements:
Slide headings must fol low the THEME of the DR Question but should NOT di rectly address the Question itself
Think of related business areas, adjacent topics, or supporting themes that would naturally appear in an enterprise presentation
Headings should sound professional and realistic for this industry and company size
Each heading should be 3-8 words and use proper business terminology
(additional.sle slwdeelllslddeeeeee wee.sssiie.errreeeeeeeeeee requittt
Conclusion Requirements:
Write a professional 2-sentence maximun conclusion for the presentation closing
Should summarize key takeaways and steps
Must align with the persona's perspective and recommendations
Should provide acticnable insights for the intended audience
Return ONLY a valid Python dictionary with this exact structure: foutput.structurc)
IMPORTANT: (important.notes)
Prompt 24: Powerpoint Outline Generation Prompt. This is the first step for generating powerpoint slides. The LLM is asked to generate outline of the slides that the insight be injected.
54

---

Published as a conference paper at ICLR 2026
Powerpoint Insight Injection Prompt
You aze expert business presentation writer creating realistic enterprise PowerPoint content. Given Deep Research (DR) Question, company context, and a specific insight, generate professional slide content that naturally incorporates the insight information to help answer the DR Question.
Company: (company.nama) (compnyydescriptiinn)
Industry: (industry)
Company Size: {company.siee (employee.counl)
Annual Revenue: revenu])
Persona Context: (persona.context)
DR Question: [dr.questicn]
External Market Context (for reference) (external context]
Target Insight:
Specific Question: (specific..ueatioo))
Answer: (answer)
Justification: {justification)
Slide Heading: (subsoction.hcadinng)
Content
Generation Requirements:
Generate realistic business content for the given slide heading
The content must contain exactly 5-8 bullet points with substantial detail
Each bullet point should be 1-2 sentences with specific business information wew.itiioaagggeeattionn requi iegnreaattoorrreurreeeeesss
Content Strategy:
Present the insight information as business findings, analysis results, or operational data
Embed the key metrics within broader business context
Implications
Use natural business language to discuss the same information as in the answer
(additional content requi rements]
Justification Requirements:
Explain specifically how this content helps answer the DR Question
Reference the key information that would be useful for decision-making
Keep just justifications concise but clear (25 words maximum)
Return ONLY a valid JSON object with this exact structure:
foutput.structure)
IMPORTANT: (important_úetails)
Prompt 25: Powerpoint Insight Injection Prompt. This is the second step for generating powerpoint slides. The LLM is asked to generate slide content with the insight embedded.
55

---

Published as a conference paper at ICLR 2026
Powerpoint Distractor Injection Prompt
You expert business presentation writer creating realistic enterpriss PowerPoint content. Given Deep Research (DR) Question, company context, and slide headings, generate distractor content for each slide that is Chematically related but does NOT help answer the DR Question.
Company: (company.name) - (company.esccription)
Industry: (industry)
Company Size: {company.size) (<employco.count))
Annual Revenue: (avenu]
Persona Context: (persona_context)
DR Question: (dr.questicn]
External Market Context (for reference): (external.context)
Slide Headings: (subsection heacingsiohhhacinngs))
Content Generation Requirements:
Generate realistic business content for each slide heading - Each slide must contain exactly 5-8 bullet points with substantial detail
Each bullet point should be 1-2 sentences with specific business information
Content should be professional and sound like semething this persona would present
Distractor Strategy:
Focus adjacent business areas that don't directly impact the DR
Queation
Discuss historical context, general industry trends, or procecural
information
Include operational details that are realistic but tangential Reference related but non-essential business metrics or activities
Return ONLY valid JSON array with this exact structure: (output.structure)
IMPORTANT: (Important.detalle))
Prompt 26: Powerpoint Distractor Injection Prompt. This is the third step for generating powerpoint slides. The L.M is asked to generate slide content with distractor information.
56

---

Published as a conference paper at ICLR 2026
Email Setup Prompt
You are an expert in enterprise se commnicattion systens and organi zat onal structures. Your task is to generate a realistio c. setup of users for an email system based the given insights and company context.
Company Context:
Company Name: (company-nane's
Description: (company.doseription)
Industry: (industry)
Size: (company.size) ((employee.count) employees)
Annual Revenue: (annual.
Persona Context: (persona.context]
Specciiic Question** (specilic..uestion) * **ssser to Speci fic Questicn** (answer)
reqenttmtnts:
Users that would realistically discuss these insights
Generate a minimal but sufficient setup to support (nun.messages)
emails discussing the insights
To make it realistic, generate at least users
Use realistic nanes for people/teanss/chnnnle based on the company context
Return ONLY JSON array of users with this exact structure: foutput structure)
IMPORTANT:
Do NOT include any preamble, explanation, or extra text Ireturn only the Python dictionary
Ensure the structure realistic for the company size and industry Make sure the persona included as user
Prompt 27: Email Setup Prompt. This is the first step for generating email chain. The LLM asked to generate the necessary setup for the email chain.
57

---

Published as a conference paper at ICLR 2026
4 Email Insight Injection Prompt
You are an expert at creating realistic business email conversations. Your task is to create an email thread that contains the actual insight that helps answer the Deep Research question.
Company Context:
Company Name:
(company .ame)
Description: (cempany description)
Industry: (industry)
Company Size: (company-size)
Employee Count: {employee_count]
Annual Revenue: (annual.revenue]
Persona Context: [persona_conter)
Deep Research Question: (dr.question)
Email Setup: (email.setup)
Target Insight:
Specific Questien: {epeeific..quet.oon)
Answer: (answer)
Justification: (justification
Requirements:
Create realistic email thread of (num messages) that contains the target insight
2. This thread should provide information that directly helps answer
the DR question
insight should naturally embedded the email content
4. The emails should feel realistic and business- appropriate
5. The sender should be someone who would naturally have access to
this insight
6. The persona needs to be either recipient or the sender of any email
Content Strategy:
The thread should discuss the speci fic question and provide the answer as part of a natural business conversation
Include the justification as supporting context
reasoning
Make the insight feel like natural part email, not forced
The content should be directly relevant to answering the DR question
Use realistic business language and formatting
Example approaches: [example approachee)p))
Output Format: Return ONLY a JSON array with the following structure: (output.structure]
IMPORTANT: (important_úetails)
Prompt 28: Email Insight Injection Prompt. This the second step for generating an email chain. The LLM is asked to insert an insight into the email chain.
58

---

Published as a conference paper at ICLR 2026
4 Email Distractor Injection Prompt
You are an expert at creating realistic business email conversations. Your task is to create (num_messages) emails that discuss topics related to the company but will NOT help answer the Deep Research queation.
Company Context:
Company Nane:
(company .nme)
Description: (company_descriptioon)
Industry: (industry)
Company Size: (company size}
#mployee Count: (employee count] }
Annual Revenue:
(annual revenue]
Persona Context: (persena.context)
Deep Research Question: (dr.question)
Email Setup: {email.setup)
Requirements:
1. Create (num.messages) realistic email messages between the
users
2. These emails should discuss business topics
that are thematically related to the company but DO NOT provide information that heips answer the DR question
3. Each email should have realistic subject line, sender, recipients,
and content
The conversations should feel natural and business-aprropriat
5. Topics should be relevant to the company's operations but unhelpful for the DR question
Content Strategy:
Focus on daily business operaticns, team collaboration, projects,
company processes
Include realistic business language, project updates, and operational discussions
Avoid topics that directly relate to the DR question would provide insights for Make the content engaging and realistic while being intentionally unhelpful
Example topics to discuss (but should NO help answer the DR question):
(example.topics]
Output Format: Return ONLY JSON array with the following structure: foutput.structurc)
IMPORTANT:
Return ONLY the JSON array, nothing else
Do not add any text before or after the JSON
The response
parseable JSON
Make sure the persona either a recipient or the sender of any email
Prompt 29: Email Distractor Injection Prompt. This is the third step for generating email chain. The
LLM is asked to insert distractor information into the email chain.
59

---

Published as a conference paper at ICLR 2026
Chat Setup Prompt
You are expert in enterprise commnicatiin systems and organizational structures. Your task is to generate
a realistic setup for teams, channels, and users for Mattermost chat system based on the given insights and company context.
Company Context:
Company Name: (company.nane)
Description: (company descrrppiooci
Industry: (industry)
Size: (company.size) ((employee.count) employees)
Annual Revenue: (annual.revenue)
Persona Context: (persona.contoxt]
Specific Questien** (specific.qqustion)) *nnseer Specific Questien** (answer)
Requi rements:
Cenerate teams, channels, and users that would realistically discuss these insights
Make sure the teams and channels are realistic for persona to be member
Each channel must be associated with tean
Each user must be rember of least team and channel
Generate a minimal but sufficient setup to support (nuntturns) chat messages discussing the insights
To make it realistic, generate at least teams, channels and 3 users
Use realistic nanes for people/teams/ceeeeeeeepp/ channelssssssss based on the company context
The persona needs to be part of all teams and channels
Return valid Python dictionary with this structure:
IMPORTANT:
NOT include any preamble, explanation, extra text Ireturn only the Python dictionary
Make sure the persona is included as user and member of all
teams/channelll
Reuse the username of the persona provided in the persona context
Prompt 30: Chat Setup Prompt. This is the first step for generating a Mattermost chat. The LLM is asked generate the necessary setup for the chat system.
60

---

Published as a conference paper at ICLR 2026
4 Chat Insight Injection Prompt
You are an expert at creating realistic bus business chat conversations. Your task is to create chat conversation that contains insight that helps answer the Deep Research question.
Company Context:
Company Nane:
comppany-nanel
Description: (company_descriptiion)
Industry: (industry)
Company Size: (company size}
#mployee Count: (employee count]
Annual Revenue: (annual revenue]
DR Question: (dr.questien)
Chat Setup: {chat_setup)
Target Insight:
Specific Question: (specificcquesttion)
Answer: (answer)
Justification: (justification)
Requirements:
Create realistic chat conversation (could be muitiple messages)
that contains the target insight
2. This conversation should provide information that di directly helps answer the DR question
3. The insight should naturally embedded in the message content
4. The conversation should feel realistic and business-an resiness-appropriate
5. The sender should be someone who would naturally have access to
this insight
the teams, channels, users from the chat setup only
Content Strategy:
The conversation should discuss the specific question and provide the answer as part of a natural business conversation
Include the justification as supporting context or reasoning Iadditiionl.content...o www..onnttetrrruiireeeettttt
Example approaches: (example.approachee)
Output Format: Return ONLY JSON array of the chat messages with the following structure: foutput.strueture)
IMPORTANT:
Return ONLY the JSON object, nothing else not add any text before after the JSON
The response must be parseable JSON
Prompt 31: Chat Insight Injection Prompt. This the second step for generating Mattermost chat. The LLM is asked to insert an insight into the chat system.
61

---

Published as a conference paper at ICLR 2026
4 Chat Distractor Injection Prompt
You are an expert at creating realistic business chat conversations. Your task is to create (num,turns) chat messages that discuss topics related to the company but will NOT help answer the Deep Research question.
Company Context:
Company Name: {company.name)
Description: (company.descripcion)
Industry: (industry)
Company Size: {company size}
Employee Count: (empleyee count]
Annual Revenue:
revenue
Deep Research Question: fdr.question)
Chat Setup: fchat.setup)
Requirements:
1. Create (num.uurns) realistic chat messages between the
users
2.
These messages should discuss business topics that thematically related to the company but
DO NOT provide information that helps answer the DR question
Each message should have realistic sender, channel, and content
3.
The conversations should feel natural and ausiness-aprooriitt
5. Topics should be relevant to the company's operations but unheipful
for the DR question
6. Use the teams, channels, and users from the chat setup only
Content Strategy:
Focus on daily business operations, team collaboration, projects, and
company processes Include realistic business Language, project updates, and operational discussions
Example topics to discuss (but should NOT help answer the DR question):
(example.topics]
Output Format: Return ONLY JSON array of the chat messages with the following structure:
foutput.structure)
IMPORTANT:
Return ONLY the JSON array, nothing else not add any text before after the JSON
The response must be parseable JSON
Prompt 32: Chat Distractor Injection Prompt. This is the third step for generating Mattermost chat. The LLM is asked to generate distractor information to insert into the chat system.
62

---

Published as a conference paper at ICLR 2026
4 Chat Confidential Info Prompt
You are an expert at inserting confidential information into chat conversations. Your task is to create business confidential information that is irrelevant to the Decp Research question or the specific question and insert it into existing chat conversations in realistic way.
Company Context:
Company Nane: (company-nane)
Description: (ccmpany.description)
Industry: (industry)
Company Size: fcompany-size)
Employes Count: femployee.couu count
Annual Revenue: (annual_revenus
DR Question: (dr.question)
Chat Sctup: fchat_sctup)
Chat Conversation: (chat.messages)
Target Insight:
Specific fic Question: (speci Fic_quest: on)
Answer: (answer)
Justification: (justification)
Requirements:
1. Create business confidention information that are secrets within the company and should not be revealed to the outside world
2. Create 1-2 new chat messages that contain the confidential information
3. Insert the new messages into the chat conversation and return the chat conversation
Also return the confidential information and the keywords
ianfidential Information Content Strategy:
The messages containing the confidential information should be necessary to address the DR question or the speci fic question the insight
The confidential information should be relevant to the cempany context and sonething that the people in chat setup naturally have to
The confidential information should be specific so that leakage of it will result in a significant disadvantage or loss to the company
Example confidential information:
(example confidential info]
Output Format: Return JSON object of the new chat messages and the confidential information with the following structure: foutput.structure)
Prompt 33: Chat Confidential Info Prompt. This prompt also generates distractor information insert into chat system like prompt 32. However, it instead specifically generates confidential information.
63

---

Published as a conference paper at ICLR 2026
Table 29: DRBench Questions and Statistics for the new tasks added (Part 1).
Domain ITSM
DR Question
Applications
Insights
Distrractos
Reual
Wist ITSM stralepies such governance improve- ments, workfoww autometion, system integrations.
10
could provide insighis
incident, problem.
and change munagement at Let's Mirket N0267
Retill
NSM
Lee's Market is getting a high rumber of service dosk
3
10
enaiss temes ou stores, so how could we roduce the 2020?
Retai
ISM
By Q3 2025, how can Lee's Market expand and optiiize its IT sell service cupubiinies to reduce service desk dependency, sccelerate resolation times, and improve both employes susemer experience across stores
10
Retail
CSM
How can Lac's Market, a ogiienal Asimn aspemarket clain, tillored Al including chathors and convenational AI, to personaiie their shuppers experiences in onlaring groeries, finding recipes. accesiig peoduer information. and entinncing beyond?
15
self-crrioe (hough
Retuil
CSM
How can Lee's 5 Market use conversttinal Al and cata driven chathets to capnase the growing Conten- ninl market providing them with isstant feedback and responding to cultural among us diverse castomer base by May 2026'
5
15
Enguisti nuncess
Retail
CSM
How can Lat's Market, regional Asim sspemarket chain, use comernatioal auress media, live clui, and tetting to improwe customer inagement
15
and loyalyy among Centernial shoppers while
ensuring
handing customer interacioons through 2006
Retal
Knowledgs Managemens
Given the rise Al based knowledge management systems, what striegges can Lee's Marker's knowt. edge management low amploaye timover rites throggh
4
14
implement
maintann
Retal
Cyoeecurity
How can Lee's Market, gailed by information secu- rity mansger Jason Wong design and implement cybenacurity awureness and training program the end 2023 fhat milgptes security risk from high on- playce tumover and seasonal hiring while mizamizing incidents caused human erre and supporting the compuny's 's ynowh both US Cansaian markets?
14
33
Retil
Cynenecurity
How can Jason Wong given Lat' Markers' limital IT resouices in 205, strengtien empleyes eyherseu-- rity riining to opention dismptions and firencial loss Q3 2027?
14
33
risk
heratticks that
Retuil
CRM
Berwee 2025 2027, Andhew
stratgies that
put into place Lee's redice ieputational rick asoociated with corporate social responsibllit communication and the same levetage the oppotuniiess positio isself an ahemutive to lurge retulers across Canada and the United States?
Ratai
CRM
Which community purmesslii progeaas that berwees hundted nillion dollars and hundred million dollars the best return on invesiment during the period of 2022-1033, messurll by media coverage value,
3
regional food ratilers
and improvemets how people fell about the bemi??
Retai
CRM
What curbon looiptit metics and methods of communication did online groeery retillers shase about delivery opertions thooghout 2024, and how did being open about these environmenal
ampocts affect customer perception competiiie mackets Canada the US?
Healtheare
CRM
The WTT Solution aricle from March 2025 indicates that Custumer Rehaiomship Management (CRM) sollware trending the healthesre industry. How can MediConn Solutions leverage sultwure drive the gnowth of its healtheare services, using porient benefiss from the software is key to incressed business the 2028?
Healtheare
Soluncos Morch 2015, Cuarmer Relationéip Management (CRM) software is nending. What kidd of business this sottore analyse, which would sid the powth Medicnn Solutios' vinua heilhhre clienele going into the you 20267
According an uniele in
Healthiare
Mariet Amalysis
Conedering the iole heallt MedCCom Solutioss improve potenn experience vemss of trust, satidaction, and rotention in Canada Q4 2026 stay ahoad af competines?
sends discusee
14
the articke,
64

---

Published as a conference paper at ICLR 2026
Table 30: DRBench Questions and Statistics for the new tasks added (Part 2).
Industry Hecalincree
Domain
DR Quedion
Applicationns
Ineights
Distractors
Marker Analyss
Staring in 3026 through QI 2007, how cun Meci- Coan Sohmonn integrase the use of Al in creating the putestt experiencs, starting from the first
3
14
louch point through care delivery follow up incresse the sumber of patiens that engage with their iols healts services
Healliorre
Market Analyis
CIIG Healthhare unicle June 2025, which the firm McKinary and Compony estimotes that mom thin 50 million inspenson visis, could be curversed vinual visios. how could MediConn Solution's morketing deportment promone service to their current patients through Q4
5
Healtheare
FISM
How would incopocating AL-T Service Mattgemmee SMI allow MediC Solutions' IT Service Desk employess become more efficient and effective 2026?
10
24
Healfhaae
CSM
The arricl published TechTBrget in Jameary 2024 the MGMA's findings showing that communicatinn technology adkresses digiaal frunt doer isues like pour booking systems. How might MediConn Solutions entance puientt presciption minagomentt services in QI 2026 to reduce wait times and boost sutisfaction and resenton?
13
30
virial consltttion
Healihaae
Knowkelge Management
How can MudiComn Seiutions snengffir knowt- edge messggmenn sysee by Q3 2036 help virtual care towus prevent knowledge related erros that are shown be lealing came of melical eners.
2
Ticaliocre
Knowledge Management
Medt Soluzons levenge its lodge managemeut systems in 2026 ensure accurate and sale presciption management in respoese to newly upproved medications, minimizing the risk medication crmes for patients?
2
Hosltheaee
Knowledge Minagement
In 3126, how oan Medi Solutions use it virtuel knowledge management platform deliver targeted continumus training for healtheare professionls milidisiipiiury care teams, affresingg knowlldge pops and improving puiinn care onsoomes in head and neck concer munagemen?
2
Hicllbore
sintegies can MediC Solations
2
ogt for to boost sales for their digital health services
Candda order to reduce readmission rates and lifttine reteation by 2026?
Hellhare
Sakes
Whut sales stsingies can MidiCam bunch Ql 2026 yow its aistamer heve for virual healfheam services while manging the key challenges of entering morket??
2
Healtheae
Seles
After coneering a clien into fill-ame user thair virtual healthiare digital services in 2026, what key factors should be considered
2
long elient scteation MediCoon Solutoos?
Hoalioaaee
Cy byhnscurity
In 2026, since a ramermane altnck would nui be just every function of Mod Solations what specific unchttetural procedural controls would syberscurity specialist need to design and implement to minimize the blast radius estended
12
issue
and MediConn's clinical comtimity curing
one thinl-purt purty
rewidee??
Healtheare
Cyhersocurry
By Q4 2055 how can Med Cont Solatone intogrure HHS Cyberseurity Performance Goolks (CPCGa into
4
12
besithowe platform to cnaute compli-
ance fox both interaal systems and thirk part vendaes,
while effectively miligring emering syher threes?
Hallboo
Cybenecurriy
2026, can Medic Solazons its vatual healtheare platfors against cocrdiaazed
12
ransomware attccs facilinted by furiig nation staee
eyber threat acturs ensaring uninterspted access clinccal systems and patient saley?
Flamic Vehiole
Seies
How will Flasion Autmntive's adoption of complete direct cucooommer siles model 2026 eycle dunaton,
13
30
avenge revemee por customer soms the United
Stiees with franchise live?
Eleecic Vehack
Sales
Elesio Ausemstive daiis from dealershp
13
franchising to dinect-b-cussmmer sales model could it cagture the benefis of the imssition to drive sales and higher mangits in 2027?
Eleoric Vehicle
2015, can Elesion Automutive use data peremted through its cusoomer rehtiioshipp management sysem analyze customer behavior online and idently poleems that drive salss
by
13
30
65

---

Published as a conference paper at ICLR 2026
Table 31: DRBench Questions and Statistics for the new tasks added (Part 3).
Industty
Donaii
DR Question
Applicaaionss
Insights
# Distractors
Eleetr: Velicle
Sales
the 2025, how Lietion Automotive relationhip
13
30
genersed thouug
managemeent system to analyze customer behavice
uniine and stentily pullems faa drive sales?
Hletee Velicle
Research
How well Eletoo posttoned to
and take
lesming in afvnnce ADAS techoology by June 2025 while providing friendy and-ise experience for drives?
alvnttae
technology
potemaal to war
drivers about husmls they night not see in fiomt of them. How can Elexion uttiiz Al lest our ADAS while mainaining compliance certiiictions by
Fleari Velicle
Reseandh
How can All he red help Elesion desiity problems with their EV' before they become souce when EU's
policy
poss offect
Bletuee Vehicle
Cybenecuriy
Caven the Help Net Securiy artilee from Apeil
2025, what stanggies should Elesicn Autoustivv
going base from
remote cybenecurity
Ekanee Vehicle
Cybereourity
Reterencing the infommation sharel in the April 2015 article from Help Net Suarrty what external cybenscuritt
involving
industry should Elesion Astomative analyse by the mil Q4 2015 in protect conamer dats and solary? Securiyy Elesion Automottve poised adapt and take proactive meusures to pravent liseli and conmaner base from the litss cyhersecurity threats approach 2026?
Hleme Veliele
Cyherscurity
resposse
2025
artclle
Bleune Velicle
Quality Asurance
Loo can Eleston Automotie's is control imcrraue design stralgy for 2026 EV models priontize physical bumons and wwiches Q3 2025 reduce
controldisply
and achiov sccess below the 268 EV average? What impucr would 20% incesse dealer-ial cus- education EV infortimentt reducing
Flearie Valiee
Quiity Asurance
3
frequency by
2027
Electre Velicle
Asse Managament
Q2 2026, how Elexion Autumotive leveruge insegnaion digital eagineering methodologies with diginl twin to uptimize EV batery perfernacc and lifecylle mungemmnt, w whike enaring regalatory complancee North Amercan markew!
Velicle
Mastigenentt
Autonotive use from I/T sensors embedded in vehicles and manffactting apipmenn combinod with digita eiglneering-etuble prodictive muineennce and minimize downtime in prodiction lines by the and of 2005?
can
data
cahance
Hlettr Velicle
Market
2004 rppor TTenns electic lea. competior stateglcs positioning shaping the North American sector by qE N06?
Market Analysis
marker tend seeking and trading down to discoumiens what specific metios should Lee's Market uilize to mensore the incremental market share grined within ungeet
valio-
community divens urban markuts by Q4 2026, assmmnng stralggcc on expendingin its ailueslly antheniippivan fabel pro- primary value prppssiion? competitive contest, incloding the railed Alhensons Kroge merger and the persis- tont threst from discoutters what specific compentive dan should Lee's Mirke's Market Research team mekk autyye over next moorts messure the shift in consumer behavior across its target diverse urban centers, specifically conceming the simultaeems prirrtization of valuo-wecking wellnes??
purel lodk-odlerings light
Market Analysis
Retail
Market Analysis
Caven the indusry socus on operational eltaciency tend of using technology like electronic shelf menicese 3025 the projected return investment (ROI) that Lee's Marke's ' Canadimn openations can expect from full rollout techualogy (J 2020, considering unique challenge managing high volume bilngul/mililinnai product informaion?
66

---

Published as a conference paper at ICLR 2026
Table 32: DRBench Questions and Statistics for the new tasks added (Part 4).
Industry Healinar
Donain ITSM
DR Question
*
Applications
Insights
INsrractors
As the CPPA Act (Bill C-27) awatts pelfiammntryy decision. can
15
build pesparedness measures to enoure smooth trasition the even the Act comess into effect by Q 2026?
Retal
ways for Lec's Marker enhhnnc on-door custoner eigagemert Canda's July 2025 renill saless deeline of 0.8% and drive messanble schound in the semsining months
What
imjoctt
Healineare
bersecurry
In algaament with Canada's new call for proposuls to strenthenn the county's is eyber resilience and addiess evolving Security Cooperatio Program (CSCP) some considerations to keep mind while integratngg new Al momalyy dotection sool into MediConn's infrastracure by 2026 without disnupting serviee?
5
under 2025 Cyter
Blatric Vehicle
Quiliy Asurmer
Given that one in seven now vehicles sold in Canasla in 2034 were renoemisiio, how Fletion Auti- motive's QA tom enhance cold coll-wether tesing pro- loods by 3036 to improve EV batteyy endumance and relibility the countyys key 7FV markes?
Retal
How Lee's Market
20%
by targeting younger consamers (ages 18 35) and differeatioee liseff from docisnnt playas in the Canadian retail market by 2026?
Considering companies
superior
realls by foctiing um enhuncing the experience of exising contomers what CS service improements could be implemented 2026 to make customers recugnized
daring inseattion?
Healthaaee
Compliance
What adiinonl complance conpols should be integeated into MediCian's platfcrm Q2 2036 authentcation and patient venfication
16
Iedigmoous perient low-cunnettikity anviron- menss?
Heilhor
Compliance
In the fockral povermen's 205
the Coen Solutions do to aldress cumplimcee risks frum
Health
legal precedents provincial variaiions by Q4 2028
Healinere
Cyhersouuitt
What stops Med Comn Solation to opeinise cyberiecurty and protection
FV FY2066
the bealthowe indistrys i growing fooss on diginl
engagemenn and opentienal officiency?
Rotal
What considentions
need made
Markei is redesigning its loyalty program by Q2 2025 to serve both Gen 2/Millennial digital preferenzes
Gen X-Boomer tradioonl
expectation!
Ratai
per report by the U.S. Cenes Rurem's
indication of rise in food service sales since Augus 2034, how Lec's Market optimise in-ttoe byoets product displays
U.S. locations
incruse bukery sales 15% the end of 2026
Healtocure
1ISM
How can MediConn streamline its EI workdlows in 2025 boost clinician efficiancy per-onsultation costs, given virual healticare expansion has improved access but incressed overall expenses?
masage
drop Canadian
sales brought
by US tarifls 2025, what insights should Lee's Market derive from produx- level sales trenss ocnoss uniff-exposed and turift insuleed categorios uphite sales
by
Bleenic Veticle
Compliance
How shouid Hizuno Autonnotvve update its compli.
5
ance documentatio framework by 2026 to verify record suprly-chiin
credits under
Canada's sevsed swndune?
Healooure
LISM
Which El modemization eltorss should MesiCion emphasize aligned with national vittual care sundmm and shured digital health infradructure 2026
16
67

---

Published as a conference paper at ICLR 2026
Table 33: DRBench Questions and Statistics for the new tasks added (Part 5).
Industry Ranil
DR Quedion
Applications
Ineights
Distractors
Complinnce
How is Lee's Market positioned to communcate
2
with all of its suppliers, big and small, conceminn
FSMA regulations, specifically the trakingg codes!
Ratall
Market Analysas
Given the 2025 murket trend of consumers value- seoking and inding down discoumers, what specific metics
Lee's Market messur the incremenaa market store gained within its unget Asian community und divers urbn center murkets by Q4 2026, asuming sinaegic focus expunding culturally authemti privase and pre- pured foods ofictigss us its primary value propodtion?
Healtheare
Complinnce
What regulanory diallenges should MadiCion
3
pupure for deckded expind its cosb-coly services
United Suyss?
Healtheare
Marketing
With the mpdid ierense in virtall consulations for healiscure visis reported 2023, what marksting stalngies can MediConn
3
10
adopt continue
atracting new customens!
Retail
Knowksdge Minagement
Considering the Retii Knowladge Management artille published
4
14
Knowmat website June
2025, what integaation sintegies could Lee's
Market consider for its knowlalge systems to help schiew fimmiial brpps for 2030
Knowledge Manzgement
Given growing prominence
souutons
4
diseaseed in the June 2025 Retil Knowledge Management amiicee. what Al-dives knowledge management stateiies should Lee's Market adopt
ultionndl enltnnc castomer experience by 2026?
Elaoric Velicle
Compliance
How can Elesion Automotive ensare product
2
compliance with CARB's 100% ZEV sales mandate 2055 Califoriia
Retul
Cybereouray
Between QJi 2025 and (44 2025, how can the adoption
14
33
of managed security service providess (MSSPs) be lovenged by Wong address Lee's Markee's intemal cyhersccuitt resurm constmintss what mesernalle improsement in threat mausgemenr com- planne, and custmmer tust can he achieved compared muintaining tnditional in-hunse appoaches??
Ehreric Vehicle
CSM
With the reported skordwn in the sales of electtic vehicles (EV), how cun after sales products and service straegies help cumpany remain suscessful 2026?
3
Heilfhoee
CRM
Accedling to an article in WTT Solutions in March 2025, Cistomer Relations Managenent (CRM) an importunt component any healticue
4
12
business development reprosmtative look for in CRM software that would coin Bute to
MedCCom Solutions business growth in 20267
Heallbocre
Medil Solutions use integated Service Miragemen: (TTM) to impeove its support of semore care and mobile healhh apps while mamtaining
10
suadurd of regulatrry complance and delivesing a seamless experieace healtheare pretessonals by Ql of 2027?
potcents and
Healtheane
ITSM
By Q3 2025, how can MediConn Sclution' IT Servioe Maragemenn (ITS)) manage security puttodss enare regalatory complince. and implement prevenivee messures againsi netwerk and information sysem securty risks, given the senstivity electrniic medical dats"
10
34
Healihaare
CSM
A 202 patient expesience rpport found that six in 10 patients ideniiied poor online booking tools and comollted call
13
30
burners to making appirnments. When individual potients or copprnte employess stuggle to book appocaments on ModCCon Solutions platfomm, how cantt I determine whether
post booking toal design convoluted customer suppart processes?
Hecalieare
CSM
Auconing an artice published on lechi larget.com 2024, excellent paizent communication criical successfill customer service deparmennt How can MediCom omn Solution improve patient communi- cation in order reducce service calls into Q 122055 2036 which specific feonaes of Solesforr's Autocnotive Cloud, Management, or Mahick Comall, could most effee- tively eshunce Flexien's castemer retoniion ssategies mid-inoome families markets, koeping mind Flesion Astomotive's focma on sustinabilly mesaging and oftersunthuse charping sutinn suppart?
13
30
Flamiic Velicle
CRM
Drive Chnsole, House
North Amseicaa
68

---

Published as a conference paper at ICLR 2026
Table 34: DRBench Questions and Statistics for the new tasks added (Part 6).
Indusiry
Donsin
DR Question
Application
Insights
Distrccorr
Eleetre Velicle
CRM
Caven than manutactures typically lose all contact
14
original dealer, how Automotive Cloud's vehicle
their vehicles
tacking capabilines help Blesion Automutivve incresse enpagement with socced-haad lwyees of lts 25% Nonhh
2055
Blectee Vehicle
CRM
How could Elexse Aucomative unlize Ausomotive Cloud's deaker performance management wolls by enly 2027 sinigthem relutionchips with fis North American dealer
bin 90% direct wo govenmmnt sales shannel?
Electric Vehicle
Quality Asunnoe
How Elesion Automative's quality neuuneee printto Apple CuePlay Andrid
connectvity performnnc by Q4
2006
EV models to diffrretiate from compottoos and
atteet 50S of Apple users and 42% Sumsung oaes who
smarphone
Blectre Vehicle
Asser Mantagement
By QJ 2025, how can Eleson Autometie apply cigi- ui twns enhunced with Al drive digital engineering capabilines enable sealable cusomizatioon of models susainable resouree usage and reducing production costs?
maintaning
Eletrii Vehicle
Market Analysis
2014 report "Trends electrcc leuorg, which developments butlery technulogy and EV supply chiins are liley to impoct global production and delivery dysoeicc Q2 2036?
Analysis
evolving benery price mends. supply chain dynamics, and regional material avallabiity highlighnd in the 3024 IEA report "Teeds Flaatie Elexion
Cars,
optimize
banory sourcig procurement stroggy
cos efficincy and production rsilience by QI 2026 How can Elexion Autumotive sinngthen nuun
Electri Vihicle
Markei Analyis
10
desiership gresence incrase
visbiity in Q4?
Eleatee Veliele
Market Analysis
How can Elexson use virual healtheare remote diagoostiss
4
10
wanarry rejair downtime??
Eleure Velilee
Market Anulyss
How sbould Elexion aluust its batury souving strakgy mitigate lithium price volutiity expected 2025?
Analysis
Wiuch compllance gaps
meet
uptansl CARB ZFV 2025 reporting eqpirement?
69


Reviews:
[{"review_id": "KeePFrE4s5", "reviewer": "Reviewer_jNFK", "summary": "This paper introduces DRBench, a benchmark for evaluating AI agents on multi-step, long horizon enterprise deep research tasks. It consists of 15 persona grounded tasks across 10 domains. The tasks comprises of retrieval and insights generation from public web content as well as private enterprise data (emails, chats, documents, spreadsheets) to answer business related queries. It proposes 4 evaluation metrics - Insight Recall, Factuality, Distractor Avoidance, Report Quality. It introduces DRBench baseline agent and evaluates it on the benchmark across multiple planning strategies and backbone models.", "questions": "1. Results show relatively poor performance in insights recall metric across models and methods. Why is it so? What % of it is due to incorrect / noisy insights extraction?\n2. How good is atomic insights decomposition? Is there any quantitative analysis done to measure the performance of insights decomposition as well as the different metrics scoring? \n3. The decision to choose number of ground-truth insights plus five for calculating insights recall score seems arbitrary. Was some other methods for penalising copying all content into the generated report explored?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper tackles an important challenge of enterprise deep research which is a problem space that remains highly unexplored. \n2. The inclusion of private datasources simulating real world applications (such as cloud storage, chat, file system etc) and containing diverse file formats creates a more realistic evaluation environment. The benchmark incorporates private datasources distributed across realistic enterprise data sources such as cloud storage, chat, file system etc containing diverse file formats, resulting in a highly authentic simulation environment.\n3. The evaluation framework consists of multiple complimentary metrics that help in evaluating agentic systems across both precision and recall and quality of report.\n4. The paper includes ablation studies across planning strategies, backbone llms and environmental settings (local vs app based).", "weaknesses": "1. A major limitation of the benchmark is its limited size. 15 tasks and 114 insights makes the benchmark significantly smaller which raises questions on statistical significance of the evaluation.\n2. Extraction of atomic insights from the final report is a very important step in the evaluation method since 3 of the 4 metrics depend on it. However due to the use of llms, this step will be noisy which will lead to less reliable metric scores.\n3. Having LLM-as-a-judge as the only method of evaluation raises question about the accuracy of evaluation since llms can halucinate and show biasness. Even though the authors talk about correlation with human preference, It does not really indicate how accurate the llm evaluations are.\n4. Synthetic data generation, even though it makes the benchmark generation approach more scalable, raises questions about the internal enterprise data being realistic in nature. Combined with the fact that LLM is also used for evaluation, it can lead to more noise and biasness in evaluation results.\n5. Even though the paper includes several ablation studies, it does not provide indepth analysis of why the results are the way they are. It simply states that certain model / approach is better than the other without trying to provide any explanation as to why it might be so. A main example of this is stating the fact that no agent managed to successfully source external knowledge without providing any explanation to why it happened.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Xcg13tMYaV", "reviewer": "Reviewer_qNkw", "summary": "This paper introduces DRBench, a benchmark for evaluating deep-research agents that must integrate public web information with private, enterprise-like data (e.g., files, chats, emails) inside a realistic multi-app environment. It proposes three evaluation axes—Insight Recall & Distractor Avoidance, Factuality (via evidence-checked citations), and Report Quality—and presents a baseline DRBench Agent (DRBA) with variants (SRP/CRP/AAP). Experiments span 15 persona-grounded tasks across 10 domains, with analyses of planning strategies, backbone LLMs, and app-based vs. local file access.", "questions": "Can you quantify LLM-judge and human agreement for each metric (beyond small samples), and report Fleiss’ κ or Krippendorff’s α per dimension?\n\nWhat are the exact artifacts you will release (images, VM snapshots, container specs, synthetic email/chat generators, grading scripts)? Any non-redistributable components?\n\nHow robust is Insight Recall to paraphrase or partial matches? Do you evaluate span-level alignment and evidence coverage per insight?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper convincingly argues that prior deep-research benchmarks are predominantly web-only and do not measure whether agents surface the most salient enterprise insights or ground claims with citations. \n\nTasks require tool use across storage, chat, email, and documents, which distinguishes DRBench from web-only retro-search settings such as DeepResearchGym and Deep Research Bench, both of which rely on fixed corpora or “frozen web” for reproducibility rather than mixed private+public sources. \n\nMulti-axis evaluation design. The insight recall vs. distractor avoidance split is well motivated; factuality uses RAG-style evidence checks; report quality is judged on structured dimensions. The methodology reflects current best practice in LLM-as-a-judge evaluations.", "weaknesses": "Evidence of external validity & task coverage. While the 15 tasks are persona-grounded, the coverage across industries and the depth of internal knowledge heterogeneity remains modest. Benchmarks such as DeepResearchGym and BrowseComp-Plus now report hundreds to thousands of instances or large curated corpora; DRBench’s small task count risks overfitting and limited statistical power.\n\nLLM-as-judge reliance & bias. All key metrics (recall alignment, factuality judgments, report quality) ultimately depend on LLM judges. The paper would benefit from more thorough human-vs-LLM agreement studies and inter-rater reliability beyond the limited assessments reported.\n\nHow robust is Insight Recall to paraphrase or partial matches? Lack of evaluating the span-level alignment and evidence coverage per insight.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "G54mNE4BEY", "reviewer": "Reviewer_hv9E", "summary": "The paper introduces DRBench, a benchmark for answering deep research questions that require synthesizing information from both Web pages and enterprise documents embedded in files or apps like emails. The 15 tasks are curated using LLMs with a human-in-the-loop approach that involves generating a company and persona, collecting relevant public URLs and extracting insights from them, generating questions based on the public insights, generating internal insights and distractors for those questions, and generating internal documents to embed those insights. The answers are evaluated based on insight recall, factuality, distractor avoidance and report quality measure using LLM-as-a-Judge. A baseline DRBench Agent (DRBA) is also developed that consists of research planning, action planning, adaptive action planning and report generation. Experimental results show that even SoTA models struggle with these tasks, particularly on insight recall.", "questions": "1. Although some tables show (e.g., Table 11) show number of questions answered successfully, why is such an accuracy not reported for all tasks and models? This seems to be the most important evaluation criterion. For example, an agent might retrieve all insights but still not synthesize them into the correct answer. LLM-as-a-Judge could be used for this criterion too. Does the benchmark include gold answers for all questions?\n2. Which model is used for LLM-as-a-Judge? This could impact results as the judge LLM is known to be biased towards models from its own family.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Novel and challenging task that involves assimilating information from various sources and also interacting with apps like emails\n2. Systematic pipeline for benchmark creation\n3. Extensive experiments and analyses to measure the performance of many models on four comprehensive criteria. Results demonstrate benchmark complexity\n4. Human evaluation to validate both benchmark creation and evaluation metrics", "weaknesses": "1. Missing relevant work \"Benchmarking Deep Search over Heterogeneous Enterprise Data\" by Choubey et al.\n2. Very small dataset consisting of only 15 tasks\n3. No agent identified public insights. There should be some analysis done to understand if that is due to lack of web indexing, retrieval approach, tool limitations, benchmark design, agent extraction etc. For example, are the public insights inaccessible by the search tool?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "loeg2r8Glm", "reviewer": "Reviewer_9AoK", "summary": "The paper proposes DRBench, a benchmark and reproducible enterprise environment for evaluating deep-research agents that must synthesize evidence from both public web sources and private organizational data real across applications. It provides 15 persona-grounded tasks spanning 10 domains with injected ground-truth insights and distractors, generated via a structured LLM+human-in-the-loop pipeline and anchored to dated, authoritative URLs. The evaluation framework scores reports along insight recall, factuality via per-claim citation verification using a fixed RAG pipeline, and multi-dimensional report quality via LLM-as-a-judge. A baseline DRBench Agent with planning variants is analyzed; results show adaptive planning boosts recall while lightweight planning better preserves factuality, and app-based environments are notably harder than local file access. The paper includes ablations across multiple backbone models, browser-only baselines, and a small human study validating metric alignment.", "questions": "- What do the stars in L948–L949 mean?\n- MinEval selects only the retail domain. Given that stratifying the same total number of tasks across industries (retail, healthcare, EV) need not increase computational/evaluation cost, why not adopt stratified sampling or include at least one representative task per industry?\n- Can the agent access public resources beyond the Task URL? If not, how do you limit the behavior of agents like OpenAI Deep Research?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The benchmark genuinely bridges public web retrieval with private enterprise data across heterogeneous formats and real apps, grounded in personas and company context; this goes beyond web-only research settings.\n- The evaluation is thoughtfully designed—atomic insight extraction, strict per-claim citation checks with a controlled RAG pipeline, explicit distractor avoidance, and nuanced report quality scoring—plus an anti-gaming cap on evaluated insights.\n- A clear, reproducible pipeline with human verification produces distractor-rich files and stable, dated public sources; the environment is containerized and well-documented, enhancing reproducibility.\n- Ablations on planning strategies, backbone models, and local vs app-based settings expose concrete failure modes, offering actionable guidance for future agent design.\n- DRBench fills a gap between deep research benchmarks and computer-use agent tests, with code/scripts promised and a setup that feels close to real enterprise workflows.", "weaknesses": "- The paper’s presentation could be slightly improved; \n  - Figure 8 could be better presented.\n  - The paper introduces “golden insight” (e.g., Sec. 6 and Prompt 15) without prior definition, seemingly synonymous with “ground-truth insights,” which creates confusion in the evaluation description. Please unify terminology and define the term at first occurrence\n  - There are two versions of labels in Figure 4 that overlap with each other.\n- Only one backend evaluation (GPT-4o) is conducted. The stability of the evaluation across different backends is not evaluated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)