LiveResearchBench: A Live Benchmark for User-Centric Deep Research in the Wild
Authors: Jiayu Wang
Yifei Ming
Riya Dulepet
Qinglin Chen
Austin Xu
Zixuan Ke
Frederic Sala
Aws Albarghouthi
Caiming Xiong
Shafiq Joty
Venue: ICLR 2026 Poster (2026)
Keywords: Benchmark
Evaluation
Deep Research
LLM Agents
Multi Agent Systems
Primary area: datasets and benchmarks
TL;DR: We introduce LiveResearchBench, a benchmark for realistic, dynamic deep research tasks, and DeepEval, a human-aligned suite for evaluating citation-grounded reports. Our study reveals key strengths and failure modes of frontier research agents.

Abstract: Deep research---producing comprehensive, citation-backed reports by searching across hundreds of live websites---marks an important frontier for agentic systems. To rigorously evaluate this ability, three principles are essential: tasks should be (1) user-centric, reflecting realistic information needs, (2) dynamic, requiring up-to-date information beyond parametric knowledge, and (3) unambiguous, ensuring consistent interpretation across users. Existing benchmarks fall short of these principles, often focusing on narrow domains or posing ambiguous questions that hinder fair comparison. Guided by these principles, we introduce LiveResearchBench, a benchmark of 100 expert-curated tasks spanning daily life, enterprise, and academia, each requiring extensive, dynamic, real-time web search and synthesis. Built with over 1,500 hours of human labor, LiveResearchBench provides a rigorous basis for systematic evaluation. To evaluate citation-grounded long-form reports, we present DeepEval, a comprehensive suite covering both content- and report-level quality: checklists for coverage and presentation, rubric-tree assessments of citation accuracy and traceability, and metrics for consistency and depth of analysis. Using LiveResearchBench and DeepEval, we conduct a comprehensive evaluation of frontier deep research systems, including single-agent web search, single-agent deep research, and multi-agent systems. Our analysis reveals current strengths, recurring failure modes, and key system components needed to advance reliable, insightful deep research. Our code is available at: https://github.com/SalesforceAIResearch/LiveResearchBench.

Published as a conference paper at ICLR 2026
LIVERESEARCHBENCH: A LIVE BENCHMARK FOR USER-CENTRIA DEEP RESEARCH IN THE WILD
Jiayu Wang', Yifei Ming",, Riya Dulepet Qinglin Chen', Austin Xu², Zixuan Ke², Frederic Sala', Aws Albarghouthi', Caiming Xiong'. Shafiq Joty University of Wisconsin-Madison
Stanford University
SSaeesforee Al Research
ABSTRACT
Deep research- producing comprehensive, and synthesizing information from hundreds of live web sources-man marks an important frontier for agentic systems. To rigorously evaluate this ability, four principles are essential: tasks should be (1) user-centric, reflecting realistic information needs, (2) dynamic, requiring up-to-date information beyond parametric knowledge, (3) unambiguous, ensuring consistent interpretation across users, and (4) multi-faceted and search-intensive, requiring search over numerous web sources and in-depth analysis. Existing benchmarks fall short of these principles, often focusing on narrow domains or posing ambiguous questions that hinder fair comparison. Guided by these principles, we introduce LiveResearchBenhh, a benehmark of 100 expert-curated tasks spanning daily life, enterprise, and academia, each requiring extensive, dynamic, real-time web search and synthesis. Built with over 1,500 hours of human labor, LiveResearch provides rigorous basis for systematic evaluation. To evaluate citation- grouudedd long-form reports, we introduce DvepEval, a comprehensive suite covering both content- and report-level quality, including coverage, presentation, citation accuracy and association, consistency and depth of analysis. DeepEval integrates four complementary evaluation protocols, each designed to ensure stable assessment and high agreement with human judgments. Using LiveResearchBench and DeepEval, we conduct a comprehensive evaluation of frontier deep research systems, including single-agent web search, single-agent deep research, and multi- agent systems. Our analysis reveals current strengths, recurring fallure modes, and key system components needed to advance reliable, insightful deep research. Our code available at: (https: / pettps://githaee /ttpgggttbtttttgg////////g com/ Sa htte//aallssssoo forcee TeerrceARRReeee TResearch I.i TpeResearchBente
reports by searching
requiriin
INTRODUCTION
Deep research is the process of addressing complex, open-ended questions that require long- horizon, multi-step reasoning and planning. It involves exploring hundreds of live web sources and synthesizing them into comprehensive, citation-grounded, and structured outputs such as reports (OpenAl 2025). This process closely mirrors how human researchers work bork-iterativelly seeking. validating, and integrating evidence. Building systems capable of this marks an important frontier in AI: moving LLMs from simple chatbots to independent mroblem-solvens However, progress is bottlenecked by a benchmarking and evaluation crisis: Which tasks are truly representative? How should ambiguities in query formulation be resolved? How can we reliably measure comprehensiveness (breadth and depth), consistency, factuality, and citation accuracy in long-form outputs while safeguarding against data contamination (Nguyen et al. 2025
Evaluating agentic tasks is challenging in general, and deep research is particularly difficult to benchmark due to its open-ended nature. To address this, we first propose four key principles for benchmarking deep research systems: tasks should be (a) user-centric, reflecting realistic information needs; (b) dynamic, requiring up-to-date information beyond parametric knowledge;
"Equal contribution. Correspondence to: milawang@css.wssc.ddt (yifei.ming, sjoty) @salesforce.com

---

Published as a conference paper at ICLR 2026
Deep Research Bench (Bosse al.):
How many IM and GM account closures did chess.com report for 2024?
No support for
Search intensive but
Fime-bound & static; Lacks support for evolving data
long-for -form answer
low reasoning load
DeepResearch Bench (Du et al.):
Write a paper to discuss the influence of Al interaction on interpersonal relations, considering Al's potential to fundamentally change how and why individuals relate to each other.
Missing
Missing content
Ambiguous scope limited search & analysis depth
target audience
format requirement
LiveResearchBench (Ours):
Create a comprehensive report about the evolution of artistic styles across different historical periods and national characteristics. Cover major historical periods from Ancient civilizations up to the present [date)). Facus primarily on visual arts (painting, sculpture, architecture). Include analysis of key regional/national characteristics including European traditions, East Asian art (China,Japan), Islamic art, and Indigenous American art, Target the report for undergraduate-Bn -level students with substantial academic depth, including proper citations and references. Structure as formal academic report examining how political, social, religious, and technological factors influenced artistic development. Include analysis of major artists, techniques, materials, and stylistic characteristics for each period and region.
Multi-faceted scope evaluation friendly
Targeted audience output expectation
Require wide search in-depth analysis
Comparison of LiveResearchBench with existing deep research benchmarks. ( ((ate)) } will be replaced by the evaluation date. LiveResearchBenchh is explicitly user-centric, unambiguous, and time-varying, featuring multi-faceted,
Figure 1:
tasks across diverse domains.
(c) unambiguous, ensuring consistent interpretation across users; and (d)
and search-
intensive, requiring multi-hop search across diverse web sources and in in-depth analysis. Recent benchmarks have introduced search-intensive tasks (Gou et al. 2025), J, but these are often static (and thus prone to contamination) (Bosse et al.) 2025), domain-specifie (Xu et al.] 2005b]] Patel et al.) (025). coarse-gras granned,, or ambiguous (Du et al./ (025)-frequent
omitting details such as the intended audience, required output format, or scope. Consequently, we still lack reliable tools to assess whether models are producing genuinely high-quality research or merely assembling plausible-so sounding yet insufficient responses.
We address this gap with Bineh,, a benchmark of 100 expert-curated queries paired with detailed checklists, constructed with over 1,500 hours of human labor. The tasks we benchmark span seven diverse domains (e.g., science, business, healthcare) and ten categories such as market analysis, literature review, policy evaluation, and topic exploration. Each query is multi-faceted: it specifies scope, audience, and format whenever appropriate, and many require real-time search at evaluation time. For example, one task asks for comprehensive report on the evolution of artistic styles "up to the present ( [date) )," requiring breadth across historical periods, depth of analysis on techniques and materials, and presentation as formal academic report for undergraduate students (Figure D. Compared to prior work, tasks in Live are realistic and unambiguous. It makes possible accurately gauge retrieval, synthesis, and reasoning in deep research.
While these queries capture more diverse scenarios, we must also ensure that the way we evaluate the resulting reports provides fine-grained insights into deep research agent capabilities. Evaluating the resulting long-form/ope fong///o/opp/dedeeddd reports poses unique challenges. For example, we cannot use simple approaches, while real-time queries imply that there is time-sensitive variation, without fixed ground truths. Most critically, tasks are inherently multi- aittiiddieniiial::: we must assess coverage. reasoning, evidence use, and presentation quality. Human evaluation costly and hard to scale, while naive L.L--as-a--udgee setups produce inconsistent results.
To overcome these obstacles, we propose EvepEval, a comprehensive evaluation suite for research reports that spans both content-level and report-level metrics. Content-level metrics assess coverage and comprehensiveness, analysis depth, citation association, and factual accuracy. Report-level metrics evaluate presentation & organization, and logical & factual consistency. Each metric is paired with a tailored evaluation protocol (e.g., checklist-base pointwise, pairwise, or rubric-tree), selected to ensure stable model assessment and high agreement with human evaluations. This design enables scalable, fine-grained, and reliable evaluation of system performance in dynamic research settings, particularly for open-ended, long-form reports.
2

---

Published as a conference paper at ICLR 2026
We summarize the key contributions follows:
We establish four core task design principles for deep research, grounded in user survey and comparative analysis, that ensure tasks are realistic, unambiguous, and evaluation-ready.
We introduce LiveResearchBench, a benchmark of 100 expert-curated, checklist-validate tasks across diverse domains and categories, explicitly designed for dynamic, multi-faccted research.
We propose DeepEval, a comprehensive evaluation suite for open-ended/long-fome reports, covering six complementary dimensions and closely aligning with expert assessments.
We conduct comprehensive evaluation of 17 state-of-the-ee art open-sourced and proprictary single- and multi-agent systems. We reveal systematic vulnerabilities and error patterns, and that most models are yet incapable of writing insightful reports.
RELATED WORK
Single- and Multi-Agent Deep Research Systems. Equipping LLMs with tools referred to as agents-for grounded answers to complex queries has become a major research direction. Pioncering systems such Deep Research using o3 (OpenAl 2025 combined browsing and code execution and generate long reports. Many successors have since emerged (Xu & Peng 2025 Java et al., 2025). Architectures generally fall into two broad categories. Single-agent systems place all tool-use decisions in one model. Examples include function-calling LLMs (Yang et al.. 2025 Agarwal et al. (025)) and ReAct-style agents (Yao et al.] 2022). Multi-agent systems ( (MAS)
instead coordinate specialized roles such planner, researcher, and writer in predefined workflow. Proprictary multi-agent systems such as Grok Heavy (xAl 2025a) and Manus (Manus) (2225)
remain undisclosed, while examples of open-source systems include Open Deep Research (Alzubi et al. 2025 DeerFlow [ante 2025). and OpenManus (Liang et al. (025). Despite rapid progress, it still unclear which dimensions favor MAS single-agent systems, what fundamental bottenecks remain, and whether MAS truly deliver consistent gains (Kapoor et al. (2025). The lack of standardized benchmarks and evaluation frameworks further prevents rigorous comparison. Our work directly addresses this gap by developing a comprehensive evaluation framework. enabling rigorous comparison of single- and multi-agent systems for dynamic and
research tasks.
Deep Research Benchmark. Several benchmarks have been proposed for evaluating deep research. DeepScholarBench (Patel et al. 2025) targets related-work generation; DeepResearch Bench (Du et al. (025) include 100 short open-ended questions; Deep Research Bench (Bosse et al.) (025). LincD (Java al., (025), and Mind2Web2 (Gou al. (025) primarily target closed-ended informatiion-sekinng tasks. While valuable, existing benchmarks exhibit some combination of the following limitations: they are domain-specific, restricted to short-form or closed-ended answers, and static, which hinders fair comparison across agents and prevents evaluation evolving information. Recent efforts such as DeepRescarchGym (Coelho et al.) 2025)
comparison is shown in Figure [ adopts short and static tasks, while ResearcherBench (Xu al.) (025b) focuses narrowly Al domain. In contrast, our LiveResearchBenchh provides user-centric, unambiguous, and time-varying tasks that are multi-faceted, search-intensive. and span diverse domains. A detailed discussion of benchmark comparison is provided in Appendix B.I
Evaluation of Long-form Answer. Recent work has moved beyond simple reference-based metrics. Some benchmarks emphasize coverage and factual grounding (Laban et al.) 2023 2024 Huang al., 2024; Xu et al. (025a), such as PROXYQA (Tan al. (2024) (sub-question coverage) and LongFact (Wii et al. 2024) (claim-leve verification). Others target structure and overall quality, including Longhvval Wu et al. 2025) (plan-based vs. direct generation) and HelloBench (Que et al., 2024) (human-aligned protocols). However, these efforts typically isolate single dimension. In contrast, DvepEval offers a comprehensive suite: report-level metrics (presentation, consistency) and fine-grained dimensions (depth, checklist-based coverage, citation traceability, rubric-tree accuracy). Crucially, DvepEval is reliable and human-aligge -llnandd instead of holistic scores, we use structured checklists (c.g.. factual errors atomic tests) and aggregate results, yielding stable grades that closely track expert judgments. A detailed discussion of evaluation methods for open-ended long-form reports is provided in Appendix B.2
3

---

Published as a conference paper at ICLR 2026
Domain Distribution (%)
Task Coverage
Media & Entertainment
Education & Knowledge
Society & Calture
Science & Technology
31
Law & Governance
Market Analysis
Top Rankings
Technical Support
Decision Support
Pollcy Regulltion
Economy Bustness
Health Wellbeing
Literoture Reviow
Competitive Analysia
Pros & Cona
Wide Info Search
Topoc Explosation
Figure 2: Domain distribution and task coverage of LiveResearchBench.
LIVERESEARCHBENCE
In this section, we present LiveResearchBench, our benchmark for evaluating agentic systems on challenging deep research tasks. We begin by outlining the key principles that govern task design (Section B.I). then provide high-level overview of the benchmark itself (Section 3.2). We then describe the data generation pipeline (Section 3.3) and conclude with our verification procedure to ensure benchmark quality (Section 3.4)
3.1 TASK DESIGN PRINCIPLES
To inform the design of our deep research benchmark, we conducted a user survey with participants from diverse backgrounds and occupations, including enterprise professionals, academic researchers and students, and users performing everyday tasks (see Appendix[]) The survey aimed to identify realistic and representative use cases deep research systems across varied contexts. From this study, we distilled four key principles to guide task design:
User-centric: Tasks should reflect the needs of the intended audience. For example, query posed by researchers may require technical jargon and fine fine-grained detail, while such complexity could overwhelm non-expert users.
Tasks should be clearly specified, since the quality of the final output depends heavily on how well user requirements are interpreted.
Unambiguous:
Tare-varying: Task requiring real-time search represent large share of user queries. Such tasks are inherently resistant to data contamination from pretraining corpora, unlike static or time-bounded queries that risk becoming outdated or leaked LLMs evolve.
Multi-faceted and search-intensive intenssveeeee: Tasks should be sufficiently challenging, demanding multi- hop search across diverse web sources and in-depth analysis. This ensures evaluation of deep research systems extends beyond simple fact retrieval.
3.2 BENCHMARK OVERVIEW
LiveResearchBench is designed as a user-centric, unambiguous benchmark for deep research tasks that require broad, real-time information gathering. It consists of 100 expert-curated questions, each paired with detailed checklists, created through a six-stage curation pipeline (Figure 3 and validated via a five-step quality control process (Figure 4.. It spans seven domains-Secie Science & Technology, Economy & Business, Health & Wellbeing, Law & Governance, Society & Culture, Education Knowledge, and Media Entertainment- ten task categories, including market analysis, technical support, decision support, policy and regulation, literature review, competitive analysis, pros-and-come comparison, wide information search, and topic exploration (Figure Example queries is shown in Figure with additional examples in Appendix F
Compared to prior deep research benchmarks. LiveReseeaccBennch tasks are designed to be more unambiguous and realistic (Figure [I]. For example, Deep Research Bench (Bosse al.) (025) includes search-intens intensive queries such as How many IM and GM account closures did chess com report for 2024?, 4? but these are static, time-bounded, and require minimal
4

---

Published as a conference paper at ICLR 2026
User interview Crowd Survey
Domainn
Clarification Questions
Reflined Research Question
Distribution
Domain Task
922
Ql Areyou looking for a globol view of the enterprtee Al services kec, focused
USO 2025 what 1 the moin acoption drivers among entergpiic pris..
SIFE
Asw??
Initial Research Question
What quantiors wourd you puus fore deep research agant?
Hou lange erter tras
you vaw USil) growth natmes for 2024- 20257
Unit Test Checkilit
(1 Does the report provice the size oftire mar kat for 2024 and 2025?
crnowes among entterprive slleeted? Which oustomnrr segmontss..?
C Oces the report focus on the US erar kic?
Figure 3: Six-stage data generation pipeline for LiveResearchBench. We begin with user interviews and surveys to capture realistic research needs, followed by expert-drafted research questions aligned with the identified domains. Clarification questions from frontier LLMs help ensure unambiguous scope, which human experts then refine into finalized queries. Finally, GPT-5 generates checklists that decompose each query into verifiable unit tests, enabling consistent coverage evaluation across systems. The quality of these checklists is further validated by human experts (Figure
reasoning-limitiin their ability to evaluate evolving real-world research. Likewise, DeepResearch Bench (Du al. 2025) often omits crucial elements such as the target audience, content and format requirements, or a clearly defined scope. Such omissions leave queries open to multiple interpretations, even across runs of the same model, undermining evaluation reliability.
In contrast, tasks in LiveResearchBench are explicitly multi-faceted and grounded in user needs. For instance, the query in Figure [] Create a comprehensive report about the evolution of artistic styles Cover major historical periods from Ancient civilization ... Target the report for undergraduate-Ieved as a formal academic report Include analysis of major artists, techniques, materials, and stylistic characteristics for each period and region. requires dynamic, up-to-date information search (with ((date) replaced by the evaluation date), while also specifying the breadth and depth of coverage, the intended audience (undergraduate students), and the desired output format (a formal academic report). This level of clarity ensures consistent interpretation across models and imposes rigorous demands on information retrieval and in depth reasoning.
up to the present ((date)).
students ...uuuuuurre
3.3 BENCHMARK CONSTRUCTION
We curate the queries and checklists in LiveRescarchBench through a six-stage pipeline (Figure 3p. To capture realistic research needs, we start with user interviews and crowd survey involving enterprise professionals, academics, and the general public. Participants were asked: What quest i ions would you ask deep research agent? (see Appendix[]. The collected responses are then used to determine the task and domain distribution. Next, we hire domain experts from enterprise and academia to draft initial research questions aligned with these distributions.
To make the questions unambiguous and consistently interpretable across both models and human evaluators, we then prompt two top-tier deep research models, OpenAl o3 Deep Research and Gemini Deep Research, to generate possible clarification questions. Relying on single model risks bias or omission, so using multiple models provides complementary perspectives. Human experts then review these clarifications, combine them with their domain expertise, and refine each query to establish a clear research scope and ensure a shared understanding of intent.
After the queries are finalized, we use GPT-5 to generate initial checklists. Each checklist decomposes query into unit questions that test specific aspects of the required output. For example, for the query How large is the US enterprise AI services market in 2024-2025 (provide specific market size in USD and growth rates) ? ..... the corresponding checklist items include: 1) Does the report provide the market size for 2024 and 2025? 2) Does the report focus on the US market?... ... (9 checklist items in total). These checklists function unit tests for the coverage dimension, verifying whether generated reports address all essential points in the query. This design enables consistent, reliable evaluation across different agentic systems.
5

---

Published as a conference paper at ICLR 2026
3.4 DATA VERIFICATION
Quality Control
Research Questions
Independent Assesement
06
Quality Control
Checklists
Aspropriate
Detact
5
Approprate
Aspropriate
Approor
Agree
Desearth
Valid
necessary
Doggres
Finall Verification & O Cross Check
Modification
Figure 4: Five-stage data verification pipeline. Research questions and checklists are first independently assessed by expert annotators under detailed guidelines. Two rounds of quality control then ensure agreement across annotators and detect systematic issues. Finally, a separate group of experts conducts verification and cross-checking to resolve conflicts and finalize the dataset.
A rigorous verification process by human experts is essential for benchmark construction. We adopt a five-stage verification pipeline (Figure (4. After collecting the research questions and corresponding checklists, we prepare detailed annotation guidelines for expert annotators (Appendix (]. Following these guidelines, annotators independently assess each query and checklist item. Queries are labeled as either appropriate or not appropriate, while checklist items are labeled as appropriate, nor appropriate, valid but not necessary, or needs modification. Next, a separate group of quality-control experts reviews these annotations and independently decides whether they agree or disagree with the initial assessments. A second quality-control stage then samples annotation results to detect potential systematic issues and to ensure that overall quality deviates by no more than 5% from the true data quality. Finally, a third group of experts performs the last round of verification and cross- choss-ch They resolve conflicts, refine the queries and checklists needed, and finalize the dataset.
4 DEEPEVAL: A COMPREHENSIVE EVALUATION SUITE FOR DEEP RESEARCH
Evaluating long-form, open-ended generation poses unique challenges: responses are too diverse for string matching, real-time queries lack fixed ground truths. and tasks are inherently multi-faceted. To address this, we introduce DeepEval, a comprehensive evaluation suite for research reports. DeepEval assesses both report- and content-leve quality across six complementary dimensions: O Presentation & Organization, @ Factual & Logical Consistency, @ Coverage & Comprehensiveness, 4 Analysis Depth, @ Citation Association, and Citation Accuracy.
Evaluation Protocol. As the metrics differ in granularity, no single evaluation protocol is sufficient. For metrics 0 we consider three options (Gu et al. 2025): (1)
where the judge evaluates the generated report against checklist and provides binary score (0/1) for each item in the checklist. (2) Pointwiss (additive) where the judge aims to find concrete errors in the report according to the criteria. The final score is determined based on the number of errors. (3) Pa Pairwise comparison where the judge compares two reports side by side and selects the preferred one (or declares tie). Based on preliminary experiments, we narrowed down the choice of protocol for each metric, and subsequently conducted a human alignment study on the selected protocols. As a result, we adopt Checklist-bare
for O Presentation & Organization and @ Coverage & Comprehensiveness: Pointwise (additive) for @ Factual & Logical Consistency and @ Citation Association; Pairwiss twise compari for O Analysis Depth. These protocols achieve high agreement with human judgments (see Appendix C).
Remarks: Another possible evaluation protocol is to prompt an LLM judge to assign a single rating within a fixed range (e.g., 0- 0-)),, guided by rubrics that define the meaning of each score. However, our initial study showed that this approach is unsuitable for any of our metrics. Even with SoTA judges (Gemini-2.5 Pro and GPT-5), agreement with human judgments fell below 60%, and the assigned scores exhibited large variability across runs, for example, differences exceeding 50 points when directly rating analysis depth. Upon closer examination of the judge responses, found that the judge models often produced conflated scores with shallow reasoning (e.g., "the report is overall well written because...") "... rather than systematically identifying issues sentence by sentence. In other words, once a model commits to an overall positive assessment, it rarely examines the report in detail, leading to missed errors and inflated scores.
@ Citation Accuracy requires verifying (statement, URL) pairs throughout the report. This task is naturally handled using the rubric tree protocol detailed in Section 4.2

---

Published as a conference paper at ICLR 2026
Seentt--nseembe-------uuuee To mitigate inductive bias from relying single model, conducted a pilot study to identify which top-tier models align best with human preferences. Among the candidates- Memimdt 2.5 Pro, GPT-5, and Claude 4 Sonnet-resulty showed that Gemini 2.5 Pro aligned most closely, followed by GPT-5, while Claude 4 Sonnet exhibited inconsistent and low agreement with human experts (Appendix []. Based on these findings, we adopt multi-judge ensemble protocol for all LLM-as-a-Judge evaluations. Specifically, we use Gemini 2.5 Pro and GPT-5 as independent judges and report final scores the average of their assessments.
4.1
COARSE-GRAINED. REPORT-LEVEL. METRICS
Presentation & Organization. Poorly organized reports or those with frequent grammatical errors undermine user trust. We evaluate presentation quality with checklist of 10 common error patterns from our pilot study (Table 2 covering structure, grammar, citations, duplicates, and formatting. Each item is scored in binary form (0/1), and we report the average success rate across items and an ensemble of judge models as this configuration best aligns with human judgment-achievinn agreement in 98.3% of cases (Appendix (). Examples of checklist items include: (1) Does the report contain any grammar or spelling errors? (2) If figures or tables are included, do they present complete data or valid visual elements? (3) Does every reference entry correspond to least one in-text citation? ? Full checklists and prompts are provided in Appendix A.
Factual& & Logical Consistency. As reports grow longer, they are more likely to contain factual and logical inconsistencies, or even contradictory claims and numbers. This metric captures whether claims remain coherent and factually consistent throughout the document. We adopt the pointwise (additive) evaluation protocol based on the human alignment study, where the LLM judge is tasked with identifying as many substantive inconsistencies as possible. The final score is then computed mechanistically based on the number of issues detected, scaled from 0 to 100: report with no contradictions receives a score of 100, while each additional inconsistency leads to a deduction. Detailed prompts provided in Appendix A.2
4.2 FINE-GRAINED: CONTENT-LEVER METRICS
@ Coverage & Comprehensiveness. A strong report should directly address all aspects of the multi-faceted query. To make evaluation consistent and interpretable, we leverage the human curated and verified checklists introduced in Section 3 Each checklist item functions as a unit test: the judge assigns binary score (0/1) indicating whether the requirement is met, and we report the average success rate across all items and queries. The uudge-model ensemble aligns with human judgment- 100.0% of preferences favor either or both. The evaluation prompt is provided in Appendix A.33
Analysis Depth. Beyond coverage, strong research report should provide substantive insights rather than superficial information gathering. We assess depth along five dimensions: (1) granularity of reasoming (causal chains), (2) multi-layer insights (trade-offs and implications), (3) critical evaluation (critiques and limitations), (4) analytical use of evidence, and (5) insight density. Ensemble judges conduct pairwise comparisons between the evaluated and baseline reports, assigning 1-5 scores to each along every dimension. To mitigate positional bias, we adopt the position- swap averaging method of Wang al. (2023), scoring each pair twice with reversed order and averaging results. Dimension scores are summed, averaged across judges, and used to derive win rates: A win, B win, or Tie (if scores differ by S 1). Ties excluded from the denominator. The human agreement rate is 92.5% with the judge ensemble. Detailed prompts can be seen in Appendix A.4
3 Citation Traccability. This metric checks whether all factual claims are linked to corresponding sources. For example, a statement like "medium-lift launch vehicles held 66.63% of the market in 2024" should include proper source URL. The LLM judge aims to flag claims not properly cited, and compute a score scaled 0-100: full coverage yields 100, with deductions for missing or misplaced citations, similar to the scoring mechanism in @ The human agreement rate 85.9% with the judge ensemble. Detailed prompts can be seen in Appendix A.5
@ Citation Accuracy. Complementary to association, this metric evaluates whether citations genuinely support their claims. In citation-grounded reports, links may be hallucinated, irrelevant to the topic, or fail to substantiate the associated statement. To capture this, we employ an agentic judge with web access that performs structured validation process: (1) identify each claim with its linked source URL, (2) check whether each URL is accessible, (3) if accessible, assess whether the content sufficiently supports the claim. This procedure is organized as rubric tree in Figure| 13
7

---

Published as a conference paper at ICLR 2026
5 MAIN RESULTS AND ANALYSIS
Average report length across models
Report length distribution across tasks
Ocn el 2ccp Resorrsh
Centin Deep Sexearch
Resea
12916
33757
0G Deep Research
Gemie-2.5
1:44
LPC-
10003
Resoeshh
Sener Doep Tesearch
1871
14003
Oycn Door foorrc
Manul
2P52 Searhh
Deerlizu
144
3761
7:0
+14
61K
363
7UM
60M
0F1- Sears
104
107
cFT5 Max Search
04 Crap Froeach
(91- Searth
5967
Hix Deep Seueacch
4rd-4 Doep Research
2044
Search
200
G014,1 Search
Drep
2432
Cook Opes 4.1
Craah Sonver
1M1
IKI
LA
2000
Sonor Rleozring
IN0
Opus
Soner Resoning
Clucke Screet
Destiion Firtoe
Werker desipte
Polccy Reyntades
Becosiial Gugton
Kamsloge
Posi Exetication
Wide Inlo Seancs
Sonar Reaooiing
1791
Scea Seasoring
Singleggent Deep henaarc 000 Mulls-gem Deep Rerearch
Compeiiiun
2000
4000
9032
8500
10050
12000
14000
19032
Figure 5: Distribution of report lengths across systems and tasks.
Agentic Systems. We conduct a comprehensive evaluation of 17 state-of- ofthe-of open-sourced and proprietary agentic systems, which can usually be grouped into three categories: (1) Single-
agent systems with web search cupabilities, including GPT-5 (OpenAl (025a) GPP-4.1 (OpenAl
2024), GPT-5-mini (OpenAl 2025b), Gemini 2.5 Pro (DepMind)) 2025b). Gemini 2.5 Flash
DDepMMnnd 2025a), Claude 4 Sonnet (Anthropic 2025a). Claude 4.1 Opus (Anthropic 2025b). Perplexity Sonar Reasoning (Perplexity] (025a)). and Perplexity Sonar Reasoning Pro (Perplexity
2025b; (2) Single-agent deep research systems, which feature extended reasoning depth and longer thinking time, including OpenAl o3 Deep Research (OpenAl 2025c). OpenAl o4-mini Deep Research (O 2025d), Perplexity Sonar Deep Research (AI (025b), Grok-4 Deep Research (Expert) (025b),, and Gemini Deep Research (DeepMind 2025c); (3) Multi-agent deep research systems, which coordinate team of specialized agents decompose complex queries. These systems typically employ a planner to dispatch sub-agents for browsing, searching, synthesis, and verification, while maintaining a large shared context across tasks. This category includes Manus (Manus 2025). Grok- 4 Heavy Deep Research (xAl 2025a), Open Deep Research (AI 2025a), and Deerflow+. Deerflow+ is our enhanced implementation of Deerflow (ByteDance) (025) that resolves frequent failure modes and early termination by adding inline citations, reference mapping, and robust long-context management.
Implementation details are provided in Appendix D Figure 12 shows side-by-side outputs for the same prompt in Figure T
Remarks: Non-reasoning models and multi-agent systems that do not excel at generating long-form reports are excluded due to poor performance in our pilot study. For each system, we configure settings to maximize performance. For example, setting output length to the maximum token limit (128,00 2,00 for GPT-5 models) and enabling retries for low-quality or failed generations. For open-source systems like Deerflow+ and Open Deep Research, we use GPT-5 as the backbone model.
Obs. I Longer reports common, but not consistently better. Figure 5 the distribution of report lengths across models and tasks. Single-agent deep research systems backed by strong base LLMs (Gemini Deep Research, OpenAl o3 Deep Research) produce substantially longer reports than both multi-agent systems and single-agent systems with web search. Yet, multi-agent pipelines do not automatically yield longer reports. For example, Deerflow+ and Open Deep Research rely on the final synthesizer to produce the report given searched findings, which constrains length. We also observe task-level variation on report length smaller than model-level variation. Length can also be inflated by formatting choices rather than substance. Some systems emit very long, provider-specific redirect URLs in citations (c.g.. "grounding" redirect URLs for Gemini 2.5 Pro), or duplicate links both inline and in the bibliography (OpenAl o3 Deep Research), effectively counting the same URL twice. Since we use shorten links and cite once for multi-agent systems for Deerflow+ and Open Deep Research, they yield fewer URL tokens. Hence, token counts reflect citation handling as much content. As we show later, longer reports do not necessarily translate into higher quality; improvements in citation discipline and factual consistency not monotonically related to length.
8

---

Published as a conference paper at ICLR 2026
Presentation Organization
Fact Logic Consistency
Coverage Comprehensiveness
Citation Association
Agent Name
Single-Agent with Web Search
GPT-5
71.6
68.3
83.4
67.6
GPT-4.1
66.0
65.9
63.6
50.6
GPT-5-mini
66.9
80.5
55.5
Gemini
76.5
38.5
Claude Sonnet
67.3
49.2
37.9
Claude 4.1 Opus
67.5
50.8
35.6
Perplexity Sonar Reasoning
73.0
40.7
52.6
Perplexity Sonar Reasoning Pro
71.9
46.7
50.1
Single-Agent Deep Research
OpenAl o3 Deep Research
71.3
64.2
85.0
25.6
OpenAl o4-mini Deep Research
74.3
62.3
78.6
27.2
Perplexity Sonar Deep Research
83.5
67.4
65.5
36.6
Grok-4 Deep Research
69.1
57.4
86.3
49.5
Gemini Deep Research
62.1
63.0
75.8
52.1
Multi-Agent Deep Research
Manus
75.0
63.1
73.3
45.6
Grok-4 Heavy Deep Research
75.9
59.4
89.3
48.0
Deerflowe GPT-5)
78.8
69.9
61.6
77.0
Open Docp Research (w. GPT-5)
81.0
71.3
65.3
76.9
Table 1: Evaluation of single-agent and multi-agent deep research systems across four dimensions. Scores (0-100) are normalized evaluation outcomes averaged across all tasks and LLM judge ensembles. We bold the best across three categories and underline the best agent for each category.
Obs.e Models struggle most with citation correctness and formatting, rather than surface fluency. Across systems, repeatedly observe failure modes that trivial for humans but challenging for current agents, such as mismatched in-text citations and references (Figure [4]. missing or incomplete URLs (Figure [5], inconsistent citation formats (Figure (6)) uncited references appearing in the bibliography (Figure 17 broken or incomplete table formatting (Figure 18 and out-
of-order reference numbering (Figure These recurring issues motivated the design of DeepEval's
Citation Association dimension (Section and foreshadow the quantitative gaps we analyze next.
Table summarizes performance across four DeepEval dimensions. Below we highlight key insights.
Obs. Multi-agent families lead on average. At the system level, Open Deep Research achieves the highest average (73.6), followed by GPT-5 (72.7) and Deerflow+. Averaged by family, multi-agent deep research systems (69.5) outperform both single-agent web (62.8) and single-agent deep research, indicating that collaborative decomposition and aggregation provide reliable gains.
Obs. O Single web agent excels in consistency, multi-agent systems in association, while single- agent deep research lags. Single-agent web models are strongest on Factual Logical Consistency (69.7 avg). benefiting from single persistent memory stream that maintains reasoning continuity. Gemini 2.5 Pro achieves the highest consistency score overall (76.5), with Perplexity Sonar Reasoning and Perplexity Sonar Reasoning Pro close behind (73.0 and 71.9). For example, single web agents such as Gemini 2.5 Pro maintain coherence because the absence inter-agent handoff's reduces the risk of logical drift. Similarly, Perplexity Sonar Reasoning benefits from producing relatively concise reports, which lowers the probability of factual or logical mistakes while keeping reasoning aligned. Multi-agent systems dominate Citation Association (61.9 avg), where explicit steps for aligning sub-agent outputs with citations or dedicated citation agents yield cleaner evideee links. By contrast, single-agent deep research models perform worst on Citation Association, primarily because many key statements remain uncited and URLs are often mismatched. For example, OpenAl o3 Deep Research often produces long reports but leaves key statements/facts uncited, while Grok-4 Heavy Deep Research often cites URLs that fail support the associated claims. Another common recurring issue the use fictional or non non-existent links, as seen in Manus.
Obs. S Multi-agent systems lead presentation, but surface polish does not imply grounded quality. Multi-agent systems achieve the highest presentation scores (77.7 avg.), with Open Deep Research (81.0) and Deerflow+ (78.8) producing the most polished outputs. Yet, presentation is weakly correlated with association or mossistenc-visualle well-organized reports may still contain inconsistent claims.

---

Published as a conference paper at ICLR 2026
Obs. G Coverage benefits from specialization, but scaling retrieval scope and system complexity strain memory capacity. On average, single agents like GPT-5 series and the Grok family perform strongly on Coverage & Comprchensiveness, with Grok-4 Heavy Deep Research achieving the highest score (... By contrast, Deerflow+ (61.6) and Open Deep Research (65.3) lag despite strong citation association. The bottleneck arises the search context scales: even hundreds of retrieved webpages can quickly exceed the available context window. In multi-agent systems, increasing planning steps and subagents can easily push retrieval into thousands (>3,000), where the problem compounds. Without robust mechanisms to label related content, track what has already been processed, and compress information without discarding key evidence, systems lose critical coverage. Thus, the open challenge is not the synthesizer alone, but the design of system-wide memory architectures and compression strategies that allow agents to retain breadth while maintaining fidelity.
Obs. Most systems are deep searchers, Figure G
not deep researchers. benchmarks analysis depth as win rate over Open Deep Research (with GPT-5 as backbone). Only Deerflow+ and Gemini Deep Research exceed ODR. Models that generate very long reports (e.g., OpenAl o3 Deep Research) often fail to synthesize across sources: Grok-4 Heavy Deep Research, though best on Coverage & Comprehensiveness, barely outperforms ODR in depth. This suggests that many systems behave more like information collectors and organizers than writers of grodence-grounded, argumentative reports functioning deep searchers rather than deep researchers that provide
Dew
BJ.3
36.7
Gemini Decp Research
44.8
GPT
GPPS Min
$4.1
Sonar Decp Research
14.3
85.7
Deep Research
13.3
86.7
Deep Research
Grok Heavy Decp Research
93.9
Grok Deep Resea rch-
95.0
Gemini-2.5 Rlash
96.0
Sonar Reasoning
Cluude 4.1 Opus-
99.0
Cla Sonnet
99.5
Gemin 2.5 Pro
99.5
Siile agent Web Search
Marus-
100.0
Segle-egent Research
GPT-4.1
100.0
Muli-geet
Sonar Reasoning
100.0
Open Deep Research
0%
25%
50%
75%
100%
Win Rate (%)
Figure 6: Analysis depth (win rate over Open Deep Research). Most systems collect and organize information but struggle to synthesize deeper insights. in-depth analysis. We provide further analysis with human judgment in Appendix
Obs. Even So systems are far from citation error-free. Using our rubric-tree framework, we evaluate the top performers from Table (GPT-5, Grok-4 Deep Research, and Open Deep Research) on the two most search-intensive tasks: market analysis and wide information search. The results are shown in Table 7 where for each agentic system and task category, we report the average number of E1, E2, and E3 errors in the generated reports. We can see that all models produce non-trivial citation errors. For example, in wide information search, most errors stem from unsupported claims (claims not verifiable from the cited link) rather than invalid or irrelevant URLs, highlighting that hallucinations persist even with web access. Further details are provided in Appendix E
Obs.9 Coherence, verifiability, breadth, depth trade off under context limits. Single web agent systems favor coherent reasoning; multi-agent systems achieve stronger association and presentation; and specialized designs lead in coverage. Yet no system achieves all four. Sustaining both high coverage and depth will require: (i) long-horizon memory with update operations, (ii) hierarchical modules.
information compression without information loss, and (iii) explicit synthesis/arummentuu Effective compression should address two challenges: (1) when information is partially redundant, the system should merge overlapping content without discarding unique evidence; and (2) when all information is related, compression should instead operate by assigning importance levels to retain critical details within the context limit. Because different users may emphasize different aspects of information, such representations must be both importance- and preference-awen capturing what is essential for reasoning or verification while adapting to user priorities. Without these capabilities, scaling retrieval primarily increases report length rather than reliable, user-aligned insight.
6 CONCLUSION
We introduced LiveRescarchBench. a benchmark of 100 expert-curated, dynamic tasks for deep research, and DeepEval, a comprehensive evaluation suite spanning six dimension for citation- grounded reports. Our evaluation of 17 leading systems shows that while agents can gather and organize information, they struggle with citation reliability and analytical depth. LiveResearchBench and DeepEval establish a rigorous foundation for benchmarking and point toward future advances in memory, compression, and synthesis as key to enabling truly insightful deep research.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENT
The authors would like to thank ICLR anonymous reviewers for their insightful feedback and helpful discussions. This project funded by Salesforce. We are also grateful for the support of the National Science Foundation (NSF) (CCF2106707, CCF2446711), the Defense Advanced Research Projects Agency (DARPA Young Faculty Award), and the Wisconsin Alumni Research Foundation (WARF).
REFERENCES
Sandhini Agarwal, Lama Ahmad, Jason Ai. Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K Arora, Yu Bai, Bowen Baker, Haiming Bao, et al. gpt-oss- 120b & gpt-oss-20b model card. arXiv preprint arXiv:2508. 10025, 2025.
LangChain AI. Open deep research. research 2025a. Open source deep research agent using modular agents and long context.
wttp://wwww.....aaaaaaaaaaaaaa/a aopppen...pppppeeeeeeeeeeeeeeeee deep
Perplexity AL. Perplexity sonar deep research.
https://docs per.lxxxt................ ai/ setttig-started//eeeeee ottpa////e..o.aaa.//ooa/a///aae/ reasoning depth variant.
2025b.
Extended
Salaheddin Alzubi, Creston Brooks, Purva Chiniya, Edoardo Contente, Chiara von Gerlach, Lucas Irwin, Yihan Jiang, Arda Kaz, Windsor Nguyen, Sewoong Oh, et Open deep search: Democratizing search with open-source reasoning agents. arXiv preprint arXiv:2503.20201, 2025.
Anthropic. Claude sonnet 4. httee://www.nattrrss.ccco/////oe 2025a.
Hybrid-reasoning Claude model.
Anthropic. Claude opus 4.1. httpss//www..aaatooocccooo//////
2025b. Upgraded version of Claude Opus
Nikos Bosse, Jon Evans, Robert G Gambee, Daniel Hnyk, Peter Mühlbacher, Lawrence Phillips, Dan Schwarz, Jack Wildman, et al. Deep research bench: Evaluating ai web research agents. arXiv preprint arXiv:2506.00 06287, 2025.
ByteDance. Deerflow: community-driven deep research framework. https://github.cr com/ bytedance/deer- flow, 2025. GitHub repository.
João Coelho, Jingjie Ning, Jingyuan He, Kangrui Mao, Abhijay Paladugu, Pranav Setlur, Jiahe Jin, Jamie Callan, João Magalhães, Bruno Martins, et al. Deeprescarchgym: A free, transparent, and reproducible evaluation sandbox for deep research. arXiv preprint arXiv:2505.19253, 2025.
Google DeepMind. Gemini 2.5 flash. httess//deeeniiuu.oooo////////// flash/ 2025a. Fast performance variant of Gemini 2.5.
Google DeepMind. Gemini 2.5 pro. /ttp::://deeemtie http:///w..ggggg//o///////////// 2025b. High-capacity reasoning variant of Gemini 2.5.
Google DeepMind. Gemini deep research.
Sheeprreeaacch 2025c. Deep reasoning variant of Gemini.
Mingxuan Du, Benfeng Xu, Chiwei Zhu, Xiaorui Wang, and Zhendong Mao. Deepresearch bench: A comprehensive benchmark for deep research agents. arXiv preprint, 2025.
Boyu Gou, Zanming Huang, Yuting Ning, Yu Gu, Michael Lin, Botao Yu, Andrei Kopanev, Weijian Qi. Yiheng Shu, Jiaman Wu, Chan Hee Song. Bernal Jimenez Gutierrez, Yifei Li, Zeyi Liao, Hanane Nour Moussa, TIANSHU ZHANG, Jian Xie, Tianci Xue, Shijie Chen, Boyuan Zheng, Kai Zhang, Zhaowei Cai, Viktor Rozgic, Morteza Ziyadi, Huan Sun, and Yu Su. Mind2web 2: Evaluating agentic search with agente In The Thirty-ninth Annual Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2025. httpss net. www.eeew.ooomiiiimmm / o
7/openreviee www.iii...eeeeeeeeeeemee

---

Published as a conference paper at ICLR 2026
Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Saizhuo Wang, Kun Zhang, Yuanzhuo Wang. Wen Gao, Lionel Ni, and Jian Guo. A survey im-as-a-judge, arXiv preprint arXiv:2411.155500 2025.
Saiang Huang. Philippe Laban, Alexander Fabbri, Prafulla Kumar Choubey, Shafiq Joty, Caiming Xiong, and Chien-Sheng Wu. Embrace divergence for richer insights: A nuuti-docummett summarization benchmark and a case study on summarizing diverse information from news articles. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume I: Long Papers), pp. 570-593, Mexico City, Mexico, June 2024, Association for Computational Linguistics. doi: https::/acclanthote www.ttthhwooo...oo/222222222 http:///ww.oooo2222aaaaaa-a----- www.naaaallnng... 32/
hn2.022....caloonneee URL
Abhinav Java, Ashmit Khandelwal, Sukruta Midigeshi, Aaron Halfaker, Amit Deshpande, Navin Goyal, Ankur Gupta, Nagarajan Natarajan, and Amit Sharma. Characterizing deep research: A benchmark and formal definition. arXiv preprint arXiv:2508. 2at.::5500.0488... 0413 2025.
Sayash Kapoor, Benedikt Stroebl, Zachary S Siegel, Nitya Nadgir, and Arvind Narayanan. AI agents that matter. Transactions on Machine Learning Research, 2025. ISSN 2835-8856 URL
/ictps://pperrett net. www.ee//wwweeeeeeoeuuuuuu forum? / 2MV12
Philippe Laban, Wojciech Kryscinski. Divyansh Agarwal, Alexander Fabbri, Caiming Xiong, Shafiq Joty, and Chien-Sheng Wu. SummEdits: Measuring L.LM ability factual reasoning through the lens of summarization. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 9662-96627779 9676,9676, Singapore, December 2023, Association for Computational Linguistics. doi: 10. 118653/v/20222 emnlp-main.600 URL http:////caaaathlooooooo org/20233 emnlp ma inainn. 6007
Philippe Laban, Alexander R Fabbri, Caiming Xiong, and Chien-Sheng Wu. Summary of haystack: A challenge long-context Ilms and rag systems. arXiv preprint oxtirhhtpss//arrivxx///////27..0 013077 2024.
Minghao Li, Ying Zeng. Zhihao Cheng, Cong Ma, and Kai Jia. Reportbench: Evaluating deep research agents via academic survey tasks. arXiv preprint arXiv:2508. 108008..58044 15804, 2025.
Xinbin Liang, Jinyu Xiang, Zhaoyang Yu, Jiayi Zhang, Sirui Hong, Sheng Fan, and Xiao Tang. Openmanus: An open-source framework for building general ai agents, 2025. URL https
1t////10.111//1111 5281 wewo://eeoooo.oooo 15186401111144
Monica / Manus.
Manus ai: autonomous agent system. https: //ap:///mmaua.. im/] 2025. General- purpose autonomous agent.
Xuan-Phi Nguyen, Shrey Pandit, Revanth Gangi Reddy, Austin Xu, Silvio Savarese, Caiming Xiong, and Shafiq Joty. Sfr-deepresearch: Towards effective reinforcement learning for autonomously reasoning single agents, 2025. URL /
06283
OpenAl. 1.t-4.1. with enhanced performance.
2024. Successor to GPT-4
OpenAl. Deep research system card. Technical report, OpenAI, August 2025. URL https
//
http:///ww.aaccmr/eer/////aaaaaa 2025a. Flagship
model with agentic and reasoning capabilities.
OpenAL. mini. Lower-cost/ lighttwitttttttttghh variant of GPT-5.
2025b. b025b.
OpenAI. Openai o3 deep research.
scttss//platformt httplllfoam..pppaaaaccccooo////o
33-deep-researee 2025c. Single-agent deep research model.
OpenAl. Openai 04-mini deep research. http::/ppplaffooooopa...cccooo www.ap.enac....... htt.://c...ppaaaaccaoooc///ooooa
04-m 34-mini-cae -eee-iideeee--e-eeeeeeeeee-----e
2025d. Lightweight deep research variant.
12

---

Published as a conference paper at ICLR 2026
Liana Patel, Negar Arabzadeh, Harshit Gupta, Ankita Sundar, lon Stoica, Matei Zaharia, and Carlos Guestrin. Deepscholar-bent bench: A live benchmark and automated evaluation for generative research synthesis, 2025. https:///arrr http:////aaaiv...//a//////////// http//0ab/222.... 20033
Perplexity. Perplexity sonar reasoning. reasoning model with web search.
ai/ 2025a. Agentic
Perplexity, Perplexity sonar pro api. https://wwy perp wwwpx/it.aaaab///bp og/ intrroddcingg-b----oorr---
api 2025b. Enhanced API variant Sonar.
Haoran Que, Feiyu Duan, Liqun He, Yutao Mou, Wangchunshu Zhou, Jiaheng Liu, Wenge Rong, Zekun Moore Wang, Jian Yang, Ge Zhang, Junran Peng, Zhaoxiang Zhang. Songyang Zhang, and Kai Chen. Hellobench: Evaluating long text generation capabiliiies of large language models,
2024. URL https: org. abs /209
Haochen Tan, Zhijiang Guo, Zhan Shi, Lu Xu, Zhili Liu, Yunlong Feng. Xiaoguang Li, Yasheng Wang, Lifeng Shang, Qun Liu, and Linqi Song. Proxyga: An alternative framework for evaluating long-form text generation with large language models, 2024. URL abs/2400 15042
Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. arXiv preprint 230.:22305.77966, 2023.
Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. Browsecomp: A simple yet challenging benchmark for browsing agents. arXiv preprint arXiv:2504. /25/6, 2025.
Jerry Wei, Chengrun Yang. Xinying Song, Yifeng Lu, Nathan Hu, Jie Huang, Dustin Tran, Daiyi Peng, Ruibo Liu, Da Huang, Cosmo Du and Quoc V. Le. Long-form factuality large language models, 2024. URL http:::t////.... htt///iivvv.. org/ abs 240.. 1104443.18800 18802
Siwei Wu, Yizhi Li, Xingwei Qu, Rishi Ravikumar, Yucheng Li, Tyler Loakman, Shanghaoran Quan, Xiaoyong Wei, Riza Batista-Navarro, and Chenghua Lin. Longeval: A comprehensive analysis of long-text generation through plan-based paradigm, 2025. URL /ttps://aax..voor///aaa
2502. 19103
XAI. Grok 4 heavy, with parallel reasoning.
aw /ttt:::://.a///ew///rkkk 2025a. Flagship multi-agent model of Grok
XAI. Grok 4 deep research (expert). deep research variant of Grok 4.
2025b. Single-agent expert
Austin Xu, Srijan Bansal, Yifei Ming, Semih Yavuz, and Shafiq Joty. Does context matter? Contextualludgeeencch for evaluating LLM-based judges in contextual settings. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume I:
Long Papers). pp. 9541-9564, Vienna, Austria, July 2025a. Association for Computational Linguistics. ISBN 979- 979-8-89176-251-40 doi: 1/225 lc1-lon URL https:
//aclanthology.com ottp://...oa//aat/t
4707|
Renjun Xu and Jingwen Peng. A comprehensive survey of deep research: Systems, methodologies, and applications. arXiv preprint arXiv:2506.12554 2025.
Tianze Xu, Pengrui Lu, Lyumanshan Ye, Xiangkun Hu, and Pengfei Liu. Researcherbench: Evaluating deep ai research systems on the frontiers of scientific inquiry. arXiv preprint arXiv:2507. 16280 2025b.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint Xiv::5555.0938., 2025.
13

---

Published as a conference paper at ICLR 2026
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022.
Junting Zhou, Wang Li, Yiyan Liao, Nengyuan Zhang. Tingjia Miaoand Zhihui Qi, Yuhan Wu, and Tong Yang. Scholarsearch: Benchmarking scholar searching ability of llss, arXiv preprint arXiv:2506.13784, 2025.
14

---

Published as a conference paper at ICLR 2026
Appendix
A
Evaluation Details
16
A.1
Presentation Organization
16
A.2
Factual & Logiccl Consistency
16
A.3
Coverage & Comprehensiveness
18
A.4
Analysis Depth
19
A.5
Citation Traceability
B
Expanded Related Works
21
B.
Benchmark Comparison
B.2 Evaluation Protocols Comparison
22
Alignment of LLM Judges with Human Experts
23
D Deerflow+
24
Evaluating citation accuracy with rubric tree
33
LiveeesearchBench Demonstration
34
Error Pattern Examples
34
Information the Annotation Team
45
User Survey
45
J
Human Verification Guidelines
45
J.1
Annotation Tasks
45
U.2
Concrete Examples
46
J.3 Remarks and Tips
47
Results Breakdown
47
15

---

Published as a conference paper at ICLR 2026
A EVALUATION DETAILS
A.1 PRESENTATION & ORGANIZATION
Grading prompt and checklist. The prompt for presentation and organization shown Figure 7
and the corresponding checklist questions provided in Table 2 research questions.
The same checklist is used for all
Evaluation Prompt for Presentation & Organization (Checklist-based)
<system syysemmrooe>>
You are an expert evaluator assessing research reports for presentation quality and formatting standards. Your sole task is to determine if the report meets specifie presentation criteria with binary scoring (0 1).
EVALUATION CRITERIA:
-Score 1: The report fully satisfies the presentation criterion with no issues.
Score 0: The report fails to meet the presentation criterion or has any issues that prevent it from passing.
INSTRUCTIONS:
Read the research report carefully assess presentation quality, formatting, and structural elements.
For each presentation criterion, determine the report fully meets standard.
Provide a binary score (0 or 1) for cach criterion.
Provide clear justification for your
referencing specific elements in the report.
IMPORTANT GUIDELINES:
Strict Binary Assessment: score requires complete satisfaction the criterion. Any failure meet the standard results in score of O.
Evaluate only presentation quality, formatting, structure, grammar, citations, and formatting consistency. Do not evaluate content accuracy, research quality, factual correctness. Citation Standards: For citation-related criteria, carefully check that all in-text citations have corresponding reference entries vice versa.
Focus on Presentation:
Grammar and Spelling: For grammar/spellingg criteria, any errors result in a score of O. Formatting Consistency: Check for consistent use of formatting elements like headers, citation styles,
etc.
Respond with JSON object containing your evaluations for each presentation criterion.
</system_roobe role>
<user_ promm_tropp>>
ORIGINAL RESEARCH QUERY: (query)
PRESENTATION CHECKLIST ITEMS TO EVALUATE:
(checklist_seciee
RESEARCH REPORT TO EVALUATE: (report_content) </user_prompt>
Figure 7: Instruction for evaluating presentation and organization quality of research reports. Placeholders such ).ser_prompt),, (query), (checklist_secte and (report_- content) } are filled at runtime.
A.2 FACTUAL & LOGICAL CONSISTENCY
Grading prompts and rubrics. The grading prompt for factual and logical inconsistency is provided in Figure g The scoring rubrics are provided Table The scores for Factual Logical Consistency are inversely correlated with the number of detected inconsistencies. Users can adjust the penalty weight assigned to each error depending on the evaluation need.
Evaluation Prompt for Factual and Logical Consistency (Pointwise Additive)
<system_roke role> You are an expert evaluator assessing research reports based on the Criterion Description: Factual and Logical Consistency.
INSTRUCTIONS:
16

---

Published as a conference paper at ICLR 2026
No.
Checklist Questions for Report Presentation Quality
Does the report present a clear, coherent, and logically ordered structure so that the organization easy follow and directly addresses the research question?
Does the report contain zero grammar and spelling errors?
Does every entry in the reference list correspond least one in-fext citation?
Does every in nextext citation have corresponding entry in the reference list?
5
there exactly one "References" (or "Bibliography" ""Sources") section, and its entries sorted according to a single, consistent scheme?
6
a single, consistent citation style used throughout the entire document?
7
Are all in-text citations placed logically at the end of clause or sentence, without interrupting grammatical flow?
8
If the report includes figures lables, does each contain complete data valid visual element? (If are included, the report automatically passes this test.)
the formatting correct and consistent? For example: (a) If delivered Markdown, are proper heading levels (#, ##, etc.) used instead of plain text for section titles; (b) if Markdown tables are included, is their syntax valid and renderable?
10 If the citations are numbered, missing) and no duplicates (two different sources assigned the same number, or one souree assigned multiple numbers)?
there skipped numbers (c.g.. [23], [25], [26] with [24]
Table 2: Checklist questions for evaluating report Presentation & Organization quality.
Be very careful and detail-oriented. Read the report sentence by sentence and identify concrete issues.
Be critical and thorough in your evaluation: do not overlook prohlems give inflated scores.
Find many substantive issues relevant to the criterion possible.
Issues must be genuine violations that clearly dissatisfy the criterion, not minor nitpicks or subjective
preferences.
Do not include issues that are irrelevant to the criterion.
Criterion Description: Factual and Logical Consistency
Score rubrics:
(Score rubrics table]
Focus on:
Logical inconsistencies
Factual contradictions (facts, claims, numbers, dates, names, etc.)
Remarks:
1. Factual accuracy (e-g., whether the report is free factual errors) separate criterion and should
NOT be considered here.
2. The same source can be used to cite multiple claims. This does NOT constitute contradiction.
OUTPUT FORMAT:
Respond in JSON with the following fields: speci fic_issues
A list specific problems, with exact quotes or locations the text (e.g., "In section X..." " "The claim that '..'' unsupported").
total total_issuee issues: Total number of issues (must match the length of speci facliffc_sssssee sc_issues). score: An integer from 1-10 based the rubric and issues identified.
reasoning: A detailed explanation (2-3 sentences) summarizing your assessment referencing the identified issues.
dsystem_rolcs <user_prompt>
TASK:
(task)
REPORT TO EVALUATE:
(report)
Please evaluate this report the criterion and provide your assessment in ISON format. <luser prompt>
17

---

Published as a conference paper at ICLR 2026
Figure 8: Instruction for evaluating research reports on logical and factual inconsistency. The rubries Placeholders such as (task) and (report) are dynamically filled at table is provided in Table 3 runtime.
Score
Issue Count
Description
100
Perfect No contradictions or inconsistencies detected
90
1-2
Excellent Very minor inconsistencies
80
3-4
Very Good Few minor inconsistencies
70
5-6
Good Some inconsistencies
7-8
Above Average - Several inconsistencies
50
9-10
Average Multiple inconsistencies
40
11-12
Below Average Many inconsistencies
30
13-14
Poor Extensive inconsistencies
15-17
Very Poor Pervasive inconsistencies
18+
Unacceptable Overwhelming inconsistencies
Table 3: Scoring rubric for evaluating inconsistencies.
A.3 COVERAGE COMPREHENSIVVENESS
Grading prompt and checklist. We present the prompt for evaluating Coverage & Comprehensiveness in Figure 9 An example coverage checklist is provided Table Note
that the checklist for every research question is different (Sections 3.3 and 3.4
Evaluation Prompt for Coverage & Comprehensiveness (Checklist-based)
<system_ _roe>>
You are expert evaluator assessing research reports against specific checklist criteria derived from the original research query.
Your task determine if the report fully and completely delivers specific data information requested for each item.
EVALUATION CRITERIA:
SSoore 1: The report fully and completely provides all specific data and information required by the
checklist item.
SScore 0: The report fails to provide the required data or provides an incomplete response.
INSTRUCTIONS:
Read the original research query to understand the exact data being requested.
Read the research report find the data that directly and completely answers the query.
each checklist item, determine if report delivers all required components the answer.
Provide a binary score (0 or 1).
Provide clear justification for your score, referencing the completeness or incompleteness of the provided data.
IMPORTANT GUIDELINES:
checklist item. any part of the requested information for that item is missing, incorrect, or incomplete, the score must be 0. For example, if a checklist item requires name, date, and valid link, the report must provide of 0.
Strict Requirement for Completeness: score of requires 100% fulfillment
of Providing only of the three results in Data vs. Methodology: Your evaluation must distinguish between report that provides answer versus one that describes process to find the answer. methodology, plan, or description of how the data could he found is not substitute for the data itself and must be scored O.
all three receive
No Credit for Placeholders: report that acknowledges checklist item but explicitly states the information is missing, unavailable, that item.
provided (e.g., "Information provided," "Data found")
must receive score
Focus on Delivery: Base your evaluation strictly on the final data delivered in the report. Do not score based on what the report promises, implies, outlines its goal. Respond with JSON object containing your evaluations for each checklist item.
</system_rolf
<user prompp
ORIGINAL RESEARCH QUERY:
(query)
CHECKLIST ITEMS TO EVALUATE:
18

---

Published as a conference paper at ICLR 2026
Cistcklist_an sect ion) ion) RESEARCH REPORT TO EVALUATE:
(zeport_content proer_prompp>
Figure Instruction for evaluating Coverage Comprehensiveness of the generated reports. Placeholders (query), (checklist_secae _ectiiiissseino and [report_content? are filled at runtime.
ID Sample Checklist Questions
Does the report cover major historical periods from Ancient civilizations up to the present (evaluation date)?
2 Does the report focus primarily on visual arts, specifically painting, sculpture, and architecture?
Does the report provide analysis of key regional or national characteristics for European, East Asian (including China and Japan), Islamic, and Indigenous American art?
Is the report structured as a formal academic report, including proper citations and references?
5 Does the report analyze how political, social, religious, and technological factors influenced artistic development?
6 Does the report include analysis of major artists, techniques, materials, and stylistic characteristics for each period and region?
Table 4: Checklist questions for evaluating Coverage & Comprehensiveness, corresponding to the example in Figure
A.4 ANALYSIS DEPTH
Grading prompt. The prompt for Analysis Depth shown in Figure 10 We use pairwise comparison, detailed in Section 4.2
Evalaution Prompt for Analysis Depth (Pairwise)
<system_ rolerrrle>>___>eee
You an impartial research report evaluation expert. Your task to compare Report A and Report B side-by-side and decide which demonstrates greater depth of analysis.
What "Depth Analysis" Means
Depth = how far the report goes beyond surface description into reasoning, insight, and layered analysis. Scoring Dimensions (0-5 each)
Assign each report
these five dimensions:
1. Granularity of Reasoning (0-5)
0 = purely abstract or vague
3 some unpacking into mcchanisms or details
5 consistently breaks down ideas into specific causal chains subcomponents
2. Multi-Layered Insight (0-5)
only surface-level restatement
3 includes some implications or second-order effects
5 consistently explores trade-off's deeper implications, "so what" insights
3. Critical Evaluation (0-5)
passive description only
some questioning assumptions limited contrast alternatives
5 strong critique, weighing of scenarios, prohing of limitations
4. Analytical Use of Evidence (0-5)
() = mentions facts/examples without connecting them to reasoning
some evidence linked to argument
5 evidence consistently advances or sharpens analysis
19

---

Published as a conference paper at ICLR 2026
5. Insight Density (0-5)
-0 mostly filler or generic phrasing
3 mix of insight and filler
5 highly concentrated with substantive analysis per token
Total Depth Score
Sum the five criteria - total score (0-25).
Judging Rules
1. Compare both reports only depth, using the rubric above.
Give cach report 0-25 depth score.
Decide the outcome:
one report's i score more than point higher - it wins.
If the difference is 1 point - call it tie.
Exclude Entirely
Coverage (breadth of topies).
Factual correctness or consistency.
Presentation, formatting, style, grammar.
Citation traceability.
LLength alone (verbosity 전 depth).
Output Format (JSON only)
"winner": "A
tie",
"scores": {
"A": {
"gramularity"; 0-5,
"insight": 0-5,
"critique": 0-5,
"evidence"; 0-5,
"density"; 0-5,
"total": 0-25 },
"B": { "granularity"; 0-5,
"insight": 0-5,
"critique": 0-5,
"evidence"; 0-5,
"density"; 0-5,
"total": 0-25)
"justification" "less than 80 words explaining which report shows deeper reasoning and why, referencing specific differences
depth.",
"major "major_flaks": flaws":
"A": I"notes on shallow reasoning or lack of insight if any" "B": ("notes shallow reasoning lack of insight if any"]
</system_rols role>>
cuser_promppe
Please compare these two reports ONLY terms depth of analysis relative the research problem and determine which demonstrates greater analytical depth. : query)
REPORT A:
REPORT [otoort___coottt
dreport_b_conted cluser_prmmpt
Figure 10: Instruction for evaluating analysis depth of the generated reports. Placcholders (query), ) (report_a (_tott_t__cntnt)....._ and ceeport__ccnntee are filled runtime.
20

---

Published as a conference paper at ICLR 2026
A.5 CITATION TRACEABILITY
Grading prompts and rubries. We provide the grading prompt for citation traceability in Figurel The scoring rubrics provided in Table As the scores determined by the number of claims properly cited, users can adjust the penalty weight assigned to each error depending on the use case.
Evaluation Prompt for Citation Traceability (Pointwise Additive)
<system byystmm_roe>>>>>
You are an expert evaluator assessing research reports based on the Criterion Description: Citation Traceability.
INSTRUCTIONS
Be very careful and detail-oriented. Read report sentence by sentence identify concrete issues.
Be critical and thorough in your evaluation: do not overlook problems or give inflated scores.
Find as many substantive issues relevant to the criterion as possible. Issues must be genuine violations that clearly dissatisfy the criterion, not minor nitpicks subjective preferences.
Do not include issucs that are irrelevant to the criterion.
Criterion Description: Citation Traceability
Score rubrics:
[Score rubrics table]
Focus on:
Proper association of claims with citations. Each factual claim that requires evidence should be accompanicd by corresponding citation. Flag cases where claims lack citations or where citations are clearly mismatched. For example, a healthcare URL attached to a statement on market share such as "medium-lift launch vehicles held 56.63% of the market in 2024".
Remarks:
1. A URL may not always be attached immediately after a claim; some cases, appears at the end a paragraph and is intended to support the preceding claims within that paragraph.
2. Factual accuracy (e.g.. whether the report is free of factual errors) is separate criterion and should
NOT be considered here.
OUTPUT FORMAT:
Respond in JSON with the following fields:
specii fic ssuss: A list of specific problems, with exact quotes or locations in the text (c.g.." "In
section X..." "The claim that '...' is unsupported").
ssues: Total number of issues (must match the length of speci fic_i ssues).
score: An integer from 1-10 hased
rubric issues identified.
reasoning: A detailed explanation (2-3 sentences) summarizing your assessment and referencing the identified issues. </system role>
cuser_prompt>
TASK:
(task)
REPORT TO EVALUATE:
[report]
Please evaluate this report the criterion and provide your assessment JSON format. cjuser_prompt>
Figure 11: Instruction for evaluating research reports on citation traceability. The rubrics table is provided in Table Placeholders such as taakk and (report) } dynamically filled at runtime.
B EXPANDED RELATED WORKS
This section provides an expanded comparison of LiveResearchBench with representative benchmarks and evaluation protocols for long-form open-ended reports on information seeking and deep research.
B.1
BENCHMARK COMPARISON
Table summarizes the key dimensions, including human-verified rubrics/answers, expert-curation, open-ended task design, domain coverage, dynamism, and evaluation methodology.
21

---

Published as a conference paper at ICLR 2026
Score
Uncited Claims
Description
100
0
Perfect All major claims/facts have citations; fully traceable
1-2
Excellent Very few claims lack citations; minimal impact
80
3-4
Good Few claims lack citations; minor omissions
70
5-6
OK Some claims lack citations
7-8
Above Average Several claims lack citations
50
9-10
Average Many claims lack citations; significant association issues
40
11-12
Below Average Most claims lack citations
30
13-14
Poor
Extensive uncited claims; report poorly supported
20
15-17
Very Poor Overwhelming lack of citations
10
18+
Unacceptable Report largely untraccable
Table 5: Scoring rubric for evaluating citation traceability.
Human- verified
Expert- curated
Open- ended
Multi- domain
Dynamic Queries
Agent-ensemble as-a-judge
Benchmark
ScholurSearch (hou al.
2025
X
X
BrowseComp Wei ct al 2025
X
X
/
X
RescarcherBench
Xu al.
2025b
V
/
X
X
X
DeepScholar-Be
PPaeel et
2025
J
X
ReportBench [Li ct al. 2025
X
K
X
V
V
X
DeepResearch Bench
2025
/
X
Mind2Web2 Gou et al. 2025
V
X
V
X
X
LiveDRBench Java 2025
X
V
Deep Research Bench [Bosse et al
2025
V
X
DeepResearchGym (Coelho al.
2025
X
X
/
X
LiveResearchBench (Ours)
Table 6: Comparison of deep research benchmarks across key dimensions.
As shown in Table 6 no existing benchmark satisfies all criteria simultaneously. In contrast, LiveResearchBench combines human-verified rubrics, expert-curated tasks, open-ended multi- domain queries, tire-narying scenarios, and an aaattt--enemmbel------eeeee ensure consistency and reliability.
B.2 EVALUATION PROTOCOLS COMPARISON
We analyze how prior benchmarks evaluate long-form open-ended research outputs and how LiveResearchBench addresses their limitations.
DeepResearch Bench (Du et al. 2025) evaluates model-generater research reports using a single to assign holistic scores across four dimensions: comprehensiveness, and readability. These are then aggregated using LLM-predicted weights.
LLM call depth, cnstruction-followiee However, our pilot study indicates that angle-judge protocols exhibit high variance. Independent judging runs using frontier models (GPT-5, Gemini-2.5-Pro. Claude-4-Opus. Claude-4-Sonnet frequently differ by over 50 points in analysis depth, severely limiting reproducibility. To address this, DeepEval employs Agent-ensemmlee-s----juug combined with checklist-based, additive, pairwise, and rubric-tree evaluation protocols. We leverage LLMs with web search APIs enabled, allowing agents retrieve external information during reasoning. These methods substantially reduce variance and provide finer-grained diagnostics across dimensions.
ResearcherBende (Xu et al.) 20256) evaluates long-form outputs using a single LLM judge to score coverage, faithfulness, and groundedness, which roughly correspond to our metrics of Coverage & Comprehensiveness, Citation Traceability, and Citation Accuracy. While valuable, this setup omits several key aspects necessary for evaluating open-ended research reports. However, DeepEval expands the evaluation space through three additional dimensions: 1) Presentation Organization. Assesses report's i structural clarity, stylistic coherence, formatting hygiene, and citation organization. Poor presentation increases cognitive load and can obscure otherwise valid reasoning, directly affecting interpretability and traceability. 2) Analysis Depth. Evaluates model's ability to perform nontrivial
22

---

Published as a conference paper at ICLR 2026
analytical reasoning beyond surface-level retrieval or summarization. This guards against superficial optimization strategies that produce shallow but well-cited text. 3) Factual & Logical Consistency. Measures global coherence across the entire aerorrtcapturngg contradictions, reasoning gaps, and logical discontinuities that local checks may miss. These additional dimensions help prevent gaming behaviors, e.g., producing concise summaries that appear coherent and well-cited but lack substantive analytical reasoning. We further observed that when presentation is not evaluated separately, LLM judges tend to conflate stylistic issues with factual or logical errors. Treating presentation as its own axis improves clarity, granularity. and diagnostic usefulness.
(Patel al., 2025) focuses on generating related-wolt sections for academic papers, a task that is narrower in scope compared to our domain-diverse open-ended research reports. Its evaluation relies on a single LLM judge (GPT-4.1 or GPT-4o) and lacks human- verified rubrics. LiveResearchBench introduces several improvements: 1) Organization reliability: DeepScholla-Bel Bencch reports a 78% buman-LLM agreement, whereas our checklist-base Presentation & Organization metric achieves 98.3% agreement. 2) Coverage quality: Their non-ven verified rubric yields 72% agreement, while our human- suman-validated Coverage & Comprehensiveness rubric achieves 100.0% human agreement for Gemini or GPT-5 judgement. 3) Evaluation scope: DeepScholar-Bente Benchh does not assess Analysis Depth, Factual & Logical Consistency, or Citation Traceability, all of which central components of our six-dimensional DeepEval framework.
DeepScholar-Benan
C ALIGNMENT OF LLM JUDGES WITH HUMAN EXPERTS
As deep research reports often long-form and open-ended, we employ LLM judges evaluate different aspects of the reports. To ensure the reliability, it is crucial that the judgments of LLMs align closely with those of humans. Recall that we use the following evaluation protocols: for O Presentation & Organization and @ Coverage & Comprehensiveness: Pointwise (additive) for @ Factual & Logical Consistency and @ Citation Association; Pairwise comparison for + Analysis Depth. In the following, we discuss in more detail the procedure and results of our human alignment experiments on the chosen protocol for each metric.
Presentation & Organization. We observe 67.8% agreement across the three LLM judges (Gemini 2.5 Pro. GPT-5, and Claude 4 Sonnet). In particular, Gemini 2.5 Pro and GPT-5 exhibit strong pairwise agreement (83.6%). To measure alignment with human judgment, we randomly sample 300 questions. For each question, a human expert is presented with the (Query, Report) pair and independently make binary decisions against the checklist without seeing LLM judge outputs. Subsequently, the human expert reviews the responses and rationales from the three LLM judges and asked to indicate their preference, choosing one judge, two jointly, declaring tie, or none preferred. The human agreement rate for Gemini 2.5 Pro or GPT-5 is 98.3 of cases (Gemini 2.5 Pro alone 95.0%, GPT-5 alone 85.0%). Based on these results, we adopt an ensemble of Gemini 2.5 Pro and GPT-5 as our final judges for Presentation & Organization where each question is independently scored by both judge models and we report the average results.
Coverage & Comprehensiveness. The overall agreement rate across the three LLM judges (Gemini 2.5 Pro, GPT-5, and Claude 4 Sonnet) reaches 81.5%, with particularly strong consistency between Gemini 2.5 Pro and GPT-5 (89.4%). To further assess aligmment with human experts, we select 72 checklist questions and follow the same evaluation procedure used for Presentation & Organization. The human evaluator first provides independent judgments without access to model outputs, then reviews the L.M judges' decisions and rationales, and finally indicates which model's assessment best aligns with their own or none preferred. Across all 72 questions, the final human- aligned judgment corresponds to either Gemini 2.5 Pro GPT-5 in 100.0% of cases, with no instance uniquely favoring Claude 4 Sonnet. Based on these results, we adopt an ensemble of Gemini 2.5 Pro and GPT-5 for Coverage & Comprehensiveness evaluation, where each question is independently scored by both judge models and the final result is averaged.
Analysis Depth. 78.8% of the judgments are consistent across the candidate judges (Gemini 2.5 Pro. GPT-5, and Claude 4 Sonnet). In particular, Gemini 2.5 Pro and GPT-5 agree in 89.6% of cases. To evaluate alignment with human judgment, we randomly select 40 samples (agreement set) where all judges agree and 40 samples (disagreement set) where they disagree. A human expert is presented with triplets (Query, Model A output, Model B output) and asked to make independent judgments (A wins, wins, or a tie) without seeing the LLM judges' decisions. For the disagreement set, the human expert also review the decision and the reasoning of each judge. Next, human experts indicate
23

---

Published as a conference paper at ICLR 2026
their preference among the judge candidates, choosing either single judge, joint preference for two judges (c.g.. Gemini 2.5 Pro and GPT-5), a tie, or none preferred. On the agreement set, humans agree with the LLM judges 95% of the time. In the disagreement set, we have similar observations as in other metrics, where humans prefer Gemini 2.5 Pro or GPT-5 in 92.5% of cases.
Consistency and Citation Association. Our preliminary analysis shows that the pointwise (additive) scheme for evaluating factual and logical inconsistency as well as citation traceability achieves substantially higher human agreement rates than direct scoring (with GPT-5 and Gemini-2.5 Pro ensemble as the judges). Specifically, for consistency, the agreement rate with human judgments 82% over 128 inconsistencies identified by GPT-5 and Gemini-2.5 Pro. For citation traceability, the agreement rate reaches 85.9% over 128 issues identified by these judges. Note that our default weighting scheme decreases the score for every two issues detected. Therefore, the actual alignment of final scores with human judgment would be even higher.
Citation Accuracy. For citation accuracy, we randomly sample 200 (claim, URL, judge decision) pairs and conduct human agreement study. The results encouraging, where human evaluators agree with either Gemini or GPT-5 on 87.1 % of (claim, URL) support judgments.
D +EERFLOW+
To broaden our multi-agent comparison and study cross-system differences, we included DEERFLOW, However, in our setting, the vanilla system was brittle on long, search-heavy queries and frequently terminated with the error max ttttttttall... tokens (xxx) must be less than or equal to max max_seq______e ses_see____e__e___e__eee______ee len (xxx). Inspection indicated unchecked context growth across agents, retrieval traces, and retries. We also observed that the absence of inline-citation control made claims harder anchor to sources and increased hallucination risk for the same set of retrieved pages. Therefore, adding inline citations both simplifies verification and reduces hallucinations, improving overall report quality.
Stabilization. We therefore introduce DEERFLOW+, enhanced variant that adds a lightweight context manager and inline-citation support. For stabilization, each agent is equipped with context manager that (i) estimates tokens and enforces total context budget before every model call, (ii) reserves response capacity via an input/output split, (iii) compresses histories and large research payloads with progressive, recency-preserving truncation, and (iv) supports per-agent budgets so retrieval-heavy roles compress more aggressively than synthesis roles. The mechanism is model- agnostic and transparent to the agent logic.
Citation Control. We added mnline-citation support (the original system only emitted a terminal bibliography) and a validation pass that enforces one-to-one mapping between in-text citations and reference entries while checking numbering continuity and duplicates.
Results. With these changes, DEERFLOW+ completed the full evaluation suite without token- limit failures and produced higher-quality reports: better retention of retrieved evidence, improved formatting and factual consistency, and more reliable performance on presentation checks tied to citation management, particularly P4 (Citation Completeness) and P9 (Format Consistency) in Figure 22 Overall, these fixes substantially reduce early termination and bring DEERFLOW on par with other multi-agent systems in our experiments. F2 example outputs for the same DEERFLOW exhibits higher information retention, stronger evidence structuring
prompt in Figure [ with inline citations, and no loken-limit failures.
(a) Deerflow (vanilla)
Evolution of Artistic Styles Across Historierl Periods and National Characteriscies
# Key Points
Artistic styles have evelved significantly across historical periods, influenced by politica", social, religious, and technologinal factors.
Ancient laid foundstionsl conventions, with sgyptian, Greek, and Reman
amphasizing religious, realistic, and grandione themes, respectively. Renaissance marked pivotal
towarda realiam and humantam, with notaole contributions from artists like Leonardo da Vinei and Michelangelo. end contemperary art movenents, such Impressicnism, Cubism, Abstract Expressionism, reflect technological advancements and global perspectives.
24

---

Published as a conference paper at ICLR 2026
Regional characteristics in art, such as European, East Asian, Islamic, and Indigenous American ast, highlight
influences stylistie approschea.
The evolution of actistic styles is complex Lapeatry woven from the threads historical cultural, and technelogical influences. Prom the monumentsl structures of ancient kgypt to the digital instellatiens of contemporary art, each period and region has contributed uniquely to development visual acts. progression of artistic styles across different historicel periods and national characteriscies, examining the impact of political, social, religious, technologioal factors art, By analyzing major artista, tectniques, materials, and sty-istic characteristics, this report aime to provide comprehensive understanding of artistic expression.
report explores
of the dynemic
Betal led Analysis
## Ancfent Art.
Pesiod/Region Charscteristics Major Actists Teciniquess/hterziils
Egyptian Art Religious themes, colassal statues, pyramida Information not provided Hieroglyphics, stone, gold
Sreek Art Realism, harmony, lifalike sculptures Phidica, Fraxiteles bronze, pottery
| Marble,
Roman Art Grandeur, funetionality, architecture Information provided Concrete, frescoes, mesaies
## Medieval Art.
Period/Ragion Characteristios Majar Artists
Taciniquuss/attrralll
Bysentine Art
Religious imagesy, mosaics Information not provided Gold leaf, glass
tesserae
Romanasque Robust architecture, sculpture Information not provided Stone, wood Verticality, Light, stained glaaa Information not provided Stone,
Gothic Art stained glass
##4 Renaissance Art
Period/Region Charocteristics Major Artists 'Ecchiiques/attariall
Eazly Renalasance Pezapective, proportion Giotto, Masaccio Fresco, tempeca Idealized beauty, sechnical mastery
High Renaissance
Vinsi, Kichelangelo
011 paint, marole
Northern Rensissance Detail, symboliem Jen van Byck, Albrecht Duzer Oil paint, woodcut
Modern Contemporary
dovemeat/Region Characteristics Bajor Artists
techniquus//aateriaal
Impressicnism Light, coler, everyday subjects Claude Kenet, Edgar Degas 0il paint
canvas Subism
Fragmented forms, multiple perspect ivve Pablo Picasse, Georges Braqua
paint, collace
Contemporary Aet Diverse mediums, globel perspectives Infermation not provided Digital media, installations
#ff Regional Characteristics
### European Art
+>Chstacteristiesw
styles, realism, perspective.
+*Majo Artistses: Leonardo da Vinci, Michelengelo, Xonet, Van Gogh.
w.Techaiiquo//ttteeslll Frescoss, oil paintings, installations.
#ftt East Asian Art (China, Japan)
C-Chaaatterlatlon cigaphy,,hy, harmony, minimalism.
v* jor Actistsva; Xu Bing, flokusai.
w=fechaique//kateeiillee Ink, paper, ceranics.
### Islamic
Chharacteristioose Geometric patterna, calligraphy,
Majjr Artists*:: Collective craftamanchip.
Tcchhitque//attrrialsrr
textiles, metalwork.
Indigenous American
Characterinticssvr Symbolic motifs, storyselling.
Majar ArtistsuAs Jaune cuick-to-See Smith, T.S. Cannon.
25

---

Published as a conference paper at ICLR 2026
+ Wood, stone, naturel fibers.
*
Survey Note
## Litersture Review Theoretical Franework
The lilerature on the evolution artistic styles highlights the Interplay between cultural, politicsl, and technologiesl influences. Theoretical frameworks such cultural mstericlism and semiotics provide insights into how art reflacts and shapes values.
#ff Methodclogy Data Analysia
Thia report synthesizes firdings from historical analyses, art eriticiss, and cultural studies to present comprehensive overview of actistic evolution. Data on ma jor artists, techniques, and materials systematically categorized by period and region.
#+ Critical Discussion
The findings underseore dynamte rature art as reflection human expert lance While political and religioua influences have historically diotated arcistic themes, technological advancements have expanded the possibilities for artistic expression.
##t Future Research Directions
explore the impact globalization on contemporaty of cigital tachnologiea in shaping future artistic movement...
the role
# Bey Citations
(The Evolution of Art styles: From Classical Contemperary Madium; peestts://mediiie /educart on/the-evo (ath---ootttoo------rrr---- oo--o-arr--ty sarooylll-roooolaaeaee -as-icl-----ooott -tto--cottttpprrr----9/6e---ee
(Art Bistory 101: : Regirner's Guide to Key Movementa and Artists] thttps://
se:enadenaggelleeerrrc bistotttottttttstss.oy sissoor--11--
aen -egianers--uuiee todeeett-keyy movemmntt aadala--nd--rtad artiats/)
[The Evolution of Artistic Nevements: From classical to Modern Art] (https:// foo:::mmpptte..o...a/ttoeeeeele /)
com.an/the-evolut wttp:lloooooooo--attaattoeeomm wrw.rrtteoecooeeeeeeeeeeooo---e -tt-mmor------------a-------
[Art of Purope]
[Best Asian arts]
(Tslanio Art] (https://pattoen
[Netive American act] (https:// www.britannica.co..t /Nativv Amee-cen-eeeeeeeeeeeeeee
[Politics of Visual Acts in Changing World Globel Taought] (#stpe://cgt..oouurl http:poojeet//pottttov-sss------
Politica Influence and Each Other LEVES] (nttps://dltteerenttleee) /how art and politics influeace and shape each othec/)
(b) Deerflow+ (ours)
Evolution of Viausl Arta Acrosa Regions: Structured, Sourge-Rased Dosster with Analytical
Normalized Data
Key Foints
Croas-regions: trade and imporial politics (silk Road, Indion and Vanetion-Islamia networks) mediated the movement of artisana, motits, and technologies, ahaping ceramics, textiles, glass, and architectural vocebulacies
Burasia between
th and 17th centuries [17] [18] 120 [21: [22] [25].
Techniesl innevatiens
underglaze blue China; stonepaste fritwarel, luster painting, and augarnas in the Islamis world; silk-weave structures and codified notifa in Utteman workshops; polychrome woodblock printing in Jepan; and Industrial and revivalaat
meterialss-porcelaain
processea -thh-eenuuy Furope-proved decisiva in [24] [26] [28].
generating new stylistic systems and macketa
and patronage esclogies (imperial academies, pelace nakkashane, urban publishers, and commercia: galleries) reorganized visual production, moving court idicms into broader consumption and global circuits (e.g., Ottomon ailks, ukiyo-e),
influencing European moderniams
Japonisme [25] (2]]
Dchatas over figuration, aniconiam, and iconoclasm in Rysontine-ialams transitions were latec locally
secular apheres, rather
uni across regions,
bane, underscoring the interdependence of 1aw, piety, and patronage [23].
normalized dataset natrices align timelines and
crass-culeural erazamiacions (e.g., Iranien cobalt to chinase parcelain; thinese blue
26

---

Published as a conference paper at ICLR 2026
-and-waite models reinterprezed in iranian stonepaste; Venetian adoption islamic glass Lechniques; Japonisme), and flag gaps inotably Indigenous Americas paris Ancient and Contemporary coveragel for targeted enrichment [13] [24] [26] [27] [28].
# Overview
this report presents a strectured, evidence-hased account of stylistic avolution in architec.ure acress key regions European traditioas, Asia (China and uspan), the islanic world, and ladigenzus America-emmhesiziin -eerrss--mmhsszziiggg how politica' economy, social institurions, religious idnologies, innovations shaped artistic deve The dossier integrates methodologlca Iramewocke (iconology, social art history, feminist end postcoloniel critiques, museum /coilecting studies) with normalized data tables, comparative matrices, and cultural tranamission pathways darived from suthoritative scholarly rasources (notably Metropol Museum of Art's Heilbrunn Timeline, specializes easays, collection entries).
painting, scuiptuze,
technological
The evidence base is Lemporally developments in Bast Asid, the Islemic world, and Burope, with connected Sub Saharan African contexts; however, is lacks substantive entries for indigenous American and provides troompleta covarage for Ancient ard Cortemporary endpoints. All claima constrained the provided sourcea; mlasing Information not providad.
The dataset robustly decumenta 9th-19th 9th-19th-century
explicitly indicated
# Uetailed Anslysis
#84 Methodological Historiographical Prameworks
This study forogrounds interpretivs approzches central art-hlatorica" ana ysis:
Iconology and humenistic themes (Penofeky) guide the decoding of symbolic programs and their historical meanings, particularly in suropean traditions [1].
Social formation of pictorial style (Baxandall) situates soc1oeconomic conditions, pacronage, and pract relevant to Italian Renatasance comparable contexts of court Social art hiatory and modernity (T. J. Clark) elucidate the urean public aphere's role in shaping modera visualily, bridging 19th century Buropean printa and Japonisme (3]. Peminist critique (Mochlin) problematizes institutions of artistic training
"periad eys! within
production
consuptiion [2].
recagnition, lena applicable to reading exclusionary structures within canonical. narratives
Postcolonial critique (Said) interrogates the diacursive construction of the "Orient"" Kestern knoxledge systems, framing historiogrephical issues Islamicate Asian receptions [3].
Museums, collecting, and ethnography (Clifford) analyze the politics of display and knowledge production, crucial to understanding the recsption of non-Western
(6). Slobal art history and African indi knowladge frameworks expand beyond center-
pertphery models, underacoring multi-nodal exchanges [7] The critique of "ppinitiviee narratives modern art
exhibitions (McDvilley) signals the stakes of curatorial [10].
Images (Eramework references)
[Panoftky, Studies in iconology (cover) (tttps://pictuure..abbbbb
(30951609232.1p2) (pg) [1]
[Baxandall, Painting
Experience (cover) (htips:/ -amazaa wwt.c//////ag/////aaaaaagg SY
[2]
[Nochlin, why There seen
homen
(cover)
(http:///hamsssnddhiioo
-965c. www.n. http:///aaacoaaaareooo/////////a
/3/7/9790500023866 0arr0000000000.tyyaaaeeeeeeeeeee [Said, Crientalisn (caver) (tttps://pictursr abchooks. htwpehoos.co//aa/////199 /ttp://abn/970000005555550000
gre-ggroeeeeeteteee wwwen-ortisteeee wwwttssssseeeeefeeeeeeeee wwweer-fat..... fpgt [4]
frameworks inferm, they are cited here as historiographic context.
extend, beyoad provided empirical sources;
#8# Timeline Regional
(Normal ixed)
The rollowing master timeline aggregates peclods, regions, and computed apans strictly from provided dataset, aligning interregional exchanges technologicel developments [16] [18] [26] (2221122000 [21] (22) [25] (27] (27) [23].
period_id reglon period_name apan_text start_year end_year
TRADE, MONGOT, Talamio
Mongolica exchanges 13th-14th 1200
TRADE_ MONGOL MONGOL_PPA__OOO PAX_CHINA 1200
East Asia
China Fax Kongolica exchanges | 13th 14th e.
TRADE_ MONCOL_ EU 1200
Europe
Pax Kongolica exchanges (Venation links) 13th-14th
27

---

502-1400, advances printing (o.g., the Diamond 868), porcelain chemlatry (Xing, Ding), and underglaze cobalt decoracion tby Yuan ca. literati [18]- literati painting emphaatzed brushwerk, monochrome ink, and poetic inseription, while academies cultivated refined mineral colors and precise shaped further under foreign dynasties, including Published as a conference paper at ICLR 2026
TRADE SOADE_VENICE VENICE_ASLAG | Surope Venice and the islamic world | 13th-16th c.
1200 1600
IRADE WWW.VIIIIIIILLLLLC ISLAABB_SSLALLL
Norld Venice
world 16th
1200 1600
TRADE_ INDIAR IN
World
Desan
circuits 9th-16th e.
803 1600
TRADE_ TRADE_INEIANS CEEIINNNCEEAANN S88
SJb Saharan ACrica Indian Ocean a Red Sea circuits (Bast
AEzicen coasts) 1tth 800 1600
islamic World kgypt: Mamluk-ottoman transition 1400-1600
1400
IKANS SaAMS__AARAA_LCE
Scb-Saharan Africa Trans-Saharan networks and Guinez Coast 15
th-16th c. I
#CSUDAN_TSMAL Sub-Saharan Africa Western and Central Sudan Maana-MMll-Songile
1000 1450 1000 1400
SHINA_50O_1000
Last Asis China China
and innevations 500-1000 500
1050
CHTNA_ 1000_1400
Asta Chira China innovations (Song-Yuan) 1000-1400
1400
TSTAMTC__ TST.AMC_STTNNPIAA STONFPASTE Tslamic World Stonepaste caramic bod (8yria/Tran) 12th-13th
1100 1300
OTTOMAN_SILE Islamic World Octoman silk weaving (Burse, nakkashane codificationi
15th-17th
1400
1700
JAPAN CKTYOR
______KYT___PYYYY POLY
East Asia Japan Ukiyc-e polychrome prints emarge e. 1765
1765
BURCFE_198 1B____99___RR_____ PRINT Europe Print in the ninetcenth century 19th century 1800 1900
Coverage gaps: Indigenous Americas have no perieds pepulated in the provided detaset Information providad).
484 Regional Analyses
##f# 3.1 East Asia China
Contexts and innovationa:
1350) energed
systems production
draftsmenchip; these assthetics
the Mongol Yuan [18].
Representat
technology likaage:
Diamond Sutra (868): woodblock printed text exemplitying early maas reproduction
technologies :17].
Desamic technclogy and transmission: Iranian cobalt supplied Chinese underglaze blue; Chinese blue and white models were subsequently zeintespreted in Isanian stonepaste (16. [13].
Selected
work_id
ds_descriptien region date_text year
WORK DIAKOND SUTRA Diamond Sutra (world's oldest dated printed book) East Asis China 868 868
[Incerpretive
interplay between court/academy and literati milieus appears structure stylistic preferences, while frencier dynastic consexts mediate material exchange notif adaptation;
limited to sited essays.) (17][11))
#itt 3.2 East Asia Japar
Udiyo-e technclogy and markats: Full-soler polychreme weodblock printa iniahiki-e) emerged ca. 1765 vaing separate blocks for each color and kento registration for precise alignnent, produced shrough collaboration among designer,
carver, printer, and publisher [26]. Subjects enccapassed courtesans, kabuki actors, current events;
landscapes,
imprassions la"ge ed tiona, ref rectecting urban cultura). conauaption [26][2?]
(mulberry) paper anabled
Artists and genres: Suzuki Harunabu pioneered full-color prints; Hokunai and Hiroshige
expanded landacape genres [25] [27].
Images (sechnique
subject)
![kiyyo- temple view bereath
lantern] (http:/ //wps////www////////////e// http:///ww..mmtteeeeeeee/////aa/
hb_JF2519. (26) [27]
[Print shop and travelers 1117////111100/mminnmmmgg (27)
woodblock printer's]
Comperative print Lechnologies (presence in dataseL)
ragion drypcint etching lithography ukiyo-o polychrame (nishiki-e) woad
engraving woodblock printing
28

---

1498 dtarupted did Published as a conference paper at ICLR 2026
Bast Aaia
Bast Asia Japan
Europe
[Technological descript ion and macket orientation strictly per cited essays; no additional sttribution beyond listed arcists.] [27]
Islamic World
Frade and naterials: The Fax Mongelica catalyzed mebility of artisans and Lexury geods, disseminoting motifs (dragons, shoenix/ssmurgh)
materiala (cabelt) scross Chinese, Tikhanid, Timrrid, ottoman, and Safavid comairs [16] Stonepaste (tritware) developed in Syris/Izan (12th 13th e.) emalated Chinese porcelain's whiteness, ensbling underglaze blue and lustez painting; these bodies supported both vessel and architestural tilework [16].
Architecture: Muqarras, callular syster artioulating trans'tional
(e.g.,
squinches, portelsi, reached netable refinement in Timurid contex:s [23] Tmage object: [Mugarnas le, Samarkand artribation |Met 26, 12.12..1891 Ihttps:// http://p///lleeio/////////o/
[24]
Textiles and imperial style: Octomen Burss workshops (15th-17th, e.) preduced kemha lampas), catma (velvetsl,
the nakkashane codified saz leaves, stylized florals (tulip, cannation, ayacinth, rosel, ogival lattices, and consumptionn
(slott-of-gold/ssiiee)
chintamani
export [25].
Figuration and aniconism; Courtly figurel trzditions existed alongaide aniconic preferences in sacred contexis; iyzantine-aalanne transitions reveal negotiated, local practices zather than monolithic bans [23].
Ceramics and architecture (presenss)
region
painting percelain stonepaste (frilware) undergiaze
|-.......
East Asia China 3 11 0111
Islamic World 111011111
region muqornas
Islamic World
[Clains anchored in Met educator and essay materials, collection object data, and symposium volume. [16; [23] (25]
3.4 Europe
Venice and the Islamic world: 13th 16th
carpets, architectural 1ith-16th-cceuur century exchanges linked enameled glass, silke, Venetian makers emulated
techniques, while Octoman textile motila circulated into Venetlan. workshops via commerce and diplomacy [16; [25].
Ninetsenth-century
culture and Japonisme: industrial print proliferation (woed engraving, lithogrophy, photo-praceases) coexisted with an atching revival (including artisana_ approaches).
influx Japaneae ukiyo-e (post-1854) influenced composition, color, and moders subjecta in Buropean painting and prints, including Gauguin, lithographic poster (e.g.r Toulouse-
by Mary Cassatt, Lautree) [2a].
Artists (subset)
artist_id
region -------]
AST_CASSATT Mary Cassatt Europe peintes/prritmaker (color drypoints)
ART @RT_CAUGUTN Poul Gouguin Europe peinter/printmaker (carved blocks)
AKV_TLautrec Henri de Yoslouse-Lautxes Lautred
surope lithographic poster artist |
[Analytica] links to aapenisme Limited to Ket easay; no additional styliatic claims introduced. :28]
#ft# Sub-Ssharan
Centexts (Interregional Links)
Fast Advican coasta (9th-16th 6.1: Urban Talamto culture (mosques, aces in coral atone flourished through Indian dcean trade in goid, ivory, and slaves, exchanging for 1ka, porcalain; Fortuguese intrualons not erase eatablished nełworks [19].
eloth,
(e.g., brasses, Akan gold weights) reflact interactions with islamic and, later, Fostuguese trade; ahaped larchpr monumental. adobe architactura chana-Mal1-
Ceast Westeza/Central
(1000-1608)
and ritual
Songhai polities [21] [22].
(subset,
provided)
id sitle_or_descrippte region
date_text
29

---

Published as a conference paper at ICLR 2026
AORK_ BOUL_LGGLLSS Akan gold weights (ornamental imprint transitioning to Africa 16th Benin brass plaques/heads (ccurt arts linked to trade) Sab- Saharan Africa 15th-16th p.
vernaculac jecta) BENIN_ BRASS PLAQUES
[Included to contextualise Iadian ocean Liade nodea referenced in Ialamic and Bast Asian ciscuits; details restricted to cited essays.) (19) [21] [22]
Indigenous Americas information not previded.
## Cross-Cultural Transmission Fathways
The table nummarizes documentad vectors of material and stylistic transfer acrogs regions odrlodizec provided) [18) [25] [28].
pertod_span xfer_id Bource_rogtee regton target_region vector -------------
|----------
XFER_ XFER_COBALL SOBALT ROBALL_IRAN_CCE IRAN_CHINA
Islamic World East Asis China cobalt (For underglaze blue
porselainl 13th-14th
XFER, SLJFWATTR SATTT___TTAA_RAAA TRAN East Asia China Talamic World blue-and-white models
reinterpreted stonepaste
XFER_ (i____OONIIX_XXE0 SIMURCE East Asia -- China | Islamic World phoonix//lood--ereee motifs
to stmurgh/vegetal spirals 13th c. VENETEIAN
Islamic World Europe enameled glass Lechniques adopted
by Venice 13th-16th c. PETTOMANTEXTTIES mot ta in Venetian workahops I 16th c. XPIR_JAPONISME
Islamic World Europe
soz/Chintamoss/forral textile
Asia Japan Burope
ukiyo composition/coior influencing
Casaatt, Couguin | 19th century
These pathways exemplify bidirectionel trenslation of materials and motifs imperial, mercentile, and artisansl nodes.
184 Artista, Works, Techni quee, and Contexts (Normal ised Sntities) All entries dezive from the provided dataset; absences reflect source limitations.
Artista (selected)
actist_id name zegior
roles
[------]]
[RT_SUZURT__R SUZURT_JAROHOBU HARUNOBU
Suzuki Farunobu East Asia -- Japan designez of ukiyo-e prints Katsushika llokussi | Bast Asia Japan ukiye print designer utagews kiroshige zast Asis Japan ukiyo-e print designer Xara Memi Islamia World Ottomen court designar (florala) Islamic
ART_HOXUSAI
ANT_H18OSHIGE
ART KARA, MEMT
ART_SHAH_ONN
court designer Mary Cassatt Europe peinte:/printtmske (color drypoints)
style)
ART_CASSATT
ART ART_GAUGUIN
Paul Gauguin Europe peintes/printmaker (carved blocks)
AST AsT_Tlautrac
Henri de Toslause-iautrer surope lithographic postar artist
works (selected) missing dates attributions flagged such)
Litlle_o_ddescrpptio
region date_lext year
------!
Takht-i Sulciman palace tiles phoenix/simazgh (1lkhanid, late 13th e.l Talamic Norld | late 13th
WORK BORK_MS__MON MLCT_AUGARMAS_TLL rile from Squinch (mugarnas element), Met 20.120.189 islamic
World
MORK _ETYOO_TTMPIRR,
TEMPLE, TANTERN Ukiyo-e print: temple view beneath paper lantern (Met: JP2819 Asia Japan
image)
WORK_ U | Print shop and travelers at weadbleck printer's (Meija Japan
depiction)
WORK_AMAN_RN poTT_TAT Akan gold weights (ornamental imprint tranaftioning to vernacular subjects) Sub-Saharon Africa I 15th-16th e. plaques/heads (court Hinkd trade) Sub-
SENTN_RRASS, PTAGUES
Saharan Africa 15th-16th c.
(world's oldest dated printed
China 866 868
Techniques, materials, and motifs toontrolled)
category
chintameni motif motif saz laaf motif motif
koza barrr)) paper material
30

---

Published as a conference paper at ICLR 2026
porcelain material
s.onepaste (Iritware) material
color drypoint technique
drypoint technique
atchinn technique
kento registiation Lechnique
lithogrephy technique
luster painting technique
augacnas technique
squinea technique
ukiyo-e polychrome (nishiki-e) technique
underglase blue | tecanique
woodblock printing | Lechnique
wood engraving technique
Contextual
period (domains indloated]
period_id key #actors (as provided)
IRADE PAX Interregicnal trader artisan mobility; transmission of notifs Idzagons /phcenix-simargh) materials (cobalt) [16]
TRADE, VENTCC
Venne//an-Kamluu--attmmm commerce; emulation in glass/textiles: [16]
dip_omacy
TRADE_ CANDIAN_CCEAN_SSS East African coastal exchanges; Portuguase intrusions from 1498
[19:
EGYPT_ JLLMMMMOLOOOTOAAA Spice route shifts; OLtoman integratien and patronage
reorientation [20]
ACSUDAN_ISLAK Islam's rola; monumental adobe architesture; scholarshipi coinage/trado
[22.
OTTOMAN_SILRS
worksheps;
codification palace
export;
ser culture shifts [25]
JAPAN_UKTYOR_POII Collaborative production/ keate registration/ urean subjects [26] [27]
BURCFE_ 13C_PRINE
Industrisl proliferation; etching revival, Japonisme [20]
#84 Debatea, Institutiors, and Marketa Piguzation and image practices: The Age of Transition materials demonstrate that eacly Islamic aniconism and iconeclasm defacement of church mossics in 8th-century Palestine suggests lacelized Christian agency rather Institutions and macketa: Imperial sumptuary syatems and palace workshops (nskkaehane) codified textile designs
context-specific and negotiated; selective
blanket Talamto prohibit' [23].
diffuzed bazaars; uxiyo-e's publisher-led networks produced large editions for urban consumers; 19th-certury Burope saw artieenal revivels alengside industrial proceszes and mass advertising
[25: (2)])
Image (ouratorial discourse
[Artforum page zeferencing McBvilley's critique (letters)] Sosttcnt//olooss////////aattttti 359.ggggg
com/v/
## 7. Validation and Data Quality Flags
dalaset uadezwent nozmalization basic integrity checks; issues zeported than corrected.
Region ccvarage (counts from provided entries)
region period_count work_count
Buzope
East Asia Chins
East Asia Japan
Talamic Korld 6 2
Sub-Saheran Africa | 1 |
Tnd gadiganous Americas
multiple (e.g., Takht Sulaimen tiles) retain parsed or absent dates.
missing numeric
miasing any date: sevaral antrias date information entirely (e.g., Met mugarnas element; uklyo-e images).
Works miasing artist attribution: =ll listad works are unasaigned to named artists in the workahop/anongmog production many cited contexts. Contcolled Lerms: All work Lechnique links conform to the controiled vecabulary. parsing: spans parsed successfully; start/end inconsistencies.
dataset; aligrs
Timitationa:
31

---

Published as a conference paper at ICLR 2026
Indigenous American art: information not provided. "Ancient" "Costemporazy"" endpoints: Only partial coverage Burope (19th century) is present; earlier classical antiquicy and later coatemporary materials are absent.
1000)
Sculpturg and architecture: Rocumented for Tslanic contexes (muqarnast and African urbaniam; otherwise undecrepresen:ed la the provided corpus.
* Survey Note
## Literature Review Theoretical Pramework
Iconology (Panofeky) provides a method for reading symbolic content within histerical herizons [1]. saxandall's "pperiod eye'! thesis situstes style within socis. practices and patronage, applicable to workshop systema and market logics evidanced in Venation exchanges
uklyo-e publishing [2] (266, Social
history (Clark)
frames 19th-century urban modernicy and prist culture, contextualizing capanisme's impact [28,2) Feminist ortziqua [Nochlin] and costcolortal analysts (Said)
interrogate institutional exclusions and erientaliat Laxononies shaping canons and displey [5) Clifferd's museun critique and "gichal ar: history"! orientations foreground circulation, collecting, and malti-sited exchange [6] [8]. Kekvilley's critique underscores curatorial mediation of non-Nestern arts within modernist. narratives [9, [10].
#4 Methodelogy Data Analysta
Source basis: Met leeilrruna essaya and ject records; educator curriculum; symposium volume; and methodelogicel texts [1] (6) [10] (11) Nosmalization: Periods
atandardized numeric apens; entities (regions, artists, works, techni ques/mteeriall, contexts) were cortrol led and Linked; croas-ragtenal Lranemissions Labulated.
Comparative matrices: Presence/absence matrices acroas prints, ceramies, es, textiles, and architectural features synthesize zegional technical emptasea.
Validstien: Controlled vocabulaties enforced; duplicates checked; coverage geps Elagged.
## Critical Discussion
Findings underscore how technologies (porcelain, atonepaste, mugaznas, silk-weave structures, color woodblock regis::ation! intersested with patronage systems (imperial workshops, academtes, publiahersi to reorgan' stylistic posaibillties and markets. Crose cultural Lzanemissions operated bidirestionally, with materials (cobalt) and models (blue-and-whiter arabesque systems) translating Iconographic debates in sysantine-lslamic contexts argue against reductive dichotomies of figuration/ant The dataset's saymettzies (notably Indigenous Americas and "Ancciet/Conie /ooneeppraay gaps) caution against overgsneralization.
medis and courts.
emphasi negotisted practice [18] [25) (26) [27] [28].
## Puture Research Directions
Populate indigenous American
periods with parsllel attention technolegies o.g., fiber arts, lithics, pigments), institutions (temple, court, communty), and transmisaion
Extend Duropean coverage to Ancieat, Medieval, Renaissance, and Contemporary with controlled stylistic technical vocabularies aligned terns [12].
Art Online and
Deepen erchitectural datasets for East Asia ttimber freming, bracket sets) and Burope vaulting, iron/steel in modarn/contemmorrryy. and sculpture
all regions. Integrate authority control (AAT/UAAN/GGN) stable identifierd and expanded coject-level
metadata from peer-reviewed galeways (Met, Smarthistory, Oxford Art Online, JSTOR,
Project XCS2) [11] [12] [13] [14] [15].
# Key Citations
[1] Studies in Tconology: Fumaniatic Themes in the Art of the Renalasance: https://www routledge, com/Studles Iconology Ilunanetii
The of Renaissance/
[2] Pointing Experience pifteenth-Contury Italy (Internet Archiva iten): https://
archive..rg//eee http//taall/ppaaaaae ingeeeeeeeeeeee
[3] The #rinting of Nodern Life (Princeton University press]:
[4] Mhy Htav There Been No Great Women Artista? (Thames à Eudson product page): https://wwx
.chamesenchudeon. wttp://.ccmmpooocs////eeeee have Seveethere-ee no- no-grest- women artis:s
[5] Grientalism (Scodreads editions pagel: orientalion
[6] The Predicament of Culture (Oxtam Books Listing): https::/ww..oxttambkksss surpppoo
[7] Global Art History (De Gruyter/Brilll:
[8] Africen Indigeaous Anexledges in Postcolonial (Routledge) https://www
rout lodge. wwwp////.iiae..eeeecceeeeeeeeeee //tp://aaaaaaallll-tttthhaohhppa
32

---

Published as a conference paper at ICLR 2026
[9] Artforum (letters image associated with scivilley's critiquel: https: bttpp:://ww.eee
/wg http////oood//////////aaaa/
[10] Actforum ladditional page image referencing the critique):
-content//up httt///001/////tticceeeeee
[11] The Met Rei Ibrunn Timel of Art Ristory: httpa:/ httes///wwwmmteee http://wwwmttmmmmorrro///o//////
[12] Cxford Art Online (Grove Art Onlinet: https://digitall wtt.::itta....irra........eeeeee
oxfordart
[13] Smarthistory (About/Essays); httpa://smarthsstoryyy..
[14] JSIOR Archive Jeurnals (about access): https://eif..ne/// htwpre/.e...eeeersuuussese///ee
resources/ archive
journals
[15] Project MUSE (About): https://about..e www.mmuuu..........mmmmm
[16] Unit 7: Trade and Artistic Exchange (Met educater curriculum, Islanic world): httpa://
http://mmmmceeeeeeeeeeeeeeeeeeee [17] China, 500-1010 A.D. Hoilbrunn Timeline of Art Ristory: http:///www.tmsssmmmommmmm
Educators/Pub. icat:oost2.ffrr22eeceeeee
Undi7.ppff
toah/ht/06/eac..
[18] Chine, 1000-1400 A.D.: Sttps:// /tte:://ww.mmtt aetmuseum.org/toeeeeee httphh//////ccctt/e//
[19) Fastern and Southern Africa, 1400-1 600 A.D.: httpp:// hem-
[20] Egypt, 1400-1600 A.D.: attps:/ www.mmemmusuum.c.mmmm /tt.://wwww.oogeoee/////////////
[21] Guinea Coast, 1400-16C0 A.D.: https:/ http///ww//meeeeeeeeeeeeeee www.w./taaeum.ccommmmm wttp:/ttt.tt/////.//e/ hamm..mmll
[22] Neatern and Centrol Sudan, 1000-1400 A.D.: https:/ htwpp/////w////////p/////pp/ettt www.nm.. www.:/mmttsssumrrooo... https//wwwwrroooooo/////////// htm
[23] Aga of Transition: Byzentire culture in the Islamic Korld (Net symposia volume) https
Age_ef_Trans Ltt_Tiaaaniioon_yaaaaaa htwp://gggeeeeerreeeteeuuuu www.octttooooo//oeeeeooeee tons/pdr/
://resourcea.. www.mmmemmmmmmmmmmmmmmmmm
Sewee_sssmmee__aee Pddd
[24] Tile from a Squinch imuqarnas element; Net 20.1200119)) https:/ htttps://www... /ttp///wmmmettt www.metnuseum.org/eee org/artt./
[25] Silks from Ottoman Turkey (Ruraa sarfculture; nakkashane; htt/ch/n ) httpa://wwx
.eemmuaeum ozg/essays/silke fzom ottoman Lurkey
[26] ooc Printa in the Ukiyo-e Style (technique) kento; col collaboration): https://wwv.
metnuseum, wettmseuu.oog//eeaaae// htteeessayy//ooolloooooooooooooo
[27] Art of the Plessure Quarters and the Ukiyo-e Style lurban culture; subjects; markets):
https:///ww. http///wwttssmm.rr//p/poo hww.://mwgggaaaaaaatato-----e--e
[28) The Print the Ninsteenth Contury (industrial prints; stching revival; coponiama):
httpa: ///////wwwww http///www..e//e///.e/////// mettmseuu. org.eaay///saee/eeeeeee hawpaayyhheeeeitttt th-tthhe-eeeeeentteeeee
Notes scurces and images: All images embedded above are drawn exclust from the provided resources. Assections are restricted to the content of the ciced materials, provided.
Saps acknoxledged explicitly where information
Figure 12: Qualitative outputs for the same prompt in Figure from DEERFLOW (a) and DEERFLOW+ (b). Our DEERFLOW+ retains more retrieved information, presents stronger evidence structuring and citation discipline, while avoiding token-limit failures seen in the vanilla version.
E EVALUATING CITATION ACCURACY WITH RUBRIC TREE
Evaluating citation accuracy with a rubric free. In citation grrounded reports, links may be hallucinated, irrelevant to the topic, or fail to substantiate the associated statement. To capture this, we employ an agentic judge that performs structured validation process: (1) identify each claim with its linked source URI., (2) check whether the URL accessible, and (3) if accessible, assess whether the content sufficiently supports the claim. The entire process can be visualized rubric tree (Figure 13)
Optimization. As reports may contain extensive citations, verifying each link individually would be time-consuming and costly. To improve efficiency, we group all claims associated with the same link, requiring only a single web retrieval for the grouped claims. In addition, before checking whether the content supports a claim, we first perform coarse-grained relevance check by examining the initial portion of the webpage (e.g., title or first few hundred tokens). For example, if the task concerns technology article, we terminate early without further evaluation.
healthcare but the link directs In total, we identify three types of errors. E1: URL is inaccessible or does not resolve; E2: URL content irrelevant the topic of the task; E3: URL content does not support the specific statements.
SoTA Deep research systems are far from citation error-free. Using our rubric-tree framework, we evaluate the top performers from Table D (GPT-5, Grok-4 Deep Research, and Open Deep Research) on the two most search-intensive tasks: market analysis and wide information search. report the
The results are shown in Table 7 where for each agentic system and task category, average number of E1, E2, and ES errors in the generated reports. We can see that all models produce non-trivial citation errors. In wide information search, most errors stem from unsupported claims
33

---

Published as a conference paper at ICLR 2026
Report Example
Rubric Tree Citation Accuracy
Time Emfiieett Training: HOT MICT Adults with Limited Time
(Ecaaim 1, Claim 2, Claim K}, URL
cefinee
75% HR_
moderate pacalee] aligns with traditional jagging or eyclingats
steadly g
training with: public guidalines recommending minutes per ieek moderate exercise[7] 71, (Claim
Does U Resolves?
NO
Link Invalid (E3)
Cotrien Treen
iigorou--inennity exercise (ilke HDT can confer equivalent health benefits moderate exercise roughly half the total time [7]. [Clamm 2] The WHO's 2020 guidance, for recommends 75-150 minutes vigorous activity atternative 150-300 minutes for time poor
Trelevant to the task?
NO
Link Inrelevant (B2)
activity [7). [Claim 3] can yleld meaningful spureed
ostablihes
YES
"minimal
Claim is suppected by
NO Unsupported cialm (R3)
References
VRS
[2] World Health Organization 2020 guideline on physical actinity and sedentary behavtour https:// pmc nebi wtwp///iccee/////////// #: P--tott-.qqupe http:///wooinneeooooooooooooooee
Y
PASS
Figure 13: Illustration of rubric tree- based evaluation of citation accuracy. Compared to evaluating all claim-URL pairs individually, the rubric tree groups claims by link, which significantly reduces evaluation costs while enabling finer-grained categorization of citation errors: invalid links (E1), irrelevant links (E2), and unsupported claims (E3).
(claims not verifiable from the cited link) rather than invalid or irrelevant URL highlighting that hallucinations persist even with web access. The problem worse for market analysis, where all systems generate large numbers of unsupported claims (Open Deep Research averages 91.9 errors per report), underscoring citation accuracy as a persistent bottleneck in deep research.
E1
E2 Errors
D3 Errors
Agent Name
Total
Errors
Task: Wide Info Search GPT-5
4.2
1.7
13.3
19.2
Grok-4 Deep Research
6.8
6.8
33.4
47.0
Open Deep Research
5.0
5.2
19.7
29.9
Task: Market Analysis
GPT--5
11.1
10.1
43.8
65.0
Grok-4 Deep Research
6.3
6.4
61.5
74.2
Open Deep Research
11.9
11.6
68.4
91.9
Table 7: Evaluation of leading research agents in citation accuracy for search-intensive tasks. For each agentic system, report the the average number of errors for each task (less better). We consider three types of errors: URL invalid (EI), URL irrelevant (E2), and Unsupported claim (E3).
F LIVERESEARCHHBENCH DEMONSTRATION
We provide concrete queries LiveResearchBench across different task categories in Table Table and TQ]
G ERROR PATTERN EXAMPLES
In this section, we provide concrete examples of error patterns observed in our pilot study. This includes mismatched in-text citations and references (Figure [4), missing links or incomplete URLs (Figure 15 inconsistent citation formats (Figure uncited references appearing in the bibliography broken or incomplete table formatting (Figure 18 oft-of-order reference numbering
(Figure 17
(Figure 19 embedded citations breaking text flow (Figure 20 and hallucinated information
'E1 and E2 are counted at the URL level (each problematic URL contributes one error). E3 is counted at the statement level: all citations attached to claim expected to support it (e.g., if claim about vaccination policy cites [5,10, 12,24,2,, the E3 emrrr count incremented by one for every source supporting the claim.
34

---

Published as a conference paper at ICLR 2026
Category Query
Competitive Analysis
Compare Spotify, Apple Music, Youtube Music, Amazon Music, Pandora, and Tidal on music discovery features (including algorithmic recommendations, curated playlists. and social discovery). audio quality (with reference to both technical specifications and perceived listening experience), and podcast selection (considering global availability), user interface design and case of navigation, music library size and catalog completeness, exclusive content and early releases, pricing structures for individual and family plans, artist compensation and payout rates, and comparibility with high-end audio equipment as of now.
Literature Review
Help produce detailed academic report (literature review) the evolution of evaluation practices for single-ages aagett and multi-agent systems (based on large language models). Focus on the works published from 2023-2025 (including high- quality published papers, technical reports, and preprints). The report should include the following dimensions: (1) Benchmarks and sandbox enviromments: How do benchmarks for single-agent systems differ from those for multi-agent research in terms task diversity. scalability. and realism? What are trade-off's made to create benchmark, their strengths and limitations? (2) Metrics: What families of evaluation metrics most frequently applicd single-agent systems multi-agent systems, and what trade-off's do they entail? Does the metric rely M.I--as-juge, agent-as-judge, human, string matching? (3) Implementation and cost: How is the benchmark/sandbox implemented? Is human annotation involved in creating the benchmark or sandbox and how is human annotation involved (if applicable)? Is the henchmark/sandhox scalable and reliable? (4) Future Directions: W hat concrete research directions and design principles can advance the development of unified, generalizable evaluation pipelines?
Market Analysis
What has been the size, growth rate, and geographic distribution of the COVID- 19 vaccine market between 2021 and 2025? What were the key demand phases (initial rollout, booster campaigns, variant-specifie vaccines), and how have public health policies and supply constraints influenced vaccine adoption globally? How do Moderna, Pfizer- BeoN Johnson & Johnson, and Novavax compare in terms of global dose distribution volumes, clinical efficacy (trial versus real-world data), and manufacturing capacity? What flagship vaccines define their market strategies, and how is the market expected to evolve considering waning demand and shifting public health priorities?
Table 8: Example queries in LiveRescarchBench (Part I).
(Figure 21 These representative cases illustrate the diverse range of structural and citation-related issues that can undermine the clarity and overall reliability of generated research reports. These observations motivated the tailored evaluation metrics in DeepEval as described in Section
35

---

Published as a conference paper at ICLR 2026
Category
Query
Decision Support
In adults aged 30-70 70 with limited weekly training time (< 150 min/week; with < 60 min/week subgroup), what is the comparative effectiveness of high-inlensity interval training (HIIT; including low-volumc/prrnn-intee prrerttinttevall variants), moderate- intensity continuous training (MICT) and hybrid HITT-MITT protocols on (a) cardiorespiratory fitness VV,.max/V.ppaak mL-kg ---min "nninn),, (b) cardiometabolic outcomes (HbA I c//asting glucose, HOMA-IR, blood pressure, lipids, adiposity/liver fat), and (c) patient-centered outcomes (adherence, enjoyment, adverse events)? How are these protocols defined and evaluated across international randomized trials and meta-analyses (2018-2025), with supportive context from observational studies and major guidelines (e.g.. ACSM/WIIO), and which progression/maintennnn (session frequency. intensity prescription, minimal effective dose; supervised vs. nosee-bsedd delivery) sustain gains over 26-12 months? Consider moderators including sex, age band (30-49 vs. 50-70), baseline fitness, and cardiometabolic risk (overwcigh/obesity, prediahetess/ottholiic syndrome, hypertension), as well as medication status.
plans
Policy & Regulation
I work information security and responsible for incident disclosure. compare CTSA's CTRCIA regulations, the SEC's cybersecurity disclosure rules, and sector- specific mandates under NERC CIP Help me provide cross-sector analysis (energy. finance. healthcare, and other critical infrastructure), focusing on the most current compliance challenges from the past 1-2 years. Analyze across reporting timelines, covered-entity definitions, enforcement mechanisms, and penalty structures. Provide: A short description of each regulatory framework A multi-column comparison table covering all dimensions Multiple real-world examples enterprise compliance challenges per framework An analysis strategic implications for U.S.-based global enterprises operating under U.S. jurisdiction across multiple critical infrastructure sectors
Pros & Cons
Conduct a comprehensive amalysis on whether social media is good for society, with primary focus on Europe (including the UK. EU member states, and Nordic countries), The analysis should focus specifically on teenagers (ages 13-18) and their relationship with social media. For this report, define "good society" across four dimensions: mental health and well-being (e.g.. anxiety, depression, self-esteem, and loneliness), 2) educational and developmental impacts (c.g., lcarning, attention span, and digital literacy). social and civil outcomes (e.g.. friendships. community engagement, political awareness, and misinformation exposure), and 4) safety and risk factors (e.g- cyberbullying. privacy. exploitation, and screen-time effects). In the report, I would like to have 6 pro arguments and 6 con arguments. Each pro con argument should include: (1) A clear thesis statement (2) Multiple supporting points substantiated with credible evidence (including but not limited to statistics, direct quotes, case studies, expert testimony, industry or government reports, academic research, news sources, online forums, and social media), followed by in-depth reasoning and analysis. Prioritize data and rescarch from the past years where available (2020-2025).
Table 9: Example queries in LiveResearchBencch (Part II).
36

---

Published as a conference paper at ICLR 2026
Category
Query
Topic Exploration
As someone who created online photo editing tool targeting social media marketers and small businesses, hclp understand how Al transforming the photo editing space and what need do stay competitive. My tool currently offers basic editing features like cropping, filters, and text overlays, and I'm considering adding Al cupabilities through third-par purt-pprtyyyy APIs rather than building in in-house. I want to know (I) which specific Al features would be most valuable for social media marketers that I could integrate without massive development costs, (2) whether Al will make traditional photo editing skills obsolete or there's still place for manual editing tools alongside AT features, (3) how major players like Canva, Photoshop, and newer Al-first competitors are using to attract users, and (4) what the real costs and benefits are integrating development timeline, server load, impact on engagement and premium conversions. Give examples from photo editing tools serving similar markets, with specific Al features, Identify U.S. and Canadian banks and broker-dealers that, according to publicly available sources, launched (i.c., production, not pilot) client-facing Al portfolio- management or financial-planning 'copilots' during 2023-2025, and compile due- diligence table covering: institution; assistant/rroduet name; launch month; customer tier; geography languages; core Al stack; safcty & compliance; integrations (c.g.. custodian/portiolio accouning/CRMMplannnn
into my platform, including monthly fees,
and results.
Wide Info Search
tools, SSO, audit logging); access & pricing: KPTs (e.g.. adoption, satisfaction, ROI) where available; plus links to the bank/OEM press release. product page demo, and at least one independent wealth- tech or fintech-trade article confirming the launch.
Top Rankings
What top 20 gaming monitors released or available for purchase in 2025, based refresh (Hz), response time (ms), and price per inch of screen size? Please assume all three criteria weighted equally. No specific panel technologics size requirements set; consider all currently available monitors within this timeframe.
Table 10: Example queries in LiveResearchBench (Part III),
37

---

Published as a conference paper at ICLR 2026
Mismatched in-text citations and references
Report from Manus:
The Electric Truck Market: 2024-2025 Analysis
Executive Summary
The electric truck market is undergoing transformative period, characterized by rapid growth, evalving technological advancements, and increasing adoption across various customer segments. This report provides comprehensive analysis of the market's size, demand trajectory, and segmentation for 2024-2025.
the key customer segments driving growth-commmrcial fleets, consumers, government-and examines the critical infrastructure and regulatory factors influencing adoption.
1.1 Market Size
The global electric truck market is experiencing significant growth. Several reports provide varying but consistently upward trends for market valuation:
[7] to USD 29.69 billion [1, 6] and USD 3.99 billion 14]. The discrepancies likely stem from different methodologies and inclusion criteria (e.g. light-duty medium/ heavy-duty, regional global). ""0055"* The market projected to grow to USD 4 71 billion [14], USD 5 52 billion [5, 13], or USD 5.88 billion [8] in 2025, Some reports indicate growth to USD 29.7 billion in 2024 and USD 124.97
*2024. Projections the global electric truck market
range
billion by 2030 [1, 4, 6).
References
Electric Truck Market Fundamentals (2024-2025)
https://www.carrorr.coom http://wwwwww..ooooooo//eeetttte htwps//rrwrcttcc.iiiiiiiiiiinini
[2] http:://wwww.gmiiiiigee.. com httmm//wuuurrraaissssaeaieaaeeee
[3] https://wwwwiea
[4]
Customer Segments and Adoption Drivers
[1] http://wwwmrctaaooooo/////////te
solutions
[3] http://wwwssseetttiiiieeeeediiia
[4] http://wwwww.......oo/o/otttttte
poweg----o---gggiiiigg [5] http::/wwwww.............eeettte
Analysis:
The report reuses citation numbers across multiple reference lists, causing ambiguity in mapping in- text citations to sources.
Figure 14: Error pattern: mismatched in-text citations and references.
38

---

Published as a conference paper at ICLR 2026
Missing Links (URLs)
Report from GPT-4.1:
Clinical Presentation & Differential Diagnoses
1.1 Extensor Carpi Ulnaris (ECU) Tendon Pathology
""ECU Tendinopathy"" Repetitive microtrauma during supination combined with ulnar deviation can cause overuse injury to the ECU tendon, common in racket sports [1][2]. Features include pain on the ulnar side of the wrist, swelling, and discomfort during resisted wrist extension with ulnar deviation- tendinopathy [2][9].
symptomatic
"EECU Instability Subluxation". Sudden onset "bolt of blue" pain the ulnar wrist,
especially during double-handed backhand strokes, may indicate traumatic ECU instability (subluxation
dislocation) [1].
References
Montalvan Parier J, Brasseur JL, Le Drape Extensor carpi injuries players:
study of 28 cases. "Br Sports Med'. 2006:40(55:42444277 (Identifies ECU tendinopathy, instability, rupture; outlines immobilization and surgical protocols.) 2. Physio sources on ECU tendinopathy-overuse mechanism, symptoms, common in racket sports [2015+ general knowledge).
Logli AL et Ulnar wrist pain tennis player: case review and discussion, "JBJS Reviews". 2022. (Discusses TFCC, ligament injury arthroscopic repair)
Analysis:
In-text citations (e.g. [2], [9]) either lack proper reference entry appear the reference list without URLs or identifiable sources.
Figure 15: Error pattern: missing links (URLs).
39

---

Published as a conference paper at ICLR 2026
Inconsistent Citation Format
Report from Gemini-2.5 Pro:
Major U.S. Bank Received a Less-Than-Satisfactorr Community Reinvestment Act Rating Recent Years
"Washington D.C." A comprehensive review of Community Reinvestment Act (CRA) ratings for U.S. banks with assets exceeding $50 billion reveals that only one institution, City National Bank, received a lles-than-attisfattoor rating in its most recent evaluation between 2019 and 2024. The bank was rated 'Needs Improve" by the Office the Comptroller the Currency (OCC) an examination dated 27, 2023,
the years 2019 through 2021.
Below are the detailed findings for City National Bank:
City National Bank
Rating: Needs to Improve
Exam Date: March 27, 2023 (Evaluation Period: January 1, 2019 December 31, 2021)
Federal Reserve/OCC CRA Disclosure: [Office Comptroller Currency (OCC) Performance
"Reeuttblee News Source: [C [City National Bank Press fetess///tttp://wwwccnccoo//m//
newsroom/
the vast majority of large banks maintained "Satisfactory" or "Outstanding" ratings during City National Bank underscores the continued importance the Community
this period, Reinvestment Act ensuring equitable access credit for all communities.
Analysis:
The report mixes multiple citation styles within the same section. For example, the "Federal Reserve/ OCC CRA Disclosure' entry combines descriptive hyperlink with a numbered reference (1], while other entries only use hyperlinks.
Figure 16: Error pattern: inconsistent citation format.
40

---

Published as a conference paper at ICLR 2026
References Not Mentioned
Report
Report from Manus:
U.S. Electric Vehicle Market Analysis
1.1 Market Size
In 2024, the U.S. electric vehicle market demonstrated robust expansion in both sales volume and revenue, Full-year EV sales in the United States ranged from approximately 1.3 million to 1.6 million units [1, 3, 5]. This volume translated into an EV sales share of around 10% of all light-duty vehicles sold during the year [1, 3]. In terms economic value, the U.S. electric vehicle market was estimated USD 131.3 billion in 2024 [11].
worth
1.2 Growth Rate
The market's growth trajectory remained positive, albeit with some variations in reported figures. EV sales in 2024 increased by 7.3% compared to 2023 [5]. The broader U.S, and Canadian market collectively saw 9% growth in 2024 [9]. Looking ahead, the market is projected to maintain a strong upward trend, with an anticipated annual growth rate (CAGR) 12.61% between 2025 2029, which expected to result
US$154 billion by 2029 [7].
Adaptation to Pressures: Ford is adapting competitive and regulatory pressures by broadening its electrification strategy and focusing on competitive battery costs [78]. The company also navigating evolving emissions and safety stanclards globally [79]. Ford's price cuts on the Mach-E reflect its adaptation to compotitive
where pricing key factor [81].
References
95. Ford Broadens Electrification Strategy to Reach More (2024, August 21). Fromtheroad.fonc.comm www.frrrtmhrdaddddooo more-customers-
Ford's Role Shaping the 2024 Electric Vehicle Market. (n.d.). Willisford.com, https://
www.willisford com 222e
Integrated Sustainability and Financial Report 2024. (2024, April 22). Corporate.ford.com https:// corporate.ford. htt.:/oow..ooooo.eeeeeee
pdf
98. Mustang Suv 2024. (2025, September 6). Cartkeettneewittbbbsttaooddd.. https.//cert/test
new.itlab stantfor..eduu/mussaaggsuuuu
Analysis:
The reference includes entries (e.g., [95]-[98]] that are not cited anywhere in the report body. This misalignment citations
Figure 17: Error pattern: references not mentioned in the report.
41

---

Published as a conference paper at ICLR 2026
Broken Table Format
Report from Gemini-2.5 Flash:
Comprehensive Research Report: Global ESG Reporting Strategies (2025)
sustainability consultant, understanding evolving landscape global ESG reporting is critical for multinational enterprises. This report provides detailed comparison the most recent versions of the SEC Climate Disclosure Rule, the EU Corporate Sustainability Reporting Directive (CSRD), and California's climate disclosure legislation (SB-253 and SB-261) as 2025. addresses key dimensions, offers insights compliance challenges costs, analyzes strategic implications for companies operating across these jurisdictions.
Multi-Column Comparison Table
SEC Climate Disclosure of 2025, currently stayed) Corporate Sustainability Reporting Directive (CSRD) (as 2025)l California SB-253 SB-261 (as of 2025)
Dimension
It's important to note that the regulatory landscape for ESG reporting is dynamic, and some of the rules mentioned, particularly the SEC Climate Disclosure Rule, are facing legal challenges and potential amendments. My analysis based "most recent versions available September 2025*, considering reported proposals statuses.
Analysis:
The report contains incomplete comparison table. While the header row generated, content rows are provided, leaving only blank space before resuming the main text. This broken formatting disrupts readability and prevents the table from serving its intended comparative function.
Figure 18: Error pattern: broken table format.
42

---

Published as a conference paper at ICLR 2026
Out-of-order References
Report from Manus:
Sleep Body-Clock Strategies Enhanced Alertness, Learning, and Mood
Specific Considerations for Shift Work:
Shift workers face significant challenges maintaining circadian alignment. Strategies include:
"Fixed Shift Schedules:" Where possible, consistent shift schedules (e.g.. permanent night shifts rather
than rotating shifts) can help the body adapt
effectively [70).
""Light Management." Use bright light during night shifts to promote alertness and wear dark glasses on the commute home to minimize light exposure before daytime sleep [66].
"Melatonin Supplementation." Under medical guidance, melatonin can used to help adjust to new
sleep-wake cycles, particularly when transitioning between shifts dealing with jet lag [39].
References (continued)
[63] Sleep and Academic Excellence: Deeper Look. (2024). "Stanford Longevity" [nttps:// longevity.sanffrdd hwwfforrdddu//ffff
(httpp////
oongevitystanfort
(64) Polyphasic Sleep: Pros and Cons of Short Sleep Schedule. (2018). "Verywell Health". [https://
www.werywellhealth htom/poo--aat pros-aadd-coos-of---plllyyyy
www.werywellheathhhco com/ www.://ww.....ooo...eeeeeeeeeeed
[65] Sleep hygiene: Simple practices better (2025). "Harvard Health Publishing". [https://
www.health. www.aathhhhaaar....m hawvard.edu/staay.............ee
www.health.hor. harvard hww.:tthhhhaaaaddduussssaaayyyy
References
[1] The impact of extended sleep on daytime alertness, vigilance and mood. (2025). "ResearchGate"
http:://www.researchg..... net!
https://www..rssarchgtee.. aameeeemat__oom
publicationn/8304477.. Theri77T_eeeimne
seaanndd_seee__eee _o__ooond_e__________e__ sanytime_alettnesss... vigilance_ane sac____ndddddd_llleeeee
Jublication/8370444 1ee/04477__he_eiee Tha_ehhaettt__teee of extended, setodddd_seee_eeee __lee_eeen... @eeep_on_dyytiiine ____ddyytmm____ _alerttess_sesss____ vigilance and mood)
(2) Circadian Rhythms National Institute of General Medical Sciences. (2025). "NIGMS" [https://
www.nigms.nih.. cowo/education/fccttheeeeee
Analysis:
The reference misordered: instead starting [1] and proceeding sequentially, begins at [63]-[65] then loops back to [1]-[62]. This disrupts expected citation order makes difficult trace inxtext citations accurately.
Figure 19: Error pattern: out-of-order references.
43

---

Published as a conference paper at ICLR 2026
Embedded Citations Breaking Text Flow
Report from Gemini-2.5 Pro:
Mitigating the Mental Toll: Research Report Stress and Burnout Prevention for Early-Career Academics
A 2022 study demonstrated both aerobic and strength training significantly reduced emotional exhaustion, cynicism, and academic inefficacy in university students. Art](https:// vertexalsearch.coo cloud..cccccooo.ooo...oo..... www.wdooogleeoooooooooooo google.com... wooundnng-apl-redee
UPOMMBLLAMCC111g1266Woo wraw _u41df
wtt.o/snccdaae/MMMMM0000000 study
found strong negative correlation between physical activity and burnout, E[3](https://
vertexaisearch.c www.arrccccco....o... www.uddgoooee.ooooooo http://ewmooooooooooe/e/e http://goooooonooogeeggge reierrrreec/tr
V3CJ3LD _ww__NNNPPPPSSSSS_ OPeIZSExxeiDKTaxxFi,, otw.apawwwwi....a.ioooooooootntn
xer
cise not only mitigates the physiological sympomms stress but also selfffffccc resilience.
wwt.ghhwwww..ooooooooooooooo.o.. J4axgE8v9CGol/xile
JJD04PiBXIbOWWaaKKKjj] http://wwwww.......o..oooooottta
_==)) "TTh Strength
Peer and Support:"" Graduate
early-career research isolating. peer
support programs, including peer mentoring and peer-led support groups, have been shown enhance
social connection, reduce loneliness, and provide sense of community.
Analysis:
report disrupts fluency by embedding citations within words (e.g. "A[2]nother," "E[3]xercise"), which breaks the reading flow appears unedited.
Figure 20: Error pattern: embedded citations breaking text flow.
Hallucinated information
Report from Manus:
Top Running Shoes Analysis (August 2025)
Top Running Shoes Ranking
Based the analysis, the following running shoes ranked:
1. "Saucony Endorphin Elite 2
Weight: 0.0 oz
Drop: 0.0
Runner's World Rating Category: Best Lightweight Racer
Composite Score: 1.11
6. "BBrooss Caldera 8**
Weight: oz
Drop: 0.0
Runner's World Rating Category: Best Overall Trail
Composite Score: 0.53
9.
*Lululemon "
Weight: oz
Drop:
7.0
Runner's World Rating Category: Best Hybrid Composite Score: 0.33
Analysis:
The report fabricates data and produces logically inconsistent ranking. presenting hallucinated information (e.g. implausible weights and scores) that appears plausible at glance but undermines factual accuracy and composition quality.
Figure 21: Error pattern: hallucinated information.
44

---

Published as a conference paper at ICLR 2026
H INFORMATION ON THE ANNOTATION TEAM
For data annotation and verification, worked with professional annotators from Salesforce. All annotators hold advanced degrees in fields including English Literature, Social Science, Computer Science, and Finance, with 2-6 years of experience in data annotation and analysis. Their expertise spans multi-modal evaluation (text, image, and audio) as well as quality control for advanced annotation pipelines.
USER SURVEY To inform the design of our deep research benchmark, we conducted an online user survey with participants from diverse backgrounds and occupation. We group participants into three categories: enterprise professionals (including sales, marketing, legal, and software engineering; 27%), researchers (including professors, industrial researchers, postdoes, and Ph.D. students; 24%), and general users (the rest; 49%). Participants were asked: What quest ieestions would you ask a deep research agent?, and What aspects of the model's response matter most to you?
We analyzed the collected responses and observed the following trends: (1) many questions are multi- faceted and require real-time search; (2) users value responses that capture both the surface-level request and the implicit intention of the query; and (3) they expect the final report to tailored the target audience, in content, format, and presentation style.
For the collected questions, we distilled them into ten representative task categories (Figure 2 Among these, the three most common were topic understanding (24.5%), wide information search (22.8%), and top rankings (16.7%). followed by market analysis (8.2%), technical support (6.5%), decision support (5.2%), policy & regulation (4.8%), literature review (4.3%). topic exploration (3.5%), pros & cons (2.0%), and competitive analysis (1.5%). In addition, we also analyzed the domain distributions based on the collected user responses: Science and Technology, Economy and Business, Health and Wellbeing, Law and Governance, Society and Culture, Education and Knowledge. Media and Entertainment. Together, these findings were instrumental in shaping the design principles and the curation of tasks in LiveResearchBench.
J HUMAN VERIFICATION GUIDELINES
Goal.
Verify the quality of deep research queries and their corresponding checklists that aims to assess the comprehensiveness and coverage of the report.
Data Format
The CSV contains one row per grading criterion, with the following key columns:
Column
Description
qid
Query identifier.
Deep Research Query
The detailed deep research query.
Checklist
Individual grading questions (Multiple checklist items per qid).
Table 11: Key columns the annotation CSV format.
You will evaluating two components: (1) Deep Research Query Quality and (2) Checklist Quality.
J.1 ANNOTATION TASKS
J.L.1 TASK 1: DEEP RESEARCH QUERY QUALITY Rate each unique Deep Research Query according to Table 12
Rating
Description
Appropriate
Query is well-structured, complete, clear, and answerable through research.
Appropriate with modification
Good foundation but needs improvement (please provide modified version).
Not appropriate
Fundamentally flawed not worth fixing.
Table 12: Rubric for rating deep research query quality.
45

---

Published as a conference paper at ICLR 2026
Consider the following criteria:
Answerability: Can this question be reasonably answered using web search and analysis?
Clarity: Is it clear what information is being requested?? Useful clarifiers include but not limited to temporal bounds, geographic scope, and specific entities.
Completeness: Is the query complete? Would you, the user, add or remove specifics to make it more complete?
format: When appropriate, is the format and report organization requirement clear? This criterion may not apply to all questionsåATuse it judiciously.
When appropriate, is the query framed appropriately for its intended readers (e.g., technical experts. policymakers, or general public)? Does the wording match the expected background knowledge and level of detail of that audience? This criterion may not apply to all Ausse it judiciously.
Target audience:
J.1.2 TASK 2: GRADING CRITERIA QUALITY
Rate each Check fisckkisst item according to Table 13
Rating
Description
Appropriate
Relevant, verifiable, and important.
Valid but not necessary
Correct but redundant or overly granular (please comment why; modify if needed).
Needs medification
Poorly phrased too narrow (plcase provide modified version). Irrelevant, unverifiable, or impossible (please comment why; modify if needed).
Not appropriate
Table 13: Rubric for rating checklist quality.
Remark: If additional checklist items should be included, you should append the new checklist items for the corresponding qid. When evaluating each checklist item, consider the following criteria:
Relevance: Does the criterion correspond to the requirements of the research question?
Verifiability: Can it be objectively verified from a research report?
Importance: Does it capture a meaningful aspect of the research question?
Non-redundancy: Does avoid substantial overlap with other checklist items?
J.2 CONCRETE EXAMPLES
J.2.1 TASK 1: DEEP RESEARCH QUERY
RQI: What is the size, growth rate, and segmentation of the U.S. electric vehicle market in 2025? What are the key drivers (policy incentives, charging infrastructure, consumer adoption) and challenges (supply chain, cost pressures)? How do Tesla, BYD, Volkswagen's 's ID series, and Ford Mustang Mach-E compare market share, production capacity, and pricing philosophies across major regions?
Rating: Appropriate. The query well-structured with clear sub-questions, specifies concrete temporal scope (2024) and geographic focus (U.S.), mentions relevant entities (Tesla, BYD, etc.), and is answerable through standard market research. Note that the query well scoped with with a business/market research context, and it doesn't hinge on whether the target audience is, say, policymaker, an investor, or general public-all of those groups could use essentially the same data. Therefore, there is no need to further specify the target audience in this case.
RQ2: How has LLM technology evolved and what are the current trends?
Rating: Needs Modification or Inappropriate. The query is overly broad ("LLM technology" is unspecified), lacks temporal bounds, and sets vague expectations ("current trends" without context).
46

---

Published as a conference paper at ICLR 2026
In addition, it does not reflect the needs of specific target audience-wiiln researchers may require technical detail, the general phrasing here fails to indicate whether the query is meant for experts, practitioners, the broader public.
J.2.2
TASK 2: CHECKLIST ITEMS
C1: Does the report provide data on the electric vehicle market specifically in U.S.?
Rating: Appropriate. This checklist item directly relevant to the research question, as it captures the query's explicit focus the U.S. electric vehicle market. It objectively verifiable, since one can check whether the report includes U.S.-specific market data. addresses an important dimension of the query by ensuring geographic precision, and it does so without overlapping with other checklist items.
C2: Does the report mention Tesla by name?
Valid but not necessary. This checklist item is correct and verifiable, since Tesla is explicitly part of the query. However, is overly granular: focusing only on Tesla misses the broader comparison with other key entities such BYD, Volkswagen, and Ford. Redundant if there is already a broader criterion covering multiple entities.
Rating:
Comment:
C3: Does the report identify ALL electric vehicle companies in the market?
Not appropriate. This checklist item is unverifiable in practice, as there is no authoritative ground truth for identifying all electric vehicle companies. also sets an impossible standard, since completeness across the entire market cannot reasonably be expected from single report. A more suitable criterion would focus on specific entities mentioned the query (e.g., Tesla, BYD, Volkswagen, Ford) rather than requiring exhaustive coverage.
Rating:
J.3 REMARKS AND TIPS
Each deep research query should be rated once for its corresponding qid, while each checklist item should be rated individually. Several tips for annotation: (1) avoid checklist items that demand unverifiable completeness, such as those asking for "all" "every" entity fact; (2) watch for redundancy, where multiple criteria check essentially the same requirement; (3) be cautious of scope creep, in which checklist item evaluates something not actually specified in the query; (4) exclude subjective judgments; checklist items should focus on verifiable facts rather than opinions.
K RESULTS BREAKDOWN
reports the breakdown of the ten presentation checklist items (Table for each system. High-level organization largely under control: models perform well on P1 (section-level structure), and straightforward behaviors such placing citations at clause/sentence boundaries (P7) and ensuring figures/tables contain valid entries (P8) are near-saturated. The harder cases are the mechanics of style and bibliographic control: stylefformatting (P2, P5-P6, P9) and citation management (P3-P4, P1O) show wide variance across systems. We also observe provider-level regularities: sibling models tend share strengths and weaknesses (e.g., on P6, GPT-5 GPT- 4.1 GPT-5 Mini score 31 %, 32%, 24%), likely due to shared data and formatting designs. System design can shift these profiles: multi-agent wrappers around common backbone often improve P4 by enforcing explicit aggregation/mmaping from in-text citations to references, whereas numbering accuracy in P10 varies by pipeline, with some single-agent web systems exhibiting steadier numbering, Overall, consistent pattern emerges across families: high-level coherence and simple placement/validity are easy; citation correctness, style uniformity, grammar, and global formatting discipline remain the hard parts
Figure 22
47

---

Published as a conference paper at ICLR 2026
P1: Stricture & Coherence
P2: Grammer & Spalling
P2: Referance Completaness
PE: Ciaation Complatenses
PD: Format Condestency
PID: Citation Numbering
Singleagant, Wal Search
Single-per: Derp Tessurch
NoBoagent Deep Boasrch
Fine-grained pass rates (PI-PIO) for the Presentation & Organization metric. Most systems are reliable on PI (Structure & Coherence) and near-saturated on P7 (Citation Placement) and P8 (Figures & Tables Validity). The largest dispersion appears in grammar, citation style/formattin, and reference alignment (P2-P6, P9, P10). Models from the same provider exhibit correlated patters;; pipeline design (e.g., multi agent wrappers) can help
Figure 22:
tests.
48


Reviews:
[{"review_id": "bcqtOVVqoG", "reviewer": "Reviewer_m18a", "summary": "This paper addresses LiveResearchBench, a benchmark of 100 expert-curated, time-varying tasks designed around four principles: user-centricity, unambiguous scope, temporal dynamism, and multi-faceted reasoning. The authors further propose DeepEval, an evaluation suite measuring presentation, factual consistency, coverage, analytical depth, and citation reliability through rubric-tree scoring and ensemble LLM-as-a-Judge evaluation. Evaluations on 18 single- and multi-agent systems show existing models excel at information collection but fall short in analytical depth and citation faithfulness.", "questions": "(1) How stable are the system rankings across repeated runs or different judge ensembles? \n(2) Could you provide inter-judge agreement or human–LLM agreement for each evaluation dimension separately?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) The paper is well-motivated by the lack of realistic, dynamic benchmarks for deep research. LiveResearchBench is constructed with clear design principles—user-centric, unambiguous, time-varying, and multi-faceted—and backed by a solid six-stage curation and five-stage verification process. The use of clarification questions, checklist-based unit tests, and expert validation ensures task clarity and evaluation consistency.\n\n(2) DeepEval provides a six-dimensional framework that goes beyond accuracy to assess coverage, analytical depth, and citation reliability. The rubric-tree and checklist design make LLM-as-a-Judge both scalable and interpretable, achieving strong alignment with human evaluation.\n\n(3) The evaluation and analysis are comprehensive, offering insights into the current bottlenecks and future directions for deep research agents.", "weaknesses": "One of the benchmark’s core strengths—its dynamic, time-varying nature—also poses practical challenges. Time-sensitive webpages and evolving content make long-term maintenance difficult. It would be helpful if the authors could provide evidence or analysis showing the benchmark’s stability and validity over time.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ifNbh6hCgv", "reviewer": "Reviewer_M3tr", "summary": "This paper proposes **LiveResearchBench**, a benchmark for evaluating *Deep Research* agents. It comprises **100 expert-curated, time-sensitive, multi-faceted tasks**, each specified with a clear audience, scope, and expected output to minimize ambiguity. The authors also present a comprehensive evaluation suite (DeepEval) covering six dimensions that span both report-level and content-level assessment. The paper evaluates 18 agent systems and reports a range of observations about the current agents.", "questions": "- (Some questions are already mentioned in the weakness section)\n- The paper mentions a very large time investment (1500 hours), but it is not clear what is included (task ideation, rubric creation, judge prompts, code, pilot runs, re-annotation, etc.). A breakdown would help much to understand the number.\n- DeepEval appears to involve many LLM calls. What is the end-to-end cost and latency to evaluate a single system on the full benchmark? \n- Will the benchmark be fully opensourced? (including the tasks, task-specific checklists, code, etc.)\n- I'd suggest including a summary table (ideally in the main paper) contrasting LiveResearchBench with representative benchmarks across key dimensions. The current writing is not clear enough and will be hard for people to comprehend.\n- OpenAI Deep Research seems more like a multi-agent system, given the information revealed?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- While there are a surge of new deep research benchmarks, this work remains different from vey recent works (e.g., unambiguous and time varying research tasks). \n\n- The paper includes a user study and human-agreement check for parts of the evaluation pipeline. These are important for the benchmark design as well as the reliability of the evaluation.\n\n- Comprehensive evaluation on most existing frontier systems.", "weaknesses": "Some high-level points:\n\n- Although the benchmark aims to evaluate open-ended research and report generation, the tasks still appear biased to tasks that are over-prescribed (explicit audience, formatting requirements, and tight scopes). This is understandable for the convenience of the evaluation, but still undermines its claimed scope.\n- The evaluation is central to the contribution, yet the paper does not clearly explain how each of the six metrics was designed, what prior work they extend, and why these particular choices were made. \n- The evaluation heavily depends on LLM-as-a-Judge, which could make it both expensive and unreliable. Given this, the details of human agreement studies should be further detailed in the paper.\n- The paper defines six metrics but the main table averages only on four of them? Why omit **Citation Accuracy** in the main comparison when accurate grounding is arguably the linchpin for user need? This is particularly concerning given the text itself notes that models struggle most with **citation correctness/formatting**. Even more worrying, the method that tops the main table (Open Deep Research) is reported to have extremely high overall error rates on some tasks\n\nRegarding the individual metrics:\n\n- Report-presentation metric may have overweighted style and citation formatting (and actually many of them are directly related to citation formats). If the presentation score is driven by citation entries, duplicates, or formatting, it's very likely an agent can overfit to style easily. Might as well elaborate more about how the 10 error patterns were selected and discuss the risk of RL/finetuning leading to superficial gains on these patterns.\n- Factual & Logical Consistency at scale. Longer documents are more likely to contain contradictions, but they also become harder for an LLM judge to evaluate reliably. This may be a challenge for the judge model.\n- Analysis Depth: Looking at the prompt, it looks too heavy for the LMs to reason accurately over all the points.\n- Citation-related two: \n  - It is **unclear how claims are extracted** from long reports and how accurate this extraction is (likely another LLM call; but now the writing in the main text, Appendix D, the figure are not clear enough). This step is crucial and should be evaluated itself given that the extraction could have been blocked by the websites.\n  - The **rubric-tree framing** of citation accuracy feels unnecessary for what is essentially a simple three-step pipeline\n\nWriting:\n\n- Writing should be further polished. Some details are hard to comprehend and some details in the appendix appear to be made in a rush. The references are also loose and arbirtrary.\n- In the relatde work section, the \"Evaluation of Long-form Answer\" part should discuss more about how previous (search-related) benchmarks handled long-form answers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MpfB3C0Kzf", "reviewer": "Reviewer_wzms", "summary": "This paper introduces LiveResearchBench, a benchmark designed to evaluate deep research agents that produce comprehensive, citation-grounded reports through multi-source web investigation. The benchmark consists of 100 expert-curated, realistic, and dynamically updating tasks across domains such as daily life, enterprise, and academia, constructed under three principles: user-centricity, dynamism, and unambiguity. Alongside it, the authors propose DeepEval, a multifaceted evaluation suite assessing report quality through coverage, presentation, citation accuracy, and analytical depth. Empirical evaluation across single- and multi-agent setups highlights limitations in current deep research capabilities.", "questions": "1. The clearly defined specific questions are not practical. In reality, are people always asking such specific questions?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The experiment is extensive, including 17 deep research agents' performance on the proposed benchmark.\n2. The finding that existing deep research agents struggle with citation reliability and analytical depth is useful.\n3. The checklist design is interesting.", "weaknesses": "1. The novelty is not enough and the description of related work is not precise. E.g., Mind2Web2 is ambiguous and focuses on short answers, etc. This makes the distinction from the existing work not convincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q63smI9qmj", "reviewer": "Reviewer_Mkd1", "summary": "This paper presents LiveResearchBench, a new 100-task benchmark for \"deep research\" agents, and an evaluation suite, DeepEval. The authors use this framework to test 18 different systems, offering some useful insights into their common failure points. \nOverall, this is a valuable and thoughtfully designed contribution to the field.", "questions": "1. How do you define `wide information search`? Any idea related to the existing Wide Research?\n2. How did you handle potentially harmful queries or unreliable web sources during the evaluation?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper’s main strength is its design philosophy. The authors do an excellent job motivating why this kind of evaluation needs to be user-centric, dynamic, and unambiguous, and they've built the tasks accordingly. The benchmark itself feels realistic, with tasks that specify an audience and output format while requiring up-to-date information, which is a convincing improvement over prior work. \nThe breadth of the DeepEval suite is also good. It combines high-level checks with more granular metrics, like checklist-based coverage and a rubric-tree for citation validation, which seem to align well with human judgment. Evaluating 18 systems provides a solid snapshot of the current landscape and surfaces recurring errors that will be helpful for the community.", "weaknesses": "My main issue is the evaluation pipeline's heavy reliance on LLMs for generating checklists and judging outputs. While the authors took steps to mitigate bias, it's still a risk. I'd be more convinced by more quantitative stats on model-human agreement or some ablation studies on the judging process.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)