ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agents
Authors: Manasi Sharma
Chen Bo Calvin Zhang
Chaithanya Bandi
Clinton Wang
Ankit Aich
Huy Nghiem
Tahseen Rabbani
Ye Htet
Brian Jang
Sumana Basu
Aishwarya Balwani
Denis Peskoff
Marcos Ayestaran
Sean M. Hendryx
Brad Kenstler
Bing Liu
Venue: ICLR 2026 Poster (2026)
Keywords: Deep Research
Large Language Models
Benchmarks
Rubrics
LLM-as-a-judge
Multi-step Reasoning
Cross-document Synthesis
Long-form Question Answering
Evidence-based Reasoning
Evaluation Frameworks
Natural Language Processing
Primary area: datasets and benchmarks

Abstract: Deep Research (DR) is an emerging agent application that leverages large language models (LLMs) to address open-ended queries. It requires the integration of several capabilities, including multi-step reasoning, cross-document synthesis, and the generation of evidence-backed, long-form answers. Evaluating DR remains challenging because responses are lengthy and diverse, admit many valid solutions, and often depend on dynamic information sources. We introduce ResearchRubrics, a standardized benchmark for DR built with over 2,800+ hours of human labor that pairs realistic, domain-diverse prompts with 2,500+ expert‑written, fine‑grained rubrics to assess factual grounding, reasoning soundness, and clarity. We also propose a new complexity framework for categorizing DR tasks along three axes: conceptual breadth, logical nesting, and exploration. In addition, we develop human and model-based evaluation protocols that measure rubric adherence for DR agents. We evaluate several state‑of‑the‑art DR systems and find that even leading agents like Gemini's DR and OpenAI's DR achieve under 68% average compliance with our rubrics, primarily due to missed implicit context and inadequate reasoning about retrieved information. Our results highlight the need for robust, scalable assessment of deep research capabilities, to which end we release ResearchRubrics (including all prompts, rubrics, and evaluation code) to facilitate progress toward well‑justified research assistants.

Published as a conference paper at ICLR 2026
RESEARCHRUBRICS: A BENCHMARK OF PROMPTS AND RUBRICS FOR EVALUATING DEEP RESEARCH AGENTS
Manasi Sharma -1, Chen Calvin Zhang Chaithanya Bandi 2 Clinton Wang Ankit Aich 1, Huy Nghiem 2, Tahseen Rabbani , Ye Htet 4, Brian Jang 1 Sumana Basu 5, Aishwarya Balwani ' Denis Peskoff 6 Marcos Ayestaran 1 Sean M. Hendryx 1
Brad Kenstler! ', Bing Liu
Sccle AI University of Maryland *Washingoon University, St. Louis Work conducted while at Scale AI
University of Chicago
*MeGill University
UUiiveriiyy of California, Berkeley
ABSTRACT
Deep Research (DR) is an emerging agent application that leverages large language models (LLMs) to address open-ended queries. It requires the integration of several capabilities, including multi-step reasoning, cross-document synthesis, and the generation of evidence-bacee backed,, long-form answers. Evaluating DR remains chal- lenging because responses are lengthy and diverse, admit many valid solutions, and often depend on dynamic information sources. We introduce standardized benchmark for DR built with over 2,800+ hours of human labor that pairs realistic, domain-diverse prompts with 2,500+ expert-written, fine-grained rubrics assess factual grounding, reasoning soundness, and clarity. We also propose a new complexity framework for categorizing DR tasks along three axes: conceptual breadth, logical nesting, and exploration. In addition, we develop hu- man and model-based evaluation protocols that measure rubric adherence for DR agents. We evaluate several state-of-the-art DR systems and find that even leading agents like Gemini's DR and OpenAl's DR achieve under 68% average compli- ance with our rubrics, primarily due to missed implicit context and inadequate reasoning about retrieved information. Our results highlight the need for robust, scalable assessment of deep research capabilities, to which end we release RE- RearchRUrrIcS (including all prompts, rubrics, and evaluation code) to facilitate progress toward well-justified research assistants.
INTRODUCTION
An exciting development in the growing capabilities of large language models (LLMs) is the emer- gence of Deep Research agents: autonomous LL.M-based systems that conduct multi-step web exploration, targeted retrieval, and synthesis answer open-ended queries. Industry leaders have begun deploying such systems (e.g., OpenAl's "Deep Research" OpenAl (2025a) and Google's "Gemini Deep Research" Google (2025)), which have demonstrated strong performance on certain benchmarks (for instance, scoring 26.6% on the expert-level HLE benchmark Phan et al. (2025)). However, evaluating deep research agents poses significant challenges. Deep Research (DR) tasks are inherently open ppeneended: they require reasoning across multiple documents, often with no single "correct" answer, and their outputs can be long and varied. Consequently, existing evaluation methods fall short. Typical QA benchmarks, both general Yang et al. (2018); 8); Mialon et al. (2023); Phan et al. (2025); Krishna al. (2025) and deep research specific Java et al. (2025); Coelho al. (2025), focus on short, casily-verifiable factual answers and do not capture the foomm multi-source synthesis
"Correspondence manasi ..w.rhaaaaa... rma@sca com.

---

Published as a conference paper at ICLR 2026
required by DR, e.g., "Which material has band gap 0.9 eV, dislocation density 4 10 2?" with the unique answer "Gallium nitride (GaN)".
Octa
Jakyo
lonvry
snseread Priae for
Madi
Doceaiivily doemain Adalyske
Cherr soce for
Haran gelige
1PA
Teay Fackrtss
lipert our mate di
Podlcc Densoort Pocosee
LLM Judge
+2
Figure 1: Overview of RESEARCHRRBRICS and its evaluation pipeline.
Several recent efforts to benchmark deep research agents directly have also revealed important limitations: for example, some benchmarks introduce LLM-generated rubrics and evaluation metrics reliant upon LLM-generated reference reports Du et al. (2025), thus raising concerns about circularity and limited oversight Dorner et al. (2025), while others are far more narrow in their scope, assessing only one specific angle of research in a technical domain (e.g., generating a "Related Works" section) Patel et al. (2025); Li et al. (2025); Wan et al. (2025). In practice, however, users direct deep research systems toward broad array of everyday topics, ranging from business reports to consumer-related queries, underscoring the need for benchmarks that combine domain diversity with expert-authored, fine-grained rubrics.
To better characterize these challenges and motivate our approach, we introduce a task complexity framework for deep research. Each query can be described along three independent axes: (1) its conceptual breadth (the number and diversity of distinct topics or domains involved), (2) its logical nesting depth (the number of reasoning or decision steps required, including sub-questions and conditionals), and (3) its exploration level (the degree of open-endedness or underspecification of goals). This tri-axial view highlights how DR queries differ from simpler QA tasks and helps articulate the shortcomings of existing methods: simple QA benchmarks lack sufficient breadth, depth, and exploration, while many current DR benchmarks fail to cover this full, multi-axial complexity.
We introduce RESEARCHRUBRICS which pairs realistic, diverse prompts with expert-authored, fine- grained rubrics for deep research. We curate queries from nine broad domains (including business planning, historical analysis, technical documentation, and common consumer questions) to reflect real-world use cases. Each prompt comes with a detailed rubric: in total, we provide 2.593 rubric criteria that check factual grounding, coherence of reasoning, completeness, relevance, and clarity of the answer. The benchmarks also include negative rubrics that specifically aim to penalize extraneous or incorrect content. Importantly, all rubrics are written and reviewed by human experts (not auto- generated), ensuring they capture nuanced, domain-specific requirements. We also develop evaluation protocols for both human and automated scoring. Following the LLM-as-judgee paradigm, we use powerful LLMs to assess rubric compliance, and we systematically experiment with improving this process comparing binary vs. ternary grading for each criterion and the level of detail in the rubrics. Finally, we apply our framework to leading DR systems (OpenAl's DeepResearch OpenAl (2025a), Google Gemini's 's Deep Research Google (2025), and Perplexity's 's Deep Research AI (2025)). The results show that even the strongest agents fall below 68% average rubric compliance. revealing substantial room for improvement in multi-document synthesis and rigorous justification.
Our contributions are as follows:
A human-crafted benchmark for deep research. We present RESEARCIIRURRICS, a suite of open-ended research tasks across diverse domains, each with an expert-written rubric (2,593 total criteria). Crucially, each rubric is both written and reviewed by humans, thereby mitigating potential anchoring biases that may arise when only verifying LLM-generated rubrics.
A task complexity framework. We formalize deep research queries along three axes breadth, depth, and ambiguity- to distinguish them from conventional QA tasks and to guide the construc- tion of balanced benchmarks that reflect real- wolld deep research queries.
2

---

Published as a conference paper at ICLR 2026
Table I: Comparison of RESEARCHRUBRICS with representative Deep Research benchmarks.
Benchmark
Human-authored Rubrics
Expert-Curated
Open-Ended Tisks
Technical Domains
L..Maar-Jurle
Rubrics
per task
BeoeemicBoosee Zhou all (2025)
K
RowseComp Wei (202))
ResearehBanchh
(2025)
ResearcherBescch Xa ct. (2025)
14
heepScholar-Reechh Patel (2025)
X
ReportBench Li et al. (2025)
X
X
Deepitesearch Bench Du (2025)
25
Wind2Weh2
LiveRssorhBeeech Wang (2025)
LiveDRBemch lava et al. (2025)
8
Expent regRench Ruan (2025)
16
DeepResearch Arena
al.
DeepReseachhemm
(2025)
SPOT San et al. (2025)
RESEARCHRUSRICS (Ours)
26
Rubric-based, open-ended evaluation. We introduce sitcome-based, fine-grained rubrics that provide rigorous evaluation of long-form research answers and closely align with expert judgments. We also separate mandatory (required for sufficiency) from optional criteria, addressing a key gap existing benchmarks.
Ternary Grading, We propose ternary grading scheme for rubrics-based benchmark that supports partial credit assignment, and examine its suitability for automated evaluation.
Rubric design impact on LLM-as-a-judge. We introduce practical recommendations for rubric design that improve agreement with human evaluators and are validated through ablation studies.
By releasing RESEARCHRUBRICS, aim catalyze progress toward trustworthy, well-justified DR assistants for complex, open-ended research tasks in multitude of domains.
2 RELATED WORK
Early benchmarks have largely taken two approaches: deriving constructing tasks from static corpora or relying on expert-curated questions.
Derived Benchmarks AcademicBrowse Zhou et al. (2025) and BrowseComp Wei et al. (2025) assess retrieval from academic papers or the web, while ResearchBench Liu et al. (2025) builds complex queries from static data. More recent work goes further and derives tasks from dynamic, real-world scenarios. DeepScholar-Bente writing using live queries from arXiv papers, though it is specialized to academic synthesis and automated metrics. ReportBench Li et al. (2025) leverages published surveys ground truth, measuring overlap with expert-written reviews but prioritizing replication. DeepResearch Arena Wan al. (2025) automatically curates 10,000 open-ended tasks from academic seminars, pairing them with adaptively generated rubrics, though automatic rubric generation can miss domain nuances.
Patel et al. (2025) evaluates systems on related work
Expert Curated Benchmarks Expert-authored benchmarks include Humanity's Last Exam (HLE) Phan al. (2025), which provides expert-written written short-answer questions across advanced domains, but does not target more ambiguous/open-endee analysis directly, and DeepResearch Bench Du al. (2025), which introduces 100 PhD-level problems requiring long-form reports, but struggles with critical weaknesses including LLM-generated rubrics for specialized domains, evaluation metrics reliant on LLM--gnneraed reference reports, and simplistic reference overlap metrics. Newer works such as Mind2Web2 Gou et al. (2025) and ResearcherBench Xu et al. (2025) extend this approach, but an effort to ease evaluation, either target narrowly defined domains (e.g.. only Al-related topics) or restrict the scope of the prompt that the agent-as-a-Judee framework can operate effectively with --geeeeratted rubrics. LiveResearch Wang et al. (2025) tries to focus on more realistic prompts but still relies on LLM- rubrics that are only human-reviewed, which may introduce an anchoring bias. Our rubrics, by comparison, are fully numan-writted and reviewed from the outset. ExpertLongBencch Ruan et al. (2025) is similar to our benchmark in that it targets expert-level, long-fone tasks across nine domains with domain-speciff rubrics. However, relies high-quality existing references for evaluation using the CLEAR framework, which limits

---

Published as a conference paper at ICLR 2026
Rubrnce
Expert
Expert
Pronge
Rewded Promot
beal colden
Firal Ficrpt
Inital Rubricc
Rorised Rubric
Dipert Enger and Copart auaapa prangt arcc Erpor1
The three-stage pipeline for creating and refining prompts and rubrics. An initial draft by Expert is iteratively improved with Expert 2 before a final review and adjustment by Expert 3.
Figure 2:
the scope of prompts highly academic professional queries (e.g., Clinical Note Generation, Legal Multi-Document Case Summarization, Molecule Description Generation), whereas we also include general consumer research queries. Additionally, our benchmark has much higher average number of rubrics per task than many of these benchmarks, resulting in superior evaluation granularity
To summarize, existing benchmarks thus face two primary limitations: reliance on static datasets or answer keys Liu et al. (2025); Li et al. (2025), and the use of non-expert or automated evaluation, including coarse metrics Patel al. (2025) or auto geooggerrrted rubrics Wan al. (2025); Du al. (2025); Gou et al. (2025). In contrast to these approaches, RESEARCHRUBRICS offers middle ground: realistic research queries (academic and everyday domains) paired with expert- expert-written rubrics assessing grounding, synthesis, reasoning, clarity, and citation usage. By using human-writted rubries with LLM judges, we avoid simplistic overlap measures while maintaining scalability. RE- SEaRCHRUBriCH complements efforts like ExpertLongBencch while emphasizing domain diversity, rubric quality, and focus on scep-research specific tasks.
3 OVERVIEW OF RESEARCHRUBRICS
RESEARCHRUBRICS consists of 101 single-turn prompts, each paired with a set of 20-43 prompt- specific rubric criteria. Every prompt and criterion in RESFARCHRUBRICS was written and iteratively refined by human experts to ensure clarity and relevance (no criteria were seeded or generated by LLMs). The prompts cover a wide range of topics and inquiry types to emulate real user questions that deep research agents receive. In total, the benchmark contains 2,593 unique rubric items, enabling a fine-graine assessment of open-ended, realistic research queries. Figs. 1 and 2 provide overview of our benchmark design and evaluation process.
Hitatiaa Aoa
3.1 DATA COLLECTION AND TASK DOMAINS
Basines Parning SauEacT
Our data collection pipeline consists of three expert participants, as shown in Fig. 2. In this context, we define an "expert" as an individual with a strong STEM background who is skilled in task design and evaluation, rather than a domain-specific spe- cialist for each prompt. All participants in our data collection only chose and worked on domains they were familiar with.
Thchnical Documentaiion
Crustie Willing
The pipeline involves three experts, each assigned
Figure 3: Distribution of task domains in our collected data.
to distinct and separate role. Expert 1 initially proposes a prompt and a set of rubric criteria. This proposal is then passed Expert 2 for review. Expert provides feedback and iterates with Expert
4

---

Published as a conference paper at ICLR 2026
1 until the pair is approved. Finally, Expert 3 conducts final, independent review and makes any last adjustments. This three-parte parteppattii setup ensures that each component is thoroughly reviewed multiple times, guarantecing high quality in the final data.
To ensure realism and variety. initial prompt ideas were drawn from user forums, Q&A sites, and brainstorming sessions, then adapted to represent the range of research- like questions deep reasoning agent might encounter. The result is a collection of prompts that span both breadth (a wide variety of domains) and depth (challenging multi-step problems).
For each finalized prompt, experts developed a detailed rubric specifying what an ideal response should include and which common errors to avoid, following the pipeline detailed in Fig. 2. We weighted each criterion based on its importance (see Section 3.3) and included negative criteria targeting likely pitfalls, such as factually incorrect statements, off-topic tangents, or disallowed content.
We curated prompts from nine broad categories (see Table 10 in the Appendix for a detailed description of each category) to maximize diversity. These range from technical documentation to historical analysis, creative writing, and current events.
Fig. 3 shows the distribution of categories in RESFARCHRUBRICS. The distribution is fairly even, with AI/MI and historical analysis queries constituting the largest portions closely, followed by domains like general consumer research, reflecting both specialized academic topics and everyday research questions. Other categories provide targeted challenges (e.g., creative synthesis or real-time news retrieval). This diversity ensures that DR agent must draw on wide range of knowledge sources and adapt to different task structures.
3.2 PROMPT COMPLEXITY DIMENSIONS
Not all research prompts are equal some involve broader knowledge base, others require deeper reasoning, and others are underspecified and exploratory. We categorize each RESEARCURURRI task along three orthogonal complexity dimensions: Conceptual Breadth, Logical Nesting Depth, and Exploration (Table 2). This framework helps ensure our benchmark covers a balanced mix of task types and allows analysis of where agents struggle most.
Table 2: Prompt complexity categories used annotate each task in RESFARCHRURRICS.
Complexity Level
Definition
Exnmple
Axis
Conceptnal Simple
Lawoves a siagle domain or topic; solvable using I prinary information source or conceptual framewock.
A math word probless or factual lookup froms one source.
Breadth
Muderate
Integrates 2-5 disinct subtopics or data sources that are weakly cospled; limited cross-domain ressoning.
A peonye combining two fields te-g- plysies concept applied medical device contest).
High
Requires sy synhess auross
> information sounces or clearly
"Analyse the environmenal, ecunomic, and pulitical factors affecting renewable
disjoint domains (e.g.. science, economics) ressoning depends multiple perspectives.
adeption in Asia."
Logical Nesting
Sholoow
Single-tep infermnce or direct retrieall answer derived from one
"What the esgital of X country" or a single lookup query.
reasoning operation query.
Intermediate Multi step reasoning (2 to 3 dopendent sub questions) where later
"Find the sales of Company A and Company B last year and desermine who grew faster. then idemiity reason for that difference."
steps depend earlier intermediatee resulss.
Deep
Requires dependeut rensoning steps hicruschical planning
"Develop evidence-bocked investment stritgyy given economic indicaters, stress against two historical scenaiias and suggest contingeccy plans."
(e.g- analysis syuthesis -a evalnation revision).
Exploration Low
Fully specified mambiguous prompt cantains explicit goals. constraints, and evalution enteria.
"Summarize metiodology of referenced papec." The uskk is clean-cut.
Medium
Modkratey open-ended (1-2 inspecifeed factors's requires
"Discuss the benefits ricks bealtheure""
limited prioiitizaionn snong known aspects.
standard themes (privacy. accuncy, etc.)
Undenpecifeed exploratory; key factors unspocified,
"I change careers something with strong
requiring chatiiicaiio of objectives or creative refirming..
growth what should I consider" The agens must chaify the criteri and explore multiple puths.
Every task in RESEARCHRRBRRIS is annotated with triplet of (Breadth, Depth, Ambiguity) labels. In our evaluations, we amalyze model performance across these dimensions to see, for example, if a model struggles more with breadth (integrating many sources) or with depth (long reasoning chains). This also helps researchers filter the benchmark for specific experiment focuses (c.g., testing only high-depth reasoning tasks).
5

---

of the positive weights (the maximum possible score the model can Published as a conference paper at ICLR 2026
3.3 RUBRIC DESIGN
RESEARCIRRURRCS is a rubric-based benchmark: each prompt is judged against a tailored set of criteria that define the requirements of a good answer. RESEARCHRUBRIC also separates mandatory (required for sufficiency) from optional criteria, addressing a key gap in existing benchmarks.
Table 3: Rubric criteria used to evaluate responses, with illustrative examples for cach category.
Criterion
Description
Example
Explicit Requirements
Checks whetber the addresses all points explicilly in prompt does correctly.
Promp:: "Compure X and and recommend one" The compares
relevant and recommendation.
Implicit Requirements
Covers points that a well informed penson would expect. directly nsked. Encourages completeness and undensanding.
Prumpi: "Explain medical treatmen."" - A good answer also requested.
mesticus side effects
Synthesis of Information
Evaluates whether the model connects and synthesizes sub parts of
Pronge "Summarize several stadies on renewable energy adoption."
information multipp query, rather merely listing
The idemitiies overurching trends draws integrated conclusions.
Use of References
Assesses inclusion and uppropriaeness of cinations or evidence where expected. Checks references specific, relevant, actually support clains.
Prompt: "Summarize recent findings on large language models." The cites key pupers (e "Aitention is All You Need and links claims sources.
Communication Quality
Evaluttes charity organization, and tume. A response may factually but still poor discrganized misaligned with
Prompi: "Write a sbort blug post for general andiennee"" - The logically stractured, concise, and avoids jargon.
needs.
Instruction F Following
Checks adherence explicit user instructions oc constraints g., required format, exclusions).
Prompe: "Sammarize this withous memioning Togic 2." The answer umits Topic Z instuused.
Table 3 presents the six broad evaluation axes used to assess response quality, Each axis contains multiple rubric criteria, which are categorized either mandatory or optional.
Mandatory criteria define the minimum requirements for valid response, i.e., core elements that must be satisfied for the answer to considered correct or adequate.
Optional criteria capture desirable but non-essential qualities ("nice-to-have" behaviors) that distinguish strong responses from merely sufficient ones.
Each criterion assigned a numerical weight in the range [-5, 5]. reflecting its relative importance. Weights of #4 or +5 correspond to mandatory criteria, while criteria with weights in [-3, 3] are optional. Positive weights reward the presence of valuable attributes, while negative weights penalize common failure modes such factual inaccuracies, irrelevance, verbosity. These weights aligned with a calibrated human preference scale (Table 7) spanning six levels, from Critically Detrimental to Critically Important. This mapping encourages more consistent human-model agreement during grading.
3.4 EVALUATION METHODOLOGY
Each model response is evaluated against all the rubric criteria using a model as a grader, in an LLM-as-a-judge setup. The model-based grader outputs temary judgment verdicts for each rubric, which are (Satisfied Partially Satisfied Not Satisfied), This scoring process the same for negative criteria, which are phrased that the negative weights applied the sum the negative criteria are met. The final task score is the weighted sum of all positive and negative weights, normalized by sum achieve).
if is satisfied, 0.5, if r3 is partially satisfied, (1) if r; is not satisfied,
1,
Er,ec Wr, Wr. Mr. Sk En,e, w. w. >0 Wr.
m, Judge( Pk, Res, rz)
where Sy is the final task score for the task with prompt Pk and model response Res. C' is the set of all criteria, Wr, is the (possibly negative) weight assigned to criterion Piss and my, is the ternary indicator returned from the model- based judge, Judge('s :)), representing the level of satisfaction for criterion ri.

---

Published as a conference paper at ICLR 2026
To calculate the breakdown of failures per rubric category in average task, we employ the following formula (where a failure is only when a rubric receives a Not Satisfied verdict).
F
Σ fat TE..
Σ
Maill,
(2)
Tel
Tel
Mail
TE..
where f'aill, ett the number of failed rubrics from category e task i, Afaall, x is the total number of failed rubrics across all categories in task I, se,t is the failure rate of category within task 4, Ta is the set of tasks in which category с occurs at least once, and F is the average failure rate of category c across tasks.
This allows to understand that when rubrics fail, which categories responsible for the highest contribution of failures in an average task (as opposed to just how often rubrics from a certain category fail). An important feature note is that since the failure rate breakdown is averaged across only those tasks in which those rubric categories occur (to minimize the effect an imbalanced rubric category distribution), the failure rate ratios do not necessarily add up to 1.
Human Consistency Analysis Similar to HealthBench Arora et al. (2025), we utilize the Macro F1 score to validate the effectiveness of using a model-based grader as proxy for human judgment. In our setup, we compare the ground truth judgement of experts and model-based graders for each task, and compute the Fl scores for each the classes (Sat (Satisfied, Partially Sat Satisfied. Not Satisfied).
Fi = 2. precision recall precision + recall
TP
TP
where precision
TP + FP and recall
(3)
TP + FN
where TP, FP,, and FN are the True Positive, False Positive, and False Negative values, respectively, We also run ablation studies to isolate the most significant factors in the level of alignment between the model-based grader and human judgments. For more details, see Section 4.4.
EXPERIMENTAL RESULTS AND ANALYSIS
We evaluate three commercial Deep Research (DR) agents on RESEARCHRUBRICS to measure their capabilities across multi-step synthesis, implicit reasoning, and evidence-backed justification. Our benchmark introduces 2,500+ expert-written rubric criteria across 100+ prompts, providing more granular evaluation than existing frameworks. This granularity enables atomic-level quality assessment that allows us to identify specific failure modes invisible to coarse-grained metrics.
4.1 EXPERIMENTAL SETUP
Evaluated Systems We benchmark OpenAl Deep Research OpenAI (2025a), Gemini Deep Re- search Google (2025), and Perplexity Deep Research AI (2025). Each system produces struc- tured PDF reports that we convert to markdown for evaluation across six dimensions: Explicit Requirements, Implicit Reasoning, Synthesis of Information, References, Communication Qual- ity, and Instruction Following. Our evaluation employs both binary (met/not (tts:/nottettt)) and ternary (fully/ps iarartilll iiallll ciiilll//nee lynoo satisfied) grading schemes understand the impact of partial credit on system rankings.
ALM-as-Judge Implementation We deploy three state-of-the-aa LLMs automated judges: GPT- 5 OpenAl (2025b), Claude-Sonnet-4.5 Anthropic (2025), and Gemini-2.5-PPa DeepMind (2025). Under binary grading, we collapse Partially Satisfi: isii verdicts to Not Satisfied mea- suring strict compliance. Homan--oode alignment is quantified using Macro Fj scores, with nine expert annotators providing ground truth across 303 responses.
Table 4: Scores
Overall Compliance
4.2 MAIN RESULTS
Compliance Scores Table 4 reveals that current system exceeds 70% rubric compliance, with the bert-performing Gemini DR achieving only 67.7% under ternary grading and
Model
Ternary
Binary
Gemini DR
0.677
0.615
OpenAl DR
0.664
0.597
Perplexity DR
0.566
0.487
7

---

Published as a conference paper at ICLR 2026
61.5% under binary evaluation. This aligns with findings from LiveResearchBench. where leading systems score below 74% on comprehensive metrics, DeepResearch Bench, where leading sys- tems score below 50% comprehensive metrics. The consistency across benchmarks suggests fundamental architectural limitations rather than benchmark-specific challenges.
Failure Rates Fig. 4 decomposes failure rates across evaluation dimensions, revealing that implicit reasoning and synthesis jointly account for 45-50% of all failures. This corroborates the findings in Multi-Agent System Taxonomy (MAST) Cemri et al. (2025), identifying reasoning- mismatch (13.98%) and disobedience of task specifications (10.98%) systemic issues. While agents excel explicit factual retrieval and communication quality (failure rates below 20%)., they consistently fail infer unstated requirements or integrate multi-document evidence into coherent arguments.
54
49.2
49.0
45.0
Tatreitata (3)
28.3
24.2
25.4
25.0
23:7
16.4
19.0
19.2
36.6
15.6
16.5
36
16
11.7
13.7
12.9
Poppeds
Exiic
Ingict Crtarts
Systheb inforaaioo
Fabrecm
Communiaaionn Ocn
Inpruction buing
Figure 4: Rubric-axis failure rates across Deep Research agents. Dark bars represent ternary grading; light bars show binary grading. Implicit reasoning and synthesis show markedly higher failure rates compared communication quality and references. The pattern holds across all three systems, indicating architectural rather than implementation limitations.
Mandatory Optional Criteria RESEARCHRURRICS separates mandatory and optional criteria, and using this differentiation, we observe (from Fig. 6) that, while mandatory criteria drive failures in explicit requirements and synthesis of information, optional criteria account for most implicit reasoning failures. This suggests current systems meet basic implicit requirements but miss nuanced quality indicators that distinguish professional from adequate research.
This finding contextualizes HealthBench's 's worst-at- 16 analysis showing 33% performance degrada- tion from average to minimum systems achieve moderate average scores by satisfying mandatory criteria while systematically missing optional quality dimensions. The mandatory/optional distinction proves essential for deployment decisions: a 60% overall score might indicate either dangerous gaps in core requirements or merely missing polish on otherwise solid foundations.
Performance Stratified by Complexity Dimension Fig. 5 presents model compliance scores stratified by conceptual breadth, logical nesting, and exploration level under binary and ternary grading schemes, respectively. Gemini DR consistently leads, achieving roughly 70% average rubric compliance across most complexity tiers, followed closely by ChatGPT DR, and Perplexity DR lagging slightly behind. A clear pattern emerges: performance degrades monotonically with increased logical nesting depth. Whereas shallow reasoning tasks (single-hop or two-step queries) are handled well, multi-step analytical evaluative problems see sharp drops, particularly for models relying sirievall-entric architectures. Conceptual breadth also correlates with difficulty, though less steeply; systems handle multi-domain synthesis better than extended inferential chaining.
Citation Analysis The implicit reasoning gap explains the breadth-accuracy trade-off documented in citation analysis: Gemini DR produces 111 citations with 81% accuracy while Perplexity achieves 90% accuracy with only 31 citations. Systems optimized for comprehensive coverage sacrifice precision, while those targeting accuracy miss crucial perspectives neither strategy successfully handles the implicit judgment of source relevance and authority.
8

---

Published as a conference paper at ICLR 2026
603
140
120
Vederate High
intamadiins
Deep
High
221
01.8
Becraje Score
Concepturl Sroaath
Log Kat Nesing
Experation
Figure 5: Performance across Conceptual Breadth, Logical Nesting, and Exploration (Ternary Evaluation)
$4.0
54
68.8
36.7 39.2
$6.8
35.3
27.5
20.3
17.2
Figure 6: Failure rate stratification by criterion importance. Mandatory criteria show systemati- cally higher failure rates across most dimensions, with the notable exception of implicit reasoning, where optional criteria failures dominate. This inversion suggests implicit requirements primarily distinguish excellent from merely sufficient responses. Dark bars represent ternary grading; light bars show binary grading.
4.3 HUMAN-LLM JUDGE ALIGNMENT FOR AUTO-EVALUANT
Our human evaluation study (Table 5) demonstrates that binary grading achieves substantial agree- ment (0.72-0.76 Macro Fi). approaching the best-performing LLMM-juges for rubrics benchmarks in recent literature. The shift from ternary to binary evaluation increases agreement by approxi- mately 20 percentage points, confirming that partial credit introduces ambiguity without improving discriminative power.
Table Human consistency with LLM judges. Macro F1 scores between human annotators and automated evaluation across grading schemes and judge models.
Agent
Judge Model Agreement
GPT-5
Claude-4.5
Gemini-2.5-Pro
Perplesity DR
0.717
0.718
0.724
Binary
Gemini DR
0.732
0.741
0.760
OpenAl
0.742
0.721
Perplesity DR
0.538
0.528
0.559
Teraryy
Gemini DR
0.532
0.567
OpenAl DR
0.546
0.527
0.557
The consistency levels validate automated evaluation feasibility for RESEARCHRUBRICS'S 2,593 criteria, exceeding HealthBench's 's 0.709 Macro Fi score. Gemini-2.5-Pe emerges the most reliable judge, achieving 0.76 agreement on binary grading, though at least the 12-17 17 percentage point gap best human agreement indicates remaining room for improvement.

---

Published as a conference paper at ICLR 2026
4.4 RUBRIC DESIGN IMPACT
To better understand how rubric design impacts evaluation reliability, we conducted a series of ablation studies focusing
key factors: (1) the inclusion of concrete examples within rubric criteria, and (2) the use of LLM-based augmentation to automatically rephrase those criteria. The goal of these experiments was measure how such modifications affect alignment between automated LLLM-as-judee and human evaluations. We present the results of the ablation study in Table 6.
We began with the original, expert-authored rubrics as our control condition. Example Detail tests whether providing brief, inline examples for each criterion improves agreement between human and model judges (in the format "(e.g., examplel, example2, example3)") The "Low" condition uses minimal guidance (the baseline criteria only), whereas "High" includes short, task-relevant examples (e.g., cited study, policy name, relevant item). LLM Augmentation evaluates whether prompting a large language model to automatically expand rephrase rubric text adds clarity. In the "Absent" setting, rubrics are the original ouman-written ones; the "Present" setting, each rubric was rewritten by an LLM with added qualifiers and examples.
We find, in Table 6, that including concrete examples within rubric criteria improves alignment by 3-4% (binary) and 2-3% (ternary). However, LLM-based rubric augmentation, i.e., automatically expanding criteria with synthetic elaboration, catastrophically degrades alignment by 15-20 %.
Table 6: Impact of rubric design on evaluation reliability. Adding examples improves human-L.LM alignment while automated augmentation degrades it.
Agent
Example Detail
LLM Augmentation
Low
High
Absent
Present
Perplexity
0.724
0.724
0.508
Binary
Gemini DR
0.733
0.760
0.760
0.564
OpenAl DR
0.709
0.721
0.721
0.528
Perplexity DR
0.523
0.559
0.559
0.371
Termary
Gemini DR
0.539
0.567
0.567
0.417
OpenAI
0.557
0.557
This finding challenges assumptions about verbosity improving clarity. Human-authored con- cise rubrics with targeted examples outperform machine-generater verbose descriptions, likely because augmentation introduces semantic drift and emphasis distortion. The implication for RE- SEARCHRUBRICS' 2,593 criteria is clear: expert curation cannot be replaced by automated expansion, and clarity emerges from precision rather than elaboration.
5 CONCLUSION AND FUTURE WORK
We introduced RESEARCHRUBRICS, new benchmark and evaluation framework for deep research agents that emphasizes fine-grained, human-aligned assessment. Through 101 diverse research challenges and exptrt-written rubric criteria, our benchmark provides a multi-dimensional lens on an agent's performance-chess source usage, and clarity of its responses. RESEARCHRUBRICE 's granularity enables us to identify specific capability gaps invisible aggregate metrics, and the mandatoryloptional distinction gives us a way to place an agent on the sufficiency effficinncy-exeelleeeee continuum, aiding deployment decisions by focusing on minimum viable performance rather than average scores. Our experiments reveal that today's 's best agents achieve only around 67% compliance with these rigorous rubrics, often falling short in integrating information across documents and providing well-justified answers with proper citations. Most critically, our findings suggest that improving Deep Research agents requires architectural innovation rather than incremental refinement: systematic failures in implicit reasoning. multi-documeet synthesis, and sustained sequential reasoning point fundamental limitations in how current systems represent and manipulate complex information structures.
not just factual recall, but the completeness, reasoning soundness,
10

---

Published as a conference paper at ICLR 2026
REFERENCES
AL. Introducing perplexity deep research, 2023. URL /
/hub/ blog/ hotg/intruuuc no- perplerpppe ex -eep Accessed. 2025-09-18.
Anthropic. Claude sonnet 4.5, 2025.
URL https://ww..nntrrooo..cooom
R. K. Arora, Wei, S. Hicks, P. Bowman, J. Quinonero-Candela. F. Tsimpourlas, M. Sharman, M. Shah, A. Vallone, A. Beutel, J. Heidecke, and K. Singhal. Healthbench: Evaluating large lan- guage models towards improved human health, 2025. URL https: https://: //dppps:://..../// wwwp:/o....ooooo.ooo/eeoooo 48550/ arXiv. 2505. 08775.
M. Cemri, M. Z. Pan, S. Yang, L. A. Agrawal, B. Chopra, R. Tiwari, K. Keutzer, A. Parameswaran, D. Klein, K. Ramchandran, M. Zaharia, J. E. Gonzalez, and Stoica. Why do multi-agent 1lm systems fail? arXiv preprint arXiv:2503. 13657, 2025. URL https: http::://axxvv.oro////o wwtp:/iiorr//aa....../aa 2503. 13657.
Coelho, J. Ning, J. He, K. Mao, A. Paladugu, Setlur, Jin, J. Callan, J. Magalhães, B. Martins, and C. Xiong. Deeprescarchgym: A free, transparent, and reproducible evaluation sandbox for deep research, 2025.
G. DeepMind. Gemini 2.5 pro: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. https://store googleap.p...ooooooo com/ deepmind meddi gemi ni ///g//nniii....... sar_i__________________ pddf, 2025. Accessed: 2025-09-25.
F. E. Dorner, V. Y. Nastl, and M. Hardt. Limits to scalable evaluation at the frontier: Lim as judge won't beat twice the data. In The 13th International Conference on Learning Representations (ICLR) 2025, January 2025. URL https://arxii. wttp:////ab/a....o/aa///// 13341.
M. Du, B. Xu, C. Zhu, X. Wang, and 7. Mao. Deepresearch bench: A comprehensive benchmark for deep research agents. arXiv preprint arXiv:25006. 1763, 2025.
Google. Gemini deep research your personal research assistant, 2025. URL httph:://gmmmiiii google/ beogge//veerii wwwp/vewwew//ee//ee/ooe/oeeeeett wwwprrviwwwdeeeereeeeeeeeeeeeeee
Accessed: 2025-09-18.
J. Gottweis, W.-H. Weng, A. Daryin, T. Tu, A. Palepu, P. Sirkovic, A. Myaskovsky, F. Weissenberger, K. Rong, R. Tanno, K. Saab, D. Popovici, J. Blum, F. Zhang, K. Chou, et al. Towards ai co-scientist. arXiv preprint arXiv:2502. 1 18864, 2025.
B. Gou, Z. Huang, Y. Ning, Y. Gu, M. Lin, W. Qi, A. Kopanev, B. Yu, B. Gutiérrez, Shu, C. H. Song. J. Wu, S. Chen, H. N. Moussa, T. Zhang, J. Xie, Y. Li, T. Xue, Z. Liao, K. Zhang, B. Zheng, Z. Cai, V. Rozgic, M. Ziyadi, H. Sun, and Y. Su. Mind2web 2: Evaluating agentic search with agent-as-a-judge In The Thirtty-inth Annual Conference on Neural Information Processing Systems, Datasets and Benchmarks Track, 2025. URL https::/p///ccccc.t. openreview. net/ forum? Ad=AUaW6DS9si.
Y. Huang, Y. Chen, H. Zhang, K. Li, M. Fang. L. Yang. X. Li, L. Shang, S. Xu, J. Hao, K. Shao, and Wang. Deep research agents: A systematic examination and roadmap. arXiv preprint arXiv:2506. /8096, 2025.
A. Java, A. Khandelwal, S. Midigeshi, A. Halfaker, A. Deshpande, N. Goyal, A. Gupta, N. Natarajan, and A. Sharma. Characterizing deep research: benchmark and formal definition. arXiv preprint, arXiv:2508.04138 04188,,,,,,,,, August 2025, preprint.
S. Krishna, K. Krishna, A. Mohananey, S. Schwarez, A. Stambler, S. Upadhyay, and M. Faruqui. Fact, fetch, and reason: A unified evaluation of serieval-auggenteed generation, 2025. URL https: /ww../oiii////////////// wrg/10......o. 0tt.g/.........
48550/ www.i...424.... 29....229999... 12941.
M. Li, Y. Zeng, Z. Cheng, C. Ma. and K. Jia. Reportbench: Evaluating deep research agents via academic survey tasks. arXiv preprint arXiv:2508. 15804, 2025.
Y. Liu. Z. Yang, T. Xie, J. Ni, B. Gao, Y. Li, S. Tang, W. Ouyang, E. Cambria, and D. Zhou. Re- searchbench: Benchmarking llms in scientific discovery via inspiration- based task decomposition. arXiv preprint 25X3.22503.2122 2025.
11

---

Published as a conference paper at ICLR 2026
G. Mialon, C. Fourrier, C. Swift, T. Wolf, Y. LeCum, and T. Scialom. Gaia: A benchmark for general ai assistants, 2023. URL htttt:///ooi... http://...........///...///.../. http://aaw............ 1291.
OpenAl. Introducing deep research, 2025a. URL https: http:://pppeaiiccc ctt.e/nii.co/innniaaaeee index/ introducing deep research/ Accessed: 2025-09-18.
OpenAI. Introducing gpt-5, 2025b. URL htL httptt/pp////pppaaaa.cco//t///// cttp:/tt.c/p/ptt////
Patel, N. Arabzadeh, Gupta, A. Sundar, Stoica, M. Zaharia, and C. Guestrin. Deepscholar- bench: A live benchmark and automated evaluation for generative research synthesis. arXiv preprint MXXIv:2508.2003. 2025.
L. Phan, A. Gatti, Z. Han, N. Li, et al. Humanity's last exam. arXiv preprint arXiv:2501.14249 2025.
J. Ruan, I. Nair, S. Cao, A. Liu, S. Munir, M. Pollens-Dempsey, T. Chiang, L. Kates, N. David, S. Chen, R. Yang, Yang. J. Gump, T. Bialek, V. Sankaran, M. Schlanger, and L. Wang. Expertlongbench: Benchmarking language models on expert-level long-form generation tasks with structured checklists. arXiv preprint 2xXiv:5506602241,, 2025.
G. Son, Hong, H. Fan, H. Nam, H. Ko, S. Lim, J. Song. J. Choi, G. Paulo, Y. Yu, et al. When ai co-scientists fail: Spot-a benchmark for automated verification of scientific research. arXiv preprint ar arXiv:2505.1 11855, 2025.
H. Wan, C. Yang. J. Yu, M. Tu, J. Lu, D. Yu, J. Cao, B. Gao, J. Xie, A. Wang, W. Zhang, P. Torr, and D. Zhou. Deepresearch arena: The first exam of Ilms' research abilities via seminar-grounded tasks. arXiv preprint arXiv:2509.00 0139,, 2025.
J. Wang, Y. Ming, R. Dulepet, Q. Chen, A. Xu, Z. Ke, F. Sala, A. Albarghouthi, C. Xiong, and S. Joty. Liveresearchbeneh: A live benchmark for user-centric deep research the wild, 2025.
Wei, Z. Sun, S. Papay, McKinney. J. Han, I. Fulford, H. W. Chung. A. Passos, Fedus, and A. Glaese. Browsecomp: simple yet challenging benchmark for browsing agents. arXiv preprint arXiv:2504. /2516, 2025.
R. Xu and J. Peng. A comprehensive survey of deep research: Systems, methodologies, and applications. arXiv preprint arXiv:2506. 12594, 2025.
Xu, P. Lu, Ye, Hu, and P. Liu. Researcherbench: Evaluating deep ai research systems on frontiers of scientific inquiry, 2025. URL 16280.
org /ot.ooo.///./000.. 00w.0..8550050880 2507.
Z. Yang, P. Qi, S. Zhang, Bengio, W. W. Cohen, R. Salakhutdinov, and C. D. Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369-2380, Brussels, Belgium, 2018. Association for Computational Linguistics. doi: 10. 10.18653/1/D118-12555 URL https: aaaaaclanhhhc http://aaaannthcl..ogogg/// -
J. Zhou, W. Li, Y. Liao, N. Zhang, T. Miao, Z. Qi, Y. Wu, and T. Yang. Academicbrowse: Bench- marking academic browse ability of 1ms. arXiv preprint arXiv:2506. 13784, 2025.
12

---

Published as a conference paper at ICLR 2026
A EXTENDED RELATED WORK
The rapid emergence of deep research agents has been accompanied by several efforts to characterize and evaluate their capabilities. Recent surveys and roadmap papers highlight the promise and challenges of autonomous LLM-based research assistants. For example, Huang et al. (2025) provide a systematic examination of Deep Research agents, analyzing their tool integration and planning strategies, while Xu and Peng (2025) offer a comprehensive survey of deep research systems and applications. These works underscore the need for robust evaluation frameworks aligned with the complex, open-ended nature of research tasks.
Early benchmarks for deep research agents have largely taken one of two approaches: constructing tasks from static corpora relying on expert-curated questions. In the first category, benchmarks like AcademicBrows, Zhou et al. (2025) and BrowseComp Wei et al. (2025) assess an agent's ability to navigate and retrieve information from academic papers or the web. AcademicBrowse focuses literature-basse queries (e.g.. browsing academic papers for answers), and BrowseComp comprises over 1,200 web questions that demand multi-hop searching across sites. While these benchmarks test long-horizon retrieval and factual accuracy, their questions tend to have a predetermined scope or "ground truth" answers, which simplifies evaluation matching reference facts. This limits their ability to capture the open-ended synthesis and exploratory aspect of real research inquiries. Another example is ResearchBench Liu al. (2025), which builds complex search questions from static data; however, static benchmarks risk data leakage (i.e., answers appearing in training data) and cannot adapt to newly emerging information.
The second category of benchmarks uses expert-authored tasks to evaluate research reasoning. Humanity's Last Exam (HIF)) Phan et al. (2025) an expansive evaluation of 2,500 expert- written questions covering advanced domains ranging from mathematics to medicine. HLE revealed significant gaps in state-of-the-an models' knowledge, but it primarily consists of challenging short- answer questions, rather than multi-document analytical tasks. Closer to our setting, DeepResearch Bench Du et al. (2025) introduced 100 PhD-Ievel research problems across 22 fields (e.g.. scientific analysis, legal reasoning), each requiring long-form report. Their evaluation combines reference- based metrics and adaptive criteria, including measuring the number and accuracy of citations. This benchmark confirmed the difficulty of deep research tasks, where no model exceeded roughly 30% on their overall metrics, yet its scoring approach leans heavily on overlap with reference solutions and simple citation counts. Similarly, ExpertLongBencc Ruan et al. (2025) targets expert-level, long-form tasks in 9 domains (law, finance, healthcare, etc.), providing 11 complex prompts each accompanied by domain-specifi checklist or rubric. ExpertL.ongBench introduced the CLEAR evaluation framework, which extracts a structured checklist from both the model's 's output and a gold reference, then compares them for alignment. This method enables fine- me-ggrinedd assessment of content requirements, but it depends on high-quality reference outputs for each task. In contrast, our work uses expert-written criteria without assuming an ideal reference answer, and evaluates responses directly via LLM-as-a-judge
avoiding potential biases from any single ground truth essay.
More recent benchmarks have moved toward dynamic, real-world research scenarios. DeepScholar- Bench Patel et al. (2025) focuses generative research synthesis: it draws live queries from recent arXiv papers and evaluates systems on writing a related work section by retrieving and summarizing up-to-date literature. Its evaluation emphasizes three axes (knowledge synthesis, retrieval quality, and verifiability)) rewarding comprehensive coverage of relevant work and correct citation of sources. However, DeepScholar-Berr Bench is specialized academic writing tasks, and uses automated metrics (including LLM L..--ggeeetted scores) which may introduce evaluation circularity. ReportBench Li et al. (2025) takes another automated approach by leveraging existing survey articles as ground truth for evaluation. It generates academic survey-style prompts and measures the overlap between the AI agent's citations and statements and those in a published survey on the same topic. This provides a concrete correctness signal (since an expert- wrrtten literature review is treated as the gold standard), but inherently prioritizes replication of the reference content over creative or divergent but valid answers. Meanwhile, DeepRescarch Arena Wan et al. (2025) addresses the authenticity of research prompts: it automatically curates over 10,000 open-ended tasks from transcripts of academic seminars across 12 disciplines. By capturing questions that arise organically in expert discussions, DeepResearch Arena aims to evaluate agents on more ill-defined, exploratory problems. Their evaluation combines factual grounding checks with adaptively generated rubrics (checklists) to
13

---

Published as a conference paper at ICLR 2026
handle the breadth of tasks. One limitation, however, is that fully automatic rubric generation miss domain nuances or implicitly favor certain solution paths.
In parallel to benchmarking efforts, researchers have begun exploring Al "co-scientist" systems that autonomously propose hypotheses or experimental plans beyond just information retrieval. Notably, Gottweis et al. (2025) present AT Co-Scientist built multi-agent Gemini 2.0 system, which iteratively generates and refines scientific hypotheses (demonstrated in drug discovery and biology domains). The advent of such systems raises the stakes for evaluation: beyond finding correct facts, we must assess whether an Al's reasoning and conclusions hold up to expert serutiny. Initial work in this vein includes benchmarks like SPOT Son al. (2025), which checks Al-generated scientific papers for logical errors or inconsistencies. Overall, as deep research agents expand from answering questions performing nuanced scientific investigations, the need for fine-grained, human-aligned evaluation becomes ever more critical.
Our work builds directly on these prior insights. In contrast to previous benchmarks that either rely coarse-grained metrics, RESEARCURUBRICS offers a new middle ground:
on static answer keys a broad collection of realistic research queries (spanning academic and everyday domains) paired with expertly crafted rubrics that detail the requirements of good answer. This approach enables evaluation of multiple dimensions - factual grounding, cross-source synthesis, reasoning validity, clarity, and citation usage within single unified framework. By using numan-writter rubrics and having LLM judges apply them, we avoid reward hacking hased on simplistic overlap measures, while still achieving scalable scoring. RESEARCHRURRICS is complementary
contemporaneous efforts like ExpertLongBench and DeepResearch Arena: those benchmarks target either highly specialized expert tasks or massive automatically generated task suites, whereas we prioritize diversity of domains and manually quality-checked criteria. Together, these efforts push toward a more rigorous and comprehensive assessment of deep research capabilities.
B EXTENDED RESULTS
This appendix expands the quantitative analysis composition, complexity, and error structure, and clarifies the relationship between output length and rubric compliance.
B. BENCHMARK COMPOSITION AND RUBRIC COVERAGE
Fig. shows the number of rubric touched per task (mean 4.74). This multi-axis coverage reflects our goal of measuring holistic research ability rather than single-skill performance. Fig. reports the criteria count per task (20-43; mean 26). Fig. 10 decomposes axis proportions by domain, illustrating that domains differ not only by content but by the expected mix of explicitness, synthesis, and citation behaviors.
Ireich Oters
Salireces Chadce
Peblcc Ortess
Quelty
(a) Distribution of task complexity dimensions in RE- SEARCHRUBRICS.
(b) Distribution of rubric criteria categories. Implicit and explicit criteria dominate the benchmark.
Figure 7: Overview of task complexity dimensions and rubric criteria category distributions in RESEARCHRUBRICS.
14

---

Published as a conference paper at ICLR 2026
Mean: 4.74
Number Tasks
20
12
10
Number of Unique Axes Task
Figure 8: How many evaluation axes does each task cover? Distribution the number rubric axes per prompt. Most tasks require 4 to 5 distinet dimensions of quality simultaneously, encouraging balanced capabilities rather than single-axis optimization.
14
12
10
Number of
Number of Criteria per Task
Figure 9: Number of rubric criteria per task.
B.2
RUBRIC SCORING SCHEME
Table 7: Rubric scoring scale with mandatory and optional criteria.
Score Range
Description
[+4, +5]
Critically important criterion without which the response fundamentally flawed or incorrect. Required for a minimally viable response.
[-5, -4]
Critically detrimental A criterion identifying an enrr severe that it makes the response actively harmful, deeply unethical, completely invalidates its reasoning.
[+2 33
Important - key feature of strong response, but not absolutely essential. Slightly Important- A "nice-to-have" detail that improves good response but does significantly change overall quality.
+1
-1
Slightly Detrimental A minor issue, tangent, or stylistic weakness that does not impact core reasoning validity.
[-3, -2]
Detrimental significant error that detracts from the response quality. introduces faulty logic, offers poor advice, but does make it fundamentally harmful.
B.3 PERFORMANCE STRATIFIED BY COMPLEXITY DIMENSION
Figs. and 12 present model compliance scores stratified by conceptual breadth, logical nesting, and exploration level under binary and ternary grading schemes, respectively. Across both settings,
15

---

Published as a conference paper at ICLR 2026
Technical Decumentatio
Hypotheticals & Philosophy
20%
29%
325
Historical Ana lyss
Currant Evants
22%
Creative Wrlting
21%
31%
32%
Business Flanninng Research
10%
40
80
100
Percentage of Criteria
Communication Quality
Implicit Criberia
References Citation Quaityy Synthesis of Information
Explidt Criteria
Instruction Fallowing
Figure 10: Axis mix by domain. Stacked proportions of the six rubric
domains.
Gemini DR consistently leads, achieving roughly 65-70% average rubric compliance across most complexity tiers, followed closely by ChatGPT DR at around 60-65%, and Perplexity DR lagging near 50%.
A clear pattern emerges: performance degrades monotonically with increased logical nesting depth. Whereas shallow reasoning tasks (single-hop two-step queries) are handled well, multi-step analytical or evaluative problems see sharp drops, particularly for models relying on retrieval-centric architectures. Conceptual breadth also correlates with difficulty, though less steeply; systems handle multi-domain synthesis better than extended inferential chaining.
Poderate
Nector
23
Nratecty
Cancouiil Braadh
Log Mesting
Selvndke
Figure 11: Performance across Conceptual Breadth, Logical Nesting, and Exploration (Binary Evaluation)
B.4 DOMAIN-WISS FAILURE STRUCTURE
The heatmap in Fig. 13 shows how failure rates distribute across within each domain.
16

---

Published as a conference paper at ICLR 2026
300
LXC
Simple
Stallor
Enereskial Brooth
Log Misling
Figure 12: Performance across Conceptual Breadth, Logical Nesting, and Exploration (Ternary Evaluation)
ML
19.3
52.8
21.0
11.9
14.5
14.0
Business Paanning Research
22.3
44.1
13.5
15.7
12.9
Creative
23.0
Domain
General Consumer Research
33.5
46.6
16.8
10.7
10.4
13.6
Rale
Historical Analysis
24.6
37.2
30.1
14.8
19.4
13.0
Hypotheticals & Philesophy
24.44
35.1
32.5
9.0
11.3
12.6
Other
26.8
11.2
8.3
Decumentation
19.2
Explict
Implict
Synthezs
References
Communication
Instructins
Figure 13: Heatmap of failure contribution by rubric axis across domains.
B.5 MISCLASSIFICATION FAILURES IN HUMAN-LLM JUDGE ALIGNMENT DURING AUTO-EVALUATION
Fig. 14 illustrates the relationship between grading mismatches, i.e., disagreements between the LLM-as-a-judge and human evaluators, and various analytical dimensions across both binary and ternary classification settings. Specifically, the top row compares mismatch distributions across rubric categories, the middle row examines mismatches with respect rubric importance (mandatory vs. optional). and the bottom row presents mismatch rates by rubric category, normalized by the size of that category the dataset. We observe that Implicit Criteria account for the majority of misclassifications, which is unsurprising given that many rubrics in the dataset belong to this category. However, when normalized by category size, References & Citation Quality and Synthesis of Information show slightly higher proportion of disagreements, suggesting that models may struggle to assess what constitutes an adequate mention of reference or argument in response. We also note that mandatory criteria exhibit a lower proportion of mismatches, which is reassuring, as implies the model and human raters tend to align more closely on the mandatory aspects of the response.
17

---

Published as a conference paper at ICLR 2026
Mirmatch Proportion by Gategory (lisary)
Nisreatch Prepartion by Category (Teraar)
Gemri
thanett
Implict Ortere
Following
Catagery
Inatruction Fallowinn
0255
buplit Cireri
Syttesb of Informttie
Systhe of aitermaat
Anfernnce Chatinn quality
0.03
Ruferonccc Clacoon Quality
644
hrssrlaan Mamalanes
Proporton Mamatthho
(a) Mismatch by category (binary).
(b) Mismatch by category (ternary).
Proportion Importance (Binary)
Preportion Inportance (Temary)
ee Gameni
Auplinaity
Surir
0.033
0.004
0.5
X44
1.541
0.00
0.2
5.1
0.6
Mandatary
6.0
Mardatory
Optional
Rubric Tipe
Rubric Type
(c) Mismatch by importance (binary).
(d) Mismatch by importance (ternary).
Mermaliced Mamatthh Rete Category (Binary)
Hormeliced Momatthh Rate Categary (Tumary)
coat Gamin
Poptorty
Conmutiation Qualtty
Communiation Quality
Che
0332
Impliia Cineria
Insruction followin
Coopury
Instrvccion Fallower Folloingg
C.145
kigliit Ortaris
0.074
0.188
Systesk Difommitee
1200
Syrties of intemmatioonn
0383
Adtrencc Chation
6.103
neterence Claat Quality
B114
(240
211
2.33
2.1%
0.20
5.75
6:06
011
0.33
(c) Mismatch by category (binary).
Mismatch rate by category (ternary).
Figure 14: Comparison of mismatch metrics (by category, importance, and mismatch rate) across binary and ternary settings.
C PROMPT AND RESPONSE LENGTH ANALYSIS
C.I PROMPT WORD COUNT ANALYSIS
To understand the scope and complexity of the evaluation tasks, we analyzed the word counts of all 101 prompts included in RESEARCHRUBRICS, Prompt length serves as a useful proxy for task complexity, longer prompts tend to encode more contextual background, sub-questions, and open-ended reasoning requirements.
Across all tasks, prompt lengths moderately distributed, with mean of 87.6 - 58.6 words (median 68, range 13-315). As shown in Fig. 15, most prompts cluster below 100 words, though long right-tail distribution reflects the presence of prompts well over 200 words.
18

---

Published as a conference paper at ICLR 2026
Prompts vary substantially by domain (Table 8). Tasks from General Consumer Research, Technical Documentation, and Business Planning & Research exhibit the longest average prompt lengths, often exceeding 100 words. In contrast, domains such Al & ML, Current Events, and Other tend to be more concise.
Prompt length also scales with the benchmark's complexity dimensions (Fig. 16). Prompts with higher conceptual breadth, deeper logical nesting, and greater exploration are systematically longer, often doubling in average length compared to simpler tasks. This pattern underscores that more open-ended research problems require not only deeper reasoning but also more extensive prompt scaffolding.
Mcan: 87.6
Median:
12
Number Promipps
10
Figure 15: Distribution of prompt word counts across all 101 tasks. The distribution is right- with mean of 87.6 words and median of words.
10C
26.1
Averga Coat.
83
Stnck
Paccett
High
Plarne ae
Deep
Modkun
Figure 16: Prompt word count by task complexity dimensions (Conceptual Breadth, Logical Nesting, and Exploration). Longer prompts are consistently associated with higher complexity levels.
C.2 RESPONSE LENGTH AND COMPLIANCE
To contextualize the prompt statistics, compared the word and token counts of responses generated by three Deep Research agents: ChatGPT DR, Gemini DR, and Perplexity DR.
On the Markdown outputs (Table 9), Gemini produces the longest responses average (7.500-7.60) words), followed by ChatGPT (6.300-6,400 words), while Perplexity outputs are substantially shorter 1~00000 words). These differences are consistent across both words and tokens, and between text and rendered formats. High variance (standard deviations above 2,000-3,000 words) reflects substantial prompt-dependent variation in response verbosity.
To understand whether output verbosity correlates with perceived quality, we examine the relationship between response length (in tokens and words) and overall rubric compliance. Figs. 17a to 17d display these correlations for Gemini DR, ChatGPT DR, and Perplexity DR.
Moderate positive correlations (r z 0.20 0.28 for Gemini and ChatGPT) indicate that longer responses generally achieve higher scores. Perplexity DR, with the shortest outputs, achieves the lowest correlations.
19

---

Published as a conference paper at ICLR 2026
Table 8: Prompt Word Count Statistics across Domains and Complexity Dimensions
Category
Subset
Count
Mean
SD
Median
Min-Max
95% CI
Overall Statistics
All Prompts
87.6
58.6
68.0
13-315
[76.0, 99.2]
By Domain
AI & ML
17
61.8
44.8
46.0
13-169
[38.0. 85.5]
Business Planning & Research
12
111.0
56.2
98.5
36-224
[73.7. 148.3]
Creative Writing
69.2
19.9
66.0
40-103
[46.2, 92.11
Current Events
$1.0
55.0
21-76
[23.1,
General Consumer Research
11
138.4
80.2
131.0
35-315
[81.9, 194.91
Historical Analysis
13
81.5
50.2
70.0
30-227
149.9, 113.1]
Hypotheticals & Philosophy
11
78.3
45.4
69.0
22-187
[46.3, 110.2]
Other
6
61.2
22.6
51.0
40-107
[35.2, 87.2]
STEM
8
80.0
43,9
64.0
30-174
[40.8, 119.21
Technical Ducumentation
12
112.3
69.1
75.5
49-271
[66.5, 158.2]
By Conceptual Breadth
Simple
36
79.1
58.7
60.5
13-271
[59.0, 99.21
Moderate
52
91.2
60.7
72.0
22-315
[74.1, 108.3]
High
13
96.8
45.4
95.0
29-195
[68.3, 125.4]
By Logical Nesting
Shallow
88.9
67.0
21-227
[56.4, 121.5]
Intermediate
72.4
61.5
13-197
[60.3, 84.5]
Deep
36
106.4
68.0
79.5
22-315
[83.0, 129.71
By Exploration
Low
29
75.6
55.1
56.0
13-227
[54.3, 96.9]
Medium
55
80.8
48.9
66.0
21-271
[67.5, 94.2]
High
130.1
72.7
111.0
47-315
[91.5, 168.6]
Table 9: Word and Token Statistics per Model
Type
Model
Mean
SD
Median
Min
Max
ChatGPT
6269.73
3684.21
5481
1328
18824
Words
Gemini
7519.32
2447.70
7562
2909
14640
Perplexity
1828.61
1127.70
1579
128
7352
ChatGPT
10169.57
5885.79
9075
2103
30233
Tokens
Gemini
12153.31
4028.00
11710
4530
26421
Perplexity
3664.36
2006.01
3241
539
14148
This supports the length-qualic conflation hypothesis: longer reports often perform better because they cover more rubric criteria, not necessarily because evaluators prefer verbosity. Nonetheless, since RESeARCHRUBRICS scores criterion-basee rather than holistic, the observed correlation partly reflects genuine informational density rather than stylistic inflation.
C.3 DISCUSSION: SYSTEMATIC PATTERNS AND THEIR IMPLICATIONS
We next present some of our key findings from our analysis.
Domain and Task Complexity Effects Our analysis reveals surprising performance inversions across domains. Agents achieve 76% coverage on open-ended consulting questions but struggle with technical precision tasks, contradicting intuitive difficulty expectations. This aligns with ResearcherBench Xu et al. (2025) findings that systems excel at exploratory reasoning while failing deterministic requirements. The pattern suggests current architectures inherently favor creative synthesis over systematic execution, explaining why even leading systems achieve below 40% on technical nugget coverage despite 85% scores organizational structure.
Task complexity analysis confirms the depth-wind width decomposition framework: performance degrada- tion accelerates with sequential reasoning requirements (depth) more than parallel capability demands
20

---

Published as a conference paper at ICLR 2026
Gemite
ChaNOPT
Perplecty
200, 201s
Mi.
0.2
Tekio Cramt (Xto))
Telrn 107
Takeo Coan
Length score (tokens; binary).
ChaIAPT
Perpledly
2-232---0000 2005
0.22 1.5
02
15
Tokan Quitixto
Toean Charc 10
Tokan
(b) Length vs. score (tokens; ternary).
Perpleaty
and 200, 2045
13
(c) Length vs. score (words; binary).
Perpleaity
3300 ma00
C4
36
12 14
24 12
15
30
(d) Length vs. score (words; ternary).
Figure 17: Comparison of length vs. score across token and word counts for binary and ternary settings.
(width). Tasks exceeding 4 sequential inference steps or 35 minutes of human-equivalent time show universal performance collapse across all evaluated systems (see Fig. 5). With ReSEARCHRUBRICS averaging 25.7 criteria per prompt (see Fig. 9), approaching the 2" 1 component complexity for n features, we operate near the theoretical saturation point for reliable evaluation.
The Length-Quality Conflation Problem Deep Research agents produce outputs 10-100 times longer than standard LLM responses (5,000-50,000) tokens; see Table 9), raising questions about whether length drives perceived quality. Our criterion-level analysis reveals nuanced relationship:
21

---

Published as a conference paper at ICLR 2026
longer responses correlate with higher scores (see Fig. 17), but this primarily reflects legitimate information density rather than padding. Systems generating comprehensive reports with 100+ source synthesis necessarily require length, yet evaluators show documented bias toward verbosity independent of content quality.
RESEARCHRUBRICS atomic evaluation partially mitigates this bias. Each of 2500+ criteria checks specific content presence rather than holistic impressions. However, the correlation persists even at the criterion level, suggesting that either (1) comprehensive responses naturally satisfy more criteria, or (2) length bias operates even on supposedly objective checkpoints. Distinguishing these explanations requires controlled experiments varying response length while holding information content constant.
Architectural Limitations Beyond Prompt Engineering The consistency of failure patterns across systems-45-500 implicit criteria failures (see Fig. 4), poor multi-hop reasoning, synthesis Nottlenecks-indicnn fundamental architectural constraints rather than implementation differences. Multi-hop reasoning studies Yang et al. (2018) demonstrate that while agents achieve 80%+ success on first-hop inference, bridge entity resolution in early neural layers creates hard limits on subsequent reasoning depth. This explains the limited improvements from prompt engineering alone.
The breadth-accuracy trade-off further illustrates these constraints. No system successfully balances comprehensive coverage with precision. Gemini's 11 1--iitation breadth sacrifices accuracy (81 %) while Perplexity's 90% accuracy comes from restrictive 31 31-citation coverage. This isn't tuning prob- lem but reflects incompatible optimization objectives that current architectures cannot simultaneously satisfy.
D SUPPLEMENTARY FIGURES AND TABLES
We provide concise descriptions of the ten prompt domains used in RESEARCHHRBRICS in Table 10.
E PROMPTS
The prompt we sent to the LLM-as-a-judge can be found in Fig. 18.
We used two prompt templates in the ablation experiments: one for example removal and for rubric augmentation. Both are shown below for reproducibility.
22

---

Published as a conference paper at ICLR 2026
Category
Description of Prompts
AT & ML.
Tasks centered on artificial intelligence, machine learning, and data science, including model evaluation, algorithmic comparisons, ethi- cal considerations, and emerging applications. Prompts often require synthesis technical papers, applied case studies, and discussions of interpretability, safety, or deployment challenges in real-world AI systems.
STEM
Science, technology, engineering, and mathematics prompts outside core AI/M domains. These tasks reguire synthesizing information from textbooks, research papers, technical reports (e.g., explaining physical principles, analyzing chemical processes, or modeling engi- neering systems).
General Consumer Research
Everyday research with complex constraints (e-g-, finding an apart- ment under budget, multi- product comparisons, travel itineraries, personal finance or legal advice, health-related questions requiring rep- utable sources).
Technical Documentation
Prompts involving explanation of complex technical concepts, code, APIs using official documentation repositories (e.g.. troubleshooting programming error with library does, comparing software architecture patterns).
Hypotheticals Philosophy
Open-ended prompts asking speculation, hypotheticals, or philo- sophical analysis, often requiring synthesis of diverse viewpoints (c.g.. "How might society change X.. ?", ethical dilemmas, future predic- tions in technology).
Historical Analysis
Questions about historical events, figures, periods that require pulling from archives. historical texts, and scholarly interpretations (e.g.. lyzing causes of historical conflict with primary source references).
Business Planning & Research
Prompts related business entrepreneurship (e.g., developing go-to-market strategy. analyzing company's ' financial health, legal considerations for startup, HR or marketing plan), often requiring use of industry reports or case studies.
Creative Writing
Long-form creative tasks that incorporate factual elements or research (c.g., writing historical fiction scene with accurate period details, or sci-fi story grounded in real science).
Current Events
Prompts focused on recent or ongoing events, necessitating retrieval of up-to-date news data (e.g.. analysis of recent policy change, comparison of current market trends).
Other
Miscellaneous prompts that do not neatly fit in the above categories, including cross-domain questions niche topics.
Table 10: Prompt domains in RESEARCHRUBRICS.
23

---

Published as a conference paper at ICLR 2026
SYSTEM:
You expect evaluator tasked with asseasing whether decument satiafles specific rubric criteria. Your avaluation must be precise, abjestive, and based solely the
present the document.
Evaluation Framework
You will evaluata each rubric criterion using three-tier satiafaction scale: valot Satlafled (Score: 0.0).r: docurent fails meet the criterton. Key elementa are mLesing, incorrect, inadequalely addressed..
2. **Partially Satisfled (Score: 0.51+ The document partially meats the criterion, Some
present incomplete, lacking depth, misaing
important aspects.
*saaissfid (Score: 1.01 document fully meets the criterion. required el elmannt are present, wel vell--avelopen and appropriately detai
Evalustion Process
v*Understend the Criterionss: Carefully read and interpret what the rubric
asking
2. .*Search for Evidencess: systenatically zeview the document for relevant criterion.
content that aédresses
3. **Assesa Complet completeneasee: Rvaluate whether the evidence fully, partially, or
fails to satisty the crlterion.
->Provide Reasoninges; Explain evaluation with specific references the document content.
# Important Guidelines
Base your evalustion
what exp icitly present the document
Do not make assumptions about implled or missing conten:
Consider the quelity, completeness, and relevance of the evidence
all criteris Provide specific exarples from the document to support your verdict
consistent your evalustion standards
Note:
Example lists in these rubrics are intended i'lustrate possible reascning patterns or relevant toptes. These example I'sta contair correct answera but not exhaustive. Use then guidance, but also make your finel judgment about what qualifies as correct when appropriate.
USER:
Decument Content
[dccument_contene
Rubric Criterion Eveluate
*iitlee:: (subric_titlel
*>Catgooyyиw (rabrie_category) <-Meight (rubric_weigh)
# Your Task
Evaluate whether the above document sacisties this specific rubric criterion.
Required Response Formet
Provide your avaluation in the following JSON format:
"*ison
"verdiet"; "oot Satistied/Partiall Satisflec/Satietttee]]
"score": [0.0/0.5/1.0],
"confidence"; [0.0-1.0',
"reasoning": "Detailed explanstion with specific avidence from the documens", "evidence : rect quote 1", rDirect quote .....
bniseinggelementet" ["Element 1 that would improve satiefaction", ...)
Ensure your zesponse is ONLY the J80N objecs, with no addicionel text.
Figure 18: Prompt used for example removal during rubric preprocessing.
24

---

Published as a conference paper at ICLR 2026
SYSTEM:
You are tasked with removing examples from rubric text while keeping everything else EXACTLY the same.
Your job is to:
1. Identify fy portions of text that contain 1ist of examples,
typically in the form (e.g., examplel, example2, example3) or similar.
2. Remove ONLY these example portions.
3. Keep all other text, formatting, punctuation, and structure EXACTLY the same.
4. Do not rephrase, reword, change anything else.
Do not add any now content.
6. Simply return the text with the example portions removed.
Examples of what remove:
" (e.g., diagrosis code block, free-text note anippet without PHI, tabular data contexting text and numerical data) "(..e. programmatic text extractions, more rigorous NLP and machine Learning techniques, etc.)"
((()) National Library of Medicine, (2) CDC Wonder or (3) publications from well-known universities)"
Be very careful with maintaining the exact same structure and wording for the rest of the rubric.
USER:
Please renove the examples from the following rubric text while keeping everything else EXACTLY the same:
Frubric_texe
Figure 19: Prompt used for grading via the LLM-as-judge framework.
25

---

Published as a conference paper at ICLR 2026
SYSTEM:
You are an expert at improving rubrics that are used to evaluate nodel responses. Make the rubrics more detailed, both in terms of facts the models should cover and any definitions or examples that should be added, while still keeping the rubrics somewhat concise.
CRITICAL FORMATTING REQUIREMENTS:
Return exactly ONE cohesive sentence (NO newlines, NO line breaks).
The rubric should be ONE SINGLE SENTENCE but can contain multiple phrases, subparts, clauses, and run-on components. Target approximately 100 words average, but you exceed that when necessary for completeness.
Do NOT create multiline, paragraph-style, or bullet-poing rubrics.
IMPORTANT: You will receive exactly ONE rubric improve, and you must return exactly ONE enhanced version of that same rubric. Do not create multiple rubrics or variations.
Your job is
Keep ALL original information from the rubric EXACTLY as it is - do not delete or remove any coze information, knowledge or intent from the rubric.
2. Make the rubric more detailed and concrete by adding specific examples inline (e.g., specific answers or patterns that might help the model to generalize)
3. Clarify vague terms with more precise descriptions within
the same sentence flow.
4. Add any information that may be missing.
5. Make the rubric as actionable and unanbiguous as possible
while staying concise.
Focus adding inline:
Concrete examples in parentheses (e.g., speci fic technical details, data formats), which need not exhaustive. Clear boundary conditions. Any definitions for unclear terms.
Do NOT:
Remove any original content.
Change the fundamental meaning intent of any rubric. Add an entirely new rubric.
Create multiple versions variations (don't generate more than one rubric output).
Use newlines, bullet points, or multiline formatting.
Break the rubric into multiple sentences.
Return only the aingle improved rubric one cohesive sentence.
USER:
Enhance this zubric by adding specific examples and details while formatting as ONE cchesive sentence (no newlines, but the rubric can contain multiple phrases and clauses)
[rubric_text]
Return only the enhanced additional text.
rubric with no
Figure 20: Prompt used for LLM-based rubric augmentation.
26


Reviews:
[{"review_id": "60BSeKLeZz", "reviewer": "Reviewer_DnDN", "summary": "The paper proposes Research Rubrics, a structured framework to evaluate how well large language models (LLMs) perform scientific reasoning. It scores models across four dimensions: problem understanding, reasoning process, solution design, and scientific contribution. Experiments on multiple state-of-the-art models show that LLMs can follow scientific logic and structure but still lack genuine creativity and practical validation.", "questions": "- Can the framework generalize beyond computer science to other scientific domains such as biology and physics?  \n- To what extent does the framework measure true reasoning ability rather than the effects of prompt engineering?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper focuses on an underexplored but important problem: evaluating whether LLMs can reason like scientists rather than just answer questions. \n- The proposed Research Rubrics provides four well-defined dimensions: Problem Understanding, Reasoning Process, Solution Design, and Scientific Contribution. \n- The experiments benchmark multiple leading models (GPT-4, Claude 3, Gemini, Qwen2, Mistral) and compare different prompting strategies (chain-of-thought, critique loop, research-plan prompting). The setup is comprehensive and systematic.  \n- The paper shows that while LLMs can follow scientific logic and structure, they still lack genuine creativity and originality, which is an important insight for future research on AI scientific reasoning.", "weaknesses": "- The study mainly focuses on computer science problems (e.g., ICLR/NeurIPS-style tasks), so it’s unclear how well the framework generalizes to other scientific domains.  \n- The framework evaluates the quality of reasoning, but it does not assess whether the model’s ideas could actually produce valid or impactful scientific outcomes.\n- Model performance depends heavily on prompt design (e.g., “research-plan” prompts boost scores), which suggests the framework might partly reflect prompt engineering skill rather than pure reasoning ability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sJLUjPi3Cf", "reviewer": "Reviewer_Un9g", "summary": "This paper introduces a benchmark for evaluating deep research agents. It contributes 75 prompts and 75 rubrics across 8 domains, of which there are 1868 human written criteria for what the question designers expected to see in a good answer. The rubrics are created through the following process:\n\nHuman 1 drafted initial prompt and rubric terms.\nHuman 2 reviewed draft, provided feedback, and iterated with Human 1 until both agreed it was clear and complete.\nHuman 3, final reviewer, checked for clarity balance and bias and made final adjustments.\n\nThe rubrics are created before any agents are asked to generate answers; rubrics can involve both positive and negative criteria and have different attributes at different weights. Deep research agents were then asked to complete the queries, and their answers were scored with these rubrics. No inter-annotator agreement was measured, but instead the paper does human-LLM agreement where a human and a LLM scored the same response with a rubric, and this showed moderate human-LLM agreement.\n\nGenerally, the finding is that SOTA deep research agents achieve at maximum 0.59 rubric compliance, with good factual recall but weak synthesis, implicit reasoning, and citation quality.", "questions": "NA", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Diagnostic clarity. Not many works exist to evaluate the output of deep research agents, and this work provides a meaningful contribution towards that front. The rubric design makes sense- and helps to evaluate the open-ended nature of these very long responses. The work provides a meaningful step to make sense and introduce clarity into the messy subjective space of research quality into a more structured and reproducible measurement framework.", "weaknesses": "The paper's main weakness is on validity.\n\nDisagreements occur in rubric creation, so the final produced rubric through the 3 stage process masks inherent disagreement and tries to measure progress against a rubric that would be created by the average human. No IAA measures or analysis of disagreements occur; the main question, of a benchmark, is if progress on said benchmark would demonstrate a meaningful improvement on the task for end users. It is unclear from the rubric creation process that this can be convincingly said. Moreover- in the creation process, rubrics are fixed before seeing any output. The overarching paper claim seems to be that hey, these deep research agents are producing outputs that misalign with human preferences. However, it is unjustified that this creation process of rubrics is able to measure true human preference of deep research outputs- it's very easy to say that hey, I may think I want to see certain aspects before I actually see it, but in reality this shifts significantly from what I said when I actually see the output. It is unclear and unjustified that this rubric creation process can actually result in rubrics that reflect true human preference.\n\nMinor nits, but it's important to note that deep research agents also produce a ton of stochasticity due to their long executions; controlling for this in some degree would make the results more robust. As it currently states, each agent was run once per prompt, with no reported stochastic variation or repeated trials. Finally, 75 rubrics and outputs is pretty low for this type of benchmark, especially when spread across so many domains; this is no concern if the authors can justify that performance on their benchmark translates to demonstratable improvements in real world deep research agent use, but this is not yet convincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "211KRFg4Ag", "reviewer": "Reviewer_pY9n", "summary": "- The paper introduces ResearchRubrics, a benchmark for evaluating “Deep Research” (DR) agents—LLM-powered systems performing multi-step web exploration, retrieval, and synthesis to produce evidence-backed, long-form answers.\n- The benchmark comprises 75 prompts across eight broad domains (STEM, health, finance, legal, consumer research, current events, historical analysis, creative writing, business planning), each paired with expert-authored, fine-grained rubrics (1,868 total criteria). Rubrics include both positive and negative criteria, with weights capturing importance and severity.\n- The authors propose a tri-axial task complexity framework: conceptual breadth, logical nesting depth, and exploration level, to characterize and analyze DR tasks systematically.\n- They develop an LLM-as-judge protocol with ternary and binary grading schemes and conduct human consistency analysis (Macro F1), showing moderate-to-substantial agreement depending on the grading regime. They perform ablations on rubric detail and LLM-augmented rubrics.\n- They evaluate leading commercial DR agents (OpenAI DR, Gemini DR, Perplexity DR) and baseline LLMs with search tools, reporting sub-60% average rubric compliance, with primary failures on implicit reasoning, synthesis, and citation rigor. They plan to release the dataset, rubrics, and tools.", "questions": "- Rubric and weighting validation:\n  - How were weights calibrated? Did multiple annotators assign weights independently? Please provide inter-rater reliability or calibration metrics for weights and for the ternary labels on a subset.\n  - Consider sensitivity analysis: How do final scores and agent rankings change under different weighting schemes or partial-credit values?\n- Category failure percentages:\n  - Clarify how failure rates per category are computed. If criteria have multiple labels (e.g., both synthesis and implicit), explain how percentages and totals are derived to avoid summing >100%. Consider reporting per-axis normalized failure proportions.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Clear motivation: DR tasks are open-ended, dynamic, and require long-form synthesis; existing QA-style benchmarks and short-answer datasets underrepresent these needs.\n- Human-authored rubrics: The choice to use carefully designed, expert-written rubrics (including negative criteria and weighted mandatory vs. optional items) is a thoughtful departure from purely automated reference-based metrics and helps capture nuanced expectations.\n- Fine-grained evaluation: The rubric axes (explicit, implicit, synthesis, references, communication, instruction following) provide multidimensional diagnostics beyond factual correctness, surfacing where current agents struggle (implicit context and synthesis).\n- Task complexity framework: The breadth-depth-exploration triad is sensible and useful for labeling, stratifying tasks, and analyzing performance across different dimensions of difficulty.\n- LLM-as-judge methodology with human consistency checks: The inclusion of ternary vs. binary verdicts, Macro F1 consistency with human annotators, and ablations on rubric clarity are solid evaluation engineering contributions. Practical insights (binary verdicts yielding higher consistency; examples helping; LLM-augmentation of rubrics hurting) are valuable to the community.\n- Diversity of domains and realistic tasks: Inclusion of everyday research queries alongside STEM and business/legal topics reflects real user scenarios for DR agents.\n- Transparency about evaluation window and pipeline: Fixing the evaluation window for dynamic content (July 2025) and describing the collection (PDF -> markdown chunks -> LLM judge) helps reproducibility.", "weaknesses": "- Scale and representativeness: 75 tasks and 1,868 criteria are substantial in rubric detail but relatively small in task count for a general-purpose benchmark, raising questions about coverage and generalization across the wide variety of DR use cases.\n- Expert definition and domain specialization: “Experts” are defined as strong STEM generalists rather than domain specialists (e.g., legal, medical). For domains with high stakes or regulatory complexity, lack of specialist involvement may reduce rubric validity and could introduce bias or omit critical domain-specific requirements.\n- Judge reliability and circularity: While the paper argues against circularity by using human-written rubrics, reliance on closed-source LLM judges and moderate human agreement in ternary settings (Macro F1 ~0.48–0.55) raise concerns about robustness and potential judge bias. There’s limited analysis of inter-annotator agreement among humans or calibration strategies for judges beyond F1.\n- Weighting scheme and scoring design: The rubric uses weighted sums normalized by absolute weights, with partial credit (0.5) for “Partially Satisfied.” The justification for specific weights (e.g., what constitutes “Critically Important” vs. “Important”) is largely qualitative. Sensitivity analyses to weight choices and to the partial-credit scale are not reported.\n- Inconsistencies and clarity issues:\n  - Category failure percentages in Table 3 appear to sum well above 100%, which suggests overlap or double-counting across axes; this needs to be clarified (e.g., whether criteria carry multiple axis tags or how proportions are computed).\n  - Minor inconsistencies in judge model naming across the main text and appendix (e.g., GPT-4.o vs. GPT-4.1; inclusion of o3 later).\n  - Editorial issues (duplicate “Creative Writing” in the domain list, typos visible in figure text) detract from presentation quality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iLDuQ4Z8AZ", "reviewer": "Reviewer_opWo", "summary": "This paper proposes ResearchRubrics, a benchmark to evaluate Deep Research (DR) agents. It covers eight domains (i.e., STEM, health, finance, legal, and common consumer questions), and the rubric criteria, written by human experts, check factual grounding, coherence of reasoning, completeness, relevance, and clarity of the answer. Applying LLM-as-a-judge to assess rubric compliance, the existing DR systems achieve a poor performance.", "questions": "**Questions**:\n1. Section 3.1 Data Collection and Task Domains: There are only \"three expert participants\" with \"a strong STEM background\". However, the proposed benchmark covers eight domains, four of which are not STEM. How could the data quality of those domains be guaranteed?\n2. Section 3.2 Prompt Complexity Dimensions: How is each ResearchRubrics task categorized? Is it categorized by human experts/annotators or LLMs? What are the criteria for categorizing?\n3. Section 3.2: The proposed \"task complexity framework\" measures three dimensions: \"(1) conceptual breadth (the number and diversity of distinct topics or domains involved), (2) logical nesting depth (the number of reasoning or decision steps required, including sub-questions and conditionals), and (3) exploration level (the degree of open-endedness or underspecification of goals).\" What are the statistics of it? For example, as in Table 1, how many instances are categorized as \"Simple\", \"Moderate\", or \"High\" Conceptual Breadth?\n4. Section 3.2: What are the insights of knowing the prompt complexity categories? Did such categories guide the construction of the proposed ResearchRubrics benchmark?\n5. Why are the proposed DR rubric criteria (as in Table 2) valid and enough to evaluate DR agents? Are they devised by pure heuristics/intuitions, or are they based on a certain theory or relevant literature?\n\n**Typos**:\n1. Line 65: \"LLM-as-judge\" $\\to$ \"LLM-as-a-Judge\"", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The proposed benchmark is intuitive, well-designed, and well-curated.\n2. Experimental results on multiple DR systems demonstrate that the current DR agents struggle with the proposed benchmark.", "weaknesses": "1. Only three human experts with a STEM background are involved in the benchmark construction process. (See Question 1 below)\n2. Some points to be clarified. (See Questions 2, 3, and 4 below)\n3. The proposed DR rubric criteria need further justification. (See Question 5 below)\n4. No case study or error analysis to provide insights into why the current DR agents fail to achieve a higher score on each of the proposed rubric criteria.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)