ScholarEval: Research Idea Evaluation Grounded in Literature

## Abstract

Abstract As AI tools become increasingly common for research ideation, robust evaluation is critical to ensure the validity and usefulness of generated ideas.
We introduce ScholarEval , a retrieval-augmented
evaluation framework that assesses research ideas based on two fundamental criteria: soundness —the empirical validity of proposed methods based on existing literature, and contribution —the degree of advancement made by the idea across different dimensions relative to prior research. To evaluate ScholarEval , we introduce ScholarIdeas , the first expert-annotated dataset of multi-domain research ideas and reviews, comprised of 117 ideas
across four disciplines: artificial intelligence, neuroscience, biochemistry, and ecology. Our evaluation shows that ScholarEval achieves significantly higher coverage of points mentioned in the human expert annotated rubrics
in ScholarIdeas compared to all baselines. Furthermore, ScholarEval is consistently preferred over our strongest baseline o4-mini-deep-research, a reasoning and search-enabled agentic system by OpenAI, in terms of evaluation actionability, depth, and evidence support. Our large-scale user study also shows that ScholarEval significantly outperforms deep research in literature engagement, idea refinement, and usefulness.
We openly release our code, dataset, and ScholarEval tool for the community to use and build on. 1 1 1 Code and data can be found at https://github.com/skai-research/ScholarEval

## 1 Introduction

Research ideation stands out as one of the most critical and challenging steps in scientific research, where the success of a research project fundamentally hinges on the technical soundness of the underlying idea and its potential to advance the field. To accelerate this stage, multiple works have developed AI-based systems for research ideation (Wang et al., 2024; Si et al., 2025b; Garikaparthi et al., 2025; Gottweis et al., 2025; Baek et al., 2025). Although AI-generated ideas score higher than human ideas on criteria such as human-evaluated novelty at the ideation stage (Si et al., 2025b)
, many of these seemingly promising ideas turn out to be ineffective when executed (Si et al., 2025a). The execution of faulty ideas can lead to substantial costs, particularly in fields requiring significant computational resources or wet-lab experiments. There is thus a critical need to rigorously evaluate research ideas pre-execution to prioritize the strongest ones for resource investment.

Figure: Figure 1: Left: Given a research idea ScholarEval generates a literature-grounded evaluation based on soundness and contribution. Right: To evaluate ScholarEval, we measure the degree of coverage of expert-annotated review rubrics in ScholarIdeas. The final coverage score is the average over all rubrics.
Refer to caption: x2.png

Despite this need, automatic research idea evaluation remains an underexplored area. Some works narrowly frame it as a prediction task: deciding which idea among a pair would lead to better results on predefined benchmarks (Wen et al., 2025). Others focus on one-dimensional approaches to idea evaluation (e.g., novelty) (Afzal et al., 2025; Shahid et al., 2025),
or target only specific sub-disciplines (e.g., AI) (Si et al., 2025b). Most importantly, these systems either only generate scores or are limited to sparse feedback such as short rationale statements (Feng et al., 2025; Wen et al., 2025; Shahid et al., 2025). However, to create AI co-scientists that can generate and refine research ideas, giving dense, actionable, and multifaceted feedback is crucial (Wu et al., 2023; Cao et al., 2024).
To the best of our knowledge, no existing work addresses the challenge of comprehensive research idea evaluation across disciplines within a framework that provides detailed and actionable feedback for idea refinement.
To address this gap, we introduce ScholarEval ([Figure 1](https://arxiv.org/html/2510.16234v1#S1.F1)), a
research idea evaluation system
grounded in the most recent literature. ScholarEval evaluates research ideas based on two fundamental criteria: soundness and contribution. (1) Soundness refers to the empirical validity of each proposed method in the research plan, assessed by systematically examining whether similar applications of this method in existing literature have demonstrated success or failure.
(2) Contribution refers to the degree of advancement a research idea offers across different dimensions–e.g., its proposed methodology, data, evaluation approaches, and conceptual framework–relative to existing literature.
The rationale for dimension-based evaluation is that novelty is multi-faceted by nature, and an idea can be considered novel relative to certain aspects of prior work, rather than being categorically novel or not (Rubaiat et al., 2025; Radensky et al., 2025).
Given a research idea detailing the problem, proposed methodology, and planned experiments, ScholarEval employs a multi-stage pipeline that first generates targeted search queries to retrieve a large volume of related literature from Semantic Scholar (Kinney et al., 2025) ([Figure 2](https://arxiv.org/html/2510.16234v1#S2.F2)). It then extracts key information from the retrieved literature to assess the research plan’s soundness and contribution. Finally, it synthesizes detailed feedback supported by relevant citations.

To evaluate ScholarEval, we construct ScholarIdeas, a multi-disciplinary dataset of 117 research ideas and their corresponding reviews validated by subject-matter experts across four disciplines: artificial intelligence, biochemistry, neuroscience, and ecology. As showcased in [Figure 1](https://arxiv.org/html/2510.16234v1#S1.F1), reviews in ScholarIdeas are composed of multiple rubrics, each focusing on a specific point that the evaluation should address, for 1076 rubrics in total.
We develop a multi-faceted automatic evaluation framework to assess ScholarEval against strong baselines, namely state-of-the-art LLMs and deep research systems.
Our results show that ScholarEval achieves greater coverage of the expert-annotated review rubrics in ScholarIdeas, significantly outperforming all baselines and surpassing o4-mini-deep-research
by over 20% relative improvement. Our results also demonstrate that ScholarEval is consistently preferred over deep research in terms of evidence support, depth, and actionability.

Our human study involving 18 experts and 46 evaluations further supports the real-world usefulness of ScholarEval across our four target disciplines. ScholarEval outperforms the strongest baseline, deep research, by a significant margin in metrics tied to our core contributions: literature engagement, citation use, feedback validity, idea refinement, and focus on relevant evaluation aspects. The expert evaluators also found our system more useful and were eager for its official release.

Our work makes the following major contributions:

- •
ScholarEval, a literature-grounded framework that comprehensively evaluates research ideas based on their soundness and contribution with actionable feedback. We openly release the ScholarEval tool.
- •
ScholarIdeas, an expert-annotated dataset for research idea evaluation spanning four disciplines and composed of 117 research ideas with 1076 detailed review rubrics.
- •
A multifaceted evaluation for long-form research idea review responses, including automatic metrics and a carefully designed human expert evaluation.

## 2 ScholarEval

ScholarEval is a retrieval-augmented, multi-stage pipeline designed to give an in-depth evaluation of research ideas based on their soundness and contribution.

Task Formulation. Given a research idea $I$,
the task is to find papers $P=\{p_{1},p_{2},p_{3},\ldots\}$ that are highly relevant to $I$ and synthesize their findings in the context of $I$ to generate a comprehensive evaluation $\mathcal{E}=(S,C)$ covering the soundness and contribution of the research idea. The evaluation is accompanied by citations, ensuring that all claims are supported by evidence from the literature.

Overview of ScholarEval. As shown in [Figure 2](https://arxiv.org/html/2510.16234v1#S2.F2), ScholarEval is composed of two main modules: Soundness and Contribution. We present both their workflows in §[2.1](https://arxiv.org/html/2510.16234v1#S2.SS1) and §[2.2](https://arxiv.org/html/2510.16234v1#S2.SS2) and provide further details in [Appendix A](https://arxiv.org/html/2510.16234v1#A1).

Figure: Figure 2: Overview of ScholarEval and its two modules. Top: The soundness module extracts the methods proposed in the research idea and conducts a thorough literature search for similar applications of each method to determine its potential effectiveness. Bottom: The contribution module identifies the dimensions along which the research idea is making contributions and conducts detailed comparisons with related papers along each dimension to identify areas of novelty or lack thereof.
Refer to caption: x3.png

### 2.1 Soundness Evaluation

The soundness module evaluates the methodological rigor of an idea by extracting methods, searching for relevant literature, and synthesizing evidence to assess whether the proposed approaches are well-supported or contradicted by existing work.

Method Extraction. The first objective of the soundness pipeline is to extract distinct methodological components including algorithmic approaches, experimental designs, evaluation protocols, ablation studies, or analytical frameworks from the idea. Formally, we leverage an LLM, referred to hereafter as $\mathcal{M}$, to extract all methods $M=\{m_{1},m_{2},\ldots,m_{k}\}$ from the research idea $I$.

Context Retrieval. This step gathers information from the literature about the effectiveness of the proposed methods. For each extracted method $m_{i}\in M$, $\mathcal{M}$ generates a relevant query for [Semantic Scholar snippet search](https://api.semanticscholar.org/api-docs/#tag/Snippet-Text/operation/get_snippet_search) (Kinney et al., 2025), which indexes 285.6M passages extracted from the title, abstract, or body of research papers (Singh et al., 2025). Since the queries are constructed from the description of the method $m_{i}$, it is likely to retrieve snippets within relevant methodology sections. We extract all papers referenced in these snippets, which provides a dense collection of relevant sources to broaden the understanding of $m_{i}$. We download the full text of these papers and parse them using [GROBID](https://github.com/kermitt2/grobid), a state-of-the-art document parsing tool, to extract three key sections from each paper: the methods section, to compare its similarity with the current method $m_{i}$; the results section, to judge the method’s effectiveness in a given context; and the abstract for a holistic view of the paper. At the end of this process, we obtain a list of related papers $P_{i}=\{p_{i,1},p_{i,2},\ldots,p_{i,n_{i}}\}$ for each method $m_{i}$, where each paper is represented as a triplet (abstract, methods, results). This list constitutes essential literature context to evaluate each method’s effectiveness.

Methods and Results Summarization. This stage serves two key functions: first, it filters out extracted papers that are not relevant to assessing the method $m_{i}$, and second, it condenses the most vital information from relevant papers, since retaining all paper data results in prohibitive context lengths. Specifically, for each paper $p_{i,j}\in P_{i}$, we instruct $\mathcal{M}$ to identify whether the methodology described in the paper is relevant to the method $m_{i}$, and if so, generate a compact summary of its methods and results grounded in the context of the method $m_{i}$ and the overall research idea $I$.

Soundness Review Synthesis. Grounded in the condensed context, the soundness pipeline concludes by synthesizing method-level soundness evaluations. Specifically, $\mathcal{M}$ analyzes the paper summaries in the context of the method $m_{i}$ and the research idea $I$ to synthesize three main sections: (1) Support: the support for the method $m_{i}$ based on the literature. This section details whether there are similar methods in the literature that have shown successful results, and uncovers how they relate to the current $m_{i}$. (2) Contradictions: the contradictions to the method $m_{i}$ based on the evidence extracted from the literature. In relation to the proposed method $m_{i}$, this section highlights the limitations that methods in a similar context have faced, signaling
its potential ineffectiveness. (3) Suggestions: based on the strengths and limitations of the method identified in the previous sections, $\mathcal{M}$ generates actionable suggestions for improvement to refine the proposed methodology.

### 2.2 Contribution Evaluation

The contribution module assesses the novelty and significance of an idea by identifying contribution dimensions, discovering related papers, conducting pairwise comparisons to determine how the idea advances the literature, and synthesizing a dimension-level contribution review.

##### Dimension Extraction.

The initial step of contribution evaluation consists of extracting the dimensions along which the research idea is making contributions to the field. Dimensions represent the facets of the idea’s potential contributions that are specific and comparable across related literature (e.g., system design, data collection, evaluation methodology, etc.). Instead of imposing pre-defined dimensions as in Radensky et al. (2025), we use LLM extraction to ensure flexibility based on the nature of the research idea (Rubaiat et al., 2025). Specifically, given the research idea $I$, we use $\mathcal{M}$ to extract dimensions $D=\{d_{1},d_{2},\ldots,d_{l}\}$. Examples of dimensions include tool or system design, conceptual framework, evaluation methodology, etc.
Each $d_{i}$ also includes the reasoning for how the idea makes contributions along that dimension. These statements will provide important context to ground the query generation
in the subsequent step.

##### Paper Discovery.

In this step, we conduct a broad search over the literature to identify relevant papers to compare $I$ against. However, unlike the soundness module, which requires searching paper content for methodological details, contribution evaluation can be performed using only abstracts, since a paper’s main contributions are typically highlighted there. This also allows us to cast a wider net and gather a broad set of related papers. Such breadth is essential for contribution evaluation, as determining truly novel contributions requires an exhaustive view of the literature.
The paper discovery thus consists of the following steps: (1) For each extracted dimension, $\mathcal{M}$ generates queries to search for relevant papers using [Semantic Scholar paper search](https://api.semanticscholar.org/api-docs/#tag/Paper-Data/operation/get_graph_paper_relevance_search) and retrieve their abstracts. (2) $\mathcal{M}$ assesses similarity in contributions of each candidate paper abstract relative to the research idea $I$ and assigns a score on a scale from 1 to 5. Papers that are deemed highly relevant (i.e., score $\geq 3$) are then used as seeds for the paper augmentation stage. (3) Paper augmentation leverages the [Semantic Scholar Recommendations API](https://api.semanticscholar.org/api-docs/recommendations) to find similar papers, and we additionally extract the publications cited by each seed paper. (4) This augmented list of candidate papers then undergoes another stage of relevance assessment by $\mathcal{M}$. However, due to the typically large volume of papers at this stage, we first filter this list to the top $n$ papers based on semantic embedding(^2^22We use Titan Text Embedding v2 (Amazon Web Services, 2025).) similarity between abstracts and $I$ before forwarding to $\mathcal{M}$.

##### Pairwise Comparison.

Once the final set of papers $P_{D}=\{p_{1},p_{2},\ldots,p_{m}\}$ is identified, we conduct a series of pairwise comparisons to uncover how these papers’ contributions compare to those proposed in $I$. Specifically, we prompt $\mathcal{M}$ to compare each paper’s abstract to $I$ along each dimension $d_{i}\in D$. This comparison produces a granular view of the areas in which $I$ is making novel advancements and those in which it is lacking in novelty, compared to existing work.

##### Contribution Review Synthesis.

The final stage of contribution evaluation consists of synthesizing dimension-level contribution assessments. For each $d_{i}\in D$, $\mathcal{M}$ uses the results of the pairwise comparisons to synthesize an evaluation composed of three sections: (1) Strengths: the novel contributions that $I$ makes along this dimension compared to prior work. (2) Weaknesses: the areas in which the contributions of $I$ are lacking or limited compared to prior work. (3) Suggestions: actionable recommendations to improve the novelty of $I$ along this dimension.

## 3 ScholarIdeas : A dataset for research idea evaluation

We design ScholarEval to provide a holistic evaluation of research ideas within a domain-agnostic framework.
A meaningful assessment of ScholarEval’s quality thus hinges on a multi-domain collection of research ideas paired with ground-truth reviews.
However, existing datasets and benchmarks fall short: they focus on full paper reviews rather than research ideas (Kang et al., 2018; Weng et al., 2025), capture singular dimensions such as novelty (Shahid et al., 2025), or are restricted to one discipline (e.g., AI; Si et al., 2025b).

To this end, we curate ScholarIdeas, a dataset containing research ideas and reviews covering four disciplines: artificial intelligence, neuroscience, biochemistry, and ecology. We employ a semi-automatic pipeline to construct this dataset, where each example is validated by subject-matter experts (§[3.1](https://arxiv.org/html/2510.16234v1#S3.SS1)). Our primary motivation for developing ScholarEval is to evaluate AI-generated ideas. However, collecting detailed expert reviews for AI-generated ideas at scale is prohibitively expensive and practically infeasible. Our dataset of existing human-written ideas with expert reviews serves as a reliable proxy.

The evaluation of long-form responses, such as the ones generated by ScholarEval, is also inherently challenging. While other scientific tasks admit easily verifiable success criteria (e.g., code generation (Jansen et al., 2025; Chen et al., 2025; Li et al., 2025b)), such evaluation metrics are not readily available for our use case.
We develop
a multi-faceted automatic evaluation pipeline detailed in §[3.2](https://arxiv.org/html/2510.16234v1#S3.SS2). We further corroborate our automatic evaluation with a large-scale human evaluation described in §[5](https://arxiv.org/html/2510.16234v1#S5).

### 3.1 Data Curation and Annotation

Data selection. We manually select papers and reviews from two sources: [OpenReview (ICLR 2025)](https://openreview.net/group?id=ICLR.cc/2025/Conference) for AI-related research ideas and [eLife](https://elifesciences.org/) for life sciences. Both sources include high-quality reviews by multiple reviewers for all submitted papers,
in contrast to many other sources that do not openly release reviews or restrict them to those of accepted manuscripts. To ensure the suitability of the reviews to be used as ground-truth, we only select papers satisfying the following criteria: (1) The paper must be reviewed by at least two reviewers. (2) All reviews of the paper are in agreement and offer a general consensus about the quality of the work. (3) The reviews mention criticism about the underlying research idea and not exclusively about obtained results or other details known post-execution. For each of the sources we use, we only retrieve the first version of the submission prior to any revisions, along with the first round of reviews. We also annotate each research idea with its publication date, which we use as a cutoff date for literature search during the execution of baselines and ScholarEval.

LLM-based extraction. After identifying 130 papers across the four disciplines of interest satisfying our criteria, we follow a multi-step approach to extract the research idea and its corresponding reviews. (1) Document parsing. We first use GROBID to parse the paper into different sections and employ a heuristic on section names to only keep those related to the background and methodology, while dropping all sections mentioning experiment execution and results. (2) Research idea extraction. We instruct an LLM(^3^33We use Claude 4 Sonnet.) to extract the research idea based on the parsed documents such that each research idea is composed of three components (Problem, Methods, Experiments). (3) Review rubrics extraction. Finally, we provide an LLM with the full reviews corresponding to the paper along with the extracted research idea and instruct it to extract statements from the reviews pertaining to assessments of the underlying research idea
These statements are further classified by the LLM based on their type (strength or weakness), severity (major or minor), and axis (soundness or contribution) to construct our review rubrics.

Expert Validation. We invite 6 subject-matter experts (PhD students, postdocs, and professors) in artificial intelligence, neuroscience, biochemistry, and ecology to validate the quality of each (research idea, review rubrics) pair extracted by the LLM. Namely, the experts are instructed to ensure that the research idea is a faithful representation of the paper at the ideation stage, that the review rubrics do not have mentions to execution, results, or paper presentation, and that the research idea and review rubrics are consistent (i.e. the review only addresses aspects contained in the research idea). Annotators are also asked to verify the correctness of the dimension assignments made by the LLMs. Experts are instructed to make necessary corrections and to discard instances that fall out of their specific area of expertise.

At the end of this process, we obtain 117 validated (research idea, review rubrics) pairs balanced across four disciplines. [Figure 3](https://arxiv.org/html/2510.16234v1#S3.F3) shows the statistics of ScholarIdeas and an example pair. Further details about ScholarIdeas curation are given in [Appendix C](https://arxiv.org/html/2510.16234v1#A3).

Figure: Figure 3: Overview of ScholarIdeas. Left: ScholarIdeas includes 117 research ideas across four disciplines. Right: Example of a neuroscience research idea and review in ScholarIdeas. Each review is composed of multiple rubrics, for a total of 1076 rubrics across the dataset. This idea-review pair is adapted from (Bloem et al., 2025).
Refer to caption: x4.png

### 3.2 Evaluation Protocol

A high-quality research idea evaluation should highlight all strengths and weaknesses that an expert reviewer would mention, ground its claims in relevant literature, engage deeply with the points discussed and the works cited, and go beyond simple good or bad judgment to offer actionable suggestions for improvement. Guided by this desiderata, we develop automatic metrics to assess the performance of ScholarEval and baselines. Full details of prompts and implementation are provided in the [Appendix D](https://arxiv.org/html/2510.16234v1#A4).

.
This is a recall-based metric which measures the extent to which an evaluation covers the rubrics from the corresponding review in ScholarIdeas. Specifically, we use [Prometheus-Eval](https://github.com/prometheus-eval/prometheus-eval) with a GPT-4 backbone (OpenAI et al., 2024), a framework shown to follow detailed evaluation rubrics effectively (Kim et al., 2024). We instruct it to assign a 1–5 score based on how well each evaluation addresses the points in the reference rubric. The final score is the average over all 1,076 rubrics in ScholarIdeas.

. To compute the reference invalidity rate, we automatically check the references (i.e., paper links) cited in the evaluation to verify whether they correspond to existing papers. This is done by examining the status code returned for each link. Since edge cases arise where status codes do not clearly indicate validity, we conservatively report a lower bound: the proportion of references that are clearly invalid.

, Act, Depth. We use an LLM-as-judge, Claude 4 Sonnet, to give pairwise preferences between ScholarEval and our strongest baseline along three criteria: Evidence Support, which measures how well claims are grounded in literature and supported by relevant citations; Actionability, which captures the clarity, usefulness, and feasibility of the suggestions; and Depth, which evaluates the level of engagement with each point and whether the evaluation mentions specifics about the work it cites rather than relying on generic statements. We also compute agreement with human annotators (see [Appendix D](https://arxiv.org/html/2510.16234v1#A4)).

## 4 Experiments and Results

### 4.1 Experimental Details

Baselines. Because there are no existing systems in the literature that evaluate research ideas with feedback at the same scope as ScholarEval, we compare against strong baselines that are most likely to be used for idea evaluation and feedback. Specifically, we select both frontier open-source and closed-source LLMs—Llama-3.3-70B (Grattafiori et al., 2024), GPT-4.1 (OpenAI, 2025a), Claude-4-Sonnet (Anthropic, 2025), and GPT-4o-search-preview (OpenAI, 2025c) for a web-connected LLM baseline—as well as a deep research system, o4-mini-deep-research (OpenAI, 2025b). We restrict our choice of deep research systems o4-mini-deep-research since it is available via API.
All baselines are prompted with a detailed template aligned with ScholarEval, including method and dimension decomposition and dedicated sections for strengths, weaknesses, and suggestions.

ScholarEval instantiation. We evaluate three variants of ScholarEval: ScholarEval${}_{\text{Llama}}$, ScholarEval${}_{\text{GPT}}$, and ScholarEval${}_{\text{Claude}}$, which use Llama-3.3-70B, GPT-4.1, and Claude-4-Sonnet as backbones, respectively.

Full prompts and additional experimental setup details are provided in [Appendix E](https://arxiv.org/html/2510.16234v1#A5).

### 4.2 Results and Analysis

**Table 1: Coverage $\uparrow$ of baselines and variants of ScholarEval overall and per-discipline. ^∗ and ^† indicate significant improvement over 1 or all baselines, respectively. Best results are bolded. Statistical significance details are provided in [Appendix F](https://arxiv.org/html/2510.16234v1#A6).**
|  | Overall | AI | Neuroscience | Biochemistry | Ecology |
| --- | --- | --- | --- | --- | --- |
| Llama-3.3-70B | 1.83 $\pm$ 0.89 | 1.88 $\pm$ 0.92 | 1.82 $\pm$ 0.86 | 1.81 $\pm$ 0.90 | 1.74 $\pm$ 0.87 |
| GPT-4.1 | 2.18 $\pm$ 1.08 | 2.21 $\pm$ 1.12 | 2.18 $\pm$ 1.10 | 2.14 $\pm$ 1.04 | 2.13 $\pm$ 1.00 |
| Claude-4-Sonnet | 2.18 $\pm$ 1.04 | 2.28 $\pm$ 1.02 | 2.15 $\pm$ 1.08 | 2.01 $\pm$ 1.01 | 2.11 $\pm$ 1.03 |
| GPT-4o-search-preview | 1.90 $\pm$ 0.98 | 1.95 $\pm$ 0.97 | 1.84 $\pm$ 0.96 | 1.86 $\pm$ 1.00 | 1.95 $\pm$ 1.02 |
| o4-mini-deep-research | 2.28 $\pm$ 1.07 | 2.35 $\pm$ 1.07 | 2.25 $\pm$ 1.04 | 2.08 $\pm$ 1.06 | 2.35 $\pm$ 1.11 |
| ScholarEval ${}_{\text{Llama}}$ | 2.04 $\pm$ 1.16^∗ | 2.06 $\pm$ 1.14^∗ | 2.05 $\pm$ 1.18^∗ | 2.04 $\pm$ 1.19 | 1.94 $\pm$ 1.13 |
| ScholarEval ${}_{\text{GPT}}$ | 2.72 $\pm$ 1.47^† | 2.84 $\pm$ 1.39^† | 2.74 $\pm$ 1.55^† | 2.61 $\pm$ 1.42^† | 2.52 $\pm$ 1.51^† |
| ScholarEval ${}_{\text{Claude}}$ | 2.77 $\pm$ 1.40^† | 2.91 $\pm$ 1.34^† | 2.55 $\pm$ 1.45^† | 2.64 $\pm$ 1.40^† | 2.90 $\pm$ 1.39^† |

Figure: Figure 4: Win rate based on Depth, Act, and Evidence between ScholarEval ${}_{\text{Claude}}$ and o4-mini-deep-research using LLM judge.
Refer to caption: x5.png

##### ScholarEval outperforms baselines in rubric coverage across disciplines.

As shown in [Table 1](https://arxiv.org/html/2510.16234v1#S4.T1),
ScholarEval${}_{\text{GPT}}$ and ScholarEval${}_{\text{Claude}}$ achieve statistically significant gains on Coverage $\uparrow$ over all baselines overall and in every discipline. ScholarEval${}_{\text{Llama}}$ also surpasses its backbone (Llama-3.3-70B) in overall coverage and in AI and Neuroscience, indicating that ScholarEval delivers improvements across backbones. Importantly, these gains are not explained by emphasizing minor rubric items only; ScholarEval also provides better coverage of expert-annotated *major* points (see [Table 13](https://arxiv.org/html/2510.16234v1#A6.T13) in Appendix).

ScholarEval eliminates reference invalidity.
As shown in [subsection 4.2](https://arxiv.org/html/2510.16234v1#S4.SS2), only ScholarEval variants achieve 0% Reference Inv. $\downarrow$ in our evaluation. As previously stated, baseline rates represent lower bounds. However, a manual audit of a sample of outputs (see §[F.2](https://arxiv.org/html/2510.16234v1#A6.SS2)) indicates that up to 80% of Llama-3.3-70B citations are hallucinated in certain evaluations. Even GPT-4o-search-preview and o4-mini-deep-research are not immune, with at least 1% of cited references not existing. Our audit also uncovers subtler failures, including misattributed authors and claims that are inconsistent with the cited sources. These patterns align with findings from recent evaluations of deep research systems (Li et al., 2025a). ScholarEval eliminates these issues, ensuring that all citations resolve to valid, traceable sources.

**Table 3: Ablations of different components of ScholarEval on ScholarIdeas-AI.**
|  | Coverage $\uparrow$ |
| --- | --- |
| ScholarEval ${}_{\text{Claude}}$ | 2.91 $\pm$ 1.34 |
| -MRE | 2.47 $\pm$ 1.42 |
| -PA | 2.42 $\pm$ 1.28 |
| -PC | 2.39 $\pm$ 1.23 |

Quality of ScholarEval evaluations exceeds that of deep research. We compare ScholarEval${}_{\text{Claude}}$ to the strongest baseline, o4-mini-deep-research based on the metrics Evidence, Act and Depth. Results in [Figure 4](https://arxiv.org/html/2510.16234v1#S4.F4) indicate that ScholarEval${}_{\text{Claude}}$ is consistently better at giving evidence-based evaluations, making actionable suggestions to improve the research idea, as well as giving sufficient details and deeply engaging with the literature it cites. Our user study detailed in §[5](https://arxiv.org/html/2510.16234v1#S5) further corroborates these results by the preference of human users.

Ablation studies. We conduct ablations to assess the effectiveness of individual components of ScholarEval. Namely, we remove each of the methods and results extraction (MRE), paper augmentation (PA), and pairwise comparison (PC). Details about the setup for each ablation experiment can be found in [Appendix E](https://arxiv.org/html/2510.16234v1#A5). Results in [Table 3](https://arxiv.org/html/2510.16234v1#S4.T3) indicate that the removal of each of these components leads to degradation in performance of ScholarEval on Coverage $\uparrow$. This is likely due to the reduced information given to the model (i.e. the effectiveness of the methods from prior work based on the reported results, the dense list of relevant papers, and the granular dimension-level paper comparisons). These results showcase the importance of these steps in providing ScholarEval with essential context from the literature to generate effective idea evaluations.

## 5 Expert User Study

Design. We gauge the real-world usefulness of ScholarEval by recruiting 18 experts for 46 total evaluations in a blind experiment with OpenAI’s o4-mini-deep-research OpenAI (2025b). Each expert had an education level of PhD student or beyond and was verified to have at least one paper published in their field (see [Table 16](https://arxiv.org/html/2510.16234v1#A7.T16)). We create a blind interface for experts to interact with ScholarEval (see [Figure 6](https://arxiv.org/html/2510.16234v1#A8.F6) and [Figure 7](https://arxiv.org/html/2510.16234v1#A8.F7)).

Procedure. We ask each expert participant to first write a research idea that includes the problem they aim to tackle and the suggested methodology. This can be one they have yet to experiment with, or an idea they have already published.
After receiving the soundness and contribution evaluation from a single system (either ScholarEval or OpenAI deep research), they complete a detailed rubric related to the core components of idea evaluation (e.g. the usefulness of the suggestions, the engagement with literature, etc.).
The entire process for creating an idea, generating the feedback, and scoring the rubric took an average of 1 hour across participants, and we allow up to 4 unique idea submissions. The exact questions, user demographics, blindness and randomization validity, and compensation details can be found in [Appendix G](https://arxiv.org/html/2510.16234v1#A7).

Evaluation.
We group questions into six dimensions. Citations captures the number of references experts would actually use in their research. Because research ideas can be complex and domain-specific, we use Faithful to measure the extent to which the feedback aligns with nuances of the research idea. Useful reflects the helpfulness of each system and the expert’s enthusiasm for future use. Focus measures the degree to which each system targets the most important factors of each research idea. LitEngage indicates the depth each system uses when making detailed comparisons with specific components of relevant literature. Refine gauges whether the system provides valuable, targeted, feasible suggestions for improvement that experts believe would actually improve their research idea. The questions related to each dimension are available in [Table 15](https://arxiv.org/html/2510.16234v1#A7.T15). We refer readers to §[G.2](https://arxiv.org/html/2510.16234v1#A7.SS2) for details on linear mixed effects model, which we adopt as our statistical method.

**Table 4: Mixed effects modeling results from the expert user study. We report the standardized regression coefficient ($\beta$) and statistical significance (* $p<.05$, *** $p<.001$). Deep Research refers to o4-mini-deep-research.**
| Dimension | $\beta$ | ScholarEval | Deep Research |
| --- | --- | --- | --- |
| Citations | 0.81^*** | 2.66 $\pm$ 2.43 | 1.09 $\pm$ 1.28 |
| Faithful | 0.69^* | 3.88 $\pm$ 1.50 | 2.91 $\pm$ 1.19 |
| Useful | 0.63^* | 7.80 $\pm$ 1.71 | 6.56 $\pm$ 2.05 |
| Focus | 0.61^* | 7.48 $\pm$ 1.26 | 6.46 $\pm$ 2.09 |
| LitEngage | 0.61^* | 7.62 $\pm$ 1.75 | 6.41 $\pm$ 1.72 |
| Refine | 0.60^* | 7.31 $\pm$ 1.62 | 6.12 $\pm$ 2.32 |

Results and analysis. Results in [Table 4](https://arxiv.org/html/2510.16234v1#S5.T4) show a statistically significant preference for ScholarEval over o4-mini-deep-research across all six dimensions measured in our expert user study. Experts found 1.5 more useful Citations using ScholarEval and scored LitEngage 1.2 higher, demonstrating the effectiveness of our multi-stage literature retrieval pipeline. A strong effect on Faithful and Focus underscores the benefit of systemically breaking down the research plan into smaller, controlled comparisons with relevant literature. The actionable feedback generated by ScholarEval had clear advantages as well, with experts reporting a 1.2 increase in Refine. The higher score on Useful further underscores the overall usefulness of ScholarEval as research evaluation framework as judged by experts.

## 6 Related Work

Literature Grounded Systems for Research. Multiple works have been proposed that use LLMs in literature grounded systems to assist in research. A commonly targeted use case for these systems is literature synthesis and related work generation (Asai et al., 2024; Kang et al., 2023; Agarwal et al., 2025). In addition, other systems have been developed for literature understanding and question answering (Skarlinski et al., 2024; Singh et al., 2025). ScholarEval builds on techniques inspired by these works but focuses on the underexplored problem of literature-grounded research idea evaluation.

Research Ideation. Recently, there has been growing interest in using LLMs for generating research ideas and scientific hypotheses, often from a literature-driven perspective. For example, Wang et al. (2024) proposed SciMON, a framework that retrieves inspirations from past scientific papers to generate novel ideas. Similarly, Baek et al. (2025) employ an iterative approach over related papers and a knowledge store to generate scientific hypotheses. Other works, such as (Radensky et al., 2025; Garikaparthi et al., 2025; Pu et al., 2025), emphasize human-LLM collaboration for idea generation, while Gottweis et al. (2025) use a multi-agent approach to generate and refine hypotheses. Furthermore, Si et al. (2025b) conducted a large-scale study assessing both human- and LLM-generated ideas in the AI field, finding that LLMs are capable of generating more novel ideas based on human evaluation.

Research Idea Evaluation. Many of the aforementioned ideation systems also incorporate modules for idea review and refinement (Baek et al., 2025; Radensky et al., 2025). For instance, Baek et al. (2025) use a review agent that is prompted with a rubric induced from human preference. Similarly, Shahid et al. (2025); Radensky et al. (2025) introduce a retrieval-augmented framework that generates novelty classifications with brief reasoning. Additionally, Feng et al. (2025) introduced a graph-based LLM framework to score research ideas. A concurrent work by Afzal et al. (2025) proposes an LLM framework specifically for paper novelty evaluation. Wen et al. (2025) tackle the problem of idea evaluation by building a system that predicts empirical outcomes in AI research, choosing the most promising idea among a pair.
ScholarEval addresses many of the limitations of these systems. First, rather than relying on the parametric knowledge of LLMs for idea evaluation as in Baek et al. (2025), ScholarEval incorporates a carefully designed retrieval pipeline to ground evaluation in the most recent literature. Second, it goes beyond one-dimensional evaluation (e.g., novelty) (Shahid et al., 2025; Afzal et al., 2025) to also assess the validity of proposed methods, since novelty alone is not a sufficient condition for successful research execution (Si et al., 2025a). Finally, ScholarEval places strong emphasis on generating dense and actionable feedback for idea refinement, in contrast to systems that only produce scores (Feng et al., 2025; Wen et al., 2025) or sparse feedback (Shahid et al., 2025).

## 7 Conclusion

We introduced ScholarEval, a framework for research idea evaluation that assesses ideas based on their soundness and contribution, grounded in scholarly literature. We also presented ScholarIdeas, a multidisciplinary dataset of research ideas paired with expert-annotated review rubrics. Our experiments demonstrate that ScholarEval achieves higher coverage of points raised by human reviewers and consistently delivers higher-quality evaluations compared to baselines. Moreover, our user study shows a strong preference for ScholarEval as a useful idea evaluation tool, providing a confident indication of its potential to augment the ideation process in both AI and human workflows.

## Limitations and Future Work

We recognize the following limitations and future work directions:

Limitations of ScholarEval. First, as a literature-grounded framework, ScholarEval relies heavily on the retrieved literature to evaluate research ideas. Hence, it is possible that it might misjudge a method’s effectiveness if it is not yet proven in the existing literature. Similarly, the contribution of a research idea might be misrepresented if some relevant papers are not retrieved for comparison. A meaningful direction for future work would be to improve the retrieval capabilities of ScholarEval to reduce such occurrences. Second, as we mention in Appendix [F](https://arxiv.org/html/2510.16234v1#A6), depending on the choice of the model backbone, running an idea evaluation using ScholarEval can take around 12 min and cost up to $3. Although not too high in absolute terms, these costs can add up if ScholarEval is used to evaluate a large batch of ideas. Future work may explore more lightweight versions of ScholarEval to enhance its usability in such cases.

Limitations of ScholarIdeas and automatic evaluation. We have made a considerable effort to select papers with high-quality reviews. However, idea evaluation remains a subjective task and human reviews cannot always be considered as ground-truth. Although the degree of coverage of human reviews can be one signal of evaluation comprehensiveness, it is not a definitive judge of overall quality. To that end, we have included other metrics in our setup to give a more holistic assessment, but the evaluation of open-form responses remains challenging, and there are other facets of evaluation that we have considered but that were challenging to scale or automate (e.g. citation factuality and relevance, usefulness, etc.). Additionally, our evaluation is limited to the four disciplines included in ScholarIdeas, and although our framework is discipline-agnostic, our results might not necessarily generalize to other disciplines. Upon acceptance, we will release our entire dataset and evaluation pipeline, including the UI we used for the user study, seeking feedback from the wider scientific community on ScholarEval’s utility.

Limitations of Expert User Study.
Some measures, such as those related to Focus, are inherently subjective to the user, and could be influenced by stylistic factors rather than actual improved quality. Additionally, resarch idea quality is a source of variance, as participants were only allowed to submit unique research ideas to a given system. However, we collect a self-report of research idea detail, which is not statistically significantly different according to a Welch’s t-test. Additionally, the overall scores of our results may be inflated by our expert sample, who report a 7.2/10 on AI use for research. Our results may be different in a sample who use AI less often. This does not affect our relative improvement over o4-mini-deep-research, as we show concrete evidence that the study was adequately blinded (see [Appendix G](https://arxiv.org/html/2510.16234v1#A7)). Future work should consider the barriers to adoption for automated research idea evaluation, especially for those who are less keen on using AI for their research cycle. Another strong direction would study how use of ScholarEval impacts short and long-term research success.

## Reproducibility Statement

To ensure the reproducibility of our work, we provide full prompts and implementation details of ScholarEval ([Appendix A](https://arxiv.org/html/2510.16234v1#A1)), the detailed process of constructing ScholarIdeas ([Appendix C](https://arxiv.org/html/2510.16234v1#A3)), full information about our evaluation ([Appendix D](https://arxiv.org/html/2510.16234v1#A4)), experimental setup ([Appendix E](https://arxiv.org/html/2510.16234v1#A5)), and expert user study ([Appendix G](https://arxiv.org/html/2510.16234v1#A7)). We also openly release our code and dataset.

## Ethics Statement

ScholarIdeas is collected from papers on Openreview and eLife licensed under a Creative Commons Attribution (CC BY) license, which allows reuse with attribution. We provide the full list of papers used to create ScholarIdeas in [Appendix C](https://arxiv.org/html/2510.16234v1#A3).

## Author Contributions

- •
Project leadership: Hanane Nour Moussa, Patrick Queiroz Da Silva
- •
Project conception: Bodhisattwa Prasad Majumder, Sachin Kumar, Hanane Nour Moussa, Patrick Queiroz Da Silva
- •
Development of ScholarEval: Hanane Nour Moussa, Patrick Queiroz Da Silva
- •
ScholarIdeas paper curation and LLM-based extraction: Hanane Nour Moussa
- •
ScholarIdeas expert validation: Hanane Nour Moussa, Daniel Adu-Ampratwum, Alyson East, Zitong Lu, Nikki Puccetti, Mingyi Xue
- •
Evaluation pipeline and ablation experiments: Hanane Nour Moussa
- •
Expert recruitment for user study: Patrick Queiroz Da Silva, Hanane Nour Moussa
- •
ScholarEval interface for user study: Patrick Queiroz Da Silva, Hanane Nour Moussa
- •
User study results analysis: Patrick Queiroz Da Silva
- •
Manuscript writing and revision: Hanane Nour Moussa, Patrick Queiroz Da Silva, Sachin Kumar, Huan Sun, Bodhisattwa Prasad Majumder
- •
Advisory: Sachin Kumar, Bodhisattwa Prasad Majumder, Huan Sun

## Acknowledgments

This material is based upon work supported by the Ai2 Faculty Research Award.
We thank Aaron Jencks, Jiwoo Park, and Yusen Peng for helping with data annotation of the quality metrics of ScholarEval and o4-mini-deep-research. We are also grateful to colleagues from Ai2 for their valuable feedback on an early version of the project, as well as colleagues from OSU NLP for their comments on the first draft of the manuscript.

## References

- Afzal et al. (2025)
Osama Mohammed Afzal, Preslav Nakov, Tom Hope, and Iryna Gurevych.
Beyond "not novel enough": Enriching scholarly critique with llm-assisted feedback, 2025.
URL [https://arxiv.org/abs/2508.10795](https://arxiv.org/abs/2508.10795).
- Agarwal et al. (2025)
Shubham Agarwal, Gaurav Sahu, Abhay Puri, Issam H. Laradji, Krishnamurthy DJ Dvijotham, Jason Stanley, Laurent Charlin, and Christopher Pal.
Litllm: A toolkit for scientific literature review, 2025.
URL [https://arxiv.org/abs/2402.01788](https://arxiv.org/abs/2402.01788).
- Amazon Web Services (2025)
Amazon Web Services.
Amazon titan text embeddings models.
[https://docs.aws.amazon.com/bedrock/latest/userguide/titan-embedding-models.html](https://docs.aws.amazon.com/bedrock/latest/userguide/titan-embedding-models.html), 2025.
Accessed: 2025-09-21.
- Anthropic (2025)
Anthropic.
Claude sonnet 4, 2025.
URL [https://www.anthropic.com/claude/sonnet](https://www.anthropic.com/claude/sonnet).
- Asai et al. (2024)
Akari Asai, Jacqueline He, Rulin Shao, Weijia Shi, Amanpreet Singh, Joseph Chee Chang, Kyle Lo, Luca Soldaini, Sergey Feldman, Mike D’arcy, David Wadden, Matt Latzke, Minyang Tian, Pan Ji, Shengyan Liu, Hao Tong, Bohao Wu, Yanyu Xiong, Luke Zettlemoyer, Graham Neubig, Dan Weld, Doug Downey, Wen tau Yih, Pang Wei Koh, and Hannaneh Hajishirzi.
Openscholar: Synthesizing scientific literature with retrieval-augmented lms, 2024.
URL [https://arxiv.org/abs/2411.14199](https://arxiv.org/abs/2411.14199).
- Baek et al. (2025)
Jinheon Baek, Sujay Kumar Jauhar, Silviu Cucerzan, and Sung Ju Hwang.
ResearchAgent: Iterative research idea generation over scientific literature with large language models.
In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), *Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)*, pp. 6709–6738, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics.
ISBN 979-8-89176-189-6.
doi: 10.18653/v1/2025.naacl-long.342.
URL [https://aclanthology.org/2025.naacl-long.342/](https://aclanthology.org/2025.naacl-long.342/).
- Bakalarski et al. (2016)
Corey E. Bakalarski, Yutian Gan, Ingrid Wertz, Jennie R. Lill, and Wendy Sandoval.
Rapid, semi-automated protein terminal characterization using isdetect.
*Nature Biotechnology*, 34(8):811–813, August 2016.
doi: 10.1038/nbt.3621.
URL [https://doi.org/10.1038/nbt.3621](https://doi.org/10.1038/nbt.3621).
- Bloem et al. (2025)
Ilona M. Bloem, Leah Bakst, Joseph T. McGuire, and Sam Ling.
Dynamic estimation of the attentional field from visual cortical activity.
*eLife*, 14:RP104222, 2025.
doi: 10.7554/eLife.104222.1.
URL [https://doi.org/10.7554/eLife.104222.1](https://doi.org/10.7554/eLife.104222.1).
- Brotherton & Balskus (2013)
Carolyn A. Brotherton and Emily P. Balskus.
A prodrug resistance mechanism is involved in colibactin biosynthesis and cytotoxicity.
*Journal of the American Chemical Society*, 135(9):3359–3362, March 2013.
doi: 10.1021/ja312154m.
URL [https://doi.org/10.1021/ja312154m](https://doi.org/10.1021/ja312154m).
- Cao et al. (2024)
Meng Cao, Lei Shu, Lei Yu, Yun Zhu, Nevan Wichers, Yinxiao Liu, and Lei Meng.
Enhancing reinforcement learning with dense rewards from language model critic.
In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), *Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, pp. 9119–9138, Miami, Florida, USA, November 2024. Association for Computational Linguistics.
doi: 10.18653/v1/2024.emnlp-main.515.
URL [https://aclanthology.org/2024.emnlp-main.515/](https://aclanthology.org/2024.emnlp-main.515/).
- Chen et al. (2025)
Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang, Botao Yu, Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, Vishal Dey, Mingyi Xue, Frazier N. Baker, Benjamin Burns, Daniel Adu-Ampratwum, Xuhui Huang, Xia Ning, Song Gao, Yu Su, and Huan Sun.
Scienceagentbench: Toward rigorous assessment of language agents for data-driven scientific discovery.
In *The Thirteenth International Conference on Learning Representations*, 2025.
URL [https://openreview.net/forum?id=6z4YKr0GK6](https://openreview.net/forum?id=6z4YKr0GK6).
- Feng et al. (2025)
Tao Feng, Yihang Sun, and Jiaxuan You.
Grapheval: A lightweight graph-based LLM framework for idea evaluation.
In *The Thirteenth International Conference on Learning Representations*, 2025.
URL [https://openreview.net/forum?id=5RUM1aIdok](https://openreview.net/forum?id=5RUM1aIdok).
- Garikaparthi et al. (2025)
Aniketh Garikaparthi, Manasi Patwardhan, Lovekesh Vig, and Arman Cohan.
IRIS: Interactive research ideation system for accelerating scientific discovery.
In Pushkar Mishra, Smaranda Muresan, and Tao Yu (eds.), *Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)*, pp. 592–603, Vienna, Austria, July 2025. Association for Computational Linguistics.
ISBN 979-8-89176-253-4.
doi: 10.18653/v1/2025.acl-demo.57.
URL [https://aclanthology.org/2025.acl-demo.57/](https://aclanthology.org/2025.acl-demo.57/).
- Gottweis et al. (2025)
Juraj Gottweis, Wei-Hung Weng, Alexander Daryin, Tao Tu, Anil Palepu, Petar Sirkovic, Artiom Myaskovsky, Felix Weissenberger, Keran Rong, Ryutaro Tanno, Khaled Saab, Dan Popovici, Jacob Blum, Fan Zhang, Katherine Chou, Avinatan Hassidim, Burak Gokturk, Amin Vahdat, Pushmeet Kohli, Yossi Matias, Andrew Carroll, Kavita Kulkarni, Nenad Tomasev, Yuan Guan, Vikram Dhillon, Eeshit Dhaval Vaishnav, Byron Lee, Tiago R D Costa, José R Penadés, Gary Peltz, Yunhan Xu, Annalisa Pawlosky, Alan Karthikesalingam, and Vivek Natarajan.
Towards an ai co-scientist, 2025.
URL [https://arxiv.org/abs/2502.18864](https://arxiv.org/abs/2502.18864).
- Grattafiori et al. (2024)
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al.
The llama 3 herd of models, 2024.
URL [https://arxiv.org/abs/2407.21783](https://arxiv.org/abs/2407.21783).
- Jansen et al. (2025)
Peter Jansen, Oyvind Tafjord, Marissa Radensky, Pao Siangliulue, Tom Hope, Bhavana Dalvi Mishra, Bodhisattwa Prasad Majumder, Daniel S Weld, and Peter Clark.
CodeScientist: End-to-end semi-automated scientific discovery with code-based experimentation.
In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), *Findings of the Association for Computational Linguistics: ACL 2025*, pp. 13370–13467, Vienna, Austria, July 2025. Association for Computational Linguistics.
ISBN 979-8-89176-256-5.
doi: 10.18653/v1/2025.findings-acl.692.
URL [https://aclanthology.org/2025.findings-acl.692/](https://aclanthology.org/2025.findings-acl.692/).
- Kang et al. (2018)
Dongyeop Kang, Waleed Ammar, Bhavana Dalvi, Madeleine van Zuylen, Sebastian Kohlmeier, Eduard Hovy, and Roy Schwartz.
A dataset of peer reviews (PeerRead): Collection, insights and NLP applications.
In Marilyn Walker, Heng Ji, and Amanda Stent (eds.), *Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)*, pp. 1647–1661, New Orleans, Louisiana, June 2018. Association for Computational Linguistics.
doi: 10.18653/v1/N18-1149.
URL [https://aclanthology.org/N18-1149/](https://aclanthology.org/N18-1149/).
- Kang et al. (2023)
Hyeonsu B Kang, Tongshuang Wu, Joseph Chee Chang, and Aniket Kittur.
Synergi: A mixed-initiative system for scholarly synthesis and sensemaking.
In *Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology*, UIST ’23, New York, NY, USA, 2023. Association for Computing Machinery.
ISBN 9798400701320.
doi: 10.1145/3586183.3606759.
URL [https://doi.org/10.1145/3586183.3606759](https://doi.org/10.1145/3586183.3606759).
- Kim et al. (2024)
Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo.
Prometheus 2: An open source language model specialized in evaluating other language models, 2024.
- Kinney et al. (2025)
Rodney Kinney, Chloe Anastasiades, Russell Authur, Iz Beltagy, Jonathan Bragg, Alexandra Buraczynski, Isabel Cachola, Stefan Candra, Yoganand Chandrasekhar, Arman Cohan, Miles Crawford, Doug Downey, Jason Dunkelberger, Oren Etzioni, Rob Evans, Sergey Feldman, Joseph Gorney, David Graham, Fangzhou Hu, Regan Huff, Daniel King, Sebastian Kohlmeier, Bailey Kuehl, Michael Langan, Daniel Lin, Haokun Liu, Kyle Lo, Jaron Lochner, Kelsey MacMillan, Tyler Murray, Chris Newell, Smita Rao, Shaurya Rohatgi, Paul Sayre, Zejiang Shen, Amanpreet Singh, Luca Soldaini, Shivashankar Subramanian, Amber Tanaka, Alex D. Wade, Linda Wagner, Lucy Lu Wang, Chris Wilhelm, Caroline Wu, Jiangjiang Yang, Angele Zamarron, Madeleine Van Zuylen, and Daniel S. Weld.
The semantic scholar open data platform, 2025.
URL [https://arxiv.org/abs/2301.10140](https://arxiv.org/abs/2301.10140).
- Kuswanto et al. (2023)
W. Kuswanto, G. Nolan, and G. Lu.
Highly multiplexed spatial profiling with codex: bioinformatic analysis and application in human disease.
*Seminars in Immunopathology*, 45(1):145–157, 2023.
doi: 10.1007/s00281-022-00974-0.
URL [https://doi.org/10.1007/s00281-022-00974-0](https://doi.org/10.1007/s00281-022-00974-0).
- Li et al. (2025a)
Minghao Li, Ying Zeng, Zhihao Cheng, Cong Ma, and Kai Jia.
Reportbench: Evaluating deep research agents via academic survey tasks, 2025a.
URL [https://arxiv.org/abs/2508.15804](https://arxiv.org/abs/2508.15804).
- Li et al. (2025b)
Yifei Li, Hanane Nour Moussa, Ziru Chen, Shijie Chen, Botao Yu, Mingyi Xue, Benjamin Burns, Tzu-Yao Chiu, Vishal Dey, Zitong Lu, Chen Wei, Qianheng Zhang, Tianyu Zhang, Song Gao, Xuhui Huang, Xia Ning, Nesreen K. Ahmed, Ali Payani, and Huan Sun.
AutoSDT: Scaling data-driven discovery tasks toward open co-scientists.
In *The 2025 Conference on Empirical Methods in Natural Language Processing*, 2025b.
URL [https://openreview.net/forum?id=tPhdEL0NYV](https://openreview.net/forum?id=tPhdEL0NYV).
- Lin et al. (2016)
Jia-Ren Lin, Mohammad Fallahi-Sichani, Jia-Yun Chen, and Peter K. Sorger.
Cyclic immunofluorescence (cycif), a highly multiplexed method for single-cell imaging.
*Current Protocols in Chemical Biology*, 8(4):251–264, December 2016.
doi: 10.1002/cpch.14.
URL [https://doi.org/10.1002/cpch.14](https://doi.org/10.1002/cpch.14).
- OpenAI (2025a)
OpenAI.
Introducing gpt-4.1 in the api, 2025a.
URL [https://openai.com/index/gpt-4-1/](https://openai.com/index/gpt-4-1/).
- OpenAI (2025b)
OpenAI.
Introducing deep research, 2025b.
URL [https://openai.com/index/introducing-deep-research/](https://openai.com/index/introducing-deep-research/).
- OpenAI (2025c)
OpenAI.
Gpt-4o search preview, 2025c.
URL [https://platform.openai.com/docs/models/gpt-4o-search-preview](https://platform.openai.com/docs/models/gpt-4o-search-preview).
- OpenAI et al. (2024)
OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, and Paul Baltescu et al.
Gpt-4 technical report, 2024.
URL [https://arxiv.org/abs/2303.08774](https://arxiv.org/abs/2303.08774).
- Pu et al. (2025)
Kevin Pu, K. J. Kevin Feng, Tovi Grossman, Tom Hope, Bhavana Dalvi Mishra, Matt Latzke, Jonathan Bragg, Joseph Chee Chang, and Pao Siangliulue.
Ideasynth: Iterative research idea development through evolving and composing idea facets with literature-grounded feedback.
In *CHI*, pp. 145:1–145:31, 2025.
URL [https://doi.org/10.1145/3706598.3714057](https://doi.org/10.1145/3706598.3714057).
- Radensky et al. (2025)
Marissa Radensky, Simra Shahid, Raymond Fok, Pao Siangliulue, Tom Hope, and Daniel S. Weld.
Scideator: Human-llm scientific idea generation grounded in research-paper facet recombination, 2025.
URL [https://arxiv.org/abs/2409.14634](https://arxiv.org/abs/2409.14634).
- Raudenbush & Bryk (2002)
Stephen W. Raudenbush and Anthony S. Bryk.
*Hierarchical linear models: applications and data analysis methods*.
Advanced quantitative techniques in the social sciences. Sage Publications, Thousand Oaks, 2nd ed edition, 2002.
ISBN 978-0-7619-1904-9.
- Rubaiat et al. (2025)
Sajratul Y. Rubaiat, Syed N. Sakib, and Hasan M. Jamil.
Mapping the evolution of research contributions using knovo, 2025.
URL [https://arxiv.org/abs/2506.17508](https://arxiv.org/abs/2506.17508).
- Shahid et al. (2025)
Simra Shahid, Marissa Radensky, Raymond Fok, Pao Siangliulue, Daniel S Weld, and Tom Hope.
Literature-grounded novelty assessment of scientific ideas.
In Tirthankar Ghosal, Philipp Mayr, Amanpreet Singh, Aakanksha Naik, Georg Rehm, Dayne Freitag, Dan Li, Sonja Schimmler, and Anita De Waard (eds.), *Proceedings of the Fifth Workshop on Scholarly Document Processing (SDP 2025)*, pp. 96–113, Vienna, Austria, July 2025. Association for Computational Linguistics.
ISBN 979-8-89176-265-7.
doi: 10.18653/v1/2025.sdp-1.9.
URL [https://aclanthology.org/2025.sdp-1.9/](https://aclanthology.org/2025.sdp-1.9/).
- Si et al. (2025a)
Chenglei Si, Tatsunori Hashimoto, and Diyi Yang.
The ideation-execution gap: Execution outcomes of llm-generated versus human research ideas, 2025a.
URL [https://arxiv.org/abs/2506.20803](https://arxiv.org/abs/2506.20803).
- Si et al. (2025b)
Chenglei Si, Diyi Yang, and Tatsunori Hashimoto.
Can LLMs generate novel research ideas? a large-scale human study with 100+ NLP researchers.
In *The Thirteenth International Conference on Learning Representations*, 2025b.
URL [https://openreview.net/forum?id=M23dTGWCZy](https://openreview.net/forum?id=M23dTGWCZy).
- Singh et al. (2025)
Amanpreet Singh, Joseph Chee Chang, Chloe Anastasiades, Dany Haddad, Aakanksha Naik, Amber Tanaka, Angele Zamarron, Cecile Nguyen, Jena D. Hwang, Jason Dunkleberger, Matt Latzke, Smita Rao, Jaron Lochner, Rob Evans, Rodney Kinney, Daniel S. Weld, Doug Downey, and Sergey Feldman.
Ai2 scholar qa: Organized literature synthesis with attribution, 2025.
URL [https://arxiv.org/abs/2504.10861](https://arxiv.org/abs/2504.10861).
- Skarlinski et al. (2024)
Michael D. Skarlinski, Sam Cox, Jon M. Laurent, James D. Braza, Michaela Hinks, Michael J. Hammerling, Manvitha Ponnapati, Samuel G. Rodriques, and Andrew D. White.
Language agents achieve superhuman synthesis of scientific knowledge, 2024.
URL [https://arxiv.org/abs/2409.13740](https://arxiv.org/abs/2409.13740).
- Wang et al. (2024)
Qingyun Wang, Doug Downey, Heng Ji, and Tom Hope.
SciMON: Scientific inspiration machines optimized for novelty.
In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), *Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pp. 279–299, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
doi: 10.18653/v1/2024.acl-long.18.
URL [https://aclanthology.org/2024.acl-long.18/](https://aclanthology.org/2024.acl-long.18/).
- Wen et al. (2025)
Jiaxin Wen, Chenglei Si, Yueh han Chen, He He, and Shi Feng.
Predicting empirical ai research outcomes with language models, 2025.
URL [https://arxiv.org/abs/2506.00794](https://arxiv.org/abs/2506.00794).
- Weng et al. (2025)
Yixuan Weng, Minjun Zhu, Guangsheng Bao, Hongbo Zhang, Jindong Wang, Yue Zhang, and Linyi Yang.
Cycleresearcher: Improving automated research via automated review.
In *The Thirteenth International Conference on Learning Representations*, 2025.
URL [https://openreview.net/forum?id=bjcsVLoHYs](https://openreview.net/forum?id=bjcsVLoHYs).
- Wu et al. (2023)
Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A. Smith, Mari Ostendorf, and Hannaneh Hajishirzi.
Fine-grained human feedback gives better rewards for language model training.
In *Thirty-seventh Conference on Neural Information Processing Systems*, 2023.
URL [https://openreview.net/forum?id=CSbGXyCswu](https://openreview.net/forum?id=CSbGXyCswu).

## Appendix

We include herein details omitted from the main text as follows:

- •
[ScholarEval details and prompts](https://arxiv.org/html/2510.16234v1#A1)
- •
[ScholarEval interface](https://arxiv.org/html/2510.16234v1#A2)
- •
[ScholarIdeas curation details](https://arxiv.org/html/2510.16234v1#A3)
- •
[Evaluation metrics details](https://arxiv.org/html/2510.16234v1#A4)
- •
[Experimental setup details](https://arxiv.org/html/2510.16234v1#A5)
- •
[Evaluation results](https://arxiv.org/html/2510.16234v1#A6)
- •
[User study details](https://arxiv.org/html/2510.16234v1#A7)
- •
[ScholarEval output examples](https://arxiv.org/html/2510.16234v1#A8)
- •
[LLM Usage](https://arxiv.org/html/2510.16234v1#A9)

## Appendix A ScholarEval details and prompts

In this section we elaborate on implementation details omitted from the main text due to space limitations and present our full system prompts.

### A.1 Soundness Details

#### A.1.1 Additional Implementation Details

Snippet Search. We employ the [snippet search endpoint](https://api.semanticscholar.org/api-docs/#tag/Snippet-Text) from Semantic Scholar to get paper snippets (from the title, abstract, or body) that are relevant to the method being evaluated. To retrieve all papers referenced in the snippet, we use the field refMentions from the returned snippet data, which allows us to match every referenced paper to its Semantic Scholar corpusID.

Paper Downloading. Once all referenced papers are identified, we use their corpusID to retrive their Semantic Scholar entry and use the field OpenAccessPDF to get the url to download the full text. Since there are cases where the paper might be behind a paywall, we use [Unpaywall](https://unpaywall.org/) with our institution emails to retrieve open access version of the papers, if available.

Methods and Results Extraction. To extract key sections from each paper, we first use GROBID to parse the paper PDF into its XML representation. Then, we search for the methods and results section by matching the section names to an exhaustive list of section titles that could potentially be used to reference the methods and results sections (e.g. Methods, Methodology, Protocol etc.).

TL;DR Summary. Since the method-level soundness review can be lengthy, we also include a summarization step where we prompt an LLM to generate a TL;DR summary highlighting the top three most important strengths, weaknesses, and suggestions to address. In our ScholarEval user interface ([Appendix B](https://arxiv.org/html/2510.16234v1#A2)), both this summary and the method-level evaluation are shown to the user, with the latter being expandable.

Citation Checking. As a post-processing step, we call a citation checking module that uses an LLM to perform the following functionalities: ensures that all citation worthy statements are followed by relevant citations and formats the bibliography section of the evaluation report.

#### A.1.2 Prompts

In this section we provide all the prompts used in the Soundness Module.

### A.2 Contribution Details

#### A.2.1 Additional Implementation Details

Paper Search. To retrieve relevant papers for contribution analysis, we use the [Semantic Scholar paper relevance search](https://api.semanticscholar.org/api-docs/#tag/Paper-Data/operation/get_graph_paper_relevance_search) which returns the top n most relevant papers based on a query. For subsequent processing in the contribution module (i.e. relevance assessment and pairwise comparison) we retrieve the paper abstract from the returned abstract field.

Paper Downsampling. Before the pairwise comparison step, we sample 25 papers from the final list. This downsampling serves two functions: first it reduces the latency and computational cost of the pairwise comparison step, and it reduces the overall context forwarded to the final synthesis step to avoid prohibitive context lengths.

Citation Checking. Similar to the Soundness module, we also apply a final post-check on the citations to ensure proper attribution and bibliography formatting.

#### A.2.2 Prompts

In this section we provide all the prompts used on the Contribution Module.

## Appendix B ScholarEval interface

To conduct the user study and for wider feedback from the community (upon acceptance), we create a ScholarEval user interface. As shown in [Figure 5](https://arxiv.org/html/2510.16234v1#A2.F5), our interface includes an input box for the user to enter their research idea or upload it from a file. Optionally, the user can specify a literature search cutoff date, This is especially useful if ScholarEval is used to evaluate an already published research idea. The user can then generate a Soundness review or switch tabs to generate a Contribution review.

Figure: Figure 5: ScholarEval user interface.
Refer to caption: x6.png

## Appendix C ScholarIdeas curation details

### C.1 Paper List

We provide the full list of 117 papers used to construct ScholarIdeas, at the end of the manuscript in LABEL:tab:ai-paper-list, LABEL:tab:neuro-paper-list, LABEL:tab:biochem-paper-list, and LABEL:tab:eco-paper-list. Each entry in the table links to the paper on OpenReview or eLife.

### C.2 LLM-based extraction

#### C.2.1 Details

We use Claude 4 Sonnet to extract the research idea and review rubrics. The extraction of the review rubrics is done in two stages: (1) we instruct the LLM to extract the verbatim excerpts from the review that give assessments about the research idea. (2) we instruct the LLM to remove redundancies and format the extracted text into standalone statements classified based on type, axis, and severity. The full prompts used in this process are in §[C.2.2](https://arxiv.org/html/2510.16234v1#A3.SS2.SSS2).

#### C.2.2 Prompts

### C.3 Expert Validation

#### C.3.1 Details

We invited six experts of Ph.D. students, postdoctoral researchers, and professors (1 from computer science, 2 from biochemistry, 2 from neuroscience, and 1 from ecology). We first conducted a training session to explain the validation task and we have provided the annotators with detailed instructions, shown in §[C.3.2](https://arxiv.org/html/2510.16234v1#A3.SS3.SSS2). We made an effort to match every annotator to research ideas in their specific area of specialty, and annotators were instructed to skip a research idea if they felt that it fell out of their area of expertise.

#### C.3.2 Validation Instructions

#### C.3.3 Validation Results

Table [5](https://arxiv.org/html/2510.16234v1#A3.T5) shows the distribution of edits for each of the research idea and review rubrics. Common issues that required edits include the mention of results in the research idea, errors in the classification of statements based on their severity, and the inclusion of rubrics mentioning the results obtained in the paper or the referring to the presentation (tables, figures, etc.).

**Table 5: Distribution of edits across research ideas and review rubrics.**
| Category | Percentage of Edits |
| --- | --- |
| Research Idea | 13.40% |
| Review Rubrics | 79.10% |

## Appendix D Evaluation metrics details

### D.1 Coverage Metric

To compute the coverage metric, we use Prometheus-Eval (Kim et al., 2024) framework with GPT-4 (OpenAI et al., 2024) backbone. We provide Prometheus with the detailed rubric given below to give a score on the degree of coverage of each rubric of ScholarIdeas. These scores are subsequently averaged to compute the final Coverage score.

### D.2 Reference Invalidity

We compute reference invalidity as the fraction of non-resolving citations by issuing an HTTP HEAD request for every paper link referenced in the output of ScholarEval and baselines, and inspecting the returned status code. In our initial implementation, we noticed that this approach undercounts failures: many publishers return 403 (bot blocking) or even 200 (soft-404 pages etc.) for URLs that do not actually resolve to the referenced papers upon manual inspection. We thus resort to reporting a lower bound for reference invalidity; we label a link as invalid only when it returns 404, 410, or any 5xx status. However the true of reference invalidity for baseline systems is higher. We provide failure cases observed in our manual audit in §[F.2](https://arxiv.org/html/2510.16234v1#A6.SS2). Reference invalidity is entirely eliminated in ScholarEval.

### D.3 LLM Metrics

We use Claude 4 Sonnet as our LLM judge and instruct it to choose the winner between a pair of reports (ScholarEval and o4-mini-deep-research) using the prompt below.

To validate the reliability of the LLM judgments, we conduct a small scale study with 33 report pairs and ask 6 Ph.D. students in artificial intelligence, neuroscience, and chemistry - who are unfamiliar with our system and would thus not be able to identify it in the pair - to choose the better report based on the three criteria. We provide our annotators with the same instructions given to the LLM judge. Each report pair was rated by 1, 2, or 3 annotators, and we set the final human label for each report pair based on majority vote. Reports that had equally split ratings were discarded, resulting in 18 remaining pairs. We compute the inter-annotator agreement between the human and LLM labels based on percent agreement and Cohen’s Kappa. The results shown in [Table 6](https://arxiv.org/html/2510.16234v1#A4.T6) underscore a relatively high percent agreement for all metrics in addition to a fair agreement based on Cohen’s kappa for both Evidence Support and Actionability.

Additionally, we provide the human preference results based on the full sample of 33 report pairs in Table [Table 7](https://arxiv.org/html/2510.16234v1#A4.T7). Despite the low agreement on the depth metric from Table [Table 6](https://arxiv.org/html/2510.16234v1#A4.T6), these preference results showcase that ScholarEval is substantially preferred over o4-mini-deep-research across all three metrics and corroborate the LLM-judge preference results.

**Table 6: Inter-annotator agreement (human vs. LLM) across metrics.**
| Metric | Percent Agreement (%) | Cohen’s $\kappa$ |
| --- | --- | --- |
| Evidence support | 66.67 | 0.1818 |
| Actionability | 72.22 | 0.1667 |
| Depth | 61.11 | -0.0161 |

**Table 7: Human preference results on sample of 34 report pairs.**
| Metric | ScholarEval | o4-mini-deep-research | Tie |
| --- | --- | --- | --- |
| Evidence Support | 70.6 | 14.7 | 14.7 |
| Depth | 79.4 | 11.8 | 8.8 |
| Actionability | 82.4 | 11.8 | 5.9 |

## Appendix E Experimental setup details

### E.1 Baselines

In this section we provide the detailed prompts used to generate the Soundness and Contribution evaluation for all baseline systems. To ensure a fair comparison, we use optimized prompts that instruct all baseline systems to generate detailed results in the same format as ScholarEval.

Similar to Li et al. (2025a), to avoid cases where the retrieval-augmented models (i.e. GPT-4o-search-preview and o4-mini-deep-research) retrieve the paper corresponding to the research idea being evaluated, we include an instruction to limit the retrieved literature to publications released before the cutoff date (i.e. the publication date of the paper that the research idea was extracted from). Although not error proof, in our experimentation we have observed that this greatly reduces the instances where the target paper is retrieved.

### E.2 ScholarEval

We use the following parameter values to ensure a balance between the system performance, latency, and context window limit concerns:

- •
Snippet search queries generated per extracted method: 1.
- •
Snippets returned per query: up to 20.
- •
Paper search queries generated per contribution statement: 3.
- •
Papers returned per query: up to 20.
- •
Papers returned by the recommendations for each seed paper: up to 8.
- •
Papers returned via references augmentation for each seed paper: up to 10.
- •
Papers sampled from the final list for pairwise comparison: up to 25.

Additionally, we implement functionality that restricts the retrieved snippets and papers to those published prior to the cutoff date (i.e. the publication date of the paper that the research idea was extracted from).

### E.3 Ablations

In our ablation experiments, we use the same setup and parameters described in [subsection E.2](https://arxiv.org/html/2510.16234v1#A5.SS2).

To ablate the methods and results extraction (MRE), we remove the steps where we extract references from the snippets, download the papers, and extract the methods and results from each. Instead, the snippets are directly summarized and forwarded for final soundness review synthesis.

The ablation of the paper augmentation (PA) is conducted by removing both the recommendation based augmentation and citation based augmentation. As such, only the seed list of papers identified after the initial paper search and relevance assessment are forwarded for pairwise comparison.

Finally, to ablate the pairwise comparison (PC) step, we sample 25 papers from the final augmented paper list and forward them directly to the contribution review synthesis step.

## Appendix F Evaluation results

### F.1 Additional Results and Significance Analysis

In this section, we include detailed evaluation results supplementing the results we presented in the main paper. Namely, [Table 8](https://arxiv.org/html/2510.16234v1#A6.T8), [9](https://arxiv.org/html/2510.16234v1#A6.T9), [10](https://arxiv.org/html/2510.16234v1#A6.T10), [11](https://arxiv.org/html/2510.16234v1#A6.T11), and [12](https://arxiv.org/html/2510.16234v1#A6.T12) show the pairwise significance analysis of the coverage results between ScholarEval variants and every baseline overall and for each discipline. Additionally, [Table 13](https://arxiv.org/html/2510.16234v1#A6.T13) shows the coverage results across all systems by type, axis, severity. [Table 14](https://arxiv.org/html/2510.16234v1#A6.T14) outlines the latency and cost of each of ScholarEval and baseline systems.

**Table 8: Pairwise comparisons for Overall coverage: Welch’s t-tests comparing ScholarEval variants to baselines (n=1076 per model). Two-sided p-values shown; Sig.? indicates ScholarEval $>$ baseline with $p<0.05$.**
| ScholarEval | Baseline | MeanSE (SD) | MeanBL (SD) | $p$ | Sig.? |
| --- | --- | --- | --- | --- | --- |
| ScholarEval ${}_{\text{Llama-70B}}$ | Llama-3.3-70B | 2.04 (1.16) | 1.83 (0.89) | 2.6e-06 | Yes |
| ScholarEval ${}_{\text{Llama-70B}}$ | GPT-4.1 | 2.04 (1.16) | 2.18 (1.08) | 0.004 | No |
| ScholarEval ${}_{\text{Llama-70B}}$ | Claude-4-Sonnet | 2.04 (1.16) | 2.18 (1.04) | 0.003 | No |
| ScholarEval ${}_{\text{Llama-70B}}$ | o4-mini-deep-research | 2.04 (1.16) | 2.28 (1.07) | 6.6e-07 | No |
| ScholarEval ${}_{\text{GPT-4.1}}$ | Llama-3.3-70B | 2.72 (1.47) | 1.83 (0.89) | 4.4e-60 | Yes |
| ScholarEval ${}_{\text{GPT-4.1}}$ | GPT-4.1 | 2.72 (1.47) | 2.18 (1.08) | 8.3e-22 | Yes |
| ScholarEval ${}_{\text{GPT-4.1}}$ | Claude-4-Sonnet | 2.72 (1.47) | 2.18 (1.04) | 2.6e-22 | Yes |
| ScholarEval ${}_{\text{GPT-4.1}}$ | o4-mini-deep-research | 2.72 (1.47) | 2.28 (1.07) | 3.4e-15 | Yes |
| ScholarEval ${}_{\text{Claude-4-Sonnet}}$ | Llama-3.3-70B | 2.77 (1.40) | 1.83 (0.89) | 9.5e-71 | Yes |
| ScholarEval ${}_{\text{Claude-4-Sonnet}}$ | GPT-4.1 | 2.77 (1.40) | 2.18 (1.08) | 4.0e-27 | Yes |
| ScholarEval ${}_{\text{Claude-4-Sonnet}}$ | Claude-4-Sonnet | 2.77 (1.40) | 2.18 (1.04) | 8.4e-28 | Yes |
| ScholarEval ${}_{\text{Claude-4-Sonnet}}$ | o4-mini-deep-research | 2.77 (1.40) | 2.28 (1.07) | 1.7e-19 | Yes |

**Table 9: Pairwise comparisons for AI coverage: Welch’s t-tests comparing ScholarEval variants to baselines (n=425 per model). Two-sided p-values shown; Sig.? indicates ScholarEval $>$ baseline with $p<0.05$.**
| ScholarEval | Baseline | MeanSE (SD) | MeanBL (SD) | $p$ | Sig.? |
| --- | --- | --- | --- | --- | --- |
| ScholarEval ${}_{\text{Llama-70B}}$ | Llama-3.3-70B | 2.06 (1.14) | 1.88 (0.92) | 0.011 | Yes |
| ScholarEval ${}_{\text{Llama-70B}}$ | GPT-4.1 | 2.06 (1.14) | 2.21 (1.12) | 0.053 | No |
| ScholarEval ${}_{\text{Llama-70B}}$ | Claude-4-Sonnet | 2.06 (1.14) | 2.28 (1.02) | 0.003 | No |
| ScholarEval ${}_{\text{Llama-70B}}$ | o4-mini-deep-research | 2.06 (1.14) | 2.35 (1.07) | 1.4e-04 | No |
| ScholarEval ${}_{\text{GPT-4.1}}$ | Llama-3.3-70B | 2.84 (1.39) | 1.88 (0.92) | 7.2e-30 | Yes |
| ScholarEval ${}_{\text{GPT-4.1}}$ | GPT-4.1 | 2.84 (1.39) | 2.21 (1.12) | 8.1e-13 | Yes |
| ScholarEval ${}_{\text{GPT-4.1}}$ | Claude-4-Sonnet | 2.84 (1.39) | 2.28 (1.02) | 4.1e-11 | Yes |
| ScholarEval ${}_{\text{GPT-4.1}}$ | o4-mini-deep-research | 2.84 (1.39) | 2.35 (1.07) | 1.2e-08 | Yes |
| ScholarEval ${}_{\text{Claude-4-Sonnet}}$ | Llama-3.3-70B | 2.91 (1.34) | 1.88 (0.92) | 2.7e-35 | Yes |
| ScholarEval ${}_{\text{Claude-4-Sonnet}}$ | GPT-4.1 | 2.91 (1.34) | 2.21 (1.12) | 5.7e-16 | Yes |
| ScholarEval ${}_{\text{Claude-4-Sonnet}}$ | Claude-4-Sonnet | 2.91 (1.34) | 2.28 (1.02) | 3.7e-14 | Yes |
| ScholarEval ${}_{\text{Claude-4-Sonnet}}$ | o4-mini-deep-research | 2.91 (1.34) | 2.35 (1.07) | 3.2e-11 | Yes |

**Table 10: Pairwise comparisons for Neuroscience coverage: Welch’s t-tests comparing ScholarEval variants to baselines (n=314 per model). Two-sided p-values shown; Sig.? indicates ScholarEval $>$ baseline with $p<0.05$.**
| ScholarEval | Baseline | MeanSE (SD) | MeanBL (SD) | $p$ | Sig.? |
| --- | --- | --- | --- | --- | --- |
| ScholarEval ${}_{\text{Llama-70B}}$ | Llama-3.3-70B | 2.05 (1.18) | 1.82 (0.86) | 0.004 | Yes |
| ScholarEval ${}_{\text{Llama-70B}}$ | GPT-4.1 | 2.05 (1.18) | 2.18 (1.10) | 0.107 | No |
| ScholarEval ${}_{\text{Llama-70B}}$ | Claude-4-Sonnet | 2.05 (1.18) | 2.15 (1.08) | 0.209 | No |
| ScholarEval ${}_{\text{Llama-70B}}$ | o4-mini-deep-research | 2.05 (1.18) | 2.25 (1.04) | 0.011 | No |
| ScholarEval ${}_{\text{GPT-4.1}}$ | Llama-3.3-70B | 2.74 (1.55) | 1.82 (0.86) | 2.0e-24 | Yes |
| ScholarEval ${}_{\text{GPT-4.1}}$ | GPT-4.1 | 2.74 (1.55) | 2.18 (1.10) | 5.7e-11 | Yes |
| ScholarEval ${}_{\text{GPT-4.1}}$ | Claude-4-Sonnet | 2.74 (1.55) | 2.15 (1.08) | 8.2e-12 | Yes |
| ScholarEval ${}_{\text{GPT-4.1}}$ | o4-mini-deep-research | 2.74 (1.55) | 2.25 (1.04) | 1.1e-08 | Yes |
| ScholarEval ${}_{\text{Claude-4-Sonnet}}$ | Llama-3.3-70B | 2.55 (1.45) | 1.82 (0.86) | 1.4e-17 | Yes |
| ScholarEval ${}_{\text{Claude-4-Sonnet}}$ | GPT-4.1 | 2.55 (1.45) | 2.18 (1.10) | 4.3e-06 | Yes |
| ScholarEval ${}_{\text{Claude-4-Sonnet}}$ | Claude-4-Sonnet | 2.55 (1.45) | 2.15 (1.08) | 6.0e-07 | Yes |
| ScholarEval ${}_{\text{Claude-4-Sonnet}}$ | o4-mini-deep-research | 2.55 (1.45) | 2.25 (1.04) | 1.1e-04 | Yes |

**Table 11: Pairwise comparisons for Biochemistry coverage: Welch’s t-tests comparing ScholarEval variants to baselines (n=147 per model). Two-sided p-values shown; Sig.? indicates ScholarEval $>$ baseline with $p<0.05$.**
| ScholarEval | Baseline | MeanSE (SD) | MeanBL (SD) | $p$ | Sig.? |
| --- | --- | --- | --- | --- | --- |
| ScholarEval ${}_{\text{Llama-70B}}$ | Llama-3.3-70B | 2.04 (1.19) | 1.81 (0.90) | 0.094 | No |
| ScholarEval ${}_{\text{Llama-70B}}$ | GPT-4.1 | 2.04 (1.19) | 2.14 (1.04) | 0.460 | No |
| ScholarEval ${}_{\text{Llama-70B}}$ | Claude-4-Sonnet | 2.04 (1.19) | 2.01 (1.01) | 0.827 | No |
| ScholarEval ${}_{\text{Llama-70B}}$ | o4-mini-deep-research | 2.04 (1.19) | 2.08 (1.06) | 0.772 | No |
| ScholarEval ${}_{\text{GPT-4.1}}$ | Llama-3.3-70B | 2.61 (1.42) | 1.81 (0.90) | 1.3e-06 | Yes |
| ScholarEval ${}_{\text{GPT-4.1}}$ | GPT-4.1 | 2.61 (1.42) | 2.14 (1.04) | 0.003 | Yes |
| ScholarEval ${}_{\text{GPT-4.1}}$ | Claude-4-Sonnet | 2.61 (1.42) | 2.01 (1.01) | 1.2e-04 | Yes |
| ScholarEval ${}_{\text{GPT-4.1}}$ | o4-mini-deep-research | 2.61 (1.42) | 2.08 (1.06) | 5.5e-04 | Yes |
| ScholarEval ${}_{\text{Claude-4-Sonnet}}$ | Llama-3.3-70B | 2.64 (1.40) | 1.81 (0.90) | 3.7e-07 | Yes |
| ScholarEval ${}_{\text{Claude-4-Sonnet}}$ | GPT-4.1 | 2.64 (1.40) | 2.14 (1.04) | 0.001 | Yes |
| ScholarEval ${}_{\text{Claude-4-Sonnet}}$ | Claude-4-Sonnet | 2.64 (1.40) | 2.01 (1.01) | 5.2e-05 | Yes |
| ScholarEval ${}_{\text{Claude-4-Sonnet}}$ | o4-mini-deep-research | 2.64 (1.40) | 2.08 (1.06) | 2.4e-04 | Yes |

**Table 12: Pairwise comparisons for Ecology coverage: Welch’s t-tests comparing ScholarEval variants to baselines (n=190 per model). Two-sided p-values shown; Sig.? indicates ScholarEval $>$ baseline with $p<0.05$.**
| ScholarEval | Baseline | MeanSE (SD) | MeanBL (SD) | $p$ | Sig.? |
| --- | --- | --- | --- | --- | --- |
| ScholarEval ${}_{\text{Llama-70B}}$ | Llama-3.3-70B | 1.94 (1.13) | 1.74 (0.87) | 0.054 | No |
| ScholarEval ${}_{\text{Llama-70B}}$ | GPT-4.1 | 1.94 (1.13) | 2.13 (1.00) | 0.083 | No |
| ScholarEval ${}_{\text{Llama-70B}}$ | Claude-4-Sonnet | 1.94 (1.13) | 2.11 (1.03) | 0.126 | No |
| ScholarEval ${}_{\text{Llama-70B}}$ | o4-mini-deep-research | 1.94 (1.13) | 2.35 (1.11) | 3.8e-04 | No |
| ScholarEval ${}_{\text{GPT-4.1}}$ | Llama-3.3-70B | 2.52 (1.51) | 1.74 (0.87) | 2.2e-09 | Yes |
| ScholarEval ${}_{\text{GPT-4.1}}$ | GPT-4.1 | 2.52 (1.51) | 2.13 (1.00) | 0.003 | Yes |
| ScholarEval ${}_{\text{GPT-4.1}}$ | Claude-4-Sonnet | 2.52 (1.51) | 2.11 (1.03) | 0.002 | Yes |
| ScholarEval ${}_{\text{GPT-4.1}}$ | o4-mini-deep-research | 2.52 (1.51) | 2.35 (1.11) | 0.212 | No |
| ScholarEval ${}_{\text{Claude-4-Sonnet}}$ | Llama-3.3-70B | 2.90 (1.39) | 1.74 (0.87) | 8.1e-20 | Yes |
| ScholarEval ${}_{\text{Claude-4-Sonnet}}$ | GPT-4.1 | 2.90 (1.39) | 2.13 (1.00) | 1.6e-09 | Yes |
| ScholarEval ${}_{\text{Claude-4-Sonnet}}$ | Claude-4-Sonnet | 2.90 (1.39) | 2.11 (1.03) | 9.3e-10 | Yes |
| ScholarEval ${}_{\text{Claude-4-Sonnet}}$ | o4-mini-deep-research | 2.90 (1.39) | 2.35 (1.11) | 2.6e-05 | Yes |

**Table 13: Coverage results of different variants of ScholarEval and baselines across type, axis, and severity dimensions (mean scores with standard deviations).**
| System | Type | Axis | Severity |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | Strength | Weakness | Soundness | Contribution | Major | Minor |
| Llama-3.3-70B | 2.63 (0.82) | 1.64 (0.80) | 1.72 (0.84) | 2.14 (0.95) | 1.90 (0.88) | 1.72 (0.90) |
| GPT-4.1 | 3.21 (1.05) | 1.94 (0.95) | 2.00 (0.99) | 2.71 (1.16) | 2.28 (1.05) | 2.01 (1.12) |
| Claude-4-Sonnet | 3.04 (0.97) | 1.98 (0.95) | 2.06 (1.01) | 2.51 (1.08) | 2.28 (1.03) | 2.01 (1.04) |
| GPT-4o-search-preview | 2.83 (0.98) | 1.69 (0.85) | 1.76 (0.91) | 2.32 (1.05) | 1.99 (0.98) | 1.77 (0.98) |
| o4-mini-deep-research | 2.98 (0.90) | 2.12 (1.04) | 2.17 (1.05) | 2.60 (1.05) | 2.41 (1.05) | 2.08 (1.06) |
| ScholarEval ${}_{\text{Llama}}$ | 2.93 (1.19) | 1.83 (1.05) | 1.89 (1.10) | 2.46 (1.21) | 2.16 (1.14) | 1.83 (1.15) |
| ScholarEval ${}_{\text{GPT}}$ | 4.16 (1.07) | 2.40 (1.34) | 2.51 (1.43) | 3.36 (1.40) | 2.92 (1.44) | 2.41 (1.45) |
| ScholarEval ${}_{\text{Claude}}$ | 3.65 (1.21) | 2.56 (1.36) | 2.61 (1.37) | 3.24 (1.37) | 2.93 (1.38) | 2.51 (1.39) |

**Table 14: Latency and cost per run (soundness and contribution) for ScholarEval and baselines.**
| System | Latency (mins) | Cost (USD) |
| --- | --- | --- |
| Llama-3.3-70B | 1.23 | 0.00 |
| GPT-4.1 | 2.90 | 0.03 |
| Claude-4-Sonnet | 7.06 | 0.08 |
| GPT-4o-search-preview | 1.94 | 0.15 |
| o4-mini-deep-research | 3.33 | 0.49 |
| ScholarEval-Llama | 6.63 | 0.00 |
| ScholarEval-GPT | 10.23 | 2.54 |
| ScholarEval-Claude | 12.10 | 3.38 |

### F.2 Manual Audit

Our manual audit of papers referenced in the evaluation reports generated by baseline systems underscores various failure modes that undermine the reliability of these systems for literature-grounded research idea evaluation.

First, our inspection of the links indicates a much high reference invalidity rate than the conservative lowerbound reported in §[4.2](https://arxiv.org/html/2510.16234v1#S4.SS2). For Llama-3.3-70B, some our inspected reports had up to 90% invalid references, and stronger LLMs such as Claude 4 Sonnet had up to 50% reference invalidity. We also observed that this issue is not eliminated by retrieval, as 22% of the links included in an inspected report generated by GPT-4o-search-preview were invalid.

We have also noticed subtler failure modes. In the example below, Claude 4 Sonnet references a paper by Gong et al. However, upon inspection, we notice that the linked paper is in fact authored by Bakalarski et al. (2016).

> While [(Gong et al., 2016-08)]([https://www.nature.com/articles/nbt.3621](https://www.nature.com/articles/nbt.3621)) demonstrated DNA-barcoded antibodies for multiplexed imaging, their approach relied on conventional chemical conjugation methods that can compromise antibody function.

These errors in attribution are also made by GPT-4o-search-preview. In the example below, it wrongly attributes a publication by Brotherton & Balskus (2013) to Kazane et al.

> These techniques have been widely used to evaluate the impact of conjugation on antibody performance, ensuring that the modifications do not adversely affect antigen-binding capabilities [(Kazane et al., 2013)]([https://pubs.acs.org/doi/10.1021/ja312154m](https://pubs.acs.org/doi/10.1021/ja312154m)).

We also observed that o4-mini-deep-research commits similar attribution errors. In the example below, it attributes CyCIF (Lin et al., 2016) and CODEX (Kuswanto et al., 2023) to the wrong authors.

> The proposed system could show improved signal (via HCR) and straightforward reagent generation (via MaMBA). The plan’s demonstration of 12-plex imaging is on par with existing methods like CyCIF (Gerdes et al. 2013) or CODEX (Goltsev et al. 2018), indicating strong competitive impact.

These observations showcase the limitations of even strong baselines (web-connected LMs and deep research systems) in generating reliable literature-backed research idea evaluations.

## Appendix G Additional Details from the Expert User Study

### G.1 Setup and Materials

Rubric.
We include our full rubric for the user study in [Table 15](https://arxiv.org/html/2510.16234v1#A7.T15). Answers were collected via Google Forms.

**Table 15: Questions asked during the expert user study, and their respective dimension, module, and scale.**
| Dimension | Module | Scale | Question |
| --- | --- | --- | --- |
| IdeaFaithful | Soundness | 1-10 | The number of extracted methods/experiments that are faithful to my original intention |
| IdeaFaithful | Contribution | 1-10 | The number of dimensions that are faithful to my original intention |
| Focus | Soundness/Contribution | 1-10 | The strengths reinforce the most valuable aspects of the proposal |
| Focus | Soundness/Contribution | 1-10 | The weaknesses highlight the most deficient aspects of the proposal |
| Focus | Soundness/Contribution | 1-10 | The suggestions for improvement are the most important aspects to address |
| Focus | Contribution | 1-10 | The criteria used to compare your proposal to related work are based on relevant metrics and/or frameworks and are NOT arbitrary |
| Refine | Soundness/Contribution | 1-10 | The top suggestions for improvement offer valuable, targeted, feasible modifications to your experiment |
| LitEngage | Soundness/Contribution | 1-10 | The evaluation uses detailed comparisons with specific components of relevant literature, rather than providing superficial citations |
| Citations | Soundness/Contribution | YYYY | Please estimate how many citations from your MAIN FIELD of study were new to you and you would use in your work |
| Citations | Soundness/Contribution | YYYY | Please estimate how many citations from OUTSIDE your main field of study were new to you and you would use in your work |
| Useful | Soundness/Contribution | 1-10 | I found the [Module] evaluation useful |
| Useful | Soundness/Contribution | 1-10 | I would use the [Module] evaluation again in the future |

Recruitment, Demographics, and Compensation. Participants were recruited via X (Twitter) and graduate department emailing lists internationally. [Table 16](https://arxiv.org/html/2510.16234v1#A7.T16) shows the number of evaluations completed per discipline, as well as years of experience. Experts were paid $\mathdollar 25$ per research idea evaluated, with a bonus $\mathdollar 10$ awarded if they completed written feedback.

**Table 16: User study demographics grouped by domain. This includes number of experts, evaluations, and years of experience per discipline.**
| Domain | Experts | Evaluations | 2-4 YoE | 4+ YoE |
| --- | --- | --- | --- | --- |
| Computer Science | 8 | 21 | 5 | 16 |
| Neuroscience | 4 | 12 | 6 | 6 |
| Chemistry | 4 | 8 | 1 | 7 |
| Ecology | 2 | 5 | 4 | 1 |
| Total | 18 | 46 | 16 | 30 |

User Interface. Experts were shown the interface shown in [Figure 6](https://arxiv.org/html/2510.16234v1#A8.F6) and [Figure 7](https://arxiv.org/html/2510.16234v1#A8.F7). We give experts a unique research id along with 4 unique idea keys, one for each research idea. Each is linked to a replicable, semi-random, and pre-calculated order of systems. The semi-random component comes from the fact that we forced the random order to include at least 2 of each system to ensure each person had the opportunity to contribute evaluations on either system. The script to regenerate these keys will be released with our code. We ensure validity of system blindness by asking users to guess which system they are evaluating, which has a $-0.12$ pearson correlation ($p=0.44$) with the actual assigned system.

### G.2 Statistical Methods

Because we collect multiple research ideas per person, our data is no longer independent, and we cannot use a mean-differences t-test. Instead, we use a Linear Mixed-Effects Model (Raudenbush & Bryk, 2002), which models both fixed effects (ScholarEval vs deep research) and random effects (ideas within and across participants). This helps to stabilize the ratings across our experts, as some may be consistently higher or lower raters. This additionally helps to account for some variability from resarch idea quality. The Linear Mixed-Effects Model is defined as follows:

$$ $\mathbf{y}=\mathbf{X}\bm{\beta}+\mathbf{Z}\mathbf{b}+\bm{\varepsilon}$ (1) $$

This adds an additional term, $\mathbf{Z}$, on top of the standard Linear Model to capture the variance of random effects, where $\mathbf{Z}$ is a matrix of shape $n{\times}m$, where $n$ is the total observations and $m$ is the number of unique participants. Each row is one-hot encoded for the participant who contributed the data point.

## Appendix H ScholarEval output examples

In this section we provide an example of a neuroscience research idea from ScholarIdeas along with an excerpt of a method soundness review and contribution dimension generated by ScholarEval.

Figure: Figure 6: User interface for the Expert User Study. The landing page includes inputs for a research id, proposal id, email for unpaywall, research proposal text box or file upload, and a literature cutoff date.
Refer to caption: x7.png

Figure: Figure 7: User interface for the Expert User Study, with an example from a user who gave permission to share their output. Reviews for both soundness and contribution are displayed in markdown format. There is a collapsible text box to show the method-level evaluations.
Refer to caption: x8.png

## Appendix I LLM Usage

The authors would like to declare the usage of LLMs for some aspects of code generation, LaTeX formatting and minor cosmetic improvements to the manuscript writing. However, LLMs were not used in the ideation of the project, designing the ScholarEval framework, or major content writing.

| [Marwa Abdulhai et al., “Defining Deception in Decision Making”](https://openreview.net/forum?id=YaRzuMaubS) |
| --- |
| [Philipp Guevorguian et al., “Exploring the Recall of Language Models: Case Study on Molecules”](https://openreview.net/forum?id=DlZ97cVwr0) |
| [Seungwon Oh et al., “Recovering Plasticity of Neural Networks via Soft Weight Rescaling”](https://openreview.net/forum?id=DnBjhWLVU1) |
| [Yuwei Yan et al., “OpenCity: A Scalable Platform to Simulate Urban Activities with Massive LLM Agents”](https://openreview.net/forum?id=qK6U4Ahfms) |
| [Moritz Glaser et al., “ESMGain: Effective and Efficient Prediction of Mutation’s functional Effect via ESM2 Transfer Learning and robust Benchmarks”](https://openreview.net/forum?id=vVlNBaiLdN) |
| [Svetlana Pavlova et al., “Flow Matching for One-Step Sampling”](https://openreview.net/forum?id=WxLwXyBJLw) |
| [Sasan Tavakkol et al., “Less is More: Adaptive Coverage for Synthetic Training Data”](https://openreview.net/forum?id=NpsgBKlApa) |
| [Lorenzo Pacchiardi et al., “100 instances is all you need: predicting LLM success by testing on a few instances”](https://openreview.net/forum?id=UoWslU6hsX) |
| [Zhihan Zhou et al., “GenomeOcean: Efficient Foundation Model for Genome Generation”](https://openreview.net/forum?id=c8sEgxG2c0) |
| [Eduardo Sánchez et al., “Linguini: A benchmark for language-agnostic linguistic reasoning”](https://openreview.net/forum?id=QiyQJqpcYe) |
| [Zekun Li et al., “MMSci: A Dataset for Graduate-Level Multi-Discipline Multimodal Scientific Understanding”](https://openreview.net/forum?id=DEOV74Idsg) |
| [Alexander Shypula et al., “Does Instruction Tuning Reduce Diversity? A Case Study Using Code Generation”](https://openreview.net/forum?id=hMEHnLJyrU) |
| [Naman Gupta et al., “MAC-CAFE: Multi-actor, Centralized Critic Architecture for Feedback-driven Editing”](https://openreview.net/forum?id=Ql7msQBqoF) |
| [Magnus Müller et al., “Large-Scale Multi-Agent Reinforcement Learning for Traffic Signal Optimization”](https://openreview.net/forum?id=hWF0HH8Rr9) |
| [Aohan Sun et al., “Mitigating Privacy Risk of Adversarial Examples with Counterfactual Explanations”](https://openreview.net/forum?id=gaa7gWPZBz) |
| [Gwok-Waa Wan et al., “GenBen: A Genarative Benchmark for LLM-Aided Design”](https://openreview.net/forum?id=gtVo4xcpFI) |
| [Kyeongrok Park et al., “Toward Human-Interpretable Explanations in a Unified Framework for GNNs”](https://openreview.net/forum?id=N0MnPLK6r7) |
| [Wenjie Tang et al., “StarCraft II Arena: Evaluating LLMs in Strategic Planning, Real-Time Decision Making, and Adaptability”](https://openreview.net/forum?id=o3V7OuPxu4) |
| [Chen Gao et al., “EmbodiedCity: A Benchmark Platform for Embodied Agent in Real-world City Environment”](https://openreview.net/forum?id=y15LAM4u0A) |
| [Sungmin Han et al., “Improving Transformer Interpretability with Activation Contrast-Based Attribution”](https://openreview.net/forum?id=irCuIdCdAl) |
| [Hong Xie et al., “Multiple-play Stochastic Bandits with Prioritized Resource Sharing”](https://openreview.net/forum?id=9e5syenoVE) |
| [Santiago Yeomans et al., “From Abstract Noise to Architectural Form: Designing Diffusion Models for Efficient Floor Plan Generation”](https://openreview.net/forum?id=skJLOae8ew) |
| [Chinmay Mittal et al., “FCoReBench: Can Large Language Models Solve Challenging First-Order Combinatorial Reasoning Problems?”](https://openreview.net/forum?id=CFKZKjrQ5r) |
| [Haihong Yang et al., “scKGOT: Intercellular Signaling Inference with Knowledge Graph Optimal Transport for Single-cell Transcriptomics”](https://openreview.net/forum?id=Y9yQ9qmVrc) |
| [Changliang Zhou et al., “ICAM: Rethinking Instance-Conditioned Adaptation in Neural Vehicle Routing Solver”](https://openreview.net/forum?id=gyTkfVYL45) |
| [Yisheng Xiao et al., “Path Selection Makes BERT-family Good Generators”](https://openreview.net/forum?id=7jDv1RrNQX) |
| [Xiang Liu et al., “ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference”](https://openreview.net/forum?id=8sglLco8Ti) |
| [Teng Yan et al., “Vision-Based Pseudo-Tactile Information Extraction and Localization for Dexterous Grasping”](https://openreview.net/forum?id=xcHIiZr3DT) |
| [Pavel Strashnov et al., “Towards Robust Evaluation of Protein Generative Models: A Systematic Analysis of Metrics”](https://openreview.net/forum?id=1S8ndwxMts) |
| [Zhenlei Wang et al., “Robust Heterogeneous Treatment Effect Estimation under Covariate Perturbation”](https://openreview.net/forum?id=glgvpS1dD1) |
| [Yao Shiyi et al., “BID: Broad Incremental for Android Malware Detection”](https://openreview.net/forum?id=ctzGqxE3O0) |
| [Leo McKee-Reid et al., “Honesty to Subterfuge: In-Context Reinforcement Learning Can Make Honest Models Reward Hack”](https://openreview.net/forum?id=to4PdiiILF) |
| [Tingzhou Wei et al., “Multivariate Time-series Forecasting with SPACE: Series Prediction Augmented by Causality Estimation”](https://openreview.net/forum?id=v5BouOktUP) |
| [Anuradha Kumari et al., “ZEPHYR GAN: REDEFINING GAN WITH FLEXIBLE GRADIENT CONTROL”](https://openreview.net/forum?id=f6GMwpxXHG) |
| [Andrei Chertkov et al., “Tensor Train Decomposition for Adversarial Attacks on Computer Vision Models”](https://openreview.net/forum?id=WVzYMa68Of) |
| [Zhenghan Chen et al., “Advancing Drug-Target Interaction Prediction via Graph Transformers and Residual Protein Embeddings”](https://openreview.net/forum?id=S2WHlhvFGg) |
| [Zhenghan Chen et al., “Non-Commutative Spectral Geometry for Adaptive Quantum-Classical Drug-Target Interaction Prediction”](https://openreview.net/forum?id=kvCKoKfqTd) |
| [Chris Cameron et al., “Foundation Models for Boolean Logic”](https://openreview.net/forum?id=qeY25DwmKO) |
| [Haoxuan Li et al., “Principle Counterfactual Fairness”](https://openreview.net/forum?id=TLgDQ0Rr2Z) |
| [Yuntian Wu et al., “Invariant Spatiotemporal Representation Learning for Cross-patient Seizure Classification”](https://openreview.net/forum?id=TkbjqexD8w) |

| [Benas et al., “Modeled grid cells aligned by a flexible attractor”](https://elifesciences.org/reviewed-preprints/89851v1#tab-content) |
| --- |
| [Wittkamp et al., “The neural dynamics of positive and negative expectations of pain”](https://elifesciences.org/reviewed-preprints/97793v1#tab-content) |
| [Cui et al., “Dysfunctional S1P/S1PR1 signaling in the dentate gyrus drives vulnerability of chronic pain-related memory impairment”](https://elifesciences.org/reviewed-preprints/99862v1#tab-content) |
| [O’Leary et al., “Natural forgetting reversibly modulates engram expression in hippocampal feedforward circuits”](https://elifesciences.org/reviewed-preprints/92860v1#tab-content) |
| [Klaassen et al., “Basolateral amygdala inhibition impairs updating of appetitive and aversive values by interacting with the prefrontal cortex”](https://elifesciences.org/reviewed-preprints/90930v1#tab-content) |
| [Haupt et al., “The transformation of sensory to perceptual braille letter representations in the visually deprived brain”](https://elifesciences.org/reviewed-preprints/98148v1#tab-content) |
| [Liu et al., “Cell class-specific long-range axonal projections of neurons in mouse whisker-related somatosensory cortices”](https://elifesciences.org/reviewed-preprints/97602v1#tab-content) |
| [Campbell et al., “Human single-neuron activity is modulated by intracranial theta burst stimulation of the basolateral amygdala”](https://elifesciences.org/reviewed-preprints/106481v1#tab-content) |
| [Derkaloustian et al., “Fine Touch Perception Relies on Frictional Instabilities”](https://elifesciences.org/reviewed-preprints/104543v1#tab-content) |
| [Lee et al., “The influence of temporal context on vision over multiple time scales”](https://elifesciences.org/reviewed-preprints/106614#tab-content) |
| [Setogawa et al., “Acquisition of auditory discrimination mediated by different processes through two distinct circuits linked to the lateral striatum”](https://elifesciences.org/reviewed-preprints/97326v1#tab-content) |
| [Cooper et al., “Ultraslow serotonin oscillations in the hippocampus delineate substates across NREM and waking”](https://elifesciences.org/reviewed-preprints/101105v1#tab-content) |
| [Mollá–Albaladejo et al., “Molecular characterization of gustatory second-order neurons reveals integrative mechanisms of gustatory and metabolic information”](https://elifesciences.org/reviewed-preprints/100947v1#tab-content) |
| [Bloem et al., “Dynamic estimation of the attentional field from visual cortical activity”](https://elifesciences.org/reviewed-preprints/104222v1#tab-content) |
| [Xu et al., “Neural Representation of Time across Complementary Reference Frames”](https://elifesciences.org/reviewed-preprints/107273#tab-content) |
| [Rieser et al., “Multifaceted Role of Galanin in Whole Brain Excitability”](https://elifesciences.org/reviewed-preprints/98634v1#tab-content) |
| [Lu et al., “The interplay between homeostatic synaptic scaling and homeostatic structural plasticity maintains the robust firing rate of neural networks”](https://elifesciences.org/reviewed-preprints/88376v1#tab-content) |
| [Ecker et al., “Assemblies, synapse clustering and network topology interact with plasticity to explain structure–function relationships of the cortical connectome”](https://elifesciences.org/reviewed-preprints/101850v1#tab-content) |
| [Dash et al., “Rules for reactivation across REM sleep microstates following sensory fear learning”](https://elifesciences.org/reviewed-preprints/102475v1#tab-content) |
| [March et al., “The Hungry Lens: Hunger Shifts Attention and Attribute Weighting in Dietary Choice”](https://elifesciences.org/reviewed-preprints/103736v1#tab-content) |
| [Molkov et al., “Introducing perturbations in point-process models of excitable systems”](https://elifesciences.org/reviewed-preprints/101959v1#tab-content) |
| [Huang et al., “Neural coding of multiple motion speeds in visual cortical area MT”](https://elifesciences.org/reviewed-preprints/94835v1#tab-content) |
| [Liu et al., “Striatal Crosstalk Between Dopamine and Serotonin Systems”](https://elifesciences.org/reviewed-preprints/107252#tab-content) |
| [Kang et al., “Rapid rebalancing of co-tuned ensemble activity in the auditory cortex”](https://elifesciences.org/reviewed-preprints/104242v1#tab-content) |
| [Praegel et al., “Age and Learning Shapes Sound Representations in Auditory Cortex During Adolescence”](https://elifesciences.org/reviewed-preprints/106387v1#tab-content) |
| [Zhang et al., “Oxytocin restores context-specific hyperaltruistic preference”](https://elifesciences.org/reviewed-preprints/102756v1) |
| [Hall et al., “A cortical–hippocampal communication undergoes rebalancing after new learning”](https://elifesciences.org/reviewed-preprints/107370#tab-content) |
| [Zhang et al., “Humans underestimate their body mass in microgravity”](https://elifesciences.org/reviewed-preprints/107472#tab-content) |
| [Barnby et al., “Self–other generalisation shapes social interaction and is disrupted in borderline personality disorder”](https://elifesciences.org/reviewed-preprints/104008v1#tab-content) |
| [Tardiff et al., “Normative evidence weighing and accumulation in correlated environments”](https://elifesciences.org/reviewed-preprints/100258v1#tab-content) |
| [Wu et al., “The Self-Interest of Adolescents Overrules Cooperation in Social Dilemmas”](https://elifesciences.org/reviewed-preprints/106840#tab-content) |
| [Chen et al., “Synchronous Ensembles of Hippocampal CA1 Pyramidal Neurons During Novel Exploration”](https://elifesciences.org/reviewed-preprints/96718v1#tab-content) |
| [Wang et al., “The relationship between cognitive abilities and mental health as represented by cognitive abilities at the neural and genetic levels of analysis”](https://elifesciences.org/reviewed-preprints/105537v1#tab-content) |

| [Zhong et al., “Modular DNA Barcoding of Nanobodies Enables Multiplexed in situ Protein Imaging and High-throughput Biomolecule Detection”](https://elifesciences.org/reviewed-preprints/105225v1/reviews#tab-content) |
| --- |
| [Majhi et al., “Non-autonomous cell redox-pairs dictate niche homeostasis in multi-lineage stem populations”](https://elifesciences.org/reviewed-preprints/96446v1#tab-content) |
| [Krwawicz et al., “Introduction of cytosine-5 DNA methylation sensitizes cells to oxidative damage”](https://elifesciences.org/reviewed-preprints/103432v1#tab-content) |
| [Xiu et al., “Action mechanism of a novel agrichemical quinofumelin against Fusarium graminearum”](https://elifesciences.org/reviewed-preprints/105892v1#tab-content) |
| [Chang et al., “Cancer cells differentially modulate mitochondrial respiration to alter redox state and enable biomass synthesis in nutrient-limited environments”](https://elifesciences.org/reviewed-preprints/107123#tab-content) |
| [Mohanty et al., “Deep Learning Reveals Endogenous Sterols as Allosteric Modulators of GPCRs”](https://elifesciences.org/reviewed-preprints/106397#tab-content) |
| [Wang et al., “Structure and evolution of Alanine/Serine Decarboxylases & S-Adenosylmethionine Decarboxylases in plants”](https://elifesciences.org/reviewed-preprints/91046v1#tab-content) |
| [Wei et al., “Crystal structure and catalytic mechanism of PL35 family glycosaminoglycan lyases with an ultrabroad substrate spectrum”](https://elifesciences.org/reviewed-preprints/102422v1) |
| [Govorunova et al., “Blue-shifted ancyromonad channelrhodopsins for multiplex optogenetics”](https://elifesciences.org/reviewed-preprints/106508#tab-content) |
| [He et al., “Coordinated regulation of chemotaxis and resistance to copper by CsoR”](https://elifesciences.org/reviewed-preprints/100914v1#tab-content) |
| [Jandu et al., “Membrane mimetic thermal proteome profiling (MM-TPP) enables proteome-wide target engagement in membranes”](https://elifesciences.org/reviewed-preprints/104549#tab-content) |
| [Chong et al., “Establishing the foundations for a data-centric AI approach for virtual drug screening”](https://elifesciences.org/reviewed-preprints/97821v1#tab-content) |
| [Schulze et al., “Effects of residue substitutions on the cellular abundance of proteins<br>”](https://elifesciences.org/reviewed-preprints/103721#tab-content) |
| [Maus et al., “Screening the MMV Pathogen Box reveals the mitochondrial bc1-complex as a drug target in mature Toxoplasma gondii bradyzoites”](https://elifesciences.org/reviewed-preprints/102511#tab-content) |
| [Lefroncois et al., “The Role of ATP Synthase Subunit e (ATP5I) in Mediating the Metabolic and Antiproliferative Effects of Biguanides”](https://elifesciences.org/reviewed-preprints/102680#tab-content) |
| [Ntourmas et al., “Endogenous oligomer formation underlies DVL2 condensates and promotes Wnt/$\beta$-catenin signaling”](https://elifesciences.org/reviewed-preprints/96841v1#tab-content) |
| [Marks et al., “Determining the off-target activity of antibiotics and novel translation initiation sites in mitochondria”](https://elifesciences.org/reviewed-preprints/103699#tab-content) |
| [Leanza et al., “Increased bone inflammation in type 2 diabetes and obesity correlates with Wnt signaling downregulation and reduced bone strength”](https://elifesciences.org/reviewed-preprints/102146#tab-content) |
| [Zhang et al., “Distinct mechanisms of inhibition of Kv2 potassium channels by tetraethylammonium and RY785”](https://elifesciences.org/reviewed-preprints/101855#tab-content) |
| [Luo et al., “Isobaric crosslinking mass spectrometry technology for studying conformational and structural changes in proteins and complexes”](https://elifesciences.org/reviewed-preprints/99809v1#tab-content) |
| [Antenucci et al., “Reassessing the substrate specificities of the major Staphylococcus aureus peptidoglycan hydrolases lysostaphin and LytM”](https://elifesciences.org/reviewed-preprints/93673v1#tab-content) |
| [Zhou et al., “Structural insights into human propionyl-CoA carboxylase …”](https://elifesciences.org/reviewed-preprints/98885v1#tab-content) |
| [Liu et al., “Genome-wide mapping of native co-localized G4s and R-loops …”](https://elifesciences.org/reviewed-preprints/99026v1#tab-content) |
| [D’Oliveira et al., “Recognition and Cleavage of Human tRNA …”](https://elifesciences.org/reviewed-preprints/91168v1#tab-content) |

| [Rucci et al., “Effects of blood meal source and seasonality on reproductive traits of Culex quinquefasciatus (Diptera: Culicidae)”](https://elifesciences.org/reviewed-preprints/89485v1#tab-content) |
| --- |
| [García-Ruiz et al., “Fitness drivers of division of labor in vertebrates”](https://elifesciences.org/reviewed-preprints/105501#tab-content) |
| [Nakagawa et al., “An illusion of a macroecological law, abundance–occupancy relationship in birds”](https://elifesciences.org/reviewed-preprints/95857v1#tab-content) |
| [Jiang et al., “Assessing plant phenological changes based on drivers of spring phenology”](https://elifesciences.org/reviewed-preprints/106655#tab-content) |
| [Howard–Spink et al., “Old age variably impacts chimpanzee engagement and efficiency in stone tool use”](https://elifesciences.org/reviewed-preprints/105411v1#tab-content) |
| [Rebindaine et al., “Developmental constraints mediate the summer solstice reversal of climate effects on European beech bud set”](https://elifesciences.org/reviewed-preprints/107554#tab-content) |
| [Smit et al., “Risk-taking incentives predict aggression heuristics in female gorillas”](https://elifesciences.org/reviewed-preprints/107093v1#tab-content) |
| [Croijmans et al., “Strip cropping shows promising increases in ground beetle community diversity compared to monocultures”](https://elifesciences.org/reviewed-preprints/104762v1#tab-content) |
| [Wang et al., “Loss of olfaction reduces caterpillar performance and increases susceptibility to a natural enemy”](https://elifesciences.org/reviewed-preprints/105585v1#tab-content) |
| [Tao et al., “Partitioning changes in ecosystem productivity by effects of species interactions in biodiversity experiments”](https://elifesciences.org/reviewed-preprints/98073v1#tab-content) |
| [Yang et al., “Interpreting prediction intervals and distributions for biologically meaningful effects”](https://elifesciences.org/reviewed-preprints/103339#tab-content) |
| [Gao et al., “Pesticide-induced resurgence in brown planthoppers is mediated by action on a suite of genes that promote juvenile hormone biosynthesis …”](https://elifesciences.org/reviewed-preprints/91774v1#tab-content) |
| [Fargeot et al., “Genetic diversity affects ecosystem functions across trophic levels …”](https://elifesciences.org/reviewed-preprints/100041v1#tab-content) |
| [Seltzer et al., “Female Moths Incorporate Plant Acoustic Emissions into Their Oviposition Decision-Making Process”](https://elifesciences.org/reviewed-preprints/104700#tab-content) |
| [Seguchi et al., “Vasopressin 1a receptor antagonist disrupts male–male affiliative relationships formed by triadic cohabitation in large-billed crows”](https://elifesciences.org/reviewed-preprints/103406/reviews#tab-content) |
| [Gatt et al., “Integrating microscopy and transcriptomics from individual eukaryotic plankton (Ukiyo-e-Seq)”](https://elifesciences.org/reviewed-preprints/102991#tab-content) |
| [Rydhmer et al., “Automating an insect biodiversity metric using distributed optical sensors: an evaluation across Kansas, USA cropping systems”](https://elifesciences.org/reviewed-preprints/92227v1#tab-content) |
| [Diaz–Colunga et al., “Full factorial construction of synthetic microbial communities”](https://elifesciences.org/reviewed-preprints/101906v1#tab-content) |
| [Zhang et al., “Neuropeptide bursicon and its receptor mediate the transition in seasonal polyphenism of Cacopsylla chinensis”](https://elifesciences.org/reviewed-preprints/97298v1#tab-content) |
| [Zhang et al., “Birds migrate longitudinally in response to the resultant Asian monsoons of the Qinghai–Tibet Plateau uplift”](https://elifesciences.org/reviewed-preprints/103971v1#tab-content) |

## Peer Reviews

confidence: 4
contribution: 4
initial_rating: 6
presentation: 4
questions: 1. The paper acknowledges that a failure to retrieve relevant papers can misrepresent soundness or contribution. What is the measured recall or F1 score for the crucial "Context Retrieval" step in the Soundness module? Could the authors provide an ablation study that systematically quantifies the impact on final coverage when retrieval is limited (e.g., reducing the number of papers/snippets retrieved) to better understand the system's robustness to retrieval noise or omissions?
2. Given the multi-stage reliance on the LLM, please provide an ablation where the full ScholarEval pipeline is executed using the same (presumably weaker) LLM backbone used for the "deep-research" baseline16. This is crucial to definitively isolate the performance gain stemming from the novel pipeline architecture versus the improved capabilities of the underlying language model (M).
soundness: 4
strengths: 1. ScholarEval fills a critical gap by offering a system that provides comprehensive, multifaceted evaluation across both methodological soundness and multi-dimensional contribution, moving beyond one-dimensional scoring or sparse feedback.
2. The explicit use of retrieval-augmentation ensures that all claims and suggestions are supported by direct evidence from the literature via citations, resulting in feedback that is demonstrably more actionable, valid, and useful for idea refinement compared to baselines.
3. The system significantly outperforms the strongest baseline (o4-mini-deep-research) by a large margin in coverage of expert review points and is consistently preferred by human experts in terms of depth, actionability, and literature engagement.
4. The introduction of ScholarIdeas (117 ideas, 1076 rubrics across four disciplines) is a major, high-quality contribution to the community, enabling future standardized benchmarking in this area.
summary: This paper introduces ScholarEval, a novel, retrieval-augmented framework designed for comprehensive evaluation of research ideas, a critical and underexplored area in AI co-science. The system assesses ideas based on two core, literature-grounded criteria: Soundness (empirical validity of proposed methods) and Contribution (advancement across multiple dimensions relative to prior work). The pipeline involves targeted search, information extraction from the full text and abstracts of scholarly literature (Semantic Scholar), and synthesis of dense, actionable, and cited feedback. To benchmark the system, the authors introduce ScholarIdeas, the first expert-annotated, multi-domain dataset for idea evaluation. ScholarEval significantly outperforms strong baselines, including a state-of-the-art agentic system, in terms of review coverage, depth, evidence support, and overall usefulness, as validated by automatic metrics and an expert user study.
weaknesses: 1. The framework's core strength is also its primary vulnerability. It relies entirely on retrieved literature, meaning it may misjudge the effectiveness of a truly novel method not yet represented in the literature or misrepresent the contribution if relevant contrasting papers are missed.
2. The reported high cost (up to $3) and time (around 12 minutes) per evaluation could pose a significant barrier to adoption for large-scale or batch ideation workflows.
3. Dependency and Sensitivity to LLM Backbone: The entire multi-stage process, from method extraction and summarization to synthesis, is highly dependent on the capability of the underlying LLM (M). The paper does not fully isolate the performance gain attributable to the pipeline structure versus the quality of the specific LLM used.

---

confidence: 4
contribution: 2
initial_rating: 2
presentation: 3
questions: See weaknesses.
soundness: 2
strengths: S1: The paper is well written. The structure of the paper is well organized.
S2: The paper contributes a new dataset SCHOLARIDEAS for evaluation of the idea evaluation works.
S3: The proposed framework SCHOLAREVAL outperforms the baselines. An expert study is also conducted and shows a preference for SCHOLAREVAL on all measured dimensions.
summary: This paper introduces SCHOLAREVAL, a retrieval-augmented framework designed to evaluate the quality of research ideas. The proposed framework consists of two modules to assess ideas: a soundness module for the empirical validity and a contribution module for novelty. Each module synthesizes the review following a retrieval-augmented way. To validate the system, the authors create a new dataset of 117 expert-annotated research ideas and detailed review rubrics from OpenReview and eLife. Experiments show that SCHOLAREVAL outperforms LLM baselines. An expert study is done to highlight its strength.
weaknesses: W1: The construction of the dataset needs more consideration. The research ideas used for evaluation are extracted from published papers by LLMs, not preliminary ideas before execution. The extraction is made by removing execution sections of papers to simulate research ideas at an early stage. The gap between the actual ideas and the extracted ones cannot be ignored.
W2: The paper highly relies on LLMs providing evaluation. More is expected on validation of the LLM-based metrics.
W3: The compared baselines are weak. The proposed framework is compared against multiple general LLMs and only one deep research system (o4-mini-deep-research). The Related Work section mentions several papers specifically designed for research idea evaluation but they are not used in the experiments. This makes the experimental results less convincible.

---

confidence: 4
contribution: 2
initial_rating: 2
presentation: 2
questions: 1. Regarding the dataset and evaluation scope:
    * Can the author explain the rationale for retrospectively extracting "ideas" from published papers to construct ScholarIdeas? This seems to be contrary to the goal of evaluating "pre-execution" ideas.
    * Why is the automatic evaluation limited to the ScholarIdeas dataset? Have the authors considered testing on other contemporary, real-world "testbeds" (such as data from ACL 2025, NeurIPS 2025) to verify the generalization ability of the framework?
2. Regarding the baseline comparison: Could the author explain why all relevant academic baselines (such as DeepReview, THE-Tree, etc.) were omitted from the experimental comparison? Without such a comparison, how can the superiority of ScholarEval over the domain SOTA be verified?
3. Regarding the validity of the evaluation protocol: Given that the appendix (Table 6) shows extremely low agreement (Kappa value close to 0) between LLM-as-judge and human evaluators, why do the authors still use it as the core evidence to support the superiority of the system in Figure 4?
soundness: 2
strengths: 1. The problem addressed by the paper—automatically evaluating the quality of research ideas—is a core challenge in the field of AI scientific discovery and holds high value.
2. The paper focuses on generating dense, literature-supported, actionable feedback rather than just a single assessment score. This focus on actionability is commendable.
3. Section 5 of the paper, "Expert User Research," is its most rigorous part. Different from the main body of the paper (which is based on ScholarIdeas evaluation), this study adopted the correct methodology (real experts evaluating fundamental pre-execution ideas) and demonstrated that the system outperforms a key baseline in practical applications.
summary: This paper presents ScholarEval, a retrieval-augmented generation (RAG) framework designed to automatically evaluate research ideas. The framework divides evaluation into two dimensions: "Soundness" (the empirical validity of the method) and "Contribution" (progress relative to previous research). To evaluate the system, the authors constructed a new expert-annotated dataset, ScholarIdeas, which contains 117 research ideas (extracted from published papers) from 4 disciplines (including AI and life sciences) and their corresponding review rubrics.

The main advantages and contributions of the paper are claimed to lie in: (1) proposing a multi-stage evaluation framework that can provide dense and actionable feedback; (2) constructing a new evaluation benchmark, ScholarIdeas; (3) experiments show that ScholarEval significantly outperforms baselines including general LLMs and 04-mini-deep-research in both automatic metrics (covering expert review points) and human evaluation (User Research).
weaknesses: - The crisis of effectiveness in the core dataset (ScholarIdeas).
    - Task misalignment: The paper claims to evaluate pre-execution creativity, but the dataset was extracted ex post facto from already completed papers. This represents a fundamental task misalignment.
    - Narrow evaluation scope: The automatic evaluation of the paper relies entirely on ScholarIdeas. The lack of testing on other real-world data (such as ACL 2025, NeurIPS 2025, etc.) makes it impossible to prove the generalization ability of this framework.
    - Timeliness issue: How can we ensure that this data will still be valid next year? Also, isn't the evaluation of 117 ideas too few?
- The baseline comparison is weak and misleading.
    - Baseline is seriously insufficient: The experimental comparison in the paper is seriously inadequate. It avoids comparison with real academic competitors (i.e., other published creativity assessment/novelty assessment tools).
    - Omission of key literature: The paper completely overlooks directly relevant contemporary frameworks such as DeepReview (with its 'deep thinking process') or THE-Tree ('historical evolution'), which are designed to deepen evaluation and reasoning.
    - "Straw Man" Argument: The paper chooses to compare with a general LLM and a commercial Black box system (04-mini-deep-research). This is a "straw man" style of argument and does not prove its superiority over other research work in the field.
- Inherent bias against breakthrough innovation: This framework is entirely "literature-based," which means it is methodologically unable to fairly evaluate truly novel, interdisciplinary, or paradigm-challenging ideas that lack literature precedents.
- Invalid and contradictory evaluation protocols.
    - **Automated evaluation of the loop**: The primary metric "Coverage" measures the degree of match between the system (an LLM pipeline) and the "gold standard" (a benchmark extracted by another LLM), rather than aligning with the true human ability to evaluate creativity.
    - **Unreliable LLM-as-judge**: The paper uses LLM-as-judge (Figure 4) to claim its superiority. However, the validation conducted by the authors themselves in the appendix (Table 6) shows that the agreement between LLM-judge and human annotators is extremely low (e.g., Cohen's Kappa for the "Depth" dimension is -0.0161), rendering its conclusion invalid.

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 3
questions: 1. How to ensure an exhaustive search of relevant literature in the framework? As AI papers might be open-sourced largely, does it apply to other disciplines to retrieve papers?

2. How is the efficiency of the framework to evaluate a research idea?

3. The standard deviation of coverage for the proposed method in Table 1 seems to be larger than the baselines across disciplines and variants. Does that indicate an unstable performance of the proposed method?
soundness: 2
strengths: 1. This paper contributes a framework to evaluate research ideas and a dataset of ideas and reviews verified by human experts. The framework evaluates the ideas from two aspects and also provides feedback. It shows superiority in multiple aspects over several baselines.

2. The dataset includes human reviews collected from public platforms and covers four disciplines: artificial intelligence, neuroscience, biochemistry, and ecology. 

3. The experiments evaluate the proposed framework using various LLMs to show its effectiveness across LLMs. The evaluation measures on various metrics and includes human verification.
summary: This paper presents a retrieval-augmented evaluation framework named SCHOLAREVAL to evaluate the research ideas by the soundness and contribution, and an expert-verified dataset named SCHOLARIDEAS of 117 research ideas and 1076 review rubrics from four disciplines. The framework adopts a multi-stage pipeline, starting with finding relevant papers of a research idea and synthesizing the existing findings, then measuring the soundness and contribution by evaluating the method components in existing work and comparing the contribution to related work. The experiments show that the proposed framework evaluates the ideas more similarly to human reviews on the proposed dataset.
weaknesses: 1. The evaluation of soundness and contribution is dependent on the performance of LLMs in retrieving and comparing with prior work, which might be tied to LLM capacity and undermine the evaluation of soundness for ideas of new tasks or novel evaluation.

2. The soundness evaluation works by extracting the method components and evaluating their effectiveness in prior work. This module seems to overlook the connections or interplay between the method components or the significance of model components in different contexts or assumptions.

3. It's unclear if unsuccessful ideas are included in the proposed datasets in Sec 3.1 and whether the framework is capable of distinguishing them from successful ones.