EvidenceBench: A Benchmark for Extracting Evidence from Biomedical Papers
Authors: Jianyou Wang
Weili Cao
Kaicheng Wang
Xiaoyue Wang
Ashish Dalvi
Gino Prasad
Qishan Liang
Hsuan-lin Her
Mingwang
Qin Yang
Gene W. Yeo
David E Neal
Maxim Khan
Christopher D. Rosin
Ramamohan Paturi
Leon Bergen
Venue: COLM 2025 (2025)
Keywords: Biomedical Benchmark
Scientific Information Extraction
Large Language Models
BioNLP
TL;DR: A large biomedical benchmark to evaluate LLMs' evidence-finding ability for scientific hypotheses and an automatic pipeline to create such benchmarks.

Abstract: We study the task of automatically finding evidence relevant to hypotheses in biomedical papers. Finding relevant evidence is an important step when researchers investigate scientific hypotheses. We introduce EvidenceBench to measure models performance on this task, which is created by a novel pipeline that consists of hypothesis generation and sentence-by-sentence annotation of biomedical papers for relevant evidence, completely guided by and faithfully following existing human experts judgment. We demonstrate the pipeline’s validity and accuracy with multiple sets of human-expert annotations. We evaluated a diverse set of language models and retrieval systems on the benchmark and found that model performances still fall significantly short of the expert level on this task. To show the scalability of our proposed pipeline, we create a larger EvidenceBench-100k with 107,461 fully annotated papers with hypotheses to facilitate model training and development. Both datasets are available at https://github.com/EvidenceBench/EvidenceBench

Published as a conference paper at COLM 2025
CvidenceBenche A Benchmark for Extracting Evidence from Biomedical Papers
Jianyou Wang"; Weili Cao Kaicheng Wang', Xiaoyue Wang', Ashish Dalvi', Gino Prasad,, Qishan Liang', Hsuan-lin Her?, Ming Wang*, Qin Yang", Gene W. Yeo?, David Neal?, Maxim Khan?, Christopher D. Rosin", Ramamohan Paturi Leon Bergen't
'Laboratory for Emerging Intelligence, University of California, San Diego
Flsevier
Department of Cellular and Molecular Medicine, University of California, San Diego
*Sichuan Cancer Hospital & Institute
The Third People's Hospital of Chengdu (jiwl01, w2cao, rpaturi, 1bergen]@ucsd. edu
Abstract
We study the task of automatically finding evidence relevant to hypothe- ses in biomedical papers. Finding relevant evidence is an important step when researchers investigate scientific hypotheses. We introduce EvidenceBench to measure models performance on this task, which is created by novel pipeline that consists of hypothesis generation and mentence-by-seneent annotation of biomedical papers for relevant evi- dence, completely guided by and faithfully following existing human ex- perts judgment. We demonstrate the pipeline's validity and accuracy with multiple sets of human-expert annotations. We evaluated a diverse set of language models and retrieval systems on the benchmark and found that model performances still fall significantly short of the expert level on this task. To show the scalability of our proposed pipeline, we create larger EvidenceBench- 100k with 107,461 fully annotated papers with hypotheses to facilitate model training and development. Both datasets are available at https:/ github.com. com EvidenceBench, EvidenceBench
Introduction
There are more than 1 million biomedical papers currently published per year, and more than 35 million papers collected in the PubMed database of biomedical literature Márquez et al. (2024). The scale of the literature has made it increasingly labor-intensive to determine what is known about a research question. Systems such as Open Deep Research and Elicit aim to automate large-scale analysis of the scientific literature. Notably, it is not enough for these systems to merely produce high-level summaries or restate an article's claims. To accurately assess what paper contributes research question, such systems should identify the supporting evidence the experimental or observational data that underpin the claims.
We focus on this critical step of evidence retrieval: given a hypothesis, system must locate the key parts of a paper that provide pertinent experimental details, numerical findings, or other forms of evidence that address that hypothesis. This step provides grounding to the judgments of an automated system, allowing researchers to quickly judge whether specific claim is supported by empirical evidence. However, creating and evaluating such systems remain non-trive con-trivial. On the one hand, few annotated benchmarks exist in the biomedical domain to measure how effectively models can identify and extract evidence. On the other hand, curating such a dataset is time-consuming and expensive, especially such annotations need be performed by experts.
"equal contributions
"equal semior authors
1

---

Published as a conference paper at COLM 2025
To address these challenges, present EvidenceBench and its large-scale extension Evidence a new benchmark for sentence-level evidence retrieval in biomedical research papers. Our proposed datasets are fully open-sourced under the CC-BY license and encompass a comprehensive range of biomedical topics.
We introduce novel pipeline for creating EvidenceBench and EvidenceBBnch-11000 pow- ered by Large Language Models (LLMs). The pipeline has two main components: hypothesis generation and an alignment annotator that matches hypotheses with sentences from pa- pers. Informally, we use existing evidence summary from review papers to generate hypothesis and use the same evidence summary find sentences that provide evidence for the generated hypothesis. See Section and Figure 2 233oo for details.
Our pipeline is highly scalable, reducing the construction time of EvidenceBench from over 3,000 human hours and $120,000 in wages to just 3 API hours and $5,000 in API costs, using state-of-the-an LLMs the time of data creation, Claude3-Opus for hypothesis generation and GPT4-0125 for alignment annotation. During the construction of EvidenceBench-1000 we used GPT4-o-ming for alignment annotation and kept the construction time and cost under 24 hours and $5000. For details about human cost estimate, see Section 3.4
Table demoonssttesss EvidenceRench-1000 is suitable for fine-tuning LLMss and embedding models as we observed significant improvements of fine-tuned models over their pretrained baselines.
We conduct benchmarking study on a variety of large language models (LLMs) and embedding models, which provide several insights. First, although LLMs still fall short of expert-level performance have the potential to assist them. Second, embedding models consistently underperform compared to large language models due to lack of global context from research papers. Third, we report that current LLMs trained for long document understanding still get "Lost in the Middle" (Liu et al. 2024). See Appendix for details of our analyses.
this task, indicating that they cannot replace humans, they
We highlight some important sections in the paper:
Section 2.1
explains two critical concepts: Study Aspect and Source of Information.
Section 2.3
presents Aspect Recall, the evaluation metric for our experiments.
Figure illustrates our key methodological contribution: leveraging expert-written evidence summaries from review papers guide an LLM-based alignment annota- tion process (Figure 3 ensuring our pipeline adheres to human expert judgments.
Section 3.3.1 shows experts confirm the scientific value of our generated hypotheses.
Section 3.4 and Table Table provide statistical evidence that our alignment annotation procedure achieves comparable performance to biomedical PhD students.
2 Task Formulation
The EvidenceBench task is to identify the most important pieces of evidence relevant to hypothesis. This is formulated sentence retrieval task. Given paper, the task is to retrieve a set of sentences that jointly provide the most important pieces of evidence. See Figure for example.
2

---

Published as a conference paper at COLM 2025
2.1 Definitions
The full-text of a research paper is presented as a list of sentences ! This ordered list of sentences the Candidate Pool. Very importantly, research papers and review papers are completely different.
Candidate Pool:
An evidence summary is written by human experts. It included in open-sourced review papers, such as surveys, monographs and systematic reviews. An evidence summary directly linked one single research paper. contains all pieces of evidence (from this research paper) that the human experts believe are important and relevant to hypothesis. Evidence summary could take the form of a normal paper summary, or a bulleted list, or even tabular format.
Evidence Summary:
Study Aspect: A study aspect, denoted as fú, single piece of information. For instance, an experimental outcome or detail from study design. Note, study aspects are decomposed from a evidence summary written by experts. Each study aspect identified by human experts and represent an important detail relevant for hypothesis. Therefore, study aspects are considered to be human experts' judgment. See Figure | for example study aspects.
Source of Information: A sentence in research paper is considered source of information for a study aspect it satisfies the following two criteria.
1. The content of the sentence implies most of the study aspect.
2. For any part of the study aspect that the sentence does not cover, the information must be easily deducible from the surrounding context.
Given a sentence s, and a study aspect fú, define the source-of-informatio indicator function Sf),
s(f,s,) {
if sy is a the source of information for fi
otherwise
Note, if sentence is source of information for study aspect, informally say the sentence covers this aspect. Note, since study aspect is decomposed and only represent one piece of information, one single sentence is enough to cover it.
Hypothesis: A hypothesis a scientific generalization, usually expressible in one line (less than 50 words). It should not be tied specific details of an experiment. See Figure for example and Section 3.3 for the generation and validation process.
The evidence set is the set of study aspects which provides evidence relevant to hypothesis, according to human expert judgments. Very importantly, we use an Evidence Summary to derive an evidence set, see Figure right side. A study aspect in this set may provide evidence on its own, or only in combination with other study aspects.
Evidence Set:
2.2 Task Definition
We now introduce our primary task, Evidence Retrieval @K (ER@K). Informally, the task to find K sentences in research paper which provide the greatest amount of evidence relevant to hypothesis. This is operationalized as finding sentences in the research paper which cover the most study aspects from the evidence set.
Formally, given a hypothesis and candidate pool, the task for system is to retrieve K sentences from the candidate pool which provide evidence relevant to the hypothesis. The retrieved sentences are then evaluated against the evidence set, which contains ground-truth study aspects (i.e. pieces evidence relevant to the hypothesis identified by human experts).
Figures and tables excluded for two reasons. First, EvidenceBench is designed to evaluate textual models. Second, important information from figures and tables often restated in the paper's text. In the rare case where an evidence summary is largely based on figures or tables from a paper, it means most study aspects cannot be covered by sentences from the paper. Such cases will be discovered at the alignment annotation stage and filtered from EvidenceBench. Specifically, we filter out cases where less than 70% of study aspects from the evidence summary are covered by sentences.
3

---

Published as a conference paper at COLM 2025
Model Input
Annotation Alignment
Evidence Retrieval@S Evalustion
Artstolochic Acid Hypothesis:
mutation
Study Aspects:
ranall carcinoma (RCC).
Candidate Pool Sentences:
99 Source Information for Aspect
Study Aspect 1: Cumuletive ingestion of more than 250 mg of AA increased the risk csRCC (OR.
Sentence 8: Cumulative ingestion
risk
s69 is Source of Information for Aspect 1
(OR, 1. 25), and we detecied dA-AL-I adducts 76% of ccRCC patients.
Ner Model selected)
Study
A distinctive AA mutational sequenced
signature was evident coRCC axomes from AA exposed patients.
Sentence 69: The results indicato an adjusted OR 1.25 (1.004-1.647) for coRCC consuming than 250 onlectd
Study Aspect ariver gane was VHL..
Irequently mutated
S106 is Source Information for Aspact
frequently mutated driver gene (7/10 tumorr) AA. exposed coRCOs
Sentence
Study Aspect lumors.
$106 is Source Information for Aspect 4
(é Model selectes
Aspect Recall is 3 /
Figure 1: In the task of ER@3, model hypothesis and the full sequence of sentences from paper as candidate pool and must select up sentences. The model selects S9, S69, and S106 the set of retrieved sentences. When compared against the ground-tre aspects, these 3 sentences only cover Aspect 1, and 4, since S9 and S69 are repetitive and cover the same Aspect Aspect 2 is missed, resulting in 75% Aspect Recall.
evidence set which contains
The goal is for the K retrieved sentences to be sources of information for as many of the study aspects in the evidence set possible. During the retrieval task, the system does not have access to the ground-truth evidence set; the evidence set is only used for evaluation.
In the second version of the task, only study aspects related to the results and analyses are considered. Study aspects related to background and methods are filtered out of the evidence set. This task is called Result-ER@K, and focus on system's ability identify numerical and experimental results.
2.3 Evaluation Metrics
To determine the quality of a system's retrieved sentences, we use Aspect Recall. Let {k be the of retrieved sentences, and fmr be the set of study aspects in the evidence set. The Aspect Recall is defined as
Σj 1 [Ia S(snf)) 21
(1)
This measures the fraction of study aspects that can be covered by retrieved set of k sentences. See Figure 1 for example calculation of Aspect Recall
Dataset Construction Pipeline
3.1 Data Sources
There two data sources for EvidenceBench and EvidenceeBench-10k. First, collection of 107 887 CC-BY open-sourced biomedical research papers where each research paper represents datapoint. Second, collection of 44,772 review papers from PubMed Central.
2Our objective is to find the smallest amount of sentences to cover the maximum amount of study aspects. We choose Aspect Recall over precision metrics because precision metrics could inadvertently encourage models to select redundant sentences, which is counterproducctvv our objective.

---

Published as a conference paper at COLM 2025
Reviow Paper
Study Aspects
Aristolochic
Cancer Humnens
Nephoopetty
Study Aspect Cursaaa incestionn
Cancer
of more then 250 mg ir nowared the nox of cCRDC (DR, 25)
Since the 1993s, studies have been trying
Sumonndnn Content
prove cianal ins betwoer AA-DNA and variou
Hypothesis
Section Urothel
mutetiona is gratuue was enicent zix of the sequemoed poRCC
Study
distnnctie AA
Is Artesischhic
Section 2.5.5 Renal concers
Mvexpooed
Adid (AA) incuced DRAA musalassn
Hypothesis generation
Plssuts: from clanns at (ants showed that a carrictive ingeocion mow then my iecrosedd
Evidence Summary goes
causal for nenall (PC))?
munational
sudert
wnced osfCo
through Aspect
Suacy Aupaett moat Ireepurrlyy gene
oxcmzc from AV eopoepooo posinnts Among theso tumors the most frqquertly musated dliver gana was VHL., which
Decomposition
was musated
of surmples
201 Chen al inveriigtedd
Surounding
the roee Ai-induoed DMA adducts in tumor supprescor gere parkenta
Btacy Aspact VHL was
Section Cannstoaiity
Section 3. Cancer in Experimental Animal
Figure 2:
The highlighted paragraph in the review paper is the evidence summary. The left side shows how a hypothesis is generated (extracted) from the evidence summary and its surrounding context in the review paper. The right side shows the evidence summary being decomposed into study aspects.
Each biomedical research paper has corresponding evidence summary included in review paper. Specifically, EvidenceBench has 426 datapoints and EvidenceBench- has 107 461 datapoints. Our datasets cover wide range of biomedical topics, see Appendix A.3 for details.
3.1.1 Train/Test Split
We use random train/test split of 133/ 293 task instances for the original EvidenceBench. We use random train/test split of 77,461/20,00 for EvidenceBench-100k All of the prompt optimization is performed the train sets and all of the few-shot examples used in the prompts are selected from the train sets. All datasets have CC-BY licenses.
Table 1: EvidenceBench Test Set. Optimal Number of Sentences refers to the smallest number of sentences that are sources information for the most of study aspects in an evidence set.
Candidate Tokens max
Sentences
Study Aspects avg
Optimal Number of Sentences
Dataset
min
avg
min
avg
avg
1691
5378
23960
9.5
Test Set (Result Retrieval)
288
1691
5382
23980
48
168.4
794
4.2
18
2.1
3.2 Dataset Pipeline Overview
A task instance (i.e. datapoint) is constructed as follows. After we harvest an expert- written evidence summary from a review paper, we generate hypothesis from it. Since the evidence summary was written to summarize a research paper, we take the sentences from this research paper as the candidate pool for the task instance. We then decompose the expert-written evidence summary into set of study aspects, also known as the evidence set. Each study aspect is used to annotate each sentence in the candidate pool. This is the alignment annotation process, which determines sentences in the research paper that are sources of information for the study aspect, and consequently, are relevant for the hypothesis. In the next sections, explain the procedures of harvesting evidence summary from review papers, hypothesis generation, aspect decomposition, and alignment annotation.
Harvesting the Evidence Summary: The highlighted portion in the review paper in Figure is an evidence summary. Evidence summary has an XML citation embedded in it, so can be identified and harvested by deterministic algorithm. See Appendix H for details.
Aspect Decomposition: For each evidence summary, we decompose into study aspects.
These study aspects comprise the evidence set for the research paper. Table shows
5

---

Published as a conference paper at COLM 2025
summary average contains 10 study aspects and half of them results aspects. In EvidenceBench, decomposition is done by GPT4-0125 and inspected by human researchers. In EvidenceeBench-100k the decomposition is done by GPT-to-mini-0771 and 200 randomly sampled instances are inspected by human researchers and are found to be of high quality.
3.3 Hypothesis Generation
A review paper focuses on specific hypothesis and survey a number of research papers, summarizing the evidence that each provides for the hypothesis. For each evidence sum- mary, our goal is to extract the hypothesis that it is providing relevant evidence to. Think of hypothesis as an explicit or implicit question waiting to be recovered. In order to do this, we provide an LLM (Claude3-Opus) with the evidence summary well surrounding para- graphs. The model is then prompted to recover the hypothesis being discussed in the review paper. See Figure left side. To ensure high quality hypotheses for EvidenceBBench1100 Claude3-Opus is also used.
3.3.1 Expert Validation of Hypotheses
We perform an expert evaluation of the hypotheses extracted from review papers in Evi- denceBench, focusing on two questions:
1. Does the hypothesis have sufficient scientific value?
2. Does the corresponding evidence summary provide evidence which is relevant to the hypothesis?
The annotation team for this task consisted of three medical doctors. The first expert defined the annotation guidelines and provided feedback on an initial set of 20 extracted hypotheses. This feedback was also used to perform prompt optimization for Claude3-Opus.
After finalizing the guidelines and prompt, separate set of 50 hypotheses was generated. The two other annotators each evaluated 25 hypotheses. Annotation guidelines are in Appendix B
For Question 1, 50/50 hypotheses were judged to have sufficient scientific value. For Question 2, 47/50 hypotheses were judged to be relevant to the corresponding evidence summaries. This demonstrates that hypotheses were correctly extracted from review papers.
3.4 Alignment Annotation of Study Aspects and Sentences
We have far described the procedure for decomposing study aspects and recovering hypotheses from the review papers. A list of study aspects describes the evidence that specific research paper provides relevant a hypothesis. The final step is to identify which sentences of the original research paper serve as sources of information for each study aspect. Because study aspects can, in general, come from any part of the research paper, this requires annotating every sentence in the research paper according to whether it matches each study aspect.
Table shows that each paper has approximately 168 sentences and 10 study aspects average. EvidenceBench contains more than 400 research papers. Sentence-by-sentene annotation requires approximately 700,000 sentence annotations, which is infeasible given the use of expert annotators; we estimate that it would require more than 3000 hours of annotation We therefore develop a pipeline for automating the annotation process, and perform human evaluation of its reliability. We observe that the task of labeling sentence
'Our experiments showed that single bioinformatics PhD student camnot reliably annotate one paper for one aspect in less than 20 minutes. Reliable annotation requires two Bio PhD students collaborating, taking 20-30 minutes per aspect. Only with this collaborative approach did annotations show high inter-annotator agreement across different teams. The calculation breaks down as follows: paperr
2 PhD students 25 minutes/ /aspect 10 aspects/ sxpecss/paper" * 400 papers 3,333 human hours. Each PhD
level expert has expected hourly wage between $40 to $95 (see wage standard in Rein al. 2024

---

Published as a conference paper at COLM 2025
according to whether it is source of information for a study aspect is considerably simpler than the benchmark's ' full sentence retrieval task. It only requires a judgment of whether single sentence from the research paper contains most of the same information study aspect.
GPT4 for Alignment Annotation
Candidate Pool
Sentence 9:
Cumulaaive ingediion of more AA inceassed risk D2RC
S9 is Source of Information Aspect
(OR, 1,25), detected dA-AL-I adducts ccRCC patlants.
Study Aspect
Study Aspect 1: Cumulative ingestion of more then 250 mg of AA incressed the risk of ccRCC (OR, 1.25).
969 Source Information for Aspect 1
Sentence 68: The results indicate an adjusted
OR of 1.25 (1 (1.004-1.55 1,47) for acRCC in persons consuming more than 250 mg of AA.
:S Source Information for Aspect 1
Sentence 106: VHL was the moet Trequently musated driver gene (7/10 tumorr) in our AA- exposed ccRCCs
Figure 3: The process for matching sentences with study aspects. In this example, GPT4 sees the study aspect, one candidate sentence from the research paper with the context, and is prompted to determine whether the candidate is a source of information for the study aspect.
For the annotation pipeline, GPT-4 is shown target sentence from the research paper and study aspect (as well as some additional context: the 10 surrounding sentences from the research paper, and the evidence summary from the review paper). It is then asked to evaluate whether the target sentence implies most of the information contained in the study aspect. See Figure
The optimization and evaluation of the pipeline were performed using a development set/ /tst set split. The prompt and annotation methodology were optimized on develop- ment set of 37 research papers. The prompt and optimization procedure is provided in Appendix
After the pipeline was finalized, it was evaluated on test set of 50 randomly sampled research papers. For each research paper, a single study aspect was selected, and every sentence in the paper was annotated for this study aspect. The labeling was performed by four annotators, who are Ph.D. researchers in bioinformatics. The annotators were split into two teams. Each annotator first performed the annotation task independently. The pairs within each team then consulted with each other to reach consensus judgments. Finally, inter-annotator agreement was calculated by comparing the judgments of the two teams. Full annotation guidelines are provided in Appendix
Each team labeled 8111 (sentence, study aspect) pairs in total. Table inter-annotator agreement between the two human teams, and between the human teams and GPT-4. Human and GPT-4 judgments match each other more than 98% of the time. Because of class imbalance (positive labels are rare, around 150 out of 8111), other measures of agreement such as Cohen's are in the mid 60's, indicating substantial agreement between the human teams and between the human teams and GPT-4 Bootstrapped hypothesis tests find no significant difference between the human numan/human agreement rate and the human agreement rate. To account for potential correlations among annotations from the same research paper, we additionally computed confidence intervals using hierarchical bootstrap sampling, and found similar results, shown in Appendix
For the larger EvidenceBBnchh-10k, over 150 million sentence-aspect pair judgements would need to be made. We switch the annotator to GPT-4o-mini-0718. We validated the quality of its annotation by re-running the hypothesis test on the same 50 randomly sampled papers, and found very similar Human & GPT agreement. See Appendix D for 4o-mini's results.
7

---

Published as a conference paper at COLM 2025
Table Hypothesis lesting Results for Automatic Alignment Annotation
Metrics
Human & Human Average Human & GPT Average
p-value
Exact Accuracy
98.8 + 0.3
98.7 0.2
0.21
F1 Binary
66.0 + 6.5
64.6 + 5,6
0.64
Cohen's
65.4 + 6.6
63.9 5.6
0.63 /
Spearman's
65.4 + 6.6
64.0 + 5.6
0.65
Experiment Setup
EvidenceBench Tasks: we consider four evidence retrieval tasks with different settings.
Evidence Retrieval @Optimal: denoted as ER@optimal. The smallest number of sentences to cover all aspects is denoted as Optimal. The average optimal number is only 4.6. The task for model retrieve no more than the Optimal number of sentences to form the best evidence set relevant to the hypothesis. Evidence Retrieval @10 is denoted as ER@10. See Appendix E.1 for the prompt.
Result Evidence Retrieval @ @Optimal: denoted as Desult-ER@Optimal. The task restricts the model to retrieve no more than the optimal number of sentences, which is calculated by the minimum number of sentences required to cover all study aspects labeled as "Results". This labeling is done by GPT4-0125. Empirically, half of the aspects are labeled as "Results', Table Result Evidence Retrieval @5 is denoted as Result-ER@5. See Appendix E.2 for the prompt.
Experiment Models We test the following models Claude3-Opus (Anthropic 2024), Gemini
1.5 (Google 2024), GPT-4o (OpenAl AT (024),, Llama3-70B, Llama3-88 (AI 2024), E5-v2 (Wang et al. 2022), ApenAl Embedding v3 Ap 2024 Voyage v2 (Voyage 2024)), GritL.-
7B Muennighofff et al. 22024), E5-Mistral-7B (Wang et al.) 2023), NV-Lmbed-v2 et al.
2024) which the leader of the Massive Text Embedding Benchmark MTEB (Muennighofl et al. 2022).
4.1 Evaluation strategies
We implement three standard practices for evaluating LLMs on long-context benchmarks (Bai et al.] 2023: Zhang et al. 2024). Chain-of-Thought (CoT): our default evaluation strategy, optimized from our train set. In-Context-Leaming (ICL): From the train set, we randomly sample 8 example hypotheses and their corresponding ground-truth set of retrieved sentences, as per standard ICL practices (Wei et al., 2022 Min et al., 2022), in addition to our default CoT prompt. Section-by-Section bSec- We divide research paper by its natural sections. In the first stage, LLM only retrieves from each section one at time. In the second stage, all retrieved sentences are presented to the LLM and final selections are made.
Our evaluation metric is Aspect Recall, defined in Section 2.3 In Appendix E we presented all prompts used in our experimental evaluation for LLMs, instructions for embedding models as well standard errors for model evaluations.
Table 3: Four tasks are reported on EvidenceBench test set. For each model, the highest number is reported if multiple strategies are used.
GPT-4o Claude3 Gemini LLama3-70B OpenAI Voyage GritLM E5-Mistral NV-Embed
ER@Optimal ER@10
51.4
47.6
48.3
46.7
25.1
22.7
27.0
22.7
25.2
71.6
66.4
65.4
65.4
42.2
42.0
46.4
41.9
44.7
Result-ER@Opt Result-ER@5
52.6
51.7
46.7
46.2
18.3
18.9
19.3
20.1
70.8
68.7
65.4
63.7
33.1
31.9
39.1
33.6
35.6
8

---

Published as a conference paper at COLM 2025
5 Results
From Table list the best performance for each model (LLM embedding model) on the four constrained Evidence Retrieval tasks in the original EvidenceBench. GPT-4o consistently outperforms others across all tasks, while Gemini, Claude3-Opus, and Llama3- 70B closely trail behind. Llama3-70B can only be evaluated using the Sec-by-Sec strategy due to its limited context window, but it shows robust performance across all tasks. There qualitative difference between LLMs and embedding models, partially because embedding models are not context-aware when calculating sentence embeddings. This invites future work on general-purposed context-aware embedding models.
In-context learning: From Table 4 we see an 8-shot ICL does not significantly alter perfor- mances for LLMs. In particular, GPT-4o and Gemini-1.5 slightly improve, while Claude3- Opus slightly degrades. This indicates the primary difficulty is context-length and not failure to understand task requirement, suggesting that ICL is less effective on long-context benchmarks.
Section-by-Sectton Processing: On the other hand, from Table Sec-by-Sec considerably improves Gemini and Claude's performances, suggesting that the default longer-context version of the task hinders their retrieval abilities. Section-by-section is by far the most robust strategy observed here. We performed further analyses in Appendix
sassoning-based LLMs: We tested two reasoning-based LLMs, Gemini 2.5 Pro Google 2025) and Claude 4 Sonnet (Anthropic (025), using the baseline prompt for ER@Optimal. As we in Table 5 they scored 52.5 and 49.1 ER@Optimal, respectively, numerically higher than GPT-40's performance. We leave a more extensive evaluation of reasoning models EvicenceBench for future work.
Table 4: Comparison of different strategies for LLMs the EvidenceBench test set.
Baseline
ICL
Sec-by-Sec
ER @ Optimal
ER@
ER @ Optimal
ER@
ER @
ER@
Model
10
10
Optimal
10
GPT-4o
48.1
69.6
51.4
68.7
50.9
71.6
Claude3
41.1
53.6
38.3
55.4
47.6
66.4
Gemini
42.7
63.0
43.2
62.4
48.3
65.4
Llama3-70B
46.7
65.4
Table 5: Evaluation for reasoning-based LLMs using baseline prompt
Model
Gemini 2.5 Pro
Claude 4 Sonnet
GPT-4o
Claude Opus
Gemini 1.5
ER@Optimal
52.5
49.1
48.1
41.1
42.7
Fine-tuning and Evaluation on EvidenceBench-100k
EvidenceBench-1100k is split into a 80k train set and a 20k test For cost reasons, we randomly sample 3000 datapoints from the 20k test set to evaluate. Table 6a shows Evidence test set can be used evaluate and clearly differentiate various models' performance.
Furthermore, we fine-tune two models: E5-v2 335M and Llama3-8B (sec-by-sec strategy) using the 80k training datapoints. We test them on the original EvidenceBench test set for the task of Result-EROOpimaa. We notice both fine-tuned models show significant improvements over their baselines shown in Table [bb This shows the EvidenceBench- 100k train set can be used for model developments. See full details of fine-tuning at Appendix[]

---

Published as a conference paper at COLM 2025
Table 6: Comparison of Model Performances on EvidenceBench Datasets
(a) EvidenceeBechh100K test set
(b) EvidenceBench test set.
Model
Result-ER@Optinal
Model
Result-ER@Optimal
GPT-4o
42.84%
Pretrained Llama3-8B
35.8%
Claude3 35.12%
Finetuned Llama3-8B
41.0%
GritLM 14.59%
Pretrained E5-v2
15.2%
OpenAl 10.95%
Finetuned E5-v2
32.9%
5.2 Effect of Hypothesis Negation
In EvidenceBench, most the sentences which are annotated as relevant to a hypothesis provide evidence in support of the hypothesis, rather than against it. To evaluate whether models are able to identify information which provides evidence against a hypothesis, we created negated version of EvidenceBench by negating each hypothesis. See Appendix E.8 for the prompt and some examples of negated hypotheses. By negating each hypothesis, we have transformed the support relationships into refutation relationships.
Using the baseline prompt, we evaluated GPT-4o and Claude3-Opur on this negated version of the original Bvich test set. In Table 4 the ER@Optimal for GPT-4o is 48.1 on the original set, the new experiment shows the ER@Optimal drops to 46.7 on the negated set. The ER@Optimal for Claude3 41.1 on the original set and the ER@Optimal drops 36.7 on the negated set. The results demonstrate that both models show decreased performance when evaluating negated hypotheses, indicating that the task becomes more challenging when models must identify evidence that refutes rather than supports claim. However, the decrease in performance small for GPT-4o.
6 Related Work
Hypothesis Generation Recent works explore using LLMs to generate scientific hypotheses (O'rre et al. 2024 Tong et al., 2023 Park et al., 2024 Baek al., 2024 Abdel- et al. 2024). Qi et al. 2023) fine-tune LLMs on biomedical literature that pairs background knowledge with corresponding hypotheses, and then use the LLMs generate hypotheses when prompted with background knowledge.
Evidence Retrieval Claim-based retrieval [Chen et al. 2023) retrieve evidence by breaking down complex claim into specific aspects and retrieving each aspect. On the other hand, our pipeline uses a novel approach, by decomposing summarized evidence from review papers into study aspects (instead of claims), which serves as ground-truth human domain experts knowledge that would guide our LLM-annotator to match sentences from research papers to these study aspects.
LLM in Biomedicine Researchers have shown strong performance of LLMs BioNLP tasks, including relation extraction, question answering, document classification, name entity recognition, and summarization (Luo et al. 2022 Chen et al.) 2024 Monajatipoor et al.
2024 Jahan et al. 2023 2024 Munnangi et al. (0024). LLMs also being used to extract
specific information from report, e.g. Interventions, Outcomes, and Findings by Wadhwal
lt al. (2023).
Conclusion
We introduced EvidenceBench and EvidenceeBench-100k, benchmark for retrieving evi- dence for scientific hypotheses from biomedical literature. EvidenceBench was constructed using an automated, scalable pipeline that transforms expert-written summaries into fine- grained annotations linked to specific sentences in research papers.
10

---

Published as a conference paper at COLM 2025
References
Abbi Abdel-Rehim, Hector Zenil, Oghenejokpeme Orhobor, Marie Fisher, Ross J. Collins, Elizabeth Bourne, Gareth W. Fearnley, Emma Tate, Holly X. Smith, Larisa N. Soldatova, and Ross D. King. Scientific hypothesis generation by large language model: Laboratory validation in breast cancer treatment, 2024.
Meta AI. Llama 3. https://llame comellammm 2024. Accessed: 2024-06-05.
Anthropic. Introducing the next generation of claude, 2024. URL WtLLttts:///ww.athhrrgg
comm/news/claude-3-ff Accessed: 2024-05-22.
Anthropic. Introducing claude 4, 2025. URL http:://ww..attrrrppppccccc////s Accessed: 2025-06-01.
Jinheon Baek, Sujay Kumar Jauhar, Silviu Cucerzan, and Sung Ju Hwang. Researchagent: Iterative research idea generation over scientific literature with large language models, 2024.
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308. 145088 2023.
Jifan Chen, Grace Kim, Aniruddh Sriram, Greg Durrett, and Eunsol Choi. Complex claim verification with evidence retrieved the wild, 2023.
Qingyu Chen, Jingcheng Du, Yan Hu, Vipina Kuttichi Keloth, Xueqing Peng, Kalpana Raja, Rui Zhang, Zhiyong Lu, and Hua Xu. Large language models in biomedical natural benchmarks, baselines, and recommendations, 2024.
language processing:
Rita Conzález-Márquez, Luca Schmidt, Benjamin M. Schmidt, Philipp Berens, and Dmitry Kobak. The landscape of biomedical research. Patterns, 2024. ISSN 3666-3899. doi:
20.1016/jpattee.22241000009 URL http:tts///ooo.rr////// 1016// pwtter.tte..... 2024. 100968
Google. Introducing gemini 1.5, google's next-generrtinnnn ai model, 2024. URL https:
blog. hooggggooggee/ehhoooogyoo/ /wooggoooleeeeooo
http://ioo.o......ooe....rooonnn #sundar-note, Accessed: 2024-05-22.
Google. Gemini 2.5: Our most intelligent ai model, 2025. URL htttt::://blog.googl.
-ttpe//mmmnnoemmiiiimmmeoe-----o cessed: 2025-06-01.
Ac-
Margaret L Hoang, Chung-Hsin Chen, Pau-Chung Chen, Nicholas Roberts, Kathleen G Dickman, Byeong Hwa Yun, Robert Turesky, Yeong-Shiau Pu, Bert Vogelstein, Nickolas Papadopoulos, Arthur P Grollman, Kenneth W Kinzler, and Thomas A Rosenquist. Aris- tolochic acid in the etiology of renal cell carcinoma. Cancer Epidemiology, Biomarkers & Prevention, 260020:1600-1608 2016. ISSN 1538-7755. doi: htRps: hcbi. nlm. nih.
Israt Jahan, Md Tahmid Rahman Laskar, Chun Peng, and Jimmy Huang. Evaluation of ChatGPT on biomedical tasks: A zero-shot comparison with fine-tuned generative transformers. In Dina Demner-fushmant Sophia Ananiadou, and Kevin Cohen (eds.), The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks, pp. 326-336, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653 /20/2222.iiooop1..00 URL httpttttp//// httpp::/acllantholllgg
org.org////222 2023. http://222....ooo11111e. 1.30
Israt Jahan, Md Tahmid Rahman Laskar, Chun Peng, and Jimmy Xiangji Huang. A compre- hensive evaluation of large language models on benchmark biomedical text processing tasks. Computers in Biology and Medicine, 171:108189, 2024. ISSN 0010-4825. doi:
108189. URL https::////// swweecedireeccctt
11

---

Published as a conference paper at COLM 2025
Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Nv-embed: Improved techniques for training llms as generalist embedding models. arXiv preprint arXiv:2405.17425, 2024.
Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12, 2024.
Renqian Luo, Liai Sun, Yingce Xia, Tao Qin, Sheng Zhang, Hoifung Poon, and Tie- Yan Liu. BioGPT: generative pre-trained transformer for biomedical text generation and mining. Briefings in Bioinformatics, 23 09 2022. ISSN 1477-4054. doi: 10.1093/bib bbac409. URL /ttp::/:://d http:///..19//bbbbbbbb 093/bib/bbac409
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 11048-11064, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL htttps://allatthollla org/2202 emnlp-main... 759
Masoud Monajatipoor, Jiaxin Yang, Joel Stremmel, Melika Emami, Fazlolah Mohaghegh, Mozhdeh Rouhsedaghat, and Kai-Wei Chang. Lims in biomedicine: A study on clinical named entity recognition, 2024.
Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. Mteb: Massive text embedding benchmark. arXiv preprint arXiv:2210.07316. 2022.
Niklas Muennighoff, Hongjin Su, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, and Douwe Kiela. Generative representational instruction tuning. arXiv preprint arXiv:2402.09906, 2024.
Monica Munnangi, Sergey Feldman, Byron C Wallace, Silvio Amir, Tom Hope, and Aakanksha Naik. On-the-fly definition augmentation of lims for biomedical ner, 2024.
Thomas O'Brien, Joel Stremmel, Léo Pio-Lopez, Patrick McMillen, Cody Rasmussen-lvey, and Michael Levin. Machine learning for hypothesis generation in biology and medicine: exploring the latent space of neuroscience and developmental bioelectricity. Digital Discovery, 3:249-263, 2024.
doi: (D00099D3DD0085C. URL http::/dx.dee doi. org/10
1839/D3000@185G
OpenAL. Hello gpt-4o, 2024. URL https:://ppeai..com http://////////eloloottt-------/ h/ Accesseeeeeeeeeee 2024-05-22.
OpenAl. New embedding models and api updates, January 2024. URL https://opeaaa
Com/ htt//index/neee wewnde///wwwemeeddiieee enwe--eeeednng-dddlll...... and api apida-ap-uddtes// Accessed: 2024-05-19.
Yang Jeong Park, Daniel Kaplan, Zhichu Ren, Chia-Wei Hsu, Changhao Li, Haowei Xu, Sipei Li, and Ju Li. Can chatgpt be used to generate scientific hypotheses? Journal of Materiomics, 10(3):578-584, 2024.
Biqing Qi, Kaiyan Zhang, Haoxiang Li, Kai Tian, Sihang Zeng, Zhang-Ren Chen, and Bowen Zhou. Large language models are zero shot hypothesis proposers. In NeurlPS 2023 Workshop on Instruction Tuning and Instruction InstructionFollln Following, 2023.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: graduate-level google- proof q&a benchmark. In First Conference on Language Modeling, 2024. URL https: //openreview. // (tt.rummmmdii17784505500
Song Tong, Kai Mao, Zhen Huang, Yukun Zhao, and Kaiping Peng. Automating psycholog- ical hypothesis generation with ai: Large language models meet causal graph, Nov 2023.
URL if htt.ff.///prerrrtssssssaav//////
12

---

Published as a conference paper at COLM 2025
Voyage AL on mteb, May 2024. URL https://bblog.voyagees com/ 2024/05/05/ soyage-lare http:///wwwuuuuut.tttootoottootr
voyage-lange---nnstuut
Instruction-tuned
and rank
Accessed:
2024-05-19.
Somin Wadhwa, Jay DeYoung, Benjamin Nye, Silvio Amir, and Byron C Wallace. Jointly extracting interventions, outcomes, and findings from rct reports with Ilms. In Machine Learning for Healthcare Conference, pp. 754-771. PMLR, 2023.
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Ma- jumder, and Furu Wei. Text embeddings by weakly-superviee contrastive pre-training, 2022.
Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Improving text embeddings with large language models. arXiv preprint ar0iv:24010000:::000000 100368, 2023.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.077 076622 2022.
Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Khai Hao, Xu Han, Zhen Leng Thai, Shuo Wang, Zhiyuan Liu, et al. Vinfty bench: Extending long context evaluation beyond 100k tokens. arXiv preprint 2rXiv:2402.13718 137188 2024.
13

---

Published as a conference paper at COLM 2025
Appendix: Table of Contents
A. Dataset.
1. Dataset License and Code License...
15
2. Dataset Hosting, Accessibility and Maintenance
Dataset Coverage Analysis.. Analysis........................
Motivating Example....
15
Dataset Collection and Processing...
16
6. EvidenceBench Dataset Structure.
16
B. Annotation Guidelines...
17
Guidelines for Hypothesis Validation
17
2. Annotation Guidelines for Alignment of Study Aspects and Sentences
19
1. First Annotation Stage
T9
2. Second Annotation Stage
20
C. Automated Alignment Procedure
20
Study Aspect Decomposition..
D. Hypothesis Testing Results
E. Experiment Details
24
1. Default Prompt Template for Evidence Retrieval
24
2. Prompt Template for Results Evidence Retrieval @Optimal and @5
24
ICL Prompt for Evidence Retrieval @Optimal and @10
26
Section-by-Section Prompt for Evidence Retrieval @Optimal and @10
27
Regeneration Prompt.... comppt..........................
28
6. Instructions for Embedding Model
28
7. Standard Errors for Model Evaluations
29
8. Negation Prompt...
E. Model Sensitivity to Paraphrased Hypothesis..
30
G. Hypothesis Collection
H. Data Preprocessing.
Harvesting Evidence Summary
32
Further Processing Evidence Summary...
32
3. Identifying Suitable Evidence Summaries
4. Human Verification
32
I. Fine-tuning and Evaluation on EvidenceBench-100:
33
J. Qualitative Analysis for GPT-4o on the Original EvidenceBench.
33
K. Further Analyses Analyses........
14

---

Published as a conference paper at COLM 2025
A Dataset
A.1 Dataset License and Code License
The EvidenceBench dataset uses the following licenses:
Test set: Provided under CC-BY license.
Train set: Provided under CC-BY-NC-SA license.
Dev set: Provided under CC-BY-NC-SA license.
The EvidenceBenchh-10 dataset uses CC-BY license.
A copy of the full license can be found at http:://hhhhhooooot//eeeeeee///d Note that the test set has the most permissive license.
All code is released under the MIT License. The full license can be found at (https:/ http://wwwo//ddeeeeeeeeeeeeeeeee http:///nnnn////b//aaaaoaeaaaaaa http:///hhccocoommmiiiic/lAnn/.
A.2 Dataset Hosting, Accessibility and Maintenance
The EvidenceBench and EvidenceBBnch-1100 datasets
can be accessed at
A.3 Dataset Coverage Analysis
In this paper, we introduce EvidenceBench with 426 papers and FridenccBench-100k with 107,461 papers. Our two datasets are fully open-sourced under the CC-BY license and encompass a comprehensive range of biomedical topics. Specifically, we quantified topical diversity for the 100k papers using 2025 MeSH headings to demonstrate representativence The dataset covers all top-level MeSH branches (A-N, V, Z), confirming coverage of every major biomedical domain. With 17,505 distinct MeSH descriptors representing 56.5% of the entire 2025 vocabulary (30,956 terms), the breadth matches that of a full-year MEDLINE slice. The branch-level distribution shows Gini-Sin Sini-Simpoon diversity of 0.81 (where indicates perfect evenness), demonstrating that no single area dominates the corpus. Additionally, MeSH tree numbers reach a median depth of 5 and 90th-pereentile of 7, indicating the benchmark encompasses both fine-grained molecular and procedural topics alongside high-level concepts.
A.4 A Motivating Example
Aristolochic Acid (AA) is toxin that is naturally occurring in traditional Chinese herbal medicines and has been known to cause many types of cancer in animals and humans Hoang et al. (2016). However, 20 years ago, the causal relationship between AA and kidney cancer not yet confirmed. In this section, present example data instance related to AA.
Hypothesis:
Aristolochic Acid (AA) induced DNA mutation is causal for renal carcinoma (RCC).
Evidence Summary
Results from Hoang et al. 2016) showed that cumulative ingestion of more than 250 mg of AA increased the risk cCRCC with odds ratio (OR) of 1.25. A distinctive AA mutational signature was evident in 6/ 10 sequenced ccRCC exomes from AA-exposed patients. Among these tumors, VHL, the most frequently mutated gene, mutated in 7 out of 10 samples.
Study Aspect Decomposition:
1. Cumulative ingestion of more than 250 mg of AA increased the risk of ccRCC (OR, 1.25). [Sentences 9 and 69 are sources of information for Aspect 1].
15

---

Published as a conference paper at COLM 2025
2. A distinctive AA mutational signature was evident in six of the 10 sequenced ccRCC exomes from AA-exposed patients. |Sentence 163 is the source of information for Aspect 2]
3. The most frequently mutated driver gene was VHL. |Sentence 106 is the source of information for Aspect 3].
4. VHL mutated in 7 out of 10 tumors. [Sentence 106 is the source of information for Aspect 4]
Full Paper:
All 216 sentences from Hoang et al. (0116 are indexed, starting from to 215. For brevity, we will not reproduce the entire paper here. Figure shows sentences 9, 69, 106 from the full list of sentences are retrieved by model.
Selected Sentences from Full Paper:
Sentence 9: Cumulative ingestion of more than 250 mg of AA increased risk of ccRCC (OR, 1.25), and we detected dA-AL-I adducts in 76% of Taiwanese ccRCC patients.
Sentence 69: The results (Table 1) indicate an adjusted OR of 1.25 (1.004-1.547) for ccRCC persons consuming more than 250 mg AA during the period of 1997 to 2003.
Sentence 106: VHL was the most frequently mutated driver gene (7/10 tumors) in AA-exposed ccRCCs (Table 2).
Sentence 163: Whole-exome sequencing confirmed that the AA mutational signature was present in 6 of 10 ccRCC patients studied.
A.5 Dataset Collection and Processing
We use BioC API download biomedical papers which are available in the PMC database. Papers unavailable in PMC are manually downloaded. We use GROBID to parse papers from PDF format to XML format. We use Stanza to split paragraphs of text into sentences. We manually copied and pasted all required open-access review paper sections, and do not distribute any contents these review papers.
A.6 EvidenceBench and EvidenceBench-100k Datasets Structure
EvidenceBench uses a train, dev, test split. EvidenceBBech--000 uses train and test split. All datasets have the same structure. Each data instance (stored in a JSON) has the following features:
hypothesis: the biomedical hypothesis in string format.
paper_as watas_candddate__pooo an ordered tuple of strings. Each string is one sentence from the paper. This serves as the candidate pool for all of the evidence retrieval tasks.
aspect_list_ids: a list of strings. Each string is an id for a study aspect.
results_ www.accpplssstttee ids: list of strings. Each string an id for an aspect related to the study's results.
aspect2sentence. indi ces: mapping (i.e. dictionary) from each aspect to all sentence indices that are sources of information for that aspect.
poatence_indde22speettss mapping (i.e. dictionary) from each sentence index to all aspects that this sentence is a source of information for.
dictionary that contains informa- tion for evaluating model's 's performance on the task Evidence Retrieval @Optimal.
optimal: A positive integer, which is the smallest number of sentences needed to cover all study aspects.
16

---

Published as a conference paper at COLM 2025
one_selection_or ofw.tesssneees: list of sentence indices, containing the smallest number of sentences needed to cover all aspects. Note, there are potentially other lists of sentences of the same size which cover all aspects. the list of aspects that are covered, which is all aspects in
this case.
wwidence_retrreeal_aee 10_ evaluation: dictionary that contains information for evaluating model's performance on the task Evidence Retrieval @10.
one nre_selection_o offiecttoonoo_sseeenneesss sentences: a list of 10 sentence indices. This list covers the maximum number of aspects which can be covered by 10 sentences.
doveredaappccee the list of aspects that covered, which may be fewer than all aspects.
dictionary that contains the information for evaluating a model's performance on the task Results Evidence Retrieval @Optimal. The structure is similar to
results_evidence. retrieval_aatetteeeeeeeeeeeeeeee aatriievalat.oppimaell aet__ootimal_evlluttttii evaluatioo:
A dictionary that con- tains the necessary information for evaluating a model's performance on the task Results Evidence Retrieval @5. The structure is similar to evidence_retrieval..tt..0 10__allt_uvvltt.oonn
sentence_types lerees_inn_____ hti___addddat_eoeoo
tuple of strings. Each string is sentence type. There are three possible sentence types: aaction_name, abstract, and nor- mal mal_ppaagraph. For example, if the third string is 'abstract', that means the third sentence comes from the abstract.
paper_id: the id of the paper used as the candidate pool.
B Annotation Guidelines
B.1 Guidelines for Hypothesis Validation
Below, we show the annotation guidelines for evaluating the hypotheses extracted from the review papers. These guidelines were co-designed and approved by medical doctor who did not see the 50 hypotheses which were evaluated.
Overall:
IARC is WHO organization that invites field experts to write review about the potential carcinogenicity of certain where they survey many relevant papers.
A review is typically organized into the following sections:
Exposure Data (e.g., how humans and animals come into contact with the substance).
Animal Study.
Human Study.
Mechanistic Evidence (e.g., the mechanism for carcinogenicity).
Others.
For each relevant paper, the field experts will extract certain information from the paper, for specific purpose, which does not have to align with the original goal of the paper.
Annotation Task:
17

---

Published as a conference paper at COLM 2025
Each task is in docx file. In each docx file, you will see:
A hypothesis.
A paragraph of extracted information from paper.
reference page (For reference only).
Potentially more context for the hypothesis (i.e., potential
connection between the hypothesis and the extracted information from paper.
The review that contains the extracted information from the paper.
Link for the paper.
You have two tasks.
Determine if the hypothesis is a reasonable hypothesis, given your understanding of the hypothesis and your external knowledge and experience.
The hypothesis might be about the carcinogenicity of substance (for human or animal), or might be about how humans get exposed to substance, or might be about experimental procedure, or something else.
Determine if the hypothesis is a valid statement with scientific value, it could be false statement, but disproving it would have scientific value. In other words, you should not judge the accuracy of the hypothesis. You should only judge if the hypothesis contains scientific value.
Determine if the hypothesis looks like hypothesis, i.e., has the format of real hypothesis.
Make your judgment based only on the contents of the hypothesis, which is usually just one sentence. Your decision should not be influenced by the other task or other materials you see, though for better comprehension, you refer to the links the reference page.
Record your decision or No), and leave any optional comment if you want. If you think the answer is not binary, then you do not have to write yes or no, but you have to give an explanation.
Determine if the extracted information from the paper contains evidence that can potentially help support or refute the hypothesis.
Answer Yes or No, followed by a brief explanation. One or two sentences. If you think the answer is not binary, then you do not have to write yes or no, but you have to give an explanation.
Notes:
You have to pledge the following conditions are during annotation for each task packet.
No consulting with AI and LLM.
words or concepts that you are not familiar with and believe are important for comprehension, search for them and understand their meaning.
If you do not understand the hypothesis or the extracted information from the paper, you should read the IARC review better understand the hypothesis and read the full paper to better understand the concepts mentioned in the hypothesis and extracted information from the paper.
You not required to read the whole paper nor the full IARC review, just to the point when you believe you understand the hypothesis and extracted information from the paper well enough.
18

---

Published as a conference paper at COLM 2025
B.2 Annotation Guidelines for Alignment of Study Aspects and Sentences
B.2.1 First Annotation Stage
This section shows the annotation guidelines for the first stage of aspect-sentence alignment. In this stage, each annotator had to independently annotate 50 papers.
You have total of 50 annotation task packets. Each task packet is docx. file that contains the following information.
An aspect (one piece of important information/detail).
The context for the Aspect (summary or a collection of extracted details from a paper).
The URL for the paper (pmc or pubmed link).
The list of indexed text elements of the paper (a text element could be sentence or section title).
You have to pledge the following conditions are met during annotation for each task packet.
No consulting with AI or LLM.
For words you are not familiar with and believe are important for comprehension, conduct search and understand its meaning.
Click the paper URL and find full contents either in HTML, XML, PDF format, and read through it from start to finish, at least once.
For every text element in the list, you must look at it and read it at least once.
You cannot talk to other annotators about anything related to your task, including progress and insights.
You have to take a mandatory 5-minute break after every hour of performing annotation.
You cannot exceed hours of annotation per day.
Below is the recommended procedure for annotating each packet.
Read and understand the aspect, and the context of the aspect.
Decide what details in the aspect count as information, and what details count as context. At least one detail needs to be identified as information, but an aspect can do without context if it is self-contained and very clear.
You can decide what information is, but geographical, temporal, and numerical data are all information.
Typically, context is recurring and ubiqui information throughout the paper.
For each text element, you decide if a very significant amount of information identified in the aspect is also explicitly present in the text element. Note, information has to be explicitly present, it cannot be from inference or allusion. Acronyms, abbreviations, and different presentation formats of the same information (e.g., rounding of numbers) are acceptable, as long as it is clear you.
Even you find significant information overlap, you have make sure the sentence is in the same context as the aspect.
Same context typically refers to the same study experiment.
Check if the sentence refers to the same experiment the aspect, since different experiments could be in one paper.
19

---

Published as a conference paper at COLM 2025
Do not do complicated mental inference. Once you have clear understanding of the aspect and sentence, do not try to invent a spurious connection between sentence and aspect.
Specifically, do not do complex computations of numbers.
B.2.2 Second Annotation Stage
Below we show the annotation guidelines for the second stage of the aspect-sentence alignment. this stage, two annotators from the same team come consensus on any disagreements from the first stage.
Go through task 0-49.
Resolve your difference, check if you made a mistake, or you missed something. If you made a conceptual error (e.g. you failed to understand some terminology), you may have to go through the paper again quickly.
For sentences that you cannot resolve your difference after discussion, i.e., one person says yes, and the other person says no, you should include them as well.
C Automated Alignment Procedure
This section describes prompt optimization for the LLM alignment of study aspects and sentences.
Prompt optimization was performed with GPT4-0125 on an independent development set of 37 (aspect, paper) pairs, i.e., 37 tasks. There was no overlap with the papers labeled by the two teams of human annotators.
paper (Including tables, figure captions, sentences, others). Text element followed by index, in this format, Index: text element [End of text element index..
given
Focus solely the statement, through each text element the list, determine If itseff alone contains ALL Information in statement. In other words, you must determine If the element could the source of information for the statement. For example, the statement contains numerical, numerical, geographical, and timeidate details mentioned the
geographical, and timerdate detalis, suitable text element must include statement, subject to different formats, such as different rounding, spalling, acronyms.
Statement: (aspect)
(context)
Peper's list of text elements: (list_of_ fext__leemntts
Make you through the entire list text elements, matter long. you find the statement cannot be found in any text element. For each text element you choose because you think It can be the source of information for the statement, explain why. you
explicityy explain why information
satisfactory explanation,
Output your explanation
and reasoning after the keyword "REASON."
Once you finalize your selection, should include the numerical Index of each text element you choose. Finally, output chosen numerical Index python [Index1, index2, ...] after keyword "DECISION."
Figure 4: Prompt for aligning a sliding window of consecutive text elements with study aspect.
In order to reduce the frequency of GPT-4 forgetting information from the papers, we use sliding window (window-length windows. GPT-4 sees a sliding window of sentences and annotates each sentence according to whether it is source of information for the aspect.
10 sentences) with overlap of 5 text elements across
20

---

Published as a conference paper at COLM 2025
Since the sliding windows are overlapping, each text element (except for the first 5) is considered twice by GPT-4. A sentence is labeled as positive is selected in either sliding window.
Figure Isoows the final prompt template used for aligning text elements with aspects. Each template uses one aspect, 10 text elements in sliding window, and the context around the one aspect (i.e., the evidence summary of the paper).
21

---

Published as a conference paper at COLM 2025
C.1
Study Aspect Decomposition
There are two steps in aspect decomposition.
The first step is decomposing evidence summary into list of study aspects where each aspect represents a single piece of information. The granularity of the decomposition is determined by the following rule:
Each decomposed study aspect must be able to align with at least one sentence from the paper. If a study aspect contains so much information that no one sentence can cover significant portion of these details, then this study aspect is considered too coarse-grained and must be further decomposed.
See Figure 5 for the prompt template that achieves the first step of aspect decomposition.
You are given expert statement about a biomedical article (paper_kg) Hd). Your job to decompose each sentence into pieces of information.
You are also given sentences the original paper reference. should determine level specifictt for the extracted information according each information
the following criteria: decomposing the statement, the original paper: you need more than one sentence to cover the information point, you must decompase that information point further.
covered by
sentence from
important decomposition:
Each extracted information should
specific aspect about the statement.
Do not delete any information from the expert statement.
Only output the extracted information.
Here expert statement:
(summary)
the original paper:
(candidate_pon
Figure 5: Step 1 of decomposing a evidence summary into aspects, using the granularity condition.
The second step is checking if the decomposed list of study aspects only contains informa- tion from the evidence summary, and if no other paper-specific information leaked into the decomposed list. See Figure 6 for the prompt template. Any datapoint whose decomposed list of aspects did not pass the second step verification is filtered out. Fewer than 10% of datapoints are filtered at this step. Empirically we noticed those filtered datapoints have evidence summaries that are not self-contained. Therefore, we did not attempt to recover these datapoints.
Given statement, and decomposed list aspects that statement. Your task determine the aspects the list contains information, details terminology not explicily and directly written in the statement. Note, aboreviations and typos are allowed. If you found any aspect contains information (any information not the statement, only contain Information from the statement, output True after the keyword "Judgment""
after the keyward "Judgment", otherwise, you determine
(summary)
Decomposed list of Aspect:
(indexed_aspect))
Figure 6: Step 2 of decomposing a evidence summary into aspects. This step confirms that aspects only contain information from the evidence summary.
22

---

Published as a conference paper at COLM 2025
D Hypothesis Testing Results
During hypothesis testing for automatic alignment annotation, although only 50 papers were annotated, each paper contains roughly 160 (sentence, aspect) pairs, totaling 8,111 pairs. In the paper, correlation and agreement metrics are calculated based on these 8,111 pairs and not on the 50 papers. Because annotations for sentences in the same paper are partially correlated, the effective sample size therefore lies between 50 and 8,111. In order to account for this correlation structure, we can use hierarchical bootstrap sampling (first sampling papers, then sampling sentences within papers) to compute confidence intervals and p-values. We provide results for GPT-4o and GPTT-o-mini on standard bootstrap sampling and hierarchical bootstrap sampling in Tables 7
8
9
10
Metrics
Human & Human Average Human GPT Average
p-value
Exact Accuracy
98.8 + 0.3
98.7 0,2
0.21
F1 Binary
66.0 1 6.5
64.6 5.6
0.64
Cohen's K
65.4 1 6.6
63.9 1 5.6
0.63
Spearman's p
65.4 + 6.6
64.0 5.6
0.65 /
Table 7: hypothesis testing result for GPT-4o on 8,111 pairs using standard bootstrap sampling
Metrics
Human & Human Average Human GPT Average
p-value
Exact Accuracy
98.7 + 0.5
98.6 0.5
0.46
F1 Binary
66.5 1 9.9
65.8 1 8.8
0.91 /
Cohen's
65.8 1 10.1
65.1 9.0
0.90
Spearman's
65.9 + 10.0
65.3 + 8.9
0.91 /
Table 8: Hypothesis testing result for GPT--4o using hierarchical bootstrap sampling.
Metrics
Human & Human Average Human & GPT Average p-value
Exact Accuracy
98.7 + 0.5
98.6 t 0.5
0.134 /
F1 Binary
66.5 + 12.3
64.0 9,9
0.362 /
Cohen's
65.8 + 12.6
63.3 + 9.9
0.356 /
Spearman's p
65.9 + 12.6
63.4 + 9.9
0.360 /
Table 9: Hypothesis testing result for GPT-4o-mini on 8,111 pairs using standard bootstrap sampling.
Metrics
Human & Human Average Human & GPT Average
p-value
Exact Accuracy Binary Cohen's K
98.7 + 0.6
98.6 + 0.6
0.339 /
66.5 + 9.9
64.0 + 7.4
0.541
65.8 +'2 10.1
63.3 + 7.6
0.536 /
Spearman's p
65.9 + 10.0
63.4 7.6
0.533 /
Table 10: Hypothesis testing result for GPT-4o-mini using hierarchical bootstrap sampling.
23

---

Published as a conference paper at COLM 2025
E Experiment Details
There are several prompt templates used for experimental evaluation, which are variations on default template.
E.1 Default Prompt Template for Evidence Retrieval
The default prompt template asks an LLM to retrieve no more than K sentences for the Evidence Retrieval tasks. Figure shows the default prompt template for ER @Optimal or ER @K.
hypothesis, biomedical paper in experiment in the paper that provides evidence relevant the hypothesis, You must explain how this experiment would provide evidence relevant the hypothesis, Determine what details (l.. text elements) about the experiment would jointly provide the most effective evidence to support or refute the hypothesis, Your task find NO MORE than (number_cm of aallowwww text elements) text elements. Since are allowed find limited amount text elements, you must only select text elements that together would cover the most amount of details of the experiment that you belleve are relevant the hypothests.
(Including sentences (sub)section tites), Identify
Hypothesis:
(hypothesis)
Peper:
(Indexed__t _istt_tttttt of text____tt elements)
effective evidence hypothesis. Output your reasoning first, and based on your reasoning, make your final selections of text elements. You should only keep the most effective details. Make you not effective (number__o of lll.llowed_ees text tet_teeex__eeeeeee elements) elements from that jointly would provide the best evidence relevant to the hypothesis.
Provide explanation of details you identified from experiment jointly provide
to
(number_od w_l_allowed_eee text elements) text elements. you do, you choose
Finally, output the list of indices of these text elements in a Python list [index1, index2, ... after the keyword "DECISION."
Figure Default prompt template for evaluating LLMs on tasks Evidence Retrieval @Opti- mal and Evidence Retrieval @10.
E.2 Prompt Template for Results Evidence Retrieval @Optimal or @5
(including sentences (sub)section tites), identily experimental results and analyses that provide evidence which relevant the hypothesis. Do not include methods background. You must explain how results analyses provide evidence for against hypothesis. Your task (number__g of L_aallowed__rg text_ elements) text elements. Since you only allowed find a limited amount of text elements, you must determine and select representative text elements that would jointly provide the most effective evidence to support or refute the hypothesis.
hypothesis, and
paper
an
As a demonstration, you will first see a sample hypothesis, and a sample list of representative text elements that are about experimental results and analyses, together the most of details relevant to the hypothesis.
Sample Hypothesis A, you no
sentences)
Sample List sentences]:
[..]
Now will see the actual hypothesis, and section of the blomedical paper. Recall, your task to find NO MORE than (number__o of awllowed_tteee text elements) text elements.
Hypothesis: (hypothesis)
paper (indexed_ list_ of text elements)
Note, you must only consider text elements related experimental results and analyses. Provide explanation of how the details you identilldd from the experimental
analyses would jointyy provide
evidence relevant to hypothesis. Output your reasoning
based reasoning, make your final selections text elements. Make elements. you do, you have to choose the most effective (number_ _o___o______ wlllowl_w_eeee wwwpeeeddeeddeeeeeeeeeeeeeeeeeee elements) elements from it that jointly would provide the best evidence relevant the hypothesis.
find more than (number_co of fllllllwwdd__.. text, elements) text
Finally, output the list indioas these elements python list [index1, Index2, ...] keyword "DECISION."
Figure 8: Step 1: Prompt template for evaluating LLMSs on tasks Results Evidence Retrieval @Optimal and Evidence Retrieval @10. Here, the number of allowed text elements denotes Optimal or 10. This prompt uses a mixture of one-shot ICL and Section-by-Section
24

---

Published as a conference paper at COLM 2025
Step 1 of Prompt template with one-shot ICL and Section-by-Section
Figure showws the prompt template for any tasks that only focus on retrieving sentences related to experiment results analyses based experiment outcomes. Note, this prompt uses a mixture two strategies: one-shot ICL umn-contex--larning)) and section-by-section processing. These two strategies are proven effective in the other two tasks, Evidence Retrieval @Optimal and @K. Due to budget limitations, we can only provide this mixture strategy, which proves to be the best strategy on the training set. Note, for fairness, for each section, we can only instruct the LLM to retrieve no more than K sentences, even though paper could have 10 sections. Consequently, the total number of retrieved sentences for all sections combined sometimes exceed to maximally allowed number of sentences K. Therefore, we have the second step of processing.
Step 2 of Prompt template with one-shot ICL and Section-by-Section:
In step 2, we show the LLM all its retrieved sentences and ask it to select the top K sentences. See Figure 9 for its prompt template.
Given hypothesis, and an indexed list of text elements (Including sentences and (sub)section titles retrieved by you from biomedical paper, identify experimental
analyses provide Find NO MORE than (number_cd of od_allowed_ text, elements)))) text elements that would jointly provide the most effective evidence to support refute the hypothesis,
hypothesis.
background.
Hypothesis:
(hypothesis)
of text elements:
(Indexed_ Jistt_oo of text eleemets))
Note, you must only consider text elements related to experimental results and analyses. Provide explanation of how the details you identified from the analyses jointly provide based on reasoning, make your final selections of text elements, Make elements, you you to choose the most effective (number, of aallowed text elements) elements from it jointly would provide the evidence relevant the hypothesis.
experimental
hypothesis. Output your reasoning evidence find more than (number_ of all.llowwe___cm textttt elements) text
Finally, the list indices these text elements
list [index1, Index2. ...] after the keyword "DECISION."
Figure 9: Step 2: Prompt template for evaluating LLMs on tasks Results Evidence Retrieval @Optimal and Evidence Retrieval @10.
25

---

Published as a conference paper at COLM 2025
E.3 ICL Prompt for Evidence Retrieval @Optimal and @10
We randomly selected 8 pairs of examples from the development set, which was completely disjointed from the test set. We experimented with different versions of in-context learning. In one attempt, we gave the full paper for each example pair (i.e,, sample hypothesis, sample full paper, sample optimal number of or 10 sentences that cover the most amount of study aspects.). However, no LLM improved on the training set using N-shot with full paper, even when =1 or 2. Therefore, we decided to not use the full paper. Instead, for each example pair, we give only the sample hypothesis and the sample list of sentences that cover the maximum amount of aspects (size = Optimal or 10). See Figure 10 for its prompt template.
Given hypothesis, and biomedical in the format of an indexed list of text elements (Including sentences and (sub)section tites), identify the experiment in the paper that provides evidence relevant the hypothesis, You must explain how experiment would provide evidence relevant the hypothesis. Determine what detals text elements) about the experiment would jointly provide hypothesis, Your task find MORE than (number, of allowed, text, elements) text elements. Since you are only allowed to find liittd amount text elements, you the hypothesis.
effective evidence support refute the
select elements that together would cover the most amount details the experiment that you belleve relevant
As a demonstration, you will first see pairs of examples, each example will have sample hypothesis, and a sample list of text elements that are representative together most
experiment.
Sample Hypothesis A, (if you have to find no more than m sentences)
Sample List sentences):
[..]
Sample Hypothesis you find sentences)
Sample sentences):
[..]
Sample Hypothesis H, you have find no more than p sentences)
Sample List (p sentences]:
[..]
hypothesis, paper. your
MORE (number_ www.lllwwwdeee.. teapedd___teeeeeeeeoeeeeeeeeet elements)
elements.
Hypothesis: (hypothesis)
Paper:
(indexed Jist of text elements)
Provide an explanation of how the details you identifled from the experiment would jointly provide the most effective set of evidence relevant to the hypothesis. your reesoning first, effective details, Make sure you do not find more than (number__d of slallowwed_de.. text, elements) text elements, If you do, you have to choose the most (number_
your reasoning, your final
text_ett_eeee_etee elements)
that jointly
hypothesis.
Finally, output the indioes these elements in python list [index1, Index2, ...] after the keyword "DECISION."
Figure 10: 8-shot In-context Learning Prompt template for evaluating LLMs tasks Evi- dence Retrieval @Optimal and Evidence Retrieval @10.
26

---

Published as a conference paper at COLM 2025
E.4 Section-by-Section Prompt for Evidence Retrieval @Optimal and @10
Section-by-section is strategy to counter the long-context difficulty posed by Evi- denceBench. Instead of processing the full paper at once (typically consisting of more than 5000 tokens), each paper is divided into its naturally defined sections, i.e. introduction, methodology, results, etc. Each time, an LLM only retrieves sentences from one single section. Note, for fairness, for each section, we can only instruct the LLM to retrieve no more than K sentences, even though paper could have 10 sections. Consequently, the total number of retrieved sentences for all sections combined sometimes exceeds the maximally allowed number of sentences ask the model to select the best K sentences from all sentences retrieved from all sections. See Figure for the prompt template of the first step. See Figure for the prompt template that asks the LLM to choose the best K sentences from all its retrieved sentences from all sections.
Therefore, we have the second step of processing where we
(includingg sentences (sub)section lifee). Identiy the experiment the section that provides evidence relevant to the hypothesis. You must explain how this experiment would provide evidence relevant the hypothesis. Determine what details (l.e. text elements) about the experiment would jointly provide the most effective evidence support refute the hypothesis, Your task find NO MORE than (number__g of __aallowe___t text, elements) text elements, Since you only allowed find a limited amount of text elements, you must only select text elements that together would cover most amount details of the experiment that you belleve relevant hypothesis.
hypothesis,
paper
Hypothesis:
(hypothesis)
Paper:
(indexed_ Jist of text elements)
Provide explanation of how the details identified from the experiment would jointly provide hypothesis. Output your ressoning first, and based on your reasoning, make your final selections of text elements. You should only keep the most effective details. Make you do not find more than (number__d L__allowed_tes text elements) text elements, you you have choose the most effective (number__m of all.lllwwdddee. text elements) elements from that jointly would provide the best evidence relevant to the hypothesis.
effective of evidence relevant to
Finally, output the list of indioes these text elements in python list [index1, Index2, ...] after the keyword "DECISION""
Figure 11: Step Section-by-section Prompt template for evaluating LLMs on tasks Evi- dence Retrieval @Optimal and Evidence Retrieval @10.
hypothesis, indexed elements (incuuding sentences (subjsection tites) retrieved by you blomedical paper, MORE than (number__g of allllaldddeeeee text elements) text elements that would jointly provide details an experiment support refute the hypothesis.
Hypothesis:
(hypothesis)
List of text elements:
(indexed_ Jist of, text elements)
Explain how the text elements would collectvely describe an experiment to or refute the hypothesis. Output your reasoning first, after the keyword "REASON"", make your
sure you do more (number, aallloodd elements) text elements, do, have choose most effective (number_od wil_sllowd__eee text_eett_eee [ext_eeeeents) elements from that would most effectively support refute the hypothesis.
Finally, output the list of indioes of these text elements in python list [index1, Indax2, ...] after the key word "DECISION."
Figure 12: Step Section-by-section Prompt template for evaluating LLMs on tasks Evi- dence Retrieval @Optimal and Evidence Retrieval @10. The LLM is asked to choose the best K number of sentences.
27

---

Published as a conference paper at COLM 2025
E.5 Regeneration Prompt
an LLM retrieved more than allowed despite explicit instructions, will be sent a regener- ation prompt, with the following format in Figure 13
(Previous conversation)
(tole:: "content" output (number__m than (number_sg of __alllwwwd__.mm text elements) number of text elements. You must choose the best (number__m of of_allowed__te text elements) text elements in previous output.)
elements)
requirement
Figure 13: Regeneration prompt template if LLM exceeds the maximally allowed number of text elements.
E.6 Instructions for Embedding Model
For Evidence Retrieval @Optimal and @10, for embedding models, there are two strategies, with instruction or without instruction. The instruction is:
"From biomedical experiment, find important and representative details that would form the most effective set of evidence relevant the hypothesis"
Recall, "with instruction strategy" means concatenating the instruction with the hypothesis and creating an embedding for the concatenated text as new hypothesis vector. Each text element in the candidate pool (i.e. the paper) is still independently embedded as its own vector.
See Table 11 for embedding models' performance with instruction and without instruction, on the tasks of Evidence Retrieval @Optimal and @10.
Table 11: Embedding Models Comparison. Standard Error calculated by bootstrapping.
No Instruction
Instruction
Model
ER @ optimal
ER @
ER @ optimal
ER @ 10
BM25
16.5 + 1.1
34.4 1.7
OpenAl
14
42.2 1.7
+ 1.4
41.3 1.7
VoyageAT
21.6 + 1.3
42.0 1.8
22.7
+ 1.3
41.9
1.8
GritL
21.5
+ 1.2
39.7 + 1.7
27.0
+ 1.3
46.4
+ 1.7
F5-Mistral
22.7 1.4
41.9 + 1.8
21.9 + 1.3
40.8 + 1.8
For tasks such Results Evidence Retrieval @Optimal and @10, embedding models will take task-specific instruction. Instruction has the following format:
From biomedical paper, find important and representative details about experiment outcomes, results and analyses that would form the most effective set of evidence relevant the hypothesis.
Note, in the instruction, we have concisely and explicitly informed the embedding model that the embedding of the hypothesis should only have high cosine similarity with text elements related to results or analyses. Since we are not changing the embedding for the results and analyses related text elements, we only change the embedding for the hypothesis to fit this purpose. This is an emergent and specially fine-tuned ability of some of the newer and more powerful embedding models, such as GritLM. See Table 12 lower right quadrant for the performance of embedding models with instruction on tasks such as Results Evidence Retrieval (ER) @Optimal and @5.
28

---

Published as a conference paper at COLM 2025
E.7 Standard Errors for Model Evaluations
In this section, we reproduce the main results from the paper, showing standard errors for all estimates.
In table 12 we show overall results for model performance on the 4 tasks. For Evidence Retrieval Tasks, three strategies are considered: default, in-context learning, and section-by- section for each LLM. Default refers to Figure prompt. ICL refers to Figure [0, Section- by-Section refers to Figure 11 and 12 For each model, the strategy that achieved the best performance is selected, and that result t is reported as the model's performance. For example, for the task @optimal, gpt-4o achieves the best aspect recall using ICL, whereas for the task ER @10, gpt-4o achieves the best aspect recall using eeciionby-secction Note, for embedding models, there are two strategies: with instruction or without instruction. The best performance for each embedding model is recorded.
Note, for the two tasks Results Evidence Retrieval (ER) @Optimal and @5, only one strategy is considered for LLM, which is one-shot ICL with section-by-section processing, Figure 8 and Figure
For Results Evidence Retrieval (ER) tasks, embedding models must only have strategy, Le. the strategy with instruction. Since the task is about retrieving sentences related to results and analyses, embedding model must understand this constraint through its instructions.
Table 12: Aspect Recall for the full retrieval task (top) and the result retrieval task (bottom). For each model, the highest number is reported multiple strategies are used. Standard Error calculated by bootstrapping.
Max Random|
GPT-4o
Claude3-Opus Gemini-1.5 LLaMA3-700| Emb
Emb VoyageA1 GritLM
E3-Mistral
ER 6 optimal
100.0
9.6
51.4 + 1.4
47.6 + 1.5
48.3 + 1.4
46.7 +.4
25.1++14
22.7 +
1.3 + 1,3 22.7+ 1.4 41.9
99.3
22.3
71.6
664 166
65.4 +
65.4 1.6
42.2 1.7
42.0
46.4
Result ER @ optimal
100.0
4.4
52.64 2.1
51.74 2.1
46.74 2.0
46.2 2.2
19.1
18.3
1.8 18.9
1.7 19.3 1.8 336 2.2
Result ER
99.8
11.4
70.8=
1.9
68.74
65.4% 2.0
63.74
3.
22
Table 13: Comparison of prompting strategies for LLMs. Baseline refers to Figure prompt. prompt. ICL refers Figure 10 Sec-by-Sec refers to Figure and 12 Standard errors are calculated by bootstrapping.
Baseline
ICL
Sec-by-Sec
ER @ optimal
ER@
ER @
ER@
ER @ optimal
ER@
Model
10
optimal
10
GPT-4o
48.1 1.5
69.6 1.5
51.4 + 1.4 68.7 + 1.6
50.9 1.4 71.6
1.5
Claude3-opus
41.1 + 1.6
53.6
1.6
38.3 + 1.4
55.4
+ 1.7
47.6 + 1.5
66.4
+ 1.6
Gemini-1.5
42.7 + 1.5
63.0 1.6
43.2 + 1.5 62.4 + 1.7
48.3 + 1.4
65.4
1.6
LLaMA3-70b
46.7
1.4 65.4 + 1.6
Table shows model performance for the three prompting strategies.
We observe that all standard errors are less than 2%, indicating we have sufficiently large test set size to effectively distinguish different models and prompting strategies.
29

---

Published as a conference paper at COLM 2025
E.8 Negation Prompt
We evaluate models on negated hypotheses using prompt in Figure 14 We give two example pairs of original hypothesis and negated hypothesis:
Original Hypothesis: Clucocorticoids cause cleft palate by inhibiting growth of the palatal shelves, preventing shelf contact and fusion.
Negated Hypothesis: Glucocorticoids prevent cleft palate by promoting growth of the palatal shelves, facilitating shelf contact and fusion.
The mortality rate from bladder cancer higher among painters than in the general population.
Original Hypothesis:
Negated Hypothesis: The mortality rate from bladder cancer is lower among painters than in the general population.
Given tollowing blomedical hypothesis, rewrtte conweys oppostla diaim, statement would retuted the same avidence that originally supporied the hypothesis. Output the rewnitton hypothesis after the keyword "Hyaathesis:"
Original Hypothesis: (hypathesis)
Figure 14: The prompt to negate hypothesis
Model Sensitivity to Paraphrased Hypothesis
We test models' sensitivity to different paraphrased versions of the hypothesis. We use variety of LLMs to paraphrase a hypothesis. Note, paraphrase would still keep the scientific terminology to make sure it is still the same hypothesis. As shown in Figure 14 GPT-4o and the two embedding models are less sensitive to paraphrased hypotheses, while Claude3-Opus is more sensitive to them.
Table 14: Models evaluated under paraphrased versions of the hypothesis. All experiments are under task Results ER @ optimal.
Model
Original Hypothesis
GPT-4o Paraphrased
Claude Paraphrased
Llama3-70B Paraphrased
Llama3-8B Paraphrased
GPT-40
52.6
49.9
51.7
50.8
50.3
Claude3-Opus
51.7
44.6
41.9
43.3
43.9
GritLM
18.9
22.8
19.4
20.8
19.1
OpenAl Emb
19.1
19.1
18.1
19.7
18.3
30

---

Published as a conference paper at COLM 2025
G Hypothesis Collection
There are two stess for collecting hypotheses from the review papers.
Step 1:
See Figure 15 for the prompt template. Note that Claude3-Opus is used for this step. The model is instructed to answer, "What is the motivation for the expert reviewer extract these pieces of information from the paper and summarize them?".
provided excerpts from blomedical monograph. Some these excerpts context. also highlighted excerpt, which was writtnn by domain experts after they read a specific paper.
The highlighted text describes specific pleces of evidence that were extracted from the paper. They may describe experimental background, setup, procedure, methodology, results, analyses.
The domain experts extracted these specific pieces of evidence from the paper because they were trying provide evidence for certain hypothesis. Your task identity hypothesis.
performing you should think about the supported by these pieces of evidence? The hypothesis should be specifically tallared for these places of evidence from the highlighted text.
the domain expert extracted specific
evidence. Which hypathesis best
The hypothesis should not include superficial details from the highlighted excerpt, le. it should not be a mere reformulation of the evidence. The surronding the highlighted excerpt, should write the hypothesis mainly using terminoloyy, phrases and sentences from the surrounding context. Do not use wording from the highlighted excerpt hypothesis should discuss limitations the study.
hypothesis should be
semantically
surrounding
You
"Excerpts from biomedical monograph"" (context)
"'The Highlighted Excerpt""
(paper_ summary)
hypothesis by laking account above requirements. reminder, hypothesis cannot use details from the highlighted excerpt, the hypothesis must arise from the surrounding context, and the hypothesis must be fully supported by the entirety highlighted excerpt. hypothesis not discuss limitations study.
Think by step writing
Finally, output the hypothesis after keyword "HYPOTHESIS""
Figure 15: First step of hypothesis collection from review papers
Step 2:
In order to make sure the hypotheses have the correct format and do not have any sum- marized evidence, we perform second step where we trim the first-step output off any unnecessary details and specifications and only preserve the central biomedical question. See Figure for the prompt template.
Consider paragraph from biomedical monograph:
(context_ )
And Consider the following claim:
(longer_ (oonge_hhypotheile
Disregard any summarized ft into the central hypothesis, Importantly, if the claim contains speculative question, keep the part of the diaim that contains the speculative question. Think step by step. output your reasoning. After you done reasoning, you should have identified the central component the hypothesis (with respect to the paragraph from the blomedical manograph), and the speculative question, if the claim has such a question. Explicity state the central central component, transform a natural blomedical hypothesis.
Using paragraph as context,
hypothesis being
claim not
component, make speculative question definiive. Incorporate it The transformed hypothesis should be clear, concise, definiivee and unambiguous. Remember, the transformed hypothesis cannot have any detalled or summarized evidence from the claim, make sure check this!
After have transformed the hypothesis, remember to check that you DID NOT ADD information that not explicity stated the claim. Also remember check no summarized evidence or details from the claim in any form included in the hypothesis. Output the central hypothesis after the keyword "HYPOTHESIS."
Figure 16: Second step of hypothesis collection, trimming the output of step 1.
31

---

Published as a conference paper at COLM 2025
H Data Preprocessing
H.1 Harvesting Evidence Summary
We extract expert-written evidence summaries from review papers using an algorithmic procedure. Summaries that cite multiple papers are excluded. We also exclude any paper that is not licensed under Creative Commons or is not in the public domain. Finally, remove any paper that is cited twice in one section a review paper, ensuring that the extracted evidence summaries complete.
H.2 Further Processing Evidence Summary
The primary difficulty in ensuring the high quality of evidence summary extracted by the algorithm lies in the variability in the placement of the citation (e.g., (e.g. Hoong et al. (2016)). Since review papers sometimes have hundreds of papers cited in one section, each with evidence summary surrounding the citation, it challenging heuristically delineate the boundaries of the evidence summaries, and motivates the use of an LLM for this step. See Figure 17 for prompt template
Given paragraph
report.
Paragraph:(papp carrr_syurrrr
Determine sentence(s) describe experimental details observations
particular (paper_umt Must include the sentence contains the (paper_urfy, not DO NOT include summaries, comments, opinions limitaions about this specific paper. Limitatons are usually enclosed by [] and might include phrases such "the working group".
Output sentence(s) VERBATIM keyword
Figure 17: Prompte for extracting evidence summary for cited paper
H.3 Identifying Suitable evidence summaries
An evidence summary that is suitable for EvidenceBench contains experimental outcomes, results, analyses, or methodology. However, some evidence summaries only have high-level information about paper and do not have the desired level of specificity. We use an LLM to determine if evidence sunmary suits EvidenceBench. See Figure 18 for prompt template.
Statement:
(paper_ (ypper_summar)
The above statement is from the paper cited. The text content in the statement regarded as evidence from the paper. any part evidence or Do any part of these describe results, findings and conclusions? experimental or these evidence nelther describe experimental details nor describe results/lindings.
experimental designs, setup, procedures, methodologies?
results/findinns?
Simply and only output decision choosing of categories (experiment, results, both, nelther), chosen category after the keyword "DECISION":
Figure 18: Prompt template to determine if a evidence summary is suitable for Evi- denceBench. Here, "both" or "results" is acceptable.
H.4 Human Verification
We randomly sampled 50 extracted evidence summary using the entire harvesting procedure and LLM preprocessing. Only in 1 case did we notice where the extracted evidence summary includes a sentence that does not belong to this specific paper. This ensures the high quality and accuracy of the harvesting procedure and L.M preprocessing.
32

---

Published as a conference paper at COLM 2025
I Fine-tuning and Evaluation on BvidenceBench-100k
EvidenceBench-100k also split into 80k train set and a 20k test set. For cost reasons, we random sample 300 datapoints from the 20k test set. We evaluated various representative LLMs and embedding models on the Task of Result ER@Optimal on this new 300 points test set from EvidenceBBench-1006
From table 6a| we clearly see LLMs dominate over embedding models, while GritLM-7B outperforms other embedding models. This shows EvidenceBench-100 100k test set can also be used to evaluate and differentiate various models' performance.
Furthermore, to demonstrate the quality of Beichnccssncc-100k training set, we fine- tuned two models. E5-v2 335M is light-weight embedding model. Since there are over 100 million sentence-aspect pair judgments from the train set, we randomly sampled million triplets, where each triplet contains anchor (the study aspect), a positive (a sentence that is considered as source of information for the study aspect), a negative (a sentence that is not considered as source of information for the aspect). Using margin-based triplet-loss with margin=0.05, AdamW optimizer (weight decay =0.01), peak learning rate of 5e-4, 10% linear warmup then cosine-annealing, batch size =256, we trained E5-v2 with full-parameter tuning for one epoch. We also fine-tuned Llama3-8B using all 80k training datapoints. Llama3-8B is trained with LoRA rank-8, alpha =16, a batch size of 16, Adam optimizer (weight decay =0.01), and the same learning rate scheduler, we fine-tuned the model for one epoch. Note, during inference time we trained LLama3-8B to only output sentence indices, during training, we trained it to output both sentence indices, sentence texts and the summarized list of study aspects. This has shown to be more effective than only training it on sentence indices.
We made sure none of the papers in the original EvidenceBench is used in any way in the Beichenceench-1000 to avoid contamination. Our fine-tuned E5 and Llama3-8B are tested on the original EvidenceBench test set for the task of Result ER@Optimal.
Notice that the fine-tuned E5 model has dramatically improved its performance on the original test set, surpassing all larger embeddding models, see Table 6b] It only trails behind the performance of large language models. The fine-tuned Llama3-8B has performance of 41.0% which trails behind much larger SoTA LLMss This shows that Evidence is suitable for developing and training both embedding- based information retrieval systems as well as large language models.
J Qualitative Analysis for GPT-4o on the Original EvidenceBench
For the task Result ER@ Optimal, there are a total of 1228 results study aspects that the review papers identified.
Out of these 1228, GPT-40's retrieved sentences fail to cover 583 study aspects, from which 50 pairs (missed "Result" aspect, set GPT-4o retrieved sentences) are randomly sampled and manually inspected by two researchers. Out of the 50 cases, 10 cases should not be considered GPT-4o errors. In 7 cases, GPT-4o retrieved an almost sufficient set of sentences but missed one aspect due to the upper limit the number of sentences. This issue arose from a failure in the optimization step during retrieval, not from reasoning or retrieval error. In 1 case, our alignment annotation procedure missed one of the GPT-4o retrieved sentences as the source of information for a study aspect. In case, parsing error occurred where our algorithm did not extract the full summary from the review paper. Finally, in case, one method-relate study aspect is mistakenly labeled as result-related study aspect.
33

---

Published as a conference paper at COLM 2025
K
Further Analyses
Effectiveness of Current LLM Solutions: Table clearly indicates that current embedding models are inadequate for assisting or replacing human experts in identifying relevant evidence for biomedical hypotheses. We now examine the rest-performing LLM solution, GPT-4o. In the task ER@Optimal, GPT-4o retrieves an average of 4-5 sentences per hypoth- esis, according to Table In this setting, GPT-4o achieves a 50% Aspect Recall, covering half of the study aspects identified by human experts. This demonstrates that current LLMs cannot fully replace human experts in finding relevant evidence for hypotheses.
Conversely, per the task definition for ER@10, models retrieve 10 sentences. In this setting, GPT-4o achieves aspect recall of 70%. According to Table [ typical research paper contains 168 sentences. Therefore, instead of reviewing the entire research paper, humans can the 10 sentences retrieved by GPT-4o
efficient starting point, while searching for additional sentences that cover potentially missing aspects. This demonstrates that, in this setting, GPT-4o can meaningfully assist human experts in locating and presenting evidence from research papers for hypotheses, a crucial step in writing review papers.
From Table we observe that the performance of embedding models falls significantly short of the performance of similar- sized generative models. The embedding models GritLM-7B, E5-Misstall-7B and the state- of-the-art NV-Embed-7B cover at most 20.1% of aspects, while the pretrained LLama3-8B covers 35.8% of aspects. The shortcomings of GritLM-7B and NV-Embed-7B suggest that naive local embedding of sentences, without contextual awareness, is insufficient for this task. Our empirical observations confirm that reasoning beyond individual sentences necessary to solve this task effectively. For instance, in Figure [ Sentence and Sentence 69 convey the same information and both address study aspect Only by comparing them together (i.e., reasoning beyond single sentence) can models eliminate one of these sentences to reduce redundancy.
Embedding Models Underperform Generative Models:
Section-level Reasoning is Sufficient: Table 4 shows that retrieving evidence section-by- section (Sec-by-Sec) achieves strong performance on the task. With Sec-by-Sec, a model can only read one section at time, preventing it from reasoning across multiple sections. The strong performance of this method indicates that global reasoning across the entire research paper not essential for retrieving evidence. This can be explained by the structure and organization of biomedical research papers, where the content of each section is relatively self-contained, and interaction across sections is sparse. This suggests that LLMs trained for much longer contexts (e.g,, over 10,000 tokens) may not be necessary for this task.
LLMs Still Get "Lost in the Middle". We observed that some papers have most of their important sentences concentrated at the beginning and end of the document, as shown in Figure 19 We categorize these papers into two groups. In the first category, the first 10 and last 10 sentences of paper cover over 80% of the study's aspects. In the second category, these sentences cover less than 20% of the aspects. Out of 3,000 randomly sampled points from the Evidencceeench-100 test set, 1,115 papers fall into the first category, while 1,111 papers fall into the second category. GPT-4o's s aspect recall performance is 51.6% in the first category and 34.9% in the second category. Claude3-Opus aspect recall is 42.8% in the first category and 28.2% in the second category. Note, during evaluation, we neither explicitly nor implicitly instruct LLMs to focus on any specific parts of the paper.
This indicates that LLMs perform significantly better when the important tokens are not located in the middle of the document. Our manual inspection also reveals that LLMs tend to focus on the beginning and end of the document. The "Lost in the Middle" phe- nomenon, as reported in previous LLMs (Liu et al. 2024), seems to persist in current LLMs on EvidenceBench.
34

---

Published as a conference paper at COLM 2025
All Aspects
Result Aspects
Abstract
2.50%
Abstract
2.00%
Method
Result
2.00%
Method
Result
Frequency (%)
1.50%
Frequency (9)
1.50%
1.00%
0.50%
0.50%
0.00%
0.00%
0.00
0.10
0.20
0.30 0.40 0.50 0.60 0.70 0.80 0.90 1.00
0.00
0.10 0.20
0.30 0.40 0.50
0.60 0.70 0.80 0.90 1.00
Paper
Paper
Figure 19: Original Bench. Left figure shows the distribution for the relative position in the candidate pool of all sentences that are considered source of information for least one study aspect. Right figure shows the same but for aspects labeled as 'Results'. Abstract sentences have much higher chance to be matched with aspects. However, all abstract sentences (around 10 sentences per research paper) only cover around 50% of all aspects, indicating no heuristic algorithm can cheat EvidenceBench.
35


Reviews:
[{"review_id": "MEolDJTBty", "reviewer": "Reviewer_F3p6", "summary": "This paper introduces EvidenceBench to measure LLM performance on finding evidence relevant to hypotheses in biomedical papers. The paper proposes a LLM driven pipeline for hypothesis generation and sentence-by-sentence annotation of biomedical papers for relevant evidence. The paper demonstrates the pipeline’s validity and accuracy with a small set of 50 human-expert annotations. The paper also presents evaluation of language models and retrieval systems on the benchmark and find that model performances still fall significantly short of the expert level on this task.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper introduces EvidenceBench to measure LLM performance on finding evidence relevant to hypotheses in biomedical papers. The paper proposes a LLM driven pipeline for hypothesis generation and sentence-by-sentence annotation of biomedical papers for relevant evidence. The paper demonstrates the pipeline’s validity and accuracy with a small set of 50 human-expert annotations. The paper also presents evaluation of language models and retrieval systems on the benchmark and find that model performances still fall significantly short of the expert level on this task.", "reasons_to_reject": "The paper demonstrates the pipeline’s validity and accuracy with a small set of 50 human-expert annotations. I find this a very low number to claim that the annotation generated by the proposed pipeline is accurate enough to train/fine-tune and test solutions that aim to be state-of-the-art for this task.\n\nThe paper states \"There are two data sources for EvidenceBench and EvidenceBench-100k. First, a collection of 107,887 CC-BY open-sourced biomedical research papers where each research paper represents a datapoint. Second, a collection of 44,772 review papers from PubMed Central.\" This is again a very small benchmark compared to the tens of millions of articles in PubMed and PubMed Central. Also, authors do not provide any evidence on whether the papers selected in the benchmark are good representation/sample of the full biomedical publication repository.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VVnXr3IEhW", "reviewer": "Reviewer_4ogA", "summary": "This paper presents EvidenceBench and EvidenceBench-100k, a pair of similarly constructed benchmarks to evaluate the abilities of (text-only) LLMs to retrieve evidence for a given set of aspects of a hypothesis from a given paper. The benchmark construction occurs in two stages: generating hypotheses and aspects from paragraphs in review articles, and aligning the generated aspects with candidate sentences in the papers referred to by the review. The authors benchmark several recent LLMs, finding that they do not match human performance. I think this paper presents a thoughtfully built, useful benchmark whose construction is well described.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This is a great example of well-validated use of model-based annotations for dataset construction. I appreciate that there is well-described expert annotation for each stage of the dataset construction — hypothesis/aspect extraction and candidate/aspect matching — and that that annotation process is repeated before scaling the dataset up to the 100k version. The authors report correlation/agreement/accuracy scores for all of this. \n\nI think this is also a clever use of review articles as a source of expert data. These summaries already exist and are written for the target audiences of systems that this benchmark proposes to evaluate. I do have some concerns with this choice of dataset, which I discuss in the section below, but I don't think this is a major fault of this work.\n\nThe authors also clearly document licensing information for all of the collected data.", "reasons_to_reject": "Though I noted above that this seems like a good use of review articles, I do still have a few concerns about this use of review articles, largely due to my own unfamiliarity with the form. The authors state at several points that sentences are chosen for their \"relevance\" to aspects of the hypothesis. I would appreciate some degree of qualitative exploration of the data to estimate the distribution of that relevance - how much of the candidate claims in papers support, vs. refute the hypothesis? Without some estimation of this, it is possible that this benchmark would fail to assess systems that become a confirmation bias machine - i.e. systems that consider all of the evidence that supports a piece of evidence or an aspect, but ignores evidence that contradicts it.", "questions_to_authors": "The authors very carefully describe the process of finding and validating hypotheses, but do not actually describe how these hypotheses are used in the final evaluation of models. I assume that the hypothesis is provided as input to the retrieval/embedding models; is this the case?", "extra_scores": {}}, {"review_id": "xl1NYYNkO9", "reviewer": "Reviewer_b6XD", "summary": "This paper introduces EvidenceBench, a benchmark dataset for extracting relevant evidence sentences from biomedical research papers given a hypothesis. The authors propose an efficient LLM-powered pipeline for dataset construction, including hypothesis generation, decomposition of expert summaries into study aspects, and aligning aspects with sentences. This method allows for the creation of large-scale datasets with significantly reduced cost and time compared to purely manual annotation. The quality and validity of the pipeline outputs (hypotheses and alignments) are evaluated against human experts, showing comparable performance to biomedical PhD students for the single-sentence annotation task. The paper evaluates various LLMs and embedding models on the benchmark, but the performance still falls significantly short of the expert level. The task is important, and the datasets (EvidenceBench and EvidenceBench-100k) are original and potentially useful resources. The paper is mostly well-structured, but following the experimental details is challenging as much of the important information is located in the appendix rather than the main body.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The creation and release of EvidenceBench and the large-scale EvidenceBench-100k dataset is a major contribution to the field, providing a valuable resource for research on evidence extraction from biomedical literature.\n- The LLM-powered pipeline for dataset construction is a novel approach that enables efficient and scalable creation of high-quality annotated data, significantly reducing manual effort.\n- The paper provides human evaluation results demonstrating the validity and accuracy of the dataset construction pipeline outputs, lending credibility to the generated annotations.\n- The paper provides a benchmark evaluation of various models on the task, offering insights into the current state-of-the-art performance and highlighting the significant gap that still exists compared to human expert performance, indicating the task's difficulty.", "reasons_to_reject": "- The definitions used in the annotation scheme, specifically regarding the decomposition of summaries into study aspects and the criteria for defining a \"source of information\" sentence, appear subjective. It is unclear how consistently these subjective definitions can be applied by different annotators or if they depend significantly on expert interpretation, which could impact the consistency and reliability of the dataset annotations.\n- The benchmark shows LLMs performing significantly below human expert level despite the dataset being created using an LLM-powered pipeline. The paper does not fully clarify the implications of this result or explicitly discuss why evaluating the end-to-end performance of the dataset creation pipeline (or a similar guided approach that leverages the multi-step process) on the extraction task was not included in the benchmark. This could lead to ambiguity regarding the benchmark's objective or the potential for achieving higher performance with different methodologies not evaluated.\n- A considerable amount of crucial detail regarding the experimental setup and methodology for the model evaluation is placed in the appendix (Appendices C and D) rather than the main body. This structure makes it challenging for readers to easily follow and fully understand the experimental procedures and interpret the results presented in the main tables without frequently referring to the appendix.", "questions_to_authors": "- Could the authors provide more detailed information or analysis regarding the subjectivity in the decomposition into study aspects and the definition of the source of information among experts? How was consistency ensured for these potentially subjective judgments during annotation?\n- The dataset was annotated using an LLM-powered pipeline, yet LLMs show relatively low performance in the end-to-end benchmark evaluation. This makes the reviewer wonder about the primary objective of the evaluation: is it solely to measure the performance of current standalone SOTA LLMs on this task, or is there a broader aim? Given the LLM-based creation process, could the authors discuss why evaluating an approach that leverages a similar pipeline flow for the end task was not considered or presented as a potential direction for achieving higher performance?\n- The authors consider recall-oriented evaluation criteria (Aspect Recall@k), but it is unclear from the description how this was considered in designing the instructions provided to the LLMs during the benchmark. Did the authors consider to explicitly instruct LLMs to optimize for recall within a specific number of top relevant sentences?\n- The paper mentions filtering cases where less than 70% of study aspects from the evidence summary are covered by sentences. How frequent are such cases, and what is the distribution of the percentage of covered study aspects in the final dataset?", "extra_scores": {}}, {"review_id": "O6GQkI4qy6", "reviewer": "Reviewer_Gnr9", "summary": "The authors propose a novel pipeline generating hypotheses and finding relevant evidence, which is useful for researchers investigating scientific hypotheses and finding supporting evidence. They show their pipeline has a close accuracy and validity compared with human experts. Also, they benchmark different models' performances based on the two datasets they created, which shows there is still space for improvement on the models.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is written clearly and is easy to follow.\n- The authors provide a comprehensive illustration and explanation of the pipeline.\n- The datasets are constructed in a way that is comparable to human-expert level, with the assistance of some GPT models.\n- The task is important and useful for the researchers for finding supporting evidence of a hypothesis when using generative AI.", "reasons_to_reject": "- The authors target biomedical papers, but they do not say why they target such a field/area, or the significance of providing such a benchmark in this field. Why is identifying the most important pieces of evidence relevant to a hypothesis crucial for this domain? Why not other domain papers?\n- It seems that the GPT models have the highest scores using the evaluation of this benchmark. Would it be possible because the authors use GPT in the procedure of generating the dataset [1]? How will it affect the validity of the benchmark?\n- EvidenceBench-100k is only assessed using 50 randomly sampled papers for quality purposes; also, it seems that I could not find any table describing the Human & GPT agreement on this.\n\n[1] LLM Evaluators Recognize and Favor Their Own Generations.", "questions_to_authors": "How do you calculate the p-values in Table 2?", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces EvidenceBench, a corpus instantiating an "evidence extraction" task in which the goal is identify the sentences in an article that are most relevant to a given hypothesis. There was consensus amongst reviewers that this is an important task, and that the authors have done a laudable job of validating each annotation step. The authors did a nice job of addressing remaining concerns during the discussion period.
Decision: Accept