Can Large Language Models Match the Conclusions of Systematic Reviews?
Authors: Christopher Polzak
Alejandro Lozano
Min Woo Sun
James Burgess
Yuhui Zhang
Kevin Wu
Chia-Chun Chiang
Jeffrey J Nirschl
Serena Yeung-Levy
Venue: ICLR 2026 Poster (2026)
Keywords: Benchmarks
Multi-document Reasoning
Medical AI
Primary area: datasets and benchmarks
TL;DR: We introduce MedEvidence, a benchmark to test if LLMs can replicate expert systematic reviews.

Abstract: Systematic reviews (SR), in which experts summarize and analyze evidence across individual studies to provide insights on a specialized topic, are a cornerstone for evidence-based clinical decision-making, research, and policy. Given the exponential growth of scientific articles, there is growing interest in using large language models (LLMs) to automate SR generation. However, the ability of LLMs to critically assess evidence and reason across multiple documents to provide recommendations at the same proficiency as domain experts remains poorly characterized. We therefore ask: **Can LLMs match the conclusions of systematic reviews written by clinical experts when given access to the same studies?** To explore this question, we present MedEvidence, a benchmark pairing findings from 100 medical SRs with the studies they are based on. We benchmark 25 LLMs on MedEvidence, including reasoning, non-reasoning, medical specialists, and models across varying sizes (from 7B-700B). Through our systematic evaluation, we find that reasoning does not necessarily improve performance, larger models do not consistently yield greater gains, and knowledge-based fine-tuning degrades accuracy on MedEvidence. Instead, most models exhibit similar behavior: performance tends to degrade as token length increases, their responses show overconfidence, and, contrary to human experts, all models show a lack of scientific skepticism toward low-quality findings. These results suggest that more work is still required before LLMs can reliably match the observations from expert-conducted SRs, even though these systems are already deployed and being used by clinicians.

Published as a conference paper at ICLR 2026
CAN LARGE LANGUAGE MODELS MATCH THE CON- CLUSIONS OF SYSTEMATIC REVIEWS?
Christopher Polzak**
Alejandro Lozano*1
Min Woo Sun*
James Burgess!
Yuhui Zhang'
Kevin Wu
Chia-Chun Chiang
Jeffrey J. Nirschl-
Serena Yeung-Levy'
'Stanford University
Mayo Clinic
ABSTRACT
Systematic reviews (SR), in which experts summarize and analyze evidence across individual studies to provide insights on specialized topic, are cornerstone for evidence-based clinical decision-making, research, and policy. Given the ex- ponential growth of scientific articles, there is growing interest in using large language models (LLMs) to automate SR generation. However, the ability LLMs to critically assess evidence and reason across multiple documents to pro- vide recommendations at the same proficiency as domain experts remains poorly characterized. We therefore ask: Can LLMs match the conclusions of systematic reviews written by clinical experts when given access to the same studies? To explore this question, we present MedEvidence, a benchmark pairing findings from 100 medical SRs with the studies they are based on. We benchmark 25 LLMs on MedEvidence, including reasoning, non-reasoning, medical specialists, and models across varying sizes (from 7B-700B). Through our systematic evaluation, we find that reasoning does not necessarily improve performance, larger models do not consistently yield greater gains, and knowledge-
fine-tuning degrades accu- racy on MedEvidence. Instead, most models exhibit similar behavior: performance tends to degrade token length increases, their responses show overconfidence, and, contrary to human experts, all models show a lack of scientific skepticism toward low-quality findings. These results suggest that more work is still required before L.Ms can reliably match the observations from expert-conducted SRs, even though these systems are already deployed and being used by clinicians.
INTRODUCTION
As the number of published articles grows exponentially (Bornmann et al. 2021 manually synthe- sizing findings from multiple sources has become highly time-consuming. Thus, there growing interest in developing automatic tools to process, synthesize, and extract insights from scientific literature (Lozano al.
2023 Scherbakov al. 2024). In particular, large language model (LLM)- based systems could offer a promising solution for supporting and automating tasks in conducting systematic reviews (SRs), which typically take an average of 67 weeks of intensive human effort (Fabiano et al. 2024 Riaz et al. 2024 Wang et al. 2025). For example, several L. tools such as Deep Research (OpenAl 2025) Google 2025). Elicit (Elicit 2025), and Open Evidence OOpenEvidence 2025) have already been deployed and can be incorporated into the SR process to improve efficiency (Fabiano et al. 2024). The momentum behind these technologies further exemplified by the U.S. Food and Drug Administration's launch of an MLM-assisted scientific review pilot on May 2025 (FDA 2025).
However, despite multiple deployments and efforts assessing scientific synthesis generation. the behavior of LLMs across key variables that influence generation remains poorly understood. In particular, their ability synthesize findings from multiple studies-edt varying in study type,

---

Published as a conference paper at ICLR 2026
Domain Expertise
Conflict Resolution
Scientific Skepticism
Homatolngy
la the ovenall surviral rate higher, lowar, orehe come when comparing peruuanenus ethanal injection sercutancous
Studies
hesplet clecherge lowar, when compuring untrrred bystandor CPR with curdi usus chisi compression untrained tystonder CPR compressiion interrapted pausee brealting?
Studies
Q: Gheen the stuchee, in the rak of chppocooccal
injeciion (PAi??
events lightr Inwer same when intsation delayed ART intiation??
Psychiatry
whhenn higher, placebo?
LLM
Internal Medichae Subspaciatiis
Rationale: survival to hospital dischenge not significantly different. as shown by simitar sunvival rates studies
Rationale:
sign iRcant differinco...Howw Howover a'dde sugguade
statidicaally
Q: is patient fundtin measured by HAG scorw light: the same
signikianty legher
comparing bologio monotherapy to placebo'?
(4)
Modal Anawer nigher
Pedeetiis Mecnatology
Human Rationale: "Thore were significant ciffrrerzzss teltwwn the two groupe in the tnwis. The pooled result showed better survivaf for contimeuuu compression slore group (RR
Human Answer nigher
Human Answer: uncoriaii affec! Human Rationate: "We are uncertain, (due wide CIs and very few clinical events...IRE IRIE outzome acsesoors wern unblinded diagoosing
Q: the incldence of tronchopulmonary apellasi (BPD) higher lower or xame comparing prophylactic GRIP to very early CPAP
very subjectiive
Figure Core skills evaluated by MedEvidence including: medical domain expertise across 10 different specialties, synthesizing conflicting evidence, and applying scientific skepticism when studies exhibit high risk of bias (e.g. due small sample sizes or insufficient supporting evidence).
population size, and risk of bias-and and to navigate conflicting evidence (as medical findings may contradict one another) not well-characterized. Understanding these behaviors essential, as medical knowledge is continually reshaped by new clinical trials, cohort studies, and expert opinions. Thus, like medical professionals do, LLMs must be capable of integrating the latest findings (e.g. via retrieval augmentation) (Ke et al. 2024), weighing the strength of varying evidence, and applying appropriate skepticism when needed to produce reliable, up-to-date recommendations (as shown in Figure DT.
While prior work has successfully evaluated LLMs on their internal "static" medical knowledge (Liévin et al. 2024 Fleming et al. 2024), assessing LLMs' capability reason across multiple sources and draw expert-level conclusions remains a significant challenge. Specifically, previous efforts have often evaluated LLMs' ability to generate summaries a given topic. This approach requires a thorough review of every detail in the generated content and lacks easily verifiable ground truth; therefore, medical experts are typically needed assess output accuracy (Wallace al. 2021
Schopow et al. 2023) Qureshi et al. 2023 Lai et al. 2024 O D'oherrty et al. 2024). making evaluation time-consuming and hard scale.
To address this, we remove the complexity of evaluating long-format summaries and retrieving relevant papers to pose an even simpler, but fundamental question: Can LLMs replicate the individual conclusions of expert-written SRs when provided with the same source studies? We explore this question in a controlled setting by collecting open-access SRs along with their associated reference articles. We then extract individual findings and reformat them into closed question -anttiinnnn (QA) task to simplify evaluation. To this end, we introduce the following contributions:
MedEvidence Benchmark We introduce MedEvidence, a human-curated benchmark of 284 questions curated from the conclusions of 100 open-access SRs across 10 medical specialties. Each question evaluates comparative treatment effectiveness on clinical outcomes. All questions are manually transformed into closed-form question answering to enable large-scale evaluation. In addition, human annotators extract evidence quality (based on the SR's analysis), determine whether full-text access is necessary, and collect the relevant sources needed replicate the SR findings.
Large-scale evaluation on MedEvidence We leverage MedEvidence to perform an in-depth analysis of 25 LLMs spanning general-domaiee utilizing MedEvidence's 's metadata, we dissect and examine success and failure modes, helping to identify targeted directions for future work.
and reasoning models. By
2

---

Published as a conference paper at ICLR 2026
2
RELATED WORK
Table I: Comparison of factuality and evidence reasoning benchmarks with medical focus. We compare MedEvidence to prior datasets across attributes relevant to systematic review-style reasoning. MedEvidence is the only dataset satisfy all criteria.
Expert- Grounded Answer
Automated Evaluation
Multiple Sources
Evidence Quality
Source- Level Concordance
Dataset
Size
Topic
Curation
Reason al.
Medicine
Human
X
Schopow et al.
Medicine
Human
X
/
x
MedREQAL
2786
Medicine
LLM
x
HealthFC
750
Consumer Health
Human
V
V
X
ConflictingQA
238
Multi-Domain
LLM
X
/
X
MedEEvidence
Medicine
Human
/
Table presents an overview of related works and their key distinctions with respect to our work.
LLM-based medical systematic review Numerous studies have explored the potential of L.MSs to automate various aspects of scientific literature review, including literature search, query augmenta- tion, screening, data extraction, bias assessment, narrative synthesis, and answering simple clinical
inquiries (Lieberum et al. 2025 Clark et al. 2025). However, larger-scale and closed QA evaluations of -ased SR meta- meta-analysee analyses generation remain relatively underexplored. Wallace al. (202T)
evaluates neural models on generating narrative summaries from RCTs referenced in Cochrane reviews, using the review as ground truth, ultimately requiring expert oversight to assess correctness. Building this line of inquiry. O DDoherty et al. (2024) conducts similar assessment across 45 systematic reviews. Reason et al. (2024) further explore whether LLMs can extract numerical data from abstracts and generate executable code for meta-analy meca--nnalysss. While their findings encouraging, the study is limited to four case studies, restricting generalizability. [Schopow et al. (2023) and Qureshi et al. (2023) investigate LLM usage across a range of systematic review stages, including meta-review and narrative evidence synthesis, but also present findings on a very small-case study scale (N < 10) and rely on comparison humans. Overall, these investigations require substantial amounts of review from medical experts, highlighting the need for automated benchmarks to facilitate evaluation.
Verification of medical faats derived from systematic reviews Several studies have leveraged SRs to benchmark LLMs' ability to perform medical fact verification, where a model must decide whether to support or refute a claim. For instance, MedREQAL (Vladika et al. 2024a) is an LLMMccuraed closed QA dataset designed to investigate how reliably models can verify claims derived from Cochrane SRs. However, it does not provide the sources used by the SRs. Instead, the dataset evaluates models on their internal knowledge, making the task a form of fact recall. HealthFC (Vladika et al. 2024b), on the other hand, tasks models with verifying claims analyzed by the medical fact-checking site Medizin Transparent, but only provides pre-synthesized analysis from the web portal as evidence. In contrast real SRs, this task primarily involves retrieving information from a pre-synthesized source, removing the complexity of reasoning across unsynthesized evidence. Unlike prior work, MedEvidence requires extracting, reasoning over, and synthesizing relevant information across single or multiple sources (each with different levels of evidence) to match the expert-derived conclusion of SR (without access the original SR itself). resembles the intricacies of SR analysis, as the raw sources (articles/abstracts) are directly provided to the model.
LLM Behavior in the Presence of Conflicting Sources ConflictingQA Wan al. 2024 examines how models respond to conflicting arguments supporting or refuting a claim. However, it focuses on inherently contentious questions without definitive answers, spans domains beyond medicine, and uses diverse online sources rather than peer-reviewed literature. ClashEval (Wu et al. (025) investigates conflicts between a model's internal knowledge and external evidence, including drug- related (medical) subset, but limits evaluation to single-source conflicts with artificially perturbed values. ConflictBank (Su et al. 2024) and KNOT (Liu et al. 2024) assess model performance on specific conflict types such as temporal inconsistencies. misinformation, and logic-based contradic- tions bonns-buturn rely on factoid-style questions sourced from Wikipedia. These benchmarks only leverage relatively succinet and synthesized inputs.
3

---

Published as a conference paper at ICLR 2026
A Systematic Review Selection
B Conclusion Conversion
Pub Med
Systenatc PubMled
TITLE [MMID]
Abstract
lolooru fightt lowar same when cemooraiinng two shose placebo? Aowowe
Main Result
Two doses of Shandhol souces cases cholcra ol hwo your folowou (onaiit
MedEvidence
Eridence Certainty.
C Relevant Study Selection
Question Validation
Poid
PublMed
Relevant studies
Study
Meta-Anilyeis
Skdy PN
Study
Poppattion
Stady 3 Total 15%% CD
Saty pie3
Chilags
Shandel
Figure 2: Overview of the dataset curation process for MedEvidence.
To the best of our knowledge, existing studies or datasets provide richly annotated data to systematically benchmark models' ability to align with the conclusions of medical systematic reviews while using the same underlying research documents as the original medical experts.
DATASET CURATION PROCESS
Data provenance We collect open-source systematic reviews, available via PubMed, conducted by Cochrane, international non-profit organization dedicated to synthesizing evidence healthcare interventions through contributions from over 30,000 volunteer clinician authors (Henderson et al.) 20T0). Cochrane is a long-standing and widely respected source of clinical evidence (Petticrew et al.
2002, Cipriani et al. 201T), offering open-access content and analyses presented in a standardized format. Additionally, for each SR, collect all the cited studies that are relevant for given conclusion (we refer to these studies as 'sources') ) When the source article's full text is available (i.e.
the article is seen-s,rre)), we obtain using the existing BIOMEDICA dataset (Lozano et al. 2025): ;
otherwise, abstracts are retrieved directly via PubMed's Entrez API (PubMed, 2010-). All retrieved full-text articles use CC-BY 4.0 license, which allows for re-distribution.
Dataset curation pipeline The core challenge in creating our dataset is ensuring that an LLM is provided with sufficient information to reproduce a given conclusion. To ensure high-quality dataset, we developed four-stage pipeline of: (1) systematic review selection, (2) conclusion to questions conversion, (3) relevant study selection, and (4) question feasibility validation (as shown Figure
1. Systematic review selection We use Entrez to retrieve all Cochrane SRs published between January 1, 2014 April 4, 2024 (PubMed 2019p. We only include systematic reviews for which all sourced studies are indexed in PubMed (with at least an abstract available). We additionally retrieve all data and metadata for the sourced studies, including: full-text via BIOMEDICA (when it is available), abstract, mesh terms, title, and publish date.
2. Conclusion to question conversion. Cochrane reviews follow a standardized format, al- lowing for a systematic conversion process. To identify potential questions, we followed the protocol below: Human annotators were instructed review the SR abstract and ex- amine the "Main Results" subsection (see Appendix Figure for an example) to identify individual conclusive statements that statistically compare an intervention with control group. These individual statements were then converted into preestion-asswe pairs by the annotators, with answers belonging to a fixed set of classes. To be clear, insuffica cient data was used for statements by the SR authors explicitly indicating that study in- vestigated- or included sufficient data to analyze- the combination of treatment, control, and outcome; uncertain effect referred cases where analysis performed but definitive conclusions could not be made (see Appendix Section B.2 for more conversion details). Evidence certainty was extracted only when it was explicitly provided by the
4

---

Published as a conference paper at ICLR 2026
Out Cecteee atratieed medical specialy
Bo Calaset stratdies
[CT Delonad strattted by soutte
Spine dayp Nesth Syrero
JUIN
2576
13.3%
Calca Dar
WHARE stat
Figure 3: Key statistical characteristics of the questions in MedEvidence. (a) shows the dataset distribution stratified by medical specialty. (b) presents the distribution stratified by outcome effect. (c) shows the distribution stratified by source concordance with the expert-assessed treatment outcome effect (i.e. the correct answer).
original SR authors, who use the standardized GRADE framework (Bezerra et al. 2022) to assess the quality of evidence in the included studies. This certainty is often stated in the abstract, indicating the strength or quality of each observation.
3. Relevant study selection To identify relevant studies for a given SR, annotators used the analysis section provided in the appendix, which "weighs" the contributions of sources supporting each conclusion. For questions with insufficient data (where it not possible to determine weights), reviewers were instructed to include studies cited in the SR that either (1) discuss the specified treatment and control but not the outcome, or (2) evaluate the treatment and outcome but compare against different control.
4. Question feasibility validation Finally, given the question-answer pair and the source studies, annotators were tasked with determining whether the question was answerable based on the provided information. A question was considered answerable if at least 75% of the total weight in the analysis came from "valid" studies included in the meta-analysis. We define study "valid" if (1) provides numerical data on both the intervention and control groups specified in the question, and (2) includes statistical or numerical details about the difference between the groups the specified outcome- -usoome--cchhheo raw counts, p- p-values, confidence intervals, or risk ratios. The most common reason for discarding conclusions when review authors pooled outcome data across studies, but the outcome omitted or discussed without clear statistical detail in the abstracts of relevant studies.
In addition to these human-curated metadata, we use an LLM to assess the percentage of individual source studies whose answer to the question aligns with the final answer provided in the systematic review. Thus, to calculate source-level agreement (which we call 'source concordance') we prompt DeepSeekV3 (the strongest model in our benchmark) to answer the question using only one single relevant source; the source is deemed to 'agree' with the final answer if and only if the LLM's classification with the one source matches the ground truth classification.
Medical domain taxonomy assignment To identify the relevant medical specialties in our dataset, we extract the Medical Subject Headings (MeSH terms)-a controlled vocabulary used by PubMed to index papers-frov the 100 systematic reviews included in dataset. We then feed this list into DeepSeek to generate a simplified categorization of specialties, resulting in 10 categories. Finally, we prompt DeepSeek to assign each question to the most relevant category, or to an "Other" category if no specific specialization is applicable.
4 DATASET DESCRIPTION
MedEvidence contains a total of 284 questions derived from 100 systematic reviews with 329 referenced individual articles, of which 1 14 have full-text available (see Appendix Figure 3 for
5

---

Published as a conference paper at ICLR 2026
Table Sample question from the dataset. Fields marked with asterisk (+) use LLMs to assist the generation. Relevant source details are omitted here for brevity.
Question
Is stroke prevention higher, lower, or the same when comparing Tran- scatheter Device Closure (TDC) to medical therapy?
Answer
difference
Relevant Sources (PubMed IDs)
22417252, 23514285, 23514286
Systematic Review (PubMed ID)
26346232
Review Publication Year
2015
Evidence Certainty
n/a
Open-Access Fext-Text Needed
*Souree Concordance
1.0
Medical Specialty
Surgery
cohort diagram of the dataset). Questions were systematically collected by three human annotators with between one and five years of graduate education. Figure 3 shows the dataset distribution stratified by specialty, outcome effect, and source concordance with the expert-assesset treatment outcome effect (i.e. the correct answer). The benchmark covers topics from medical specialties (e.g. public health, surgery, family medicine, etc.), five different outcome effects (highe lower, no di ffe Fencee uncertain. effect, insufficien
data), and three broad levels of concordance hetween the source paper and the correct answer (full agreement, no agreement, mixed agreement). Additional characteristic distributions of the dataset can be found in Appendix Figure [1]
Data format. MedEvidence is grouped by question; each question includes core data for evaluation, metadata, as well as the content details for the relevant sources. The core data consists of: a human-generated question of the form "Is [quantity of medical outcome] higher, lower, or the same when comparing [intervention] to [control]?"; the taxonomized answer to the question (higher: lower, no difference. uncert uncertain effect, Insufficiens data); and the list of relevant studies (sources) used by the review authors to perform the analysis, identified by their unique PubMed IDs. We additionally provide the following metadata: the systematic review from which the question was extracted; the publication year of the systematic review; the authors' confidence in their analysis, also referred to as the "evidence certainty' (high, moderate, low, very low, or n/a if not provided); Boolean identification of whether full-text is available and needed to answer the question; the exact fractional source concordance; and the medical specialty associated with the question. Separately, for each source, provide the unique PubMed ID, title, publication date if available, and content (full-text available PMC-OA, abstract otherwise). An individual data point example is shown in Table 2
5 BENCHMARKING LLM PERFORMANCE
5.1 EXPERIMENTAL SETTINGS
LLM selection We selected 25 LLMs across different configurations, including a variety of sizes (from 7B to 67 IB), reasoning and non-reasoning capabilities, commercial and non-commercial
licensing, and medical fine-tuning. This selection includes GPT-ol OOenAA 2024b). DeepSeek R1 (DeepSeck-Al 2025a). OpenThinker (Team) 2025a). GPT-4.1 (OpenAl 2024a). Qwen3 (Team) 2025c). Llama 4 (AI@ Meta 2025), HuatuoGPT-o (Chen et al. 2024a), OpenBioLLM (Ankit Pal 2024). and more (please see Appendix TableIar to see details of all selected models). This selection is non -xh-ussaustive; rather, is designed to investigate overarching trends across different model types.
Prompting setup
Basic prompt We evaluated all models in a zero-shot setting, prompting them to first provide a rationale for their answer, followed by 'answer' field containing only one option from the list of five valid treatment outcome effects (higher, lower, no difference, uncertain effect, or insufficient data). We provided minimal guidance in the prompt beyond specifying the required response format, and supplied the abstracts or full text of the relevant studies as context (see Appendix Figure 12 for the exact prompt).

---

Published as a conference paper at ICLR 2026
(b)
OpenBicLLM tor
Wes Ourpue
Uama Scout
Resesring
Lama 2.4 35
hontlusscringg
Medicaly
OpentiolUM 700
Time Condrannd
Human Excett
Limms 1.1 ab
Lbma 30
ano Hoverick
631
0.0
02
5.5
0.8
10
Ovaral Accuracy Nercentage Valld
Peatment Cuscome Dffect
Figure 4: (a) Average model accuracy (and 95% CT) on MedEvidence, overlaid on the percentage of questions where the model provided valid output (details in Appendix Best expert performance is shown in a pink dashed line (more details in Appendix Rj. No model matches or surpasses the best expert performance, even though experts are time-constrained. (b) Average recall grouped by ground truth treatment outcome effect, aggregated across all models (with 95% CT). A per-model average recall by treatment outcome effect is shown in Appendix Figure 18
(a)
(b)
0.6
0.
0.0
moderate
high
0.0
1-40%
50%
51-99%
100%
Exitence Certainyy Leve
Percentage of intiedua Sources Matching the Correct Arweer (Binred
OpenticLLM
Open 700"
GPhet
Qoen5-255-421-175
OpanThinser2-35
DeapSemk
Jama Soout
MuckuzEPi a1 729*
Uama 3.1 4038
Uame
Lamu 3.3 10s INL D610
Non Rezsoring Reasoring
Usma 3.0
Lama 3.1 86
Usira 3.5 108 Instrctt
GPT ooz 1208
DespSeek R1 GP771
Owen2 3.70 retruct
Lama 3- 700
Owen2 5:310 retruct
GPT11 mini
..w. Average
Limma Maverck
Ower2 5-728 ritruct
OwD 328
Figure 5: (a) Accuracy as function of evidence certainty, shows monotonically increasing trend. (b) Accuracy as a function source concordance, defined as the percentage of relevant sources that agree with the final systematic review (SR) answer, also exhibits a monotonically increasing trend.
2. Expert-guided prompt LLMs may not natively understand how to handle multiple levels of evidence, which can lead to unfair evaluations. To address this, we explicitly design prompt that instructs the LLM to summarize the study design and study population, and to assign grade of evidence based on established definitions of grades of recommendation (see Appendix Figure for the full prompt).
For both cases, the input exceeded the LLM's context window, we used multi-step refinement (via LangChain's is RefinelDeemmnttChain to iteratively refine the answer based sequence of article chunks. All models were evaluated with zero temperature to maximize reproducibility.
LLM evaluation Model performance evaluated using accuracy based on exact match between the answer field and the ground truth. Model outputs were lower-cased and stripped of whitespace before comparison. If no 'answer' field was provided, or if its content was exact alle-baedd
7

---

Published as a conference paper at ICLR 2026
1.0
OpenBieLLM 83
HuatuaGPP-o1--28
OpenBieLLM
328
706
HuatuaGFTF01-708
0.8
Qwen2.5
Liema 3.3 708 (AI-Distill)
Accuracy
0.6
OpenTninker2-326
QwQ-32B
GPT-oss
0.4
Owen3-235B-A222
Liama 3.3 70B-Instruct
Liama
Liema 3.0
0.2
DeepSeek
Non-Reasoring
Reasoning
0.0
Medical*
100
200
300
400
500
600
700
Figure Average model accuracy as a function of model size. We observe diminishing returns beyond 70 billion parameters. Arrows point from base models to their medically -fnnttuued counterparts (arrow between HuatuoGPT-ol 7B and Qwen2.5 7B omitted due to very similar performance).
match with the correct answer, the output was deemed incorrect. Confidence intervals (CIs) were calculated via bootstrap (95%, N=1000) (Eiron Tibshirani 1994). Models were evaluated both under zero-shot and few-shot settings.
Compute Environment Experiments were performed in local on-prem university compute environ- ment using 24 Intel Xeon 2.70GHz CPU cores, 8 Nvidia H200 GPUs, 16 Nvidia A6000 GPUs, and 40 TB Storage. Large-scale models that could not be run locally in this environment were queried in the cloud using public APIs available from together.ai OpenAl.
6 DISCUSSION
As shown in Figure 4 (a), even frontier models such as DeepSeek V3 and GPT-4.1 demonstrate rela- tively low average accuracy of 62.40% (56.35, 68.45) and 60.40% (54.30, 66.50), respectively-far from saturating our benchmark. More importantly, model performance still lags behind expert clinical accuracy (< 0.75), even when clinicians are limited by time and unable conduct the in-depth analysis performed by the original SR authors. We identify four key factors that influence model performance
benchmark: (1) token length, (2) dependency treatment outcomes, (3) inability to assess the quality evidence, and (4) lack skepticism toward low-quality findings. Additionally, we found that (5) medical fine-tuning does not improve performance, and (6) model size shows diminishing returns beyond 70 billion parameters. We explore each of these factors in more detail below using the basic prompt setup.
Reasoning vs non-reasoning LLMs We highlight that, in general, reasoning models do not consis- tently outperform non-reasoning models of the same class size on MedEvidence (Figure g (a)), as evidenced by DeepSeek V3 outperforming its reasoning counterpart (DeepSeek RI) while LLaMA 3.3 70B distilled from DeepSeek R1 outperforms the LLaMA 3.3 70B base model.
Model performance decreases as token length increases Generally, performance on MedEvidence drastically reduces as the number of tokens increases (Appendix Figure 15) even though all but two models can fit 80% of the dataset within one context window (see Appendix D). Naturally, training ILMSs on long contexts does not guarantee improved long-context understanding, as models may still
struggle to utilize information from lengthy inputs (Chen et al. 2024b Li ct al. 2024).
Model performance dependency on treatment outcome effect Figure (b) shows the per-class recall stratified by treatment outcome effect. Overall, all models perform best questions where the correct answer corresponds to higher or 1 lower effects cases where strong stance can be taken. They slightly less successful questions, where a definitive conclusion is available but there is no clear preference for either treatment. Performance is lowest the most ambiguous class, uncertain effect. Notably, as
di difference and insufficient data
8

---

Published as a conference paper at ICLR 2026
shown in Appendix Figure 16 models are generally reluctant to express uncertainty, often committing to a more certain outcome that appears plausible. Notably, previous work has observed LLMs are
verbally overconfident (Sun et al. 2025 Xiong al. 2023) and shown that reinforcement learning
via human feedback (RLHH) amplifies this effect ( (Leng et al. 2024).
Model performance improves with increasing levels of evidence We leverage the evidence certainty levels reported by experts in each systematic review (SR). As shown in Figure 5 a), the overall ability of models to match SR conclusions improves the level of evidence increases. We therefore explore whether model performance is also associated with the level of source concordance. As shown in Figure b])), models' ability to match human conclusions increases as the proportion of sources agreeing with the correct answer increases (e.g., DeepSeek V3 achieves 92.45% accuracy at 100% source agreement vs. 41.21 % at 0% source agreement). Prior work has noted that I.L.s struggle to aggregate findings across multiple documents Shaib et al. (2023). Our results show that this limitation amplified when sources present conflicting conclusions. This suggests that, unlike human experts, current LLMs struggle to critically evaluate the quality of evidence and to remain skeptical of results. We observe that this behavior persists even when models are prompted (using the expert-guided prompt) to consider study design, population, and level of evidence (Appendix Figure I9
Medical finetuning does not improve performance Figure 7 compares the average performance of medi- cally finetuned models to their base model counterparts. Across all comparisons, medical finetuning fails im- prove performance (even for medical-reesoning models) and, in most cases, actually degrades it. Indeed, fine- tuning without proper calibration can harm generaliza- tion, sometimes resulting in worse performance than the
Reesoning Non Reasoring Medically Finetured*
OpenBioLLM 88*
Liama 3.0
HustuoGPT
Qwan2 5-78-instruct
base model (Mai et al. 2024] Kong et al.) 2020 Wu et al. 2024). Similar behavior has been previously re- ported in long-context medical applications (Fleming
OpenBialL 708*
ct al. 2024).
HustuoGPT 708*
Model size shows diminishing returns beyond 70B parameters As shown in Figure 6 within the same model families, increasing size from 7B to 70B param- eters yields substantial accuracy gains on MedEvidence. However, beyond this point, we observe rapidly dimin- ishing returns, both within specific model families and across suite of evaluated models more broadly.
Liama
0.0
0.1
0.2
0.3
0.4
0.5
0.6
Figure 7: Medically-fintunee models vs their base generalist counterparts. Pairs of medical and base models are adjacent. 95% confidence intervals are calculated via bootstrapping with N 1000.
Additional evaluations We further re-evaluate top per- forming models and find that randomizing the order of sources and omitting thain-of-thougee prompting both do not significantly affect performance. Few-shot evaluation yields slight improvements. but high eгror rates persist and the performance gap to clinical experts remains. For detailed results, further experiments, and qualitative analysis, please see the Appendix.
Combined, our results suggest that synthesizing information across sources to match individual systematic reviews' conclusions eludes current scaling paradigms. Increasing test-time compute (i.e., reasoning) does not necessarily improve performance, larger models do not consistently yield greater gains, and knowledge-basse
fine-tuning tends to degrade performance. Instead, most models exhibit similar behavior: model performance tends to degrade as token length increases, their responses show overconfidence, and all models exhibit a lack of scientific skepticism toward low-quality findings. These results suggest that more work is required before LLMs can reliably match the observations from expert-conducted SRs, despite L.M systems already being deployed and used by clinicians.
Linitations Our study has several limitations. First, the dataset is subject to selection bias, include a SR if all its sources are available (either full text/abstractt. Second, while our benchmark designed to isolate and provide a controlled environment to test LLMs' ability reason over the same studies experts used to derive conclusions, it does not assess the full SR pipeline, including literature search, screening, or risk-of-bias assessment. Future work could incorporate multi-expert consensus or update findings based on newer studies to strengthen benchmark reliability.
only

---

Published as a conference paper at ICLR 2026
7 CONCLUSION
Benchmarks drive advancements by providing a standard to measure progress and enabling re- searchers to identify weaknesses in current approaches. While LLMs already deployed for scientific synthesis, our understanding of their failure modes still requires broader investigation. In this work, we present MedEvidence, benchmark derived from gold-standard medical systematic reviews. We use MedEvidence to characterize the performance of 25 LLMs and find that, unlike humans, LLMS struggle with uncertain evidence and cannot exhibit skepticism when studies present design flaws. Consequently, given the same studies, frontier L.Mss fail to match the conclusions of systematic reviews least ~37% of evaluated cases. We release MedEvidence enable rescarchers to track progress.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
AetMeta. The llama 3 herd of models, 2024, URL http::ss//artt..orrr//////0-0..0
A Meta. The llama herd, 2025.
URL
imoda - inte qence/
Malaikannan Sankarasubbu Ankit Pal. Openbiollms: Advancing open-source large language models for healthcare and life sciences. https://uuggingface.cc Co/aadityaapppeniooLLLLLLLLLL
2024.
Camila Torres Bezerra, Antonio José Grande, Vivianny Kelly Galvão, Douglas Henrique Marin dos Santos, Álvaro Nagib Atallah, and Valter Silva. Assessment the strength of recommendation and quality of evidence: Grade checklist. a descriptive study. Sao Paulo Medical Journal, 140(6); 829-88 836, 2022.
Lutz Bornmann, Robin Haunschild, and Rüdiger Mutz. Growth rates of modern science: latent piecewise growth curve approach to model publication numbers from established and new literature databases. Humanities and Social Sciences Communications, 8(1):224, 2021. doi: 10.1057/
URL
Junying Chen, Zhenyang Cai, Ke Ji, Xidong Wang. Wanlong Liu, Rongsheng Wang, Jianye Hou, and Benyou Wang. Huatuogpt-ol, towards medical complex reasoning with llms, 2024a. URL
https:
abs attp:/bs//22222 18925
Longze Chen, Ziqiang Liu, Wanwei He, Yunshui Li, Run Luo, and Min Yang. Long context is not long at all: A prospector of long-dependency data for large language models. arXiv preprint arXiv:2405. 17915, 2024b.
A Cipriani, T Furukawa, and C Barbui. What is cochrane review? Epidemiol Psychiatr Sci, 20 (3):231-2333 Sep 2011.
Justin Clark, Belinda Barton, Loai Albarqouni, Oyungerel Byambasuren, Tanisha Jowsey, Justin Keogh, Tian Liang, Christian Moro, Hayley O'Neill, and Mark Jones. Generative artificial intelligence use in evidence synthesis: A systematic review. Research Synthesis Methods, pp. 1-19, 2025. doi: 00.1017/sm.20255
DeepSeek-AL. Deepseek-r1: Incentivizing reasoning capability in iims via reinforcement leamingg. 2025a.
12948
DeepSeck-AL. Deepseek-v3 technical report, 2025b. URL 19437
Bradley Efron and Robert Tibshirani. An introduction to the bootstrap. Chapman and Hall/CRC, 1994.
Elicit. Elicit: 15.
The ai research assistant, 2025. URL wtt.htttts//eii...tttt com Accessed: 2025-05-
Nicholas Fabiano, Amav Gupta, Nishaant Bhambra. Brandon Luu, Stanley Wong, Muhammad Maaz, Jess G Fiedorowicz, Andrew Smith, and Marco Solmi. How to optimize the systematic review process using ai tools. JCPP advances, 4(2):e 1223,, 2024.
U.S. FDA. Fda announces completion of first ai-assisted scientific review pilot and aggressive agency-wide ai rollout timeline, May 2025. FDA News Release.
Scott L Fleming, Alejandro Lozano, William J Haberkorn, Jenelle Jindal, Eduardo Reis, Rahul Thapa, Louis Blankemeier, Julian Z Genkins, Ethan Steinberg, Ashwin Nayak, et al. Medalign: A dataset for instruction following with electronic medical records. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 22021-22030, 2024.
Google. Gemini deep research your personal research assistant, 2025. reweaaeorreearchc/eaea
Accessed: 2025-05-15.
Loma K Henderson, Jonathan C Craig, Narelle S Willis, David Tovey, and Angela C Webster. How to write cochrane systematic review. Nephrology (Carlton), 15(6):617-62) Sep 2010.

---

Published as a conference paper at ICLR 2026
YuHe Ke, Liyun Jin, Kabilan Elangovan, Hairil Rizal Abdullah. Nan Liu, Alex Tiong Heng Sia, Chai Rick Soh, Joshua Yi Min Tung, Jasmine Chiat Ling Ong, and Daniel Shu Wei Ting. Development and testing of retrieval augmented generation in large language models- case study report. arXiv preprint 2xXi::2422.01333, 2024.
Lingkai Kong, Haoming Jiang, Yuchen Zhuang, Jie Lyu, Tuo Zhao, and Chao Zhang. Calibrated language model fine-tuning for in-and out-of-distribution data. arXiv preprint arXiv:2010. / 1506, 2020.
Honghao Lai, Long Ge, Mingyao Sun, Bei Pan, Jiajie Huang, Liangying Hou, Qiuyu Yang. Jiayi Liu, Jianing Liu, Ziying Ye, Danni Xia, Weilong Zhao, Xiaoman Wang, Ming Liu, Jhalok Ronjan Talukdar, Jinhui Tian, Kehu Yang, and Janne Estill. Assessing the risk of bias in randomized clinical trials with large language models. JAMA Netw Open, 765):22416677, May 2024.
LangChain. Refinedocumentschain.
URL
reffereence/langghhatti/haaaaaar angchain. refine. her.inee..eiinneeecuuttttccaaaid html Accessed: 2025-05-16.
Jixuan Leng, Chengsong Huang, Banghua Zhu, and Jiaxin Huang. Taming overconfidence in Ilms:
Reward calibration in rlhf. arXiv preprini arXiv:2410.09724, 2024.
Tianle Li, Ge Zhang, Quy Duc Do, Xiang Yue, and Wenhu Chen. Long-conter iims struggle with long in-context learning. URL https://arxiv. org/abx/2404.02000 2024.
Judith-Lisa Lieberum, Markus Töws, Maria-Inti Metzendorf, Felix Heilmeyer, Waldemar Siemens, Christian Haverkamp, Daniel Böhringer, Joerg Meerpohl, and Angelika Eisele-Metzger. Large language models for conducting systematic reviews: on the rise, but not yet ready for use-a scoping review. Journal of Clinical Epidemiology, 181:11 1746, 2025.
Valentin Liévin, Christoffer Egeberg Hother, Andreas Geert Motzfeldt, and Ole Winther. Can large language models reason about medical questions? Patterns, 5(3), 2024.
Yantao Liu, Zijun Yao, Xin Lv, Yuchen Fan, Shulin Cao,, Jifan Yu, Lei Hou, and Juanzi Li. Untangle the knot: Interweaving conflicting knowledge and reasoning skills in large language models, 2024.
URL httpttttt///xiitii
org/ www.////.4444 03577
Alejandro Lozano, Scott I. Fleming, Chia-Chun Chiang, and Nigam Shah. Clinfo. ai: An open-source retrieval-augmeened large language model system for answering medical questions using scientific literature. In PACIFIC SYMPOSIUM ON BIOCOMPUTING 2024, pp. 8-23. World Scientific, 2023.
Alejandro Lozano, Min Woo Sun, James Burgess, Liangyu Chen, Jeffrey Nirschl, Jeffrey Gu, Ivan Lopez, Josiah Aklilu, Austin Wolfgang Katzer, Collin Chiu, Anita Rau, Xiaohan Wang, Yuhui Zhang, Alfred Seunghoon Song, Robert Tibshirani, and Serena Yeung-Lew Biomedica: An open biomedical image-caption archive, dataset, and vision language models derived from scientific literature, 2025. URL
07171
Zheda Mai, Arpita Chowdhury, Ping Zhang, Cheng-Hao Tu, Hong- You Chen, Vardaan Pahuja, Tanya Berger- Wolf, Song Gao, Charles Stewart, Yu Su, et al. Fine-tuning is fine, if calibrated. Advances in Neural Information Processing Systems, 37:136684-1161119220 2024.
OpenAl. Gpt-4 technical report, 2024a. ht
OpenAl. Openai ol system card, 2024b. URL https//aarav..or//bb/////2
OpenAl. Deep research system card, 2025. Accessed: 2025-05-15.
URL
feee--reeearchh-syttenn----
OpenAI, Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K. Arora, Yu Bai, Bowen Baker, Haiming Bao, Boaz Barak, Ally Bennett, Tyler Bertao, Nivedita Brett, Eugene Brevdo, Greg Brockman, Sebastien Bubeck, Che Chang, Kai Chen, Mark Chen, Enoch Cheung, Aidan Clark, Dan Cook, Marat Dukhan, Casey Dvorak, Kevin Fives, Vlad Fomenko, Timur Garipov, Kristian Georgiev, Mia Glaese, Tarun Gogineni, Adam Goucher,
12

---

Published as a conference paper at ICLR 2026
Lukas Gross, Katia Gil Guzman, John Hallman, Jackie Hehir, Johannes Heidecke, Alec Helyar, Haitang Hu, Romain Huet, Jacob Huh, Saachi Jain, Zach Johnson, Chris Koch, Irina Kofman, Dominik Kundel, Jason Kwon, Volodymyr Kyrylov, Elaine Ya Le, Guillaume Leclere, James Park Lennon, Scott Lessans, Mario Lezcano-Casado, Yuanzhi Li, Zhuohan Li, Ji Lin, Jordan Liss, Lily, Liu, Jiancheng Liu, Kevin Lu, Chris Lu, Zoran Martinovic, Lindsay McCallum, Josh McGrath, Scott McKinney, Aidan McLaughlin, Song Mei. Steve Mostovoy, Tong Mu, Gideon Myles, Alexander Neitz, Alex Nichol, Jakub Pachocki, Alex Paino, Dana Palmie, Ashley Pantuliano, Giambattista Parascandolo, Jongsoo Park. Leher Pathak, Carolina Paz, Ludovic Peran. Dmitry Pimenov, Michelle Pokrass, Elizabeth Proehl, Huida Qiu, Gaby Raila, Filipppo Raso, Hongyu Ren, Kimmy Richardson, David Robinson, Bob Rotsted, Hadi Salman, Suvansh Sanjeev, Max Schwarzer, D. Sculley, Harshit Sikchi, Kendal Simon, Karan Singhal. Yang Song, Dane Stuckey, Zhiqing Sun, Philippe Tillet, Sam Toizer, Foivos Tsimpourlas, Nikhil Vyas, Eric Wallace, Xin Wang, Miles Wang, Olivia Watkins, Kevin Weil, Amy Wendling, Kevin Whinnery, Cedric Whitney, Hannah Wong, Lin Yang. Yu Yang, Michihiro Yasunaga, Kristen Ying. Wojeiech Zaremba, Wenting Zhan, Cyril Zhang, Brian Zhang, Eddie Zhang, and Shengjia Zhao. gpt-oss- 120b and gpt-oss-20b model card, 2025. URL arw.v.iiivv................vvv.. org. abs 2500 10925.
OpenEvidence. Open evidence: Ai-powered medical information platform, 2025, Accessed: 2025- 05-15.
James O' D'Dheert,, Cian Nolan, Yufang Hou, and Anja Belz. Beyond abstracts: A new dataset, prompt design strategy and method for biomedical synthesis generation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop), pp. 358-377, 2024.
Mark Petticrew, Paul Wilson, Kath Wright, and Fujian Song. Quality of cochrane reviews. quality of cochrane reviews is better than that of non-cochrane reviews. BMJ, 324(7336):545, Mar 2002.
PubMed.. Entrez Programming Unilities Help [Internet]. Bethesda (MD): National Center for Biotechnology Information (US), 2010- URL
nlm. gov/
books/NBK25500
PubMed. Search strategy used to create the pubmed systematic reviews filter, 2019. URL https www.. nih. wwtphsooooobbooeee pubmed
strategy. html
Riaz Qureshi, Daniel Shaughnessy, Kayden A. R. Gill, Karen A. Robinson, Tianjing Li, and Eitan Agai. Are chatgpt and large language models "the answer"to bringing us closer to systematic review automation? Systematic Reviews, 12(1):72, 2023.
Tim Reason, Emma Benbow, Julia Langham, Andy Gimblett, Sven Klijn, and Bill Malcolm. Artificial intelligence to automate network meta-analyses: Four case studies to evaluate the potential application of large language models. Pharmacoecon Open, 8(2):205-220, Mar 2024.
Irbaz Bin Riaz, Syed Arsalan Ahmed Naqvi, Bashar Hasan, and Mohammad Hassan Murad. Future of evidence synthesis: Automated, living, and interactive systematic reviews and meta-analyses. Mayo Clinic Proceedings: Digital Health, 2(3):361-365, 2024.
Dmitry Scherbakov, Nina Hubig, Vinita Jansari, Alexander Bakumenko, and Leslie A Lenert. The emergence of large language models (Ilm) as a tool in literature reviews: an Ilm automated systematic review. arXiv preprint 24Xiv:2409.046600, 2024.
Nikolas Schopow, Georg Osterhoff, and David Baur. Applications of the natural language processing tool chatgpt in clinical practice: Comparative study and augmented systematic review. JMIR Med Inform, 11:e48933, Nov 2023.
Chantal Shaib, Millicent Li, Sebastian Joseph, lain Marshall, Junyi Jessy Li, and Byron C Wallace. Summarizing, simplifying, and synthesizing medical evidence using gpt-3 (with varying success). In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 1387-1407, 2023.
Zhaochen Su, Jun Zhang, Xiaoye Qu. Tong Zhu, Yanshu Li, Jiashuo Sun, Juntao Li, Min Zhang, and Yu Cheng. Conflietbank: A benchmark for evaluating the influence of knowledge conflicts in Ilm, 2024. URI.tttpss httptttttt/trxxtt www./orggooooo abb 2400. 12076
13

---

Published as a conference paper at ICLR 2026
Fengfei Sun, Ningke Li, Kailong Wang, and Lorenz Goette. Large language models overconfident and amplify human bias. arXiv preprint arXiv:2505.02151, 2025.
ThenThoughts Team. Open Thoughts. https//open-tthuuhtsaaii January 2025a.
Qwen Team.
Qwen2.5 technical report, 2025b. URL /ttp://///aaa...g/aaaa//////
15115
Qwen Team. Qwen3, April 2025c. URL http://ppp///eee.....ii.b.b...b.
Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025d. URL
pettps://qwen.ae wtt..h/.....ooo//oaggaaaa------.
Juraj Vladika, Phillip Schneider, and Florian Matthes. MedREQAL:
Examining medical knowledge recall of large language models via question answering. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, Pp.
Bangkok, Thailand, August 2024a. Association for Computational Linguis- tics. doi: 10. 11511111222222....0 fini2/2//.nnniiggaa....///// URL
Find Andingg-- ac 8607
Juraj Vladika, Phillip Schneider, and Florian Matthes. HealthFC: Verifying health claims with evidence-based medical fact-checking. In Nicoletta Calzolari, Min Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue (eds.), Proceedings of the 2024 Joint In- lernational Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pp. 8095-8107, Torino, Italia, May 2024b. ELRA and ICCL. URL /ttis::/aclanntoooogyyo wwwpeelooyooogg////e.ooo org/ 1rec- main. 7097
Byron C Wallace, Sayantan Saha, Frank Soboczenski, and Iain J Marshall. Generating (factual?) narrative summaries of rets: Experiments with neural multi-document summarization. AMIA Summits on Translational Science Proceedings, 2021:605, 2021.
Alexander Wan, Eric Wallace, and Dan Klein. What evidence do language models find convincing?, 2024. URL /ttpt/t/////xww.////////
Zifeng Wang, Lang Cao, Benjamin Danek, Qiao Jin, Zhiyong Lu, and Jimeng Sun. Accelerating clinical evidence synthesis with large language models. npj Digital Medicine, 8(1):509, 2025.
Eric Wu, Kevin Wu, and James Zou. Finetunebench: How well do commercial fine-tuning apis infuse knowledge into lims? arXiv preprint arXiv:2411.050:: 2024.
Kevin Wu, Eric Wu, and James Zou. Clasheval: Quantifying the tug of between Ilm's internal prior and external evidence, 2025. URL httpp::/acxxv..ogg////// wbtp:///bbb//aeee 10199
Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. Can llms express their uncertainty? empirical evaluation of confidence elicitation in llms, arXiv preprint 33Xi: 2023.
14


Reviews:
[{"review_id": "viSJ7dfCLT", "reviewer": "Reviewer_WvC2", "summary": "The paper introduces MedEvidence, a benchmark of closed-form QA items distilled from Cochrane systematic reviews, pairing each conclusion with the same source studies experts used. LLMs (reasoning/non-reasoning; medical-finetuned/generalist) are evaluated; key findings: reasoning/scale/medical-finetuning don’t reliably help, performance drops with long inputs, models are overconfident and insufficiently skeptical of low-quality evidence. Even frontier models trail a time-constrained expert baseline.", "questions": "Q1) On source concordance -- Beyond using DeepSeek, did you validate concordance against human judgments on a subset to ensure this variable itself isn't model induced? \n\nQ2) For time-constrained expert line (Fig 4a), how many experts participated, how were specialities matched to questions, and what agreement (e.g., κ/ICC) did you observe?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "S1) This is a well-posed task and this paper makes targeted contributions. The paper removes retrieval and long-summary grading by converting conclusions into closed-QA and evaluating exact match answers. \n\nS2) Evaluation is reasonbly transparent with limited uncertainty. Metrics (e.g. per-class recall, accuracy, evidence uncertainty w/ source concordance) reliably support key findings. Work is methodically sound. \n\nS3) Clear empirical takeaways -- I am largely satisfied with their takeaways of frontier models underperforming time-constrained experts, etc.", "weaknesses": "W1) I think perhaps a reasonable weakness/questions here is re conceptual novelty and whether this paper is suited for ICLR. Essentially authors do a dataset+evaluation work with closed-class answers. While useful, it advances prior factuality/evidence-reasoning datasets incrementally (i.e. Table 1) and centers on mapping mapping SR conclusions to a QA rather than introducing new modeling/eval methods. \n\nW2) LLM-derived source concordance. While I think this is a reasonable thing to do for eval/metric purposes, I think there is a potential circularity/model bias element to it. And we're introducing those limitations into a key explanatory variable. \n\nW3) Evaluation pipeline (maybe) order-sensitive. When context overflows, and answers are refined over seqs of artile chunks; I am not sure how randomization effects play into the analysis of main text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZsQF4D0c1P", "reviewer": "Reviewer_zrHx", "summary": "This work presents two contributions: (1) a dataset of systematic reviews from Cochrane, and (2) an assessment of LLM ability to generate a conclusion over the SR evidence strength.", "questions": "1. There is also a difference between general systematic reviews and randomized control trials (RCTs). Since this is sourced from Cochrane, I assume they are all RCTs? Could you confirm this? \n    (a) Often there are established research questions that are specific and provided in these reviews. Why are these questions not being used directly? \n\n2. For the dataset generation, there does not appear to be any assessment of annotator agreement or the quality of the question conversions, apart from the source concordance analysis. However, this measure alone is insufficient, it primarily reflects whether a question can be answered using a single source, which does not capture true evidence synthesis. How is quality is assessed among annotators? \n\n3. Could you clarify the motivation for mapping to *new* labels for the model to synthesize? I am concerned that *“X increases Y”* or *“X may reduce Y”* don’t imply “higher” or “lower” labels, since direction alone doesn’t indicate desirability or certainty of effect.\n\n4. Could you clarify the difference and novelty introduced by your dataset relative to the listed prior works? \n\n(Clarity) L246 typo: should be “use an LLM” not “use an LLMs”", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "This is a well written paper with a scoped contribution that explicitly tests modern LLMs (incl. reasoning models) ability to synthesize evidence across systematic reviews; the analysis is solid and has high relevance for clinical settings where these models may be deployed.", "weaknesses": "My main concern with this work is the lack of adequate comparison with prior work. It’s unclear whether the dataset itself is a novel contribution; there is a lack of interaction with prior work on evaluating LLM evidence synthesis and existing cleaned SR datasets:  \n\n[1] Three datasets built similarly: TrialReviewBench (https://arxiv.org/html/2407.00631v2), https://arxiv.org/abs/2008.11293, and https://aclanthology.org/2024.acl-srw.42/. \n\n[2] There is some prior work that finds similar conclusions regarding overconfidence in LLM responses, and lack of ability to synthesize evidence (https://pmc.ncbi.nlm.nih.gov/articles/PMC11613457/; https://jamanetwork.com/journals/jamanetworkopen/fullarticle/2838106) \n\nI do believe this paper has a good set of experiments over models that benefits the community, especially since it’s clear that these modern models still have the same issues as prior iterations. I am happy to increase my score given the better grounding against prior work and datasets, and justification for why new labels/questions were generated for this task (see questions below).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IFRewAbROF", "reviewer": "Reviewer_4Htb", "summary": "The paper introduced MedEvidence, a benchmark based on human-annotated systematic reviews, to test the LLMs’ ability in question answering based on medical context. Various LLMs are then evaluated and compared on the benchmark.", "questions": "* Line 252-257: The categorization is directly based on DeepSeek, which is not rigorous enough. Also, in the remaining part of the paper, the authors did not utilize the categorization in the evaluation/analysis.\n* Figure 6: It is hard to distinguish the medically finetuned models from the others via the thickness of the margin. Making notes on the y-axis labels can be helpful (for example, adding * for medical models)\n* The results include a few surprising results. For example, reasoning models, larger models, and domain-adapted models do not improve performance. Case studies discussing potential causes for these results can be helpful.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "* All test cases are manually curated based on existing Cochran meta-analyses.\n* A large number of LMs are evaluated on the benchmark.", "weaknesses": "* The soundness of the benchmark is heavily based on the annotation quality, but there is no discussion about the annotators’ background.\n* The only task provided by MedEvidence so far is a multiple-choice/classification task, as the model has to answer one of five given treatment outcome effects. Please see the questions for details.\n* There are no numerical experimental statistics in the main paper. All results are presented in plots.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)