Beyond Classification Accuracy:  Neural-MedBench and the Need for Deeper Reasoning Benchmarks
Authors: Miao Jing
Mengting Jia
Junling Lin
Zhongxia Shen
Huan Gao
Mingkun Xu
Shangyang Li
Venue: ICLR 2026 Poster (2026)
Keywords: vision-language models
benchmark dataset
medical AI evaluation
reasoning-intensive tasks
Primary area: datasets and benchmarks
TL;DR: We introduce Neural-MedBench, a reasoning-intensive benchmark that exposes how state-of-the-art vision-language models fail at clinical diagnosis despite excelling on standard medical AI benchmarks.

Abstract: Recent advances in vision-language models (VLMs) have achieved remarkable performance on standard medical benchmarks, yet their true clinical reasoning ability remains unclear. Existing datasets predominantly emphasize classification accuracy, creating an evaluation illusion in which models appear proficient while still failing at high-stakes diagnostic reasoning. We introduce Neural-MedBench, a compact yet reasoning-intensive benchmark specifically designed to probe the limits of multimodal clinical reasoning in neurology. Neural-MedBench integrates multi-sequence MRI scans, structured electronic health records, and clinical notes, and encompasses three core task families: differential diagnosis, lesion recognition, and rationale generation. To ensure reliable evaluation, we develop a hybrid scoring pipeline that combines LLM-based graders, clinician validation, and semantic similarity metrics. Through systematic evaluation of state-of-the-art VLMs, including GPT-4o, Claude-4, and MedGemma, we observe a sharp performance drop compared to conventional datasets. Error analysis shows that reasoning failures, rather than perceptual errors, dominate model shortcomings. Our findings highlight the necessity of a Two-Axis Evaluation Framework: breadth-oriented large datasets for statistical generalization, and depth-oriented, compact benchmarks such as Neural-MedBench for reasoning fidelity. We release Neural-MedBench at https://neuromedbench.github.io/ as an open and extensible diagnostic testbed, which guides the expansion of future benchmarks and enables rigorous yet cost-effective assessment of clinically trustworthy AI.

Published as a conference paper at ICLR 2026
BEYOND CLASSIFICATION ACCURACY: AND THE NEED FOR DEEPER REASONING BENCHMARKS
Miao Jing l.15..1.5, Mengting Jia 1.2, Junling Lin", Zhongxia Shen', Huan Gao2.6,
Mingkun 2u SShangyang Li
Schooo of Physics Science and Technology.
Beijing University of Posts and Telecommunications Beijing, China "Guangdong Institute of Intelligence Science and Technology, Hengqin, Zhuhai, China "Beijing Chaoyang Hospital, Capital Medical University, Beijing, China "Sleep Medical Center, Huzhou Third Municipal Hospital, Affiliated Hospital of Wenzhou Medical University, Huzhou, China University Macau, Macau, China
Renyixuu Health Technology Co., Ltd, Beijing. China
7 "Academy for Advanced Interdisciplinary Studies, Peking University, Beijing, China
'Euual contribution *Corresponding authors
shangyang 1 foxmai com, xumingkun@gdi int.cn
ABSTRACT
Recent advances in vision-language models (VLMs) have achieved remarkable per- formance standard medical benchmarks, yet their true clinical reasoning ability remains unclear. Existing datasets predominamtly emphasize classification accu- racy, creating evaluation illusion in which models appear proficient while still failing at high-stakes diagnostic reasoning. We introduce Neural MedBBenhh a compact yet ceessonnnn-intensiinn benchmark specifically designed probe the lim- its of multimodal clinical reasoning in neurology. Neural Peural-MedBas BBnchh integrates multi multi-sequence MRI scans, structured electronic health records, and clinical notes, and encompasses three core task families: differential diagnosis, lesion recognition, and rationale generation. To ensure reliable evaluation, we develop a hybrid scor- ing pipeline that combines LLM-based graders, clinician validation, and semantic similarity metrics. Through systematic evaluation of state-of-the-art VLMss includ- ing GPT-4o, Claude-4, and MedGemma, we observe sharp performance drop compared to conventional datasets. Error analysis shows that reasoning failures, rather than perceptual errors, dominate model shortcomings. Our findings highlight the necessity of a Two-Axis Evaluation Framework: breadth-oriented large datasets for statistical generalization, and depth-oriented, compact benchmarks such Neural Mataal-MeeBBaee -MecBench for reasoning fidelity. We release Neural Maatral-MedBaa -MedBench https: /ttp:::/neeuronee hnnpoo/////uuromebenhhh www.hhh..tthh.......... ioo agnostic testbed, which guides the expansion of future benchmarks and enables rigorous yet cost-effective assessment of clinically trustworthy Al.
open and extensible di-
INTRODUCTION
Recent advances in vision-language models (VLMs) have led to striking improvements across a wide range of medical AT tasks. On standard benchmarks such as MedMNIST v2 Yang al. (2023) and MultiMedQA (Singaal et al. (2023), state-of-the-ar models achieve near-human or even superhuman performance label prediction and image-text alignment. These results have created an impression that medical VLMs are nearing clinical readiness. Yet, the clinical reasoning literature has recently underscored Schwartzztein (2024). safe and effective diagnostic practice, especially in high-stakes fields such as neurology, demands more than classification accuracy: it requires multimodal synthesis, ambiguity resolution, and the capacity to justify conclusions in a manner consistent with clinical logic. Current benchmarks, despite their scale, rarely capture these aspects. We argue that this discrepancy

---

Published as a conference paper at ICLR 2026
Depth
MMLLI-ooMMeliaa))
DeVD-Borch (Diugaooc) Naue-MolBerch
Now Mottleoc
C2MQ4
A-161 D-112
HeahhBeoch
SLAKE
VQA-Med
SEEDlock
MMMM
MEGA RENCH
OaniMeddVQA
P
Width
GPT-66
17 1.00m2 5-VL-32D
Claude3.7 Deuhao--istt--pn
Figure I: Conceptual Positioning of Neural-MedBende and Empirical Evidence of the Eval- uation Illusion. (Left) An illustrative mapping of existing medical VLM benchmarks along the proposed axes of Breadth (dataset size and diversity) and Depth (reasoning complexity). Most benchmarks occupy the uation, which Meural-MedBent Iwo Breadth-Axis benchmarks (MMLU-Pro and DrVD-Bench) versus our Depth-Axis benchmark (Neural MedBench). The stark performance drop on Neural-MedBench provides clear empirical evidence of the "evaluation illusion" and the disconnect between the two evaluation axes. Table shows full results.
low-Depth space, leaving critical gap in high-Depth eval- is designed to fill. (Right) Performance of leading VLMs on
creates evaluation illusion, misleading sense of model capability, where strong performance on shallow benchmarks obscures deep weaknesses in reasoning fidelity and clinical trustworthiness.
To address this gap. we propose new perspective on evaluation: Two-Axis Evaluation Framework for medical Al. Along the first axis, Breadth, large-scale datasets assess statistical generalization and coverage across populations and conditions. This where almost all existing benchmarks operate. However, recent work such as Diagnosis Zhu et al. (2025) has shown that relying solely on such breadth-oriented evaluations leads to overestimation of model capability, as they fail to capture true clinical reasoning challenges. Along the second axis, Depth, compact but diagnostically complex, expert-curated benchmarks assess reasoning fidelity, forcing models to integrate multimodal signals, reason under uncertainty, and provide structured justifications, approach that echoes trends in MeddAgentsBench Tang et al.] (2025). Importantly, we hypothesize that these axes are largely uncorrelated: success breadth benchmarks does not guarantee competence in depth scenarios. This hypothesis is supported by findings that even expert LLMs struggle with self-awareness and metacognitive reasoning challenging cases, as shown in MetaMedQA Griot et al. (2025). A complete picture of model readiness therefore requires evaluation along both dimensions.
To operationalize the Depth axis in clinical neurology, we introduce Meural-MedRencee agnostic benchmark deliberately designed as a "stress test". Neural-Med expert-annotated multimodal cases, including multi-sequence MRI scans, structured electronic health records (EHRs), and clinical narratives, yielding 200 reasoning-intensin tasks. Its design mirrors practices in medical education, such as Objective Structured Clinical Examinations (OSCEs), where carefully constructed cases are used to assess reasoning skills under uncertainty Geathers et al.) (2025): Siegelman et al. 2024). Unlike prior resources focused on breadth, Neural -MedBench is compact by design, prioritizing reasoning density over volume, enabling cost-effective yet high-signal evaluation of clinical reasoning.
di- comprises 120
Our evaluation of leading models, including GPT-4o, Claude 4, and MedGemma, reveals a striking pattern: models that excel on large-scale benchmarks fail systematically on Error analysis shows that these failures stem not from perception lexical mismatch, but rather from breakdowns in clinical reasoning, limitations that remain hidden under existing evaluation paradigms. This disconnect provides the first empirical evidence for the independence of the two axes, underscoring the urgent need for benchmarks that explicitly capture reasoning depth.
This work makes the following contributions:
2

---

Published as a conference paper at ICLR 2026
We propose the Two-Axis Evaluation Framework, arguing that trustworthy clinical Al requires complementary assessments of both breadth (statistical generalization) and depth (reasoning fidelity).
We release signed to operationalize the Depth axis, comprising 120 multimodal, expert-eurated diag-
the first neurology-focue benchmark explicitly de-
nostic cases with 200 reasoning-intensive tasks.
We provide empirical evidence of the disconnect between breadth and depth, showing that state-of-the-art VLMs fail primarily at reasoning, despite strong performance on existing large-scale datasets.
We present a systematic error analysis and a human performance baseline to contextualize model results, and we release Neura -eedBBncch an open, extensible resource with public leaderboard and roadmap for expansion.
RELATED WORK
2.1 MEDICAL VLM BENCHMARKS
A wide range of benchmarks have been developed for evaluating medical VLMs. Early datasets such as ChestX -ray Wang et al. (017), CheXXert IIriin al. (2019), and MIMIC-CXR
et al. (2019) primarily enabled large-scale classification and report generation. Organ- or disease- LUNA for pulmonary nodules Setio al. (2017). LiTS for liver tumors Bilic et al. (2019), BraTS for brain tumors Menze al. (2015), and ISLES for stroke Maier al.
specific challenges-incllnne
evaluation to segmentation and lesion characterization. More recent mul- timodal benchmarks target richer tasks: ROCO v2 Rückert et al. (2024) for radiology captioning, RadGraph Jain et al. (202I) for entity-relation extraction, and McdFMC Wang et al. (2023) for federated multi-center classification. VQA-style benchmarks, such VQA-RAD
al. (201g) 8),
VQt He et al.) (2020), and OmniMedVQA Hu et al.] (2024), test text- imtg-imae reasoning but often rely on templated questions, limiting their coverage of authentic diagnostic challenges. Recent meta-beecchmars (e.g.. Rad-Bench Kuo et al. (2024), DiagnosisArenal Zhu et al. (2025), MedAgents- Bench Tang et al. (2025) have emphasized comprehensiveness, yet remain breadth-oriented: they focus on scale and diversity rather than probing reasoning under uncertainty.
In contrast, MeurallMedBBnnn explicitly operationalizes the Depth axis of evaluation. By curating 120 multimodal, diagnostically complex neurology cases into 200 iissoinnginttnsive tasks, it emphasizes reasoning fidelity over volume. This design mirrors clinical examinations such as OSCEs, and captures underexplored challenges like multi-lesion differential diagnosis, contextual interpretation, and longitudinal integration across MRI sequences.
2.2 VLM EVALCATION METHODOLOGGIES
Medical VLMS typically evaluated via three strategies: reference-based, reference-free, or hybrid metrics. Traditional reference-based methods include BLEU,, ROUGE, and CIDEr for surface overlap, as well semantic similarity scores such as BERTScore Zhang et (2019). CLIPScore Hessel et al.
(2021) and domain-specific adaptations like CheXbert Smit et al. (020) or RadGraph-F1 Jain et al.
(2021) While scalable, these methods cannot assure that generated outputs are clinically grounded.
Reference- frrencee-feeeeeee approaches, such as saliency attribution (e.g., Grad-CAM Selvaraju et al. (2020)
or image-text text alignment tests, serve as indirect proxies for reasoning but lack explanatory power. More recent frameworks have introduced LL.M-based grading systems for evaluating factual accuracy and reasoning plausibility
free-form clinical responses. For example, the Expert-of- Verification and Alignment (EVAL) framework streamlines clinician-led evaluation in bleeding scenarios Giuffrè et al. (2025), while LLMEval-Mer combines expert-designed checklists with LLM- as-Judge dynamics improve reliability in medical QA and reasoning tasks Zhang et al. (2025) Zhi et al. (2025) Fang et al. (2026): Zheng et al.) (2026). Despite their automation advantages, these approaches remain subject ongoing reliability validation.
employs a hybrid evaluation strategy: semantic fidelity is quantified using BERTSccoee, while clinical reasoning scored via an M-be-b evaluator calibrated with neurologist.
3

---

Published as a conference paper at ICLR 2026
Expert Curation
Data
Validation
Baagook
JAMA
ADNI
Radinlagy
Gemini
aimtigneuus presstiations
Low
Radiopaedia...
Couple
es
Phodos
mult step reasocing
Lark Type
Demandayici
Diler
vead
Clociial
Others
syndromes
Voc
Neural MedBench
Pooling Filtering
Evaluation
meeking eook
Model Test
+ Ouers
Faltaa SPPT
1.
? tee
Cizade
CONST
Dig nigr
the Cuee Aeproeo
Faer Premoion
Mormmi Tea
Melled Madii 1o
Leve
Level 2.83
A&M
Figure 2: Overview N'Nurra-MedBenchh The workflow begins with pooling and filtering data from diverse clinical sources, followed by a multi-stage expert curation and model-based validation pipeline to select for diagnostically complex cases. The resulting benchmark is used to test a cohort of models, whose outputs are assessed via hybrid evaluation suite including accuracy, semantic similarity, and a clinician-validated LLM grader.
review. This design allows scalable yet clinically faithful assessment of both "what" is predicted and "why," exposing nuanced failure modes that conventional metrics tend to overlook.
NEURAL-MEDBENCH
Design Philosophy and Motivation. As established in our proposal for a Two-Axis Evaluation Frame- work, current medical benchmarks overwhelmingly emphasize Breadth, rewarding statistical general- ization but failing to probe deep, integrative reasoning. To address this gap, -MarralM-edBBccc was conceived operationalize the Depth axis. Its design philosophy is therefore fundamentally different from large-scale datasets.
Instead of prioritizing data volume, we prioritize reasoning density. Our goal was to create a compact but diagnostically rich "stress test," inspired by high-stakes clinical examinations like OSCEs, where the ability to synthesize, reason, and justify paramount. This approach, illustrated Figure [ allows for rigorous cost-efficient evaluation of the core clinical reasoning capabilities that remain anser-dd by existing resources.
Case Sources and Curation. We curated 120 authentic neurology cases from multiple rigorously vetted sources: ADNI and OASIS (research cohorts), Radiopaedia (expert-verinf imaging cases), and peer-reviewed case reports (e.g.. JAMA Neurology). These sources span both common diseases (e.g., Alzheimer's disease, ischemic stroke, epilepsy) and rare or diagnostically complex conditions (c.g., autoimmune encephalitis, CNS infections).
The final 120 cases were selected from an initial pool of over 2,000 candidates using funnel- shaped, multi-stage pipeline: (1) Initial screening retained only cases with sufficient multimodal completeness (e.g., imaging. neuropsychological scores, patient histories); (2) Expert curation by two senior neurologists and neuroradiologist, who reviewed cases for plausibility, diagnostic complexity, and educational value; (3) Annotation of ground-trutte answers, including final diagnosis, differential diagnoses, lesion characterization, and explanatory reasoning: (4) Consensus review and
4

---

Published as a conference paper at ICLR 2026
challenge validation, where disagreements were resolved through discussion, and trivial cases filtered out using baseline models, ensuring each retained case posed a meaningful diagnostic challenge.
This rigorous process ensures that ground truths are not merely single labels but structured narratives that model how neurologists articulate reasoning. Each case supports on average reasoning tasks (200 tasks in total), spanning direct diagnosis, complex disease inference, and multi-turn dialogue. This compact yet high-density design prioritizes diagnostic richness and evaluation efficiency, enabling fine-grained reasoning assessment without the prohibitive annotation and compute costs of large-scale datasets.
Task Families and Difficulty Stratification. From the curated cases, we derived 200 reasoning- intensive tasks, grouped into three core families:
Task Family
Description
Differential Diagnosis
Given imaging and patient history, models must provide a ranked diagnostic hypothesis with justification.
Lesion Recognition Rationale Generation
Identify lesion type and location, testing multimodal spatial reasoning. Generate explanatory reasoning for diagnostic choices, mirroring case discussions or board exams.
To probe model reasoning at different depths, these tasks are further stratified into three difficulty levels (see examples in Fig. 3):
Level (Direct Diagnosis): Involves straightforward cases with classic, unambiguous signs primarily in modality (e.g., clear tumor on an MRI). This level tests core pattern recognition.
Level 2 (Complex Diagnosis): Features cases with ambiguous or conflicting evidence that require integrating information from at least two modalities (e.g., linking subtle imaging findings with specific details from the clinical history) to resolve the uncertainty.
Level 3 (Iterative Diagnosis): Simulates a multi-turn clinical consultation where the model must progressively refine its diagnosis by interpreting new information. This tests its ability to dynamically adjust its reasoning chain.
This design provides controlled lens into how models scale from factual recall to higher- ooder clinical reasoning.
Evaluating deep clinical reasoning is notoriously resource-intensive. To address this, Neural -Murral-eedBnch introduces a scalable, hybrid evaluation pipeline that is rigorously validated by clinicians, but fully automatable for community use. The protocol consists of two stages:
Evaluation Protocol.
Stage 1: Grader Validation. The core of our protocol is a dedicated LLM--ased grader, guided by detailed, neurology-specific rubrics. To ensure this automated grader serves reliable proxy for expert judgment, we performed rigorous clinician -eniiiinniin-hhe--ee -n-tthe--ttooopppp- validation. A panel of board- certified neurologists independently scored a large subset of model outputs using the same rubric. We found very high correlation between the LLM grader's scores and the consensus of human experts (e.g., Pearson's r > 0.9), scientifically validating our grader robust instrument for automated assessment.
Stage 2: Automated Community Evaluation. Following this one-time, intensive validation, the cliically-calibrate LLM grader is released as part of the benchmark. This allows any researcher to evaluate new model in a fully automated fashion, without requiring access to clinicians. Users can simply run their model on Neural- Mourrl-Medder -MedBench and use provided, validated grader to obtain scores that are highly correlated with expert neurological assessment.
To further contextualize results from this automated pipeline, also establish human performance baseline by evaluating clinicians of varying expertise on the benchmark. This crucial addition anchors all model scores, providing a clear, realistic measure of the gap between current AI capabilities and human-leve clinical reasoning. This two-stage protocol thus offers the best of both worlds: the gold-standard rigor of clinician validation and the scalability of automated evaluation.
5

---

Published as a conference paper at ICLR 2026
Level
Level 2
Level 3
wode, rlack right The pokeal bat legrression topelipthenss, diandice melitta, and scall libillatonn Naundogic ecancration revssed discorrazoon ster-oem mersory incarmunn, usa sagitaikes. Canbrratta Mose cdii cocat of cclbyii....
scafiutio wad difiealies.
iepuitmen,, and faher copertedinn havd
pregresod panudon howde
Fatemall lut MIA relurtt okema.
coguative kepeiimerr The prttent viking coniiattous with tood of moli eiren cooentad in single
Pokest Data
Aac: 15 yaa
Repoeted corrert tee of aeassendde nist way exfeermsory daago assecagalada lada in lacks dings talogeoooth, aik-oppereed
Grxder Mple
Qr: Wiasr is the dkkey dlaggnnss bened dests lmaginy?
dlevnee white
addiicn, had hakkyy of peperieniess hosteyy of ducheta, hout desses or atsic and had nocert Lesotrii
eikass?
tisest helatice, had bod repattal....
What noeds be done
isaglng
cays
prissory fooils
118, and sthjects Meed pressue (atting desntii prestar was 6l. Aucidy occured the pad fow meets GDS tone scare war on
Sabbert slaad
(osingg cystali
Reference
Reference
Ash aestilppis
.MI shewe lopprinnase vepprnprrirrll regien with introuee 1977 rigual
lenppnngariittl leslons
right
libzlyy fives peisacy, givce bonggnsonsly cndoecing in intreooust rass with sarmunsinng weegens: edcna da eldelft patlorr
brats metattade
The
with
regltn,
(decccael oppereai difiaace socfliaent, low peleptiiley-eeee maaging sigall, rurrased egral
and
Diegnose the patienr's
ssagoral lobe. CNS cnlanning loamn ecosagauiao isere mnd Liwe dgsal and sacre spparesi diffacce coethoeet The panertt claccal veaging festoes comiittaa ond for forther
strirr XIII testres
indicatin vassapale edemaa
Reference
The leila more comiitees with scclsttaai
Abfoitee''s Olsaae
unyled capopeetay
a bemagemooely catancing centical aunica/ maso with vaagasiic
bitskied
Figure 3: Examples of questions and their reference answers in Neural Meural-MedBench, illustrat-
ing typical reasoning tasks.
Dataset Analysis. Table I summarizes task statistics. Text inputs were normalized in length to control for spurious effects of input size model performance. Imaging data include balanced representation of T1, T2, FLAIR, and CT modalities. Each difficulty tier contains both common and rare pathologies, ensuring robustness tests beyond frequency bias
Table
Text Lengths and Image Pixels by Subset
Subset
Token Chinese
Token English
Image Pixel (x10,000)
Level 1
374
225
$35
Level 2
504
309
145
Level 3
67
37
1 1147
Summary. By integrating multimodal data, stratified diagnostic difficulty, structured reasoning targets, and a cost-efficient hybrid evaluation protocol, -Meral--edBenode operationalizes the Depth axis of clinical AI evaluation. complements large-scale breadth benchmarks with a compact, reproducible, and extensible testbed for probing reasoning fidelity, advancing toward clinically trustworthy multimodal AI.
4 EXPERIMENTAL SETUP
Our experimental setup is designed to comprehensively assess the clinical reasoning capabilities of state-of -fateoof-the-art VLMs on Neural-Me and to contextualize their performance against realistic human baseline.
4.1 MODELS BENCHMARKED
We evaluate 16 representative VLMs spanning three categories, all in zero-shot setting to probe intrinsic reasoning capabilities without task-specific fine-tuning:
We benchmark three categories of models provide comprehensive view of current VLM capabili- ties. First, we include proprietary frontier systems with strong multimodal reasoning ability, namely GPT-Series ? and Gemini Comanici et al. (2025), which serve reference baselines. Second, evaluate powerful open-source generalist models such as Claude Anthropic (2024), Doubao Team (2024). Twen-VL.-Plus Bai al. (023). Finally, assess the effect of domain-specifie adaptation,

---

Published as a conference paper at ICLR 2026
benchmark medical-speciaalzed VLMs including LLaVA-Med Li et al. (2023). RadFM Wu et al.
(2023). Huatuo-GPT--ision Chen et al. (2024), Med-Flamingo Moor et al. (2023) Lingshu LASA Team et al. (2025) and MedGemma Sellergren al. (2023), which represent current efforts to tailor
VLMs for clinical applications. full list of models, versions, and hyperparameters provided in
Appendix C
4.2 HUMAN PERFORMANCE BASELINE
To establish a realistic anchor for task difficulty, we conducted a human evaluation with two clinician groups: Medical Students (n = 5) and Senior Physicians (n = 5). Each participant completed the entire set of 200 tasks, blinded to both model outputs and gold-standard answers. All responses were scored using the same evaluation protocol as applied to the models, ensuring a direct and fair homen-model comparison.
4.3 TASK AND PROMPTING STRATEGY
To simulate clinical consultations, we adopt a structured role-prompting approach. Each model initialized "experienced neurologist" to encourage adherence to medical reasoning and terminology. Tasks are presented in full, combining textual narratives, structured patient history, and imaging (MRI/CT encoded base64). Full prompt templates are detailed Appendix B
4.4 EVALUATION METRICS
Given the multi-faceted nature of clinical reasoning, we employ a suite of complementary metries:
Diagnostic Accuracy (pass@k): k): For tasks with definitive outcomes, we report top-1 (pass @ I) and top-5 (pass@5) accuracy. The latter measures whether the correct diagnosis appears within model's differential.
Semantic Fidelity (BERTSccore): Used for free-form rationale generation, capturing semantic alignment between model outputs and expert references. We use it only secondary similarity indicator.
Reasoning Fidelity (LLM Grader): A clinically calibrated GPT-4o grader evaluates correct- ness, logical coherence, and evidence grounding (see Section
Error Taxonomy: To understand failure modes, incorrect responses were manually annotated by neurologists into five categories: Perceptual Failure, Reasoning Failure, Knowledge Cap. Grounding Error, and Visual Hallucination.
This combination of automated metrics, human baselines, and error analysis provides a rigorous and clinically grounded evaluation protocol.
EXPERIMENTAL RESULTS AND ANALYSIS
Our experimental analysis is structured to answer three central questions: (1) How do state-of-the-art VLMs perform on Depth-Axis benchmark like Neural -MedBennh,, and does this performance align with their success Breadth-Axis benchmarks? (2) How large is the performance gap between these models and practicing human clinicians? (3) What are the primary underlying reasons for model failures in complex clinical reasoning?
5.1 OVERALL PERFORMANCE: A STARK DISCONNECT BETWEEN BREADTH AND DEPTH
We evaluated diverse cohort of VLMs on all 200 tasks in Neural -MedBench, with full results in Table 2 The findings reveal a profound difficulty that stands in stark contrast to conventional bench- marks. Across all models, pass@) accuracy remains strikingly low, with even the gop-erforming specialized model MedGemma-22 2-B-it only reaching 30.0% on the simplest Direct Diagnosis tasks. Performance plummets
complex subsets, averaging below 15% for most models, stark contrast to the Senior Physician's 's performance of over 35% on the same tasks. Allowing for five attempts (pass@55) improves scores, with the generalist model Gemini 2.5-Pro achieving the highest
7

---

Published as a conference paper at ICLR 2026
score of 50%. However, model consistently surpasses the 50% mark, indicating that the correct diagnosis often not even within their top considerations.
Table 2: Evaluation metrics across different models and tasks
Direct diagnosis
Complex diseases
Multi-round dialogue
Models
1 [%) |Ss]
BertSowre 1/4 (a)
pascon5 (%]
BertScore passie 1%]
passie5 [%]]
Base VLMs
GPT-5
36.7 (11)
43.3 (13)
0.80
28.3 (17)
45.0 (27)
0.81
19.5 (39)
27.5 (55)
GPT-4o
20.0 (6)
36.7 (11)
0.70
8,3 (5)
40.0 (24)
0.73
8,5 (17)
16.5 (33)
Gemini 2.5-Pro
30.0 (9)
50.0 (15)
0.77
15.0 (9)
38.3 (23)
0.72
11.5 (23)
19.5 (39)
Gemini 2.5-Flash
26.7 (8)
46.7 (14)
0.76
13.3 (8)
35.0 (21)
0.68
10.5 (21)
18.5 (37)
General
VLMs
Gemini 2.0-Flash
20.0
30.0 (9)
0.68
11.7 (7)
23.3 (14)
0.67
918)
16.0 (32)
Claude 4.0-Sonnet
16.7 (5)
43.3 (13)
0.78
13.3 (8)
31.6 (19)
0.77
6.5 (13)
18.0 (36)
Claude 3.7 Sonnet
16.7 (5)
26.7 (8)
0.73
8.3
31.7 (19)
0.66
7.5 715)
12.0 (24)
Claude 3.5 Sonnel
6.7 (2)
16.7 (5)
0.68
(5)
18.3 (11)
0.66
7.4)
10.5 (21)
Qwen-VL-22
10,0 (3)
30.0 (9)
0.79
5.0
21.7 (13)
0.69
(8)
10,0 (20)
Doubao-I.3-visionn
6.7 (2)
40.0 (12)
0.64
10.0
13.3 (8)
0.63
5.5 (11)
11.5 (23)
Medical VLMs
LLaVA-Med
10.0 (3)
16.7 (5)
0.66
10.0 (6)
28.3 (17)
0.69
(12)
13.0 (26)
MedGemma
30.0 (9)
36.7 (11)
0.80
18.3 (11)
38.3 (23)
0.75
(21)
(315
Lingshu
26.7 (8)
40.0 (12)
0.75
21.7 21.7
35.0 (21)
0.73
8.7)
(400
RadFM
(.)
20.0 (6)
0.62
3.3 (2)
13.3 (8)
0.67
(55
6.0(12)
Med-Flamingo
0.0 (0)
15.7
0.67
3.3
10.0
0.66
(35
10.0 (20)
HuatuoGPT
10.0
(6)
0.60
5.0
13.3 (8)
0.68
3.0
7.4
Human
Medical Student
3.3 (1)
3.3 (2)
6.2
Scnior Physician
40.0 (12)
35.5 (21)
15.0 (30)
This stark performance drop provides the first empirical evidence for our central hypothesis: the two evaluation axes (Breadth and Depth) are largely uncorrelated. To visualize this "evaluation illusion," Figure Souureecootttn contrasts the high scores on a typical Breadth-Axis benchmark against the low scores Neura 1 -MedBen ch, demonstrating that success in pattern recognition does not translate to competence in deep clinical reasoning.
5.2 THE GAP TO HUMAN EXPERTISE: A SOBERING REALITY
To provide crucial, real-world anchor for model performance, we benchmarked leading VLMs against both the ideal Ground Truth (expert consensus) and realistic human baselines. The results, visualized in FigureFLeet), reveal sobering chasm. On Direct Diagnosis tasks, the top-performing VLM, MedGemma, reached 30.0% pass@) accuracy. falling 10 percentage points short of the Senior Physician's 's 40.0% and barely surpassing Medical Student. This performance gap widens dramatically in the Complex Diseuses subset, where the Senior Physician's 's performance (35.5%) is nearly double that of MedGemma (18.3%). This quantitative and clinically meaningful gap validates benchmark's ss difficulty and underscores the immense distance to human-level clinical competence.
The performance gap between medical students and VLMs reflects distinct reasoning paradigms. Students' lower pass@1 scores stem from clinical training that prioritizes broad differentials avoid premature closure, whereas VLMs benefit from style pattern matching for decisive answers. However, students better in multi turlt--tummaa dialogues due to superior metacognition the ability to self-correct using new evidence. contrast, VLMs exhibit significant anchoring bias, often failing to revise initial hypotheses even when presented with disconfirming clinical data.
8

---

Published as a conference paper at ICLR 2026
Ground
Senor
GFT-4p
VacGenica
9%
100
Reasering Perceptual Falture
Visual Hallzziattion Knowledge Sap
25
27%
The Performance Gap to Human Expertise and Systematic Diagnosis of Model Failures. (Left) direct comparison of pass@1 accuracy. The "Ground Truth" represents the expert consensus answer. The "Senior Physician" bar shows the performance of board-certified neurologist a blinded test, providing a realistic human expert baseline. All evaluated VLMMs perform significantly below this realistic baseline, widening the gap as task complexity increases. (Right) Distribution of primary error types from a systematic analysis of incorrect model responses. Reasoning Failure the dominant cause of error, suggesting the primary bottleneck is cognitive, not perceptual.
Figure 4:
5.3 WHY Do MODELS FAIL? A SYSTEMATIC ERROR ANALYSIS
To diagnose the root cause of this performance gap, we performed systematic analysis of 100 incorrect model responses. The distribution primary enror types (Figure Right)) reveals that the primary bottleneck for current VLMs is not perception, but reasoning. A striking 51% of failures were classified as Reasoning Failures, where models correctly identified key findings but failed synthesize them into correct diagnosis. This rate is nearly double that of Perceptual Failures (27%). This key finding allows us to define the "lower bound" of current VLM capabilities. Our results suggest this capability floor is set not by lack of perceptual acuity, but by fundamental deficit in integrative clinical reasoning. This directly explains the observed disconnect between high linguistic fluency and low diagnostic accuracy, highlighting the necessity of Depth-Axis benchmarks for diagnosing and ultimately fixing these core reasoning deficiencies.
5.4 ADDITIONAL FINDINGS
Foundation Models vs. Medical-Specializze Models. Our results reveal a nuanced relationship between generalist and specialist VLMs. On single-best-guess accuracy (pass@ I), the medical- specialized MedGemma-22BBitts is good, demonstrating the value of domain-specific fine-tuning for diagnostic precision. However, on differential diagnosis generation (pass@5), large generalist models like Gemini 2.5-Pro retain edge, likely due to their broader generative capabilities. This suggests that optimal clinical AI may require a synthesis of both deep specialization and generative breadth.
Evaluation Efficiency. As a Depth-Axt Axis benchmark, Neural -MedBench is deliberately de- signed for high signal-to-noise ratio and evaluation efficiency. Our diagnostic prompts are carefully engineered to elicit rich, multi-faceted reasoning from compact set of cases. As summarized in Table this high-density design reduces inference token costs by over an order of magnitude (10x) compared to large-scale Breadth-Axis benchmarks like GMAI-MMBench Ye et al. (2024) and Omn- iMedVQ.A Hu et al. (2024), while maintaining high diagnostic difficulty. This cost-effictivvenes is a key feature, it lowers the barrier for academic labs and enables practical, multi-sample robustness analyses (c.g., via temperature sweeps), which are often computationally prohibitive on larger datasets but essential for assessing model reliability.

---

Published as a conference paper at ICLR 2026
Table 3: GPT-4o performance on different benchmarks
Benchmark
Number of Images
Cost of Image Tokens
Passing Rate (%)
GMAI-MMBench
12K
$30.00
53.96
OmniMedVVQA
128K
$320.00
29.74
Beuaal-MeBBench
IK
$2.50
9.67
LIMITATIONS AND FUTURE WORK
While Neural Meuural-MedBerl -MedBench was deliberately curated for diagnostic depth, we acknowledge several limitations its current composition. First, the total number of tasks (200) is modest, reflecting a deliberate trade-off for the high-density, expert-intensive annotation required for reasoning evaluation. Second, the distribution of conditions is biased toward diseases with pronounced imaging correlates (e.g., stroke, tumors), with less representation of disorders that are primarily functional metabolic. Third, although cases are drawn from multiple sources, the benchmark does not yet systematically capture the full spectrum of domain shift across hospitals, scanners, reporting conventions. Fourth, the benchmark focuses on depth-orien depth-oriented reasoning via multimodal and multi-field integration, leaving context-length robustness to complementary benchmarks. Finally, the current release represents only an initial stage of the benchmark, which will be progressively expanded over time.
These considerations underscore that Neural-MeBBench is best interpreted as a high- resolution stress test for reasoning fidelity, not a comprehensive benchmark for statistical generalization. To address these limitations, future iterations will continuously grow both scale and disease spectrum, with particular emphasis on underrepresented conditions, longitudinal follow-up, and multi-center variability, ensuring that the benchmark remains a continually evolving resource for the community.
CONCLUSION
In this work, we challenged the prevailing paradigm of evaluating medical VLMs through the lens of classification accuracy. We argued that this approach creates an "evaluation illusion," and proposed more complete Two-Axis Framework that mandates complementary assessment of both Breadth (statistical generalization) and Depth (reasoning fidelity). To operationalize this missing Depth axis, we introduced Neural-MedBench,, compact, diagnostically complex benchmark for clinical neurology. Our extensive experiments yielded a sobering conclusion: a model's 's success on large- scale, Breadth-Aae -hiid benchmarks is poor predictor of its ability to perform deep clinical reasoning. We found that even state-of-the-ar models are consistently outperformed by human clinicians and that their failures stem primarily from fundamental breakdowns in reasoning, not perception.
While Neural -Muural-MedBechh provides a crucial new perspective, we acknowledge its limitations. Its intentionally compact scale restricts broad statistical claims, and its current focus is on diagnostic reasoning rather than the full clinical workflow. The sample size for our human baseline, while offering vital insights, also modest, These limitations, however, illuminate clear and exciting roadmap for future work. The next generation of Depth-Axis benchmarks must expand in both scale and clinical scope, incorporating more diverse and underrepresented conditions, longitudinal data, and multi-center variability to systematically study domain shift. Notably, Neural 1araal-eesn -Med is designed as continually evolving resource: the present set of cases and tasks reflects the state at submission, and its scale and coverage will be continuously extended.
Ultimately, N -edBBnch is more than a dataset; it is a diagnostic tool for the field itself, providing reproducible methodology to probe the core reasoning capabilities of our most advanced models. By offering this high-resolution lens and committing to its ongoing evolution, we aim to shift the community's focus from narrow pursuit of accuracy towards the development of AI systems that possess genuine, trustworthy clinical reasoning. This benchmark serves as a foundational step in that critical journey.
10

---

Published as a conference paper at ICLR 2026
8 ETHICS DETAILS
Data used in this study were obtained from the Alzheimer's Disease Neuroimaging Initiative (ADNI) database (adni.loni, usc.uus,.dd)), the OASIS dataset, Radiopaedia, and case reports from articles published in The Journal of the American Medical Association (JAMA). All data were accessed with prior approval and are fully de-identified.
ADNI: Investigators within ADNI contributed to data collection but did not participate in the analysis or writing this manuscript. A comprehensive list of ADNI investigators available at: ADNI Acknowledgement List. Data collection and sharing for ADNI were funded by the National Institute on Aging (U01 AG024904), the Department (W8 LXWH-12-2-00122 and other sponsors listed on the ADNI website. This manuscript was reviewed by the ADNI Data and Publications Committee prior to submission.
Defense
OASIS: portion of the OASIS dataset was restructured comply with privacy protection protocols, ensuring that no personally identifiable information was included. Throughout the process, we adhered strictly to OASIS's privacy guidelines, safeguarding participant confidentiality.
Radiopaedia: Radiopaedia data used in this study are provided under the Creative Com- mons Www--ooonommriall-SSaaa were utilized for non-comen commmearraaal purposes with proper attribution to Radiopaedia and are redistributed under the same licensing terms.
3.0 (CC-BY-NC-SA 3.0) license. The data
REPRODUCIBILITY STATEMENT
All experiments in this study based our -Medraeh dataset. Its sources, filtering, and construction process are described in Appendix A. The full dataset is publicly available on Hugging Face http://wuginngaaa.e..vtttttsssss Bww.e//eeeennneeeeeeeeeeeeeeeed for reuse and verification.
10 ACKNOWLEDGMENTSS
This work supported in part by the Government Special Support Funds for the Guangdong Institute of Intelligence Science and Technology; in part by the Beijing Renyixun Health Technology Co., Ltd.; part by the Young Scientists Fund of the National Natural Science Foundation of China under Grant 62506084 (M. Xu); and in part by the Young Scientists Fund of the National Natural Science Foundation of China under Grant 32500997 (S. Li).
REFERENCES
Anthropic. Claude haiku, 2024. URL httpp::/www..atttrrpic.cco
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2388.12966 2023. URL httts:://arx:
org//bs/ 2308. 12966
Patrick Bilic, Patrick F Christ, Eugene Vorontsov, Grzegorz Chlebus, Hao Chen, Qi Dou, Chi-Wing Fu, Xu Han, Pheng-Ann Heng, Jürgen Hesser, et al. The liver tumor segmentation benchmark (lits). arXiv preprint arXiv: 1901.04066, 2019.
Junying Chen, Chi Gui, Ruyi Ouyang, Anningzhe Gao, Shunian Chen, Guiming Hardy Chen, Xidong Wang, Ruifei Zhang, Zhenyang Cai, Ke Ji, Guangjun Yu, Xiang Wan, and Benyou Wang. Huatwogpt-vision, towards injecting medical visual knowledge into multimodal 1lms at scale, 2024.
URL httpttttss//axi....ttm org/ abs/ 62../2406..222
Gheorghe Comanici. Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.0626) 2025.
11

---

Published as a conference paper at ICLR 2026
Tao Fang, Jiayan Guo, Mingkun Xu, Yuanzhe Zhao, and Shangyang Li. Medsimsearch: Sim2real agentic learning for medical visual reasoning, 2026. URL
net/
[
Jadon Geathers, Yann Hicke, Colleen Chan, Niroop Rajashekar, Sarah Young, Justin Sewell, Susannah Cornes, Rene F Kizilcec, and Dennis Shung. Benchmarking generative ai for scoring medical student interviews in objective structured clinical examinations (osces). In International Conference on Artificial Intelligence in Education, pp. 231-245. Springer, 2025.
Mauro Giuffrè, Kisung You, Ziteng Pang, Simone Kresevic, Sunny Chung, Ryan Chen, Youngmin Ko, Colleen Chan, Theo Saarinen, Milos Ajcevic, et al. Expert of experts verification and alignment (eval) framework for large language models safety in gastroenterology. npj Digital Medicine, 8(1): 242, 2025.
Maxime Griot, Coralie Hemptinne, Jean Vanderdonckt, and Demet Yuksel. Large language models lack essential metacognition for reliable medical reasoning. Nature communications, 16(1):642. 2025.
Xuehai He, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie. Pathvqa: 30000+ questions for medical visual question answering. arXiv preprint arXiv:2003. 10286, 2020.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference- free evaluation metric for image captioning. arXiv preprint wwxiv:2/04.087// 2021.
Yutao Hu, Tianbin Li, Quanfeng Lu, Wenqi Shao, Junjun He, Yu Qiao, and Ping Luo. Omnimedvqa: A new large-scale comprehensive evaluation benchmark for medical Ivlm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 221 22170-2218.. 2024.
Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-lleus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, et al. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. Proceedings of the AAAI Conference Artificial Intelligence, 33(00)5590-555 2019.
Saahil Jain, Ashwin Agrawal, Adriel Saporta, Steven Q H Truong, Du Nguyen Duong, Tan Bui, Pierre Chambon, Yuhao Zhang, Matthew P Lungren, Andrew Y Ng, et al. Radgraph: Extracting clinical entities and relations from radiology reports. arXiv preprint arXiv:2106. 14463, 2021.
Alistair E W Johnson, Tom J Pollard, Seth J Berkowitz, Nathaniel R Greenbaum, Matthew P Lungren, Chih-ying Deng, Roger G Mark, and Steven Horng. Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports. Scientific Data, 6(1):317, 2019.
Tzu-Lin Kuo, Feng- Ting Liao, Mu- Wei Hsieh, Fu-Chieh Chang, Po-Chun Hsu, and Da-Shan Shiu. Rad-bench: Evaluating large language models capabilities in retrieval augmented dialogues. arXiv preprint arXiv:2409. /2558, 2024.
LASA-Team, Weiwen Xu, Hou Pong Chan, Long Li. Mahani Aljunied, Ruifeng Yuan, Jianyu Wang, Chenghao Xiao, Guizhen Chen, Chaoqun Liu, Zhaodonghui Li, Yu Sun, Junao Shen, Chaojun Wang, Jie Tan, Deli Zhao, Tingyang Xu, Hao Zhang, and Yu Rong. Lingshu: A generalist foundation model for unified multimodal medical understanding and reasoning, 2025. URL
https::/arxiv.. org/ abs 2506. 07044
Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. A dataset of clinically generated visual questions and answers about radiology images. Scientific data, 5(1):1-10, 2018.
Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training large language- andvvisionn assistant for biomedicine in one day. In NeurlPS 23 Datasets & Benchmarks Track (Spotlighr), 2023. httpt:ttt///tttm wht..b.tttttt.tgggg
Oskar Maier, Bjoern H Menze, Janina Von der Gablentz, Levin Häni, Mattias Heinrich, Matthias Liebrand, Stefan Winzeck, Abdul Basit, Paul Bentley, Liang Chen, et al. Isles 2015-a public evaluation benchmark for ischemic stroke lesion segmentation from multispectral mri. Medical image analysis, 35:250-269. 2017.
12

---

Published as a conference paper at ICLR 2026
Bjoem H Menze, Andras Jakab, Stefan Bauer, Jayashree Celpathh-Cramert Keyvan Farahani, Justin Kirby, Yuliya Burren, Nils Porz, Johannes Slotboom, Roland Wiest, et al. The multimodal brain tumor image segmentation benchmark (brats). IEEE Transactions Medical Imaging, 34(10): 1993-2024, 2015.
Michael Moor, Qian Huang. Shirley Wu, Michihiro Yasunaga, Cyril Zakka, Yash Dalmia, Ed- uardo Pontes Reis, Pranav Rajpurkar, and Jure Leskovec. Med-flamingo: a multimodal med- ical few-shot learner. In Stefan Hegselmann, Antonio Parziale, Divya Shanmugam, Shengpu Tang, Mercy Nyamewaa Asiedu, Serina Chang, Tom Hartvigsen, and Harvineet Singh (eds.), Proceedings of the 3rd Machine Learning for Health Symposium, volume 225 of Proceed- ings of Machine Learning Research, pp. 353-367, PMLR, 10 Dec 2023. URL https
//proceedings.mo ml wwwppreaa//pa wwtpppeeeee////ooooeeee/ mooo22/mmmooo23aa3m html
Johannes Rückert, Louise Bloch, Raphael Brüngel, Ahmad Idrissi- Yaghir, Henning Schäfer, Cynthia S Schmidt, Sven Koitka, Obioma Pelka, Asma Ben Abacha, Alba G. Seco de Herrera, et al. Rocov2: Radiology objects in context version 2, an updated multimodal image dataset. Scientific Data, 11 (1):688, 2024.
Richard M Schwartzstein. Clinical reasoning and artificial intelligence; can really think? Transac- tions of the American Clinical and Climatological Association, 134:133, 2024.
Andrew Sellergren, Sahar Kazemzadeh, Tiam Jaroensri, Atilla Kiraly, Madeleine Traverse, Timo Kohlberger, Shawn Xu, Fayaz Jamil, Cían Hughes, Charles Lau, al. Medgemma technical report. arXiv preprint arXiv:2507.05200 2025.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: visual explanations from deep networks via gradient-based localiza- tion. International journal of computer vision, 128:336-359, 2020.
Arnaud A A Setio, Andrea Traverso, Timo de Bel, Max S Berens, Chris van den Bogaard, Piergiorgio Cerello, Hoo-Chang Chang Chen, Qi Dou, Maria E Fantacci, Pierre Geurts, al. Validation, compari- son, and combination of algorithms for automatic detection of pulmonary nodules in computed tomography images: the lunal6 challenge. Medical Image Analysis, 1231-13, 2017.
Jeffrey Siegelman, Lisa Bernstein, Jennifer Goedken, Linda Lewin, Jason Schneider, Martha Ward, and Hugh Stoddard. Assessment clinical reasoning during a high stakes medical student osce. Perspectives on Medical Education. 13(1):629, 2024.
Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. Large language models encode clinical knowledge. Nature, 620-7972):172-180, 2023.
Akshay Smit, Saahil Jain, Pranav Rajpurkar, Anuj Pareek, Andrew Y Ng, and Matthew Lungren. Chexbert: combining automatic labelers and expert annotations for accurate radiology report labeling using bert. arXiv preprint 26979 2020.
Xiangru Tang, Daniel Shao, Jiwoong Sohn, Jiapeng Chen, Jiayi Zhang, Jinyu Xiang, Fang Wu, Yilun Zhao, Chenglin Wu, Wenqi Shi, et al. Medagentsbench: Benchmarking thinking models and agent frameworks for complex medical reasoning. arXiv preprint -xXxx:2533.07459 2025.
Doubao Team. The doubao visual language model officially released with general model capability fully comparable gpt-4o, 2024. URL http:::ttteeettt dwu.ao.ooooooooo.ooo com/
Dequan Wang, Xiaosong Wang, Lilong Wang, Mengzhang Li, Qian Da, Xiaoqiang Liu, Xiangyu Gao, Jun Shen, Junjun He, Tian Shen, et al. A real-world dataset and benchmark for foundation model adaptation in medical image classification. Scientific Data, 10(1):574, 2023.
Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Sum- mers. Chestx-ray8: -ospital-scale chest x-ray database and benchmarks weakly- supervie classification and localization of common thorax diseases. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
13

---

Published as a conference paper at ICLR 2026
Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang. and Weidi Xie. Towards generalist foundation model for radiology by leveraging web-scale 2d&3d medical data. arXiv preprint N.XY::2308.02466,, 2023.
Jiancheng Yang. Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, and Bingbing Ni. Medmnist v2-a large-scale lightweight benchmark for 2d and 3d biomedical image classification. Scientific Data, 10(1):41, 2023.
Jin Ye, Guoan Wang, Yanjun Li, Zhongying Deng, Wei Li, Tianbin Li, Haodong Duan, Ziyan Huang, Yanzhou Su, Benyou Wang, al. Gmai--mmeench: A comprehensive multimodal evaluation benchmark towards general medical ai. Advances in Neural Information Processing Systems, 37: 94327-94427, 2024.
Ming Zhang, Yujiong Shen, Zelin Li, Huayu Sha, Binze Hu, Yuhui Wang, Chenhao Huang, Shichun Liu. Jingqi Tong, Changhao Jiang, al. Limeval-men mende A real-world clinical benchmark for medical ilms with physician validation. arXiv preprint arXiv:2506.0409 040788 2025.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger. and Yoav Artzi. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv: 009.5,,0,067, 2019.
Xianda Zheng, Huan Gao, Meng-Fen Chiang, Michael J. Witbrock, Kaiqi Zhao, and Shangyang Li. Evo-PI: Scaling medical reasoning via evolving chincipll-guidee reinforcement learning, 2026. URL
net forum? wotmmmddd=ooggo 1x=
Weihai Zhi, Jiayan Guo, and Shangyang Li. Medgr Breaking the data barrier for medical reasoning via generative reward learning. arXiv preprint arXiv:2508 20549, 2025.
Yakun Zhu, Zhongzhen Huang, Linjie Mu, Yutong Huang, Wei Nie, Jiaji Liu, Shaoting Zhang, Pengfei Liu, and Xiaofan Zhang. Diagnosisarena: Benchmarking diagnostic reasoning for large language models. arXiv preprint arXiv:2505. 2025.
A ADDITIONAL DATASET DETAILS
Data Souree Breakdown. To ensure transparency and reproducibility, we provide a detailed overview of the composition of Neural Neural-MedBence As shown in Table 4 the benchmark draws on four carefully vetted sources spanning public neuroimaging cohorts, curated online repositories, and peer-reviewed clinical case reports. Together these sources yield heterogeneous pool of conditions ranging from common neurodegenerative diseases to rare atypical presentations, enabling broad coverage of diagnostic challenges.
Table 4: Proportional breakdown of all cases in Neural-MedBench by source.
Source
Proportion of Cases (%)
Primary Conditions Covered
ADNI OASIS Radiopaedia
18.2
Alzheimer's Disease, Mild Cognitive Impairment
9.1
Alzheimer's Disease, Normal Aging
45.5
Stroke, CNS Infections, Autoimmune Encephalitis, Tumors Rare Diseases, Atypical Presentations
JAMA Ncurology
36.4
Case Curation Protocol. As illustrated in Figure 5 the construction of Heural-MedBench followed a rigorous, four-stage, funnel-shaped pipeline to distill an initial pool of over 2,000 candidate cases into the final 120 high-density cases. The stages were:
1. Pooling and Filtering: We first pooled cases from sources and filtered them to retain only those with sufficient multimodal completeness (e.g., clear imaging, patient history, and clinical notes).
2. Expert Curation: A panel of senior neurologists reviewed the filtered cases for clinical plausibility, diagnostic complexity, and educational value, selecting for scenarios that require deep reasoning.
14

---

Published as a conference paper at ICLR 2026
3. Annotation of Ground Truth:
For each selected case, the expert panel systematically annotated the ground truth, which includes not only the final diagnosis but also differential diagnoses, key lesion characteristics, and the full explanatory reasoning chain.
4. Consensus Review and Challenge Validation:
Finally, all annotated cases underwent a consensus review to resolve any disagreements. In this stage, also performed challenge validation using baseline models to filter out diagnostically trivial cases and ensure every in the final benchmark presents meaningful reasoning challenge.
This multi-stage process ensures that each of the 120 cases is diagnostically rich, clinically authentic, and serves as potent of model's reasoning capabilities.
Puthology
Others
Rudiology CT, Mikl..
Photo
Stestical Pat...
Pooling & Filtering
Retain cases with sufficient multimodal completetess.
funnd shaped, Patttgggggggge pipeline
Carred Anavie
Expert Curation
Reviewed cases for plusibility, diagnosiic complexity, and educational value.
Annotation of Ground Truth
dfizettaid id Irnen tosen deaeeeettrrrree
Collected cases were sysienutically armotated with respect to the key aspects specified the ground
expleatery renweing wor.
Consensus review & Challenge validation
Gemini
model completely incomectly, or only arrived at the coneet answer after multiple attempes.
Direct Diagnosis
Complex Diseases
Multi-round Dialogue
Total: 55 cases
Figure 5: The four-stage, funnel-shaped curation pipeline for Neural-MedBench. The process distills over 2,000 initial candidates into 120 final cases by systematically filtering for multimodal completeness, curating for diagnostic complexity, annotating rich ground truth, and validating the final reasoning challenge.
Distribution of Clinical Conditions and Reasoning Tasks. providess a dual view of the benchmark's 'ss diversity. The top panel (sunburst chart) illustrates the hierarchical distribution of the 120 clinical cases across wide spectrum of neurological diseases. It demonstrates balanced coverage of both common conditions like Alzheimer's disease and a long tail of rare or diagnostically challenging disorders (e.g., PERM, PAVF), ensuring robust test of model generalization beyond frequency bias. The bottom panel details the distribution of the 200 questions across the three core cognitive capabilities probed by our benchmark: Differential Diagnosis (29%). Lesion Recognition (25%), and Rationale Generation (46%). The benchmark is thus heavily weighted towards higher-order reasoning, moving far beyond simple recognition.
15

---

Published as a conference paper at ICLR 2026
Normal
Aizhelmerss disease
Tasks & Subtasks
PERM
Normal
Differential Diagnosis
Lesion
Rationale
MO!
Recegnition
Generation
AIND
Sercening
58
Localzztion
38%
Reasening
68%
Prioritzzaion
Classification
10%
Mechanism
6%
mitracn Nesconaan
Macinngons
Corelaaion
27w
Integration Characterizztion
4Tw
Argumentation
18s
Uncertainty
26
Presentation
and Choleer desiies
GUlero PAVE
29%
25%
46%
Figure 6: Hierarchical Distribution of Neurological Diseases and Reasoning Tasks in (Top) The sunburst chart illustrates the distribution of the all clinical cases across a wide range of neurological conditions, from high-level categories (e.g., Neurodegeneratiie Disorders, Intracranial Neoplasms) to specific diagnoses (e.g.. Alzheimer's disease, Glioblastoma), including significant proportion of rare and complex disorders. (Bottom) The task distribution across the 200 questions, categorized into three core reasoning families; Differential Diagnosis (29%), Lesion Recognition (25%), and Rationale Generation (46%). Each family further broken down into specific cognitive subtasks.
Length Normalization. We first segmented notes using standard clinical section headers (c.g.. "History of Present Illness," "Neuroloiia Examination," "Impression/ "iapressioooAsssssmeent."" "Plan," etc.). We prioritized and always retained the reasoning-rell relevant sections: HPI, Neuro Exam, and Impression/Ae Impression/Assessmeen Low-yield sections such as repeated "Past Medical History," exhaustive medication lists, or administrative details were truncated or removed if they exceeded a small threshold and were clearly redundant. We imposed an upper token budget per modality (e.g., per note) to avoid extreme outliers. When needed, we truncated from low-importance sections first, only truncating within high-priority sections as last resort. Structured EHR fields (age, sex, key lab values) were not truncated. We did not pad or artificially inflate shorter notes. Normalization therefore acts mainly by trimming pathological "very long" tails, not forcing all samples to be identical in length. Natural variability in realistic note lengths (within a moderate range) is preserved.
Content Statistics. Figure 7 presents the detailed distributions of token counts and image sizes for the dataset. Panel (a) shows the distribution of English token counts in the clinical narratives, confirming that while we controlled for extreme length variations, natural diversity remains. Panel (b) illustrates the variability in total image pixels per case, reflecting the real-world heterogeneity in MRI/CT acquisition protocols (e.g., number of sequences and slices). These statistics underscore that Neural Neural-MedBent -MedBench preserves authentic data characteristics while maintaining a balanced design for fair evaluation.
Limitations of Dataset Composition. While Neural Maarral-MeBBBncc was deliberately curated for diagnostic depth, we acknowledge several limitations in its current composition. First, the total number of tasks (200) is modest, reflecting a deliberate trade-off for the high-density, expert-intensive annotation required for reasoning evaluation. Second, the distribution of conditions is biased toward diseases with pronounced imaging correlates (e.g., stroke, tumors), with less representation of disorders that are primarily functional or metabolic. Third, although cases are drawn from multiple sources, the benchmark does not yet systematically capture the full spectrum of domain shift across hospitals, scanners, or reporting conventions. Finally, the current release represents only initial stage of the benchmark, which will be progressively expanded over time. These considerations underscore that Neural-MedBench is best interpreted fidelity, not a comprehensive benchmark for statistical generalization. To address these limitations, future iterations will continuously grow in both scale and disease spectrum, with particular emphasis
high-resolution stress test for reasoning
16

---

Published as a conference paper at ICLR 2026
Saneilaghdd
1405
12
280
0900
5.00
0.29
0.33
0.55
2.00
3.2
1.00
4.7
28p
Token Distribution Comparison
Image Size Distribution
Figure 7: Detailed content statistics for Neural-MedBench.
on underrepresented conditions, longitudinal follow-up, and multi-center variability, ensuring that the benchmark remains a continually evolving resource for the community.
B PROMPT TEMPLATES
We employed structured role- pole-prompting strategy to ensure that model behavior adhered to clinical logic.
"You are an experienced neurology expert, skilled in analyzing complex cases and providing preliminary diagnoses based on combination of medical imaging and clinical descriptions. Please carefully answer the following questions based the provided information."
Example Task Prompt.
Patient history: 47-year-old female with progressive motor weakness and inter- mittent visual disturbances.
Imaging: Multi selti-sequence MRI (T2, FLAIR) uploaded below.
Task: Please provide your most likely diagnosis, differential diagnoses, and explain your reasoning.
All multimodal inputs were interleaved with text. Imaging data were encoded in base64 and inserted < <image> tokens when supported by the model.
We employed a bascline model as an automatic judge to assess the agreement between the model's outputs and the reference answers. The evaluation is conducted using fixed prompt.
You are an expert medical evaluator judging the accuracy of Al-generated diagnoses against reference answers.
model [nsweranswer]]
ref answer: [nswer))
Guidelines:
1. The model's diagnosis is correct if it captures the main pathologs/comdition mentioned in the reference diagnosis.
2. Minor differences terminology or additional details provided by the model are acceptable as long as the core diagnosis is correct.
3. If the reference diagnosis contains multiple conditions, the model should identify least the primary condition to be considered correct.
4. Ignore differences in formatting, grammar, or level of detail if the core diagnosis is correct.
Please provide direct answer at the heginning, [yes] or [no]-
17

---

Published as a conference paper at ICLR 2026
C
MODEL CONHIGURATIONS AND HYPERPARAMETERS
To ensure reproducibility, list the exact model versions evaluated:
GPT-4o: gpt-4o-2025-03-26
GPT-5: gpt-5-chat:
Gemini 2.5-Pro: gemini semini-2.5-prr 2..5-rro-0225:
Gemini 2.5-Flash: genthnn-22.5-1flaah-peeeee-----
Gemini 2.0-Flash: gemini-2.0-flashhes.
Claude 4.0 Sonnet:
claude-somnet-4220255544
Claude 3.7 Sonnet: clauude33---oonnee22255500
Claude 3.5 Sonnet: claude-3---ssomee-
Qwen-Vile VL-2.5: qwen-v1-2.5-33
Doubao:
LLaVA-Med: oa
RadFM: chaoyl-wu/RadFM
Med-Flamingo: 1
Huatuo-GPT-vision: Hustuc-gpt-viiiion-
MedGemma: MedGemma-27B-it
Lingshu: Ling
C.1 HYPERPARAMETERS
Table Summarizes the key hyperparameter settings used across all experiments. For diagnostic accuracy (pass@1 1, @ass@55,, predictions were generated with nucleus sampling = 0.95, T 0.7). For text similarity evaluation, we used BERTScore with roberta-largeeeeee All confidence intervals reported using the Wilson score interval.
Table 5: Decoding hyperparameters used for each model
Model
Hyperparameters
GPT-4o
temperature=007, top.p=0.95 seed=22
GPT-5
temperature=00.7, top p=0.95
Gemini 2.5-Pro
temperature=0077 00p
Gemini 2.5-Flash
lemperature=077. top p=0.95
Gemini 2.0-Flash
temperature=0.7, top-p=0.95
Claude 4 Sonnet
lemperature=0.7, top-p=0.95
Claude 3.7 Sonnet
temperature=0.7,, top p=0.95
Claude Sonnet
temperature-0.7, top p=0.95
DW-2-VVL-2.5
0. top.p=0.9p
DouBao
(emperature=0.7,oop top-p=0.95
aLLVVAMedd
temperature=0.7, top p=0.95
MedGemma
temperature=0.77, top-p=0.95
RadFM
temperature=0.7,, ttt.ttpppppl.. p=0.95
Med-Flamingo
temperature=0.7,, top-p=0.95
HuatuoGPT
temperature=0.7, top p=0.95
Lingshu
temperature=0.7, top. 00.95
18

---

Published as a conference paper at ICLR 2026
D CONFIDENCE INTERVALS
For diagnostic accuracy metrics, report 95% confidence intervals using the Wilson score interval:
p(1
p)
22
CI = 1+
2n +
4n
where ý is observed accuracy, is sample size, and 1.96 for 95% confidence.
E GRADER BIAS CONSIDERATIONS
potential concern arises from the use of GPT-4o both as a benchmarked model and as the primary LLM-based grader. To mitigate self-evaluation bias, conducted rigorous clinician-in-ther toop calibration, finding a very high correlation with expert consensus scores (Pearson's 's r > 0.9). In addition, our evaluation protocol is hybridized with semantic similarity metrics (BERTScore) and direct clinician validation, ensuring that grading robustness does not rely solely on GPT-4o. To ensure our ALM--rader is reliable proxy for expert judgment, we conducted rigorous validation. The key steps and findings are:
Human Gold Standard: A panel of three board-certified neurologists independently scored large subset of 100 model responses to establish a human consensus score.
Quantitative Correlation: The crucial finding was very high correlation (Pearson's 's r>0.9) between our LLM-grader's scores and the human consensus, scientifically validating its reliability.
Bias Check: We specifically checked for self-evaluation bias and found no systematic advantage for GPT-4o's 's own outputs compared to those from other models.
For future iterations, we plan to expand the grader pool by incorporating multiple LLMs (e.g.. Claude, Gemini) and additional clinician review to further enhance fairness and reproducibility.
F ERROR TAXONOMY ANNOTATION PROTOCOL
To better understand failure modes, two board-certified neurologists independently annotated 100 randomly sampled incorrect responses using the following taxonomy:
Reasoning Failure: Correctly observed features but incorrect causal inference.
Perceptual Failure: Misinterpretation of visual features (e.g., lesion missed).
Visual Hallucination: Fabricated findings not present in the input data.
Knowledge Gap: Missing medical knowledge required for correct diagnosis.
Across the incorrectly answered cases, the distribution of primary error types was: Reasoning Failure 5 51%. Perceptual Failure 27%, Visual Hallucination 13%, and Knowledge Gap 9%. These proportions suggest that current multimodal language- systems for neuroimaging are limited more by inference and recisionnmakingg than by raw visual recognition or encyclopedic recall.
Disagreements were resolved by consensus. Inter-rater agreement was = 0.82, indicating strong consistency.
Reasoning Failure (51 %).. The dominant failure mode reflects cases in which salient image find- ings were noted but mapped to an incorrect diagnosis or pathophysiologic explanation. Typical patterns included misweighting across MRI sequences (e.g., over-interpreting FLAIR hyperinten- sity without integrating diffusion restriction), conflating acute and chronic stigmata, and collapsing multi-lesion presentations into single etiology. These errors point to gaps in causal and temporal modeling- models can "see" but struggle to structure evidence, adjudicate competing hypotheses, or apply exclusion criteria.
Perceptual Failure 2)). Pure decection/charrcteriiatiin mistakes were the next largest category. Contributing factors included low signal-to-noise, motion artifact, and small lesion size.
19

---

Published as a conference paper at ICLR 2026
Visual Hallucination (13%). In nearly one-fifth of failures, the model asserted findings that present (e.g., "midline shift," "ring-enhanne lingeenhancing lesion") and then built a diagnosis on those fictitious observations. This is distinct safety risk because it couples perceptual fabrication with persuasive language.
Knowledge Gap (9%). Pure deficits in medical knowledge were least common. Notably, the relatively small share here underscores that, for common entities, the models "know enough" but still reason unreliably.
G ERROR CASES
To provide more granular and intuitive understanding of the failure modes quantified in our main analysis. this section presents a series of representative error cases from Neural -MeddeMeMchhhh Figure presents Level 2 complex diagnosis task. This task specifically tests the model's ability to link pharmacological history to specific neuroimaging findings, common real-world clinical challenge. The subsequent pages display additional failure cases, presented in groups figure (Figure 9 to Figure [2). Together, these qualitative examples provide concrete evidence for the limitations current VLMs and highlight the critical importance of Depth-Axis evaluation.
Level 2 Error Case
A 4.41-yea-old quadriplegic male patient, due traffic accident, had multiple bedsores been bectridden for years metronidazule powder (wilh gel) the bedsores for past to 10 months. She presented the emergency room our hospital with complaints slurred speech facial tilting for days, accompanied by seizure. She had history of type 2 diabetes mellitus. Physical examination: The patient was conscious, oriented, slurred speech. could follow simple instructions, and had signs meningeal iritation. His pulse rate was 101 beats per minute and blood pressure was 120/80 minHg.
applying
Q: Combine the patient's medical records and pictures to analyze and diagnose the most likely disease? ?
Reference & Model Answers
Gemini
Ground truth
Metronidazele encephatopathy
Central Pontine Myelinobyzs
Motnbolic Encophalopahies
Acute Ischemic Stroke
Ronsoning Fallure
Knowniedgee Gop
Roasoning Failure
Figure 8: A representative Level 2 failure case demonstrating both Reasoning Failure and Knowledge Gap.
20

---

Published as a conference paper at ICLR 2026
Level Error Case
Level 1 Error Case
female with marned living wat sprus, presents defktiency, but no Iypertension, recent itsamnia, and vitamin 1512 diabetes, heact discase, strake. Site takes 10 medications, incloding NSAIDs antioggt.ans ans-anntllatseets antdepressunts, FDA-- approved drugs Alzheimer's cognitie impaimment her Giteer. l'uetionallyy she canages finances independenlly but requires some aesieace with meals and rnvel. She good connected vision, good besting stable BMI 18 bload mild
77-year-oh old White,
edicaation, history of
The potient is an 82-jear-old fomale, con-Hispanicl enino, Asine, Finglshh-ppaakn to perfani cetivities daily living completaly living unkgaandaily. ciller retacament commonil ss inkgeankn aal grong living. least fins--égnoe rolut was repertod have cognitive impairment,
ciucation (20 yeurs),
Family hisiory
repered The pulienl was on medacation, with single visit. Beta beta-blooorr lipid lowering medication, and discer es medications curreatly being used. The GDS total adjusted
cogaltive impoinmantt of medications recurded
and on MISE and MoCA scores were missing, but she showed good word learning and animal naming shilk. atdition, she had history of inperttassion, no disbotes, heart ciseaso.
presuue
Neumlogical showed porkimsoiee tradykinees. unstable podture, lnfl- comiioocnorr loss, and speoch spexiis with cortical deficts (aplaiis, sprasia, neglect) suggeslive of subecctiaal ischemic vasculas demeutis.
Blind
decumcuted.
In daily life. she was able to handle fioancial affirs independently, some help peeparing meals and traveling
Diagnase patient's disense.
Diagnose the patient's 's's disense.
Model Answers
Model Answers
Gemini
Gemini
Selesetical bohesic Parcslor Dementia
AD
Seleotical hobenic Parcwior Desevila
MCT
X
X
X
X
X
X
X
oogastne A0 Allewee's Aness
W Cladsshsaant
(a)
(b)
Level Error Case
Level Error Case
years married and living with sponse, fully independent in duily activities Family history notable for patemaal coguttie impoiment He moees medications, including sathyperiensives (ACE inhibitors, antiadrnnergic drags),
66l-year-okt Liggniic
years marned, living independently with spozse. No family bistory of cognifive impairment, She takes medications, including beta blockees and anddegressanss. History inchides vitamin B12 deficiency, sleep apnco, and price 18I, Smokee past mooths. BBM 32 5. blood pressure 156622 GDS score Nemoolaicaa enam showed ocly ago sgo-reloted or nou nou-dementic findings. Cogrative testing: MOCA 26, namiin Henson ligure capy
78-year-old lisgenic
atticoaddants uniplateless, antiepressanss,
years
anxiolytics. BMI is 36.3, bland gressure 118/74, Recent depressed inizable moce cad emotional lnbility reported GUS soace 5. Neurological
testing: MoCA gond language flunncy short-tera monsory, fignc eopy
Diagnose the pationt's disease.
Diagnose the patient's 's disease.
Model Answers
Model Answers
Geriii
Med Gemmin
Mediamnn
Vartiar Inpainnest
depeeshhve
Cognstire Inpoirment
MC7
Nornal CogattiveAgggggg
MCT
X
X
X
X
X
X
X
X
X
oagetise www.deeeersss.. aberr
cloced SCaR
Cha.solcaeed
(c)
(d)
Figure 9; Additional error cases.
21

---

Published as a conference paper at ICLR 2026
Level Error Case
Level Error Case
Prescttation: Acute beadache and impaired comsiioueces
Nerzelogical exaza: non- on--oocal
The patient is yeay old female, right handed. Specific information about her educational backgroand seciaeconomic missing MMSE cognitive function and demestta status cannot directly arcevsed. Imaging data showeed that her estimared tomal intracranial valnme was 1749 was
Age: 50 years
currently curean
Graker Male
lacking,
07: What abvormality Indicatted by the incressed
densly within the deep cerebrat with on aun-cnalruu
CT?
bram valume tange, indieating stracture was not sigr. ficantly ahnormal in volume The unteranna artery scnre was 1,44,, soggering that there may be certain cerchrovaseular hoslth prostems, evaliation. the missing resulls of delayed necill impossible to fully undestted ber jong- neg--em memory abilit.
stundandred
(1: Wher the liheey coure of bilatoral thatinnic and
baval gongle repostionaaion obserned on CT?
(3: What the rigntficncc of abnonce of noresal
flae rlgual in the deep cerebral veins on MRV?
speeific ingect
Hose bilateral thalanic hemerrhages
costribate the
Q5:
likely
given panern of abnormalitia lmapping?
Reference Model Answers
Diagnose the patient's disease.
Greand
conolall nele feceddnti
Pansun Theambort
Deep cembral throsor
Model Answers
Fonse ooegerion edloee
Zbparic-bctemsi excephatipatt
Gemini
Adfinction
rignal
Condrenareviar
Normal
Cerebronmcslor
Contemarriirr inno
Shrandorts
Penouz Aircnboohh Nyperteaaio
Bilatcall shaloosic
X
X
X
lcosrrlage
Avieveal cenoloal
Neerai cenobral
Midffe and cuericrr cmnerrl ensenns
rein dhoombrtt
veler
"ow Pigroso compa airgortte gromsnt by
Level 2 Error Case
Level Error Case
year old maz pessented with 1-mooth histery of headacbe, diplopia. setaures, and altered consconancs for I week. The poinest lad had sciruess chanattenzzd by totae lect devistion the bead and eyebeils, follwed by buief totic clonic activity of the eatremities, and confasion lozing 10 15 nimutes after viiom: The paliets detaed hisker Jewe, bead
year mat pessented with montth histry of headackte diplopia. sciaures, and altred conscionaness for I week. The pelienl had had seirares
chansestzed tmii lett deviatonz of the besd and syoballs, followed by beief tosic clonic activity
extrenttess and contisioo listing 10 to 15 minnuees limb
dnax linb
acher ihe seizm. The detiel bisirr ol leve best The
The
patiert histor
Moysical
slanted. deried histry exyyme inmuavnnhea awayy was iurretive..
Hoysical tests
and ködney aucymelinkcal mumunatdeen cosay was unaktive.
patient
patient
Chest X-my thows. Twe- dinessiocal echooecioggapte did
shoved exygen svantion 50%, Blood cultrre results were sterlie.
20 J2.
diveane this
Anslye and
diseane
Reference & Model Answers
Reference ModelAnsserr Answers
Ground
Ground trath
eilecuz
aleneu
PAPM
Insie
Organophaphate pobaeing
Organophonphott paizoning
poiseiin
monomiil
X
X
(c)
(d)
Figure 10: Additional error cases.
22

---

Published as a conference paper at ICLR 2026
Level 2 Error Case
Level Error Case
the right The paiient bad typertenson, lyperlipkeemii, mellitas, sapication..... Cer.......rsseeeeee.... oepinal (CSF) cumination showel nocaal bloud cell coum, persoit and ginceee lovel, and mibdly elrowed while bloed cell coun of cellajy.......
....ittemniiien bendschos for arveral works, which grigresaal beaduchhes
past years experiecce wamennnn helevinr discedar, notless syndmme daylime rapid cogrutive dacline. ... was trated wit levodopa conserazide cup to 200 mi mng tanes daily), but las Unifed Parsimen's Discase Rating Scale score dil not inpeove, depaminagje medicstion was disceminued Neurologi: exanitation showed reduced expression, symaetrical bradykinesi. ana revealed ahnenmdiiis orgritive fasctioning affacting both soavertal
44.year-eld mal: referied piggessive Pullianns with respoese levedopa.
fibcilaaion... memocyy impriment
stuflaess, difliculty walking,
reporid
balluciratinrs,
Eye movenntie wen will-noging, with
Neveepyycbollgical evatration
Brain-specific FDCs TIG-PETMMRE
shnwed prosante Duc mpid detenortioo
function
Prikizsea's symploms,
taderwiut fromal certical brain buopoy,
diagnose dieare diis
Reference & Model Answers
Analye diegneee Hekeyy
Ground trank
Mediianna
Reference Model Answers
En-hecin Cherker dboste
Hesplecti jeacess
Adaltai Sclersth
llk-- grair prizns
eine- gmalk glisn
Autusmal recenahe
Lerr bost
ecar lachenac
Zesy leely
Levy lonh
X
X
X
X
X
X
X
Yiee sigeete wwleeseenneee Sechegee bo
(a)
(b)
Level 2 Error Case
Level Error Case
wite diagneeal Alchwiner'ss dirvatee afler two years of progessive shom-rem memory loss doe to perrisezz ahtormal behavinc, and an XIII scan of beain showed higpocns mipl anoply. His wife brought him neeeology olitie. His medural hisury clancteried by well-oommded typerteniinn ead type diabenes melimus. He did aove any personal or Simily hiviory merad illees.
Prosezia with confesion word finding diTicollss.
Enterzal CI left MCA infaret with edema.
Patient Data
Age: 75 yees
Garder Mile
Patient's regular mediettios
imaging? litaly diagaasis baved
(lnaniggine
musdeenaal enalapeil: metiommen)
... ....began experimns
Q2: What featares in the Image support the presence of vasogenic edoms?
symntous the risk video
BMSE ware 20 out 30.
cntonarion (-1), recorting (-3)
Based feature, does terion liketi primary brain
recall 33,, atteatic and
contimitt oun 10.
Eeerzeenophaloggnnnee magnetic
Reference & Model Answers
findings
Ground
Anelyrr
Nikety disease dkts
Seltery the poreet-- accipial regine
Deheneic Stroke
bclenni Stoke
Reference & Model Answers
Perlocionall white wuttor byposionnatio
Low deedly while mustrr cround
Servosnalig fow doasity skovo rasagesic ediuo biveal-main
alompiion. Aecossst
Misdestifeaton attic /M500)
MED
X
X
X
degradieerrroooo dlgpectl
(c)
(d)
Figure 11: Additional error cases.
23

---

Published as a conference paper at ICLR 2026
Level 3 Ervor Case
Whar
diagnosis based traii imaging?
Wher fearures tmage rapport presence of ofvooggoiic
Q3: Based on imaging features, does lesion more libely represent a primory bratn mumor or muttentaris?
there imaginng inforction, how they differ from lesion?
suggest mass offect herniation? If so, to whal estent?
Reference & Mosee-Answers
Sultury Avale
Misife cerebral oniogy straee
Acsle deep gra PyStention
axtustaris
Perfissiond cxymauery
Rypoodeniiy dfeer
shif
Ainoronii oker glvwe
Dichesic
Frneun slrombonh
MCA
Penous dinosbotti de teperteniiv
Deficte oovte roscalaa patholog
Anfornt
(a)
Level Error Case
Prescotatoo:: beadreches fox maaato.
presaue.
Age:
QT:
filelly diggnesis besed inaging features?
Watch imaging foettrr
patlent's sympionee
Inracranfol pressure?
What imaging sign inolivates trenspendy noal flue hystroophhaar??
fesiere
prlor zurgical Intermontion inforved from imsgingg?
Reference Model Answers
Ground tradi
with ohrmaetive Tyshocopiotts
of 20sborsphahe ventrioalor
Traniopendreal wdomo
Pertvessrouder hyperhessity
Perivstriesdo Inasugendyyeal
Nermal
sentorial
fuarsh unseriile
indestation
(b)
Figure 12: Additional error cases.
24


Reviews:
[{"review_id": "d6BqcY6Pnf", "reviewer": "Reviewer_Kijm", "summary": "This paper introduces Neural-MedBench, a focused and reasoning-intensive benchmark designed to evaluate the clinical reasoning capabilities of vision-language models (VLMs) in the neurology domain. This benchmark integrates multi-sequence MRI scans, structured electronic health records, and clinical notes, and encompasses three core task families: differential diagnosis, lesion recognition,\nand rationale generation. A hybrid scoring pipeline is proposed for better evaluation.", "questions": "1. The details of grader validation and validation process are missing. \n2. Some important VLMs are missing, such as Gemini 2.5 pro and GPT-5.\n3. What the different of Base VLMs and General VLMs? \n4. It is intersting to provide more discussion for the following case: Why the pass@1 acc of medical student is significant lower than VLMs? Why the pass@1 acc of medical student in Multi-round dialogue is significantly better than Direct diagnosis?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "A two-axis evaluation framework is proposed, which can be complementary assessments of both breadth (statistical generalization) and depth (reasoning fidelity). Neural-MedBench is the first neurology-focused benchmark explicitly designed to operationalize the Depth axis, comprising 120 multimodal, expert-curated diagnostic cases with 200 reasoning-intensive tasks. Experimental results show that current VLMs fail primarily at reasoning, despite strong performance on existing large-scale datasets. A systematic error analysis and a human performance baseline are also provided.", "weaknesses": "1. The definition of three difficulty levels is unclear. \n2. A limitation section should be added in main body.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "opAStRiMtr", "reviewer": "Reviewer_RaHn", "summary": "This paper introduces Neural-MedBench, a compact yet reasoning-intensive benchmark designed to evaluate clinical reasoning capabilities of vision-language models (VLMs) in neurology. The authors propose a \"Two-Axis Evaluation Framework\" distinguishing between Breadth (statistical generalization on large datasets) and Depth (reasoning fidelity on complex cases). The benchmark comprises 120 expert-curated neurology cases with 200 tasks spanning differential diagnosis, lesion recognition, and rationale generation. Evaluation of state-of-the-art VLMs (GPT-4o, Claude-4, MedGemma) reveals significant performance drops compared to breadth-oriented benchmarks, with error analysis showing reasoning failures (51%) dominate over perceptual errors (27%).", "questions": "1. Prompt Sensitivity: Did you test the sensitivity of the evaluation prompts to variations in wording?\n2. Medical Student Performance: What explains the substantially lower performance of medical students compared to existing large models in Table 2?\n3. BERTScore Discrimination: The BERTScore differences in Table 2 appear minimal across models. Does this metric provide sufficient discriminative power?\n4. Senior Physician Baseline: Given the relatively low performance of senior physicians on the benchmark, were all questions validated by multiple experts to ensure correctness and clinical appropriateness?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well-written. \n2. The paper presents the Two-Axis (Breadth/Depth) evaluation framework provides a compelling perspective on AI evaluation that addresses a genuine gap in current medical AI benchmarking.\n3. The rigorous multi-stage expert curation process involving senior neurologists and neuroradiologists, with consensus review, ensures benchmark quality and clinical validity.", "weaknesses": "1. The bechmark is small-scale benchmark compared with exists medical benchmark, such as OmniMedVQA, GMAIMMbench.\n2. Some medical VLM method need to added to compared，such as, Huatuo-vision, Lingshu. While, GPT-4o maybe a outdate model, gpt5 is welcome.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rRpxBlRVRk", "reviewer": "Reviewer_NAbK", "summary": "This work provided Neural-MedBench, which is an expert-curated, neurology-focused benchmark that shifts evaluation from breadth to depth by testing multi-modal clinical reasoning, showing current VLMs chiefly fail for reasoning rather than perception despite strong shallow scores. However, there remain some unresolved issues, with single-judge reliance, opaque challenge validation, and unclear length normalization tempering the claims.", "questions": "See the weaknesses", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. High reasoning density: a depth-oriented benchmark with greater clinical relevance than shallow QA-style responses.\n2. The benchmark and items are curated by medical experts, which is more credible than LLM-generated or auto-converted content.\n3. Annotations are not single-point labels but structured, narrative explanations that mimic clinical reasoning, yielding high clinical value.\n4. Items are challenging and come with explicit difficulty tiers.", "weaknesses": "1. The paper should specify which baseline model(s) were used for the pre-release “challenge validation,” including the exact prompts and their performance.\n\n2. Grading relies primarily on GPT-4o as the automatic judge; despite high correlation with clinicians, model-specific bias may persist. Please explain why a multi-model, anonymous voting scheme was not adopted.\n\n3. “Normalized in length to control for spurious effects of input size on model performance”: length-related performance is itself part of model capability. Why normalize for this task, and how exactly was it implemented (e.g., truncation, abstractive summarization, padding to a fixed length, specific upper/lower thresholds, etc.)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)