Medical thinking with multiple images
Authors: Zonghai Yao
Benlu Wang
Yifan Zhang
Junda Wang
Iris Xia
Zhipeng Tang
Shuo Han
Feiyun Ouyang
Zhichao Yang
Arman Cohan
hong yu
Venue: ICLR 2026 Poster (2026)
Keywords: Multimodal diagnostic reasoning
Vision language models (VLMs)
Medical VQA
Thinking with images
Primary area: datasets and benchmarks

Abstract: Large language models perform well on many medical QA benchmarks, but real clinical reasoning is harder because diagnosis often requires integrating evidence across multiple images rather than interpreting a single view. 
We introduce MedThinkVQA, an expert-annotated benchmark for thinking with multiple images, in which models must interpret each image, combine cross-view evidence, and solve diagnostic questions under intermediate supervision and step-level evaluation. The dataset contains 10,067 cases, including 720 test cases, with an average of 6.68 images per case, substantially denser than prior work (earlier maxima $\leq$ 1.43). 
On the test set, the best closed-source models, Claude-4.6-opus, Gemini-3-pro, and GPT-5.2-xhigh, achieve only 54.9%--57.2% accuracy, while smaller proprietary variants, GPT-5-mini/nano, drop to 39.7% and 30.8%. Top open-source models perform worse overall, with Qwen3.5-397B-A17B (52.2%) and Qwen3.5-27B (50.6%) leading, followed by Lingshu-32B (43.2%), InternVL3.5-38B (40.7%), and MedGemma-27B (31.8%).
Further analysis points to a single-core bottleneck: current models struggle with grounded multi-image reasoning, i.e., reliably extracting, aligning, and composing evidence across views before higher-level inference can help. 
This is supported by three consistent findings: adding expert-provided single-image cues and integrating cross-image evidence improve performance, whereas replacing them with models’ self-generated intermediates reduces accuracy. 
Step-level analysis shows that over 70% of errors come from image reading and cross-view integration, with reasoning failures increasing on decisive steps. 
Scaling results show that while accuracy increases with more images, additional inference-time computation is beneficial only when the underlying visual grounding is already reliable. When early evidence extraction is weak, longer reasoning yields limited or unstable gains and can even amplify misread cues. 
Together, these results show that the main barrier is not simply insufficient reasoning length or depth, but the lack of reliable mechanisms for grounding, aligning, and composing distributed evidence across real-world, cross-view, multimodal clinical inputs.

Published as a conference paper at ICLR 2026
MEDICAL THINKING WITH MULTIPLE IMAGES
Zonghai Yau Benlu Wang", Yifan Zhang 2:4, , Junda Wang 1.2 Iris Xia 3, Zhipeng Tang '
Shuo Han 4, Feiyun Ouyang 4 Zhichao Yang Arman Cohan Hong Yu 1,2,4
1 Manning College of Information and Computer Sciences, UMass Amherst
"Center for Healthcare Organization and Implementation Research, VA Bedford Health Care *Department of Computer Science. Yale University
'Miner School of Computer and Information Sciences, UMass Lowell
zonghaiye yaoh@eiiyyeeee umass. edu, bennu. wang@yalee edu
ABSTRACT
Large language models perform well on many medical QA benchmarks, but real clinical reasoning is harder because diagnosis often requires integrating evidence across multiple images rather than interpreting a single view. We introduce Med- ThinkVQA, expert-anno appert-nnottted benchmark for thinking with multiple images, in which models must interpret each image, combine cross-view evidence, and solve diagnostic questions under intermediate supervision and step-level evaluation. The dataset contains 10,067 cases, including 720 test cases, with an average of 6.68 images per case, substantially denser than prior work (earlier maxima 1,43). On the test set, the best closed-source models, Claude-4.6-opus Gemini- 3-pro, and GPT-5.2-xhigh, achieve only 54.9% 50.557..% 37.2% accuracy. while smaller proprietary variants, GPTT---mini/nano, drop to 39.7% and 30.8%. Top open- source models perform worse overall, with Dwee3..-397B.
(52.2%) and Qwen3.5-27B (50.6%) leading, followed by Lingshu-32RB (43. 3.).% IntervVI.... 38B (40.7%), and MedGemma-2777 (31.8%) Further analysis points to a single- core bottleneck: current models struggle with grounded multi-image reasoning, i.e., reliably extracting, aligning, and composing evidence across views before higher-level inference can help. This is supported by three consistent findings: adding expert- provided single-image cues and integrating cross-image evidence improve performance, whereas replacing them with models' self-generated in- termediates reduces accuracy. Step-level analysis shows that over 70% of errors come from image reading and cross-view integration, with reasoning failures in- creasing on decisive steps. Scaling results show that while accuracy increases as more images are provided, additional inference-time time computation beneficial only when the underlying visual grounding is already reliable. When early evidence extraction weak, longer reasoning yields limited or unstable gains and can even amplify misread cues. Together, these results show that the main barrier is not simply insufficient reasoning length or depth, but the lack of reliable mechanisms for grounding, aligning, and composing distributed evidence across real-world, cross-view, multimodal clinical inputs. Code and example data are available at https:://gtthii com Biiapper-illy/Medddiee nVV
INTRODUCTION
Medical question answering has advanced quickly with large language models (LLMs) and vision-
language models (VLMs) Yan al.] (2033 Din al. 2024 Yang et al.] 2025 Li al. 2024| Chen et al. 2024b Yao & Yu 2025 Xie et al. 2024 Jiang et al. 2025). Many benchmark scores are now
high, and several exam- exam-style settings appear close to saturation (Jin et al. 2020 Hendryeks et al.
2020 Pal et al.) 2022, Jin et al.) 2019 Yao et al.) [224 Chen et al. 2024a Gilson et al. 2023). But real diagnostic reasoning is usually not a single question over a single image. In practice clinicians read the clinical scenario, inspect several views, combine findings across images, and only then narrow the differential diagnosis. Figure| (left) illustrates this workflow. This gap matters because a
"Equal contribution

---

Published as a conference paper at ICLR 2026
Medical thinking with images (TwI)
Diagnasing- Stayll Evaluation
ayed 72 yours complenned of ont-wesk chest pain and thrae-day exertional dyspnoee, There and no post histery of condievascular, reapirstery. ineetinll diceas...
Clinical Scenarios:
Step1: per-image radiological annotations Foure J: Chest X (PA Weal deterciratng situ Piyra 2/3/4 (S)
Steal steaa:
MedTwZ Eval
Clinical Sconario Understoniing
MedTwI Error Analysta
otegti step step
Image Perceptuol Error
Dieece Gorret?
Knoslacige
Medical Agent
Erer Anabys
Resstaning Process Errer
Case Discussion Evol
Education Volue Anolysis
Stepp:: case-level integnated imoging findings
Sectice by sectior and
D secse Celvetion chrecal
Cheat X-nay demeretrened
Each sectoo cintttte
Dagnatti kauder 5. Touelfinaee Ohe
gastre
bubille
sugaesfiot situs meeeruu totalis.
Medical Education Case Discussion
Eda senonstrated
Backgraudd
of thoaac and sbdoe
Step3: differential diagnosis reasoning
Situ Perspectives
The imaged show a clen miver image koysut heart sper parnts rihht werki traoee right, and obdennnal CT the enmechvippless are nigh--aceed shile the livr left-aded: slonse shut mverna fonnis with
Nohort with afts reveru teriho are feopuertly cove coveptanacic.
Tmaging Perspective:
Cottel theat natlo melogrohh saggeted dextrosondo
CT thomas
clivical Sigerficacc::
Recogsii sotceii on tevors tetele viua inclliidd and
Iaskted domroorrs (8) would keep rereel
Ouficoenc: con The patuert
Based provided images together the textual context, select the single best diagnosts from the options.
pernoleyy
Kartayonor systrone (5) neads
Retshen artifact (0) samot
Take Home Messege/ Teaching Palnts:
Sirul lnvissal NosaRENits cestrocrdd Dentmcaarde amua tnveriss
A cangnattel motforration with peritonaa doromeility (8)
Anarevers auch phomace vto essertel for acneett drggees fnaetmer paaning
Kartegoner
rstation orteecct Orggnital cardicc woffommatinn with pazit and obraredity
organ net this symmetric marrer paftere. Honce only (A) Sttx ierrrs tntala with deonscende fadinge
Ponert inforned then coedition, expecisity prepontion energnn
pietell rale confirming clegrent meee sted
Figure 1: Medical Thinking with Images (TwI): task and diagnosing-style evaluation. Left: sample case with clinical scenario, multi-view images (e.g., radiograph CT), and five-option single-best-anse diagnosis. Middle: TwI's three supervised steps: (1) Per-Image Findings (detect and name key radiological signs for each image, expert-annotated, brief statements); (2) Case-level Integrated Imaging Summary (synthesize cross-view evidence into a single case summary); (3) iitereential-iaannoost (DDx) reasoning (align the summary with candidate diagnoses, rule out distractors with image-grounded arguments, and pick the most consistent answer). Right: Beyond- accuracy evaluation. Steps 1-2 use automatic metrics (ROUGE/ RadCliQ), while Step 3 and the Medical Education Case Discussion are assessed with structured human- and LLM-judge rubrics that check clinical correctness and educational value, localizing failures in image reading, cross-view fusion, and teaching quality (see Section 3).
model can perform well on standard medical QA while still failing at the harder step of grounding, aligning, and composing evidence across multiple images. We therefore need a benchmark that tests diagnosis in the way it is often performed, with multiple informative views and explicit intermediate reasoning.
MedThinkVQA is designed around this (middle), the benchmark defines a three-step think-with-image (TwI) process: models first produce per-image findings, then synthesize them into case-level integrated imaging summarry, and finally perform differential -diaiiioi-s reasoning to choose the best diagnosis. This structure makes the diagnostic process observable, instead of evaluating only the final answer. MedThink also in- cludes a Medical Education Case Discussion task, where models generate teaching-styl explanations grounded in the case. This better reflects how clinicians explain findings and share reasoning in practice. Figure (right) further shows our diagnosing-style evaluation setup, which goes beyond answer accuracy examine stepwise correctness, error types, and education value, so failures in image reading, cross-view fusion, and higher-level reasoning can be localized rather than hidden inside a single accuracy number.
setting. As shown in Fig. D
Table T places MedThinkVQA in the context of prior multimodal medical QA datasets (Hu et al. 2024 Chen et al.) 2024c| Zuo et al., 2025). Compared with earlier benchmarks, MedTTiink VQA combines several properties that rarely appear together: expert annotation, real clinical scenarios, multi-modal imaging, longitudinal follow-up studies, intermediate supervision for tiirk--wtt-immagee reasoning, and beyond-accuracy evaluation. To our knowledge, it is the only benchmark in Table that satisfies all of these conditions. The dataset contains 10,067 cases, including 720 test cases, with an average of 6.68 images per case. Prior expert-level medical VQA benchmarks use far fewer images per case, with a previous maximum of at most 1.43. This difference is not only a matter of scale. changes the task itself, from recognizing clues in one image to integrating distributed evidence across
2

---

Published as a conference paper at ICLR 2026
Expert Ammutation
Clinical Scenarke
Iing per Cas
Multi-Mod
Longitud Studies
Think-wwith-Imang
Beyond-ACC
Benchmark
Case
Imaging
Intermediate Signals
Evaluation
Rad
Lau
451
K
0:45
VQA-Moc
Alacha th
500
1,00
Path-QQA
0.13
SLAKE-BIT
1,061
0.89
PMC-VQA VQA Zrogie [t
33,430
0.87
OmniMedlV que Sreeeeeeee
127,995
K
0.92
X
X
X
GMAL-MMBRnTT CennnniCCoo
21,281
1.00
GEMcX Liu
1,05,5775
1.00
ModirameDA
2,851
ICG.CXK Ma Bra
11,439
2.00
MadRAX Filtaliporr ct
2,500
185
X
GFMeX-TTime
MMMU
et
1,752
MMMU-Prr & clal
316
1,25
S-Chain Le-Duc el
12,000
1.00
MadXponqg Mtwiyiii er
2,000
1.43
MedThinkYQA
10,067
Table 1: Comparisons with multimodal medical QA benchmarks. MedThinkVQA is expert- annotated, averages 6.81 images per case (prior maxima 1.43; 2 4.5x more), and is the largest corpus at the expert-annotatio annotaaonooi level; checkmark in the Expert Annotation column indicates that items are curated and labeled by clinicians rather than automatically generated. Clinical Scenarios. Prior work lacks broad, fine-grained coverage of real diagnostic scenarios; only MedThinkVOn and MedXperiQA-MM) include scenario labels. Multi-Moda Imaging/ Longitudinal Studies. We mark Multi-Modal Imaging when at least some questions require joint interpretation of images from multiple distinct medical imaging modalities for the same case (c.g., radiograph+CT), and Longitudinal Studies when questions built from follow-up imaging of the baseline vs follow-up studies). Think-with- with--maaess Intermediate Signals. This merged column indicates whether benchmark provides intermediate supervision for think-with-imagee reasoning, including per-image findings, a case-level imaging swnmary, and a case discussion (teaching note). Beyond-ACC Evaluation. Leveraging these signals, only MedThinkVQA supports graigedin and-to-end assessment of think-with-iman -intei reasoning and teaching discussions: stepwise checks, error-type tags, education-van value scoring, and automatic intermediate metrics, rather than accuracy alone.
patient different time points (c.g..
views, modalities, and time. We further design MedThink VQA so that questions depend on imaging evidence rather than textual shortcuts; Section details the ICD-10 coverage, rare-disease cases, and the filtering and option policies used to control distractors, leakage, and surface biases.
Figure shows that this setting remains difficult even for the strongest current models. On the test split, the best closed-source systems, Claude-4.6- Sonde-4.6-ppuss 6-Opus, Gemini-3-pro. and GPT-5.2-xhigh, reach only 57.2%, 55.3%, and 54.9% accuracy, respectively. Strong open-source models trail behind, led by 3ren3..-397B-AI1 on the hardest prior benchmark of a related kind and clinician performance on our manually reviewed subset, indicating substantial headroom. The lower panels of Fig. 3 further clarify the source of the difficulty. First, accuracy rises monotonically as more images are provided, which shows that additional views contain real information gain and that the benchmark truly rewards using more visual evidence. Second, accuracy also tends to increase with greater reasoning effort, measured by output tokens. Third, within both Dense and Mixture-of-Expert (MoE) model f'amilies, larger models generally perform better, but their ability to benefit from extra reasoning budget differs: the medium-to-large (> 2B) Qwen3.5 family translates additional reasoning more effectively as model size grows, whereas Qwen3- VL and small-scale Qwen3.5 models show limited and unstable returns. Together, these trends point to a central conclusion: longer reasoning can help, but only after the model has already extracted and aligned the right visual evidence.
52.2%. These results remain far below both the strongest result reported
This conclusion is reinforced by our intermediate-signat and error analyses. Providing expert per- image findings and case summaries improves final accuracy, while replacing them with models' self-generated intermediates yields only small gains or even hurts performance. This means the main bottleneck is not simply the last reasoning step, but the earlier stages of grounded visual understanding and cross- cross-image composition. The expert audit shows that 88.05% of images are supportive for the final diagnosis, the test set contains about 2.30 imaging modalities per average, and 30.4% of test cases involve longitudinal follow-up up studies. In other words, most cases require aggregating multiple useful views rather than finding decisive image. Our step-level
3

---

Published as a conference paper at ICLR 2026
analysis supports the same story: across 202 labeled steps, 44 contain non-empty error tags, among these error- bearing steps, 77.27% involve muugeeunderstanding errors and 22.73% involve reasoning and scenario-setup eπrors much rarer. Even within Critical
errors, while medical-know/ln error-bearing steps, image understanding remains dominant. These findings align with the abstract and with Fig. 3 current models still struggle most with reliably extracting. aligning, and composing evidence across views before higher-level inference can fully help.
Contributions. (I) We introduce BedThhinVQA,, benchmark for multi-image diagnostic reasoning that supervises three explicit think serkk--wth-images steps. (2) We provide a beyond-accuracy evaluation suite, including automatic intermediate metrics, structured error-type type tagging, and education value scoring, together with scoring scripts and output formats. (3) We release a large, image-densse expert-annotated corpus (10,067 cases, 6.68 images per case) that, to our knowledge, the only benchmark that checks all columns in Table [ (4) We present evidence that the main bottleneck of current medical VLMS is not raw reasoning length alone, but reliable cross-image evidence extraction and integration.
MEDTHINKVQA
2.1 SOURCE CORPUS
MedThinkVQA is adapted from Eurorad, peer-reviewed online database of radiology teaching cases curated by the European Society of Radiology (eur). The corpus covers major subspecialties (neuro, musculoskeletal, thoracic, abdominal, pediatric, etc.) and common imaging modalities (X-ray, CT, MRI, ultrasound, etc.). Each case includes: (i) a brief clinical history; (ii) a multi-image set (average 6.68 images per case); (iii) radiologist-annotaee
per-image hints; (iv) a case-level Integrated Imaging Summary section; (v) an Expert Reasoning & Teaching Note that interprets the findings, highlights key diagnostic reasoning, and links clinical relevance:(vi) the final diagnosis; and (vii) differential diagnosis iifierentialldiannosiis list.
Cases are contributed by radiologists and researchers worldwide, typically based on real clinical examinations. Submissions are reviewed by the Eurorad Editorial Board (radiology experts) before publication to ensure authenticity and educational value (eur). We collected 10,067 cases and curated them into MedThink VQA. After post-processing, we formed a held-out test set with 720 cases. These cases span major imaging modalities in the test split and typically involve more than two distinct modalities per case; detailed modality statistics are provided in Appendix L For concreteness, detailed field-to- solld-toannntttionn examples and six representative Eurorad case studies are provided in Appendix E Details of the MCQ transformation and option policy are provided in Section 2.3
Eurorad materials use CC BY-NC-SA 4.0; MedThink VQA follows the same license and is for research and education only, with attribution and Alire,like, and commercial use. We worked with Eurorad and use the materials with permission. Cases are de-identified to the best of our knowledge; did not collect new personal data; IRB review was required; we remove items if residual identifiers are suspected. The benchmark is not a clinical device and must not be used for diagnosis, treatment, or triage. To lower leakage risk, we release collection and filtering scripts, run de- dupllicaation, and drop items that text-only models can solve; we also keep a path to refresh held-out items.
2.2 DATASET COVERAGE
Task framing. We characterize dataset coverage along two orthogonal axes: (i) a disease axis using ICD-10 chapters, and (ii) rudiology/medical imaging axis grouped by anatomy and subspecialty. The ICD-10 taxonomy contains 19 chapters. Using GPT-5.2 to map case labels to ICD-10, our test set covers 308 categories. providing coverage of long-tail conditions. A complete breakdown of ICD-10 chapters and subcategories is reported in the Appendix Q
To assess breadth from imaging perspective, we aggregated the full dataset by radiology subspe- cialties (anatomy & subspecialry). Figure 2 shows the distribution. The cases are not concentrated in single region but span across all major clinccal domains. The largest share comes from abdominal imaging (22.0%), followed by neuroradiology (16.0%) and musculoskeletal (14.3%). Mid-sized cate- gories include chest (9.6%), paediatric (8.0%), and urogenital imaging (7.5%), while cardiovascular
4

---

Published as a conference paper at ICLR 2026
(6.7%) and head & neck (5.7%) also make substantive contributions. Smaller but non-negligible proportions are represented in breast and interventional radiology. with hybrid imaging appearing only rarely (<0.1%). From temporal-structune perspective, roughly quarter of MedThink cases are longitudinal follow-up studies (multiple time points for the same patient), so temporal disease evolution is explicitly represented; detailed longitudinal statistics summarized in Appendix M
2.3 MCQ CONVERSION AND OPTION POLICY
Each case presented five-choice dongle-best-anweer MCQ: Given the clinical history and associated radiol- ogy images, select the most likely di- agnosis from the options. The ground- truth label is the case's s final diagnosis. While only the clinical history and im- ages are provided as imput context for
Ungrdbdogy Gental
Paodintic cadsinge
Robeenalingge
Rreastraate
21
67%
the QA task, we also retain other cu- rated textual fields (expert caption, In- tegrated Imaging Summary, and Ex- pert Reasoning & Teaching Note) in the dataset files for potential future use. If the source differential diagno- sis list has 2 5 candidates, we prune five using a confusion-awe aware ranking (keep the correct answer plus four distractors that models most often confuse with the truth). Duplicates or contradictions are rejected.
Mondestekee systes
Hond imuging
Head & noch imageo
Figure Distribution of radiology imaging main categories
2.4
TEST SET
We design the test split to stay as close as possible to expert reasoning and image-based decision making:
Overall
Samples Images
720
(1) Expert differential diagnosis as starting point. We first use cases where the expert differential list has 2 entries. The final diagnosis serves as the key, and the differential entries form the distractor pool. This ensures all candidate options come directly from experts and filters 2,996 data points.
5845
Per-sample
Imgs/sample
length
1338.6
Find. length
875.8
Disc. length
2732.8
(2) Leakage Detection. To ensure the rigor of the dataset, we conduct leakage detection each clinical history to verify whether it directly reveals the correct diagnosis. Specifically, we examine whether (i) the diagnosis label itself (exact name or ICD-standard term) appears in the text, (ii) synonyms, abbreviations, or eponyms are explicitly present, (iii) uncertain mentions of the label its variants occur (e.g., "?X." "rule out X," "suspected X," "possible X"). The detailed prompt used for this detection is provided Appendix H In total, 137 leaked cases are identified and removed from the dataset.
Option Length
Avg.
Num. Density
Macro
0.0033
Pos. cornect Mcan mod, ent All med. types Longit. shure (%)
2,84
30.4
(3) Confusion-aware pruning. Moreover, there are more than five distractors, we check which wrong answers the reasoning VLM (GPT- o4-mini) model picks mistakenly. We keep these frequently confused distractors when possible and sample the rest at random. Only deletions are made; the original Expert Reasoning & Teaching Note is lightly edited (via GPT-5 mini) remove references to deleted options (Appendix No new medical content introduced.
Table 2: Test stats (Cap/Find/Disc. = cap- tion, findings, discussion; Pos. correct = avg. po- sition of correct option; Mean mod. ent = mean # of imaging modali- ties; see Appendix [ modality statisties and M for longitudinal-study statisties.
(4) Remove text-solvable cases. To ensure that images genuinely necessary for solving the task, we conduct a two-stage text-only filtering procedure.
First, we evaluate each provisional item using four large-scale text-only models Saaells--Qwen--eeext-e-00 GPT-oss- 120B, MedGeema--2B-eenn and LJama-3.3-70B. Any item
5

---

Published as a conference paper at ICLR 2026
that all four models answer correctly is removed. This step filters out questions that can be reliably solved from the textual context alone, retaining only cases where imaging information is essential or plays substantial role. This stage removes 1,074 cases.
Second, to further climinate potentially text-solvable samples even after supervised fine-tuning (SFT), test with four SFT LLMs (Qwen2.5-3B, Mistral-7B-v0.22 Qwen3-4B, and Llama-3.1-8B). We remove cases in which all four models answer correctly. This stricter criterion ensures that even smaller-capacity models cannot consistently solve the question without visual input. This additional filtering step removes 180 cases.
(5) Surface Bias Mitigation We observed systematic surface bias in option length: in over half of the cases, the correct answer was the longest choice, far above the uniform expectation of 20% under five-option setting. This bias likely arises because correct diagnoses tend be phrased more specifically and with greater clinical detail for given patient, whereas distractors are shorter and more generic. Importantly, models achieved 5- 10 points higher accuracy on such items, suggesting partial reliance on superficial length heuristics rather than genuine multimodal reasoning. To prevent shortcut learning, we pruned items accordingly.
In addition, to reduce dataset-level structural skew, we further rebalanced the distribution of ICD disease categories and imaging modalities. We removed additional samples to improve the balance of labels and modalities across the dataset. In total, this stage removes 706 cases, improving both surface-level fairness and macro-level distributional balance.
2.5 MEDICAL EDUCATION CASE DISCUSSION
In clinical practice, difficult or representative cases are often documented as teaching notes and shared with trainees and colleagues, and the Eurorad "Discussion" sections already serve this role. The human-expert study in Appendix B further shows that even experienced clinicians find subset of MedTTinkVQA cases very difficult, motivating our Medical Education Case Discussion task, in which models asked to generate structured teaching content rather than predict single diagnosis. To make this evaluation well defined, we focus on cases whose discussions follow a clear five-section template, namely Background, Clinical Perspective, Imaging Perspective, Outcome, and Take-HHme Messages, yielding a subset of test cases that strictly conforms to this structure and supports section- by-section comparison. Details about Case Discussion Automatic/Human Evaluation and Results can be found at Appendix D
EXPERIMENTAL SETUP
Model Baseline We establish baselines using a diverse set of VLMS to ensure fair and representative evaluation. The selection spans both Inference-Time Scaled Large Multimodal Models (e.g., GPT-5.2 and Gemini-3-Proi and Vanilla Large Multimodal Models, which include open-weight generalist and medical-tuned families such as In family, MedGemma, and Phi, at different parameter scales (2B-397B).
Automatic evaluation 1. Intermediate imaging metrics For the per-image findings and the case- level integrated imaging summary (Steps 1-2 in Fig. [ follow recent radioloyy-repon evaluation work (Yu et al., 2023] Ostmeier et al. 2024) and compute ROUGE as a lexical-overlap baseline together with RadCliQ, which correlates more strongly with radiologist preferences. We apply these metrics to compare model outputs against the expert-written findings and summaries, providing automatic, fine-gre aieeggriied signals for how well models capture clinically salient details. 2. Stepwise Reasoning Evaluation We split each model explanation into atomic steps using GPT-5-MINI, then used GPT-5 as an L.M judge to label each step: factual correctness, whether is critical to the final diagnosis, and, if incorrect, an eтror type. When step is incorrect, the judge assigns one of four error types: clinical-scenario misunderstanding, missing or misread image evidence (Image Understanding Err), medical knowledge error, or flawed reasoning; this taxonomy is reused for both automatic and human evaluations. Overall, most failures stem from image misineepreaationniffomttioo extraction, especially on critical steps (69.23%). When answers wrong, Reasoning Err and Medical Knowledge Err become more prominent alongside the image errors (details in Appendix]) and Appendix 0).

---

Published as a conference paper at ICLR 2026
thinking
par
Ald
oven AMd
Like
Model sise scaling (Danta and HoE)
gpts-2
Scaling Reasoning Effort
Owend.a Pamily
QwendAVL Family
Larger reapesing modals ecoton
gpt-S-eano
232
Figure 3: MedThinkVQA benchmark results on the test set, highlighting the core challenge of thinking with multiple images: performance is limited less by raw reasoning length alone, and more by whether model can reliably ground, align, and compose evidence across views. Top: Overall test accuracy (%). When available, paired bars report the same model in thinking and matched non-thinking mode (gray); single bars report the thinking result when such mode cxists, otherwise the standard configuration. Detailed results are provided in Appendix C Bottom: (left) Masking fraction of images shows stable gains benchmark contains real cross-image information gain. However, larger reasoning models benefit more, suggesting that success depends how well additional images are actually used, rather than simply being available, The gains of thinking over non-thinking also peak at +8. 1/+6.5/43.7 points for GPT-5/mini/nana
visual evidence is provided, indicating that the
more visual evidence becomes available. (middle) Accuracy generally increases with greater reasoning effort, as measured by average output tokens, indicating a clear scaling trend. (right) Both Dense and MoF. L.MS within each family broadly follow model-size scaling, but their ability to convert extra reasoning budget into better answers differs. Within the medium-to- large (> 2B) Qwen3.5 family, reasoning gains increase as models scale, whereas Qwen3-V VL and small-scale Qwen3.5 models shows limited and unstable returns, suggesting that extra inference-tine time computation helps mainly when early visual evidence extraction is already reliable and cannot reliably compensate for weak multi-image grounding.
Human evaluation Two medical experts did a Stepwise Reasoning Evaluation on 50 cases (202 steps) for step factuality and error types. In total, 44 steps contained errors (21.78%), with Image Understanding Err dominant (77.27%), followed by Reasoning Err, supporting the automatic evaluation conclusion that image misinterpretation is the primary source of mistakes. Inter-rater agreement was high: Cohen's 's λ 0.82 between the two experts, and human-LLM,, LLM-LLMM-- agreement ranging from x 0.70 to K 0.84, confirming the reliability of the automatic judge.
4 RESULTS AND DISCUSSION
4.1 MAIN RESULTS
FigureBJreprte reppoott the overall MedThink leaderboard. Performance is still limited across all models.
The best result 57.2% from Claude-4.6-Opus, followed by Gemini-3-pro at 55.3% and GPT-5.2 2 at 54.9% in xhigh thinking mode. Strong open models are lower, ranging from the low-50s to the low-20s, including the Qwen3.5 family (21-52.2%) IntemVL3.55335 (40.7%), Qwen3-VL-32B (32.9%), MedGemma-277B (31.88 .)).. and Phi-4 (22.2%). Detailed results provided in Appendix C Overall, these results show that fith-k-ith-multiple--in diagnosis remains difficult even for current top multimodal systems.
7

---

Published as a conference paper at ICLR 2026
Inference- time thinking helps, but the gains moderate. For models with matched non non-thinking variants (gray bars in Fig. top), thinking usually improves accuracy by about 5-7 points. For example, GPT-5.2 increases from 49.9 to 54.9, GPT-5-mini from 33.2 to 39.7, and GPT-5-nano from 24.4 to 30.8. Similar improvements also appear in the Qwen3.5 MoF models. This shows that a larger reasoning budget is useful, but it does not eliminate the task's main difficulty.
S this bottleneck clearer. It evaluates several representative models under controlled input settings. When models rely solely on images, performance remains limited. When expert- written imaging text is added, accuracy improves substantially across models. This suggests that the main difficulty lies in extracting and combining diagnostic evidence from multiple images, rather than in reasoning over text once the key evidence is already provided.
4.2
IMAGE REASONING CAPABILITIES
Expert imaging summaries sharply boost accuracy. Across all four models, providing expert- written text extracted from images leads to large gains on MedThink VQA (Fig. 55. Feeding the Integrated Imaging Summary (expert) raises accuracy by +41.5, + 50.5, +41.5, and +42.0 points for MedGemma-277. GPT--5nano, GPT-5-mini, and GPT-5, respectively, corresponding to 2.19x, 2.60x, 1.95x, and 1.92x their baselines. Once this diagnosis-oriented summary available, adding an Image Hint (expert) provides only modest extra gains (-0.5-5.0 points), and the summary consistently outperforms the hint alone by -8.0-15.0 points. These patterns indicate that structured findings matter more than caption- descriptions: the summary encodes laterality, location, pattern, and extent in way that provides highly discriminative cues for differential diagnosis. The relative gains are larger for weaker baselines (e.g., GPT-5-nano 2.60x vs. GPT-5 1.92x). suggesting that once visual evidence is correctly verbalized, the remaining language inference is largely adequate-ann and the core obstacle is extracting and structuring pixel-level radiological evidence from multi-view inputs.
Self-generated text is fragile and consistently hurts. When models first write their own Hint/Summary and then condition on it, performance consistently drops across all models and all self-generated settings (Fig. 5). All four models underperform their baselines: MedGemma-27B decreases by 3.5 points with self-hints and by 12.5 points with sus--uummiics/ottti GPT-5-nam decreases by 3.0-6.0 points; GPT-5-mini decreases by 6.5 points; and GPT-5 decreases by 7.0-9.5 points. Tab. Tab. explains why: (ss 0.13-0.16) and imperfect RadCliQ-vi scores, meaning that self-produced descriptions often miss laterality, precise locations, or key patterns and may introduce subtle inaccuracies. In addition, noisy text increases sequence length and can dilute attention over multi-view inputs, and current VLMs may over-trust trust erroneous text when image- text grounding is weak.
generations achieve low ROUGE-L
4.3 SCALING ANALYSES: VISUAL EVIDENCE AND REASONING EFFORT
Larger reasoning models exploit visual evidence more effectively, but the payoff' is family- dependent. Figure (bottom-leef left) ablates visual evidence by masking a fraction of images. Accuracy improves monotonically more images are revealed, confirming that MedThinkVQA genuinely rewards multi-view evidence aggregation rather than single-image shortcuts. Importantly, the slope of this improvement is steeper for larger reasoning models: GPT-5 benefits more from increasing image ratios than GPT-5-mini and GPT-S-nano, suggesting a stronger capability to select, align, and fuse cross-view signals. Moreover, the advantage of thinking grows visual evidence increases and peaks at high image ratios, reaching +8.1/46.5/+3.7 points for GPT-5/mini/nanoo respectively. This interaction indicates that additional reasoning budget most useful when there is sufficient visual evidence to integrate and when the model can reliably ground its intermediate steps in that evidence.
The same principle helps interpret the contrasting behavior of Qwen families in Fig. 3 (bittom-right). Across both Dense and MoE variants, performance broadly follows model-size scaling, yet the marginal return of extra reasoning differs sharply by family and model performance: scaling the reasoning budget helps most Qwen3.5 increasingly as models grow, whereas Qwen3-VL and small Qwen3.5s fail to reliably translate additional reasoning tokens into accuracy gains. A plausible explanation is that Qwen3.5's stronger multimodal stack better supports evidence selection and views-view alignment, so added "thinking" tends to refine fusion decisions. In contrast, if Qwen3- VL and small Qwen3.5's 's visual grounding and fusion are weaker, additional tokens may preferentially
8

---

Published as a conference paper at ICLR 2026
elaborate misread or misaligned cues, effectively amplifying early perceptual errors rather than correcting them, yielding flat or unstable gains even with larger reasoning budgets.
This phenomenon is particularly pronounced in Qwen3.5-0.8B. due to its limited visual feature extraction capacity, the reasoning process largely reiterates incorrect visual interpretations. Errors accumulate across reasoning steps, and the final prediction can be substantially worse than its non-reasoning counterpart.
Scaling reasoning effort helps when the visual-evidence substrate is reliable. Figure 3 (bottom- middle) shows that accuracy generally increases with reasoning effort, operationalized by average output tokens (from minimal/none medium/high and xhigh). This pattern supports "more inference-ting compute + better diagnosis" trend, but the benefit remains limited. Together with the Qwen3.5 vs. Qwen3-VI contrast, these results suggest that inference-time scaling conditional: additional reasoning tokens are beneficial primarily when the model's 's image reading and cross-view alignment are sufficiently accurate to serve as a trustworthy substrate. When that substrate is noisy, longer reasoning may not help and can even entrench spurious evidence chains.
Implications and Future Directions. Inference-time scaling is useful but secondary lever in MedThinkVOA Our results suggest that extra reasoning tokens help only when the model can first read, align, and combine evidence across multiple images with reasonable reliability. In other words, stronger thinking cannot compensate for weak multi-image grounding, and may even reinforce early perceptual mistakes. This makes the main research target clearer: progress will likely depend less on simply extending reasoning length, and more on improving cross-view evidence selection, image-text grounding, and mechanisms that keep intermediate reasoning tied to image-supported evidence. From this perspective, several directions appear promising. First, process- supeevsse SFT and distillation use our per-image findings, integrated summaries, and option- option-wise eliminations as step labels, extending shain-of-thoughhe and SFT ideas from general medical MedVQA rationales to multi-image radiology /Zhang et al.) (2024); Gai al. (2024). Second, data-centric and alignment methods can better steer the model toward the right views, including medical-imase augmentation, generative counterfactual editing, and semi- radiology QA expansion Kebaili et al. (2023):
Shoer & Kementchedjhieval (2025): Wang et al. (2024), together with preference-base based objectives such as DPO and reinforcement learning that reward faige-faithful reasoning while penalizing shortcut solutions Rafailov et al. (2023): Liu et al. (2024b); Guo et al. (2025). Third, test-time enhancement may be more effective when reasoning paired with tool use rather than longer free- form deliberation alone. For multi-image diagnosis, models may benefit from calling external tools or knowledge sources to better interpret visual findings, resolve ambiguous patterns, and connect image evidence to domain knowledge. This could include retrieving relevant radiology knowledge, consulting structured medical references, or using lightweight intermediate modules that help map visual cues to clinically meaningful concepts before final reasoning. In this setting, the goal is not simply to produce more tokens, but to make each reasoning step better grounded in both the images and the external knowledge needed to interpret them.
4.4 DATA CONTAMINATION ANALYSIS
We assess potential test leak- age using strict sliding- window variant of MELD (Memorization Effects Leven- shtein Detector), which mea- sures character-level overlap between each model's 's gener- ated answer and its input ques- tion on the MEDTHINE VQA test set. Across seven representative LLMs/VLMs, MELD similarities cluster around 20-24% with narrow IQRs, and no item reaches the commonly used high-risk threshold of 50%. The distributions are also similar across text-only and vision-language models, suggesting no family-specific effect. Overall, we find no evidence of severe contamination.
Error type
All error steps (N-1509)
Critical error steps (N-182)
Image Understanding Er Reasoning Err
959 (63.55%)
126 (69. (99)
583 (38.63%
71 (39.01%)
Medical Knowledge Err
362 (23.99%)
60 (32.97%)
Clinical Scenario Err
191 (12.66%)
22 (12.09%)
Table 3:
LLM-judge error-type coverage. Note: categories are multi- label; percentages are step-level coverage over error steps and may sum to > 100%. Full per-split (answer-correct vs. wrong) breakdowns are in the Appendix.
Full details and boxplots are provided in Appendix P

---

Published as a conference paper at ICLR 2026
5 RELATED WORK
Early MedVQA corpora set task forms but had small scale or shallow reasoning (Ben Abacha et al.
2019 Lau et al. 2018 Liu et al. 2021 He et al. 2020 Zhang al. ) Later unified benchmarks grew breadth across modalities and specialties (Hu et al. 2024 Chen et al. 2024c). General expert- level suites also add a Health/Medicing subset and try to reduce shorteuts Yue et al., 2024a b). But most questions are single-imag or short-contexi, and many use automatic labels. Many datasets are built from image captions, so labels do not encode diagnostic reasoning or multi-image context. They also lack the detailed clinical information that real cases need. Coverage of medical image types is still limited compared to practice. Within chest radiography and longitudinal imaging, large-scale corpora such as GEMeX (Liu et al. 2025b), Medical-Diie -iaall--iii----- VQA (MIMIC-Dif's VQA) (Hu et al., 2023), ICG-CXR (Ma et al. 2025). and the multi-image MedFrameQA benchmark (Yu et al.) 2025) expand data scale and introduce explainable, difference-awand image settings. However, their QAs and rationales are largely produced by cule-based or GtT-style pipelines rather than per-item expert traces, most items remain single-view or image-pair based, and they do not provide the per-image findings, case-level imaging summaries, or teaching notes needed for stepwise diagnostic supervision. So evaluation stays answer-centric and lacks stepwise diagnostic supervision, as reflected in the upper rows of our comparison.
counterfactual, explicitly multi-
MedXpertQA raises difficulty and realism and has multimodal track with images and histories (uoo et al. 2025). It also provides scenario labels. But it does not release expert per-image findings or a case-level imaging sunmary, and it does not annotate option-wise eliminations. Items also use far fewer images per case (max S. 1.43), so cross-view fusion is not stressed. We fill these gaps with expert step labels (per-image findings and a case summary), with option-wise eliminations, and with reproducible beyond-accurage suite (step metrics, error types, and education scoring).
Eurorad-based studies often prompt models with textual descriptions from case reports (Kim et al.. 2025). This probes language use, but it does not test reading raw images. Text-only prompting cannot test multi-image fusion or image dependence. In parallel, agent-style evaluation on chest X-rays BeeeRAAAChestAAeeBBenchhh orchestrates segmentation, grounding, classification tools on curorad-deriver multiple-choice cases, but the released benchmark exposes only questions and images without the underlying expert step traces, complementing rather than replacing multi-image diagnostic supervision (Fallahpour et al. 2025). So our setting requires direct multi-image reading and option-wise, pridence-grounded elimination.
and
Work on reasoning supervision trains or audits how models explain answers (Gai et al. 2025 Liu
ct al. 2024a| Wang al. 2025) Fan et al. 2025). Prior efforts include chain-of-though generation. visually grounded reasoning, and cycle consistency. Recent corpora such as GEMeX- -Tink (Liu
ct al. 2025a) and S- S-Chain (Le-Duc et al., 2025) further introduce stsp-by-step rationales with explicit visual grounding and localization metrics (e.g., beyond answer-only evaluation while still focusing mainly on single-image cases without clinical scenarios or multi-view, case-level synthesis. These help transparency and stability. But most corpora traces, do not release expert, item-specific diagnostic traces tied to options. Without option-aligne contrastive fidelity checks and step-level rubrics are hard to standardize. We release expert per-image findings and a case-level summary, and we pair them with option wwiee climinations. This enables contrastive fidelity checks, step-level scoring, and LLM judges. Teaching discussions are also a standard product of medical education, yet benchmarks rarely evaluate this skill.
scores, A-score, mloU), moving
evaluation with human and
CONCLUSION
We introduce MedThinkVQA., a large-scale benchmark for multimodal diagnostic reasoning in radiology, built from authentic multi-image cases with expert-authored reasoning traces. Our results show that this setting remains difficult even for the strongest current models, with performance mainly limited by failures in grounded multi-image evidence extraction and cross-view integration. We hope MedThink VQA will serve as a rigorous testbed for developing models that can not only answer correctly but also reason more reliably in clinically realistic settings.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
Research reported in this study was supported by the National Center on Homelessness Among Veter- ans (NCHAV) and by the National Institutes of Health (NIH) under award number IR0INR020868, and 1101 1 -1. This study was also in part supported by NIH under award numbers RO1DA056470-AI and IROI AG0S0670-01, and by the U.S. Department of Veterans Affairs (VA) Health Systems Research. The content is solely the responsibility of the authors and does not necessarily represent NIH, VA, or the US government.
We sincerely thank Juncheng Huang. MBBS (Department of Diagnostic Imaging, National University Hospital, Singapore) and Chin Siang Ong. MBBS PhD MPH (Assistant Professor Surgery, Division of Surgical Outcomes, Surgery Center for Health Services and Outcomes Research, Department of Surgery, Yale School of Medicine) for their invaluable clinical expertise and dedication to this project. They contributed substantially to the expert annotation process of MedThinkVQA. Their rigorous review helped ensure the clinical fidelity of per-image findings, integrated imaging summaries, differential diagnosis construction, and case discussions. In addition, their professional insights guided the refinement of the held-out test set, the auditing of image-level supportiveness, and the identification of clinically challenging and longitudinal cases. Their careful clinical judgment and thoughtful feedback significantly strengthened the reliability, educational value, and medical realism of this benchmark. We are deeply grateful for their time, expertise, and commitment to advancing trustworthy clinical AI research.
REPRODUCIBILITY STATEMENT
We provide full details ensure reproducibility. Dataset sources and splits in Section im- plementation details and training practices are in Section We attached various prompts for data construction, LLM Judge in Appendix J We also include an anonymized code repository link in Abstract.
ETHICAL STATEMENT
Data source, licensing, and legal compliance. All cases adapted from Eurorad, a peer-reviewed educational database maintained by the European Society of Radiology. Eurorad materials are licensed under Creative Commons
4.0 International license. MedThinkVQA follows the same license. Released data are for research and education only; commercial use is prohibited. Derivative datasets must preserve attribution, non-commercial use, and Alike terms.
Human subjects and privacy. Eurorad cases are intended for education and de-identified to the best of our knowledge. We did not collect new personal data and did not recruit patients or lay participants; IRB review was not required. We reviewed materials for residual identifiers and removed items when concerns arose.
Evaluation reliability. We combine automatic scripts, expert review, and LLM-judges. On step- level labels, human-human agreement is Cohen's x = 0.822833, human 1-LLM- M-judge agreement 0.838357, and -LMM agreement is н 0.701566. These results support the stability of our automated judging, but LLM-judges do not replace expert oversight.
Bias and fairness. Educational repositories can encode geographic, demographic, and practice-style biases. Rare conditions and certain protocols are unevenly represented. Models trained or tuned on this benchmark may inherit such biases. We encourage stratified analyses and external validation before any deployment.
Safety and misuse. Models evaluated here are research artifacts. They must not be used for diagnosis, treatment, triage, other high-stakes tasks without added clinical validation, regulatory clearance, and domain oversight. Generated discussions may sound authoritative yet still be incomplete or wrong. Any downstream requires human supervision, documented fail-safes, and monitoring.
Transparency, reproducibility, and environment. We document data construction, metrics, and judging protocols. We release code, scoring scripts, and example data, subject third-party licenses.
11

---

Published as a conference paper at ICLR 2026
No hidden reward models, private test sets, special samplers used. We report hardware and runtime where relevant and encourage efficient evaluation to limit environmental impact.
Conflicts of interest and ethics compliance. All authors have read and will adhere to the ICLR Code Ethics for submission, reviewing, and discussion. Any sponsorships or competing interests will be disclosed in the author checklist.
Data leakage assessment and mitigation. As discussed in Section 4.4 we conducted internal checks for leakage and found obvious overlap between test items and publicly released training artifacts that we were aware of. We remove text-only solvable items, strip explicit textual shortcuts, and stress cross-image fusion. Still, the risk of leakage cannot be ruled out. To reduce risk further, will (i) release the full data collection and processing code for public audit, and (ii) maintain rolling test set covering the most recent 6-12 months of newly curated cases, with periodic updates and refreshed scores for reported models. We will also publish de-duplication scripts coxact/near-duplice filters on images and texts) and document all split procedures.
Others. MedThink VQA is a research benchmark, not a clinical tool. Expert-authored traces are pedagogical; they may overlook interpersonal nuances, local workflows, and institutional contexts. The multiple-choice setting enables standardized scoring; it also simplifies real diagnostic work and stops before treatment planning and longitudinal follow-up. Coverage broad but not complete across body regions, patient groups, vendors, devices, and acquisition protocols. Although cases span many conditions, some specialties (c.g.. pediatrics, psychiatry) and rare diseases remain underrepresented. All cases originate from single educational repository, populations, and imaging pipelines are likely. The dataset is currently English- multilingual generalization has not been tested. Annotations, while expert-writtent stylistic variation. Our LLM-as-Judde components improve scalability, but they can be prompt- sensitive and may reflect judge-model biases; we therefore report human agreement and keep experts informed. Finally, we evaluate stepwise reasoning for differential diagnosis; reference- free evaluation of clinical reasoning without ground-truu steps left for future work.
distribution shifts across hospitals,
still contain noise or
12

---

Published as a conference paper at ICLR 2026
REFERENCES
Eurorad - radiology teaching cases. https: https::/www.eurorad.cc org/
European Society of
Radiology, accessed 2025-08-28.
Asma Ben Abacha, Sadid A Hasan, Vivek V Datla, Dina Hommner-Fushmantd and Henning Müller. Vqa- med: Overview of the medical visual question answering task at imageclef 2019. In Proceedings of CLEF (Conference and Labs of the Evaluation Forum) 2019 Working Notes. 9-12 September 2019, 2019.
Hanjie Chen, Zhouxiang Fang, Yash Singla, and Mark Dredze. Benchmarking large language models on answering and explaining challenging medical questions. arXiv preprint arXiv:2402.18060
2024a.
Junying Chen, Ruyi Ouyang, Anningzhe Gao, Shunian Chen, Guiming Hardy Chen, Xidong Wang, Ruifei Zhang, Zhenyang Cai, Ke Ji, Guangjun Yu, et al. Huatuogpt- visiin, towards injecting medical visual knowledge into multimodal iims at scale. arXiv preprint arXiv:2406. 19280, 2024b.
Pengcheng Chen, Jin Ye, Guoan Wang. Yanjun Li, Zhongying Deng, Wei Li, Tianbin Li, Haodong Duan, Ziyan Huang, Yanzhou Su, et al. Gmai mmminch A comprehensive multimodal evaluation benchmark towards general medical ai. Advances in Neural Information Processing Systems, 2024c. URL htl.https::/eee //apxiv///axvvv///// www.oogg//bbbbe/eeoooooe 2408 03361
Adibvafa Fallahpour, Jun Ma, Alif Munim, Hongwei Lyu, and Bo Wang. Medrax: Medical reasoning agent for chest x-ray. arXiv preprint arXiv:2502.02677 2025.
Lin Fan, Xun Gong, Cenyang Zheng, Xuli Tan, Jiao Li, and Yafei Ou Cycle-vqa: A cycle-consistent framework for robust medical visual question answering. Pattern Recognition, 165: 1609, 2025.
Xiaotang Gai, Chenyi Zhou, Jiaxiang Liu, Yang Feng, Jian Wu, and Zuozhu Liu. Medthink: Explaining medical visual question answering via multimodal decision-making rationale. arXiv preprint arXiv:2404. 123.2 2024.
Xiaotang Gai, Chenyi Zhou, Jiaxiang Liu, Yang Feng, Jian Wu, and Zuozhu Liu. Medthink: A guidedd framework for explaining medical visual question answering. In Findings of the Association for Computational Linguistics: NAACL 2025, pp. 7438-7450, 2025.
Aidan Gilson, Conrad W Safranck, Thomas Huang, Vimig Socrates, Ling Chi, Richard Andrew Taylor, David Chartash, et al. How does chatgpt perform on the united states medical licensing examination (usmle)? the implications of large language models for medical education and knowledge assessment. JMIR Medical Education, (11)e45312, 2023.
Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in lmm via reinforcement learning. arXiv preprint arXiv:2501. 12948, 2025.
Xuehai He, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie. Pathvqa: 30000+ questions for medical visual question answering. arXiv preprint arXiv:2003.10288 2020.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint NEH::20009003000, 2020.
Xinyue Hu, Lin Gu, Qiyuan An, Mengliang Zhang, Liangchen Liu, Kazuma Kobayashi, Tatsuya Harada, Ronald M Summers, and Yingying Zhu. Expert knowledge-aware image difference graph representation learning for difference-aweir aware medical visual question answering. In Proceedings of the 29th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pp. 4156-44 4165, 2023.
Yutao Hu, Tianbin Li, Quanfeng Lu, Wenqi Shao, Junjun He, Yu Qiao, and Ping Luo. Omnimedvqa: A new large-scale comprehensive evaluation benchmark for medical Ivlm. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pp. 221 22170-22183, 2024.
13

---

Published as a conference paper at ICLR 2026
Songtao Jiang, Yuan Wang, Sibo Song, Tianxiang Hu, Chenyi Zhou, Bin Pu, Yan Zhang, Zhibo Yang, Yang Feng, Joey Tianyi Zhou, et al. Hulu-med: A transparent generalist model towards holistic medical vision- lsnonnlanguaae understanding. arXiv preprint arXiv:2510.08666. 2025.
Di Jin, Edward Pan, et al. What disease does this patient have? large- open-mmmaii medical qa dataset. arXiv preprint arXiv:2009.1300 13081, 2020. URL
13081
Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering. arXiv preprint arXiv: /99,0666,, 2019.
Qiao Jin, Fangyuan Chen, Yiliang Zhou, Ziyang Xu, Justin M Cheung, Robert Chen, Ronald M Summers, Justin F Rousseau, Peiyun Ni, Mare Landsman, et al. Hidden flaws behind expert-level accuracy of gpt-4 vision in medicine. arXiv preprint ar arXiv:240I.0S8: 00396, 2024.
Aghiles Kebaili, Jérôme Lapuyade-Lahorgen and Su Ruan. Deep learning approaches for data augmentation in medical imaging: review. Journal of imaging, (4):81, 2023.
Su Hwan Kim, Severin Schramm, Lisa C Adams, Rickmer Braren, Keno K Bressem, Matthias Keicher, Paul-Sören Platzek, Karolin Johanna Paprottka, Claus Zimmer, Dennis M Hedderich, et al. Benchmarking the diagnostic performance of open source Ilms in 1933 eurorad case reports. upj Digital Medicine, 8(1):97, 2025.
Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. A dataset of clinically generated visual questions and answers about radiology images. Scientific data, 5(1):1-10, 2018.
Khai Le-Due, Duy MH Nguyen, Phuong TH Trinh, Tien-Phat Nguyen, Nghiem T Diep, An Ngo, Tung Vu, Trinh Vuong, Anh-Tien Nguyen, Mau Nguyen, et al. S-chain: Structured visual chain-of- thought for medicine. arXiv preprint arXiv:2510.22728 2025.
Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang. Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems, 36, 2024.
Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang. and Xiao-Ming Wu. Slake: A semantically- labeled www.leedeeenaneedd dataset for medical visual question answering. In 2021 IEEE 18th international symposium on biomedical imaging (ISBI), pp. 1650-1654. IEFF, 2021.
Bo Liu, Xiangyu Zhao, Along He, Yidi Chen, Huazhu Fu, and Xiao-Ming Wu. Gemex- An enhanced med-vqa dataset for region region-aware aware multimodal chain-offthhuueeee reasoning. In Proceedings of the 33rd ACM International Conference on Multimedia, pp. 13213-13220. 2025a.
Bo Liu, Ke Zou, Li-Ming Zhan, Zexin Lu, Xiaoyu Dong, Yidi Chen, Chenggiang Xie, Jiannong Cao, Xiao-Ming Wu, and Huazhu Fu. Gemex: large-scale, groundable, and explainable medical vqa benchmark for chest x-ray diagnosis. In Proceedings of the IEEE/CVF International Conference on Computer Vision pp. 21310-21320, 2025b.
Jiaxiang Liu, Yuan Wang, Jiawei Du, Joey Tianyi Zhou, and Zuozhu Liu. Medcot: Medical chain of thought via hierarchical expert. arXiv preprint arXiv:2412. 2024a.
Ziyu Liu, Yuhang Zang, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Haodong Duan, Conghui He, Yuanjun Xiong, Dahua Lin, and Jiaqi Wang. Mia-dpo: Multi-image augmented direct preference models. arXiv preprint arXiv:2410.17637 2024b.
optimization for large rision-language
Chenglong Ma, Yuanfeng Ji, Jin Ye, Lu Zhang, Ying Chen, Tianbin Li, Mingjie Li, Junjun He, and Hongming Shan. Towards interpretable counterfactual generation via multimodal autoregression. In International Conference Medical Image Computing and Computer-Assiste Intervention. pp. 611-620. Springer, 2025.
Sophie Ostmeier, Justin Xu, Zhihong Chen, Maya Varma, Louis Blankemeier, Christian Bluethgen, Ame Edward Michalson Md, Michael Moseley, Curtis Langlotz, Akshay S Chaudhari, et al. Green: Generative radiology report evaluation and error notation. In Findings of the association for computational linguistics: EMNLP 2024, pp. 374-390, 2024.
14

---

Published as a conference paper at ICLR 2026
Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Medmeqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Conference on health, inference, and learning, pp. 268-260. PMLR, 2022.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36:53728-53747 53741,, 2023.
Belal Shoer and Yova Kementchedjhieva. A simple data augmentation strategy for text-in- -ext-in-image scientific vqa. In Proceedings of the 9th Widening NLP Workshop, pp. 100- 2025.
Xiangru Tang, Daniel Shao, Jiwoong Sohn, Jiapeng Chen, Jiayi Zhang, Jinyu Xiang, Fang Wu, Yilun Zhao, Chenglin Wu, Wenqi Shi, et al. Medagentsbench: Benchmarking thinking models and agent frameworks for complex medical reasoning. arXiv preprint arXiv:2503.07454 2025.
Junda Wang, Yujan Ting, Eric Z Chen, Hieu Tran, Hong Yu, Weijing Huang, and Terrence Chen. Semihvision: Enhancing medical multimodal models with semi-human annotated dataset and fine-tuned instruction generation. arXiv preprint arXiv:2410. 14948, 2024.
Yuan Wang, Jiaxiang Liu, Shujian Gao, Bin Feng, Zhihang Tang, Xiaotang Gai, Jian Wu, and Zuozhu Liu. V2t-cot: From vision text chain-of- -f-thughht for medical reasoning and diagnosis. arXiv preprint arXiv:2506. 1961:, 2025.
Yunfei Xie, Ce Zhou, Lang Gao, Juncheng Wu, Xianhang Li, Hong- Yu Zhou, Sheng Liu, Lei Xing, James Zou, Cihang Xie, et al. 25d: A large-scale multimodal dataset with multigranular annotations for medicine. arXiv preprint arXiv:2408 029000 2024.
Zhiling Yan, Kai Zhang, Rong Zhou, Lifang He, Xiang Li, and Lichao Sun. Multimodal chatgpt for medical applications: an experimental study of gpt-4v. arXiv preprint arXiv:23 10. 2023.
Zhichao Yang, Zonghai Yao, Mahbuba Tasmin, Parth Vashisht, Won Scok Jang, Feiyun Ouyang, Beining Wang. David McManus, Dan Berlowitz, and Hong Yu. Unveiling gpt-4v's hidden challenges behind high accuracy on usmle questions: Observational study. Journal of Medical Internet Research, 27:e651 776,6514,,20 2025.
Zonghai Yao and Hong Yu. A survey on Ilm-based multi-agent ai hospital. 2025.
Zonghai Yao, Zihao Zhang, Chaolong Tang, Xingyu Bian, Youxia Zhao, Zhichao Yang, Junda Wang, Huixue Zhou, Won Seok Jang, Feiyun Ouyang, et al. Medqa-cs: Benchmarking large language models clinical skills using an ai-sce framework. arXiv preprint arXiv:2410.01555 2024.
Feiyang Yu, Mark Endo, Rayan Krishnan, Ian Pan, Andy Tsai, Eduardo Pontes Reis, Eduardo Kaiser Ururahy Nunes Fonseca, Henrique Min Ho Lee, Zahra Shakeri Hossein Abad, Andrew Y. Ng, Curtis P. Langlotz, Vasantha Kumar Venugopal, and Pranav Rajpurkar. Evaluating progress in automatic chest x-ray radiology report generation. Patterns, 4()): 2023. doi: 10.1016/j. patter.2023. 100802. URL https:::p... /tw. http:://oooo.oo..oooo/oooooo 10tp://...11111/....1 pttppppaat...... 2023 200..1008222.0022222222002000020 100802
Suhao Yu, Haojin Wang, Juncheng Wu, Luyang Luo, Jingshen Wang, Cihang Xie, Pranav Rajpurkar, Carl Yang, Yang Yang, Kang Wang, al. Medframeqa: multi-image medical vqa benchmark for clinical reasoning. arXiv preprint arXiv:2505.1696 2025.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discin discipline multimodal under- standing and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9556-9567, 2024a.
Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun. al. Mmmu-pro A more robust dusti-discipline multimodal 2024b.
understanding benchmark. arXiv preprint arXiv:2409.022
Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Weixiong Lin, Ya Zhang, Yanfeng Wang, and Weidi Xie. Pinc-vqa: Visual instruction tuning for medical visual question answering. arXiv preprint arXiv:2305. 10415, 2023.
15

---

Published as a conference paper at ICLR 2026
Xuan Zhang. Chao Du, Tianyu Pang, Qian Liu, Wei Gao, and Min Lin. Chain of preference optimization: Improving ofiin-of-houugh reasoning in llms. Advances in Neural Information Processing Systems, 37:333-355 2024.
Yuxin Zuo, Shang Qu, Yifei Li, Zhangren Chen, Xuekai Zhu, Ermo Hua, Kaiyan Zhang, Ning Ding. and Bowen Zhou. Medxpertqa: Benchmarking expert-level medical reasoning and understanding. arXiv preprint arXiv:2501.18362, 2025.
16

---

Published as a conference paper at ICLR 2026
A LLM USAGE
In accordance with the ICLR 2026 policies on LLM usage, we disclose how LLMs were used in this work. LLMs were employed to assist with grammar polishing, wording improvements, and drafting text during paper preparation. All technical content, proofs, experiments, and analyses were conceived, implemented, and validated by the authors. Authors remain fully responsible for the correctness of the claims and results.
No L.MS used generate research ideas, write code for experiments, produce results. No confidential information was shared with LLMs, and no prompt injections or other inappropriate uses were involved.
This disclosure aligns with the ICLR Code of Ethics: contributions of tools are acknowledged, while accountability and verification rest entirely with the human authors.
B EXPERTS PERFORMANCE AND DATA QUALITY ANNOTATION
clinicians in active climical practice| We randomly sampled 96 test cases and ran a two-round expert study aligned with MCQ and stepwise evaluation. In Round 1, experts study images and selected one diagnosis out of five options, matching our VLM setup. In Round 2, they additionally received the full reference materials, including image captions, per-image findings. the integrated imaging summary, the teaching discussion, and the ground-truth answer, and audited each case for internal consistency, difficulty, and image redundancy, distinguishing supportive from redundant views. The same 96-case subset is used to evaluate VLM baselines for a fair human-model comparison.
Annotators and protocol. All annotations provided by two
only the clinical history and all
Round 1: Diagnostic Performance. Experts answered 74/96 cases correctly (77.10% accuracy).
Round 2: Data Quality and Image Redundancy. In the audit phase, experts marked 2/96 cases (2.1%) possibly inconsistent and 18/96 (18.8%) very difficult, indicating that the benchmark largely reflects coherent teaching cases while retaining a non-trivial proportion challenging items. For image redundancy, experts judged whether each view provided supportive evidence toward the final diagnosis. In 65/96 cases (463 images), all views were deemed supportive (100%). The remaining 31/96 cases contained 315 images, of which 222 (70.48%) were judged supportive. Overall, 685/778 images were rated as supportive (88.05%) %), with the rest considered redundant for determining the final diagnosis.
Case group
#Cases
#lmages
#Supportive imgs
Supportive ratio (%)
All images supportive Mixed supportive redundant
463
463
100.00
31
315
222
70.48
Overall (96 cases)
96
778
685
88.05
Table 4: Round expert audit: image-level redundancy support. Most cases use all images as supportive evidence; even in mixed cases, the majority of views remain supportive rather than redundant.
The expert study shows that experienced clinicians still clearly outperform state-of-the-an VLMs on MedThinkVQA. that the curated items overwhelmingly consistent with only small fraction flagged as potentially problematic, and that most cases require aggregating evidence from many supportive views. As shown in Fig. 3 and when image_ ratio 0 the task reduces choosing one diagnosis out of five options with nearly a random success probability of 20%, and accuracy then rises steadily larger proportion of case images is revealed across all models; together with the expert audit in Table 4 where 88% of images are rated supportive, this monotonic gain suggests that
'One annotator is diagnostic radiologist at tertiary academic hospital in Asia with 7 years of post- training experience, and the other is academic surgeon at U.S. medical school with 5 years of post-training experience.
17

---

Published as a conference paper at ICLR 2026
additional views are rarely pure redundancy and usually contribute useful diagnostic information, even though overall performance still remains well below human experts. At the same time, the realistic minority of redundant / non-supportive images means models must both integrate multiple supportive views and learn to down-weight redundant ones, mirroring how radiologists select and prioritize key views before forming diagnosis.
18

---

Published as a conference paper at ICLR 2026
Model class
0.45
Large models (>=)
Small models (<27B)
Accuracy (mean +/- std)
0.40
0.35
0.30
0.25
0.20
0
20
40 Image Ratio (%)
Figure Performance of open-source multimodal models on MedThink VQA under varying visible image ratios (0%-100%). Models are grouped by parameter scale into small (It 27B, n=14) and large (gt 27B, n=11) categories. See Appendix C for full list of models
19

---

Published as a conference paper at ICLR 2026
C ACCURACY BY IMAGE RATIO
We report accuracy (acc) and the number of correct predictions (acc_correch) on the full test (N 720). The image ratio r denotes the fraction of available images used as visual evidence: when r 0%, the input contains only clinical history (text-only); when r = 100%, all available images are provided (full visual evidence).
Reasoning Thinking setting. To make "reasoning effort" (OpenAl GPT models) and "thinking mode" (Qwen models) directly comparable and easier read, we separate the base model name from the Reasoning/ Thinking setting in the tables below. For models without an explicit reasoning/thinking control, we mark the setting default.
C.I
FULL TEST RESULTS (N 720, FULL VISUAL EVIDENCE; r 100%)
Table 5: Full-test results on the full test set (N 720) with full visual evidence (r 100%).
Model
Reasoning/Thinking
acc
#correct
OpenAl (GPT-5 family)
GPT-5.2
none
0.4958
GPT-5.2
low
0,5292
381
GPT-5.2
medium
0.5444
392
GPT-5.2
high
0,5333
384
GPT-5.2
x-high
0.5486
395
GPT-5 mini
minimal
0.3306
238
GPT-5 mini
low
0.3903
281
GPT-5 mini
medium
0.3958
285
GPT-5 mini
high
0.3972
GPT-5 nano
minimal
0.2444
176
GPT-5 nano
low
0.2806
202
GPT-5 nano
medium
0.3125
225
GPT-5 nano
high
0.3083
222
Google (Gemini)
Gemini 2.5 Flash
default default default
0.4270
307
Gemini 2.5 Pro
0.5097
367
Gemini 3 Pro
0.5569
401
Anthropic (Claude)
Claude 4.6 Opus
default
0.5722
412
Microsoft (Phi)
Phi-4-muliimdal-innntte
default
0,2222
160
MedGemma
MedGemma 1.5 4B
default default
0.2958
213
MedGemma 27B
0.3181
229
Qwen-Vr VI.
Qwen3-VL.C. V-BBBB
non-thinking
0.2972
214
Owen3-VL--22
thinking
0.2792
201
Qwcn3-VL--BB VL-4B
non-thinking
0.2697
194
Qwen3-VL-4B3
thinking
0,2895
208
Qwen3-VI..c VL-88B
non-thinking
0,3125
225
Owen3-VL-8P
thinking
0.2910
210
non-thinking
0.3109
224
thinking
0,2952
213
Qwen3-VL--225
non-thinking
0.3425
247
Owen3-VL-32B VL-32B
thinking
0.3292
Continued on next page
20

---

Published as a conference paper at ICLR 2026
Model
Reasoning/Thinking
acc
#correct
Qwen2.5-VI.-72B
non-thinking
0.4000
Qwen3-VL-235B--220 (FP8)
non-thinking
0.3528
254
Qwen3-V1.-25BB-222 (FP8)
thinking
0.3528
254
Qwen3.5
Qwen3.5-0..
non-thinking
0.3139
226
Qwen3.5-00 3.0.BB
thinking
0,2097
151
Qwen3,5-28
non-thinking
0.3250
234
Qwen3.5-2B
thinking
0,3000
216
Qwen3.5-4B
non-thinking
0.3444
248
Qwen3.5-4B
thinking
0.3764
271
Qwen3.5-98
non-thinking
0.3694
266
Qwen3.5-9h
thinking
0,4417
318
Qwen3.5-27B
non-thinking
0,4500
324
Qwen3.5-27B
thinking
0,5056
364
Qwen3.5-35B-A33
non-thimg non-thinking
0.4111
296
Qwen3.5-35B-A33.
thinking
0.4500
324
Qwen3.5-122B-AN
non non-thinking
0.4528
326
Qwen3.5-122BB-A100
thinking
0.5125
369
Qwen3.5-397B-AII (FP8)
non thinking
0,4569
329
Qwen3.5-397B-AII (FP8)
thinking
0.5222
376
Lingshu (vLLM)
Lingshu-7B
default
0.3694
266
Lingshu-I-SB
default
0.2917
210
Lingshu-32B
default
0,4319
311
InternVL.3.5 (yLLM)
V13.5-1.3.
default
0.3778
272
Intem 5
default
0.3556
256
VL..5BL.3..54B
default
0.3375
243
IntemVL3.5-8B
default
0.3347
241
Intern VL3.5113
default
0.3569
257
IntemVL3..530BB
default
0.3542
255
default
0.4069
293
C.2 SMAGE-RATIC ABLATION (r {0,20 20, 40, 60, 80)%
The r = 100% results coincide with the full-test results in Table 5 and are therefore not repeated here. Each cell shows newace (#correct); missing entries indicate that model not evaluated that ratio in se selected weleeeed_mddee_aeee
csv.
Table 6: Image-ratio ablation results (r E {0, 20, 40, 60, 80)%).
Model
Reasoning/Thinking
20%
60%
80%
OpenAT (GPT-S family; APT)
GPT-5.1
0.3389
GPT-5,2
low
0.3653 (263)
GPT-5
medium
0.3264 (235) 0.3736 (269)
0.4319 (311)
0.4583 (330)
0.4736
(202)
(274)
0.3931 (283)
GPT-5 mini
medium
0.2847 (205) 0.3528 (254)
0.3542(255
0.3667 (264) 0.3861 (278)
GPT-5 mini
minimal
0.2625 (189) 0.3097 (223)
0,3153 (227)) 0.3278 (236) 0.3250 (234)
GPT:5 nano
low
0.2097 (151) 0.2653 (191)
0.2708 (195)
0.2639 (190) 0.2806 (202)
GPT-5 nano
medium
0.2250 (162)
0.2819 (203)
0,3000 (216)
0.3111 (224)
0.3111 (224)
GPT:5 namo
minimal
0.2125 (153) 0.2236 (161)
0.2208 (159)
0.2417 (174) 0.2375 (171)
MedGemma (VI.LM)
MedGemma 4B (TT)
default
0.2319 (167) 0.2611 (188) 0.2569 (185) 0.2597 (187) 0.2597 (187)
Lingshu (vLLM)
Continued page
21

---

Published as a conference paper at ICLR 2026
Model
Reusoning/Tinnkin
0%
20%
40%
60%
80%
Lingshu-7B
default
0.2792 0.311 (224)
0.3458 (249)
0.3694 (266)
0.3694 (266)
Lingshu-I-8B
default
0.2903 (209)
0.2900 (209)
0.2900 (209)
0.2903 (209)
0.2875 (207)
Lingshu 32BB
default
0.3208 (231)
0.3625 (261)
0.4069 (293)
0,4250 (306)
0.4194 (302)
I.ternVI.3.5 (VI..MM
default
0.3292
0.3639
0,3667 (264) 0.3639
0.3708 (267)
In15m
default
0.2944 (212) 0.3181 (229)
0.3250 (2344)
0.3189 (244)
0.3389 (244)
IntemVL3.5-4B
default
0.2375 (171)
0,2819 (203)
0.2931 (211) 0.3153 (227)
0.3389 (244)
default
0.2208
0.2681 (193)
0,2847 (205)
0.3222 (232)
1n1emWL33.--14B
default
0.2500 (180) 0,2944 (212)
0,3222 (232) 0.3333 (240)
0.3361 (242)
IntemVLL3.520B-AABB
default
0.2458 (177)
0.2958 (213)
0,3208 (231) 0.3458 (249)
0.3542 (255)
IntemVL.3.5.300
default
(173)
(229)
0.3569 (257)
(267)
0.4133 (298)
Qwen-VI. (vLLMM)
Qwen3-VL-28
non-thinking
0.2875 (207) 0.2944 (212)
0,3014 (217) 0.2931 (211)
0.2931 (211)
Qwen3-VL-28
thinking
0.2750 (198) 0.2806 (202)
0.2625 (189) 0.2861 (206)
0.3000 (216)
Qwen3-VI.4B
non-thinking
(171)
(181)
0,2389 (172)
0.2569 (185)
0.2625 (189)
Qwen3 VI.-BB
thinking
0.2236 (161) 0.2806
0,3042 (219) 0.2972 (2
0.3153
Dwenn--V--BB
non- thinking
0.2347 (169) 0.2569 (185)
0.2778 (200) 0.2917 (210)
0.3181 (229)
Qwen3-VL-SBB VL-SB
thinking
0.21111 (152) 0.2639 (190)
0,3014 (217) 0.3056 (220)
0.2944 (212)
Wwva3-VL-32B
nea-thinking
0.2444
(211)
0.2944 (212)
0.3250 (234)
Owen3-VL-3322
thinking
0.2361 (170)
0.2694 (194)
0,3097 (223) 0.3069 (221)
0.3014 (217)
Qwen2.5 72B
non thinking
0,2250 (162)
0.2889 (208)
0.3111 (224)
0.3264 (235)
0.3278 (236)
Qwen3.5 (vLLM)
Qwen3.5-27B
non-thinking
0.3278 (236) 0.3514 (253)
0.3806 (274) 0.4097
0.4319 (311)
Qwen3.5-27B
thinking
0.3875 (279)
0.3806 (274)
0.4347 (315)
0.4569((32)
0.4861 (350)
Qwen3 --35B-ABB
non-thinking
0,2736 (197)
0.3083 (222)
0,3639 (262)
0.3819 (275)
0.3889 (280)
thinking
(227)
(267)
0.3917 (282)
(310)
(324)
Ow053.5-12BB-A108
thinking
0.4681 (337) 0.4806 (346)
22

---

Published as a conference paper at ICLR 2026
ROUGE-L (1)
RadCliQ-vl (f)
Model
Caption
Findings
Caption
Findings
gpt-5-nano
0.1435
0.1585
0.8080
0.6781
gpt-S-mini
0.1510
0.1636
0.8317
0.6931
GPT-5
0.1534
0.1627
0.8341
0.6818
medgemmaa27bbiit
0.1336
0.1621
0.7810
0.7192
Table 7: Scores of VLMs for Image->Caption and Image-+Findings across metrics (ROUGE-L and RadCliQ).
Beseline Image Hart iself Annotation)
Imaging Summary (Copert Annotation)
(uell Arnotation)
prated Imaging Summary tsetr Asnstation)
90
08.0
$6.0
$2.0
80
76.5
70
Accuracy (%)
62.0
60
50
40
26.0
31.5
30
28.5
26.5
22.5
22.5
20
MedGemma-278
GPT-S-nano
GPT-5-mini
GPT-5
Figure: 5: Accuracy on 200 randomly sampled cases from the MedThinkVQQ when augment- ing images with text. We compare Image Hint (caption-like) and Integrated Imaging Summary (diagnosis-oriented findings), each provided either by an expert or generated by the model itself (self), Both combines the two.
23

---

Published as a conference paper at ICLR 2026
D MEDICAL EDUCATION CASE DISCUSSION
D.1 CASE DISCUSSION AUTOMATIC EVALUATION
We implemented comprehensive automatic evaluation framework assess the quality of generated case discussions using GPT-5 as evaluators. Each generated discussion contained multiple subsections including background, clinical perspective, imaging perspective, outcome, and take-home messages. Our evaluation employed two-stage approach: first, we conducted sentence-level factual correctness assessment by splitting each subsection into individual sentences and tasking prompted LLM (GPT- 5) to judge the correctness of each sentence based the provided case context, imaging findings, differential diagnosis list, image captions, and medical images. The evaluator was instructed to mark sentences true if explicitly supported reasonably inferable from the context, and false only if clearly contradictory or incorrect. Second, we performed quality assessment using an expert-curated rubric that scored discussions five key criteria: disease overview, clinical pathophysiology. imaging analysis, reasoning and differentials, and transferable learning, with each criterion rated on a 0-2 scale. The LLM evaluator provided both numerical scores and brief justifications for each rubric criterion, focusing on medical accuracy, completeness, educational value, and integration of clinical and imaging perspectives. For the automatic evaluation, we randomly sampled 20 case discussions from our dataset for GPT-5 to evaluate using this framework.
D.2 CASE DISCUSSION HUMAN EVALUATION
To validate our automatic evaluation framework, we conducted human evaluation using two medical experts who independently assessed radiology discussions. Each evaluator presented with one case discussion randomly selected and generated by three different models, ensuring blinded assessment without knowledge of the generating model. Following the two-stage methodology as the automatic evaluation, the human evaluators first performed sentence-level factual correctness evaluation and then the evaluators applied the expert-curated rubric to provide quality scores. This human evaluation served as the gold standard for assessing the reliability and validity of our automated evaluation approach.
D.3
CASE DISCUSSION EVALUATION RESULT
The generated case discussions demonstrated high factual accuracy across all tested models, with overall correctness rates ranging from 92.81% % to 99.22% shown Tab. g The GPT-5 series consistently achieved the highest factual correctness, while the Clinical Perspective subsection scored highest across all models (97.89-100%). The Outcome subsection showed some performance differences, with MedGemma-27B achieving 85.71% compared other models' which scored above 95%. The rubric-based evaluation revealed GPT-5 achieving the highest overall score of 9.9/10, MedGemma-27B scored 7.05/10, showing particular weakness in clinical pathophysiology (1.15/2) and reasoning differentials (1.1/2), while all models demonstrated consistent strength in disease overview and imaging findings (Tab.
Model
Background (%)
Clinical (%)
Imaging (%)
Outcome (%)
Take-Home (%)
Overall (%)
gpt-5 gpt-5-mini
100.0
100.0
97.81
98.70
100.0
99.08
98.59
98.65
99.10
100.0
100.0
99.22
gpt-5-nano
97.87
98.99
97.39
95.89
98.46
97.76
medgemma-27b-it
89.0
97.89
94.93
85.71
93.65
92.81
Table 8: Sentence-level factual correctness evaluation across discussion subsections
24

---

Published as a conference paper at ICLR 2026
Model
Total
Disease Overview
Clinical Pathophys.
Imaging
Reasoning Different.
Transfer
Learning
gpt-5
9.9
2.0
1.9
2.0
2.0
2.0
gpt-5-mini
9.4
1.95
1.6
2.0
1.85
2.0
gpt-S-nano
8.4
1.7
1.25
2.0
1.45
2.0
medgemma-27b-it
7.05
1.4
1.15
1.85
1.1
1.55
Table 9: Rubric evaluation scores across different models
E EXAMPLE MAPPING FROM EURORAD FIELDS TO MEDTHINKVQQ ANNOTATIONS
To make the supervision signals MedThinkVQA concrete, this section uses single Eurorad case from the test split, "Ureteropelvic junction laceration following blunt trauma" whose processed ISON is stored at cases/ hwrgggnnerraaaeeee /ases/general caseeaaseeaaa///ee 9 case. jon. We list the main ISON fields and show how they correspond to the supervision concepts used in the main text.
Clinical scenario.
JSON field: CLINICAL_HISTIN (string).
Content: brief free-text description of the presenting complaint and relevant history (for case 219, an elderly patient with cardiovascular comorbidities presenting with right-sided thoraco-abdominal trauma and microscopic haematuria).
Main-text concept: this field is used verbatim the Clinical Scenario shown to models before any images or answer options (Fig. left).
Per-image hints (Image Hint per-image findings).
JSON field: img (list). Each element dictionary with keys img iin path, img_al t (short legend), and img_att2 (full descriptive caption).
Content structure (case 219):
Images 1 1-3: prior multidetector CT study months earlier, showing bilateral peripelvic renal cysts with otherwise normal renal morphology.
Images 4-5: current CT for abdominal trauma, with right perirenal and fascial fluid and dependent hyperattenuation in the renal pelvis compatible with acute blood.
Image 6: anterial-phass CT and MIP reconstructions, without contrast extravasation, again emphasising hyperattenuation in the renal pelvis and peripelvic cyst.
Images 7-8: delayed excretory -pase images, showing medial perirenal extraluminal opacified urine and normal wawpacccmmmm/ollleiingyyyee
opacification.
Images 9-10: delayed images showing extraluminal opacified urine arising from focal breach at the ureteropelvic junction and an opacified proximal ureter.
Main-text concept: Emg_alt? provides the expert per-image hinr used in Step (Jm- age Hint / per-image findings). In the TwI setting, models are asked to produce concise radiological finding sentences that are consistent with these captions.
Case-level Integrated Imaging Summary.
JSON field: TMAGING_F SINGG_FINDINGN INGS (string).
Content: case-level narrative integrating all imaging examinations (prior CT, current CT, delayed acquisitions), key abnormalities (peripelvic cysts, perirenal fluid, extraluminal opacified urine from UPJ breach), and absence of other traumatic lesions, plus immediate management (e.g., ureteral stenting).
Main-text concept: this field is the expert reference for the Integrated Imaging Summary (Step 2 in Fig. models must fuse per-image findings into a single summary that matches this cross-view evidence.
25

---

Published as a conference paper at ICLR 2026
Differential diagnosis and MCQ construction.
JSON field: LLF_DIAGNOSSL LIST (string with comma-separated diagnoses).
Content (case 219, simplified): contains the target diagnosis "Ureteropelvic junction lacera- tion following blunt trauma" and related entities such "Ureteropelvic avulsion", "Renal parenchymal laceration with calyceal disruption", "Urinoma", "Perinephric haematoma", and "Subcapsular haematoma".
Additional JSON fields used for the MCQ:
options: dictionary mapping option letters ("A"-"E") to diagnosis strings.
correct_answer: the correct option letter (e.g., "C").
the correct diagnosis string (e.g.. "Ureteropelvic junction laceration following blunt trauma.").
Main-text concept: these fields instantiate the five-option single-best-nnswe MCQ used in Step 3 (Differentill-iaagnoss
Reasoning); models compare their Integrated Imaging Summary to the opt i ons and must select sarreet_aswwe_ttee
Medical Education Case Discussion.
JSON field: DISCUSSION (string).
Content: long-form teaching text covering epidemiology (e.g.. rarity of UPJ injuries), mechanisms, imaging pitfalls, management strategies, and prognosis.
Main-text concept: this field is the expert reference for the Medical Education Case Discus- sion task, where models generate a structured explanation (Background, Clinical Perspective, Imaging Perspective, Clinical Significance, Outcome, Take-Home Notes) that graded against DISCUSSION for clinical correctness and educational value.
Overall, this case illustrates how raw Eurorad sections and figure captions are mapped onto the Clinical Scenario, Image Hint, Integrated Imaging Summary, Differential Diagnosis, and Medical Education Case Discussion supervision signals defined in the main text and implemented as JSON fields in MedThinkVQA.
26

---

Published as a conference paper at ICLR 2026
E. MULTIMODAL CASE STUDY: PRIMARY CARCINOMA OF THE RECTOVAGINAL SEPTUM
This MedThinkVQA case is 61-year-old woman with pelvic pain and inguinal lymphadenopathy, ultimately diagnosed with primary carcinoma of the rectovaginal septum. the JSON, all four modalities (Endoscopy, CT, MRI, Histology pathology) share the same CLINICAL HISTORY, BAAGINGGFIINDNNGN DISCUSSION, and each image is referenced by its img_idi id and stored under www.iiaaaaaaaaa /oodallss/ddlllla mto// tty/ (img_id) jpg with img_a a 1t and img_all a22 captions.
Endoscopy. The endoscopic modality contains single sigmoidoscopy frame (img_id 191MrGt3) that documents both focal bulging of the sigmoid wall and a cm vegetative rectal lesion; these findings encoded in the corresponding imq_al alt and img_at2 fields.
Figure 6: Endoscopy image for this case (img_id id = 19iMrGt3),
CT. The CT modality consists of two axial contrast-enhancee CT images (euIJtKe- pEX97IC8) that show a heterogeneous mass in the pouch of Douglas, invasion of adjacent structures, struccuurss and inguinal lymphadenopathy; the excretory-phar -phase scan with rectal contrast further clarifies rectal wall involvement.
Axial contrast- enhanced CT with pelvic mass and enlarged left inguinal nodes.
BulJtKe-
PfX97TC8. Excretory-phase CT with rectal contrast, better defin- ing rectal wall involvement.
Figure 7:
CT images (img_id = 3uIJtKe-, pfx97Ic8)
stored
under
images casesaaass.eeee / moda/ity/.....oo.oo///mm
MRI.
The MRI modality includes five 12-weighted images (O5kEGVZa, IAVIh4UN, BAAhh4N,,FJWW.A F POKI jEeq, 014cWn_5 __ that jointly characterise location (rectovaginal septum), extension to cervix and myometrium, intimate contact with the rectal wall, nodal disease, and preservation the right ovary and inner cervical stromal layer.
27

---

Published as a conference paper at ICLR 2026
O5kEGV74. Sagittal T2: mass in pouch of Douglas extending to cervix and myometrium.
IAVIh4UN, Preserved low-signal inner cervical stroma.
FjWYFZXB. Mass inseparable from the anterior rectal wall.
FOKIjEeq. Multiple enlarged
U14cWn 5. Coronal T2: normal right ovary separately identified from the mass.
pelvic and abdominal lymph nodes.
Figure 8: MRI images (img_id O5kEGVžq, IAV1h4UN, FJWYF FOKI jEeg, U14cWn_5) 5) with uniform image size and aligned top edges.
Histology pathology. The histology / pathology modality contains a single composite slide (3xrCMRPY) showing solid tumour growth with marked atypia and immunostaining for CAM5.22 2, CK7, and WT1, all encoded the img_alt2 description.
Figure 9: Histology pathology image for this case (img_id id 3xrCMRPY).
Multimodal reasoning signal. In the dataset, this case is formatted five-option diagnostic MCQ with ground-truth label primary carcinoma of the rectovaginal septum. Endoscopy and CT highlight an extraluminal pelvic mass with rectal involvement; MRI localises the tumour to the rectovaginal septum and shows preserved cervix and ovaries with nodal spread; histology confirms Müllerian-type carcinoma. A model must integrate all four modalities together with the shared textual fields in the JSON to distinguish this entity from rectal, cervical, and ovarian primaries.
28

---

Published as a conference paper at ICLR 2026
E.2 LONGITUDINAL CASE STUDY: CYSTIC PULMONARY TUBERCULOSIS
This MedThinkVQA case a nine-year-old boy with severe cystic pulmonary tuberculosis, followed radiologically over almost a year. In the JSON, the shared CLINICAL_HISS _HTTOOYY, IMAGING DING_FINIINGSSS and DIsCUSSIO ION fields are linked to chest radiographs and CT scans multiple time points. Below, we group images by clinical time point illustrate longitudinal disease evolution.
Baseline imaging admission. Baseline chest radiographs (posteroanterior and lateral views) show a bilateral diffuse micronodular acinar infiltrate. A same-day chest CT (lung and mediastinal windows) reveals a diffuse micronodular pattern with random distribution throughout both lungs. suggestive of an inflammatory or infectious process.
PA view. Baseline chest radiograph with diffuse mi- cromodular infiltrates.
Lateral view. Baseline chest radiograph confirm- ing bilateral involvement.
CT, lung window. Dif- fuse micronodular pattern in both lungs.
CT, lung window. Randomly distributed nodules throughout the parenchyma.
CT, mediastinal window. No large focal mass; dif- fuse micronodular disease.
Figure 10: Baseline chest radiographs and CT at admission, all displayed with a uniform relative size.
Early course with pneumothoraces. During the ICU stay. the patient develops spontaneous pneumothoraces requiring chest drainage. Serial radiographs show persistent diffuse micronodular infiltrates with evolving unilateral and bilateral pneumothoraces and multiple chest tubes in place.
Development of confluent cystic disease. A subsequent contrast-enhanced CT demonstrates extensive confluent cystic lesions predominantly in the upper lobes and posterior regions, consistent with cystic pulmonary tuberculosis and explaining the recurrent pncumothoraces.
Persistent cysts and larger pneumothoraces. A further CT shows similar cystic disease but larger bilateral pneumothoraces, pneumomediastinum. and multiple chest drains in place, underscoring the mechanical complications of cystic tuberculosis.
Pre-discharge CT. Before discharge, CT still shows cystic lesions and residual pneumothorax, but with improved overall ventilation. The patient tolerates these sequelae after chest tube removal and leave the hospital.
Late follow-up. At eight months after discharge, follow-up CT shows near- mear-complete resolution of the cystic and nodular lesions, with only subtle residual cysts and fibrotic sequelae.
29

---

Published as a conference paper at ICLR 2026
Chest radiograph. Right pncumothorax with chest tube in situ.
Chest radiograph. Persis- tent diffuse micronodular opacities.
Chest radiograph. Bilat- eral lung involvement with scattered nodules.
ZAC
Chest radiograph. Multi- ple thoracic drains for current pneumothoraces.
Chest radiograph. Per-
Chest radiograph. Diffuse cydtic-nodular
sistent residual pneumoth- orax.
changes on a background of severe disease.
Figure 11: Serial chest radiographs during ICU stay, showing pneumothoraces and multiple chest drains.
CT, lung window. Multi- ple cystic lesions in both upper lobes.
CT, lung window. Coa- lescent cysts forming large fillei spaces.
CT, lung window. Cys- tic lesions with surround- ing ground-glass opacities.
CT, coronal view. Upper- lobe predominance of eys- tic disease.
CT, lung window. Poste- rior lung involvement with confluent cysts.
CT. mediastinal window. Multiple large cystic spaces without solid mass.
Figure 12: CT during peak disease severity, with confluent cystic lesions and diffuse parenchymal involvement.
Longitudinal reasoning signal. In MedThinkVQA. this case is encoded five-option diagnostic MCQ with the correct answer cystic pulmonary tuberculosis. A model must integrate longitudinal information across all time progrssnoo from micronodular infiltrates confluent cystic dis-
30

---

Published as a conference paper at ICLR 2026
CT. lung window. Large right pneumothorax on eystic background.
CT, lung window, Ex- tensive bilateral changes.
CT, coronal view. Bilat- eral pncumothoraces with upper-lobe cysts.
cystic
CT, mediastinal window. Pneumomediastinum and thoracic drains.
CT, lung window. Persis- tent cystic lesions despite drainage.
Figure 13: CT with larger pneumothoraces and pneumomediasinnum, a background of cystic pulmonary tuberculosis.
CT, lung window, Resid- ual cystic changes with im- proved aeration.
CT, lung window. De- creased extent of parenchy- mal disease.
CT, coronal view. Tra- cheostomy in place with residual cysts.
Figure 14: Pre-discharge CT: persistent cystic lesions but improved clinical tolerance and removal of chest drains.
ease, recurrent pneumothoraces requiring multiple drains, and eventual radiologic recovery-together with the clinical text distinguish this entity from other cystic lung diseases (e.g., Pneumocystis jirovecii pneumonia, Langerhans cell histiocytosis, Birt- Hoog- Dubé syndrome). The unified, uni- formly sized image panels highlight how temporal evolution a single patient can be represented as structured longitudinal multimodal item in our dataset.
31

---

Published as a conference paper at ICLR 2026
CT, lung window, Marked improvement with near- normal parenchyma.
CT, lung window. Few residual discrete cysts.
CT, coronal view. Almost complete radiologic recov- ery.
Figure 15: Late follow-up CT eight months after discharge, demonstrating almost complete recovery with limited sequelae.
E.3 GPT-5 CORRECT CASE STUDY: HIBERNOMA OF THE CHEST WALL
A 28- 28-year-old woman underwent conventional chest examination for suspected pneumonia, which incidentally revealed a right paraspinal chest-wall mass. Radiography showed a paraspinal opacity with scoliosis and rib deformities. CT demonstrated a solid, non-mineralised paravertebral lesion with fatty components slightly denser than subcutaneous fat and prominent internal serpiginous vessels. MRI confirmed predominantly fatty signal intensity that slightly less bright than subcutaneous fat, with incomplete fat suppression, streaky soft-tissue components and slow, inhomogencous enhancement after contrast contrast-features typical of a hypervascular brown-fat tumour (hibernoma).
32

---

Published as a conference paper at ICLR 2026
Chest Right paraspinal mass with scoliosis and rib deformities.
radiograph.
CT, coronal view. Fatty mass slightly denser than subcutaneous fat.
CT, axial view. Solid le- sion with internal serpig- inous enhancing vessels.
T2 Dixon water-only
T1Gd
MRI Tl-weighted. Pre- dominantly fatty high signal, not as bright as subcutancous fat.
MRI T2 with fat sup- pression. Incomplete fat suppression with hetero- geneous appearance.
MRI T1 post-contrast. Slow, inhomogeneous
enhancement of soft- tissue components.
MRI T1-weighted. Fatty tissue with streaky muscle-like strands.
MRI T1 post-contrast. Prominent internal ves- with flow voids and enhancement.
Imaging from paraspinal chest-wall mass later diagnosed as hibernoma: chest radiograph, contrast-enhanced CT. and multiparametric MRI (T1, T2 with fat suppression, and post-contrast). All images are stored in the dataset as individual files associated with this case.
Figure 16:
GPT-5 answer. For the five-option question (Atypical lipomatous tumour: Clear cell sarcoma of the soft tissues; Hibernoma of the chest wall; Liposarcoma; Resolving haematoma), GPT-5 selected Hibernoma of the chest wall and produced the following explanation:
Young woman with a paraspinal chest-wall mass showing predominantly fatty density with prominent internal serpiginous vessels on CT and marked, heterogencous enhancement on MRI. The lesion is slightly less T1-hyperintense than subcutaneous fat and demonstrates avid enhancement with flow voids, features characteristic of hypervascular brown-fat tumor (hibernoma). Atypical lipomatous tumor well-differentiated liposarcoma and other liposarcomas typically occur in older patients, are less vascular and show thick septa or nodular non non-fatty components rather than prominent intralesional vessels. Clear cell sarcoma lacks macroscopic fat and classically arises in distal extremities. A resolving hematoma would not show this degree internal vascularity or persistent avid enhancement. Thus, hihernoma the chest wall is most likely.
Why this reasoning is correct. GPT-5's explanation closely matches the expert interpretation of this case. It correctly recognises fatty but not quite normal fat signal pattern (slightly denser than
33

---

Published as a conference paper at ICLR 2026
subeutaneous fat CT, slightly less T1-bright MRI, incomplete fat suppression) and emphasises the key hallmark of hibernoma: a hypervascular intramuscular lesion with prominent serpiginous internal vessels and slow, inhomogencous enhancement rather than homogeneous pure- mass. then uses these imaging cues, plus the patient's 's young age and paraspinal chest-wall location, to rule the main differentials: atypical lipomatous tumour and other liposarcomas (typically less vascular, in older patients, with thick septa and nodular non-fatty components), clear cell sarcoma (no macroscopic fat, usually distal extremities), and resolving haematoma (lacking persistent vascular flow voids and avid enhancement). This stepwise, modality- reasoning is consistent with the teaching point for hibernoma and leads the correct diagnosis for this MedTTink VQA item.
34

---

Published as a conference paper at ICLR 2026
E.4 GPT-5 ERROR CASE STUDY
E.4.1 GPT-5 ERROR CASE STUDY: COMBINED WILKIE, NUTCRACKER, AND MAY-THURNER SYNDROMES
26-year-old woman with a three-year history of weight loss and postprandial abdominal discom- fort, prior anorexia nervosa, and known pelvic congestion syndrome underwent contrast-enhanced abdominal CT. Imaging demonstrated: (i) compression of the third portion of the duodenum between the superior mesenteric artery (SMA) and aorta with reduced aortomesenteric angle and distance (Wilkie SMA syndrome), (ii) compression of the left renal vein between the aorta and SMA with the classic "beak sign." " proximal left renal vein dilatation and engorged left ovarian and pelvic veins (Nuteracker syndrome), and (iii) compression of the left common iliac vein by the right common iliac artery against the lumbar spine (May-Thurnel syndrome), together with gastrie and proximal duodenal dilatation.
FARE
Sagittal CT. Third portion of the duodenum compressed between
Aortomesenteric angle. Markedly reduced SMA- aorta angle.
Aortomesenteric
distance. Narrowed
aortomesenteric distance with duodenal compres- sion.
aorta and SMA.
LRV beak sign. Abrupt marrowing the left re- mal vein between aorta
LRV dilatation. Proxi- enlargement of the renal vein.
Beak geometry. An- gulated, tapered appear- ance of the compressed left renal vein.
and SMA.
Pelvic veins. Tortu- left ovarian vein dilated pelvic ve- nous plexus.
May-Thurner. Com- pression of the left common iliac vein by the right common iliac artery.
Global effect. Dilata- tion of stomach and proximal duodenum from outloow obstruc- tion.
Figure 17: Nine CT slices from single case demonstrating three concurrent vascular compression syndromes: Wilkie (SMA) syndrome, Nutcracker syndrome with pelvic venous congestion, and May-Thurner syndrome. All images are stored as separate files but belong to one MedThinkVO.. VQA item.
35

---

Published as a conference paper at ICLR 2026
GPT-5 answer (incorrect). For the five-option question (iliac vein compression by occult pelvic mass; hereditary megaduodenum; scleroderma with duodenal involvement; superior mesenteric artery syndrome; Nutcracker syndrome and May-Thurner syndrome), GPT-5 selected superior mesenteric artery syndrome and produced the following explanation:
CT demonstrates markedly reduced aortomexenteric angle (R: 18²) and aortomesenteric distance (~ 2 mm), which are diagnostic measurements for compression of the third portion of the duodenum by the SMA. that is, superior mesenteric artery (SMA) syndrome. The clinical context of a young plus postprandial abdominal discomfort, is classic for SMA syndrome mesenteric fat pad. While other vascular compression findings may be present, the imaging and symptoms most directly support SMA syndrome as the primary diagnosis.
with significant weight loss and anorexia nervosa,
loss the
Why GPT-5 wrong: failure multi-image reasoning. Although this explanation is internally coherent for an isolated Wilkie / SMA syndrome, it systematically ignores the additional information carried by the other six images. Three CT slices do quantify the reduced aortomesenteric angle and distance and show duodenal compression, but the remaining views encode two further, independent haemodynamic abnormalities:
Several slices through the renal hilum show an abrupt, "beak"-shappe" narrowing of the left renal vein between the aorta and SMA with proximal dilatation and engorged ovarian and pelvic veins, which is classic for Nutcracker syndrome with pelvic congestion.
A slice at the iliac bifurcation demonstrates focal compression of the left common iliac vein by the right common iliac artery against the spine, characteristic of May-Thurner syndrome.
The ground-truth label for this MedThinkVQA item reflects the combination of vascular compression syndromes documented in the full nine-image not just SMA syndrome. However, GPT-5 effectively behaves as it were solving a single-image task: focuses on the most salient slice showing
compression, maps this to the textbook pattern of superior mesenteric artery syndrome, and then dismisses the rest with the remark that other compression findings "may be present."
This behaviour illustrates a central limitation of current multimodal LLMs on genuinely multi- image cases. Instead of integrating heterogeneous evidence across different anatomical levels and mechanisms (duodenal obstruction, renal venous outflow obstruction, iliac venous compression), the model implicitly treats the images as redundant views of one problem and optimises for single, locally consistent story. In other words, GPT-5 correctly explains one mechanism, but it fails at joint, cross-region reasoning over curated set of complementary images, which precisely what this multi-syndrome case is designed to test.
36

---

Published as a conference paper at ICLR 2026
E.4.2 GPT-5 ERROR CASE STUDY: LONGITUDINAL CYSTIC PULMONARY TUBERCULOSIS
This MedThinkVQA item is longitudinal case of previously healthy nine-year-old boy with cystic pulmonary tuberculosis. The clinical course spans almost eleven months: initial presentation with fever, cough, dyspnoea and diffuse micronodular infiltrates; multiple bilateral spontaneous pneumothoraces requiring repeated chest drainage and intensive care; evolution on CT from diffuse micronodules and ground-glass opacities to extensive thin-walled cystic change with recurrent pneumothoraces; and, after prolonged anti-tuberculous therapy, near-complete radiologic recovery with only a few residual cysts and fibrotic sequelae. All time points are encoded as separate images but linked to single question whose ground-truth diagnosis is cystic pulmonary tuberculosis.
GPT-5 answer (incorrect). For the five-option question (Pneumocystis jirovecii pneumonia: Lym- phoid interstitial pneumonia; Langerhans cell histiocytosis; Cystic pulmonary tuberculosis; Birt- Hogg-Dubé syndrome), GPT-5 chose Pneumocystis jirovecii pneumonia and produced the following explanation:
The imaging shows initial diffuse interstitial/ground -lass process (crazy siizy--pvvingaoo appear- ance) followed by the development numerous thin-walled tysts/pneumatoceles throughout both lungs and recurrent bilateral spontaneous pneumothoraces. This evolution diffuse GGO progressing to widespread pneumatoceles with high risk of pneumothorax - is classic for Pneumocystis jirovecii pneumonia, especially in children with underlying occult immunodeficiency. Langerhans cell histiocytosis typically affects young adult smokers with upper-lobe probe---ddoiiantt irregular cysts and costophrenic angle sparing: lymphoid interstitial pneumonia is a chronic disease with lower-lobe cysts and not an acute febrile presentation; tuberculosis produces cavities rather than numerous thin-walled cysts; Birt- Hogg -ubé syndrome presents in adults with basilar subpleural cysts without acute diffuse ground-glass opacitics. Thus, Pneumocystis jirovecii pneumonia best fits the overall picture.
Error analysis: ignoring the longitudinal time axis. GPT-5's reasoning is internally consistent for an acute, immunodeficienne--elaateee Pneumocystis pneumonia, but it fails as longitudinal interpretation of this case. Several key aspects of the time series are either misread or ignored:
Disease duration and follow-up. The patient is immunocompetent nine-year-old boy followed over almost eleven months, with documented near- near-complete radiologic recovery after prolonged tuttruuubeuuulus therapy. This long clinical evolution with structured follow- up CT's is far more typical of tuberculosis than of uncontrolled Pneumocystis infection, which in undiagnosed immunodeficient child would be expected to progress or relapse rather than steadily resolve.
Full temporal chain, not a single snapshot. GPT-5 effectively compresses the sequence "diffuse micronodules/ground glaass - extensive thin- walled cysts - gradual resolution" into standard short-course template for Pneumocystis pneumonia. It focuses on the middle phase (GGO with cysts and pneumothoraces) and treats the early and late time points as redundant, rather than evidence of a slowly evolving, ultimately reversible granulomatous infection under long-term treatment.
Misconception about tuberculosis morphology. The explanation assumes that tuberculosis "produces cavities rather than numerous thin-walled cysts," implicitly excluding cystic or pneumothorax-pin pprone forms of TB. However, the discussion explicitly describes cystic pulmonary tuberculosis as a rare but recognised entity in which diffuse nodules and ground- glass change can evolve into confluent dalled cysts with recurrent pneumothoraces. followed by gradual radiologic improvement once therapy is effective.
Underuse of sequential improvement as a diagnostic cue. A core signal in this longitudi- nal case is the progressive improvement across follow-up CT's: cysts shrink, pneumothoraces resolve, and only sparse cystic or fibrotic sequelae remain. Such a trajectory is strongly sup- portive of treated infection (here, tuberculosis) and less compatible with chronic interstitial or cystic diseases (such as LCH, LIP, Birt-Hogg-Dube that typically show persistent or progressive structural damage.
37

---

Published as a conference paper at ICLR 2026
E.4.3 GPT-5 ERROR CASE STUDY: MULTIMODAL-HYDATIO
This MedThink item is multimodal, multi-organ case of disseminated hydatid disease in 19-year-old man presenting with seizures and low-grade fever. Cranial CT and MRI show a solitary parietal cystie lesion with mild peripheral enhancement and no diffusion restriction; chest X-ray and body CT demonstrate multiple, well-defined peripheral pulmonary nodules with water attenuation; thoracic ultrasound confirms the cystic nature of the lung lesions; and abdominal CT reveals cystic hepatic lesion with peripheral calcification. Serology for Echinococcus granulosus confirms the diagnosis, and the ground-truth label for the item disseminated hydatid disease.
GPT-5 answer (incorrect). For the five-option question (Disseminated hydatid disease; Lung: pulmonary septic emboli; Pulmonary metastases; Brain: cystic brain tumour; Brain abscess), GPT-5 selected Brain abscess and produced the following explanation:
The clinical presentation of seizures and fehricula along with laboratory findings of leuko- cytosis and neutrophilia suggests an infectious or inflammatory process. The absence of respiratory symptoms makes options related to pulmonary pathology less likely as the primary diagnosis. Given the neurological symptoms. the most plausible diagnosis would involve the brain. Between options D and E, brain abscess (E) is more likely to present acutely with seizures and fever, and is often associated with leukocytosis and neutrophilia, whereas a eystic brain tumor (D) might not typically present with such acute infectious symptoms. The imaging findings, although not described, would likely show a localized area of infection in the brain for an abscess, which aligns with the clinical picture provided.
Error analysis: failure to integrate multimodal, multi-organ evidence. GPT-5's reasoning treats this case as if it were a purely neuroradiologic vignette: it focuses on seizures, low-grade fever and inflammatory markers, implicitly assumes the key information is "ring-enter tring-enhancing brain lesion plus infection," and then chooses brain abscess by comparing only options D and E. This single-modaliy shortcut neglects almost all of the structured multimodal evidence presented:
Pulmonary imaging is downgraded to "background". Chest X-ray and chest CT clearly show numerous, well-defined, peripheral pulmonary nodules with water attenuation and no features of suppurative consolidation or infarction. Thoracic ultrasound further confirms that these nodules are true cysts (anechoic with posterior acoustic enhancement), pattern much more typical of hydatid disease than septic emboli or metastases. GPT-5's statement that pulmonary options are "less likely" because of absent respiratory symptoms ignores that hydatid cysts are often asymptomatic in the lungs and that imaging, not symptoms, carries the main diagnostic weight here.
Hepatic cyst is completely ignored. Body CT demonstrates a classic hydatid cyst in the left hepatic lobe with a well-defined cystic lesion and peripheral calcification of the pericyst. This second non-brain, non-lung organ involvement is a strong clue for systemic parasitic disease. GPT-5'ss explanation does not mention the liver at all, indicating that this modality and organ channel are effectively dropped from its reasoning.
Brain MRI is interpreted through generic "ring-enhancement abscess" template. The brain lesion in this case is a solitary, CSF-like cyst with a thin rim, mild peripheral en- hancement, no diffusion restriction, and only moderate oedema. These features, particularly the absence of diffusion restriction and the characteristic low-signal rim on T2-weighted images, are more consistent with a hydatid cyst than with pyogenic abscess. GPT-5 instead imagines a typical abscess pattern and even states that the imaging findings "would likely" show a focal infection, revealing that it is reasoning from a mental template rather than actually integrating the provided MRI sequences.
Cross-organ pattern is never assembled. The correct diagnosis requires noticing a triad: (i) multiple cystic pulmonary lesions, (ii) a calcified hepatic cyst, and (iii) a solitary brain eyst with hydatid-lik, like MRI characteristics. Taken together, these represent a classic multi-organ, haematogenously disseminated parasitic infection. GPT-5 never composes this cross-organ, cross-modality picture; instead, it chooses the most salient single modality (brain MRI/CT) and maps the entire case to a focal intracranial infection.
38

---

Published as a conference paper at ICLR 2026
E.5 STEE-LEVEL EVALUATION CASE STUDY: BILATERAL SUBAREOLAR ABSCESSES
E.5.1 ERROR TAXONOMY FOR MODEL RESPONSES
Reasoning Error (Reasoning Err). The imaging and clinical facts themselves are correctly stated, but the model misconstructs the causal chain or diagnostic logic, reaches an incorrect conclusion, selects an inappropriate differential diagnosis, uses correct facts to support an incorrect judgment.
Image Understanding Error (Image Understanding Err). The model misreads or hallu- cinates objective visual information that is directly apparent the images (e.g., location, morphology. signal intensity, enhancement pattern, extent), and the error originates from image interpretation rather than downstream reasoning.
The model misunderstands, misquotes, or inaccurately restates clinical information provided in the stem (e.g.. symptoms, age, duration, physical findings, laboratory data), treats information that is not given if it were explicitly provided.
Clinical Scenario Error (Clinical Scenario Err).
Medical Knowledge Error (Medical Knowledge Err). The model violates general medical knowledge textbook-level facts-such as typical manifestations of a lesion, epidemiol- ogy, key features for differential diagnosis, or common complications-inn ways that are independent of how the specific images or clinical history are read.
Clinical scenario and imaging summary. A 39-year-old woman presented with unilateral, single- orifice nipple discharge (galactorrhoea) from the right breast and no fever, palpable mass, or skin inflammatory changes. Initial mammography (craniocaudal and oblique views) and ultrasound of both breasts were reported as normal. Galactography of the right breast showed an intraductal filling defect in canalised nipple duct and raised the suspicion of an intraductal papilloma. Subsequent contrast-enhanced breast MRI revealed bilateral subareolar fluid-filled cavities without internal enhancement, with intensely enhancing peripheral capsules and surrounding inflammatory tissue. A repeat ultrasound, performed after the patient clarified that the discharge was actually bilateral, showed bilateral subareolar hypoechoic nodules with anechoic centres, consistent with abscesses. Cytology confirmed bilateral subareolar abscesses.
GPT-5 diagnosis and explanation. The multiple-choice options included galactocele, fat necrosis, intracystic papillomas, and subareolar abscesses (correct answer: subareolar abscesses). GPT-5 instead chose intracystic papillomas and produced the following free-text explanation:
Unilateral, single-duct nipple discharge suggests focal intraductal process, Mammography is essentially negative. Ultrasound shows a dilated retroarcolar duct containing an intracystic solid nodule. MRI demonstrates a small enhancing subareolar intraductal lesion without sur- rounding inflammatory change. Findings are classic for intraductal/itrrcystie papilloma, not galactocele (no fat fluid level). abscess (no pain, erythema or inflammatory changes). or fat necrosis.
Step-level parsing and expert error annotation. We use GPT-5 to decompose its own explanation into discrete reasoning steps, then ask a breast imaging expert to judge the factual correctness of each step and assign an error type when applicable. Error types follow our taxonomy: Clinical Scenario Err, Image Understanding Err, Reasoning Err, and Medical Knowledge Err as defined in Section ??
1. Step 1: "Unilateral, single-duct nipple discharge suggests a focal intraductal process." Expert factual judgment: Incorrect. Error type: Clinical Scenario Err. Although the original history was recorded as unilateral, single-orifice discharge, the case narrative later clarifies that the galactorrhoea actually bilateral. GPT-5 treats the initial, incomplete history definitive and over-anchor on classic teaching pattern for intraductal papilloma, misrepresenting the true clinical scenario.
2. Step 2: "Mammography is essentially negative." Expert factual judgment: Correct. Error type: None. The report states that mamnography did not show any pathological findings, so this step accurately reflects the imaging description.
39

---

Published as a conference paper at ICLR 2026
3. Step 3: "Ultrasound shows dilated retroareolar duct containing intracystic solid nodule." factual judgment: Incorrect. Error rype: Image Understanding Err.
In reality, the initial ultrasound was normal, and the follow-up ultrasound showed bilateral subareolar hypoechoic nodules with anechoic centres (abscesses), not a duct with an intra- cystic solid nodule. GPT-5 hallucinates very specific retroareolar papilloma pattern directly from the images, so the primary error is misreading the imaging rather than subsequent reasoning.
4. Step 4: "MRI demonstrates small enhancing subareolar intraductal lesion without rounding inflammatory change."
Expert factual judgment: Incorrect. Error type: Image Understanding Err.
The MRI actually showed bilateral fluid-filled cavities without internal enhancement, sur- rounded by intensely enhancing capsule and inflammatory tissue with moderate enhance- ment. GPT-5 inverts this pattern (solid enhancing nodule, no inflammation), again indicating failure correctly translate the described MRI findings into its intermediate reasoning.
5. Step 5: "These imaging findings are elassic for intraduutaa/intracystii papilloma." Expert factual judgment: Incorrect. Error type: Reasoning Err.
Given the true imaging pattern (bilateral subareolar fluid collections with enhancing capsule and surrounding inflammation), the correct diagnosis is abscess. Here GPT-5 generalises from its misread ultrasound/MRR steps and commits diagnostic reasoning error: it asserts a papilloma pattern even though the combination of bilateral disease, fluid-filled cavities, and inflammatory enhancement better matches abscess.
6. Step 6: "The findings are not compatible with galactocele because there is no fat-fluid level."
Expert factual judgment: Correct as stated. Error rype: None.
The expert accepts that there is no explicit fat-fluid level described. While the accompanying textual explanation overstates how definitively this excludes a galactocele, the literal step ("no fat-fluid level is seen") is factually consistent with the report, so no image scenario error is assigned at the step text level.
7. Step 7: "The findings are not compatible with an abscess because there no pain, erythema, or inflammatory changes."
Expert factual judgment: Incorrect. Error types: Reasoning Err, Clinical Scenario Err, Medical Knowledge Err.
This step bundles three distinct problems:
Clinical Scenario Err: It equates the absence of external pain, erythema, skin signs with the absence of inflammation, ignoring the MRI description of surrounding inflammatory tissue and the later clarification of bilateral symptoms.
Medical Knowledge Err: It assumes that breast abscesses cannot present without overt systemic cutaneous signs, which contradicts the broader discussion that non- puerperal central abscesses can be subtle.
Reasoning Err: It uses this flawed premise to exclude the true diagnosis (abscess), despite direct imaging and cytology evidence to the contrary.
8. Step 8: "The findings are not compatible with fat necrosis."
factual judgment: Correct. Error type: None.
The described lesions are fluid-filled cavities with enhancing capsules and inflammatory tissue, without oil cysts or internal fat signal; this pattern is more characteristic of abscess than fat necrosis, so excluding fat necrosis this step is reasonable.
40

---

Published as a conference paper at ICLR 2026
F OPTION & DISCUSSION AUGMENTATION PROMPT
To ensure reproducibility, we document the exact prompts used for augmenting Options and expanding the Discussion in the medical multiple-choice QA setting.
F.1 SYSTEM PROMPT
medical assistant.
Prompt Option Gemeration
Task
Given medicel multiple-choics question of the form "Selact the single best diagnosia" pssed on CUINTCAT_HISTORE several patient 'mages, the current provided optiona, answet,
existing discussion (Lecluding reasoning
options), please:
Generate additional
options is exactly 5 (n more, no leas).
answer
Expand and refine the provided discuesion, ensuring it Lhoroughly explaine how answer appropri ate,
eliminate all incorrect using reasoning grounded In the CITNICAL_HSSTON and images.
and
** Suggested Approaches
Consider Erroneous Ferspectives ivea: distractors misinterpres or images.
overemphaaize aspects of the Create distractors based on common diagnostic Leverage Common Misconceptions
frequently coned conditions.
3. Logical Xisdirecting lon: Introduce distzactors grounded in logical reasening that appear plausible
ultimately incorrect.
General Requi rements
1. Maintain ConsLatencys Ensure options match the original ones in length, structuze, and professionsl wording. Avold Oversimplitied Distractors.
3. Ensure High Plausibility.
Expand Jiscussion:
Includa reasoning for the newly generated distracters. strregteen explanations for ruling out incorrect answers. Deepen justification for selecting the correct
Final Cutput Format:
Return valid J80N with exactly these fields: options (A-E), correct_ansves discussion.
#4 Important Cutpot Rules
Keep all voriginals options text unchanged; only add new distractors
zeach exectly five total optiens.
Uo NUT reorder existing options; append only the missing letters
(e.g., add D/EI that are fillad.
The final correct_answer must correspond to the original correct option's text.
extra commentery outside the JSDN body.
41

---

Published as a conference paper at ICLR 2026
G DISCUSSION PRUNING PROMPT
This section documents the prompts used to prune Discussion paragraphs by removing references to extra differential diagnoses that are among the allowed answer options.
G.1 SYSTEM PROMPT
clinical editor. job MINIMALGY edit medical DISCUSSION.
Goal: remove refereaces to extra differential diagneses that appear in ALLOWED OPTIONS.
DLAGNDLSG LIST Preserve all content related to ALLOKED OРTTONS.
sUT among
Keep the original clinical reasoning Ilow, tone, and meaning, not add new racta,
Rules:
NEVER
any ALLONED-OPTIONS OPTIONS (even if an EXTRA item partially overlags). sentences/elane
Lnformation
whose main role introduse, justify, stst Itens in EXTRA_TO, RO_RR,
If sentence mixes allowed and extra diagnosee, keep the allowed part and delete only y
part, then fix gramnar to remain fluent. 31 Keep general disease detinitions, inaging/laa reasoning, and conclusions that suppert MLLONED_CFIIONS.
4) Maintain coherence and clinicai correctness; do Nor invent new clains. Output strictly
one key: discusaton_nec,
IE EXTRA_TO_RRMOVE is empty, return the original discussion ciacussion_new.
G.2 USER PROMPT TEMPLATE
Edit the DISCUSSTON by de delating only the parts about the extra iffferentiala.
ATTOWED__Om _PTTTONS (keep anything related these):
KALLONED SEWEEED__PTTONNEE CP__CINNS_NNO0
>4F_DLAGNOS_E StL.___SS______ CLEAN:
DOTF_OTAGNOSTS_LOO _CL_AN___OON>>>>
EXTRA_TO_RENCE REOOVE Idalete content only shout thesel
<EXTRA_TO_ TXXTA__T__TRWOT__1505S
DESCUSSION:
<nTSCUSSTOV>
Return JSON: (*dlacussion_new": "..."!
42

---

Published as a conference paper at ICLR 2026
H PROMPTS FOR DATA LEAKAGE AUDITING
SYSTEM MESSAGE
You ace a met iculous clinical QA auditor for multiple cheice diagnosis questioas. Your Jobs Given ONLY the cLiNICaL HISTORY text and the list of candidate diagnosis OPTIONS, decide whether the history text DIAECTLY REVEALS any cption(s).
Decinition of DIRECT REVEAL lalagnesis labe- appears in the text itself, not inferred:: diagnosie "piagnosis: x", "Eiopsy-proven x".
Explicit label: the
or ICD/standard
appears, or patterns
lenguage variant of the diagnosis label appears te.g.r "Mi" for myocardial inferction; "Osler-xeber-
Explicit
for Hill
Li Explicit but uncertain mencion of the dizgnasia label (or its symonym/accrnymm/pppyyy e.g., "2x", *r/o "rule x", "query "suspected "consistent with x", "concern for x", "Ixx of/known case of x*.
"possib e/probable x*,
symptoms, signa, rick factors, imaging descriptors, or lab pasterns that merely sueGest a diagnosis, Only mark a leak if the diagnosis LABEL itself (or i:s standard synony//scromm//pponne
NOT leak:
the
OPTΣONS solely diotionary candidate labels their widely-used synonynz/ / wwwrnnnseepoonneeeeee.ccooeeeeee to search for DIRECT textual mentions. Do NOT infer diagnosee based on reasening.
from context.
For each leaked option, return: cption_id, option_text.
overall leak_lavel lavel (max severity across its ev' dences; 13>12>111
verbatim enippet (atact,ead) character indices iste EXACT Clinical history string brief justification
If no option is lecked, set has_leak=false and provide nen_leak_reasen.
Return ONLY valid 28o8 following the sequired achema. No extra prose.
USER MESSAGE (TEMPLATE)
CLINICAL (use exact string when computing spans) cOSCHISTORY2939
(CI.NNCCA__
CC<<END_E18TORY12
OPTIONS (candidate diagneses; NoT infer--use only label dictionary): A) Cop_ion_A_text
(option_H_text
[option___text]
costion___tex [eption_E_text]
(continue as needed, preserving order)
Taak: Tdentify ALL options (If any) that are direct revealed by the ATSTORY text under 21/12/L3 desinitions. Extract verbatin evidence snippet (st 0-based [start,
char spans into the exact RISTORY string above. "f none, set haa_leakesfalse.
I PROMPTS FOR DISCUSSION GENERATION
SYSTEM PROMPTS
elinical history, imaging mare images (with captionsl, write diseussion Background; Clintes) Perspect Tmagi Perspect Gutcome; Take Home Message, Be accurate, concise, and grounded in the provided info.
are cestified rediologist.
findings, differentiel diagnosis list, the final diagnesis, and cne
sections:
Return strict JSON with keys exactly:
"Background"; "..."
"Clinical Forspective"; "..."
"Imaging Perapective"; "...", "Outcome": "...",
"rake kome Message": "..."
example of tone/strusture (content is just an example; Do NOT capy text):
43

---

Published as a conference paper at ICLR 2026
"Background": "May and Tharner described for the tirat time 1956 like formation the left cormon iliet vein in 224 of autopsies. May-thurner syndrcme, also known as ilias Venous Compression Byndrome (IVCS), condition usually ly the left common 1lfa wein ay the right common 1lfac artery.",
compression the overlying artery,
iclini Perspective" ive*: "This df is reported to be more frequent in women and the main clinical presentation is deep vein thrombosia. aucopsies
The true prevalence of this condition is unknows, buut series reported 224 to 334. May-thuzner syndrome is a progressive vascular disease with long-term diaabling comp lcations.",
compreasion, thronboals, should be treated if symptomatic. The procedure includes an ascending venogzam through the iliae vein to show the stenotic azea. guidewire advanced threugh the lesion and ever-the-wire.',
"Imaging Perspective":
is than placed
"Outcome"; *Since 1995 venous stents have been placed into the nacrowed area. Stents seen
beneficial, improving elinical cuzcome and the quality of life of these patlents.",
"Take Message": *if pasient discomfort, swelling deep venous thrombosis (NVT), in the iliofemoral vein territary, aspecially side
May-Thurner syndrome."
J LLM JUDGE PROMPT
J.1 SYSTEM PROMPT
are aveluator radiclogy casa analyses. Judge correctness step based on the provided context (Clinical history, Captions, Imaging findings, Discusator) and relevent Leaching velue/demain knowledge. Rulea:
11 Evaluate whether each step is correct oz reasonably supported; reasonable analysis counts as correct.
step explicitly supported, cerrectly implied, or logically reasonable given context
and your teaching value/damsin knowLedge.
False only the step clearly wrong, contradictory, or cannot reasonan's y inferred the context or standard domain knowledge.
4) Ignore style, zedundsncy, er zeasoning quality- -ocius only on cesrectness.
Provide exectly cancise 1-2 sentence explanstion per stap.
6) Return ONLY JSON follewing the provided schema; one verdict per step, same ozder.
J.2 USER PROMPT (TEMPLATE)
Taak: For each step be low, judge if it is supported by tha provided context and rel relevart
Leaching value/domain knewledge,
Title: [[title)]
Clinical history: [[clinics] ([cliniiss_hhsttrry)
Imaging findings: ([imagiag_Tindinnss
Discussion: {(discussion))
Captiona (all): [icaption__lock)] e.gg, lines [lcagtion_i)1 none, (none)"
Stepa to judge (in order):
(:atepa_block]] e.g., "1. [1step_1) [1step_2]]\r...>
Output strictly JSON; verdict in the ordes, using this schema:
"vezdicts":
foc_tact true, "explanation": *A brief, self-contained justification 11-2 sencencea).. If true, mention supporting (hrase(s) from the context when possible; false, state the contradiction or ' 'not supported by the provided context'. (2-300 chars)
... object per step, in order
44

---

Published as a conference paper at ICLR 2026
J.3 LLM AS JUDGE FOR CASE DISCUSSIONS
You are board-oert Fed radiologise tasked with eva evallaating the factual correctness of radiclogy case discussions.
Judce the correctneas of each seatence from the Discuasion section (Background Clinical Perspective Inaging Perspective Outcome Take-Home) based on the provided case context (C-inizal history, imaging findings, Differential list), the image captions, and imagea themselves.
Rules:
Mark
sentence is expl ty aupported, correctly implied, or logically ressonable given the context and standard domain knowledge, False only clearly wrong, contradictory, or reasonably inferable.
Tgnore style rodundanay--"ccua only on correctness. Provide exactly concise 1-2 sentence explacation sentence. Return ONLY J90N for the schems below.
Retura STRICT JS08 with this schema:
"sontence_ jutienett__udgeeetst:
@sertence_keys**"
"Lext"; #coriginal sentence>",
"Eactual": true "explanstion"; *<ONE concise 1-2 explanations*
J.4 RUBRIC EVALUATION PROMPT
You are a beard-certifion radiologist tasked with evalusting the quality radiology discussions.
Evaluate Discussion using standardized rubric.
provided radiology case
ANTERIALS PROVIDED:
Clinics) history and imaging findings
Differential diagnosis 1ist
Medical inages with captions
viscussion
(containing: Background, clinical perapective,
Imaging perspective, Outcome, Take Hooe messages)
EVALUATION INSTRUCTIONS:
1. Read the entire Diacussion section carefully
criteria 3. Foz each rubric score, provide a brief 1-2 sentence justification calculate
scale.
score (sum
rubrics, range 0-101
#UCUS Uk:
Medical accuracy and dendeene-bdded
Completeneas of information
Educational
radiology
clear communication of key concepts Integration clinical imaging perspectivas
OUTPUT FORMAT:
val id object following apect ried achema. Do not include any additional text or explanations outalde the structure.
Return STRICT JSON
schema:
"rubric_ scores"; *rubrio_soores":
"subrice @aa__elllaaaaaaaaaaialllaaaaaaea
"ubbicc_ [ebric_1_diseseee se____ddesee_vvee diseesseoverrvvee ("score"; 021 "explanation"; *<1-2 sentences>"),
("score": 01112, "explanation"; *sl sentences>"), sentences>"),
"subric_3_ine iimagiing_mmm |"score"; 011/2, "explanatien".
at differentiall" (*score": 0 12, "explanation": *c1-2 sentences>"), ("score*: 12, *explanstion"; +e1-2 sentences>"),
"Lotal": 10
45

---

Published as a conference paper at ICLR 2026
K ADDITIONAL EVALUATION TABLES FOR TEXT-SOLVABLE CASES
All results below are evaluated on the same raw test set of 2,859 items. The four-model joint-corrett intersection contains 1,074 items.
Table 10: Per-model accuracy and four-model joint-correct intersection on the 2,859-item raw test set.
Model
Correct
Total
Accuracy
Ovven--Ner--00--AA---nnnse upt-oss-1206
1,657
2,859
0.580 (57.96%)
1,747
2,859
0.611 (61.11%)
medgemma-27b-textti
1,644
2.859
0.575 (57.50%)
1amma-333-70B--nttrte
1,613
2,859
0.564 (56.42%)
Four-model joint-correct intersection
1,074
2,859
0.376 (37.57%)
46

---

Published as a conference paper at ICLR 2026
L MODALITY STATS IN DATASET
Our dataset provides explicit modality annotations for every image. In the training split, each case contains up 49 image slots, and each slot stores coarse modality label (image {k] m_modmoaiitt lyyy and a fine-grained subtype (image _t___uuu_mddditttta img list where each image contains modality and sub modality. We use these provided labels to summarize modality distributions at both the image and case levels.
In the test split, each case stores an
L.1 COARSE MODALITY CATEGORIES (LEVEL-1)
We use the Level-1 field modality as the coarse modality category. Both splits cover the same nine Level-1 modalities: CT, MRI, X-ray, Ultrasound, Non-modality Workflow / Post-processing, Pathology, Climical photography, Nuclear medicine & &Molecular imaging, Endoscopy.
L.2 PER-IMAGE MODALITY DISTRIBUTION
Table [T] reports the per-image distribution of Level-1 modalities. The training split contains 61,414 images, while the test split contains 5,845 images.
Table 11:
Per-image distribution of coarse (Level-1) modalities in the training and test splits. Counts absolute image counts, and percentages are relative to the total number of images in each split.
Modality
Train images
Train (%)
Test images
Test (%)
22,422
36.51
1,929
33.00
MRI
19,749
32.16
2,179
37.28
X-ray
7,790
12.68
540
9.24
Ultrasound
5,092
8.29
609
10.42
Non-modality Workflow Post-processing
3,070
5.00
230
3.93
Pathology
1,495
2.43
151
2.58
Clinical photography
852
1.39
73
1.25
Nuclear medicine& & Molecular imaging
718
1.17
114
1.95
Endoscopy
226
0.37
20
0.34
Total
61,414
100.00
5,845
100.00
CT and MRI dominate both splits, accounting for 68.67% of training images and 70.28% of test images, followed by X-ray and Ultrasound.
L.3 PER-CASE MODALITY COVERAGE
Because cases can contain multiple images, Table 12 additionall reports the fraction of cases that contain at least one image from each Level-1 modality.
Table 12:
Per-case coverage of coarse (Level-1) modalities. case is counted for modality if contains at least one image annotated with that modality. Percentages are relative to the total number of cases in each split.
Modality
Train cases
Train (%)
Test cases
Test (%
CT
5,859
62.68
455
63.19
MRI
4,069
43.53
393
54.58
X-ray
3,415
36.54
248
34.44
Ultrasound
2,126
22.75
220
30.56
Non-modality/ Workflow Post- frot-ppocessiig
1,769
18.93
151
20.97
Pathology
764
8.17
78
10.83
Clinical photography
561
6.00
48
6.67
Nuclear medicine & Molecular imaging
373
3,99
50
6.94
Endoscopy
145
1.55
16
2.22
Total
9,347
100.00
720
100.00
47

---

Published as a conference paper at ICLR 2026
L.4 FINE-GRAINED SUB-MODALITY DISTRIBUTION (LEVEL-2)
Each image also has a fine-grained sub modality label capturing protocol or acquisition type (e.g., contrast-enhanced CT, diffusion MRI, catheter angiography/DSA, mammographyy hybrid PET- CT, and pathology slide types). The training split contains 47 distinct sub-modalities, while the test split contains 42; all test sub-mo modaliiaaiie are covered by the training split. Table 13 reports per-image frequencies for every Level-2 category.
Table 13: Per-image distribution of fine- aneegrained (Level-2) sub-modalities, grouped by their corre- sponding coarse (Level-1) modality.
Modality
Sub modality
Train images
Train (%)
Test images
Text (%
Contrast-enhuneed
11.626
1,011
17.81
CT
contratt CT
6.252
10.18
503
8.61
CT
CT Angiography
1,941
3.16
132
2.26
CT
HRCT / Thin CT
1,730
2,82
202
3.46
CT
Other_Cr CT
841
1.37
43
0.74
CT Perfusion
32
0,05
0.14
MRI
Conventional MRI
17.054
27.77
1,852
31.69
MRI
Dillusion
2.61
MRI
MR Angiography Venography
471
0.77
34
0.58
MRI
Other_MiC
310
0,50
27
0.46
MRI
Perfusion MRI
191
0.31
25
0.43
MRI
MR Spectrszopy
112
0.18
0.27
MRI
Functional MRI
0.01
X-ray
Plain radiograph
4,438
7.23
315
5.39
X ray
Calbeter angiography
X X-ray
Fluomscopy
1,186
1.93
40
0.68
X-ray
Mammography
512
0.33
91
1.56
X ray
Other Xray
0,00
0.00
Ultrasound
B-mode ultrasound
3,591
5.85
405
6.93
Ultrasound
Doppler ultrasound
1,282
2.09
2.72
Ultraound
Contrast-en enaannced ultrasound
117
0.19
26
0.44
Ulraaouudd
Interventional Procedure
Ultrasound
Elastography
21
0.03
5
0.09
Ultrasound
Other, US
0.00
0
0.00
Con-modality/ Workflow Post- processing
3D post- processing
1,450
2.36
81
1.39
Non-modality/ Workflow Post- Post-procsssing
Annotated figure/ diagram
801
1,30
82
1.40
Non-medality Workflow Post-poocesing
Recenstruction Image manipulation
734
Non modality Workllow Post processing
PACS/ Teleradiology screenshot
78
0.13
0.02
/on-modality/ Workflow Post noa-ppocessing
Other Workflow
Pathology
Histology (H&E)
932
1,52
91
1,56
Pathology
Patho_otyylogyy
292
0.48
20
0.34
Pathology
Immunshistochemistry
244
0.40
0.65
Pathology
Cytology
27
0.04
2
0.03
Clinical photography
Extemal clinical photo
Clinical photography
Intraoperative photo
394
0.64
35
0.60
Clinical photography
Other ClinicalPhoto
Clinical photography
Ophthalmic photo
20
0.03
0.02
Clinical photography
Dermoscopy
0.00
0
0.00
Nuclear medicine Molocular imaging
Hybrid: PET-CT
367
0.60
1.42
Nuclear medicine & Molocular imaging
Planar scintigraphy
199
0.32
10
0.17
Nuckeat medicine Molecular imuaing
PET
Nuclear medicine & Molecular imaging
Hybrid: SPECT-CT
44
0.07
0.12
Nuclear medicine Molecular imaging
Nuclear medicine Molocular imaging Endoscopy
Hybrid: PET-MR
10
0.02
0.02
Gl endoscopy
128
0.21
0.17
Endoscopy
Other Endoscopy
0.10
Endossopp
Breachoscopy
37
0.06
2
0.03
Total
61,414
100.00
5,845
100.00
L.5 PER-CASE MODALITY DIVERSITY
Beyond per-image counts, we characterize case-level multimodality using modalities modalities__connt count, defined the number of distinct Level-1 modalities present in case. Table 14 summarizes its distribution. The training split contains 9,347 cases with an average 2.04 modalities per case, while the test split contains 720 cases with average of 2.30 modalities per case.
48

---

Published as a conference paper at ICLR 2026
Table 14:
Distribution of the number of distinct coarse (Level-1) modalities per case
moodalities_coure in the training and test splits.
# Modalities
Train cases
Train (%)
Test cases
Test (%)
3,093
33.09
I65
22.92
3,656
39.11
292
40.56
1,875
20.06
166
23.06
592
6.33
74
10.28
109
1.17
22
3.06
6
16
0.17
1
0.14
6
0.06
0
0.00
Total
9,347
100.00
720
100.00
the training split, 72.20% cases contain at most two modalities. The test split more multimodal: 63.47% of cases have two modalities, and 36.53% contain three more modalities.
Analogously, we define sub modalities ieliiiees_ccutttttt as the number distinct Level-2 sub- modalities per case. The average is 2.70 for training and 3,09 for test. Table 15 provides the full distribution.
Table 15: Distribution of the number of distinct fine- tne-grained (Level-2) sub-modalities per case in the training and test splits.
Sub-modalitte
Train cases
Train (%)
Test cases
Test (%)
1,974
21.12
94
13.06
2,717
29.07
183
25.42
3
2,334
24.97
196
27.22
1,351
14.45
132
18.33
589
6.30
66
9.17
252
2.70
30
4.17
94
1.01
15
2.08
8
23
0.25
3
0,42
0.09
0.14
10
4
0.04
0
0.00
0.01
0.00
Total
9,347
100.00
720
100.00
L.6 COMMON MODALITY COMBINATIONS AT CASE LEVEL
We also examine which Level-1 modalities co-occur at the case level. Here, modality combination defined as the set distinct Level-1 modalities present case. For the test split (720 cases), the 5 most frequent modality combinations are:
CT + MRI (70 cases, 9.7%).
CT (70 cases, 9.7%).
MRI (68 cases, 9.4%),
CT + X-ray (48 cases, 6.7%).
MRI + Ultrasound (33 cases, 4.6%),
For the training split (9,347 cases), the most frequent modality combinations are:
MRI (1,188 cases, 12.7%).
CT (1,127 cases, 12.1%),
CT + X-ray (851 cases, 9.1 1)),,
CT + MRI (732 cases, 7.8%),
X-ray (513 cases, 5.5%),
49

---

Published as a conference paper at ICLR 2026
M LONGITUDINAL STUDIES IN MEDTHINK
We annotate whether each case contains longitudinal follow-up imaging (i.e., multiple imaging time points for the same patient), In both the training split and the held-out test split, we pro- vide: (i) is ae__lonnituudioo tudinal, (ii) t immeoint_couree
and (iii) interval_text, a free-text description of the follow-up interval or overall span between time points when available.
Table 16 summarizes the prevalence of longitudinal cases. The held-out test set contains 219 longitudinal cases out of 720 (30.4%). The training set contains 2,703 longitudinal cases out 9.347 (28.9%). Aggregating both splits, MedThink VQA includes 2,922 longitudinal cases out of 10,067 total cases (29.0%). indicating that nearly one third of the dataset requires reasoning over temporal disease evolution.
Table 16; Prevalence of longitudinal studies in MedThinkVQA.
Split
# Cases
# Longitudinal
Share (%)
Train 9,347
2,703
28.9
Test
720
219
30.4
Overall
10,067
2,922
29.0
Longitudinal cases are typically short sequences. Across longitudinal cases, the median LLmepoint_courte 2 and the mean is 2.34. About 24.2% of longitudinal cases contain three or more time points, with maximum of 9 time points in training (8 in test). Table 17 reports the distribution timepoint_couee among longitudinal cases.
Table 17: Distribution of the number of imaging time points (t (timepoind) soapoint__oovii nt__count among longi- tudinal cases. Each cell shows # cases and the percentage relative to the longitudinal subset of the corresponding split.
imepoint_couu
Train
Test
Overall
2
2,051 (75.9%)
163 (74.4%)
2,214 (75.8%)
464 (17.2%)
41 (18.7%)
505 (17.3%)
4
149 (5.5%)
9 (4.1%)
158 (5.4%)
5+
39 (1.4%)
6 (2.7%)
45 (1.5%)
For longitudinal cases, interval_teeet _eeet provides free-teett description of the follow-up interval(s) or overall temporal span between imaging time points. Interval descriptions are available for 74.0% of longitudinal cases (2, 2,922), while the remaining 26.0% are marked as unknown/unspeciifed. Among cases with known interval descriptions, the most common timescales are months (36.5% and years (2).1%). Table T8 summarizes coarse interval categories derived from interval_tent text.
Table 18: Coarse distribution of follow-up interval descriptions (interva 1 l_text) among longitu- dinal cases. Categories are assigned by keyword matching to temporal units and explicit calendar spans; percentages are relative the longitudinal subset of each split.
Interval category
Train
Test
Overall
Unknown / Unspecified
703 (26.0%)
56 (25.6%)
759 (6.
Hours
62 (2.3%)
12 (5.5%)
74 (2.5%)
Days
406 (15.0%)
29 (13.2%)
435 (14.9%)
Weeks
272 (10.1%)
16 (7.3%)
288 (9.9%)
Months
721 (26.7%)
69 (31.5%)
790 (27.0%)
Years
448 (16.6%)
30 (13.7%)
478 (16.4%)
Date range Span
75 (2.8%)
6 (2.7%)
81 (2.8%)
Other
16 (0.6%)
1 (0.5%)
17 (0.6%)
50

---

Published as a conference paper at ICLR 2026
We additionally estimate the maximum follow-up interval/span described in Anterval_text by parsing explicit numeric durations (c.g.. "6 months", "3 years") and calendar spans (e.g., "997 to 2000"); when multiple time references occur in the same text, take the maximum. This produces parsable maximum follow-up for 69.4% of longitudinal cases (2,029) / 2,922). The maximum parsed follow-up interval reaches 60 years in training (20 years in test). Table 19]eert the distribution of these maximum follow-up intervals.
Table 19:
Distribution of the maximum follow-up interval/span (estimated from interval_teet among longitudinal cases. Each cell shows cases and the percentage relative to the longitudinal subset of the corresponding split.
Max follow-up
Train
Test
Overall
week
287 (10.6%)
(13.7%)
317 (10.8%)
week-1 month
354 (13.1%)
20 (9.1%)
374 (12.8%)
1-6 months
538 (19.9%)
(25.6%)
594 (20.3%)
6-12 months
252 (9.3%)
22 (10.0%)
274 (9.4%)
1-3 years
245 (9.1%)
(6.4%)
259 (8.9%)
3-5 years
84 (3.1%)
8 (3.7%)
92 (3.1%)
> 5 years
113 (4.2%)
(2.7%)
119 (4.1%)
Unparsed / Unknown
830 (30.7%)
63 (28.8%)
893 (30.6%)
51

---

Published as a conference paper at ICLR 2026
Pairwise comparison
Cohen's x
Expert vs. Expert 0.822833
Expert LLM judge
0.838357
Expert 2 vs. LLM judge
0.701566
Table 20: Inter-rater reliability on step factuality (Cohen's s). High agreement with Expert and substantial agreement with Expert 2 support the reliability of the L.M judge.
N PROMPTS FOR STEPWISE EXPLANATION EXTRACTION
N.1 SYSTEM PROMPT
meliculous elinicel reasening editor. Convert given explanation paragraph into ordered List of numbered steps that preserves the originel meaning and evidence.
11 Preserve content: do NOT introduce facts not present in the explanation.
Decompose into stonic inferences oz observations each step one concise sentence (<- +30 words).
3) Order staps to reflect the reasening flow le.g., findings -> interpretation +> decisionl Rewrite references like "option A/3/C' into piain statements; avoid eprion
5) if the explanaticn contrasts entities (e.g.r 'x not Y"), separate them into distinct steps. (typically English). It the explanation is very short, return aingle clear step. Return ONLY the J8ON that matches the provided schema.
the same Language
explanation
N.2 USER PROMPT (TEMPLATE)
Convert following explanation into ordered 1ist steps.
Context (for referent clarity only do NOT add facts not present in the explanation):
Title: (title)
climical history: (clinicsl_histerrs
Imaging findings: [
Explanation <<<
truth):
(explamation)
>>>
strictly JSON foll lowiing schema text)
52

---

Published as a conference paper at ICLR 2026
O LLM JUDGE STATS
GPT-5 was evaluated on the entire test set, whereas the other three models were evaluated on random sample of 100 test cases due to cost and time constraints. Error-type coverage is computed over erroneous steps; since a step may bear multiple error labels, the percentages can exceed 100%.
0.1
GPT-5 (FULL TEST SET WITH 6,425 STEPS
Correctly answered (is jo_correctt=re).
Steps (with valid is is_factual); 3,903
Step factual accuracy: 3311/3903 (84.83%)
Critical steps: 1,264
Critical-step factual accuracy: 1212/1264 (95.89%)
Erroncous steps (all): 592
Error-type coverage (among erroneous steps):
Reasoning Err: 167/592 (28.21%)
Image Understanding Err: 374/592 (63.18%)
Clinical Scenario Err: 53/592 (8.95%)
Medical Knowledge Err: 91/592 (15.37%)
Other/Unspecified:: 60/592 (10. 14%)
Erroneous critical steps only: 52
Error-type coverage (among erroneous critical steps):
Reasoning Err: 14/52 (26.92%)
Image Understanding Err: 37/52 (71.15%)
Clinical Scenario Err: 6/52 (11.54%)
Medical Knowledge Err: 11/52 (21.15%)
Incorrectly answered (is so_ccorrect=False.
Steps (with valid is is_factual); 2,522
Step factual accuracy: 1605/2522 (63.64%)
Critical steps: 520
Critical-step factual accuracy: 390/520 (75.00%)
Erroneous steps (all): 917
Error-type coverage (among erroncous steps):
Reasoning Err: 416/917 (45.37%)
Image Understanding Err: 585/917 (63.79%)
Clinical Scenario Err: 138/917 (15.05%)
Medical Knowledge Err: 271/917 (29.55%)
Other/Unspeccfied:: 9/917 (0.98%)
Erroneous critical steps only: 130
Error-type coverage (among erroneous critical steps):
Reasoning Err: 57/130 (43.85%)
Image Understanding Err: 89/130 (68.46%)
Clinical Scenario Err: 16/130 (12.31%)
Medical Knowledge Err: 49/130 (37.69%)
53

---

Published as a conference paper at ICLR 2026
O.2 INTERNVL3_5-14B_
(100-SAMPLE SUBSET)
Overall.
Total number of steps (all samples): 607.
Correctly answered (is (s__corret==rre).
Steps (with valid is is_factual): 247
Step factual accuracy: 189/247 (76.52%)
Critical steps: 91
Critical-step factual accuracy: 88/91 (96.70%)
Erroneous steps (all): 58
Error-type coverage (among erroneous steps):
Reasoning Err: 23/58 (39.66%)
Image Understanding Err: 29/58 (50.00%)
Clinical Scenario Err: 9/58 (15.52%)
Medical Knowledge Err: 24/58 (41.38%)
Erroneous critical steps only: 3
Error-type coverage (among erroneous critical steps):
Reasoning Err: 0/3 (0.00%)
Image Understanding Err: 3/3 (100.00%)
Clinical Scenario Err: 0/3 (0.00%)
Medical Knowledge Err: 0/3 (0.00%)
Incorrectly answered (is correct=False).
Steps (with valid is factual): 360
Step factual accuracy: 195/360 (54.17%)
Critical steps: 61
Critical-step factual accuracy: 52/61 (85.25%)
Erroneous steps (all): 165
Error-type coverage (among erroneous steps):
Reasoning Err: 104/165 (63.03%)
Image Understanding Err: 84/165 (5091%)
Clinical Scenario Err: 33/165 (20.00%)
Medical Knowledge Err: 81/165 (49.09%)
Erroneous critical steps only:
Error-type coverage (among erroneous critical steps):
Reasoning Err: 4/9 (44.44%)
Image Understanding Err: 6/9 (66.67%)
Clinical Scenario Err: 2/9 (22.22%)
Medical Knowledge Err: 2/9 (22.22%)
54

---

Published as a conference paper at ICLR 2026
O.3 MEDGEMMA273 1001100__a00 100_EAMPLE L000-S SUBSET)
Overall.
Total number of steps (all samples): 1,074.
Correctly answered (is fi__correct=Tre).
Steps (with valid is_factual); 376
Step factual accuracy: 285/376 (75.80%)
Critical steps: 102
Critical-step factual accuracy: 97/102 (95.10%)
Erroneous steps (all): 91
Error-type coverage (among erroneous steps):
Reasoning Err: 22/91 (24.18%)
Image Understanding Err: 50/91 (54.95%)
Clinical Scenario Err: 15/91 (16.48%)
Medical Knowledge Err: 36/91 (39.56%)
Erroneous critical steps only: 5
Error-type coverage (among erroneous critical steps):
Reasoning Err: 3/5 (60.00%)
Image Understanding Err: 4/5 (80.00%)
Clinical Scenario Err: 0/5 (0.00%)
Medical Knowledge Err: 1/5 (20.00%)
Incorrectly answered (is correct=False).
Steps (with valid is fa__fcctal):: 698
Step factual accuracy: 383/698 (54.87%)
Critical steps: 114
Critical-step factual accuracy: 78/114 (68.42%)
Erroneous steps (all): 315
Error-type coverage (among erroneous steps):
Reasoning Err: 156/315 (49.52%)
Image Understanding Err: 221/315 (70.16%)
Clinical Scenario Err: 72/315 (22.86%)
Medical Knowledge Err: 119/315 (37.78%)
Erroneous critical steps only: 36
Error-type coverage (among erroneous critical steps):
Reasoning Err: 16/36 (44.44%)
Image Understanding Err: 22/36 (61.
Clinical Scenario Err: 9/36 (25.00%)
Medical Knowledge Err: 14/36 (38.89%)
55

---

Published as a conference paper at ICLR 2026
0.4 QWEN2.5VL-320 (100-SAMPLE SUBSET)
Overall.
Total number of steps (all samples): 781.
Correctly answered (is fi__correct=Tre).
Steps (with valid is is_factual): 337
Step factual accuracy: 274/337 (81.31%)
Critical steps: 103
Critical-step factual accuracy: 100/103 (97.09%)
Erroneous steps (all): 63
Error-type coverage (among erroneous steps):
Reasoning Err: 22/63 (34.92%)
Image Understanding Err: 36/63 (57.14%)
Clinical Scenario Err: 6/63 (9.52%)
Medical Knowledge Err: 31/63 (49.21%)
Erroneous critical steps only: 3
Error-type coverage (among erroneous critical steps):
Reasoning Err: 0/3 (0.00%)
Image Understanding Err: 3/3 (100.00%)
Clinical Scenario Err: 0/3 (0.00%)
Medical Knowledge Err: 0/3 (0.00%)
Incorrectly answered (is correct=False).
Steps (with valid is faa_acctal): 444
Step factual accuracy: 236/444 (53.15%)
Critical steps: 67
Critical-step factual accuracy: 35/67 (52.24%)
Erroneous steps (all): 208
Error-type coverage (among erroneous steps):
Reasoning Err: 130/208 (62.50%)
Image Understanding Err: 113/208 (54.33%)
Clinical Scenario Err: 52/208 (25.00%)
Medical Knowledge Err: 109/208 (52.40%)
Erroneous critical steps only:
Error-type coverage (among erroneous critical steps):
Reasoning Err: 21/32 (65.62%)
Image Understanding Err: 26/32 (81.25%)
Clinical Scenario Err: 4/32 (12.50%)
Medical Knowledge Err: 11/32 (34.38%)
56

---

Published as a conference paper at ICLR 2026
P MELD-BASED DATA CONTAMINATION ANALYSIS (FULL DETAILS)
Detector: We use MELD (Memorization Effects Levenshtein Detector) a stricter, sliding-window form. For model output y and its corresponding question æ. we compute normalized Levenshtein similarity over fixed-width windows on the longer string and take the maximum across windows. Scores are reported as percentages; higher values indicate longer, more verbatim copying. Following prior medical-OQ -A practice Tang et al. (2025), samples with similarity 2 50% are flagged as high-risk for contamination.
Protocol. We run the exact inference setup used in main experiments the MEDTHINKVQA test set and apply MELD between each generated answer and its input question. We evaluate seven models spanning both LLMs and VLMs: Qwen3-32B, Med-Gemme Gemma-27m-BBiee 27B-ir, Nod-Gemma-2275 27B-n----B------tt it, 1PT--,1-nano, GPPT-4.1mini, S.ven2.5-VL-72B-Instree
and 1aama-.3--70BInssti
Results. Appendix Figure ?? plots the full distributions. Across all models, medians lie near ~20-24% with tight interquartile ranges, and the upper tails are short. Importantly, we do not observe any case with MELD similarity 50%; the largest outliers remain below that threshold. Text-only LLMs and VLMs exhibit highly similar distributions, suggesting that the presence of images does not drive overlap behavior.
Context prior benchmarks. MedAgentsBench Tang et al. (2025) reports broader spreads and heavier right tails (with many outliers above 50%) on several widely used QA datasets (e.g.. MMLU, MedQA, MedMCQA). contrast, MEDTHINK VQA shows uniformly low overlap and high-similarity spikes, indicating substantially lower contamination risk.
Limitations. MELD surface-form detector; heavy paraphrasing or template-level memorization may evade detection. Our analysis should therefore be viewed as strong negative evidence for verbatim leakage rather than proof of absence of forms contamination.
57

---

Published as a conference paper at ICLR 2026
P.1 MELD AND OUR WINDOWED VARIANT
We first restate the original MELD procedure (Algorithm 1. and then present our implementation (Algorithm 2. which adds (i) exed-denominator Levenshtein ratio with respect /ga). (ii) length- [g2l sliding window over the model's continuation restricted to its early prefix, and (iii) length-aware bucketing and generation caps for efficient parallel decoding.
Algorithm 1: MELD (original reproduction)
Data: Generative model y; dataset of guestion-answer pairs; tokenizer T; threshold
YE [0, 1J.
Result: 7: percentage (or average strength) of completions with overlap above Y.
Initialize an empty list L
2 foreach (q, a) D do
Split q into two halves: 91 and Q2
Tokenize: t + T(4)) and f2 f (g2)
Set sampling temperature 0 and pass qi context to y
Let k < [/2] and generate a continuation x consisting of k tokens from
Compute the (paper-style) Levenshtein- based overlap ratio
int( int(round( 2.0x x 100))
M9
100
where !gl is the total number of characters in both strings and M is the number of matches.
if Y then
append e to L 10 Z + mcan(1) (L)
11 return Z
Algorithm 2: MELD (ours, concise): windowed Levenshtein with length- batching Data: Model gr dataset D; tokenizer T; threshold Y; cap multiplier e²> 1; min gen tokens m, batch size B.
Result: 7 (near-exact rate), & (mean similarity).
Build items. For each E D: form text e (r); empty, continue. Tokenize ids < Tg)): split at max(1,,, [/ds)/2]); set Fi =T (ids[: | hl), 42 = - (ids(hh :I), ks lids) - h. Collect
tuples (g), 92, k; l9al).
Bucket. Group tuples into batches of size < B with similar & (length-aware).
foreach batch b do
G max(mm m, maxieb k;); set decoding (temp 0, top-p 1, max tokens G)
Generate in parallel 2; < s(qi.s) for all
foreach item in do
L + 142,1l
region <- first CL. characters of z;
(1
Lev(region j j L). 92.1)
max region
10
1 lp Y
11 Z +-
Σ, 81i E t- Σ, Pii
12 return Z,
58

---

Published as a conference paper at ICLR 2026
Q DISEASE CATEGORY BREAK DOWN
Training set size: n. = 9347.
Test set size: 720.
ICD coverage: Train spans 22 chapters/ / 184 blocks / 776 categories; Test spans 19 chapters / 123 blocks / 308 categories. Train-test overlap: 19/19 test chapters, 120/123 test blocks, 290/308 test categories appear in the training split.
SUBCATEGORY DETAIL (WITHIN EACH ICD-10 CHAPTER)
Percentages are relative to the corresponding split; block counts are reported as Train Test. 1. Certain infectious and parasitic diseases (Train: 475; 5.1%, Test: 55; 7.6%)
1.1 A00-A09 Intestinal infectious diseases 22 3
1.2 A15-A19 Tuberculosis - 118/9
1.3 A20-A28
Certain zoonotic bacterial diseases - 17 /
1.4 A30-A49 Other bacterial diseases - 75 12
1.5 A50-A64 Infections with a predominantly sexual mode transmission 9
1.6 A65-A69 Other spirochactal diseases - 1/0
1.7 A70-A74 Other diseases caused by chlamydiae - 0
1.8 A80-A89 Viral infections of the central nervous system - 19/0
1.9 A92-A99 Arthropod- borne viral fevers and viral haemorrhagic fevers 4/0
1.10 B00-BOK B09 Viral infections characterized skin and mucous membrane lesions 8
1..11 B15-B19 Viral hepatitis - 2
1.12 B20-B24
Human immunodeficiency virus [HIV] disease 10 /
1.13 B25-B34
Other viral diseases 10/0
1.14 B35-B499 Mycoses- 57
1.15 B50-B64
Protozoal diseases 7 /2
1.16 B65-B83 Helminthiases - 113 12
1.177 B85-B89 Pediculosis, acariasis and other infestations - -20
1.118 390-BB4 Sequelae of infectious and parasitic diseases 1/0
2. Neoplasms (Train: n 2578; 27.6%, Test: = 202; 18.1%)
2.1 C00-C14 Malignant neoplasms of lip, oral cavity and pharynx 16/
2.2 C15-C26 Malignant neoplasms of digestive organs - 252/ 13
2.3 C30-C39 Malignant neoplasms of respiratory and intrathoracic organs 107 8
2.4 C40-C41 Malignant neoplasms of bone and articular cartilage 71/8
2.5 C43-C44 Melanoma and other malignant neoplasms of skin 15/1
2.6 C45-C49 Malignant neoplasms of mesothelial and soft tissue 154 / 19
2.7 C50-C50 Malignant neoplasm breast 52
2.8 C51-C58 Malignant neoplasms of female genital organs 60 /
2.9 C60-C63 Malignant neoplasms of male genital organs 41 /
2.10 C64 C66 Malignant neoplasms of urinary tract 102 /
2.11 C69-C C72 Malignant neoplasms of eye, brain and other parts of central nervous system
96 17
59

---

Published as a conference paper at ICLR 2026
2.12 C73 C75 Malignant neoplasms of thyroid and other endocrine glands 41 /5 2.16 D00-D09 In situ neoplasms 6/0
2.13 C76-C80 Malignant neoplasms of ill-defined, secondary and unspecified sites 107 /
2.14 C81-C96 Malignant neoplasms, stated presumed to be primary, of lymphoid, haematopoietic and related tissue - 210/ 13
2.15 C97-C97 Malignant neoplasms independent (primary) multiple sites -
2.117 DI0-D36 Benign neoplasms 1109 80
2.18 D37-D48 Neoplasms of uncertain or unknown behaviour 135/22
Diseases the blood and blood-forming organs and certain disorders involving the immune mechanism (Train:
177: 1.9%, Test: 2.3 3.3%)
3.1 D50-D53 Nutritional anaemias 3/0
3.2 D55-D59 Haemolytic anaemias 18/1
3.3 D60-D64 Aplastic and other anaemias - 1/0
3.4 D65-D69 Coagulation defects, purpura and other hacmorrhagic conditions 11/0
3.5 D70-D77 Other diseases of blood and glood-forming organs - 11
3.6 D80-D89 Certain disorders involving the immune mechanism 45 12
4. Endocrine, nutritional and metabolic diseases (Train: n 259; 2.8%, Test: 14; 1.9%)
4.1 E00-E07 Disorders of thyroid gland 26 0
4.2 E10-E14 Diabetes mellitus 8 /
4.3 E15-E16 Other disorders of glucose regulation and pancreatic internal secretion - 0
4.4 E20-E35 Disorders of other endocrine glands - 92 7
4.5 E50-EE64 Other nutritional deficiencies 6/0
4.6 E68 Obesity and other hyperalimentationn - 6/0
4.7 E70-E90 Metabolic disorders 118/7
Mental and behavioural disorders (Train: n 3; 0.0%, Test: n 0; 0.0%)
5.1 FO0-F09 Organic, including symptomatic, mental disorders - 1/0
5.2 F10-F19 Mental and behavioural disorders due to psychoactive substance use
5.3 F50-F59 Behavioural syndromes associated with physiological disturbances and physical factors 0
Diseases the nervous system (Train: n 445; 4.8%, Test: n 37; 5.1%) %)
6.1 G00-G09 Inflammatory diseases the central nervous system - 58 /
6.2 G10-G14 Systemic atrophies primarily affecting central nervous system
6.3 G20-G26 Extrapyramidal and movement disorders - 22
6.4 G30-G32 Other degenerative diseases of the nervous system 13
6.5 G35-G37 Demyelinating diseases of the central nervous system - 36/6
6.6 G40-G47 Episodic and paroxysmal disorders - 9
6.7 G50-G59 Nerve, nerve root and plexus disorders 52
6.8 G60-G64 Polyneuropathies and other disorders of the peripheral nervous system 6/0
60

---

Published as a conference paper at ICLR 2026
6.9 G70-G73 Diseases of myoneural junction and muscle 4/0
6.10 G80-G83 Cerebral palsy and other paralytic syndromes 2/
6.11 G90-G99 Other disorders of the nervous system - -231 / 14
Diseases the eye and adnexa (Train: π. 27; 0.3%, Test: n 0.7%)
7.1 HO0-HO6 Disorders of eyelid, lacrimal system and orbit 11 /1
7.2 H15-H22 Disorders of sclera, cornea, iris and ciliary body 0/1
7.3 H25-H28 Disorders of lens - 1/0
7.4 H30-H36 Disorders of choroid and retina 1/1
7.5 H40-H42 Glaucoma-a 1/0
7.6 H43-H45 Disorders of vitreous body and globe 2/0
7.7 H46-H48 Disorders of optic nerve and visual pathways - 8/0
7.8 H49-H52 Disorders of ocular muscles, binocular movement, accommodation and refraction 2
7.9 H55-H59 Other disorders of eye and adnexa 1/0
8. Diseases of the ear and mastoid process (Train: 25; 0.3%, Test: n 2; 0.3%)
8.1 H60-H61 Diseases of external ear 3 //
8.2 H65-H75 Diseases of middle and mastoid 8
8.3 H80-H83 Diseases of inner ear 10/
8.4 H90-H95 Other disorders ear - 4/0
Diseases the circulatory system (Train: 935; 10.0%, Test: n 58; 8.1%)
9.1 I05-109 Chronic rheumatic heart diseases - 2/0
9.2 120-125 Ischaemic heart diseases - 17 /
9.3 126-128 Pulmonary heart disease and diseases of pulmonary circulation
9.4 130-152 Other forms of heart disease 73 10
9.5 160-169 Cerebrovascular diseases 145 / 10
9.6 170-179 Diseases of arteries, arterioles and capillaries 455 / 20
9.7 180-189 Diseases of veins, lymphatic vessels and lymph nodes, not elsewhere classified 187 11
9.8 195-199 Other and unspecified disorders of the circulatory system - 4/0
10. Diseases of the respiratory system (Train: 271; 2.9%, Test: 23; 3.2%)
10.1 J00-J06 Acute upper respiratory infections 4/
10.2 J09-J18 Influenza and pneumonia 23
10.3 J30-J39 Other diseases of upper respiratory tract - 26
10.4 J40-J47 Chronic lower respiratory diseases - 24
10.5 J60-J70 Lung diseases due to external agents - -29-/999 /3
10.6 J80-J84 Other respiratory diseases principally affecting the interstitium 62/9
10.7 J85-J86 Suppurative and necrotic conditions of lower respiratory tract 8
10.8 J90-J94 Other diseases of pleura 31 /
61

---

Published as a conference paper at ICLR 2026
10.9 J95-J99 Other diseases of the respiratory system 64/2
11. Diseases of the digestive system (Train: 1174; 12.6%, Test: n 66; 9.2%)
11.1 K00-K14 Discases of oral cavity, salivary glands and jaws 77 6
11.2 K20-K31 Diseases of oesophagus, stomach and duodenum 143/ 6
11.3 K35- (335-K38 Diseases of appendix - 61 /
11.4 Hernia - 145 7
11.5 450 Noninfective enteritis and colitis - 51 11.6 K55 455-K66 Other diseases of intestines 3100 20
11.7 K65-K66 Diseases of peritoneum 77 5
11.8 K70-K77 Discases of liver 105 /
11.9 K87-KK7 Disorders of gallbladder, biliary tract and pancreas 176/9
11,10 K90-K93 Other diseases of the digestive system - 29 /
12. Diseases of the skin and subcutaneous tissue (Train: n 47; 0.5%, Test: n 0.8%)
12.1 L00-L08 Infections of the skin and subcutancous tissue- 17/0
112.2 1.50-1.54 Urticaria and erythema- - 4/0
122.3 1.00-1.75 Disorders of skin appendages 8
12.4 L80-L99 Other disorders of the skin and subcutaneous tissue 18/2
13. Diseases of the musculoskeletal system and connective tissue (Train: 636; 6.8%, Test:
71 61; 8.5%)
13.1 M00-M03 Infectious arthropathies 10/2
13.2 M05-M14 Inflammatory polyarthropathies - 28 5
13.3 M15-M19 Arthrosis -
13.4 M20-M25 Other joint disorders - 5
13.5 M30-M36 Systemic connective tissue disorders 66 10
13.6 M40-M43 Deforming dorsopathies - 0
13.7 M45-M49 Spondylopathies 2
13.8 M50-M54 Other dorsopathies 10/1
13.9 M60-M63 Disorders muscles - 70/3
13.10 M65-M68 Disorders of synovium and tendon - -53 /
13.11 M70-M79 Other soft tissue disorders 87 13
13.12 M80-M85 Disorders of bone density and structure 62.5
13.13 M86-M90
Other ostcopathies 114 11
13.14 M91-M94 Chondropathies - 41 /
13.15 M95-M99 Other disorders of the musculoskeletal system and connective tissue - 8
0
14. Diseases of the genitourinary system (Train: 450; 4.8%, Test: 64; 8.9%)
14.1 N0O-N08 Glomerular diseases 4/
14.2 N10-N16 Renal tubulo-interstitial diseases 52/6
62

---

Published as a conference paper at ICLR 2026
14.3 N17-N19 Renal failure - 2/
14.4 N20-N23 Urolithiasis - 21
14.5 N25-N29 Other disorders of kidney and ureter - 52/3 33
14.6 N30-N39 Other diseases urinary system - 69 /
14.7 N40-N51 Diseases of male genital organs 92 13
14.8 N60-N64 Disorders of breast 45 14
14.9 N70-N77 Inflammatory diseases of female pelvic organs - 8/5
14.10 N80-N98 Noninflammatory disorders of female genital tract 105 / 16
15. Pregnancy, childbirth and the puerperium (Train: n = 43; 0.5%, Test: n = 7; 1.0%)
15.1 000-008 Pregnancy with abortive outcome - -22/
15.2 010-016 Oedema, proteinuria and hypertensive disorders in pregnancy, childbirth and the puerperium 0
15.3 020-029 Other maternal disorders predominantly related to pregnancy 0/
15.4 (030-048 Maternal care related to the fetus and amniotic cavity and possible delivery problems 7
15.5 060-075 Complications of labour and delivery 5/
15.6 O85-O92 Complications predominantly related to the puerperium - 4/
16. Certain conditions originating in the perinatal period (Train: = 30; 0.3%, Test: = 0;
0.0%)
16.1 P10-P15 Birth trauma - /
16.2 P20-P29 Respiratory and cardiovascular disorders specific to the perinatal period - 7
0
16.3 P35-P39 Infections specific to the perinatal period - 1 /
16.4 P50-P61 Haemorrhagic and haematological disorders of fetus and newborn - 5
16.5 P70-P74 Transitory endocrine and metabolic disorders specific to fetus and newborn
3 3/0
16.6 775-P78 Digestive system disorders of fetus and newborn 6/0
16.7 P90-P96 Other disorders originating the perinatal period - 7/0
17. Congenital malformations, deformations and chromosomal abnormalities (Train: 984; 10.5%, Test:
60; 8.3%)
17.1 Q00-Q07 Congenital malformations of the nervous system 136 / 10
17.2 2210-Q15 Congenital malformations of eye, ear, face and neck 30/2
17.3 Q00QQQQ--0000 -28 Congenital malformations of the circulatory system 274
17.44 Q30-Q34 Congenital malformations of the respiratory system 60/7
17..5 Q35-Q37 Cleft lip and cleft palate - 1/0
17.6 Q38 Q45 Other congenital malformations of the digestive system 85
17.7 Q50 Q50-Q56660 Congenital malformations of genital organs 49 /
17.88 Q60-Q64 Congenital malformations of the urinary system - 55
17.99 Q65-Q79 Congenital malformations and deformations of the musculoskeletal system-
1555 /
17.10 Q80-Q89 Other congenital malformations -122
63

---

Published as a conference paper at ICLR 2026
17.11 Q90-Q09000 Q99 Chromosomal abnormalities, not elsewhere classified 7/0
18. Symptoms, signs and abnormal clinical and laboratory findings, not elsewhere classified (Train: n 42; 0.4%, Test:
1; 0.1%)
18.1 R00-R09 Symptoms and signs involving the circulatory and respiratory systems -
18.2 R19 Symptoms and signs involving the digestive system and abdomen 12/0
18.3 R20-R23 Symptoms and signs involving the skin and subcutaneous tissue 1/0
18.4 R30-R39 Symptoms and signs involving the urinary system - 3
18.5 R40-R46 R46 Symptoms and signs involving cognition, perception, emotional state and behaviour 1/0
18.6 R50-R69 General symptoms and signs - 6/0
18.7 R83-R89 Abnormal findings on examination of other body fluids, substances and tissues, without diagnosis - 1/0
18.8 R90-R94 Abnormal findings on diagnostic imaging and in function studies, without diagnosis - 14/0
18.9 295-R99 Ill-defined and unknown causes of mortality - 1 /
19. Injury, poisoning and certain other consequences of external causes (Train: 698; 7.5%, Test: = 31; 4.3%)
19.1 S00-S09 Injuries the head 47
19.2 S10-S19 Injuries to the neck 37 /
19.3 S20-S29 Injuries to the thorax - 73/1
19.4 S30-S39 Injuries to the abdomen, lower back, lumbar spine and pelvis 126/5
19.5 S40-S49 Injuries to the shoulder and upper arm -33/
19.6 S50-S59 Injuries the elbow and forearm 20 /
19.7 S60-S69 Injuries the wrist and hand 23
19.8 S70-S79 Injuries the hip and thigh 28 /
19.9 $80-S89 Injuries to the knee and lower leg - -43 8
19.10 S90-S99 Injuries to the ankle and foot - 22 2
19.11 TO0-T07 Injuries involving multiple body regions - 1 1/0
19.12 T15-T19 Effects of foreign body entering through natural orifice -48
19.13 T36-T50 Poisoning by drugs, medicaments and biological substances 4/0
T51-T65 Toxic effects of substances chiefly nonmedicinal
source 17/1
19.15 T66-778 Other and unspecified effects of external causes 12/
19.16 T79-179 Certain early complications of trauma - 6/0
19.17 T80-T88 Complications of surgical and medical care, not elsewhere classified - 158
/6
20. External causes of morbidity and mortality (Train: 4; 0.0%, Test: 0; 0.0%)
20.1 760-Y69 Misadventures to patients during surgical and medical care 4/0
21. Factors influencing health status and contact with health services (Train: 24; 0.3%, Test:
72 2; 0.3%)
64

---

Published as a conference paper at ICLR 2026
21.1 Z00-Z13 Persons encountering health services for examination and investigation 12 /0
21.2 Z400754 Persons encountering health services for specific procedures and health care -7/1
21.3 Z80-799 Persons with potential health hazards related to family and personal history and certain conditions influencing health status 5
22. Codes for special purposes (Train: 20; 0.2%, Test: 2; 0.3%)
22.1 U00-U149 Provisional assignment of new diseases of uncertain etiology or emergency use /
Note:
For each split, block counts within a chapter sum to the chapter total, and chapter totals across all chapters sum the split size.
Abbreviations: NEC elsewhere classified.
65

---

Published as a conference paper at ICLR 2026
R
RUBRIC FOR DISCUSSION EVALUATION
R.1
RUBRIC 1: DISEASE OVERVIEW CORF DEFINITION (0-2 POINTS)
Focus: Understanding of the discase's s fundamental attributes, including: nomenclature, classification, and etiology.
points: Unable to identify or define the disease.
point: States the disease name, but classification etiology is vague inaccurate.
2 points: Accurately states the standard medical name, clearly defines its essential nature, and identifies principal etiologies key risk factors.
R.2 RUBRIC 2: CLINICAL PRESENTATION & PATHOPHYSIOLOGY (0-2 POINTS)
Focus: How the disease manifests and its underlying mechanisms.
points: Unable to describe any clinical features.
Descrihes some common symptoms/signs but cannot explain the underlying pathophysiology. or omits critical features.
point:
2 points:
Systematically outlines the typical clinical presentation and clearly explains the core pathophysiologic mechanisms.
R.3 RUBRIC 3: KEY IMAGING FINDINGS & INTERPRETATION (0-2 POINTS)
Focus:
Recognition, description, and interpretation of discase-spccific imaging features modalities.
0 points:
Unable to describe any imaging characteristics.
point: Provides only generic descriptors (e.g.. "mass," "opacity") without coddaltty-speciffe features (CT, MRI, radiography. ultrasound), or fails to distinguish key benign versus malignant signs.
relevant modal- ities (e.g., morphology, attenuation/signn siggaal characteristics, margins, enhancement pattern, diffusion restriction), and interprets their clinical significance (c.g., stage, aggressiveness, complication risk).
2 points: Clearly and accurately describes characteristic findings
R.4 RUBRIC 4: DIAGNOSTIC REASONING DIFFERENTIAL DIAGNOSIS (0-2 POINTS)
Focus: Integrating clinical and imaging data to reach diagnosis and distinguish differential considerations.
points: Unable to articulate a diagnostic approach.
point: Arrives at the correct diagnosis but does not present a coherent, integrated reasoning process, does not propose appropriate differential considerations.
points: Clearly demonstrates how clinical information and imaging findings are synthesized to close the diagnostic loop, and lists at least two nigh-triority differential considerations with brief imaging discriminators (key features that separate each mimic from the index diagnosis).
R.5
RUBRIC 5: TRANSFERABLE LEARNING & GENERALIZATION (0-2 POINTS)
Focus: Lessons that extend beyond single case.
0 points: Teaching points are confined to this case.
point: Some generalizability suggested but remains vague and lacks actionable takeaways.
2 points: Clcarly summarizes transforable lcarning points and explains how to avoid misinterprctation improve diagnostic accuracy in similar future scenarios.
66


Reviews:
[{"review_id": "sHegsnHKN0", "reviewer": "Reviewer_8tFF", "summary": "This paper introduces MedThinkVQA, a large-scale benchmark explicitly designed to evaluate multimodal large language models (MLLMs) in multi-image diagnostic reasoning. Built from 8,481 expert-curated teaching cases (average 6.5 images per case), it formalizes a three-step workflow that mirrors clinical reasoning: (1) per-image findings, (2) cross-view imaging summary, and (3) differential-diagnosis reasoning, followed by a medical-education discussion task. \nEvaluation goes beyond simple accuracy by introducing step-level correctness, error-type tagging, and educational-value scoring. Baseline experiments on diverse MLLMs, show that even the best model reaches only 57.4 % accuracy, revealing that cross-image integration is the major bottleneck for current medical VLMs.", "questions": "See above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This is a high-quality benchmark paper with clear motivation and substantial novelty. It moves beyond single-image VQA toward multi-image, step-supervised diagnostic reasoning, something not covered by prior datasets such as OmniMedVQA or MedXpertQA. The design is conceptually elegant mirroring how clinicians think and technically meticulous, from option-wise pruning and leakage checks to fine-grained error analysis and human-LLM judge validation (κ≈0.8). \nThe beyond-accuracy evaluation framework, including RadCliQ metrics and teaching-discussion scoring, sets a new standard for interpretable benchmarking in medical AI. The authors also demonstrate strong ethical and reproducibility practices, releasing code, annotation scripts, and bias audits. Overall, the work is novel, comprehensive, and clinically grounded, offering a meaningful step toward trustworthy multimodal reasoning in medicine.", "weaknesses": "While the dataset is impressively detailed, it is built entirely from Eurorad cases, which may bias the distribution toward educational rather than real clinical imaging; cross-institutional validation or inclusion of temporal cases would further strengthen robustness. \nThe current evaluation focuses on four families of models and could benefit from broader comparisons to recent foundation-level MLLMs such as Gemini 2.5 Pro or Claude 3.5 Sonnet to fully situate difficulty. The study identifies the image-fusion bottleneck clearly but offers relatively limited prescriptive insight—there is little discussion of architectural directions or training strategies that could overcome this barrier.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "shyjpvNXQk", "reviewer": "Reviewer_zQ3B", "summary": "This paper introduces MedThinkVQA, a benchmark for evaluating multi-image diagnostic reasoning in medical imaging. The benchmark features 8,481 cases (751 test) averaging 6.51 images per case, sourced from Eurorad and expert-annotated with three-step supervision: (1) per-image findings, (2) case-level imaging summaries, and (3) differential diagnosis reasoning. This paper evaluates various VLMs (GPT-5, Qwen2.5-VL, MedGemma, InternVL) and find that current models struggle significantly (GPT-5: 57.39% accuracy), with the primary bottleneck being cross-image evidence extraction and integration rather than language reasoning. The benchmark includes beyond-accuracy evaluation with error-type tagging and medical education case discussion generation.", "questions": "1.  How do you ensure that Eurorad cases haven't been seen during pre-training of evaluated models?\n2. Only the train set (small scale dataset) is used to train the vlm? the detailed train (sft) strategy.\n3. Why not compared with the expert radiologist?\n4. The present question from benchmark is verified by human expert for quality control? If not, why?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper presents the largest expert-annotated multi-image medical QA benchmark.\n2. This paper presents the well-designed three-step evaluation framework that mirrors clinical diagnostic workflow.\n3. This benchmark performs rigorous dataset curation with multiple quality control measures, such as, leakage detection, confusion-aware pruning.", "weaknesses": "Weakness:\n1. Dataset relies entirely on Eurorad cases, potentially limiting generalizability despite broad coverage of radiology subspecialties.\n2. The benchmark data sourced from Eurorad may have been included in the pre-training corpora of some evaluated models.\n3. The case analysis is not presented, such as, failed case analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NX1gLvkeEX", "reviewer": "Reviewer_prRm", "summary": "This paper introduces MedThinkVQA, a benchmark built from Eurorad teaching cases, each pairing a clinical scenario with multi-image studies and expert reasoning artifacts. The task is framed as multi-choice diagnosis with a multi-step “think-with-images” pipeline: per-image findings, case-level integrated summary, differential-diagnosis reasoning, and a long-form medical education discussion generation. The evaluation goes beyond accuracy with stepwise checks, error-type tags, ROUGE/RadCliQ, and rubric-based/LLM-judge scoring of discussions, with human studies supporting reliability.", "questions": "Do cases in MedThinkVQA include redundant images (e.g. imaging study that does not provide valid information, or sometimes even contradictory information)? This aligns more to the practice where clinical users would not always do an image pre-filtering for the MLLM assistant. If that is the case, will such data produce a robust MLLM when used for model training and test models to ignore noisy information when used for model evaluation?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The problem formulation is well-motivated: prior medical VQA sets tend to be single-image, answer-centric, or automatically labeled; here the authors emphasize cross-view fusion and expert-authored intermediate signals that better mirror real diagnostic practice. \n\n2. Dataset quality and evaluation design are thoughtfully engineered. The paper details option sets derivation from expert differentials, textual leakage detection, pruning of items solvable by text-only LLMs, mitigation of surface biases, and wide coverage spanning 20/22 ICD-10 chapters. The evaluation is also staged and fine-grained.\n\n3. Engineering efforts in data curation and model evaluation bring concrete analysis and conclusions.\n\nGenerally, this is a good paper and I did not see major flaws.", "weaknesses": "1. Dataset statistics are inconsistent. Both the abstract and Table 1 state an average of 6.51 images per case, yet Line 200 in Sec. 3.1 mentions 8.3. \n\n2. Some details about the dataset are missing. For example, how are multiple images in one case gathered? Are they longitudinal studies from the same patient, or complementary imaging modalities (X-ray, CT, MR, etc), or both? If both are involved, I suggest analyzing them separately since two scenarios assess different capabilities.\n\n3. The evaluation relies on commercial LLMs (GPT-5) as LLM-judge, which might entail model/version drift risk since neither the API nor any specific snapshot is guaranteed to be available forever. Given this, I am wondering whether open-source models (e.g. Qwen series) are able to serve this and how the evaluation results will vary (e.g., will the scores differ drastically, or will there be any bias?). \n\n\n4. Lack of discussion with related work. For example, Medical-Diff-VQA [1] , ICG-CXR [2], MedFrameQA [3] are not mentioned in Table 1 and the manuscript, although they explicitly feature in visual reasoning with multiple imaging studies from the same patient.\n\n[1] Expert Knowledge-Aware Image Difference Graph Representation Learning for Difference-Aware Medical Visual Question Answering (KDD 2023)\n\n[2] Towards Interpretable Counterfactual Generation via Multimodal Autoregression (MICCAI 2025)\n\n[3] MedFrameQA: A Multi-Image Medical VQA Benchmark for Clinical Reasoning (arXiv 2025.05)\n\n---\n\nBelow are minor issues:\n- Each abbreviation should be expanded when it first appears for better readability. For example, “QA” (question-answering) in Line 010 and Line 035; “MCQ” (multiple-choice question) in Line 104.\n- In Lines 362--363: “Table shows representative model accuracy on the held-out test set.” Table index seems missing. \n- The prompts in the appendices (Secs. E, F, and G) are overflowing the right margin. Enabling automatic hyphenation or manually inserting hyphens may fix these issues.\n- I suggest copying the “without SFT” results in Fig. 3 to Tab. 8 which presents model performance after SFT, or combine these results in one single figure. In this way, the readers will see the value of the curated training examples more clearly.\n- I suggest adding a graph showing the imaging modality distribution, instead of plain description in Sec. I.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8IdRg1Zxrs", "reviewer": "Reviewer_qXc8", "summary": "This paper introduces an expert-annotated benchmark that probes the capabilities of multimodal LMMs on multi-image cross-view medical reasoning. Moreover, a step-wise evaluation protocol is proposed that can help pinpoint where models fail in their reasoning. State-of-the-art models are evaluated on the benchmark, and their mistakes are categorized into different error groups, highlighting the tendency for these models to fail on medical image understanding.", "questions": "- I would recommend restructuring the paper with clearer delineation between different sections. \n- How are steps defined in the step-wise evaluation and how the error categories are defined and determined?\n- How is the benchmark strictly probing multi-view reasoning? What happens if we remove some images? If the benchmark is truly probing cross-view capabilities, this would significantly degrade performance.\n- What is the expert performance on the benchmark?\n\nMinor: \n- Table number is missing on line 362.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "- The motivation of the paper is clear: it is important to assess multi-view capabilities of current models as it is closer to realistic diagnostic scenarios.\n- The step-wise evaluation and breakdown into error categories is valuable. It helps us better understand where these models fail in complex medical reasoning tasks. \n- The study on the utility of adding expert hints vs. self-generated captions is interesting and highlights the benefits of human-in-the-loop diagnostic workflows.", "weaknesses": "- Overall, the paper is hard to follow at parts due to multiple factors. First, the structure is unusual. The introduction could be more high-level without much details about methodology. Details about methodology is dispersed between the introduction and various other sections. Second, the paper assumes significant background knowledge on the source dataset used to create the benchmark. Is the ground truth reasoning trace part of the source dataset? What are the different sections of the source data and what kind of information do they exactly contain (e.g. \"Integrated Imaging Summary\" or \"Image Hint\")?\n\n- The step-wise evaluation is a bit unclear to me. How exactly is the generated output broken down into these steps? Furthermore, a rigorous definition of the error categories would be helpful, e.g. what does \"Clinical Scenario Error\" entail?\n\n- How is it supported in the work that multiple images are *necessary* to answer the questions accurately? If the problems can be tackled by a single view, then the key claim of probing cross-view synthesis is not well-supported.\n\n- Human performance is missing in the benchmark, making it difficult to gauge the gap between SOTA models and human experts on cross-view medical reasoning.\n\n- It is unclear what is the practical use-case of generating the teaching note in this benchmark. It seems only loosely connected to cross-view medical reasoning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)