Synthesizing High-Quality Visual Question Answering from Medical Documents with Generator-Verifier LMMs
Authors: Xiaoke Huang
Ningsen Wang
Hui Liu
Xianfeng Tang
Yuyin Zhou
Venue: ICLR 2026 Poster (2026)
Keywords: Medical VQA
Large Multimodal Models
Data Synthesis
Medical Literature
Vision-Language
Open-Weight Models
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We present MedVLSynther, a transparent, open-weight generator–verifier pipeline that synthesizes high-quality, context-aware medical VQA from the open subset of PubMed literature

Abstract: Large Multimodal Models (LMMs) are increasingly capable of answering medical questions that require joint reasoning over images and text, yet training general medical VQA systems is impeded by the lack of large, openly usable, high-quality corpora. We present MedVLSynther, a rubric-guided generator-verifier framework that synthesizes high-quality multiple-choice VQA items directly from open biomedical literature by conditioning on figures, captions, and in-text references. The generator produces self-contained stems and parallel, mutually exclusive options under a machine-checkable JSON schema; a multi-stage verifier enforces essential gates (self-containment, single correct answer, clinical validity, image-text consistency), awards fine-grained positive points, and penalizes common failure modes before acceptance. Applying this pipeline to PubMed Central yields MedSynVQA: 13,087 audited questions over 14,803 images spanning 13 imaging modalities and 28 anatomical regions. Training open-weight LMMs with reinforcement learning using verifiable rewards improves accuracy across six medical VQA benchmarks, achieving averages of 55.85 (3B) and 58.15 (7B), with up to 77.57 on VQA-RAD and 67.76 on PathVQA, outperforming strong medical LMMs. A Ablations verify that both generation and verification are necessary and that more verified data consistently helps, and a targeted contamination analysis detects no leakage from evaluation suites. By operating entirely on open literature and open-weight models, MedVLSynther offers an auditable, reproducible, and privacy-preserving path to scalable medical VQA training data.

Published as a conference paper at ICLR 2026
SYNTHESIZING HIGH-QUALITY VISUAL QUESTION ANSWERING FROM MEDICAL DOCUMENTS WITH GENERATOR- -EERRR LMMS
Xiaoke Huang"', Ningsen Wang Hui Liu Xianfeng Tang", Yuyin Zhou'
UC Santa Cruz 2 Fudan University
Amazon Research
* indicates equal contribution
https: //t::://titt thub. /tW//UUSSCVVLAAAUAAA
ABSTRACT
Large Multimodal Models (LMMs) are increasingly capable of answering medi- cal questions that require joint reasoning over images and text, yet training general medical VQA systems is impeded by the lack of large, openly usable, high-quality corpora. We present MedVLSynther, a rubric-guie guriied venrffr frame- work that synthesizes high-quality multiple-choice VQA items directly from open biomedical literature by conditioning on figures, captions, and in-text references. The generator produces self-contained stems and parallel, mutually exclusive op- tions under a machine- -chekkble JSON schema; a multi-stage verifier enforces essential gates (self-containment, single correct answer, clinical validity, image- text consistency), awards fine-grained positive points, and penalizes common fail- ure modes before acceptance. Applying this pipeline to PubMed Central yields VQA: 13,087 audited questions over 14,803 images spanning 13 imaging modalities and 28 anatomical regions. Training open-weight LMMs with rein- forcement learning using verifiable rewards improves accuracy across six medi- cal VQA benchmarks, achieving averages of 55.85 (3B) and 58.15 (7B), with up 77.57 VQA-RAD and 67.76 Path outperforming strong medical LMMs. Ablations verify that both generation and verification are necessary and that more verified data consistently helps, and a targeted contamination analysis detects no leakage from evaluation suites. By operating entirely on open literature and open-weight models, MedVLSynther offers auditable, reproducible, and preeecypreservinggg path to scalable medical VQA training data.
INTRODUCTION
Large Multimodal Models (LMMs) are rapidly becoming capable assistants for biomedical discov- ery and clinical education, where questions must be answered by jointly interpreting medical images (e.g., X-ray, CT, microscopy) and the surrounding textual context (e.g., figure captions, narrative de- scriptions, etc.). Despite fast progress, training general medical VQA systems remains difficult because the community lacks large, openly usable, and high-quality training corpora.
On the evaluation side, recent benchmark (Hu et al., 2024 Ye al., 2024 provide broad and chal- lenging test suites, but they are designed for assessment rather than training and therefore offer no training splits. On the training side, existing datasets fall into three categories, each with limi- tation. 1) Manually curated sets (Lau et al. 2018: Liu et al., 2021: He et al.) (020) are carefully annotated but are either small or bound to narrow modalities and topics, limiting coverage and trans- fer. 2) Automatically generated sets (Zhang et al.) 2023b] Chen et al. (024c) scale more easily but are typically produced by text-only LLMs that ignore visual evidence and figure-text relations, yielding noisy stems, ambiguous options, and medically dubious answers that can impede model learning. 3) Closed, large-scale resources (Li et al. 2024) exist but are not publicly shareable due to patient privacy, licensing, and institutional agreements, which slows open research and repro- ducibility. Collectively, these constraints lead to a practical bottleneck: we can evaluate medical VQA systems comprehensively, but cannot train them broadly and transparently.

---

Published as a conference paper at ICLR 2026
(<) Stage Generation
(c) Performance Improvement Lome
Secttrenow (Don)
Medical
Rubric-Basee -asee LMM Generator
Medical VQA
Document
Stage Verification
Medical
Document
Multi-Stage Rubric-Based LMM Verifier
Filter
Not
Medicall
VQA
PatiVVAA
VQA
Figure 1: (a) Stage-1 generation: a rubric-guided LMM converts PubMed figures and captions into multiple-choice VQA items. (b) Stage-2 verification: multi-stage, rubric-based LMM verifier screens items and filters low-quality ones. (c) Training open-weight students (3B/7B) on MedSyn- VQA yields consistent gains over strong medical LMM baselines.
This paper asks a simple question: can we synthesize high-quality, auditable medical VQA data di- rectly from open biomedical literature? Our answer is MedVLSynther generator-verifier frame- work that leverages state-of-the-art open-weight LMMs (Zeng et al. 2025 Wang et al., 2025 Bai et al. 2025 to produce and automatically vet VQA triplets from figures and surrounding text in PubMed articles (ozanoo et al. [025 Roberts 2001). The key design choice is to make both gener- ation and verification explicitly rubrie-driven and context-aware.
Rubric-guided context-aware generation (Figure (ig) Given a figure, its caption, and the fig- ure's in-text reference paragraph when available, the generator LMM instructed to propose VQA item, including question stem, multiple-choice options, and the correct answer, under a com- prehensive rubric. The rubric enforces that stems are self-contained and anchored in the provided visual-textual context, that options are parallel and mutually exclusive, and that the answer can be justified from the figure and caption, not from world knowledge alone. The rubric also specifies set of accepted question archetypes (e.g., recognition, localization, comparative, reasoning) and a ISON schema/format that simplifies downstream filtering and training.
Multi-stage rubric-based verification (Figure I (b)). To ensure quality, we feed the same con- text and the generated VQA to a verifier LMM and score it in three stages: 1) Essential criteria form strict pass/fail gates. Any failure discards the item. 2) Fine-grained criteria award positive points with justifications, and allow the verifier to surface additional criteria opporuuistically. 3) Penalty criteria investigate common failure modes and subtract points when detected. We sum the fine-grained and penalty scores and apply a threshold to filter surviving items. This verifier is model-agnostee and be instantiated with any open- weight LMM; in practice we find that verifier different from the generator improves robustness.
The verifetor-verifier loop yields data pipeline whose rules are transparent and auditable end-to-end. Because we build on open literature rather than protected clinical data, the entire pipeline, including prompts, rubric, and metadata, can be inspected and reproduced. At the same time, recent open-weight LMMs rival proprietary systems on many multimodal tasks (Zeng et al.) 2025). allowing us benefit from strong perception and reasoning while staying fully open.
The resulting medical VQA dataset, MedSynVQA. covering diverse modalities, subspecialties, and question archetypes. Models trained on this data with Reinforcement Learning with Veri- fiable Rewards (RLVR) (Guo et al., [225]] Shao et al. 2024) outperform counterparts trained on PMC-VQA (Zhang et al. (023b), as well as the strong baseline trained on text-only medical cor- pora (Huang et al. (025b). As summarized in Figure T (c), our training improves accuracy on MMMU-Med (Yue et al. 2024), PathVQA (He et al.] 2020). and VQA-RAD (Lau et al., 2018) over strong baselines (Alshibli et al. 2025 Li et al. 2023). Meanwhile, ablations reveal that (i) both generation and verification are necessary: their synergy yields the best accuracy, and (ii) scale matters: more verified data consistently helps. We analyze topic coverage, modality distribution, and question types, and most importantly, conduct a contamination analysis tailored for synthetic medical VQA; we find no detectable leakage from the evaluation sets.
2

---

Published as a conference paper at ICLR 2026
Table 1: Comparison among medical VQA datasets. MedSynVQA is open and reproducible, cov- ering 13 modalities and 28 anatomical regions, with 13,087 questions over 14,803 images. "N/A" indicates missing statistics. "# Rate" denotes ratio of images/questions.
Questions
Images
Modality
Anatomyy
Availability
Training
MedXpenQA-MMe
2,000
2,852
1.43
10
11
Expert
Open access
Yes
GMAL-MMBench
25,831
25,831
1.00
38
N/A
Automatic
Open access
Yes
No
Automatic
Open
SLAKE
14,028
642
0.05
Expert
Open access
No
VQA-RAD
3,515
315
0.99
3
3
Expert
Open access
No
Yes
PadlVQA
Automatic
Open
PMC-VQA
226,946
149,075
0.66
N/A
Automatic
Open access
GMAL-VL-S VL-S SM
20: 5,500,000
N/A
NIA
13
NIA
Automatic
Not Open
Yes
Yos
MedSymVQQ
13,087
14,803
1,13
13
Automatic
Open
Yes
Yes
Flating
13.9%
33.0%
1a.m
Ora CarRy
Makay
Question type
(b) Modality
(c) Anatomy
ul ultrasound Labe ed
pat ienent
Datienti
observed
modal
pattern
Drann
ure
surgin gig.icalln
Mele right
Lumor
augafiica
Compar
red
Lattiyy lesion trea tmen
tresen
Seterr oup
eglon
moaaaaa indings
clinical. S
(d) Word cloud for generated questions.
Figure 2: MedSynVQA statisties: 1) Dataset distributions for question type, imaging modality, and anatomy. 2) Word cloud for generated questions.
Our contributions are summarized as follows:
MedVLSynther, a rubric-guided, context-aware generator-verifirr pipeline that syn- thesizes reliable medical VQA from open biomedical articles.
A comprehensive rubric for medical VQA quality, spanning essential gates, fine-grained positive criteria, and penalty criteria, together with a machine-checkable schema that sup- ports automatic filtering and auditing.
A synthetic medical VQA training set (MedSyn that substantially improves med- ical LMMs on multiple medical VQA benchmarks and complements existing resources without relying on private patient data.
Transparency and reproducibility: our pipeline operates entirely on open data and open models, enabling the community inspect prompts, scoring rules, and filtering decisions end-to-end.
While synthetic data cannot replace carefully curated clinical datasets, our results indicate that high-quality, auditable synthesis both feasible and useful for medical VQA. We hope MedVLSyn- ther provides practical path scalable training data that respects privacy, encourages openness, and accelerates progress in multimodal medical intelligence.
2 RELATED WORKS
Multimodal medical VQA. Early, expert-curated datasets (Lau et al. 2018 Liu et al. 2021 He
al. 2020) established Med-VQU but remain small mosslity- limiting transfer. Later,
3

---

Published as a conference paper at ICLR 2026
(a) Extraction & Pre-Filtering
(b) Rubric-Based Context-
Publ
Figure(s)
Aware Generation
Question Options
Decument
Medical
Primary Tog:
Secondary Tag:
Clivical imaging, Microscopy....
Caption [Ref Para.
Open Weight LAW
Cphoed Ornere
Encertil Enterio
Le Eepporten Onfens
X-ray, Optical, Shut,, Brais...
Former Corehnirt
(c)
Multi-Stage Rubric-Basse -untext-Aware Verification
Figure(s)
Essentiol Criteria
Fine grained Chitena
Peralit ty Criterio
13 Siri Self cortieet
(21 fordilal Carlens
(P3 Forsidoo Tome:
Caption
() Weodeleyy Costroint
RRef Para.
If all poss
(-3 2S0N Scrvene Conal
Merice Ieccuuscy (-1) Swonee b--ft
Filter or Not Threshold
siglt
dyries
(4) tten Coscaiee
Question
1- Optice "We Centemency
(12) and Fens
Options
13 Cliid Nasdiny
Answer
Smage Ter:
Figure 3: From PubMed documents we extract figures and reference text, then apply (a) extraction and pre-filtering by primary/secondary tags; (b) rubric-based, context-aware generation with format constraints and question archetypes; (c) multi-stage verification with essential, fine-grate ane-grained, and penalty criteria. Items retained if their rubric score exceeds threshold.
broad benchmarks (Hu et al. 2024] Ye et al. 2024 Zuo al. 2025) consolidated evaluation across many modalities and anatomies yet offer little no training data, creating a supervision bottleneck. In contrast, large cevrature-deriver corpora (Subramanian et al., 2020; Rückert al., 2024) and especially Lozano et al. (2025)'s s 24M image-caption pairs provide open, scalable raw material. Our work converts this open substrate into exam-quality VQA by coupling context-aware generation with rigorous verification, bridging the gap between expansive evaluation suites and accessible training data.
Synthetic data generation for multimodal medical VQA. Prior synthetic pipelines scale super- vision but suffer quality issues: [i al. (2023)'s s self-instruct approach and ZZhang et al. (2023b)'s 227k auto-generated pairs (largely from text-only LLMs) can omit modality cues, produce am- biguous stems, and yield visually ungrounded answers; broader compilations like [i et al. (2024) closed, while speccit (Hu et al.) sets remain narrow. These limitations motivate a quality-firr strategy: we condition on figures, captions, and in-text references and enforce a rubric-guided generator plus multi-stage verifier filter low-quality items, yielding reliable, open data suitable for training medical LMMs without relying on private images.
2024 Multimodal models, medical adaptation, and reasoning. General LVLMs (Hurst et al.
Comanici et al.) 2025 Wang et al., 2025 Bai et al. 2025] Liu et al., 2024 An et al. 2025
An et al.) acquire instruction following via visual SFT. while medical variants TT al. 2024 Luo
ct al. 2023| Alshibli et al. 2025 Liu et al. 2023) Wu et al. 2025 Chen et al. 2024b]] Zhou et al.
Wu et al. 2023) add in-domain pretraining and SFT/RL. for clinical competence. Recent deliberate-reasoniing models (Jaech et al. 2024) show that reinforcement learning with verifiable
2024
rewards (e.g., GRPO (Guo et al.J 2025 ) can surpass SFT-only methods on multi-step problems, and early medical efforts point the same way but lack open, high-quality multimodal supervision. Our rubric-verified VQA corpus supplies that missing signal and pairs naturally work for RLVR, contributing auditable and trustworthy visual reasoning in open- gpinhweight medical LMMs (Chen et al., 2024c Su al. 2025).
3 MEDVLSYNTHER AND MEDSYNVQA
Our goal is to synthesize high-quality, clinically valid multiple-choice VQA (MC-VQA) exam- ples directly from biomedical papers (Lozano et al.] 2025). We cast the task as a Genera- tor-Verifier pipeline driven by Large Multimodal Models (LMMs)): rubric-guide generator produces MC-VQA items from figures and text, and multi-stage rubric- guided verifier performs automatic quality control before data are admitted to the final corpus (Figure 3.
4

---

Published as a conference paper at ICLR 2026
3.1 DATA SOURCE, EXTRACTION, AND FRE-FILTERING
Source. We build on Biomedica (Lozano et al. 2025). large-scale extraction of figures and figure-level metadata from the PubMed Central Open- Open-Aceess Access (PMC-OA) collection (Roberts) 200T). For each paper we ingest: 1) The figure image(s) (a single caption may reference up to 6 images), 2) The figure caption. 3) The corresponding figure references in the main text (when present).
Samples missing either images caption are discarded.
Pre-filtering. We retain items annotated by Lozano et al. 2025 with the primary labels: Clin- ical imaging and Microscopy, and 25 secondary subtypes (c.g., x-ray radiography, optical coher- ence tomography, skull, brain, etc.). After pre-filtering we obtain 23,788 tigure-caption(-releeeeee triplets.
We denote each pre-filtered sample by
x= (I,
where I is one more images, C the caption, and R the in-text references.
Choice of generator and verifier LMMs. We state-of-the-art open-weight LMM pable of long-context vision-language reasoning: GLM-4.5V-108B Zeng et al. (025). InterVL-3.553385 /Wang& et al. 2025), and Qwen2.5-VL-72B (Bai al. 2025 Unless otherwise noted, GLM T08B serves as the default generator due to its strong estruction-followwee and image-grounding performance. The rubric and striet JSON schema make the output predictable and machine-verte verifablee
3.2 RUBRIC- BASED, CONTEXT-AWARE VQA GENERATION
Given the generator LMM Go produces 5-option MC-VQA instance in strict JSON format: {g. options(A.. answer E {A..E)}- Generation is contexf-aware the model receives the image(s) together with C and R. To ensure exam-quality items, the prompt instills the role of an expert
item writer and enforces a self-check rubric.
Essential (must pass before output): 1) Stem self-contained (no "caption/context" men- tions); 2) Image-content alignment (requires inspecting specific visual features); Implicit use of caption facts without answer leakage; 4) Exactly one best answer; 5) Medical cor- rectness (modality, anatomy, terminology).
Important (strongly recommended): cognitive level is over application; strong, parallel distractors; clear focus single concept.
Optional: localization quantitative details when clearly supported.
A small set of question archetypes (i.e., Finding/Abnormalin Identification, Modality Recognition, matoom/LLocalizatio Other Biological/Technica Attributes, Disease Diagnosis, Next Step, and Lesion Grading) reduces prompt entropy and encourages clinically meaningful questions.
3.3 MULTI-STAGE, SUBRIC-BASED. CONTEXT-AWARS VERIFICATION
While the generator reliable, automatic verification is essential for scale and precision. Given and candidate MC-VQA the verifier LMM Vg, is prompted to operate in two roles, Referee and Critic, and return only structured rubric with binary scores. Verification is also context-aware: V, sees the same images, caption, and references as Go plus the proposed MC-VQA.
Stage-1: Essential screening (hard gate). The Referee evaluates seven non-negotiable items; a sample must pass all to proceed: 1) Stem Self-contained; 2) Vocabulary Constraint (no unsup- ported clinical facts); 3) Diagnosis Leak (no verbatim restatement from sources); 4) Single Correct Option; 5) Option Type Consistency (same semantic type); 6) Clinical Validity (terminology/modaa. ity/anatomy); 7) Image-Text Consistency.
5

---

Published as a conference paper at ICLR 2026
Items scored (0,5) 5) with fair, rule-based mindset. During this stage, remove instances that the verifier cannot grade (e.g,, malformed JSON), leaving 23,635 candidates. After applying the essential filter, 22,903 remain.
Stage-2: Fine-grained positive criteria (bonus points), The Critic now assumes the item is not excellent and awards points only on irrefutable evidence We query 4-8 bonus criteria (binary, with weights Important = 3 or 4, Optional = or 2), including: 1) Plausible Distractors (every distractor a strong near-miistt miss); 2) Parallel Options (length/structure uniformity): 3) Stem Concision (less than two sentences and concise); 4) Clarity and Focus (single, unambiguous question); 5) Answer-field Validity (answer exists, matches an option); 6) JSON Schema Compliance (exact keys, no extras).
The Critic denies criterion if it can imagine slightly better wording distractor, pushing precision over recall.
Stage-3:
Penalty criteria (error hunting). Finally, the Critic actively searches for pitfalls (neg- ative weights): 1) Forbidden Terms (-2; stem contains "caption/contex(") 2) Synonym Drift (-1; introduces unsupported specific facts); 3) Multiple Keys (-2), and Medical Inaccuracy (-2).
Each pitfall is triggered only with concrete reason.
3.4 AGGREGATION AND ACCEPTANCE RULE
Let P be the set of positive (Important U Optional) criteria with weights w; > 0 and binary scores 8 E {0, w.). Let N be the pitfalls with wy and scores Py E f0, wy We compute normalized quality score:
EiEp 8: + +EjENN Eiep W;
S(x, v) clip
Candidates passing Stage-1 are accepted S(x, y) 2 with = 0.9670. This high threshold emphasizes precision while keeping a useful yield; results in 13,087 MC-VQA items, which call MedSynVQA.
3.5 TRAINING MEDICAL LMMs WITH MEDSYNVQA
We use our synthesized corpus to train medical LMMs with two LMM finetuning approaches.
Supervised Fine-Tuning (SFT). Following MedVLThinker, elicit thinking traces with GLM-45V--08B and perform SFT on (thinking trace, answer) pairs. The supervision emphasizes clinically grounded reasoning paths while preserving the strict answer format.
RL with Verbal Rewards (RIVR). We then apply GRPO on answers only (no trace optimization), again mirroring bypem--arameters from Huang et al. (2025)). The reward promotes exact-match accuracy and adherence to the schema without over-fitting to any single imaging modality.
4 EXPERIMENTS
4.1 SETUP
Unless otherwise stated, we finetune two open-weight LMMs Qwen2.5- VL 3B and 7B Instruct (Bai et al. 2025), using the same training schedule, image resolution, tokenization, and optimization papae-parameters as Huang et al. 2025b). We use our rubric-guided generator-verifier pipeline to synthesize training items from PubMed figures and captions (Figure (3), and train students either with SFT or RLVR. Unless otherwise noted, experiments use 5K samples.
Models.
Benchmarks and metric. We follow Huang et al. (2025b) evaluation suite and scripts, re- porting multiple-choice accuracy six medical VQA benchmarks: MMMU medical split (MMMU-Med) (Yue et al.) 2024). (MedX-M) (Zuo et al. 2025). PathVQA (He et al.] 2020). PMC-VQA (Zhang al. 2023b), SLAKE (Liu al. 2021), and VQA-RAD (Lau al. 2018).

---

Published as a conference paper at ICLR 2026
Table 2: G pipeline ablation. Rubric-guided generation outperforms PMC-Sttle Text-Image Generation, and adding verification yields the best average accuracy. Cells are shaded by accuracy; darker better.
Model
MMMU
MedX-M
PathVQA
PMC
SLAKE
VQA-RAD
Avg.
Qwen2.5 V1.-3B-Instris
44.12
20.69
61.96
44.77
61.30
62.01
49.14
AMyl-tyle Text-only Generation
46.47
20.8
62.43
$1.03
73.08
71.69
54.25
PMC-Style Text-Image Generation
20.40
73.08
72.06
54.80
Rubric Context-Awaid Aware Generation
52.35
20.60
62.49
51.83
70.43
70.59
54.72
Rubric Context-Aware Verification
52.35
21.40
62,82
50.23
74.76
73.53
53.85
Qwen2.5 1--n2.5-VL-7--Instrre
52.94
18.89
65.39
49.30
65.71
68.75
53.50
PMC-Style Text-only Generation
54.71
22.15
63.89
$3.23
67.07
65.44
54,41
PMC-Style Text-Image Generation
51.76
21.70
53,43
71.69
55.15
Rubric Context-Awer Aware Generation
58.24
23.50
65.41
53.83
68.03
75.00
57.33
Rubric Context-Aware Verification
57.06
23.15
66.36
53.78
67.79
77.21
57.56
Table 3: Dataset scale ablation. Effect the number MedSynVQA training items (1k-13k) on downstream accuracy. Performance improves with scale, with diminishing returns beyond 5k examples. "N/A" denotes zero-shot (no additional training). Cells are shaded by accuracy.
Model
Scale
MMMU
MedX-M
PathVQA
PMC
SLAKE
VQA-Rad
Avg.
N/A
44.12
20.69
61.96
44.77
61.30
62.01
49.14
Qwen2.5-VL
1000
50.59
20,20
63.18
48.37
65.87
67.65
52.64
2000
47.06
19.95
64.07
47.27
74.04
76.84
54.87
3B-Instruct
5000
52.35
21.40
62.82
50.23
74.76
73.53
55.85
10000
48.82
20.55
63.44
49.87
72.84
74.63
55.03
Full
51.76
22,30
63.03
48.92
72.60
72.43
55.17
N/A
$2.94
18.89
65.39
49.30
65.71
68.75
53.50
Qwen2.5-VL
1000
57.65
21.60
65.53
50.93
68.27
65.81
54.96
7B-
2000
60.00
22.35
67.76
51.18
67.31
73.16
56.96
5000
57.06
23.15
66.36
53.78
67.79
77.21
57.56
10000
$7.06
22.45
66.86
52.73
71.88
73.90
57.48
Full
55.88
22.10
65.56
55.43
72.36
77.57
58.15
Baselines. We compare against strong general-pun purpose and medical LMMs used in Huang et al. (2025b), including Gemma3 4B (Team et al.] 2025), Bwen2.5-VL-3BB77B-mm 4B (Sellergren et al. 2025). LLaVA-Med (Li et al. 2023), HuatouGPT-Vision--1 (Chen et al.) 2024c), and VedVLThinker (Huang et al. 2025b), strong baselines trained solely on text-only data.
MedGemma
4.2 RESULTS
Ablation the Cenerator-Verifeer pipeline. Table 2 studies each stage of our pipeline. We begin from zero-shot Qwen2.5-Ve VI. students and add 1) PMC-style text-only question generation, 2) rubric-guided context-aware generation, and 3) rubric-aware verification. For 3B student, the base model averages 49.14. Text-only generation lifts the average to 54.80. Switching to rubric-guided, context-aware generation performs similarly on average (54.72). Adding verification yields the best average, 55,85, with large gains on clinically grounded datasets. For 7B student, the base model averages 53.50. Text-only generation yields 55.15, rubric- rubric-guided generation 57.33, and with veri- fication we obtain 57.56 and again improving across benchmarks. Overall, rubric guidance already outperforms PMC-style text-only recipe, and multi-stage verification supplies the remaining head- room, producing the best average in both scales (Table 2 The trend aligns with the high-level improvements visualized in Figure]
How much synthesized data do we need? We vary the number of MedSyn VQA training items from IK to 13K (Table 3. For 3B student. Accuracy increases from 52.64 (IK) to 55.85 (5K), then plateaus at 55.03 (10K) and 55.17 (Full). For 7B student. The curve is similar: 54.96 (1K), 56.96 (2K), 57.56 (5K) with slight dip to 57.48 10K, and peak of 58.15 with the full dataset. This tendency suggests the potential for further refinement of the filtering method. Moreover, to reduce computational cost, use 5K items as the default scale in subsequent experiments.

---

Published as a conference paper at ICLR 2026
Table Choice of generator and verifier LMMs. We vary the generator and verifier. Higher-capacity generator/verifier pairs produce higher-quality data and consistently improve the final average accu- racy. "N/A" indicates the zero-shot performance. Cells are shaded by accuracy.
Model
Generator
Verifier
MMMU
MedX
PathVQA
PMC
SLAKE
VQA-Rad
Avg.
61.96
44.77
62.01
Qwen2.5-VI.
GLM-4.5V 108B
Qwen2.5-VL. 728
52.35
21.40
62.82
50.23
74.76
73.53
55.85
38-Instruct
GLMV4.5V 1088
GLM-4.5V
1088
51.18
20.30
63.56
50.63
71.63
70.22
54.59
Owen2.5-VL
GLM-4.5V
108B
47.65
21.50
62.37
48.87
69.85
53.93
113e 38B
GLM-4.5V 108B
49.41
21.90
61.81
51.98
74.76
71,32
55,20
N/A
N/A
52.94
18.89
65.39
49,30
65.71
68.75
53.50
Qwen2.5-VL.
GVV
Qwen2.5-VI.
77.21
7B-Instruct
GLM-4,5V 108B
GLM-4.5V 1088
58.82
23.65
67.22
54.48
71.15
73.16
58.08
QWen2.5.VL. 72B
GLM-4.5V 108B
56.47
22.55
67.25
52.38
67.07
72.79
56.42
VL.5 38B
GLM-4.5V 108B
57.65
23.30
66.12
53.58
70.67
75.37
57.78
Table 5: Training approach and data source ablation. Comparing SFT and RLVR using three sources: PMC (image-text), m23k (text-only), MedSynVQA. RL consistently outperforms SFT, and MedSynVQA leads to the highest averages across all benchmarks. Cells are shaded by accu- racy.
Model
MMMU
McdX-M
PathVQA
PMC
SLAKE
VQA-RAD
Avg.
Qwen2.5-VL V----B----sttuucc
44.12
20.69
61.96
44.77
61.30
62.01
49.14
SFT (PMC)
47.84
21.46
52.76
54.55
65.79
58.58
50.16
SFT (m23k)
32.55
16.00
42.74
28.53
43.91
33.09
32,80
SFT (MedSynVQA))
48.82
20.90
63.12
47.57
54.33
59.93
49.11
RL (PMC)
48.43
21.51
51.61
54.22
75.56
62.38
52.28
RL (m23k)
52.16
22.90
62.28
47.32
63.38
71.08
53.19
RI. (MedSSyynVQAA
52.35
21.40
62.82
50.23
74.76
73.53
55.85
Qwen2.5- -L--2-5-VL7B--nnsiin
52.94
18.89
65.39
49.30
65.71
68.75
53.50
SFT (PMC)
49.80
21.39
53.02
$4.67
67.71
57.72
50.72
SET (m23k)
46.86
16.40
56.35
34.58
54.97
53.80
43.83
SET (MedSynVQA)
49.41
20.90
64.81
50.08
59.62
66.54
$1.89
RL (PMC) RI (m23k)
55.29
57.09
55.38
66.59
53.66
56.86
24.43
66.83
50.67
65.79
64.71
$4.88
RL (McdSynVQA)
57.06
23.15
66.36
53.78
67.79
77.21
57.56
Which generator and verifier LMMs should we use? We next vary the capacity and identity of the generator and verifier LMMs used during data synthesis (Table 4 For the 3B student, pairing GLM-4.5V-I0SB generator with Qwen2.5-VL-77 VLL-BBB verifier yields the best average 55.85; other high-capacity pairs are close. For the 7B student, the same open-weight verifier gives 57.56 with GLM-108B generator, while using GLM-108B both generator and verifier further nudges the average to 58.08. We keep the Qwen2.5-VL-722 verifier for the main results to maximize repro- ducibility with open weights, but Table indicaaes that stronger verifier capacity translates to higher downstream accuracy.
Training approach and data source ablation. Table 5 compares SFT RL from verification
reward (RL) across three data sources: PMC-VQA (image-tes pairs) (Zhang et al. (023b), m23k (text-only) (Huang et al.] 2025a). and MedSynVQA.
RL outperform SFT for both 3B and 7B models, across all data source. 2) Under RI. the MedSynVQA signal is the strongest, giving the best average on both 3B (55.85) and 7B (57.56). The results indicate that rubric-based context- aware MedSynVQA dataset are more effective training source than the previous synthetic PMC- VQA (Zhang et al. 2023b) and the text-only one (Huang et al. 2025b).
Comparisons. Table summarizes head-to-head results the full benchmark suite. Our stu- dents trained with MedSyn achieve 55.85 (3B) and 58.15 15 (7B), state-of- of-t-oo-tthe-an averages among open-wen weight models considered. Notably, our 3B student surpasses MedVLThinker-7B by +0.97 and all other 3-7B baselines; the 7B student improves over the best prior MedVLThinker--75 by +3.27. Gains consistent across datasets, with strong results VQA-RAD (up 77.57).
8

---

Published as a conference paper at ICLR 2026
Table 6: Comparison to basclines. Average and per-benchmark accuracy of general- peneral-purpose and medical LMMs versus models trained with MedSynVQA, Both MedVLSynthn 3B and 7B achieve the best average across benchmarks, demonstrating strong gains small and medium scales.
Model
MMMU
MedX-M
PathVQA
PMC
SLAKE
VQA-Rad
Avg.
General LLM
Gemme 4B
21.89
59.24
44.42
66.59
56.86
Swen2.5-VLL-3B--
44.12
20.69
61.96
44.77
61.30
62.01
49.14
Owen2.5-VL-7B-Instmie
52.94
18.89
65.39
49.30
65.71
68.75
53.50
Medical I.I.M
MedGemma 4B
32.55
8.17
59.64
42.73
83,49
78.55
50.86
MedGemma 27B
35.88
12.13
62.09
36.75
77.40
72.67
49.49
Llava Med V1.5 7B
31.37
22.56
56.52
34.28
62.82
56.74
44.05
HuatuoGP"T-Visoon
50.59
22.00
63.53
53.39
75.00
63.60
54.69
MedVLThinker-2
52.16
22.90
62.28
47.32
63.38
71.08
53.19
McdVL.Thinker-77
56.86
24.43
66.83
50.67
65.79
64.71
54.88
MedVI. Synthernt McdVLSSntthr-77
52.35
21.40
62.82
50,23
74.76
73.53
55.85
55.88
22.10
65.56
55.43
72.36
77.57
58.15
Case study. Figure presents two cases, revealing deep comprehension with context for our gen- erator and the leakage rejection by our verifier. Please refer to the appendix for more details.
Contamination analysis. We practice contamination analysis between MedSyn and the eval- uation suites in the appendix. No overlaps were found under this protocol.
5 CONCLUSIONS
MedVLSynther shows that high-quality, auditable medical VQA data can be synthesized at scale from open biomedical literature by pairing rubric-guided, context-aware generation with multi-stage verifier. The resulting MedSynVQA delivers consistent gains for open-weight LMMs across six benchmarks and ablations confirm that both the generator and verifier are necessary. Op- erating entirely open data and models, the approach offers reproducible, sewecy.preserving. and transparent path to supervision for medical VQA.
ACKNOWLEDGMENTS
This work was partially funded by an unrestricted gift from Google.
REFERENCES
Ahmad Alshibli, Yakoub Bazi, Mohamad Mahmoud Al Rahhal, and Mansour Zuair. biooln Large vision language model for visual dialogue in biomedical imagery. Biomedical Signal Pro- cessing and Control, 103:107433 2025.
Zhaochong An, Guolei Sun, Yun Liu, Runjia Li, Min Wu, Ming-Ming Cheng, Ender Konukoglu, and Serge Belongie. Multimodality helps few-shot 3d point cloud semantic segmentation. In The Thirteenth International Conference on Learning Representations.
Zhaochong An, Guolei Sun, Yun Liu, Runjia Li, Junlin Han, Ender Konukoglu, and Serge Belongie. Generalized few-shot 3d point cloud segmentation with vision-language model. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 16997-17007, 2025.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502, 13923, 2025.

---

Published as a conference paper at ICLR 2026
Case I: Context-Aware Generation
Question: Which of the following best describes the vertebral anomaly shown in this image?
Options: (A) Six lumbar vertebrae with incomplete sacral integration
(B) Five lumbar vertebrae with complete sacral fusion...
Answer: (A)
Caption: Lumbar transitional vertebrac
Context: The subject was diagnosed as having CLBP and lumbar
transitional vertebra. Computer tomography (CT) showed six lumbar
vertebrae, which one more lumbar vertebra than a normal person... This indicates the first sacrum is not completely integrated...
Pass Verification: True
Case 2: Leakage Rejection by Verifier
Question: A 37-year-old woman presents with palpable mass in the upper outer quadrant of the right breast. Imaging and biopsy reveal a circumscribed hypoechoic mass with flow, invasive duc- tal carcinoma, grade 1, with predominant tubular formation, mild nuclear atypia, and strong diffuse ER positivity. Which histopatho- logical feature is most consistent with the tumor grade?
Options: (C) Well-differentiatee with predominant tubular for- mation and mild nuclear atypia...
Answer: (C)
Caption: ...invasive ductal carcinoma, grade 1, ...shoning predominant tubular formation (arrows), mild nuclear atypia ... ER immunostain showing strong diffuse positivity ... Context:
Pass Verification:
False
Figure 4: Examples of context-aware generation and leakage rejection by the verifier.
Jianly Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. Bge m3-embedding: Multi-lingual, multi-functionality multi-granularing text embeddings through self-knowledge dis- tillation. arXiv preprint arXiv: 2024a.
Junying Chen, Zhenyang Cai, Ke Ji, Xidong Wang, Wanlong Liu, Rongsheng Wang, Jianye Hou, and Benyou Wang. Huatuogpt-ol, towards medical complex reasoning with 1lms. arXiv preprint arXiv:2412.18925 2024b.
Junying Chen, Chi Gui, Ruyi Ouyang, Anningzhe Gao, Shunian Chen, Guiming Hardy Chen, Xi- dong Wang, Ruifei Zhang, Zhenyang Cai, Ke Ji, et al. Huatuogpt-vision towards injecting medi- cal visual knowledge into multimodal llms at scale. arXiv preprint arXiv:2406. /9280, 2024c.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities. arXiv preprint arXiv:2507.06261, 2025.
Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre- Emmanuel Mazaré, Maria Lomeli, Lueas Hosseini, and Hervé Jégou. The faiss library. arXiv preprint arXiv:2401.0822 008288281,,,1,,, 2024.
Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma. Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, arXiv preprint arXiv:2501. 12948, 2025.
10

---

Published as a conference paper at ICLR 2026
Xuchai He, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie. Pathvqa: 30000+ questions for medical visual question answering. arXiv preprint arXiv:2003.10286, 2020.
Xinyue Hu et al. Medical-cxr-vqa dataset: A large-scale Ilm-enhanced medical dataset for visual question answering on chest x-ray images.
Yutao Hu, Tianbin Li, Quanfeng Lu, Wenqi Shao, Junjun He, Yu Qiao, and Ping Luo. Omnimedvqa: A new large-scale comprehensive evaluation benchmark for medical Ivlm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20170-21 221822 2024.
Xiaoke Huang, Juncheng Wu, Hui Liu, Xianfeng Tang, and Yuyin Zhou. ml: Unleash the po- tential of test-time scaling for medical reasoning with large language models. arXiv preprint (rXiv:2504.00860 2025a.
Xiaoke Huang, Juncheng Wu, Hui Liu, Xianfeng Tang, and Yuyin Zhou. Medvithinker: Simple baselines for multimodal medical reasoning. arXiv preprint arXiv:2508.02660 2025b.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint vrXiv:2410.21276 2024.
Gabriel Ilharco, Mitchell Wortsman, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannanch Hajishirzi, Ali Farhadi, et al. Openclip. Zenodo, 2021.
Aaron Jacch, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412. 16720, 2024.
Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushnen dataset of clinically generated visual questions and answers about radiology images. Scientific data, 5(1):1-10, 2018.
Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Nau- mann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training a large language-and as- sistant for biomedicine one day. Advances Neural Information Processing Systems, 36: 28541-28564 2023.
Tianbin Li, Yanzhou Su, Wei Li, Bin Fu, Zhe Chen, Ziyan Huang, Guoan Wang, Chenglong Ma, Ying Chen, Ming Hu, et al. Gmai-vl & gmai-vl-5.5 m: A large vision-language model and a comprehensive multimodal dataset towards general medical ai. arXiv preprint 2r 2024.
Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma. Yan Yang, and Xiao-Ming Wu. Slake: A semantically- labeled mnnneeee-ennhaneed dataset for medical visual question answering. In 2021 IEEE 18th international symposium on biomedical imaging (ISBI), pp. 1650-1654. IEEE, 2021.
Fenglin Liu, Tingting Zhu, Xian Wu, Bang Yang, Chenyu You, Chenyang Wang. Lei Lu, Zhangdai- hong Liu, Yefeng Zheng, Xu Sun, et al. A medical multimodal large language model for future pandemies. NPJ Digital Medicine, 6( 6(1):226, 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF conference computer vision and patlern recognition, pp. (6296-266366 2024.
Alejandro Lozano, Min Woo Sun, James Burgess, Liangyu Chen, Jeffrey J Nirschl, Jeffrey Gu, Ivan Lopez, Josiah Aklilu, Anita Rau, Austin Wolfgang Katzer, et al. Biomedica: An open biomedi- cal image-caption archive, dataset, and vision-language models derived from scientific literature. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 19724-19735, 2025.
Yizhen Luo, Jiahuan Zhang, Siqi Fan, Kai Yang, Yushuai Wu, Mu Qiao, and Zaiqing Nie. Biomedgpt: Open multimodal generative pre-trained transformer for biomedicine. arXiv preprint ar0iv:2308. 09442, 2023.
11

---

Published as a conference paper at ICLR 2026
Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Medmeqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Conference on health, inference, and learning, pp. 248-260. PMLR, 2022.
Richard J Roberts. Pubmed central: The genbank of the published literature, 2001.
Johannes Rückert, Louise Bloch, Raphael Brüngel, Ahmad Idrissi Yaghir, Henning Schäfer, Cyn- thia S Schmidt, Sven Koitka, Obioma Pelka, Asma Ben Abacha, Alba G. Seco de Herrera, et al. Rocov2:
Radiology objects in context version an updated multimodal image dataset. Scientific Data, 11(1):688, 2024.
Andrew Sellergren, Sahar Kazemzadeh, Tiam Jaroensri, Atilla Kiraly, Madeleine Traverse, Timo Kohlberger, Shawn Xu, Fayaz Jamil, Cían Hughes, Charles Lau, al. Medgemma technical report. arXiv preprint arXiv:2507.0520/ 2025.
Zhihong Shao, Peiyi Wang. Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint :xXi:22422.03000, 2024.
Yanzhou Su, Tianbin Li, Jiyao Liu, Chenglong Ma, Junzhi Ning, Cheng Tang, Sibo Ju, Jin Ye, Pengcheng Chen, Ming Hu, et al. Gmai-vl-rl: Harnessing reinforcement learning for multimodal medical reasoning. arXiv preprint arXiv:2504.0188 2025.
Sanjay Subramanian, Lucy Lu Wang, Sachin Mehta, Ben Bogin, Madeleine Van Zuylen, Sravanthi Parasa, Sameer Singh, Matt Gardner, and Hannaneh Hajishirzi. Medicat: dataset of medical images, captions, and textual references. arXiv preprint halh:2000006000 2020.
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv preprint arXiv:2503. 2025.
Tao Tu, Shekoofeh Azizi, Danny Driess, Mike Schaekermann, Mohamed Amin, Pi-Chuan Chang, Andrew Carroll, Charles Lau, Ryutaro Tanno, Ira Ktena, et al. Towards generalist biomedical ai. Nejm Ai, 16):A1oa2300138, 2024.
Siddharth Tumre, Sangameshwar Patil, and Alok Kumar. Improved near-duplicate detection for aggregated and paywalled news-feeds. In Proceedings of the 2025 Conference of the Nations of the Americas Chupter of the Association for Computational Linguistics: Human Language Technologies (Volume 3: Industry Track), pp. 979 9199997, 2025.
Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al. Internv13. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508. 18265, 2025.
Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. Towards generalist foundation model for radiology by leveraging web-scale 2d&3d medical data. arXiv preprint urXiv:2308.02463. 2023.
Juncheng Wu, Wenlong Deng, Xingxuan Li, Sheng Liu, Taomian Mi, Yifan Peng, Ziyang Xu, Yi Liu, Hyunjin Cho, Chang-In Choi, et al. Medreason: Eliciting factual medical reasoning steps in Ilms via knowledge graphs. arXiv preprint wrXiv:2504.0099) 2025.
Jin Ye, Guoan Wang. Yanjun Li, Zhongying Deng, Wei Li, Tianbin Li, Haodong Duan, Ziyan Huang, Yanzhou Su, Benyou Wang, et al. Gmai-mmbened A comprehensive multimodal evaluation benchmark towards general medical Advances Neural Information Processing Systems, 37: 94327-94427, 2024.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, al. Mmmu: A massive multi-discipline multi- modal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9556-9567, 2024.
12

---

Published as a conference paper at ICLR 2026
Aohan Zeng, Xin Lv, Qinkai Zheng, Zhenyu Hou, Bin Chen, Chengxing Xie, Cumxiang Wang, Da Yin, Hao Zeng, Jiajie Zhang, et al. Glm-4.5: Agentic, reasoning, and coding (are) foundation models. arXiv preprint arXhv:2508.0647/ 2025.
Sheng Zhang, Yanbo Xu, Naoto Usuyama, Hanwen Xu, Jaspreet Bagga, Robert Tinn, Sam Pre- ston, Rajesh Rao, Mu Wei, Naveen Valluri, et al. Biomedclip: multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs. arXiv preprint arXiv:2303.00915, 2023a.
Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Weixiong Lin, Ya Zhang, Yanfeng Wang, and Weidi Xie. Pmc-vqa: Visual instruction tuning for medical visual question answering. arXiv preprint arXiv:2305./0415. 2023b.
Juexiao Zhou, Xiaonan He, Liyuan Sun, Jiannan Xu, Xiuying Chen, Yuetan Chu, Longxi Zhou, Xingyu Liao, Bin Zhang, Shawn Afvari, et al. Pre-trained multimodal large language model enhances dermatological diagnosis using skingpt-4. Nature Communications, 15(1):5649, 2024.
Yuxin Zuo, Shang Qu, Yifei Li, Zhangren Chen, Xuekai Zhu, Ermo Hua, Kaiyan Zhang, Ning Ding, and Bowen Zhou. Medxpertqa: Benchmarking expert-level medical reasoning and understanding. Xixiv preprint arXiv:2501. 18362, 2025.
13

---

Published as a conference paper at ICLR 2026
APPENDIX
A
THE USE OF LARGE LANGUAGE MODELS (LLMS)
During the preparation of this manuscript, used OpenAl's GPT-5 model for minor language finement and smoothing of the writing. The Al tool was not used for generating original content, conducting data analysis, formulating scientific ideas. All conceptual development, experi- mentation, and interpretation were conducted independently without reliance on AI tools.
B REPRODUCIBILITY STATEMENT
We provide detailed instruction to reproduce our research. The prompts for generation and ver- ification are presented in Scction Section 3 provides instructions for initial data curation and filtering, question generation and verification, and our implementation details. Section 4.1 demon- strates experimental settings, including benchmarks and comparison baselines. All models are trained on 8 A100 GPU machines using mixed precision. The hyperparameters are following "Med- VLThinker" (Huang et al. (0255).. We full-stack release our data, code, and models the pursuit of open science
C ETHICAL CONSIDERATIONS AND LICENSING
MedVLSynther operates solely PMC-OA literature. We include source metadata and original licenses in dataset manifests and exclude items whose licenses are incompatible with redistribution for research. We provide attribution source articles in per-item metadata. MedVLSynther is intended exclusively for research; no clinical use authorized. We release prompts, rubric, and filtering logs to enable community auditing.
D MORE DATA STATISTICS
Table 7: Image secondary label from original Biomedica statistics.
Image Secondary Label
# Images
Ratio
Image Secondary Label
# Images
Ratio
clinical imaging
1039
7.17
intraoral imaging
393
2.71
brain
984
6.79
eye
379
2.62
x-ray radiography
879
6.07
electrocardiography
377
2.60
light microscopy
829
5.72
mammography
374
2.58
computerized tomography
785
5.42
patient photo
357
2.46
immunohissoceemisty
676
4.67
procedural image
335
2.31
magnetic resonance
673
4.65
laryngoscopy
314
2.17
surgical procedure
565
3.90
microscopy
65
0.45
skin lesion
563
3.89
scientific illustration
31
0.21
functional magnetic resonance
525
3.62
line plot
27
0.19
angiography
519
3.58
bar plot
23
0.16
intraoperative image
504
3.48
immunoblot
12 0.08
specimen
500
3.45
confocal microscopy
11 0.08
optical coherence tomography
492
3,40
aerial photography
2 0.01
ultrasound
486
3.35
table
2
0.01
endoscopy
477
3.29
tree
1
0.01
teeth
470
3.24
signal plot
0.01
skull
415
2.86
user interface
0.01
plot
400
2.76
ambiguous
0.01 1
https://ucac-viee iit.aa...gght........... thub. ic/MedVV..ynther/
14

---

Published as a conference paper at ICLR 2026
shhan (ianeN
20433
(a) MedSyn question token length distribution
(b) Stage-2 verification rubric part pass rate
120
300
(c) Owenn..-VV-Innsrruet 3B pass on MedSyn-
(d) Qwen2.5--L-IInsrruct 7B pass rate MedSyn-
VQA 5K suhset
VQA 5K subset
Figure 5: More data statisties about MedSynVQA
E PROMPTS
Generator Prompt
[SYSTEM ROLE]
You are expert medical-education item writer. Your job is to generate high-quality multiple-choice question (MCQ) from a biomedical figure (the image) and its accompanying paper caption/context. The MCQ must be self-con self-contained, clinically valid, and solvable by carefully inspecting the image together with general domain knowledge implicitly derivable from the caption/context: WITHOUT quoting the caption revealing the answer verbatim. INON-NEGOTIABLE RULES]
1) Do NOT write "according the cuppion/ddeccriitionnee""" or similar.
2) Do NOT copy answer text verbatim from the caption; paraphrase and compress.
3) Exactly ONE best answer. Distractors must be plausible and mutually exclusive with the key.
4) Use only information supported by the image and facts that a competent clinician could
infer from the caption/context; no speculative claims.
5) Keep clinical terminology precise; avoid brand names/PHI: no patient identifiers.
6) Output MUST follow the JSON schema below-no extra keys, commentary.
7) Do NOT include chain-of -ff--foootthttttt Keep rationales concise and factual is NOT required-
only output the JSON.
[RÜBRIC (Self-check before you output)]
Essential Criteria (must PASS)
BI. Stem is self-contained; no mention of "ccaption/descriptionn: no hidden assumptions.
E2. Image-content alignment: the question requires inspecting specific visual features.
15

---

Published as a conference paper at ICLR 2026
E3. Caption-derived facts integrated implicitly (paraphrased) but do not leak the
swer.
E4. Single correct option; remaining options incorrect for clear clinical reason.
E5. Medical correctness: terminology, anatomy, modality, and pathophysiology are accu- rate.
Important Criteria (strongly recommended)
III. Cognitive level: 2 application (identification, interpretation, next step, best explana-
tion).
12. Distractors: near-misses,, common confusions, or plausible alternatives not trivial.
13. Parallelism: options have similar length/structure; avoid "all/none of the above."
- 14. Difficulty labeled (Fasy/Moderatt//arr) with a brief justification. (Do NOT include this in the output JSON.)
-15. Balanced scope: focuses on one primary concept (finding, diagnosis, step, location).
Optional Criteria (nice have)
01. Localizes the key finding (e.g.. lobe/ssegment/organ subregion) if appropriate.
02. Uses quantitative details ssize/scale/rrade/sttee only when clearly supported.
[ALLOWED QUESTION ARCHETYPES] Pick ONE that best fits the image + derivable
facts:
Finding identification ("Which abnormality is present?")
Best diagnosis/ most likely explanation
Next best step (diagnostic management)
Localization ("Which structure/region is affected?")
Modality/sequence recognition (e.g., T1 T2, phase, stain)
[GENERATION WORKFLOW]
Step 1 (Privately) derive few concise, paraphrased facts from captions/contexts that clinician could reasonably infer.
Step 2 Choose an archetype so that both image inspection and those facts are needed. Step 3 Write self-contained stem that integrates the derived facts implicitly (no mention of "caation/(escripiionnee"" and does NOT reveal the answer.
Step 4 Author 5 options (A-E): one correct, four high-quality distractors (near- miss/opposite/irrlleeant busppppsseeeiilllaaaauu..aaaaaaa
Keep options parallel.
Step 5 Run the RUBRIC. If any Essential item fails, regenerate (intemnalyy.. Output only JSON.
[OUTPUT FORMAT STRICT JSON]
Return exactly one JSON object with keys:
""question": string
""options": object with keys "A""""""""""""""""""""""""""""""
"annwer": one of "C""""""""""""""""""""""""""""""
[SOURCE MATERIAL]
CAPTIONS: [CAPTION BLOCK)
CONTEXTS: (CONTEXT.BLOOO
Think silently. Output ONLY the JSON object.
Verifier Prompt
[SYSTEM ROLE]
You are a biomedical MCQ verifier with two distinct roles, executed in order:
1. **The Referee (for Essential Criteria):** You are an objective, rule-based official. Your job is fairly determine if the absolute minimum standards for usability met. #*The Critic (for Bonus Criteria) ** After the essential check, you become relentless perfectionist. Your default assumption is that the MCQ is NOT excellent. Your goal is to deny bonus points unless perfection demonstrated beyond any doubt. Judge each MCQ "only" using
[DATASET POLICY MUST ENFORCE]
Stem is self-contained and MUST NOT say "caption" or "context".
16

---

Published as a conference paper at ICLR 2026
Paraphrasing allowed, but DO NOT introduce unsupported clinical facts (age/sex/histo- wyllocation/indigss/digggggoo beyond CAPTION/CONTEXT or clearly visible image cues.
CAPTION names diagnosis. do NOT restate that diagnosis verbatim the stem.
Exactly one correct option; all options must be the same semantic type.
Clinical correctness poooaliyyysainlannnoyyyaaattpppp htw.ap/wwwwaiiiitooooooooooooooo
must supported by
sources.
ISCORING MODEL] Output ONLY "rubric":[...)
Each item has:
-idx (1..N), title, deseription (ONE sentence starting with "Easemial/mmprrran//pttoon////t
fall Stiteria:....).
category: Essential [Important [Optional |Pitfall,
weight: Essential=5; Important in 3,4; Optional in 1,2; Pitfall in -1,-2,
score: Dusseniil//mpoortt/////pp/
in O, weight; Pitfall in O, weight (0 not triggered), notes: - 12 words (brief reason; no chain-of-thought)..
**coring Mindset:**
**Esenttial Items;** ** Award full points the rule met. Be fair and impartial referee. "Important/Optional Items (Bonus Points):** **These are bonus points, not entitle- ments." ** The score **0 by default. ** You must find **irrefutable evidence of perfec- tion** to award points. If there is *any* subjective room for improvement, the score remains
[FIEDD ESSENTIAL ITEMS- MUST INCLUDE with EXACT titles]
1) Stem Self-contained
2) Vocabulary Constraint
3) Diagnosis Leak
4) Single Correct Option
5) Option Type Consistency
6) Clinical Validity
7) Image-Text Consistency
[BONUS CRITERIA FOR EXCELLENCE ZERO-TOLERANCE JUDGEMENT]
(You must assess against a diverse set of 4-8 items. For this section, your mindset is "guilty until proven innocent." ** single, minor flaw any sub-point means an instant score for the entire item. *#)
Plausible Distractors (Important, 3-4)
**Every single** distractor must be strong, clinically relevant alternative given the sources.
Each must differ from the key by exactly ONE clear axis.
**If you can imagine a slightly more plausible distractor that wasn't used, score 0. There must be no weak links.
Paaaalel Options (Important, 3)
Grammatical structure, length, and specificity must be ** **rigorously uniform**
**y noticeable outlier** in form (e.g., one starts with a verb, others with nouns) or length fails this. No unique cues the key.
-Stem Concision (Optional, 1-2)
SStem must be < sentences AND < 35 words.
If the stem can be rephrased to be even **slightly more elegant direct** without losing critical meaning. **score *
Clarity and Focus (Optional, 2)
The stem poses a single, perfectly unambiguous question.
If the question could be worded **any more clearly or is even slightly awkward* ** **score *
Answer Field Validity (Important, 3)
'answer' exists, is one of A-E, and exactly matches an option key.
No duplicate options; all option strings non-empty.
-JSON Schema Compliance (Important, 3)
MCQ has exactly required keys question, optionsA..E., answer; no extras/missing.
-Fobiiddnn Terms (Pitfall, -2)
Stem contains the exact word 'caption' or 'context'.
17

---

Published as a conference paper at ICLR 2026
Synonym Drift (Pitfall, -1)
Stem introduces a **specific** clinical fact that is absent from sources and not visible in the image.
Multiple Keys (Pitfall, -2)
>> option reasonably correct ** given sources -Medical Inaceuracy (Pitfall, -2)
Any statement directly contradicts sources.
[HOW TO JUDGEJ
For Essentials, be referee. For the Bonus section, be rival looking for a weakness.
Use only ([CURES)CCAPTOONCCOTEEY
an MCQ asserts anything unsupported,
fail the relevant item.
[OUTPUT FORMAT - STRICT JSON ONLY]
Return exactly: ("rubric": (("idx":1," ""ttlet"""" """""""."""""""""""""""""""""""" """description""""" "EEsenttal Criteria: ".."" "Esset"Essential",, ""esegn "5o,,""c""""d or :""notes":".."" "...""....]]
No totals. No extra keys, No commentary. [INPUTS]
IMAGES: <attach in order
MCQ: (MCQ JSON}
CAPTION: (CAPTION BLOCK)
CONTEXT: (CONTEXT_BLOCK,
[CONDUCT]
If inputs are insufficient to judge, output ("error":" iaapoeoiiinnffiiinteeeeeeeeeee ONLY. Think silently. Output ONLY the ISON object above.
F DATA CONTAMINATION ANALYSIS
PMC-VGA MRI abr ormalitiss
Dataset: VQA-RAD
maaging modality desoribed
content?
Optiona: A. No
Options:
Options:
A. Magnetic Reconance Imaging
temporal lobe occipital tbee mombil lobe
Yes
B. Computec Tamography
G. Piostron Emesion Tomagraphy
Answer: A. Vee
D. D.X.Ray
pariotal
Tomography
Dataset: MedSynVVGA
Dataset: MedSynVVA
Dataset: MedSynVVA
imaging
surrc region provided image?
madbiity imaga (x)?
MRU?
Options:
Magnelic resomaneti imaeing
Optiona:
Options:
B. Computed tomography
Froncal labe
Lef interol caroti artery
C. Xeay
"Tampponal lobe
a. fiight mebdle cersbeal artery A/1 segmant
D. Posizon emission
Farietal kobe Insuiay Oco pitipitalln
Ci Left modee central artey MI' segmant sotwvor cemebill intery
tomography
Utresound
cendraal arteyy
Oasipiall
antery
MI
Figure 6: Examples from our contamination analysis across MedSyn VQA and our testset. On the left, this case shows that top- text-embedding neighbor shows similar options but entirely different images. In the middle, find exact duplicates by MD5; pHash retrieves same-source size variant with different question/answers. On the right, image-embedding neighbor reflects modality clustering (high similarity) without instance identity; questions also differ.
Text decontamination. We audit overlap between the training pool (13,087 rows) and the held-out test set (8,220 rows) over the fields question and options, after lowercasing, whitespace cleanup, digit masking (all numbers to "<NUM>"), and normalizing options "A./B./..". 1) MinHash Levenshtein hashes 3-character n-grams and confirms candidates with normalized edit similarity (T 0.90, following Pal al. 2002 This conservative pass finds near-duplicate strings
18

---

Published as a conference paper at ICLR 2026
(pairs=0; hit queries=0; hit rate=0.0). 2) Embedding FAISS (Douze et al. 2024 encodes text with BAAl/bge-m3 (Chen et al. 2024a) n.2-morma,,zed), retrieves top-k=5 train neighbors per test item, and flags pairs with cosine larger then 0.88 (Tumre et al., 2025). This yields 47 pairs across 23 test queries (hit rate about 0.280% with per-query hits mean=2.0 (max=5). The top-1 similarity for those 23 queries averages 0.899 (median 0.894; p95=0.926), and qualifying pair similarities concentrate in [0.88, 0.95). Considering all test items, the best-neighbor ("MaxSim") distribution mean 0.651 (median 0.644; p95=0.778) 778), and Overlap@0. 0.88 23/8,220 0.280%, indicating minimal distributional leakage beyond trivial lexical matches. We found no overlap of texts.
Image decontamination. We repeat the two-pass audit on images decoded directly from the dataset (using the first image per record by default). 1) A hashing pipeline detects exact and near duplicates via MD5 of raw pixels and 64-bit perceptual hash (pHash); pHash neighbors are re- trieved with binary FAISS index, and we summarize each test image's minimum Hamming dis- tance and "Overlap@d" (fraction with best neighbor less than or equal to "d"") 2) semantic pipeline encodes images with OpenCLIP or BiomedCLIP (Ilharco et al.] 2021 Zhang et al., 2023a), searches the train set with FAISS (Douze et al., (024) (inner product on L2 12-normalized features), and reports per-test MaxSim, Overlap@ and histogram of high-similarity test-train pairs (e.g.. at r = 0.88). Together, hashing probes pixel-level or layout-level duplication while embeddings probe semantic or style reuse; all statistics are computed from the images stored in parquet under this seaele-imaa---ee--eeon protocol. We found no overlap of images.
G EXPERT EVALUATION STATISTICS
To complement the downstream results and decontamination analysis, we conducted a small-scale human evaluation of MedSynVQA questions to assess intrinsic item quality and the behavior of the verifier. Five board-certified imaging experts participated, with clinical backgrounds covering oncologic CT/MR, abdominal and gastrointestinal CT/MR, brain CT/MR, diagnostic radiology, and gastrointestinal endoscopy. Some questions were jointly reviewed by more than one expert when they spanned multiple organs or modalities.
G.1 VERIFIER-ACCEPTIE ITEMS
We first assessed the precision of the verifier on items it accepts. We randomly sampled 15 multiple- choice questions (MCQs) that had been accepted under the main verification threshold. Each item was rated on four 1 to 4 Likert scales (4 excellent, 1 poor): medical correctness and uniqueness of the keyed answer, clarity and professional wording, image grounding, and option design. In addition, experts provided binary judgment whether the item was acceptable for use medical MCQ.
Table summarizes the aggregate statistics, and 10 reports per-item ratings. 14 of the 15 items (about 93% )) were judged acceptable overall; one item was rejected because the brain MR image had such poor contrast that the expert considered the image non-evaluable, independent of the ques- tion text. The mean scores across criteria were 3.35 for medical correctness, 3.13 for clarity and professional wording, 3.46 for image grounding, and 3.40 for option design, corresponding to good to excellent quality on average. Two items received score of 1 on the clarity and wording scale, indicating that while their medical content and image use were sound, the phrasing could be sub- stantially improved. This consistent with our finding that the current vocabulary related checks in the verifier do not fully capture wording quality and suggests that prompt refinement for clarity is promising direction.
G.2 VERIFIER-EIEECEE ITEMS
We then examined whether the verifier is justified when rejecting items. We randomly sampled 20 MCQs that had been filtered out by the verifier and, for each item, selected one essential rubric that had received zero score. The audited rubrics and their frequencies, along with human agreement, are summarized in Table Table [T provides per-item details.
Across the 20 rejected items, the audited essentials were: stem self- containdd (5 items), vocabulary constraint (4), diagnosis leak (I), single correct option (2), option type consistency (4), clinical valid- ity (2), and image-text consistency (2). For rubric types that do not require strong medical expertise
19

---

Published as a conference paper at ICLR 2026
(stem self-contained, vocabulary constraint, option type consistency). the authors first inspected the items. Stems marked as non self-contained were indeed unusable without external context, for ex- ample explicitly referring "according to the context" rather than being answerable from the stem and image alone. Option type consistency failures involved heterogeneous option formats or options literally written "none" (for example, option E is just "none" an empty placeholder, rather than a meaningful choice such as "no abnormality" or "none of the above"), so these rejections were appropriate.
For the vocabulary constraint rubric, we identified a systematic issue. In the detailed cases examined, the verifier incorrectly flagged sex, age, or clinical history as "unsupported." even though these attributes were present in the caption or context. These are genuine false negatives: the current rubric prompt does not reliably capture linguistic professionalism or terminology use and can misfire when the model fails to correctly read the context. We therefore regard vocabulary constraint as weaker signal and a primary target for future prompt and rubric refinement.
For the medically heavy essentials (diagnosis leak, single correct option, clinical validity, and image- text consistency), items were evaluated by clinicians. In all inspected cases, experts agreed with the verifier. Items rejected for diagnosis leak did reveal the diagnosis directly in the stem; items re- jected for single correct option or clinical validity had multiple plausible keys or medically incorrect statements; items rejected for image-text consistency did not match the visual evidence.
G.3 SUMMARY
Although this human study is necessarily small due to limited clinician time and the difficulty of re- viewing full image-based MCQs, it already involves five specialists and includes items that required cross-domain input. The audit shows that a large majority of verifier- medically sound and usable, and that the essential medical rubrics driving rejection (diagnosis leak, single correct option, clinical validity, image-text consistency) align well with clinician judgment. The main systematic discrepancy is concentrated in the vocabulary-related check, which we now explicitly highlight a limitation. Overall, Tables I provide intrinsic evidence about the liability and failure modes of the verification stage and complement the downstream metrics and decontamination analyses in the main paper.
items are judged
Table 8: Summary of expert evaluation on questions.
Subset
Items
Experts
Human
Mean medical conrectness
Mean clarity
Mean image grounding
Mean option design
Verifier accepped MCOs
15,00
5.00
14/ (93%)
3.35
3.13
3.46
3,40
Verifser-rejecie rejected MCQs
Table Distribution of audited essential rubrics for rejected items.
Essential rubric
Human agreement verifier
Stem self-contained
Yes (all)
Stems depended on external context, not answerable from stemeimage
Vocabulary
cases)
Verifier wrongly flagged sexlagehisstory unsupported
Diagnosis leak
Yes (ull)
Diagnesis explicilly revealed in stem
Single correct option
Yes (all)
Multiple plausible keys amhiguuus
Option type consistency
(all)
Heteragenenus option formats or meaningless "none" placeholder
Clinical validity
Yes (all)
Medically incervect inconsistent statements
Imago-test text consistency
(all)
Text description not match the visual findings
Table 10: Per-item expert ratings for verifier-accepted MCQs.
specialty
accoptaNee
Carity sonting Imago grording (1-4) Option design
(ngticeal)
A0I
Siagnostic indlog
Yes
A02
diagnonii tadieoggs
Yes
AB5
diagnostic filllggg
Yos
diagnostic indillogs
A16
diagnoatc medielgg
AM7
peneittraira andoorggy
Ye
A18
aasocittsiinl zedoicons
Yes
assosiatesind andoscops
pensintocinal
CTMM
A12
bean CT/ER
NAA
contast images
A15
lean CTAEE
AlL
beai CTMR
Yo5
20

---

Published as a conference paper at ICLR 2026
Table 11: Per-item audit for verifier-rejected MCQs.
Item
Reviewer specially
Essential rubric checked
Human agrees with rejection?
B01
author self-review author self review author self-review
Stem self-contained
Yes
B02
Stem self-cuntained
Yes
B03
Stem self-contained
Yes
B04
author self-review author self-reviow
Stem self-contained
Yes
B05
Stem self-contained
Yes
B06
author self-review author self-review
vocahulary constraint
Yos
B07
vocabulary
No
B08
author self-review
vocabulary construint vocabulary construint
No
B09
author self-review
No
B10
oncologic CT/MR, diagnostic radiology
diagnosis Icak single correct option single correct option option type consi option lype consistency option type consistency option type consistency
Yes
B11
oncologic CT/MR
Yes
B12
gastrointestinal CT/MR, gastrointestinal endoscopy
Yes
B13
author self-review
Yes
B14
author self-review
Yes
B15
author self-review
Yes
B16
author self-review
Yes
B17
gastrointestinal CT/MR, gastrointestinal endoscopy
clinical validity clinical validity image-text consistency image-texi consistency
Yes
B18
gastrointestinal CT/MR, gastrointestinal endoscopy
Yes
B19
gastmintestinal CT/MR, gastrintesttial endoscopy
Yes
B20
brain CT/MR
Yes
H CASE STUDY OF SYNTHETIC DATA
Essential Criteria: Stem Self-contained
Generated Question: Which finding is
most consistent with the severe osteolysis
described in the context?
Generated Choices: (A) Intact ischial ra-
mus on AM model (B) Complete destruc-
tion of ischial ramus (C) Stable acetabular prosthesis (D) Minimal bone loss around
acetabulum (E) Intact pubic ramus AM model
Generated Correct Answer: (B)
Verifier Key Output:
"idx": 1, "title": "Stem Self-contained", "description": "Essential Criteria: Stem does not rely on caption or context.", "category":: "Essential", "weight": 5, "score": 0, "notes":
"Stem references 'context', violating self-containment.)
Analysis:
This item should fail with score 0 on Stem Self- celf-contained. The phrase in the context makes the answer contingent on external text; without that context, the solver cannot infer that severe osteolysis corresponds to complete destruction of the ischial ramus, option B is not reuchable from the stem alone. The abbreviation AM model is also undefined in the stem, further harming self-containment.
21

---

Published as a conference paper at ICLR 2026
Essential Criteria: Vocabulary Constraint
Caption: six-year-old girl with right side neglected modified Severin classifica- tion type IV developmental dysplasia of hip.
Context: 6-year-old girl was admitted to outpatient clinic with complaints of limping June 2009...
Generated Question: Which abnormality is most consistent with the radiographic findings in this six-year-old girl?
Generated Choices: (A) Right hip sub- luxation with acetabular dysplasia (B) Left hip dislocation with femoral head avascu- lar necrosis (C) Bilateral femoral neck frac- tures (D) Pelvic osteomyelitis (E) Sacroil- iac joint ankylosis
Generated Correct Answer: (A)
Verifier Key Output:
"iidx": 2, "title": "Vocabulary Constraint", "description": "Essential Criteria: No unsup-
ported clinical facts introduced.", "category": "Essential", "weight": 5, "score": 0, "notes": "Stem introduces age and visible in image." }.
Analysis:
This item should receive score on Vocabulary Constraint. The stem introduces age and sex as visible facts, while the evaluation setling treats the image as the sole evidence; these demographics are not directly verifiable from the image and are not required to answer Including them constitutes unsupported clinical facts and also injects bias by priming pedi- atric diagnoses such as developmental dysplasia of the hip, reducing the emphasis visual reasoning. This one relatively acceptable example for vocabulary constraint rejection.
Essential Criteria: Diagnosis Leak
Caption: 37-yyer-olld woman with a pal- pable mass on the upper outer quadrant of the right breast. (a) Cranio-caudal mam- mogram. (b) Mediolateral Oblique mam- mogram. Arrows in (a and b) mark the sion. The triangular skin marker indicating the location of the palpable mass is visi- ble in (b). (c-e) Ultrasound imaging show a cireumscrihed 1.6 x 1.8 x 1.5 cm hypoe- choic mass with flow and post enhancement at 1l o' clock position and 10 em from nip- ple corresponding
in mammo- grams. (f) Ultrasound image from biopsy procedure showing the needle. (g) invasive ductal carcinoma, grade 1, showing infiltrating malignant glands within desmoplastic stroma (H&E stain). (h) high-power magnification showing predominant tubular formation (ar- rows), mild nuclear atypia with rare/no mitosis (H&E stain). (i) ER immunostain showing strong diffuse positivity with anti-ER antibody.
the
Context: Tumor differentiation graded
ordinal scale (3: doryy-differentiatee, 2: moderately-differenttate and 1: well-differentiatee)) was positively correlated with US measured tumor size...
Generated Question: A 37-year-old woman presents with palpable mass in the upper outer quadrant of the right breast. Imaging and biopsy reveal circumscribed hypoechoic
22

---

Published as a conference paper at ICLR 2026
with flow, invasive ductal carcinoma, grade 1, with predominant tubular formation, mild nuclear atypia, and strong diffuse ER positivity. Which histopathological feature is consistent with the tumor grade?
Generated Choices: (A) Poorly differentiated with marked nuclear atypia and frequent mi- toses (B) Moderately differentiated with solid nests and moderate nuclear atypia (C) Well- differentiated with predominant tubular formation and mild nuclear atypia (D) In situ carci- noma with comedonecrosis (E) Metastatic carcinoma with signet ring cells
Generated Correct Answer: (C)
Verifier Key Output:
The stem does not restate the diagnosis verbatim.", "category": "Essential", "weight": 5, "score": 0, "notes":
"iixx: 3, "title": "Diagnosis Leak", "description": "Essential Criteria:
"Stem restates diagnosis from caption."
Analysis:
This item exhibits clear diagnosis leakage. The stem reproduces the caption's 's diagnostic content verbatim, namely invasive ductal carcinoma, grade 1, predominant rubular forma- tion. mild nuclear atypia, and strong ER positivity. Option C mirrors these phrases, so question only baseline can answer without using the image, producing direct stem to option match. This dual overlap, caption to stem and stem to option, invalidates the assess- ment.
Essential Criteria: Single Correct Option
Caption: Panoramic view after implant in- sertion.
Context: None
Generated Question: Which of the fol- lowing best describes the radiographic ap- pearance of the dental implants in this panoramic view?
Generated Choices: (A) All implants are fully osseointegrated with no peri-implant radiolucency (B) Multiple implants show peri-implant radiolucency indicating early failure (C) Implants positioned in the maxillary arch (D) Implants surrounded by dense cortical bone (E) Implants exhibit significant angulation errors
Generated Correct Answer: (B)
Verifier Key Output:
"idxx" "title": "Single Correct Option", "description": "Essential Criteria: Only one op- tion is correct based on sources.", "category"": "Essential", "weight": 5, "score": 0, "notes": "Option is not clearly supported by image." }.
Analysis:
With panoramic rudiograph of limited resolution, option B asserts both the presence of peri-implant radiolucency around multiple implants and the clinical inference of early fail- ure. The first claim is not clearly visible this single image, and the second cannot be concluded from imaging alone. Option A, "no peri-implant radiolucency" is likewise non- falsifiable here and may seem plausible if details are indistinct, creating an A vs B ambiguity. Options D and E are subjective without quantitative thresholds, and while C is clearty in- correct, its presence does not make any remaining option uniquely correct. The net effect is that more than one answer can be defended from the same image.
23

---

Published as a conference paper at ICLR 2026
Essential Criteria: Option Type Consistency
Caption: Scoliosis series and kyphosis series images of the patient. The scoliosis anteroposterior view im- age (2A) shows the patient's trunk flexed to the left side by 12.31 degrees; the kyphosis series lateral image (2B) shows anterior flexion by 16.18 degrees of the patient's s trunk.
Context: None
Generated Question: Which measurement represents the degree of trunk flexion in the lateral view?
Generated Choices: (A) 12.31 degrees (B) 16.18 de-
grees (C) Anterior flexion (D) Left-side flexion (E) Trunk
flexion angle
Generated Correct Answer: (B)
Verifier Key Output:
('idx": 5, "title": "Option Type Consistency", "description": "Essential Criteria: All op- tions are the same semantic type.", "category": "Essential", "weight": 5, "score": 0, "notes": "Options mix numeric values and deseriptive terms.")
Analysis:
The stem asks for "which measurement represents the degree of trunk flexion in the lateral view." the expected answer type numeric value. The options, however, mix numbers and descriptive phrases: A and B are numeric, whereas C, D, and E are textual descriptors. This type mismatch violates the requirement that all options be of the same semantic class and also weakens alignment with the stem't 's focus on a measurement.
Essential Criteria: Clinical Validity
Caption: 41-year-old male with cervi- cal tuberculosis suffered from neck pain, spastic quadriparesis, and dysphagia due to retropharyngeal abscess. Neutral lateral cervical spine radiograph (A) indicates cer- vical kyphosis of 51 degree at C3-6, and C4 the apical vertebra. The kyphosis Cobb angle was 54 dgree and 49 degree on flexion and extension, respectively (B, C), which indicated that the flexibility was only 3.9%. MRI showed the formation of retropha- ryngeal abscess and ventral compression of the spinal cord. The Cobb angle was 21 de-
gree on MRI, which indicated that the flex- ibility was 58.8% (D). The x-ray and CT
scans the final follow-up showed that the correction was well maintained without sig-
nificant loss, and solid bone fusion was finally achieved. Context: ...Radioloiical examina- tions (cervical X-ray, MRI, and CT) were performed to assess the bone fusion state, flexibil- ity and correction of kyphosis. The flcxibility of kyphosis was evaluated on both dynamic X-rays and cervical MRI
Generated Question: Which imaging modality best demonstrates the flexibility of cervical kyphosis in this patient?
Generated Choices: (A) Neutral lateral cervical spine radiograph (B) Flexion lateral cervi-
cal spine radiograph (C) Extension lateral cervical spine radiograph (D) MRI (E) CT scan
Generated Correct Answer: (D)
Verifier Key Oufput:
24

---

Published as a conference paper at ICLR 2026
(iixx: 6, "title": "Clinical Validity", "description": "Essential Criteria: Clinical correct- ness supported by sources.", "category": "Essential", "weight": 5, "score": 0, "notes"; "MRI does not best demonstrate flexibility." }.
Analysis:
The question asks which modality best demonstrates the flexibility of cervical kyphosis, but standard assessment of flexibility is by dynamic flexion and extension lateral radiographs, which directly show angle change. MRI is primarily for soft-tissue and cord compression assessment, is usually obtained supine, and is not dynamic study, so it is not the best modality to demonstrate flexibility, Although the case text reports percentage derived from the difference between the neutral X-ray angle and an MRI angle, that does not make MRI the best demonstration of flexibility.
25

---

Published as a conference paper at ICLR 2026
1 CASE STUDY OF MODEL OUTPUTS
MedVLSynther-3B3 5K output for MMMU-medical
Question: In the diagram below, the wave represents (). the QRS segment rep- resents () and the wave represents () Choices: (A) atrial depolarization, ventric- ular depolarization, ventricular repolariza- tion
(B) ventricular repolarization, atrial depo- larization, ventricular depolarization
(C) ventricular depolarization, ventricular repolarization, atrial depolarization
Correct Answer: (A)
Model Answer: (A)
Model Response:
<think> The P wave is typically associated with atrial depolarization, which is the ini- tial depolarization of the atria. The QRS segment represents ventricular depolariza- tion, which is the depolarization of the ven- tricles. The T wave is associated with ven- tricular repolarization, which is the process
S
of the ventricles returning their resting state after depolarization. Therefore, the correct sequence is P (atrial depolarization), QRS (ventricular depolarization), and T (ventricular re- polarization). </think> <anserr.A. atrial depolarization, ventricular depolarization, ven- tricular repolarization<</npe </answer>
Analysis:
The model correctly matches the wave, QRS complex, and T wave with their respective physiological events: P wave for atrial depolarization, QRS complex for ventricular de- polarization, and wave for ventricular repolarization. However, its explanation could be enhanced by linking the morphology of these waves their function. For instance, could mention the small, rounded P wave representing atrial depolarization, the narrow, tall QRS complex signifying rapid ventricular depolarization (with atrial repolarization buried within), and the broader T wave for ventricular repolarization. Additionally, the model could have clarified the terminology, specifically that the "QRS complex" the correct term, not the "QRS segment." Furthermore, while the explanation is correct, it would have been helpful briefly exclude options B and C, which incorrectly describe the sequence of depolarization and repolarization.
26

---

Published as a conference paper at ICLR 2026
MedVL.Synther-3B 5K output for MedXpert-QA
When performing tension band fixation for olecranon fracture, illus- trated in Figure A, which nerve is most vul- nerable to injury if the Kirchner wire pene- trates too deeply through the anterior cortex of the proximal ulna? Choices: (A) Median nerve
Question:
(B) Ulnar nerve
(C) Posterior interosscous nerve
(D) Radial nerve
(E) Anterior interosseous nerve
Correct Answer: (E)
Model Answer: (E)
Model Response:
Athink>Olecranne fractures are located at the elbow joint, and the ulna is the bone that forms the elbow joint with the radius. The anterior cortex of the proximal ulna is where the Kirchner wire is typically placed
for tension band fixation. The anterior interosseous nerve runs along the anterior surface of the ulna, just below the anterior cortex. If the Kirchner wire penetrates too deeply through the anterior cortex, it could potentially injure the anterior interosseous nerve, which is the vulnerable nerve in this location. <<tbink>><nswer>>> Anterior interosseous nerve. </answer>
Analysis:
The model demonstrates a solid grasp of surgical anatomy and spatial reasoning by cor- rectly identifying the anterior interosseous nerve (AIN) the nerve most at risk when the Kirschner wire penetrates too deeply through the anterior cortex of the proximal ulna during olecranon crssion-bnd fixatiio.. This shows that the model not only recalls factual anatomy but also understands the directional logic of the operative field field-the thht danger arises from an- terior cortical breach rather than posterior or medial misplacement. While ils explanation slightly simplifies the anatomy, the overall reasoning is sound. The AIN indeed originates from the median nerve, passes deep to the pronator teres, and descends along the anterior interosseous membrane within the deep volar driven wire could encroach upon. By contrast, the posterior interosseous nerve lies pos- terolaterally around the radial neck, the ulnar nerve remains protected posterior-medially within the cubital tunnel, and the median nerve trunk more superficial and radial. These distinctions support the model's 's conclusion.
cettaartenntt-recciiseee the region a deeply
27

---

Published as a conference paper at ICLR 2026
MedVL.Synther-3B 5K output for Path VQA
does microscopy show branch- ing papillae having flbrovascular stalk cov- ered by single layer of cuboidal cells hav-
Question:
ing ground-glass nuclei? Choices: (A) Yes
(B) No
Correct Answer: (A)
Model Answer: (A)
Model Response:
<think> The question describes a histolog-
ical feature that is characteristic of papillary thyroid carcinoma. Papillary thyroid carcinoma is type of thyroid cancer that typically presents with branching papillae, which are struc- tures that resemble branching tree-like structures. These papillae are covered by a single layer of cuboidal cells that have ground-glass nuclei. The presence of these features is consistent with the histological appearance of papillary thyroid carcinoma, which often diagnosed through microscopic examination. <atinr>><asswe>>AAA Yes. </answer>>
Analysis:
The model accurately identifies the defining microscopic features of papillary thyroid car- cinoma (PTC)-branching papillae with fibrovascular cores, lined by single layer of cuboidal cells showing ground-ge glass ("Orphan Annie eye" ") nuclei-and correctly concludes "Yes." " This demonstrates solid pattern recognition and understanding of the diagnostic mor- phology. While its reasoning leans toward a textbook summary rather than slide-specific de- scription, the conceptual mapping is on point. It could be improved by explicitly referencing visible features such as the fibrovascular stalks, nuclear grooves, and intranuclear pseudoin- clusions, as well as by differentiating from benign papillary hyperplasia or follicular variant PTC. Nonetheless, the model's s response shows a commendable ability to link structural, cy- tologic, and diagnostic hallmarks coherently- an impressive level of interpretive accuracy for 3B-scale pathology model.
28

---

Published as a conference paper at ICLR 2026
MedVLLSynther-3B 5K output for PMC-VQA
Which region of the brain is con- nected with the habenula that is highlighted
Question:
in green? Choices: (A) Basal forebrain
(B) Periaqueductal gray
(C) Medial prefrontal cortex
(D) Raphe nuclei
Correct Answer: (C)
Model Answer: (C)
Model Response:
<think The image shows brain diagram
with various regions labeled. The habenula is highlighted in green, and it is connected to several other regions. The green label next to the habenula points the "medial pre- frontal cortex," which is one of the areas associated with the habenula. This suggests that medial prefrontal cortex region that functionally associated with the habenula. </think><<nswe>> C. Medial prefrontal cortex. </answer>
Analysis:
The model correctly selects the medial prefrontal cortex (mPFC) as the region function- ally associated with the habenula, demonstrating that it interpret the visual of the green-highlighted region and map it to the correct brain area. This shows sound grasp of color-coded neuroanatomical representations and context-based reasoning. However, its explanation remains somewhat superficial: it does not reference the figure legend, which clarifies that green indicates functional association while blue denotes direct anatomical connectivity, nor does it rule out the other options (basal forebrain, periaqueductal gray, raphe nuclei), which are blue rather than green in the diagram. Despite this, the reasoning puthway-linkee color cue to functional correlation- is accurate and efficient. Overall, the model demonstrates solid spatial comprehension and context mapping, with room for improvement in precise figure interpretation and differential exclusion.
29

---

Published as a conference paper at ICLR 2026
MedVL.Synther-3B 5K output for SLAKE
Question: Does the picture contain lung?
Choices: (A) Yes
(B) No
Correct Answer: (A)
Model Answer: (A)
Model Response:
<think> The image provided a cross- sectional view of human body, likely CT scan. The visible structures include bones, soft tissues, and air-filled spaces. The air- filled spaces are characteristic of the lungs, which are responsible for gas exchange in the body. Therefore, the image does contain lung structures. </think>> A Yes. </answer>
Analysis:
The model correctly recognizes that the im- age contains lung parenchyma, identifying the tow-attenuation, air-filled regions as pulmonary structures and noting the axial (cross- sectional) orientation the scan. This indicates a proper understanding of how lung fields appear on CT-dark, symmetric areas surrounding the mediastinum. However, the explana- tion remains somewhat generic: does not explicitly anchor its reasoning to clear anatomic landmarks such as the trachea, bronchi, ribs, or vertebral body, nor does it exclude other po- tential air-filled structures like bowel loops. stronger rationale would mention the bilateral distribution of aerated areas within the thoracic cavity, bordered by ribs and mediastinum, confirming their identity as lungs rather than abdominal gas. Overall, the model shows correct interpretation and solid conceptual grounding, though finer spatial referencing and exclusion reasoning would further strengthen its diagnostic precision.
30

---

Published as a conference paper at ICLR 2026
MedVL.Synther-3B 5K output for VQA-RAD
Question: there a small bowel obstruc-
tion? Choices: (A) Yes
(B) No
Correct Answer: (A)
Model Answer: (A)
Model Response:
<think>>A A small bowel obstruction typi- cally presents with signs of bowel dilation and fluid accumulation. In this CT scan, the small bowel loops appear dilated and there is a significant amount of fluid accumulation, which are characteristic signs of small bowel obstruction. The presence of these findings strongly suggests the presence of small bowel obstruction.
Yes. </answer>
Analysis:
The model correctly identifies small bowel obstruction (SBO) by recognizing the presence of bowel dilation and fluid accumulation, demonstrating solid grasp of the key radiologic pattern. Its reasoning is directionally accurate and clinically valid, but somewhat general- ized-it does not explicitly describe the hallmark CT features that confirm SBO. A stronger explanation would reference multiple centrally located dilated small-bowel loops (23 em in diameter), numerous air-fluid levels, and possibly the small-bowel feces sign, contrasted with a collapsed distal colon. It would also mention the need to identify transition point to confirm mechanical obstruction and distinguish from paralytic ileus or large-bowel obstruction. Overall. the model's ss answer is correct and displays good diagnostic reason- ing, though could reach expert-level precision with more specific anatomic anchoring and differential exclusion.
31

---

Published as a conference paper at ICLR 2026
MedVL.Synther-7B 13K output for MMMU-medical
This line of myelinated axons in layer IV of visual cortex represents the axons of cells in the Choices: (A) Superior colliculus.
Question:
Axon
(B) Lateral geniculate.
Tyrosine
(C) Retina.
(D) Medial geniculate.
L-DOPA
NE
Correct Answer: (B)
Model Answer: (B)
NE PNMT
Model Response:
Ca
<think> The image shows cross-section of the brain, specifically the visual cortex, which is part of the cerebral cortex. The layer IV in the visual cortex is known as the "cytoarchitectonic layer" and is charac- terized by the presence of large pyramidal cells. The myelinated axons in layer IV of the visual cortex are the axons of cells in the lateral geniculate nucleus (LGNN), which a relay nucleus the visual pathway. The LGN receives input from the retina and sends it to the primary visual cortex (V1) for further processing.
DA
Ca
i
NE
ATP
Ca
ADP
Pr G,
02
NE
MAO
Post-synaptic cell
Given the options: A. Superior colliculus
This part of the midbrain involved in visual and auditory processing, not the visual cortex. B. Lateral geniculate This is the correct answer the myelinated axons in layer IV of the visual cortex are the axons of cells in the lateral geniculate nucleus. C. Retina The retina is the sensory organ that sends information to the LGN, not the LGN itself. D. Medial geniculate This is part of the auditory pathway, not the visual pathway. Therefore, the correct answer is B.
B. Lateral geniculate. </answer>
Analysis:
The model correctly identifies that the myelinated axons in layer IV of the visual cortex originate from the lateral geniculate nucleus (LGN), Its reasoning demonstrates accurate knowledge of the visual pathway-that retinal ganglion cells project to the LGN, and LGN neurons, in turn, send their axons (optic radiations) to layer IV of the primary visual cor- tex (VI). The exclusion of other options (superior colliculus, retina, medial geniculate) is logically consistent and anatomically precise, showing solid understanding of sensory sys- tem segregation. The answer is well well-grounded in neuroanatomy and pathway logic, though could be further improved by explicitly linking the layer IV (stria of Gennari) feature to the LGN afferents and noting that the medial geniculate instead projects auditory tex. The model correctly traces the visual pathway, recognizing that myelinated axons in layer IV of the visual cortex originate from lateral geniculate nucleus neurons. Its reason- ing is anatomically precise and stru-ssde effectively excluding distractors from other sensory systems.
32

---

Published as a conference paper at ICLR 2026
MeedVL.Syther-7B 13K output for MMMMM
Question: What the diagnosis? Choices:
(A) Echinococcal infection
(B) Pneumatosis intestinalis
(C) Trichobezoar
(D) Meckel's diverticulitis
(E) Adrenal cancer
Correct Answer: (B)
Model Answer: (B)
Model Response:
<think The image shows CT scan of the abdomen. The presence of gas within the intestinal wall, which is not normally seen, is key feature of pneumatosis in- testinalis. This condition characterized by the presence of gas within the intesti- nal wall, which can be seen as cystic or cyst-like structures imaging. The other options do not typically present with the same imaging findings
in the im- age. <tthink><answerr B. Pneumatosis </answer
Analysis:
The model correctly recognizes pneumato-
sis intestinalis by identifying gas within the bowel wall on abdominal CT. IIs reasoning captures the defining radiologic feature and excludes unrelated options, showing solid di- agnostic understanding. However, it lacks detailed image anchoring (e.g., cystic or linear intramural lucencies separated from luminal gas) and fails discuss differential cues or clinical severity markers. The model accurately identifies pneumatosis intestinalis based on intramural gas CT. Its reasoning is directionally correct but could be strengthened by specifying the cystic/linear gas pattern along the bowel wall and differentiating from mimics such hepatic cysts intraluminal masses.
33

---

Published as a conference paper at ICLR 2026
MedVL.Synther-7B 13K output for Path VQA
Question: does mycobacterium avium in- fection in duodenal biopsy from pa- tient with aids show massive intracellular macrophage infection with acid-fast organ- isms filamentous and pink in this acid-fast stain preparation? Choices: (A) Yes
(B) No
Correct Answer: (A)
Model Answer: (A)
Model Response:
<think> The image provided shows a high magnification view of duodenal biopsy stained with an acid-fast stain, which is typically used to detect Mycobacterium
species. The organisms appear pink, filamentous structures within the cells, which is char- acteristic of Mycobacterium species. The presence these organisms within macrophages also consistent with mycobacterial infection. Given the appearance of the organisms and their location within macrophages, the answer is likely to be positive. Therefore, the answer is: </thin>><answer>>
c/ans//ns
Analysis:
The model correctly identifies that in an AIDS patient, a duodenal biopsy showing nu- merous pink acid-fast bacilli within macrophages on AFB stain diagnostic of Mycobac- terium avium complex (MAC) infection. Its reasoning appropriately connects the presence of acid-fast organisms and intracellalar accumulation in macrophages with the character- istic histopathologic pattern of MAC enteritis. The answer is accurate and clinically well- grounded, though the description "filamentous" is slightly imprecise since these organisms slender rods rather than true filaments. more complete explanation would note that these AFB-laden foamy macrophages occupy the lamina propria and are highlighted by Ziehl-Neelser Fite stains, confirming the diagnosis of massive intracellular mycobacte- rial infection.
34

---

Published as a conference paper at ICLR 2026
MedVL.Synther-7B 13K output for PMC-VQA
Question:
What does the surrounding
edema suggest? Choices: (A) A healthy
brain tissue
(B) The cause of intraparenchymal hemor-
rhage
(C) An underlying mass
(D) A normal imaging finding
Correct Answer: (C)
Model Answer: (C)
Model Response:
<think> The image shows a CT scan of the brain with annotations indicating mass within the right ventricle and hemor- rhage within the right ventricle. The term "edema" is used to describe the surround- ing area, which typically refers to swelling or fluid accumulation. In the context of a brain scan, edema often suggests an un- derlying mass or pathology that is causing the brain tissue to swell. The presence of edema in the vicinity of mass is a com- mon finding and is often indicative of the mass's effect on the surrounding brain tis-
sue. </think>> <answer>>. An underlying mass. </answer>
Analysis:
The model appropriately links the presence of surrounding edema to an underlying mass, recognizing that perilesional hypodensity non-contrast brain CT typically reflects vaso- genic edema caused by tumor-induced blood-brain barrier disruption. This interpretation is anatomically and sathiplyyiolooicalll sound, as such edema tracks along white matter, spares cortical ribbon, and often produces mass effect- fffeutefeatures consistent with a neoplastic process rather than normal tissue or benign change. While the model's 's reasoning correct, could be improved by specifying the edema type (vasogenic vs. cytotoxic) and explicitly excluding other causes such as reactive changes or hemorrhage origin. Overall, the con- clusion is accurate and clinically coherent: surrounding edema in this context indicates an underlying mass.
35

---

Published as a conference paper at ICLR 2026
MedVL.Synther-7H 13K output for SLAKE
Question: Does the picture contain kidney?
Choices: (A) Yes
(B) No
Correct Answer: (B)
Model Answer: (B)
Model Response:
<think>> The model correctly identifies the image thoracic CT slice and reasons that kidneys are absent because they lic in the abdominal region (around LI-L3). Its inference is anatomically sound, grounded in visible thoracic structures such as the heart, lungs, and ribs. The reasoning shows solid spatial and physiological un- derstanding, though it could be strength- ened by explicitly citing thoracic landmarks and noting that lower chest cuts may oc- casionally include the renal upper poles. </think><<nswerr B. No</answer>>
Analysis:
The model correctly identifies the modality as MRI, which matches the image (a coronal brain slice with high soft-tissue contrast). can capture clues such as soft tissue contrast and arrows indicating lesions, demonstrating its sensitivity to imaging features.
36

---

Published as a conference paper at ICLR 2026
MedVL.Synther-7B 13K output for VQA-RAD
Question: there evidence of pulmonary
edema? Choices: (A) Yes
(B) No
Correct Answer: (B)
Model Answer: (B)
Model Response:
<think> To determine if there evidence of pulmonary edema in the given chest X- ray, we need to look for certain signs. Pul- monary edema is characterized by the pres- ence of fluid in the alveoli, which can lead increased lung markings, blunting of the costophrenic angles, and possible areas of consolidation or hazy opacities in lung fields.
In this X-ray, the lung fields appear rela- tively clear without significant opacities increased markings that would suggest pul- monary edema. The heart size and medi- astinal structures also appear within normal limits. There no evidence of blunting of the costophrenic angles that would be in- dicative of pleural effusion, which often associated with pulmonary edema.
Given these observations, there is no evidence of pulmonary edema in this image. <athie>><nnswer>> B. No<//nsser>>
Analysis:
The model correctly concludes that there is no evidence of pulmonary edema, basing its answer the absence of diffuse opacities. normal heart size, and clear costophrenic gles-all all consistent with a non-edematous chest radiograph. Its reasoning reflects sound general understanding, but it remains somewhal superficial, lacking mention of hallmark interstitial and alveolar edema signs such as cephalization of pulmonary vessels, Kerley lines, peribronchial cuffing, indistinct vascular margins. or perihilar "bat-wing" opacities. A stronger explanation would explicitly state that vascular markings are crisp, lung fields remain lucent without interstitial or alveolar filling, the cardiothoracic ratio normal, and no effusions are seen. These combined findings confidently exclude pulmonary edema.
37


Reviews:
[{"review_id": "zJbbrcwr1N", "reviewer": "Reviewer_57hD", "summary": "This paper presents MedVLSynther, a well-designed framework for automatically generating and verifying medical VQA data from open biomedical literature. The proposed generator–verifier pipeline ensures that the resulting questions are self-contained, clinically valid, and image–text consistent, leading to the creation of a high-quality dataset (MedVLSynther-13K).\n\nOverall, I am positive about the paper. While the novelty is somewhat limited—as the work shares similarities with PMC-VQA—it nonetheless represents a solid contribution with comprehensive experimental results and clear methodological rigor. Please refer to my detailed comments regarding minor typos and identified weaknesses in the Questions section. I am very willing to raise my score if the concerns have been solved.", "questions": "Minor Questions:\n\n1. Several tables mention “PMC,” but the paper does not clearly define what it refers to. It is likely related to PMC-VQA, but the authors should explicitly state this to avoid confusion and improve clarity.\n2. “Text-only generation lifts the average to 54.80=.” The “=” seems a typo.\n3. The tables use too many green background highlights, which actually reduce readability instead of improving it. In addition, Table 6 lacks the green markings used elsewhere, leading to inconsistency in presentation. The authors should simplify the color scheme and maintain consistent formatting across all tables.\n4. The paper references “Biomedica: An open biomedical image-caption archive, dataset, and vision-language models derived from scientific literature” but provides insufficient context about it. The authors should briefly introduce this dataset to help readers understand its role without needing to consult external papers—for example, clarifying whether compound figures were separated and describing the general distribution of the figures used.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Provides a new and useful medical VQA dataset.\n2. Carefully analyzes the data generation pipeline.\n3. Conducts large-scale evaluation showing the dataset’s usefulness.", "weaknesses": "Main concerns:\n\nMy main concern falls in two aspects:\n\n1. The improvement is limited. As shown in Table 2, most gains appear to come from using PMC figures, while the additional rubric-based filtering contributes little to performance improvement. Can authors explain the results more?\n2. The paper lacks a direct evaluation of data quality. All reported results rely on downstream task performance rather than explicitly measuring the dataset’s intrinsic quality. It is strongly recommended that the authors include a more direct assessment—such as human expert evaluation—to better validate the reliability and usefulness of the generated data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JeiNhqPgXf", "reviewer": "Reviewer_n1Sq", "summary": "The authors proposed a multi-stage pipeline that leverages open biomedical databases (Biomedica Index) together with open large language models to generate a synthetic medical VQA dataset. Specifically, the pipeline involves a set of rubrics to guide the generator and verifier LLMs in generating and verifying the synthetic VQA samples, resulting in a dataset of around 13,000 examples. The authors claimed that the proposed dataset is beneficial for improving performance on several benchmarks. Further ablation studies also demonstrated the potential advantages of using the proposed dataset. The complete openness throughout the entire pipeline paves a potential way to create and verify the training of medical LLMs.", "questions": "Question and suggestion\nQ1: Could author explain why not using the latest Qwen3 series?\nQ2: Could author explain more about the data contamination analysis? The supp. only mentioned 8220 rows in a test data which does not appear in the main paper.\nS1: It would be nice to know how other open source LLM's performance on the benchmark (e..g, Qwen3 or GLM-4.5V 108B ) so that we can better understand how practical of the proposed dataset.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The author proposed a first fully open pipeline to create synthetic medical VQA dataset. From the dataset, model, to the generation and verification process are transparent, which can be easily customized to various setting and is  beneficial to the research community.\n\n- A deep investigation of the generator and verifier are presented in the paper, which shed light on how people can leverage the proposed pipeline.", "weaknesses": "In summary, my concerns are more related to the resulting data evaluation:\n- Potential data leakage: Although we cannot say much about the other open models compared in the baseline, it seems to me that the resulting MedVLSynther has data leakage issue when evaluated on PMC benchmark because Biomedica is dereived from PMC. \n\n- Inconsistency and confusing table and graph results: The numbers of MedVLThinker-3B/7B presented in Figure 1c does not match those in Table 6. Feel like the numbers in Figure 1c is directly copy from the MedVLThinker paper. If so, what is shown in the table 6?\n\n- The benefit of data for performance boost seems to be minor: When looking table 5, it seems to me that most of performance gain is from RL, when doing pure SFT, there is no clear advantage when compared to model trained on PMC. Since the main selling point of the paper is the new data not RL training method, it is questionable where the proposed data is truly better than the existing dataset\n\n- Lack of baseline results: It would be nice to at least know how well the generator (GLM-4.5V 108B ) perform on the downstream task. From the application perspective, If GLM easily outperforms other methods in the benchmark, it seems to be unclear why bother to train a model on this new dataset instead of just using an open model. \n\n- The baseline is a bit dated: It's unclear why did the author choose Qwen2.5 series instead of Qwen3. Based on the Qwen3 release date (2025.04.29) and ICLR 2026 deadline (2025.09.24), the timeline should not be an issue.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mmlE2B6XUF", "reviewer": "Reviewer_XqLm", "summary": "This paper presents MedVLSynther, a rubric-guided generator-verifier framework for synthesizing medical VQA data from PubMed literature. The authors employ large multimodal models to generate multiple-choice questions from medical images, captions, and in-text references, followed by a three-stage verification process. The resulting dataset contains 13,087 questions over 14,803 images spanning 13 imaging modalities and 28 anatomical regions. Models trained on this data with reinforcement learning demonstrate improvements over baselines on six medical VQA benchmarks, achieving averages of 55.85 for 3B models and 57.56 for 7B models.", "questions": "1. If we train with 100k or all samples from PMC-VQA, would the performance be comparable or better? The paper does not answer this question.\n\n2. Is it possible to have a much large scale dataset based on dataset like BIOMEDICA[1], what would the cost be? \n\n\n\n[1] Lozano, Alejandro, et al. \"Biomedica: An open biomedical image-caption archive, dataset, and vision-language models derived from scientific literature.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper introduces a generator-verifier architecture that separates generation and verification using different LMMs, enabling auditable quality control through a rigorous three-stage process of essential gates, fine-grained scoring, and penalty detection.\n\n2. The context-aware generation approach incorporates not just image captions but also in-text reference paragraphs from the surrounding literature. \n\n3. The comprehensive rubric design with seven essential criteria, four to eight fine-grained criteria, and four penalty criteria, along with explicit weights and a machine-checkable JSON schema.", "weaknesses": "This paper is technically competent and presents a well-engineered pipeline. However, I have concerns about whether it makes a sufficient scientific contribution to a top-tier venue. Apart from the proposed datasets, what are the main insights people can get from this paper?\n\n1. The paper positions itself as achieving high quality despite a small scale (13K samples), but lacks a direct empirical comparison with larger-scale alternatives. It does not have direct comparison using PMC-VQA's full 227K samples for training; Table 5 only compares against PMC-VQA in a limited setup, not at full scale. \n\n2. The paper claims to improve medical VQA, but all evaluation benchmarks are also derived from literature or educational materials, not real clinical data. The paper's questions (Figure 2) are mostly \"recognition\" (33.6%) and \"modality identification\" (27.7%) — these are not representative of clinical reasoning. \n\n3. The paper motivates multimodal generation by criticizing text-only approaches (L83-84: \"text-only LLMs that ignore visual evidence\"). However, Table 2 shows that for 3B models, multimodal generation (54.72) performs worse than text-only generation (54.80), suggesting that visual grounding provides minimal or even negative value for smaller models. Is it necessary to have multimodal generation ?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "viPKzDTsmt", "reviewer": "Reviewer_7G9c", "summary": "The paper introduces MedVLSynther, a rubric-guided generator verifier pipeline that synthesizes multiple-choice medical VQA items directly from PubMed Central figures, captions, and in-text references.\n\nA context-aware LMM generator produces self-contained stems and mutually exclusive options under a strict JSON schema, while a multi-stage LMM verifier enforces essential gates, awards fine-grained points, and applies penalty criteria to ensure quality.\n\nThe resulting MedVLSynther-13K dataset (13,087 audited questions over 14,803 images, 13 modalities, 28 anatomical regions) is used to train open-weight LMMs with reinforcement learning using verifiable rewards, achieving state-of-the-art averages among open models across six benchmarks.\n\nAblations show both generation and verification are necessary, that more verified data helps (with diminishing returns beyond 5k), and contamination analysis finds no leakage from evaluation suites.", "questions": "What proportion of accepted items pass a small human audit, and how do human judgments correlate with the verifier’s fine-grained/penalty scores?\n\nHow sensitive are downstream gains to the chosen acceptance threshold τ and to the balance of question archetypes; could adaptive sampling increase difficult, clinically salient items?\n\nCan you report per-modality and per-anatomy breakdowns of improvements and failure cases to guide targeted data augmentation or rubric refinement?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality: The explicit, rubric-driven generator–verifier loop that is both context-aware (image + caption + references) and schema-constrained is a fresh, auditable approach versus prior text-only or loosely filtered syntheses. The design of essential, fine-grained, and penalty criteria operationalizes “exam-quality” medical VQA in a reproducible way.\n\nQuality: The empirical study is thorough, spanning dataset characterization, multi-benchmark evaluation, ablations on pipeline stages, scale, and generator/verifier choices, plus contamination analysis. The use of RL with verifiable rewards and consistent gains over strong medical LMMs strengthen the causal link between verified data quality and downstream performance.\n\nClarity and significance: The pipeline is clearly illustrated with stage-wise prompts/criteria and concrete acceptance rules, including a normalized score and high threshold. MedVLSynther-13K offers a practical, privacy-preserving resource that advances training for medical VQA and is likely to catalyze open, reproducible research.", "weaknesses": "Verification robustness: While multi-stage verification improves precision, the reliance on LMM verifiers introduces potential model biases and false accept/reject risks; human audits or inter-verifier agreement analysis are limited in the main text. Adding a small-scale human evaluation and measuring inter-model/verifier consistency would better quantify verification reliability.\n\nScope and coverage: Pre-filtering to Clinical imaging and Microscopy plus selected subtypes may bias question distribution (e.g., heavy digital photography share) and underrepresent complex radiology workflows. Expanding coverage to additional modalities/workflows and reporting performance stratified by difficulty and question archetype would improve generality claims.\n\nPractical reproducibility/cost: Although models and data are open, the use of very large generators/verifiers (up to 108B) may hinder replication and scaling by typical labs. Providing cost/compute estimates, a “small-model” variant with trade-offs, and release of prompts/rubrics/scripts would enhance accessibility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)