MedLesionVQA: A Multimodal Benchmark Emulating Clinical Visual Diagnosis for Body Surface Health
Authors: Deli Yu
Shengzhi Wang
Kai WU
Xiaozhong Ji
Bo Cui
Jieqiong Cao
Huichao Wang
Boyuan Jiang
Xu Wang
Qian Xu
ChaoGao
Yi Zhao
Dian Chen
Meng Li
Haifeng Wu
Yijun He
Haihua Yang
Venue: ICLR 2026 Poster (2026)
Keywords: Medical Multimodal Benchmark
Body Lesion Images
Medical VQA
Primary area: datasets and benchmarks

Abstract: Body-surface health conditions, spanning diverse clinical departments, represent some of
the most frequent diagnostic scenarios and a primary target for medical multimodal large
language models (MLLMs). Yet existing medical benchmarks are either built from publicly
available sources with limited expert curation or focus narrowly on disease classification,
failing to reflect the stepwise recognition and reasoning processes physicians follow in
real practice. To address this gap, we introduce MedLesionVQA, the first benchmark
explicitly designed to evaluate MLLMs on the visual diagnostic workflow for body-surface
conditions in large scale. All questions are derived from authentic clinical visual diagnosis
scenarios and verified by medical experts with over 20 years of experience, while the
data are drawn from 10k+ real patient visits, ensuring authenticity, clinical reality and
diversity. MedLesionVQA consists of 12K in-house images (never publicly leaked) and
19K expert-verified question–answer pairs, with fine-grained annotations of 94 lesion types,
110 body regions, and 96 diseases. We evaluate 20+ state-of-the-art MLLMs against
human physicians: the best model reaches 56.2% accuracy, far below primary physicians
(61.4%) and senior specialists (73.2%). These results expose the persistent gap between
MLLMs and clinical expertise, underscoring the need for the multimodal benchmarks to
drive trustworthy medical AI. The dataset can be found in https://github.com/bytedance/MedLesionVQA.

Published as conference paper ICLR 2026
MEDLESIONVQA: A MULTIMODAL BENCHMARK EMULAT- ING CLINICAL VISUAL DIAGNOSIS FOR BODY SURFACE HEALTH
Deli Yu Shengzhi Wang 1=t Kai Wu Xiaozhong Ji Bo Cuil", Jieqiong Cao Huichao Wang"", Boyuan Jiang', Xu Wang', Qian Xu', Chao Gao', Dan Wang', Yi Zhao", Dian Chen', Meng Li', Haifeng Wu', Yijun He', Haihua Yang
Xiiaoee Team, Bytedance Dermatology, Beijing Tsinghua Changgung Hospital
dely-yuêqq. com,
com, wukaiwork8gmail com, yunghalhialbyreelacee.cc
ABSTRACT
Body-surface health conditions, spanning diverse clinical departments. represent some of the most frequent diagnostic scenarios and primary target for medical multimodal large language models (MILMss. Yet existing medical benchmarks are either built from publicly available sources with limited expert curation or focus narrowly on disease classification, failing to reflect the stepwise recognition and reasoning processes physicians follow in real practice. To address this gap, we introduce MedLesion VQA, the first benchmark explicitly designed to evaluate MLLMs on the visual diagnostic workflow for sudy-surface conditions in large scale. All questions are derived from authentic clinical visual diagnosis scenarios and verified by medical experts with over 20 years of experience, while the data are drawn from 10k+ real patient visits, ensuring authenticity, clinical reality and diversity. MedLesion VQA consists of 12K in in-house images (never publicly leaked) and 19K expert-verified question-answwr pairs, with fine-grained annotations of 94 lesion types, 110 body regions, and 96 diseases. We evaluate 20+ sate-off-the-ar MLLMs against human physicians: the best model reaches 56.2% accuracy. far below primary physicians (61.4%) and senior specialists (73.2%). These results expose the persistent gap between MLLMs and clinical expertise, underscoring the need for the multimodal benchmarks to drive trustworthy medical AI. The dataset can be found in https: https:// com/ bytedance/ www.ddannce/eeeeeiiioooo /MedLesionVVAA
1 INTRODUCTION
Photo-based interaction with multimodal large language models has recently gained attention as potential pathway for addressing body-surface health concerns, including the skin, nails, hair, oral cavity, genitals, and other visible areas areas(AlSand AlSaad et al. 2024 Zhou al. 2024a). It requires MLLMs (Saab et al. [2024 Moor et al.
2023
et al. 2023 Chen et al. [024b] Lin et al. [2025] Nath et al., 2024) and medical MLLMs (Tian et al.
2023
Chen al. 2023 Wei Zhu & Wang 2023 Wang et al.. 2025 to give visual diagnosis results according
to body lesion images photographed by users via smartphone or other device, Although current MILMs have
Equal contribution
1 Work done during internship Corresponding author

---

Published as a conference paper ICLR 2026
Fine-Grained Annotations by Experts
Align With Real Clinical Practice
Dataset Examples
Lesion Recognnzton
Recognition
Expert Distribution (Years)
Quesiion What losion ii this pietaee!
Lesion Recognition
Attribute Recognition
Location Recognition
50%
20%
18%
12%
Options: arytheme, papule; C. plaque;
Answer:
1:55
5-10
10:20
>20
Location Recoggniton
Undorstanding
94 Lesions 96 Discuses
Question: What is the body region shown this picture?
Lesion Reasoning
Spacial Relation
Ci eczema
Optione froo: army leg
Answer AB
dermatitis 110 Body Regions
Artribute Recognition
Clinical Outcome
Evaluation categories
Question th
peanut-lkee
Diagnosis
Answer: Yes
Zaion
19%
Suggestion
Lasion
Reba
1755
Qusstion:
Lesion Reasoning
considkoeal maculet
Treatment
Dogg
Answer: The lesion presents localized and non mised color change in the picture, thorefore
20%
Diagaosis
Spatiol Relaziin
Suggestion Treapmont -----
Question: disenes
Question: treat
diegnosed based pictures?
picture? list least topical anti infective drugs?
Answer:
parrissis and chronic orveru
Asswer: on the surface
Answer: Mepimein uintmere, fisiéic acid cintment
Figure 1:
Overview Medl eesion VQA. The benehmark is designed to emulate visual diagnostic workflow physicians (top-left), covering seven core abilities with fine-grained annotations. Expert physicians with over 20 years of experience validated annotations (middle), which include detailed identification of 94 lesion types, 96 discases, and 110 body regions (bottom).
shown the ability for medical assistance (Esteva et al., 2017 Coustasse et al. 2019 Tschandl et al. 2020). they still struggle replicate the visual diagnostic workflow (Weller et al. 2014) that physicians rely for body-surface health- soath-nnnnnnn fine-grained recognition, reasoning, diagnosis, and treatment suggestions across departments such dermatology. dentistry, and general surgery. The critical challenge is how evaluate whether MLLMs can truly align with this workflow and perform like physicians in authentic clinical settings.
Existing medical benchmarks are either assembled from publicly available sources with limited expert curation focus narrowly on disease classification, failing to capture the visual diagnostic workflow for body-surface health that physicians follow in practice. General-purpose benchmarks, such as GMAI-MMBench (Ye et al. 2024) and OmniMedVQA (Hu et al., 2024), extend to up to 38 modalities by aggregating data from open-source websites. Although these datasets are extensive, publicly sourced information often includes outdated or basic-level data and lacks expert annotations critical for lesion interpretation and treatment recommendations. Conversely, specialized datasets such SkinCon (Daneshjou al., (022b) and DDI (Daneshjou et al. 2022a) integrate expert annotations focus narrowly on singular tasks, such as disease classification, not adequately reflecting real- real-world clinical practice. For instance, SkinCon (Daneshjou al. 2022b) introduces lesion concepts, which are visual symptoms disease, without open-ended diagnostic queries. DDI employs binary labeling (e.g., malignant vs. benign), which oversimplifies the real-world clinical complexities. Additionally, SkinCon contains only 3,700 images, and DDI encompasses merely 656 cases (Daneshjou al. 2022a),, which insufficient for robust evaluation.
2

---

Published as a conference paper ICLR 2026
To address these issues, we introduce MedLesionVQA, the first benchmark explicitly designed evaluate the visual diagnostic workflow for body-surface health. To ensure authenticity and close alignment with physician practice, we collaborated with senior medical directors with over 20 years of experience and defined seven core diagnostic abilities by referring to authoritative textbooks and clinical literature (Weller et al. 2014). These abilities span lesion recognition, reasoning, diagnosis, and treatment across dermatology dentistry, and surgery. Our dataset comprises 12K images collected directly from real patients, guaranteeing that none originate from internet sources or leaked repositories. With 12K images and 19K question-ana answer pairs, MedLesion VQA is substantially larger than prior expert-curated benchmarks for body-surface health, enabling more robust and diverse evaluation. Beyond its authenticity and scale, MedL_esion eesion implements a tne-grained annotation system, covering 94 lesion types, 96 diseases, and 110 anatomical regions. For example, human hand is subdivided into nine distinct regions, from the purlicue to the fingertip, enabling highly detailed evaluation of model performance.
STD,
Furthermore, our QA generation pipeline is grounded in real clinical questions, which serve as templates for automatic generation and are then refined through rigorous expert review. This yields over 19K diverse, high-quality QA pairs with expert-level accuracy and statistical reliability, addressing gaps left by prior benchmarks. After extensive prompt tuning and iterative refinement, we establish an LLM-based scoring system developed with physicians, ensuring strong consistency between automated assessments and human judgments. Our key contributions are summarized as follows:
The first body-surface benchmark aligned with visual diagnostic workflow. We introduce the first multimodal benchmark explicitly designed to evaluate the visual diagnostic workflow for body-surface health, moving beyond narrow disease classification. MedLesionVOL VQA evaluates the stepwise diagnostic abilities of state-of-the-an MLLMs, providing a foundation for their ad advancement toward real-world clinical use.
Expert-level and fine-grained annotation system. Our benchmark benefits from valuable expert annotations, covering over 96 prevalent diseases, 110 body regions and sub-regions and 94 distinct lesion types. All annotations are conducted and rigorously verified by clinical experts following a systematic clinical lexicon tree.
Comprehensive evaluation. We conducted an extensive evaluation involving more than 20 widely-used MLLMs. Additionally, we established human baselines by engaging general practitioners and senior physicians, enabling a thorough and systematic comparison between MLLMs and medical experts.
2
RELATED WORKS
2.1 MULTIMODAL LARGE LANGUAGE MODELS
Numerous Multimodal Large Language Models have been developed, focusing primarily on improving image captioning, visual question answering, and cross-modal retrieval (Achiam al.] 2023 Anthropic [2025a]
Bai et al. 2023 Chen et al. 2024e fi Liu et al. [023c] Chen et al. 2024f] Representative models include
the GPT-VV (Achiam et al. 2023). DeepSeek series (Guo al.
(025). LLAVA series (Li et al. 2024 Liu
et al. 2023c). Intern VL series (Chen et al. 2024 d), Qwen series [Bai et al.] 2025) Wang et al. 2024b). and
CogV LM series (Wang al.] 2024c] Hong et al. (024), among others (Laurençon al. [0023] Ding et al.
2021). These works have significantly contributed to the development of the community. To address specific
medical tasks, researchers have trained and fine-tuned MILMS using specialized medical data, leading to
the development of medical vision- language models (Li et al. 2023a| He al. 2024]] Wu et al. 2023 Liu
ct al.] 2023d). which integrate medical images (such as X-rays, MRIs, and CT scans, etc.) with clinical data (including patient records, diagnosis, and treatment plans, etc.) (Ye et al. 2024 Antonelli et al. 2022
Irvin et al., (201)). However, achieving precise medical question answering and fine-grained multimodal diagnostics remains significant challenge.
3

---

Published as a conference paper ICLR 2026
Table 1: Difference between MedLesion VQA and other existing benchmarks/datasset.. aaapaa//////aaaatttomimmm/a////a Onn (Hu et al. 2024) and GMAI-MMBBnch Ye al. 2024) contains subset of lesion images for dermatology-related
eva uation.
Benchmark
Images/QA
VQA
Data source
Anno./Eval. dimension
lesion (unknown)
OmniMedVQA * (Hu et al. 2024
119K / 128K
public
body region (25)
GMAI-MMBench" Ye ct al. 2024
26K / 26K
V
public
discasc (umkrrow))
Fitzpatrick17K (Groh et al. DermNet dde 2025
202T
17K / null
X
public
discase (1 (114)
19K / null
X
public
disease (23)
SkinCon Daneshjou et al. 2022b
3230 mull
X
public
lesion concepts (48)
DDI (Daneshjou 2022a)
656 null
X
in-house
disease (2)
SNU-134 (Hnn 2019
2101 mull
X
in-house
disease (133)
lesion (94) and attribute (7)
body region (110)
McdLesion VQA
12K / 19K
in-house
disease (96) suggestion treatment
2.2
BENCIMARRS
The field of MLLMs has experienced rapid advancements, both in terms of models (Achiam et al.
2023 Bai
ct al.
2023
Anthropic 2025a) and benchmarks (Bitton al., 2023 Zhu et al. 2024 Li et al. 2025 Ray
ct al., 2024
Lim et al. 2024 Yu et al., [203) 2024] Xu et al. 2023 Lee et al.) 2024 Roberts et al., 2022).
Evaluating the medical capabilities of MLLMs requires specific benchmarks, and the representative medical
benchmarks include VQA-RAD (Lau et al.
2018), SkinCon (Daneshjou et al.
20226). SkinCAP (Zhou et al.
2024b), DDI (Daneshjou al. 2022a). SCIN (Ward et al. [2024). SLAKE (Liu et al., 2021), RadBench
[Wright & Reeves] 2016). MMMU (uue al. (024), GMAL-MMBench (Ye et al. (2024). OmniMedVor (Hu
et al. 2024) and MediConfusion (Sepehri et al. 2024), etc.. Among which, OmniMedVQA (Hu et al. 2024)
introduces the largest medical VQA dataset to date, covering 12 data modalities and 20 anatomical regions, with over 100k images. GMAI-MMBench (Ye et al. 2024) includes various medical imaging data, such as X-rays. CT scans, MRIs, and ultrasounds, along with corresponding clinical information. RadBench (Wright & Reeves 2016) focuses on radiology, involving tasks such as modality recognition and disease diagnosis. Derma VOA Yim et al. 2024) introduces multilingual dermatology VQA dataset built from remote-care patient portal messages and focusing on response generation for dermatology consultations. DermIM (Yan ct al. 2025). in contrast, is million-scale dermatology vision-language training dataset aligned with a clinical ontology. In this work, we introduce MedLesion VQA, which consists of 12K+ in-house body lesion images and 19K expert-verife verified QA pairs. It uniquely targets the stepwise visual diagnostic multimodal abilities that are central to real visual diagnosis workflows.
3 ESTABLISHMENT OF MEDLESIONVQA
3.1 OVERVIEW OF BENCHIARR
MedLesion VQA comprises 12K in-house patient images collected under approved ethical protocols. All images undergo a rigorous pre-processing pipeline, including image quality filtering, content inspection, de-identification of personal information, and distribution balancing. We cooperate with senior physicians to design and implement an anmotation protocol, referencing authoritative materials (Weller et al., 2014; James ltt 2011). The protocol covers 96 prevalent diseases, 94 lesion types, and 119 body regions. Then, inspired by diagnosis and treatment pipeline in clinical practice, we construct 19K diverse question-answa samples involved with 7 stepwise visual diagnostic abilities, and some examples are shown in Fig. T Finally,
4

---

Published as conference paper ICLR 2026
Body Region
Disease Diagnosis
There are multiple papules the skin the junction of the cheek and the lateral neck. The lesions and their corresponding characteristics consistent with the typical symptoms folliculitis. Folliculitis commonly occurs on the face, head neck. Therefore, folliculitis is considered.
scalp, face, postauricular area, neck, hand, fingernail
Lesion
primary lesions: papules
location: cheek, lateral neck
size: 0.1-0.3cm diameter
Suggestion Treatment
color: brown
The treatment of folliculitis mainly focuses on topical medication. recommended to select medications for treatment under the guidance of doctor.
shape: subcircular
distribution: scattered
number: multiple
Commonly used medications include Ichthammol Ointment, Mupirocin Ointment, etc.
boundary: relatively well-defined
secondary lesions: pustules, reddening
The Annotation procedure. The physicians sequentially annotate the body regions, lesions, attributes, discase diagnosis, and finally suggestion & treatments.
Figure 2:
propose automated scoring pipeline to calculate the metric of MLLMS' benchmark results, and the scoring pipeline tuned align physician judgment metric with negligible difference.
3.2 ANNOTATION PROTOCOL
More than tens of physicians are invited into the image annotation process, which contains image filtering, annotation labeling, and annotation reviewing. First, group of annotators check the quality of each image, such as its clarity, and discard the unqualified images as well as those that do not show the exposed human skin the oral cavity. Second, body region type, lesion type, lesion attribute type, disease type, and suggestion treatment annotations are labeled under annotation rules, which are developed by an expert panel of senior experts. Finally, other senior experts review the annotation results and correct any errors, ensuring the annotation quality with entity-level precision and recall of over 95%.
Body region. The physicians asked annotate all visible parts of the human body and the internal parts of the oral cavity. We have respectively constructed the corresponding lexical trees for part division, and the annotation is carried out according to the secondary nodes of the lexical trees. More information of the lexical trees is detailed in Appendix A.2.
Lesion. Our dataset has annotations for types of lesions. For each lesion, describe its key attributes. These attributes are: size, color, shape, quantity, distribution, and boundary. We also pinpoint the exact location of each lesion. To do this, we use very detailed body map, much like the fine branches of tree. All our labels have multiple options, not just "yes or no," and most come with at least different text descriptions. Finally, we identify primary and secondary lesions. We also describe their relationship and how often they appear together.
Disease. Each image provided with up to differential disease diagnosis by two independent physicians, which are sorted in the order they consider the most reasonable. Then, the inverse of the rank is used the weight to combine the annotation results the two physicians, to obtain the final sorting result. For the list of total discase labels in the annotation data, please refer to Table of the supplementary material. The logic of diagnostic reasoning is also provided during annotation.
Suggestion & Treatment. For each image, physicians required to provide corresponding treatment suggestions based on the unique disease diagnosis or differential disease diagnosis, including advice on seeking medical treatment, medication, matters needing attention in daily life, and
5

---

Published as a conference paper ICLR 2026
3.3 QUESTION-ANSWER CONSTRUCTION
This section introduces the process of question generation, including category balance, prompt design tailored for assessing different cognitive abilities, and the development of various question types.
Evaluation category balance. We balanced the distribution of questions across seven abilities to closely reflect their real-world distribution clinical practice, as illustrated in Fig. T Lesion, attribute, and location recognition questions comprise 61% of the MedLesion VQA dataset, as accurate fine-graines graiind recognition is fundamental for subsequent diagnostic tasks. Specifically, the evaluation assigned equal weighting to each lesion type according to the real-world distribution, ensuring comprehensive coverage for accurate skin lesion identification and analysis.
QA construction prompts. In the context of real-world question examples, design different QA generation templates for different evaluated abilities in order to test the corresponding capabilities. Two typical prompts are displayed in Fig. 3 aa), and the rest will be included in the supplementary materials.
Diverse question types. The generated questions are categorized into two types: multi-choice and open-ended questions, while open-ended questions include judgment, fill-i-thhebblank, and short-answer questions. For multi-choice questions, we create similar distracted options based on the correct answer and then randomize the order of all options, ensuring that the correct answer has an equal likelihood of appearing in any position. To prevent answers from being overly diverse and difficult to assess, the answers to open- open-ended questions are kept relatively concise. This approach enables the judging model to provide more consistent scores in the subsequent evaluation.
Lavion Recoynihion
Eieenonis reasoning
Multi-
Oper-anded
Thes description: You will be prodtte with
prcoo of medkal informaiin reuted to a suturee Questors
ralsed regirding
Task desription: Provile you with a piace meskall irior mutien
patare.
jingle queatons will provides with one
nutch the anasr rault with mullip
respanie the cuastion the prockted amoordiddg whitter the prodited srowe cometty indeeeette the quedias, posile woringg give corposseskee 05 lputlally camall, jumpkoddy comeal).
descripiin::
and
that all info information sourzed from
for asdiin thown the picture reeds
the
ctung, and the recogittio db of the model
requredioo
luaeed on the isformation iduarel From the cattrr,
making dagnees
daiky of the model make shocse deep dlagnzi
diggootic
the key intermation the
susvine resoving
enithing clo. the monning al optiars are varifarrlyy ai flarent from tse anien
reults, supurr 2. You shouts eurnut singl or miltple captos
quectoo, multyle rettors, and rutiple mnser
do Ouisut
optioss
The a1 sooe most eutpst forror
Cuetiod coraples fron mall ovril scarorios
Coetted ecumpiee from nol world seworbee>
4
Medical information: <Medical information
dytions sUptiorge
Question +Queiilme Correct answer <trawer predicte srower -rrddctions
Arsser results: cknowars
Prompts automatic QA construction for evaluation abilities.
(b) Prompts for extracting answers and scoring predictions.
The prompt template used on MedLesionVO Medical information includes body region, lesion, attribute, disease diagnosis, and suggestion & treatment information annotated above.
Figure 3:
Manual review and improvement. To enhance the medical accuracy and ensure appropriate difficulty in QA sets, physicians manually review all auto-generated QA pairs. This review focuses primarily on verifying the correctness of critical medical information within both the questions and the answers. Ambiguous questions are clarified, and non-standard answers are revised accordingly. Additionally, distractors in multi-choice questions are assessed regarding their accuracy and difficulty. A few open-ended questions, particularly those concerning suggestion & treatment and lesion reasoning, are converted into multi-choice format due to the inherent complexity of determining definitive answers. The final benchmark comprises 19,843 question- answer pairs (QAs), which are partitioned into a validation subset containing 1,499 QAs (7.55% of total samples) and test subset consisting of 18,344 QAs (92.45% of total samples).
3.4 AUTOMATIC SCORING PIPELINE
For multiple-respon questions, since MLLMS occasionally fail to output exact option answer, need extract the option answer from the answer set and the raw prediction output using saraattin-answer prompt and then compare it with the correct answer. To calculate score, we have set the following rules: 1) If the
6

---

Published as a conference paper ICLR 2026
predicted answer contains options that are not in the correct answer set, is considered completely wrong and receives score of 0; 2) If the predicted answer fails to identify all correct answers, the score is calculated based on the ratio of the number of correctly answered options to the total number of correct answers.
For open-ended questions, the prompt for the judge model is designed as indicated in Fig. Bb). With this prompt, the judge model will analyze the predicted answer, compare its similarity to the correct answer, and most importantly, determine whether the question has been answered.
Evaluation consistency test. We use GPT-4 as judger to score the model's predicted answers for open-ended
QAs. Moreover, we invite physicians to score the answers, also using the three scoring levels of 0 - 0.5 1.0.
Through the analysis inconsistent cases, we find that the model is too strict in scoring for attributes such color and size. For example, or color descriptions like "pink" and "skin tone", and size descriptions like "pinpoint" and "millimeter", due to the lack of specialized medical knowledge, the judge model tends to be overly strict according general criteria. When supplement the evaluation details for color and size in the prompt, therefore the high consistency rate between the judge model's scores and manual scores can be ensured. The details can be found in Appendix A.5.
4 EXPERIMENTS
4.1 EVALUATION
MLLMs baseline. For closed-source models, evaluate several well-known models, including GPT series models (Achiam et al. 2023). Gemini series models (Google 2025 DeepMind 2024). and Claude4-
opus(Anthropic/ (225a)). For open-source models, we comprehensively evaluate model parameters ranging from 0.256 billion to 72 billion, including the famous LLaVA series(Liu et al. 2023b Li et al., 2023b), Qwen2.5 series (Wang et al. 2024a), VL series(Chen et al., 2024(), DeepSeek-Vi VL series (Wu et al.
2024) and several medical-domain models, like MedGemma (Sellergren et al. 2025). HuatuoGPT-vvsiio
(Chen et al. 2024b), Bimedix2 (Mullappilly et al. 2024). LLava-med (Li et al. 2023b), lingshu-7b (Xu et al. 2025 and iinssuu32b(Xu et al. 2025).
Physician baseline. We invite two groups of primary and 15 senior physicians to answer the 1499 questions in the validation set, respectively. Primary physicians are general practitioner, while senior physicians are specialized expert from dermatology or dentistry departments. Questions are randomly distributed, and each question is completed by at least 2 different physicians. The physicians are not allowed to consult textbooks or search the Internet during the question completion task.
Evaluation Implementation. The evaluation is conducted using the VLMEvalKit (Duan et al. 2024) framework under zero-shot setting. There are text-only set and vqa-set. We additionally add a text-only baseline input to isolate the contribution of the visual modality, helping to evaluate the model's reliance on visual versus textual information.
MedLesionVQU. has around 9% questions, which can be answered without significant reliance visual input. These questions mainly belong to the category of treatment and suggestion. We intentionally retained all questions rather than dropping the text-only ones out, since both visual and textual information are integrated real-world clinical diagnostic process. Each question will be assigned a label of multi-modal or text-only in the open-resource version.
4.2 MAIN RESULTS
2 compare the performance of 22 vision-language models MedLesion VQA which includes 7 medical tasks aligned closely with real clinical setting, assessed through both multiple-choice and open-ended question formats. Fig 4 presents the performance of 10 representative MLLMs MedLesionVOL VQA. In general, Gemini-
7

---

Published as a conference paper at ICLR 2026
Table 2:
The overall accuracy of open-source and flosed-souree models on the test set and validation set. *:Some closed-sous souree commercial models are evaluated only on the valid set due to API access limitations. The table is sorted in descending order based on the AVG test score.
Recognition
Undersfanding
Lesion Recogpilion
Locotion
Attribute
Spotial Reiation (1183)
Lesion
Disease Diagnuis (1698)
Segaestion
Model
(1499)
(16544)
Receguttio (898G)
Becogaition (3605)
Beassiigg (3071)
Treatuneat (1018)
(8340)
+ Image Input
Senior physicians*
0.7321
0.6826
0.7583
0.7046
0.7102
0.6533
0.7313
0.8574
Primary physicians*
0.6144
0,5932
0.6218
0.5203
0.6336
0.5412
0.6258
0.8162
Gemini-2.5-ppe 2.5-poo--- Go
2025
0.5624
0,4902
0,5166
0.4300
0.6223
0.5754
0.6048
0,8482
GPT-3 OpenAl
2025
0,5252
0,4741
0.5109
0:4039
0.6932
0,4550
0,4444
0.5684
Claude opus Anihaopic
2025b
0.5139
0,3906
0:4513
0:4488
0.7412
0.4458
0.5744
0.6076
GPT-03 OpennA 2024
0.5092
0,4379
0:4881
0:4718
0.6288
0.4302
0.3826
0.4229
GPT-4V OpenAl 2024
0.4938
0.4915
0,4071
0,4780
0,4050
0.6308
0.3393
0.5132
0,8216
Gemini 2.0-Hashh DeepMind
2024
0.4954
0.4801
0,4062
0.4453
0.3923
0.61 12
0.3443
0.5219
0.8136
Qwen2.5-VL-7225 (Wang al.
20224a
0.4904
0,4904
0.3735
0.4636
0.417
0.6618
0.3608
0.5272
0.8246
IniemV1...5 78B (Chen el al
20241
0.4790
0.4757
0.3352
0.4981
0.4259
0.6601
0.3084
0.4800
0.7963
GLM-4V-9B (GL ot al (024)
0.4654
0.4474
0,3472
0.4528
0.3584
0.5596
0.3283
0,4929
0.7281
Qwen2.5-VL-17 (Wang clal.
2024a
0:4243
0.4243
0,3255
0,4005
0.3547
0.5482
0.3356
0,4248
U,7474
Deepseck 112 small el al.
0.4142
0.4164
0,3226
0.4107
0.3627
0.5297
0.2534
0.4822
Doepsack-112 Wu ct
2024
0,3882
0.3928
0.3293
0.3383
0.3514
0.5563
0,2468
0,4309
0.7147
Lingshu-3BB Xuret
20035
0,3541
0,3539
0.21
0.2723
0.3796
0.5551
0.30
0,2811
0.7276
Qwen2-VL-22 Wang al
2024a
0.3536
0,3533
0,2876
0.3319
0.3059
0.4448
0,2057
0.4171
0.6675
Lingshu7b Xu al 2025
0,3398
0,3448
0,2058
0.271
0.379
0.504
0,2822
0.2794
0.7078
Deepoec-v/2-iity etu
2024
0.3168
0.3293
0,2660
0,2869
0,3079
0,4529
0.1817
0.3953
0,6109
LLaVA-v1.5-13B
Liu al.
0.2980
0.3008
0,2437
0.3270
0.2742
0.31
0.1798
0.3082
0.4966
Intem\L2..5-88
(Chen
20241
0.3096
0.2994
0.3035
0.3247
0.2796
0.3100
0.1474
0,2772
0,4082
ShanoGPT4V-7B Len ot al.
20240
0.2897
0,2831
0.2232
0.2914
0.2656
0.4155
0,1476
0.3256
0.4233
HoamooGTT-vssion
Chen et ml LLaVA-mistral-78
2024g
0.279
0,2281
0.1111
0.2997
0.5649
0,1897
0,282
0.676
Li et a 2025a
0.2911
0,2731
0,2205
0,2714
0,2640
0.3740
0,1585
0.2399
0.4913
Blomedic2Muullappillyy at
2024
0.2603
0.2676
0,171
0.1647
0.3118
0,4095
0.1563
0.349
0,5977
IIaVA.11.5715
Luu et al.
202.3h
0.2648
0.2595
0,2254
0,2456
0.2288
0.31
0.1605
0.3042
0.423
IntemVLL2.5448
Chen X0:e1
0,2632
0.254
0,1895
0.3151
0.2428
0.2172
0.1336
0.3121
0.2965
McdGemma-4b Sellergren ci a. 2025
0.195
0,203
0.1212
0.0915
0.1697
0.4007
0.1908
0,2339
0.5564
SmolVT
Marahoti et al. 2025
0.1898
0.17
0,1711
0. 1602
0.1897
0.2656
0.0992
0,1417
0,2190
SmoIVLMM-25M
Marahot et all 2015
0.1564
0.156
0.1397
0, 1418
0.1507
0.2172
0.0912
0.1691
0,2274
LLaVA-med-v1
(L) al 20236)
0.0885
0.0791
0.0372
0.0715
0.1104
0.1258
0.0466
0.0535
0.1426
Only Text as Input
(Wang V.2.578..
al. 2024g
0.3636
0.3839
0.3378
0,3089
0.3763
0.6606
0.2967
0.3946
0.8014
Qwen2.5v1-728 (Wan al. 2024
0.3478
0.3537
0,2640
0,2784
0.2987
0.5818
0.3194
0,3016
0.8124
IniemVL2.554B Chen et 20241
0.3403
0.3406
0.2071
0.3023
0.3190
0.5266
0.2981
0,2645
0.7446
GPT-4V /Achiam ei ail 2022
0.3089
0,3185
0,2201
0.1687
0.3200
0.6076
0.2441
0,2844
0.8140
Dwon2.5VL-7BB/Waa et al.
2024g
0,3153
0,3097
0,2217
0.2376
0.2646
0.4900
0,2939
0.2945
0,7404
Deepock-v12 at.
2024)
0.2981
0,2851
0,2452
0,16855
0.2916
0.5455
0.1996
0.3032
0,7227
Qwen2-VIL-2B Waag et al.
2024g
0.2693
0.28
0,2146
0.2384
0.2636
0.4195
0.1873
0.2232
0. 0.6389
SharoGPT4V-7B (C hcp
20144
0,2193
0.2477
0.1940
0.1171
0.2293
0.3374
0.1439
0,2668
0.4247
LLaVA-med-v1..-7B (Li
202236
0.08-42
0.0763
0.0349
0.0535
0.1096
0.1533
0.0398
0,0739
0.1899
2.5-pro(Google 20255) shows the best performance across nearly all capabilities with 56.24% average accuracy. Senior physicians achieve averaged score of 73.21% ,, far beyond the best MLLMs. Key findings from this comprehensive comparison include:
Insight 1: MILMs Cannot Function as Body Surface Health Doctors MedLesion VQA presents
significant challenges for multimodal large language models (MLLMs). The overall accuracy of representative MLLMs on our MedLesion VQA benchmark is below 57% emphasizing the need for implementing real-world visual diagnostic tests. Although many MILMs claim perform physician's 's level, Tab. [2] indicates that even the best MILMM performs notably worse than primary care physicians (by 5%) and significantly worse than expert clinicians (by 17%). The primary reason of incorrect diagnosis are errors in recognizing
8

---

Published as a conference paper ICLR 2026
lesion types, locations, attributes, or relationships-tasks that human doctors perform reliably while the best lesion recognition accuracy for MLLMs is only 49%. Our results from MedLesion VQA show that MLLMs frequently fail in diagnostic tasks and often struggle align with physicians in real clinical settings. These findings underscore the need for caution when employing MLLMs as medical practitioners and highlight the necessity to develop more advanced medical-specific MLLMs.
Insight 2: Textual Capabiliities Can Cause MILMSs to Appear More Competent Than They Are
People often perceive MILMs as highly knowledgeable experts and report positive experiences during question-ane question-and-answer and-answer interactions. However, our LeedLesiiVVAA benchmark suggests that MLLMs seem more competent than they are due to their impressive text generation abilities, even when subjective questions are minimized in MedLesion VQ VQA. A comparison between text-only and vision-text evaluations indicates that "suggestion" scores remain high regardless of the modality (82.4% vs. 81.2% with and without images). The high accuracy of treatment recommendations demonstrates that large language models can generate effective general advice, even without specialized expertise in body health images. In contrast, MI.LMs perform poorly more visually demanding tasks, such as lesion and location recognition. These findings underscore the necessity of comprehensive clinical pipeline evaluations when applying MLLMs in medical contexts.
Insights 3: Performance Improves as Model Size Increases. The results demonstrate a generally positive correlation between model size and per- formance, but with diminishing returns and no- table exceptions. Models under 1B parameters (e.g., S pabilities across all tasks (scores below 0.2), while mid-scale models (IB-1OB) 10B))) like Qwen2-VL-28 and Deepseck-v12-tiny (3.4B) exhibit significant perfor- mance jumps, particularly in recognition and diag- nostic tasks. The GLM-4V-9B model achieves near- state-of-ae fate-offthhearr the-art results, rivaling much larger models with average of 0.465 compared to the 0.309 score of Intern V1.2.5-38B. However, scaling beyond 10B pa- rameters shows inconsistent returns- while Qwen2- VL-72B dominates in most metrics, the VL2.5- 78B underperforms smaller models in key areas like disease diagnosis, suggesting current architectural or training limitations in MLLMs. Generally, closed- source models consist hundreds of billions of parameters and provide the relatively high performance.
lasion adéribute
recoynition
body region recognition
show limited ca-
Spebial
reletion
lesion rasoningg
anddeie pkpineer Quaer
041
lesion recognition
Sugpestion
diseate dioanosis
treatmend
Figure 4: Results of 10 representative MLLMs across the 7 ability dimensions defined in MedLesion VQA.
The Need to Rethink Domain-Specith Models. These models in the medical domain mostly exhibit suboptimal performance compared to general models on similar model-size scale. For example, the comparison between LLaVA-v1.5-7B and 1.-----med-1.--7 highlights the trade-off between spe- cialization and generalization. LLaVA-med-vi v1.5-BB performs 18% worse than ALLVA-v1.55-77 on the MedLesion VQA dataset, yet demonstrates superior performance on VQA-RAD. Similarly, MedGemma-4t also shows worse performance than Intern VL2.5-4B. Reliance supervised fine-tuning (SFT) for domain adaptation. Although SFT is effective, it often leads to overfitting to domain-specific patterns (e.g., radiol- ogy report generation), which may constrain model's ability to generalize to unseen fasks-such as our body-surface-foccee focusse benchmark and even degrade underlying reasoning capabilities.
Insight 4:
9

---

Published as a conference paper at ICLR 2026
4.3 PER-ABILITY ERROR ANALYSIS
Physicians v.s. Models. We provide a general per-ability comparison between physicians and Gemini- 2.5-pro, which has the best performance in MedLessonVQA. As shown in Tab. [ several patterns emerge: (i) physicians consistently outperform Gemini-2.5-pro on visually demanding recognition and reasoning abilities (e.g., lesion and attribute recognition, region localization), especially for location recognition and disease diagnosis; (ii) the gap narrows, and in some cases reverses, on more text-centric abilities such as suggestion & treatment. We believe this quantitative breakdown already sharpens which capabilities most require architectural or training improvements.
Error Cases, We also analyze the error instances sampled from the GPT-4V's predictions. The distribution of these errors illustrated in Fig (a), including lack of knowledge, text misunderstanding, and judgment error, etc. The fundamental deficiency lies in the predicted answer's failure to perform targeted visual analysis. As shown Fig 5 (b), While generically mentioning textbook features of folliculitis and acne, it critically neglected to anatomically map the documented clinical findings. This representational gap between generalized knowledge and case- sess---ecffic application resulted in diagnostic inaccuracy. The specific error cases can be found in Appendix B.2 and B.3.
Lnderstanding
Match Log 0
Lack of knowledge
Firromm the parspective of the Wost! information pratented in the enage, what wartly the naand ciat alout for defintion dagassic of folloster and coos?
judge Error
knowledge
Arran the image. the deferive of fulkaa and zore requies conidration of the tallowin mg feature:: hollou its typicaly presents as red popais in puatuess around hal toliiie and monty rebited bainbeorin weat arre commonly mardeets corredoses, saniee, pustiss, and rodalas oten busd on feee. ctea back where sabscesua glants shunsaer
Sentor cleviiene srowe The potiand preseety
chaakr, ranatting sygoficus dee sweting. sccampanted by mubbi' red popuine- The Aveoeetor: The image shows puetloy of the sho the hexd, fice, and cheeps,
types
with multiple milke sized red round papaler reattane and densely Avtriilated Thess lesinn footuras can serve the beds for dieponnty follculbte and aone
consilered dlapoouss are MoNcwiba and
Fmnister: The predicted arower only goncully desoribed the typial manfesttiors and detrbbtton characterittes at hallouitta and scts, withour accresing the spectt: soual
informaiion preserted in the giver image juac as son sweling on the heat, fio, cheeks. mulipre e mi aed red round popurs, acatiered demsel dassibutec), and comeetly quention
(a) Error distribution
(b) Showcase of an open-ended QA error
Figure 5: Error analysis of GPT-4V on our benchmark. Figure (a) illustrates the eиror analysis: the top-left panel shows the overall distribution of error cases, while the bottom-left panel displays the distribution for the 'Lack of Knowledge' category. The right figure presents an example of an incorrect response to open-ended question.
5 CONCLUSION
In this paper, we propose MedLesionVQA a large-scale and body surface oriented benchmark evaluating the lesion, region, diagnosis, and treatment-related recognition and reasoning ability for medical MLLMs. MedLesion VQA contains 12K body lesion images with expert-level fine-grained annotations of 96 prevalent dermatological diseases, 94 distinct lesion types and 110 body regions. The evaluation dimension of MedL VQA is built on basis of 7 multimodal stepwise visual diagnostic abilities, including lesion recognition, lesion attribute recognition, body region recognition, lesion spatial relation recognition, lesion reasoning, disease diagnosis and suggestion & treatment, which ensure the alignment with the authentic clinic senary. Mainstream MLLMs are evaluated on the benchmarks, and Gemini-2.5-pre has the best score of 56.24. 24. Furthermore, senior and primary physicians invited to answer the questions of benchmark and obtain score of 61. .44 and 73.21, respectively. The results show that there is large improvement for MILMS on the benchmark and indicates significant challenges and medical specialization of the MedLesion RedLesionVOA. VQA.
10

---

Published as a conference paper ICLR 2026
REFERENCES
Dermnet, 2023. https: hwt.eeeereeeemmeeeeeeeeeee...... com/ [2024].
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303. 088744, 2023.
Rawan AlSaad, Alaa Abd-Alrazaq, Sabri Boughorbel, Arfan Ahmed, Max-Antoine Renault, Rafat Damseh, and Javaid Sheikh. Multimodal large language models in health care: applications, challenges, and future outlook. Journal of medical Internet research, 26:e59505, 2024.
Anthropic. Claude 3 model family. httpsss/www.anthrrppi....oo
/wwp::///wwwwoo www.:wwhthrrii....mm com 2025a.
www comm Anthropic. Introducing claude Anthropic News, 2025b. 7356663216777777666633
news /c /clllanna aude-47__aa www.eeddfffoffeeeeeeeeeee
Michela Antonelli, Annika Reinke, Spyridon Bakas, Keyvan Farahani, Annette Kopp-Schneider, Bennett A Landman, Geert Litjens, Bjoern Menze, Olaf Ronneberger, Ronald M Summers, al. The medical segmentation decathlon. Nature communications, 13(1):4128, 2022.
Jinze Bai, Shuai Bai, Shusheng Yang. Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: versatile vision-language model for understanding, localization. Text Reading, and Beyond, 2, 2023.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502. 13923, 2025.
Yonatan Bitton, Hritik Bansal, Jack Hessel, Rulin Shao, Wanrong Zhu, Anas Awadalla, Josh Gardner, Rohan Taori, and Ludwig Schmidt. Visit-bench: A benchmark for lssonnlanguuae instruction following inspired by real-world use. arXiv preprint arXiv:2308 0059066....... 2023.
Junying Chen, Xidong Wang, Ke Ji, Anningzhe Gao, Feng Jiang, Shunian Chen, Hongbo Zhang, Dingjie Song, Wenya Xie, Chuyi Kong, et al. Huatuogpt-ii, one-stage training for medical adaption of llms. arXiv preprint arXiv:231 1tri::2311.0000 1.09774, 2023.
Junying Chen, Chi Gui, Ruyi Ouyang, Anningzhe Gao, Shunian Chen, Guiming Hardy Chen, Xidong Wang, Zhenyang Cai, Ke Ji, Xiang Wan, et Towards injecting medical visual knowledge into multimodal lims at scale. In Proceedings of the 2024 conference on empirical methods in natural language processing, pp. 7346-7370, 2024a.
Junying Chen, Chi Gui, Ruyi Ouyang, Anningzhe Gao. Shunian Chen, Guiming Hardy Chen, Xidong Wang, Ruifei Zhang, Zhenyang Cai, Ke Ji, et al. Huatuogpt-vision towards injecting medical visual knowledge 192200000...... 19280, 2024b.
into multimodal lims scale. arXiv preprint arXiv:2406.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao and Dahua Lin. Sharegpt4v: Improving large multi modti models with better captions. In European Conference on Computer Vision. pp. 387. Springer, 2024c.
Zhe Chen, Weiyun Wang. Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271 2024d.
Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. Science China Information Sciences, 6711))220101, 2024e.
11

---

Published as a conference paper ICLR 2026
Zhe Chen, Jiannan Wu, Wenhai Wang. Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic Visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24185-24198, 2024f.
Alberto Coustasse, Raghav Sarkar, Bukola Abodunde, Brandon J Metzger, and Chelsea M Slater. Use of teledermatology to improve dermatological access in rural areas. Telemedicine and e-Health, 25(1 I): 1022-1032, 2019.
Roxana Daneshjou, Kailas Vodrahalli, Roberto A Novoa, Mclissa Jenkins, Weixin Liang, Veronica Rotemberg, Justin Ko, Susan M Swetter, Elizabeth E Bailey, Olivier Gevaert, et al. Disparities in dermatology ai performance on diverse, curated clinical image set. Science advances, 8(31 ):abbg66111 2022a.
Roxana Daneshjou, Mert Yuksekgonul, Zhuo Ran Cai, Roberto Novoa, and James Y Zou. Skincon: A skin disease dataset densely annotated by domain experts for fine-grained debugging and analysis. Advances in Neural Information Processing Systems, 35:18 15118177-18167 2022b.
Google DeepMind. Gemini 1.5: Unlocking multimodal understanding across millions of tokens, 2024. URL Accessed: 2025-04-30.
https:://arxiz. org httgg/b///// 2203. 0
Ming Ding, Zhuoyi Yang. Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang. et al. Cogview: Mastering text-to-image generation via transformers. Advances in 2021.
neural information processing systems,
Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang. Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, et al. Vlmevalkit: An open-source toolkit for evaluating large multi-modality models. In Proceedings of the 32nd ACM International Conference Multimedia, pp. 11198-11201, 2024.
Andre Esteva, Brett Kuprel, Roberto A Novoa, Justin Ko, Susan M Swetter, Helen M Blau, and Sebastian Thrun. Dermatologistleveel classification of skin cancer with deep neural networks. nature, 542(7639): 115-118, 2017.
Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, Hongning Wang. Jiadai Sun, Jiajie Zhang, Jiale Cheng, Jiayi Gui, Jie Tang, Jing Zhang, Juanzi Li, Lei Zhao, Lindong Wu, Lucen Zhong, Mingdao Liu, Minlie Huang, Peng Zhang, Qinkai Zheng, Rui Lu, Shuaiqi Duan, Shudan Zhang, Shulin Cao, Shuxun Yang, Weng Lam Tam, Wenyi Zhao, Xiao Liu, Xiao Xia, Xiaohan Zhang, Xiaotao Gu, Xin Lv, Xinghan Liu, Xinyi Liu, Xinyue Yang. Xixuan Song, Xunkai Zhang, Yifan An, Yifan Xu, Yilin Niu, Yuantao Yang, Yueyan Li, Yushi Bai, Yuxiao Dong, Zehan Qi. Zhaoyu Wang. Zhen Yang, Zhengxiao Du, Zhenyu Hou, and Zihan Wang. Chatglm: A family of large language models from glm- 1amm-130p 130b to glm-4 all tools, 2024.
Google. Gemini 2.5 pro, 2025. URL https / cloud. google. com wwtp:/eteeeeeeeeeeeeeeeeeeee
generative-ale hateeeaaaaccs..eeeeeeeee docs/ models/ Seedeess/geini//-----1 gemini/2-5--rro Large language model; Capable of handling various modalities such as text, audio, image, and video; Supports a context window of I million tokens.
Matthew Groh, Caleb Harris, Luis Soenksen, Felix Lau, Rachel Han, Acrin Kim, Arash Koochek, and Omar Badri. Evaluating deep neural networks trained on clinical images in dermatology with the fitzpatrick 17k dataset. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1820-1828. 2021.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Iim via reinforcement learning. arXiv preprint arXiv:2501. 122444 2025.
12

---

Published as a conference paper ICLR 2026
Seung Scog Han.
SNU dataset + Quiz. 2019.
doi: 10.6084/m9 (iig00//m..igsshreee.... wig.saar..6544977......
URL https:: com/articles/aataaaaaaaaatccacaa wat.ee/eaaaae/aaeeeeaeesaeeessee SaaesstttSSU_SSSS SNNUUSNUU_UU GNU__SNUMMELAOMAA _ndd__aadd___ Reddit
Gataset_Quizz64443 6454973
Sunan He, Yuxiang Nie, Zhixuan Chen, Zhiyuan Cai, Hongmei Wang, Shu Yang, and Hao Chen. Meddr: Diagnosis-guided bootstrapping for large-scale medical vision-language learning. arXiv e-prints, pp. arXiv-2404, 2024.
Wenyi Hong. Weihan Wang, Ming Ding, Wenmeng Yu, Qingsong Lv, Yan Wang, Yean Cheng, Shiyu Huang, Junhui Ji, Zhao Xue, al. Cogvim2: Visual language models for image and video understanding. arXiv preprint arXiv:2408. 16500, 2024.
Yutao Hu, Tianbin Li, Quanfeng Lu, Wenqi Shao, Junjun He, Yu Qiao, and Ping Luo. Omnimedvga: A new large-scale comprehensive evaluation benchmark for medical Ivlm. In Proceedings of the IERE/CVF Conference Computer Vision and Pattern Recognition, pp. 20170-222211 2024.
Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Ileus Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, et al. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 590-597, 2019.
William D James, Dirk Elston, and Timothy Berger. Andrew's diseases of the skin E-book: clinical dermatology. Elsevier Health Sciences, 2011.
Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. A dataset of clinically generated visual questions and answers about radiology images. Scientific data, 5(1):1-10, 2018.
Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander Rush, Douwe Kiela, al. Obelies: An open web-scale filtered dataset of interleaved image-text documents. Advances in Neural Information Processing Systems, 36: 71683-71702, 2023.
Tony Lee, Haoqin Tu, Chi Heem Wong, Wenhao Zheng. Yiyang Zhou, Yifan Mai, Josselin Roberts, Michihiro Yasunaga, Huaxiu Yao. Cihang Xie, et al. Vhelm: A holistic evaluation of vision language models. Advances in Neural Information Processing Systems, 37:1 40632-140666,, 2024.
Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoi- fung Poon, and Jianfeng Gao. Llava-med: Training large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems, 36:28541-28566 2023a.
Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoi- fung Poon, and Jianfeng Gao. Llava-med: Training a large language-anddvision assistant for biomedicine in one day. arXiv preprint arXiv:2306.00890 2023b.
Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava- next-interleave: Tackling multi-image, video, and 3d in large multimodal models. arXiv preprint arXiv:2407.07895, 2024.
Haodong Li, Xiaofeng Zhang, and Haicheng Qu. Ddfav: Remote sensing large vision language models dataset and evaluation benchmark. Remote Sensing, 17(4):719, 2025.
Hyeonseok Lim, Dongjae Shin, Seohyun Song, Inho Won, Minjun Kim, Junghun Yuk, Haneol Jang, and KyungTaee Lim. Vlr-bench: Multilingual benchmark dataset for langua-la retrieval augmented generation. arXiy preprint arXiv:2412.10155 2024.
13

---

Published as a conference paper ICLR 2026
Tianwei Lin, Wengiao Zhang, Sijing Li, Yugian Yuan, Binhe Yu, Haoyuan Li. Wanggui He, Hao Jiang, Mengze Li, Xiaohui Song, et al. Healthgpt: A medical large vision-language model for unifying comprehension and generation via heterogeneous knowledge adaptation. arXiv preprint arXiv:2502.0988. 2025.
Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang, and Xiao-Ming Wu. Slake: A semantically-labekee
knowledge-enhancee dataset for medical visual question answering. In 2021 IEEE 18th international
symposium on biomedical imaging (ISBI, pp. 1650-1654. IEEE, 2021.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023b.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances neural information processing systems, 36:34892-34916, 2023c.
Junling Liu, Ziming Wang, Qichen Ye, Dading Chong, Peilin Zhou, and Yining Hua. Qilin-med-vl: Towards chinese large vision-language model for general healthcare. arXiv preprint arXiv:2310. 17956, 2023d.
Andrés Marafioti, Orr Zohar, Miquel Farré, Merve Noyan, Elie Bakouch, Pedro Cuenca, Cyril Zakka, Loubna Ben Allal, Anton Lozhkov, Nouamane Tazi, Vaibhav Srivastav, Joshua Lochner, Hugo Larcher, Mathieu Morlon, Lewis Tunstall, Leandro von Werra, and Thomas Wolf. Smolvim: Redefining small and efficient multimodal models. arXiv preprint arXiv:2504.05299, 2025.
Michael Moor, Qian Huang, Shirley Wu, Michihiro Yasunaga, Yash Dalmia, Jure Leskovee, Cyril Zakka, Eduardo Pontes Reis, and Pranav Rajpurkar. Med-flamingo: a multimodal medical few-shot learner. In Machine Learning for Health (MIAH), pp. 353-367. PMLR, 2023.
Sahal Shaji Mullappilly, Mohammed Irfan Kurpath, Sara Pieri, Saeed Yahya Alseiari, Shanavas Cholakkal, Khaled Aldahmani, Fahad Khan, Rao Anwer, Salman Khan, Timothy Baldwin, et al. Bimedix2: Bio- medical expert Imm for diverse medical modalities. arXiv preprint arXiv:2412.07769 2024.
Vishwesh Nath, Wenqi Li, Dong Yang. Andriy Myronenko, Mingxin Zheng, Yao Lu, Zhijian Liu, Hongxu Yin, Yucheng Tang, Pengfei Guo, al. Vila-m3: Enhancing vision-language models with medical expert knowledge. arXiv preprint arXiv:2411.12991 2024.
OpenAl. Gpt-4o, 2024. URL
http:///ccat..oee...ccc com com Large language model; Prompt: "".
OpenAl. Chatgpt (gpt-5 version), 2025. URL
openai (ommaai.comm com/ chat
Sourjyadip Ray, Kushal Gupta, Soumi Kundu, Payal Arvind Kasat, Somak dataset to benchmark the readiness of large vision language models in hospital environments. arXiv preprint 1:Xiv:24400064222 2024.
Aditya, and Pawan Goyal. Ervqa:
Josselin S Roberts, Tony Lee, Chi H Wong, Michihiro Yasunaga Yifan Mai, and Percy Liang. Image2struct: Benchmarking structure extraction for vasion-language models. Advances in Neural Information Processing Systems, 37:115058-115977 2024.
Khaled Saab, Tao Tu, Wei-Hung Weng, Ryutaro Tanno, David Stutz, Ellery Wulezyn, Fan Zhang, Tim Strother, Chunjong Park, Elahe Vedadi, et al. Capabilities of gemini models in medicine. arXiv preprint arXiv:2404. 111124444..116.... 2024.
Andrew Sellergren, Sahar Kazemzadeh, Tiam Jaroensri, Atilla Kiraly, Madeleine Traverse, Timo Kohlberger, Shawn Xu, Fayaz Jamil, Cían Hughes, Charles Lau, et al. Medgemma technical report. arXiv preprint arXiv:2507.05200 2025.
14

---

Published as a conference paper ICLR 2026
Mohammad Shahab Sepehri, Zalan Fabian, Maryam Soltanolkotabi, and Mahdi Soltanolkotabi. Mediconfu- sion: Can you trust your ai radiologist? probing the reliability of multimodal medical foundation models. arXiv preprint atXiv:2449.154777 2024.
Yuanhe Tian, Ruyi Gan, Yan Song, Jiaxing Zhang, and Yongdong Zhang. Chimed-gpt A chinese medical large language model with full training regime and better alignment to human preferences. arXiv preprint S.Ki::2311.000253 2023.
Philipp Tschandl, Christoph Rinner, Zoe Apalla, Giuseppe Argenziano, Noel Codella, Allan Halpern, Monika Janda, Aimilios Lallas, Caterina Longo, Josep Malvehy, et al. Human-computer collaboration for skin cancer recognition. Nature medicine, 1688))1229-21 1234, 2020.
Bingning Wang, Haizhou Zhao, Huozhi Zhou, Liang Song, Mingyu Xu, Wei Cheng, Xiangrong Zeng. Yupeng Zhang, Yuqi Huo, Zecheng Wang. et al. Baichuan-ml: Pushing the medical capability of large language models. arXiv preprint arXiv:2502. 2025.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge. Yang Fan, Kai Dang. Mengfei Du Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: vl: Enhancing vision- vision-language language model's perception of the world at any resolution. arXiv preprint 24.::24499.1299999 2024a.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vsionnlanguage model's perception of the world at any resolution. arXiv preprint arXiv:2409.12199 2024b.
Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Song XiXuan, et al. Cogvlm: Visual expert for pretrained language models. Advances in Neural Information Processing Systems, 37:121475-121499, 2024c.
Abbi Ward, Jimmy Li, Julie Wang, Sriram Lakshminarasimhan, Ashley Carrick, Bilson Campana, Jay Hartford, Pradeep K. Sreenivasaiah. Tiya Tiyasirisokchai, Sunny Virmani, Renee Wong, Yossi Matias, Greg S. Corrado, Dale R. Webster, Margaret Ann Smith, Dawn Siegel, Steven Lin, Justin Ko, Alan Karthikesalingam, Christopher Semturs, and Pooja Rao. Creating an empirical dermatology dataset through crowdsourcing with web search advertisements. JAMA Network Open, 711):2246611--244615, 11 2024. ISSN 2574-3805 doi: 10. 2024. 46615
16.1. URL https:
1001/ http://jamantttrrooooooo
Wenjing Yue Wei Zhu and Xiaoling Wang. Shennong-tcm: traditional chinese medicine large language model. htttss://qithni htt.tuuu..ooommiccceeaeee wwwp/wwwwwwaaeeeee Swwnnong-nnenaee TCM- LLM 2023.
Richard B Weller, Hamish JA Hunter, and Margaret W Mann. Clinical dermatology. John Wiley & Sons, 2014.
Chris Wright and Pauline Reeves. Radbench: benchmarking image interpretation skills. Radiography, 22(2): e131-e136, 2016.
Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. Towards generalist foundation model for radiology by leveraging web-scale 2d&3d medical data. arXiv preprint arXiv:2308 2...:22300.00463, 2023.
Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma. Chengyue Wu, Bingxuan Wang. Zhenda Xie, Yu Wu, Kai Hu, Jiawei Wang. Yaofeng Sun, Yukun Li, Yishi Piao, Kang Guan, Aixin Liu, Xin Xie, Yuxiang You, Kai Dong, Xingkai Yu, Haowei Zhang, Liang Zhao, Yisong Wang, and Chong Ruan. Deepseek-v12: of language models for advanced multimodal understanding, 2024. UBL/hppps//aaxiii.cc///////////
15

---

Published as a conference paper ICLR 2026
Cheng Xu, Xiaofeng Hou, Jiacheng Liu, Chao Li, Tianhao Huang, Xiaozhi Zhu, Mo Niu, Lingyu Sun, Peng Tang, Tonggiao Xu, et al. Mmbench: Benchmarking end-to-end multi-modal dnns and understanding their hardware-softwe implications. In 2023 IEEE International Symposium on Workload Characterization (IISWC), pp. 154-166. IEEE, 2023.
Weiwen Xu, Hou Pong Chan, Long Li, Mahani Aljunied, Ruifeng Yuan, Jianyu Wang, Chenghao Xiao, Guizhen Chen, Chaoqun Liu, Zhaodonghui Li, et al. Lingshu: A generalist foundation model for unified multimodal medical understanding and reasoning. arXiv preprint arXiv:2506.0744 070444 2025.
Siyuan Yan, Ming Hu, Yiwen Jiang, Xieji Li, Hao Fei, Philipp Tschandl, Harald Kittle,, and Zongyuan Ge. Derm Im: A million-scale vision-language dataset aligned with clinical ontology knowledge for dermatology. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12681- 12690, 2025.
Jin Ye, Guoan Wang, Yanjun Li, Zhongying Deng, Wei Li, Tianbin Li, Haodong Duan, Ziyan Huang, Yanzhou A comprehensive multimodal evaluation benchmark towards
Su, Benyou Wang. et al. general medical ai. Advances in Neural Information Processing Systems, 37:94327-94422 2024.
Wen-wai Yim, Yujuan Fu, Zhaoyi Sun, Asma Ben Abacha, Meliha Yetisgen, and Fei Xia. Dermavqa: multilingual visual question answering dataset for dermatology. In International Conference on Medical Image Computing and Computer-Assister Intervention, pp. 209-219. Springer, 2024.
Weihao Yu. Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.0244 022490, 2023.
Weihao Yu, Zhengyuan Yang. Lingfeng Ren, Linjie Li, Jianfeng Wang. Kevin Lin, Chung- Boing-Ching Lin, Zicheng Liu, Lijuan Wang, and Xinchao Wang. Mm-vet v2: A challenging benchmark to evaluate large multimodal models for integrated capabilities. arXiv preprint arXiv:2408.000
2024.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9556-9567 2024.
Juexiao Zhou, Xiaonan He, Liyuan Sun, Jiannan Xu, Xiuying Chen, Yuetan Chu, Longxi Zhou, Xingyu Liao, Bin Zhang, Shawn Afvari, et al. Pre-trained multimodal large language model enhances dermatological diagnosis using skingpt-4. Nature Communications, 15(1):5649, 2024a.
Juexiao Zhou, Liyuan Sun, Yan Xu, Wenbin Liu, Shawn Afvari, Zhongyi Han, Jiaoyan Song, Yongzhi Ji, Xiaonan He, and Xin Gao. Skincap: A multi-modal dermatology dataset annotated with rich medical captions. arxiv preprint 1xXi::2405,18004 2024b.
Fengbin Zhu Ziyang Liu, Xiang Yao Ng, Haohui Wu, Wenjie Wang, Fuli Feng, Chao Wang, Huanbo Luan, and Tat Seng Chua. Mmdocbench: Benchmarking large lasion-language models for fine-grained visual document understanding. arXiv preprint arXiv:2410.21311 2024.
16

---

Published as conference paper ICLR 2026
MedLesionVQA: A Multimodal Benchmark Emulating Clinical Visual Diagnosis for Body Surface Health
Supplementary Materials
A DATASET DETAILS
A.1 DETAILS FOR DATA DISTRIBUTION.
All annotation data are reviewed by another group expert physicians, and the qualified standard is that the recall and precision of the sampled data in each annotation dimension reach 95%, respectively. Among all images obtained through annotation, taking into account the coverage of body region, lesion, and disease, screen out total of 12K images including 10K images with abnormalities and 2K images without abnormalities. The histogram distribution of lesion, disease and body region is shown in Fig. 6 which is used illustrate the annotation information density. Taking the left sub-figure as example, the horizontal axis represents the number of images containing certain lesion, while the vertical axis represents the number of these type of lesions, which indicates that most type of lesions has least 50 images and there is enough lesion annotations and there is less issue of long-tail distribution.
Fig. 7 offers a detailed view of the data distribution for different clinical-oriented abilities. Taking the top-left image an example: This multi-layer ring chart illustrates the distribution of recognition questions in the dataset across four main categories. The outer ring shows the total number of questions for each category: Region Recognition contains 3,986 questions, Attribute Recognition has 3,508 questions, Lesion Recognition includes 3,340 questions, and Spatial Relation comprises 1,133 questions. The inner ring further breaks down each category into different types, including multiple-choice judgment, fill-in-the-Be bheakkk and short-answer questions, highlighting their relative proportions within each category.
Lesion distribution
Disease distribution
Body part distribution
70
35
60
30
Number
Number
8
Number
25
20
15
10
2
500
250
75
50
85
70
200
100
70
25
Frequency
Frequency
Frequency
Figure 6:
Histogram of lesion, disease, and body region distribution of Medl.esion VQA. The left figure shows that most lesions has at least 50 images.
A.2 DETAILS FOR LEXICAL TREES IN ANNOTATION PROTOCOL
We have respectively construct the corresponding lexical trees for part division. The detailed information of the lexical trees are listed below, in Tab. 3 and 4 Visible body parts annotated with the secondary node in the level-2 list, while abnormal body parts are annotated with more refined leaf node the level-2, level-3 level-4 list according to the minimum body division. For example, the leaf node of upper eyelid is in the path (head -> face -> periorbital area -> upper eyelid) at the fourth level. The leaf node anterior neck is the path (neck -> anterior neck) at the second level.
17

---

Published as conference paper ICLR 2026
Spatial
1elation(2133)
Rece connitennte (entten(333 (3340)
(CT97) (CT92) Sinuuuse Koys
Leasion
nattower
lasion reasoning
(E:00
Understanding Tasks Test Dataset
two Anwer170
Recognition Tasks
Made Choueten
abot HAS
Rand
disease diagnosis (1691)
44
Region
Spetial relation(113)
Resognition az)
Leasion
lesion reasonins (244)
Insorunde it
on Validation Dataset
Recognition
Validation Dataset
ECTONNSL Harn
Ragion
disease diagnosis
(1A5)
Figure Data distribution of MedLesion VQA benchmark.
Besides the lexicon body regions, we construct the lexicon list of lesion, disease and lesion attributes, as shown in Tab.
and
Table 3: Lexical tree for body regions.
Level
Level 2
Level top
Level
scalp
temporal region
occipital region
forehead
18
head

---

Published as conference paper ICLR 2026
cheek
perioral area chin
upper eyelid
face
periorbital area
lower eyelid
supraorbital arch
perinasal area nose
postauricular area
anterior neck
neck
lateral neck
posterior neck
breast
submammary area nipple
chest
arcola
back
waist
trunk abdomen
periumbilical area groin
axilla
lateral trunk
iliac region
shoulder
flexor side
upper arm
extensor side
extensor side
elbow
cubital fossa
flexor side
forearm
extensor side
wrist
upper limb
dorsum of hand
palm
interdigital space
hand
thenar space
lateral side of finger
palmar side finger
finger
dorsal side of finger
tip of finger
buttock
medial side
thigh
lateral side
knee
popliteal fossa
extensor side
leg
flexor side
dorsum of foot
heel
lower limb
sole
plantar aspect of foot
foot
19

---

Published as conference paper ICLR 2026
forefoot
first metatarsopha- langeal joint area lateral border of foot
ankle
interdigital
space of
toes
dorsal side of toe
plantar side toe
toe
lateral side of toe tip of toe
nail plate
nail bed
nail
nail root
nail fold
nail groove
anus
perianal skin
labia majora
female external
labia minora
genitalia
clitoris
vaginal orifice
perineum
glans penis
external urethral oni
male external genitalia
fice in male
penis
coronal sulcus
prepuce
penile shaft
scrotum
skin of undetermined
upper limb or lower limb
location
mucosa of undeter-
mined location
A.3 PROMPTS FOR AUTOMATIC CONSTRUCTING QAs
Table 89 systematically outline the task framework for automatically constructing question-answering (QAs). The QA construction prompts for lesion recognition, location recognition, lesion reasoning, spatial relation, disease diagnosis and suggestion treatment are shown Table Lesion attributes contain size, color, shape, distribution, quantity, and boundary. We design QA construction prompts for each attribute, as shown
Table g
A.4 EXPLANATION OF 7 CLINIC-ORIENTED MULITMODAL. ABILITIES
To further illustrate the evaluation dimension of MedI.esion VQA, we give detailed explanation for each evaluated ability, as follows:
Lesion Recognition. Skin lesion an abnormal condition on the surface of body skin. For example, macule lesion represents change in surface color without elevation or depression, while papule lesion is
20

---

gingiva of premolar Published as conference paper ICLR 2026
Table 4: Lexical tree for oral cavity.
Level
Level 2
Level
Level
upper labial mucosa
upper labial frenum lower labial frenum
labial mucosa
lower labial mucosa
oral mucosa
orifice of parotid duct
buccal mucosa
occlusal line of teeth
buccal frenum
mucosa of retromolar
area
tip of tongue
lingual papillae on the tip of tongue
dorsum of tongue
lingual papillae on dorsum of tongue
tongue
root of tongue
lingual papillae on the root of tongue
lateral tongue
border of
lingual papillae on the lateral border of tongue sublingual veins
ventral
surface of
tongue
lingual frenum
sublingual region
sublingual caruncle sublingual fold
hard palate
palate
soft palate
posterior pharyngeal
wall
pharynx
uvula
palatoglossal arch
palatopharyngeal arch
tonsil
tonsil/adenoid
adenoid
upper lip
lower lip
lip
margin upper lip
margin of lower lip
angle mouth
gingiva of central in-
cisor
ingiva
gingiva of lateral in- cisor
gingiva of canine
gingiva of molar
central incisor
lateral incisor
teeth
canine
premolar
molar
21

---

Published as conference paper ICLR 2026
Table 5: Lesion list.
Lesions
macule, patch, papule, plaque, mass, vesicle, bulla, pustule, wheal, alopecia, nevus, scale, scale, crust, fissure, sear, pigmentation, depigmentation, swelling, erosion, ulcer, hypertrophy, breakage, peeling, hypopigmentation, blood blister, excrescence, keratotic papule, keratotic plaque, erythema, striac atrophicac, comedone, maculopapule, fissure, rupture, maceration, excoriation, exudation, dryness, lichenification, thickening of the skin, topical preparation, papule/macule, papule/vesicle, macule/vesicle, patch/plaque, unidentifiable lesion (poor image quality), unidentifiable lesion (diffi- cult classify), unidentifiable lesion (possible physiological nature), opening, reddening, keratin plug, blackhead, whitehead, bleeding, purulent discharge, elevated edge, xerosis capillorum, trichop- tilosis, pili annulati, canities, thickening, atrophy, roughness, onycholysis, absence, longitudinal fissure, longitudinal stripe, melanonychia striata, transverse stripe, punctate depression, unevenness, subungual hemorrhage, color change of the nail plate, alopecia patch, hypotrichosis. absence of hair, receding hairline, loss of eyebrows, sparse eyebrows, loss of eyelashes, nodule, ecchymosis, petechia, striae, pseudomembrane, frenum rupture, recession, groove and fissure, desquamation of tongue coating, thickening of tongue coating, tooth mark, tonsillar hypertrophy, adenoidal hypertrophy
Table 6: Disease list.
Diseases
acne, rosacea, lupus miliaris disseminatus faciei, seborrheic dermatitis, scalp psoriasis, psoriasis, pityriasis rosea, eczema, tinea corporis, folliculitis, androgenetic alopecia, telogen efflluvium, alopecia areata, pscudopelade, trichotillomania, tinea capitis, systemic lupus erythematosus, syphilitic alope- cia, white hair, contact dermatitis, lichen simplex chronicus, tinea manus, tinea pedis, acute eczema, chronic eczema, asteatotic eczema, nummular eczema, pompholyx, stasis dermatitis, auto-sensitive dermatitis, progressive pigmented purpuric dermatosis, atopic dermatitis, scabies, exfoliative keratol- ysis, palmoplantar pustulosis, palmoplantar keratoderma, onychomycosis, onychodystrophy, tinea cruris, malassezia folliculitis, urticaria, urticarial vasculitis, dermatographism, cold contact urticaria, chronic urticaria, chronic spontaneous urticaria, cholinergic urticaria, nevus, skin tag, herpes zoster, herpes simplex, impetigo, varicella, papular urticaria. prurigo, verruca plantaris, corn, callus, ver- ruca vulgaris, verruca filiformis, verruca plana, pruritus cutis, pediculosis, insect bite dermatitis, keratosis pilaris, lichen spinulosus, pityriasis rubra pilaris, post-inflammatory hypopigmentations pityriasis alba, pityriasis versicolor, nevus anemicus, achromic nevus, vitiligo, alopecia, hyper- hidrosis, ichthyosis, scleroderma, molluscum contagiosum, diaper dermatitis, pemphigus, bullous pemphigoi, melasma, freckles, cutaneous candidiasis, furuncle, carbuncle, paronychia, erysipelas, cellulitis, dermatophytosis, lichen planus, basal cell carcinoma, squamous cell carcinoma, malignant melanoma, keloid scar, hypertrophic scar
22

---

Published as a conference paper ICLR 2026
Table 7: Value lists of attributes
Attributes
Value List
Size
needlepoint-sized, sesame-seed-sizenn millet-grain-siiee sice-grain-sized mung- bean-sized, soybean-sized fingernail-size coin-sized, walnut-sized, palm-sized, in diameter, y em cm
Color
white, pinkish white, red, light red, pink, dark red, yellow, purple, dark purple, gray, black, brown, flesh-colored
Shape
subcireular, subelliptical, subspherical, subhemispherical, elongated strip-shaped, irregular, annular, cauliflower-likk spider-like, target-shaped, crab claw-like
Quantity
1,2, 3, few, multiple
Distribution
scattered, dense, cluster-like, symmetrical, zonal, reticular, fused, partially fused, fused into sheet, adjacent, diffuse, localized
Boundary
well-defined, poorly-defined, relatively well-defined
a circumscribed and solid elevation of skin, varying in size. Thus, lesion recognition requires MLLMs the perception ability of visual semantic information of disease images.
Attribute Recognition. Lesion attributes include its color, size, quantity, boundary clarity, spatial distribution, and geometric shape. There is need to describe all attributes but some key attributes for each lesion. The key attributes of bulla lesion include its size, shape, quantity, and spatial distribution, and the reason for ignoring its color attribute is that nearly all bulla has typical color of transparent skin tones. Attribute recognition demands the perception ability of detailed visual information and understanding of general world knowledge.
Location Recognition. Region represents human body regions, such as head, face. ear, hand, foot, etc. We expect medical MILMs recognize body regions like clinical doctors do. Moreover, body region information is related to disease diagnosis because some diseases may frequently occur in certain body regions.
Spatial Relation. Spatial relationship between lesions serves evaluation dimension when there are more than one lesions on affected skin area and can be hint for analysis of primary and secondary lesions. Spatial relation evaluates the complex visual understanding about different lesion entities.
Lesion Reasoning. Lesion reasoning evaluates MI.LMs' capacity to deduce pathological mechanisms from visual information. Furthermore, it also involves the analysis of how primary lesions evolve (e-g., papules progressing pustules in folliculitis) how secondary lesion changes (scaling, scabs) are introduced.
Disease Diagnosis. Disease diagnosis requires MLLMs to exploit both multi-dimensional visual information, including lesion, its attributes, body region, and spatial relationships, and cross-referencing medical knowledge to map clinical manifestations to potential pathologies. Thus, disease diagnosis evaluates the comprehensive visual recognition and reasoning ability of MLLMs.
Suggestion Treatment. Suggestion and treatment recommendations should align with evidence-based medical knowledge while incorporating patient-per personalized context, including lesion severity, attributes, and region, which is reflected lesion images. Thus, this evaluation dimension requires comprehensive multimodal reasoning ability integrating knowledge and visual information.
23

---

Published as a conference paper ICLR 2026
A.5 JUDGE-LLM, PROMPT DESIGN, AND RELIABILITY ANALYSIS
A.5.1 SCORING PROTOCOL AND PROMPT
For the open-ended image-Qe evaluation, we use GPT-4 as a judge-LLM to score model responses. Given a question, its reference (gold) answer, and a model-predicted answer, the judge-LLM assigns score from {0.0, 0.5, 1.0), corresponding to incorrect, partially correct, and fully correct, respectively. The judge instructed to (1) compare the predicted answer with the reference answer under the context of the question, (2) provide a short explanation of the decision, and (3) output the final score in machine-readable form as <result>scores reesutl
The scoring is purely text-based: the judge-ILM only observes the question text, the reference answer, and the model's predicted answer, but not the image itself. Thus, the evaluation probes the semantic correctness of the generated textual answers, and does not depend on any visual capability of the judge-LLM.
A.5.2 RUBRIC AND DISAMBIGUATION RULES
To reduce ambiguity and avoid overly strict grading on surface- form mismatches, we provide the judge-LLM with detailed scoring rubric. The final prompt used in all reported experiments contains the following guidelines in Table TO
Color. Colors are grouped into coarse color families, and scores are assigned as: (i) exactly the same color: 1.0; (ii) same color family but not exactly the same term: 0.5; (iii) different color family: 0.0. For example, "light red" and "dark red" are in the red family, and "purple" and "dark purple" are in the purple family; browns (e.g., "brown", "dark brown") and blacks are grouped into dark-color family.
Shape. Shapes are grouped into coarse categories, with: (i) exactly the same shape: 1.0; (ii) same category (e.g., different but related shapes): 0.5; (iii) different categories: 0.0. For instance, like and ellipse-like descriptions are one category, rectangle- and square-like descriptions are in another, and irregular shapes (e.g.. "bran-like / furfuraceous") are treated separately.
Size. For size descriptions, common-sense consistency is used: (i) fully consistent or equivalent units: 1.0; (ii) close but not identical size: 0.5; (iii) substantially different scale: 0.0.
Lesions / Morphology. Certain clinically equivalent terms are treated as the same category, e.g., "blood scab" and "crust" are both mapped to "scab", and "scale" is treated as equivalent to "desqua- mation".
Diseases Diagnoses. the predicted answer uses standard synonym or an alternative yet clinically equivalent name for the same disease as in the reference answer, the response is considered fully correct (1.0).
These rules explicitly address the disagreement modes that we observed in a preliminary analysis, where the
judge-LLM tended to be overly strict on attributes such as color and size (e.g., treating "pink" vs. "skin tone", "pinpoint" vs. "millimeter" as completely different). After incorporating the above rubric into the prompt, we fix this prompt and use it unchanged for all experiments reported in the main paper and appendix.
A.5.3 AGREEMENT WITH HUMAN EXPERT
To quantify the reliability of GPT-4 a judge-LLM,, we evaluate its agreement with senior physician on all open-ended questions in the test set ( (18,32 image-QA pairs). Both the human expert and the judge-LLM assign scores from (0.0, 0.5, 1.0} using the same rubric.
24

---

Published as conference paper ICLR 2026
Exact-match consistency and average score difference. Let Shuman 0.5, 0.0, and SLLM {0.0, 0.5, 1.0] be the scores for a given sample, and define the absolute score difference
As Shuman SLLMI
(0,0.5, 1.03.
On the 18,344 open-ended QA pairs, the distribution of As is:
As
0
0.5
1.0
Proportion
91.43%
6.45%
2.12%
The average absolute score difference is therefore
As = 0 x 0.9143 + 0.5 x 0.0645 + 1.0 x 0.0212 = 0.053.
We define the consistency rate as 1 As, which yields final consistency of
1 - As = 0.947 = 95%.
other words, GPT-4 exactly matches the human score on over 91% % of the samples, and differs by most one tier (0.5) on the vast majority of the remaining cases.
Queiratic-eeighted Cohen's Kappa. To further assess inter-rater reliability, we compute the quadratic- weighted Cohen's Kappa Kw between the human expert and GPT-4 the same set of open-ended questions. Using the standard quadratic weighting scheme for ordinal categories, we obtain
Piw 0.8825,
which is typically interpreted as high or "almost perfect" agreement between the judge-LI.M and the human expert.
Empirical reliability. The high exact-match rate (95%), the very small average absolute score difference (As 0.053), and the queihet Cohen's 's Kappa of fu; 0.8825 demonstrate that GPT-4's scores closely track those a senior physician. This indicates that, despite being a proprietary model, the judge-LLM behaves stable and reliable proxy for expert assessment in setting.
MORE RESULTS
In this section, present the comprehensive experimental results, followed by detailed case study analyzing representative examples of model outputs.
B. EVALUATION OF FEW-SHOT AND COT PROMPTING
Our primary evaluation is conducted in "zero-shot" setting, which aligns with the standard practice of prominent benchmarks such as MMBench (Xu et al. 2023). OmniMedVQA (Hu et al.) (024), and MMMU (Yue al. 2024). This approach effectively assesses the model's intrinsic, of capabilities. To further explore the upper bounds of the model's performance and its adaptability, we also conducted supplementary experiments, incorporating few-shot and CoT prompting. The results are presented in Tab. I1 The following results show that there is an increasing trend but significant differences.
B.2 ERROR ANALYSIS
For a comprehensive understanding of the model's operational strengths and weaknesses, we meticulously examined error instances sampled from GPT-4''s predictions. The distribution of these errors is illustrated
25

---

Published as conference paper ICLR 2026
Fig (a). The fundamental deficiency lies in the predicted answer's failure perform targeted visual analysis. As shown in Fig (b), While generically mentioning textbook features of folliculitis and acne, it critically neglected anatomically map the documented clinical findings. This representational gap between generalized knowledge and case-specific application resulted in diagnostic inaccuracy, In addition, Current models lack explicit medical concept grounding. meaning visual features (e.g., "millet-sized papules") are not rigorously verified against diagnostic criteria (e.g., folliculitis vs. acne). If the model has low confidence in its answer, may either refuse to respond or provide an irrelevant reply.
B.3 ERROR CASES
Judge Error. As shown in Fig. 12 the judgment model incorrectly identifies a correct answer erroneous. In other words, the judge model mistakenly flags valid response erroneous. This type of error arises during the assessment phase rather than during the answer generation phase. and then potentially affects the overall accuracy of the evaluation by unfairly penalizing correct answers.
Text Misunderstanding, As shown in Fig. TT GPT-4V fails correctly understand the question and generate erroneous answers. This indicates that the model struggles with accurately interpreting the semantic meaning or the specific intent behind the input question. This misunderstanding can lead responses that are irrelevant, partially correct, or completely incorrect, highlighting limitations in the natural language comprehension capabilities of the model in this context.
Reject to answer. As shown in Fig. 8, GPT-4V outputs irrelevant responses declines to answer certain questions. This behavior suggests that the model may encounter difficulties in comprehending complex or ambiguous queries, leading either to produce answers unrelated to the question to withhold response altogether.
Lack of knowledge. We further categorize Lack of Knowledge into three enror types (as shown in Fig. (a):
Category Confusion, Perceptual Error, and Terminology Precision. 1) Category Confusion: The model fails to correctly distinguish between categories hierarchies of concepts, leading misclassification illogical associations (Fig. 9). 2) Perceptual Error: The model struggles with spatial, physical, or contextual reasoning, resulting in implausible descriptions interpretations of real-world scenarios (Fig. [Op. 3) Terminology Precision. The model uses imprecise, overly broad, or incorrect terms, especially in technical domains (Fig. 13
LIMITATIONS AND POTENTIAL RISKS.
Our proposed MedLesion is the first clinically-orientedd benchmark designed assess whether current state-of-the-art MLLMS can function similarly to real physicians and to guide the development of future MLLMs. However, publicly releasing the dataset may lead to leakage. potentially compromising its future effectiveness as gold-standard benchmark.
26

---

Published as conference paper ICLR 2026
X
Error: Reject to Answer
Question:
Based on the visual information presented in the images, why can definitive diagnosis fungal skin disease (Malassezia folliculitis) be made?
Ground Truth Answer:
The images show multiple pinpoint-torrriierrriii sized, flesh-colored to pink, roundish papules the neck and chest, with secondary erythematous manifestations These characteristics are consistent with the clinical presentation of Malassezia folliculitis, thus confirming the diagnosis.
Model Prediction:
X Visual images alone are insufficient for diagnosis- Requires clinical history and lab tests
Scoring Breakdown:
Match Log:
Fundamental Misalignment:
Expected: Justification of visual diagnosis
Provided: Refutation of visual diagnosis
Composite Score:
<result>0-0<rresutt
Figure 8: An example of reject to answer.
27

---

Published as conference paper ICLR 2026
Table Automatic QA constructing prompts
Ability Category
Prompt Description
Task Description: You will be provided with a piece of medical information related to a picture. Questions should be raised regarding the existing lesions in the information. Assume that all the information is sourced from the picture, and the recognition ability of the model for the lesions shown in the picture needs to be examined. The model is required to answer with the lesion terms the answers. Medical Information: {
Lesion Recognition
Lesions:
Task Description: You will be provided with a piece of medical information related picture. Please pose questions regarding the reasoning process for lesion identification, assuming all information is derived from the image. The questions should assess the model's s ability recognize lesions based attribute information and other key information obtained from the image in the reasoning process
Lesion Reasoning
Medical Information: ()
Lesions: ()
Task Description:
You will be provided with a piece of medical information related to picture. Please pose questions regarding the spatial relationships between lesions, assessing the model's ability to understand the spatial relationships among multiple lesions in the image. The model should respond with related terms the answer.
Spatial Relation
Medical Information: {
Spatial relationship: ))
You will be provided with a piece of medical information related picture. Provide you with a piece of medical information for picture. Raise questions about the reasoning process of making diagnosis in it, examine the reasoning ability of the model to make disease diagnosis based the information obtained from the picture, and set the key information in the diagnostic process as the tested points.
Task Description:
Disease Diagnosis
Medical Information: {
Diagnostic reasoning: )
You will be provided with a piece of medical information related to picture. Please pose questions regarding the location (body region) where the lesion appears, assuming all information is derived from the image. The questions should assess the model's 's ability recognize the location of the lesion area in the image, and the model needs to respond with body regions as the answer. Medical Information:
Task Description:
Location Recognition
Regions: ()
You will be provided with a piece of medical information related picture. Please pose questions regarding the content of suggestion treatment, assessing the model's medical knowledge ability to make such suggestions based on information obtained from the image. Medical Information: {
Task Description:
Suggestion Treatment
Suggestions: (}
28

---

Published as conference paper ICLR 2026
Table 9: Automatic QA constructing prompts for lesion attribute
Subtask
Prompt Description
Task Description: You are provided with a paragraph of medical exam point information and supplementary medical information about an image. Please pose questions regarding the size of the lesion, assessing the model's ability to identify the size attribule of the lesion in the image. The model should respond with size terms the answer. Add constraints in the question stem to specify whether the answer should use analogy (e.g., "pinpoint-sized", "grain-sized")) or specific measurements (in mm cm). Medical Exam Point Information: Attributc Size: {
Lesion Size
Task Description: You are provided with a paragraph of medical exam point information and supplementary medical information about an image. Please pose questions regarding the color of the lesion, assessing the model's ability identify the color attribute of the lesion in the image. The model should respond with color terms as the answer. If the lesion term includes color information, do not reveal the color details in the question.
Lesion Color
Medical Exam Point Information: Attributc Color: {
Task Description: You are provided with a paragraph of medical exam point information and supplementary medical information about an image. Please pose questions regarding the shape of the lesion, assessing the model's s ability to identify the shape attribute of the lesion in the image. The model should respond with shape terms the answer. Medical Exam Point Information: Attribute Shape: ()
Lesion Shape
Task Description: You provided with paragraph of medical exam point information and supplementary medical information about an image. Please pose questions regarding the distribution of the lesion, assessing the model's ability to identify the distribution attribute of the lesion in the image. The model should respond with distribution terms as the answer.
Lesion Distribution
Medical Exam Point Information: Attribute Distribution: ()
Task Description: You provided with paragraph of medical cxam point information and supplementary medical information about an image. Please pose questions regarding the quantity the lesion, assessing the model's ability to identify the quantity attribute of the lesion in the image. The model should respond with quantity terms as the answer. If the description of the quantity is vague, ignore the question type requirements and only construct multiple-choice questions to enable the model answer more definitively.
Lesion Quantity
Medical Exam Point Information: Attribute Quantity: ()
Task Description: You are provided with a paragraph medical exam point information and supplementary medical information about image. Please pose questions regarding the boundary of the lesion, assessing the model's ability to identify the boundary attribute of the lesion in the image. The model should respond with boundary terms the answer.
Lesion Boundary
Medical Exam Point Information: Attribute Boundary: {}
29

---

Published as conference paper ICLR 2026
Section
Content
Overall instruction
Based on the question, the correct answer, and the predicted answer, evaluate the correctness score of the predicted answer.
Evaluation method
Decide whether the predicted answer correctly answers the question, provide scoring details, and finally give am overall score.
The supplementary scoring rules below specify the scoring standards for different
types of questions, and are used to handle situations where the expressions in the predicted answer and the reference answer not exactly the same.
The score must be one the following values: 0.0 (completely incorrect), 0.5, or 1.0 (completely correct).
Output the final score using the format <tesulttscoree///esuul for example:
cresult>0.r 0rtpoltt...x/reeee.e.e 5</result>.
Question and answers
Question: (question)
Reference answer: greterence_anseer
Predicted answer: jpredicted_answer.
Supplementary
scoring
Colors are unified by color families, with explicit scoring rules. Same color: 1.0; same color family: 0.5; different color families: 0.
rules: Color
Red family: red-related colors such red, light red, dark red, deep red, etc.; purple- related colors such purple, dark purple. deep purple, purplish red,
Dark family: brown-related colors such brown, dark brown, tan, light tan, elc.. and black-related colors such as black.
Supplementary rules: Shape
scoring
Shapes are categorized, with explicit scoring rules. Exactly the same: 1.0; same category: 0.5; different categories: O.
Circular category: round-like ellipse-like Rectangular category: rectangle, square, quadrangular, Irregular category: bran-like (furfuraceous) shapes.
Supplementary rules: Size
scoring
If comparative descriptions or concrete numeric values correspond to a common-sense similar size, give 1.0; if they are close, give 0.5; if the gap is too large, give 0. "Blood scab" and "crust" treated equivalent to "scab": "scale" is treated as equivalent "desquamation".
Supplementary
scoring rules: Lesions
Supplementary
scoring
answer a synonym of the same discase, should be regarded as correct. answer.
rules: Discases
Table 10: English translation of the judge-LLM prompt used for open-ended scoring.
30

---

Published as conference paper ICLR 2026
Table 11: Evaluation of Few-shot and CoT Prompting
Model
AVG val
Lesion
Loc.
Attr.
Rela.
Diag.
Sugg.
Senior physicians
0.7321 0.6144
0.6826 0.5932
0.7583
0.7046
0.7102
0.7313
0.8574
Primary physicians
0.6218
0.5203
0.6336
0.6258
0.8162
GPT-4.1
(Zero-shot)
0.5276
0.4487
0.5303
0.4308
0.6401
0.4879
0.8304
(I-shot)
0.5526
0.4791
0.4623
0.4700
0.6525
0.6064
0.8634
(3-shot)
0.5582
0.4808
0.5338
0.4614
0.6260
0.5477
0.8615
(5-shot)
0.5536
0.4332
0.5404
0,4682
0.6566
0.5666
0.8565
(CoT)
0.5377
0,4235
0.5706
0,4221
0.6348
0.4868
0.8245
25mini-2.5 Pro
(Zero-shot)
0.5479
0.4435
0.5052
0.4269
0.6516
0.6368
0.8716
(1-shot)
0.5506
0.5018
0.4660
0.4540
0.6890
0.5856
0.8694
(3-shot)
0.5550
0.4774
0.5138
0.4646
0.6017
0.5661
0.8398
(5-shot)
0.5588
0.4775
0.5256
0.4840
0.6404
0.4415
0.5573
(CoT)
0.5441
0.4469
0.5360
0.4195
0.6254
0.6186
0.8378
31

---

Published as conference paper ICLR 2026
Error: Category Confusion
X
Question:
The diseases diagnosed in the image include eczema and
Ground Truth Answer:
J contact dermatitis
Model Prediction: skin infection
Scoring Breakdown:
Match Log:
Judgment: The predicted answer is incorrect- The question specifically requested identification of another diagnosis besides eczema evident in the image, with "contact dermatitis" being the correct "Skin infection" does not fulfill the required diagnostic precision
Composite Score: <result>O-0<rresule
Figure 9; An example of category confusion.
32

---

Published as conference paper ICLR 2026
Error: Perceptual Error
X
Question:
The key lesion confirming melanocytic nevus in the image is
Ground Truth Answer:
A rice-grain-sien sized black nevus
Model Prediction:
Pigmentation
Scoring Breakdown: Match Log:
Judge: The predicted answer only generally mentioned pigment deposition and did not explicitly point out the specific key lesion as in the correct answer, which states "a millet-sized black nevus," thus failing correctly answer the question Scoring details: The prediction did not provide an accurate and specific description of the "key lesion for diagnosing pigmented nevus in the image" but instead gave broad concept, so the score is 0-0-
Composite Score: Tresult>0-0</resslle
Figure 10: An example of perceptual error.
33

---

Published as conference paper ICLR 2026
Error: Text Misunderstanding
X
Question:
From the perspective of the visual symptom information presented in the image, what exactly allows for definitive diagnosis of Fungal skin disease-tinea cruris?
Ground Truth Answer:
The diagnosis is based on lesions including flat macules and raised papules, specifically pigmented macules- The macules numerous, dark red color, and round, near-oval, or irregular in shope, with stattered distribution- Model Prediction:
From the perspective of the visual symptom information presented in the image, the diagnosis of skin diseases requires consideration of the lesion's morphology, distribution, color, presence of exudate or crusting, as well as the patient's medical history and symptoms
Scoring Breakdown:
Match Log:
Judge: The predicted answer only generally mentioned pigment deposition and did explicitly point out the specific key lesion as the correct answer, which states "a millet-sized black nevus," thus failing to correctly answer the question Scoring details: The prediction did not provide accurate and specific description of the "key lesion for diagnosing pigmented nevus in the image" but instead gave a broad concept, the score 0-0-
Composite Score: <result>0-0< -0ehsesutt-///tt>>>
Figure 11: An example of text misunderstanding error.
34

---

Published as conference paper ICLR 2026
Error: Judge Error
X
Question:
The size of alopecia areata patch in the image cm
cm
Ground Truth Answer:
V 5 7-5
Model Prediction:
X 4
Scoring Breakdown:
Match Log:
The predicted values for both dimensions deviate from the correct measurements-
No partial credit awarded as the answer entirely incorrect:
Composite Score: <result>O--<<rresult>>
Comment:
Predicted dimensions (4 x 2 cm) are within <20% error margin of correct measurements (5 x 7-5 cm):
Length: (20% under)
Width: 1.5 cm (clinically negligible alopecia areata assessment).
35
Figure 12: An example of text understanding error.

---

Published as conference paper ICLR 2026
Error: Terminology Precision
Question:
The lesion appearing at the top of the image
Ground Truth Answer:
/ Desquamation
Model Prediction:
dandruff:
Scoring Breakdown:
Match Log:
judge Scoring details: "Dandruff' is essentially the desquamation of the scalp- In terms of meaning, the predicted answer and the correct answer refer to the same thing, but "desquamation"
more professional, medical term, while "dandruff" common colloquial expression Therefore, the predicted answer is generally correct but lacks precision and professionalism
Composite Score: <result>0-7<rreesltt>
36 Figure 13: example terminology precision error.


Reviews:
[{"review_id": "VzIoOT3eQM", "reviewer": "Reviewer_noDT", "summary": "This paper introduces MedLesionVQA, a multimodal benchmark explicitly designed to evaluate multimodal large language models (MLLMs) in body-surface health diagnosis. Unlike existing datasets focusing on isolated disease classification or lesion recognition, MedLesionVQA covers  seven clinical ability dimensions.\nThe dataset includes 12K ethically collected volunteer images and 19K expert-verified question–answer pairs, covering 94 lesion types, 96 diseases, and 110 body regions. Over 20 MLLMs, including GPT-5, Gemini 2.5 Pro, and Qwen2.5-VL-72B, are evaluated, alongside human baselines. Results show that the best-performing MLLM (Gemini 2.5 Pro) achieves 56.2% accuracy, lagging behind primary physicians (61.4%) and senior physicians (73.2%), indicating persistent gaps between current MLLMs and clinical reasoning capabilities.", "questions": "See above.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The benchmark is well-motivated and methodologically thorough, with a clear focus on aligning AI evaluation with authentic clinical workflows. The data collection process is commendably rigorous: recruiting real volunteers under ethical review and performing multi-stage expert validation with high inter-annotator reliability. The seven diagnostic categories mirror the stepwise reasoning of real physicians, offering a structured and interpretable assessment framework. The inclusion of both open- and closed-source MLLMs, together with human baselines, provides a well-rounded evaluation. The paper is clearly written, integrates comprehensive error analysis, and successfully highlights key challenges in general-purpose MLLMs’ ability to handle fine-grained lesion understanding and multimodal reasoning.", "weaknesses": "The benchmark’s contribution primarily lies in data curation and clinical alignment rather than introducing a new evaluation paradigm. \nThe evaluation setup, though comprehensive, leans heavily on performance reporting without deeper interpretability or diagnostic insights. For instance, while the authors provide category-wise breakdowns, the analysis stops short of identifying failure modes beyond recognition vs. reasoning. The results are descriptive but not analytical: there is little exploration of why models fail (e.g., domain shift, visual ambiguity, prompt misalignment) or how these findings could inform model development.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wHhUekPDn3", "reviewer": "Reviewer_ayvc", "summary": "This paper introduces MedLesionVQA, a multimodal benchmark designed to test how well AI systems can perform medical visual diagnosis for body surface conditions such as skin, oral, and nail diseases. It includes over 12k volunteer images and VQA pairs that cover seven diagnostic steps including lesion recognition to treatment recommendation. The dataset is curated with annotations across several lesion types, diseases, and body regions, reflecting real clinical workflows validated by clinical experts. The authors evaluate around 20 MLLMs and find that the best model achieves 56.2% accuracy, still well below physician performance. This work provides interesting insights evaluating MLLMs for body surface diagnosis.", "questions": "Please address the above weaknesses. \n- What instructions were given to the clinical experts for verification of the samples?\n- Please include the clinical lexicon tree in the appendix.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The authors experimentally show limitations of current MLLMs by comparing them against human physicians across realistic diagnostic tasks, providing quantitative evidence that even the best models are behind expert clinicians.\n- The paper offers valuable empirical analysis on scaling and specialization trade-offs, showing through systematic evaluation that while larger models generally perform better, domain specific fine-tuning can sometimes reduce performance.", "weaknesses": "- The paper does not disclose information about the diversity of participants involved in the data collection process. Metrics such as the country of origin and age groups of participants would help assess whether the benchmark is truly diverse or if its biased to any particular demographic or target group. \n- Need to include comparisons about other Medical MLLMs like – Google’s MedGemma [1], Huatuogpt [2], Bimedix2 [3]. This would help us understand if domain specific medical instruction tuning affects the model’s generalizability and how these models compare on MedLesionVQA.\n- The Appendix and the supplementary material section are missing from the submission. The authors make multiple references to it in the main paper.  Detailed prompts for the evaluation framework are missing. \n- In Figure 1 Suggestion and Treatment section “list at least two topical anti-infective drugs?”. Questions like these would help models respond without properly analyzing the image. This reduces the credibility of the benchmark. \n\n[1] *Sellergren, Andrew, et al. \"Medgemma technical report.\" arXiv preprint arXiv:2507.05201 (2025).*\n\n[2] *Chen, Junying, et al. \"Huatuogpt-vision, towards injecting medical visual knowledge into multimodal llms at scale.\" arXiv preprint arXiv:2406.19280 (2024).*\n\n[3] *Mullappilly, Sahal Shaji, et al. \"Bimedix2: Bio-medical expert lmm for diverse medical modalities.\" arXiv preprint arXiv:2412.07769 (2024).*", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UWUMGmQhXp", "reviewer": "Reviewer_UgBg", "summary": "The paper introduces MedLesionVQA, a multimodal benchmark aimed at emulating the real clinical visual diagnostic workflow for body-surface health (dermatology, dentistry, surgery). It comprises ~12K in-house volunteer images and ~19K expert-verified QA pairs, annotated across 94 lesion types, 96 diseases, and over 110 body regions, and organized into seven stepwise abilities: lesion recognition, attribute recognition, location recognition, spatial relation, lesion reasoning, disease diagnosis, and suggestion & treatment. The authors propose an automated scoring pipeline for MCQ and open-ended items (with a judge-LLM) and compare over 20 MLLMs against primary and senior physician baselines, showing a persistent performance gap.", "questions": "Will the dataset be made publicly available?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper provides a comprehensive benchmark of over 20 general-domain MLLMs and physicians over fine-grained body-surface VQA tasks, with concrete findings on the specialized capabilities of these models.\n- The dataset is of good quality with manual quality check, covering a broad range of cases, question types, body-region ontologies, and lesion attributes, supporting comprehensive assessment.\n- The writing highlights practical insights that can guide future model design and training.", "weaknesses": "**1. About the queries solvable by text-only LLM**\n\nI am curious why the authors do not drop VQA items that can be easily answered by a text-only LLM, which indicates that the question itself can be hacked without using the visual evidence and should not be included for VQA benchmarking. This is a common practice to avoid language shortcuts in curating VQA benchmarks [1--3].\n\n[1] MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark (ACL 2025)\n\n[2] PMC-VQA: Development of a large-scale medical visual question-answering dataset (Communication Medicine 2024)\n\n[3] MicroVQA: A Multimodal Reasoning Benchmark for Microscopy-Based Scientific Research (CVPR 2025)\n\n\n**2. About evaluation of medical-domain models**\n\nCurrently the benchmark only assesses general-domain MLLMs. It would provide more insights if including medical-domain [4-6] or dermatology-specialized MLLMs [7]. \n\n[4] MedGemma Technical Report (arxiv 2025)\n\n[5] HuatuoGPT-vision, towards injecting medical visual knowledge into multimodal llms at scale (EMNLP 2024)\n\n[6] GMAI-VL & GMAI-VL-5.5M: A Large Vision-Language Model and A Comprehensive Multimodal Dataset Towards General Medical AI (arxiv 2024)\n\n[7] Pre-trained Multimodal Large Language Model Enhances Dermatological Diagnosis using SkinGPT-4 (Nature Communications, 2024)\n\n\n**3. About more details** \n- The paper notes large gaps in visually demanding tasks. I am wondering if there are per-ability error taxonomies for physicians versus models. That would sharpen which capabilities (e.g., boundary/size/color perception vs. region localization) most need architectural or training changes.\n- The authors mention that they use prompt tweaks for the open-ended scoring based on GPT-4 after observing disagreements (e.g., color/size strictness). This introduces circularity and instability: results hinge on a proprietary model’s behavior and prompt details. There seems no rigorous inter-rater reliability report (e.g., Cohen’s κ) between physicians and the judge-LLM. It seems that the authors did not upload the appendix; there is no content at the end of the PDF nor supplementary material.\n\n\n**4. Discussion with highly related works is missing**. For example, DermVQA [8] and Derm1M [9].\n\n[8] DermaVQA: A Multilingual Visual Question Answering Dataset for Dermatology (MICCAI 2024)\n\n[9] Derm1M: A Million‑Scale Vision‑Language Dataset Aligned with Clinical Ontology Knowledge for Dermatology (ICCV 2025)\n\n\n**5. Question on the LLM-based scoring**\n\nThe use of commercial proprietary LLMs (GPT-4) might entail model/version drift risk since neither the API nor any specific snapshot is guaranteed to be available forever. Given this, I am wondering whether open-source models (e.g. Qwen series) are able to serve this and how the evaluation results will vary (e.g., will the scores differ drastically, or will there be any bias?). \n\n---\n\nMinor issues: \n- The “multiple-choice question” (selecting one single answer from multiple choices) in Line 277 seems to be “multiple-response question” (selecting all correct answers).\n- The first sentence in the Introduction section is a strong assertion (“Taking a photo and consulting multimodal large language models has become a main approach for addressing body surface health concerns”). I am not sure if this is the case for the global world. Any references/statistics to support such solution as “a main approach”?\n- Most of the references in this manuscript should use “\\citep” with parentheses to avoid confusion with the main texts.\n- In Figure 1, upper right part, “Leision Recognition” → “Lesion Recognition”.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "COa4WNYL0w", "reviewer": "Reviewer_PsWv", "summary": "This paper introduces a novel benchmark dataset for evaluating the capabilities of multimodal LLMs on diagnosing body-surface conditions. Data has been collected from in-house volunteers and the derived question-answer pairs are reviewed by medical professionals. The paper evaluates state-of-the-art models, including both proprietary and open-source models, and demonstrates that current AI techniques fall far behind senior professionals on diagnosing body-surface conditions.", "questions": "- Has there been an overlap between the experts creating the dataset and those solving the benchmark for the reported numbers?\n- Have authors observed consistent failure modes on the benchmark? If so, what is the take-away in terms of future directions?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This work contributes valuable new data collected in-house that can be leveraged to fairly evaluate existing techniques as it is guaranteed that such models are not trained on the dataset. \n\n- The benchmark is generated based on physician annotated data, and all QA pairs are manually reviewed. This provides solid guarantees on its reliability and correctness.\n\n- The paper evaluates human performance on the benchmark as well, thus we can gauge the gap between existing models and medical professionals.", "weaknesses": "- It is unclear if the physicians participating in annotation and QA verification overlap with the medical professionals asked to solve the benchmark for human-performance baseline. If there is an overlap, then I am doubtful about the methodology as the human performance may be inflated.\n\n- The part about automatic scoring needs more clarification. For multiple-choice evaluation, why cannot models simply output the selected answer option, which is then compared to the ground truth?\n\n- It would be great to have more analysis on *how* the models fail on the benchmark, leading to a discussion on how to improve current models to close the gap with senior medical professionals.\n\nMinor:\n- Typos on lines 198, 392.\n- The Appendix is missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)