M3CoTBench: Benchmark Chain-of-Thought of MLLMs in Medical Image Understanding
Authors: Juntao Jiang
Jiangning Zhang
Yali bi
BAI Jinsheng
Weixuan Liu
Weiwei Jin
Zhucun Xue
Yong Liu
Xiaobin Hu
Shuicheng YAN
Venue: ICLR 2026 Poster (2026)
Keywords: Chain-of-Thought
Multimodal Large Language Models
M3CoTBench
Benchmark
Primary area: datasets and benchmarks

Abstract: Chain-of-Thought (CoT) reasoning has proven effective in enhancing large language models by encouraging step-by-step intermediate reasoning, and recent advances have extended this paradigm to Multimodal Large Language Models (MLLMs). In the medical domain, where diagnostic decisions depend on nuanced visual cues and sequential reasoning, CoT aligns naturally with clinical thinking processes. However, current benchmarks for medical image understanding generally focus on the final answer while ignoring the reasoning path. An opaque process lacks reliable bases for judgment, making it difficult to assist doctors in diagnosis. 
To address this gap, we introduce a new M3CoTBench benchmark specifically designed to evaluate the correctness, efficiency, impact, and consistency of CoT reasoning in medical image understanding. M3CoTBench features  (1) a diverse, multi-level difficulty dataset covering 24 examination types, (2) 13 varying-difficulty tasks,  (3) a suite of CoT-specific evaluation metrics (correctness, efficiency, impact, and consistency) tailored to clinical reasoning,  and (4) a performance analysis of multiple MLLMs. M3CoTBench systematically evaluates CoT reasoning across diverse medical imaging tasks, revealing current limitations of MLLMs in generating reliable and clinically interpretable reasoning, and aims to foster the development of transparent, trustworthy, and diagnostically accurate AI systems for healthcare.

Published as conference paper at ICLR 2026
M3COTBENCH: BENCHMARK CHAIN-OF-THOUGHH OF MLLMS IN MEDICAL IMAGE UNDERSTANDING
Juntao Jiang" Jiangning Zhang'; Yali Bi', Jinsheng Bai', Zhucun Xue', Yong Liu't Xiaobin Hu?, Shuicheng Yan
Weixuan Liu', Weiwei Jin',
Zhefiiang University, University of Science and Technology of China, East China Normal University,
Zhejiang Provincial People's s Hospital, 6 National University of Singapore
ABSTRACT
Chain-of-Thoupe (CoT) reasoning has proven effective in enhancing large language models by encouraging step-by-step intermediate reasoning, and recent advances have extended this paradigm Multimodal Large Language Models (MLLMs). In the medi- domain, where diagnostic decisions depend on nuanced visual cues and sequential reasoning, CoT aligns naturally with clinical thinking processes. However, current benchmarks for medical image understanding generally focus on the final answer while ignoring the reasoning path. Such opaque reasoning processes lack reliable bases for judgment, making it difficult to assist doctors in diagnosis. To address this gap, we intro- duce new M3CoTBench benchmark specifically designed evaluate the correctness, efficiency, impact, and consistency of CoT reasoning in medical image understanding. M3CoTBench features 1) a diverse, multi-level difficulty dataset covering 24 examina- tion types, 2) 13 varying-difficuly tasks, 3) suite of CoT-specific evaluation metries (correctness, efficiency, impact, and consistency) tailored to clinical reasoning, and 4) a performance analysis of multiple MI..Ms. M3CoTBench systematically evaluates Co' reasoning across diverse medical imaging tasks, revealing current limitations of MLLMs generating reliable and clinically interpretable reasoning. and aims foster the development of transparent, trustworthy, and diagnostically accurate Al systems for healthcare. Project page https//juntaojjanggawi,
INTRODUCTION
In recent years, Chain-of- Thouuhh (CoT) reasoning has proven to be a transformative mechanism in enhancing the problem-solving capabilities of Large Language Models (LLMs) (Chu et al., 2024). By generating intermediate reasoning steps before arriving at final answer, CoT improves transparency and structured decision-makine in LLMs. Notable advancements include models like OpenAl's ol (OpenAI, 2024b) and 03-mini (OpenAI, 2025c), which exhibit consistent, step-by-step logical reasoning across multi-turn interactions, and DeepSeek-R1 (DeepSeek-AL al., 2025) that excels at decomposing complex tasks into fine-grained subtasks. Building on these successes, researchers have extended CoT to Multimodal Large I Language Models (MLLMs) (Wang al., 2025b), enabling joint processing of multiple modalities. Multimodal CoT frameworks now integrate visual and textual evidence into coherent multi-step explana- tions, with methods like Chain-of-So Sfot (Liu et al., 2024b). TextCoT (Luan et al., 2024), and DCoT (Jia et al., 2024) emphasizing region-of-interest analysis. Recent breakthroughs, such as OpenAl's I''s o3 (OpenAI, 2024c) model, further demonstrate CoT's potential for image-based reasoning, while applications in healthcare, robotics, and autonomous driving highlight its versatility across domains.
In medical MILMS, CoT reasoning is uniquely critical due to the complexity of medical image interpreta- tion (Liu et al., 2024a). Clinicians rely on systematic diagnostic processes that involve iterative observation, verification against key features, and knowledge- based refinement. Explicit reasoning chains are essential to ensure safety, trustworthiness, and alignment with clinical guidelines. However, current benchmarks for medical image understanding focus solely on final-answer accuracy, neglecting the quality of intermediate reasoning steps (Wu al., 2024; Ye et al.. 2024: Hu et al., 2024). For instance, state-of-the art medical MI.MM benchmarks evaluate visual question answering (VQA) performance without assessing how or
"qqual Contribution Corresponding Author

---

Published as conference paper at ICLR 2026
why a model arrives answer. This gap limits the development of clinically reliable Al systems, as two models could produce identical answers through fundamentally flawed or incomparable reasoning paths. Such a lack of scrutiny over intermediate reasoning increases the risk of unnoticed errors, misdiagnoses, and overconfidence in models that appear accurate on surface metrics.
To address these challenges, introduce novel M3CoTBench benchmark that is designed evaluate and standardize CoT reasoning in medical image interpretation. Specifically, we propose a novel curation pipeline, which includes 1) the collection of diverse and high-quality medical images, 2) automated data annotation, and 3) manual review and calibration. By bridging the gap between medical diagnostic workflows and Al-driven reasoning. M3CoTBench not only facilitates transparent evaluation but also paves the way for developing clinically trustworthy MLLMs. Our contributions redefine evaluation standards in medical imaging. emphasizing the need for interpretable, step-by- step reasoning in high-stakes applications. Our work is guided by three core principles:
Diverse Medical VQA Dataset. We curate 1,079-image QA dataset spanning 24 modalities and 13 tasks, stratified by difficulty and annotated with step-by-step reasoning aligned to clinical workflows.
Multidimensional CoT-Centric Metrics. Evaluation criteria for correctness, efficiency, impact, and consistency of reasoning steps, enabling granular performance analysis for various ML.MMs.
Comprehensive Model Analysis. We evaluate general-purpose and medical MLLMs by quanti- tative metrics and case studies, highlighting strengths and failure modes in clinical reasoning to guide future improvements.
2 RELATED WORK
2.1 MULTIMODAL LARGE LANGUAGE MODELS
Inspired by recent advances in LLMs like LLaMA (Touvron et al., 2023) and GPT (Ouyang et al., 2022), MILMs extend text-centric architectures by embedding visual features into the latent language space, enabling diverse image- grounded text generation. The LLaVA-One Vision (Li et al., 2024) family combines large-scale image/video corpora with instruction finetuning to excel across single-image. and video tasks. LLLVVA-COT (Xu et al., 2024) introduces multistage prompting strategy incorporating summarization, visual analysis, reasoning, and conclusion. Qwen-3-VI. (Bai et al., 2025) employs enhanced Interleaed-MMRoPP for improved spatial-temporal modeling. DeepStack integration to leverage multi- level ViT features for tighter vision-language alignment, and text-based time alignment for precise video temporal grounding. enabling stronger multimodal understanding and long-context reasoning. Intern VL 3.5 (Wang et al., 2025a) substantially improves reasoning capability and inference efficiency through cascade reinforcement learning and efficient vision-language deployment. Closed-source GPT-4o (OpenAI, 2024a) exemplifies integration of real-time vision, audio, and text reasoning. GPT-5 (OpenAl, 2025b) intelligently routes queries to either fast responses or deeper reasoning, delivering smarter, more accurate, and more useful performance across domains. Slaude-Sonnet-44 (Anthropic, 2025) and Gemini 3 (Google DeepMind, 2024) also show remarkable performances. In medicine, specialized MIL.Ms adapt these techniques to clinical data: Med- -lammnno (Moor et al., 2023) augments Flamingo (Alayrac et al., 2022) with medical image-text pretraining for few-shot VQA; LLaVA-Med (Li et al., 2023) aligns visual content with biomedical concepts using PubMed captions and GPT-4 instructions; Lingshu (Team et al., 2025) is generalist multimodal medical foundation model that unifies medical image-text understanding and reasoning across diverse clinical tasks. MedGemma (Sellergren al., 2025) is long-context medical vision-language model built on Gemma, optimized for clinical image analysis and medical QA. Recent advances in MILMs also catalyze the development of medical agents capable of assisting in diagnosis, decision support, and workflow integration (Hu et al., 2025).
2.2 MEDICAL MULTIMODAL BENCHMARKS
Medical multimodal benchmarks evaluate how well MLLMs interpret and reason over clinical imaging data. VQA-RAD (Lau et al., 2018) is an carly radiology VQA dataset with -anoottted QA pairs. VQt (He et al., 2020) extends VQA to pathology by pairing textbook and digital pathology images with expert-reviewed questions. SLAKE (Liu et al., 2021) offers English-Chinee radiology QA enriched with semantic labels linked to structured medical knowledge base. FMBench (Wu et al., 2024) is the first
2

---

Published as conference paper at ICLR 2026
to systematically assess fairnsss in MLLMs, incorporating clinical tasks, demographic-aware evaluation, and novel disparity metric. Quilt- (Seyfioglu al., 2024) targets histopathology VQA using real- world images and curated questions. OnniMedVQA (Hu et al., 2024) aggregates diverse datasets spanning multiple modalities and anatomy, requiring models to integrate heterogeneous inputs and justify their answers.
(Ye et al., 2024) unifies 284 global datasets into a large-scale multimodal QA benchmark covering a broad range of clinical scenarios. Med-CMR (Gong et al., 2025) is a fine- grained evaluation benchmark specifically designed to measure how models integrate medical images with clinical reasoning to perform complex multimodal reasoning. Moreover, they often lack annotations for intermediate reasoning steps, limiting their effectiveness in assessing CoT-style clinical inference in complex medical scenarios.
2.3 COT-RELATED MLLM BENCHMARKS
Research on reasoning in multimodal models has advanced through several dedicated benchmarks, Visual- CoT (Shao al., 2024) introduces a large-scale dataset of image-QA pairs, augmented with region annotations and step-by-step rationales, along with a multi-turn reasoning pipeline for interpretable, region- focused CoT tasks. CoT (Chen et al., 2024b) provides a comprehensive benchmark spanning diverse domains and requiring complex multi-step visual-textual reasoning. MME-CoT (Jiang al., 2025) extends this line work by contributing high-quality data across six domains and proposing three novel metries to assess CoT quality, robustness, and efficiency. CoMT (Cheng et al., 2025) introduces a benchmark that requires both multimodal inputs and outputs evaluate the visual reasoning abilities of MLLMs, addressing the limitations of traditional text-only outputs in multimodal CoT tasks. MMIR (Yan et al., 2025) is designed to evaluate MLLMs' ability to detect and reason about semantic inconsistencies in layout-rich multimodal content, revealing significant shortcomings in current models and highlighting the need for more advanced cross-modal reasoming capabilities. While these benchmarks have advanced CoT reasoning in natural image domains, resources remain scarce in the medical field, where rigorous diagnostic reasoning, interpretability, and domain expertise essential. This gap underscores the need for medically grounded benchmarks that can assess step-by-step clinical inference multimodal settings.
Image Collection
Data Annotation and Calibration
kaggle zenobb GC
QA Annotation
QA Calibration
Select and lilter
Prompt (ciggeechakr)
Gemini
Amwer questions by different MiLLMs
Modalities
Principles
Rodiology
Diversity
acconding origiaal libel (ltrmal) fara Image usswer.
Ulraaouund
Class Balance
Incomistant review and final review by experts
Ercloccopy
Typlcalty
Microscopy
Compitcance
Key Steps Annotation
CoT Key Steps Colibration
Gerini Seview MILMs
Steps Requirement
Examination Types
Key Fectures
Key Canclusion
Original
Fnall Reviow
Adcitional Analysis
Labels
Revlew students and experts
Curation of BeCch benchmark that encompasses three sections: 1) carefully curated medical images from various public sources, 2) multi-type and multi-diffieule QA generation via LLMs and expert calibration, and 3) structured annotation of key reasoning steps aligned with clinical diagnostic workflows.
Figure I:
3 CURATION OF M3COTBENCH
The collection of images, construction of QA pairs, the annotation of key CoT steps, and manual review/- calibration are carefully designed, as shown in Figure
3.1 DATA COLLECTION
All images in M3CoTBench are sourced from public datasets, with selection guided by principles of diversity, typicality, class balance, and compliance.
3

---

Published as conference paper at ICLR 2026
Diversity. Images are collected from 55 public medical datasets, encompassing diverse imaging modalities, examination types, and anatomical regions (Table A1), with broad geographical coverage (Figure A3) and diverse temporal ranges of publishing.
Typicality. To ensure large intra-dataset variance, image features are extracted by Biomed- CLIP (Zhang et al., 2023), and semantically distinct subset selected by maximizing the minimum pairwise feature distance.
Class Balance. Each public dataset comprises multiple subcategories, and we actively maintain a balanced representation of these subcategories during collection through careful inspection of the original labels.
Compliance. Datasets with usage restrictions or labeled as "no derivatives" are excluded, ad- dressing compliance issues often neglected in prior benchmarks.
Diverse Examination Type
Different Question Types
Tasks
X-Ray Colonoscapy
Histology
Modality Image Quality
Symptom
CT Pundus
OCT
Multi-Choice
laperoscopy
Single-Chotce
Recognition
True or False
It treetment
Simple
24 modalities or examination bypes
Tasks, perception complex reasoning
Carefully Designed CoT Annotation
4 Evaluation Dimensions Efficiency
Correctness
1. Eamination Type: Fundus photography.
To evaluste how faithfully and completelly the groundtruths.
measure how efficiently a model
Key Visual Feature: Visible superior retinal elevation; invelement.
produces correct staps
Conclusion:
detachment
CoT affects response speed.
Additional Analysis: Ratinal dotachment soparates nourosensory retina from RPE, disrupting photorecestor function....
Consistency
Impact
1. .Examination Type:
Hemataxylin easin (H&E).. 2. Key Visual Feature: Irregular clusters of opitholial calls with high nuclear cytoplasmic ratio, pleomorphic nuclet, and stromal Invasion. Invastve carcinoma.
To quantify the performance step-by-step reasoring.
To evaluate how clasely the reasoning paths across Instances the task type.
3. Conclusion:
a)
b)
Calaneryy
Brleiig Booqua
TraeTalle
Suggle Choice
Aesipation
Hanna
Simple Amoear
Malti-Chaies
Dellly
Gollec
Figure 2: Overview of BeCch benchmark. Top: amination types, 4 question types, and 13 clinical reasoning tasks. Middle: CoT annotation examples and 4 evaluation dimensions. Bottom: The distribution of image-QA pairs across a) modalities, b) question types, and e) tasks.
The benchmark covers 24 imaging modalities/ex-
3.2 DATA ANNOTATION AND CALIBRATION
Question-Answer Pairs Generation. We employ unitied pipeline for generating QA pairs, with all questions and candidate answers initially fully generated by GPT-4o, and subsequently calibrated by three different MLLMs and human experts to ensure the validity of the questions and the correctness of the answers.
Conversion of Existing Datasets. We apply different strategies to different public datasets according to their diverse original purposes. Starting with existing QA pairs from public VQA and image classification datasets, we use GPT-4o rewrite them into more diverse formats, such as single-choice, multiple-choice, true/false, and short-answer questions. For segmentation datasets, we concatenate the raw image with its corresponding mask and ask targeted questions about the masked region; for object detection datasets, we generate spatial questions, such as requesting a rough anatomical location or estimating bounding box coordinates; and for some

---

Published as conference paper at ICLR 2026
image quality assessment and disease grading tasks, we present paired images and formulate comparative questions.
Generation of Inferenceddriven Medical Questions. To enrich the complexity of QA tasks and better support reasoning capabilities, we provide GPT-40 with the original label and prompt to generate questions with corresponding answer options grounded in that information. For example, given a slit lamp image labeled "severe keratitis with corneal ulcer". GPT-4o is prompted to create a multi-choice question about causes, such as "What might he the cause of this condition? (Select all that apply)". with answer options including bacterial, viral, fungal infections, trauma, allergic reactions, etc. The correct answers align with clinically relevant causes associated with the diagnosis. This approach introduces hierarchical difficulty and inference-driven tasks that extend beyond surface-level recognition, fostering more in-depth medical reasoning.
Al and Human Expert Calibration Process. For calibration, we leverage three different MLLMs to answer each image-question pair independently.. If any MLLM's response differs from the initially generated answer, human expert, experienced doctor, intervenes to make the final judgment. Additionally, the expert reviews all images and QA pairs comprehensively to perform a final quality check and calibration. This combined AT-human validation ensures high accuracy and reliability of the dataset.
Rationale for the step design. Our CoT steps are derived from clinician interviews and established theories of medical reasoning. They consist of: (1) confirming the image's nature (modality/examination type), (2) identifying key visual features, (3) drawing diagnostic conclusions, and (4) providing further medically informed analysis.
Validation via Doctor Interviews. Before designing the CoT steps, we interview clinicians, radiologists, and sonographers from five hospitals. Most clinicians describe their workflow as: identify the imaging modality, observe key features, draw core conclusions, and then perform additional analyses such as etiology or treatment planning. Some clinicians note that intuition may guide an initial hypothesis, which then verified through feature inspection. These findings support our chosen steps as both sufficient and necessary for medical reasoning.
Theoretical Support from Medical Cognition. Our CoT design draws on established cognitive models. (1) Hypothetico-deductive reasoning (Elstein et al., 1978): Clinicians generate and iteratively test hypotheses; our steps follow this natural cycle. (2) Pattern recognition (Norman et al., 2007): Experienced doctors rapidly spot salient imaging patterns: our early focus on key features reflects this process. (3) Dual-process theory (Arvai, 2013): Intuitive and analytical reasoning interact; our annotations capture this by allowing preliminary intuitive judgments followed by feature- verification and further analysis.
CoT Key Steps Generation. To ensure effective CoT in medical VQA that mirrors clinicians' cognitive workflow from perception to judgment, we first leverage MLLMs to annotate CoT key steps, which then undergo repeated cycles of review, feedback, and revision by medical experts and students before senior experts confirm the final CoT annotations.
MLL.-BBased Annotation. For each image-QA instance, we provide GPT-4o and Gemini- 2.5-Pro with the image, the question, the answer, and any relevant contextual information from the original annotations. For example, underlying labels used to construct the question itself, complex questions about treatment, causes, prediction, function are often derived from simpler labels, such as disease type, which are also provided as input. Additionally, the model generates reasoning steps following an expert-designed four-step clinical structure: (1) confirming the nature of the image, such the imaging modality and examination type; (2) identifying key visual features; (3) drawing diagnostic conclusions, including the relevant disease, organ, or tissue; and (4) providing additional analysis based on medical knowledge, such as treatment strategies or associated symptoms. It is worth noting that we condition the model by specifying the expected reasoning steps based on the task type. For instance, modality questions omit steps 3 and 4. while diagnostic questions skip step 4. GPT-4o and Gemini 2.5 Pro then generate the corresponding key reasoning steps accordingly. Finally, the final results are generated again by GPT-4o, which integrated annotation information from both GPT-40 and Gemini 2.5 Pro.
Al and Human Expert Calibration Process. To ensure high-quality and medically reliable an- notations, we adopt a multi-stage human-Al collaborative verification process: (1) Initial Student
5

---

Published as conference paper at ICLR 2026
Review: A medically trained student manually reviews model- human-generated annotations, correcting factual, spelling, and formatting errors, and filling in missing key information. Un- certain cases are discussed with experts. (2) Automated Multi-model Checking: The image. question, and reasoning steps are validated using GPT-4o. (3) Expert Review Model Flags: Any reasoning step flagged "potentially incorrect" by any model is sent to an expert the relevant imaging modality for manual review. (4) Consensus Resolution: When experts identify issues, the involved experts and student reviewers hold brief online meetings asynchronous discussions to resolve disagreements. Three such meetings and multiple asynchronous discussions are held. Annotations are updated based on consensus. (5) Final Expert Read-through: Experts conduct a final pass each sample to ensure that the image, question, reasoning steps, and answer are medically correct, consistent, and compliant with benchmark standards.
3.3 DATA COMPOSITION AND CATEGORIZATION
As shown in Figure 2, MBCco'Bench includes diverse image-Qu QA pairs with multiple question formats and task types of varying difficulty. It covers broad range of imaging modalities and examination types across several categories. Tasks span from basie perception advanced medical reasoning, enabling comprehensive evaluation of MILMs.
QA Types. We include four question formats: single-choice, multiple-choice, true/false (judgment), and short-answer, spanning 13 task types with varying difficulty levels.
Examination Types. The dataset encompasses 24 imaging modalities and examination types, which be organized into six major categories: ophthalmic imaging, radiology, endoscopy. microscopy, ultrasound- based examinations, and surface-level inspections. Representative modalities within these categories include slit lamp photography (SLP), fundus photography (FP), optical coherence tomography (OCT), optical coherence tomography angiography (OCTA), scanning laser ophthalmoscopy (SLO), fundus flu- orescein angiography (FFA), X-ray, computed tomography (CTT) magnetic resonance imaging (MRI), ultrasound (US), infrared reflectance (IR), nuclear medicine, fetoscopy, laparoscopy, colonoscopy, gas- troscopy, capsule endoscopy, bronchoscopy, ENT endoscopy, histology, cytology, fluorescence microscopy, dermoscopy. and intraoral examination.
Task Types. To thoroughly assess the reasoning ability of MILMs, design questions spanning broad spectrum of clinical tasks, including: Examination Type, Image Quality, Recognition, Referring Recognition, Localization, Diagnosis, Grading. Prediction, Function, Symptom, Counting, Cause, and Action. These categories range from low-level perception tasks (e.g., assessing image quality) to high- level clinical reasoning (e.g., identifying causal factors or suggesting next actions). Such a taxonomy is constructed to test MLLMs' ability to bridge the gap between visual perception and domain knowledge reasoning, challenging both their vision-language alignment and medical understanding. Some example image-question pairs can be seen in Figure
Table 1: Criterion comparison for current benchmarks. /: Satisfied. X: Unsatisfied.
Dataset
#lmg/#QA
Exam. Type
Question Type
CoT Annotation
Acc.
Eval. Dimension
Carr.
Imp.
Ef
Cons,
VQA-RAD (Lau al., 2018)
3315
X
SLAKE al., 2021)
VQA (Seylioglu et al., 2024)
985 1283
x
X
x
OmmiMedVOA et al., 2024)
118010/ 127995
12'
X
X
GMAI-MMBench (Ye al., 2024)
-/ -/25531
38
x
X
X
M3CoTBench
1079 1079
24
/
The way of classifying modalities differs from this paper.
4 EVALUATION SUITE OF M3COTBENCH
We evaluate CoT reasoning based on four aspects: correctness, efficiency, impact, and consistency. Here, correctness measures whether the generated reasoning steps are accurate; efficiency reflects the additional inference time introduced by reasoning: impact quantifies the overall effect of reasoning on answer accuracy compared to direct prediction without reasoning; and consistency assesses whether similar tasks tend to follow similar reasoning paths.
6

---

Published as conference paper at ICLR 2026
Examination Type
Image Quality
Counting
What procedure image?
True or False: The image on the left is of higher quality than the right.
How many calls are there in this image?
Recognition
Referring Recognition
Localization
Which category image likelly belong to?
...Waa region right image represent the left image?
Where is the subcutancous air pollection?
Diagnosis
Grading
Symptom
What is the correct diagnesis based this image?
True False: The retinapathy the left right image.
Which of the following symptoms
than
patient likely to experience?
Action
Prediction
Cause
Function
True or Palse: Tais condition potentially
Which the following are potential condition?
During defense against parasittc infections, by which mechanism do type calls the immune response?
appropriate
treatment
options for Chis struation?
blindness,
Figure Example image-question pairs for 13 tasks in M3CoTBench, including identifying examina- tion types, image quality assessment, recognition, referring recognition, counting, localization, diagnosis, grading, symptom identification, clinical action planning, prediction, functional understanding. and causal reasoning.
Evaluation of Reasoning Correctness. To comprchensively evaluate the accuracy of the model's reasoning steps, we quantify alignment between the generated reasoning sequence expert-annotated reasoning paths. Specifically, we compute the following metrics:
N
Avg Precision
R O 1/1R Avg Recall Σ
1 N
R n
A
1/14
(I)
Here, R. denotes the set of reasoning steps generated by the model, (.4x] represents all annotated gold reasoning paths for given question, N is the number of examples. Since multiple valid reference paths may exist, we choose the reference A&- with the highest overlap with R. Precision measures the proportion of model-generated steps that are correct, while recall quantifies the coverage of reference reasoning steps.
Evaluation of Reasoning Efficiency. CoT reasoning often introduces significant computational overhead due to longer generated sequences. Excessively verhose CoT outputs increase inference time memory consumption, reducing practical usability in real-world applications. To evaluate reasoning efficiency, we compute the number of correct reasoning steps per unit time. Formally,
E
Σ
R
A /Teor.
A higher E indicates more accurate reasoning steps per unit time, reflecting more efficient reasoning, Then we define average inference latency impact as the ratio between the total CoT inference time and direct inference time, divided by the number of examples: I. Test /Tireet where Tear, Tairett are the total inference times with and without COT. respectively. larger value indicates greater average latency overhead. By jointly considering and L, we can better benchmark the trade-offs between interpretability and computational cost in CoT-enabled models.
Evaluation of Reasoning Impact. To quantify the benefit of generating step-by-step reasoning over directly producing the final answer, define the reasoning impact metrie the difference in answer accuracy between the two approaches. Let Acc denote the accuracy of the model when generating
7

---

Published as conference paper at ICLR 2026
answers with intermediate reasoning steps, and Accairect denote the accuracy when generating answers directly without explicit reasoning. The reasoning impact is computed as: ACCsup AcCairest-
A positive value of I indicates that step-by-stey reasoning improves answer correctness, demonstrating the effectiveness of CoT generation in enhancing model performance. Conversely, a negative or zero value suggests that the reasoning steps do not provide additional benefit may degrade the correctness. This metric offers a straightforward way to assess whether incorporating explicit reasoning contributes meaningfully to the model's 's accuracy.
Evaluation of Reasoning Consistency. Structured and task-specific reasoning pathways are fundamental for interpretability and play a vital role in ensuring reproducibility, transparency, and trustworthiness in high-stakes medical decision maaiiiinnmmaiig.... However, existing evaluation metrics often treat reasoning steps as unordered elements. To address this gap, we introduce a path consistency metric that explicitly evaluates how closely the reasoning paths for instances within the same task type align. We compute this score independently for each of the thirteen tasks and then average the results. For task t with N examples, represent each generated reasoning path p(() as an ordered sequence of step categories (e.g., modality, feature, diagnosis, additional analysis). To evaluate path consistency, first select the reference path by maximizing its average similarity with generated paths:
N sim p(")) sim(P, = ILCS(P, PC //aaa/ (P.(()))))
p(t) arg max
Σ
The task-level consistency score is then defined as the average similarity between each path and the canonical reference:
( path I/N
sim p(t) 111))
(4)
Average score over all tasks: Cpath indicates that the model has strong structural stability in its COT.
1/M E/
c(t)
path where M = 13. A higher Cparh E [0,
5 EXPERIMENTS
5.1 EXPERIMENT SETUP
Evaluation Models. We select top-performing MLLMs for comprehensive CoT evaluation. We evaluate open-source models such as LLaVA-CoT(1BB) (Xu al., 2025), IntemVLL3.5885 30B) (Wang et al., 2025a), Q
30B) (Bai et al., 2025), Deeen---i--hhiniigffee 30B) (Bai et al., 2025). We also include closed-source GPT-4.1 (OpenAI, 2025a), GPT-5 (OpenAI, 2025b), Gemini 2.5 Pro (Google DeepMind, 2024), and Claude-Somnet-4.5 (Anthropic, 2025) as strong baseline models. Finally, we evaluate some models specifically designed for the medical domain, like LLaVA-Med (7B) (Li et al., 2023), SiaatuoGT-Viisinn-77 70-oonn2---- (Chen et al., 2024a). HealthGPT(3.... (Lin et al., 2025), Lingshu (7B, 32B) (Team al., 2025) and MedGemma (4B, 27B) (Sellergren et al., 2025).
Implementation Details. We define the CoT prompt as: Please generate a step-by-step answer. including all intermediate reasoning steps, and provide the final answer the end. The direct prompt is defined as: Please directly provide the final answer without any additional output. For all experiments, the batch size is set to to ensure independent processing of each sample, and the temperature is uniformly set to 0.1. For evaluation, we GPT-4o and LLaMA-3.33 ---.. Turbo (Grattafiori et al., 2024) and Gemini 2.5 Pro for assessment criteria. All local inference experiments are conducted on a server with AMD GPUs. APIs are used for closed-source MI.LMs.
5.2 QUANTITATIVE RESULTS
The experimental results can be seen in Table 2, from which there are some interesting findings:
Correctness. (I) Closed-source vs. Open-source Models: Across models, closed-source systems do not exhibit uniform advantage over open-source ones in terms of CoT-ground truth alignment. A notable example is GPT-5, which achieves relatively high Precision substantially lower Recall, This pattern
8

---

Published as conference paper at ICLR 2026
Table 2: M3CoTBench results for MsLMs. 1(): the higeer(lowwr) the better. PI, R: the average F1 score(() (6), Precision( (6), and Recall(S). AcCdiret and Accsup' accuracy(%) of generated answers by directly and COT. I, E and L, and Cpath (%): Impact, Efficiency, Latency, and Consisteney score, respectively.
Optimal / sub-optimal results are highlighted in bold / underline.
Correctness
Impact
Efficlency
Consistency
Model
PI(T) PC)
HCT ) Accara
Accuss
K(t) ) EXT) D(4
(par)()
Open-auurce MLLMs
LLaVA-CoT (Xu et al., 2025)
49.80
54,08
46.15
40.13
36.75
-3.38
0.06
1.56
77.02
3,55 (Wang et 2025a)
60.61
52.88
56.81
-3.20
IntemVII 3,55330 (Wang et al., 2025a)
59.42
62.15
56.92
63.81
57.65
-6.16
0.03
16.68
76.30
Qwen3-VL VL-IIstrrcc-8B (Bai al, 2025)
55.17
52.74
57.84
51.30
-4.68
Qwen VI. Instruct 30B (Bai et al., 2025)
59.15
56.13
62.51
54,63
51.39
-3.24
0,03
35,63
83.01
Owen VL. Thinkinn
2025)
59.84
59.91
48.33
+4.50
Qwea3 VL- Thinking-308 (Bai et al., 2025)
62.15
63.34
61.01
51.95
53.47
+3.52
0.02
1.15
76.02
Closed-ssurce MLLMs
GPT-4.1 (OpenAI, 2025a)
60.76
58.32
63.42
56.77
58.11
+1.34
0.17
5.08
81.31
GPT-5 (OpenAl, 2025b)
55.13
64.15
18.34
S8.76
58.22 28.29
-0.47
0.06
1.10
65.39
Gemini 2.5 Pro (Gongle DeepMind, 2024)
66.07
62.48
70.10
60.24
60.10
-0.14
0.10
1.52
82,00
Clande-Sonnet-4.. 4.5 (Anthropic, 2025)
56.50
53.62
59.71
51.34
51.07
-0.27
0.15
2.69
85.22
Medical Specifii MIL.My
LLa (7B) (Li et al., 2023)
30.51
36,33
26.30
29.38
29.29
-0.09
0.35
3.22
72.68
HuatuoGPP Vision (?B) (Chen et al., 2024a)
49.45
51.17
47.85
41.89
34,94
-6.95
0.21
5.92
73.19
HealthGPT (3.8B) (I.in et al., 2025)
32.56
47.27
24.83
44.11
42.03
-2.08
0.06
15.36
67.72
Lingshu-7B (Team al., 2025)
57.57
63.96
52.34
50.00
42.08
-7.92
Lingshu-32B (Team al., 2025)
59.16
68.68
53.82
51.81
44.95
-6.86
0.21
10.87
71.47
MedGemma (Sellergren et al., 2025)
50.29
46.14
43.33
41.29
MedGemma 27B (Sellergren et al., 2025)
50.98
48.33
53.81
46.06
45.46
0.60
0,03
23.71
82.55
arises because GPT-5 frequently bypasses step-by-step reasoning and directly outputs final answers under CoT instructions. In contrast, GPT-4.1 and Gemini 2.5 Pro show strong and balanced P/R/FI scores, indicating that they reliably produce complete and structured reasoning chains. This comparison suggests that adherence to CoT instruction, rather than model openness, is the dominant factor in CoT quality. (2) Thinking Models vs. Instruction Models: Within the Qwen3-VL family, the Thinking variants consistently outperform the Instruct variants, demonstrating the benefit of explicitly modeling multi-step reasoning. The Instruct models are optimized for task completion and often generate concise answers with implicit reasoning, which leads to missing or compressed intermediate steps and lower CoT-GT alignment. In contrast, the Thinking models are encouraged to externalize intermediate decisions, resulting in more complete coverage of annotated steps. (3) Large vs. Small Models within the Same Series: Comparisons within the same model family show that larger models generally achieve higher Fl scores than their smaller counterparts. Beyond raw capacity, larger models exhibit greater stability in multi-step reasoning, making them less prone to skipping steps, collapsing reasoning, or introducing spurious assumptions. (4) Performance of Medical-Speeiffe MLLMs: Medical-spectee specialliee MLLMs do consistently outperform general-purpos purpose MLLMs in CoT-GTT alignment. Many medical MLLMS show moderate Precision but lower Recall. This suggests that domain specialization alone does not guarantee high-quality CoT generation.
Efficiency. In terms of reasoning efficiency, models like LLaVA-CoT and the inherently generate step-by-step outputs, so adding CoT prompt does not significantly change sequence length, resulting in low latency and stable efficiency. Within the same model series, larger models tend to have relatively lower latency than smaller ones. Closed-sone cousee models keep moderate latency growth. ALaVA-Mes achieves the highest efficiency due to its inherently fast inference speed, so generating CoT outputs does not noticeably increase latency. Additionally, the efficiency scores of larger models within the same series remain flat translate into significantly more correct reasoning steps.
series
decline, indicating that the increased inference time does not proportionally
Impact. Notably, CoT prompting fails to yield consistent gains in medical image understanding and can even reduce accuracy, likely because it introduces unnecessary or misleading reasoning steps in domains where diagnostic decisions depend more on visual cues than logical inference. The problem is especially pronounced when medical models lack robust multimodal grounding. and CoT may further raise hallucination risk distract attention from critical features (Li et al., 2025). Some prior studies have discussed this phenomenon. Mishra & Thakkar (2023) points out that CoT is highly sensitive, and unreasonable reasoning chains may substantially degrade performance, Jiang et al. (2025) measures
9

---

Published as conference paper at ICLR 2026
the effects of CoT in the MME-CoT benchmark: most perception tasks showed decreased performance, while about half of the reasoning tasks declined. Closed-source models exhibit smaller performance drops or even a positive impact under CoT prompting. This suggests that closed-sounce models possess more robust internal reasoning mechanisms and stronger caaructtio-followiin controls, allowing them to absorb the additional CoT constraint without severely disrupting prediction quality. Although both Owen3-VL-Thinkee VL-Tkinking and LLaVA-CoT generate reasoning steps by default, explicit CoT prompting affects them differently. For Qwen3-VL-Thine VLTThiikig, performance changes under CoT prompting suggest that its generated reasoning may be more tightly coupled with the prediction process, where additional structured prompting encourages more consistent step-wise outputs and slightly improves accuracy. In contrast, LLaVA-CoT shows limited performance gain, indicating that its generated reasoning may function more as an explanatory output rather than consistently influencing the final decision.
Consistency. Most models tend to generate similar reasoning steps when handling the same task, resulting in generally high path consistency scores. Most closed-source models achieve relatively high scores, benefiting from consistent generation processes and reasoning patterns. GPT-5 shows the lowest consistency because often omits intermediate reasoning steps, producing incomplete chains. Open-source models also exhibit relatively high consistency. Aside from the MedGemma series, where larger models outperform smaller ones, models within the same series generally show very similar consistency scores.
5.3 QUALITATIVE ANALYSIS
By analyzing model outputs with errors, systematic errors are emerging within the intermediate steps in COT, rather than merely the final prediction. For example, steps, where the model misinterpreted secondary features (e.g., narrow anterior chamber angle and iris deposits) as primary evidence, leading it to incorrectly conclude angle-closure glaucoma and deviate from the correct treatment path. Such qualitative inspection highlights three factors:
occurred in the intermediate reasoning
1. Incomplete Verification of Decisive Diagnostic Features. Although the CoT reasoning often identified some relevant abnormalities, it frequently omitted or misweighted critical criteria, such as the extent of epithelial involvement in the pathology case, thereby allowing early misreadings to dominate the conclusion and persist through the subsequent steps.
2. Weakened Vision- LaiiuaL Grounding During Step-wise Verbalization. By forcing the model to translate visual cues into descriptive textual representations before decision-making, CoT increased the risk of information distortion, subtle semantic drift, and gradual loss of fine visual detail. In the hematology example, this intermediate translation process leads an inaccurate verbal focus on nuclear shape while neglecting the defining cytoplasmic gramules, their relative prominence, and characteristic spatial distribution.
3. Error Accumulation Along the Reasoning Chain. Once early descriptive mistake occurred, subsequent steps propagated and rationalized the error, producing a seemingly coherent but ultimately incorrect explanation that became harder to override with additional context.
These observations indicate that the degradation with the CoT prompt reflects deeper vulnerabilities in how visual evidence is interpreted and verified across multiple reasoning stages. Representative examples and detailed error analyses are provided in Appendix D.2.
6 CONCLUSION
In this work, we introduce M3CoTBench, a novel benchmark designed to evaluate CoT reasoning in MLLMs for medical image understanding. Our benchmark addresses the critical gap between answer correctness and reasoning quality in clinical AI systems by incorporating diverse imaging modalities or examination types, step-by-step reasoning annotations, and tailored multi-ddimensionl evaluation metrics across medical cases of varying difficulty, from simple pattern recognition to complex diagnostic reasoning, enabling fine-grained analysis of model capabilities. Through comprehensive assessments of -tate-of-tte-art MLLMs, we demonstrate limitations of existing models in generating interpretable and clinically aligned reasoning. We hope this benchmark will inspire future research toward more transparent, trustworthy, and practically valuable AI systems for healthcare and beyond. More discussions about limitations and social impact can be seen in Appendix E and Appendix
10

---

Published as conference paper at ICLR 2026
ACKNOWLEDGMENTS AND DISCLOSURE OF FUNDING.
This is A Project Supported by Scientific Research Fund of Zhejiang University (XY2025026). This work also supported by the National Key R&D Program China (Grant No. 2025YFF051 1302).
ETHICS STATEMENT
We have ensured that our study and dataset construction follow ethical standards, with no direct involve- ment of human subjects, and no foreseeable risk of harm. Data usage complies with privacy and legal requirements, and have aimed to mitigate potential biases in annotations and model evaluation. We disclose no conflicts interest or sponsorship that could influence the results.
REPRODUCIBILITY STATEMENT
We have already elaborated all the models algorithms proposed, experimental configurations, and benchmarks used the experiments in the main body or appendix of this paper. Furthermore, we declare that the entire code used in this work will be released after acceptance.
REFERENCES
Entrep challenge: Advancing vision- language ai for ent endoscopy analysis, 2025. URL https:/aachhallenge. hemus.edu. wtw.:/de.uu.///mmm.e...mm m0m2022/entree..
Maruf Adewole, Jeffrey D. Rudie, Anu Gbadamosi. Oluyemisi Toyobo, Confidence Raymond, Dong Zhang, Olubukola Omidiji, Rachel Akinola, Mohammad Abba Suwaid, Adaobi Emeeookor, Nancy Ojo, Kenneth Aguh, Chinasa Kalaiwo, Gabriel Babatunde, Afolabi Ogunleye, Yewande Gbadamosi, Kator lorpagher, Evan Calabrese, Mariam Aboian, Marius Linguraru, Jake Albrecht, Benedikt Wiestler, Florian Koffer, Anastasia Janas, Dominic LaBella, Anahita Fathi Kzerooni, Hongwei Bran Li, Juan Eugenio Iglesias, Keyvan Farahani, James Eddy, Timothy Bergquist, Verena Chung. Russell Takeshi Shinohara, Walter Wiggins, Zachary Reitman, Chunhao Wang, Xinyang Liu, Zhifan Jiang, Ariana Familiar, Koen Van Leemput, Christina Bukas, Maire Piraud, Gian-Marc Conte, Elaine Johansson, Zeke Meier, Bjoern H Menze, Ujjwal Baid, Spyridon Bakas, Farouk Dako, Abiodun Fatade, and Udunna Anazodo. The brain tumor segmentation (brats) challenge 2023; Glioma segmentation in sub-saharan africa patient population (brats-africa), 2023.
Moulay A. Akhloufi and Mohamed Chetoui. Chest xr covid-19 detection. https///xx--oovidI9. cranddccalleege.orgg. August 2021. Online; accessed September 2021.
Walid Al-Dhabyani, Mohammed Gomaa, Hussien Khaled, and Aly Fahmy. Dataset of breast ultrasound images. Data in brief, 28: 104863, 2020.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Tain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in neural information processing systems, 35:23716 227316,,,,, 2022.
Shams Nafisa Ali, Md. Tazuddin Ahmed, Tasnim Jahan, Joydip Paul, S. M. Sakeef Sani, Nawshaba Noor, Anzirun Nahar Asma, and Taufiq Hasan. web-based mpox skin lesion detection system using state-of-the the-art deep learning models considering racial diversity. Biomedical Signal Processing and Control, 98: 106742. 2024.
Max Allan, Alex Shvets, Thomas Kurmann, Zichen Zhang, Rahul Duggal, Yun-Hsuan Su, Nicola Rieke, Iro Laina, Niveditha Kalavakonda, Sebastian Bodenstedt, et al. 2017 robotic instrument segmentation challenge. arXiv preprint arXiv:1902.000 1902.06426. 2019.
Anthropic. Introducing claude sonnet 4.5.
2025.
Guilherme Aresta, Teresa Araújo, Scotty Kwok, Sai Saketh Chennamsetty, Mohammed Safwan, et al. Bach: Grand challenge on breast cancer histology images. Medicul Image Analysis, 56: 16:122-139, August 2019. ISSN 1361-8415 doi: 10. 201... 2019.05.010.
URL http::/dx. www.o//w.oooo/////ooo// 10. 10.110666...mllee.

---

Published as conference paper at ICLR 2026
Joseph Arvai. Thinking, fast and slow, daniel kahneman, farrar, straus & giroux, 2013.
Shuai Bai, Yuxuan Cai, Ruizhe Chen. Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang. Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, Mei Li. Kaixin Li, Zicheng Lin, Junyang Lin, Xuejing Liu, Jiawei Liu, Chenglong Liu, Yang Liu, Dayiheng Liu, Shixuan Liu, Dunjie Lu, Ruilin Luo, Chenxu Lv. Rui Men. Lingehen Meng, Xuancheng Ren. Xingzhang Ren, Sibo Song. Yuchong Sun, Jun Tang, Jianhong Tu, Jianqiang Wan, Peng Wang. Pengfei Wang, Qiuyue Wang, Yuxuan Wang. Tianbao Xie, Yiheng Xu, Haiyang Xu, Xu, Zhibo Yang. Mingkun Yang. Jianxin Yang, An Yang. Bowen Yu, Fei Zhang, Hang Zhang, Xi Zhang, Bo Zheng, Humen Zhong, Jingren Zhou, Fan Zhou, Jing Zhou, Yuanzhi Zhu, and Ke Zhu. Qwen3-vl technical report, 2025. URL https//aarii.orr//as//111..6..
Sophia Bano, Francisco Vasconcelos, Luke M Shepherd, Emmanuel Vander Poorten, Tom Vercauteren, Se- bastien Ourselin, Anna I. David, Jan Deprest, and Danail Stoyanov. Deep placental vessel segmentation for fetoscopic mosaicking. In Medical Image Computing and Computer Assisted Intervention-MICCAA 2020: 23rd International Conference, Lima, Peru, October 4-8, 2020, Proceedings, Part III 23, pp- 763-773. Springer, 2020.
Sophia Bano, Alessandro Casella, Francisco Vasconcelos, Sara Moccia, George Attilakos, Ruwan Wimala- sundera, Anna L David, Dario Paladini, Jan Deprest, Elena De Momi, et al. Fetreg: Placental vessel segmentation and registration in fetoscopy challenge dataset. arXiv preprini arXiv:2106.0555 2021.
Asma Ben Abacha, Sadid A. Hasan, Vivek V. Datla, Joey Liu, Dina Demner- Fushman, and Henning Müller.. Vqa-med: Overview of the medical visual question answering task at imageclef 2019. In Working Notes of CLEF 2019, volume 2380 of CEUR Workshop Proceedings, Lugano, Switzerland, September 9-12 2019. CEUR- -.k--.S.or...... URL https://ccur-- ws.org/Vol- 2330pppp 272.pdf.
Sartaj Bhuvaji, Ankita Kadam, Prajakta Bhumkar, and Sameer Dedge. Brain tumor classification (mri). https///wwwkaagle... http.///tatttttaaaaaaaaaaaaaaiaa http:/rwwwaaaaaiiuuuo---o-aasaee -lwooo-lassiiicationn mri, 2020.
Federico Bolelli, Luca Lumetti, Shankeeth Vinayahalingam, Mattia Di Bartolomeo, Arrigo Pellacani, et al. Segmenting the Inferior Alveolar Canal in CBCTs Volumes: the ToothFairy Challenge. IEEE Transactions on Medical Imaging. pp. 1-17, Dec 2024. ISSN 1558-254X. doi: https://boi.rr//00.11999 TMM.2024.3523096.
Mateusz Buda, Ashirbani Saha, and Maciej A Mazurowski. Association of genomic subtypes of lower- grade gliomas with shape features automatically extracted by a deep learning algorithm. Computers in biology and medicine, 109:218-225, 2019.
Junying Chen, Ruyi Ouyang. Anningzhe Gao, Shunian Chen, Guiming Hardy Chen, Xidong Wang, Ruifei Zhang, Zhenyang Cai, Ke Ji. Guangjun Yu, Xiang Wan, and Benyou Wang. Huatuogpt-vision, towards injecting medical visual knowledge into multimodal 1mms at scale, 2024a. URL https//aarxiv.or//ab/// 2406.19280
Qiguang Chen, Libo Qin, Jin Zhang, Zhi Chen, Xiao Xu, and Wanxiang Che. M3cot: A novel benchmark for multi-domain multi-step multi-modal chain-of-thought. Proceedings of the 62nd Annual Meeting of the A Association for Computational Linguistics (Volume 1: Long Papers), pp. 81 8199-8221, 2024b.
Zihui Cheng. Qiguang Chen, Jin Zhang, Hao Fei, Xiaocheng Feng, Wanxiang Che, Min Li, and Libo Qin. Comt: novel benchmark for chain of multi-modal thought on large vision- langugee models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. -26666, 2025.
Zheng Chu, Jingehang Chen, Qianglong Chen, Weijiang Yu, Tao He, Haotian Wang, Weihua Peng. Ming Liu, Bing Qin, and Ting Liu. Navigate through enigmatic labyrinth a survey of chain of thought reasoning: Advances, frontiers and future. In ACL, 2024.
Marco Cipriano. Stefano Allegretti, Federico Bolelli, Federico Pollastri, and Costantino Grana. Improving Segmentation of the Inferior Alveolar Nerve through Deep Label Propagation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 21 105-2111 14. IEEE, Jun 2022. ISBN 978-1-6654-6947-00 doi: https///doi.rg/11....1///cc/
2022.00046.
12

---

Published as conference paper at ICLR 2026
Noel Codella, Veronica Rotemberg, Philipp Tschandl, M Emre Celebi, Stephen Dusza, David Gutman, Brian Helba, Aadi Kalloo, Konstantinos Liopyris. Michael Marchetti, et al. Skin lesion analysis toward melanoma detection 8: A challenge hosted by the international skin imaging collaboration (isic). arXiv preprint arXiv: 1et..090223.636 2019.
DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Jumxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, Xiaokang Zhang. Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang. Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang. Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang. Jin Chen, Kai Dong. Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang. Lecong Zhang. Liang Zhao, Litong Wang. Liyue Zhang. Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang. Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang. Peng Zhang, Qiancheng Wang. Qinyu Chen. Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang. R. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Yc, Shiyu Wang. Shuiping Yu. Shunfeng Zhou, Shuting Pan. S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye. Tao Yun. Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang. Wenjun Gao, Wengin Yu, Wentao Zhang, W. L.. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nic, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li. Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang. Y. X. Wei, Yang Zhang. Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang. Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He. Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma. Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang. Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong. Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma. Ying Tang. Yukun Zha, Yuting Yan. Z. Z. Ren, Zehui Ren. Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang. Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang. and Zhen Zhang. Deepseek-rl: Incentivizing reasoning capability in lims via reinforcement learning, 2025. URL https://arkiv.or//ab//550......
Jianning Deng, Peize Li, Kevin Dhaliwal, Chris Xiaoxuan Lu, and Mohsen Khadem. Feature-based visual odometry for bronchoscopy: A dataset and benchmark. In 2023 IEFF/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5557-6564, IEEE, 2023.
Simon Duchesne, Isabelle Chouinard, Olivier Potvin, Vladimir S Fonov, April Khademi, Robert Bartha, Pierre Bellec, D Louis Collins, Maxime Descoteaux, Rick Hoge, et al. The canadian dementia imaging protocol: harmonizing national cohorts. Journal of Magnetic Resonance Imaging. 49(2):55-455 465, 2019.
Arthur S Elstein, Lee S Shulman, and Sarah A Sprafka. Medical problem solving: An analysis of clinical reasoning. Harvard University Press, 1978.
Mohammad Fraiwan, Ziad Audat, Luay Fraiwan, and Tarek Manasreh. Using deep transfer learning to detect scoliosis and spondylolisthesis from x-ray images. Plos one, (755)80027785, 2022.
Wilfrido Gómez-Flores, Maria Julia Eregorio-Calas and Wagner Coelho de Albuquerque Pereira. Bus-bra: a breast ultrasound dataset for assessing computer-aided diagnosis systems. Medical Physics, (1)) 3110-3123, 2024.
Haozhen Gong, Xiaozhong Ji, Yuansen Liu, Wenbin Wu, Xiaoxiao Yan, Jingjing Liu, Kai Wu, Jiazhen Pan, Bailiang Jian, Jiangning Zhang, Xiaobin Hu, and Hongwei Bran Li. Med-cmr: A fine-grained benchmark integrating visual evidence and clinical logic for medical complex multimodal reasoning, 2025. URL https:/aariivorr//a//2511... 12200000
Google DeepMind. Gemini. http:://eemmiiddggoooeee/hhhh//s
2024. Accessed: May 2025.
Shivanand Gornale and Pooja Patravali. Digital knee x-ray images. Mendeley Data, 1, 2020.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh
13

---

Published as conference paper at ICLR 2026
Goyal, Anthony Hartshom, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez. Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang. Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cris- tian Canton Ferrer, Cyrus Nikolaidis. Damien Allonsius. Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dicuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Erie Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis An- derson, Govind Thattai, Graeme Nail, Gregoire Mialon, Guan Pang. Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, llyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang. Jade Copet, Jaewon Lee. Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong. Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang. Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Karthik Prasad, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala- Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham. Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si. Mitesh Kumar Singh. Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Ning Zhang. Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Pumit Singh Koura. Puxin Xu, Qing He. Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan. Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou. Thomas Scialom, Tobias Speckbacher, Todor Mihaylov. Tong Xiao, Ujjwal Karn. Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do. Vish Vogeti, Vitor Albiero, Vladan Petrovic, Weiwei Chu. Wenhan Xiong, Wenyin Fu. Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaofang Wang. Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song. Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos. Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples. Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf. Arkabandhu Chowdhury, Ash- ley Gabriel, Ashwin Bharambe, Assaf Fisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence. Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang. Changkyu Kim, Chao Zhou, Chester Hu, Shing-Hsianp Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li. David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang. Due Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Elconora Presani, Emily Hahn, Emily Wood, Eric-Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern Grant Herman, Grigory Sizov, Guangyi, Zhang. Guna Lakshminarayanan, Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang. Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj. Igor Molybog, Igor Tufanov, Ilias Leontiadis, Irina- Elena Veliche, Itai Gat, Jake Weissman, James Geboski. James Kohli, Janice Lam, Japhet Asher, rean-Baptiste Gaya, Jeff Mareus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang. Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U. Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun
14

---

Published as conference paper at ICLR 2026
Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo. Licheng Yu, Liron Moshkovich, Luea Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov. Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari. Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White. Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong. Norman Cheng. Oleg Chemoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittnrr, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina 7-yaagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang. Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang. Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shugiang Zhang. Shuqiang Zhang, Sinong Wang. Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephamie Max. Stephen Chen, Steve Kehoe, Steve Satterfield, Su- darshan Govindaprasad, Sumit Gupta. Summer Deng, Sungmin Cho, Sunny Virk. Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li. Tianjun Zhang, Tim Matthews. Timothy Chou, Tzook Shaked, Varun Vontimitta. Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla. Vlad lonescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang. Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaojian Wu, Xiaolan Wang. Xilun Wu, Xinbo Gao. Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang. Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary Vito,, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, Zhiwei Zhao, and Zhiyu Ma. The llama herd of models, 2024. URL https///axxivoorg/ab/////4
Hayden Gunraj, Chi en Amy Tai, and Alexander Wong. Cancer-net pca-data: An open-source benchmark dataset for prostate cancer clinical decision support using synthetic correlated diffusion imaging data. NeurlPS Workshops, 2023.
Palak Handa, Amirreza Mahbod, Florian Schwarzhans, Ramona Woitek, Nidhi Goel, Manas Dhir, Deepti Chhabra, Shreshtha Jha, Pallavi Sharma, Vijay Thakur, Simarpreet Singh Chawla, Deepak Gunjan, Ja- gadeesh Kakarla, and Balasubramanian Raman. Capsule vision 2024 challenge: Multi-class abnormality classification for video capsule endoscopy, 2025. URL https://arxiv.orr//bb////0 044400000000000444040...
Ali Hatamizadeh. An Artificial Intelligence Framework for the Automated Segmentation and Quantitative Analysis of Retinal Vasculature. University of Californa,, Los Angeles, 2020.
Ali Hatamizadeh, Hamid Hosseini, Niraj Patel, Jinseo Choi, Cameron C Pole, Cory M Hoeferlin, Steven D Schwartz, and Demetri Terzopoulos. Ravir: A dataset and methodology for the semantic segmentation and quantitative analysis of retinal arteries and veins in infrared reflectance imaging. IEEE Journal of Biomedical and Healthh Informatics, 26(7):3272-3283, 2022.
Xuehai He, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie. Pathvqa: 30000+ questions for medical visual question answering. arXiv preprint arXiv:2003. 10286, 2020.
Steven A. Hicks, Andrea Storás. Pål Halvorsen, Thomas de Lange, Michael A. Riegler, and Vajira Tham- bawita. Overview of imageclefmedical 2023 - medical visual question answering for gastrointestinal tract. In CLEF2023 Working Notes, CEUR Workshop Proceedings, Thessaloniki, Greece, September 18-21 2023. CEUR-W WS.org.
Xiaobin Hu, Yunhang Qian, Jiaquan Yu, Jingjing Liu, Peng Tang, Xiaozhong Ji, Chengming Xu, Jiawei Liu, Xiaoxiao Yan, Xinlei Yu, et al. The landscape of medical agents: A survey, Authorea Preprints. 2025.
Yutao Hu, Tianbin Li, Quanfeng Lu, Wengi Shao, Junjun He. Yu Qiao, and Ping Luo. Omnimedvga: A new large-seale comprehensive evaluation benchmark for medical Iylm. In Proceedings of the IEF.//VV Conference on Computer Vision and Pattern Recognition, pp. 22 70-2222288 183, 2024.
15

---

Published as conference paper at ICLR 2026
Md Nazmul Islam, Mehedi Hasan, Md Kabir Hossain, Md Golam Rabiul Alam, Md Zia Uddin, and Ahmet Soylu. Vision transformer and explainable transfer learning models for auto detection of kidney cyst, stone and tumor from ct-radiography. Scientific Reports, 12(1): 1440, 2022.
Zixi Jia, Jiqiang Liu, Hexiao Li, Qinghua Liu, and Hongbin Gao. Dcot: Dual chain-of-thought prompting large multimodal models. In The 16th Asian Conference on Machine Learning (Conference Track), 2024.
Dongzhi Jiang. Renrui Zhang, Ziyu Guo, Yanwei Li, Yu Qi, Xinyan Chen, Liuhui Wang. Jianhan Jin, Claire Guo, Shen Yan, et al. Mme-cot: Benchmarking chain-of-thought in large multimodal models for reasoning quality, robustness, and efficiency. arXiv preprint arXiv:2502.09621, 2025.
Bai Jieyun and Ou ZoanHong. Pubic symphysis-fetal head segmentation and angle of progression, April 2023. URL http:///di.org/11.58881 00t.005281122nooo788121113
Jakob Nikolas Kather, Frank Gerrit Zöllner, Francesco Bianconi, Susanne M Melchers, Lothar R Schad, Timo Gaiser, Alexander Marx, and Cleo-Aron Weis. Collection of textures colorectal cancer histology.
May 2016. URL https:///oii. ott/ 122.0/////2....511.....
Ural Koç, Ebru Akçapmar Sezer, Yaşar Alper Özkaya, Yasin Yarbay, Onur Taydas, Veysel Atilla Ayyıldız. Hüseeyn Alper Kızıloğlu, Uğur Kesimal, imran Çankaya, Muhammed Said Beşler, et al. Artificial intelligence in healthcare competition (Icknofest-2021)): stroke data set. The Eurasian journal of medicine, 54(3)2248 2022.
Zahra Mousavi Kouzehkanan, Sepehr Saghari, Sajad Tavakoli, Peyman Rostami, Mohammadjavad Abaszadeh, Farzaneh Mirzadeh. Esmaeil Shahabi Satlsar, Maryam Gheidishabran, Fatemeh Gorgi, Saeed Mohammadi, al. A large dataset of white blood cells containing cell locations and types, along with segmented nuclei and cytoplasm. Scientific reports, (2(1):1123, 2022.
Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. A dataset of clinically generated visual questions and answers about radiology images. Scienrific data, 5(1):1-10, 2018.
Wonkyeong Lee, Fabian Wagner, Adrian Galdran, Yongyi Shi, Wenjun Xia, Ge Wang, Xuanqin Mou, Md Atik Ahamed, Abdullah Al Zubaer Imran, Ji Fun Oh, et al. Low-dose computed tomography perceptual image quality assessment. Medical Image Analysis, 99:103343, 2025.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03322. 03326. 2024.
Chunyuan Li, Cliff Wong, Sheng Zhang. Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training large aanguage-and-vision assistant for biomedicine one day. Advances in Neural Information Processing Systems, 36:28541 -285644 2023.
Xiaomin Li, Zhou Yu, Zhiwei Zhang. Xupeng Chen, Ziji Zhang. Yingying Zhuang, Narayanan Sadagopan, and Anurag Beniwal. When thinking fails: The pitfalls of reasoning for fistrictionnfolloiinng in 1ims, 2025. URL hwtps//aaxiv.orr/////555....
11443.
Tianwei Lin, Wenqiao Zhang, Sijing Li, Yuqian Yuan, Binhe Yu, Haoyuan Li, Wanggui He, Hao Jiang, Mengze Li. Xiaohui Song. et al. Healthgpt: medical large Languale model for unifying com- prehension and generation via heterogeneous knowledge adaptation. arXiv preprint arXiv:2502.094. 098338 2025.
Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma. Yan Yang, and Xiao-Ming Wu. Slake: A semantically- labeled knowledge-enhance dataset for medical visual question answering. In 2021 IEEE 18th International Symposium Biomedical Imaging (ISBI), pp. 1650-1654, IEEE, 2021.
Jiaxiang Liu, Yuan Wang, Jiawei Du, Joey Zhou, and Zuozhu Liu. Medcot: Medical chain of thought via hierarchical expert. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 17371-17389, 2024a.
Shengjie Liu, Chuang Zhu, Feng Xu, Xinyu Jia, Zhongyue Shi, and Mulan Jin. Bci: Breast cancer inmunohistochemicl image generation through pyramid pix2pix. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 1815-1824, June 2022.
16

---

Published as conference paper at ICLR 2026
Yanzhen Liu, Sutuke Yibulayimu, Yudi Sang, Gang Zhu, Chao Shi, Chendi Liang, Qiyong Cao, Chunpeng Zhao, Xinbao Wu, and Yu Wang. Preoperative fracture reduction planning for image-guided pelvic trauma surgery: A comprehensive pipeline with learning. Medical Image Analysis, 102: 103:10,, 2025a. ISSN 1361 184 doi: htps:///oi..rgg/ wt.poo.../.../////o..o. 1016/j mefj..mdia.202.... 20..20025.130.... URL htts://www..cienecdineet
Yanzhen Liu, Sutuke Yibulayimu, Gang Zhu, Chao Shi. Chendi Liang. Chumpeng Zhao, Xinbao Wu, Yudi Sang, and Yu Wang. Automatic pelvic fracture segmentation: a deep learning approach and benchmark dataset. Frontiers in Medicine, 12:151 1487, 2025b.
Zuyan Liu, Yuhao Dong. Yongming Rao, Jie Zhou, and Jiwen Lu. Chain-of-spot: Interactive reasoning improves large vision-language models. arXiv preprint 2at.:2203..9966 2024b.
Shenghan Lou, Jianxin Ji, Xuan Zhang, Huiying Li, Yang Jiang, Menglei Hua, Kexin Chen, Xiaohan Zheng, Qi Zhang, Peng Han, Lei Cao, and Liuying Wang. Gastric Cancer Histopathology Tissue Image Dataset (GCHTID). 6 2024. doi: 10.6084/m9.ii htt.://ar..........m..mm 25954813. figshare.25954813... URL https://fggsaaee.comartrtteeeeee dataset/Gastric. Cancer Histopathology Tissue Image Dataset Gwr.eetGCHHDDDD/C (2CHIID)/5595481..
Bozhi Luan, Hao Feng, Hong Chen, Yonghui Wang, Wengang Zhou, and Houqiang Li. Textcot: Zoom in for enhanced multimodal text-rich image understanding, arXiv preprint arXiv:2404.0977 09797, 2024.
Luca Lumetti, Vittorio Pipoli, Federico Bolelli, Elisa Ficarra, and Costantino Grana. Enhancing Patch- Based Learning for the Segmentation of the Mandibular Canal. IEEE Access, pp. 1-12, 2024. ISSN 2169-3536. doi: https://doi.ooo org/ 109/ACCESS 3424.940
Christian Matek, Scbastian Krappe, Christian Münzenmayer, Torsten Haferlach, and Carsten Marr. An expert-annotated dataset of bone marrow cytology in hematologic malignancies. The Cancer Imaging Archive, la
Christian Matek, Sebastian Krappe, Christian Münzenmayer, Torsten Haferlach, and Carsten Marr. Highly accurate differentiation bone marrow cell morphologies using deep neural networks on a large image data set. Blood, The Journal of the American Society of Hematology, 138(20):917-1927) 2021b.
Aayush Mishra and Karan Thakkar. Stress testing thain-of-thought prompting for large language models. arXiv preprini arXiv:2309.166: 16621, 2023.
Agata Momot. Common carotid artery ultrasound images. Mendeley Data, 2022.
Anna Montoya, Hasnin, kaggle446, shirzad, Will Cukierski, and yffud. Ultrasound nerve segmentation. wewoooundddde-ee-eeeeee nerve- serve-semeentatinn 2016.
Michael Moor, Qian Huang. Shirley Wu, Michihiro Yasunaga, Yash Dalmia, Jure Leskovec, Cyril Zakka, Eduardo Pontes Reis, and Pranav Rajpurkar. Med-flamingo: multimodal medical few-shot learner. In Machine Learning for Health (ML4H), pp. 353-367. PMLR, 2023.
Roberto Morelli, Luca Clissa, Roberto Amici, Matteo Cerri, Timna Hitrec, Marco Luppi, Lorenzo Rinaldi, Fabio Squarcio, and Antonio Zoccoli. Automating cell counting in fluorescent microscopy through deep learning with c-resunet. Scientific Reports, 11(1):22920, 2021.
Geoff Norman, Meredith Young, and Lee Brooks. Non-analytical models of clinical reasoning: the role of experience. Medical education, 41(12):1140-1155 2007.
OpenAL. Hello gpt-4o.
2024a.
OpenAI. Learning to reason with llms. http:://ooeaaiiccocmmmddeeeeeeee with-llms/ 2024b.
OpenAI. Thinking with images. http:://ggnaaaccmmmmeecctttiiiie
2024c. 2024c.
OpenAI. Introducing gpt-4. I in the api, 2025a. URI. https///oopppaiicoo/ieeeeegg---
OpenAI. Introducing gpt-5. http://oppmiiiccmmmma-laaaaaaaaa
2025b.
OpenAl. Openai 03-mini. http:////eeiaai.oommmmee/////o/m 2025c.
17

---

Published as conference paper at ICLR 2026
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
Prasanna Porwal, Samiksha Pachade, Ravi Kamble, Manesh Kokare, Girish Deshmukh, Vivek Sa- hasrabuddhe, and Fabrice Meriaudeau. Indian diabetic retinopathy image dataset (idrid), 2018. URL httpss//dx..ooi.rr wtt.d/.orrrgg/022222/////
Bo Qian, Hao Chen, Xiangning Wang. Haoxuan Che, Gitaek Kwon, Jaeyoung Kim, Sungjin Choi, Seoyoung Shin, Felix Krause, Markus Unterdechler, et al. Drac: diabetic retinopathy analysis challenge with ultra-wide optical coherence tomography angiography images. arXiv preprint arXiv:2304. 0xX.022304.02389. 2023.
Gwenolé Quellec, Mathieu Lamard, Fierre-Hent Conze, Pascale Massin, and Béatrice Cochener. Au- tomatic detection of rare pathologies in fundus photographs using few-shot learning. Medical image analysis, 61: 61:101660, 2020.
Md Mizanur Rahman. Brain cancer- mri dataset. Mendeley Data. 1, 2024.
Tawsifur Rahman, Amith Khandakar, Muhammad Abdul Kadir, Khandaker Rejaul Islam, Khandakar Islam, Rashid Mazhar, Tahir Hamid, Mohammad Tariqul Islam, Saad Kashem, Zaid Bin Mahbub, et al. Reliable tuberculosis detection using chest x-ray with deep learning, segmentation and visualization. leee Access, 8: 19166- 2020.
Tawsifur Rahman, Amith Khandakar, Khandaker Reajul Islam, Md Mohiuddin Soliman, Moham- mad Tariqul Islam, et al. Aseptic loose hip implant x-ray database. https//www..aggle..omddduu tawsifturahmmnnassepiic.. wttp:///asppppppppeeeeeeeeeeee implant-xxay sett--xay-ddttaaaa database, 2022.
Netherlands Rotterdam Ophthalmic Institute. Rotterdam Eye Hospital. Justraigs challenge training data set, January 2024. URL
Salman Sajid. Dental condition dataset. http://www..gggggccoommmmmc///ms 2024.
Andrew Sellergren, Sahar Kazemzadeh, Tiam Jaroensri, Atilla Kiraly, Madeleine Traverse, Timo Kohlberger, Shawn Xu, Fayaz Jamil, Cían Hughes, Charles Lau, Justin Chen, Fereshteh Mahvar, Liron Yatziv, Tiffany Chen, Bram Sterling, Stefanie Anna Baby, Susanna Maria Baby, Jeremy Lai, Samuel Schmidgall, Lu Yang, Kejia Chen, Per Bjornsson, Shashir Reddy, Ryan Brush, Kenneth Philbrick, Mercy Asiedu, Ines Mezerreg, Howard Hu, Howard Yang, Richa Tiwari, Sunny Jansen, Preeti Singh, Yun Liu, Shekoofeh Azizi, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Riviere, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Elena Buchatskaya, Baptiste Alayrac, Dmitry Lepikhin, Vlad Feinberg. Sebastian Borgeaud, Alek Andreev, Cassidy Hardin, Robert Dadashi, Léonard Hussenot, Armand Joulin, Olivier Bachem, Yossi Matias, Katherine Chou, Avinatan Hassidim, Kavi Goel, Clement Farabet, Joelle Barral, Tris Warkentin, Jonathon Shlens, David Fleet, Victor Cotruta, Omar Sanseviero, Gus Martins, Phoebe Kirk, Anand Rao, Shravya Shetty, David Steiner, Can Kirmizibayrak, Rory Pilgrim, Daniel Golden, and Lin Yang. Medgemma technical report, 2025. URL https://arxiivorr/abb////// 052001..
Mehmet Saygin Seyfioglu, Wisdom O Ikezogwo, Fatemeh Ghezloo, Ranjay Krishna, and Linda Shapiro. Quilt-llava: Visual instruction tuning by extracting localized narratives from open- source histopathology videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 13183-13192, 2024.
Hao Shao, Shengju Qian. Han Xiao, Guanglu Song, Zhuofan Zong, Letian Wang, Yu Liu, and Hongsheng Li. Visual cot: Advancing multi-modal language models with comprehensive dataset and benchmark for chain-of of-thuggtt reasoning. Advances in Neural Information Processing Systems, 37:8612-866 26422 2024.
FA Sharifullin, DD Dolotova, TG Barmina, SS Petrikov, LS Kokov, GR Ramazanov, YR Blagosklonova, IV Arkhipov, IM Skorobogach, NN Cheremushkin, et al. Creation of a dataset of msct-images and clinical data for acute cerebrovascular events. Russian Sklifosovsky Journal" Emergency Medical Care". 9()):311-23) 2020.
18

---

Published as conference paper at ICLR 2026
Chaoyin She, Ruifang Lu, Danni He, Jiayi Lv, Yadan Lin, Meiqing Cheng, Hui Huang, Lida Chen, Wei Wang. and Qinghua Huang. A retrospective systematic study on hierarchical sparse query transformer- assisted ultrasound screening for early hepatocellulaar carcinoma, 2025. URL https//aaxxixorg/ab////55 03772.
shenggan, Nicolas Chen, cosmicad, and akshaylamba. Beed: Blood cell count and detection, 2018, URL Dataset.
Osamah Taher and Kasım Ózacar. Hecapsmet: An enhanced capsule network for automated heel disease diagnosis using lateral foot x-ray images. International Journal of Imaging Systems and Technology. 34 (3):223084, 2024.
LASA Team, Weiwen Xu, Hou Pong Chan, Long Li, Mahani Aljumied, Ruifeng Yuan, Jianyu Wang, Chenghao Xiao, Guizhen Chen, Chaoqun Liu, Zhaodonghui Li. Yu Sun, Junao Shen, Chaojun Wang. Jie Tan, Deli Zhao, Tingyang Xu, Hao Zhang, and Yu Rong. Lingshu: A generalist foundation model for unified multimodal medical understanding and reasoning, 2025. URL https///ariiiv.maaaa//5550.... 07044.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 2rXiv:2302.1333 13971, 2023.
Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The ham 10000 dataset, large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific data, 5(1):1-9, 2018.
Andru P Twinanda, Sherif Shehata, Didier Mutter, Jacques Marescaux, Michel De Mathelin, and Nicolas Padoy. Endonet: a deep architecture for recognition tasks on laparoscopic videos. HEEE trunsactions on medical imaging, 36(1):86 97, 2016.
Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing. Shenglong Ye, Jie Shao, Zhaokai Wang. Zhe Chen, Hongjie Zhang, Ganlin Yang, Haomin Wang, Qi Wei, Jinhui Yin, Wenhao Li. Erfei Cui, Guanzhou Chen, Zichen Ding, Changyao Tian, Zhenyu Wu, Jingjing Xie, Zehao Li. Bowen Yang. Yuchen Duan, Xuehui Wang. Zhi Hou, Haoran Hao, Tianyi Zhang, Songze Li, Xiangyu Zhao, Haodong Duan, Nianchen Deng, Bin Fu, Yinan He, Yi Wang, Conghui He, Botian Shi, Junjum He, Yingtong Xiong, Han Lv, Lijun Wu, Wenqi Shao, Kaipeng Zhang, Huipeng Deng, Biqing Qi. Jiaye Ge, Qipeng Guo, Wenwei Zhang, Songyang Zhang, Maosong Cao, Junyao Lin, Kexian Tang, Jianfei Gao, Haian Huang. Yuzhe Gu, Chengqi Lyu, Huanze Tang. Rui Wang, Haijun Lv, Wanli Ouyang, Limin Wang, Min Dou, Xizhou Zhu, Tong Lu,, Dahua Lin, Jifeng Dai, Weijie Su, Bowen Zhou, Kai Chen, Yu Qiao, Wenhai Wang, and Gen Luo. Internv13.5: Advancing open-source multimodal models in versatility, reasoning, and efficiency, 2025a. URL http://aarriv.orrrab/555.0. 18265.
Yaoting Wang. Shengqiong WU, Yuecheng Zhang, Shuicheng Yan, Ziwei Liu, Jieho Luo, and Hao Fei. Multimodal chain ofain-of-thought reasoning: A comprehensive survey, 2025b. URL https://arxiv.or//bbb/ 2503. 12605.
Alexander Wong, Hayden Gunraj, Vignesh Sivan, and Masoom A. Haider. Synthetic correlated diffusion imaging hyperintensity delineates elinically significant prostate cancer. Scientific Reports, 12(3376), 2022. doi: 10. 01118//44119-0022-066666-
Peiran WU, Che Liu, Canyu Chen, Jun Li, Cosmin Bercea, and Rossella Arcucci. Fmbench: Benchmarking fairness in multimodal large language models on medical tasks. arXiv preprint 24XXi:24100000833 2024.
Guowei Xu, Peng Jin, Hao Li, Yibing Song, Lichao Sun, and Li Yuan. Llava-cot: Let vision language models reason step-by-step. 2024. URL https://arxiv.rr/abb//11111 10440..
Guowei Xu, Peng Jin, Ziang Wu, Hao Li, Yibing Song, Lichao Sun, and Li Yuan. Llava-cot: Let vision language models reason step-by-step step. In Proceedings of the IEEE/CVF International Conference Computer Vision. pp. 2087-20829 2025.
Pusheng Xu, Xiaolan Chen, Ziwei Zhao, and Danli Shi. Evaluation of digital ophthalmologist app built by gpt4-v (ision). medRxiv, pp. 2023- 11. 2023.
Qianqi Yan, Yue Fan, Hongquan Li, Shan Jiang. Yang Zhao, Xinze Guan, Ching-Chen Kuo, and Xin Eric Wang. Multimodal inconsistency reasoning (mmir): A new benchmark for multimodal reasoning models. arXiv preprint arXi: 16033, 2025.
19

---

Published as conference paper at ICLR 2026
Jin Ye, Guoan Wang, Yanjun Li, Zhongying Deng, Wei Li, Tianbin Li, Haodong Duan, Ziyan Huang, Yanzhou Su, Benyou Wang, al. Gmai-mmbench: comprehensive multimodal evaluation benchmark towards general medical ai. Advances in Neural Information Processing Systems, 37:94327-994277, 2024.
Xu Yiming, Zheng Bowen, Liu Xiaohong, Wu Tao, Ju Jinxiu, Wang Shijie, Lian Yufam, Zhang Hongjun, Liang Tong, Sang Ye. Jiang Rui, Wang Guangyu, Ren Jic, and Chen Ting. Annotated ultrasound liver images, November 2022. URL https://doi.oooo http:::/dii.org/005288/1/ (t..22///eeodo.72776666.
Sheng Zhang, Yanbo Xu, Naoto Usuyama, Hanwen Xu, Jaspreet Bagga, Robert Tinn, Sam Preston, Rajesh Rao, Mu Wei, Naveen Valluri, al. Biomedelip: multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs. arXiv preprint vrXiv:2303.00913 2023.
Jinyu Zhao, Yichen Zhang, Xuehai He, and Pengtao Xie. Covid-19 19 lung et scans, 2020. URL https: //wwkkagle..om/4465440000
20

---

Published as conference paper at ICLR 2026
APPENDIX
This supplementary material provides more detailed information about M3CoTBench. The content of each appendix is summarized as follows:
Appendix A. Provides a detailed description of how large language models are applied in this work. This includes their use in assisting writing, guiding dataset construction, supporting annotation processes, and contributing to model evaluation.
Appendix B. Offers comprehensive information about the dataset used in this study, including the sources of the data, the diseases and abnormalities covered, the distribution of image resolutions, detailed task specifications in the benchmark, and descriptions of CoT annotations.
Appendix C. Provides an in-depth explanation of the evaluation methodology. including the metries used, the design of prompts, and additional clarifications how model performance is measured and interpreted.
Appendix D. Presents supplementary experimental results that complement the main paper, along with illustrative case studies that demonstrate model behavior and practical outcomes in various scenarios.
Appendix F. Discusses the known limitations of this study, including potential weaknesses in the methodology, dataset coverage, and model generalizability, providing a balanced view of the research.
Appendix F. Highlights potential societal implications of this work, considering both beneficial applications and possible risks, and reflecting on the broader impact of deploying such models in real-world scenarios.
A THE USE OF LARGE LANGUAGE MODELS
We use large language models solely for polishing our writing, and we have conducted a careful check, taking full responsibility for all content in this work. In addition, LLMs and MILMs were also used in the construction of the dataset and the evaluation of models, and the specific usage has been described in detail in the main text.
osis monocyte
fastion
brain
retinitis pignentosa edema
mabler Repteratiio
pro coliferatiiv ciatedic net Prtimoatit thy
chick bacckeppxx
tubber cut thuriesiion
di
diabetic
disteciee
mass
desophagitis
central
serous
chor
iortinntinyyy
benigg gn keratosia sosis
00
Tioma
tumor
thologic
ulterative
u foer
pulioraary
infiltration
colitis
monati
staleltersaste
disease
brain
non-enhancing
tumor
ing
enhancinn
lesion
on
spondylolistheese
actinic
abale
keratoses
atelectasi
reforabie
glascome
seteets
non-re
con-reeeealle
con-referee
aglaucoma
prolifern
Central
retinel
occlus
monkeypox
basophil
meningioma
high
Coats
nevus
hheporrhage
(bleeding)
in
stroke
benign
tumor
incarce
appendicitis
liver
abscess
infarct
cancer
lung.
cancer
mal Lighnnt
brain
ca
tumor
eosinophil
covid=12 dooctiin
nodule
retinal
detachment
Apperinflated
pituitary
car stor
brain
1schentc
stroke
enhancing
sever's
's
disease
cratae
cirence
Tymphocyte
tumor
cowpox
p presaaats
heet spura
pleural
1
e effusion
dermatofibrom
neutrophil
scoliosis
wadlery notessalaanns
Figure A1: Word cloud for abnormality and diseases included in Bench. The word cloud below visualizes the frequency and variety of these labels, highlighting the spectrum of diagnostic conclusions and imaging findings represented.
21

---

Published as conference paper at ICLR 2026
B MORE DETAILS ABOUT THE DATASET
B. SOURCE DATASET INFORMATION
Images M3CoTBench dataset are collected from publicly available datasets, offering a highly diverse and representative foundation for training and evaluating multi-modal medical reasoning models. Its comprehensive coverage modalities, anatomies, time periods, and geographic sources ensures broad applicability and robustness in real-world clinical scenarios. The detailed information of data sources can be seen in Table Al.
B.2 DISEASES AND ABNORMALITIES
This dataset contains a wide range of diseases and abnormalities. word cloud illustrating their distribution is shown in Figure AI.
Image Resolution Distribution
X-ray
MRR
Histo
Colonasospy
2000
leparcocopy
Dermatology
FP
Gastroscapy
FFAA
Hiight
OC1
1000
Endowcopy
Capsule Endoscopy
IR
Bronchoscopy
Nudiear Medicine
Fluarsscenco Microscopy
0
1002
2002
2003
4003
Width (pisele)
Figure A2: Image resolution distribution in M3CoTBench. Most images are concentrated below width of 1200 and height of 1500, though exhibit higher resolutions.
B.3 IMAGE RESOLUTION DISTRIBUTION
For the images, we retained their original sizes as provided in the source datasets, without applying any additional compression or resizing. Some images may have been preprocessed in their original datasets. However, for tasks such entity linking, grading, and image quality comparison, we concatenate two images side by side, which results in increased image width. The resolution distribution information can be seen in Figure A2.
Diversity in examination types: The dataset covers 24 imaging modalitics and examination meth- ods, which be grouped into six major categories: ophthalmic imaging, radiology, endoscopy, microscopy, ultrasound-based examinations, and surface-level inspections. These include slit lamp photography (SLP), fundus photography (FP), optical coherence tomography (OCT), optical coherence tomography angiography (OCTA), scanning laser ophthalmoscopy (SLO), fundus fluorescein angiography (FFA), X-ray, computed tomography (CT), magnetic resonance imag- ing (MRI), ultrasound (US), infrared reflectance (IR), nuclear medicine, fetoscopy, laparoscopy, colonoscopy, gastroscopy, capsule endoscopy. bronchoscopy. ENT endoscopy, histology, cytology. fluorescence microscopy, dermoscopy, and intraoral examination.
Diversity in anatomical regions: The datasets encompass broad spectrum of anatomical regions, including but not limited to eye, skin, chest (lungs and heart), brain, abdomen (liver,
22

---

Published as conference paper at ICLR 2026
kidney, stomach, etc.), oral cavity, uterus and fetal environment, breast, vertebrae, hip, knee, foot, blood, and bone marrow. This anatomical diversity supports the evaluation of models' capability across different clinical tasks and organ systems.
The included datasets were published across wide temporal range, from earlier benchmarks to very recent contributions. This time span captures the evolution of imaging quality, annotation practices, and diagnostic standards, making the dataset suitable for both historical benchmarking and future-proed model evaluation.
Diversity in publication years:
Geographic diversity: The data sources originate from over a dozen countries and regions, reflecting variety of healthcare environments, population demographies, and medical imaging protocols. This geographic diversity enhances the robustness, fairness, and woal-world applicability ialliiiinstiutionnll settings. The
of models trained on the dataset, particularly in cross-domale geographic distribution of data sources illustrated in FigureA3.
Loedsa
Auckio
Seatte
Toroaks Now Yodk City Birhedd
Sasza Cau
Buroieea
Allem
Deijin
Los Augetz
Kareak
Teles
Ehaighht
Edss
(Chenptal
Deta
Diiice
Ouasechvo
Ho Cha Miah
Salarai Adiia
Rio Neato
Beubens
Sydres
Figure A3: Geographic distribution of data sources in the dataset. Red flags indicate the locations of contributing hospitals institutions, where applicable. Due the complex and varied origins of some datasets, exact source locations may not always be clearly identifiable.
B.4 DETAILED INTRODUCTION TASKS
The benchmark encompasses diverse range of tasks that mirror real-world clinical challenges in medical visual-language reasoning. These tasks are designed evaluate not only a model's ability to recognize and classify visual information, but also its capacity to comprehend spatial, procedural, and diagnostic contexts. Broadly, the tasks can be grouped into mid-level visual understanding, such as identifying image modality, recognizing anatomical structures. or assessing image quality. These tasks primarily test the model's capability to extract and interpret observable features from the image. Knowledge- based reasoning tasks, the other hand, require integration of visual features with clinical knowledge, commonsense reasoning, or multi-step inference. These include complex tasks such as diagnosing diseases, predicting disease progression, grading severity, planning clinical actions, identifying causal relationships.
conceptual levels: Perceptual-leve tasks focus on low- to
Understanding and recognizing the imaging modality involved, such CT, MRI, X-ray, OCT, demonstrates the model's 's awareness of different diagnostic techniques and their clinical contexts.
Modality Examination Types:
23

---

Published as conference paper at ICLR 2026
Table Al: Data sources of different modalities in M3CoTBench
Dataset
Anatomical Region
Modality Examination Type
Oph (Xu et al., 2023)
Eye
SLP, FP, OCT, US, SLO, FTA
IDRDD (Porwal et ul., 2018)
Eye
FP
JustRAIGS (Rotterdam Oph-
Eye
FP
thalmic Institute, 2024)
RIADD (Quellee et al., 2020)
Eye
FP
DRAC 2022 (Qian ct al., 2023)
Eye
OCTA
RAVIR (Hatamizadch et al., 2022;
Eye
IR
Halamizadeh, 2020)
ISIC 2018 (Codella et al., 2019)
Skin
Dermoscopy Dermoscopy
НAM10000 (Tschandl et al., 2018)
Skin
MSLD v2.0 (Ali et al., 2024)
Skin
Dermoscopy
VQA-RAD (Lau al., 2018)
Chest, Abdomen, Brain Chest, Abdomen, Brain
X-ray, CT, Nuclear Medicine
"QA-Med-2011 (Ben Ahacha al., 2019)
X-ray, CT, MRI, US
SLAKE (Liu al., 2021)
Chest, Abdomen, Brain
X-ray, CT, MRI X-ray
Chest XR COVID-19 (Akhloufi & Chetoui, 2021)
Chest (Lung)
TB Chest X-ray (Rahman et al., 2020)
Chest (Lung)
X-ray
Heel Bone (Taher & Ozacar,
Foot
X-ray
2024)
Digital Knee X-ray (Gornale Knee
X-ray
Patravali, 2020)
Vertebrae X-ray (Fraiwan et al., 2022)
Vertebrae
X-ray
Hip Implant X-ray (Rahman et al., 2022)
Shoulder
X-ray
CT Kidney (Islam et al., 2022)
Kidney Lung
CT
COVID-19 Lung CT (Zhao et al., 2020)
CT
Brain Stroke CT (Koę et al., 2022) LDCTIQAC 2023 (Lee et al.,
Brain
CT
Abdomen
CT
2025)
MSCT-Image Dataset (Sharifullin ct al., 2020)
Brain
CT
PENGWIN (Liu et al., 2025a;b)
Pelvis
CT
ToothFairy (Bolelli et al., 2024;
Oral Cavity
CT
Lumetti et al., 2024; Cipriano al., 2022)
Brain Tumor (Bhuvaji al., 2020)
Brain
MRI
LGG Segmentation (Buda et al.,
Brain
MRI
2019)
Brain Cancer MRI (Rahman,
Brain
MRI
2024)
BRATS-SSS ASASS-SSA (Adewole et al., 2023)
Brain
MRI
Cancer-Net PCa-Data (Wong
Prostate
MRI
al., 2022: Gunraj al., 2023)
SIMON MRI (Duchesne al.,
Brain
MRI
2019)
24

---

Published as conference paper at ICLR 2026
Table Al (continued); Data sources of different modalities in M3CoTBench
Dataset
Anatomical Region
Modality Examination Type
BUSI AA--Dhabyani 2020)
Breast
US
FH-PS-AOP (Jieyun ZhanHong. 2023)
Fetal
Nerve Segmentation (Montoya
Neck
al., 2016)
Carotid Artery (Momot, 20222
Neck
Liver-US (She et al., 2025)
Liver Liver
US
Annotated Liver US Dataset (Yim-
ing et al., 2022)
BUS-BRA (Gómez-Flores et al., 2024)
Breast
US
Quilt-VQA (Seyfioglu et al.,
Multi-regions
Histology
2024)
BCI (Liu et al., 2022)
Breast
Histology Histology
Colorectal Histology
Colon and Rectum
MNIST (Kather et al., 2016)
GCHTID (Lou et al., 2024)
Stomach
Histology Intraoral Examination
Dental Condition Dataset (Sajid,
Oral Cavity
2024)
BACH (Aresta et al., 2019)
Breast
Histology
CMIA Histological Slides
Lung. Breast
Histology
Fluorescent Neuronal
Brain
Fluorescent Microscopy
Cells (Morelli al.. 2021)
BCCD (shenggan al., 2018)
Blond
Cytolngy
Raabin- WBC (Kouzehkanan et al.,
Blood
Cytology
2022)
BMC (Matek al., 2021a;b)
Bone Marrow
Cytology
VndoVis-17- 17-VLQ (Allan et al.,
Abdomen
Laparoscopy
2019)
m22cai16-tool (Twinanda al., 2016)
Abdomen
Laparoscopy
ImageCLEFmeed MEDVQA- Gl (Hicks et al., 2023)
Gastrointestinal Tract
Colonoscopy, Gastroscopy
Bronchoscopy Dataset (Deng
Airway Tract
Bronchoscopy
al., 2023)
Capsule Vision 2024 (Handa al., 2025)
Gastrointestinal Tract
Capsule Endoscopy
ENTRep Challenge 2025 (ENT, 2025)
Ear, Nose, Throat
ENT Endoscopy
FetReg (Bano ul., 2021)
Uterus/ Fetal Environment Uterus Fetal Environment
Fetoscopy Fetoscopy
Fetoscopy Placenta Data (Bano al., 2020)
25

---

Published as conference paper at ICLR 2026
Image Quality Assessment: Evaluating whether image is diagnostically adequate, and comparing the relative quality between multiple images when necessary. This reflects the model's ability to judge image usability in clinical practice.
General visual recognition tasks, including identifying anatomical structures, tissues, or medical devices, without explicit spatial reference.
Recognition:
Region-specifit identification tasks where the model must recognize or interpret a particular area in the image based on the question or accompanying text.
Referring Recognition:
Counting: Quantifying specific elements an image, such as surgical tools, lesions, polyps, or cells, often requiring precise object detection and differentiation.
Localization: Identifying the spatial location regions of interest, such as lesions, organs, or abnormal structures, testing the model's 's understanding of spatial relations and context.
Diagnosis: Inferring the presence of abnormalities. diseases, or clinical conditions based on image and text input; this is the most common and clinically important task category.
Grading: Assessing the severity or stage medical condition, such as cancer staging or diabetic retinopathy levels, requires a nuanced interpretation of visual cues.
Symptom Identification:
Recognizing observable clinical signs or inferring underlying symp- toms based on the visual features of the image and contextual cues.
Clinical Action Planning: Making decisions about the next steps in patient care, such as recommending further examinations, procedures, or treatment options, demonstrating clinical reasoning ability.
Estimating future disease progression, risks of complications, or expected outcomes, often involving mudti-modal reasoning over image and text inputs.
Prediction:
Functional Understanding: Interpreting the physiological function of organs, the intended use of medical instruments, or the purpose of surgical actions, integrating procedural and anatomical knowledge.
Identifying the cause etiology of symptom condition, requiring the model to reason about potential underlying mechanisms or prior events.
Causal Reasoning:
B.5 COT ANNOTATION
The CoT annotations are collaboratively generated by medical experts and MLLMs, generally following a four-part structure: /cxamination type, key feature, key conclusion. additional analysis). This approach aligns closely with clinical reasoning patterns used by physicians, who often begin by identifying the type of examination or modality, observing key findings, deriving conclusions, and, when necessary, conducting further interpretation or differential diagnosis. The length and structure of CoT vary depending on the task. For tasks such as recognition, diagnosis, and grading. three-step format, (cxaminntttiiiioonn type, key feature, key conclusion), ), is generally sufficient. contrast, more complex tasks like treatment planning, causal reasoning, symptom analysis, prognostic prediction, or functional interpretation often require a four-step annotation capture the depth of reasoning. When it comes identifying the imaging modality, CoT length depends on the nature the question. For example, general tasks, it may not be necessary to analyze image features to identify the modality explicitly. However, in questions specifically targeting modality identification, CoT annotations typically include two steps. focusing clues about the imaging technique used. Notably, during examination modality statisties, some subtypes are grouped into broader categories. However, in CoT annotations, these modalities are often distinguished more finely. For example, IHC and HE are treated separately, are MRI T wweeghtte and T2-weighted images. Examples of CoT annotation are shown Figure A4 and Figure AS.
characteristic visual
C SUPPLEMENTARY TO EVALUATION PROCESSES
C.1 EXAMPLES OF PATH SIMILARITY IN REASONING CONSISTENCY EVALUATION
To evaluate the structural stability of reasoning in multi-step tasks, we introduce path consistency metric that measures the similarity of reasoning paths across instances of the same task type. Unlike traditional metrics that treat reasoning steps as unordered, this metric accounts for sequential structure
26

---

Published as conference paper at ICLR 2026
a) What esaminations need to dane next?
CoT Annotation
Examination type: Scanning laser optthalmoscopp (SLO).
Key slame-shaped bemorthages, especially
perlunily dilation of
optic veins, extensive
3. Key conclusion: The likely diagnosis based on this image is central retinal vein occlusion.
Additional Analysis suggestions): Fiuorescein angtography demeastrates retinal circulation patterns identaties areas nonperfusion. maculer edlema and structural changes characteristic venous occlusive disease.
reveals
b) What might be the cause of this condition?
CoT Annotation
Examination type: Fundus photography.
Key features: The image shows with aratlered pigmentary depusits.
distortion optic
Key cunclusion: The likely disgnosis based on this image could be suspicious chromic tractional retinal detachment((RR))
Reasoning): Chronic develops from prolonged fibrovascular preliferation creatingg mechankal traction rettiaal tissuc, commonly seen in advanced prolicerative diabetic retinopathy with Inadequate glycemic control
a. Additional Analysis
<) What disease likely associated with the picture? (Sclect option) Liver Cancer Hepatills Cirrbosis
CoT Annotation
Esamination type:
Fatty disease
Key features: The liver contains an irregular, hypodense mass with heterogeneous enbancement.
conclusion: The likely diagnosis is liver
What
should be
CoT Annocation
manage this situstin??
(Select that apply)
Examination type: X-ray.
Reduca
B. Engage in regular moderate exercise
Key features: The cardiac silhouette is enlarged, occupying more than half of the Uhoracic cavily's transverse diameter.
C. Smoke more to relleve stress
Avoid Maintain healthy weight
alcohol
conclusion: The likely diagnosis is cardiomegaly.
Slap medications
consulting
G. Monitur bland pressure regularly
Addditional Analysis (action suggattioss/aptan analysis): Option reduction prevents fhald retention and hypertensonn. Option B: Moderate exercise streagthens carciovascular system under guidance. Option C: Smaking warsens carciovascular disease directly. Option D: Alcohol excess causes cardiemyopathy and artbythmias. Option Weight loss reduces cardiac workload, Option Medication acherence cracial
H. Fat more processed foods
management Option Bloud prossure monitoring treatment effectiveness. Option Processed foods worsen hypertensian and inflammelin..
CoT Annocation
surgical or instruments the image? (Select ane option)
A. No instruments present
Esamination lype: Colunewagyy
B. Scalpel
C. Metal clip
Key features: Three separate silver colored cylindrical structures in the image.
Surgical sponge
conduusio:: There three metal clips in the image.
Figure A4: Examples of CoT annotations with corresponding images and questions M3CoTBench (1). Different types of questions are annotated with different lengths of CoT steps. For example, diagnostic (c) and recognition (e) guestions involve three annotation steps, while action oetion-planning (a, d) and causal analysis (b) questions are annotated with four steps.
27

---

Published as conference paper at ICLR 2026
True Palse: This abnormality will certainly squamous cell carcinoma
CoT Annotation
Examination type: Dermatological image.
2. Key festures; Localized rough, scaly patch with central depression, yellowish adherent scales, and surrounding erythema.
3. Key conclusion: likely diagnosis
4. Additional Analysis (prediction): Only small percentage (estimated around 0.1-10% per leston per year) may evolve into squamous cell carcinoma.
b) How many polyps are the image?
CoT Annotation
Examination type: Gastroscopy.
Key mucosal appears lacks any visiible elevated or protsberant masses..
Key canclusion: There is polyp in the image.
Annotation
True False: left asteoerihviils severity than right.
Examination type: ray.
2. Key features: In the left image, joint space relatively well preserved; only mild narrowing. in the right image, abvious joint space narrowing, subchondral sclerosis, and osteophyle formation.
Key conclusion: The right image left.
knee osteoarthritis severity
of images by Image right segmentation mask specific region the image on the left. What does the white area in the right image option)
Examination type: FLAIR))
the left image? (Select one
2. Key features: The right image shows an irregular white area with incressed signal intensity, corresponding to the bright hyperintense region in the left cranial image.
Glioma
edema
Cerehrospinal fiuld Skull
Normaa brain tissue
Key conclusion: The white
image represents glloma
Which of the following symptoms is this likely experience? (Sclect
CoT Annotation
Examination type:
apply)
2. Key features: Enlarged gallbladder with low-attenuation common bill duct alongaide dilated intrahepatic bile ducts.
A) RUQ pain (Right Upper Quedrant pain) B) Jaundice C) Dark urine
clay colored stools Weight
Key conclusion: The possible diagnosis
dart and obstruction.
4. Additional Analysis (symptom analysis): Obstruction of the cystic duct and common biie duct (CBD) can lead to bile stasis, causing: KUQ pain due gallbladder distension or inflemmation. Jaundioe from buildup of bilirubin, Dark urine because conjugated bilirubin excreted urine. Clay colored pigments intestines.
bile
Figure A5: Examples of CoT annotations with corresponding images and questions in M3CohBench (3). Different types of questions are annotated with different lengths of CoT steps. For example, counting (b), grading (c) and referring recognition (d) questions involve three annotation steps, and prediction (a) and symptom questions are annotated with four steps.
28

---

Published as conference paper at ICLR 2026
by comparing generated paths to canonical reference using the normalized longest common subsequence (LCS). The final score, averaged over 13 tasks, reflects the model's ability to follow consistent, interpretable reasoning patterns, a key property for transparency and trust in medical decision-making. Here are some examples to show the specific calculation method:
Consider the following reasoning paths, where each element is of {modality, feature, diagnosis, treatment) representing a progression from identifying the imaging type, describing visual findings, inferring clinical conditions, to suggesting appropriate medical interventions.
Example 1: Pi = mmodality, feature, diagnosis]. P2 = feature, modality, diagnosis]. Then LCS is [modality, diagnosis] and [feature, diagnosis]. [LCS(P, P2) 2, thus
sim, max 3)
0.67.
(A1)
Example Pi
modality, diagnosis, treatment]. Pa modality, feature, diagnosis, treatment]. Then the LCS is [modality, diagnosis, treatment] [CS((1, = 3, thus
3 max(3, 4)
sim((PI, P2)
0.75.
(A2)
Example 3:
Pi
mmodality, feature, treatment] Then the LCS is [modality, diagnosis, treatment)
P2
modality, feature, diagnosis, treatment]. The [LCS(P.. P2) = 3, thus
3 sim (P1, Pz) max(3, 4)
3 0.75.
(A3)
Example 4:
P1 = [eatuuee modality, diagnosis, treatment] P/2 = [modality, feature, diagnosis, treatment]. Then the L.C [modality, diagnosis, treatment] and [feature, diagnosis, treatment] The LCS(P;, 3, thus
sim,(Pi,P3) max(3, 4) 0.75.
(A4)
C.2 EVALUATION PROMPTS
During evaluation, use GPT-4o and to the questions. Accuracies of the direct answers and CoT answers are both averaged from the two models. We also use GPT-4o to assess the correctness of each step, and use GPT-4o and Gemini 2.5 to determine the step order in the CoT output. The consistency scores of the steps in CoT outputs are averaged from the two models. Since the feature description and additional analysis parts are relatively subjective, with multiple valid expressions for the same meaning, adopt more lenient instructions for these components. In contrast, apply stricter criteria the examination modality and key conclusion steps.
assess the accuracy of answers
C.2.1 EVALUATION PROMPTS FOR ANSWER ACCURACY
The prompt for calculating accuracy for both direct outputs and CoT outputs is shown below:
29

---

Published as conference paper at ICLR 2026
Prompt for calculating accuracy for both direct outputs and CoT outputs
You are medical evaluation expert:
#Your tasks:
1. Extract the final answer only from the model's prediction below. 2. Judge if it matches the provided ground-truth answer. #Type instruction:
Return ONLY JSON object with the EXACT format below (no extra text):
"match": true false, "final_answer "the extracted final answer text" }}
Inputa:
Question: (question)
Ground-truth Answer:
(answer)
Model's Prediction: (prediction)
C.2.2 EVALUATION PROMPTS FOR PRECISION CALCULATION
The prompt for precision calculation is:
Prompt for calculating precision for CoT outputs
Given solution with multiple reasoning steps for an image-based problem, reformat it into well-structured steps and evaluate their correctness.
Step 1: Reformatting the Solution
Convert the unstructured solution into distinct reasoning steps while:
Preserving all original content and order.
Not adding new interpretations.
Not omitting any steps.
# Step Types
Image Modality or Examination Types
Describes the imaging type or procedure used (e.g., CT, MRL, histology, endoscopy,
Anterior segment slit lamp examination, scanning laser ophthalmoscopy).
Focuses on technical aspects without interpretation.
Key Image Feature Analysis
Pure visual observations obtained from the image..
Describes visible structures or abnormalities the image. (c.g., The eye exhibits notable redness).
Pure observation without inference.
Identification, Localization, Diagnostic Conclusions other Conclusions
Provides specific findings or diagnosis based on image features. (e.g., the likely diagnosis based on this image could be Coats" disease).
30

---

Published as conference paper at ICLR 2026
Includes reasoning and clinical conclusions.
T The location of the abnormalities organs.
Classification conclusion for cells organs. (e.g., The cell is Basophil.)
Recognition conclusion for instruments their processes. (e.g., The current state of the monopolar curved scissors is cauterization.)
Other conclusions. (e.g.. The red area in the right image represents in the left image the pubic symphysis.)
4. Knowledge-Basee Based Differential Exploratory Analysis
This not mandatory and is only required for those requiring additional analysis. Includes disease progression prediction, organ/cell function, treatment or further examination suggestions, cause analysis of disease or abnormalities, other medical knowledge, and step-by-step analysis of multiple-chor choice options.
# Step Requirements
Each step must be atomic (one conclusion per step)
No content duplication across steps
Step 2: Evaluating Correctness
Evaluate each step against:
# Ground Truth Matching
For modality or examination types:
Must strictly correspond to ground truth; different wording allowed if meaning is equiva- lent.
MRI TI/T2/DWI sequences are considered different modalities. Endoscopy answers must exactly match the type, e.g., ENT Endoscopy, gastroscopy. capsule endoscopy, laparoscopy and so on.
Various synonymous expressions for 'Section stained with hematoxylin and cosin (H&E)', e.g., H&F-sttaned slide are all acceptable.
For image feature description:
Mostly-overkap matching: Answers that match the majority of the ground truth and convey the same meaning are considered correct.
For key conclusions:
Should strictly correspond to ground truth; different wording allowed if meaning equivalent.
For additional analysis:
Mostly-overlap matching: Answers that match the majority of the ground truth and convey the same meaning are considered correct.
# Reasonableness Check
Logic is valid
Step information must not contradict any ground truth correct answer
Step information must support or be neutral to correct answer
# Judgement Categories
Match: Aligns with ground truth.
Wrong: Invalid contradictory.
N/A: For background information steps.
# Output Requirements
1. The output format MUST be in valid JSON format without ANY other content. For highly repetitive patterns, output as single step.
Here is the JSON output format:
31

---

Published as conference paper at ICLR 2026
"step_type"; "modality or examination types limage feature description key conclusions additiooal analysis
|estateeeet of the question."
"Step information": "Step result", "judgment": "Match| Wrong/N/A
Your task is to reformat the following solution into discrete reasoning steps and evaluate each step based the ground truth. Input:
[Problem]
(question)
[Solution]
(solution]
[Correct Answer]
(answer)
[Ground Truth Information]
(gt Lanioaanota)ion)
C.2.3 EVALUATION FOR RECALL CALCULATION
The prompt for recall calculation is:
Prompt for calculating recall for CoT outputs
You are an expert system for verifying solutions to medical image- problems. Your task is to match the ground truth middle steps with the provided solution.
# Input Format:
Problem: The original question/task.
2. A Solution of a model.
Ground Truth: Essential steps required a correct answer.
# Matching Process:
You need match each ground truth middle step with the solution. Match Criteria:
The middle step should match the content or be directly entailed by a certain content in the solution.
For subjective or descriptive steps such as image feature descriptions, treatment sugges- tions, discase causes, symptoms cellular/isttrutments functions, match lemiently: A step is "Matched" if the overall meaning largely overlaps with the solution and there is no contradiction, even if wording differs. Exact wording or structure is not required as long the clinical implication is preserved.
For objective steps such as specific diseases, cell names, lesion names, or image modali- ties/examination types, match strictly: The terminology must refer to the same medical
32

---

Published as conference paper at ICLR 2026
concept, though phrasing may differ (e.g.. "retinal detachment" vs. "detached retina" is acceptable).
For modality or examination types, it must strictly correspond ground truth; different wording is allowed if the meaning equivalent. Different types of MRI sequences, like T1/T2/DWI are considered different modalities. Endoscopy answers must exactly match the type, e-g., ENT Endoscopy, gastroscopy, capsule endoscopy, laparoscopy, and so on. Various synonymous expressions for 'Section stained with hematoxylin and cosin (H&E)* e.g., stainet slide are all acceptable.
In all cases, evaluate whether each ground truth step is represented in the solution, either explicitly or with clear implication. # Output Format: JSON array of judgments:
"step_index"; <integer>,
"step_type": "modality or examination types limage feature description kkey conclusions additienal analysis Iretaaement of the question" "judgment": "Matched" "Unmatched",
# Additional Rules:
1. Only output the JSON array with no additional information.
Judge each ground truth middle step in order, without omitting any step. Here is the problem, answer, solution, and the ground truth middle steps: [Problem]
(question)
[Answer]
(answer)
[Solution]
(solution]
[Ground Truth Information]
(gt_annotation)
C.2.4 EVALUATION PROMPTS FOR STEP ORDER RECOGNITION
When computing CoT consistency, it is necessary determine the order of the reasoning steps in the model's output. This requires first classifying the type of each step. Our prompt as follows:
Prompt for step order recognition
Example JSON output structure:
"modality_order 1,
"feature_order" 2,
33

---

Published as conference paper at ICLR 2026
ccnnclusioooodder 3,
oothers_se :echers_order:: 4,
modalltyysubs ["substring 1", "substring 2"),
feature__suss"; ("substring 1", "substring 2"1,
"conclusion_subs";
( ("substring 1", "substring 2"],
"others_subs": ("substring 1", "substring 2"]
System Prompt: You are a medical reasoning pathway analyzer. Analyze an Al's answer to a medical question by extracting information into four categories and determining their appearance order.
Medical Question/Tann Fastio Not provided
Categories:
1. modality _uubs- Imaging/examination methods.
Extract the image modality or examination type. Examples: "Fundus photography", "MRI (FLAIR)", "Anterior segment slit lamp exami- nation", "CT scan", "Ultrasound B B-scan", "Section stained with hematoxylin and cosin (H&E)". "microscopy".. "ENT Endoscopy"
2. feature .ubbe-ssss-s-.. Characteristics visible directly from the image.
Examples: "opaque lens with dense white structure", "ill-defined hyperintense lesion with heterogeneous internal signal the right cerebral hemisphere". "mild sulcal effacement, no significant mass effect"" contours", "bright, dense structure visible in the right kidney's collecting system"
musculoskeletal structures with visible bone
3. conclusion subs Key conclusions including diagnoses, recognition (cell/orga//instru- men//yypes/uurical processes), anatomical locations, grading, counting. Examples: "mature cataract", "consistent with grade diffuse astrocytoma". "kidney stone". "left kidney". "Coats' disease". "Monocyte cell type", "The instrument in the image Tube". "The state of ultrasound probe is idle"
4. others subs Further explanations, treatment info, action recommendations, symptoms, functions, clinical knowledge.
Examples: "Visual function assessment is needed", "Dehydration concentrates urine, increasing supersaturation of stone-forming salts", "COVID-19 is caused by the SARS- CoV-2 virus", "For benign lesions, aggressive treatments like chemotherapy are usually unnecessary", "Option A: Dehydration increases risk. Option B: High oxalate intake promotes stones"
Extraction Rules:
Extract complete phrases sentences for each category.
Extract multiple distinct substrings per category if present.
Copy exact text; do not paraphrase.
Use empty array [] if no content.
Order Rules:
Scan text from start to end to determine which category appears first, second, third, fourth.
Assign 1-4 based appearance sequence.
Assign 0 if category does not appear.
Examples: (1.2.3.4) modality -+ feature conclusion - others, (2.3,1.0) feature conclusion + modality, no others.
Example Input: "Fundus photography of the right eye shows an opaque lens with dense white structure, This appearance is consistent with mature cataract. Visual function assessment needed." Example Output:
1,
"feature_order"; 2,
34

---

Published as conference paper at ICLR 2026
cconclusionordeer 3,
"others_se otherssorderr: 4,
modalityyssbss ["Fundus photography"),
featurre_subs"; ["opaque lens with dense white structure"), "conclusion_subs ("mature eataract"), "others_subs". ("Visual function assessment is needed"]
CRITICAL: You must respond with ONLY valid JSON format. Do not include any other text before or after the ISON object. Your output must be valid JSON in this exact format: {OUTPUT FORMAT)
D SUPPLEMENTARY TO EXPERIMENTS
D.1 SUPPLEMENTARY RESULTS
Here, we present the average response time per question for each MLLM under both the direct and step-by-step settings, as shown in the Table A2. From the table, it is evident that CoT reasoning generally increases the average response time compared to direct output, generating full sequence of reasoning steps requires more computation and context processing. For smaller models, such as LLaVA-Med latency remains low even with CoT enabled, primarily due to their fast inherent reasoning speed and smaller parameter count. Larger models, like Lingshu-32B or Dwen3-VL-Inntrruct step sequences, but the relative increase latency is comparatively modest, likely owing to advantages in parallel processing and context management in larger architectures. Some models naturally display reasoning steps, such the -wen3-VL--Thinking series and LLLVA-CoT, which already have higher baseline latency direct output mode, so the additional delay from CoT prompting limited. contrast, models that normally produce direct answers experience significant latency increase when CoT is enabled, reflecting the additional computational cost of generating interpretable reasoning steps. Overall, response time depends not only on the generation mode (direct vs. CoTT) but also on model size, architecture, and built-in reasoning mechanisms: lightweight models have low latency and limited growth: larger models have higher baselines but relatively controlled increases; and models that generate reasoning steps incur substantial additional inference time.
produce more complex
Table A2: Comparison the average response time per question MLLMSs under direct and step-by-step reasoning conditions. Optimal sub-optimal results are highlighted in bold underline.
Model
Tatreet (s/smmpe/
(esr(s/ammle))
LLavvaCoTT Xu et al. (2025)
0.70
7.78
InternVLL3.588B (Wang al., 2025a)
0.93
16.91
Intem\1.3.5-30B (Wang et al., 2025a)
4.01
66.92
Qwenn-VL-Innstuct-88 (Bai et al., 2025)
0.52
49.10
Twen3-VL-Insttuet-301 (Bai al., 2025)
1.69
60.34
Dwen3-VL-Thinningg333 (Bai et al., 2025)
34.98
97.56
Swen3-VL-Thhininn--800 (Bai al., 2025)
104.03
120.08
GPT-4.1 (OpenAl, 2025a)
2.37
12.04
GPT-5 (OpenAl, 2025b)
24.26
26.58
Gemini 2.5 Pro (Google Meep 2024)
14.24
21.62
1.uueeSommet-4.5 (Anthropie, 2025)
4.68
12,60
I.LaVA-Med (7B)
al., 2023)
0.77
2.49
HuatuoGPT- Vision (7B) (Chen et al., 2024a)
1.22
7.20
HealthGPT (3.8B) (Lin al., 2025)
0.92
14.07
Lingshu-7h (Team al., 2025)
0.66
5,56
Lingshu-32B (Team et al., 2025)
0.78
8.44
MedGemma-4B (Sellergren al., 2025)
1.49
30.76
MedGemma-27B (Sellergren et al., 2025)
2.58
61.06
35

---

Published as conference paper at ICLR 2026
D.2
CASE STUDY
D.2.1 EXAMPLE
Comparison of answers from Qwen3- LL-3--V-ttnttttt-300 and the annotated CoT steps.
Q: True False: The cell shown in the image is Lymphocyte.
Correct Answer: False.
The direct answer is False, and the CoT Answer True.
The annotated CoT steps are shown below: 1. Hematology or Cytology
The cell has a prominent bilobed nucleus with abundant pinkish-orange granules in the cytoplasm. The cell type is Eosinophil.
To determine whether the cell shown in the image is lymphocyte, let's analyze the key features of lymphocyte and compare them the cell in the image.
Step 1: Identify the
image
The image shows a cell with a large, round nucleus that occupies most of the cell's volume.
The cytoplasm is scant and appears light blue or pale.
The nucleus is dense and dark purple, with smooth, round contour.
There is no visible granulation in the cytoplasm.
Step 2: Recall the characteristics of lymphocyte Lymphocytes are a rype of white blood cell.
They are typically small to medium size.
The nucleus is large, round, and dense, often taking up most of the cell's volume.
The cytoplasm is minimal and pale, often appearing as a thin rim around the nucleus. Lymphocytes do not have granules in their cytoplasm.
The nucleus is usually darkly stained (basophilic), indicating high RNA content.
Step 3: Compare the observed cell lymphocyte characteristics
The cell the image has large, round, darkly stained mucleus.
The cytoplasm minimal and pale.
There are no visible granules.
These features are consistent with lymphocyte.
Step 4: Consider other possibilities
Neutrophils: Have multi-lobed nucleus and granular cytoplasm.
Eosinophils: Have bilobed nucleus and large, eosinophilic granules.
Basophils: Have lobed nucleus and large, basophilic granules.
Monocytes: Have a large, kidney-shaper or horseshoe-shaped nucleus and abundant cytoplasm.
The cell in the image does not match any of these.
Step 5: Conclusion
The cell in the image exhibits the classic features of a lymphocyte.
Final Answer: True"
Analysis: Introducing CoT reasoning in this led worse performance hecause it caused the model to misweight and misinterpret key diagnostic features. Instead focusing on the defining characteristics an
36

---

Published as conference paper at ICLR 2026
cosinophil, namely the bilobed nucleus and abundant pink-orange cytoplasmic granules, the CoT reasoning overemphasized general patterns like a large, round nucleus and seant cytoplasm, which are typical of lymphocytes. The multi-step natural language explanation allowed the model to generate a seemingly coherent rationale, but this reasoning was not properly grounded in the observable image features, creating self-consistent ultimately incorrect inference. This demonstrates that CoT can inadvertently introduce cognitive bias and distract attention from critical domain-speciffe visual cues, leading to errors that direct answer prediction would have avoided.
D.2.2 EXAMPLE
Comparison of answers from MedGemma-22B and the annotated CoT steps.
What might be the treatment options for this condition? (Select one option)
A) Surgery as the first line of treatment
B) Topical corticosteroid drops and dilating drops, with systemic medication when necessary
C) No treatment is required
D) Laser therapy
Correct Answer: B.
The direct answer is B, and the CoT Answer is D.
The annotated CoT steps are shown below; 1. Anterior segment slit lamp examination.
2. Posterior synechiae causing pupil deformation and keratic precipitates on the corneal endothelium.
3. The likely diagnosis based on this image is anterior wvettis.
A) Surgery the first line of treatment: Not appropriate for anterior uveitis, surgery is rarely needed unless complications occur. B) Topical corticosteroid drops and dilating drops, with systemic medication when necessary: This is the standard treatment for anterior uveitis. It helps reduce inflammation and prevent complications such as synechiue. C) No trealment required:Incorrect because untreated anterior uveitis can lead to serious complications and vision loss. D) Laser therapy: Not typically used for uveitis. It is more relevant for other eye conditions such as glaucoma or retinal diseases.
The output of CoT prediction is shown below: The user wants
identify the condition shown in the image and then select appropriate treatment option from the given choices.
Analyzee the image:** The image shows a close-up view of an eye. The iris appears to be pulled forward, and the anterior chamber angle (the space benween the iris and the cornea) seems narrow. There is also white, slightly raised lesion the iris, which could be deposit The pupil is somewhat constricted. The overall appearance suggests a potential issue with anterior chamber angle and possibly inflammation or pigment dispersion.
sign of inflammation.
2. s #dennify the likely condition:" ** Based the narrow angle appearance, the condition is likely a "angle-closure glaucoma* ** or related condition like *pigmentarry glaucoma** or **pseudoexfoliatton glaucoma**, * which can also present with narrow angles and pigmen//ssuuooessttttoo deposits on the iris. The white lesion could he pseudoexfoliation flake or pigment deposit. Given the appearance, narrow angle is key feature.
s =sEvaluate the treatment options:
as **A) Surgery as the first line of treatment: * Surgery (like laser iridotomy or trabeculectomy) is often the definitive trealment for angle-closure glaucoma, especially if the angle is unalomicully narrow and causing symptoms or high pressure. However, it's always the *first* line, expecially the pressure is controlled medically or if the angle closure is intermittent. ***B) Topical corticosteroid drops and dilating drops, with systemic medication when necessary:** ++ Topical corticosteroids are used for inflammation (uveitis), which can sometimes be associated with glaucoma or mimic its symptoms. Dilating drops (mydriatics) are generally *contraindicated" in angle-closure glaucoma because they can further narrow the angle and precipitate acute attack. Systemic medication might be used for severe inflammation or pressure control, but this combination (especially the dilating drops) not the standard first-line approach for suspected angle-closure. * **C) No treatment is required:** * Glaucoma, especially angle-closure
37

---

Published as conference paper at ICLR 2026
glaucoma, is serious condition that can lead to irreversible vision loss if untreated. Therefore, no treatment is incorrect. s s.*+D) Laser therapy:* Laser therapy, specifically laser peripheral iridotomy (LPI). is a common and often first-line treaiment for "primary ungle-closure glaucoma* or "primary angle-closure suspect to create an alternative pathway for aqueous humor flow and prevent acute aftacks. can also be used in other types of narrow-ange angle glaucoma.
4. **Refine the diagnosis and treatment: ** The image strongly suggests a narrow angle, potentially leading to angle-closure glaucoma. The white lesion could be pigment or pseudoexfoliation material. If it's "primary ungle-cli glog--clouure glaucoma* or "primary angle-closure suspect* @ with a narrow angle, the standard initial treatment is often **laser peripheral iridotomy (IPI))* to prevent acute angle closure. Medical therapy (drops to lower pressure) might be used initially or concurrently, but LPI is frequently the primary intervention to address the underlying anatomical issue." * *If it's *secondary angle closure* due to inflammation (uveitis), then corticosteroids (topical or systemic) would be the primary treatment, along with cycloplegics (which dilate the pupil and can help open the angle in some inflammatory conditions, unlike mydriatics used for examination). * *f'i''s "iigmentary glaucoma" R or "pseudoexfoliation glaucoma", treatment focuses on lowering intraocular pressure, often starting with topical medications, but laser trabeculoplasty or surgery might be needed.
5. ReRe-evaluate the options based on the most likely scenario: s* The image most strongly points towards narrow angle, making angle-closure glaucoma primary concern. * Option A (Surgery) often needed but might not be the absolute "first" step if medical therapy or laser therapy can control the pressure and prevent attacks." * Option B (Corticosteroids, dilating drops) is incorrect because dilating drops are contraindicated in angle closure. & Option C (No treatment) is incorrect. * Oppiion D (Laser therapy) is a very common and often firtt first-line treatment (LPI) for preventing acute angle closure in anatomically narrow angles.
6. ** *Comppre A and D:** Laser iridotomy (D) is frequently the "initial* intervention for primary angle closure or ungle-closure suspect to preveni ucute attacks. Surgery (A) might be considered later if laser therapy fails or if other conditions are present. Given the options, laser therapy (specifically L.P)) is a very strong candidate for the first-line treatment in this scenario.
7. * Final Decision: ** The image shows shows features suggestive of angle-closure glaucoma (narrow angle, Laser peripheral iridotomy is the standard first-line line treatment to prevent acute angle closure in such cases. Therefore, Option D is the most appropriate choice among the given options, representing the common initial intervention. * **Sttep 1: Analyze the image. ** The image shows an eye with a visible iris and pupil. The iris appears somewhat pulled forward, and the angle between the iris and the cornea (anterior chamber angle) seems narrow. There is white, slightly raised lesion on the iris surface, which could represent pigment dispersion, pseudoexfoliation material, or another deposit. The pupil is relatively round but not fully dilated. **Step 2: Identify the likely condition. ** The narrow angle appearance is a key feature. This, combined with the possible iris deposits, strongly suggests condition related to narrow angles, such as primary angle-closure glaucoma (PACG), primary angle-closure suspect (PACS), pigmentary glaucoma, or pseudoexfoliation glaucoma. These conditions involve a physically narrow angle between the iris and cornea, which can impede aqueous humor outflow and lead increased intraocular pressure (IOP) and potential vision loss. **Step 3: Evaluate the treatment options in the context of the likely condition. 84 & **A) Surgery as the first line of treatment:" While surgery (like trabeculectony or tube shunt) is used for glaucoma, it's typically reserved for cases where medical or laser therapy fails, or for specific types of glaucoma. For anatomically narrow angles causing potential angle closure, laser iridotomy is often preferved initially. **B) Topical corticosteroid drops and dilating drops, with systemic medication when necessary: ** Corticosteroids are used for inflammation (uveitis), which can sometimes cause secondary angle closure, but they not the primary treatment for primary angle closure. Importantly, *dilating drops (mydriatics)* * are generally contraindicated in angle-closure glaucoma because they can further narrow the angle and precipitate an acute atlack by blocking aqueous outslow through the trabecular meshwork. a +*C) No treatment is required:** 3* Angle-closure glaucona a potentially blinding condition that requires treatment to lower IOP and prevent damage to the optic nerve. Therefore, no treatment is incorrect. **D) Laser therapy: Laser peripheral iridotomy (LPI) is common and often first-line treatment for primary angle-closure glaucoma and angle-closure suspect. It creates a small hole in the peripheral iris, allowing aqueous humor to bypass the narrow angle and flow directly from the posterior to the anterior chamber, thus relieving pupillary block and preventing treating acute angle closure. Laser trabeculoplasty might be used for open-angle glaucoma or (igmentary/pxuuoofflaaaae for narrow angles causing potential closure. *Seep 4: Select the most appropriate option. Given the
possible pigmentissedooesfolatiio
glaucoma, but LPI is the specific laser procedure
38

---

and Gemini Published as conference paper at ICLR 2026
likely diagnosis of narrow angle condition potentially leading angle closure (based the image), laser peripheral iridotomy (LPI) the standard initial treatment to prevent acute attucks and manage the condition. Therefore, laser therapy is the most fating first-line approach among the choices provided.
*"Final Answer:** The final answer is D."
Analysis: In this case, the model's CoT reasoning led to an incorrect treatment choice because it misinter- preted the key clinical features emphasized in the annotated COT. The annotated steps indicate anterior uveitis, characterized by anterior chamber inflammation and posterior synechiae, and recommend topical corticosteroids and dilating drops (Option B). However, the model's s reasoning focused secondary features such as a narrow anterior chamber angle and iris deposits, overemphasizing the possibility of angle-closure glaucoma, and consequently selected laser therapy (Option D) as the first-line treatment. This illustrates that CoT can sometimes introduce assumptions and reasoning paths that override the correct clinical interpretation, leading to errors even when the direct answer might otherwise be easier to infer.
E
LIMITATION DISCUSSION
E. ANNOTATION DISCREPANCIES BETWEEN EXPERTS, AI, AND PUBLIC DATASETS
The question-answer pairs and CoT annotations were generated through collaboration between medical experts and AI, while also referencing labels from existing public datasets, In some cases, discrepancies arose between expert judgment and dataset labels. We generally prioritized the public dataset labels as the highest authority. However, we frequently encountered inconsistencies potential errors in these labels. In such cases, we made efforts to verify through repeated reviews and multiple AT model assessments, but we cannot guarantee that every annotation step is fully accurate.
E.2 DISEASE-SPECIFIC LABELS MAY IMPLY UNJUSTIFIED DIAGNOSTIC PRECISION
Some annotations involve specific diseases (e.g., COVID-19, 19, certain cancers), directly inherited from the original dataset labels. These labels may have been informed by additional contextual information unavailable in the image alone. In reality, making a definitive diagnosis from single image is often not feasible, for trained physicians. By retaining these disease-specifie labels, the task may set unrealistically high bar MILMs, possibly exceeding what is expected human experts. To address this, we aimed to phrase our labels cautiously using formulations like "the most likely diagnosis is..."
E.3 SUBJECTIVITY IN EXPRESSION MAY AFFECT MATCHING
Although adopted relatively permissive matching criteria account for variation in wording, certain annotation statements inevitably involve subjective interpretation, particularly when describing subtle visual findings or formulating likely diagnoses. These subjective elements can introduce variability in phrasing that, despite semantic similarity, may not be captured perfectly by automated matching methods. Furthermore, medical descriptions often allow for multiple valid expressions of the same observation, and differences in terminology, level of detail, or emphasis may lead to mismatches during evaluation. This issue is particularly relevant for open-ended reasoning tasks, where the boundary between correct and incorrect answers can be nuanced.
E.4 EVALUATION FULLY CONDUCTED WITH MLLMS
All evaluation of model outputs is conducted using GPT-4o, 1-1-AA-..-77B-In-tti
2.5 Pro. While these models have demonstrated strong performance in general reasoning and medical question answering, they remain Al systems with inherent limitations. complex or ambiguous cases, the model may misinterpret medical terminology, overlook subtle differences between options, or apply inconsistent grading criteria. Additionally, its judgments may be influenced by prompt wording prior context, leading to potential evaluation bias. The absence of human pooss-alldattio may lead mis-scores. particularly in domains that require precise domain knowledge.
Regarding circularity concerns, although using greater variety of models may lead to further improve- ment, the current annotation workflow is already effective in ensuring high-quality annotations while minimizing model bias. Specifically, by integrating two models, GPT-4o and Gemini 2.5 Pro through
39

---

Published as conference paper at ICLR 2026
multiple processing steps and incorporating manual expert correction, the risk of dominance by single model has been significantly reduced. Moreover, the final evaluation based on comparing outputs with the annotated ground truth, rather than relying on the model to generate judgments, further reducing the risk of circularity independently.
E.5 No INTER-ANNOTATOR AGREEMENT SCORES ARE REPORTED
Inter-annotator agreement scores: Because this workflow is not fully parallel, we acknowledge that inter-annotator agreement scores are not reported, which is limitation of this study. However, the multi-stage review process, combining initial student review, multi-model automated checks, targeted expert verification, consensus discussions, and final read-through, ensures high-quality annotations while minimizing bias from any single reviewer model. This careful workflow allows produce reliable reference reasoning chains suitable for evaluating MI.LMs medical image understanding.
6.6 No MULTIPLE EXPERIMENTAL RUNS, AND NO CONFIDENCE INTERVALS WERE REPORTED
Due to cost and time constraints, this study only conducts double evaluations for answer accuracies and CoT steps consistencies, and a single evaluation for the correctness CoT steps and did not report confidence intervals significance tests. We acknowledge that repeating experiments and reporting confidence intervals would provide more rigorous and reliable results. In future versions, we plan to include multiple runs and statistical significance analyses to further strengthen the robustness of our findings.
E.7 LIMITED EXPLORATION PROMPTS AND ABLATION STUDIES
In this study, we do not conduct comprehensive exploration of alternative prompting strategies or perform extensive ablation experiments evaluate the impact of prompt design choices systematically. Variations such as adjusting the level of detail, explicitly guiding reasoning steps, or introducing domain-speciff constraints could potentially influence model performance. Similarly, ablation studies, such as removing specific reasoning cues, altering input formatting, testing under different context lengths, might have provided more profound insights into model behavior. absence of these experiments limits our ability to fully characterize how sensitive the results are to prompt engineering and task setup.
F SOCIAL IMPACT DISCUSSION
The proposed M3CoTBench benchmark carries several important implications for the development and evaluation of medical AI systems:
F.1 ADVANCING INTERPRETABLE MEDICAL AI
By explicitly evaluating the reasoning chains of MILMs, M3CoTBBenh encourages transparency in how models arrive at their predictions. Understanding intermediate reasoning steps allows researchers and clinicians to better align AI behavior with clinical decision-making processes, fostering trust and supporting responsible deployment in medical research and practice. In high-stakes medical applications, interpretabil- ity is critical: clinicians can verify whether model reasoning is consistent with established diagnostic criteria, and researchers can identify failure modes that may not be apparent from final predictions.
F.2 IMPROVING MODEL EVALUATION MEDICAL AI
Most existing benchmarks focus solely on final predictions, overlooking the reasoning process that leads to those outcomes. MBCco fills this gap by providing structured framework to assess the correctness, consistency, and efficiency CoT reasoning across diverse medical imaging tasks. This enables more nuanced analysis of model performance, highlighting specific strengths and weaknesses in reasoning patterns that are essential for complex diagnostic scenarios. By systematically evaluating intermediate steps, M3CoTBench supports the development of models that not only accurate but also capable of robust and verifiable decision-ni maaiiinn-maiing.....
40

---

Published as conference paper at ICLR 2026
F.3 PROMOTING RIGOROUS DEVELOPMENT OF TRUSTWORTHY AT SYSTEMS
By emphasizing the evaluation of reasoning quality rather than only accuracy, the benchmark guides the design of models that are not only correct but also interpretable and reliable. This focus on transparent reasoning can help mitigate risks associated with opaque AI decisions in clinical settings, enabling accountable AT deployment. Moreover, by providing standardized metrics for reasoning quality, MeCcTBench encourages best practices in medical Al development, fostering the creation models that adhere to both technical and ethical standards.


Reviews:
[{"review_id": "JkLIxE5gfE", "reviewer": "Reviewer_Ti8j", "summary": "This paper proposes M3CoTBench, a benchmark designed to evaluate Chain-of-Thought (CoT) reasoning in multimodal large language models (MLLMs) for medical image understanding. It constructs a diverse dataset of medical images with clinically grounded reasoning annotations and proposes multidimensional evaluation metrics covering correctness, efficiency, impact, and consistency. Experiments on both open- and closed-source models show that while CoT improves interpretability, it does not always enhance diagnostic accuracy, revealing limitations in current MLLMs’ clinical reasoning and highlighting the need for more trustworthy and efficient medical CoT systems.", "questions": "Please refer to the Weaknesses.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper tackles an emerging yet underexplored topic: evaluating Chain-of-Thought reasoning in medical multimodal LLMs, which is both timely and relevant to advancing trustworthy medical AI.\n2. The benchmark is validated on a broad range of both open- and closed-source MLLMs, providing a well-rounded comparison that highlights current model limitations and practical challenges in clinical reasoning.", "weaknesses": "1. The definition of CoT in the medical area is unclear. Although the paper claims that its Chain-of-Thought (CoT) formulation “mirrors clinicians’ cognitive workflow”, the reasoning template shown in the Appendix appears overly simplified. It typically only has four steps: examination type -> key features -> key conclusion -> additional analysis. It is unclear why this sequence represents a gold standard reasoning path in clinical diagnosis. Is it based on any references, such as guidelines in medicine?\n2. The justification for diverse reference reasoning paths is insufficient. The paper mentions that “multiple valid reference reasoning paths may exist” and evaluates by matching the generated path to the most similar reference. While this makes sense conceptually, it is unclear how the annotation process ensures both diversity and correctness of reference reasoning paths. In the medical domain, it remains questionable whether clinicians indeed exhibit substantially diverse CoTs. If so, where does this diversity arise? Is it in identifying different key features (Step 2) or in drawing different key conclusions (Step 3)?\n3. The Reasoning Impact evaluation simply measures the performance difference between models with and without CoT, which seems redundant. This metric does not provide new insight into reasoning quality.\n4. The reasoning progression across CoT steps is weak. From the provided CoT examples, the reasoning flow among steps is not clearly causal or hierarchical. Steps 3 and 4 appear to be direct deductions from Step 2, while Step 1 (examination type) is largely independent of the reasoning process itself. As a result, it is difficult to claim that the sequence truly reflects a step-by-step reasoning chain rather than a loosely connected checklist.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Dqup17tthS", "reviewer": "Reviewer_iKQE", "summary": "This paper introduces M3CoTBench, a benchmark designed to evaluate chain-of-thought (CoT) reasoning in multimodal large language models (MLLMs) for medical image understanding. M3CoTBench comprises a diverse dataset spanning 24 imaging modalities (X-rays, MRIs, endoscopy, etc.) and 13 task types, ranging from low-level tasks like image quality assessment to high-level clinical reasoning such as diagnosis and treatment planning.", "questions": "1. LLaVA-CoT exhibits relatively strong performance compared to Gemini 2.5 Pro. The authors attribute this to its architecture and training process, which emphasize structured reasoning chains while minimizing irrelevant or misleading steps. However, given that Gemini 2.5 Pro also incorporates thinking capabilities, what accounts for this performance difference?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.  This paper introduces M3CoTBench, encompassing 24 imaging modalities to evaluate MLLMs' understanding capabilities across diverse medical imaging contexts.\n2. The benchmark introduces tailored metrics to assess reasoning quality across four dimensions: correctness of each reasoning step, efficiency cost, impact on final answer accuracy, and logical consistency—providing a more nuanced evaluation beyond traditional accuracy measures.", "weaknesses": "1. M3CoTBench spans 24 modalities and 13 task types, but contains only 1,079 image-based QA pairs. Given this broad coverage, does each category have sufficient samples? The paper does not appear to provide per-category statistics.\n2. The benchmark’s dataset, while diverse, is relatively small (only 1079 Q&A pairs) compared to other medical VQA datasets, which may limit the statistical breadth of evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OnEKx1rD9D", "reviewer": "Reviewer_28Mb", "summary": "This paper introduces M3CoTBench, a benchmark for evaluating Chain-of-Thought reasoning in medical image understanding MLLMs. The dataset contains 1,079 image-QA pairs across 24 medical imaging modalities with expert-annotated reasoning steps and four evaluation dimensions.", "questions": "- How do you address evaluation circularity when using GPT-4o to assess GPT-4o reasoning?\n- What are the inter-annotator agreement scores during expert validation?\n- Why do most models show negative reasoning impact, is this a CoT implementation issue or benchmark design problem?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- Addresses Critical Gap: First comprehensive benchmark for CoT reasoning in medical imaging - important for clinical AI transparency and trust.\n- High-Quality Curation:\n  - Diverse coverage: 24 modalities from 55 public datasets\n  - Rigorous annotation: Multi-stage validation with medical experts\n  - Clinical alignment: 4-step reasoning framework mirrors diagnostic workflows\n- Novel Evaluation Framework: Four dimensions (correctness, efficiency, impact, consistency) provide comprehensive CoT assessment beyond accuracy.\n- Extensive Evaluation: Tests 13 MLLMs including general-purpose, reasoning-focused, and medical-specific models with interesting findings about CoT effectiveness.", "weaknesses": "Methodological Concerns: \n- The dataset comprises only 1,079 images, relatively small compared to other medical reasoning benchmarks (e.g., OmniMedVQA with 118K+ images).\n- Potential Bias: Although reasoning steps undergo expert validation and revision, their initial generation by GPT-4o may introduce biases inherent to its reasoning style, which might persist despite subsequent human refinement.\n- Evaluation Circularity: The study uses GPT-4o both to generate reasoning chains and to evaluate them against GPT-4o-based gold standards, creating a circular evaluation loop.\n- The paper does not specify which MLLMs were used for flagging potentially incorrect reasoning steps.\n\nEvaluation Concerns:\n- Despite the multi-expert validation process, no inter-annotator agreement scores are reported.\n- Confidence intervals and significance tests for performance differences are not provided.\n- The number of experts involved and procedures for resolving disagreements are not described.\n\n\nConceptual Issue:\n- Counterintuitive Findings: The universally negative impact of reasoning across models raises questions about the benchmark’s design, the quality of the Chain-of-Thought implementation, and the validity of using GPT-4o as the evaluation reference.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)