MedAgent-Pro: Towards Evidence-based Multi-modal Medical Diagnosis via Reasoning Agentic Workflow
Authors: Ziyue Wang
Junde Wu
Linghan Cai
Chang Han Low
Xihong Yang
Qiaxuan Li
Yueming Jin
Venue: ICLR 2026 Poster (2026)
Keywords: Medical AI
Agentic AI
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: This paper proposed MedAgent-Pro, an evidence-based reasoning agentic system designed to achieve reliable, explainable, and precise medical diagnoses.

Abstract: Modern clinical diagnosis relies on the comprehensive analysis of multi-modal patient data, drawing on medical expertise to ensure systematic and rigorous reasoning. Recent advances in Vision–Language Models (VLMs) and agent-based methods are reshaping medical diagnosis by effectively integrating multi-modal information. However, they often output direct answers and empirical-driven conclusions without clinical evidence supported by quantitative analysis, which compromises their reliability and hinders clinical usability. 
Here we propose MedAgent-Pro, an agentic reasoning paradigm that mirrors modern diagnosis principles via a hierarchical diagnostic workflow, consisting of disease-level standardized plan generation and patient-level personalized step-by-step reasoning. To support disease-level planning, a retrieval-augmented generation agent is designed to access medical guidelines for alignment with clinical standards.  For patient-level reasoning, MedAgent-Pro leverages professional tools such as visual models to take various actions to analyze multi-modal input, and performs evidence-based reflection to iteratively adjust memory, enforcing rigorous reasoning throughout the process. Extensive experiments across a wide range of anatomical regions, imaging modalities, and diseases demonstrate the superiority of MedAgent-Pro over mainstream VLMs, agentic systems and leading expert models. Ablation studies and expert evaluation further confirm its robustness and clinical relevance. Anonymized code link is available in the reproducibility statement.

Published as a conference paper at ICLR 2026
MEDAGENT-PRO:
TOWARDS EVIDENCE-BASED
MULTI-MODAL MEDICAL DIAGNOSIS VIA REASONING
AGENTIC WORKFLOW
Ziyue Wang's Junde Wu?, Linghan Cai², Chang Han Low', Xihong Yang Qiaxuan Li Yueming Jin
National University of Singapore, UUniversity of Oxford,
3 Harbin Institute of Technology (Shenzhen), *National University of Defense Technology,
The Second Affiliated Hospital Zhejiang University School of Medicine
ABSTRACT
Modern clinical diagnosis relies on the comprehensive analysis of multi-modal patient data, drawing on medical expertise to ensure systematic and rigorous rea- soning. Recent advances in Vision- Languue Models (VLMs) and based methods are reshaping medical diagnosis by effectively integrating multi-modal information. However, they often output direct answers and empirical-driven conclusions without clinical evidence supported by quantitative analysis, which compromises their reliability and hinders clinical usability. Here propose MedAgent- an agentic reasoning paradigm that mirrors modern diagnosis principles via hierarchical diagnostic workflow, consisting of disease-level stan- dardized plan generation and patient-level personalized step-by-step reasoning. To support disease-level planning, a setrieval-augmentee generation agent is designed to access medical guidelines for alignment with clinical standards. For patient- level reasoning, MedAgent-PPrs leverages professional tools such as visual models to take various actions to analyze modai input, and performs evidence- basedd reflection iteratively adjust memory. enforeing rigorous reasoning throughout the process. Extensive experiments across a wide range of anatomical regions, imaging modalities, and diseases demonstrate the superiority of MedAgent- over mainstream VLMs, agentic systems and leading expert models. Ablation studies and expert evaluation further confirm its robustness and clinical relevance. Anonymized code link is available in the reproducibility statement.
INTRODUCTION
As a core task in medical practice, clinical diagnosis entails synthesizing various clinical information to reach conclusion (Steinberg et al.) 2011: Guyatt al.) (992a).. where clinicians make decisions mainly based on visual cues from medical imaging and textual information from patient records. For example, is common for clinicians to examine radiology images identify tumor scales, or analyze pathological slides to detect potential cancer. Early Al-assisted methods focus on adopting pure image analysis models to support diagnosis, such as pneumonia severity assessment in chest radiology or cellular classification in pathology. Recently, Vision- Vision-Language Models (VLMs) have demonstrated that integrating multi-modal information can significantly benefit diagnosis, and med- ical visual question answering (VQA), where models answer textual questions based on images, has become a key benchmark in this context (Zhan et al. 2020 Chen et al.) 2025b Liang et al. 2024).
However, ordinary VQA formulated in existing methods fails to reflect the real-world diagnostic processes. Rather than performing one-hop VQA, clinical diagnosis involves standardized, step- by-step process (Steinberg et al.) 2011 Eddy 1990a/blc; Albargouni et al., 20188). The diagnostic process for each patient typically involves two stages: 1) Determining the target disease and for- mulating standardized workflow based on medical guidelines with clinical indicators support the decision; ii) Step-by-step analysis of personalized data, combining qualitative and quantitative
"Corresponding author; Yueming Jin

---

Published as a conference paper at ICLR 2026
Exomple 1: Gloucoma
Answers from VLMs/Medical Agents
Answers from MedAgent- Pro
GPI-4a: To cunform whatler paisent fis giucoma, is usuall ly necessory to comprehensvety onstge multiple factors such marphology the optic disc thickesss the optic nerve flbor byer, intracuuar pressure difficalt diagnose
According relevant documents, the liyy ind cators for diagnosing glocooma from funcus images are as fellows:
Accorting the segmentation result optic eip and dise. the VCDR is 0.681, higher than the risk thrcsold O.S.
thiscness porient's neuronetinal devistes rule,
glaucoma baved
temooral margin The potient have optic disc hemorthage with associated cup to disc and retinal plgmentation.
MMedAgeet: This is VQA tasic and should be handled by LLae Med. According to the fundis image, the patient does not have glaucoma. The image shows a normal optic nerve.
image shron undear optic dis boundary, which may ro portosolibrry atroshy.
Fundus Image
Based on above fodings the pationt are very liklyy to have glaucoma.
Exomple 2: Heart Disease
Answers from VLMs MMMs/Medicol Agents
Answers from MedAgent-Pro
CPT-Aa: help you amelyee the fevrures achroordliggeeh image, but cennot directly diugnose heart diseana deteriine whethhrr pation: has heart disssos, professional medical image anglsss and a doctor's oxaluction are requirod.
Arcording to relevant documents, the koy indizatons for dlagrosing heart discse from 30 echocart ography images are as follows Acconding to the segmentation result of lefeentrccc and mypcardium, the Left Ventiialar Ejection Fraction (LVEF) of the pat patient 20% sigeliiant below the summal value, The Left Ventricular End Diastolic Diameter (LVD0)) 62mm, which larger then the normal value of male [42 42m5mm) Information the potient's
mo0
Echocadingraphy
task and should be echorardlggapphy Image shows normal cardisc function, which moans that the patient does not have heart disease according to the information provided.
Mivied/gent:
Age
bwighht
swiight
Sax
Considoring may (SVV)-35.2), Ventricaler 130g/m? larger then narmal value 150-118 118mm) summery, patient
1700mm
5tri
Patient Information
cardoomyopathy hypertrophic.
Figure 1: Comparison of diagnostic outcomes for two diseases across mainstream VLMs, medical agentic systems, and our proposed MedAgent- workflow.
assessments to evaluate these indicators. Throughout the process, each step builds on previous as- sumptions and is supported by relevant literature or specialized tools For example, in glaucoma diagnosis, the cup-to-disc ratio is a key indicator, which relies on the accurate localization of the optic cup and disc in the preceding step. In contrast, current VQA models often generate diagnostic conclusions hastily, relying on empirical internal knowledge without fine-grained analysis.
Due to the dual requirements of patient safety and evidence traceability mandated by clinical regu- lation, current methods fall short of meeting the standard: VLMs (Achiam et al. 2023] Chen et al.
2025a) have shown strong performance across a range of tasks (Ghezloo et al.] 2025 |ennary et al. 2023). However, they lack sufficient medical knowledge and remain inadequate for in-depth medical analysis. While models like GPT-ol (Achiam et al. 2023) and DeepSeek-RI (Guo et al. (0255) in- corporate reasoning capabilities to support more structured analysis, their limited fine-grained visual perception ability impairs quantitative analysis and reduces their effectiveness in clinical applica- tions. Meanwhile, agentic systems (Chen al. 2023 Li et al., 2023) have extended the capabilities of VLMs by introducing professional tools, but current medical agentic systems (Kim et al.) [2024] Zuo et al. [204]] Fallahpour et al. 2025) integrate tools statically instead of dynamically organizing tools according to the given disease. As a result, when asked to provide a diagnosis, these systems simply conduct a fixed pipeline without selecting and orchestrating appropriate tools to support their decision-makin
In summary, existing methods treat medical diagnosis an empirical one- hop QA task, drawing exclusively from VLMss internal knowledge to make qualitative judgments. In contrast, medical practice emphasizes evidence-basee diagnosis, requiring structured reasoning and clinical indicators (Eddy) 1990a b
Zhu et al. 2025a).
To tackle these issues, present MedAgent-Proo reasoning agentic workflow tackling versatile multi-modal medical diagnosis tasks. We aim to design a workflow that aligns with modern med- ical criteria, provides decision support with medical guidelines and quantitative analysis as shown in Fig. Our MedAgent-Pro embraces a hierarchical structure to simulate the modern clinical procedure, conduct by-peb reasoning at the disease and patient levels. Disease-level planning generates standardized diagnostic plans, while Ratient-level reasoning follows these plans to analyze personalized information. To facilitate alignment with clinical guidelines. MedAgeet--Pes incorpo- rates a eetrieval-augmenter generation (RAG) agent that injects relevant medical knowledge into the diagnostic planning. In the reasoning process, MedAgent-Pin integrates expert tools such as visual models to take various actions for diagnosis support, enabling accurate qualitative and quantitative analysis of clinical indicators. Furthermore, to maintain the rigor of multi-step clinical reasoning, propose evidence- based reflection mechanism, whereby the system evaluates the reliability of each step's 's output and adjusts the memory to guide subsequent actions, ensuring that every decision is grounded in sound and trustworthy foundation. Our contributions are summarized as:
We propose MedAgent- Pro, the first agentic paradigm that presents systematic, evidence-based reasoning for accurate and reliable medical diagnosis. By aligning with the principle of modern
2

---

Published as a conference paper at ICLR 2026
medical workflow, paradigm transforms the empirical, ready- outputs of prior methods into more rigorous logical reasoning and a well-founded conclusion.
MedAgent-Prre presents hierarchical structure consisting of disease-level and patient-level rea- soning. A RAG-based method makes the disease-leve planning align with medical guidelines, while tool-based action and evidence-based reflection are devised at the patient level ensure professionalism and reliability of the step-by-step analysis.
Evaluations across 10+ imaging modalities, 20+ anatomies, and 50+ diseases show that MedAgent-Pro comprehensively surpasses mainstream VLMS, agentic systems, and task-specific models. Notably, it outperforms GPT-4o by 34% and 22% on glaucoma and heart disease diagno- sis. Clinician evaluations further highlight the diagnostic quality and reliability of our method.
2 RELATED WORK
Multi-modal Medical Diagnosis. Developing Al techniques for multi modli medical diagnosis
has become primary research objective (Bakator & Radosav 2018 Kononenko, 2001; McPhee
et al.
2010). Prior research focused on medical imaging assessment, including classification (Zhang
al. 2023a
2019
Azizi et al. 2021 Association (020), detection (Baumgartner
al.
2021
Wang et al. 2020; Bejnordi et al. 2017] and segmentation (Isensee et al. 1 Wu et al. 2025
Ma et al. (2024a).
VQA has been proposed for end-to-end multi-modal diagnosis (Zhang et al.
2023b, Khare al. 2021 Zhang al., 2025), while VLMs (Li et al. 2024b Moor et al. [023]]
Liang et al. 2025 Lin et al. 2024) have yielded competitive performance in medical VQA. Despite
these advancements, medical VQA (Lau et al.) 2018 Liu al.) 2021] He et al. 2020 remain overly
simplified compared to the diagnostic practice, and further research is highly desired.
VLM--aaeed AT Agent. Establishing autonomous intelligent system is long-standing research goal, with agent-based methods gaining increasing attention. VLMMbased agents have made great progress in diverse applications such industrial engineering (Mehta et al.; [023] Xia al. 2023) scientific experimentation (Boiko et al. 2023), embodied agents (Brohan et al. 2023 Huang et al.
2022), gaming (Gallotta al. (024), and societal simulation (Ma et al.) [024b Jinxin al. 2023)
Despite their adaptability in various scenarios, VLM-based agents remain limited in the medical domain due insufficient fine-graine mne-grained visual perception.
Medical Agentic System. Current medical agentic systems can be categorized into two streams.
The first stream enhances VLM capabilities through mechanisms such as debate or majority voting among agents to refine responses (Kim et al. 2024 Tang et al.) 2024 Zuo et al., 2024 Lyu et al., 2025 Zhu et al. (2025b) The second ones integrate an orchestrator agent with various specialized
models to solve various medical tasks (Li et al. 2024a Xia et al. 2024 Fallahpour et al. 2025
Fathi et al. 2025 Wang et al., 2026). Nevertheless, they mainly aggregate tools in a fixed manner instead of clinically cliiieallygrouuneed workflow, leading to suboptimal handling of complex diagnoses.
Diserse Lovel Lovel
Step d: oDbiect type Tooti, Result tyne i>
Step 2: Obbject bype Too12, Result cype2>
Toovitax
Medical Knowledge Base
RAG Agent
Guideline
Planning
Plan
Steem
Resat
Contiinue
Torminate
Plan
Eval
Decide
Putient-Lerel
Information
Reasoning
Result
Indicators
Disgnesis
Therefore finl diegnaus
Qualitative Analysis
Available ToolBox
Quantitative Analysis
C
G ail
L
Guideline
Analysis
Visual/Coding Models
Image
Evidence
Result
Figure 2: Overview of the MedAgent- Pro. MedAgent- Pro performs diagnosis through hierarchical structure, with reasoning guided by a VLM supported by an RAG agent and specialized tools. Evi. means Svidence-based in the figure.
3

---

Published as a conference paper at ICLR 2026
3 METHODS
3.1 OVERALL WORKFLOW
For a specific disease, clinicians typically follow guidelines to develop a standardized diagnostic workflow. Each patient's diagnosis then follows this workflow, combining qualitative observa- tions and quantitative assessments to obtain clinical indicators (Details in Appendix (D). Emulating this paradigm, MedAgent-Pro design a hierarchical reasoning workflow consisting of disease-lvel planning and patient-level reasoning, enabling personalized diagnosis under standardized, disease- specific guidance. We take VLM V as an orchestrator conduct basic tasks within the workflow.
As illustrated in Fig. 2 the disease-level planning is designed to formulate standardized diagnos- tic plans for each disease based on medical guidelines, which is assisted by an RAG agent. In the meantime, patient-leve reasoning processes each patient's personalized data individually by exe- cuting the diagnostic plan step by step and assisting the VLM performing necessary quantitative analysis through specialized tools, thereby enabling evidence-based reasoning as follows:
3.2 DISEASE-LEVEL KNOWLEDGE-BAN BASED PLANNING
Doctors typically develop standardized workflows based on their expertise and medical guidelines [ddyy] 1990b c Shaneyfelt et al.] 2006). Aligning with this practice, we introduce a Retrieval- Augmented Generation (RAG) agent R. to incorporate medical guidelines during the planning stage to guide diagnostic plan generation. MedAgent-Pro is equipped with a large-scale knowledge base K. built from MedlinePlus (Miller et al.) 2000). which includes entries on 1,000+ diseases and conditions, and 4,000+ expert-reviewed corresponding guidelines authenticated by NIH/NLM.
Ney-word
Preprocess
Search
Formulate
Medical Knowledge Basc
Chunk
Summary
RAG Agent
Selection
Guideline
Figure 3: The illustration of the RAG process, which leverages two-step retrieval.
As shown in Fig. 3 we first index K. to support efficient retrieval. Each guideline article is first assigned a short summary of its metadata like organ and disease in MedlinePlus. Given a disease query, the RAG agent R first filters documents using the summary to obtain query-relevant subset. The full article is then segmented into 300-token chunks, embedded with PubMedBERT' (Gu et al.) 2021], and stored in vector index. R then retrieve the top-5 most relevant chunks based on simi- larity. Based on the retrieved chunks, the VLM summarizes their content and generates a procedural guideline G (KC) that reflects real-world clinical practices for the queried disease.
For the diagnostic plan generation, the VLM summarizes disease-specific clinical indicators Z (I,,I2, T2,,T2..... ......... . .mm) from the guideline G. To support the analysis of I. MedAgent-Pro equipped with a toolset T and an associated action set A, where each action a A corresponds to tool t T through a predefined mapping (a(a) t. For example, may be segmentation model, and is its paired action such as "Segments the optic cup in fundus image." Based on G and A, the VLM generates disease-specific diagnostic plan = P1, P1........ ,nn. Each step P; applies an action E A to an input object o; and produces an output ry:
P. : a,(r:), E A.
Here o and r; can be the input data like fundus image, intermediate result like segmentation mask or final indicators. In practice. P is stored as a JSON file. Each P, includes an operation action as a predefined Python function from the toolset T with fixed behavior, and two data fields specifying the expected input and output data property for a, Throughout the design, each disease is assigned a diagnosis plan aligned with medical guidelines, enabling a regulated and standardized workflow.
3.3 PATIENT-LEVEL EVIDENCE-BAR BASED REASONING
Grounded in the evidence- bradence-bbsed principle of modern medicine, which integrates reliable clinical evidence with individual expertise to guide patient care, MedAgent-Pro follows the -isccfc-speccffc diagnostic plan P to analyze clinical indicators I for each patient case in the following paradigm:
4

---

Published as a conference paper at ICLR 2026
Disease-leva Knowledge- Based Planning
Patient level Evidance based Reasoning
6 Task How diagnese glaucema?
Eig Memory Init.: input Fundus image, following steps will be conducted: step !..,, step ien
12 RAG Agant Acconting to retranued infomattion [3]2) the
diggttnis of gleazzmm neads to fars on the fallow especie:
Patient History and Rise factory ast about family history of glauooma...
Optic herve Eisluation: in fundus image, look for incressed vertical cup disc
S Visual Model: resultts of optic disc and optic cup.
the segmentation
rim thinning or notching, disc hemorrhage and perfpapillary atrophy. OCT images.
aeano the retinel newe liber layet.....
Mouni Fled Teating: Detecting functional vision loss
kB Coding Module: The result of vertical Cup-to- Ratio O 0.624, the optic reaion cropped for the following steps. (indicator 1)
Aore W A The ooste hypertenoos mestrvee rtady Jeurna glaarnss i (196)) 97. 005
122 Reflection: is hard to determine whether the patlent have optic hemorrhages (DH), relevant analysis stops (Indicator 2)
2)) Modd Gonsld at "ouuereattur donage the rtznua Progrece lo retooi ond
Analyyssss: image shows an unclear optic disc boundary, which may related to periapilaaay strophy (PPA). (Indicator 3)
Too the following toos:
Seymentallam model model segment Optis Gupp modell: segment Optic Dive
Codint madul: White code/pragrom to deal with vision results
MIA Describe the Input ophenarmologe images
The rim thiciness (8T) the four direction b.., obey the ISNT rule (I=S>leeT), (lndicator 4)
Codingg Module:
X Planning: The diegnoni plan insolven:
Step Object Medical records; Tool: VIMG Besult: Personal and fisk factors
T Reflection: Dased on the above analysis, vCDR and
peripapillary atrophy are abnormal, and rim thickness narmal.
Step Object Fundas image, Tool: VLM, Resul:: Drance hemoerhages condition
Drance hemorrhages are uncertain
not considered.
Stop +1: Object Fundus image, Tool: Sogmentation model 1. Result Optic Cup mask
Step 12:
Object: Fundus image: Toal: Segmentation model 2; Reeult: Optic Disc musk
3 Risk-based Decision: The weights for the four indicators
(VCDR, RT, PPA] are [0.5, D.3, 0.2. The final risk score 0.75, exceeding threshold 0.4, indicating the patient has glaucoma.
Object:
Fundus Tool: Cocting module, Result: Dise
mntl) Object Eye shotos, Tool. VLM) Visual Defects condition
Figure 4: A detailed case study for glaucoma diagnosis. The blue text indicates the agents, while the green text indicates the reasoning steps. More cases can be found in Appendix F
Memory Initialization. For each patient's s personalized multi-modal data D, the VLM performs an orchestration process to select executable steps from P based on data availability, filtering out those requiring unavailable inputs to form a long-term memory .M for each patient's diagnosis:
M E 0; E D).
As Fig. 4 shows, in the glaucoma diagnosis, when provided with the fundus image, the orchestration process selects relevant steps and skips those requiring unavailable data such as OCT scans.
Tool-based Action. During the reasoning, the VLM checks the data property of the current input: it matches any o;, MedAgent-Pro conducts corresponding step P. stored in memory M to obtain rü- For ag, while qualitative analysis is conducted directly by the VLM, quantitative analysis relies on professional tool agents in the toolset perform specialized assessments. T includes visual models such as segmentation tools (e.g., MedSAM (Ma et al.] (024a)). Cellpose (Stringer & Pachitariu 2025)) and L.LM-based coding tools (e.g., Copilot (github team ) where thorough descriptions and ablation studies of toolset are in Appendix B As shown in Fig. specialized segmentation tools are used extract the optic cup and disc masks, while coding tools subsequently compute the cup-to- disc ratio based on the segmentation results, which serves as key indicator in glaucoma diagnosis.
Evidence-based Reflection. During sequential reasoning, the system evaluates the output r; each step P/j to determine a status s; E { (Continue, Terminate, Complete) as short-term memory, which indicates how MedAgent-Prr take the subsequent action. s; is determined by:
Complete T4 E I,
85
Terminate
Ti & I -p(r;, Oz, G): T4 & I A d(r,, o; G)-
(3)
Continue
Here, e is state assessment function implemented by the VLM, which evaluates the reliability of ry based on the quality of input data og, and the plausibility of the result (Details in Appendix [E.4) If
s; = Continue, the output r; is treated as evidence e and used as the input 0++1 for the next reasoning step. If s, Terminate, the process is halted, r; is deemed unreliable and may hinder subsequent reasoning and lead to incorrect diagnoses. This process continues iteratively until s; = Complete.
The reasoning results in set of indicators: Rfinal {ra|s; Complete). ). The VLM then assigns based weights W for R final according to clinical importance grounded in medical guideline: W V(R sma G). The final risk score p computed weighted sum:
Risk- Decision.
R Rinal |=
Σ
war, s.t.w; E W.r, E R
(4)
0
5

---

Published as a conference paper at ICLR 2026
Glancoma
Heart Disease
NEJM (Acc)
Method
bAce
Fi
bAce
Fi
All
Cell Imaging
Chest X-ray
CT & MRI
Ophi. Imaging
General VLMs
GPT-4o
56.4
21.1
56.8
28.1
70.9
74.6
54.5
63.4
70.5
Janus: Pro 7B
53.4
13.3
52.3
10.7
30.0
32.2
36.4
29.3
28.2
LLaVA-MMdd
50.0
BioMedClip
58.1
47.0
37.8
27.9
27.1
40.0
29.3
20.5
Qwen2.5-7B VL
54.3
16.3
50.0
0.0
41.8
54.2
30.9
36.6
44,9
IntemV1.2.5..8
51.8
13.8
49.7
3.6
42.2
37.3
40.0
41.5
35,9
Medical Agentic Systems
MedAgents (AC1'24)
52.1
8.9
51.1
15.9
66.1
69.9
51.5
58.7
68.2
MMedAgent (EMNLP'2 24)
52.4
16.3
55.0
26.7
71.7
73.8
56.4
65.3
70.5
MDAgent (NeurlPS' 24)
56.8
22.2
57.2
30.3
73.8
79.6
52.9
67.3
23.0
MedAgent-
90.4
77.8
72.3
69.1
72.7
89.7
Table I: Comparison with general VLMs and medical agentic systems on REFUGE2, MITEA and NEJM (%). "Opht." is the short form of Ophthalmology. Our setting is highlighted in green
Enlarged Cardiomediastium
Avg.
Atelectasis
Cardiomegaly
Consolidation
Edema
Fracture
GPT-4o
58.3
68.7
64.3
60.5
61.2
$3.2
56.3
Junus-Pro-7B
51.9
61.7
54.1
45.2
52.5
34.1
50.0
LLaVA-Med
50.2
50.0
51.5
50.0
50.0
50.0
50.0
BioMedClip
57.9
48.6
62.7
62.2
50.0
61.1
50.0
Qwen2.5-7B-Vl
55.2
69.6
58.1
54.5
48.8
50.8
50.0
LnternVL2.5-8B
51.6
57.7
48.6
51.7
47,5
50.8
43.8
MedAgent-Prro
72.0
85.5
74.2
66.3
59.2
75.4
68.5
Lung Leision
Lung Opacity
Plcural Effusion
Supporting Devices
Method
Avg.
Pncumonia
Pneumothorax
GPT-4o
38.6
59.6
47.7
63.4
Janus-Pro-7B
59.1
42.1
57.4
47.2
62.4
LLaVA-Med
50.5
BioMedClip
53.0
Qwen2.5-7B-VL
43.2
59.3
34,9
80.8
59.1
54,2
47.6
55.3
MedAgent-Pro
77.8
62.1
65.6
89.2
Table Comparison with general VLMs on the MIMIC dataset (%). "Avg." is the average perfor- mance across 12 sub-tasks. Only bAcc values are presented; F1 score can be found in Appendix A
The final diagnosis is then made by comparing with a risk threshold 9. Through wridence-based reasoning, MedAgent- Pro integrates reliable external evidence with expert knowledge to improve diagnostic decision-making thus promoting the modern diagnosis workflow.
4
EXPERIMENT
4.1
EXPERIMENTAL SETUP
Dataset. We conduct experiments on four datasets. The REFUGE2 dataset (Fang et al. 2022 is used for glaucoma diagnosis, and the MITEA dataset (Zhao et al. 2023) for heart disease diagnosis, both of which require in-depth diagnostic reasoning. To assess multi- patients, we sample 442 chest X-ray cases from 100 patients from the MIMIC dataset (Johnson al.) 2023), each case involving the identification of up to 12 potential thoracic abnormalities. We further employ the New England Journal of Medicine (NEJM) database (of Medicine), where we compile 992 real-world diagnostic cases, encompassing 10+ anatomical regions, 10+ imaging modalities, and 50+ diseases. For cases involving cell or ophthalmology imaging, visual tools are available. However, non-clinical images (e.g., daily photographs) often lack compatible tool support.
diagnosis for individual
Evaluation Metrics. Three metrics are used to evaluate performance. For REFUGE2, MITEA, and MIMIC datasets, report balanced accuracy (bAce) and the FI score. For the NEJM dataset, where tasks are framed as multiple-choice questions in accordance with its evaluation protocol, we report the accuracy rate. The best results highlighted in bold, and the second-best underlined.

---

Published as a conference paper at ICLR 2026
REFUGE2 winners
Ophthalmology VLMs
Chest X-Ray VLMs
Team Name
AUC
Method
bAce
Fl
Method
bAcc
VUNO EYE TEAM
88.3
RetiZero
50.8
18.4
Maira-2
64.1
MIG
87.6
VisionUnite
85,8
73.1
CheXagent
69.1
MedAgent-Pro
95.1
MedAgent-Pro
90.4
76.4
MedAgent-Pro
72.0
Table 3: Comparison with task-specific models on diverse datasets (%).
Implementation Details. In the MedAgent-Pro, we use GPT-4o (Achiam et al. 2023) the base- line VLM by default, and implement the RAG agent using LangChain (Topsakal Akinci (0023).
For fair comparison, all baseline medical agentic systems adopt GPT-4o as their underlying VLM.
4.2 PERFORMANCE EVALCATIONS
Comparison with General VLMs. We compared with advanced VLMs including BioMedClip
iZhan et al. 2023a), GPT-4o (Achiam et al. (023), LLaVA-Med (Li et al. (Chen et al. 2025a), Qwen (Wang et al. 2024b). and VI. (Chen et al. 2024a). As these VLMs cannot handle 3D images, sample three random slices from each 3D echocardiography in the MITEA dataset, repeat this ten times, and report the mean performance to reduce sampling variability (std. value in Appendix A). As shown in Table [ and 2 the proposed MedAgent-Pit framework significantly outperforms existing VLMs across all datasets. Compared to GPT-4o, achieves improvements in bÂcc of 34.0% and 21.0%, and gains in F1 score of 55.3% and 44.2% for glaucoma and heart disease diagnosis respectively. This highlights MedAgent-Pro workflow's effectiveness in handling complex diagnostic tasks, particularly those requiring quantitative indica- tors such as cup-tor cup-to-disc ratio or left ventricular ejection fraction. Integration of visual tools into MedAgent-Prr Pro's 'ss reasoning enables precise indicator calculation and diagnosis, addressing current limitations of VLMs. More comparison results and detailed analyses are in Appendix A
2024b), Janus-Pro
Across diverse real- worl-world diagnostic scenarios in the NEJM (details in Appendix E), MedAgent-Pro demonstrates significant performance gains in domains with visual tool support, such as cell imag- ing, chest X-rays, CT/MRI, and ophthalmology. Furthermore, even maintains strong performance in cases without visual tool support, achieving overall improvement of 7.9%, which demonstrates its strong robustness and generalizability across variety of diagnostic tasks. In addition, as shown in Table 2, chest X-ray diagnosis involves certain tasks like Cardiomegaly which rely precise quantitative measurements like the cardiothoracic ratio. Meanwhile, others such as Fracture detce- tion, require detailed step-by-step analysis to identify subtle abnormalities. MedAgent-PPrg achieves leading performance across most tasks, with an average performance gain of 13.7%.
Comparison with Medical Agentic Systems. We also compare MedAgent- gent-Pro with advanced med- ical agentic frameworks, including MedAgents (Tang et al. 2024), MMedAgent (Li et al. 2024a) and MDAAeent (Kim et al., 2024). As shown in Table [ MedAgent-Pro consistently outperforms these methods across all diseases and domains. This performance advantage stems from the fact that prior methods are primarily designed for basic question answering or as modular toolboxes, lacking the capacity to handle complex, multi-modal clinical scenarios. In contrast, MedAgent -Pro incorporates retrieval- bbased diagnostic steps and seamlessly integrates visual tools into its reasoning process, enabling effective and comprehensive decision decision-making making support in clinical applications.
Comparison with Task-Specific Models. Additionally, we compare MedAgent- -rr with state-of-
the-art task-specific methods, i.e., for glaucoma diagnosis (Wang et al. 2024a [.i et al., (024c) and chest X-ray analysis (Bannur et al.] 2024 Chen et al., 2024b). For glaucoma diagnosis, we also compare with the winners from the REFUGE2 challenge leaderboard (Fang al. 2022). As the leaderboard only reports the AUC metric, our comparison is limited to this metric.
As shown in Table 3 MedAgent-Pro outperforms these task-specific methods, despite the VLMs in MedAgent-Pri remaining zero-shot. In glaucoma diagnosis, the AUC metric has improved by 6.8%, while the bAcc and Fl scores have increased by 4.6% and 3.3%, respectively. This finding further demonstrates that integrating specialized tools with general VLMs can achieve performance comparable domain-specific models, emphasizing the potential of the MedAgent-PP Pro framework.
7

---

Published as a conference paper at ICLR 2026
Setting
Glaucoma
Heart Disease
Planning
Action
Reflection
bAce
F1
bAcc
F1
Method
bAcc
F1
56.4
21.1
56.8
28.1
75.9
36.5
63.3
45.9
GPT-4o
90.4
76.4
88.5
71.0
73.4
66.6
VisionUnite
92.9
79.1
90.4
76.4
77.8
72.3
Table 5: Ablation on qualita- tive indicators analysis.
Table 4: Ablation on key components, including Planning. Tool-based Action and Evidence-based Reflection.
4.3 ABLATION STUDY AND DETAILED ANALYSIS
Effectiveness of the Proposed Key Components. We conduct an ablation study on glaucoma and heart disease diagnosis to evaluate the effectiveness of three key modules: planning, tool-based action, and evidence-based reflection, while each module builds upon the previous one. As shown in Table 4 incorporating planning significantly improves overall performance, while integrating visual tools support diagnostic action brings further gains, with F1 scores increasing by 34.5% and 20.7%, respectively. The addition of evidence- based reflection further enhances the consistency of plan execution and the reliability of actions, thereby reaching the best performance. These results validate the effectiveness of the components and demonstrate their complementary roles in jointly enabling evidence- based medical reasoning. Other detailed ablation of the toolset are in Appendix
Impact of Qualitative and Quantitative Accuracy. Both qualitative and quantitative analyses each reasoning step may introduce errors. To assess their respective impact on final accuracy, we conduct ablation studies on glaucoma diagnosis. Since the accuracy of qualitative analysis is difficult to quantify, we use an ophthalmic-specffic model VisionUnite (Li al., (024c) to con- duct qualitative analysis instead of the original GPT-4o. As listed in Table the VisionUnite only gains marginal improvement, indicating that general-purpos VLMs (e.g., GPT-4o), when guided by medical guidelines, are sufficient for qualitative analysis without requiring additional domain- specific tools. Furthermore, we simulate noisy segmentation masks following prior works (details in Appendix E), and observe that higher segmentation accuracy consistently yields better performance, as depicted in Fig. 5) These findings highlight the importance of tool-based actions in supporting quantitative analysis and the robustness gains achieved through evidence-base based reflection.
Ablation on Tool Accessibility We further conduct an ablation study to demonstrate that the per- formance gains of MedAgent-Pro stem from its carefully-designed workflow that enhances agentic reasoning, rather than from merely integrating additional tools. In this experiment, the baseline models given the same toolset directly, the results are shown below:
BRice
Glaucoma
Heart Disease
Actual bAce
28
Baseline Model
bAcc
Fl
bAcc
72
Score 052
GPT-4o
74.4
52.3
64.3
48.0
blcce (9%)
Linus-Pro-PPe
62.6
39.4
56.8
36.4
LLaVA-Med
50.0
0.0
50.0
0.0
Qwen2.5-7B- VL
72.6
43.0
43.3
17.3
InternVL.2.5880
60.6
27.7
51.4
13.3
MedAgent-Prre (Ours)
90.4
76.4
77.8
72.3
(56)
Figure 5: Ablation on quantitative indicator analysis that reveals how segmentation accu- racy influences diagnostic outcomes.
Table 6: Comparison of VL.Ms given with toolset access but without planning.
It can be seen that even when granted access to the toolset, these models fail effectively utilize the tools for coherent reasoning. For example, despite having access to segmentation models for optic cup and disc and coding module, they unable generate code to compute the cup-to-disc ratio.
8

---

Published as a conference paper at ICLR 2026
2010
Obsercetion
Grounding
Complexity Assessment
Usage Counts
17.5
Segmentation
15.0
Cinical Complexity
12.5
10.0
Apslectacis
Candiomegaly
Consolidation
Endarged Caroomefissmnn
Edema
Paattree
Long Lesion
Lung Opscity
Pleurat Effusion
Preumonia
Preumsthora
Support Devices
Figure 6: Diagnostic plan complexity vs. clinician assessment by subtasks on MIMIC.
CLINICAL EXPERT EVALUATIONS
Alignment with Real-World Clinical Workflow. To assess how well MedAgent-Pro aligns real clinical workflows, we quantified the number of steps it performs across 12 chest X-ray diagnostic tasks. Specifically, we compared the results with thoracic clinicians' rankings, where each task was rated from to 12 based on perceived diagnostic difficulty and time demand. As shown in Fig. [ most tasks show clear positive correlation between total step count and physician- complexity. For instance, Fracture is the most complex and involves the highest number of steps, while Support Devices require the fewest step and ranked least complex. Notably, conditions like Pleural Elfusion and Cardiomegaly benefit from visual tool integration, significantly reducing workflow steps. In contrast, tasks such as Fracture and Edema remain step- inteeinteesive. This is because diagnoses relying quantitative indicators (c.g.. the cardiothoracic ratio) can be automated by visual tools, while those requiring qualitative assessment still depend on sequential reasoning. The findings demonstrate the effectiveness and practical compatibility of MedAgent-Proo structured, evidence- based workflow with real-world clinical diagnostic process.
2F1-l
Jens
Manitt
Mosiggstotts
Jeas
Mosigenthh
Pecsabddy
(a) Doctor Evaluation on Glaucoma Diagnosis
(b) Doctor Evaluation on Chest X-Ray Diagnosis
Figure 7: Evaluation on glaucoma and chest X-ray diagnosis by clinical experts.
Assessment of Generated Diagnostic Content Across Other Methods. To further assess diagnos- tie quality beyond accuracy, we conduct validation with clinical experts on glaucoma and chest X-ray diagnosis. The clinicians rate the diagnostic outputs from both VL.Ms and MedAgent-Pryy across five piocensions-rllvvann
comprchensiveness, clinical reliability, reasoning coherence, and language clarity-usie to 5 scale (details in Appendix E.. MedAgent- outperforms other VLMs across five dimensions and demonstrates strong stability across diverse cases, underscoring the alignment of our structured, wvidence-based approach with modern medical diagnosis standards.
CONCLUSION
This paper introduces MedAgent-Proo a reasoning agentic system designed to deliver accurate, multi-modal medical diagnoses. Addressing the limitations of current empirical diagnosis. This

---

Published as a conference paper at ICLR 2026
represents significant step toward the core principles of evidence- based medicine and advancing the practical application of AI in healthcare. However, the proposed framework depends on the availability of visual tools, which remain limited in certain medical domains. In addition, qualitative analysis still relies on VLMs, which are prone to VLMs' inherent hallucination. Addressing these limitations will further improve the reliability and clinical impact of computer-aided diagnosis.
ACKNOWLEGEMENT
This work was supported by Tier 2 grant, Singapore (12HP20224-0028), and Ministry of Education Tier grant, NUS, Singapore (24-1250-P000)..
THE STATEMENTS
LLM USAGE STATEMENT
In this work, LLMs were used solely as writing assistants to polish the language of the manuscript. They were not involved in research ideation, experiment design, implementation, analysis, any other scientific contribution.
REPRODUCIBILITY STATEMENT
We have made extensive efforts to ensure the reproducibility of our work. The datasets used in our experiments are publicly available, and all preprocessing steps are described in detail in the main text and appendix. The implementation tool integration, and evaluation pipeline, has been uploaded to anonymous GitHub repository at https://anonymous
MedAgent-Pro. including the workflow design,
CCCC Hyperparameter settings, model con- figurations, and evaluation metrics are fully documented in the appendix. Additional ablation studies and qualitative analyses are also provided in the appendix to further support reproducibility.
ETHICS STATEMENT
This study involves human evaluation conducted by clinical experts. All participants were clearly informed of the purpose, scope, and potential risks of the study, and their participation was entirely voluntary with informed consent obtained beforehand. No sensitive patient data collected alyzed. The study was determined not to constitute human-subject research requiring IRB approval. We have adhered the ICLR Code of Ethics throughout the study, ensuring fairness, transparency, and compliance with ethical and legal standards.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, llge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Loai Albarqouni, Tammy Hoffmann, Sharon Straus, Nina Rydland Olsen, Taryn Young, Dragan Ilic, Terrence Shaneyfelt, R Brian Haynes, Gordon Guyatt, and Paul Glasziou. Core competencies in seadence--asse practice for health professionals: consensus statement based on a systematic review and delphi survey. JAMA network open, 80)) 2018.
American Diabetes Association. 2. classification and diagnosis of diabetes: standards of medical care in diabetes-2020. Diabetes care, 43 43(Sppleement 1):tcmmeeenn.S--33111 1):514-S31,
2020.
Shekoofeh Azizi, Basil Mustafa, Fiona Ryan, Zachary Beaver, Jan Freyberg, Jonathan Deaton, Aaron Loh, Alan Karthikesalingam, Simon Kornblith, Ting Chen, et al. Big self-supervised models advance medical image classification. In Proceedings of the IEEE/CVF international conference on computer vision, pp. -4888, 2021.
Mihalj Bakator and Dragica Radosav. Deep learning and medical diagnosis: A review of literature. Multimodal Technologies and Interaction, 2(3):47, 2018. 2018
10

---

Published as a conference paper at ICLR 2026
Shruthi Bannur, Kenza Bouzid, Daniel C Castro, Anton Schwaighofer, Anja Thieme, Sam Bond- Taylor, Maximilian Ilse. Fernando Pérez-Gase Garcia, Valentina Salvatelli, Harshita Sharma, et al. Maira-2: Grounded radiology report generation. arXiv preprint arXiv:2406.044: 2024.
Michael Baumgartner, Paul F Jäger, Fabian Isensee, and Klaus H Maier-Hein. andetection: self-configuring method for medical object detection. In Medical Image Computing and Com- puler Assisted Interrention-MICCA/ 2021: 24th International Conference, Strasbourg, France, September 27-Ociober 1, 2021, Proceedings, Purt 24, pp. 530-539. Springer, 2021.
Babak Ehteshami Bejnordi, Mitko Veta, Paul Johannes Van Diest, Bram Van Ginneken, Nico Karssemeijer, Geert Litjens, Jeroen AWM Van Der Laak, Meyke Hermsen, Quirine F Manson, Maschenka Balkenhol, et al. Diagnostic assessment of deep learning algorithms for detection of lymph node metastases women with breast cancer. Jama, 318(22):2199-2210 2017.
Manuela Benary, Xing David Wang, Max Schmidt, Dominik Soll, Georg Hilfenhaus, Mani Nassir, Christian Sigler, Maren Knödler, Ulrich Keller, Dieter Beule, et al. Leveraging large language models for decision support in personalized oncology. JAMA Network Open, 6(11):e234368- e2343689, 2023.
Elvira Beracochea, Corey Weinstein, Dabney Evans, et al. Rights-based approaches to public health. Springer Publishing Company, 2010.
Daniil Boiko, Robert MacKnight, and Gabe Gomes. Emergent autonomous scientific research
capabilities of large language models. arXiv preprint arXiv:2304.05332, 2023.
Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, et al. Do as i can, not as i say: Grounding language in robotic affordances. In Conference on robot learning, pp. 287-318. PMLR, 2023.
Elif Can, Wibke Uller, Katharina Vogt. Michael C Doppler, Felix Busch, Nadine Bayerl, Stephan Ellmann, Avan Kader, Aboelyazid Elkilany, Mareus R Makowski, al. Large language models for simplified interventional radiology reports: a comparative analysis. Academic Radiology, 32 (2):888-898, 2025.
Guangyao Chen, Siwei Dong, Yu Shu, Ge Zhang, Jaward Sesay, Börje F Karlsson, Jie Fu, and Yemin Shi. Autoagents: A framework for automatic agent generation. arXiv preprint arXiv:2309. 17288, 2023.
Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal understanding and generation with data and model scaling. arXiv preprint arXiv:2501.17811 2025a.
Ying Chen, Guoan Wang, Yuanfeng Ji, Yanjun Li, Jin Ye, Tianbin Li, Ming Hu, Rongshan Yu, Yu Qiao, and Junjun Hc. Slidechat: A large vision-language assistant for whole-slide pathology image understanding. In Proceedings of the Computer Vision and Pattern Recognition Confer- ence, pp. 5134-5143,
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 24185-24199 2024a.
Zhihong Chen, Maya Varma, Jean-Benoit Delbrouck, Magdalini Paschali, Louis Blankemeier, Dave Van Veen, Jeya Maria Jose Valanarasu, Alaa Youssef, Joseph Paul Cohen, Eduardo Pontes Reis, et al. Chexagent: Towards a foundation model for chest x-ray interpretation. arXiv preprint arXiv:2401. 12200 2024b.
David M Eddy. Practice policies: where do they come from? Jama, 263(9); 1990a.
David M Eddy. Guidelines for policy statements: the explicit approach. Jama, 263(16):2239-2243 1990b.
DM Eddy. Clinical decision making: from theory to practice. practice ods. JAMA, 18331199113911888 1990c.
for meth-
11

---

Published as a conference paper at ICLR 2026
DM Eddy. Clinical decision making: from theory to practice. practice policies-guideling guidelines for meth- ods. JAMA, 263(13):1839 1841, 1990d.
Ruth Falik. Symptom to diagnosis: An evidence-basee guide. JAMA: The Journal of the American
Medical Association, 295(22):2667-222 22668 2006.
Adibvafa Fallahpour, Jun Ma, Alif Munim, Hongwei Lyu, and Bo Wang. Medrax: Medical reason- ing agent for chest x-ray. arviv preprint arXiv:2502.02673, 2025.
Huihui Fang, Fei Li, Junde Wu, Huazhu Fu, Xu Sun, Jaemin Son, Shuang Yu, Menglu Zhang, Chenglang Yuan, Cheng Bian, et al. Refuge2 challenge: A treasure trove for multi-dimension analysis and evaluation in glaucoma screening, arXiv preprint arXiv:2202.08999, 2022.
Dennis Fast, Lisa C Adams, Felix Busch, Conor Fallon, Mare Huppertz, Robert Siepmann, Philipp Prucker, Nadine Bayerl, Daniel Truhn, Marcus Makowski, al. Autonomous medical evaluation for guideline adherence of large language models. NPJ Digital Medicine, 7(1):1-14, 2024.
Nima Fathi, Amar Kumar, and Tal Arbel. Aura: A multi-modal medical agent for understanding, reasoning and annotation. In International Workshop on Agentic Al for Medicine, pp. 105-114. Springer, 2025.
Roberto Gallotta, Graham Todd, Marvin Zammit, Sam Earle, Antonios Liapis, Julian Togelius, and Georgios N Yannakakis. Large language models and games: A survey and roadmap. arXiv preprint 4rXiv:2402.18659, 2024.
Fatemeh Ghezloo, Mehmet Saygin Seyfioglu, Rustin Soraki, Wisdom O Ikezogwo, Beibin Li, Tejoram Vivekanandan, Joann G Elmore, Ranjay Krishna, and Linda Shapiro. Pathfinder: A multi-modal multi-agent system for medical diagnostic decision-making applied to histopathol- ogy. arXiv preprint arXiv:2502.08916, 2025.
The github team. Github copilot. May 11, 2025.
https://ggthbb..co//ettteee
Accessed:
Diederick E Grobbee and Arno W Hoes. Clinical epidemiology: principles, methods, and applica- lions for clinical research. Jones & Bartlett Publishers, 2014.
Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. Domain-specific language model pretraining for biomedical natural language processing. ACM Transactions Computing for Healthcare (HEALTH), 3(1): 1-23, 2021.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501. 12948, 2025.
Gordon Guyatt, John Cairns, David Churchill, Deborah Cook, Brian Haynes, Jack Hirsh, Jan Irvine, Mark Levine, Mitchell Levine, Jim Nishikawa, al. Evidence-base medicine: new approach 1992a.
to teaching the practice of medicine. jama, (68(17)2440--245,
Gordon Guyatt, John Cairns, David Churchill, Deborah Cook, Brian Haynes, Jack Hirsh, Jan Irvine, Mark Levine, Mitchell Levine, Jim Nishikawa, al. Evidence-based medicine: a new approach to teaching the practice of medicine. jama, 268(17):2420, 2422222244 1992b.
Xuehai He, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie. Pathvqa: 30000+ questions for medical visual question answering. arXiv preprint arXiv:2003. /0286, 2020.
Jacob Hershenhouse, Daniel Mokhtar, Michael Eppler, Severin Rodler, Lorenzo Storino Ramac- ciotti, Conner Ganjavi, Brian Hom, Ryan Davis, John Tran, Giorgio Ivan Russo, et al. Accuracy, readability, and understandability of large language models for prostate cancer information to the public. Prostate Cancer and Prostatic Diseases, pp. 1-6, 2024.
Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.055m 2022.
12

---

Published as a conference paper at ICLR 2026
Fabian Isensee, Paul F Jaeger, Simon AA Kohl, Jens Petersen, and Klaus H Maier-Hein. nnu- net: self-configuring method for deep learning-basc based biomedical image segmentation. Nature methods, 18(2):203-211, 2021.
Shi Jinxin, Zhao Jiabao, Wang Yilei. Wu Xingjiao, Li Jiawen, and He Liang. Cgmi: Configurable general multi-agent interaction framework. arXiv preprint arXiv:2308. 12503, 2023.
Alistainr EW Johnson, Lucas Bulgarelli, Lu Shen, Alvin Gayles, Ayad Shammout, Steven Horng, Tom J Pollard. Sicheng Hao, Benjamin Moody, Brian Gow, et al. Mimic-iv, a freely accessible electronic health record dataset. Scientific data, 10(1):1, 2023.
David L Katz. Clinical epidemiology & evidence-based medicine: fundamental principles of clinical reasoning research. Sage Publications, 2001.
Yash Khare, Viraj Bagal, Minesh Mathew, Adithi Devi, U Deva Priyakumar, and CV Jawahar. Mm- bert: Multimodal bert pretraining for improved medical vqa. In 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), pp. 1033-1036. IEEE, 2021.
Yubin Kim, Chanwoo Park, Hyewon Jeong, Yik Siu Chan, Xuhai Xu, Daniel McDuff, Hyeonhoon Lee, Marzyeh Ghassemi, Cynthia Breazeal, and Hae Won Park. Mdagents: An adaptive collab- oration of Ilms for medical maccis, In The Tiirth-eighth Annual Conference on Neural Information Processing Systems, 2024.
Igor Kononenko. Machine learning for medical diagnosis: history, state of the art and perspective. Artificial Intelligence in medicine, 23(1):89-109 2001.
Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman, dataset of clinically generated visual questions and answers about radiology images. Scientific data, 5(1):1-10, 2018.
Binxu Li, Tiankai Yan, Yuanting Pan, Jie Luo, Ruiyang Ji, Jiayuan Ding, Zhe Xu, Shilong Liu, Haoyu Dong, Zihao Lin, et al. Mmedagent: Learning to use medical tools with multi-modal agent. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 8745- 8760, 2024a.
Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Nau- mann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training large language-and- assis- tant for biomedicine in one day. Advances in Neural Information Processing Systems, 36, 2024b.
Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Com- municative agents for" mind" exploration of large language model society. Advances in Neural Information Processing Systems, (6:51991-52008, 2023.
Zihan Li, Diping Song, Zefeng Yang, Deming Wang, Fei Li, Xiulan Zhang, Paul E Kinahan, and Yu Qiao. Visionunite: A vision-language foundation model for ophthalmology enhanced with clinical knowledge. arXiv preprint arXiv:2408.022 022865 2024c.
Xinjie Liang, Xiangyu Li, Fanding Li, Jie Jiang, Qing Dong, Wei Wang, Kuanquan Wang, Suyu Dong, Gongning Luo, and Shuo Li. Medfilip: Medical fine-grained language-image pre-training. IEEE Journal of Biomedical and Health Informatics, 2025.
Yuci Liang, Xinheng Lyu, Wenting Chen, Meidan Ding, Jipeng Zhang, Xiangjian He, Song Wu, Xiaohan Xing, Sen Yang, Xiyue Wang, al. Wsi-llava: A multimodal large language model for whole slide image. arXiv preprint 24Xiv2241..02141 2024.
Jingyang Lin, Yingda Xia, Jianpeng Zhang, Ke Yan, Le Lu, Jiebo Luo, and Ling Zhang. Ct-glip: 3d grounded language- image pretraining with et scans and radiology reports for full-body scenarios. arXiv preprint urXiv:2404. 5272, 2024.
Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang. and Xiao-Ming Wu. Slake: A semantically- labeled knowledge-enhance dataset for medical visual question answering. In 2021 IEEE 18th International Symposium Biomedical Imaging (ISBI), pp. 1650-1654, IEFE, 2021.
13

---

Published as a conference paper at ICLR 2026
Xinheng Lyu, Yuci Liang, Wenting Chen, Meidan Ding, Jiaqi Yang, Guolin Huang, Daokun Zhang, Xiangjian He, and Linlin Shen. Wsi-agents: A collaborative multi-agent system for multi-modal whole slide image analysis. arXiv preprint arXiv:2507.14680, 2025.
Jun Ma, Yuting He, Feifei Li, Lin Han, Chenyu You, and Bo Wang. Segment anything in medical images. Nature Communications, 15(1):654, 2024a.
Zilin Ma, Yiyang Mei, and Zhaoyuan Su. Understanding the benefits and challenges of using large language model-based conversational agents for mental well-being support. In AMIA Annual Symposium Proceedings, volume 2023, pp. 1105, 2024b.
Stephen McPhee, Maxine A Papadakis, Michael W Rabow, et al. Current medical diagnosis & treatment 2010. McGraw-Hil Medical New York:, 2010.
Nikhil Mehta, Milagro Teruel, Patricio Figueroa Sanz, Xin Deng, Ahmed Hassan Awadallah, and Julia Kiseleva. Improving grounded language understanding in collaborative environment by interacting with agents through help feedback. arXiv preprint arXiv:2304. 10750, 2023.
Naomi Miller, Eve-Marie Lacroix, and Joyce EB Backus. Medlineplus: building and maintaining the national library of medicine's consumer health web service. Bulletin of the Medical Library Association, 88(1):11, 2000.
Michael Moor, Qian Huang, Shirley Wu, Michihiro Yasunaga, Yash Dalmia, Jure Leskovee, Cyril Zakka, Eduardo Pontes Reis, and Pranav Rajpurkar. Med-flamingo: a multimodal medical few- shot learner. In Machine Learning for Health (ML4H), pp. 353-367. PMLR, 2023.
The New England Journal of Medicine. Nejm image challenge. https: hwtp://wwwww.m ne mm.g.
smnagecchalerin Accessed: May 11, 2025.
Terrence Shaneyfelt, Karyn D Baum, Douglas Bell, David Feldstein. Thomas Houston, Scott Kaatz, Chad Whelan, and Michael Green. Instruments for evaluating education in evidence-based practice: systematic review. Jama, 296(9): 116-1111111 117,, 2006.
Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei. Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. Large language models encode clinical knowledge. Nature, 120-7972) 2023.
Earl Steinberg, Sheldon Greenfield, Dianne Miller Wolman, Michelle Mancher, and Robin Graham. Clinical practice guidelines we can trust. national academies press, 2011.
Carsen Stringer and Marius Pachitariu. Cellpose3: one-click image restoration for improved cellular segmentation, Nature methods, 22(3):592 599, 2025.
Thomas Yu Chow Tam, Sonish Sivarajkumar, Sumit Kapoor, Alisa V Stolyar, Katelyn Polanska, Karleigh R McCarthy, Hunter Osterhoudt, Xizhi Wu, Shyam Visweswaran, Sunyang Fu, et al. A framework for human evaluation of large language models in healthcare derived from literature review. NPJ digital medicine, 7(1):258, 2024.
Xiangru Tang, Anni Zou, Zhuosheng Zhang, Ziming Li, Yilun Zhao, Xingyao Zhang, Arman Cohan, and Mark Gerstein. Medagents: Large language models as collaborators for zero-shot medical reasoning. In Findings of the Association for Computational Linguistics ACL 2024, pp. 599-621, 2024.
Oguzhan Topsakal and Tahir Cetin Akinei. Creating large language model applications utilizing langchain: primer on developing IIm apps fast. International Conference on Applied Engi- neering and Natural Sciences, volume 1, pp. 1050-1056, 2023.
Dong Wang, Yuan Zhang, Kexin Zhang, and Liwei Wang. Focalmix: Semi-supervised learning for 3d medical image detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3951-3960, 2020.
Dujuan Wang, Tao Cheng, Sutong Wang, Youhua Frank Chen, and Yunqiang Yin. Smr- Enn-ageets: Synergistic medical reasoning agents for zero-shot medical visual question answering with milms. Information Processing & Management, 63419, 2026.
14

---

Published as a conference paper at ICLR 2026
Meng Wang, Tian Lin, Aidi Lin, Kai Yu, Yuanyuan Peng, Lianyu Wang, Cheng Chen, Ke Zou, Huiyu Liang, Man Chen, et al. Common and rare fundus diseases identification using vision-language foundation model with knowledge of over 400 discases. arXiv preprint arXiv:2406.093 17, 2024a.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision- -anguage model's perception of the world at any resolution. arXiv preprint arXiv:2409. 12191, 2024b.
Junde Wu, Ziyue Wang, Mingxuan Hong, Wei Ji, Huazhu Fu, Yanwu Xu, Min Xu, and Yueming Jin. Medical sam adapter: Adapting segment anything model for medical image segmentation. Medical image analysis, 102:103547, 2025.
Peng Xia, Kangyu Zhu, Haoran Li, Tianze Wang. Weijia Shi, Sheng Wang, Linjun Zhang, James Zou, and Huaxiu Yao. Mmed-rag: Versatile multimodal rag system for medical vision language models. arXiv preprint arXiv:2410.13085. 2024.
Yuchen Xia, Manthan Shenoy, Nasser Jazdi, and Michael Weyrich. Towards autonomous system: flexible modular production system enhanced with large language model agents. In 2023 IEEE 28th International Conference on Emerging Technologies and Factory Automation (ETFA), pp. 1-8. IEEE, 2023.
Cheng Xue, Qiao Deng, Xiaomeng Li, Qi Dou, and Pheng-Ann Heng. Cascaded robust learning at imperfect labels for chest x-ray segmentation. In Medical Image Computing and Computer Assisted Intervention-MICCAI 2020: 23rd International Conference, Lima, Peru, October 4-8, 2020, Proceedings, Part VI 23, pp. 579-588 008.588.77 Springer, 2020.
Yijun Yang, Zhao- Yang Wang, Qiuping Liu, Shuwen Sun, Kang Wang, Rama Chellappa, Zongwei Zhou, Alan Yuille, Lei Zhu, Yu-Dong Zhang, et al. Medical world model: Generative simulation of tumor evolution for treatment planning. arXiv preprint arXiv:2506.0233 02327, 2025.
Li-Ming Zhan, Bo Liu, Lu Fan, Jiaxin Chen, and Xiao-Ming Wu. Medical visual question an- swering via conditional reasoning. In Proceedings of the 28th ACM International Conference on Multimedia, pp.
2020.
Jianpeng Zhang, Yutong Xie, Qi Wu, and Yong Xia. Medical image classification using synergic deep learning. Medical image analysis, 54:10-19, 2019.
Minqing Zhang, Jiantao Gao, Zhen Lyu, Weibing Zhao, Qin Wang. Weizhen Ding, Sheng Wang, Zhen Li, and Shuguang Cui. Characterizing label errors: confident learning for noisy-labeled image segmentation. Medical Image Computing and Computer Assisted Intervention-MICC. 2020: 23rd International Conference, Lima, Peru, October 4-8, 2020, Proceedings, Part I 23, pp. 721-730. Springer, 2020a.
Sheng Zhang, Yanbo Xu, Naoto Usuyama, Hanwen Xu, Jaspreet Bagga, Robert Tinn, Sam Pre- ston, Rajesh Rao, Mu Wei, Naveen Valluri, et al. Biomedclip: multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs. arXiv preprint arXiv:2303.009/5 2023a.
Tianwei Zhang, Lequan Yu, Na Hu, Su Lv, and Shi Gu. Robust medical image segmentation from non-expert annotations with tri-network. In Medical Image Computing and Computer Assisted Intervention-Mil MICCAI 2020: 23rd International Conference, Lima, Peru, October 4-8, 2020, Proceedings, Part IV 23, pp. 249-258. 258. Springer, 2020b.
Wenchuan Zhang, Penghao Zhang, Jingru Guo, Tao Cheng, Jie Chen, Shuwan Zhang, Zhang Zhang, Yuhao Yi, and Hong Bu. Patho-rl: multimodal reinforcement learning-based pathology expert reasoner. arXiv preprint arXiv:2505.1 1404, 2025.
Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Weixiong Lin, Ya Zhang, Yanfeng Wang, and Weidi Xie. Pmc-vqa: Visual instruction tuning for medical visual question answering. arXiv preprint arXiv:2305. 10415, 2023b.
15

---

Published as a conference paper at ICLR 2026
Debbie Zhao, Edward Ferdian, Gonzalo D Maso Talou, Gina M Quill, Kathleen Gilbert, Vicky Y Wang, Thiranja Babarenda Gamage, João Pedrosa, Jan D D'hooge Timothy Sutton, et al. Mitea: A dataset for machine learning segmentation of the left ventricle in 3d echocardiography using subject-specific labels from cardiac magnetic resonance imaging. Frontiers in Cardiovas- cular Medicine, 9:1016703, 2023.
Haidong Zhu, Jialin Shi, and Ji Wu. Pick-and-learm: Automatic quality evaluation for noisy-labeled image segmentation. In Medical Image Computing and Computer Assisted Intervention-MICCA.. 2019: 22nd International Conference, Shenzhen, China, October 13-17, 2019, Proceedings, Purt VI 22, PD. 376-584. Springer, 2019.
Yinghao Zhu, Ziyi He, Haoran Hu, Xiaochen Zheng, Xichen Zhang, Zixiang Wang, Junyi Gao, Liantao Ma, and Lequan Yu. Medagentboard: Benchmarking multi-agent collaboration with conventional methods for diverse medical tasks. arXiv preprint arXiv:2505. 02225555..37,,, /2371, 2025a.
Yinghao Zhu, Yifan Qi, Zixiang Wang, Lei Gu, Dehao Sui, Haoran Hu, Xichen Zhang, Ziyi He, Junjun He. Liantao Ma, et al. Healthflow: A self-evolving ai agent with meta planning for au- tonomous healthcare research. arXiv preprint arXiv:2508.0262) 2025b.
Kaiwen Zuo, Yirui Jiang, Fan Mo, and Pietro Lio. Kg4diagnosis: hierarchical multi-agent IIm framework with knowledge graph enhancement for medical diagnosis. arXiv preprint arXiv:2412..6883,, 2024.
16

---

Published as a conference paper at ICLR 2026
APPENDIX
Table of content:
Additional Comparative Result
A.1
Full Comparison with Domain-specific Methods on MIMIC
A.2
F1 Score Result on the MIMIC
A.3
Results on Additional Metrics
Comparison of Methods' Robustness
A.5
Results on General Medical VQA Datasets
A.6
Results Sub-tasks in the Heart Disease Diagnosis
B
Additional Ablation Studies
[B.]
Comparison of using Different Baselines
Ablation on the RAG Agent
B.3
Ablation on Different Toolset
B. 4
Ablation on Tool Accessibility
B.5
Analysis of Decision-Making Strategies.
Other Detailed Analysis
Study Computation Cost
Discussion on Relation to Medical World Model
Background Evidence based Medicine
Implementation Details
F..1
Experimental Setup
E.
Generation of Noisy Labels
E.3
Criteria for Human Evaluation
E.
Details for Evidence-base. Reflection
E.5
Details for Toolset
Additional Qualitative Analysis
Additional Case Studies on Glaucoma and Chest X-ray Diagnosis
A ADDITIONAL COMPARATIVE RESULT
Due to space limitations in the main paper, many of our results were not fully presented; these results provided in the appendix below:
A.1 THE FULL COMPARISON WITH DOMAIN-SPECIFIC METHODS ON MIMIC
In Table of the main text, we only report the average balanced accuracy (bAcc) across the 12 tasks when comparing MedAgent-PP Pro with state-of-the-ar (SOTA) chest X-ray diagnosis models such as CheXagent (Chen et al.) (024b) and Maira (Bannur et al. 2024). Therefore, we present the complete results for all 12 tasks below:
It can be observed from Table 7 outperform GPT-4o on most tasks, and exhibit exceptional performance on certain tasks. For exam- ple, Xagent achieves leading bAcc of 81.8% on Lung Lesion and 84.2% on Pneumothorux.
that that after fine-tuning on chest X-ray data, both models significantly
However, these models inevitably suffer from data imbalance, resulting in poor performance on certain datasets. For instance, CheXagent only achieves a bAcc of 43.3% on Atelectasis, and Maira reaches just 50.9% on Pneumonia diagnosis, which severely impacts their average performance.
construct diagnosis plan for each disease using medical guidelines retrieved by the RAG agent, and utilize specialized visual tools to support pro- fessional analysis. This approach integrates domain knowledge directly into the workflow instead of fine-tuning the VLM, thereby avoiding the impact data imbalance. As result, our method achieves competitive performance across all 12 tasks, reaching the highest average bAcc of 72.0%.
Accordingly, in proposed MedAgent-Pro
17

---

Published as a conference paper at ICLR 2026
Enlarged Cardiomediastinum
Method
Avg.
Atelectasis
Cardionegaly
Consolidation
Edema
Fracture
GPT-40
58.3
68.7
64.3
60.5
61.2
53.2
$6.3
Maira
70.2
63.7
52.4
CheXagent
75.4
72.5
68.8
MedAgent-Pro
72.0
74.2
59.2
68.5
Lung Lcision
Lung Opacity
Pleural Effusion
Supporting Devices
Method
Avg.
Pneumonia
Pucumothorax
GPT-4o 58.3
38.6
63.1
62.7
59.6
47.7
63.4
Maira
64.1
68.2
74.3
76.8
50.9
50.0
77.7
CheXagent
69.1
81.8
73.6
80.7
67.0
84.2
68.3
MedAgent--Pr 72.0
72.2
67.6
77.8
62.1
65.6
89.2
Table 7: Comparison with zero-shot foundation models the MIMIC subset with task-specific models on bAcc metric (%). "Avg." is the average performance for the 12 sub-tasks.
A.2 Fl SCORE RESULT ON THE MIMIC DATASET
Due to space constraints in the main text, we only reported bAcc results the MIMIC dataset. Here, we provide the Fl scores across all 12 tasks for general VLMs, task-specific models, and the proposed MedAgent- Pro, along with their average.
Enlarged Cardiomediastium
Method
Avg.
Atelectasis
Cardiomegaly
Consolidation
Edema
Fracture
VLM-based Methods
GPT-4o
81.0
70,8
71.0
67.1
22.2
Janus-Pro-78
28.5
37.6
24.0
15.4
0.0
LLaVA-Med
45.4
84.3
81.0
72.0
0.0
BioMedClip
82.9
36.4
0.0
Qwen2,5-7B-V7
56.4
36.9
0.0
0.0
InterVVL2.5880
48.6
85.6
69.9
40.0
0.0
46.2
Agentic Systems
Maira
43.6
50.3
61.5
24.0
49.1
36.3
16.3
CheXagent
70.3
92.5
81.3
48.5
69.4
36.4
54.5
MedAAgent-rr
73.0
83.1
80.3
58.8
66.7
82.4
71.4
Lung Leision
Lung Opacity
Pleural Effusion
Supporting Devices
Method
Avg.
Pneumonia
Pneumothorax
VLM-based Methods
GPT-4o
61.3
37.5
81.4
67.2
0.0
80.7
Janus-Pro-7B
28.5
30.8
36.1
39.4
0.0
75.2
LLaVA-Med
45.4
83.1
45.9
76.0
BioMedClip
77.7
40.7
19.5
Qwen2.5-7B-VL
51.2
45.7
86.8
V1..55
Agentic Systems
Maira
43.6
53.3
66.0
74.9
7.8
0.0
83.3
CheXageat
70.3
77.8
88.7
87.5
57.5
53.8
95.0
MedAggnt-Pro
73.0
63.5
80.9
76.9
63.2
53.1
95.7
Table 8: Comparison with zero-shot foundation models and domain -ppeii- models on the MIMIC subset on Fl score (%). "Avg." is the average performance for the sub-tasks.
As shown in Table 8 our proposed MedAgent-Pro achieves state-of-the-art performance on the MIMIC dataset, attaining the highest overall average Fl score of 73.0% across all 12 diagnostic tasks. It outperforms both general-purpose VLMs (e.g., GPT-4o at 61.3%) and task-specific mod- els (e.g., CheXagent at 72.1%), and demonstrates exceptional results key clinical tasks such as Enlarged Cardiomediastinum and Fracture. Unlike other models that suffer from performance degradation due data imbalance limited task specialization, MedAgent-Pa maintains consis-
18

---

Published as a conference paper at ICLR 2026
tently strong results across all tasks, with its lowest FI score still reaching 53.1% Pneumothorax, highlighting its robustness and generalizability.
While the Fl score is widely used metric for evaluating classification performance, it also has in- herent limitations, particularly in imbalanced datasets. The Fl score emphasizes performance on the positive class and may overlook the model's ability correctly identify negative cases. As result, a model can achieve a deceptively high F1 score by predicting all instances as positive. For instance, on the Pleural Effusion task, LLa achieves the second-highest FI score (83.1%% 1%) among all models. However, its balanced accuracy (bAcc) is only 50.0%, indicating that the model predicts all samples positive without discriminating between classes. Therefore, fairly and compre- hensively assess model's performance, the Fl scores presented in Table g should be interpreted alongside the bAec metrics reported in Table of the main text.
A.3 RESULTS ON ADDITIONAL METRICS
For the binary diagnosis tasks like glaucoma and heart disease diagnosis, we also report recall and precision values here to offer more comprehensive view of the model's performance. It can be seen that our model not only performs well bAcc and F1 metric, also have significant performance gain the recall and precision metric, which shows our method presents high generalization ability across different tasks.
Glaucoma
Heart Disease
Method
bAce
Fl
Recall
Precision
bAce
F1
Recall
Precision
VLM-based Methods
GPT-4o
56.4
21.1
30.0
16.2
56.8
28.1
17.3
75.0
Janus-Pro-7B
53.4
13.3
60.0
52.3
10.7
5.7
75.0
L.LVAAMMed
50.0
0.0
0.0
50.0
0.0
0.0
BioMedClip
58.1
21.3
75.0
12.4
47.0
37.8
40.3
35.6
Qwen2.5-7B-VL
54.3
16.3
10.0
44.4
50.0
0.0
0.0
0.0
122emVL2..-8B
51.8
15.0
49.7
33.3
Agentic Systems
MedAgents (ACL'24)
$2.1
8.9
5.0
40.0
51.1
15.9
9.6
45.5
MMedAgent (EMNLP'24)
52.4
16.3
25.0
12.0
55.0
26.7
17.3
60.0
MDAgent (NeurIPS'24)
56.8
22.2
22.0
57.2
30.3
71.4
MedAgent-Pro (Ours)
90.4
76.4
85.0
69.4
77.8
72.3
65.4
81.0
Table 9: Comparison with general VI.Ms and medical agentic systems on REFUGE2 (Glaucoma) and MITEA (Heart Discase) datasets with additional metrics (%).
A.4 COMPARISON OF METHODS' ROBUTNESS
We repeated experiments on heart disease 10 times and report the standard deviations in the table below:
Method
bAce
F1
GPT-4o
4.8
12.7
Janus-Pro-7B
1.2
4.1
I.LaV--Med
0.0
0.0
Qwen2.5-7B-VI.
0.0
LntemVL2.558B
0.8
2.1
MedAgent- (Ours)
0.3
0.4
Table 10: The standard deviation of different methods the heart disease diagnosis.
As shown in the table, GPT-4o exhibits high variance. contrast, MedAgent-Pae despite using GPT-40 as the backbone, achieves significantly lower variance, indicating improved consistency.
19

---

Published as a conference paper at ICLR 2026
The std values of some methods due to consistent failure across all cases rather than true stability. MedAgent- remains stable through two main strategies: 1) The diagnostic process in MedAgent-Pro is dominated by quantitative analysis performed by specialized tools, which give certain results, significantly reducing reliance on unstable VLM outputs. 2) The evidence-based reflection paradigm verifies the intermediate outputs each reasoning step. When tool outputs unavailable or unreliable, the workflow interrupts that reasoning path and excludes the correspond- ing indicator from the final decision, thus improving robustness.
A.5 RESULTS ON GENERAL MEDICAL VQA DATASETS
Notably, our method is specifically tailored for medical diagnosis tasks that demand multi-step, evidence-based reasoning, rather than conventional VQA-style queries. Nevertheless, to demon- strate its generalizability, evaluate on a subset of 300 VQA pairs from SLAKE- VQA and 500 pairs from PathVQA, with results reported below:
still compare with VLMs general VQA tasks. Specifically,
Method
PathVQA
Slake-VOC
GPT-4o
56.2
58.3
Janus-Pro-7B
50.4
48.7
LLaVA-Med
54.4
32.0
Qwen2.5-7B- BL
47.4
47.3
VntemSVL.2.5SB
54.2
MedAgent-PPrs Pro (Ours)
62.5
63.4
Table 11: Comparison on Medical VQA datasets.
These results show that while MedAgent-Prre Pro is not designed specifically for VQA tasks, it nonethe- less achieves the best even on radiology VQA datasets, further validating the robustness and adapt- ability of our framework.
A.6 RESULTS OF SUB-TASEE IN THE HEART DISEASE DIAGNOSIS
In the main text, we simplify the heart disease task in the Mitea dataset into a binary classification task. Here present the full result all the sub-task in the Mitea dataset, which is amyloidosis, aortic regurgitation, cardiomyopathy dilated, cardiomyopathy hypertrophic, hypertrophy and trans- plant. The results shown below:
Aonic Regurgitation
Cardiomyopathy Dilated
Cardiomyoputhy Hypertrophic
Method
Avg.
Amyloidosis
Hypertrophy
Transplant
Cases
12
10
2
GI'T-4o
65.6
50.6
58.6
52.4
53.0
Janus-Pro-7B
49.3
57.6
35.8
49.1
64.6
34.5
LLaVA-Med
51.6
50.0
54.1
49.6
50.0
50.0
Qwen2.5-7B-V
50.0
50.0
48.9
IL.e5nVL...588B
52.5
54.9
51.8
48.5
48.9
MedAgent-Pro
82.8
93.0
80.2
87.4
85.1
80.7
70.5
Table 12: The bAcc value of the sub-tasks in heart disease diagnosis.
As shown, MedAgent-Pro achieves even better performance when diagnosing more specific heart- disease subtypes, substantially outperformig VLMs. This is because a clearer disease query allows MedAgent- Pro to focus on more direct clinical indicators (e.g., ventricular wall thickening for amy- loidosis), leading more accurate decisions.
20

---

Published as a conference paper at ICLR 2026
B ADDITIONAL ABLATION STUDIES
B.I COMPARISON OF USING DIFFERENT BASELINES
While we employ GPT-4o as the backbone VLM in the main text due to its reasoning capabilities and accessibility, emphasize that MedAgent-Pro is not tightly coupled to GPT-4o. 'he critical components in MedAgent-Prr including clinical knowledge retrieval and quantitative assessment, are performed by the RAG agent and specialized tools, reducing reliance on any single VLM. To verify this, we conducted additional experiments by replacing GPT-4o with two open-source VLMs, Qwen2.5-7B- VL and Intern VL2.5-8B, as the backbone the workflow:
Glaucoma
Heart Disease
Baseline Model
bAce
bAcc
F1
VLM-based Methods
Qwen-2.5-7B-VL
54.3
16.3
50.0
0.0
Intern VL2.5-8B
51.8
49.7
3.6
GPT-4o
56.4
21.1
56.8
28.1
Agentic Systems
MedAgent-Pro (Qwen)
86.3
60.6
71.7
62.8
MedAgent-Prro (InternVL)
84.3
53.7
70.7
61.2
MedAgent-PPi (GPT-40)
90.4
76.4
77.8
72.3
Table 13: The comparison of using different VLMS as backbones.
As shown in the table, while replacing GPT-4o with open-source models results moderate perfor- mance drops, MedAgent-Ping still achieves significant gains over using the backbone VLMs alone (refer to Table in the main text), demonstrating that the performance improvement stems from the design of our workflow, not from reliance any specific foundation model.
B.2 ABLATION ON THE RAG AGENT
LangChain was employed in our RAG agent implementation mainly for convenience, as it offers modular and widely adopted interface. Importantly, the retrieval module is fully interchange- able, and our framework not tied to any specific implementation. To verify this, we replaced LangChain's retriever with FAISS-based alternative, which fully open-source and independent of LLMs. The results are shown below:
Glaucoma
Heart Disease
RAG Method
bAce
F1
bAce
Fl
FAISS
91.5
76.9
77.2
71.6
LangChain (Ours)
90.4
76.4
77.8
72.3
Table 14: The impact of different implementations of the RAG agent.
As shown in the table, switching to an open-source retrieval method leads to minimal performance degradation and even slight improvements in glaucoma diagnosis, demonstrating that our framework is robust to the underlying RAG implementation.
As for the exploration of hypersparameters in the RAG module, conducted an ablation study the number of retrieved chunks l used for generating the diagnostic plan:
As shown in the table, framework exhibits strong robustness to variations in the top-k retrieval parameter. While a larger offers marginal improvements up to a point, the overall performance remains stable across wide range.
21

---

Published as a conference paper at ICLR 2026
Glaucoma
Heart Disease
k for Top-k
bAcc
F1
bAcc
F1
1
84.7
80.0
70.7
61.1
2
87.6
78.5
71.7
62.8
5
90.4
76.4
77.8
72.3
10
90.4
76.4
77.2
71.5
Table 15: The impact of the hyperparameter in the RAG agent.
We have further replaced MedlinePlus with ACC/AHA guideline documents for the Heart Disease task, which is more professional guidelines from medical socicties for heart disease diagnose. The results are shown in Table 16
RAG source
Heart Disease bAce
Heart Disease F1
MedlinePlus (Default)
77.8
72.3
ACC/AHA Guidelines
77.2
71.5
Table 16: The impact of different RAG knowledge base.
It can be seen that using ACC/AHA guidelines the knowledge base for RAG can slightly improve the overall performance, further demonstrating that MedAgent-Pris is a flexible framework, and can improve performance with more professional knowledge base tools.
We attribute this stability to two factors: (1) The structured workflow and tool-guided reasoning play dominant role in diagnosis quality: (2) Given that current VLMS have been trained knowledge bases with medical knowledge like Wikipedia, they are often able to generate reasonable diagnostic plans with intrinsic knowledge even when the retrieved evidence slightly varies.
Noted that incorporating hospital-specifie guideline or private RAG sources is feasible within our framework, such customization further enhances the clinical relevance of MedAgent-Pro.
B.3 ABLATION ON DIFFERENT TOOLSET
We have further evaluated the performance on toolset variety, including visual analysis models, coding modules, and qualitative reasoning models (qualitative models were not used the main text due to their relatively limited improvement and strong domain specificity).
Setting
Glaucoma
Visual Tools
Coding Model
Qualitative Models
bAcc
F1
56.4
21.1
87.0
55.0
78.9
38.5
90.4
76.4
92.9
79.1
Table 17: Ablation key components with different combinations.
The results still support the robustness of our workflow design. Since the qualitative analysis tools (e.g., domain-spenn -peccffc VLMs) are hard to obtain for all diagnosis tasks and can be done by VLMs themselves, we do not include them in the workflow.
22

---

Published as a conference paper at ICLR 2026
B.4 ABLATION ON TOOL ACCESSIBILITY
Meanwhile, we further demonstrate MedAgent-Prye effectiveness even without access to any ex- ternal tools, using only retrieved knowledge and structured reasoning. In cases in the NEJM dataset where no external tools are accessible, the results are shown below:
Method
Acc
VLM-based Methods
GPT-4o
74.7
Janus-Pro-7B
29.6
LLaVA-Med
25.2
BioMedClip
27.3
Qwen2.5-7B-VL VL
42.9
InternVL2.5-BB
44.0
Agentic Systems
MedAgents
69.3
MMedAgent 75.3
MDAgent
77.5
MedAgent-Pro (Ours)
84.0
Table 18: Comparison of methods on the NEJM cases where no suitable visual tools are available.
As shown in the table, method still achieves the best performance, highlighting that the reasoning structure of MedAgent-Pa is capable of leveraging tool outputs when available, and maintaining strong performance even when tool access limited due to the system's planning capability and integration retrieved knowledge.
B.5 ANALYSIS OF DECISION-MAKING STRATEGIES.
We further explore alternative decision decision-making making strategies in the risk- decision stage. addi- tion to our proposed structured fusion, which assigns risk-based weights to clinical indicators, we evaluate flat fusion, where all raw indicators are directly fed into the VLM As illustrated in Fig. X structured fusion consistently outperforms flat fusion across varying indicator counts, leading to more balanced and comprehensive decisions, whereas VLMs often focus on partial cues.
Structure Fusion bAcc
75
Flat Fusion bÂcc
Structure Fusion F1
85
Flat Fusion F1
70
F1 Score (%))
bAcc (%)
80
80
55
50
70
45
The number of indicators
Figure 8: Comparison of kinds of decision-making ways on Glaucoma diagnosis.
23

---

Published as a conference paper at ICLR 2026
C OTHER DETAILED ANALYSIS
C.I STUDY ON COMPUTATION COST
Regarding computational cost, our workflow is actually efficient: for each disease, the planning stage is executed only once across all patients, generating reusable diagnostic code and workflows. Below, we report the average inference time per case (evaluated on an A6000 GPU):
Method
time (s)
VLM-based Methods
GPT-4o
21.2
Janus-Pro-7B
14.1
LLaVA-Med
1.5
Qwen2.5-7B-VL
5.1
InternVL2.555
10.7
Agentic Systems
MedAgents
143
MMedAgent
32
MDAgent
272
MedAgent-Pro (Ours)
44.6
Table 19: Study on computational cost on Glaucoma diagnosis.
As shown, MedAgent-Pa Pro achieves competitive speed compared to existing medical agentic sys- tems. The overhead from planning and code execution is minimal (typically under 1 second), and the majority of runtime is spent on the backbone VLM, similar to other methods.
C.2 DISCUSSION ON RELATION TO MEDICAL WORLD MODEL
Recently, the Medical World Model (Yang et al. 2025) has attracted attention for its ability to provide visual evidence through generative simulation for treatment planning. Here, discuss its relation to MedAgent-Pro and clarify the differences between the two approaches.
Regarding visual evidence, MedAgent- directly integrates visual envidence through specialized quantitative tools. For example, the glaucoma workflow uses segmentation models to obtain optic disc and cup region, and in the chest X-ray diagnosis, grounding tools can show the location of abnormalities like effusions.
Mcanwhile, McWM generates visual evidence through generation model that simulates disease progression under different conditions to support treatment decisions. Although the tasks and mech- anisms differ, both approaches incorporate visual evidence aligned with their respective objectives.
D BACKGROUND OF EVIDENCE-BASED MEDICINE
Our proposed MedAgent-Pro is inspired by the principle of evidence-based medicine in modern healthcare. Here, we provide a detailed introduction to this concept to help readers better understand the design rationale behind our agentic workflow.
Background of Evidence-based Medicine. Evidence-Based Medicine (EBM) defined as "the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients. It means integrating individual clinical expertise with the best available external clinical evidence from systematic research" (Guyatt al. 1992a). The goal of EBM is to integrate clinical expertise, patient values, and the best available scientific evidence to guide decisions related to clinical management. The term originally used to describe approach
24

---

Published as a conference paper at ICLR 2026
to teaching medical practice and improving individual physicians' decision decision-making for individual patients (Guyatt et al. 1992b). Adoption of evidence-basee medicine is necessary a human rights- based approach to public health and a precondition for accessing the right to health (Beracocheal et al. 2010).
The term "reiidene-bbaedd commissioned by the Council of Medical Specialty Societies to teach formal methods for designing clinical practice guidelines (Eddy) 1990d). In 1996, David Sackett and colleagues clarified the defi- nition of EBM as "the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients." This involves integrating individual clinical exper- tise with the best available external evidence from systematic research (Grobbee & Hoes| (014)) EBM aims to make individual clinical decisions more structured and objective by grounding them in research evidence. At the individual level, EBM is further defined as the process of finding, ap- praising, and applying contemporaneous research findings as the basis for medical decisions (Katz) 2001).
first used by David M. Eddy in 1987 in workshops and manual
Link with the proposed MedAgent-Pre Workflow. In the MedAgent-Pii workflow, we employ an RAG agent at the disease level to retrieve relevant medical guidelines, ensuring that clinical expertise is explicitly incorporated into the workflow. Meanwhile, we combine the medical guideline with the plans, which integrate evidence into the reasoning
available toolset to construct process meaningfully, rather than merely aggregating facts. This approach guides a step-by-step reasoning in clear and structured manner, aligning with the EBM concept.
At the patient level, the evidence-baal based reasoning process follows disease-specific plans, conducting both qualitative and quantitative analyses step by step enable comprehensive assessment and decision-making. Furthermore, each reasoning step is followed by a verification process to ensure that only clinically valid medical data is incorporated into the final decision. This design promotes the practical implementation of the EBM paradigm within the Al diagnostic workflow.
E IMPLEMENTATION DETAILS
E EXPERIMENTAL. SETUP
In this section, we provide additional experimental details that were omitted from the main text due to space constraints. Datasets. For glaucoma diagnosis,
the REFUGE2 dataset (Fang al. 2022 which is 2D retinal fundus image dataset containing 1200 RGB images at a resolution of 2124 2056 annotated by experts. Each image includes segmentation masks for the optic disc and optic cup, along with a classification label indicating whether the patient has glaucoma.
For heart disease diagnosis, we utilize the MITEA dataset (Zhao et al. 2023). 3D echocardiogra- phy dataset comprising 536 images from 143 human subjects. Each image contains segmentation masks for the left ventricular myocardium and cavity, along with classification label identifying the patient's 's heart condition across seven categories: healthy, aortic regurgitation, dilated cardiomy- opathy, amyloidosis, hypertrophic cardiomyopathy. hypertrophy, and transplant. Due the limited number of samples for each specific heart condition (some may only have 2 or 3 cases), we simplify the task to a binary classification: determining whether the patient has heart disease.
For chest X-ray diagnosis, we use the MIMIC dataset (Johnson et al. 2023), which includes 2D chest X-rays labeled with 12 potential abnormalities, such as Atelectasis, Cardiomegaly, Edema, Enlarged Cardiomediastinum. Fracture, Lung Leision, Lung Opacity, Pleural Effusion, Pneumonia, Pneumothorax and Supporting Devices. We sample 442 chest X-ray cases from 100 patients due to the large scale of the original dataset.
Meanwhile, we further employ New England Journal of Medicine (NEJM) database (of Medicine), where we compile 992 real-world diagnostic cases, encompassing over 10 anatomical regions, 10 imaging modalities, and 50 diseases. We present several examples here to provide a more intuitive understanding below:
Evaluation Metrics. We make a detailed description of the metrics we used in the main paper: The TN bAce is defined } (TP4FN + TN+FP). the F1 score is calculated as 2 Precision x Recal! Precision+ Recall
25

---

Published as a conference paper at ICLR 2026
Dealing with MCQ questions Clinicians in practice typically narrow down to small set of sus- pected conditions before further differentiating among them Falik] (2006). MedAgent- generally follows this realistic workflow. In datasets like NEJM, questions are given in the format of multi- choice questions (MCQ). MedAgent-Pro constructs a diagnostic plan for each candidate option, per- forms analysis for each disease separately, and then lets the VLM integrate the resulting evidence and indicators to choose the most plausible answer.
E.2 GENERATION OF NOISY LABELS
In Section 4.3 of the main text, we discuss how inaceurate segmentation results can affect the final diagnostic accuracy. Due to space constraints, provide detailed description of the noise label generation process here.
To simulate noisy annotations, we implement three types of label perturbations--illanee
erosion, and affine transformation-10 simulate typical deviations found in manual annotations following (Xue et al. 2020; Zhang et al. 2020a Zhu et al., 2019). For each training sample, one noise type is randomly applied with a noise intensity controlled by the parameter 3 E (0, 1), which represents the proportion of desired label distortion relative to the original foreground area.
Dilation-based Noise. Given a binary label
E {0, [0,1) we define the foreground area as
[1] = Σ L. The target noisy label satisfies:
L
| + B-
Iterative binary dilation applied until the foreground area reaches (1 + (--3)-11 chosen to be the one closest this target ratio while avoiding overflow beyond the image boundaries.
-T]. The final mask is
Erosion-based Noise. Erosion is used to simulate under-segmmenaatio.. The goal is to reduce the foreground such that:
[ (1- 3)- [Z]
Binary erosion is applied iteratively until the foreground shrinks to the target level, again selecting the mask closest to the target size.
Affine Transformation-base Noise. This simulates geometric misalignments (e.g., due to incon- sistent viewing windows). The mask is rotated by a random angle E [-0max: max] and then translated by an adaptive step along randomly chosen direction. The amount of introduced noise quantified as:
L + L
NoiseRate
[1]
where @ denotes the symmetric difference between the original and transformed masks. A binary search is used to find the smallest step such that NoiseRate se 8, allowing fine-grained control over the perturbation strength.
The parameter 3 directly determines the target deviation ratio between the noisy and original mask, effectively simulating varying degrees of annotation error. ensures that the noise proportional the original label size, making the corruption * **scleeaawre and clinically realistic. higher simulates more aggressive annotation noise. Below are examples of some noisy labels:
E.3 CRITERIA FOR HUMAN EVALUATION
The clinicians rate the diagnostic outputs from both VLMs and MedAgent-Prr across five dimen- relevane,, comprehensiveness, clinical reliability, reasoning coherence, and language clar- ity-using to scale following previous standards.
Relevance (Content Relevance). Relevance evaluates how accurately the model's response reflects the content and clinical significance of the input image and question (Can et al. 2025 Tam et al.) 2024). A relevant response should focus visible anatomical or clinical features and stay aligned with the diagnostic task, avoiding unrelated or off-topic information. The assessment considers whether the response directly references key image findings, maintains contextual focus, and ex- cludes irrelevant details. Responses are rated on a scale from to 5, with 5 indicating fully relevant and mngeeggo observations, and I indicating largely irrelevant off-topic reply.
26

---

Published as a conference paper at ICLR 2026
Toxoplasmosis
Lisch nodules
Preeclampsia
X-linked retinoschisis
Wilson's disease
Pulmonary Sarcoidosis
Paragonimiasis
Enterobiasis
Praziquantel
bronchogram
Dexamethasone
Oleothorax
Lung herniation
Bilirubin
Histoplasmosis
Beta-thalassemia
Tobacco
Hemiplegia
Hydatid cyst
Epiglottitis
Rubella
Insect bite
Nitroglycerin
Contracture
Left main stem
Melanoma
Cell histiocytosis
Mural thrombus
Tinea faciei
Abscess
Arteritis
Lead poisoning
Reactive perforating collagenosis
Arsenic toxicity
Cardiac allograft
Bisphosphonates
Scabies
Frey's syndrome
Gardening
Figure 9: Example diagnostic cases from the NEJM dataset.
27

---

Published as a conference paper at ICLR 2026
Original Image
Ground Truth
Dilation Noise
Erosion Noise
Affine Noise
Figure 10: Example of noisy labels with different noise types
Comprehensiveness (Coverage of Factors). Comprehensiveness assesses whether the model's response takes into account all pertinent clinical features necessary for well-rounded and accurate diagnosis (Hershenhouse et al. 2024; (Singhal et al.) 2023). It evaluates the inclusion of a broad range of relevant factors, such as differential diagnoses, comorbidities, and key clinical indicators. A comprehensive response should reflect the depth and balance expected from human expert, without overemphasizing a single feature. Scores range from 5, with indicating a thorough and balanced discussion of all clinically relevant aspects, and reflecting shallow response lacking diagnostic depth.
Clinical Reliability (Expertise). Clinical reliability evaluates the extent to which the response aligns with medical standards, evidence-based medicine practices, and expert clinical understanding Fast et al. 2024). A correct answer should be accurate, appropriate terminology, and reflect reasoning that would be trusted a clinical context. The response is assessed based on its consis- tency with established guidelines and expert consensus. Scores range from I to 5, where 5 denotes expert-level accuracy and full alignment with clinical standards, and indicates clinically incorrect misleading content.
Reasoning Coherence (Logical Consistency). Reasoning Coherence evaluates whether the model's diagnostic output follows a coherent and logical clinical reasoning process (Hershenhouse) et al., 2024; Singhal et al.) (202).. A correct conclusion alone is insufficient-whal matters is that the explanation demonstrates sound medical logic, rather than guesswork unsupported assump- tions. This criterion considers whether diagnostic steps are appropriate, clinical inferences are well- justified, and the overall reasoning pathway traceable and explainable. Scores range from 1 5, with 5 indicating fully logical, medically justified reasoning, and 1 reflecting illogical or funda- mentally flawed diagnostic process.
Language Clarity. Language clarity gauges how effectively the model communicates its reasoning and conclusions (Can et al.) 2025) Tam et al.) 20244 Hershenhouse et al., 2024). The explanation should use professional, clinically appropriate language that is also clear and accessible to the in- tended audience. This metric evaluates whether the language precise and unambiguous, medical terminology is correctly used, grammar and structure sound, and whether professional reader can easily follow the rationale without confusion. Scores range from to 5, where denotes crystal- clear and professional expression, and reflects poor language quality that hinders understanding.
28

---

Published as a conference paper at ICLR 2026
Evaluation Details Two licensed clinicians participated in the study: one ophthalmologist eval- uated the glaucoma cases, and one thoracic surgeon evaluated the chest X-ray cases. Each rater assessed only cases within their clinical specialty, ensuring
judgment.
To mitigate anchoring or exposure effects, we used a randomized and blinded evaluation protocol. For each task, all model outputs (MedAgent-Pro and comparision methods) were anonymized, and both the case order and the ordering of outputs for each case were randomly shuffled for the clinician. This prevents bias arising from fixed sequencing or repeated exposure.
E.4 DETAILS OF EVIDENCE-BASED REFLECTION
For completeness, we provide a more detailed description of the Evidence- Reflection mech- anism used in patient-level reasoning. The state-assessment function & is a zero-shot evaluator implemented by the VLM without any training. Specifically, discussed in Section 3.3, each exe- cuted step P produces a result T3 from the imput object oG, which can be (i) a visual model output (e.g., segmentation mask or bounding boxes), (ii) numerical result (e.g., the cup-to-dise ratio), or (iii) a qualitative analysis. The function & evaluates the reliability of the result from the following aspects:
(1) Input-quality assessment. The reflection prompt explicitly asks the VLM to judge the quality of the input image and the feasibility of performing the specified action (e.g., "segment optic disc" or "assess pleural effusion")). The VLM evaluates whether the image is blurry, occluded, cropped, low-resolution, or missing the relevant anatomical region.
(2) Visual model output assessment. If r; is the output of visual tool, the VLM is prompted to assess whether the predicted region anatomically plausible. The VLM examines (i) whether the mask or bounding box has reasonable size (e.g., masks or boxes that cover nearly the entire image are clearly incorrect), (ii) whether the shape is reasonable (e.g., the mask contains large holes or appears fragmented), and (iii) whether the prediction is consistent with the clinical guideline (e.g.. "the cup region should lie inside the disc region"))
(3) Numerical-result assessment. When Ti is a numerical value (e.g.. vCDR ratio or LVEF esti- mate), checks whether the value lies within a physiologically valid range based on the guideline G' (e.g.. the vCDR value should be less than 1).
Qualitative textual assessment. When r; is a qualitative statement produced by the VLM or grounding tool (e.g.. "mild opacity in the right lower lobe"), the reflection prompt requires that such judgments be made only when clear visual or textual clues are present. If the VLM expresses uncertainty (e.g.. "the abnormality is not clearly visible"), the result is considered unreliable.
We do not train this module for two reasons: (I) annotated data for determining the reliability of each result is difficult to obtain, and (2) training such component would undermine the zero-shot fairness of our comparisons. Our goal is to provide a flexible workflow that can adapt to a wide range of clinical scenarios. Despite being zero-shot, the ablation studies in the main paper demonstrate that the carefully designed evidence-based reflection mechanism is effective.
Upper Bound in Evidence-based Reasoning During runtime in the letient-level wridence-based reasoning, we set a strict upper limit on the reasoning rounds 10. When this limit is reached, the corresponding branch is terminated and excluded from the final dociisionmmakigg to ensure stability.
E.5 DETAILS OF THE TOOLSET
We have added the detailed description of the tools used in MedAgent-Pro as below, As shown in the table, only the glaucoma and heart-disease datasets provide ground-truth masks, allowing us to finetuning model quantitatively evaluate the performance of the tools. All other tools used in a zero-shot manner, please refer to their original papers for their general performance, as these tasks lack annotation for quantitative assessment in our tasks.
29

---

Published as a conference paper at ICLR 2026
Usage
Task
Tool Model
Traineed
Glaucoma
Medical SAM-Adapter
No
Heart Discase
Medical SAM-Adapter
Yes
MIMIC- -XX
MedSAM
No
Segmentation
NEJM Chest X-ray
MedSAM
No
NEJM CT MRI
MedSAM
NEJM Cell Imaging
Cellpose
No
Grounding
MIMIC-CXR
Maira-2
No
Coding Utiliyy Tools
All tasks
GitHub Copilot (code execution / parsing)
No
Table 20: The details of the toolset used in MedAgent-Pro.
F ADDITIONAL QUALITATIVE ANALYSIS
Due to space limitations in the main text, we present additional comparative experiments here, in- cluding results for both general VLMs and MedAgent- Pro glaucoma diagnosis and chest X-ray diagnosis tasks. It is worth noting that we omit overly lengthy reasoning steps and redundant or meaningless repetitions from the presented content.
The comparison shows that, unlike general VLMs, the proposed MedAgent-Pii not only produces accurate conclusions but also demonstrates more comprehensive reasoning process, according to its integration of an RAG agent to generate diagnostic plans based on medical guidelines. Mean- while, quantitative analysis grounded in evidence-based reasoning further enhances the reliability and trustworthiness of the final diagnosis.
30

---

Published as a conference paper at ICLR 2026
Original Image
The fundus image shows possible signs of glauromatous optic nerve damage, most notably increased optic cup and possible thinning. However, this image alone is not sufficient to definitively diagnese glaucoma, and further clinical evaluation and diagnostic testing are necessary.
This fundus image shows the posterior segment of the eye, including the optic disc, the macula, and retinal blood vessels. assess glaucorna, we need to consider several features, especially related to the optic nerve head (optic disc appearance).....
GPT-40
This patient may have signs suggestive of glaucoma based on: A. large cup-to disc ratio. Thinning of the neuroretinal rim, particularly inferiorly.
Recommendation: Refer to ophthalmologit for comprehensive glaucama workup, including IOP measurement, visual field testing, and OCT imaging.
The image provided is a fundus photograph, which a detailed image of the retina the back of the eye. Here are the key observations:
The image provided is a fundus photograph, which a detailed image of the retina at the back of the eye. Here are the key observations:
The bright circular area on the left side of the image optic disc. where optic nerve
1. Optic Disc:
1. Normal Retina: The majority of the image shows a normal retina with reddish-brown background and visible blaod vessels. 2. Lesion or Abnormality: There is noticeable bright yellow
eve. Blood Vessels: Radiating from the optic disc are several retinal blood vessels. These vessels appear as fine, red lines. Retinal Layers: image shows the layered structure of the retina, with the photoreceptor layer the bottom and the ganglion cell layer the top.
Janus-Pro-78
the left side the image, which appears be abnormal spot lesion.3. Blood Vessels: The retinal blaed vessels are visible and appear to normal and pattern.
The overall appearance of the retina seems normal, with no obvious signs of disease or abnormalities.
General Health:
General Health:
The overall health the retina appears to be normal, with na signs of retinal detachment or other significant abnormalities.
According fundus image, patient does have glaucoma. The image shows a normal optic nerve, which is important indicator
According the fundus image, patient does not have glaucoma, The image shows a normal optic nerve, which is important indicator the absence glaucoma.
LLaVA-Med
absence of glaucoma.
The provided fundus image shows a retinal view of an does not pravide enough detail definitively diagnose glaucoma. Here are some key points to consider:
The provided fundus image shows a retinal view of an eye, but does not provide enough detail to definitively diagnose glaucoma. Here's why:
Optic Nerve Head (ONHi): The optic nerve head appears to be within normal limits terms of color and cup-to-disc ratio. However, this is just one espect of glaucoma diagnosis.
1. Optic Nerve Head (ONH) Appearance: The optic nerve head this image appears relatively normal without any visible signt of cupping or pallor that might suggest glaucomatous damage. However, definitive astessment the optic nerve requires stereoscopic view and comparison with the fellow
Qwen2.5-7B-V
Retinal Blood Vessels:The retinal blood vessels appear to be normal in caliber and distribution. There are signs significant narrowing dilation that would suggest vascular changes associated with glaucoma.
2. Retinal Blood Vessels: The retinal blood vessels appear to be within normal limits, neither excessively dilated nor constricted
To determine the patient has glaunoma based the fundus image, we need to examine the optic disc, the optic and the surrounding retinal nerve fiber layer. Glaucoma typically associated with optic disc changes, such as increased cup-to-dist disc ratio, rim thinning, and/or the presence a rim excavation.
To determine whether the patient has glaucoma based the fundus image, we need look for signs commonly essociated with this condition. Here are some key features consider....
Based on the visible features are no avert signs of glaucoma such as increased cupping, optic disc changes, nerve fiber layer loss. However, definitive diagnosis of glaucoma requires comprehensive eva lustion.. including intraocular pressure measurement, optic nerve head assessment, and visual testing. conclusion, the fundus image does not provide sufficient evidence diagnose glaucoma. comprehensive eye examination necessary for an accurate disgnosis.
fundus Image, there
InternVL-2.5-88
Based image provided, there are obvious signs of glaucoma. The optic disc and optic cup appear normal, the RN thick continuous, and there peripapillary atrophy or blood vessel changes. However, a definitive diagnosis should be made by healthcare professional based comprehensive examination and additional clinical information.
visible
According to relevant documents, the key indicators diagnasing glaucoma from fundus images are as follows:
According to relevant documents, the key indicators for diagnasing glauroma from fundus Images follows:
According to the segmentation result of optic cup and disc, the the risk thresold O.6.
According to the segmentation result optic cup disc, the vCDR 0.699, higher than the risk thresold 0.6.
0.612, higher than
The thickness of the patient's neuroretinal rim devintes from the normal ISNT rule, with a much thinner temporal margin.
The thickness of the patient's neuroretinal rim deviates from the normal ISNT rule, with a much thinner temporal margin.
MedAgent-Pro
The patient have optic disc hemorrhage with associated cup to disd and retinal pigmentation.
The patient have optic disc hemorrhage with associated cup-to-disc and retinal pigmentation.
The image shows an clear optic disc boundary, which related peripapillary atrophy.
The image shows clear optic disc boundary, which related to peripapillary atrophy.
Based on above findings, the patient is very like to have glaucoma.
Based an above findings, the patient is very like to have glaucoma.
Figure 11: Qualitative comparison on glaucoma diagnosis with general VLMs. Green texts highlight the correct conclusion made by MedAgent-PPi
31

---

Published as a conference paper at ICLR 2026
Original Image
Cardiomegaly, enlargement of the heart, is typically assessed on posterior anterior (PA) chest ray by measuring the cardiothoracie ratio (CTR) - the ratio of the cardiac width the thoracic width. CTR greater than 50% on a PA view is generally considered indicative of cardiomegaly. The heart size in the second image appears within limits. width
This chest X-ray shows complex clinical picture with multiple devices and findings, there does not appear to be an obvious acute bony fracture visible in the scanned region, including the clavicles, ribs, spine the level shown.
GPT-4o
less half the width of the thoracic cavity, suggesting normal cardiethoracic ratio. The patient does not appear have cardiomegaly.
No abvious fracture visible on this image, but further
clinical correlation and passibly advanced imaging
recommended to confirm or exclude any subtle fractures.
Based on the given images, it not possible for me to definitively determine if the patient has cardiomegely. Cardiomegaly typically assessed by comparing the size the heart to the thorackc cavitty on an ray, but these images only show the heart's location and size relative the
Based on the provided X-ray image, there is no apparent fracture visible in the chest area. The bories appear to be and aligned correctly, with visible signs of breakage or dislocation.
Janus-Pro-7B
spine. Additional imaging clinical evaluation would be needed to make a conclusive diagnosis.
Yes, the chest X-ray shows cardiomegely, which an enlargement of the heart.
Yes, the patient appears to have Fracture, as indicated by the chest X-ray.
LLaVA-Med
Based on the provided chest X-rays, it is not possible to definitively diagnose cardiomegaly /enlargement of the heart) without a detalled radiological Interpretation by qualified radiolagist. However, there are some observations that be made:
The image you provided is chest X ray, which is used to visualize the structures of the chest, including the hones, lungs, and heart. obvious signs of fracture. The ribs appear intact, and there visible fractures or breaks the bony structures.
particular X ray, there are no
The heart appears to relatively large compared to the thoracic cavity.
Qwen2.5-78-VL
However, it's important to note that a single X-ray may not always sufficient to diagnase fracture, especially if the fracture is subtle or the patient has had recent traums. If there suspicion of fracture, further imaging such as CT might necessary for
There may blunting of costophrenic angles, which could suggest pleural effusion or other conditions.
detailed evaluation.
The image shows an X-ray of the chest. To determine if the would typically look for
To determine whether the patient has glaucoma based on the fundus image, we need to look for signs commonly essociated to consider.....
patient has cardiomegaly, enlarged cardiac silhouctte, which may indicate an abnormal increase in heart size.
condition.
some key features
from the image, the heart appears to be enlarged relative thoracic cavity. This suggests that the patient may have cardiomegaly. However, a definitive diagnosis should by medical professional, additional information and context are necessary for an accurate assessment.
Besed the visible features in this fundus image, there are overt signs of glaucoma optic disc changes, or nerve fiber layer loss. However, a definitive diagnasis of glaucoma requires comprehensive evaluation, indluding intrancular pressure measurement, optic nerve head assessment, and visual field testing. In conclusion, fundus image does provide sufficient evidence to diagnose glaucoma.. comprehensive eye examination is necessary for an accurate diagnosis.
as increased cupping.
InternVL-2.5-88
Cardiomegaly, or enlargement the heart, typically assessed on posterior anterior (PA) chest X ray by measuring the cardiathoradic ratio (CTR). CTR ratio the cardiac width to the thoracic width, A CTR greater than 50% PA generally considered indicative of cardiomegall..
Common radiographic findings of fractures in the thoracic vertebrac, ribs, clavicles, and sternmm include: Cortical break: Disruption or alteration of the continulty or sharpness the cortical bone lineAngulation displacement: Loral angulation or misalignment of hone fragments. Overlapping bone fragments, abnormal changes in bone density, such as thickening, thinning, or irreullrrity the cortex.
The heart located the image below, the red box indicates the heart boundary.
MedAgent-Pro
The thoracic vertebrae located in the image. Juding from the results, the thoracic vertebrae appears reletively normal.
The thoracic cavity located the image below, the box indicates thoracic boundary.
The ribs located the image below. Juding from the results, old left-sided rib fractures are present in the image.
According the result, the CTR is colculated the ratio the maximum horizontal cardiac diameter the maximum horizontal thoracic diameter, which 0.59 in this case. Judging from the above information, the patient is likely to have stelectasis.
The clavicles are located in the image below. Juding from the results, the clavicles appears relatively normal.
Judiging from the ahove information, the patient likely to have fracture.
Qualitative comparison on chest X-ray diagnosis with general VLMs. Green texts high- light the correct conclusion made by MedAgent--Pae
Figure 12:
32


Reviews:
[{"review_id": "lTZfQMEp9z", "reviewer": "Reviewer_roDJ", "summary": "This paper introduces MedAgent-Pro, an agentic hierarchical workflow system for evidence-based, multi-modal medical diagnosis. The approach separates disease-level planning (standardized per guidelines using a retrieval-augmented generation agent) and patient-level stepwise reasoning empowered by professional tools (e.g., visual models, coding agents), integrating iterative reflection for reliability. MedAgent-Pro is benchmarked on extensive multi-modal datasets, showing performance gains over mainstream VLMs, agentic baselines, and expert models.", "questions": "see Weaknesses", "limitations": "", "rating": 2, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The hierarchical design effectively mirrors medical diagnostic principles, with disease-level planning driven by retrieval from guidelines and patient-level personalized analysis.\n2. The RAG-based incorporation of external domain knowledge into the planning stage supports clinical transparency.", "weaknesses": "1. Although the hierarchical structuring and rigorous evidence focus are valuable, the architecture closely resembles recent work in multi-agent, RAG-augmented, and tool-integrated medical AI (e.g., MedAgents, MMedAgent, MDAgent, and others in Table 1). Moreover, the overall contribution of this work appears rather limited. Many of the claimed innovations rely on techniques that have already become widely adopted or established in the field. As such, the paper seems more like an integration or system-level implementation of several existing popular paradigms, rather than presenting a novel method.\n\n2. Can the authors clarify and mathematically specify the state assessment function $\\phi$ used during patient-level reasoning? How is output reliability judged, is it a learned model, rule-based, or probabilistically estimated?\n\n3. How are risk-based weights $\\mathcal{W}$ in risk score computation (p.5) defined in practice? Are they derived from clinical guidelines directly, learned via validation, or optimized by the system?\n\n4. For clinical expert evaluation, please elaborate on the number of raters, the randomization protocol, inter-rater agreement metrics (such as Cohen’s kappa or ICC), and whether any anchoring or exposure effects could have occurred.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LdkyammdrX", "reviewer": "Reviewer_nJkL", "summary": "The paper introduces MedAgent-Pro, an agentic reasoning workflow designed to emulate evidence-based clinical diagnosis. The system operates on a hierarchical structure, using planning, RAG and reasoning agents to ensure reliability.", "questions": "Please see the weaknesses above", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The primary strength of this work is its alignment with medical practice, from an empirical one-hop VQA to a structured and evidence-based reasoning process.", "weaknesses": "My major comments are:\n\n1. The system's effectiveness, particularly its quantitative analysis which drives the performance gains, relies on the availability of specialized visual tools for a given task. The workflow's performance for diseases or modalities lacking such robust, pre-existing tools is less clear. While the NEJM results without tools are still strong, they lack the key quantitative grounding that differentiates the method.\n2. The VLM  remains a single point of failure for several critical steps. The VLM is solely responsible for generating the entire diagnostic plan from guidelines, performing all qualitative analysis, executing the evidence-based reflection, and assigning the final risk-based weights. Any hallucination or error in these steps could compromise the entire workflow.\n3. The main comparison tables compare MedAgent-Pro (with tools) against general VLMs (without tools). This confounds the benefit of the agentic workflow with the benefit of simply having tool access. A more rigorous comparison, presented only in Appendix B.4, shows that while MedAgent-Pro still wins, giving baselines tool access does significantly close the gap. This stronger, more fair comparison should have been centered in the main paper.\n4. The evaluation on the MITEA dataset (heart disease) was simplified from a 7-class problem to a binary classification task (healthy vs. heart disease) due to limited samples per category. This simplification is a significant departure from a realistic clinical scenario, where differentiating between various heart conditions (e.g., amyloidosis vs. hypertrophy) is the critical diagnostic task.\n5. The RAG agent's knowledge base is MedlinePlus. However, clinical-grade diagnosis typically relies on more complex professional guidelines from medical societies (e.g., ACC/AHA, etc.).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "35Ng4qA7dW", "reviewer": "Reviewer_HerW", "summary": "This work explores reasoning agentic system of medical diagnosis. By disease-level standardized plan generation and patient-level personalized step-by-step reasoning, MedAgent-Pro presents modern diagnosis principles via a hierarchical diagnostic workflow. They leverages professional tools such as visual models to take various actions to analyze multi-modal input, and performs evidence-based reflection. Extensive experiments across a wide range of anatomical regions, imaging modalities, and diseases are conducted.", "questions": "Please see weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tEvaluations across 10+ imaging modalities, 20+ anatomies, and 50+ diseases validate its universality.\n2.\tThe proposed system fully leverages existing models in their toolbox to provide clinical evidence.\n3.\tThe proposed model is trustworthy because of more visual cues and indicators compared to vanilla VLMs.\n4.\tComprehensive ablations further validate the effectiveness of major components in the agentic system.", "weaknesses": "1.\tThe author proposed evidence-based system. However, the proposed system is more like an engineering product, which lack of architecture innovation and similar to other agentic systems[1,2].\n2.\tThe author should provide analysis in view of visual cues compared to medical world model[3], which was also designed for treatment planning. Is the proposed system providing any visual evidence?\n3.\tWhich part of MedAgent-pro is finetuned? Is the backbone GPT-4o finetuned on downstream data? What about the compared methods? The fairness should be ensured.\n4.\tIn Evidence-based Reflection, is it possible that s_i is always continue, resulting in unexpected dead loop?\n5.\tClinical guidelines may be inconsistent across different areas and hospitals. How can the proposed system overcome such inconsistency?\n\n[1] Zhu Y, He Z, Hu H, et al. MedAgentBoard: Benchmarking Multi-Agent Collaboration with Conventional Methods for Diverse Medical Tasks[J]. arXiv preprint arXiv:2505.12371, 2025.\n[2] Zhu Y, Qi Y, Wang Z, et al. HealthFlow: A Self-Evolving AI Agent with Meta Planning for Autonomous Healthcare Research[J]. arXiv preprint arXiv:2508.02621, 2025.\n[3] Yang Y, Wang Z Y, Liu Q, et al. Medical world model: Generative simulation of tumor evolution for treatment planning[J]. arXiv preprint arXiv:2506.02327, 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xtLHv3rLAA", "reviewer": "Reviewer_ACbS", "summary": "This paper introduces **MedAgent-Pro**, an agent-based system designed for multi-modal medical diagnosis. The authors identify a key limitation in current Vision-Language Models (VLMs) and agentic systems: they tend to provide direct, empirical-driven answers without the step-by-step, evidence-based reasoning that is fundamental to real-world clinical practice.\n\nTo address this gap, MedAgent-Pro proposes a hierarchical workflow that mimics clinical procedure. This workflow is divided into two main stages: **(i)** Disease-Level Planning: A Retrieval-Augmented Generation (RAG) agent accesses a medical knowledge base to formulate a standardized, multi-step diagnostic plan based on established clinical guidelines. **(ii)** Patient-Level Reasoning: The agent executes this plan step-by-step, analyzing the specific patient's multi-modal data. This execution phase leverages a toolbox of \"professional tools,\" such as segmentation models, to perform quantitative analysis (e.g., calculating the cup-to-disc ratio) alongside qualitative assessments.\n\nThe authors present an extensive evaluation across more than 10 imaging modalities and 50 diseases, claiming that MedAgent-Pro substantially outperforms general VLMs like GPT-4o, other medical agent systems, and task-specific expert models. The system's alignment with clinical workflows is further supported by positive evaluations from clinical experts. Overall, this is a fairly strong paper.", "questions": "See \"Weaknesses\" for questions. **Note** that, if the related works section and other problems in Weaknesses are not addressed the rating can be subject to deduction.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "### **Originality and Significance:** \nThe most significant strength is the novel agentic paradigm itself. By formalizing the clinical diagnostic process into a hierarchical workflow (plan-via-RAG, execute-via-tools), the paper moves the field from \"black-box\" pattern recognition towards auditable, evidence-based reasoning. This directly addresses the critical needs of reliability, safety, and trustworthiness for medical AI.   \n\n\n### **Problem Formulation:** \nThe paper does an excellent job of motivating the work, clearly articulating the gap between current VLM capabilities (which provide \"empirical-driven conclusions\") and the needs of clinical practice (which demands \"quantitative analysis\" and structured reasoning).   \n\n\n### **Experimental Breadth:**\nThe evaluation is a major strength. The authors test their system on a very wide array of tasks, covering over 10 imaging modalities, 20+ anatomies, and 50+ diseases, providing strong evidence for the generalizability of the framework.   \n\n\n### **Clinician Evaluation:** \nThe inclusion of a qualitative evaluation by clinical experts is a crucial addition. Reporting that clinicians rated MedAgent-Pro's outputs higher on dimensions like \"reasoning coherence\" and \"clinical reliability\" provides strong, practical validation of the paper's central thesis.", "weaknesses": "### **Inadequate Contextualization with Prior Work:** \n\nThe paper claims that current medical agentic systems \"simply glued all tools together\". This may be an oversimplification that ignores recent, highly relevant work. The authors should cite and differentiate their approach from other multi-modal medical agents like \"SMR-agents\" (Wang et al., [1]) and \"AURA\" (Fathi et al., [2]), which also appear to integrate multi-modal reasoning and tool use in a sophisticated manner. A more nuanced comparison in the Related Work section is needed to clearly establish this paper's specific contributions over the state-of-the-art.\n\n### **The \"ToolBox\" is Underspecified:**\nThe system's performance is critically dependent on its \"Available ToolBox\". The paper mentions \"visual/coding models\" and gives MedSAM as an example, but a comprehensive list of all tools, their specific capabilities, and their individual performance is not provided. Without this, it's impossible to assess the agent workflow independently of the tools. \n\n### **\"Evidence-Based Reflection\" is Unclear:**\n This mechanism is highlighted as a key contribution but is operationally vague. How does the agent \"evaluate the reliability\" of a step's output? Is this a heuristic, a learned classifier, or a VLM self-critique prompt? The \"Evi. Reasoning\" block in Figure 2 is abstract. This core mechanism needs to be formalized.\n\n\n### **Conceptual Limitation to Sequential Reasoning:** \nThe proposed methodology is inherently bounded to sequential reasoning steps. The RAG agent generates a linear plan, and the execution agent follows it. The paper does not discuss or explore scenarios that might require parallel exploration of different diagnostic paths or more complex, branching decision-making. This is a notable limitation, as real-world diagnostics can often be non-linear.\n\n### **Confusing Mathematical Notation:** \nThe mathematical expressions in Section 3.2, such as representing inputs and outputs as key-value pairs (e.g., $\\mathcal{P}i:r_{i}=\\langle k_{r_{i}},v_{r_{i}}\\rangle$), add a layer of formalism that is arguably unnecessary and confusing. A-plainer textual description of how data is passed and transformed between steps would significantly improve clarity and accessibility.\n\n### **RAG Knowledge Base:** \nThe paper relies on a \"Medical Knowledge Base\" sourced from MedlinePlus. The scale, and curation of this knowledge base, as well as the indexing and retrieval process, are mentioned but could be detailed further. The quality of the RAG-generated plan is a critical bottleneck that is not explicitly evaluated.\n\n[1] Wang et al., \"SMR-agents: Synergistic medical reasoning agents for zero-shot medical visual question answering with MLLMs\"\n\n[2] Fathi et al., \"AURA: A Multi-modal Medical Agent for Understanding, Reasoning and Annotation\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)