CARE: Towards Clinical Accountability in Multi-Modal Medical Reasoning with an Evidence-Grounded Agentic Framework
Authors: Yuexi Du
Jinglu Wang
Shujie LIU
Nicha C Dvornek
Yan Lu
Venue: ICLR 2026 Poster (2026)
Keywords: Multi-modal Large Language Agent
Medical Visual Question Answering
Visually Grounded Reasoning
Reinforcement Learning with Verifiable Reward
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: CARE is an agentic, evidence-grounded Med-VLM that proposes entities, segments them, then reasons with selected visual clues—achieving state-of-the-art accuracy with auditable, clinically aligned answers.

Abstract: Large visual language models (VLMs) have shown strong multi-modal medical reasoning ability, but most operate as end-to-end black boxes, diverging from clinicians’ evidence-based, staged workflows and hindering clinical accountability. Complementarily, expert visual grounding models can accurately localize regions of interest (ROIs), providing explicit, reliable evidence that improves both reasoning accuracy and trust. In this paper, we introduce **CARE**, advancing **C**linical **A**ccountability in multi-modal medical **R**easoning with an **E**vidence-grounded agentic framework. Unlike existing approaches that couple grounding and reasoning within a single generalist model, CARE decomposes the task into coordinated sub-modules to reduce shortcut learning and hallucination: a compact VLM proposes relevant medical entities; an expert entity-referring segmentation model produces pixel-level ROI evidence; and a grounded VLM reasons over the full image augmented by ROI hints. The VLMs are optimized with reinforcement learning with verifiable rewards to align answers with supporting evidence. Furthermore, a VLM coordinator plans tool invocation and reviews evidence-answer consistency, providing agentic control and final verification. Evaluated on standard medical VQA benchmarks, our **CARE-Flow** (coordinator-free) improves average accuracy by **10.9%** over the same size (10B) state-of-the-art (SOTA). With dynamic planning and answer review, our **CARE-Coord** yields a further gain, outperforming the heavily pre-trained SOTA by **5.2%**. Our experiments demonstrate that an agentic framework that emulates clinical workflows, incorporating decoupled specialized models and explicit evidence, yields more accurate and accountable medical AI.

Published as a conference paper at ICLR 2026
CARE: TOWARDS CLINICAL ACCOUNTABILITY IN MULTI-MODAL MEDICAL REASONING WITH AN EVIDENCE-GROUNDEI AGENTIC FRAMEWORK
Yuexi Du Jinglu Wang'! Shujie Liu', Nicha C. Dvornek2.5 Yan Lu
I Microsoft Research Asia
DDepartment of Biomedical Engineering, Yale University
'Department of Radiology & Biomedical Imaging, Yale University
(yuexi, du, nicha dvornek @yalc. edu, {jinglwa, yanlu) con
ABSTRACT
Large visual language models (VLMs) have shown strong multi-modal medical reasoning ability, but most operate as end-to-end black boxes, diverging from clinicians' evidence-based, staged workflows and hindering clinical accountability. Complementary this, expert visual grounding models can accurately localize regions of interest (ROIs), providing explicit, reliable evidence that improves both reasoning accuracy and trust. In this paper, we introduce CARE, advancing Clinical Accountability in multi-modal medical Reasoning with an Evidence- grounded agentic framework. Unlike existing approaches that couple grounding and reasoning within a single generalist model, CARE decomposes the task into coordinated submodules reduce shortcut learning and hallucination: compact VLM proposes relevant medical entities; an expert entity-referring segmentation model produces pixel-level ROI evidence; and grounded VLM reasons over the full image augmented by ROI hints. The VLMs are optimized with reinforcement learning with verifiable rewards to align answers with supporting evidence. Fur- thermore, a VLM coordinator plans tool invocation and reviews evidence-answer consistency. providing agentic control and final verification. Evaluated on standard medical VQA benchmarks, our CARE-Flow (coordinator- free) improves average accuracy by 10.9% over the same size ( (10B) state-of-the-art (SOTA). With dynamic planning and reviewing, the heavily trained SOTA by 5.2% Our experiments demonstrate that an agentic framework that emulates clinical workflows, incorporating decoupled specialists and explicit evidence, yields more accurate and accountable medical Al.
CARE-Coord yields further gain, outperforming
INTRODUCTION
Recent advances in visual language models (VLMs) have delivered strong results medical image understanding and diagnostic visual question answering (VQA) (He et al. [2024] Xu et al. 2025 Sellergren et al. 2025). However, most current methods (Dong et al. 2025 Hou et al. 2024]] Li et al. 2025) adopt monolithic, single-shot formulation that maps images and text directly answers without explicitly localizing or verifying the supporting visual findings. This design invites shortcut learning and hallucination, especially under distribution shift, fine-grained, case-relevant evidence neither retrieved nor required, as illustrated in Fig. (a). Unlike human clinical workflow, which localizes abnormalities, examines them at appropriate scales, and then decides based on explicit image evidence, such black-box inference undermines clinical reliability and accountability.
In response, some works augment VL.Ms with visual grounding (Wu et al. 2025 Huang al. 2025a Luo et al.] 2024; Zhu et al.) 2025b), but typically treat grounding as an isolated perception head whose outputs are fed back into reasoning for full use (Fig. (b)). In general-domain VQA, concurrent works (Fan et al. 2025 Zhang et al. 2023a Qi et al. 2024 interleave external image manipulations,
Work done during internship at Microsoft Research Asia.
1 "oorrsponndig Author.

---

Published as a conference paper at ICLR 2026
Pulmonnyy indirmttonn
Quaetor:: daggneeke mee irage?
Quasrton: West the
90
Queotion: th me image?
Dha srage??
Threis aonamake
Locore HOTNO ther daggo33
The shea ragan rates 251
Medical Enilit Propored
local
No atvormallty
Gm
7% 3781 habal: hight Saner rabar))
necr bo nereall Foom
(a) Single-shot VLM
Quoation: disgnnsi
OP14,
Ascornne Geounding
misimagg??
Eriistdd
medgemma
Kruy thac For stone ctreaus ag disass
Nov, Fwll spornig the Ror VQ2A
Eeience Crounded
posumonts heart farn Irfectinn
Tramss C Sover HANEE werrady rad
Onvrnz Stg.iman
Reproriatiio Rosull
Ai
CARE Plwe
Mollushation dire to tock ofvassoning [b) Grounding VLM
Wrong diseaas dus (c) Visual Reasoning VLM
Conrect Connecclegneen location.
Model Sice (Biitona)
(d) CARE-Coard
(e) Model Sizo vs. Average Acc.
Figure 1: VLMs for medical reasoning. (a) Single-shot VLMs often miss local evidence. (b) Grounding VLMs do not explicitly utilize ROI in reasoning. (c) Generalist visual reasoning VLMs fail with incorrect initial focus. (d) Our agentic CARE-Coord performs grounded evidence-based reasoning and expert discussion, improving accountability. accuracy vs. model size. Models with unknown size appear in the rightmost panel.
(e) Comparison of average medical VQA
e.g,, zoom-in, crop, OCR, between grounding and reasoning to supply regions of interest (ROIs) to the chain of thought. However, these methods couple perception and reasoning inside a single generalist model. This coupling demands high-quality paired ROI- -ol-grrunding and VQA supervision data and often costly multi-turn reinforcement learning (RL) to stabilize tool use (Yang et al., 2025a Zheng et al. 2025). Both tasks can degrade when such data are scarce. Compared with specialist visual grounding models (Liu al. [2023] Ren et al., (024). VI.M-based grounding frequently misses tiny but clinically salient findings, weakening downstream reasoning. Moreover, chaining all steps inside one model amplifies error propagation: early grounding errors bias subsequent reasoning and yield (c). These limitations motivate an agentic framework that coordinates well-trained specialist tools and feeds grounded evidence back into reasoning.
confident hallucinations, as illustrated in Fig. I
To advance Clinical Accountability in mudti-modal medical Reasoning, we introduce an Evidence- grounded agentic framework, CARE As presented in Fig. [ given user query and medical image, CARE explicitly models the clinical diagnostic workflow to perform VQA in three stages: (1) Medical entity proposal: a cenasiioncoonitionnen VIM proposes candidate medical entities (e.g.. anatomical structures and findings). which are fine-tuned via RL with verifiable reward (RLVR) to improve performance and accountability; (2) Entity referring segmentation: given proposed entities, model localizes the corresponding ROIs, producing pixel-level evidence; (3) Evidence-Grounded VQA (EG-VQA): an EG-VQA model reasons over the full image and one of three evidence views commonly used in medical imaging: (i) a zoom-in crop for local detail, (ii) binary mask for positional/spatial priors, or (iii) global indicator when local evidence is unnecessary. To operationalize agentic control, we introduce a dynamic coordinator CARE- Coord that plans the ioocaiivoas,onns selects the most informative evidence view, and performs iterative answer review, mitigating hallucinations. For a coordinator-free. self-contained setting, CARE-Flow executes all three evidence views and aggregates EG- VQA outputs via simple rules.
a tailored
We evaluated CARE on four standard medical VQA benchmarks (Hu et al., 2024, Liu et al. 2021 Ben Abacha al. 2019 Lau et al. (0188) spanning over ten image modalities and multiple organs, with results summarized in Fig. (e). Extensive experiments validate the effectiveness of our clinician-inspired framework. CARE-Flow (totaling 10B parameters) shows strong competitive results on multiple benchmarks, outperforming comparable generalist models and demonstrating strong parameter efficiency. demonstrating strong parameter efficiency. Adding the agentic coordinator, CARE-Coord, further improves performance by large margin, showcasing the potential of agentic reasoning. Our contribution is summarized below:
We propose CARE, the first medical agentic framework for accountable medical visual reasoning. A dynamic coordinator plans tool use and conducts iterative answer review reducing hallucinations via explicit evidence checks.
2

---

Published as a conference paper at ICLR 2026
We design a region-groundee reasoning workflow that feeds reliable, pixel-level evidence (referring segmentation, zoom-in views, or global indicators) back into VQA, improving both accuracy and accountability via accurate entity proposal and segmentation.
Empirically, our CARE-Floww surpasses the same size (10B) SOTA baseline by 10.9%, and exceeds the heavily-trained SOTA baseline (Lingshu-32B (Xu et al. (025)) by 2.26 %; CARE-Coord further outperforms Lingshu-32B by 5.2%. %
2
RELATED WORK
Medical Multimodal Large Language Models. General-purpose VLMs (Bai et al. 2025 Zhu et al. 2025a Hurst et al.
Meta 2024; OpenAl 2025) lack expert medical knowledi ledge. Early medical VLMs (Li et al. 2023; Moor et al. 2023) used low-quality data (Zhang et al. 2023b), and
2024
while recent systems (Chen et al., 2024a| Xu et al. 2025 Lin et al. 2025 He et al. 2024) leverage
better data and RI. (Tai al., 2025 Pan et al. 2025),
most remain single-shot black boxes prone to
hallucination. Agentic pipelines (Zhu et al.) 2025c;
Tang et al. [022 Xia et al. 2025 Kim et al.
2024) or tool-use (Li al. 2024a| Wang et al.) 2025 Nath et al. [025 He al. 2025) are also
explored but typically lack the visual evidence needed for diagnostic reliability. Instead, our method, with visual- -videnee supported reasoning, provides much better accountability for the answer.
Grounded VLMs. Research on grounded VLMs (Zhang et al.
2024a
Rasheed et al.
2024 Yuan
ctt al. 2025
Zhang et al.] [2024b; Lai et al., 2024) and medical
VLMs (Huang al.
2025a Wu
et al. 2025
Lin et al. 2025] Luo et al. | Chen et al., 2025; Huang et al. 2022a
2025b) has
focused on a grounded output, but typically treats grounding as an auxiliary multi-task optimization rather than using grounded clues to improve answer quality. These methods also require large-scale, fine-grained annotated data for supervised fine-tuning (SFT). In contrast, our method treats grounded visual evidence as supporting evidence for downstream reasoning, training VLM specialized to utilize local visual clues, which naturally leads to a higher performance at testing (Chen et al.) 2024b).
Vision-Language Reasoning, Since Wei al. (2022) proposed Chain- Thought (CoT), various methods (Wei et al., [022] Yao et al. 20033] Schulman et al.) 2017: Rafailov et al.) 2023) and
RLVR (Shao et al., [024| Yu et al. (2025a) have advanced reasoning. However, vision-
[025 Fan et al.) Zhong et al. 2025 Zhang al. 2023a Li al. [0224b] Yang et al. 2025b Qi al. (024) that focus on image content are often computationally expensive (c.g., multi-turn) (Zheng el al. [025 Fan el al. 2025 Zhong et al.
reasoning methods (Yang et al. 2025a] Zheng et al.
2025 Yang al. 2025b) or require high-quality buman-annotaan
data (Fan al. [025]] Qi et al.
2024: Li et al. [2024b] Zhang et al. (2023a), limiting medical adoption. In contrast, our method uses
model- -popposd visual clues as direct inputs during both training and inference.
3 METHOD
To advance multimodal medical reasoning while mitigating shortcut learning and hallucination, we decompose it into specialized sub-tasks and integrate expert visual tools with agentic coordination. aligning the pipeline with clinical practice and improving accountability with high-quality visual evidence and staged reasoning.
We detail the CARE framework with an overview in Fig. 2 CARE takes a user question and a medical image, and executes three decomposed sub-tasks: (1) Medical entity proposal. A question-prompe compact VLM proposes relevant anatomical structures or findings. The VLM fine-tuned with RLVR for coisiste- proposals. (2) -neity-re segmentation. A tailored iieering-segmentttiion model localizes the entities and produces high-quality pixel-level VQA. A finetuned VQA model reasons over the full image augmented by the grounded evidence. We further introduce the agentic control with the coordinator CARE-Coord which plans the tool invocation sequence, selects the most informative evidence view, and performs iterative chain-of of-thought review before finalization. For coordinator- free setting, CARE-Flow follows a static workflow that executes all evidence views and aggregates EG-VQA outputs by simple rules (e.g., majority vote).
Method Overview.
evidence (ROI masks/regions). (3) Sridence-Grrundde
3

---

Published as a conference paper at ICLR 2026
Mediicll Proposal
Tratnabie
Prozen
User Imput
Trirking
Oinginal Dmage
CoT: lungs appcar have some
Ennny
Phesas and
Increwsed exeaaty which could indicate the presence of e'emsin cheeaees. Ane lung
User Queestan
most relevant
"Long"
Progosal VLM
usev quessien.
andiyy Ooicett
(b) Entity Referring Segmentation
What diseases iollrded tn picture?
Question:
Proprictary MLLM Coordinator
Stop Segraantation
Amaage Proj,
SAM
SAM Decc.
Responser Moak 3
Segmentation Model
BERT Enc
Text Proj
G heeative OoT-Ardwwer Review
'Lng"
C-95%
GT: Lung cancer,
(c)
Evidence-Grounded VQA
Shep G-VQ4
frusking
Drdtence dea
Oegeationge
Natably
Output Patennal muse nodiule
Zoons-in in
supgesting
donser area
Response:
Masii
рresence
"Long cancer
Clocol
Dasr Cuuztion
Figure 2: Method overview. The proposed CARE comprises a VLM coordinator and a set of task-specific expert models. The coordinator plans tool use and conducts answer review, invoking specialist models as needed. The expert set includes: (1) a question-conding conditioned pntity-proposal VLM that identifies relevant anatomical structures/fiddings; (2) referring segmentation model that localizes entities with pixel-level ROI evidence; and (3) an evidence-grounded VQA VLM that reasons the image augmented with selected visual evidence (zoom-in, mask, global indicator).
3.1 MEDICAL ENTITY PROPOSAL
Medical Entity. We first propose the most relevant entities in the image conditioned on the user's question, mirroring climician's workflow of hypothesizing which anatomical structures, findings, or devices are implicated. We refer these as medical entities. A compact VLM is fine-tuned with RLVR to prioritize proposals that support sedencce-consister answers. As no public dataset exists for this task, we synthesize training data: for each image, we randomly sample a segmentation mask/medical entity and generate corresponding question, yielding paired (image, question, entity/mask) examples for supervision (see Appendix D.3 in detail).
Reinforcement Fine-tuning (RFT) for entity proposal. We fine-tune the entity proposal VLM with RLVR. Instcad of a binary accuracy reward, we use an booeedddng-similaar reward to capture semantic matches. For input image-question pair, the model outputs set of P entity names E = {2) With ground truth ε {e,)2 1==1 - small embedding language model SLM (Wang al. 2020) computes pairwise cosine similarities Si.j sim (&),(&)), (L)(ee)), forming matrix PxQ. We apply the Kuhn-Munkres algorithm (Kuhn 1955) find an optimal bijection
K. = {(;; ) mii((P.Q) maximizing total similarity, and define the similarity reward
Raim (E,
8t
min Q) (εL,C;)EK
Using soft similarity reward not only avoids O-gradient issue during RFT but also helps mitigate the domain gap between synthetic data and real user questions. It is not forcing an exact match, but rather maximizing semantic similarity. We further include a count reward that discourages empty or overly long proposals, Reount (f) = 1 if < P < 5, and 0 otherwise; and a repetition penalty Reepetition r+1 with the number of repeated entities. The total reward takes the form:
Reatity
Rsim + Reount Reepetition + Riormats
where Riormat enforces the <think> and <answer> tags to wrap CoT and answer respectively. The resulting VLM proposes entities closely aligned with the user's query, mirroring the clinician's first "where to look" step. The proposed entities serve as inputs to the segmentation model.
3.2 ENTITY REFERRING SEGMENTATION
We build an entity referring segmentation model based on SA-Med-2D (Cheng et al. (023) as shown in Fig. 2 (b). Given pre-trained SAM-style segmenter with image projector Projj, SAM encoder
4

---

Published as a conference paper at ICLR 2026
layers ENCSAM, and SAM mask decoder DeCM- augment it with frozen, lightweight stRe biomedical text encoder (Alsentzer et al. 2019) Ency and an embedding projector Projy. For an input image-entity pair (x', é), where E E, we encode the image and entity into token sequences t Proj)(z') and Ir = Éncr(é), respectively. We then concatenate them with binary modality token embeddings modd to form the SAM encoder input positional encodings. mood set to 0 for image tokens and for text tokens. We only use the image tokens from the output, i.e., first [4r] tokens, as key and value for Deca. Meanwhile, we project 'T with the Projr and use it as the query. The final segmentation mask is given by:
concat tr) + modd Inspired by
M = Decar Dee.oreEnnSsAA.. (ENCsAM (t)0 : [t,l,, (ro)r(tr)).
During fine-tuning, we only update Projs. Enc,, and Projz, thereby equipping the pre- trained medical SAM model with the ability of referring segmentation. Because masks may be imperfect. we compute a confidence score C from the mask probability map Mp via C(Mp) = Entropy(M,, and pass C' downstream so the coordinator and EG-VQA model can filter low-quality segmentations.
(2)
3.3 SVIDENCE-GROUNDER VISUAL QUESTION ANSWERING
Evidence-grounded VQA. We treat the segmentation mask as an additional hint and design three visual evidence types that reflect clinical practice and avoid information loss: (1) Zoom-in: we zoom in and crop around the ROI to provide a detailed, higher-resolution local view; (2) Mask: we feed the binary mask separate signal that acts as an intention-amplfficatte positional and spatial context; (3) Global: we provide an all-ones mask when no segmentation is available or when the task depends on global context (e.g., modality or imaging axis). This scheme allows the EG-VQA model to adapt to different question types while remaining efficient. We drop input masks whose confidence falls below a confidence threshold To empirically chosen based on Appendix D. preventing low-quality segmentation from harming decisions; in the worst case, the VLM falls back to its ROI-free behavior. We append mask metadata the prompt as "<image> (instance: (NAME), conf f dence:: after the input image to form a multi-in iilli-image input. We choose not to directly overlay the mask over the original medical image since the pixel value and image contrast may have physical meaning.
prior highlighting
IDENC)) 8) ". The clue is concatenated
Fine- tane-tuning the EG-VQA VLM. We fine-tune the EG-VQA VLM in a two-stage manner. First, we use the trained entity proposal VLM and the referring segmentation model to annotate raw VQA datasets with visual clues. Second, we perform SFT followed by RFT on the combined data, including
all three clue types. For RFT we add a CoTT-length reward Riength (j) 0.25 min (1, T, where the generated reasoning and L is preset maximum reasoning length, alongside a binary accuracy
reward Race (no external LLM verifier) and a format reward Riormat following Sec. 3.1 The final
reward for EG-VQA model is:
REG. Race + Rioormat + Riength-
The fine-tuned EG-VQA model is capable of handling three different types of visual clues and making more accurate evidence-supe supported decisions.
3.4 REINFORCEMENT LEARNING WITH VERIFIABLE REWARD
We fine-tune expert VLMs using RLVR to improve the answer accountability and generalizability. We specifically employ the DAPO Yu et al. 2025a) algorithm. With outputs {v: { generated by reference model Neat for input x, we update the policy model no using the following objective:
G vd
min (rigArg, clip(res, 1- 1-1111111 -.....aa)))) +c)) (5)
(o) = Eyg (1)
E
1=1
where Ta. 3 cs)
Teat (2i, <)) and the advantage is group-normalized as Ar,j R; mean ( (R, 1
std(f Rt
e and c) are the upper and lower clip thresholds. More detail is in Appendix D.2 Recent studies CCu
et al. 2025 Ma et al.) (225)) suggest SFT injects new knowledge (memorization), while RFT improves
existing capabilities by adjusting output to generate a reasonable Chain-of Thought (CoT). We choose to apply RFT to each expert VL. improve accountability via CoT under limited data.
5

---

Published as a conference paper at ICLR 2026
Table 1: Quantitative results on medical VQA benchmarks. We report medical VQA aceuracy
(%) on four standard benchmarks: OMVQA-3K (Hu et al. 2024), VQA-RAD (Lau et al. 2018). SLAKE (Liu et al. 2021) and VQA-Med-2011 (Ben Abacha al. 2019). Open-ended questions are scored by GPT-40 against ground-trute answers. Our segmentation model is smaller than IB. We highlight medical expert VLMs in gray and ours in green.
Model
OMVQA-3K VQA-RAD SLAKE
VQA-Med-2019
Overall
Proprictary
GPT 4o Hurst et 2024
64.07
58.54
63.55
$9.60
61.44
OpenAT 2025
63.19
62.20
Open-source
Llama-3.2-11B Vision (Mela 2024
43,10
53,22
63.17
57.40
54.22
Qwen2.5-V1.778 (aii chili 2726
61.40
54.10
59.73
50.60
56.46
Qwen2.5-VL-328
65.10
61.20
65.46
51.60
60.84
IntemVL.3-8B Zhu a.. 20263
75.97
61.86
66.13
$7.40
65.34
IntemVL.3-38B (Zhi AU7A1
78.57
62.97
67.26
DeepEyes-7B Zhicin
2025
57.40
56.10
61.16
52.20
56.72
Java-maed-vv.--mmssaal
2023
45,30
41.91
50.86
37.00
43.77
MedVLM-RI (Pan et
2023
72.07
41,46
46.47
45.40
$1.35
medgemma-4b Seliergren or 2025
61.50
58.09
69.66
47.40
59.16
medgemmma 27b (Sellergnen (Saleeerre
62.75
Huatuo(GP-VVision-/ hen
Z724a
70.70
59.87
60.50
$7.20
62.07
HuatuoGPT-Vision-33 (Chon 20244
76,80
60.75
64.12
60.60
Lingshu-7B (uu et 2025
73.17
58.54
76.15
58.80
66.66
Lingshu-322 32B [Xu at. 2025
83,97
64.75
82.25
58.20
72.29
CARE-Flow-S (4B)
56.32
53.60
CARE-Coord-S
97.70
62.75
77.19
60.60
74.56
CARE-Flow-H (10B)
96.17
63.64
83.21
56.60
74.91
CARE-Coord-B
97.97
68.29
83.11
60.50
77.54
3.5
COORDINATING EXPERT MODELS FOR VISION REASONING
We propose CARE in modes that mimic clinical workflows: static pipeline and a dynamic, coordinator-driven agent. Instead of a single model, we decouple the reasoning process into collabo- rating expert models. This approach optimizes each model for its specific task and uses independent reasoning with post-verification to prevent the amplification of errors from prior steps.
Static workflow. Our static workflow framework, CARE-Flow, processes an image-question pair (ær, .T) through sequential pipeline: Entity Proposal VLM - Segmentation Model + EG-VOA VIM. The entity proposal model outputs set of P entities & {e,) The segmen- tation model then produces corresponding masks M [M)e= and confidences C {CH [CJr= and we discard masks where C, < TO- Lacking coordinator to select the best visual clue, we call the EG-VQA model three times, once for each clue type, and use majority vote decide the final answer.
Dynamic coordination. Our dynamic agent, CARE-Coord, employs a powerful VLM as the coordinator (Fig. (2. The coordinator can plan which models call, act using tool calls, and review intermediate outputs to verify reasoning quality. This dynamic process improves decision quality and mitigates hallucinations. We instruct the coordinator verify the VLMs' reasoning logic rather than making clinical judgments. The coordinator also improves efficiency by choosing the optimal visual clue, even skipping localization entirely for global questions (e.g., about image modality), which reduces tool calls. Among the models we tested, GPT-5 (OpenAl) 2025) was the best-performing coordinator. We also detail experiments on training a small VLM for this task Appendix D.4
Iterative CoT-Answer Review. A small expert VLM may generate answers that are inconsistent with its reasoning. This because our rule-based reward Race only verifies the final answer, not the preceding chain-of-thought. While adding an LLM verifier could fix this, it would be computationally expensive. Instead, we use the coordinator for iterative CoT-answer review post inference. We instruct the coordinator to check the consistency of each thought-answer pair. If they disagree, the coordinator can re-run the expert model or correct the pair using its own reasoning.
EXPERIMENTS
In this section, we evaluate the performance CARE on four standard medical VQA benchmarks and compare it against state-of-the-art baselines. Our goal is answer the following research questions.

---

Published as a conference paper at ICLR 2026
Table 2: Ablation on grounded VQA. We ablate the 8B EG-VQA components during training, varying training visual evidence and coordinator configurations. Only one type of visual evidence is used for inference. CARE-Flow and CARE-Coord are highlighted in blue and green, respectively.
Training Visual Clue
Coordinator
Datasets
ID
OOD
Overall
Mask
Zoom
Global
Planning
Review
OMVQA
VQA-RAD
SLAKE
Avg.
VQA-Mee-2019
94.5
60.5
78.8
77.9
56.0
72.4 40.0
95.4
61.6
81.8
79.6
54,0
73.2 10.3
73.8 +1.4
95.5
62.0
83.1
80.2
55.6
74.1 +1.7
Wi.
63.6
83.2
81.0
56.6
74.9 12a
42.
97,9
68.2
83.1
83.1
60.8
77.5
+5.3
Table 3: Ablation on training strategy for EG- Table 4: Ablation on coordinator. We ablate
VQA. We ablate different training strategies for EG- different coordinators. "S" denotes using a VQA VLM. We adopt the CARE-Flow in this abla- single selected visual evidence. tion exclude the coordinator's effects.
Coordinator
Method
ID
0OD
Overall
N/A
Mask
80.8
56.8
74.8
N/A
Zoom-in
$1.1
56.2
74.9
67.9
57.4
65.3 10.0
NIA
Global
30.3
54.4
74.0
77.8
47.2
NIA
Mark Zooea
74.4
GRPO
75.2
54.0
69.9
+4.8
Majonity (CARE-Floo)
DAPO
77.0
54.2
71.3
16.0
SIT + DAPO
79.3
56.2
73.5
+8.3
73.3
SFT DAPO Rength (CARE-Flow
81.0
56.6
74,9
+9.6
(CAR-CCA)
77.5
IntemMVL3380 RET
S
30.7
58.2
25.1
RQ1:
Is CARE performing better than other reasoning non non-reasoning VLMs? RQ2: Is it helpful
to include visual evidence for the VQA tasks and how to better make use of it? RQ3: Does including the coordinator improve the capability of CARE? RQ4: What is the influence of using the entity proposal and segmentation model?
Datasets. For the entity proposal model, train it with SA-Med-20M (Ye et al. 2023) dataset. We create synthetic dataset of 10k training and Ik testing question-ROI pairs. As for the segmentation model, train it with 17k image-mast pairs from SA-Med-20M YYe al. 2023) dataset. We evaluate segmentation performance on the MeCo-G (Huang et al.] (202aa) dataset. For the VQA task. we use OmniMedVQA (Hu et al., 2024). VQA-RAD (Luu et al. 2018), and SLAKE (Liu al. 2021) for in-domain (ID) training. All the ID data are combined during training. For OmniMedVVA we randomly create 4k/3k split for training/testing. Besides the test set for II) data, we further use VQA-Med-2019 (Ben Abacha et al., 2019) for out-of-domanl (OOD) evaluation. Both open & closed-ended questions are included. For onen-ended questions, we use GPT-4o (Hurst et al. 2024) to judge the accuracy of the answers.
Implementation Details. The segmentation model built on SA-Med-2D (Cheng et al. 2023), which is much smaller (600M) than large VLMs; more details are in Appendix D.5 The entity proposal VLM is finetuned using Intern VL3-2B (Zhu et al. 2025a) with similar rewards in MiniLM- L6-v2 (Wang al. 2020). The EG-VQA VLM adopts Intern VL3--B/8B models, fine-tuned the in-domain VQA datasets. We denote variants with the suffixes "-S/B," using 2B/8B EG-VQA VLMs, respectively. With the 2B entity-proposal model and relatively small segmentation module, CARE-FlOw-S/B totals 4B/10B parameters. The default coordinator (CARE-Coord) adopts GPT- OOeeAA 2025). We set the mask confidence threshold TO 70% in segmentation following our
ablation Appendix D.6 More details are available in Appendices D.5 and D.8 to D.10
Baselines. We compare our model against baselines from several categories. Proprietary Models:
GPT-4o (Hurst et al. 2024) and GPT-5 (OpenAl 2025). General VLMs: Llama-3.2 Vision (Meta)
(Bai et al. 2025), VL.3 (Zhu et al. 2025a), and the visual reasoning model (Li et al. 2023), medgemma (Sel) lergren et al. 2025), HuatuoGPT- Vision (Chen et al. 2024a). Lingshu (Xu et al. 2025). and the
2024). Qwen2.5-V2
DeepEyes (Zheng et al. 2025) Medical VLMss:
reasoning model MedVLm- R1-B (Pan et al. 2025). We also compare with reported values from (Yu
ctt al. 2025b Cui et al. 2024| Lin et al. (203)) in Tab. 9 We also benchmark different Segmentation Models: RecLMIS (Huang et al. 2024b). LISA (Lai et al. 2024), MedPLIB (Huang et al., 2025a), UniBiomed (Wu et al., 2025). and BiomedPare Parse (Zhao et al.] 2024). Since not all segmentation models accept text prompts, we only compare with BiomedParse in the VQA benchmarks.
7

---

Published as a conference paper at ICLR 2026
Table 5: Ablation on segmentation model. We evaluate segmentation models MeCo- G dataset, and their impact on medical VQA. Note that only BiomedParse can be adapted to referring segmentation. Results with are reported by Huang al. 2025a).
Table 6:
Ablation on untity-proposal VLM. We ab- late training strategies for the entity-proposal VLM and measure their impact on entity proposal, segmen- tation, and medical VQA. Baseline 1/2/3 are of our framework using the 2B entity proposal VLM trained with different matching and rewards. "G" and "KM" are greedy and KM-based matching, respectively.
Segmentation Model
MeCo-G Mean Dice Score
Overall MedVQA
Der
CT
PET
Avg.
Proposal Model
Entity Match
Reward
Entity Accuracy
Mask
Dice
MedVQA
Overall
RecLMIS
88.8
74.9
81.1
81.6
LISA-7B"
52.6
54.2
62.7
GPT-5
40.5
50.5
72.7
McdPLIB-7B'
79.9
59.8
64.5
68.1
Baseline81
Bin.
72.8
41.0
75.1
UniBiomed
63.1
8.5
18.5
24.9
BaselineW2
G
Sim.
71.7
37.0
74.2
BiomedParse
82.6
8.4
20.3
30.1
74.1
Baseline#3
KM
Bin.
74.1
53.9
75.7
Ours
92.5
75.6
82.5
81,9
77.5
Ours
KM
Sim.
85.2
73.4
77.5
4.1 MAIN RESULTS
Evaluation in Tab. demonstrates the superiority of our proposed method that mimics the clinician's 's diagnosis process, answering RQ1. As shown in Tab. []] our constant 10B model (CARE-Flow achieves an SOTA average accuracy of 74.91% %, surpassing the much larger Lingshu-32B (72.9%%) by 2.6%. This demonstrates our architecture's parameter efficiency, as even our 4B model (CARE-Flow- S) outperforms models up to 38B parameters. While these general medical expert VLMs perform better than their base model due to vast domain-specinfe pre-training, our CARE surpassed them with limited fine-tuning data and computational resources. Comparing with DeepFyes--78 7B (Zheng et al. 2025), a single-model vision reasoning VLM, our model shows much more significant improvement against its baseline, demonstrating the effectiveness of our proposed multi-stage reasoning scheme. After switching to Agent mode, our CARE-Coort gains a uniform performance improvement of ~ 3% from its workflow version. This is not only coming from more reasoning, visual clue choosing, but also from the final answer reviewing by the coordinator. We further note that introducing a coordinator can help greatly improve the generalizability of OOD data, gaining over 6% improvement on a small version of CARE. CARE-Coord is not the best performing method on SLAKE since the dataset relies less on local visual clues, and its evaluation set is relatively small.
Qualitative Study. We provide qualitative results in the Fig. 3 with a full reasoning process for a VQA pair in the test set. We note that our coordinator not only chose the correct visual clue type but also corrected an impractical proposal from entity proposal VLM, which helps the EG-VQA model focus on the local detail. This highlights the effectiveness of the coordinator in planning and reviewing during inference time. We provide more examples that require more complex reasoning or clinically specific knowledge in Appendix D.12 Figs. to 19
4.2 ABLATION STUDY
Without loss of generality, all ablations use the 2B entity proposal VLM and the 8B EG-VQA VLM.
Ablation EG-VQA Model. Ablations in Tab. confirm the effectiveness of visual clues (RQ2), as our full static model outperforms a no-clue baseline by 2.5%. More diverse visual clues can improve the robustness at test time. Furthermore, dynamic coordinator that also reviews the final answer (row 8) provides the largest benefit, improving accuracy to 77. 5.5,, while a coordinator without reviewing cannot guarantee a better result compared with CARE-Flow.
Ablation on Training Strategy. We ablate different strategies in Tab. 3 In general, using SFT or RFT individually only yields moderate improvement, while combining them results in 1% improvement. Meanwhile, we note that DAPO is generally better than GRPO. Lastly, we note that using an additional length reward also helps improve the final performance by 1.4% as forces the model to provide longer reasoning and make use of visual evidence.
Ablation on Coordinator. We evaluate the influence of using different coordinators in Tab, 4 where to note CARE-Coord with robust coordinator beats all other variants, answering RQ3. Among all three evidence views, zoom-in stands out with the best performance. We also note that using both zoom-in and mask inference may harm the performance, which may be influenced by much longer input sequence. Using a weaker coordinator like GPT-40 or open-souree Intern VL3-38B (Zhu et al.) 2025a) may lead to incorrect evidence view choice or error due to hallucination. We didn't 't use smaller
8

---

Published as a conference paper at ICLR 2026
Table 7: Ablation on REatity Ablate the in- fluence of the different entity proposal reward. We highlight our CARE-Flow-B in blue.
Table 8: Coordinator Edit Radio. X - of means successful coordinator edit, and / - X means failed coordinator edit. OOD data is highlighted in gray.
Room
R Repertoon
Entity Accuracy
Mask Dice
Overall MedVQA
Dataset
X-V -
Coordinator
A with
Overwrite
Total
85.0
72.7
74.5
OMVQA-3K
1.90%
0.57%
+1.33%
2,47%
84.7
72.7
74.3
VQA-RAD
7.09%
4.87%
+2.22%
11.96%
/
82.7
72.9
74.4
SLAKE
2.77%
3.15%
-0.38%
5.92%
VQA Med 2019
7.60%
3.60%
+4.20%
11.20%
85,2
73.4
74,9
Overall
4.84%
3.05%
+1.79%
7.89%
Table 9: Comparison with More Baselines. We compare with additional baselines using their reported metrics (Recall for open-ended, Accuracy for closed-ended). FAVP and BioMed- VITAL use larger pre-trained data and fine-tune per dataset. PMC-CLIP is a classification- incompatible with open-ended generation. Our method is highlighted in green.
BioMed-VITAL
method
Method
VQA-RAD
SLAKE
Open Recull
Closed Accuracy
Open Recall
Clused Accuracy
PAVP Vicuna Yu et al 2025h
71.90 69.72
$8.20
87.20
88.10
HHoMed-VITAL-EER
2024
84.86
91.69
90.70
PMC-CLIP [Lin al 2173
84,00
88.00
Ours (CARE-Coord-R)
66.27
78.88
87.34
19
pre-trained In models, they failed make correct tool invocations. Additionally, GPT-4o tends to over-edit the answer from the EG- VQA agent, leading to more errors. Mcanwhile, we note that our own fine-tuned coordinator (IntemVV3--B RFT) also outperforms the majority vote method, especially on the OOD dataset. Namely, we can improve the evidence choosing process and reduce unnecessary tool invocations using fine-tuned small VLM. Still, this fine-tuned coordinator cannot conduct the iterative review, which makes it a second-best coordinator. Considering the size difference, it is expected that the proprietary coordinator (CPT-5) perform the best.
Ablation on Entity Segmentation Model. We then evaluate the segmentation model on the referring segmentation benchmark MeCo-G (Huang et al.) (2025a) in Tab 5o answer RQ4. Compared with existing methods trained on the same dataset, our method achieves a generally higher performance. It also outperforms general- purpose referring segmentation models like UniBiomed and BiomedParse with fewer medical entities. We further use BiomedParse as referring segmentation model in PARE-Coord, leading to a 3.4% performance drop in the medical VQA task.
Ablation on Entity Proposal VLM. To answer RQ4, we ablate the influence of using different training strategy for the entity proposal VL. in Tab. [6] where we report the entity proposal accuracy on the synthetic test set, the segmentation performance, and the medical VQA performance with CARE-Coord. Our expert entity proposal, VLM, beats all other variants. Notably, using either greedy matching (baseline # #1/2) or simple binary reward (baseline #1/3) can lead to inferior results, as greedy matching rewards the model even with only one correct proposal, and binary reward often results in zero- carooggadienttin issue during optimization. It is also worth noting that using GPT-5 for direct entity proposal behaves sub-optimally, as it lacks task-specific training.
Ablation of Entity Proposal Reward. Our entity proposal reward Remuy is composed of four terms (Eq. (2). We further evaluate sensitivity to other terms in Tab. 7 (after ablating Raam in Tab. (). We evaluate based on CARE-Flow to isolate the Coordinator's influence. The additional format rewards, Reount and Reepetition. help the model generate better formatted entity proposals, avoiding repeated or excessive output. While this aids downstream VQA, their influence is relatively small.
Evaluation of Coordinator Edits. To understand the Coordinator's behavior and CoT review process, we evaluate the ratio of coordinator edits in Tab. 8 We report: (1) x /: Ratio where the coordinator successfully fixed the expert model's wrong answer. (2) / X: Ratio where the coordinator mistakenly overwrote the expert model's correct answer. The difference the positive contribution; the sum is the total meaningful editing ratio. Overall, the overwriting ratio is less than 12% While review can introduce errors, it generally performs better, likely due to its stronger reasoning capability. It also shows higher "X - l" rate in OOD data, demonstrating stronger generalization. We note that we instruct the coordinator (Fig. 10 only to review CoT. It serves as a verifier and fool-invocation planner, not a final answer provider, leveraging GPT-5's strong reasoning, not its internal knowledge. As shown by in-domain performance (Tab. I CARE-Coord outperforms the GPT-5 baseline (83.09% vs 68.56%) showing the coordinator is not playing the decisive role.

---

Published as a conference paper at ICLR 2026
USER:
What abnormal finding identified in radiographic image?
A:: Airspacc opecity B: Pleural effusion
C: Atelectasia
DuLung consolidation
Iteration Coordinator Planning:
Stop 2: The question asks about parenchymal and plauna abnormalites on a chest CT alica. Key regions ane the differentiae sinspcc specitylcorsolidetio from pleural effusion or atelectasiss.
lungl and pleural
Proceeding to propose ROIs relevant to lung parenchyma and pleuro.
Calling Tool Medical Entttt Proposal:
<think- Resed on radiographic imagel can observe the presence abmnrrmal opacition within lung fields. These opecities are dencer than the surrounding lung tissue indicating a possible patholog cal process. The mosil located the lower zores which typicsily where consolidarion size distribution of these opasities sugsest localized process rather than diffuse one. surrounding lung parenchyma agpears relatively clear which further supports the presence of consolidation -ithiin <answer- lung consolidation <answar?
prominmnt
X Imprectical Segmentation Proposel
Iteration #2 Coordinator Reviewing:
Step 2: To distinguish sinspace opacity ve consolidation vs stelectasis
pieural effusion, focusing the parenchyma (both lungalaand
pleural spaces crucial. will segment left and right lungs tungil to localize paranchymal spacities and provide z0omed-ift evaluntion.
Zoom-in
Calling Tool- Refering Segmentation:
Promyt: ('left lung", : "right lung"]
Coordinator Fix Prompt Organ
Confidence: (86 91%, 93.30%)
Iteration #3 Coordinator Reviewing:
Stop 2: Using the segmanted lunga provides positionsl, and soomed details of parcnchlymai opachtes, helping discriminata consolidation (derse,, lobar, bronchogiarmst from stelectacis (volume loss, crowning and plaural effusion (dependent fluid outside lung). flluse zoom
paronchyma
Calling Tool- Evidence- Grounded VQA:
Zoom-in in Evidence for More Detail
<thinkoThe The grovided image transverse section of sean focusing thorscic cewity specifically the lungs. Upon sxamining
image
there
fields
opuciity compared the surrounding lung
These
areas of increased density do not appear to be sir-fille like the majonity of the lung parenchyma but insteed show a more solid or cloudy appeeranzee, Sush opecities can indicatte the presencee of substances other than sir within the aivecli such as fluid cel la or other materiall, The
rest of the flelds appear relativlly
the typical branching patterns the branchi blood vesselsvtsible no obvious displacement abrormality their positioning, bonry structures
heart
mediastinal structures are centrally located indluding the vertabrao and ribs ancirate the thoracio cavity and appear intact without any visible fractures or lesions. </think> <answer>y, Airepace spacitys/answer>
Clinical Reasoning the Identified Abnormality
Iteration #4- Coordinator Reviewing:
Reviewing:
I proposed focusing
ranchhyma
spaces differentiate parenchymal opacities from pleural effusion
atelectasis, since the question targats antortocacic abnommalties.
Iaegmented the left and right lungitto localles and closely inspect parenchymnal changes that would distinguish consolidation (derss, lober with air bronchograms)) from stelectasia (volume lass] and plaural affusion (fuid in pleumi spacs).
Using lung masks
zoomed-in clue allowed detailed assessment intesparenchymal opecities. model identified incressed parenchymmal density within the lungs consistent with sirspace opecity, without signs of pleural fluid leyering or marked volume loes.
<answer A. Airspace opacity lanewer
Ground Truth:A
Figure 3: Case Study. We present the complete reasoning trace for CT disease identification question. Key information from the coordinator is highlighted in blue, model reasoning in green. and each model's final answer in yellow.
Additional Baseline Comparison. We further compare with additional baselines Yu et al. 2025b
Cui et al.) 2024 [Lin et al., 2023) that cannot reproduced under our settings in Tab. 9 We report the recall for open-ended and accuracy for closed-endes questions, using the exact values from the original papers (recall combines the CoT and the final answer). Our method shows competitive performance on SLAKE (Liu et al. 202T) but did not outperform these baselines in VQA-RAD (Lau
ett al. 2018). This may relate to different training and evaluation settings. For example, FAVP (Yu al. 2025b) and BioMed-VITAL (Cui et al.) 2024) use 20 more training and are fine-tuned on VQA-RAD and SLAKE separately for a longer time, which naturally improves the performance on individual datasets. Furthermore, our method uses suipence-supported reasoning to provide better accountability, which is vital in real-world applications.
DISCUSSION AND CONCLUSION
In this paper, we propose CARE, a novel medical vision reasoning agent that follows a real-world visual-guided clinical decision-making process. Instead of a single-shot, black-box output, we divide the medical decision-making into three steps with expert model: identify the entity of interest, accurately locate the ROI on the image, and use local visual clues to make final reasoning. Comparing with existing methods, our CARE not only performs better on open benchmarks, but also demonstrates better accountability and reliability. Using a robust coordinator like GPT-5 further expanded the capability of CARE, demonstrating competitive accuracy in both ID and OOD settings.
10

---

Published as a conference paper at ICLR 2026
6 ETHICS STATEMENT
This work uses only publicly available medical VQA benchmarks (OmniMedVQA (Hu et al. 2024 VQA-RAD (Lau al. 2018), SLAKE (Liu al. (02T), and VQA-Med-2019 (Ben Abacha et al. 2019) and segmentation dataset (SA-Med-20M Ye et al. 2023)); new data were collected, and no patient interaction occurred. All datasets are used under their respective licenses and provenance policies, and no attempt was made to re-identify individuals. Our system is intended for research use only and is not medical software; it must not be used for diagnosis or treatment. To minimize harm, emphasize visual grounding and fact-checking, report failure cases, and dataset limitations.
REFERENCES
Emily Alsentzer, John R Murphy, Willie Boag, Wei-Hung Weng, Di Jin, Tristan Naumann, and Matthew McDermott. Publicly available clinical bert embeddings. arXiv preprint arXiv:1904.03333, 2019.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint 2XX::2502.13922 2025.
Asma Ben Abacha, Sadid A Hasan, Vivek V Datla, Dina FonneerFuushma, and Henning Müller. Vqa- med: Overview of the medical visual question answering task at imageclef 2019. In Proceedings of CLEF (Conference and Labs of the Evaluation Forum) 2019 Working Notes. 9-12 September 2019.
Junying Chen, Chi Gui, Ruyi Ouyang, Anningzhe Gao, Shunian Chen, Guiming Hardy Chen, Xidong Wang, Ruifei Zhang, Zhenyang Cai, Ke Ji, et al. Huatuogpt-vision, towards injecting medical visual knowledge into multimodal iims at scale. arXiv preprint arXiv:2406. /9280, 2024a.
Xupeng Chen, Zhixin Lai, Kangrui Ruan, Shichu Chen, Jiaxiang Liu, and Zuozhu Liu. R-llava: Im- proving med-vqa understanding through visual region of interest. arXiv preprint arX2v:2410.23377, 2024b.
Yanyuan Chen, Dexuan Xu, Yu Huang, Songkun Zhan, Hanpin Wang, Dongxue Chen, Xueping Wang, Meikang Qiu, and Hang Li. Mimo: medical vision language model with visual referring multimodal input and pixel grounding multimodal output. In Proceedings of the Computer Vision and Puttern Recognition Conference, pp. 24732-4744,, 2025.
Junlong Cheng, Jin Ye, Zhongying Deng, Jianpin Chen, Tianbin Li, Haoyu Wang, Yanzhou Su, Ziyan Huang, Jilong Chen, Lei Jiangand Hui Sun, Junjun He, Shaoting Zhang, Min Zhu, and Yu Qiao. Sam-med2d, 2023.
Tianzhe Chu, Yuexiang Zhai. Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le. Sergey Levine, and Yi Ma. SFT memorizes, RL generalizes: A comparative study of foundation model post-training. arXiv preprint arXiv:2501.1716: 2025.
Hejie Cui, Lingjun Mao, Xin Liang, Jieyu Zhang, Hui Ren, Quanzheng Li, Xiang Li, and Carl Yang. Biomedical visual instruction tuning with clinician preference alignment. In Advances in Neural Information Processing Systems, volume 37, pp. 96449-964677, 2024.
Zhuobai Dong, Junchao Yi, Ziyuan Zheng, Haochen Han, Xiangxi Zheng, Alex Jinpeng Wang, Fangming Liu, and Linjie Li. Seeing is not reasoning: Mypbench for graph-based evaluation of multi-path visual physical cot. arXiv preprint arXiv:2505.24138 2025.
Yue Fan, Xuehai He, Diji Yang. Kaizhi Zheng, Ching-Cher Kuo, Yuting Zheng, Sravana Jyothi Narayanaraju, Xinze Guan, and Xin Eric Wang. Grit: Teaching milms to think with images. arXiv preprint arXiv:2500 15879, 2025.
Yexiao He, Ang Li, Boyi Liu, Zhewei Yao, and Yuxiong He. Medorch: Medical diagnosis with tool-augmented reasoning agents for flexible extensibility. arXiv preprint arXiv:2506.00233 2025.
11

---

Published as a conference paper at ICLR 2026
Yuting He, Fuxiang Huang. Xinrui Jiang. Yuxiang Nie, Minghao Wang, Jiguang Wang. and Hao Chen. Foundation model for advancing healthcare: Challenges, opportunities and future directions. IEEE Reviews in Biomedical Engineering, 2024.
Yifan Hou, Buse Giledereli, Yilei Tu, and Mrinmaya Sachan. Do vision- language models really understand visual language? arXiv preprint arXiv:2410.00193, 2024.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3 2022.
Yutao Hu, Tianbin Li, Quanfeng Lu, Wenqi Shao, Junjun He, Yu Qiao, and Ping Luo. Omnimedvqa: A new large-scale comprehensive evaluation benchmark for medical Ivlm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20170-222 22183, 2024.
Xiaoshuang Huang, Haifeng Huang, Lingdong Shen, Yehui Yang, Fangxin Shang, Junwei Liu, and
Jia Liu. A refer-and-groume multimodal large language model for biomedicine. In International Conference Medical Image Computing and Computer-Assitted Intervention, pp. 399-39999090 409. Springer, 2024a.
Xiaoshuang Huang, Hongxiang Li, Meng Cao. Long Chen, Chenyu You, and Dong An. Cross-modal conditioned reconstruction for language-guin guided medical image segmentation. JEEE Transactions on Medical Imaging. 2024b.
Xiaoshuang Huang, Lingdong Shen, Jia Liu, Fangxin Shang, Hongxiang Li, Haifeng Huang, and Yehui Yang. Towards a multimodal large language model with pixel-level insight for biomedicine. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39-4, pp. 3779-3787, 2025a.
Yu Huang, Zelin Peng, Yichen Zhao, Piao Yang, Xiaokang Yang, and Wei Shen. Medseg-r: Rea- soning segmentation in medical images with multimodal large language models. arXiv preprint 1046:55 2025b.
Aaron Hurst, Adam Lever, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21226. 2024.
Yubin Kim, Chanwoo Park, Hyewon Jeong, Yik S Chan, Xuhai Xu, Daniel McDuff, Hyeonhoon Lee, Marzych Ghassemi, Cynthia Breazeal, and Hae W Park. Mdagents: An adaptive collaboration of Ilms for medical decision-making. Advances in Neural Information Processing Systems, 37: 14410-79452, 2024.
Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83-77, 1955.
Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9579- 97799-959,,22 2024.
Yuxiang Lai, Jike Zhong, Ming Li, Shitian Zhao, and Xiaofeng Yang. Med-rl: Reinforcement learning for generalizable medical reasoning in lision-language models. arXiv preprint arXiv:2503.13939 2025.
Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Fusner-Fushman. A dataset of clinically generated visual questions and answers about radiology images. Scientific data, 5(1):1-10, 2018.
Binxu Li, Tiankai Yan, Yuanting Pan, Jie Luo, Ruiyang Ji, Jiayuan Ding, Zhe Xu, Shilong Liu, Haoyu Dong, Zihao Lin, et al. Mmedagent: Learning to use medical tools with multi-modal agent. arXiv preprint XrXiv:2477.00 02443,224,,40 2024a.
Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training large language-and-visine assistant for biomedicine in one day. Advances in Neural Information Processing Systems, 36: 28541-28564, 2023.
12

---

Published as a conference paper at ICLR 2026
Yijiang Li. Qingying Gao, Tianwei Zhao, Bingyang Wang. Haoran Sun, Haiyun Lyu. Dezhi Luo, and Hokin Deng. Core knowledge deficits in multi-modal language models. arXiv preprint arXiv:2410. 10855, 2025.
Zejun Li, Ruipu Luo, Jiwen Zhang, Minghui Qiu, Xuanjing Huang, and Zhongyu Wei. Vocot: Unleashing visually grounded multi-step reasoning in large multi-moda models. arXiv preprint arXiv:2405.16919 16919, 2024b.
Tianwei Lin, Wenqiao Zhang, Sijing Li, Yugian Yuan, Binhe Yu, Haoyuan Li, Wanggui He, Hao Jiang. Mengze Li, Xiaohui Song, et al. Healthgpt: A medical large vision-language model for unifying comprehension and generation via heterogeneous knowledge adaptation. arXiv preprint arXiv:2502.09838, 2025.
Weixiong Lin, Ziheng Zhao, Xiaoman Zhang, Chaoyi Wu, Ya Zhang, Yanfeng Wang, and Weidi Xie. PMC-CLIP: Contrastive aaguage-image pre-training using biomedical documents. In International Conference on Medical Image Computing and Conputer-Assisted Intervention, pp. 525-536. Springer, 2023.
Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang. and Xiao-Ming Wu. Slake: A semantically- labeled tnowledge-enhaneed dataset for medical visual question answering. In 2021 IEEE 18th international symposium on biomedical imaging (ISBI), pp. 1550-1654, IEEE, 2021.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang. Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499 2023.
Lingxiao Luo, Bingda Tang, Xuanzhong Chen, Rong Han, and Ting Chen. Vividmed: Vision language model with versatile visual grounding for medicine. arXiv preprint arXiv:2410.12694 2024.
Lu Ma, Hao Liang, Meiyi Qiang, Lexiang Tang, Xiaochen Ma, Zhen Hao Wong, Junbo Niu, Chengyu Shen, Runming He, Yanhao Li, al. Learning what reinforcement learning can't 't: Interleaved online fine-tuning for hardest questions. arXiv preprint arXiv:2506.07527, 2025.
AI Meta. Llama 3.2:
Revolutionizing edge ai and vision with open, customizable models. Meta Al Blog. Retrieved December, 20:2024, 2024.
Michael Moor, Qian Huang, Shirley Wu, Michihiro Yasunaga, Yash Dalmia, Jure Leskovee, Cyril Zakka, Eduardo Pontes Reis, and Pranav Rajpurkar. Med- dlddflaaiigoo: a multimodal medical few-shot learner. In Machine Learning for Health (ML4H), pp. 353-367. PMLR, 2023.
Vishwesh Nath, Wenqi Li, Dong Yang, Andriy Myronenko, Mingxin Zheng, Yao Lu, Zhijian Liu, Hongxu Yin, Yee Man Law, Yucheng Tang, et al. Vila-m3: Enhancing vision-language models with medical expert knowledge. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 14788 -47988, 2025.
OpenAl. Gpt-5 system card. hittps:/cchi www.ho.ppaai.coom wwt.c///ppp---ssttmtmaaaaaaaaaaa pdd. 2025. Accessed: 2025-08-25.
Jiazhen Pan, Che Liu, Junde Wu, Fenglin Liu, Jiayuan Zhu, Hongwei Bran Li, Chen Chen, Cheng Ouyang, and Daniel Rueckert. Medvlm-r1: Incentivizing medical reasoning capability of vision- language models (vlms) via reinforcement learning. arXiv preprint arXiv:2502. 19634, 2025.
Ji Qi, Ming Ding, Weihan Wang, Yushi Bai, Qingsong Lv, Wenyi Hong, Bin Xu, Lei Hou, Juanzi Li, Yuxiao Dong, et al. Cogcom: A visual language model with shain-of-manipulaan reasoning. arXiv preprini 242.02400. 2024.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36:53728-53744 2023.
13

---

Published as a conference paper at ICLR 2026
Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Eric Xing, Ming- Hsuan Yang, and Fahad Khan. Glamm: Pixel grounding large multimodal model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13009-13018, 88, 2024.
Ming-Hsuan
Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, al. Grounded sam: Assembling open-world models for diverse visual tasks. arXiv preprint arXiv:2401. 20ti:22001.444 14159, 2024.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint
2017.
Andrew Sellergren, Sahar Kazemzadeh, Tiam Jaroensri, Atilla Kiraly, Madeleine Traverse, Timo Kohlberger, Shawn Xu, Fayaz Jamil, Cían Hughes, Charles Lau, et al. Medgemma technical report. arXiv preprint arXiv:2507.0520/ 2025.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemat- ical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Xiangru Tang, Anni Zou, Zhuosheng Zhang, Ziming Li, Yilun Zhao, Xingyao Zhang, Arman Cohan, and Mark Gerstein. Medagents: Large language models as collaborators for zero-shot medical reasoning. arXiv preprint arXiv:2311.1052 2023.
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self- attention distillation for task-agnostic compression of pre-trained transformers. arXiv preprint arXiv:2002.10957, 2020. doi: 10.44550/axX1.22000.055555
Ziyue Wang, Junde Wu, Chang Han Low, and Yueming Jin. Medagent-pro: Towards multi-modal evidence-based medical diagnosis via reasoning agentic workflow. arXiv e-prints, pp. arXiv-2503, 2025.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824 24837, 2022.
Linshan Wu, Yuxiang Nie. Sunan He, Jiaxin Zhuang, Luyang Luo, Neeraj Mahboobani. Varut Vardhanabhuti, Ronald Cheong Kin Chan, Yifan Peng, Pranav Rajpurkar, et al. Unibiomed: universal foundation model for grounded biomedical image interpretation. arXiv preprint 2034 2025.
Peng Xia, Jinglu Wang. Yibo Peng, Kaide Zeng, Xian Wu, Xiangru Tang, Hongtu Zhu, Yun Li, Shujie Liu, Yan Lu, et al. Mmedagent-rl: Optimizing multi-agent collaboration for multimodal medical reasoning. arXiv preprint arXiv:2506. 00555, 2025.
Weiwen Xu, Hou Pong Chan, Long Li, Mahani Aljunied, Ruifeng Yuan, Jianyu Wang, Chenghao Xiao, Guizhen Chen, Chaoqun Liu, Zhaodonghui Li, et al. Lingshu: A generalist foundation model for unified multimodal medical understanding and reasoning. arXiv preprint arXiv:2506.0704. 2025.
Senqiao Yang, Junyi Li, Xin Lai, Bei Yu, Hengshuang Zhao, and Jiaya Jia. Visionthink: Smart and efficient vision language model via reinforcement learning. arXiv preprint a3X1v.2507.13348, 2025a.
Shuo Yang, Yuwei Niu, Yuyang Liu, Yang Ye, Bin Lin, and Li Yuan. Look-backs Implicit visual re-focusing in milm reasoning. arXiv preprint arXiv:2507.03019, 2025b.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models, Advances in neural information processing systems, 36:1 16:118800 1809-11911111 11822, 2023.
Jin Ye, Junlong Cheng, Jianpin Chen, Zhongying Deng, Tianbin Li, Haoyu Wang, Yanzhou Su, Ziyan Huang, Jilong Chen, Lei Jiang, et al. Sa-med2d-22 dataset: Segment anything in 2d medical imaging with 20 million masks, arXiv preprint arXiv:2311.1 1169, 2023.
14

---

Published as a conference paper at ICLR 2026
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint WrXiv:2503.14470 2025a.
Ting Yu, Zixuan Tong, Jun Yu, and Ke Zhang. Fine- Fine-grained adaptive visual prompt for genera- tive medical visual question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39. pp. 9662-9670, 2025b. doi: 10.1609/aaai. v39...33047
Haobo Yuan, Xiangtai Li, Tao Zhang. Zilong Huang, Shilin Xu, Shunping Ji, Yunhai Tong, Lu Qi, Jiashi Feng, and Ming-Hsuan Yang. Sa2va: Marrying sam2 with llava for dense grounded understanding of images and videos. arXiv preprint arXiv:2501.0400 2025.
Jiarui Zhang, Mahyar Khayatkhoei, Prateek Chhikara, and Filip llievski. Towards perceiving small visual details in zero-shot visual question answering with multimodal llms. arXiv preprint arXiv:2310. 2023a.
Sheng Zhang, Yanbo Xu, Naoto Usuyama, Hanwen Xu, Jaspreet Bagga, Robert Tinn, Sam Preston, Rajesh Rao, Mu Wei, Naveen Valluri, et Biomedclip: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs. arXiv preprint 00300:2303.00915 2023b.
Tao Zhang, Xiangtai Li, Hao Fei, Haobo Yuan, Shengqiong Wu, Shunping Ji, Chen Change Loy, and Shuicheng Yan. Omg-llava: Bridging image-level, object-level, pixel-level reasoning and understanding. Advances in neural information processing systems, 37:71737-71761 2024a.
Yichi Zhang, Ziqiao Ma, Xiaofeng Gao, Suhaila Shakiah, Qiaozi Gao, and Joyce Chai. Groundhog: Grounding large language models to holistic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 14227-14238, 2024b.
Theodore Zhao, Yu Gu, Jianwei Yang, Naoto Usuyama, Ho Hin Lee, Tristan Naumann, Jianfeng Gao, Angela Crabtree, Jacob Abel, Christine Moung-War et al. Biomedparse: a biomedical foundation model for image parsing of everything everywhere all at once. arXiv preprint 120771, 2024.
Ziwei Zheng, Michael Yang, Jack Hong, Chenxiao Zhao, Guohai Xu, Le Yang. Chao Shen, and Xing Yu. Deepeyes: Incentivizing" thinking with images" via reinforcement learning. arXiv preprint arXiv:2505. 2025.
I Liangyu Zhong, Fabio Rosenthal, Joachim Sicking, Fabian Hüger, Thorsten Bagdonat, Hanno Gottschalk, and Leo Schwinn. Focus: Internal milm representations for efficient fine-grained visual question answering. arXiv preprint 2rXiv:2506.21700. 2025.
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504. 10479, 2025a.
Kangyu Zhu, Ziyuan Qin, Huahui Yi, Zekun Jiang, Qicheng Lao, Shaoting Zhang, and Kang Li. Guiding medical vision languala models with explicit visual prompts: Framework design and comprehensive exploration of prompt variations. arXiv preprint arXiv:2501.025 02.302 2025b.
Yinghao Zhu, Ziyi He, Haoran Hu, Xiaochen Zheng, Xichen Zhang, Zixiang Wang, Junyi Gao, Liantao Ma, and Lequan Yu. Medagentboard: Benchmarking multi-agent collaboration with conventional methods for diverse medical tasks. arXiv preprint 123., 2025c.
15

---

Published as a conference paper at ICLR 2026
A LIMITATIONS AND FUTURE WORK
Our method is mainly designed for diagnosis tasks that require local detail in the medical image, rather than general tasks like diagram analysis or global related tasks, constraining its improvement on these tasks. Additionally, we note that our method may not be the best performing model under all evaluation settings and datasets Yuu et al. 2025b Cui et al. 2024 Lin al., 2023). But our method still stands out in terms of data efficiency and clinical accountabiligg benefiting from our evidence- grounded design. As discussed in Appendix D.12 our method still suffers from model hallucination, especially from the coordinator. Developing system that is more resistant to hallucination will be our next target. We also plan to extend the visual model toolbox for more general tasks, e.g., a coding model or image editing tool.
B REPRODUCIBILITY STATEMENT
In order to ensure the best reproducibility, we provide full details of our implementation in Sec. Appendix [D3)] and Appendix D0 including the framework used, model, prompt, hyperparameter, and other details. We further plan to release the code and pre-trained model publicly later, including our 2B entity proposal VLM, entity referring segmentation model, 2B and 8B EG-VQA model, and our locally trained coordinator model. Since have used subset for OmniMedVQA (Hu et al., 2024) dataset, we also plan to release the full data split and pre- pre-processing pipeline for all data used the experiment. Random seed is set to 42 throughout data preparation, model training, and inference. VLM's temperature is also set to 0 during inference. Note that the current GPT-5 API does allow to adjust the temperature, so variation when reproducing our results with GPT-5 as coordinator.
the default value, which may lead to small
C THE USE OF LARGE LANGUAGE MODELS
Throughout this work, we only use Large Language Models (OpenAI GPT family models) to refine the paper writing and correct grammar errors. LLMs are not used during literature review, idea formation, or implementation except for necessary experiments.
D APPENDIX
D.1 BASELINE TRAINING DATA EXPOSURE
We provide the detailed information about the medical training data for each baseline used in the evaluation in Tab. I0 Most medical expert VLMs we report on were pre-trained or fine-tuned with overlapping medical data, often with significantly larger datasets (e.g.. HuatuoGPT- IM data, Lingshu used over 12M). In contrast, our total VQA training data size is only just over 10k. While general VLMs like the Qwen series not medically pre-trained, some other baselines (proprietary GPT family and Intern VL3) did include medical data in their training.
used over
D.2 REINFORCEMENT LEARNING WITH VERIFIABLE REWARD
To endow small VLMs with test-time reasoning, we adopt reinforcement learning with verifiable
rewards (RLVR) for chain-of-then -ft-oougtttthttttt reinforcement fine-tuning (RFT). Concretely, we use DAPO (Yu
et al. 2025a) to fine-tune the base model. Given an input from reference policy Tref- These outputs are then scored with rewards {R..... Re). DAPO optimizes the policy model by maximizing PPO-style clipped objective (Schulman al.) 2017)
sample G responses (V1. ..... DG
in Eq. (5). We use DAPO instead of GRPO (Shao et al. 2024) for stability and improved reasoning in setting. Following prior work (Yu et al. 2025a Shao al., 2024), we ask the model generate outputs with intermediate reasoning and a final answer wrapped in paired <t <think> and <answer> tags.
16

---

Published as a conference paper at ICLR 2026
Table 10: Medical Training Data for each Model. We report the medical-specific training data for each baseline for fair comparison. We highlight the overlapped training data with ours in bold. We highlight results of medical expert VLMs using gray, and our model using green
Model
Medical Training
Training Medical Datasets
GPT:4n (uusst et al. 2024 / GPT:5 OpenA1 2025
Unknown
Llama-3.2-1
Vision Mela
2024
X
N/A
Qwen2.5-VL (Bai et al. 2025
N/A
InterVVL3 Zhu
20250
VQA-RAD, SLAKE,, SLAKE, and others
DeepEyes Zheng el al. 2023
X
N/A
Mlvaa-Mddv...-miii
PMC-15M
MedVLM-R1-28 (Pan et
OmniMedVVQA VQA-RAD, SLAKE, and others
medgemma (Sellergren et al. 2025
VQA-RAD. SLAKE, MIMIC-CXR, and others
HuatuoGPT- Viston (Chen el 2024a
PubMedVision
Lingshu (Xu et al.
2025
VQA-RAD, SLAKE, VQA-Mede Mod-Med-2019, and others
CARE (Ours)
OmalMedVQA. VQA-RAD, SLAKE
D.3 ENTITY PROPOSAL DATA SYNTHESIS
Benefiting from the simplicity of the RLVR method, we are able to construct VQA data for our visual entity proposal task from an existing segmentation dataset using only medical images and corresponding medical entity names. As we only care about proposing the entity/ies that are related to the user question, the actual answer of the synthetic question does not influence the model, and we can directly use sampled entity names as the ground truth answer.
We use SA-Med-20M (Ye et al., 2023) as the base, which provides image-mask pairs plus rich metadata. From all masks, we collect their entity names and, after cleaning, obtain a list of 208 entities, including anomalies, organs, anatomical structures, and external devices. This is one of the largest segmentation entity lists compared to state-of- oort----rttt-tttt- methods, such as BiomedParse (Zhao et al. 2024) with 82 entities and VILA-M3 (Nath et al., (2025) with 127 entities. We then prompt GPT-40 (Hurst et al. 2024) with the image and the set of entities present to generate a brief, clinically grounded question about one or multiple provided masks; the corresponding answer is simply the involved entity/ies. Because the answer space is restricted to the curated entity list, we mitigate hallucinations during data synthesis and supervision.
Our prompt used for data synthesis is presented in Fig. 4 where we provide both the original medical image and its corresponding meta-information derived from the dataset to the GPT-4o model as input, as shown in Fig. 5 The synthesis model can then get access to the medical entities found in the image and create questions based on this information. The model is instructed to generate questions as it can never see the ground-truth metadata, avoiding issues due to data leakage. We also provide list of possible tasks in the prompt, as shown in Fig. 4 which includes: (1) Describe the entity; (2) Find the anomaly with different difficulty; (3) Locate the entity: (4) Count the number of entities; (5) Directly segment required entity; (6) Crop the described region. These tasks can largely cover the type of questions presented in the general medical VQA benchmarks, covering variety of different requirements, and ensure a proper generalizability of the trained model.
Examples of synthetic data can be found in Appendix D.I1
D.4 REINFORCEMENT FINE-TUNING FOR COORDINATOR MODEL
We train our own coordinator based on Intern VL33-8 Zhu et al. 2025a) using RFT. Despite the pre-trained coordinator that can directly make tool calling without fine-tuning, they fall short terms of lateney Appendix D.6, cost, and visual clue selection. We decided to train an expert coordinator designed for our workflow instead.
Similarly, we choose to use RFT to train our model, and the definition of the task is defined as follows: given a user input image-question pair (Iz, 2T). the model should make plan to decide the order of calling each tool, and specifically which visual clue to for the EG-VQA model. Since focus on the medical VQA benchmark, we use the training data from these datasets to create datasets with different planning routines.
17

---

Published as a conference paper at ICLR 2026
Data Bynthesis Froop::
You aze Nodtth-eysth: foz every infut inage and 1808, outpet one independent, self-eoctained, of dialog, dialog is with question ostional tool
ore-line 3808 stalogs,
dictionary
ascistant acosages embed sttinko.s/thimm hataao//whhaaaeesttesssaaaaaee
first raplay
tobena think).
<thinks condoct step-by-ereg and metlyste a fex sentencee, analysta the appearence the image and rate the dect sion whether to call the segnentation tool cepending on the gererated question, and deterxine the rough name of the 50I to lock at. <action>
the Joo tool tool_cal (onit when ssmentation la needed) whose achers is Inamer "segrentat soot*, argunentes (o idstcinage pethat, prorgt prorpti"colaeee ("cllees" ("seqmanation, reoutt"; (omuit""""""aa""" ("oask *wrask "apakk patto", "elaso_ ("cisbll>")
cinaget.
Choose taska at rendon idesoribe, anensly-find leweys with target label /hards without target sebpat/otttier no in the inagel, lecste, count, direst sopmentation, croy desociptionl/ invoke the cool for
wwwpiingg/ccinggg/o /exiateneeeeeeee unleas obvious absence.
Act 16 you see only the image-never reference or leak JBCK keys (instance count, eto.), fi filemeres, these instructions. NEVER referance the segnentatinn ttinn moae in the <ebinio section, include taga, metadata,
vazy
phrastngg lve
Figure Prompt for Data Synthesis. We present the prompt used for the GPT-4o model to synthesize training data for the entity proposal model. We ask the model to generate questions based on the given meta-information of the provided image. The question is related to the medical entityfies in the metadata.
Exarp1: JBcK for cota syathesis
"inage path* "inage//tt 00--2097022- -001--x pnn",,
"mazin podalsty"; fet", "acbi by:" 19/6",
"dataset": "M8022222"
"originel namelts *aros 0C01",
"inssing typef: 130",
"aliiw
"alice indexty "inage resclation";
768, 768
"Lestanoo indo":
"rask_patii:: "paaks/es__ Satpe/es_00-a-000 Tww.ks/et_.00-A0000222----- 0000000000000-2----0000000000000 000002222222244400 pocpa_ooo--x____e X_00022-00000 png"" Index":
"rack net?":
"naae bounding box"s 1350, 314, 405, 350),
"mask bounding box Loose*: [350, 294, 423, 3801,
tnornal reed mask bounding box": 1456, 409, 525, 4691. "naok_pize* (268, 758),
"owteer point"s 1976, 355)
pana": "prostate
"irstance count": "ocestate and utarus":
1.
"iratanow existence"s I
"uterus"
Figure 5: Example Metadata for Data Synthesis. We present the metadata used for the GPT-4o model to synthesize training data for the entity proposal model. It includes the information about the original image, medical entities labeled from the dataset, and other related information, like the position of each mask.
Considering for each pair of user input the training data (#1, and its ground truth y, we have its prediction of using all three types of visual clue {Uzoom Imasks Dotobal), } filter the visual clue that produce a correct answer to a set viable visual clue for each training entry
=
{cl Úe v}, where c e {2oom, mask global)
(6)
Our training data is then given by (æ,, ET) and the corresponding V.
We fine-tune tune the model to generate the visual clue that most suitable for the input, which naturally leads to tool calling chain based this output, Since some of the data may have more than one
18

---

Published as a conference paper at ICLR 2026
trawa/dwward
trainireward www.daaeeeeeenninniiiiiiioe
soae zami
Gaaty hoppas
0.5
045
25
0.1
onkngpobal step
tollapooal Stop
230
200
60C
530
L2
200
400
600
203
1.24
(6) Reward Curve for Entity Proposal Model
Similarity Reward Curve for Entity Proposal Model
Figure 6: Reward Curve During Training We provide the full reward curve of the entity proposal model during training. (a) is the overall reward, and (b) the individual similarity reward.
viable visual clue, ie., V 1, we reward the output if is a subset of V.
E V, [1, 0, otherwise,
Reuordinator (2)
We use a similar format and repetition reward to encourage unique output with correct tags.
D.5
IMPLEMENTATION DETAILS
In addition Sec. we include more details about our implementation in the following section.
D.5.1 BASELINES.
For all our baselines, we use pre-trained models with their official weights. For the proprietary VLMs, access the GPT family model with the Azure OpenAl API. All baselines are instructed to answer the question with short phrase rather than a long description of the image. As for the MedVLM-RI ------1--2BB (Pan et al.) (2025) with reasoning capability, we follow its original prompting and extract its final answer after reasoning for evaluation.
D.5.2 ENTITY PROPOSAL VLM FINE-TUNING.
For our entity proposal VLM, we use Intern L3-2B Zhu et al. 2025a) as our base model. We fine-tune the model using the DAPO (Yu et al. 2025a) algorithm. We set the number of generations
for each rollout to be 8, and we set B = 0, é, 0.2, e = 0.28 following (Yu et al. 2025a). We freeze the vision encoder and projection MLP while fine-tuning the large language model using LoRA (Hu et al. (022) with rank = 32, alpha a = 64, and LoRA dropout of 0.05. We use a learning rate of I 10 with linear learning rate decay. We the mini-batch size be 8 and use gradient checkpointing of 2, which gives us a total batch size of 64 for 4 GPUs. The max completion length is set to 2048. We trained the model for 1200 steps using DeepSpeed Zero-2 on a single machine with 4 NVIDIA A 1100-80G GPUs. The fine-tuning process takes roughly 10 hours. The prompt for this model can be found Appendix D.8 All random seed is set 42. The reward curve during training is in Fig.
D.5.3 SEGMENTATION MODEL ARCHITECTURE.
As mentioned in Sec. we build our referring segmentation model based on SA-Med-2D (Cheng ct al. 2023). We use BioClinicalBent (Alsentzer et al. 2019) as our frozen text encoder, as it is pre-trained vast amount of medical data while maintaining relatively small model size. We use an additional linear layer as the text projector to project text tokens to the SAM decoder embedding space. We use binary modality embedding token moo to distinguish between image and text tokens. use eood, = 0 the i-th token belongs to the image sequence; otherwise, we set it to 1. The overall size of the segmentation model is therefore 600M.
19

---

Published as a conference paper at ICLR 2026
tsain/eward
0.3
15
04
14
taingjoobe 0.2
Reward Curve for G-VQA Model
(b) Accuracy Reward Curve for D-VQA Model
Figure 7: Reward Curve During Training We provide the full reward curve of the G-VQA model during training. (a) is the overall reward, and (b) is the individual accuracy reward.
Table 11: Confidence Threshold Ablation. We ablate different mask dropping confidence thresholds during training. We use CARE-Flow model here to isolate the influence of the coordinator and only focus on different mask confidence thresholds. Our choice of To is highlighted in green.
Overall Arg. Acc.
TC
VL3-2B InternVI.3-88
0%
70.60
74.16
30%
70.82
74.66
50%
71.02
74.84
70%
71.14
74.92
69.72
72.48
D.5.4 SEGMENTATION MODEL FINE-TUNING.
We train our referring segmentation model based on SA-Med-2D (Cheng et al. 2023) pre-trained weights and using frozen BioclinicllBERT (Alsentzer
2019) language encoder. We use learning rate of 8 x 10-5 and cosine learning rate decay during training. The weight decay is set to 0.1. The batch size is set to 64. We train the model for 30 epochs on single NVIDIA A100-80G GPU, which takes roughly 18 hours.
D.5.5 EG-VQA VLM FINE-TUNING.
We two-stage fine-tuning recipe for EG-VQA model, and the same recipe for both 2B and 8B models. We use the visual clue generated with our entity proposal model and segmentation model for this model.
Stage 1: We fine-tune our model with SFT in stage I based on the Intern VL official code. We unlock the projection MIP, and fine-tune the vision encoder and large language model using LoRA with rank == 16 and alpha a = 32. We set the maximum CoT length to be 200 for the length reward. The learning rate
2 x 10-5 and weight decay is 0.05. We use a cosine learning rate decay with warm-up of 3% training steps to update our learning rate. We set the mini-batch size to be 4, and gradient checkpointing
4, which results in total batch size of 64 given GPUs. The max sequence length is set to 16384. We trained the model for epoch using DeepSpeed Zero-1 on single machine with 4 NVIDIA A A100-80G GPUs. The training process takes roughly hour for 2B model and 2 hours for the 8B model. Our training data during SFT combines all three in-domain datasets (~ 10k entries), each corresponding to different types of visual clues, resulting in 3 times more data.
Stage 2: Then, we fine-tune the model using the DAPO algorithm with the same settings as our entity proposal model, for both the 2B and 8B models. We use the rollout setting, learning rate, batch size, and LoRA settings. We train the model on the same data as for SFT for 1 epoch, which takes roughly one and half days random seed to 42, The full reward curve for both models is in Fig. 7
machine with 4 NVIDIA A A100-80G GPUs. Similarly, we set the
The prompt for the EG- EG-VQA model can be found in Appendix D.9
20

---

Published as a conference paper at ICLR 2026
Table 12: Inference Speed Evaluation We compute the time cost for each component our CAREmodel during inference time on OmniMedVVA (Hu et al. 2024) dataset. We report the time in seconds used to finish one single VQA request.
Inference Time Per-input (scconds)
Model
Avg. Tool Invocations
Coordinator
Entity Prop
Segmentation
GVQA
Overall
Fixed Visual Clue
3.00
1.31
0.06
2.03
3.44
Majority Vote (CARE-FlOw)
5.00
1.27
0.06
5.81
7.22
IntemVL3-33B
3.09
11.57
1.06
0.09
1.97
18.63
GPT-4o
2.82
17.44
1.24
0.07
2.08
21.08
GPT-5 (CARD-CooRd)
2.50
39.32
1.09
0.12
2.28
43,51
IntemVL13-88 RFT
3.00
1.83
1.35
0.11
2.89
6.24
D.5.6 HEURISTIC MAJORITY VOTE COORDINATION,
As mentioned in Sec. 3.5
majority vote for static workflow coordination. However, for cases like open-ended questions or diverged answers, we default to using the best-performing zoom-in visual clue as our final answer according to Tab. 4 where zoom-in performs the best over other types of clues. Eventually, our workflow will produce reasoning-basee final answer to the user input, along with series of filtered segmentation masks of the ROIs.
D.6 ADDITIONAL EXPERIMENTAL RESULTS.
Ablation of Confidence Threshold. We ablate different confidence thresholds drop the masks during the final grounded VQA in Tab. [1] While the gap between different thresholds is generally small, our choice of TC: 70% generally optimal.
Inference Speed Evaluation. We evaluate the inference speed of each module in our CAREin Tab. 12 using 2B+8B version. While our CARE-Flow is very fast, the agent version with a large coordinator takes much longer to output. Our locally fine-tuned coordinator achieves better efficiency than the heuristic majority vote, as it does not need to iterate through all three types of visual clues. As for the proprietary VLM APIs, it is easy to notice that the major latency comes from the proprietary coordinator rather than our developed expert models, which demonstrates the trade-off between more robust and intelligent coordination and the system's 's efficiency.
Full Evaluation Results. We present the full benchmarking results in the Tab. 13 in this table are the same as Tab. [ but we include all the models that we have evaluated, mainly QwenVL2.55 (Bai et al. 2025) and Intern (Zhu et al.) 2025a) at different model sizes. Overall, our 10B-level model can still outperform the largest Qwen and InternVL models with more than 70B parameters, which further highlights its parameter efficiency.
The settings
Fine-tuned Baseline. For fair comparison, introduce Intern Wit-VLL---inntedd new Intern VL3 baseline fine-tuned with our exact training data configuration to isolate the training dataset's 's influence. We report its performance in Tab. [4] We compare this new baseline with static CARE-Flowmodel to isolate the influence of the external coordinator. Note that CARE-Flow-S uses the Intern VL.3-BB architecture for entity proposal and EG-VQA, whileCARE-Flow- uses VL for entity proposal and Intern VL3-8B for EG-VQA. The results show that, even with the same training and fine-tuning settings, our method consistently outperforms the baseline using the same base model by an average of over 4%. This highlights the contribution of our decomposed and evidence-groune VQA pipeline.
Conservative Coordinator Strategy. We further explored a conservative coordinator strategy that forces the coordinator to use answers from the expert VLM when the corresponding reasoning confidence is high. Since the reasoning VLM lacks direct confidence, we asked it to output confidence score (0-100) based on its own reasoning. We apply a hard threshold a: if the local VLM's s confidence is 2 we use the expert model's answer; otherwise, we use the coordinator's final answer. We report this final performance alongside our CARE-Flow-B observe that adapting the final answer based directly on the expert VLM's confidence does not improve performance, consistent with previous evaluations. Our coordinator's 's CoT review process already implicitly considers the confidence of the expert VLM's reasoning trace, as ask it
the Tab. 15 We
generally
21

---

Published as a conference paper at ICLR 2026
Table 13: Full Benchmarking Results on Medical VQA Datasets. We present the full benchmarking results on Medical VQA datasets (Hu et al. [2224] Liu et al. 2021; Ben Abacha et al. 2019 Lau
al. 2018) and report their accuracy (%). The open-ended questions are evaluated using GPT-
4o (Hurst et al., 2024) against ground-truth. We use the Instruct fine-tuned model whenever available.
We use a 3% subset of OmniMedVQA (OMVQA) Hu et al. (2024) dataset for benchmark. We
highlight results of medical expert VLMs using gray, and our model using green.
Model
OMVQA-3K VQA-RAD
SLAKE VQA-Med-2009
Overall
Proprietary
2024 GPT-4o Huurst et al.
64.07
58.54
63.55
59.60
61.44
GPT--5mmn [OpenAI 2025
68.57
59.87
65.94
63.99
GPT-5 OpenAL
74.73
63.19
67.75
62.20
66.97
Open-source
Llama-3.2-11B-Vishe Mcta 2024
43.10
53.22
63.17
57.40
54.22
Qwen2,5-VL-3B Bai al 2025
$7.07
54.77
53.24
43.00
$2.02
Qwen2.5-VL-7B [Bai ct 2025
61.40
54.10
59.73
50.60
56.46
Owen2.5-VI.-22B
65.10
61.20
65.46
$1.60
60.84
Qwen2.5-VL-72BB (Bar
65.27
62.75
67.37
53.00
62.10
InternVL.3-2B (7hi
20258
75.43
55.65
63.07
54.00
62.04
IntemVL3-8B Zhu et
20258
75.97
61.86
66.13
57.40
65.34
InternVL.3-14B (Zhu et al
2025a
77.70
63.64
68.61
$7.60
66.89
InternVL3-38B ZZhu el al 20258
78.57
62.97
68.70
$8.80
67.26
IntemVLL3-78B (7hh al. 2025a
80.73
65.85
72.42
61.40
70.10
DecpByes-7B (Zheng ct al. 2025
57.40
61.16
52.20
56.72
-wa-mmcd-vl..5mmmstaa----
2023
45.30
41.91
50.86
37.00
43.77
MedVLM-RI-2B (Pan et al..
2025
72.07
41,46
46.47
45.40
$1.35
medgemma-4h (Sellergren et al.
2025
61.50
58.09
69.66
47.40
59.16
medgemma-27b (Sclicrgren
2025
64.23
62.75
70.52
48.40
61.47
HuatuoGPT-Visidree hCh ot al
2024a
70.70
59.87
60.50
57.20
62.07
JnatuoGGT-Visioon3333 Vision-34B (C hen al.
2024a
76.80
60.75
64.12
60.60
65.57
Lingshu-7B (Xu al. 2025
73.17
76.15
58.80
66.66
Lingshu-32B eXu
83,97
82.25
$8.20
72.29
CARE-Flow-S (4B)
94.53
56,32
78.44
53.60
70.72
CARE-Coord-S
97.70
62.75
77.19
74.56
CARE-Flow-B (10B)
96.17
83.21
$6.60
74.91
CARE-Coord-B
97.97
68.29
83.11
60.80
77.54
Table 14: Comparison with Fine-tuned Baseline. We report medical VQA accuracy (%) four standard benchmarks: three in-domain OMVQA-3K (Hu et al. 2024). VQA-RAD (Lau et al. (2018).
SLAKE (Liu al. 202T) and out-of-domain VQA-Med-20DD 2019 (Ben Abacha al. 2019). We
compare with the AaternVL3 (Zhu et al. 2025a) bascline fine-tuned with the same training data to isolate the influence of different training data. Our results are highlighted in blue.
Model
OMVQA-3k
VQA-RAD
SLAKE
VQA-Med-2019
Overall
IntemVL3322 (zero-shot)
75.43
55.65
63.07
54,00
62.04
IntemVL3-28 2 2--nnnuued
87.97
57.43
69.56
51.00
66.49
CARE-Flow-S
94.53
56,32
78.44
53.60
70.72
VL3-8B (zero-shol)
75.97
61.86
66.13
57.40
65.34
Intem La--VLLL-BB--neeune
91.13
61.86
76.53
53.80
70.83
CARE-Flow-B
96.17
63.64
83.21
56.60
74.91
review the CoT quality. Consequently, the coordinator insists on the expert VL.M's output when its reasoning trace is confident.
Judde Stability. We also evaluate our results with different LLM-as-judges (both propri- etary (Hurst et al. 2024) and open-source (Zhu al. 2025at) to demonstrate the variance stemming from the judger. We skip OmniMedVQA (Hu et al. 2024) in this evaluation as it only contains closed-ended questions. We report the performance in Tab. 16 We note that the variance between different LLLM-a--juder is very small (less than 1%). Our performance improvement is significant enough considering this variance.
22

---

Published as a conference paper at ICLR 2026
Table 15: Conservative Coordinator Review Strategy. We report medical VQA accuracy (%) on four standard benchmarks (Hu et al. [024] Lau et al. 2018 Liu et al. [021] Ben Abacha et al. 2019) compare our coordinator strategy against conservative strategy that votes for EG-VLM'S answer when its confidence is high. We report the performance with different confidence threshold a. Our
results are highlighted in green.
Model
OMVQA-3K
VQA-RAD
SLAKE
VQA-Med-20119
Overall
25
96.70
66,29
83.77
56.40
75.79
50
96.74
67.40
83.87
56.60
76.15
75
97.47
67.85
83.68
56.80
76.45
CARE-Coord-B
97.97
68.29
83.11
60.80
77.54
Table 16: Different LLM-assJudge. We report the performance on the datasets with open-ended questions Lau al. 2018); ; Liu et al. ( Ben Abacha al. (2019) to demonstrate the variance
of different LLM as judge. We experimented with both proprietary (Hurst et al. 2024) and open-
source (Zhu et al. 2025a) LLMs of different sizes. We report the averaged performance across
different L.MSs and the corresponding standard deviation.
LLM Judger
VQA-RAD
SLAKE
VQA-Med-2011
Overall
GPT-4o
68.29
83.11
60.80
70.73
GPT-44-mini
68.51
84.35
60.60
71.15
MMlemnLL3-38B-Isttee
82.44
69.62
IniernVL.3-78--nnsee
67.41
83.30
60.20
70.30
Avg. STD
67.91
83.30
60.15
70.45
10.58
_079
10.81
10.66
Fine-tuning Strategy for Entity Proposal Model. We report the performance of the entity proposal VLM with SFT and SFT + RFT to validate our choice of using RFT on the entity proposal VLM. We use the static CARE-Flowframewort here to isolate the influence of the coordinator review. As shown Tab. 17 using SFT alone or combining SFT and RFT underperforms our RFT model, which validates our choice of training recipe. This suggests that applying SFT with synthetic data may be harmful to our task. Applying RFT over SFT indeed helps general performance, but for a task that is already within the model's capability, directly applying RFT could be the best option.
D.7 HUMAN EVALUATION
To better quantify the accountability of our method, we conduct a human evaluation to assess the quality of our model's reasoning trace.
Experimental Setting. We randomly sample 35 correctly answered cases from the four test datasets evaluate the accountability of the reasoning traces. We develop web-based evaluation platform, where human evaluators can examine the full reasoning process and assign True/False judgment for each case (the user study interface is shown in Fig. 24). We recruit nine medical students to perform the evaluations through the platform. For comparison, we also included GPT-4o as the coordinator baseline.
Experiment Participant. Due to the limited time, we are only able to contact participants with PhD/MD-level knowledge for our experiments. We gathered pieces of feedback from 9 participants with no prior knowledge about our work, all of whom have either obtained or are pursuing PhD/MD degree related the medical and imaging domain. We plan to further collaborate with experts with clinical experience in future research, and we agree that this is critical step towards real-world application.
We report the human evaluation pass rate alongside their original overall medical VQA performance in Tab, 18 CARE-Coord-B achieved a human evaluation pass rate of 82. 14%, surpassing the GPT-4o baseline (73.94%) This result demonstrates that proposed framework not only achieves higher accuracy but, more importantly, generates reasoning traces that are more factually accurate and visually grounded, thereby offering superior clinical accountability.
Results.
23

---

Published as a conference paper at ICLR 2026
Table 17: Entity Proposal Model Fine-tuning Strategy Ablation. We ablate different fine-tuning strategies for the Entity proposal VLM on the entity proposal, referring entity segmentation, and medical VQA tasks. Our choice of training strategy is highlighted blue.
Method
Entity Accuracy
Mask Dice
Overall MedVQA
SFT SFT RFT
76.70
72.69
74,26
80.92
74.03
74.44
RFT (CARE-Floo-BB
85,28
73.48
74.91
Table 18: Reasoning Trace Human Evaluation. We conduct human evaluation that evaluates the quality of the reasoning trace of our model on subset of samples. We report the pass rate of human evaluation. We compare our CARE-Coord-B against our model with the GPT-4o coordinator. Our method is highlighted green.
Coordinator
Human Evaluation Pass Rate (%)
GPT-40 GPT-5 (CARE-Coord-B)
73.94
82.
Extity Prepocel VLV Prospt
13111 Promph2
Yeu are redical VoA wes stant.
Ese the main image and any axtra mask-inages balov to answer the questicn. Each mask is only the
hist
assistanti
confidance
ia
SQuestion templates
[Queation] Aeply wish exactly two tager sthinkyyour think propeart/ thiikke then sinal anaves in censver> POSTITON, HAYE (o.tl../////srrr each answer tuple with 8138 from ivery tiny sire, ciny a se, smsil sise, rediur sixe, larpe sise, very large size: and PoSTITc from tupper, lowez, laft, right, conter, upper left, upper sight, lover left. lever zight] followed ustity Names progose to zeat selevant no.a based the queatien-do not answer the question directly. ven
Figure 8: System Prompt for Entity Proposal Model. We present the system prompt for our entity proposal model, where we instruct the model to name the most relevant medical entity related to the user question. We note that even if we asked the model to also generate the size and position of the entity, we only use the entity name for the downstream tasks.
D.8 ENTITY PROPOSAL MODEL PROMPT
We provide the prompt for the entity proposal model in Fig. [ Notably, ask the model generate the size and position information of the proposed entity, but we choose not to use it during the segmentation and downstream inference, as we notice this information can introduce unexpected hallucination, as the spatial reasoning capability is not strength of our base model. Still, including this information can serve as a self cafffprompting and help generate the final entity proposal. Meanwhile, the size and position information also do not participate in the calculation of the similarity reward; we only use the entity name Eq.
D.9 EG-VQA MODEL PROMPT
We present the prompt of the EG-VQA model in Fig. 9 where we mainly highlight the meaning of each type of visual clue. This can help the model better understand the properties of different visual clues and focus on the corresponding region or local detail when fed with different clue.
D.10 COORDINATOR PROMPT
We provide the full prompt used for the coordinator in Fig. 10 coordinator model in the system prompt and instruct it to plan and review. We instruct the coordinator model to review and double-check if the chain of thought and the answer from the tool VLM are consistent, and ask it to correct the answer when necessary. We also require the coordinator always
We define the behavior of the
24

---

you Published as a conference paper at ICLR 2026
Proundes-Pot VCH Fxonpt
BEYSTEM Pronpts
You nedical Vot assiocant,
the
extra regios clue inager below to the quastion. there alveas
31 Globel agrees
region only proposed by inoge its contidence high, othervisc dicregard Anovez concisely nedically accurately.
Rnueation on Templates
[Qusation] Fizst ourput the thinking proccco in <thinkk </thinks togs and than output the
final anNARE Sanaweck s/unawers
Figure 9: System Prompt for G-VQA Model. We present the system prompt for our grounded VQA model, where we provide information about different types of visual clues and guide the model to focus differently when given these visual clues.
Coordinee con Agent Sysrcws Promp:
You ace ixpprts nedical viauel quest timn-nnseering (VDA) coordinstor Al, Your to dynaniiallly plan and orchestrate the usaqe of apacialized sedical vision coola to accusately snewer el nnfeel questioons based on provided redios! images.
You have acoeza to these toola:
1. "cod propoasl"; identifies sogiona-af-Laterenr
question
2. "segrent segion"; Conasates prosise scgmentation mases mediaal regions
geumpta proposed by "zoi proporaaa""""
3, "ega from regiee"; Serforms visusl question anevering based opeional reglon-of-interes segnentation on nask, the neek will be processed into tiree different vixusl clue typea: 11 Sinary 802 zask for pesitiencl infornation/ 2) Zosn-in viax of local BOI detalls/ 31 Global nank for full
context. Anavez concisuly and nedically acourately.
quection inage then, use "searent, secion" cbtain aemmntaatin napss ezitical nedicul atructures Indlisated by the BATe:
Flrst, 955 *rst propocsl" propece
ROIS
Fiesliy, use "yes fr00 sceson wit 32 vithoat the secennteiicn Basks to Esach an 3ccurate
nedtesl di3gnosie or proviée detalled crintcal arswer. Rerember chasse the nest suitable slue based on the context, either "mask" for binezy "soon, is" for zooned-
foll-trege
"vgu fror_region" vasify your answer, sugardiess
have seqnentation reason ng procees recoened by the soot alagned
not.
klasys veriey
morregperding
SaSNOE, F2x 250 final aBORer DOG
Always choose the neat suitable tool based the current available data. claariy apecify your reszoning for each tool invocssion sten-by-stee and provide concize olinicelly somrate shoeld be sical reasening folloved single <anawee></Ammnee>>>>
final
#inel
Figure 10: System Prompt for Coordinator Model. We present the system prompt for coordinator model. We introduce the overview of each tool and define the general workflow here. The section highlighted in blue defines the coordination and planning behavior, and the section highlighted in green defines the iterative answer review process during inference.
make at least one tool call to the BG-VQA model, the EG-VQA model is more convincing on the medical task, and the coordinator model is only for action planning and answer review.
We also provide the tool schema to the coordinator model during inference, which is defined in Appendix [.13
D.11 EXAMPLE SYNTHETIC ROI PROPOSAL DATA
We provide randomly sampled synthetic data examples in Figs. and 12 The full data will be released upon acceptance.
D.12 MORE CASE STUDY
We provide randomly sampled cases during model inference using CARE-Coord-B in Figs. 14 to 19 We also include example of common failure cases in Fig. 20, where the reason for failure due to hallucination introduced by the coordinator. And the coordinator over-e the answer from the model insist the wrong answer.
25

---

Published as a conference paper at ICLR 2026
Excmple of Synthetic Data
Non-enhencing
Edema
Enhancing Turnrr
"idt:
iiagess/n S101--- Braf819 D01RO2 $71_1-111 p___l______pp000 p110.2n0"
"ccevaraations" [
"Econ" "baman",
"value": "eirace>lAAee thaze enhancing, sen-enhancing Luzora, or eduna the rezicu?"
EEcnn "gpt",
"oslue"; "acc:ioss ("semencation tool"", V"rguuennta"" (Pproppt)"
Vechanncnng [eaas//=]//cccioottteeeell ""s""""""tttt""""""""""""""""""" V"segnctatiion tooli", V"arguentaa """oropp"""; ("non enhancing tunory*: "/////ttteoaactioo (atttiaaaattooonnaantett \"nane"" ("segnenzation, tool)", Marquuenta'l" to (>Ppornt)"s ("edira)" ///etiioos"""
Figure 11: Example of Synthetic Data for Entity Proposal Task. An example of synthetic data for the entity proposal task, we wrap the ground truth entity name in the JSON object.
Example of Synchatie Dara
Input Image
Clavicula Left
"id": 3968,
"Inagee/st oe--Totaleereettaeor dotase:-s07: (415-png"
"concrsoatonn":
"fron": "huran",
"yalue": "cinzgorlala there any abnoralltt in the left claviele reçion this cp
inege?
"fron": "apt",
"oalue": "Gaction>///aand ("eegnactation tooll", V"regumettrl (pprompe)
clavioula [oft\")ss/actiop>
Figure 12:
Example of Synthetic Data for Entity Proposal Task. An example of synthetic data for the entity proposal task, we wrap the ground truth entity name the JSON object.
D.13 TOOL FUNCTION DEFINITION
We provide the function definition for the coordinator model in Figs, 21 to 23
26

---

Published as a conference paper at ICLR 2026
Sxample Synthecis Data
Input Image
"idn; 1276,
"ixage" P0----PAE2---- 25_0044-4_0007..0
"cutvertat.ong
"Eros": "humaa",
"Ve lue": "Cinageonnta there any abnorrality in the liver on this or acan?s
"Erom":: "apt",
"ractiors/) ("eggentattion V'rggnmeees' (Pproppt)
/e/action"""
Figure 13: Example of Synthetic Data for Entity Proposal Task. An example of synthetic data for the entity proposal task, we wrap the ground truth entity name the JSON object.
USER:
Whet lesion located in?
Coordinator Planning:
Step 1:1 will relevant regions interest (ROIs) related the question to localize the lesion identily the
spesific brain
Invoking roi pllpprollellttcc to detect key enstomical regions (e cerebrai lobes) and the lesion.
Calling Tool
Proposal:
<thinic Based on the MRI image the lesion appears be located in the right hemisphere of the brain, it reletively
well-defined with dietinet boundarj, the position in
right quadrent the brain.s/thinke
Enhancing tumor, enhancing tumor-sisnswen <lanswer>
Iteration #2 Coordinator Reviewing:
Stop sog regmontte proposed region obtain precise mask, help anatomical
brain Tobes Invoking segment, region for the "enhancing turor" and "non enheiniing tumor".
Calling Tool Segmentation:
Promat: ('cnhancing tumar*, "non onhoncing tumor"
Confidence: (21 103% 92.28%)
Maak
Iteration #3 Coordinator Reviewing:
Stop With the lasion mask ready, will partorm VQA analyze
using
Insion acouples. Using vqa from region with Blue. typé
"masic
positional
reletive
anatomy
21.03%
Calling Tool Grounded VQA:
scan. Upon examining image, Drop the mask aitusted the rightt herisshheee of the when viewed from this perspective. the lesion appears hyperintense compared the surrounding tissue indicating difference signal intensity. location the lesion relation the standard anstomical Landmarks the brain responsible processing tory information
<thinic provided limegt
prominent
temporal labs, the temporal involved in the comprehension on of meaning memory formation and the processing of semantic memory. </think>>
suggests that iss positioned
Right temporal </onswer>
Iteration #4 Coordinator Reviewing:
Reviewing:
Step (ROI proposel]: identified the key structure se "enhanning turorr and "non enhancing tumor" localize lober determinetion.
Step (Sepmentationi: Segmented the lesion obtain precise masks for positional assessment relative to cerebrai lo
type: mask) analyce the image, which corresponds to the patient's night herisshere on exial MRI Its position enterolateral and inferios, consistent with the temporat lobe.
(VQA
MRI, The
Right temporal
Ground Truth: Right tomporal
Figure 14: More Case Study. We provide a full reasoning process user's question about a brain MRI image here. We highlight the key information from the coordinator in blue and key information from the tool in green.
27

---

Published as conference paper at ICLR 2026
USER:
airepece
Iteration #1 Coordinator Planning:
propose
regiana interest (RCIs) based the
the lung fields
sirepace consolidation would sppear.
Proceeding to RDI proposel.
Calling Tool Entity Proposal:
<thinke Based on the ray image wiil analyze the left side for any sighs airspace consolidation. will look these
incrassed opacity may indicate consolidation, will consider the
areas, sreank/
Fanswer- [Airspace Consolidation
Iteration 2 Coordinator Reviswing:
Step 2: The intial torm was too gonerié, accurately assess alrspace consolidation on the loft side, Jwill sagment anatomical
anelysis the relevant parenchyma. Proceading to segment
Calling Tool Sugmentation:
Prompr: ('left side lung"]
[96.78%]
Iteration #3 Coordinator Reviewing:
Zoom-in
Stop S: will partorm VQA focused on the segmented
lung zoom-in
evaluste parenchymal aisspace consolidation.
Calling Tool Grounded VQA:
<think- The provided chest image
anteroposterfor view thorscic cavity highlighting the lungs heart surrounding strustures Upon examinstion lung
appsers have an area of increesed opacity sempared to the right lung suggesting the presence of airspace consolidation. This indicated by
typicaily air-filied lung fields. heart madiestinal structures are controlly located and the diaphmamm apacars intact. The increased opacity on the left side consistent with fluid cells or other material filingg the alveolar
within
characteristic feature of airspace consolidaton.k/thin
conswer-
84
Reviewing:
Reviewing:
propoeel: Targeted region relevant
question perenchyma
airepace consolidation manileett
Segmentation: Isolated the lung to restrict analysis to the correct sici (mask confidence -85%).
VQA with zoom-in: Focused perenchymal details within the left lung: the zoomed assessment showed region increesed opacity stent filling, supporting airspace consolidation.
Ground Truth: Yed
Figure 15: More Case Study. We provide full reasoning process of user's question about Chest X-ray image here. We highlight the key information from the coordinator in blue and key information from the tool in green.
USER: any anomaly A: No
detectable picture?
B: Yes.
Iteration Coordinator Planning:
Stop 7: Start with global clue essess overoll normalcy, then confirm with segmentation mask neaded. Procesding with VQA using global
Calling Groundded VQA:
<thinkcThe
transverse
clearly
anstomical structures including the vertebra in the center surrounded by soft tissues and organs such the
intestines, The
appears be normal abdominal with ocvvoous of abnormalllies like tumors
masses, vernebre and surrounding bones appear Intact soft tissues da not show any apparent signs of distress or Irrogulanties. The distribution and dansity of the tisques scem consistent with typical
sinifficnt fluid socurnulition
abdorninal anstomy </thinks
canswer- A-janswer>
Iteration #2 Coordinator Reviewing:
Reviewing:
VQA verification; Lised
first evaluate entire abdomen any anomalies, VQA assessment of the global image indicated abrormallty.
deviction. The
Conclusion: No detectable anomaly on this abdominol CT alice.
censwere No Nicaanswrr>
Ground Truth:
Figure 16: More Case Study. We provide a full reasoning process of user's question about body
CT image here. We highlight the key information from the coordinator in blue and key information from the tool in green. This example skipped the entity proposal and segmentation, the question is about global information the image.
28

---

Published as a conference paper at ICLR 2026
USER:
abnormality displayed image?
A: Eczema
B: Poorissis
C: Melanoma
D: Aone
Iteration #1 Coordinator Planning:
Step will propose the key region interest related to the question (the škin lesion) using roippoopaaal This
helps focus subeequent enalysis.
Step segment the proposed
region using segment_region
mask, provide
positionaal information end enable a zoomed-in viewe
Step vea__rom__eeeeeeeeeoooo
the sogmentation mask perform diaghosis the provided
leverading both local lesion details and global skin context.
Calling Tool- ROI Proposal Agent:
Based image the obnormality appeers to pigmented irregularr borders and varying shades of brown. description is consistent melanoma, oczema psoriasis and acne da not typically present with such pigmented lesion--thiikke
Pigmented lesioric/anseen c/nsswee
Iteration #2 Calling Tool Segmentation:
"Pigmentad Lesion
Zoom-in
Confidence: 92.1
Iteration #3 Calling Tool Grounded VQA:
<think The image depicts a
with distinct cheracteristios, the lesion
Birculer
dolll-dfiinddddd border displays
ringingg dari, appears unifform
lew lighter aress towards the centee. The surrounding skin relatively clear with a pinkieh tone indicating the lesion is isolated, These
features are typical of melanocyyic lesion which can include moles or other rypes skin growtha briginating from melanocytas The uniform edges bey canswer C. Melanome <lanswer>
distribution
identifying
abnormailty shown,</thnne
84- Coordinator Reviewing:
Reviewing:
question skin lesion) using proposel then segmented the losion with segment, region to create precise masks of the pigmented lesion, providing positional context and ansibling close inspection).
identiffied the key region
enelysis.
region zoom-In elue the segmented irreularr borders with peripheral projections, and veriegsted brown pigmentation feetures concerning for malignent melanocytic lesion rather
Finally, vqa.
assess dermoscopic fectures. lesion shows asymmetry.
psoriasis,
canswer- C. Melanome clanswer>
Ground Truth: Melanoma
Figure 17: More Case Study. We provide full reasoning process of user's question about digital image here. We highlight the key information from the coordinator in blue and key information from the tool in green.
29

---

Published as a conference paper at ICLR 2026
USER:
condition depicted image lung?
A: Pulmonary fibrasts
BOVIDD11
C: Lung cancor
D: Bronchitis
Iteration #1 Coordinator Planning:
Step
which condition desicted, focusing lung where parenchymal abnormailties (e.s. fe....roun grounc- gloss spacities, consolidations sppeas.
the relevant regions the Xhaat
Proceading ROI propesel.
Calling Tool-
Entity Proposal:
<thinic Based on the image the lung shows signs of incressed opacity and possible consolidation which are
Characteristic cavid11., other
liste not typically
specific radiographic
features, pulmonary fibrosie lung cancer and bronchitis would Likely show different patterne on chest
rey-s/thikk
Lung
Iteration Coordinator Reviewing:
Step 2: Segment the proposed lung fields to localize parenchymal findings that differentiate COVID-19 from other aptions.
Procoading
these
Calling Tool- Referring Segmentation:
Promot: ["Lung"]
Condiaienc: [94.58%
Mask
Iteration #3 Coordinator Reviewing:
Step Use VQA with the lung
characteristic COVID-19
findings (bilateral periheeral ground- plass gparttess@omsolideetnnngg..
select the correct option among the choices.
Proceeding with VQA using the mask for positional emphasis and local detail.
Calling Tool Evidenca--roune Grounded
sthinic The image provided is a chest s-ray showing incressed interstitisl markings and ground-ge glase opecities which are characteristic consistent with involvement patients infected the The distribution ion and pattern opecities Bo not metch the typical presented kome pulmonary fibrosis lung cancer bronchilis on chest
findings cavid-19 preumonis. These
<tthinks
COVID-19</answee
Iteration #4 Coordinator Reviewing:
Reviewing
first used roi, poL_rrposal to icentify the most relevant region for difforentiating among the listed conditions (pulmanary parenchymai, The propesed "lung" the RDI.
Noxt, used segment region to obtain a precise lung mask (cantidence 94, 5355) focus analysis on paronchymal findings across both
lungs.
patterns (e.g.. paripheral ground-glass opacitisuttonsoiddiii The took concluded that the patter consistent with COVID-19 ritherr then pulmonery fibrosss,
Finally,
region lung (clue. type: masi) evaluate
Kenswer-B COVID-19 10an--9e
Ground
Figure 18: More Case Study. We provide full reasoning process of user's question about Chest
X-ray here. We highlight the key information from the coordinator in blue and key information from the tool in green.
30

---

Published as a conference paper at ICLR 2026
USER:
What is the largest organ the picture?
Iteration #1 Coordinator Planning:
Stop 1:1 will propose likeyy regions af-interest relevent to dentiffyng the langest organ in an axial, chest CT, such as the lungs and hear.
Proceeding progosal.
X Incorrect organ proposal
Calling Tool Medical Entity Proposal:
sest-sactional human bady and the largest organ the humen body the Lvag will look for the largest organ in the imaga.s/iikk
<think-Basae Based image identiffy largest organ.
Coordinator Fix Proposal Error
Iteration #2 Coordinator Revlewing:
Step The initial ROI proposai suggested "liver", but this thoracic CT siice where the Proceeding segment reg
the lunge and heart segment
lungs,
occupies largest
Calling Tool Referring Segmentation:
Promet: L'Lungs", "Heart))
V "mask" for size comparison
Confilienc:: (92.17%, 70.64%)
Iteration #3 Coordinator Reviewing:
Mask
Step 3:1 will perform VQA using the segmented masks evaluste which organ in this CT slice. I'll include masks for the lungs, and heart, type leveregee positional
information.
Calling Tool- Evidence- Gronnnddd
<thinic-Thhe provided image is transverse (axial) slice from a scan
the thoracic region. It displays the side the image indicating they are with eir hence appeer lees diense on the sean. freert la centrally located and appears lighter area compared to the lungs surrounded by the darker oegions of the lung fields, The bonees such es density, image provides
within the chest including the heart surrounding
The lungs
appeaa es large Bark areas
visible and appeer white ellowing for the essessment their size and relative positions.
structures
Kanswer Lung Kanswer>
Reasoning each organ found in the image...
Iteration #4 Coordinator Revlewing:
Reviewing:
right lungs I first proposed ROIs; the initial autometic proposal incorrectly suggested "liver" whichh is not present in this thoracio level. corrected
principal organs
lunga heart.
segmented the lung, left lung, and heart to compare their extents within slice Using the mesks es positionallsize clues, lungs occupy the largest amaa bilaterally compared with the heart.
canswar- Lung cinswwrs
Ground Truth:Lung
Figure 19: Case Study. We present the complete reasoning trace for CT question. Key information from the coordinator is highlighted in blue, model reasoning in green, and each model's final answer in yellow. Notably, during review, the coordinator corrects an initial error from the entity proposal.
31

---

Published as a conference paper at ICLR 2026
USER:
What abnermal CT scan?
iteration #1 Coordinator Planning:
Step 1:1 will propose regiens ofvinterest relevant to the question (abdominal CT) locate potential. kddeyaluustors, gel gilinladder, liver, spleen,
abnormalities
free airiiluid
Proceading to ROI proposal.
Vinitial proposal is correct
Calling Agent- Entity Proposal:
ethinkc> Based on scan image to identity any abnormal the scan shows abdominal ragian with various structures, will look for any unusual masses fluid collections or other iniegllariiess </tnink>
[dddays, pancrens)
Iteration #2 Coordinator Reviewing:
Hallucination introduced
Step Segment the proposed ROI (dddneys pancrees) to potential ebnormalites
sp hycronepphooiis
Calling Agent Soggentation:::
Prompt: ('kicneys', "pancrees"]
Condidence: 772.51%, 20.3556]
Iteration #3 Coordinator Reviewing:
Zoom-in
Stop 2: Parform VQA using the kidney mask to detect
specific abnormality liy (e.g. rereal stores) with zoom-it view local high-sontrast fool. initial VQA result suggessts pantrestic maes, but this seems inconsistent slice prominently
C = 20.35% Tc
Drop the mask
enhanced kidneys where calcilications may be present.
will verily using a global context view without relying the kidney meok resesess the abnocshality.
Calling Agent- Grounded VQA:
Insist focus on Kidney of
<<
upon examination the image rovelis-sgvend abnormalities. Firstly there is mociceeble mass located in the pancrestic which appears to be causing displacement and compression of adjacent structures, the mass has heterngenenus dansity indicating varying tissue composition within Additionally, thare is evidence of surrounding
sthinic-Tho provided
bissue infiltrtiion
possible involvement nearby organs, kidneys are visible appear be normal size and shape position is slightly altared due to the mass effect, the spinal column and other bony structures seem intact without any abvicus signs metastasis other abnormalities. Overail,
significant
pethology that pequires turther clinical
comelation possibly blopay for definitive diagrosis. </thinks
sanswers Pancreatic mass s/answere
Correct reasoning and answer from G-VQA
Iteration #4 Coordinator Reviewing
The VQA result sugjests pancrestic mees, seems inconsistent enhanced kidneys where calerticatios may be present. will verify using a global context view without relying an the kidney mask to reassess
promitently shows contrast
abnormality.
X Insist on hallucinated answer
Calling Agent- Grounded VQA:
kyy abnormalitiss. Firstly, there is noticeeble mass located the pancrestic region which appears to be ceusing displacement and compress on of adjacent has heterogeneous density indicating verying tiesue composition within Additionaily, there eVidence of surrounding appear
<thinie-The The provided
image
upon examination image
structures the
bssuu infiltretion possible involvement nearby organe, kidneys are
normel size Shaps but position relative to the mass suggests that the mess everting pressure on them. the overall lsyout and deneity variations within the mass requires further medical evalustion determine property potential impac the patient's
suggest that abnornal
h/thnt
sanswere Pancreatic mass s/answer
Correct answer from G-VQA with different clue
Iteration #5 Coordinator Reviewing:
Reviewing:
first prosesed the kidneys as the most relevant ROIs for an abdominel. CT when asked about an abnormality. seggnented the kidnneys focus eveiustion polentisl urinery tract pathology.
VQA outputs (both zoomed kidney and global context repestedly susgested "pancreatic mass, Which inconsistent with akial slice that prominently displays contrasi-entanneed kidneys and punctate hyperdense foci typical of calculi. discrete hyperdense within collecting dystems,
Reassessing the
with attention the kidneys stent with renal celculi rather than a pancreatic massi
X Insist on hallucinsted enswer
Kanswer Renal Calculi
Ground Truth: Poncreatic adenacarcinoma..
Figure 20: Failed Case Study. We provide full reasoning process of user's question about a CT image with a failed result here. We highlight the reason for failure during the reasoning chain, where the G-VQA model gives a correct answer, but the coordinator model insists on hallucinated answer. We use the red dashed arrows to illustrate how the hallucination propagates during the inference.
32

---

Published as a conference paper at ICLR 2026
Entity Propesal
Function
Schema
"Lype"; "funcLion", "Sunction"; "nane": "rro_proppsal"" "description": "Proposes medically regions (entity name] the image based the question.",
"paramatera";
"type": "object",
"properLies"; ("type": "string"), "inage @image_u=l";
"question": ("type"; "atring")
1, "required"; ("image_uri) url", "question"),
Function Definition for Entity Proposal Model We provide the full definition of the function schema of the entity proposal model used during coordination. We explain in detail the definition of each parameter here.
Figure 21:
SegmenLa PuncLion
Schema
"Lype": "function", "function";
"nane"; "segment__egiion
"desoript ption"; "Segments specf ric
regions from med! cal image gl ven textual
prompt."
"paramecers" "type": "object",
"properties": {
"image_uri" url"; ("type": "string"),
"prompi";
"type": "array", "items": { "type": "string"
"description"; "List of medical entity (e.g., lung, liver)
},
"raquired"; red": ["Imago_u-l1 "M"", "prompt"),
Figure 22: Function Definition for Segmentation Model We provide the full definition of the function schema of the referring segmentation model, which accepts two inputs and outputs the segmentation mask along with its confidence.
33

---

"iLems": "type": "string" Published as a conference paper at ICLR 2026
G-VOA Punction ton Schena
"type": "Function",
"Function":
"name"; "vgo from region",
"description": "Performs medical visuel question answering based on optional visuel ROT).",
image (either
indicacor, binary
"parameters":
"type": "object",
"progerties";
"image_uza" uzl*: ("type": "string"),
"question"; ("type"; "string"),
"maak url":
"Eype": "array",
"icoms": "type": "string"
"deacription":
hinary path segmented region"
"maak name":
"type": "array",
"dascription"; "List of entity name of the mask, e.g., 'lung', "liver',
"uubor'
"mask confidence::
"Lype": "array",
"i "itams": "type": "number", "minisun": 0, "maximum": 100
"dsacription": "Mis of con't dence score each mask, between
*clue_ Lype:=__p_::
"Eype": "string",
"enun": ["global", "mask", "2con in"),
Steacription" "Type of visual alve to provide: 'global' ' for global hinary 'zoom
context indicator,
zoomed-in image. Global
highlights ovezall centext, binery highlights positional information and zoon-in view highlighta local decaila."
"required": ["image_uri] url", "questicn", "clue "clve_type")
Figure 23: Function Definition for G-VQA Model We provide the full definition of the function schema G-VQA model. Note that we only set the input image, question, and clue type to be required parameters, as when using global clue type, there is no need to use other parameters about the mask.
34

---

Published as a conference paper at ICLR 2026
USER:
Which part of the body does this image belong to?
Ibenaiin #1 Coundinwtor Plaoning:
Step 1: will propose relivant anatomical regions based on the quediin understard the oveall contest of the image.
ROIt.
Calling Tool Medical Entity Proposal:
ethinibhuused the
crove-tent fonal witu the abdraen.
the kidneys and other abdresina) structures suggest thix the sbilssen. </dl inio
<answer> ""intasinn": "idneyy, "spineess//annnsss
Calling Tool Evidence Grounded VQA:
<thinlothe Image is a tronoverse (axia1] slice froe et scan of the abdowen, vartous anatorical structures can be Ldentified Including the vertebral colurn at center bottow of the lange which appears as shite circular bone structure. flanking verbebral column are the kidseys which are as synaetrical hean-ahaped structures the kidwes wetellnd presense sar lmops
sarroudding apparing as arou, ifiahle by sunrounding ongans providing contrast that helps dir ireetttnigg presence spectric organs
distinuuleablle cortex and lares intest
structures, tmage
regfon.c/thine>
comnee>>abdomee /annswrr>
Glssl Vins.
Heration #3 Coordinator Reviewing (Final):
Ressoring usager
poprued iklly reieantt incieetiilnngg bodly part (intecinn: citney, spine) becatiee mecognningg ley whaiomme helps debeeiin the region Give the queiiar & douut cwnal ladly region, lusel VQA globael cunder la aoans the full inae rallor the focuning un single musk The VQJA ilso lified an anial CT sioe shawing vatiebra bodyy balataral kidness and bowal hops, confeming the abdominal legion.
response
canswers Abdomen (ettomen-/jannnnn Jannwer>
Does reasoning trace make sense?
X
Figure 24: User Interface of Human Evaluation. We provide the full user interface of our human evaluation.
35


Reviews:
[{"review_id": "umHEapFyNd", "reviewer": "Reviewer_G9oY", "summary": "The paper introduces CARE, an agentic framework for medical VQA that explicitly mirrors clinical workflows. CARE decomposes the task:\n(i) a medical entity proposal VLM (trained with RLVR/DAPO using verifiable rewards), (ii) an entity-referring segmentation module (SA-Med-2D) providing pixel-level evidence (ROI masks/boxes), and (iii) an evidence-grounded VQA (EG-VQA) reasoner that answers using one of three evidence views: Zoom-in, Mask, or Global (a full-context “no localization needed” indicator).\n\nTwo operating modes are studied: CARE-Flow (deterministic pipeline) and CARE-Coord (a planner–reviewer agent that selects tools, iterates, and performs answer reviews). Across four benchmarks, the 10B CARE-Flow establishes strong new results; CARE-Coord further improves performance, surpassing prior 32B baselines. However, it's heavily dependent on using relatively bigger proprietary models such as GPT-5.", "questions": "Also refer to **Weaknesses**.\n\n### **Baseline Training Exposure and Evaluation Fairness**\nFor each baseline reported in Table 1, please explicitly indicate whether the model was:\n(a) fine-tuned on any of the in-domain datasets (OmniMedVQA, VQA-RAD, SLAKE),\n(b) fine-tuned on other medical VQA datasets, or (c) evaluated zero-shot using public checkpoints. This transparency is critical to disentangle architectural advantages from training-data advantages. A compact supplementary table summarizing this information would substantially improve interpretability and fairness.\n\n### **Quantitative Evaluation of Accountability**\nBeyond qualitative case studies, have you considered measuring accountability quantitatively (e.g., the overlap between predicted answer entities and the segmented ROI, or expert ratings of reasoning trace correctness)? Even a limited-scale human evaluation or automatic alignment metric would strengthen the accountability claim.\n\n### **Coordinator Error and Review Conservatism**\nIn Figure 18, the coordinator overrides a correct specialist answer, illustrating a potential hallucination at the review stage.\nHow frequently do such coordinator-induced reversals occur across the evaluation set? Have you explored conservative review heuristics (e.g., vetoing coordinator changes when specialist confidence is high) to mitigate these errors?\n\n### **Evaluation Stability and LLM-Based Scoring**\nFor open-ended questions evaluated by GPT-4-class models, have you measured inter-rater agreement (e.g., multiple LLM or human graders) or compared results with deterministic metrics like normalized exact match? Providing a measure of scoring variance would improve confidence in reported accuracy differences.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper's primary strength lies in its principled and clinically inspired architecture. The decomposition of medical reasoning into discrete, specialist-driven stages [hypothesize (entity proposal) → localize (segmentation) → reason (grounded VQA)] is a significant conceptual advance. This design directly addresses the \"black box\" problem by providing explicit, pixel-level evidence for each decision, thereby enhancing both interpretability and debuggability. Empirically, the work is robust, demonstrating strong performance across multiple datasets and supporting its claims with thorough ablation studies that dissect the contribution of each component, from the matching strategy to the agentic coordinator. The use of reinforcement learning with verifiable rewards (RLVR/DAPO) is a sophisticated technical choice that aligns model incentives with checkable signals, avoiding the need for costly human preference data. Finally, the paper is commendably detailed in its implementation description, providing prompts and tool schemas that greatly exceed the reproducibility standards of many contemporary agentic AI papers.", "weaknesses": "### **Baseline Comparison Fairness and Transparency**\nThe validity of the primary results in Table 1 depends critically on understanding each baseline’s training regime. It remains unclear which comparison models were fine-tuned on the same in-domain datasets (OmniMedVQA, VQA-RAD, SLAKE) and which were evaluated zero-shot. It is indeed unjust to call the columns in-domain and OOD solely based on your configurations and then compare them against other models. The authors should explicitly disclose the training exposure for every baseline—ideally in a small supplementary table—and group results into “Fine-tuned” vs. “Zero-shot/Unknown” categories. Without this disclosure, the magnitude of CARE’s improvement cannot be properly attributed to architectural design versus training-data advantage. The current Table also needs to be modified or at least clearly states the differences and point to the more complete Table which would be in the appendix.\n\n### **Reward Function Analysis and Sensitivity**\nWhile the EG-VQA reward is ablated, the Entity Proposal reward R_{\\text{Entity}} = R_{\\text{sim}} + R_{\\text{count}} + R_{\\text{repetition}} + R_{\\text{format}} lacks empirical justification for its composition.\nA component-wise ablation analysis or a reference to another paper that is using the same reward with clear ablations would clarify which sub-rewards most influence stability and downstream performance. This is particularly important since R_{\\text{Entity}} governs the quality of the entity grounding that underpins the rest of the pipeline.\n\n### **Isolating Framework Contributions (CARE-Coord-Open Variant)**\nThe performance of CARE-Coord depends in part on its proprietary coordinator (GPT-5). To distinguish architectural benefit from raw model capacity, the paper should include a CARE-Coord-Open variant using the strongest open-source VLM employed elsewhere (e.g., InternVL3-8B). Reporting this gap would isolate how much of the gain stems from the agentic framework itself versus the closed-model advantage.\n\n### **Misleading Parameter-Scaling Visualization**\nFigure 1(e) plots proprietary models with unknown parameter counts on a “Billions of Parameters” axis, inadvertently implying knowledge of their scale and suggesting a false comparison. The authors should redesign the plot; for instance, use a separate panel, shaded region, or qualitative marker to avoid misrepresenting scaling relationships.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AQ9z7PhcQe", "reviewer": "Reviewer_ojM1", "summary": "This paper proposes CARE, an agentic framework that mirrors the clinical diagnostic workflow. Instead of treating medical visual question answering (VQA) as a monolithic black-box task, the authors decompose it into interpretable sub-steps: (1) a medical entity proposal stage, (2) a segmentation model to localize regions of interest (ROIs), and (3) an evidence-grounded VQA model that reasons over the full image along with different evidence views (zoom-in, masked, or global). Additionally, an LLM-based coordinator performs planning, reasoning, and quality control over these steps, improving performance and interpretability.\n\nOverall, I find this to be a promising and well-motivated direction. The framework reflects clinical reasoning in a meaningful way, and the results appear solid across multiple datasets.", "questions": "- Reproducibility and openness. Since the paper claims superior performance, open-sourcing the models, weights, and codebase would be critical for community validation. Without this, it’s difficult to verify the robustness and generality of the framework.\n\n- Justification for reinforcement learning. The ablation study supports the inclusion of RL fine-tuning, but the paper should better articulate why reinforcement learning is necessary for both the entity proposal and the VQA stages. From the description, it appears to combine symbolic and correctness-based rewards; please clarify why supervised fine-tuning (SFT) would not suffice.\n\n- Demonstration of diagnostic effectiveness. If the paper aims to position CARE as a clinically meaningful diagnostic model, the current visual examples (e.g., Figure 3) feel too trivial. The task shown (“Which organ has the largest area?”) is not representative of real diagnostic challenges and could likely be handled by standard detection models (e.g., YOLO). Consider adding more realistic examples—such as differentiating between cancerous vs. non-cancerous lung images—and illustrating how the coordinator or CoT reasoning contributes to that decision.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Well-motivated workflow design. The authors tackle the core problem—controlling hallucinations in large medical VLMs—through a structured, interpretable workflow rather than pure end-to-end training. This decomposition aligns well with clinical reasoning and is conceptually elegant.\n\n- Coordinator design. The introduction of an LLM-based coordinator for planning and quality control is intuitive and methodologically sound. It also adds an additional layer of reliability that most prior medical VQA systems lack.\n\n- Strong empirical performance. The reported results show consistent improvement over prior works, demonstrating the practical value of the agentic workflow.\n\n- Extensive experimentation. The authors conduct a large number of experiments, including multiple baselines and ablations, which adds credibility to the study.", "weaknesses": "See questions", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9c6LW0SWIZ", "reviewer": "Reviewer_fS8R", "summary": "The paper proposes a framework for multi-modal reasoning specifically designed for medical VQA tasks. The task is split into three separate stages: (i) identifying medical entities relevant to the question, (ii) segmenting those entities in the image, (iii) visual reasoning using different views generated from the segmentations. They further propose the inclusion of a coordinating agent for planning and review. They report in-domain and out-of-domain improvements over comparable methods and include a comprehensive ablation study evaluating various components of their method.", "questions": "I would like to see some statistics on how often the final answer is overwritten by GPT-5 and the accuracy of these overwrites. \n\nWhat is the binary modality token?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper proposes a novel framework for agentic visual reasoning with tool usage in medicine. They identify core workflow steps helpful for medical visual reasoning and train dedicated agents for these tasks. While the core concept of tool-supported visual reasoning exists for generalist VQA with similar training strategies, the originality and significance of this work arises from a well-executed specialization for the medical domain that I can see to be easily built upon.  \nThe method, modules and training strategy is well motivated and thoroughly evaluated and ablated.  \nThe manuscript is well structured and explanations and figures are clear.", "weaknesses": "1. ##### **Missing related work:**\n\n   The paper claims to beat state-of-the-art on VQA-RAD and SLAKE, however, there are multiple methods with better performance on VQA-RAD and SLAKE not mentioned in the comparison, e.g. \\[1,2,3\\]. These works should be included for a complete contextualization of the work and the claim adapted.\n\n2. **Unclear contribution of the coordinator**\n\n\tIn figure 3 and 18 we can see that the coordinator (GPT-5) is able to overwrite the answers of the medical entity proposal module as well as the final answer of the system. Especially on the OOD dataset, where GPT-5 alone outperforms CARE, I am wondering how often the coordinator makes use of this power, essentially circumventing the contributions of all the proposed modules and degenerating to an API call to ChatGPT (albeit with the input of the CARE modules). A different reading of the OOD results would be that GPT-5 reaches an accuracy of 62.20 but if we include all the info from the CARE pipeline, GPT-5 becomes worse.\n\n\\[1\\] Yu, Ting, et al. \"Fine-grained Adaptive Visual Prompt for Generative Medical Visual Question Answering.\" *Proceedings of the AAAI Conference on Artificial Intelligence*. Vol. 39\\. No. 9\\. 2025\\.  \n\\[2\\] Cui, Hejie, et al. \"Biomedical visual instruction tuning with clinician preference alignment.\" *Advances in neural information processing systems* 37 (2024): 96449-96467.  \n\\[3\\] Lin, Weixiong, et al. \"Pmc-clip: Contrastive language-image pre-training using biomedical documents.\" *International Conference on Medical Image Computing and Computer-Assisted Intervention*. Cham: Springer Nature Switzerland, 2023\\.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)