ExGra-Med: Extended Context Graph Alignment for Medical Vision-Language Models
Authors: Duy Minh Ho Nguyen
Nghiem Tuong Diep
Trung Quoc Nguyen
Hoang-Bao Le
Tai Nguyen
Anh-Tien Nguyen
TrungTin Nguyen
Nhat Ho
Pengtao Xie
Roger Wattenhofer
Daniel Sonntag
James Zou
Mathias Niepert
Venue: NeurIPS 2025 poster ()
Keywords: multi-modal LLM
multi-modal learning
representation learning
healthcare
Primary area: deep_learning
TL;DR: a new learning scheme for multi-modal LLM, LLAVA

Abstract: State-of-the-art medical multi-modal LLMs (med-MLLMs), such as LLaVA-Med and BioMedGPT, primarily depend on scaling model size and data volume, with training driven largely by autoregressive objectives. However, we reveal that this approach can lead to weak vision-language alignment, making these models overly dependent on costly instruction-following data. To address this, we introduce ExGra-Med, a novel multi-graph alignment framework that jointly aligns images, instruction responses, and extended captions in the latent space, advancing semantic grounding and cross-modal coherence. To scale to large LLMs (e.g., LLaMa-7B), we develop an efficient end-to-end training scheme using black-box gradient estimation, enabling fast and scalable optimization. Empirically, ExGra-Med matches LLaVA-Med’s performance using just 10\% of pre-training data, achieving a 20.13\% gain on VQA-RAD and approaching full-data performance. It also outperforms strong baselines like BioMedGPT and RadFM on visual chatbot and zero-shot classification tasks, demonstrating its promise for efficient, high-quality vision-language integration in medical AI.

EXGRA-MED: Extended Context Graph Alignment for Medical Vision-Language Models
Duy M. H. Nguyen 1,2,3
Nghiem T. Diep 1-
Trung Q. Nguyen Hoang-Bao Le
Tai Nguyen Tien Nguyen 45
TrungTin Nguyen 6.7 Nhat Ho
Pengtao Xie
Roger Wattenhofer James Zou 13
Daniel Sonntag 18t Mathias Niepert 2.3
German Research Centre for Artificial Intelligence (DFKI,
2 Max Planck Research School for Intelligent Systems (IMPRS-IS). University of Stuttgart, University Medical Center Gottingen, Max Planck Institute for Multidisciplinary Sciences,
ARC Centre of Excellence for the Mathematical Analysis of Cellular Systems, 7 School of Mathematical Sciences, Queensland University of Technology, University of Oldenburg. 9 University of Texas at Austin,
10 University of California San Diego, MBZUAI, 12 ETH Zurich, 13 Stanford University
* Co-second contribution Co-senior authors.
Exgra-Med
Abstract
satte-of-the-arr medical multi-modal LLMs (med-MLLMs), such as LLAVA-MED and BIOMEDGPT, primarily depend on scaling model size and data volume, with training driven largely by autoregressive objectives. However, we reveal that this approach can lead to weak vision-language alignment, making these models overly dependent costly cnstruction-followiig data. To address this, introduce EXGRA-MED, novel multi-graph alignment framework that jointly aligns im- ages, instruction responses, and extended captions in the latent space, advancing semantic grounding and cross-modal coherence. To scale to large LLMs (e.g., LLaMa-7B), we develop efficient end-to-end training scheme using black-box gradient estimation, enabling fast and scalable optimization. Empirically, EXGRA- MED matches LLAVA-MEL achieving a 20.13% gain on VQA-RAD and approaching full-data performance. It also outperforms strong baselines like BIOMEDGPT and RADFM visual chatbot and zero-shot classification tasks, demonstrating its promise for efficient, high-quality vision-language integration in medical AI.
performance using just 10% of pre- pre-training data,
1 Introduction
Generic Multi-Modal Large Language Models (MLLMs) such as GPT-4V 7 LLaVa [57], and Next-GPT (97 unify text, image, and audio processing for tasks like captioning and visual reasoning. cntructioo-followinn (IF) data [60] which involves complex, often multi-turn interactions grounded in image content [8] In the medical domain, specialized IF datasets, including medical images, clinical notes, and diagnostic criteria, have been curated adapt general-purpose MILMs while leveraging their pre-learned knowledge and minimizing training costs [98]. For example, VA-Med (50) samples 600K image-text pairs from PMC-15M [106), using GPT-4 to generate around 60K multi-modal IF examples. The training involves two pretraining steps: (i) aligning vision encoders and language decoders via projection layers, and (ii) jointly training the model (excluding the vision encoder) medical IF data using an auto-regressive objective. The resulting model then fine-tuned for downstream medical tasks.
key component training MILMMs
Following the above approach, most later works have focused on scaling up the amount of medical IF data 98
30 or increasing the model size by incorporating larger vision encoders language
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

decoders 196 39 while relying on the same standard autoregressive learning scheme. Contrary to this, we question the effectiveness of autoregressive objective functions when learning medical-MLLM with IF data. Surprisingly, our findings reveal that autoregressive learning is highly data-hungry during pre-training, i.e., without sufficient medical IF samples, model performance plummets for downstream tasks, even after fine-tuning- To illustrate this, we pre-trained LLAVA-Med using only 10% of the data and compared it to the version trained on 100%. Both models were fine-tuned on two medical visual anestion-answerte performance on open- and close-ended questions are compared. The results show a dramatic decline: from 72.64% 52.39% on VQA-RAD (Figure I and from 64.06% to 56.15% Path (Table [. This underscores the instability of medical-MLLA. trained with autoregressive methods and highlights the problem that these methods require the curation of enough medical IF data to achieve satisfactory performance.
tasks VQA-RAD 48 and Path VQA [3] and their average
To address the limitations of autoregressive train- ing under limited cerrrcctio-foolowinn data, we propose EXGRA-MED, novel multi-graph align- ment framework that strengthens cross-modal un-
Downstram Performance on VDA-RAD
70
derstanding in multi-modal large language models (MLLMs). At the core of our approach is the con- struction of three modality-speciff graphs: for visual features extracted by a vision encoder, and two for different textual variants of the in- struction. These graphs represent semantic rela-
65
Hed
Mes
60-
(LAW (general)
MesiMoE (PhI))
Meditoo (Stable LM
55
BloMeoGPTE
BloMed-CUP
tionships within and across modalities, and we formulate a combinatorial multi-grap alignment problem learn consistent triplet-level associa- tions between the image, its instruction, and a semantically enriched variant. This alignment ob- jective is jointly optimized with the autoregressive language modeling loss, enabling the model to en- hance semantic depth, coherence, and instruction- following ability. To generate the enriched in- struction variant, we use a frozen LLM (GPT-4 7 to produce a contextually extended version of each instruction that highlights key concept rela- tionships without altering the original intent. The vision encoder and language model (LLaMa (93)) then process the image, instruction, and extension independently to produce node embeddings used in the alignment step. Unlike naive data augmenta- tion, our use of GPT-4 enriches supervision and facilitates fine-grained graph-based correspondence learning across modalities (Figure 3).
10
40
100 Instruction-Tuning Percentage
70
Figure 1: Our EXGRA- EXGRA-MMED versus LLaVA-Mon across varying instruction-follwwii (IF) pre-training data sizes, highlighting the data-hungry behavior of rugo-segressive modeling. Both models are fine- tuned on the same VQA-RU RAD training set after the pre- training stage at each IF rate. At 100% IF pre- training. ExGra-Med and LLaVA-Med are bench- marked against other state-of-the-an models, all fine- tuned the same VQA-RAD training set (except GPT- 4, which is evaluated without fine-tuning). Circle ra- dius represents the number of model parameters.
Our method stands apart from prior multi- alignment approaches for LLMs 772
16 in
two key ways. (i) Instead of merely learning projection layers between frozen vision and language models, we train the LLM directly via multi-graph alignment framework. (ii) We also extend pairwise contrastive learning by integrating global structural graph constraints, enabling alignment not just between individual image-caption pairs but across entire datasets. This graph-based design captures both feature and relational consistency, critical for handling similar entities in medical data. While multi-graph alignment is typically non-differentiable (80) and computationally intensive (74) we address these challenges using implicit maximum likelihood estimation [70] [63]. This enables efficient gradient- based training over large LLMs (c.g.. LLaMa-7B) using barycenter graph [8 for alignment, allowing our model to scale effectively while preserving strong alignment performance.
In summary, we make the following key contributions:
We reveal the data- data-demgn nature of autoregressive modeling in pre-training medical- MLLM (LLaVa-Med), showing that insufficient caarructionfollowwiii data leads to signifi- cant performance drops on downstream tasks, even after fine-tuning.
We introduce a new multi-graph alignment objective that establishes triplet correlations among images, their pnstruction-followinn context, and their enriched versions. Additionally, we developed an efficient solver for training with L.LMs and outlined theoretical properties related to distance and the shortest path in the geodesic space of multi-moda graphs.

---

We empirically demonstrate that using small amount of pre are-training data, EXGRA-MED can achieve performance comparable to La-MMMe trained on 100% data. Additionally, when trained on larger datasets, EXGRA- MED outperforms several state-of-the-ar med- ML..Ms and multi-modal pre-training algorithms across three Medical VQA tasks, medical visual chat, and the average zero-shot image classification performance on 23 datasets.
2 Related Work
Medical Multi-Modal LLMs. Recent developments in medical-MLLM like Biomed-GP? [04].
MedFlamingo [64, Med-Dr [00, LLAVA-Med (50), and Med-Pal.Ms 83 94 are transforming
healthcare by integrating diverse data types and scaling medical instruction data. Biomed-GPP excels with multiple biomedical modalities, MedFlamingo focuses on few-shot learning for medical visual question answering, and LLAVA- leverages large-scale biomedical image-text pairs for improved performance. Commonly, these models emphasize scaling medical instruction data and increasing model parameters to enhance accuracy and applicability in real-world medical scenarios. In contrast, our approuch examines the widely used autoregressive pre-training algorithms and demonstrates that incorporating enriched context multi-graph alignment of existing instruction samples can significantly enhance Medical-MLLMM performance without requiring larger models or extensive datasets.
Visual Instruction Tuning. Visual instruction tuning techniques aim to bridge the gap between frozen Maiion- models and frozen LLMS trained on unimodal data, enabling them to work effectively in molti-modal context. These methods involve (i) learning a multi-layer perceptron (MLP) layer to map embeddings from the vision model to the language model as LLa [57], VideoLLM [16]: (ii)
using adapter-based adjustment as LLaMa-adapter 105., Voxposer 35 or (iii) learning multi-modal
perceiver by gated cross-attention 9 or Q-Former as BLIP-2
51 Pre-training algorithms to
train these models can be combined with both auto-regressive and contrastive learning 772 102) or
image-text matching as in [12. Our algorithm differs from those by focusing on directly training LLMs rather than lightweight projectors. This requires fast solver capable of efficiently handling forward and backward passes through large-scale LLMs with extensive parameters.
51).
Vision-language Pretraining Algorithm. Pre-training algorithms commonly applied for vision- language models, like CLIP 777, employ various strategies. Among these, generative approaches are widely used, including masked prediction in language models [24 85]. or autoregressive algorithms that predict sequential text LLMs [57 [OS]. Another prominent direction focuses on discriminative methods, which learn contrastive distances between image-text pairs (58) 102 45), optimal transport
15 68, , or impose clustering constraints (72). Toward multi-modal learning across three or more modalities, there also exist works such PAC-S(81), GeoCLAP (46), and IMAGEBIND (26), extending the CLIP or InfoNCE (71) to align embeddings across multiple modalities simultaneously.
formula- tion across cross-domain graphs. While LVM-Med [22 is the most similar to our approach, targets aligmment within vision tasks, whereas we align images, instruction-follont following data, and extended contextual information. We provide a more comprehensive comparison between EXGRA- and related works in the Appendix.
Our function departs from these by generalizing them into a combinatorial
Scalable Multi-Graph Alignment. Graph alignment across K domains (K 2 3) is highly computa- tionally intensive. Current methods, such as multi-marginal optimal transport [54] 75), Wasserstein barycenters [69]. and adjti-adjacency matrix assumptions [13 89] relax the problem but are limited to small-scale tasks and require multiple solver steps, making them inefficient for LLM training. In contrast, our algorithm leverages heuristic solvers (90 [0] and modern gradient estimation techniques for black-box optimization [70] 63) enabling scalable and efficient performance for large language models. A deeper analysis of this factor discussed in the Ablation study.
3 Multi-graph Alignment Learning
We denote the vision encoder, projector, and large-language model (LLM) models are fo()). h.).), (.), respectively. Figure 3 illustrates parameters for these models by solving a triplet alignment between modalities in instruction tuning data. Below, we summarize the notations used before describing each component in detail.
EXGRA-MED algorithm, which learn
Notation. Given any tensor T (Tij...) and matrix M = (MR.), we use 8) M to denote the
consor-matri multiplication, i.e., the matrix (Ek, Tes.N.AM,,,)i, Given Y [y1, 32, .... yNl
3

---

RNxd, we define EY) = Σ= Di E R'. Moreover, we define the matrix scalar (or inner) product associated with the Frobenius norm between two matrices M = (Ma,j) and N (Nv,)) as (,,,,e, Le., (M,N) = Ei, Mi,NNa, We write [M] = 2 M) for any natural number M.
3.1 Extended context enriched medical instruction following data
Recent research has demonstrated that incorporating longer context significantly enhances LLMs' ability to process complex inputs and improves Salruction-following by retaining more relevant information [59 [1) Building on this insight, we expand medical instruction-folloving data by generating contextually enriched paraphrased versions of existing samples, offering complementary perspective to the original dataset. There are two key motivations for incorporating both original and extended captions in our multi- gralig alignment framework. (i) First, aligning with original captions preserves precise, domain-speen specific details, while extended captions enhance semantic richness, leading to more robust image embeddings. (ii) Second, this approach helps the LLM generate contextually rich yet semantically consistent responses, improving alignment across diverse linguistic forms (Table G).
In partieular, a typical instruction sample includes (X., [X.. x.),... X x where X, is an input image, X question, and X an answer round / in multi-round L of a conversation. In the medical domain, most of the questions are generic, and the information answer usually covers the question, so we only focus on extending the answer X.. We leverage the GPT API with a prompt to form an extended context for each X by:
You identify insues in blonedion ansverx
versions shille (omplnng conbent
and genezate
Constion What type of tincue is chown in the image?
acamag:
Vie inace shows hiscopatholon'
ce section of en endonasell nese, whichh tho casal cavity.
tlesee sarple taken from inside
Extended Answer: The Lnage desicta a histopathoiogten pection of an endo- nasel mase, whilch typicslly conciatx wnithellal ailsoee neoplastic process. in taken froo saael covity and nay chow fectures ouch as collular atypia or inflamsa Lioe, depending on the underlyyng patholoay.
X de GPT (X) g: X',, prompt), VI E [L]. (1)
indicating sarple
The details for prompt are presented in the Appendix. In short, we ask GPT to provide additional explanations for concepts that appeared in the original answer X, while keeping the content consistent. An example output for is illustrated in Figure 2 and Table 8 It's worth noting that other frozen LLMS like Gemini are also valid in our method (Table 4
Illustration for creating the ex- tended context instruction-folle folloowin data powered by GPT-4o.
Figure 2:
X! ar
3.2 Multi-graph construction on vision-language embedding
Each image X. E 33x HxW where (H, W) are the original spatial dimensions, is divided into sequence of visual patches U [ with N = (H x W)/U with U the patch size. Using pre-trained ViT model fe, we extract patch-wise features as V fU) E N and apply another projector to map it into the projected embedding Z = h V) ) E R xd We then pool the features from the image patches to define global description as Z, EE (Z) E Rd For each language input x! E (X!- X with c E {a,ae), we assume it has M tokens, i.c., x! = و 1 E RM and feed it into the LLM model to extract set of embedding Z 90 [xx 1 ||, Mxd. We subsequently concatenate all multi-round L in each single instruction tuning to define Z.- = T, E EZZ which collects average text embedding of original answers (c a) and their longer- context extended versions (c - ae) respectively. Though we adapt simple average pooling feature mechanisms, it remains an effective approach with a clear observed margin of separation between the distinct distributions (Table in the Ablation study).
Given a batch size of B instruction-tuning samples, we now construct three graphs G. = (Vo, E.). G. = (Va, E, and 9a. (Vae, Eac) representing for visual image features, text embedding encoded by L.LM for original answers and their extended context embedding extended by GPT. Specifically, for each triplet pair {x!(²) [x]]()) [X] (*) }R, (k [B]), we add node representing x(!)) to V., a node for (x(](k) to Ve. and finally a node for [ (k) to Voe- This results in ae). We equip node-level feature matrices for these graphs using their embedding computed above, i.c.,
set of nodes V. { .... x(()). J: V. =[x]] .... (B) for each
F. = (2(2) .... z.8²). F. (z (1) ...., z.. The edges for E,, E, afterward can be created through the k-nearest neighbors algorithm given the feature node matrices F,, F.. Finally, we can run message- paasaigg network (.).) on three built graphs to learn richer node representations.

---

This approach has proven effective for representation learning 91 42 resulting in aggregated feature-node matrices (Z.
(B))
ma (F,, Es), with s {u, a, ae).
Forward Backwarrd Pass
Multi-Graph Alignment
DC DM0
2000 Daa
Projector
Ap
Lerge Language Model
Vision Transfomer
Vacee (R.)))
(fare( (f.wr)
Gasrded Anawari cid fer ale indosiing
sebeet yeanoll lmmk.
2
The siomaa on ls sun har damogrphic profii
Ger
Whells gender pollent?
Medical Images
X
Language Insmuctions
XP
g Vicne (Fan v2)
Figure 3: Overview of EXGRA-MED. The large language model go and the projector h& trained jointly by aligning triplet of modalities input image, a structure-aware multigraph aligmment (Eq.(3). This alignment operates over graphs G.. Ga. and Gae, representing the visual, instruction, and extended textual information. respectively, via shared barycenter graph. The entire model is optimized end-to-end using modern black-box gradient estimation techniques enable efficient learning across modalities 770 63
data, and extended captions through
3.3 Second-order graph alignment problem
We first provide background information about the second-order graph alignment problem between two arbitrary graphs 91 &V,&&) and G2 (V2, E2), which is known as quadratic assignment problem. It occurs several problems vision and computer graphics to find correspondences between two graph structures under constraints on the consistency of node features and the graph structures [101 25
We denote by VE {0. 1) with V1 M and |2l N, the indicator matrix of matched vertices, that is, Vi, if vertex 2 E 1 is matched with vy E V22 and Va, = 0 otherwise. That non-zero entry in each row and column. Similarly,
is, V is binary matrix with exactly the indicator tensor of match edges, that is, Ej 1 if V., 1 and Vi,1 = and E, = otherwise. This implies that the tensor is fully determined by the matrix V. that is, EjA.j V1, VR.1. We also define the vertex affinity matrix and edge affinity tensor as
set E E {0, 11
A" IDa and A E Rl fal, respectively. The set A(G1, (2) indicates for all admissible pairs
(V, E) that encode a valid matching between 91 and G2-
A(91 G2)
V E (0. MXNNN
Vn =1
V.. j=1
(2)
The second-order graph alignment (SGA) problem now is defined as: SGA(A", A²) = arg min (A" A VV.
V=A(S1.92)
arg min VeA(G
Σ Vi. + Σ N.J.A,
(3)
G2)
R.
3.4 Scalable Multi-graph Alignment
Our aim is to solve the graph alignment between Go, Ga. and Gae to form triplet constraint between input image embedding. its original instruction embedding, and the extended version ones. However, solving structure- aware graph alignment between K domains (K > 3) is computationally expensive. One potential solution is perform pairwise graph alignments (k) times, shown in Eq. while applying specific constraints to maintain consistency between correspondences [3] 89 (Table (
However, these approaches become impractical as K increases, making them unsuitable for large- scale problems or the integration of multi-modes inputs such as text, images, audio, or medical health records.
Another direction leverages the barycenter concept from optimal transport 27 10,
which identifies
central distribution that minimizes the weighted sum of Wasserstein distances to the given input
5

---

distributions. follow this idea to reformulate the alignment of K graphs into K separate alignments with barycenter graph. Unlike previous unsupervised methods that estimate the barycenter before aligning, we directly define the barycenter using known triplet pairs across the three graphs. This significantly reduces complexity, making our solver more efficient in L.M settings.
Specifically, we define new barycenter graph Gor (Vbr, where Vor =
( {
(2)
(B)
For
[x]] [x } and correspondence feature node as For 3 E. - ,..., Σ. m! with E {v, a, ae). The edge set Ehr is formed similarly to another graph by running the k-nearest neighbor on feature node For. We now state the multi multi-graph alignment as:
with 6br = X (k)
(h)
(x
SGA(A", A.)= = arg min
mrg V.E.(g. (E(U,,,))
E
(A) - A 8 V.. Va):
(4)
where V, is the indicator matrix representing for valid mapping between 93 and 9or, A" E IRI
and A E RIE, Evrl be vertex affinity matrix and edge affinity tensor between Go and Gor- For e.g..
(4%) = d 2 (1) Σ. 2(30 with d(-) be a distance metric (e.g., cosine distance) measuring similarity between node tth in Go and node jth in Gor.
To address the NP-Hard nature of aligning each graph the barycenter graph Go which arises from its combinatorial complexity, we employ efficient heuristic solvers utilizing Lagrange decomposition techniques 90 80]
3.5 Backpropagation with Black-box Gradient Estimation
Given V. = SGA(A", A be solution obtained from the solver, we aim to learn feature repre- sentation for LLMs such that V. be identical to true triplet alignments explicitly indicated by the barycenter graph. By denoting V: be an optimal mapping between the graph Ge to Gor, we compute the following total of hamming loss function:
L(V,, V:
Σ (V. V.)) + (V., (1 V.)).
(5)
sEfe,a,ae)
However, computing the gradient of the loss function with respect to the matching problem inputs (4", A.). i.e., A Ve, V:¹) poses a challenge due to the piecewise constant nature of the
graph matching objective in Eq. (4) 76 79]. To address this, we resort the IMLE 70 63
method that estimates gradients and enables backpropagation through the alignment algorithm by taking the difference between solutions of noise-perturbed alignments.
In particular, given (c, ~ Gumble(0, 1) and for each s E {u, a, ae), we compute:
0L OL
= V, SGA (A A. A, where V. = SGA (A" + c, A + +c),
0Ag' 0A
(A AS (A + A + +6') A L( V,",, with À is a step size.
3.6 Graph Distance Properties via Structure Alignment
We provide theoretical insights into the graph-matching problem in F3. Once the optimal matching is found, it defines a valid distance metric between graphs (Theorem[1 based on joint node-edge representations. Additionally, the geodesic path connecting two graphs (Theorem (2 can be derived from these alignments, enabling dynamic formulations in optimal transport and sampling strategies in representation learning. In particular, we define a discrete distance between two graphs given solution to the matching alignment as follows:
dsGA (91, (1))2)
min VeA(91.92)
(E + Σ Vi..
1.
i.j.A.
We provide an informal definition of the space of structured graphs below to introduce the main theorems. A more formal treatment, including proofs and definitions, is presented in the Appendix.

---

Definition 1 (Space of all structured graphs). S(F) is the space of all structured graphs defined over a node metric feature space (F.dy), where each graph is associated with an edge structure space (S, da) and a mixing measure je- = (i==1, over the product space (F x S).
Theorem 1 (Metric properties). The distance dSGs in Eq. 0 defines a metric in S(F).
The dsGA distance above is zero if and only if there exists a one-to-one mapping between the graph vertices that preserves both shortest paths and features and both graphs have the same number of vertices.
Theorem 2 (Geodesic space). The space S(F) equipped with the dsca distance is geodesic.
4 Experiments
4.1
Implementation Details
Table I: Fine-tuning performance on MedVQA tasks (pre-trained 10%
Table 3: ExGra-Med versus
Bold indicates the best values among pre-training algorithms (Sec.
LLaMe- when pre-trained
excluding LLaVA-Med.
with extended captions (ext.cap)
VQA-RAD
Closed
Ang.
Open
SLAKE
Closed
Ang.
PatNYQA
Closed
Overall
Performance is reported
2 (1015)
65.65
K1.67
77.64
$1.04
$5.41
81.43
55.73
MI
64.0
71.37
VQA-RAD downstream after
VL (105)
12.19
101
(0:70/11
35.09110
56.15jrm
61.0%
77.57
71:43
fine-tuning.
PLOT
21.67
37.81
11.79
$1.35
SULiP
VEAP
56.90
$7:40
77.94
76.47
67.47
65.08
50.85
90.05
$0.55
$2.31
81.13
90.69
32.21
18.08
91.15
50.85
61.885
34-463
69:55
60.38
Methel
VQA-RAD
GeeCLAP
68.83
52.64
85.10
91.15
71.68
aLd (B0ens)
63.65
81.02
13.44
11 11 Med (199))
IMAGEBIND EvGra-Mol
75.74
66.53
7066
esteap
66.02
79.14
72.52
$4.2
$5.10
85.01
37.25
91.45
64.34
73.96
Rxfars-Med Med (1055)
66.82
11.52
LLakA-Maa (40%)) LLaVi Mad (40%) csleup
62.23 La 7841111 7.82
Table 2: Fine-tuning performance on MedVQA tasks (pre-trained 40%). ).
63.58
Bold indicate for best values among pre-training algorithms (Sec. 4.3 ex Table 4: EXGRA-MED results with different frozen LLMs It shows that Gemini and Qwen is
cluding LLaVA-Xed.
Method
YQ4-RAD
SLAKR
PathVQ4
Ovurall
Open
Closed
Aug.
Open
Cloued
Avg
Open
Cloued
Avg:
also effective within our method.
LLaVA- Med (100%)
72.64
83.43
36.78
91.33
LadVA-Med (405))
16-4221183..
14.04
86.6
(8..9)
InGNNCE
77.57
82:68
33.58
50.63
61.6
71.74
Mathed
VQA-RAD
SLAKE
PLOT
66.36
79.41
71.89
83.28
82.93
$3.16
35.11
$9.59
62.35
72.45
SiPLIP
YLAP
63.02
63.37
79.03
61:25
71.11
72.14
M3.38
81.26
K3.89
80.29
$3.64
30.77
35.02
36.01
$0.86
90.83
63.435
63.225
72.12
72.65
Exti Med (Damiin 50%
Extit Medi (Quaal. 10%
Mad (Reckac)
21.05
$2.39
7013
$3.05
$4.7
GeoCLAP
62:28
81.72
72.5
82.64
85.2
83:92
33.2
75.05
54.13
70.18
Extirs Sted (LPF-4) ard
24.37
85,15
PAC-S
63.77
79.41
71.99
84.52
85.5 K5.58
85.05
27:11
53.34
56.23
70.96
ExGia Med (Ceetii $355
71.3
IMAGEBIND
64.73
78.68
71.71
82.31
84.02
83:47
38.76
$7.05
61.42
72.20
ExGre
senceymss
ExGra-Med
66.01
82.72
74.37
84.47
85.82
85.15
37.41
91.27
64.34
74.57
Med (Ruerlin))
30.87
Model Architecture Configuration. We use the LLaMA-7B large language model [93.. the
LLIP-ViT-L-Patch14 visual encoder 177]. and MLP projection similar to LLaVA 1.5 [56]. Stage follows the standard Med (50) setup, while stage 2 incorporates our multi- graph alignment with autoregressive training. For multi-graph alignment, 2-layer graph convolutional network is applied to the output of the Projection and LLM Decoder (handling both image and text modalities). We train for epoch in stage I and epochs in stage 2 using the same dataset as LLava-Med. The model is optimized using Adam [47] with Coneaainnl scheduler and learning rates of 2e 3 and 2e 5 for stages and 2, respectively.
Pre-training data. We use the same dataset as LLaVA-Med [50]. Stage includes 600K image-text pairs filtered from PMC-15%: 15M, converted into instructionn-ollloiing data with simple image descriptions. Stage 2 comprises 60K image-text pairs from PMC-15M across five modalities: CXR. CT, MRI, histopathology, and gross pathology. GPT-4 is then employed to generate multi-round Q& A in tone mimicking visual interpretation, converting these pairs into an colrructioo-oollowiin format.
Rumning-time. We train EXGRA-MED using 4 A100-GPUs per with 80GB for both stages and complete the training process for stage 6.5 hours and for stage 2 in 7.5 hours. With original LLaVA-Med (version 1.5) [50], the training process for stage I finishes in 6.5 hours, and for stage 2 finishes in hours. In total, we need an extra 0.5 hour to complete the whole pre-training process compared to the VLaVa-Med.
7

---

4.2 Autoregressive Training is Data-hungry
We highlight the data-intensive nature of autoregressive training by evaluating LLaVA-Med, a state-of- the-art biomedical multimodal language model. LLaVA-Med follows a two-stage training process: Stage aligns image-text tokens with biomedical concepts, and Stage 2 fine-tunes the model for instruction tasks. We pre-trained LLaVA -Med on varying data amounts (10%, 40%, 70%) and fine- tuned on the VQA-RAD dataset. As shown in Figure || performance drops significantly pre- pre-training compared to full training, revealing the autoregressive mechanism's 's data dependency issue in Medical-MLLMM, This highlights the challenge of weak connections between visual features and text embeddings without sufficient instruction-tune tuninng data.
10%
In contrast, our EXGRA-MED specifically learns image-text alignment by enforeing semantic consistency across images, instruction responses, and extended contexts. Using the same setup as LLaVA-Med, EXGRA-MED excels even with limited data, achieving 72.52% at just 10% pretraining, far above LLaVA-Medi 52.39% and consistently outperforms LLaVA-Med across instruction tuning rates (40%- 100%).
Impact of Longer Training and Enriched Captions on LLAVA-MED Performance. We conduct deeper analysis of the data- aata-hungry phenomenon by examining LLAVA- MED when (i) trained longer in Stage (with an additional hour) and when (ii) incorporating extended captions in an autoregressive manner, as done in EXGRA-MED. The experiments were performed using 10% and 40% pre-training settings, followed by fine-tuning on the VQA-RAD dataset. From the results in Table we draw two key conclusions. First, extended captions improve LLAVA-MED performance, particularly in the data-scarce 10% pre-training scenario. However, both settings, ExGRA-MEi demonstrates superior performance with significant margins, highlighting the effectiveness of its multi-graph alignment strategy in mitigating the data-hungry issues of the autoregressive mechanism.
4.3 Multi-modal Pre-training Comparison
To further demonstrate the strengths of EXGRA-MED, we compare its performance against other vision-language pre-training methods currently used for visual instruction tuning enhance frozen vision-language models
Datasets. We test pre- trainre models on three prominent biomedical VQA datasets: VQA-RAD [48), SLAKE [55), and PathVQA BT. VQA-RAD includes 3,515 questions across 315 radiology images, while SLAKE contains 642 radiology images from various body parts and over 7k QA pairs. PathVQA, focused on pathology, features 5k images and 32.8k questions. All datasets include open-ended (e.g., what, why, where) and closed-ended (yes/no two-option) question types. We report performance using recall for open-ended, which evaluates the proportion of ground- tokens that appear in the generated sequences and accuracy for the closed-ended questions.
Baselines. We compare seven approaches, including:
two modalities- based methods such as InfoNCE-bass methods 45 58 SigLIP [022.
PLOT [3]. VLAP [72]. Among this, SigLIP adapts the Sigmoid loss on image-text pairs to break the global view of the pairwise similarities for normalization. resulting in scaling in large batch sizes. PLOT defines optimal transport distance between visual image patches and text embedding. In contrast, VLAP uses assignment prediction to bridge the modality gap between the visual and L.LM embeddings.
multi-modal learning across three modalities such as image, text, voice, or augmented im- age-is explored by methods like PAC-S [8]]. GeoCLAP [46. and IMAGEBIND [26]. PAC-S integrates contrastive losses across multiple modality pairs: (image-text), text), and (text- augttnue text). GeoCLAP applies CLIP-style contrastive learning to all cross-domain pairs, such as audio-tex text and text- image. Similarly, IMAGEB IND generalizes the InfoNCE loss to learn unified embeddings across diverse modalities.
We train the baselines under the same settings as EXGRA-MED with varying pre-training data rates and compare their performance on downstream tasks.
Results. Tables I 16 show our performance and baselines under 10%, 40%, and 100% instruction- tuning data. While most contrastive baselines improve over LLaVA-Med at 10%, EXGRA- AEMD consistently outperforms all methods across settings. excels particularly on open-ended questions requiring external knowledge and maintains stable gains across all VQA datasets. In contrast, some
8

---

Table 5: Comparison with other Med-MLLMs on MedVQA tasks
All models (except GPT-4) are fine-tuned the same training set each VQA task. These Med- Med-MLLMss differ notably in model size,
Table 6: ExGra-Med ablation study
Results are presented as average scores on VQA-RAD and SLAKE, using pre- trained weights 10%, 40%, 100%.
training data volume, and pre-training strategies e.g., ExGra-Med (7B, 60K GPT-4 augmented samples) vs. MedDR (40B, 2M samples).
APura
VQA-RAD
SLAKE
Path
Methad
VQ+-RAD SLAKE
Open
Closed
Arg.
Open
Closed
Avg.
Open
Closed
Avg.
VadVA-Med
S1.62
91.33
64.06
(Full 10% Pocii--Mal (Pull, 10% 0.5g
82.9
BioneoKPPT-B
18EM
81.3
89.9
88
67.73
42:51
M212
61.8
81.5
71.7
74.7
91.1
82.9
36.3
88
62,15
72.25
ExGra (Flll 40%)
74.37
$4.99
CLIP
422M
67.6
79.8
73.7
82.5
89.7
16.1
OID toxti Med wit ext conteat
72. 12
$1.11
Mad-DD
40R
17.5
78.9
58.2
14.2
83.4
78.
33.5
90.2
61.85
66.28
(o ExGns Med mesegg pessing
captins
72.58
52.31
LLAVA (general)
70
50
65.1
57.55
78.2
63.2
70.7
1.7
63.2
35.45
54.57
GPT-4
200B
39.5
78.9
59.2
33.6
43.6
38.6
sigge
Med Mot: (Phuz)
3,68
5M.55
X2.72
70,64
15.08
35.58
55.32
34.74
91.98
63.36
73.11
ExGra Med (Full, 100%)
74.91
$5.46
Mad-MaR (Stalle M)
2R
50.08
50,07
65.05
83.16
83.41
83.20
53.79
91.30
62.55
70.3
(vi) HsGn-Maa weo beryorntee graph
75.81
54.35
EvGra-Med
7B
66.35
83,46
71.91
85.34
85.58
85,45
90.92
63.87 74.25
ExGra-Med (DCI)
7B
67.03
83.46
75.25
$4.85
55.58
85,23
37,77
91.86
64.82
75.1
methods like SigLIP peak at 40% (e.g., 72.14% Avg on VQA-RAD) but drop notably at 100% (down over 6%),, whereas EXGRA-MED improves further to 74.91% (Avg) and 74.75% (Overall).
4.4 Med Comparison with Medical MLLMS
We now compare EXGRA-MED pre-trained with 100% data against other medical foundation models, each trained on varying datasets and employing different architectures or model sizes.
Baselines. We benchmark eight competitors, both generic or medical foundation models, including
LLaVA (57) LLaVA-Med
[64], Med-Dr (30), Biomed-GP GGPT [104, M212 (53). 139 Whilst LLaVA and GPT-4o have no medical background, the others are pre-trained on a variety of biomedical knowledge. With the exception of LLaVa, which we reproduced, the results for the other baselines are taken from the literature. Moreover, we also present an enhanced version, EXGRA-MED + DCI, which integrates multi-scale visual features from vision encoders [000], potentially benefiting medical image analysis by considering both local (detailed) and global (contextual) features.
(5).
Med-Flamingo
GPT-4o 7 and Med-MoE
Results. Overall, both versions of EXGRA-MED outperform baseline models (Table 5). with the DCI variant achieving the best results VQt (64.82% average, 75. 1% overall). Compared to LLaVA-Med, it shows notable gains: +2.01 % on VQA-RAD, +2.03% on SIAAKE, and +0.76% on Vat Despite using only 7B parameters, both ExGRA- MED models surpass the 40B Med-Dr across all datasets.
4.5 Medical Visual Chatbot Evaluation & Zero-shot Image Classification
Medical Visual Chatbot. We present in Section E Appendix EXGRA-MED results compared against several SOTA general and Med-MLLMs such as LLaVA, GPT-40, LLaVA-Med, Med- Flamingo, Med-Dr, and Biomed-GPT. Among these, we observe that EXGRA-MED is the best model across question types.
Zero-shot Image Classification as MedVQA. Results are presented in Section of the Appendix. In short, we outperform other models across all datasets, particularly excelling in microscopy. where it surpasses RadFM by 8.2%
4.6 Additional Analysis
Table 7: Results of fully finetuning vs LoRA finetuning on VQA-RAD dataset.
Method
VQA-RAD Open VQA-RAD Closed VQA-RAD Avg Param
LLaVa-Med (LoRa)
62.06
75.00
68.53
2.1B
ExGra-Med (LoRa)
63.55
79.41
71.48
2.IB
LLaVa-Med (Fully-finctuning)
63.65
81.62
72.64
7B
ExGra-Med (Fully- Fullt-finetuning)
66.35
83.46
74.91
7B
Potentially Hallucination in Extended Captions. We conducted a user study with five general practitioners from top public hospitals (Appendix Section K). In Stage 2 of pre-training, each expert evaluated 200 image-text pairs (1,000 total) across five modalities chest X-ray, CT, MRI, histology, and others rating the completeness and accuracy of GPT-4-generated extended captions. As shown

---

in Figures 10-14, most scores ranged from 3 to 5, with few low ratings, confirming the overall consistency and quality of the extended outputs. Also, these extended captions are used only during pre-training to guide latent space alignment. They are excluded during fine-tuning on downstream tasks. As such, we argue that a small amount of noise in the extended captions should have minimal impact on overall performance, since they do not directly affect the model's task-specific adaptations.
Other Factors. We examine ExGra-Med results under following settings:
(i) generalization to other frozen LLMs (GPT-4, Gemini [92], and Qwen3-8B LLM [99 to generate extended captions and how the method works with simple synonyms (Table
ii) contribution of coefficient (a) combine multi- graph alignment with auto-regressive. iii) without using extended contexts, i.c. simplifying from the three graph alignment to two cross-domain alignments (image vs. original captions).
(iv) without using original captions, i.c., only extended ones are used.
(v) applying message passing to enhance node features.
(vi) using multi-graph alignment in both steps (default uses only Step-2).
(vii) solving three pair-wise graph alignments in multi-graph alignment rather than solving through a baryeenter graph (Eq.4) in Sec. 3.4).
(viii) using parameter-efficient finetuning with LoRa [33 rather than fully finetuning on downstream tasks.
Tables 4 show results (i)(vvii) where the most important factors highlighted. We further observe that ExGra-Med generalizes effectively across distinct LLM paraphrase generators such as GPT-4, Gemini, and Qwen3-8B. The stable performance across these models indicates that ExGra-Med not tightly coupled to specific language model architecture, but instead captures transferable alignment mechanisms applicable a wide range of paraphrastic contexts.
In Table 7 we report the results of ExGra-Med and LLaVA-Med on the VQA-RAD dataset for the (vii)) setting. As shown, ExGra-Med consistently outperforms LLaVA- even when both models are fine-tuned using LoRA, demonstrating the robustness of our approach under parameter-effiee efficieet adaptation. Nevertheless, expected, MoRA-based fine-tuning yields modest performance drop compared to full fine-tuning. We believe that to bridge this gap, pre-training on a larger-scale medical instruction-cuiing dataset (e.g., the 21M samples curated from MedTrinity (98)) would allow the LoRA setup to more closely match the performance of fully fine-tuned models. Additional analyses on average pooling features, k-nearest neighbors for graph construction (Section 3.2
Current Limitations and Future Work. While our experiments have primarily focused on the LLaVa model, it is essential to validate the effectiveness and adaptability of ExGRA-MED with alternative architectures, such as the Flamingo model [9]. which has shown promising results in vision-language tasks. Expanding the evaluation to include other of-e-oo--hhe-rt models can provide broader perspective on the generalizability and robustness of EXGRA-MED. Furthermore, incorporating vision encoders or large language models (LL.Ms) that specifically pre-trained medical datasets 107 17 103) presents a compelling opportunity to enhance both performance and domain- specific understanding. These specialized models are designed to capture the nuanced characteristics of medical data, which could further enhance the robustness of EXGRA-MEN in complex biomedical scenarios. Moreover, extending our mechanism to the setting of medical visual
19
41 49 reasoning represents a promising direction for improving both the overall performance and the trustworthiness of the model.
5 Conclusion
We have shown that enforcing triplet correlations among images, instructions, and extended cap- tions significantly enhances vision-language alignment- area where autoregressive models like LLAVA-MED struggle, particularly under limited data and domain shift, which manifests as a strong dependence on large-scale pre-training data. To this end, introduce EXGRA-MED multi-grap graph alignment algorithm that trains efficiently, matches LLAVA- MED using only 10% of the data, and outperforms other ofte-of-theear models across tasks. These results highlight that selecting an effective learning algorithm for LLMS is as crucial as scaling model size or data volume.
10

---

Acknowledment
This work supported by Deutsche Forschungsgemeinschatt (DFG, German Research Foundation) under Germany's Excellence Strategy EXC 2075 - 390740016, the DARPA ANSR program under award FA8750-23- 2-0004, the DARPA CODORD program under award HRO01 25900889. The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Duy M. H. Nguyen. Duy M. H. Nguyen and Daniel Sonntag are also supported by the No-IDLE project (BMBF, 01IW23002), the MASTER project (EU, 101093079), and the Endowed Chair of Applied Artificial Intelligence. Oldenburg University.
References
[1] Chest ct-scan images dataset, 2024-09-30. (Cited page 30
Accessed:
[2] Covid ct dataset.
Accessed: 2024-09-
30. (Cited on page 30
[3] Blood cell images. blood- cells 2023. Accessed: 2024-09-30. (Cited on page 30
https://wwz www.wwaagll..cmm http:://w.....g........aaaattta
[4] Covid-19 image dataset: 3 way classification covid-19, 19, viral pneumonia, normal. https
//tianchi.
wtt.:/ccoomaaaaaaa wtt.///atteeeeeeeeee.... 2023. (Cited on page
[5] Nlm malaria data. https://lhncbc.n www.bbc..mm..... mih. Lee.-r--eeeearrrH---------
wtw.:/eiiiinnnmaaaaaaaaaaaaaaaaa html 2023. Accessed: 2024-09-30. (Cited
page
[6] X-ray hand small joint classification dataset (based on bone age scoring method rus-chn). 2023. Accessed: 2024-09-30.
baidu. http://....ici..cccaaaaaaaeeeeee
(Cited page 30
[7] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint
arXiv:2303.08721 2023. (Cited on pages 2 )
[8] M. Aguch and G. Carlier. Barycenters in the wasserstein space. SIAM Journal on Mathematical Analysis, 43(2):904-924 2011. (Cited on page 2
[9] J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lene, A. Mensch, K. Milli- can, M. Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716-23736, 2022. (Cited on pages 3nd 10
[10] J. M. Altschuler and E. Boix-Adsera. Wasserstein barycenters are np-hard to compute. SIAM Journal on Mathematics of Data Science, 4(1):179-203; 2022. (Cited on page 3
[11] C. An, F. Huang. J. Zhang, S. Gong, X. Qiu, C. Zhou, and L. Kong. Training-free long-context scaling of large language models. arXiv preprint 240. 2024. (Cited on page
[12] A. Asraf and Z. Islam. Covid19 pneumonia and normal chest x-ray pa dataset. mendeley data (2021), 2021. (Cited on page 30)
[13] F. Bernard, J. Thunberg, P. Swoboda, and C. Theobalt. Hippi: Higher-order projected power iterations for scalable matti-matchhng. In Proceedings of the ieee/cyf international conference on computer vision, pages 10284-10293, 2019. (Cited on pages and
[14] D. Cao and F. Bernard. Sclf-supervised learning for multimodal non-rigid 3d shape matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17735-17744 2023. (Cited page 25
[15] G. Chen, W. Yao, X. Song, X. Li, Y. Rao, and K. Zhang. Plot: Prompt learning with optimal transport for vision-language models. International Conference on Learning Representations, 2022. (Cited on pages and

---

[16] G. Chen, Y.-D. Zheng, J. Wang. J. Xu, Y. Huang. J. Pan, Y. Wang, Y. Wang, Y. Qiao, T. Lu, et al. Videollm: Modeling video sequence with large language models. arXiv preprint arXiv:2305. 13292, 2023. (Cited on pages 2 and
[17] H. Chen, R. Shukla, R. Wu, S. Yang, D. Duong-Tran, D. M. H. Nguyen, M. Niepert, C. Beeche, J. Gee. J. Duda, et al. Adapting rision-languee models for 3d ct/mri understanding on pmbb via slice selection and explanation analysis. In Proceedings of the IEEE/CVF International
Conference on Computer Vision, pages 2273-2282, 2025. (Cited on page 10]
[18] P. Chen. Knee osteoarthritis severity grading dataset, 2018. (Cited on page 30
[19] Z. Chen, A. H. Cano, A. Romanou, A. Bonnet, K. Matoba, F. Salvi, M. Pagliardini, S. Fan, Köpf, Mohtashami, al. Meditron-7000 Scaling medical pretraining for large language models. arXiv preprint vrxiv:2311116077
2023. (Cited on page 10
[20] L. Chizat and F. Bach. On the Global Convergence of Gradient Descent for Over- parameterized Models using Optimal Transport. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. (Cited page 20
[21] M. E. Chowdhury, T. Rahman, A. Khandakar, R. Mazhar, M. A. Kadir, Z. B. Mahbub, K. R. Islam, M. S. Khan, A. Iqbal, N. Al Emadi, et al. Can ai help in screening viral and covid-19 pneumonia? leee Access, 8: 22665-336766 2020. (Cited page 30)
[22] J. P. Cohen, P. Morrison, L. Dao, K. Roth, T. Q. Duong, and M. Ghassemi. Covid-19 image data collection: Prospective predictions are the future. arXiv preprint arXiv:2006.11988, 2020. (Cited on page(30
[23] M. Cuturi. Sinkhorn distances: Lightspeed computation optimal transport. Advances in neural information processing systems, 26, 2013. (Cited on page 25
[24] J. Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 11X.::/810004805, 2018. (Cited on page 3
[25] V. Ehm, M. Gao, P. Roetzer, M. Eisenberger, D. Cremers, and F. Bernard. Partial-to-partial shape matching with geometric consistency. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 27488-27497, 2024. (Cited on page
[26] R. Girdhar, A. El-Nouby, Z. Liu, M. Singh, K. V. Alwala, A. Joulin, and I. Misra. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15 15180-15190, 2023, (Cited on pages and g
[27] W. Guo, N. Ho, and M. Jordan. Fast algorithms for computational optimal transport and wasserstein barycenter. In International Conference on Artificial Intelligence and Statistics, pages 10088-2097, PMLR, 2020. (Cited page(5
[28] A. Gupta and R. Gupta. Isbi 2019 c-nme challenge: Classification in cancer cell imaging. Select Proceedings, 2, 2019. (Cited on page 30
[29] S. Haller, L. Feineis, L. Hutschenreiter, F. Bernard, C. Rother, D. Kainmüller, P. Swoboda, and Savchynskyy. A comparative study of graph matching algorithms in computer vision. In European Conference on Computer Vision, pages 636-653. Springer, 2022. (Cited page 5
[30] S. He, Y. Nie, Z. Chen, Z. Cai, H. Wang, S. Yang, and H. Chen. Meddr: Diagnosis-guided boot- strapping for scrge-scale medical vision-language learning. arXiv preprint 24X4v 2024. (Cited on pages and
[31] X. He, Y. Zhang, L. Mou, E. Xing, and P. Xie. Pathvqa: 30000+ questions for medical visual question answering. arXiv preprint arXiv:2003. 10286, 2020. (Cited on pages 2 and
g
[32] T. Heimann and H.-P. Meinzer. Statistical shape models for 3d medical image segmentation: a review. Medical image analysis, 13(4):543-56) 563, 2009. (Cited on page 25
[33] Hu, Y. Shen, P. Wallis, 7. Allen-Zhu, Li, S. Wang, I. Wang, W. Chen, et Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022. (Cited on page 10
12

---

[34] Y. Hu, Li, Q. Lu, W. Shao, J. He, Y. Qiao, and P. Luo. Omnimedvqa: A new large-scale comprehensive evaluation benchmark for medical Ivim. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22170-22183, 2024, (Cited on
page 28)
[35]
W. Huang, C. Wang, R. Zhang, Y. Li, J. Wu, and L. Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models. arXiv preprini arXiv:2307.05977 2023. (Cited on pageBi
[36] J. Hyun, M. Kang, D. Wee, and D.-Y. Yeung. Detection recovery online multi-object tracking with sparse graph tracker. In Proceedings of the IEEE/CVF winter conference applications of computer vision, pages 4850-4859, 2023. (Cited on page 25
[37] S. Jaeger. S. Candemir, S. Antani, Y.-.J J. Wáng, P.-X. Lu, and G. Thoma. Two public chest x-ray datasets for computer-aided screening of pulmonary diseases. Quantitative imaging in medicine and surgery, 4(6):475, 2014. (Cited on page 30
[38] S. Javadi and A. Mirroshandel. A novel deep learning method for automatic assessment of human sperm images. Computers in biology and medicine, 109:182- 194, 2019. (Cited on page 30
[39] S. Jiang, T. Zheng, Y. Zhang, Y. Jin, and Z. Liu. Moe-tinymed: Mixture of experts for tiny medical large vision Aision-language models. arXiv preprint arX1v:2444..02237, 2024. (Cited on
pages 2 9)d
[40] H. K. Jin, H. E. Lee, and E. Kim. Performance of chatgpt-3.5 and gpt-4 in national licensing examinations for medicine, pharmacy, dentistry, and nursing: systematic review and meta- analysis. BMC Medical Education, 24(1):1013, 2024. (Cited on page 33
[41] A. E. W. Johnson, T. J. Pollard, S. J. Berkowitz, N. R. Greenbaum, M. P. Lungren, C. Y. Deng, R. G. Mark, and S. Homg. Mimic-exr, de-identified publicly available database of chest radiographs with free-text reports. Scientific Data, 6(1):317, 2019. (Cited on page I0]
[42] W. Ju, Z. Fang, Y. Gu, Z. Liu, Q. Long, Z. Qiao, Y. Qin, J. Shen, F, Sun, Z. Xiao, et al. A comprehensive survey on deep graph representation learning. Neural Networks, page 106207, 2024. (Cited on page 5)
[43] J. N. Kather, N. Halama, and A. Marx. 100,000 histological images of human colorectal cancer and healthy tissue, April 2018. Accessed: 2024-09-30. (Cited on page 30
[44] D. S. Kermany, M. Goldbaum, W. Cai, C. C. Valentim, H. Liang, S. L. Baxter, A. McKeown, G. Yang, X. Wu, F. Yan, et al. Identifying medical diagnoses and treatable diseases by image-based deep learning. cell, 172(5):1 122-1111, 2018. (Cited on page
[45] Z. Khan and Y. Fu. Contrastive alignment of vision to language through parfmeter-efficient transfer learning. International Conference on Learning Representations, 2023. (Cited on pages
and 33)
[46] S. Khanal, Sastry, A. Dhakal, and N. Jacobs. Learning tri-modal embeddings for zero-shot soundscape mapping. The British Machine Vision Conference (BMVC), 2023. (Cited on pages]
and
[47] D. P. Kingma, Adam: A method for stochastic optimization. arXiv preprint 141.v:441..6980 2014. (Cited on page 7
[48] J. J. Lau, S. Gayen, A. Ben Abacha, and D. Hunner-Fushman. A dataset of clinically generated visual questions and answers about radiology images. Scientific data, 5(1):1-10, 2018. (Cited pages
2 and 88)
[49] K. Le-Due, P. T. H. Trinh, D. M. H. Nguyen. T.-P. Nguyen, N. T. Diep, A. Ngo, T. Vu, T. Vuong, A.-T. Nguyen, M. Nguyen, T. Hoang, K.NN. Nguyen, H. Nguyen, C. Ngo, A. Liu, Ho, A.-C. Hauschild, K. X. Nguyen, T. Nguyen- -Tng, P. Xie, D. Sonntag, J. Zou, M. Niepert, and T. Nguyen. S-chain: Structured visual of-ihoof-thought for medicine, arXiv preprint, 2025.
(Cited page [0
13

---

[50] C. Li, C. Wong, S. Zhang, N. Usuyama, H. Liu, J. Yang, T. Naumann, H. Poon, and J. Gao. Llava-med: Training a large language-and-ving vances in Neural Information Processing Systems, 36, 2024. (Cited on pages
assistant for biomedicine one day. Ad-
7 and 9
[51] Li, D. Li, S. Savarese, and S. Hoi. B Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730-19744 PMLR, 2023. (Cited on pages 2 and
[52] J. Li, D. Li, C. Xiong, and S. Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning, pages 12888-1299 PMLR, 2022. (Cited on page 3
[53] P. Li, G. Liu, L. Tan, Liao, and S. Zhong. Self-supervised vision-language pretraining for medial visual question answering. In 2023 IEEE 20th International Symposium Biomedical Imaging (ISBI), pages 1-5, 2023. (Cited on page
[54] T. Lin, N. Ho, M. Cuturi, and M. Jordan. On the complexity of approximating multi-margina optimal transport. Journal of Machine Learning Research, 23:1-43 2022. (Cited on page 3
[55] B. Liu, L.-M. Zhan, L. Xu, L. Ma, Y. Yang, and X.-M. Wu. Slake: A semantically-labelee knowledge-enhanced dataset for medical visual question answering. In 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), pages 1650-1654, 2021. (Cited on
page
[56] H. Liu, C. Y. Y. Li, and J. Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 2024. (Cited on page 7
[57] H. Liu, C. Li, Q. Wu, and Y. J. Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. (Cited on pages and
[58] L. Liu, X. Sun, Xiang, Zhuang, L. Yin, and M. Tan. Contrastive vision-language alignment makes efficient instruction learner. arXiv preprint arXiv:2311 2023. (Cited on pages
and 33
[59] N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157-173, 2024. (Cited on pagel4
[60] R. Lou, K. Zhang, and W. Yin. A comprehensive survey on instruction following. arXiv preprint arXiv:2303. 10475, 2023. (Cited on page I)
[61] X. Ma, X. Chu, Y. Wang, Y. Lin, Zhao, L. Ma, and W. Zhu. Fused gromov-wassersee graph mixup for graph-level classifications. Advances in Neural Information Processing Systems, 36, 2024. (Cited on page 25
[62] MH Nguyen, H. Nguyen, N. Diep, T. N. Pham, Cao, B. Nguyen, P. Swoboda, N. Ho, S. Albargouni, P. Xie, et al. Lvm-med: Learning large-scale self-supervised vision models for medical imaging via second-order graph matching. Advances Neural Information Processing Systems, 36, 2024. (Cited pages] 10 and 25
[63] P. Minervini, L. Franceschi, and M. Niepert. Adaptive perturbation-base gradient estimation for discrete latent variable models. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 9200-9208, 2023. (Cited on pages 2
6
[64] M. Moor, Q. Huang, S. Wu, M. Yasunaga, Y. Dalmia, J. Leskovec, C. Zakka, E. P. Reis, and P. Rajpurkar. Med-flamingo: a multimodal medical few-shot learner. In Machine Learning for Health (ML4H)), pages 353-367. PMLR, 2023. (Cited on pages 3 and 9
[65] F. Mémoli. Grossv-Wasserstein Distances and the Metric Approach to Object Matching. Foundations of Computational Mathematics, 11(4):417 487, Aug. 2011. (Cited on page 25
[66] F. Mémoli and G. Sapiro. A Theoretical and Computational Framework for Isometry Invariant Recognition of Point Cloud Data. Foundations of Computational Mathematics, 5(3:3113-47,,
July 2005. (Cited on page 25
14

---

[67] L. Nanni, M. Paci, F. L. Cactano dos Santos. H. Skottman, K. Juuti- Uustiiuo,,,,o,, and J. Hyttinen. Texture descriptors ensembles enable image-based classification of maturation of human stem cell-derived retinal pigmented epithelium. PLOS One, 11(2):0114399, 2016. (Cited on page 30
[68] D. M. Nguyen, A. T. Le, T. Q. Nguyen, N. T. Diep, T. Nguyen, D. Duong- Tran, J. Peters, L. Shen, M. Niepert, and D. Sonntag. Dude: Dual distribution-aware context prompt learning for large vision-language model. Asian Conference Machine Learning, 2024. (Cited on page 3
[69] D. M. Nguyen, N. Lukashina, T. Nguyen, A. T. Le, T. Nguyen, N. Ho, J. Peters, D. Sonntag, V. Zaverkin, and M. Niepert. Structure-aware e (3)-invariant molecular conformer aggregation networks. International Conference on Machine Learning, 2024. (Cited on pages and
[70] M. Niepert, P. Minervini, and L. Franceschi. Implicit mle: backpropagating through dis- crete exponential family distributions. Advances in Neural Information Processing Systems, 34: 15577-1479, 2021. (Cited on pages
and 25
[71] A. v. d. Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. 2018. (Cited on page
arXiv preprint
[72] J. Park, J. Lee, and K. Sohn. Bridging vision and language spaces with assignment prediction.
International Conference on Learning Representations, 2024. (Cited on pages 3 and
[73] S. Pawar, S. Tonmoy, Zaman, V. Jain, A. Chadha, and A. Das. The what, why, and how of context length extension techniques in large language models- a detailed survey. arXiv preprint arXiv:2401.07872, 2024. (Cited on page 4
[74] P. A. Pevzner. Multiple alignment, communication cost, and graph matching, SIAM Journal Applied Mathematics, 52(6):16631177, 1992. (Cited on page 2
[75] Z. Piran, M. Klein, J. Thorton,, and M. Cuturi. Contrasting multiple representations with the multi-marginal matching gap. International Conference on Machine Learning, 2024, (Cited on page
[76] M. V. Pogančić, A. Paulus, V. Musil, G. Martius, and M. Rolinek. Differentiation of blackbox combinatorial solvers. In International Conference on Learning Representations, 2020. (Cited
page
[77] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, pages 8748-8763. PMLR, 2021. (Cited on page
[78] P. Rajpurkar, J. Irvin, A. Bagul, D. Ding, T. Duan, H. Mehta, B. Yang, K. Zhu, D. Laird, R. L. Ball, al. Mura: Large dataset for abnormality detection in musculoskeletal radiographs. arXiv preprint arXiv:/712.0699
2017. (Cited on page 30
[79] M. Rolínek, V. Musil, A. Paulus, M. Vlastelica, C. Michaelis, and G. Martius. Optimizing rank-based metries with blackbox differentiation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7620-7630, 2020. (Cited page
[80] M. Rolínek, P. Swoboda, D. Zietlow, A. Paulus, V. Musil, and G. Martius. Deep graph matching via blackbox differentiation of combinatorial solvers. In Computer ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXVIII, pages 407 424. Springer, 2020. (Cited on pages
and 25
[81] S. Sarto, M. Barraco, M. Cornia, L. Baraldi, and R. Cucchiara. Positive- augmenee contrastive learning for image and video captioning evaluation. In Proceedings of the IEEE//VF confer- ence on computer vision and pattern recognition, pages 6914 6924, 2023. (Cited on pages and
[82] F. Shaker, S. A. Monadjemi, J. Alirezaie, and A. R. Naghsh-Nilchi. A dictionary learning approach for human sperm heads classification. Computers in biology and medicine, 91:181- 190, 2017. (Cited on page 30
15

---

[83] K. Singhal, T. Tu, J. Gottweis, R. Sayres, E. Wulczyn, L. Hou, K. Clark, S. Pfohl, H. Cole- Lewis, D. Neal, al. Towards expert-level medical question answering with large language models. arXiv preprint 2rX1v:3305.09617, 2023. (Cited on page 3)
[84] E. Soares and Angelov. large dataset of real patients scans for covid- 19 identification. Harv. Dataverse, 1:1-8, 2020. (Cited page 30
[85] Song, X. Tan. T. Qin, Lu, and T.-Y. Liu. Mpnet: Masked and permuted pre-training for language understanding. Advances in neural information processing systems, 33: 33:16857-16867, 2020. (Cited on page
[86] F. A. Spanhol, L. S. Oliveira, C. Petitjean, and L. Heutte. A dataset for breast cancer histopatho- logical image classification. leee transactions on biomedical engineering, 63(7):1455-1444 2015. (Cited on page 30
[87] J. Suckling. The mammographic images analysis society digital mammogram database. In Exerpta Medica. International Congress Series, 1994, volume 1069, pages 375-378, 1994. (Cited on pageBi/ 30
[88] Y. Sun, C. Liu, K. Zhou, J. Huang, R. Song, W. X. Zhao, F. Zhang, D. Zhang, and K. Gai. Parrot: Enhancing multi-trr instruction following for large language models. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 9729-97259577 99500,975000000 2024. (Cited on pagel
[89] P. Swoboda, A. Mokarian, C. Theobalt, F. Bernard, et al. A convex relaxation for multi-graph matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11156-11165 2019. (Cited pages and
[90] P. Swoboda, C. Rother, H. Abu Alhaija, D. Kainmuller, and B. Savchynskyy. A study of lagrangean decompositions and dual ascent solvers for graph matching. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1607-1616, 2017. (Cited
pages
6 and 25
[91] S. Tang, F. Zhu, L. Bai, R. Zhao, C. Wang, and W. Ouyang. Unifying visual contrastive learning for object recognition from a graph perspective. In European Conference on Computer Vision pages 649-667. Springer, 2022. (Cited on page
[92] G. Team, R. Anil, S. Borgeaud, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, Hauth, K. Millican, al. Gemini: a family highly capable multimodal models. arXiv preprint ar. 11805 2023. (Cited on page 10
[93] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023. (Cited on pages 2 a
[94] T. Tu, S. Azizi, D. Driess, M. Schaekermann, M. Amin, P.-C. Chang, A. Carroll, C. Lau, R. Tanno, I. Ktena, et al. Towards generalist biomedical ai. NEJM AI, Aoa23138230 2024. (Cited on page
[95] L. Wang, Z. Q. Lin, and A. Wong. Covid-net: tailored deep convolutional neural network design for detection of covid-19 cases from chest x-ray images. Scientific reports, 10(1):19549, 2020. (Cited on page 30)
[96] C. Wu, X. Zhang, Y. Zhang, Y. Wang, and W. Xie. Towards generalist foundation model for radiology. arXiv preprint arXiv:2308.02463, 2023. (Cited on pages] and 2nd
[97] S. Wu, H. Fei, I.. Qu, W. Ji, and T.-S. Chua. Next-gpt: Any-to-any multimodal 1mm. arXiv preprint 230::2309005119 2023. (Cited on page I
[98] Y. Xie, C. Zhoou, L. Gao, J. Wu, X. Li, H.-Y. Zhou, S. Liu, L. Xing, J. Zou, C. Xie, et al. Medtrinity-22 A large-scale multimodal dataset with multigranular annotations for medicine. arXiv preprint arXiv: 200.00040,,, 2024. (Cited on pages and 10
16

---

[99] A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09380 00888, 2025. (Cited on page [0
[100] Yao, W. Wu, T. Yang. Song, M. Zhang, Feng, Y. Sun, Z. Li, W. Ouyang, and J. Wang. Dense connector for milmss. arXiv preprint arXiv:2405. 13800, 2024, (Cited on page ②
[101] A. Zanfir and C. Sminchisescu. Deep learning of graph matching. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1684-26933 2018. (Cited on page
[102] Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer. Sigmoid loss for language image pre- training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975-11986, 2023. (Cited on pages] and 8
[103] Zhang, Yu, J. Dong, Li, Su, Chu, and Yu. Mm-llms: Recent advances in multimodal large language models. arXiv preprint arXiv:2401. 13601, 2024. (Cited on page 10
[104] K. Zhang, J. Yu, Z. Yan, Y. Liu, E. Adhikarla, S. Fu, X. Chen, C. Chen, Y. Zhou, X. Li, et al. Biomedgpt: a unified and generalist biomedical generative pre-trained transformer for vision, language, and multimodal tasks. arXiv preprint arXiv:2335.17100, 2023. (Cited on pages and
[105] R. Zhang, J. Han, C. Liu, A. Zhou, P. Lu, Y. Qiao, H. Li, and P. Gao. Llama-adapter: Efficient fine-tuning of large language models with zero-initialized attention. In The Twelfth International Conference on Learning Representations, 2024. (Cited on page
[106] S. Zhang, Y. Xu, N. Usuyama, J. Bagga, R. Tinn, S. Preston, R. Rao, M. Wei, N. Valluri, Wong, al. Large-scall scale-scaleeae domain domain-specific pretraining for biomedical vision-language processing, arXiv preprint 1xXiv:2303.00113 2(3):6, 2023. (Cited page
[107] T. Zhao, Y. Gu, J. Yang, N. Usuyama, H. H. Lee, S. Kiblawi, T. Naumann, J. Gao, A. Crabtree, J. Abel, et al. A foundation model for joint segmentation, detection and recognition of biomedical objects across nine modalities. Nature methods, pages 1-11, 2024. (Cited on page 10)
17

---

SUPPLEMENTARY MATERIAL FOR "EXGRA-MED: EXTENDED CONTEXT GRAPH ALIGNMENT FOR MEDICAL VISION-LANGUAGE MODELS"
Contents
A
Proofs of the Main Theoretical Results
19
Proof of Theorem
19
A.2 Proof of Theorem
20
Proofs of Technical Results
22
B.1 Proof of Proposition]]
22
B.2 Proof of Lemmall
24
Examples of Extended Contexts Generated Using GPT-4
25
Expanded Discussion of Related Work
25
D.I Graph Perspective and Optimal Transport for Alignment Problems Combinatorial Alignment for Representation Learning
25
D.2
25
Medical Visual Chatbot
27
Zero-shot Image Classification as MedVQA
28
LLM Prompting for GPT-4 Generate Extended Captions
29
H Additional Results for Multi-modal Pre-training Comparison
29
H.T MedVQA datasets
29
H22 Results
31
Further Ablation Studies
32
K Nearest Neighbor in the Graph Construction Step
32
1.2 Feature representation analysis using average pooling for visual and language tokens
32
J
Qualification Test on the GPT-generated Extended Captions
33
18

---

A Proofs of the Main Theoretical Results
In this section, we provide detailed technical proofs of our main theoretical results. To accomplish this, we first introduce the fundamental concepts and materials that will be utilized in the proofs of Theorems 1 andd 2
We consider labelled graphs as tuples of the form 9 - Cy, where the labelling function C :. + assigns each vertex v; E V feature f; (v:))) in some feature space (F, dr). Similarly, we denote L. V + S as structure function which links each vertex es E V with its structure information 8; (u:), e.g. edge information, in some structure space (S, de). By associating a weight to each vertex, we allow the graph & to be represented by fully supported mixing measure
Ei=1 w.d( over the product between feature space and structure space F x S. Notably, fl is not necessarily a probability measure as the summation of its weights can be different from one. We have the vertex affinity matrix between two graphs as A" RMM,,, N where A = (dy j. Structural similarity is measured by pairwise distances within each graph, represented by A E R Eal, with A Al,, (d(s),, 5k) d, (s). where d.(-) models node distance, such the shortest path. We then define the space of all structured graphs (F x S, dd. over a metric feature space (F, dy) as S(F), where (S, d.) is a metric structure space and
EE W.d(Fas)) is mixing measure over F S.
A.1 Proof of Theorem 1
For the sake of simplicity, we denote the labeled graphs G and structured graphs discussed above only by f the whole structured graph.
To prove Theorem for any two graphs G1 and G2 in the structured graph space S(F), described respectively by their mixing measure > wird(frnas)) and #2 = ンÿ= (f2))))))))))) respec- tively, we wish to prove the following properties:
1. Positivity: dsGA G2) for any 91 + G2.
2. Equality relation: dSGA (9i, G2) 0 if and only if 91 G2.
3. Symmetry: dSSGA(1113 G2) dSGA (s,A(G2,91)) 91).
4. Triangle inequality: dsca (91, dsaa(S1, G2) dsas 9s) for any graph G3.
Note first that 1. Positivity and 3. Symmetry hold trivially.
Proof of 2. Equality relation. The equality relation immediately follows the following Proposition
I which is proved in Appendix B.I
Proposition (Equality relation). For any two graphs Gi and G2 in the structured graph space S(F), described respectively by their mixing measure μn = EM wisd(frorr) and P2 =
De= == W2j0 it holds dsGa (91, 92) if and only if M N and there exists a bijection
N
[ + [M] such that:
E1. Vi E [M] W1i W2)(())
E2. Vi E [M] f11 = [20(6)-
E3. Vi, k E [M]2 dd((11, = d. (8(00(()) 820(k))
Proof of 4. Triangle inequality. Let us consider two arbitrary graphs 91 and G2, described respectively by their probability measure μa = EM wird(f and = N (25 For any graph Gy described by its probability measure μ4 = ΣK= Wax (fsk we define P E A(G1, G2) and Q E AlG2, Ga) as two optimal couplings of the SGA distance between fn and μl2 and μ2 and μ3. respectively, i.e.,
M
N
P A() = Pe (0,1)MM
Σ
Pts = W1j = 1, Σ
P..j = W2 = =13
j=1
19

---

N
K Ui. waj
Q E A(G,,
Q (0,1)
E
Qj
122
1,
We then construct R =
Then it holds that R E A(9i, 93). Indeed, we have
way
1,k
En = jj.
Qjk
Qikk
ΣΣ
P.J.
Σ
wij
Wdm==
W2j
W2j
W2)
By the suboptimality of R. the triangle inequalities of dy and -1. we have
dsca(Gr (91,
Σ (dy(Sin Jaj) + (d1(,11 d, (s(j, sx)l] Ri. RE.
PraQu.
Pk. Qa.
[dy(fin. faz) + (1(s. - do(s3s s3]il] Σ Goikd
Σ
W2t
W2d
Qr.
Pa.aQaa
E iskdddd
[dy(fie, + (da(sii. 81K) - do(s3j, s3:)1l
W2r
Pr. PR.dQa.
Σ [dy(fu,, dy faj]
W22
w2d
ProQi.. Pa.aQa.
Σ
[1ds(s1.
d=(s2 s2a)l
[d, s2d)
ds(s3j- s3))l
W2c
W2d
iskhted
PuePui Qu.
= E
[ds(Jie, + - do (82e, s2a)ll
U22
W2d
PurQus PR.dQu.
Σ ..i.ttd
(ds();; + (d,(s2); 82a) - du (835, 831)1]
W22
W2d
Qu
Qa.
Σ
[dy(fis: f2x) (111, (s s2a)1] Prcckk. kdd
W2e
W2d
Pa.
Pk.d
Σ
[ds(Ser + Id (321, s2a) - do do(s3). s31)ll Qr Qa,1
102c
W2d
Note that we have
Qui
Qa.
Pa.
Pk.d 1. W2d
W2t
W2d
W22
This is how we achieve the desired result, because
dsGA (91, Ga)
Σ [fr f2x) + (ds($11881)) - do(s2e, s2a) Pr..
i.k.ddd
Σ
[dy(fer, fas) + (d2( s2a) - do(s2j, s31) Qu, Qu1
3.l.add
dsaa (91, dsas (92, 9a) (since P and Q the optimal plans).
A.2 Proof of Theorem 2
Theorem 2 enables us to characterise the optimal transport problem between two measures curve in the space of measures, with the objective of minimising its total length. Furthermore, this formulation is beneficial for deriving global minima results for non-convex particles in gradient descent in an optimisation context, which is valuable application of gradient flows [20]. By definition, a geodesic between 9i and G2 a shortest path between these two graphs. particular, the computation of distances along constant speed geodesic paths is a relatively straightforward process, as these paths are directly embedded into the real line as follows: dSGA(G1, 92) =
u| -'sGs (p(u), p(t)), for all 0 < # < and for any path (continuous map) connect G1 to
20

---

32 such that p(u) = 91 and p(t) = G2. To prove Theorem A.2 it is necessary to collect fundamental material using Definition 2 from metric geometry for general metric space (MI d).
Definition 2 (Length and geodesic spaces), Let (MI d) be metric space and two points x, y E M. We say that path (curve) p: [0, 1] + MI connect or join x to y if p(0) = z and p(1) y and is continuous map.
We also define the length L(p) E IR of a path p: [0, M as
L(p) sup Σ p(t+1))
where we take the supremum over all n 2 1 and all n-tuples ti <... < t, in [0, 1].
We denote metric space M. length space if for all x, M, d(x,y) inf, T(p) where the infimum is taken over all paths connecting x to y-
We call length space geodesic space for all x, E MI, there exists path p(x, v) [0, + M such that
d(x, v) min L(p(z, y)). p(a.y)
We also denote the path p(x, y) as a geodesic between x and Finally, define path p: [0, 1] + M. as constant speed geodesic if and only if d(p(u).p(t) = uld(p(0)) Vu,t E [0, 1].
For the proof of Theorem 2 we first consider an optimal coupling V for SGA distance between two graphs 91 and Gz i.e.
(9 G2)
min VEA(G, G2)
A V) 0(A A.."".
described respectively by their mixing measure Huu Σx= W1 (1yd( Moreover, for any t E [0, we define 14 : F such that
EM ( (fos
and #1
+ So x Si
M N
ve(fo. sp, f1, =
so, s1), and μ4 == -V# Σ
(1 ((0(0)))fotthh
and on the metric space Sg x S₁, we define the distance
de = -(1 (1-t)dsp @td,, (1 (1-t)da,, @td,, ((sor, Soj). (SIA, s1)) (1- (Sd, +tx) (Soj s1)) for any ((Sai, Soj): (81k, s1)) E Sg x Sy. Here, we denote # the push-forward operator such that d (A) = V ("T (A)) for any Borel sets of a o-albegra. For simplicity, we only consider (F,d)) = (RRd, 11 II) where || is the Euclidean norm.
Then we aim to prove that (F x Sp x S1, -1- t)da, @ td.,, Pt)tejo.1] is a constant speed geodesic joining (F x So, dso, po)) and (F x S1,da1,Pu )), for arbitrary elements (F x So, dao, (o)) and x Si,
in the metric space (S(F), dsGA).
To do so, we consider any u, t E [0, 1] such that + t. By definition, we have to prove that dsGA (Mus Me) It u/ u/dsGA (Ho,P)). p1).
Indeed, to prove equation
we first recall that
Pr = VV
Vi, (0
N
14 ::=U#VV
(1
dSGA (Po, (1) Σ
[dy f19) + |d, (Soi, S16) do(soj, (1)0))8811) V Vi
21

---

We then define the coupling 7" = (Ro x #o E A(pus H4). By the suboptimality of ym*f, it holds that:
d Me)
Σ
[dy(fos. f1z) + (de( Soj): (111; du ((S0e suj), ($1k. s1)) N4 Mk
u.t Tk.
Σ
dd{(11- - + tfis, (1 uo + ufis)
1(1 18dd,(00s (802; S11) + td (805. s11) (1 ())s(80i; 81k) udz (80j swile Vis Vhl
Σ [[e (For f12) ((t - -n)dd 81k) - (t (-)ddd((00; s1))l Vij Vie
m|
Σ
[dy(Jois (15) |d (50i; 81R) du (805s su)il Vi, Vi.t
i.j.k.l
= It - u/ uldsua(Ho M1). Here, we used the fact that d, is the Euclidean norm, hence dr((L_t)fo (1 (1-u) fo. tufig))) = [[(1-)fo: +tfy -1--fff---1--11 (L- ufiill = utt fiz). Therefore, we have
dSGA (Mess Me) S 1t - u dssa (Mo, p1).
(8)
The remaining task is to prove that
dsGa(Mus pt) |1 - u/ dsda (Mo.
(9)
To show that this inequality, note that via the triangle inequality of dSGA and for any 0 S S 1 holds that
dSGA (No, M1) S dsGA (No, Uw) + dSGA (Pus ms) dsGa (Me, 11)
S udssa (Jor (1) + (1 u)dsoa (10:114 + (1 t)dsGa (Mo,
dSGa (Ho, m1).
Hence, for any 0 S u St < 1, we obtain dsGa (Ho, pa) + dSGA (Juu Me) + dsGs (Mt, p1) udsan (No, m1) + (t (10, + (1 ()dsaa (No, 11). (10)
Suppose that
(Hus H4) < (t -u)dsaA (2))
Then combining with the fact that
dSGA (Hos Hu) S udsus (Ho, p1), and dsca ()4, m1) $ t)dsaa (Hos p1);
have
dsGA(Ho: Pa) + dssA((Hu pe) + dsGa (1) < (do P1) + (t - 2 u)dsGA (No, (1) + (1 +t)ssa (Ao M2).
This leads to the contradiction with the equation 10 Hence the desired inequality in 9 holds. Finally, we obtain
dsGa (Mus Me) It u/dsaa (Ho, p1).
(11)
B Proofs of Technical Results
B.1 Proof of Proposition]
1
First, let us suppose that dsGA (91, G2) = O. We wish to prove the existence of a bijection satisfying E1, E2, and E3. Indeed, let V A(91, G2) be any admissible transportation plan that encode valid matching between G1 and G2. Then we define:
d(s S1k)
[dy fix) + do s1)], Vi, [M]
(12)
d(s2j, s2))
[d/ fa) + ds s2)).
Vj. [M]
(13)
2
22

---

Recall that we then define SGM discrepancy as:
dsGA (G1,92)
min VEA(CGi.Ga)
Σ
Vis + Σ Alsiaaia L.R.
min Ve)(Cr
0(A", A V)
min VEA(G9192))
(A, V) O, V)).
(14)
It should be recalled that the vertex affinity matrix A RMxx defined as A = (dj(fi f2y))., was introduced in the previous section. The edge affinity tensor, denoted by A", is defined follows: Aig. = [d $1k) dz($2 s2)).
Let V* be the optimal coupling for dsGA (91, 92). Then we have
Oy(A", v-) (A V*)= VEA(S1.0))
min [0,(4", V) + O, (AA.))) V)] dsaa (91, 92) = 0.
(15)
Since both (A v*) and OA,A, V*) are non-negative, we conclude that Oy V*) (A, V*) O. Now we wish to use the following Lemma [B2 which is proved in Appendix B.2
Lemma 1. Given the definition of A, j.k. = Id(s 81k) - d(82j- 821)l where d(s11, 81k) and
d( 821) are provided in equations 12 and 13). respectively, it holds that
O,(A", V*)
AUSRI Vis VEL
[d(si;; 816) - d(s2j, 821) s2,) VV, S.j.ke
(16)
kjj.k.
Moreover, there exists a bijective a: : [M] + [N] with M = N satisfies the weight and distance preserving isometry as follows:
EI. Vi E [M] wii = (2)
E3*, Vi, k E [M] d(s1 $14) dl (1)) (2)()))
Because we have V* is the optimal coupling w.r.t. the distance d such that
0.(A* V*)
min OA(A", V) = 0, VEA(CL.G2)
(17)
V is supported by a and satisfies V = IMN x a. Therefore, O,(A", V*) = Eng (f(fss the fact that
E, di (fie f20()) E, Vig Σ, (f1n.fo(()) 0. Here, we used
V E A(G
V Ve (0.
Vis wij 1, Σ VUS W2:
Note that (fi, ( E [M] are all non-negative. This leads to df f2=(1) = 0, Vi E [M]. This is equivalent to fni f2o(0) Vi E [M] since d, is metric, which is the desired E2. Therefore,
we also have (fss fix) = dy (foo())) f2a())) Vi, E [M]. Combining equations 12 13). and
E3*, we have
1
d(s16 S1k)
2 (fi fik) + ds(s 8u)), 1
(18)
(()20()) (T(K)) 2 (dr(f2o()) f2o(s)) + d, d.( (2002)() Vi e [M]
(19)
This leads to the desired result, i.e., E3. d, (816, 81R) (.( (2)( Vi, k E [M]².
Now, let us suppose that M = N there exists a bijection a [M] H [N] satisfying E1, E2, and E3. We wish to prove that (S, G2) = 0. Then we can consider the transportation plan V = IMxN x a, Le., V is associated with i + and j + a(1). Using E1, it holds that V E A() Moreover, via E2 and E3, we also have
dsaa (91, G2) = min O(A", A V) < Σ
V + Σ A. VG Vi.
VEA(Gi.G.)
i.
Σ
(111
de(s S2)lV Val
k.j.k.l
23

---

= =Ia((fi (20t1) + Σ (ds(S - ds( s2a(k)) Viy 0.
1.j.k,l
This leads the desired result that dSGa (91, 92)
B.2 Proof of Lemmal L
By definitions and the triangle inequalities of the metric dy and ds. we have [d(sis, s14) d(s25, V2)] VE.
A,,,", VT
Σ is
Σ
+
+ dd 2 fai + do(s2)) 821]] Vi.
kj,kd
1
Σ
(fss fix) df fa)]
dd (515, do( s21)l Vi Vi.
2
$ 2
1
Σ
Vha +
1
Σ
Ido(s1c: d, (525, s21) Vi Vid
2
1
Σ
V. Vil + +2 (.((As)
2
Σ
- (f(s fai)l V.g VE, (since (A,, V*) = 0).
(20)
2
Using the triangle inequality of the metric d again, we have
dy(Sin: S1x) S dy(Sis f2j) + dz(S2s,
fai f2)
ds (f21: Sik),
dy(f2s, S (f f1i) + df fix) + (fi, fz)..
This is equivalent to
dy(Sis: fıx) dr (225, f2i S dd J2s) di(Jix, dy(ʃ2j, - (f fIx) S df(fi,, d (fik, f2).
(21)
We consider two sets 1 = ji. : d, (fr. fix) (J fx) S 0} and T2 (i,j.k,
df(f2s, f2i) - df fIx) 0}. Combining equations (20 and 21 it holds that
1 Σ
A V*) S 2
(dy(Sr, dz(s2s, f2i V
Light
Σ
[dg(f2s, - (f f1x)] V., Vi.
2
1
+
Σ
(ix(fin dy f2i) Vi Vi
2
Σ
[dy(fi, + dr (fix, f2)) Vi Vi.
2
k.j.klel,
Σ
[dy(fnn, + (f fa)) ViJ V
2
1
Σ
(dy(Sro + +d f21) Vi, Vi.
2
M
M
df(fis Vi' +
(fu = (A,, V*) = 0. (22)
2
2
RJj
Hence. O, V*) 0 since O,(A", V*) 2 0. Here, we have V* the optimal coupling such that
0, A V*) = min O,(A,, V).
(23)
(z)
24

---

Hence, in accordance with Theorem 5.1 from 665 66]. there exists an isomorphisms between the metric spaces associated with 91 and G2, described respectively by their mixing measure μl Ei (1:6(Sne.)))) and μl = Σÿ= W2y (2fd(Fay.sa)> This means that there exists a bijective with weight preserving isometry o [M] + M]. This implies that M = N and there exists a bijective M + [N] satísfies the weight and distance d preserving isometry follows:
E1. Vi E [M] : Wh, = (2)(())
E3*. Vi, h E [M]; d(s1s, $1s) = d( (( (2n(k))
C Examples of Extended Contexts Generated Using GPT-4
We present several examples of enriched captions generated using the GPT-4 API in Table These extended captions offer multiple advantages: (i) they enrich the model's ability to associate images with detailed, domain-specific descriptions that go beyond conventional captions; (ii) they better reflect weal-world medical workflows, where clinicians utilize domain expertise, thereby facilitating multi-scale understanding by bridging local and global features while reducing ambiguity in learning; and (iii) from a representation learning perspective, these captions diversify the embedding space and capture hierarchical relationships between input images and captions, potentially enhancing performance in complex pre-training tasks.
D Expanded Discussion of Related Work
D.1 Graph Perspective and Optimal Transport for Alignment Problems
EXGRA-MED formulates the graph alignment to solve the node-to-node correspondences under edge constraints indeed can be formulated using optimal transport, namely fused Fromov-wasserstern optimal transport (FGW-OT) [69 61]. However, two main challenges hinder from using optimal transport in EXGRA-MED:
First, performing the forward pass compute alignment between graph pairs using optimal transport is computationally expensive [69], making it impractical for scaling to large-scale LLM training with hundreds of thousands of samples. This obstacle is further challenged in EXGRA-MED, where three separate graph alignment problems have to be solved, sig- nificantly incurring computational costs. In contrast, adopting graph-based formulation enables the use of heuristic solvers specifically designed for combinatorial graph matching (90 [80, providing an efficient solution to address the high computational complexity of graph alignment.
Second, our training loss (Hamming loss Eq. requires gradients from graph alignments to leam feature representations. Using optimal transport would necessitate backpropagation through its Sinkhorn iterations (23 (50-100), adding substantial computational cost and GPU memory usage for storing intermediate variables. Our graph-base based formulation addresses the second challenge by leveraging modern gradient estimation techniques for black-box optimization 70 making the backward step efficient for L.M training.
D.2 Combinatorial Alignment for Representation Learning
Combinatorial graph alignment is key problem in computer vision, aiming to establish correspon- dences between elements of two discrete sets, such as image key points or 3D mesh vertices. is widely used in tasks like 3D vision [14. tracking (36), and shape model learning [32] In contrastive learning, LVM-MED [62] is the most relevant to our EXGRA-MA MED toward learning feature represen- tation by prok-propa through combinatorial training loss. However, there are key differences
between them:
Motivation: EXGRA- MED identifies the data- iata-intennsiv nature of auto-regressive modeling in LLa and addresses it by introducing a multi-graph alignment approach across vision, image captions, and extended captions. LVM-MED, in contrast, is designed as single-modality pre-trained model (vision) that learns through contrastive learning between images and their augmented versions.
25

---

Table 8: Example of a conversation extended with enriched caption explanations.
Original Conversation
User
What is the organ shown the image?
INSTRUCTION
The inage is CT scan of the abdenen and palvis, rocuaing on the adrenal glands.
The organ depicted in the image the adrenal glands, which located top of each kidnay. In this CT scan adrenal
and pelvis, glands appear as snall, triangula--shppe structures that play a crucial role hornone production, including cortisol and strenaline which are vital for stress reaponse and netabolian.
User
What do the black arows indicate?
black arrous the image are pointing to
image indicate bilateral adrenal entargenent, shich rignifies that both adrenal glands are enlarged bayond their normal size. This condition can be associated with varioua clinical implicatione, such hormonsl imbalances underlying diseases, and require the cause.
arroes
the bilateral adrenal enlargenent, which that both adrenal glanda larger nornal.
evaluation
User
Are both adrenal glands enlarged!
INSTRUCTION
Yes, both adrenal glanda appear to be enlarged scan. This bilateral adrenal enlargement may due various
Yea, both adrenal glands anlarged in the CT scan. This bilateral adrenal enlargement can
from
conditions, including: Adrenal adenonas Cuehing's syndrome Hyperplasia Metastatic disease Further evalustion by a healthcare professional is essential identify the apecific finding.
and further avaluation by a
healthcare professional is necessary to deternine the undarlying reason this finding.
User
What the age and gender the patient?
INSTRUCTION
The patient 23-year-old fenale.
patient 13-year-old indicating her age and gender clearly. information on essential for understanding her demographic profile clinical context.
User
What is the main finding the chest x-ray
INSTRUCTION
The main finding in the cheat x-ray
The main finding in the chest x-ray 1ymphadenopathy, which indicates the enlargement of 1ymph nodes. This condstion may suggeat undarlying isaues euch infection, malignancy, autoimmae disordere, varranting further investigation doternine the and appropriate nanagement.
lymphadenopathy, which
refera the anlargenent
of lynph nodea.
User
What the patient's medical condition?
INSTRUCTION
The patient has respiratory symptons and confirmed B1N1 infection. additionally, the patiant bas history of addiction.
The patient is diagnosed with respiratory confirmed HINI
symptona Additionally, the patient has infection. significant history of crack addiction, which may inpact their treatment options.
bealth and
Optimization Solver: EXGRA-MED integrates multiple modalities (vision, captions, and extended contexts) into barycenter graph-based solver, making it computationally efficient. LVM-MED relies on pairwise graph matching, which becomes computationally expensive for large-scale models. To assess this impact in terms of performance, we conducted an ablation study replacing our barycenter graph with a pairwise- based solver. As shown in Table (bottom row), this alternative approach resulted in lower records.
Graph Construction: EXGRA-ME MED works in the vision languae domain, making aug- mentation complex since captions must remain semantically meaningful. introduces extended contexts generated by LLMS (e.g., GPT-4, Gemini) to enhance representation learning. LVM-MED in the other direction works with the vision domain and thus can define two graphs one based on input images and another on their augmented versions.
26

---

EXGRA-MED introduces new theoretical insights, proving that its multi- gratt distance valid metric and demonstrating that the shortest path in the geodesic space of multi-modal graphs enhances learning. Opposingly, LVM-MED lacks theoretical contributions, focusing primarily on empirical performance.
Theoretical Analytic:
Performance Comparison: We compared EXGRA-MED which leverages triplet align- ment between images, captions, and extended captions, against the LVM-MED solver, which employs contrastive learning between images and captions. Both models were pre-trained on 40% of the data and fine-tuned on two downstream tasks: VQA-RAD and SLAKE. The results show that EXGRA-MED consistently outperforms LVM-MED achieving scores of 74.37 vs. 72.12 on VQA-RAD and 84.99 vs. 81.95 on SLAKE.
E Medical Visual Chatbot
Datasets. Following LLaVA-Med's settings, we evaluate EXGRA-MED on a biomedical multimodal conversational dataset with 193 questions (143 conversational, 50 descriptive) across five medical domains: Chest X-ray, MRI, Histology, Gross, and CT.
Table 9: Medical visual chatbot evaluation Results are reported using GPT-4 as the scorer.
#Para Question Type
Domain
Method
Overall
Conver.
Deser.
CXR
MRI Hist. Gross
CT
LLAVA
7B
39.40
26.20
41.60 33.40 38.40 32.91
33.40
36.1
LLAVA-Med 1.0
7B
47.4
33.99
51.31 36.32 45.61 41.09 44.87
43.93
LLAVA-Med 1.5
7B
46.78
34.58
54.58 36.5 41.85 40.3
45.02
43.62
MedFlamingo
8.3B
28.58
13.89
26.93 21.34 22.09 32.71
22.25
24.77
Med-Dr
40B
35.61
19.28
38.98
26.28 29.10 35.40
28.30
31.38
Biomed-GPT
182M
20.71
17.99
27.53
18.50 17.18 14.72
22.08
20.01
GPT-4o
200B
42.04
25.47
42.77
39.74 38.68 31.40 35.59
37.75
ExGra-Med
7B
48.49
34.32
58.37
36.82 46.05 45.19 38.24
44.82
ExGra-Med (DCI)
7B
48.99
34.01
59.9
32.34 51.88 42.53 38.28
45.11
Baselines, We evaluate with several SOTA multimodal large language models, including general models like LLaVA and GPT-4o, as well as medical-focused models such as LLaVA-Med and its variants, Med-Flamingo, Med-Dr, and Biomed-GPT. We use the officially provided weights for all comparison baselines without additional reproduction steps. The details of the evaluation protocol using GPT-4 as scorer are presented in the Appendix section.
Evaluation Protocol We evaluate the ability of models serve as medical visual chatbot as follows: each of the 193 novel questions in the dataset has a corresponding ground-truth answer. We generate responses to each question from the LMM. Next, ask GPT-4 to score the helpfulness, relevance, accuracy, and level of detail of the response from the LMM relative to the ground-truth answer. The score is, therefore, performance. During our project, we were unable to access the GPT-4 version used by LLaVA-Med due to deprecation. Therefore, we opt for the GPT-4o version as a judge. We also this GPT version to reproduce the results reported in the LLaVA-Med paper and observe a decrease in performance. This may be due to the fact that GPT-4o serves better judge than the previous version and thus judge the model's response harder.
scale of 0 to 100, where higher score indicates better overall
Results Table g shows the experimental results of EXGRA-MED alongside competitive methods, with the highest scores in bold. Our two method suriants-basee on LLaVA 1.5 with and without the DCI cactniiuu-ouutperrie
others conversation samples and achieve comparable results to LLaVA-Med 1. on description samples. In evaluations across five medical domains, our methods surpass the baselines in three (CXR, Histology, and Gross), positioning EXGRA- AMD the state-of- the-art overall. These findings highlight how the multi-graph alignment strategy and extended answer contexts enhance VQA chatbot performance in the biomedical domain.
Qualitative results are shown in Table TO where EXGRA-MED'S generated responses are compared against series of example questions and image contexts. The top and middle parts of the figure
27

---

illustrate a detailed description of a chest X-ray image, while the bottom part showcases a conversa- tional exchange regarding the origin and extent of mass found in a medical image. As demonstrated. EXGR A-MED effectively provides clinically relevant answers with high-level details, enhancing the interpretability and usability of its outputs. In cases where the given image and/or question ambiguous, as shown in the middle dialogue, EXGRA-MEL should produce a clear explanation without adding any additional information or showing any signs of hallucinations.
Table 10: Examples of the medical visual chatbot. Top: a detailed description for Chest X-Ray (CXR) image; Middle: a detailed description for CT scan image; Bottom: a conversation exchange regarding medical condition.
Detailed Description
Caption User
signs netastatic lesions Malk through important detaila
I-ray:
inage
EXGRA-MED
The inage is cheat X-ray (CXR) taken on the 1at postoperative day. provides vieu of the patient's chest, including the lunga, heart, and aurrounding structures. Since this postoperative inage, may patient'a ls condition after surgical procedure, monitor for any complications,
the
offectiveness of
Caption
Viral Staladenitiz. Axial computed tonography in 14-yoar-old female with bilateral parotid pain and swalling denonstrates eyametris trietric enlargenent of the parotid glands with subtle stranding adjacent (shite arrovs). Blood vork positive for parainfluenza virus.
User
Offer thorough analysis the
EXGRA-MED
the actual image, but help you interpret the inage based on the visual aspecta you describe. If you any questione
AI, cannot
content context the inage, feel free to ask, and I'll do best to provide assistance.
Answer (Short)
Chest X ray showing dilated cardiomyyopatyy
Conversation
Caption
computed tonography scan posterior costal are of the eighth rih on the patients left side and does not lnvolve the nedulla.
mass arises
User
What is the origin of the mass?
EXGRA-MED
The nass appears to be originating from the right lung. Does the nase involve the nedulla?
User
EXGRA-MED
According to the inage, the nass does not appear to involve the nedulla. The nedulla involvement would indicate that the has extended to this area. In this case, the mass seens to be affecting other parts the brainsten and carebellum.
part the brainsten, and its
(Short)
F Zero-shot Image Classification as MedVQA
Datasets. We assess the generalization of ExGRA- on zero-shot image classification by adapting public datasets from 34
We convert image classification tasks into visual question answering, where the model selects the correct answer from input options based on the given image (Figure We focus on three key data modalities prevalent in our pre- pre-training: Microscopy, Computed Tomography (CT), and Chest X-Ray (CXR). This evaluation spans several downstream tasks, including 8 datasets for Microscopy, for CT, and 11 for CXR, totaling 23 datasets.
Baselines. We use checkpoints from LLaVa-Med, Ked-Flamings and RadFM [96] for zero-shot inference on the collected datasets. Notably, RadFM pre-trained on 16M 2D and 3D medical scans, while EXGRA-ME MED is trained on just 600A instruction-follwwee data. For baseline models, we follow the prompts proposed by (34] with detailed evaluations using third-party software align model outputs with ground-truth answers.
Evaluation method. Following (34), we use Question-anae -anseeiinnnnwering Score metric report the
performance the models. Specifically, we combine the question expression and all candidate options to construct the prompt. Our prompt template therefore is as follows:
"This is a medical question with several Options, and there is only one correct answer among these options. Please select the correct answer for
28

---

OmniMedVQA Benchmark
45
LLaVA-Med
Med-Flamingo
RadFM
ExGra Med
40
37.3
35.0
35
33.7
32.0
32.0
30
29.1
28.8
Accuracy (%)
26.1
27.7
25
23.2
17.5
15
5
# images
4,906 5,531
2,044 3,281
5,752
# QA items
7,245
CT Figure 4: EXGRA-MED performance 23 zero-shot image classification tasks within three data
Microscopy
CXR
modalities.
the question. Remember, you can only select one option. The Question is: (Question). ### The candidate Options are: (Options))
The MLLM receives this prompt and corresponding image and is asked to generate a response. We then utilize difflib, standard Python package to compare two strings, to calculate the similarity of the response with each the candidate options and pick the option with the largest similarity as the final prediction. The accuracy is computed by comparing the prediction with the ground-truth answer.
Results. Figure fillustrates the average performance of ExGRA-MEET across Microscopy, CT, and Chest X-Ray modalities, with the total number of images and question-answe items listed below. Detailed results for each dataset within these modality groups Overall, EXGRA-MED still outperforms other models across all datasets, especially excelling microscopy modality, where it exceeds the runner-up, RadFM, by 8.2%. We attribute these benefits to the strong alignment between visual features and language embeddings achieved through triplet constraints, which compel the model to capture deeper semantic relationships.
provided in Tables I1 [3 and 12
Figure provides several examples of microscopy and CT images. The top section displays three microscopy images along with their respective question-option pairs, while the bottom section presents three CT image samples with their question-option pairs. The ground truth correct options are highlighted in blue. In total, the number of images and guestion-answeringg items across the three groups various medical image modalities are shown Figure
Results We provide detailed results for datasets on each data modality in Tables II
13 and 12
G LLM Prompting for GPT-4 to Generate Extended Captions
We illustrate in Figure how to leverage the GPT-4 API to analyze and extend the original answers. For detailed responses specific cases, refer Table g
H Additional Results for Multi-modal Pre-training Comparison
H.1 MedVQA datasets
We train and evaluate ExGra-Med on three biomedical VQA datasets, including VQA-RAD, SLAKE, and PathVQA. The dataset statistics are summarized in detail in Table 14
29

---

Table 11: Performance comparison on various microscopy image datasets.
Microscopy
Method
Image Dataset
LLa
Med-Flamingo
RadFM
Ours
CRC100k
24.74
17.18
27.48
28.06
ALL Challenge 28
29.24
13.20
39.88
27.49
Tichh 67
39.14
16.08
47.84
46.97
Blood Cell 3
21.11
15.25
16.95
29.87
BreakHis
86
23.27
13.62
18.26
33.74
NLM-Malaria
30.67
6.76
32.43
66.67
HuSHeM 82
16.85
18.18
11.36
25.84
MHSMA
38
29.64
39.66
38.41
39.70
Avg.
26.83
17.49
29.08
37.29
Table 12: Performance comparison across CXR datasets.
Method
CXR Dataset
LLaVA-Men
Med-Flamingo
RadFM
Ours
RUS CHN 6
28.05
20.19
29.88
41.88
Mura 78
20.70
25.91
43.47
30.19
Pulmonary Chest MC
337
21.05
27.03
10.81
47.37
MIAS 87
25.35
38.30
28.37
42.96
Pulmonary Chest Shenzhen 37
26.35
32.54
36.95
19.93
COVIDx CXR-4 (95
28.25
25.83
48.14
22.68
Knee Osteoarthritis
11.20
22.24
6.19
8.69
Chest X-Ray PA 12
29.06
38.04
38.28
49.41
CoronaHack [22
19.74
33.67
22.99
47.81
Covid-19 tianchi
16.67
45.26
33.68
30.21
Covid19 heywhale 21
22.03
56.31
23.37
29.28
Avg.
23.18
32.01
28.84
33.67
Table 13: Performance comparison various CT (Computed Tomography) datasets.
Method
CT Dataset
LLaVA-Med
Med-Flamingo
RadFM
Ours
Chest CT Scan EARS-CoVAL CT
25.72
20.00
25.06
20.09
28.79
40.92
44.55
34.95
Covid CT 2
22.61
21.72
28.79
37.19
OCT & X- X-Ray 2017 44
27.21
28.08
29.46
47.89
Avg.
26.08
27.68
31.97
35.03
VQA-RAD dataset is collection of 2248 QA pairs and 515 radiology images which are evenly distributed over the chest, head, and abdomen. Over half of the answers are closed-ended (i.c., yes/no type), while the rest are open-ended with short phrase answers.
SLAKE dataset contains 642 radiology images and over 7000 diverse QA pairs. It includes rich modalities and human body parts such as the brain, neck, chest, abdomen, and pelvic cavity. This dataset bilingual in English and Chinese, and in our experiments, we only considered the English subset.
PathVQA dataset contain pathology images. It has a total of 32795 QA pairs and 4315 pathology images. The questions in this dataset have two types: open-ended questions such why, where, how, what, etc. and closed-ended questions.
30

---

System Prompt
You possese in-depth biomedical knoxledge checking the quality of the answer to given instruction. From the given impat, which is pair of instruction and answer, your tack involves the rollowing stepe:
Explain why the given
good for instruction. Please analyze based the Helpfulness, Relerance, Accuracy, Level of Betail, and Structure fields.
2. Gonerate a hetter ansver based on the reasons pointed out above, while preserving the level details, add bullet compreheneive words, etc. Because these ansvers about bionedical knowledge, you must keep all the medical terninology and inportant words in the nev The bettar inage and ansvering the question.
aane content. To achiove that, you may vant to adjuat points,
better
should be
that you also seeing the
3. Output JS08 object containing the following kaya (note that double quotes should not be ueed)) "explamation": ("helpulaees"" "eefulleee""Kommen on helpfulness, max tokens?, <eommennt relevance, nax 20 tokens?, sccuracy" "eruucture"<<camnee" on max 20 tokens>, "detail":<commat" on datail, max 20 tokens>,
accuracy,
structure, 20
},
"revision": <impreved version of the answer, max 2x tokens of imput if 2 tokens, otherwise tokens>
Figure 5: Instructions provided to the system for analyzing the quality of answers based on different criteria and generating revised response in JSON format.
Table 14: Dataset statistics for 3 medical VQA datasets: VQA-RAD, SLAKE, and Path VQA.
VQA-RAD
SLAKE
PathVQA
Dataset
Train
Test
Train
Val
Test
Train
Val
Test
# Images
313
203
450
96
96
2599
858
858
# QA Pairs
1797
451
4919
1053
1061
19755
6279
6761
# Open
770
179
2976
631
645
9949
3144
3370
# Closed
1027
272
1943
422
416
9806
3135
3391
H.2 Results
Tables 15 and 16 present the results using 70% and 100% of the data. Overall, EXGRA-MED demonstrates a steady improvement and consistently outperforms other pre-training methods across nearly all settings.
Table 15: Performance fine-tuning on MedVQA downstream datasets (pre-training 70%). Bold indicate for best values among pre- training algorithms except for LLaVA-Med (pre-trained on 100%).
Method
VQA-RAD
PathVQA
Overall
Open
Closed
Avg.
Open
Closed
Avg.
Open
Closed
Avg.
LLaVA Med (100%)
63.65
81.62
72.64
83.44
83.41
83.43
36.78
91.33
64.06
73.37
LLaVA-Med (20S))
65.961731
73.79
84.1670m3
83.17/jo3
InfoNCE
64.18
77.94
71.06
70.9
82.69
76.80
33.58
88.5
61.04
69.63
PLOT
60.13
78.31
69.22
82.48
83.89
83.185
29.23
85.7
57.478
69 96
SigLIP
VLAP
64.08
79.41
71.75
84.94
85.1
85.02
36.44
91.51
63.98
73.58
ExGra-Med
67.12
81.99
74.56
84.81
84.86
84.84
37.26
91.77
64.52
74.64
Table 16: Performance fine-tuning MedVQA downstream datasets (pre-training 100%).
VQA-RAD
SLAKE
PathVQA
Method
Overall
Open
Closed
Avg.
Open
Closed
Avg.
Open
Clused
Avg.
LLLVA-MMd (100%)
81.62
91.33
73.37
InfoNCE
66.01
79.41
72.71
83.23
83.41
83.32
35.01
89.53
62.27
72.77
77.21
82.44
83.65
62.21
72.09
SigLIP
57.11
74.26
65.69
85.07
83.41
84.24
36.47
89.38
62.925
70.95
VLAAP
79.78
84.74
83.17
83.955
62.755
72.36
ExGra-Med
66.35
83.46
74.91
85.34
85.58
85.46
36.82
90.92
63.87
74.75
31

---

I Further Ablation Studies
I.1 K Nearest Neighbor in the Graph Construction Step
We conduct experiments to assess the impact of different K values in the graph construction step. Table 17 presents model performance on the VQA-RAD dataset along with the training time for Step-2 pre-training using 10% of the data for each K value. Our findings indicate that K = achieves the best balance between performance and efficiency.
Table 17:
Impact of Nearest Neighbors Count on Graph Construction. Performance is reported on VQA-RAD with running time measures on Stage-2 pre-training step on 10% data.
VQA-RAD
Settings
Open
Close
Avg.
Run Time
ExGra-Med (Full),
55.9
73.9
64.9
Ih
ExGra-Med (Full), 5
66
79.04
72.52
1h4'
ExGra-Med (Full), K =
55.52
73.16
64.37
1h17'
Table 18: Comparison of pre-training algorithms with different feature embedding methods. Models are pre-trained on 40% of the data and evaluated on the average performance across three medical visual guestion-answering datasets.
Method
VQA-RAD SLAKE
PathVQA
EXGRA-MED
74.37
84.99
64.34
InfoNCE (avg.feature)
70.34
83.29
61.6
PLOT (optimal transport)
71.89
83.16
62.4
I.2 Feature representation analysis using average pooling for visual and language tokens
We investigate using average pooling token features in EXGRA-MED with experiments:
We trained ExGRA- on 70% of the pre-training data, randomly sampling 1000 unseen image-text pairs. The trained model extracted features using average pooling, and a box plot (Figure 6 visualized the central tendency, spread, and skewness of 1000 positive and negative pairs. The results show: (i) the median similarity for positive pairs is significantly higher than for negative pairs, indicating clear separation; (ii) while some overlap exists in the interquartile ranges (IQRs), the shift in central tendency confirms the distinction; and (iii) outliers are present, particularly among negative pairs, but they minimally overlap with the core distribution of positive pairs.
Doxplct of Positive Negetive Siillorty Unseen
0.
08
0.7
08-
0.5
Posttiv
Megaltive
Category
Figure 6: Visualization of similarity values between positive and negative pairs based on features computed by EXGRA-MED.
32

---

We compare against two pre-training algorithms, InfoNCE 45 utilize the same contrastive loss, but InfoNCE relies on cosine distance with averaged features, while PL.OT directly computes optimal transport over sets of visual and text tokens. The results for these baselines are summarized in Table We observe that using a more sophisticated distance metric, such as optimal transport, provides a slight improvement (around 1%) over the averaging approach. However, the performance gain is relatively modest. Based on the above evidence, we conclude that using average pooling for distance feature extraction a reasonable and practical approach.
58 and PLOT [1]] Both
Qualification Test on the GPT-generated Extended Captions
We adopt the GPT-4 as a tool for paraphrasing image captioning due to its improved performance compared with GPT-3.5, especially in healthcare [40]. During our implementations, we also randomly checked for a hundred samples and found consistency between extended context and original ones. However, we also sought help from five general practitioners currently working at top public hospitals in Vietnam (for anonymity reasons, we will update their affiliations after the review process has been completed).
In particular, we randomly chose 1000 samples in Stage 2 of pre-traaiie pre-training, covering five data modalities: chest X-ray, CT scan, MRI, histology, and others. Each doctor is assigned specific data modality given their expertise, including 200 image-text pairs and corresponding captions. We then build an annotation tool for them to verify data where each sample is asked with two questions (i) whether the extended caption covers the original caption; and (ii) whether new concepts appearing in extended captions are correct. For (i) and (ii), doctors can rate with five levels (from I to 5), each indicating an increasing level of correctness (Figures) 7
We provide statistical correctness evaluated by general doctors for these domains in Figures 10 13 and [4] It can be seen that most rating scores fall between and 5, with only a small number of samples rated or 2, validating the overall consistency of GPT-4 outputs. While concerns may arise regarding the impact of low-scoring extended captions (rated or 2) on the L.M,, it's important to note that these extended captions are utilized solely for contrastive learning during pre-training to align the model's 's latent space representations. They are not used in ruto-segressive training, which involves predicting target ground-truth tokens. Additionally, the model is fine-tuned with the given training samples from downstream tasks after pre-training (no extended captions are used). Thus, we argue that the presence of a small number of noisy extended captions should not significantly affect the performance of the LLM.
H ExGra-Med Rating: ct scan R
herbol
Teatbo
Terbao
There ditt seeeeeetccoon ceoo apanmpiuul
A seropt nat alscoo from yourr cruoe,
2130000 F3c,pen
infectios, auch ylcoccas arwuee
Start yecesco
pockes
Siro Secckwocrs pertio which can sercall oornr
wich arout from enjeent in uotarm or
thouug
Singia cam
infucion..
comgle prrgriiekee en wi gyme on duringeen selertee
Kemee indeee be morey ven, cor
disitt. in pertanttto wheel
antiation th underfying
bant the a aacuer?
peat weturatttie the of merte uniieeeee
budi an paal wnaaeaaeeitte encembiit.
INPORIATTON connett
Wey
Peor
Wedertt
Figure 7: (Part 1) Demonstration of our annotation tool for general practitioners to validate the quality of extended captions generated by GPT-4.
33

---

Suidelines
Mery Feor Conditiany
coptoonggsiin sightiiatty wrgest the ongins recering
Daampfes:
oriinnlcaption
desalle inpyiipa
Coes mun saln coferracc whth ong condent...
descri rioes enteedaded captionmaaie cemen demnenti oftihe cww.gracltiinnntnaooooooooo http://wwww...oooeeooocccaoooooo www.ttteeeeeeeeeeeeeeeoooo. conten thorew
www.oeeoyooooooeeeoeeoeoooooo theorgg ong contense ireeewrrt
Dumplees
Part ohehe
mitoor thrsuld seroocer coweations.
ensianee
3: Medorate Consistoncy
mpttoonmnnt slig wigiisl caps
righty
adds ary resettive
eoverty seicand reciical
Conditerncy
Daample::
orighttt
Mamuano ligh fectaal exurcsy and will
Comuenency
Seariiption: vdwaeed
criginal reviching www.eaateeeeeeecoo
Freharon wwwp
Euamples
cuptoon
Rully accusata and train darity and iile aca
Figure 8: (Part A detailed guideline for scoring, ranging from I to 5, is provided.
34

---

Q: What is the diagnosis of the histopathology in this image? Breast hyperplasia without
Q: What are the types of cells de- picted in this image?
Q: What is the probable diagnosis depicted in this image?
atypia histopathology
Neutrophils
B: Normal breast histopathology
Chronic myeloid leukemia
Melanocytes
C: Benign breast histopathology
B: Multiple myeloma
C: Lymphocytes
D: Fibrocystic breast histopathol-
C: Hodgkin' 's lymphoma
D: Hepatocytes
ogy
D: Acute lymphoblastic lcukemia
Q: What the diagnosis cancer seen in this image?
A: Adenocarcinoma of the right hilum. T3 NI MO. Stage Ilb Mesothelioma of the right hilum, T2 MO, Stage IIb
Q: Which imaging technique was utilized to obtain this image?
A: Ultrasound
C: Large cell carcinoma
left
Q: Is COVID-19 apparent this
B: Optical Coherence Tomography
hilum, T2 N2 MO, Stage IIaa
CT scan image?
C: Magneticc Resonance Imaging (MRI)
Non-small cell carcinoma of the left hilum,
A: No
MO, Stage
Yes
Thermography
Figure 9: Examples from the OmniMedVQA dataset: microscopy (top) and CT images (bottom) with corresponding questions and options, with the correct answers highlighted in blue.
Correctness of new Information for Chest X-ray
Inclusion for Chest X-ray
100
50
60
10
Figure 10: Statistical correctness of extended captions generated by GPT-4 on Chest X-rays.
35

---

Correctress of new informationn for CT Scan
Inclusion for CT Scan
129
100
150
20
Scones
Figure 11: Statistical correctness of extended captions generated by GPT-4 on CT scans.
new
140
120
100
#halnngs
Figure 12: Statistical correctness of extended captions generated by GPT-4 on MRI.
Correctness of new Information for Grass
Inclusion for Gross
100
140
170
100
Figure 13: Statistical correctness of extended captions generated by GPT4 on mixed domains.
36

---

Correctress of new information for Nistology
Inclusion for Histology
70
80-
60
20
20
Scones
Socres
Figure 14: Statistical correctness of extended captions generated by GPT-4 on histology samples.
37

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
We provided experiments to support our claims.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We included it at the end of the paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
38

---

Justification:
All assumptions are already stated.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: we provided experiment descriptions in the main paper and appendix. Further- more, we will release GitHub implementation if the paper is accepted.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
39

---

Answer: [Yes]
Justification: We will publish the code when the paper is accepted or through an anonymous link if some reviewers ask for it.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https
ssicnyolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provided details for experiments in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We conduct experiments on diverse datasets and follow the protocol used by previous works for fair comparisons.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
40

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
This information is included in our results.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httppp////.errre...coo/o//////aa
Answer: [Yes]
Justification: Our paper follows the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Our work helps to reduce the carbon footprint when training large models using ViT. There no negative societal impact.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
41

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work does not pose such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have properly cited papers and resources used in experiment.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
42

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/mmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: We don't have experiments involving crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation,
or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
We do involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
43

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [Yes]
Justification: We use LLM to paraphrase the caption in the medical instruction-follee following dataset, which will be used to do cross-alignment among modalities, improving model feature representations. The details for this step are provided in the paper.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, ntn-standard components.
Please refer to our LLM policy htteop///eaaageecccco/ofrtrrrrrs for what should or should not be described.
44


Reviews:
[{"review_id": "gUny0G3TNr", "reviewer": "Reviewer_Usn8", "summary": "This paper presents ExGRA-Med, a multi-graph alignment framework designed to improve vision-language alignment in medical multimodal large language models. The authors address a significant limitation of existing approaches (e.g., LLaVA-Med, BioMedGPT) that suffer from weak cross-modal alignment due to their heavy reliance on autoregressive training objectives.", "questions": "1. The proposed method uses GPT-4o-powered extended context instruction-following data for multi-graph alignment. How does this compare with methods that directly optimize instruction-following data using GPT-4o?  Direct comparison with GPT-4o optimization methods would strengthen contribution claims.\n2. Current medical VQA datasets are typically more complex than the Figure 2 examples. Can your method maintain effectiveness on sophisticated medical datasets involving complex clinical reasoning? Demonstrating effectiveness in complex medical scenarios would enhance practical significance.\n3. Please incorporate more challenging benchmarks such as MMMU Health & Medicine, MedXpertQA, and OmnimedVQA to better assess model capabilities. Results on advanced benchmarks would provide stronger evidence of superiority.\nLimitations\n4. The complex alignment architecture requires more thorough validation beyond simple benchmarks. Effectiveness has not been sufficiently demonstrated across diverse scenarios.\n5. Unclear whether performance gains persist with stronger base models or comprehensive VQA corpora. Robustness to state-of-the-art foundation models needs validation.", "limitations": "1. The complex alignment architecture requires more thorough validation beyond simple benchmarks. Effectiveness has not been sufficiently demonstrated across diverse scenarios.\n2. Unclear whether performance gains persist with stronger base models or comprehensive VQA corpora. Robustness to state-of-the-art foundation models needs validation.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The proposed multi-graph alignment mechanism effectively tackles the vision-language alignment problem by jointly aligning images, instruction responses, and extended captions in the latent space.\n2. Demonstrates impressive data efficiency, achieving comparable performance to LLaVA-Med using only 10% of the pre-training data.\n3. Shows substantial improvement (20.13% gain) on VQA-RAD benchmark, indicating better medical visual question answering capabilities.\n4. The black-box gradient estimation approach enables scalable training for large LLMs.\n\nWeakness:\n1. The proposed multi-graph alignment framework introduces significant architectural complexity that makes the paper difficult to follow and understand. The intricate design may hinder reproducibility and practical adoption due to unclear implementation details and a lack of intuitive explanations. \n2. The proposed approach extends VQA capabilities primarily through simple conversational formats as shown in Figure 2, limiting applicability to complex medical reasoning scenarios. The evaluation lacks assessment on more challenging conversational AI tasks that require sophisticated multi-turn dialogues or clinical decision-making processes. \n3. The experimental evaluation is constrained to relatively simple benchmarks (VQA-RAD, SLAKE, PathVQA) and primarily compares against LLaVA-Med, an earlier generation model. The absence of comparisons with recent state-of-the-art medical multimodal models limits our understanding of the method's true competitive standing.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the thoughtful and detailed feedback. We greatly appreciate your recognition of the strengths of our work, including the effectiveness of the multi-graph alignment mechanism, the strong data efficiency, the significant gains on VQA-RAD, and the scalability enabled by our black-box optimization approach. We also appreciate the constructive suggestions regarding architectural clarity, evaluation scope, and broader applicability. Below, we address each of these concerns in detail.\n\n**Q1. Architectural complexity of multi-graph alignment, reproducibility, and practical adoption**\n\n**A1**. We thank the reviewer for highlighting these important aspects. Regarding the **architectural complexity** of the multi-graph alignment module, if the paper is accepted, we will simplify the mathematical formulations and include more intuitive explanations - such as **visual illustrations and video teasers** on the project website to make the method easier to understand and follow.\n\nIn terms of **reproducibility** and **practical adoption**, we are committed to open-sourcing:\n- The **pretrained checkpoints** are used for downstream adaptation tasks, and\n- The **complete training pipeline** for multi-graph alignment\n\nThis will ensure that our method can be fully reproduced and easily integrated by the research community.\n\n**Q2. Limitation of ExGra-Med’s conversational format in handling multi-turn dialogues or clinical decision-making**\n\n**A2**. We thank the reviewer for this question. We acknowledge that Figure 2 provides only a simplified illustration of how we extend the original captions. However, in practice, ExGra-Med is capable of handling multi-turn dialogues, as it is trained on curated multi-turn conversational data provided in the pre-training set of LLaVA-Med (Figure 5 in the LLaVA-Med paper).\n\nSpecifically, for each input image, medical experts generate sequential follow-up questions based on the model’s prior responses, enabling the system to learn context-aware dialogue behavior. This design allows ExGra-Med to better support extended interactions that resemble clinical reasoning and decision-making processes.\n\n**Q3. Evaluations are limited to simple benchmarks (VQA-RAD, SLAKE, PathVQA) and primarily compared against LLaVA-Med. More challenging benchmarks such as OmmiMedVQA and recent SOTA medical multimodal LLMs are needed.**\n\n**A3**. We thank the reviewer for these valuable suggestions and fully agree that including more challenging benchmarks and broader comparisons strengthens the paper’s contributions.\nFortunately, we have already included these evaluations but may not have highlighted them clearly enough in the main text:\n\n(i) The zero-shot results on **23 segmentation tasks** are actually sub-tasks **derived from the OmmiMedVQA benchmark** (Appendix Section G, Figure 4). Additionally, we conducted a **medical visual chatbot evaluation** (zero-shot) using **GPT-4 as the evaluator due to the complexity of the model responses** (Appendix Section F).\n\n\n(ii) We have also compared ExGra-Med with **state-of-the-art medical MLLMs** beyond LLaVA-Med, such as **RAD-FM** (pre-trained on 16M 2D/3D scans vs. 600k for LLaVA-Med), **MedDr** (40B parameters vs. our 7B model, trained on 2M instruction-tuning samples), and **GPT-4o API** for zero-shot evaluation on the medical visual chatbot task.\n\nBeyond medical MLLMs, we have **benchmarked the multi-graph alignment** module itself against advanced alignment algorithms across different domains such as **PLOT, SigLIP, VLAP (2-domain alignments)**, and **GeoCLAP, PAC-S, ImageBIND (3-modality alignments)**.\n\nIn summary, we believe these diverse benchmarks (from standard to highly challenging) and comparisons with multiple SOTA baselines effectively validate ExGra-Med’s strengths. If the paper is accepted, we will revise the manuscript to emphasize these points more clearly to address the reviewer’s concerns.\n\n**Q4. Comparison with baselines trained using GPT-4-generated instruction-following data, not just used in multi-graph alignment**\n\n**A4**. We appreciate the reviewer’s suggestion and agree that clarifying this point is important. We would like to clarify that we have **already conducted analyses** to assess the effect of using GPT-4-generated extended instruction-following data beyond just multi-graph alignment.\nSpecifically:\n\n(i) We trained **LLaVA-Med models using the GPT-4-extended instruction data**, and the results are presented in Table 3, denoted as LLaVA-Med (10%) ext. cap and LLaVA-Med (40%) ext. cap, representing training with 10% and 40% of the extended captions, respectively.\n\n(ii) For **all multi-modal pretraining algorithms in Tables 1 and 2**, including ExGra-Med and relevant baselines, the **same GPT-4-extended instruction-following data** were used to ensure a fair comparison. This setup helps isolate the effect of the multi-graph alignment itself, and results confirm the consistent improvements of ExGra-Med across all three VQA datasets.\n\n\nWe will revise the manuscript to better highlight these evaluations and ensure this point is clearly communicated to readers.\n\n\n**Q5. Unclear whether the performance gains of ExGra-Med are robust across other foundation models beyond LLaVA-Med**\n\n**A5**: We thank the reviewer for raising this important point. To evaluate the robustness of ExGra-Med beyond the LLaVA-Med backbone, **we have integrated the multi-graph alignment framework to the Qwen-VL 7B model**, a strong vision-language foundation model.\n\nWe compared the following three configurations:\n- Qwen-VL 7B baseline without additional pretraining on medical instruction-following data,\n\n- Qwen-VL 7B pre-trained with 10% GPT-4-enhanced medical instruction-following data, and\n\n- ExGra-Med uses the Qwen-VL 7B architecture, also pre-trained with the same 10% instruction-following data, but incorporating our multi-graph alignment strategy.\n\nAll models were fine-tuned on the VQA-RAD and SLAKE datasets (closed-ended questions, evaluated with accuracy). Our results show that ExGra-Med consistently improves performance over both baselines on these benchmarks, with gains of 3–6%, demonstrating that our approach is model-agnostic and generalizes well across different foundation architectures.\n\nWe will add these findings more explicitly in the revised manuscript to highlight the generalizability of our method.\n\n| Model                                           | VQA-RAD | SLAKE  |\n|------------------------------------------------|---------|--------|\n| LLaVA-Med (medical pre-trained 10%)            | 61.4%   | 80.3%  |\n| **ExGra-Med (LLaVA** architecture, pre-trained 10%)| 79.4%   | 85.10% |\n| Qwen-VL (general weights)                      | 53.85%  | 70.5%  |\n| Qwen-VL Med (medical pre-trained 10%)          | 52.73%  | 86.53% |\n| **ExGra-Med (Qwen-VL**, pre-trained 10%)           | 58.10%  | 89.75%    |"}, {"review_id": "aklEaXBUtA", "reviewer": "Reviewer_Fpu5", "summary": "This paper introduces EXGRA-MED, a novel multi-graph alignment framework for medical vision-language models that addresses the data-hungry nature of autoregressive training in medical multimodal LLMs. The authors identify that models like LLaVA-Med suffer significant performance drops when trained on limited instruction-following data, even after fine-tuning. To address this, EXGRA-MED constructs three modality-specific graphs representing visual features, original instruction responses, and GPT-4 generated extended captions, then jointly aligns them using a barycenter graph approach with black-box gradient estimation for scalable training.", "questions": "1. You mention the ExGra-Med's language model backbone is LLaMA-7B, but which LLaMA version exactly? From what I know, LLaVA-med uses mistral-7b-v0.2 as the language model backbone. Usually better language model backbone will result in better downsteam performance, will that cause unfair comparison? \n2. Please try using the term \"pre-training\" for stage 1 and \"fine-tuning\" on stage 2 properly, according to the original LLaVA paper. On line 281, I assume you want to say \"complete the whole fine-tuning process compared to the LLaVa-Med\".", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "## Strengths\n1. The paper clearly demonstrates the data-hungry nature of autoregressive training in medical MLLMs through systematic experiments showing dramatic performance drops (e.g., 72.64% to 52.39% on VQA-RAD) when using only 10% of pre-training data.\n2. The novel fine-tuning with multi-graph alignment\n\n## Weaknesses\n1. Training efficiency analysis. According to section 4.1 the actual training time of ExGra-med is 0.5 hour longer than the original LLaVA-Med. How much additional cost (e.g. GNN training) will the multi-graph alignment training will bring should be clarified.", "was_revised": true, "final_justification": "The rebuttal solves my concerns", "rebuttal": "We thank the reviewer for the encouraging feedback and thoughtful observations regarding both the strengths and areas for improvement in our work. Regarding the training efficiency concern:\n\n**Q1. Clarifying additional costs of 0.5 hour longer of ExGraMed compared to LLavA-Med.**\n\n**A1**. We thank the reviewer for raising this important point. The **additional 0.5-hour training time** required by ExGra-Med compared to LLaVA-Med is primarily **due to the multi-graph alignment module**. This includes the following computational steps:\n\n- Constructing graphs using the **K-nearest neighbors** algorithm,\n\n- Performing **GNN-based message passing** to aggregate features among nodes in each graph,\n\n- Solving **three graph alignment** problems during the forward pass using a barycenter graph, and\n\n- Executing the corresponding **backward steps** to compute gradients for optimization.\n\nTo ensure that the observed performance gain of ExGra-Med is not simply due to longer training time, we conducted an additional ablation experiment where **LLaVA-Med was trained for one extra hour**. As shown in **Table 3**, ExGra-Med still outperforms LLaVA-Med under these settings, confirming the effectiveness of our proposed alignment method beyond just training duration.\n\n**Q2. Clarifying the language models LLAMA versions used in ExGra-Med and LLaVa-Med**\n\n**A2**. Thank you for your careful evaluation and valuable question. To ensure a fair comparison, we confirm that both **ExGra-Med** and **LLaVA-Med** in our study are built upon the **same original LLaMA-7B model**, as provided on Hugging Face [1]. Specifically, **we do not use** any variant such as the **Mixture-of-Experts** mistral-7B-v0.2 checkpoint. This ensures consistency and fairness across all experiments.\n\n[1] huggyllama/llama-7b\n\n**Q3. Make consistent use of the terms “pre-training” and “fine-tuning”**\n\n**A3**. Thank you for the helpful comment. We acknowledge the potential confusion caused by the use of the terms “pre-training” and “fine-tuning.”\n\nFollowing the original LLaVA paper (Figure 3), Stage 1 is intended for the medical concept alignment, and Stage 2 is used for fine-tuning on generic medical instruction tuning. **However, both these Stages indeed can be seen as the pre-training phase** because the model is still not fully optimized for specific downstream tasks. **Additional fine-tuning** is required on each target dataset (e.g., VQA-RAD, SLAKE, or PathVQA) to achieve the best performance.\n\nRegarding line 281, we intended to state that ExGra-med takes slightly longer to complete the full pre-training process (Stage 1 + Stage 2) on the 600k medical instruction tuning samples compared to LLaVA-Med.\n\nWe will revise this sentence for clarity and ensure consistent use of terminology in line with the original LLaVA definitions throughout the manuscript."}, {"review_id": "7psebrrqkm", "reviewer": "Reviewer_xkmU", "summary": "This paper proposes EXGRA-MED, a training-efficient framework for medical vision-language models that aligns images, instructions, and enriched captions through a multi-graph alignment strategy. By combining this with black-box optimization, the model achieves strong performance with significantly less instruction-following data, outperforming baselines across multiple medical VQA and classification tasks.", "questions": "- Could the authors provide quantitative results when training EXGRA-MED with smaller frozen LLMs to evaluate downstream performance trade-offs?\n\n- How robust is EXGRA-MED to inaccurate or noisy extended captions, especially if GPT-4 is replaced with lower-quality generators?\n\n- Is the triplet graph structure essential, or could similar benefits be achieved using just pairs (e.g., image and extended caption)?\n\n- Can the authors release a toolkit or provide open-source code for the black-box optimization with IMLE to support reproducibility?", "limitations": "No. Authors should be rewarded rather than punished for being up front about the limitations of their work and any potential negative societal impact.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths\n- The paper introduces a multi-graph alignment framework that directly trains LLMs with triplet correlations across images, instruction responses, and extended captions, improving vision-language coherence without requiring large datasets\n- EXGRA-MED demonstrates strong performance using only 10% of the instruction-following data, reaching or exceeding LLaVA-Med's full-data performance on multiple benchmarks.\n- Employs Implicit Maximum Likelihood Estimation to scale combinatorial graph alignment efficiently to LLaMA-7B-sized models, a notable technical achievement. \n- Covers three major MedVQA datasets, chatbot benchmarks, and zero-shot classification on 23 datasets, showing consistent improvements over several baselines.\n\nWeaknesses\n- While LLaVA-Med and LLaMA-7B are used as backbones, the method’s generalization to other base models (e.g., Qwen) remains underexplored. \n- The enriched caption generation pipeline depends on proprietary GPT-4 APIs. This reliance may hinder reproducibility or deployment in restricted environments. \n- The graph alignment formulation, though elegant, is mathematically intensive and may benefit from simplified descriptions or more intuitive illustrations for accessibility.", "was_revised": true, "final_justification": "Thanks for the rebuttal. It solves all the concerns.", "rebuttal": "We sincerely thank the reviewer for the thoughtful and encouraging feedback. We greatly appreciate your recognition of the key contributions of our work, including the design of the multi-graph alignment framework, the efficient use of limited instruction-following data, and the successful integration of IMLE for scalable graph alignment. Below we address Reviewers’ concerns.\n\n\n**Q1. Examine ExGra-Med’s generalization to other base models (e.g. Qwen).**\n\n**A1**. We thank the Reviewer for your suggestion. During the short period of rebuttal, we integrated the ExGrad-Med to the Qwen-VL 2.5, 7B version. This model is pre-trained with the original 10% medical instruction-tuning data and the extended version with multi-graph alignments. We summarize below our findings confirming that ExGra-Med still helps to enhance the Qwen model in this case. We’ll update this setting with 100% pre-training to the revised version if the paper is accepted.\n\n| Model                                           | VQA-RAD | SLAKE  |\n|------------------------------------------------|---------|--------|\n| Qwen-VL (general weights)                      | 53.85%  | 70.5%  |\n| Qwen-VL Med (medical pre-trained 10%)          | 52.73%  | 86.53% |\n| ExGra-Med (Qwen-VL, pre-trained 10%)           | 58.10%  | 89.75%    |\n\n\n**Q2. Using proprietary GPT-4 APIs to generate enriched captions may hinder reproducibility or deployment in restricted environments.**\n\n**A2** We thank the reviewer for highlighting these important concerns related to **reproducibility** and **practical deployment**.\n\nTo support **reproducibility**, we will publicly release the GPT-4, generated extended instruction-following dataset used in our experiments. This will allow researchers to reproduce our results without needing API access.\n\n**Regarding deployment in restricted environments**, for example, in healthcare settings with strict data privacy constraints where sending data to external APIs (like GPT-4) may not be permissible, we address this by validating alternative caption generation using open-source models. Specifically, we employ the Qwen3-8B LLM [1], which has demonstrated superior performance compared to previous Qwen variants (e.g., QwQ, Qwen2.5) on reasoning-intensive tasks like mathematics, coding, and commonsense reasoning.\n\nDue to the time constraints of the rebuttal phase, we were only able to evaluate ExGra-Med pre-trained on 10% Qwen-generated instruction-following data. We summarize the obtained results in the table below\n\n| Model                                  | VQA-RAD Open | VQA-RAD Closed | VQA-RAD Avg | SLAKE Open | SLAKE Closed | SLAKE Avg |\n|----------------------------------------|--------------|----------------|-------------|------------|--------------|-----------|\n| ExGra-Med (Qwen-enhanced captions)     | 61.96        | 78.31          | 70.13       | 82.6       | 86.8         | 84.7      |\n| ExGra-Med (GPT-4 enhanced captions)    | 66.02        | 79.04          | 72.52       | 84.92      | 85.10        | 85.01     |\n| LLaVA-Med (baseline)                   | 43.38        | 61.4           | 52.39       | 80.94      | 80.29        | 80.62     |\n\nAs can be seen that, ExGra-Med (Qwen-enhanced captions) achieves:\n\n**+17.74%** higher average accuracy on **VQA-RAD** (70.13 vs. 52.39),\n\n**+4.08%** higher average accuracy on **SLAKE** (84.7 vs. 80.62), compared to the LLaVA-Med baseline.\n\nWhile the performance of ExGra-Med with GPT-4-enhanced captions is notably higher on VQA-RAD (72.52 vs. 70.13), the two models achieve comparable results on SLAKE (85.01 vs. 84.7), suggesting that our multi-graph alignment remains effective even when using alternative instruction-following data sources. \nWe will add these findings more explicitly in the revised manuscript if the paper is accepted.\n\n[1] Yang, An, et al. \"Qwen3 technical report.\" Arxiv 2025\n\n**Q3. The graph formulation is mathematically intensive and may benefit from simplified descriptions.**\n\n**A3**. We thank the reviewer for this valuable suggestion. We agree that the current mathematical formulation may be too technical and could benefit from a more accessible presentation to better reach a broader audience. If the paper is accepted, we will revise the text to simplify the mathematical descriptions and provide more intuitive explanations, including visual illustrations and video teasers on the project website, to make the core ideas of the method easier to understand and follow.\n\n**Q4. How can ExGra-Med remain robust when replacing GPT-4 with potentially lower-quality caption generators?**\n\n**A4**. We appreciate the reviewer’s question and have addressed this concern in conjunction with Q2. Specifically, we evaluated the robustness of ExGra-Med by replacing GPT-4 with a state-of-the-art open-source model, Qwen3-8B, to generate extended instruction-following captions.\n\nAs reported, even when using captions generated by Qwen3-8B, arguably of lower quality than GPT-4, ExGra-Med still achieves a significant improvement over the LLaVA-Med baseline, confirming the robustness of our multi-graph alignment approach to variation in caption quality.\n\n**Q5. Is the triplet graph structure essential, or would simple pairwise alignments between image and extended captions suffice?**\n\n**A5**. We thank the reviewer for this insightful question. We had the same curiosity and conducted an **ablation study** to investigate the necessity of the triplet graph structure.\n\nIn this study, we compared the full **triplet alignment** setup (image, original caption, and extended caption) against a simplified version using only **pairwise alignments** (image and extended caption). Both models were pre-trained using **40% of the medical instruction-tuning data**, and results are shown in Table 6 of the main paper. For convenience, we summarize the key results below:\n\n| Method                                  | VQA-RAD | SLAKE  |\n|-----------------------------------------|---------|--------|\n| ExGra-Med (triplet alignment)           | 74.37   | 84.99  |\n| ExGra-Med (pairwise alignment only)     | 72.58   | 82.31  |\n\nThese results indicate that the **triplet alignment structure provides a meaningful improvement**, likely due to its ability to better capture the semantic relationships across all three modalities. This richer structure helps the model learn more robust and informative representations compared to simpler pairwise alignments.\n\n**Q6. Can the authors release a toolkit or provide open-source code for the black-box optimization with IMLE to support reproducibility?**\n\n**A6**. Yes, we fully support open science and reproducibility. Upon acceptance, we will release:\n- **The pretrained checkpoints** (based on multi-graph alignment) for downstream task adaptation, and\n\n- **The complete training code**, including our efficient solver with barycenter graph construction and end-to-end learning using black-box gradient estimation via IMLE.\n\nThis will enable the community to reproduce our results from scratch, starting from the official LLaVA checkpoints, and facilitate further research based on our framework.\n\n**Q7. Quantitative results when training EXGRA-MED with smaller frozen LLMs to evaluate downstream performance trade-offs**\n\n**A7**. We thank the reviewer for this thoughtful suggestion. We fully agree that evaluating ExGra-Med with smaller language models would provide valuable insight into the trade-off between model size, efficiency, and performance.\n\nDue to the limited timeframe of the rebuttal period, we were unable to conduct additional experiments with significantly smaller models (e.g., 3B or below). However, we partially addressed this concern by validating the robustness of **ExGra-Med on the Qwen-VL 7B model, as discussed in Answer 1**. This evaluation demonstrates that ExGra-Med generalizes well beyond LLaVA-Med and retains its performance gains when applied to a different backbone.\n\nWe recognize the importance of further exploring the performance of ExGra-Med on smaller frozen LLMs to better support deployment in resource-constrained settings. If the paper is accepted, **we plan to extend our analysis to smaller models, such as Qwen-VL 3B or MiniGPT variants**, and release the corresponding checkpoints and results to benefit the community."}, {"review_id": "WZp2iUAbud", "reviewer": "Reviewer_H5Kb", "summary": "Existing medical multimodal LLMs rely heavily on large-scale data and autoregressive training, which often results in poor vision-language alignment and high dependence on instruction-tuning data. The authors propose ExGra-Med, a novel multi-graph alignment framework that jointly aligns images, responses, and extended captions to enhance semantic grounding and cross-modal coherence. ExGra-Med achieves competitive or superior performance to state-of-the-art models like LLaVA-Med and BioMedGPT using significantly less training data, showing strong efficiency and scalability for medical vision-language tasks.", "questions": "See Weaknesses.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Pros:\n\n1. The paper identifies a critical limitation in existing autoregressive medical MLLMs by revealing their heavy reliance on large-scale instruction-following data, highlighting a key inefficiency in current pre-training paradigms.\n\n2. It introduces a novel multi-graph alignment objective that effectively captures triplet-level semantic relationships among images, instruction responses, and enriched captions, enhancing cross-modal understanding.\n\n3. An efficient training solver is proposed for large LLMs, supported by theoretical insights into geodesic distances and shortest paths in multimodal graph space.\n\n4. The proposed model, EXGRA-MED, achieves competitive or superior performance to state-of-the-art med-MLLMs using only a fraction of the data, demonstrating strong efficiency, scalability, and generalization across diverse medical vision-language tasks.\n\nCons:\n\n1. Does EXGRA-MED use full fine-tuning? How does it perform with parameter-efficient fine-tuning methods such as LoRA, and could this further improve efficiency?\n\n2. The proposed method appears to be domain-agnostic---could it also be effective in the general domain beyond the medical field?\n\n3. It would be helpful to cite a recent survey on MLLMs to provide readers with a comprehensive background and better contextualize your contribution within the broader MLLM research landscape [1].\n\n[1] \"MM-LLMs: Recent Advances in MultiModal Large Language Models.\" Findings of the Association for Computational Linguistics ACL 2024. 2024.", "was_revised": true, "final_justification": "The author's response has addressed some of my concerns.", "rebuttal": "We sincerely thank the reviewer for their thoughtful and supportive feedback. We also appreciate your positive evaluation of the core contributions of our work, including identifying the limitations of current medical MLLMs, introducing the multi-graph alignment objective, and proposing an efficient solver grounded in theoretical insights. Below we address your remaining concerns.\n\n**Q1. Does EXGRA-MED use full fine-tuning? How does it perform with parameter-efficient fine-tuning methods such as LoRA, and could this further improve efficiency?**\n\n**A1** We thank the reviewer for this insightful question. To clarify, all experiments in the main paper **use full fine-tuning** to ensure fair comparisons with existing baselines, which are also fully fine-tuned.\n\nTo investigate the reviewer’s suggestion, we conducted additional experiments using **LoRA-based parameter-efficient fine-tuning** on both **ExGra-Med** and **LLaVA-Med**, and evaluated their performance on the VQA-RAD and SLAKE datasets. The results are summarized below:\n\n| Method                          | VQA-RAD Open | VQA-RAD Closed | VQA-RAD Avg | Param |\n|---------------------------------|--------------|----------------|-------------|--------|\n| ExGra-Med (LoRa)               | 63.55        | 79.41          | 71.48       | 2.1B   |\n| LLaVa-Med (LoRa)               | 62.06        | 75.00          | 68.53       | 2.1B       |\n| ExGra-Med (Fully-finetuning)   | 66.35        | 83.46          | 74.91       | 7B     |\n| LLaVa-Med (Full-finetuning)    | 63.65        | 81.62          | 72.64       | 7B     |\n\nThese results show that **ExGra-Med with LoRA still outperforms LLaVA-Med with LoRA**, indicating the robustness of our approach even under parameter-efficient tuning. However, as expected, **LoRA-based tuning does lead to a drop in performance compared to full fine-tuning**.\n\nWe believe that to bridge this gap, **pre-training on a larger-scale medical instruction-tuning dataset** (e.g., the 21M samples curated from MedTrinity [1]) would allow the LoRA setup to more closely match the performance of fully fine-tuned models. If the paper is accepted, we plan to explore this direction further and release a variety of pretrained checkpoints, including LoRA-compatible versions, to support the community.\n\n[1] Xie, Yunfei, et al. \"MedTrinity-25M: A Large-Scale Multimodal Dataset with Multigranular Annotations for Medicine.\" ICLR 2024.\n\n**Q2. The proposed method appears to be domain-agnostic - could it also be effective in the general domain beyond the medical field?**\n\n**A2**.  This is a very insightful question. We thank you for raising it. While our starting focus in this work is on medical multimodal LLMs, we highlight a broader challenge: autoregressive models tend to be data-hungry, especially when adapting across domain shifts (e.g., from natural images to medical images).\n\nWe believe this issue might not be limited to the medical domain. **For instance, in the robotics domain**, particularly in vision-language-action (VLA) models [1,2], similar challenges can arise. These models often face limited access to high-quality training trajectories and require significant adaptation when deployed in complex real-world environments. As a result, they may demand large-scale pretraining to bridge the domain gap.\n\nOverall, we see this as an important direction for future research and believe our proposed multi-graph alignment approach could generalize well to such domains, potentially improving efficiency and cross-modal alignment where labeled data is scarce.\n\n[1] Kim, Moo Jin, et al. \"OpenVLA: An open-source vision-language-action model.\" Arxiv 2024\n\n[2] Black, Kevin, et al. \"π0: A vision-language-action flow model for general robot control. Arxiv 2024\n\n**Q3. Adding a more recent survey on MLLMs to provide readers with a comprehensive background and better contextualize your contribution within the broader MLLM research landscape.**\n\n**A3**. We thank the reviewer for this valuable suggestion. We agree that incorporating recent surveys on medical multimodal large language models (MLLMs) would strengthen the contextual grounding of our work. In the revised version, we will update the Introduction and Related Work sections to include relevant and up-to-date surveys, thereby better situating our contributions within the evolving MLLM research landscape."}]

Author rebuttal:

Meta-review: The paper received mixed ratings from the reviewers. Most of the comments are positive in terms of the technical design and the significance of model efficiency at the data level. One reviewer who gave a borderline rejection had concerns regarding the model design complexity and the additional experiments on larger benchmarks. The authors provided a comprehensive rebuttal on these issues. As these issues are not directly related to the core contribution, and all the other reviewers are satisfied with the paper's contribution, AC finally decided to recommend acceptance of the submission.
Decision: Accept (poster)