Boosting Medical Visual Understanding From Multi-Granular Language Learning
Authors: Zihan Li
Yiqing Wang
Sina Farsiu
Paul Kinahan
Venue: ICLR 2026 Poster (2026)
Keywords: Multi-Granular Language Learning
Medical Image Analysis
Multimodal Learning
Primary area: foundation or frontier models, including LLMs

Abstract: Recent advances in image-text pretraining have significantly enhanced visual understanding by aligning visual and textual representations. Contrastive Language-Image Pretraining (CLIP) has played a pivotal role in multimodal learning. However, its focus on single-label, single-granularity alignment limits its effectiveness in complex domains such as medical imaging, where images often correspond to multiple labels across different levels of granularity. To address this, we propose Multi-Granular Language Learning (MGLL), a contrastive learning framework designed to improve both multi-label and cross-granularity alignment. MGLL leverages structured multi-label supervision, integrates textual descriptions across granularities, and introduces soft-label supervision with point-wise constraints to enhance alignment. MGLL employs smooth Kullback–Leibler (KL) divergence to ensure cross-granularity consistency while maintaining computational efficiency as a plug-and-play module for vision-language models. Pretrained on our constructed large-scale multi-granular datasets and evaluated across multiple datasets, MGLL outperforms other state-of-the-art methods in downstream tasks. The code is available at https://github.com/HUANGLIZI/MGLL.

Published as a conference paper at ICLR 2026
BOOSTING MEDICAL VISUAL UNDERSTANDING FROM MULTI-GRANULAR LANGUAGE LEARNING
Zihan Li'
Yiqing Wang" Duke University Durham, NC 27708, USA yq wqq.wag@dduee.e.
Sina Farsiu
University of Washington Seattle, WA 98195, USA zhanl i edu
Duke University
Durham, NC 27708, USA sina. edu:
Paul Kinahan i University of Washington Seattle, WA 98195, USA kinahan@uw edu
ABSTRACT
Recent advances in image-text pretraining have significantly enhanced visual un- derstanding by aligning visual and textual representations. Contrastive Language- Image Pretraining (CLIP) has played a pivotal role in multimodal learning, How- ever, its focus on single-label, single-gran granularity alignment limits its effective- ness in complex domains such as medical imaging, where images often corre- spond to multiple high-level labels (e.g., disease categories) across different an- notation granularities (e.g., diagnostic description, clinical explanation). To ad- dress this, we propose learning framework designed to improve both multi-label and cross-gras cooss-ranularity alignment. MGLL leverages structured multi-label supervision, integrates textual descriptions across granularities, and introduces soft-label supervision with point- wise constraints to enhance alignment. MGLL employs smooth Bellbackk-eiblee (KL) divergence ensure cross-grame coossgrraullarity consistency while maintaining com- putational efficiency as plug-and-pe module for vision-language models. Pre- trained on our constructed large-scale multi-granulla datasets and evaluated across multiple datasets, MGLL outperforms other state-of fate--ofthh--aee the art methods in down- stream tasks. The code available at https://situbbcmo/U/UUAAZZZZZZ
Language Learning (MGI.I., a contrastive
INTRODUCTION
In recent years, the large-scale image-text pretraining has significantly improved the performance of downstream computer vision tasks. Among these approaches, Contrastive Language- Pre- training (CLIP) Radford et al. (202T) has gained widespread popularity for ability to learn aligned visual and textual representations from paired data. CLIP has been extensively utilized in multimodal learning, ensuring that representations from different modalities remain semantically consistent. Consequently, CLIP has been employed for pretraining vision foundation models and fine-tuning on various downstream tasks, such as classification, image segmentation, and object detection. Despite the success of CLIP and related pretraining methods in aligning images with textual categories, simple image-text pair matching remains inadequate in medical domains such as imaging, biosignal analysis, and genomics. A single medical image signal often maps to multiple target categories, requiring both multi-label and multi- granularity alignment. As shown in Fig. []] retinal fundus image may present both Diabetic Macular Edema and Diabetic Retinopathy, along with finer-grained labels like Severe Diabetic Macular Edema and Moderate Mon-Proliferative Diabetic Retinopathy. This calls for alignment across multiple semantic levels. Existing multi- label contrastive methods |Wang et al.] (2022b): SSaporta et al. (2024); Naeem et al. (2024) focus on instance-label correlations but struggle with cross-grame gassuugr semantics and generalization. Com- pared to natural images, medical images encode more complex and hierarchical information that spans diagnoses, anatomical structures, lesions, and textures, yet they suffer from data scarcity due to privacy concerns and the high cost of annotation, which further compounds the challenge.
"quual contribution.
Co-correeponding author.

---

Published as a conference paper at ICLR 2026
(Single-Granular
Edemn
Language Learning)
Fine-luning
Diobetic Retinopathy ?
photo of (Dlabetic Edema).
Sewere Diohetic Moculer Ederno
Non Proliferative
Diobetic Retinopathy ?
Moderste Nut
Seaery DAC
Multi-lbel aligement
Various Images
Cross-ganulariity aligament
Medwoste UMC
Servre NVUM
MGLL Image Encoder
MGLL (Multi Granular Language Learning)
Abnormal
Multi- Granular Embedding
Severe Globetic Moculor Edena
Moderate Noo- Proliferative Dioberic
Fine-tuning
Rechoapochy
Retioal hemorrhages exudates observed)
Verious Visual Understanding Yosks
near mecula
Rich Text
center abserved)
Structural Dato
Probing Clossification
Finetuning Classificotion
Mederate NROR
Yes
Multi-iabe alignment
hard
MLLM Reasoning
Cross-ganndarity alignment
Modemate nur
Severe WPOR
Figure 1: The illustrative comparison of input and outcome between CLIP and MGLL.
In this study, we aim to address the challenges of multi-label alignment and eross-granularity alignment through generalizable image-text contrastive learning framework simultaneously. Here, we define "label" high-level disease category that an image belongs to, whereas "granularity" represents different levels aspects of medical annotations, such diagnostic attributes or clinical explanations. Unlike previous image-text contrastive pretraining approaches, which rely on single- gratulgr, multi-label datasets by collecting
granular, single-label supervision, construct rich textual descriptions associated with the labels. Furthermore. we extend the original CLIP image and text loss Radford et al. (202T) incorporate soft-label supervision and introduce point-wise constraints to enhance multi-label alignment. At the same time, we define contrastive learning ob- jectives for each granularity level and employ smooth Kullback-Leibler (KI.) divergence achieve cross-granularij alignment. By jointly optimizing these learning objectives, our proposed MGLL (Multi-Granular Language Learning) effectively aligns image-text pairs across both multiple la- bels and multiple granularities. Notably, our method does not introduce any granularity-sensitfe encoders, ensuring no additional computational cost. This allows MGLL function as plug- and-play module that can be integrated into any vision foundation model or large vision-language model Touvron al. (2023). We hope our method and experiments can provide new insights into medical vision-language pretraining and facilitate more effective visual representation learning. Our contributions are as follows:
We propose MGLL,, novel contrastive learning framework using that enables simultaneous multi-label and cross-granularity alignment.
language
We provide a set of architecture-agnosiic. multi-label, multi-granularity learning objectives that can be seamlessly integrated into vision-language models and foundation models to enhance medical visual understanding.
We design a structured multi-granuika multi-label system and construct large-scale multi- granular retinal and X-ray image-text datasets. Extensive experiments on over ten down- stream datasets demonstrate that MGLL consistently outperforms other state-of-the-an (SOTA) methods, exhibiting superior generalization ability.
2 RELATED WORK
2.1 IMAGE-TEX" CONTRASTIVE LEARNING
Large-scale image-tex text pretraining underpins modern multimodal learning. Contrastive methods like CLIP Radford et al. 2202 align visual and textual features via paired data. Variants such as NaLC et al. (2024) use local-to toll-to-globll pairwise learning. Symile Saporta et al. (2024) models higher-order multimodal relations, and Long-CI.oo CLII ZZang al. (2024) handles extended text via
2

---

Published as a conference paper at ICLR 2026
73" PE
He tafi lend prencle paaaee erfomation
Ap
thangesin the retioal oignent opthiiioe.
The 2fd grandeer bmgigge antomatiin
line rucuet linul Dely 2as cris.
9,72"
15751
granzler
Image
K:5
K02
1:12
1677
k:7
KAT
Image Encoder
rust
KL32
1,79
K,73
Figure 2: The overview of MGLL (Multi-Granular Language Learning) pretraining pipeline.
stretched embeddings. MedCLIP Wang et al. (2022b) addresses false negatives in medical data by semantic matching. Recent foundation models Mida--odriguezzzzz et al.] (2025); [u et al.] (2024); Li al. (2025); /Zhang et al. (2023); He et al. (2025) 2024) tailor contrastive learning to specific domains. Still, standard frameworks often underperform in medical settings due to data scarcity and complex semantics.
2.2 MULTI-LABEL LEARNING
While conventional deep networks perform well in single-label classification, real-world objects of- ten carry multiple labels across entities, actions, and attributes Zhang et al. (2022); Khattak et al.
(2024) Dai al. (2024): Wang et al. (2023). Early work by Wang et al. Wang al. (2016) learned joint image-labe label embeddings, and later introduced a recurrent attention module for interpretability Wang et al. 7011 SupCon Khosla et al. (2020) extended contrastive learning to supervised set- tings using label structures, while Zhang et al. ZZhan et al. (2022) proposed a hierarchy prrpehhi- loss. Yet, these methods remain limited vision-lann Lision-language integration, with fixed label spaces con- straining semantic flexibility.
2.3 MULTI-GRANULARIIN LEARNING
Visual and textual data convey semantics across multiple granularities. Recent work explores this via diverse frameworks Zhao et al. (2024): Li et al. (203))] | Liu et al. (2023). Wang et al. Waag et al. (202aa) bidirectional cross-ate coss-attention for mne-grained alignment, Zhao al. Zhao al. (2024) propose à multi-granularity vision flow, and Xiong et al. Xiong et al. (2022) align inter-/intra-modat features with decision fusion. Du al. Du al. 2022) capture cross-modal multi-granular seman- ties for retrieval. Most of the these approaches rely on fixed training pipelines that limit their ability incorporate heterogeneous or hierarchical annotations. Our MGLL provides a more flexible and generalizable framework for learning both multi-label and cross gusss-rrauulartty visual-language repre- sentations. MGLL can effectively utilize different types of granularity information across diverse datasets without requiring specific annotation formats or model architectures. MGLI also displays robust performance under complex scenarios such as mixed granularity and noised annotations.
MULTI-GRANULAR LANGUAGE LEARNING
3.1 OVERVIEW
To achieve multi-label and wvoss-granularigy matching between images and text, we propose MGLL,, a multi--ranuulrrtyy language-based contrastive learning framework. As shown in Fig. 2 our frame- work consists of an image encoder and text encoder, where use Vision Transformer Doso- vitskiy et al. (2020) and BERT Devlin et al. 2019) as the default choices. First, we collect rich textual descriptions for both fundus and X-ray images, constructing two multi-granularieg datasets: MGLL-Fundus and MGLL-Xray. Next, we leverage the encoded image representations and multi- granularity text representations for multi-label contrastive learning, employing smoothed KI. di-
3

---

Published as a conference paper at ICLR 2026
vergence align cross- cross-granulariy, representations. We then describe how to transform multi- granularity text into hierarchical representations and detail the MGLL objective. Furthermore, we provide both empirical and theoretical analyses, demonstrating that MGLL captures richer image- text correlations than CLIP without additional parameters. This enables the learning of more dis- criminative visual features, improving downstream vision tasks. Finally, we introduce the construc- tion of large-scale multi-granular datasets: MGLL-Fundus and MGLL-Xray.
3.2 THE MGLL OBJECTIVES
Most contrastive learning methods rely on the traditional CLIP loss, but our primary goal is to achieve simultaneous multi-label and cross-gras goassaaaaullrrtty alignment between image-text pairs. To this end, we improve the standard CLIP loss by introducing the soft CLIP loss, the point-wise loss, and the smooth KL Beulbackk-eibllee
divergence loss in our proposed multi-granularity language learning objective. The soft CLIP loss LACIIP enhances the visual encoder by enabling better align- ment with multi-label features. The point-wise loss optimizes the alignment of visual features with specific text features given granularity, further improving multi-label alignment. The smooth KI. divergence loss helps different granularity features converge toward a unified feature space, fa- cilitating cross-granularity alignment of visual representations. To quantify the similarity between image and text features, we adopt a soft alignment strategy. allowing an image Vi to align not only with a single label Ti but also with multiple related labels Tik, k 21,2...., M.} as Eqs. (I) and (2), where N is the total number of images, M; is the number of text labels associated with the i-th image, Vú and Ta represent the encoded features of the i-th image and its corresponding k-th text label, respectively. sim(V, Tix) is the similarity function measuring their alignment. The temperature parameter r controls the sharpness of the probability distribution, while the weight fac- tor Wis determines the contribution of the k-th text label to the alignment of the i-th image. The text-to-image loss lui can be obtained simply by swapping the roles of the image and text terms of the image-to-text loss ba in Eq.
exp(sim(V, V, Tu:)TT)
Lik Wik log
Σn Adn
exp(sim Vi, Tam)/T) N M.
LACLIP
ΣΣ
(la + 4sa)
2 M.
Each image feature V. is treated as multiple pairs (V,, Ta), T2), (V, TiM,). and wa is the probability of selecting Tik as a valid label for V.. Was derived from the co-occurrence matrix normalization as Eq. (3). Instead of forcing the model to align strictly with one label like CLIP, MGLL allows multi-label optimization and prevents the model from being biased toward a single label.
coocurrence (v, Ta))
Wask Σe coocurrence(44 Ti.
To further optimize the alignment between visual and textual features, we employ binary cross entropy as point-wise loss Lp to refine multi-label alignment as Eq. (4), where T.y ( Vij = sim(Vi TT) represents the similarity logits between the encoded image feature V22 and text feature T3 before applying activation. The binary label Vij {0, 1} indicates whether the image-text pair is valid match. (xx) is the Sigmoid activation function, defined as (xx) Ite-r. which normalizes the logits into probability range. T, denotes the annotation corresponding to single label at specific granularity level. .M denotes the total number of annotations, and N represents the total number of images. These annotations are consistent with those defined in Eq. ( Since the point- wise loss does not explicitly model the relationships among annotations, we omit the label subscripts of M and T, for simplicity. By explicitly supervising individual image-text pairs, this loss enhances fine-grained multi- alignment and improves the discriminability of visual representations.
M Viy log Fig + (1 - yiy) log(1 - xxa)
Lp
N
To achieve cross- granularity alignment, employ the smooth Kullback- (KL) divergence loss LXK., formulated as follows. Given m similarity logits between the encoded image feature and the text feature (P:) mm we define the mean distribution the average of all predicted distributions:
4

---

Published as a conference paper at ICLR 2026
= P. Then, compute the KL divergence between each predicted distribution and the mean distribution as Eqs. and 66. where p!ù)) represents the predicted probability of the i-th model for category j. This loss encourages consistency across different granularity levels by align- ing their predicted distributions toward the mean distribution M, which achieves cross- coos-granullarty alignment.
pl) M
(Pi(P:)M)) Σ
p( log
(5)
L.LL Σ (P((P)
(6)
The final loss with weight factors is as Eq. where on is 0.5, 02 and org by experimental setting.
LILLL = 01fSSLIP + a2 + 0r 0z3SKK
3.3
EMPIRICAL AND THEORETICAL ANALYSIS OF MGLL
3.3.1 EMPIRICAL ANALYSIS
CLIP aligns each image with a single text label, limiting its effectiveness in multi-label scenarios. MGLL addresses this by using soft CLIP loss and point- wise loss to align visual features with mul- tiple correlated text labels on a shared manifold. For each granularity in separate spaces and aligns image features accordingly. A smooth KL divergence loss further promotes consistency by aligning features across granularities with their mean distri- bution, preventing overfitting to any single level. This enables MGLL to distinguish both coarse and fine-grained categories (c.g., Glaucoma Diabetic Macular Edema, and Severe vs. Moderate Diabetic Macular Edema), where CLIP typically fails.
alignment, MGLL encodes
3.3.2 THEORETICAL ANALYSIS
We provide theoretical comparison between MGLL and CLIP. CLIP maximizes similarity between image and corresponding text features while minimizing contrastive loss for mismatched pairs, as defined in Eq. (8), where V; and T; are image and text features, sim(I,T) I.T is cosine Іللाر
similarity, and temperature parameter.
CCUp = N Σ log exp(sim( Vg, T.)/T)
(8)
Σ)= exp(sim(VV (2)r)
However, CLIP only aligns image Vi with single text label T,, limiting its effectiveness in multi-label settings. It also projects text features of different granularities into the same space, which is suboptimal when finer semantic distinctions
needed. MGLL overcomes these issues by introducing Soft CLIP Loss, Point-wise Loss, and Smooth KL Divergence Loss to support multi- label and cross-granularity alignment appropriate feature subspaces.
(1) Soft CLIP Loss: MGLL allows an image feature VA to align with multiple text features (Ta, Ti..... TiM, ). At optimality, this leads to the condition in Eq. (9) implying Eq. (10). where Vi converges to the weighted center of its associated text features. This contrasts with CLIP, which aligns each image single text feature, highlighting MGLL's advantage in multi-label learning.
Mi
V (V, Txx) =
(9)
A=1
M.
Tik
Vi
Wikk
(10)
|T
||
(2) Point-wise Loss: To enhance image-text alignment, we introduce point-wise binary cross- entropy loss C. with its gradient shown in Eq. (II). If yij = 1, the objective to maximize o (2 strengthening similarity between V; and Tj. If Yıj 0, it minimizes (x:y), suppressing similarity with irrelevant text. This encourages alignment with all valid labels while filtering out noise, improving over CLIP's single-label constraint.
5

---

Published as a conference paper at ICLR 2026
dCp
= ox(s) Vij
(11)
dry
(3) Smooth KL Divergence Loss: To enforce crossggraullartty consistency, introduce Smooth KL Divergence Loss. The mean distribution is defined as M = Σm P; where {P:} are predicted distributions. By the non-negativity of KL divergence, Eq. achieves equality only when P; = M. Minimizing LxI. thus enforces Pi = P2 = -... Pm = M, encouraging consistent representations across granularities and improving alignment in feature space.
DRL (P: M) 2 0, vi
(12)
While CLIP optimizes image-text alignment, it overlooks feature variability across granularities and lacks consistency in visual alignment. By aligning each image feature Vi with single text feature T,, it risks biased representations. In contrast, MGLL drives text features of different granularities toward a shared mean distribution M, promoting common semantic grounding and aligning visual features with all gramularity levels, not just one.
3.4 LARGE-SCALE MULTI-GRANULAR DATASETS
3.4.1 MGLL-FUNDUS DATASET
In this study, we construct a large-scale multi-granularity fundus image-text dataset, MGLL-Fundus, consisting of 246,389 pairs of fundus images and corresponding multi-granularity textual descrip- tions. The image data in HGLL--Fundus originates from 49 public datasets, covering more than 50 disease categories (details are provided in the supplementary material). The gulti-granularity textual descriptions mainly include two levels of granularity: disease category and clinical expla- nation. The disease-level granularity comprises normal/abnormal labels along with specific dis- ease categories. The clinical explanation granularity provides detailed textual descriptions derived from label explanations in datasets and Wye Wik (2024). As shown in Fig. 2 the disease- level description is "Abnormal, Age-related Macular Degeneration", while its corresponding clini- cal explanation is "Changes in the retinal pigment epithelium." " By incorporating multi- cuulti-raauulaiity textual descriptions, establish hierarchical labeling system for fundus images, including nor- mal/abnormal classification, disease categorization, and detailed clinical descriptions, which enable cross pross-granularity image-text alignment and enhance performance across different granularity levels. Our multi-gran quuti-graaullrity approach can also be adopted to other modalities.
3.4.2 MGLL-XRAY DATASET
In radiology research, the heterogeneity of study descriptions in DICOM (Digital Imaging and Communications in Medicine) headers complicates patient cohort selection, especially with man- ual methods. MIDRC (Medical Imaging and Data Resource Center) MIDRC (2024) highlight this challenge, where over 138,00 .000 studies are categorized into only 97 unique descriptions, while the rest are described by 1,300 different descriptions. Therefore, we need LOINC (Logical Observation Identifiers Names and Codes), which provides a standardized coding system to enhance data shar- ing and analysis. To facilitate data coordination with, we collect 190,882 X-ray images from the MIDRC repository MMIDRC (2024). We convert the images from DICOM to PNG format while ex- tracting key metadata. The extracted multi-granularity textual information includes modality, study description, and series description. Modality includes CR (Computed Radiography), which has lower resolution and signal-to-noise ratio (SNR), and DX (Digital Radiography), which uses flat- panel detectors for nighet-quality imaging. Study Description provides an exam-level overview, such as "Chest X X-ray", while Series Description details specific imaging sequences like "PA View" (posteroanterior) "Lateral View". These multi-granularity textual features serve as the textual component of MGL.L-rray dataset.
4
EXPERIMENTS
4.1
SETUP
We construct a large-scale multi-granularity fundus image-text dataset for pre-training, with further details provided in the supplementary material. To evaluate our model's performance, we conduct experiments on eleven downstream datasets: FIVES Jin et al. (2022). IDRiD Porwal et al. (2018).
OIA-DDR Li al. (2019b), ADAM Fang al.
(2022). PALM Fang et al. (2024), REFUGE Orlando RFMID Pachade et al. (2021). MIDRC-XP MIDRC
ct al. (2020), RIM-ONE Batista et al. (2020).
72044 HIDRC-XR-Portall MIDRC (2024). ChestX-ray 14 Wang al. (017a) under both linear

---

Published as a conference paper at ICLR 2026
Linear Probing
Mathed
Fully Finetuning
NGLL
VilerFM
RRM
13RENDA)
Un Med-d
RET
KespfI
FLAIR
RETFound
KoCtOp
70%
DARINN
SAYA
REFINE
ADAM
RAG4 ADAM
Figure 3: The quantitative comparison (AUC) between baseline methods and proposed MGLL on nine fundus downstream datasets.
probing and full fine-tuning settings. In our quantitative evaluation, we employ AUC (Area Under the receiver operating characteristic Curve), mAP (mean Average Precision), and ACC (Accuracy) as assessment metrics. As for the multi-label setting, we report the category- average accuracy as ACC. We adopt ViT-L/14 Dosovitskiy et al. (2020) as the image encoder and BiomedicalBERT Alsentzer et al. (2019) as the text encoder by default. All experiments were conducted under iden- tical settings, with baselines pre-trained on our self-constructed multi-granularity datasets to ensure fair comparison. We strictly followed the official data splits of all downstream datasets. During pretraining, we only used the training sets for model training and the validation sets for pretraining evaluation, while the test sets were never accessed.
4.2 COMPARISON WITH STATE STATE-OF-THER OF-THE-ART METHODS
4.2.1 EVALUATION ON RETINAL FUNDUS DATASETS
Utilizing the multi- quatti-gaauularity image-text fundus dataset we constructed, we pretrain our model within the Muati-Granulla Language Learning (MGLL) framework to enhance its capability in fea- ture representation for retinal fundus images. We conduct comprehensive experiments to compare the performance of MGLL against several state-of-the-art (SOTA) baseline methods across nine downstream datasets, covering a wide range of retinal diseases. The AUC results for both linear probing and full fine-tuning are presented in Fig. 3 while more detailed results can be found in the supplementary material (Tables 15 to 23). MGLL consistently achieves significant performance improvements across all nine datasets, with particularly strong gains the linear probing setting. Notably, on the multi-label dataset RFMiD Pachade et al. (2021). MGLL outperforms other methods by at least 16.6% in linear probing and 6.7% in full fine-tuning, demonstrating its superior capabil- ity in handling imbalanced data distributions. Fig. 4 visualizes class activation maps (CAMs) from CLIP and MGLL on two cases with different retinal diseases. meaningful features, instead assigning nearly uniform attention weights across the entire fundus im- age. In contrast, MGLL effectively localizes key regions of interest (ROIs) for different diseases. Specifically. MGLL accurately highlights hard exudates for chorioretinitis and the retinal pigment epithelium for rge-related macular degeneration. These quantitative and qualitative evaluations col- lectively indicate that MGLL possesses extraordinary capability in effective feature extraction and performance enhancement across diverse retinal diseases.
evident that CLIP fails to extract
4.2.2 EVALUATION ON X-RAY DATASETS
We pretrain MGLL on the MGLL-Xin -Xray dataset and conduct experiments on MGLL and other SOTA
baseline methods (Radford et al. (202): :: Tiu et al. (2022); Zhang et al. (2023): Zhou al. (003a));
Dai et al. (2024): Khattak et al.) (2024): Lai et al. (2024); Xie et al." (0255)
the MIDRC-XR
datasets, which are shown in the Table I In the linear probe setting, MGLL achieves significant advancements over the second-best method (UniChest Dai et al. (2024) on MIDRC- XR and UniMed-CLIP Khattak et al. (2024) on CO-RC--X--Prrtalle with improve- ments of 2.23% and 3.81% in AUC, respectively. indicating superior representation learning capa- bilities. The performance gap becomes even more significant in the fully fine-tuned setting. To demonstrate its generalization capability, we conduct additional experiments using multi-granular
and
7

---

Published as a conference paper at ICLR 2026
Table 1: The performance evaluation on MIDRC-XR, MIDRC-XB Pe-R--ee- and ChestX-ray 14.
Bold indicates best performance and underline shows second-bor
MIDRC-XR
MIDRC-XI-Portabbl
ChestX-ray I4
Method
Láncar Probe (S)
Fully AUCC
Fiae-tunee (%)
Lincar
Prohe (%)
Fully
Fime-tune (5)
I Lincar Prohe (%)
Fully
Fine-tune (%)
AUI
ACC
CLIPOCM (CML-21)
51.18
58.52
11:45
78.23
22.31
91.83
90105
7K302
87.58
31.79
CheXacro (NaL. BMB-22
51.31
03.85
12.71
50.11
73.26
55.46
12.84
80.13
23.56
92.47
92.42
55.23
68.72
76.98
15.97
61.81
87.39
31.52
KAD (Nai. Com- 25)
53.84
47.13
14.86
85.74
76.39
60.12
73.53
80.71
23.88
93.41
92.98
85.96
73.72
78.95
21.87
83.80
89.13
34,01
(CLR--23)
UnChest (TMI-24)
$9.02
54.78
19.32
92.51
86.32
06.93
78.69
85.28
27.37
95.44
94.32
36.38
26.15
81.72
25.52
85.4
89.99
37,97
UniMed-Cl CLIP (unXiv-24)
57.35
54,07
18.06
94.15
87:47
65.49
80.05
86.63
28.16
96.31
93.55
$6.19
75.54
81.21
24.96
82.59
88.36
32.19
CARZero (CVPR-220 24)
57.92
18.79
93.48
86,94
15.24
82.67
92,94
92.66
67
77.32
83.94
88.65
32.86
(ICML-25)
37.02
MGLL
61.25
56.57
21.19
99.08
90.06
73.33
83.86
89.06
30,62
99.75
98.80
89.87
82.94
90.41
28.53
87.37
92.71
39.17
Linear Probing
Fully Finetuning
Chorioreiiins
Age related Macoior Degenentico
Original
CLIP
MGLL
CLIP
MGLL
Figure 4: The Class Activation Maps of different diseases from CLIP and MGLL.
labels constructed from the MIMIC-CXR dataset Johnson et al. (2019). evaluating performance on the ChestX-ray benchmark WWng al.] (2017a). The results reveal its exceptional transferability, with substantial performance advantages across all other baseline methods. In the linear probe set- ting, MGLL achieves 82.94% AUC, 90.41 % accuracy, and 28.53% mAP, surpassing the second-best method (CARZero Lai et al. (2244) by 5.62%, 6.47%, and 1.65% respectively. The improvement highlights its superior representation learning capacity, suggesting that its multi-granulat approach captures more generalizable features that transfer effectively across datasets. These consistent and substantial improvements also demonstrate that MGLL enables more robust feature extraction.
4.3 PERFORMANCE WITH MGLL MLLMS
To evaluate MGLL's impact as a specialized vision encoder within multimodal large language mod- els (MLLMs) for ophthalmological diagnostics, design multiple-choice benchmark involving 2,233 clinical cases over ten ophthalmological conditions, where each fundus image prompted mod- els select the correct diagnosis from four options (one correct, three random alternatives). We replace the standard vision encoders in seven advanced MLLMs with our pretrained MGLL: In- structBLIP Dai al. (2033). Mini-Gemini Li et al. (2024), Qwen-VL VL Bai al. (2023). InternVL Chen et al. (2024a), LLaVA Liu et al. (2023), LLaVA- Med Li et al. 2023), Med-Fil -lammnna Moor al. (2023). and Janus-Pro Chen al. (2025). All MLLMs were tnnet on the target dataset to ensure fair comparison. Results demonstrate consistent and substantial improvements across all tested MI.LMs, with average accuracy gains ranging from 4.6% (In to 34.1% (L.LaVA-Me as shown in Table 2 Notably, medically-specialzzel models exhibited the most dramatic enhance- ments, with Med- Fad-iFaainngo and LLa VA-Med showing 31.7% and 34.1% 1% increases respectively. This dramatic improvement can be attributed to the alignment between MGLL's ophthalmology- specific visual feature extraction capabilities and the medical reasoning frameworks already embed- ded within these models. Even the gigg-performing general- purposs MILMs like LLaVA (72.73% to 79.98%) also achieve significant gains with MGLL. The improvements across challenging condi- tions like Tessellation and Retinitis underscore MGLL's capacity extract clinically relevant visual features from fundus images, highlighting its robust adaptability across models.
8

---

Published as a conference paper at ICLR 2026
Table Comparison of multiple-choice accuracy with MGLL in multimodal large language models
on selected ten representative diseases.
Method
AMD
Calaract
CSR
DR
Glancoma
Media Hure
Myopia
[Retmiis
DMC
Tessellation
Average
InstruciBL1P Dau ct
2023
80.17%
80.00%
0L00%%
76.315
59.30%
10.12%
44,25%
11.11%
63.79%
41.67%
47.29%
-MGLL
8366%%
85.00%
28.57%
82,55%
65.43%
45.16%
52.65%
44.44%
74.14%
58.33%
61.99% (34,7% 1>
Mini Gemmiin[icta
2024
76.6115
85.00%
14,29%
79.871
67.90%
38.715%
58.4115
33.33%
60.34%
33.339
54.78%
-MLLL
82.46%
85,00%
42.86%
84.56%
72.22%
58,06%
64.16%
55.56%
68.52%
41.67%
68.21%
(10,055 1>
Qw Rai at al
2023
81.87%
75.00%
28. 5756
80. 54%
78.40 78.4078
54.84%
76.85%
22.2276
84.48%
25,006
60: 75%
MGG..
85.96%
80.00%
42.86% 96
89.93%%
87.04%
70.97%
90.97%
33.33%
89.06
41.67%
70.245
(9:55%)
IntemVV Chen et MGGT.
3024g
81.29%
85.00%
71.43%
94.663%
59.5155
64. 5056
58.05%
44.44%
87.03%
66.67%
77. 75.5%%
86.55%
90.00%
71.43%
96.66%%
90.74%
67.74%
91.85%
55.56%
94.88%%
75.00%
81.96 Ri (4.65%
T.LVA LLu atal 2023
83.04%
90.00%
42. 86.6%
887.1.5
31.366%
48. 50%
58. 50.
44.44%
93.10%
58.33%
72.73%
MGGLL
84.00%
90.00%
57.34%
93..96%
91.98%
61.29%
90.71 9
66.07%
96.55%
66.67%
79.98 R (7.39%
T.aVA-Melfiiettl
2023
16.37%
15.00%
42. 86.86%
25.8.5%
25.316
25.81%
23.89%
33.53%
16.67%
16.67%
24 2506
MGLLL
58.48%
65.00%
57.34%
77.18%%
59.26%
51.61 %
57.08%
44.44%
55.17%
58.33%
58.37%
(14,155 13
Med-Hlamings Moor al.
2023
25.73%6
311.07%
57.14%
350.91%
24.117%
22.585
18.5815
22.22%%
24.14%
8,335%
26.97%
MGLL
69.01%
75.00%
71.43%
880,54%
61.11%
54,88%
45.58%
44.44%
51.72%
33.33%
58.70%
(31,715 1>
Janus Pre Chen et al.
2025
55.3015
75.00%%
42.861%
95.299%
90.7456
580000
87.17%
33.335%
62.007%
58.33%
68.925
MGLL
98.64%
85.00%
71.43%
96.64%
95.06%
67.74%
90.27%
55.56%
70.69%
75.00%
79.80%
(10.8816
4.4
ABLATION STUDIES
4.4.1 ABLATION STUDY ON MGLL OBJECTIVES
We conduct an ablation study to analyze the effectiveness of each objective in MGLL on the RFMiD dataset, shown in Table 3 The standard CLIP model performs the worst, highlighting its limita- tions in medical image understanding. Incorporating the point-wise Loss Lp significantly improves performance, demonstrating its ability enhance feature extraction. The soft CLIP loss LCLLPP also improves over CLIP, which enables soft alignment with multiple labels. Combining both losses (CacLIP + Lp) further boosts performance, indicating their complementary effects. Finally, adding the KLLt-KL loss Lakt. leads to the best performance, demonstrating its role in refining feature consis- tency across different learning objectives. These results validate the effectiveness of each objective.
Table 3: Ablations of different MGLL objectives on RFMiD.
Table 4: Ablations of granularity count on MIDRC- XR AR-Portable.
Linear Probe (%)
Fully Fine-tune (%) AUC
Linear Probe (%)
Fully Fine-tune (%)
Method
AUC
ACC
mAP
ACC
mAP
Method
AUC
ACC
mAP
AUC
ACC
mAP
CLJP
44.66
92.53
7,28
65.10
92.86
17.31
CLIP
71.43 78.22 22.31
91.83
90.08
83,94
Cp
70.34 92.69
23.83
88,25
94.31
56.46
EycUPP
67.86
92.63
20.52
85.13
93.58
50.67
MGLLI
80.54
86,97 28.32
95.96
94.66
86.54
Lari Cp
15.73
92.77
30.16
90.31
94.87
62.27
MGLL2
82.92
88.35
29.43
97.26
96.84
87.68
Cop+ LoKt.
79.62
92.84
34.08
92.83
95,48
64.99
MGLLs
83.86
89.06
30.62
99.75
98.80
89.87
Table 5: Ablations of image encoder on RFMiD.
Table 6: Ablations of text encoder on RFMiD.
Linear Probe (%)
Fully Fine-tune (%)
Linear Probe (%)
Fully Fine-tune (%)
Method
Method
AUC
ACC
mAP
AUC
ACC
mAP
AUC
ACC
mAP
AUC
ACC
mAP
CLIPP
44.66
92.53
7.28
65.10
92.86
17.31
CLIP
44.66 92.53
7.28
65.10
92.86
17.31
Nex-Next-Base
74.45 92.73
27.94
88.55
94.57
57.62
CLIP....
68.93
92.66
22.14
88.76
94.58
58. 14
ConvNext-Large
78.34
92.80
32.03
91.29
94.95
62.93
VIT-B/16
75.53
92.76
30.11
89.46
94.83
61.84
BERT
79.62
92.84
34.08
92.83
95.48
64.99
VIT-L/14
79.62
92.84
34.08
92.83
95.48
64.99
LLaMA
74.89
92.75
29.38
90.97
95.04
62.86
VIT-H/14
79.18
92.81
33.42
92.07
95.29
63.85
4.4.2 ABLATION STUDY ON GRANULARITY COUNT
The ablation study on granularity count demonstrates the significant impact of multi-granula lan- guage supervision on model performance. As shown in Table incrementally increasing the num- ber of granularity levels consistently improves performance across all evaluation metrics. MGLL3, which utilizes three distinct granularity levels (modality, study description, and series description), achieves superior results compared to both MGLL, (all textual information combined into sin- gle granularity) and MGLL2 (two granularity levels). Specifically, under linear probe evaluation, MGLL2 outperforms the baseline CLIP by substantial margins (+12.43% AUC. + +10.84% ACC, +8.31% % mAP) and shows marked improvement over MGLL, (+3.32% AUC, +2.09% ACC, +2.30% mAP). The similar trend persists in fully fine-tuned scenarios. These results confirm that preserv- ing the hierarchical structure of medical imaging information enables more comprehensive vision- language alignment than flattened representations, validating the core idea of the Multi-Granular Language Learning framework.

---

Published as a conference paper at ICLR 2026
4.4.3 SELECTION OF IMAGE ENCODER AND TEXT ENCODER
The ablation study on image encoders reveals significant performance variations across different architectural choices when evaluated on the RFMiD dataset as shown in Table [ Vision Trans- former (ViT) Dosovitskiy
al. (2020) generally outperforms CNN counterparts (ConvNeXt Liu et al. (022bb), with ViT-I/14 achieving optimal results across all metrics. Interestingly, the larger V/T-H/14 model shows slightly diminished performance compared to ViT-I/14, suggesting poten- tial overfitting scenario or diminishing returns with increased model complexity in this domain.
The comparative analysis of text encoders demonstrates that the choice of language model signif- icantly impacts the model's 's ability to align textual and visual representations. BERT emerges as the optimal text encoder, achieving the highest performance across all evaluation metrics as shown in Table 6 The standard CLIP text encoder (denoted as CLIPtezt) shows the limited performance among the tested alternatives, though it still substantially improves upon the baseline CLIIP model. These findings suggest that the bidirectional attention mechanisms are suitable for the structured, hierarchical medical terminology utilized in MGLL.
4.4.4 ABLATION STUDY ON IMAGE QUALITY AND TEXT QUALITY
Image resolution is critical factor in the performance of MGLL as evidenced by the ablation exper- iments on the MIDRC-XR-Portabl dataset. The performance exhibits a clear monotonic relation- ship with image resolution, with Low-Resolution (128x 128) and Ultra Low-Resolution (64x64) configurations across all evaluation metries as shown in Table 7 These findings underscore the importance of preserving fine-grained visual details in medical imaging applications, as higher resolution allows the model to capture sub- tle radiological features. However, MGLL substantially outperforms the baseline CLIP even at Ultra Now-Resolution, suggesting that MGLL provides robust improvements regardless of image quality.
(512x512) 12) significantly outperforming both
The integrity of textual information significantly impacts model performance, as demonstrated through controlled degradation experiments on the MIDRC-XR--Potabl dataset. The analysis con- trasts standard textual descriptions against two degraded conditions: "Error" (20% partial errors in modality, study, or series descriptions) and "Missing" (20% partial omissions these same fields). Standard textual descriptions yield superior performance across all metrics as show in Table g The "Missing" condition demonstrates intermediate performance, while the "Error" condition shows more performance degradation, suggesting that incorrect information is more detrimental than in- complete information. Nevertheless, both degraded conditions still significantly outperform the baseline CLIP model, indicating the robustness of MGLL to textual noise. These findings have important practical implications for clinical deployment scenarios, where reporting systems may contain documentation gaps transcription errors, and suggest that MGLL maintains considerable diagnostic utility even under suboptimal documentation conditions.
Table 7: Ablations of image quality on MIDRC- XR-Portable.
Table 8: Ablations of text quality on MIDRC- XR-Portable.
Linear Probe (%)
Fully
Fine-tune (%)
Linear Probe (%)
Fully Fine-tune (%)
Method
Method
AUC
ACC
mAP
AUC
ACC
mAP
AUC
ACC
mAP
AUC
ACC
mAP
CLIP
71.43
78.22
22.31
91.83
90.08
83.94
CLIP
71.43 78.22 22.31
91.83
90.08
83.94
Ultra Low-Res
78.82
85.68 27.49
94.48
93.69
86.22
Error
80.02 86.48 28.01
97.71
97.22
87.96
Low-Res Res
80.66
87.02
28.53
98.18
97.65
88.46
Missing
81.14
87.25
28.86
98.62
97.95
88.74
Standard-Res
83.86
89.06
30.62
99.75
98.80
89.87
Standard
83.86
89.06
30.62
99.75
98.80
89.87
CONCLUSION
This study introduces Multi-Graaullr Language Learning (MGLL), a novel contrastive learning framework that addresses limitations in existing vision-language pretraining methods. MGLL uti- lizes textual information on different granular levels while employing soft-label supervision with point-wise constraints to enhance representation quality, which advances multi-label and cross- granularity alignment capabilities simultancously. The implementation smooth Kullback-Leibler divergence also ensures cross-granularity consistency. Our evaluations across multiple downstream datasets demonstrate that MGLL consistently outperforms of-th----he-an methods downstream tasks, particularly in domains requiring multi-label understanding at various granular levels. The results validate its ability to capture complex semantics in visual data, establishing MGLL as an advancement in developing future vasion-language models.
10

---

Published as a conference paper at ICLR 2026
Ethics statement This research uses exclusively publicly available medical imaging datasets and does not require Institutional Review Board (IRB) approval. All datasets utilized in this study, in- cluding the 49 public fundus imaging datasets comprising MGLL-Fundus, the MIDRC repository for X-ray images, MIMIC-XXR, and other evaluation datasets, have been previously released for research purposes with appropriate ethical clearances and patient consent procedures handled by the original data providers. All patient identifiers have been removed from the datasets prior to our access, ensuring full de-identification in compliance with HIPAA and other relevant privacy regula- tions. We have strictly adhered to the usage terms and conditions specified by each dataset provider and have not attempted to re-identify any individuals. Our multi- multi-granularity learning approach is designed to improve automated medical image analysis, particularly for diabetic retinopathy, glau- coma, age-related macular degeneration, and chest X-ray interpretation. The MIDRC dataset ap- plications focus on enhancing radiological assessment capabilities, which could potentially assist healthcare providers in resource- limited settings and improve diagnostic consistency. However, we emphasize that our models are intended diagnostic support tools and should not replace clinical judgment. We commit to responsible Al development by ensuring transparent reporting of model limitations, encouraging rigorous clinical validation before any real-world deployment, and advo- cating for appropriate human oversight in all clinical applications. We do not claim our models are ready for direct clinical without further validation and regulatory approval.
Reproducibility statement We have made extensive efforts to ensure the reproducibility of our work. The complete implementation, including the original source code and README file with detailed instructions, is provided in the supplementary materials. Training configurations and hyper- parameters are fully documented in both the source code and Section 4. Step-by-ster mathematical derivations of the proposed methodology are presented in Section [3.3 and Appendix A All datasets employed in this study are publicly available, with references and preprocessing procedures de- scribed in Section and Appendix B The experimental setup is described in Section [4. and more To ensure fair comparison and reproducibility. all experiments used identical settings with baseline models pre-trained on our custom multi-gran granularity datasets.
details are described in Appendix C
Acknowledgments Research reported is part of the MIDRC (Medical Imaging and Data Resource Center) project. MIDRC is funded by the National Institute Biomedical Imaging and Bioengi- neering (NIBIB) of the National Institutes of Health and ARPA-H -H under contract 75N92020D00021.
REFERENCES
cataract dataset.
https://wwrr cataractdataset/ 2020. Accessed: 2025-02-20.
Vietai advance course retinal disease detection. 2020.
compettirions/vettal itgpettiiittttitaaaa-dddeeeeeeee htt.aaaaaaaaaaaaaaeaaaaaaeeeeeea [Accessed: 2025-02-20].
Augemnted ocular discases, 2021.
https:
www. http://kaggeeeee///atteeeeeeeeee
nurmukhammen 7/ augemngeeenne teatmnneeddooo -cudedoolllle casllllrrdieeeee seases
Accessed: 2025-02-20).
Derbi hackathon retinal fundus image dataset, 2022.
https:/ww..aaggle.
datasets/ninl haac ret na www.uuuuiiaagee-- [Accessed: 2025-02-20).
Michael D Abråmoff, James C Folk, Dennis Han, al. Automated analysis of retinal images for
detection of referable diabetic retinopathy. JAMA ophthalmology, 131(3)3311-377, 2013.
Kedir M Adal, Peter G van Etten, Jose P Martinez, Lucas Vliet, and Koenraad A Vermeer.
Accuracy assessment of intra-and intervisit fundus image registration for diabetic retinopathy screening. Investigative ophthalmology visual science, 56(3):1805-18122 2015.
Syed Samiul Alam, Samiul Based Shuvo, Shams Nafisa Ali, Fardeen Ahmed, Arbil Chakma, and Yeong Min Jang. Benchmarking deep learning frameworks for automated diagnosis of ocular toxoplasmosis: A comprehensive approach to classification and segmentation. IEEE Access, 2024.
11

---

Published as a conference paper at ICLR 2026
Emily Alsentzer, John R Murphy, Willie Boag. Wei-Hung Weng, Di Jin, Tristan Naumann, and Matthew McDermott. Publicly available clinical bert embeddings. arXiv preprint 19Xx::1904.03333, 2019.
Jinze Bai et al. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308. /2966, 2023.
Muhammad Naseer Bajwa, Gur Amrit Pal Singh, Wolfgang Neumeier, Muhammad Imran Malik, Andreas Dengel, and Sheraz Ahmed. G1020: A benchmark retinal fundus image dataset for computer-aided glaucoma detection. In 2020 International Joint Conference on Neural Networks (LJCNN) pp. 1-7. IEEE, 2020.
Francisco José Fumero Batista, Tinguaro Diaz-Aleman, Jose Sigut, Silvia Alayon, Rafael Arnay, and Denisse Angec-Pereira. Rim-one dl: unified retinal image database for assessing glaucoma using deep learning. Image Analysis and Stereology, 39(3):611-167, 2020.
Veronica Elisa Castillo Benítez, Ingrid Castro Matto, Julio César Mello Román, al. Dataset from fundus images for the study of diabetic retinopathy. Data in brief, 36:107066 2021.
Grace Maria Binu. Retinal occlusion dataset, 2023. htt.:/ttpss///..m www. www.wwwwwgle...o com datasets/
grztemearriiihon/eeeaaaalooooooo www.oonnndtttttttaaaaaaa [Accessed: (cgaeeeessss.ottttttttcccacaeee 2025-02-20).
Attila Budai, Rüdiger Bock, Andreas Maier, Joachim Hornegger, and Georg Michelson. Robust vessel segmentation in fundus images. International journal of biomedical imaging, 2013(1): 154860, 2013.
Ling-Ping Cen, Jie Ji, Ken-Wei Lin, al. Automatic detection of 39 fundus diseases and conditions in retinal photographs using deep neural networks. Nature communications, 12(1):4828, 2021.
Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal understanding and generation with data and model scaling. arXiv preprint arXiv:2501. 178/1, 2025.
Zhe Chen et al. Internvl: Scaling up vision foundation models and aligning for generie visual- linguistic tasks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 24185-24198, 2024a.
Zhihong Chen, Maya Varma, Jean-Benoit Delbrouck, Magdalini Paschali, Louis Blankemeier, Dave Van Veen, Jeya Maria Jose Valanarasu, Alaa Youssef, Joseph Paul Cohen, Eduardo Pontes Reis, et al. Chexagent: Towards a foundation model for chest x-ray interpretation. arXiv preprint arXiv:2401.1 12208, 2024b.
Tianjie Dai, Ruipeng Zhang, Feng Hong, Jiangchao Yao, Ya Zhang, and Yanfeng Wang. Unichest: Conquer-and add-divide pre-training for multi-source chest x-ray classification. IEEE Transactions on Medical Imaging, 2024.
Wenliang Dai et al. Instructblip: Towards general-purposs vasion-language models with instruction tuning. Advances in neural information processing systems, 36:49250 2023.
Parisa Karimi Darabi. Diagnosis of diabetic retinopathy, 2024. https: wwtp://wwwww/m kaggle... com/
Accessed: 2025-
02-20J.
Coen De Vente et al. Airogs: Artificial intelligence for robust glaucoma screening challenge. IEEE transactions on medical imaging, 2023.
Etienne Decenciere, Guy Cazuguel, Xiwei Zhang, et al. Teleophta: Machine learning and image processing methods for teleophthalmology. Irbm, 34(2):196-20) 203, 2013.
Aysen Degerli, Mete Ahishali, Mehmet Yamac, Serkan Kiranyaz, Muhammad EH Chowdhury, Khalid Hameed, Tahir Hamid, Rashid Mazhar, and Moncef Gabbouj. Covid- 19 infection map generation and detection from chest x-ray images. Health information science and systems, 9(1): 15, 2021.
12

---

Published as a conference paper at ICLR 2026
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics:: human language technologies, volume (long and short papers), pp. 4171-4186, 2019.
Andres Diaz-Pinto, Sandra Morales, Valery Naranjo, et al. Cnns for automatic glaucoma assessment using fundus images: an extensive validation. Biomedical engineering online, 18:1- 18:1-19, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICIR, 2020.
Jiawei Du, Jia Guo, Weihang Zhang, et al. Ret-clip: A retinal image foundation model pre-trained with clinical diagnostic reports. In International Conference Medical Image Computing and Computer-Assisted Intervention, pp. 709-719. Springer, 2024.
Yunhao Du Binyu Zhang, Xiangning Ruan, Fei Su, Zhicheng Zhao, and Hong Chen. Omg: OB- serve multiple granularities for natural Sanguage-baset vehicle retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3124-3133, 2022.
Emma Dugas, Jared, Jorge, and Will Cukierski. Diabetic retinopathy detection. ht https://
kaggle, wattglee.comooooooo http://coppttiioiooooooo -ew.eetio--rriiiaaatly----d-eeee detectiion 2015. Kaggle.
Team EyeWiki. The eye encyclopedia, 2024. hattps://eywwikk.c/
[Accessed: 2025-02-20).
Huihui Fang, Fei Li, Huazhu Fu, et al. Adam challenge:
Detecting age-related macular degeneration from fundus images. IEEE transactions on medical imaging, 41(10):2888-2847 2022.
Huihui Fang, Fei Li, Junde Wu, al. Open fundus photograph dataset with pathologic myopia recognition and anatomical structure annotation. Scientific Data, 11(1):99, 2024.
Taimur Hassan, Hina Raja. Bilal Hassan, Muhammad Usman Akram, Jorge Dias, and Naoufel Werghi. A composite retinal fundus and oct dataset to grade macular and glaucomatous disor- ders. In 2022 2nd International Conference on Digital Futures and Transformative Technologies (ICoDT2), pp. 1-6. IEEE, 2022.
Chunming He, Kai Li, Yachao Zhang, Guoxia Xu, Longxiang Tang, Yulun Zhang, Zhenhua Guo, and Xiu Li. Weakly-supervised concealed object segmentation with sam-based pseudo labeling and multi-scale feature grouping. NeurlPS, 36, 2024.
Chunming He. Rihan Zhang, Longxiang Tang, Ziyun Yang, Kai Li, Deng-Ping Fan, and Sina Farsiu. Scaler: Sam-enhanced collaborative learning for label-deficient concealed object segmentation. arXiv preprint arXiv:2511.18133 2025.
Qiao Hu, Michael D Abråmoff, and Mona K Garvin. Automated separation of binary overlapping trees in low-contrast color retinal images. In Medical Image Computing and Computer-Assiisedd Intervention 2013, pp. 436-443. Springer, 2013.
Shih-Cheng Huang, Liyue Shen, Matthew P Lungren, and Serena Yeung. Gloria: A multimodal
global-local representation learning framework for label-efficient medical image recognition. In Proceedings of the IEEE/CVF international conference computer vision, pp. 3942-39229999 3951, 2021.
Mir Tanvir Islam, Shafin T Mashfu, Abrar Faisal, Sadman Chowdhury Siam, Intisar Tahmid Naheen, and Riasat Khan. Deep learning-based glaucoma detection with cropped optic cup and disc and blood vessel segmentation. leee Access, 0922828-2841 2021.
Kai Jin, Xingru Huang, Jingxing Zhou, Yunxiang Li. Yan Yan, Yibao Sun, Qianni Zhang, Yaqi Wang, and Juan Ye. Fives: A fundus image dataset for artificial intelligence based vessel segmen- tation. Scientific data, 9(1):475, 2022.
Alistair EW Johnson, Tom J Pollard, Seth J Berkowitz, et al. Mimic-exr, a de-identified publicly available database of chest radiographs with free-text reports. Scientific data, 6(1):317, 2019.
13

---

Published as a conference paper at ICLR 2026
Karthik, Maggie, and Sohier Dane. Aptos 2019 blindness detection. htttps:/kkaglent
deteeestsieeeettiinnnn.o 2019. Kaggle.
Tomi Kauppi, Valentina Kalesnykiene, Joni-Kristian Kamarainen, et al. The diaretdbl diabetic retinopathy database and evaluation protocol. In BMVC, volume 1. pp. 10. Citeseer, 2007.
Muhammad Uzair Khattak, Shahina Kunhimon, Muzammal Naseer, Salman Khan, and Fahad Shah- baz Khan. Unimed-clip: Towards unified image-text pretraining paradigm for diverse medical imaging modalities, arXiv preprint arXiv:2412.10311 2024.
Prannay Khosla, Piotr Teterwak, Chen Wang, et al. Supervised contrastive learning. Advances in neural information processing systems, 33:18661-18677 2020.
Ungsoo Kim. Machine learn for glaucoma. 2018a. URL Attpss//aa...orr///./09/00//00 1YRRAC
Ungsoo Kim. Machine learning for pseudopapilledemma, [Accessed: 2025-02-20].
httpp://osf.io/2w//
Oleksandr Kovalyk, Juan Morales-Sánchers Rafael Verdúó-Monedero, et al. Papila: Dataset with fun- dus images and clinical data of both eyes of the same patient for glaucoma assessment. Scientific Data, 9(1):291, 2022.
JR Harish Kumar, Chandra Sekhar Seelamantula, JH Gagan, et al. Chákşu: A glaucoma specific fundus image database. Scientific data, 10(1):70, 2023.
Haoran Lai, Qingsong Yao, Zihang Jiang, et al. Carzero: Cross-attention alignment for radiology
zero-shot classification. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11137-11146, 2024.
Larxel Ocular disease recognition. 2021.
https: wwwp http://wwwwwkggle.ee/e/o www.kkgggl...................... cww. / anddewmad/ooulla--dleessss www.:aaaae....eeooooooooooooooia
Accessed: 2025-02-20J,
Chunyuan Li et al. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems, 36, 2023.
Liu Li, Mai Xu, Xiaofei Wang, Lai Jiang, and Hanruo Liu. Attention based glaucoma detection: A large-scale database and cnn model. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10571-10588 2019a.
Tao Li, Yingqi Gao, Kai Wang, Song Guo, Hanruo Liu, and Hong Kang. Diagnostic assessment of deep learning algorithms for diabetic retinopathy screening. Information Sciences, 501:511-522, 2019b.
Yanwei Li et al. Mini-gemini Mining the potential of multi-modality vision language models. arXiv preprint ar arXiv:2403.18814 2024.
Zihan Li, Diping Song. Zefeng Yang, Deming Wang, Fei Li, Xiulan Zhang, Paul E Kinahan, and Yu Qiao. Visionunite: A vision- langua-la foundation model for ophthalmology enhanced with clinical knowledge. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025.
Li Lin, Meng Li, Yijin Huang, Pujin Cheng, Honghui Xia, Kai Wang, Jin Yuan, and Xiaoying Tang. The sustech-sysu dataset for automated exudate detection and diabetic retinopathy grading. Scientific Data, 7(1):099, 2020.
Haotian Liu et al. Visual instruction tuning. Advances in neural information processing systems, 36, 2023.
Ruhan Liu, Xiangning Wang, Qiang Wu, et al. Deepdrid: Diabetic quality estimation challenge. Patterns, 3(6), 2022a.
and image
Yong Liu, Cairong Zhang, Yitong Wang, Jiahao Wang. Yujiu Yang, and Yansong Tang. Universal segmentation at arbitrary granularity with language instruction. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 3469-3469, 2024.
14

---

Published as a conference paper at ICLR 2026
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11976-11986, 2022b.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05100 2017.
DongAo Ma, Jiaxuan Pang, Michael Gotway, and Jianming Liang. fully open ai foundation model applied to chest radiography. Nature, pp. 1- 1-11, 2025.
Team MIDRC. The medical imaging and data resource center, 2024. https::/www.mmt [Accessed: 2025-02-20].
Michael Moor et al. Med-flamingo: a multimodal medical few-shot learner. In Machine Learning for Health (ML4H), pp. 353-367. PMLR, 2023.
Muhammad Ferjad Nacem, Yongqin Xian, Xiaohua Zhai, et al. Silc: Improving vision language pre- training with self-distillation. In European Conference on Computer Vision, pp. 38-55. Springer, 2024.
Luis Filipe Nakayama, Mariana Goncalves, L. Zago Ribeiro, al. A brazilian multilabel ophthal- mological dataset (brset). PhysioNet https://oi. org/10, 13026:2, 2023.
Arijit Nandi. Giaucooma-cassficcatooommmll 2022. URL
https: //g thub. com/ Accessed: 2025-02-20.
ing. officiialariiiff/llauoaaaaaaaaa- www.ssilaaattaaaa.
ML-
Meindert Niemeijer, Xiayu Xu, Alina V Dumitrescu, et al. Automated measurement of the arteriolar- to-venular width ratio in digital color fundus photographs. IEEE Transactions on medical imag- 2011.
José Ignacio Orlando et al. Refuge challenge: unified framework for evaluating automated meth- ods for glaucoma assessment from fundus photographs. Medical image analysis, 59: 10150,,,,, 2020.
Samiksha Pachade, Prasanna Porwal, Dhanshree Thulkar, et al. Retinal fundus multi-disease image dataset (rfmid): dataset for multi-disease detection research. Data, 6(2):14, 2021.
Sachin Panchal, Ankita Naik, Manesh Kokare, al. Retinal fundus multi-disease image dataset (rfmid) 2.0: dataset of frequently and rarely identified diseases. Data, 8(2):29, 2023.
Vu Minh Hieu Phan, Yutong Xie, Yuankai Qi, Lingqiao Liu, Liyang Liu, Bowen Zhang, Zhibin Liao, Qi Wu, Minh-Son To, and Johan W Verjans. Decomposing disease descriptions for enhanced pathology detection: multi-aspect rision-language pre-training framework. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11492-11501, 2024.
Ramon Pires, Herbert F Jelinek, Jacques Wainer, Eduardo Valle, and Anderson Rocha. Advancing hag-of-visual-werr representations for lesion classification in retinal images. PloS one, 9()): e96814, 2014.
Prasanna Porwal, Samiksha Pachade, Ravi Kamble, et al. Indian diabetic retinopathy image dataset (idrid): database for diabetic retinopathy screening research. Data, 3(3):25, 2018.
Jianing Qiu, Jian Wu, Hao Wei, et Development and validation of multimodal multitask vision foundation model for generalist ophthalmic artificial intelligence. NEJM AI, 1(12):Alo2230021,, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Manuel Alejandro Rodríguez, Hasan AlMarzouqi, and Panos Liatsis. Multi-label retinal disease classification using transformers. IEEE Journal of Biomedical and Health Informatics, 27(6): 2739-2756 2022.
15

---

Published as a conference paper at ICLR 2026
Adriel Saporta, Aahlad Puli, Mark Goldstein, and Rajesh Ranganath. Contrasting with symile: Simple model-agmostic representation learning for unlimited modalities. arXiv preprint arXiv:2411.01053 2024.
Dandan Shan, Zihan Li, Yunxiang Li, Qingde Li, Jie Tian, and Qingqi Hong. text prompt network for medical image segmentation. IEEE Transactions on Image Processing, 2025.
Stpnet: Scale-aware
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of ACL, 2018.
Julio Silva-Rodriguez, Hadi Chakor, Riadh Kobbi, et al. A foundation language-image model of the retina (flair): Encoding expert knowledge in text supervision. Medical Image Analysis, 99: 103357, 2025.
Hidenori Takahashi, Hironobu Tampo, Yusuke Arai, al. Applying artificial intelligence to disease staging: Deep learning for improved staging of diabetic retinopathy. PloS one, 12(6))e0179790, 2017.
Ekin Tiu, Ellie Talius, Pujan Patel, Curtis P Langlotz, Andrew Y Ng, and Pranav Rajpurkar. Expert- level detection of pathologies from unannotated chest x-ray images via self-supervised learning. Nature biomedical engineering, 6(1)):1399-1400) 2022.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.1397. 2023.
Fuying Wang, Yuyin Zhou, Shujun Wang, Varut Vardhanabhuti, and Lequan Yu. Multi-granularign cross-modal alignment for generalized medical visual representation learning. Advances in Neural Information Processing Systems, 35:33536-33549, 2022a.
Jiang Wang, Yi Yang, Junhua Mao, Zhiheng Huang, Chang Huang, and Wei Xu. Cnn-rnn: A unified framework for multi-label image classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2285-2294, 2016.
Linda Wang, Zhong Qiu Lin, and Alexander Wong. Covid-net: a tailored deep convolutional neural network design for detection covid-19 19 cases from chest x-ray images. Scientific reports, 10(1): 19549, 2020.
Meng Wang, Tian Lin, Aidi Lin, Kai Yu, Yuanyuan Peng, Lianyu Wang, Cheng Chen, Ke Zou, Huiyu Liang, Man Chen, al. Enhancing diagnostic accuracy in rare and common fundus dis- eases with knowledge-rich vision-language model. Nature Communications, 16(1):5528, 2025.
Xiaosong Wang, Yifan Peng, Le Lu, et al. Chestx-res rayt: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2097-2106, 2017a.
Yiging Wang. Zihan Li. Jieru Mei, Zihao Wei, Li Liu, Chen Wang, Shengtian Sang, Alan L Yuille, Cihang Xie, and Yuyin Zhou. Swinmm: masked multi-view with swin transformers for 3d medical image segmentation. In International conference on medical image computing and computer- assisted intervention, pp. 486-496. Springer, 2023.
Zhouxia Wang, Tianshui Chen, Guanbin Li, Ruijia Xu, and Liang Lin. Multi-label image recog- nition by recurrently discovering attentional regions. In Proceedings of the IEEE international conference on computer vision, pp. 464-472, 2017b.
Zifeng Wang. Zhenbang Wu, Dinesh Agarwal, and Jimeng Sun. Medelip: Contrastive learning from unpaired medical images and text. In Proceedings of the Conference Empirical Methods in Natural Language Processing. Conference on Empirical Methods in Natural Language Process- ing, volume 2022, pp. 3876, 2022b.
16

---

Published as a conference paper at ICLR 2026
Qijie Wei, Xirong Li, Hao Wang, Dayong Ding. Weihong Yu, and Youxin Chen. Laser scar detection in fundus images using convolutional neural networks. In Computer Vision-ACCV 2018: 14th Asian Conference on Computer Vision, Perth, Australia, December 2-6, 2018, Revised Selected Papers, Part IV 14, pp. 191-206. Springer, 2019.
Junde Wu, Huihui Fang, Fei Li, et al. Gamma challenge: glaucoma grading from multi-modality images. Medical Image Analysis, 90:102938, 2023.
Ruiqi Wu, Chenran Zhang, Jianle Zhang, et al. Mm-retinal: Knowledge-enhanced foundational pretraining with fundus image-texi expertise. In International Conference on Medical Image Computing and Computer-Assiser Intervention, pp. 722-732. Springer, 2024.
Chunyu Xie, Bin Wang, Fanjing Kong, Jincheng Li, Dawei Liang, Gengshen Zhang, Dawei Leng, and Yuhui Yin. Fg-clip: Fine-grained visual and textual alignment. arXiv preprint arXiv:2505.05071, 2025.
Peixi Xiong, Yilin Shen, and Hongxia Jin, Mga-vqa: multi-granularity alignment for visual question answering. arXiv preprint arXiv:2201.10666 2022.
Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, and Bingbing Ni. Medmnist v2-a lcrge-scale lightweight benchmark for 2d and 3d biomedical image classification. Scientific Data, 10(1):41, 2023.
Zhao Yang, Jiaqi Wang, Yansong Tang, Kai Chen, Hengshuang Zhao, and Philip HS Torr. Lavt: Language-aware vision transformer for referring image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pp. 18155-18165, 2022.
Hantao Yao, Rui Zhang, and Changsheng Xu. Visual-language prompt tuning with knowledge- guided context optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Pp. 6757-6767, June 2023.
Beichen Zhang, Pan Zhang, Xiaoyi Dong, Yuhang Zang, and Jiaqi Wang. Long-clip: Unlocking the long-text capability of clip. In European Conference on Computer Vision, pp. 310-325. Springer, 2024.
Jiong Zhang, Behdad Dashthozorg, Erik Bekkers, Josien PW Pluim, Remco Duits, and Bart M ter Haar Romeny. Robust retinal vessel segmentation via locally adaptive derivative frames in 2016.
orientation scores. IEEE transactions on medical imaging,
Shu Zhang, Ran Xu, Caiming Xiong, and Chetan Ramaiah. Use all the labels: A hierarchical multi- label contrastive learning framework. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16660-16669, 2022.
Xiaoman Zhang, Chaoyi Wu, Ya Zhang, Weidi Xie, and Yanfeng Wang. Knowledge-enhanced visual-language pre-training on chest radiology images. Nature Communications, 14(1):4542, 2023.
Zhuo Zhang, Feng Shou Yin, Jiang Liu, et al. Origa-light: An online retinal fundus image database for glaucoma analysis and research. In 2010 Annual international conference of the IEEE engi- neering in medicine and biology, pp. 3065-3068. IEEE, 2010.
Xiangyu Zhao, Xiangtai Li, Haodong Duan, et al. Mg-llava: Towards multi-granularity visual instruction tuning. arXiv preprint arXiv:2406.11 17770. 2024.
Hong- Yu Zhou, Chenyu Lian, Liansheng Wang, and Yizhou Yu. Advancing radiograph representa- tion learning with masked record modeling. In ICLR, 2023a.
Yung-YuPo
Hong- Yu Zhou, Julián Nicolás Acosta, Subathra Adithan, Suvrankar Datta, Eric J Topol, and Pranav Rajpurkar. Medversa: A generalist foundation model for medical image interpretation. arXiv preprint arXiv:2405.07988 2024.
Yukun Zhou, Mark A. Chia. Siegfried K. Wagner, et al. A foundation model for generalizable disease detection from retinal images. Nature, 622:156 163, 2023b. URL https: : / semant. cschol 1 aar.d wrr. crrg/oorpsttD22411635
2D:241166226666 68236
17

---

Published as a conference paper at ICLR 2026
A PPENNIX
Table of Contents
Detailed Theoretical Analysis of MGLL
19
A.I Soft CLIP Loss
19
A.2 Point-wise Loss
20
Smooth KL Divergence Loss
22
Dataset Details
24
Pretrain Datasets
24
B.
2 Downstream Datasets
26
C Setup Details
28
Evaluation Metrics
28
Implementation Details
28
More Detailed Experimental Results
29
D
Detailed Results on Retinal Fundus Datasets
29
D.2
Zero-Shot Comparisons Across Medical and Natural Domains
................................
32
D.3
Performance Evaluation on Region Segmentation
32
D.4
More Detailed Results of MGLL MLLMS
33
D.5
More CAMs on Retinal Fundus Datasets
37
D.6
Ablation Study on Weight Factors of Loss
................................
37
D.7 7
Ablation studies on the temperature coefficient (T)
38
D.8
Performance datasets with artificially introduced noise ................................
38
D.9
Performance on datasets with mixing granularity levels
38
Discussion and Future Work
39
18

---

Published as a conference paper at ICLR 2026
A DETAILED THEORETICAL ANALYSIS OF MGLL
A.1 SOFT CLIP LOSS
Formulation of the Soft CLIP Loss In contrast to standard CLIP, which forces an image represen- tation V; to align with single text label T. MGLL uses the Soft CLIP Loss to allow an image to align with multiple text features (Ti,, Ti,.... J. J. In doing so, loss designed so that optimal image feature becomes the weighted "center" of its associated text features. This mitigates bias toward any single label and better captures the multi-label nature of the data. The loss for an image-label pair is defined the following formula:
e Va..T
la Wisk log
(13)
2m
M
Sim(V,,Tumm)
where wis is the weight assigned to the k-th text label for image i, derived via the following formula:
cooccurrence V,, Tax)
Wii Σ> cooccurrence V, Tik)
(14)
where sim(V; Tak measures the similarity between image and text features (lypically cosine simi- larity). is the temperature parameter controlling the sharpness of the resulting probability distri- bution. The overall loss is given by
N
Mr
C.CLPP
Li + Lss
(15)
2Σ M2
Derivation of the Optimality Condition
(a) The Goal of the Optimization
For the purposes of our analysis, we focus on how the loss aligns V with its multiple text features Tik- At optimum, the gradient of the loss with respect to the image feature V; must vanish. That is, we require
VV = 0.
(16)
Focusing on the part of the loss involving the alignment between Vú and its labels, we can write a simplified optimality condition (ignoring symmetric contributions from text-to-image terms):
M
Vv, sim Twx) = 0.
(17)
(b) Computing the Gradient
Assume for simplicity that both image and text features are normalized to unit norm:
||] = and
Ta 1.
(18)
Under this assumption, the cosine similarity reduces to dot product:
sim(V,, Txx) = V, Tix
(19)
The derivative with respect V; is then straightforward:
Vv TTx) Tik
(20)
Thus, the optimality condition becomes
19

---

Published as a conference paper at ICLR 2026
M. Wik Tik X Vi,
(21)
where is a scalar (a Lagrange multiplier that arises from the normalization constraint on Vi). Because V, is unit norm, this equation can be re-arranged to yield
Σ
Mr Wik Tik
V;
(22)
k= Wik Tik
This result shows that at the optimal solution, the visual feature V, is aligned with the weighted average (or the "center") its associated text features.
(c) Rewriting in Normalized Form
More generally, even when the features are not explicitly normalized in the network, one can express the optimality condition in terms of normalized vectors:
M,
T&
V
Wikk
(23)
|
This is equivalent to the expression provided earlier, emphasizing that VÚ converges to the weighted centroid of the normalized text features.
Interpretation and Discussion Unlike CLIP, which uses a one-to-one image-text matching- Soft CLIP Loss allows each image be simultaneously aligned with multiple text descriptions. The weighting wikk (derived from the co-occurrence matrix) ensures that each text label contributes to the final image representation in proportion to its relevance. The optimality condition guarantees that the image representation V, is not overly biased by any single text feature but is instead the "center" all its semantic descriptors. This is crucial in multi-label settings where an image may contain several objects or concepts. In standard CLIP, the loss encourages VÚ8 to align closely with a single text label T:. Here, the Soft CLIP Loss's optimality condition shows that the ideal V is instead weighted aggregate of multiple text labels, overcoming the limitation of forcing one-to- one alignment a multi-labe context. The detailed derivation shows that under the Soft CLIP Loss, the following gradient condition
M, Wak Vv,sim(V,, Tu)
(24)
leads directly the interpretation that the optimal image feature V; the normalized weighted sum of its associated text features. This theoretical result underpins MGLL's capability perform multi- label alignment, ensuring that image representations capture the combined semantic information provided by multiple labels.
A.2 POINT-WISE LOSS
Formulation of the Point-wise Loss The point-wise loss Lp is designed to refine the alignment be- tween visual and textual features on a per-pair basis. Unlike global or batch-level losses, this loss explicitly supervises each image-text pair, ensuring that every valid pair (where Vij 1) is pulled closer together in the feature space while non-matching pairs (where Vij = 0) are pushed apart. This detailed supervision is achieved via binary cross-entropy formulation applied to the similarity logits between an image feature Vú and a text feature Tj. The point-wise wise loss is defined as
Yi log T.j + (1 Viy) log(1 - 2i5))
Cp
(25)
where my -im(V, T7) are the similarity logits (before activation) between the i-th image and the j-th text. z.g σ (21r)) = -+ is the probability computed by the Sigmoid activation. Vij
20

---

Published as a conference paper at ICLR 2026
{0, 1) is the binary label indicating whether the image-text pair is valid match. This formulation is typical for binary classification tasks and allows the network to learn to distinguish relevant from irrelevant image-text pairs.
Derivation of the Gradient To understand how the loss drives the optimization, we derive the gradi- with respect to the logits Zij.
(a) Loss for a Single Pair
Consider the binary cross-entropy loss for a single image text pair (i,
li = - |yss log (mzy) + (1 - Viy) log(1 o(x,)))
(26)
(b) Derivative with Respect to 2-j
We can let Pi = ((zr)). Using the chain rule, the derivative Es with respect to ray is:
Olg am.
dpas
(27)
Öp.
draj
0bu Step Compute 8pd
We can differentiate lij with respect to Pij as follows:
Olij
Mij
1 Vij
(28)
Öpij
Pij
Pij
Step 2. Compute
dpa2 dass""
Since P.j = ( and the derivative of the sigmoid function is as follows:
do (xits)
-( = Pij (1 -Pt))
(29)
dx
So we have
dp = Pis(1 - -Pis) dzaj
(30)
Step 3. Combine the Derivatives. We can multiply the two derivatives gives as follow:
Olij
Uij
1 -
Vij
Piis1 Pij)
(31)
Dæy
Pij
1 Piş
Expanding and simplifying, we have:
Oli
(2r (1 -ps) - -(1 v(s)p())
(32)
Dzy
Distributing the negative sign results in:
0lg Pij Vij Dzi
(33)
Thus, for each image-text pair, the gradient is:
Olis
= o((is) Vii
(34)
Drij
Since the overall wisnt-wise loss Lp is the average over all pairs, the gradient with respect to each Ti remains the same:
dCr
o(Ii)) Vig
(35)
0zij
Interpretation and Discussion This gradient expression, ( Vijs provides clear insights into the optimization dynamics:
21

---

Published as a conference paper at ICLR 2026
The gradient becomes (zzy) - 1. If the predicted probability (1ay)) less than 1, the gradient is negative, prompting an increase in 2ij (and hence o (21,)). This drives the features V; and T closer together.
For Positive Pair (Uyy 1):
For a Negative Pair (Yij = 0): The gradient simplifies to (x,y)) If oz is positive (which always is, since (xx) (0, 1)), the gradient is positive, pushing 2ij downward. This reduces the similarity, ensuring that irrelevant image-text pairs are further separated.
Thus, the gradient directs the model increase similarity for valid pairs (driving (zi;)) towards 1). And it also decreases similarity for invalid pairs (driving (m.j) towards 0). In contrast to the standard CLIP framework, which optimizes global alignment between an image and single text description, the point-wise loss enables the model to adjust each image-text pair individually, lead- ing more discriminative and robust feature space. Every potential image-text pair evaluated, allowing the model to learn subtle distinctions. By penalizing both false positives and false negatives the individual pair level, the loss helps create a more separable and robust embedding space. &Ce. (zxs) Pij- clearly shows that the optimization encourages high similarity for valid pairs and low similarity for invalid pairs. Point-wise Loss not only enhances the discriminability of the visual representations but also supports the multi-label learning methods.
The derivation its gradient, azy
A.3 SMOOTH KL DIVERGENCE LOsS
Overview and Motivation The Smooth KL Divergence Loss is introduced to enforce consistency across predictions obtained at different gramularities. In scenarios where multiple predicted logits {z:) are available (each corresponding to a different granularity or viewpoint), we wish to align their probability distributions that they all "agree" on the prediction. This achieved by first converting the logits into probability distributions via the Softmax function:
P. Softmax(z.)
(36)
and then encouraging each P. to be close the average (mean) distribution:
m
M
P
(37)
m
The overall loss is given by
L.IL = Σ
Dx (P.lMM
(38)
1=1
where for each i the KL divergence is defined as
p()
(P.())) = Σ
p(i) log
(39)
M
Properties of KL Divergence There are two key properties of the KL divergence:
1. Non-negativity (Gibbs' Inequality):
Dkl(PI 2 0
(40)
for any two probability distributions P and Q
2. Zero if and Only if Equality:
Dkl = if and only if P=Q
(41)
These properties imply that the divergence is minimized (and equals zero) when the two distributions are identical.
Detailed Derivation and Proof
(a) Expressing the Loss
22

---

Published as a conference paper at ICLR 2026
Given the predicted distributions {Pi) and the mean distribution M, the loss is
p(
SKL =
log
(42)
M
(ul(P.)
Because each DRr (P: 2 0, it follows that
LLL 2
(43)
(b) Conditions for Zero Loss
The loss for single term, (Pu equals zero and only if
p! M for every category
(44)
Since this must hold for every i, we have:
P P... -M
(45)
This is the necessary and sufficient condition for minimizing the loss:
LKL= 0
Pi
Pm
(46)
Thus, by minimizing LSLL. the model is encouraged to produce consistent predictions across differ- ent granularities. So the mean distribution defined as
M
P
(47)
If all P. are equal, then it trivial see that
M P.
Vi
(48)
Therefore, the minimization process pushes each individual P; toward the common distribution M, ensuring consistency across different predictions.
(c) Gradient Considerations
While the explicit gradient derivation is more involved due to the dependency of M on every P., we can outline the intuition of individual KL divergence and the intuition of coupled optimization.
1. Individual KL Divergence: For each i, consider the derivative of
pla)
DR. (R.(l))
Σ p6)
log
(49)
MG
with respect to p(,) : If we ignore the dependence of M on P. (as an approximation), the derivative
(Pxl(P) log
p(i
(50)
OP(3)
MG)
The condition for minimum (when this derivative is zero for all j) is then
p(i)
p(3) MO) =
log
MU) +1=0
(51)
which alone does yield P; M; however, when accounting for the normalization constraint Σ, p(u) and the simultaneous optimization across P, the equilibrium is reached only when p! (i for every and j.
2. Coupled Optimization: Since M is the average of all P., any deviation in one P. from the others will increase its corresponding KL divergence. Thus, the overall optimization drives all predictions to align with each other.
23

---

Published as a conference paper at ICLR 2026
Interpretation and Discussion By minimizing LsKL. force the different predictions (from various granularities) to become consistent. Every P. is toward the common mean M, ensuring that the predictions from different parts of the model from different feature granularities agree with each other. This consistency also contributes to a more stable and robust feature space, as the model learns reconcile variations in prediction across granularities. The Smooth KI. Divergence Loss L.KI. ii= DxL. (P) is fundamentally designed to enforce that all predicted probability distributions P. (across different granularities) become identical by driving them toward the mean distribution M.
B DATASET DETAILS
B.1 PRETRAIN DATASETS
MGLL-Fundus: We develop MGLL--uuddus, a comprehensive multi-granularity fundus image-text dataset comprising 246,38) image-text pairs. This dataset integrates fundus images from 49 public datasets, encompassing more than 50 disease categories. The data distribution of the MGLL-Fundus dataset is presented in Table 9 The textual descriptions in MGLL-Fundus are structured across two distinct granularity levels: disease category and clinical explanation. The disease-leve granularity includes normal/abnormal classification and specific disease categorization, while the clinical expla- nation granularity provides detailed textual descriptions derived from dataset label explanations and EyeWiki FikiiEyeWiki (2024). Some multi-granularity textual description examples from the dataset illustrated in Table
Table 9: The data distribution MGLL--Fundun dataset.
Num
HRF Budai et al. 2013
45
INSPIRE. AVR Niemeijer el al.
2011
IOSTAR Zhang et al.
2016
30
RITEIHU
2013
G1020
GAMMA
ORIGA
Zhang ml 2001
650
REFUGE Orlando el
2120
1200
ODIR
Larael 2021
7000
PALM
Fang al
2024
1200
RFMID
Paclade 0t
2021
3200
RFMIDv2
Panchal ar...
2023
860
APTOS
Karlink
2019
3662
DeepDKiD
1600
EyePACS
Duras
2013
35126
IDRID Porwal
2018
516
ADAM Fang et
11.2
1200
ACRIMA
Dizz Pinlo
2019
705
MESSIDOR
Abratooll
2013
1748
Ccn
AIROUS
Venie
LAG La al 4014
4854
PARAGUAY Benilez el al
2021
757
PAPILA
Kuvalyk
488
BiDR Darato
AUA4]
2838
FIVES Jt
800
FUND
Hassaa ct
60
179
E-ophta Decenctere
2013
463
BRSET | Nacayana 4013
16266
MuReD
Rounguez
2208
OIA OIA-DDR DDR ei al 2019
12522
SUSTech [
20140
1219
Catract 2022
A021
601
DGOCH
BoVW Pires
HarvardGlaucon
201
RIM ONE- Malisia el mall
485
CHAKSU Kilmir el
2023
1345
DiaReiDIi Kaunpi el
2007
89
LSD Wer 2019
175
GNG Namda 40.2
400
ADD A0M A0D
AU74
14813
DIIRI
2757
Viet
3435
ToxoFundus
2024
411
Papilledenal kim 2018b
1369
BEH [slam ot 4021
634
ROD Sinu
21225
281
ROI ROdlAda éc
Summary
246,389 images
MGLL-Xray: To enhance data compatibility, we assembled 190,882 X-ray images from the MIDRC repository MIDRC [0244. We transformed these images from DICOM to PNG format while pre- serving essential metadata. The extracted multi-granularity textual information encompasses three levels: modality, study description, and series description. The modality category distinguishes between CR (Computed Radiography), characterized by relatively lower resolution and signal-to- noise ratio (SNR), and DX (Digital Radiography), which employs flat-panel detectors achieve superior image quality. Study Description provides examination-levs context (e.g.. "Chest X-ray"), while Series Description specifies imaging protocols such as "PA View" (posteroanterior) or "Lat- eral View." These hierarchical textual elements constitute the textual component of our MGI.L-Xrayy dataset. Some multi granuiatranulrrity textual description examples from this dataset are presented in Ta- ble II]
MIMIC-CXR: To further evaluate the generalization capability of our approach, we conducted supplementary experiments using multi-granular labels from MIMIC-CXR dataset Johnson et al. 2019), with performance assessed on the ChestX-rayl1 benchmark Waag et al. (2017a). The MIMIC-CXR dataset represents of the largest publicly available collections of chest radio- graphs, comprising 377,110 images associated with 227 227,835 imaging studies. This dataset encom- passes 14 common thoracic pathologies, including atelectasis, cardiomegaly, consolidation, edema,
24

---

Published as a conference paper at ICLR 2026
Table 10: The multi-granularity textual description examples of MGI.L-Fundus dataset.
Discase Category
Clinical Explanation
Abnurnal, Mild Now-Prollion Profiffffferttive Diabetic Retinopahy
Only miczuancursms observed
Abncemal, Modenate Proliiilleettivv Dubbetic Relinouully
Retinal hemerragges or hard esudates oeserved
Abnormal, Severe Non. Proliffeeeee Dishetic Retinogathy
Many intraretinal henworhages or definite venous headligg ohserved
Abnormal, Proliferative Diahetic Retinopatiy
Neovasculatizatinn sitrssax/preretinn hemorrhage
Absomall Cataract
Opacification crystalline lens abserved
Abroomal, Myopla
Lespand fundus observed
Absomall Media Haze
Opacity of media observed
Abnormal, Brznch Retinal Vein Occlusion
Occlusion of central retinal
Abnurnal. Tossellation
The chocuidal vessels visible
redlced demityy the pigments
Abucemal, Laser Scars
Cirvalan or inegular shuped scars on the retinal surface ubserved
Ahnomull, Central Seroes Retinopathy
Fluid accumulation under the retina observed
Ahnomal, Optic Disk Cupping
The thinning of ncuronctinal rim such that optic disc appeurs excavated
Ahnormal, Central Retinal Vein Occlusion
Occlusion of the retinal and the presence flame shaped hemorrhages Markad kortoosity of the retinal bloed vessels
Abnormal, Tortoons Vessels
Abnormal, Asterii Hyalosis
Nomerous astroid bodies are dispersed in vitreous
Abnormal, Optic Disc Pallor
Pale yellow discolocation of the optic disc
Absommal Optic Disc Edema
Swelling of ogtic disc
Abnormal, Optuciliary Shunt
Presence prepapillary vascular loogs optociliary shunt vessels
Abnormal, Azietior Ischemic Optic Ncuroputhty
Optic disc swelling and pulkor
Abnormal, Paraloweal Telangietteia
Yellow, lipid-rich exudation or parafoveal graying or tocturus blood vessels Presence of traction and retinal tmatiin detachment
Ahnormal, Retinal Trnction
Abnormal, Retinitis
Presence vitreous inflammation intraretinal hemorraaee
Abscemal, Chorioretinits
The hand exndates observed
Ahnormal, Macular Hale
A small retinal break loented the center of the fovea observed
Abnomall Retinitis Pigmentosa
The presence of hone spicule deposits and arterial namroing The presence of soft exudates
Absomaal Cocton Wool Spots
Abuormal. Coloboens
missing of pertion tissue both choroid and retina Bost-shaped hemorrfage which abscares the undellynng resina Giny-white white opaque lesions with feathery olges observed
Abuormal, Prenotinal Henombage
Abnormal. Myelinated Nerve Fibers
Abnormal, Hemoerhagic Retineqully
The presence of flame-dhuped hemerhages
Abnumnal, Central Retinal Artery Occlusion
The presenee pole, whitening,
retinal swelling The rilting presence the oval optic disc
Ahnormul, Tilledd Disk
Ahnormal, Cyunid Macular Edema
The presence of muliiple systoid meas the maculs and causes retinal edems
A Ahnomal, Post-traumatic Cheroidal Rupture
The breaks the charoid, Brach's memhrane. and RPE
Abnormal, Choroidal Folds
The presence of folds in the chornid
Abnormal, Vitrcous Hemonraage
spaces created arcund vitroous body Fooilorm or round dibiion the retinal arterioles which occur the temporal reina observed The presence of inllamattion of retinal blood vessels Clear optic disk boundaries, Normal fundus color. No upparent retinoputhy
presence extavasated blood
Abnurnal. Macrounsaryam
Abnormal. Visculitis
Normal Healthy
Table 11: The multi-granulann textual description examples of MGLL-Xray dataset.
Modality
Study Description
Series Description
CR
CHEST PORT I VIEW (RAD)-CS
APishutter)
CR
XR CHEST AP PORTABLE
AP
CR
RIBS RIGHT CHEST AP
SINGLE VIEW
Ribe LOWER
CR
XR PORT CHEST IV
CXR AP GRID
CR
XR CHEST 2 VIEWS
W Chest Lat.
CR
CHEST
LATERAI.
Lateral
CR
XR PORT CHEST IV
ClearRead Bone Suppression
CR
XRAY CHEST ONE VIEW
XRAY CHEST FRONTAL AND LATERAL VIEWS
CR
XR RIGHT HIP 2+ VIEWS ORTHOPEDICS PRE OPERATIVE
HIP X-Table Lat X C.est
CR
XR CHEST PA AND LATERAL
DX
XR CHEST VIEWS, FRONTAL AND LATERAL
DX
XR WRIST LEFT (ROUTINE: APLAT.OBL) XR CHEST VIEWS AND LATERAL.
XR WRIST LEFT (ROUTINE: APLATOBBL)
DX
Chest
DX
XR THORACOL LMBAR SPINE STANDING 2 OR 3 VIEWS
Thoraco Lumbar
DX
XR THE ARC SPINE AP AND LATHRAA AI.
Thorncio-spine
DX
SCOLIOSIS STUDY XK RIGHT RIBS 2 VIEWS INILATERAL.
VIEWS (NEURO INTERPRETNTION)
Long Spine
DX
Rib AP Ext Rot(shutter Patella
DX
XR RIGHT SHOULDER VIEW
DX
XR RIGHT KNEE 4+ VIEWS (NON-TRAUMA, PAINARTHRITSS)
DX
XR RIGHT HIP 2-3 VIEWS ATERAI, WITH PELVIS WHEN PERFORMED XR CHEST FRONTAL
Hip-joint
DX
CHEST LANDSCAPE
effusion, emphysema, fibrosis. hernia, infiltration, mass, nodule, plcural thickening, pneumonia, and pneumothorax, along with a "No Finding" category for normal cases. For our multi-granularic framework, we leveraged two distinct levels of textual information available in MIMIC-CRR the structured disease labels (coarse granularity) and the detailed radiology reports (fine granularity). The disease labels provide categorical classification, while the reports offer comprehensive clinical interpretations with anatomical specificity, disease progression details, and differential diagnoses. This hierarchical representation allows model to simultaneously process high-level disease cat- egorization and nuanced clinical descriptions. Some examples of these multi -ulti-granularity textual descriptions are presented in Table 12
25

---

Published as a conference paper at ICLR 2026
Table 12: The multi-granularity textual description examples of MIMIC-CXR dataset.
Discase Labels|
Radiology Reports
Pleural Effusion
New tracheostony is midline. The approximate diameter of the tube. 11 compares to the diameter of the trachea, 27 mm. This sizing should be evaluated clinically. Pocumomediastinum outinnes the tracheal wall and extends into deep subeutaneous emphysema the neck. presumably function tracheostomy.. Followup advised. There is no pneumothorux pleural effusion. Lungs are cleur. Heart size is normal.
Edemma
The small right apical pneumathorax is stable and unchanged. The right chest tube is in stable position. Unchanged parenchymal opacity at the left lung base. Unchanged size the cardiac silhouette and stable position of the right internal jugular vein catheter.
Atelectasis
Monitoring and support devices are in stable position. Stable left refrocardiac atelectasis and right basal parenchymal opacity. No pulmonary edema. No lurger pleural effusions. No pneumothorax, Support lines and tibes are unchanged in position. The left-sided pleural effusion continues to decrense in size. There is improved acration at the left base. Partially layering right sided pleural effusion is again seen. There new small left-sided apical pacumothorax.
Cardiomegaly
Lung Opacity
Slight worsening of curdiomegaly and moloeetttmdeerte pulmonury edema, uccompanied by increasing moderate left pleural effusion and persistent small right pleural effusion. Indwelling support and monitoring devices are unchanged in position, including proximally located left PICC, terminuting the junction of the left axillary and subclavian veins.
Lung Lesion
Single portable upright AP image of the chest. There are low long volumes. There is an interval increase pulmonury edema. The cardiomediastinl silhouette appears to be somewhat ealarged from prior exam, particularly in the right mediastinum. There no lurge pleural effusion or pneumothorax. A pacer is seen overlying left anterior chest with intact leads appropriate position.
the alveolar opacities biluterally, consistent with moderate to
Pacumonia
Portable AP radiograph of the chest was reviewed with no prior studies available for comparison. Heart size is top normall Mediastinum is grossly unremarkable. Lungs are essentially clear except for right basal opacity which unclear represents true lesion summation of shadows, Repeated radiograph preferably with full inspiration is required. If finding is persistent, assessment with chest CT would be necessary.
No Finding
Trachenstomy tuhe in satisfactory position with the tip 4.5 cm ahove the carina. The right internal jugular central line and nasogastric tube are unchanged in position. The heart remains stably enlarged. Lung volumes are markedly reduced and there is a small layering left effusion with persistent retrocardiac consolidation likely reflecting partial lower labe atelectasis. No pulmonary edema. obvious pneumothorax.
B.2 DOWNSTREAM DATASETS
We evaluate our proposed MGLL alongside several baseline methods on multiple downstream datasets. The details of these datasets are presented in Table 13 The multiple-choice evaluation benchmark details are in Table [4]
Fundus Imaging Datasets:
Jin et al. 2022): : A collection of 800 retinal images categorized into four diagnostic classes: normal, age-related macular degeneration, diabetic retinopathy, and glaucoma.
FIVES
IDRDD Porwal et al. (2018): Contains 516 retinal images with annotated severity grades for diabetic retinopathy (DR) and diabetic macular edema (DME).
OLA-DDR Li ct al.
2019b:: Comprises 12,523 fundus images labeled with diabetic retinopathy
severity classifications.
ADAM Fang et al. [002]; A dataset of 1,200 fundus images specifically designed for age-related macular degeneration detection.
PALM Fang et al. (022)) Consists of 1,200 fundus images annotated for pathological myopia diagnosis.
REFUGE Orlando et al. (2020): Includes 1,200 retinal images with binary classification for glau- comatous and non- -onn-luucommtous.
RIM-ONE Batista et al.
(2020): A retinography collection of 485 images developed for glaucoma
evaluation.
RFMiD Pachade et al. (002): Encompasses 3,200 retinal images with multi-label annotations
across 45 categories. Our evaluation focuses on 12 labels where positive cases exceed 2% preva- lence.
Radiographic Imaging Datasets:
MIDRC-XR MIDRC (2024): A dataset contains 111,816 X-ray images across 14 LOINC-coded categories (including XR Chest AP views).
26

---

Published as a conference paper at ICLR 2026
Table 13: The details of downstream datasets.
Name
Numer (Train Val Test)
Label Categories
FIVES Jin al. 2022
480 : 120 200
Normal, Age-related Macular
Degencration, Diahetic retinopathy, and Glaucoma
IDRiD Porwal et al (2018)
319: 94 103
Severity levels of Diabetic Retinopathy (no apparent, mild non-proliferative. moderate non-proliferate, severe non-proliferate, proliferative) and Diabetic Macular Edema (no apparent, mild, moderate, severe) Severity levels of Diabetic Retinopathy (no apparent, mild non non-proliferative, moderate non pooo--rollfttee severe non- non-proliferate, proliferative)
OIA-DDR Li et al. (2119b
6260 : 2503 3759
ADAM Fang et al. 2022
400 : 400 400
Age-related Macular Degeneration and no Age-relatedd Macular Degeneration Pathological myopia and Healthy
PALM
Fang al. 2024
400 : 400 400
REFUGE Orlando el al.
2020
400 : 400 400
Glaucoma and Healthy
RIM-ONE Batista et 2020
270 69 146
Glaucoma and Healthy
REMiD
Pachade et al.
(2021
1920 640 640
Diabetic Retinopathy, Age-related
Macular Degeneration, Media haze, drusens, Myopia, Branch Retinal Vein Occlusion, Tessellation, Laser scars, Optic disc cupping, Optic disc pallor, Optic disc edema, and Retinitis XR Chest AP, XR Chest Views, XR Unspecified body region Views, Chest Single view, XR Chest PA and Lateral, XR Chest Views, XR Chest AP and Lateral, XR Chest and Abdomen Single view, XR Ribs Views, XR Abdomen AP, XR Abdomen Single view, XR Chest View and Abdomen Supine and Upright, XR Abdomen Supine and Upright, XR Ribs Views and Chest PA
MIDRC-XRMMIDRC 2024 V2224
89453 11182: 11181
MIDRC-XR-Portaee MIDRC12024
63253 7906 7907
Portable XR Chest AP single view, Portable XR Chest Views AP, Portable XR Abdomen AP, Portable XR Chest Views, Portable XR Chest Views W inspiration expiration, Portable Abdomen Supine and Upright Atelectasis, Cardiomegaly, Effusion. Infiltration, Mass, Nodule, Pneumonia, Pneumothorax, Consolidation, Edema, Emphysema, Fihrosis, Pleural Thickening, Hernia
ChestX-ray 14 Wang al. 201 /a
77872: 8652 25596
27

---

Published as a conference paper at ICLR 2026
MIDRC-XR-Portableeee (2024): Focuses on portable radiography with 79,066 X-ray images across 6 LOINC-coded categories (including Portable XR Chest AP single views).
ChestX-ray14 Wang et al. (2017a) 7a): A comprchensive medical imaging repository contains 112,120 frontal-view chest radiographs annotated with 14 labels. Labels are extracted from corresponding radiological reports.
Evaluation Benchmark on MLLMs (Multiple-choice Benchmark): comprehensive ophthalmic evaluation dataset comprising 2,233 images across 25 distinct diagnostic categories. The label dis- tribution of the multiple-choice evaluation benchmark is in Table
Table 14: The label distribution of the multiple-choice evaluation benchmark.
Label
Num
Label
Num
Health
890
Other disease (Other)
Myopia
226
Tessellation
12
Retinitis
9
Chorioretinitis
3
Diabetic Retinopathy (DR)
149
Drusen
30
Media Haze (MH)
31
Central Serous Retinopathy (CSR)
7
Cataract
20
Arteriosclerotic Retinopathy (AR)
2
Optic Disk Cupping (ODC)
32
Optic Disc Edema (ODE)
11
Optic Disc Pallor (ODP)
Hypertensive Retinopathy (HR)
Branch Retinal Vein Occlusion (BRVO)
16
Central Retinal Vein Ocelusion (CRVO)
11
Age-related Macular Degeneration (AMD)
171
Age-related Macular Degencration (No AMD)
Diabetic Macular Edema (DME)
58
No Diabetic Macular Edema (No DME)
11
Glaucoma
162
No Glaucoma
12
Choroidal Ncovascularization (CN)
4
Summary
2233 images
C
SETUP DETAILS
C.I EVALUATION METRICS
In our quantitative evaluation, we employ Area Under the Receiver Operating Characteristic Curve (AUC), mean Average Precision (mAP), and Accuracy (ACC) as assessment metrics. Among these, AUC serves as primary evaluation metric, as it reflects overall model performance. mAP is particularly useful for handling long-tailed label distributions. To better assess performance on the imbalanced multi-label dataset such RFMiD Pachade al. (2021), we report the category-wiis average accuracy instead of overall accuracy. As for the accuracy on the multiple-choice bench- mark, implement a four-option forced-choice classification approach utilizing the MGI.L-Fundus dataset. For each fundus image presented, the model must select the most probable diagnostic clas- sification from among four distinct disease labels. These options comprise the correct diagnostic label along with three additional labels randomly sampled from the complete disease names avail- able in the dataset. The randomized inclusion of incorrect options helps evaluate model performance in distinguishing the correct diagnosis from plausible alternatives, which also enables quantitative assessment of diagnostic accuracy.
C.2 IMPLEMENTATION DETAILS
We adopt VIT-L/14 Dosovitskiy et al. (2020) as the image encoder and (oomediialBERRT/Alseneee jet al. 2219)) as the text encoder. All images are resized to 224 x 224, and data preprocessing includes random flipping (probability 0.5) and color jittering (factor 0.1). We set the batch size to 32, the feature vector dimension to 768, and the temperature coefficient to 0.07. Optimization is performed using AdamW Loshchilov & Hutter (2017) with learning rate of 1e-4, weight decay of 0.0001, and hyperparameters 31 = 0.9, B, = 0.98, and = le All experiments are conducted with the NVIDIA RTX A6000 GPU.
28

---

Published as a conference paper at ICLR 2026
D MORE DETAILED EXPERIMENTAL RESULTS
D.1 DETAILED RESULTS ON RETINAL FUNDUS DATASETS
We present the complete experimental results for performance comparison with eight baselines (Radford et al. (2021): Yao et al. ) ; Zhou et al. (023bb Rolva-Rodrigeez et al. (2025); Wu 2024): Khattak al. (2024): Quu et (024)) on downstream retinal fundus datasets in Table 15 to Table 23 The experimental results demonstrate that the MGLL consis- tently outperforms existing approaches across nine retinal fundus datasets both linear probing and full fine-tuning evaluation settings. Notably, MGLL demonstrates particularly strong gains in the linear probing setting, where it demonstrates substantial improvements over second-bor best methods (e.g., achicving 90.02% AUC in ADAM compared to UniMed-CLID 79.33%, and 92.42% AUC REFUGE versus RET-CLIP's 84.59% To further analyze these results, we observe consistent performance improvements across multiple evaluation metrics. For instance, in the FIVES dataset, MGLL achieved 89.73% AUC, 52.00% ACC, and 75.32% mAP in linear probing, significantly out- performing RETFound (88.09% AUC, 49.00% ACC, 72.55% mAP). When fully fine-tuned on this dataset, MGLL maintained its advantage with 94.98% AUC, 72.00% ACC, and 86.34% mAP.
al. (2024);
Du Du
The robust performance of MGLL can be attributed its multi granularity learning approach, which effectively captures both local and global features in retinal fundus images. This architectural advan- tage enables MGLL to identify subtle pathological patterns that may overlooked by conventional methods. For example, in the REFUGE dataset (glaucoma detection). MGLL achieved a 7.83% improvement AUC over the second-best method in linear probing setting.
Additionally, the exceptional performance on the PALM dataset 999.66 AUC, 96.00% ACC, and 99.72% mAP in linear probing) demonstrates MGLL's capacity to achieve near-perfect diagnostic accuracy in certain retinal conditions. When compared to previous state-of-the-art methods such as VisionFM (97.12% AUC) and RET-CLIP (95.25% AUC), MGLL offers clinically significant im- provements in detection reliability. This superior performance indicates excellent feature represen- tation quality and transferability of our pretrained MGLL,, enabling effective adaptation to diverse diagnostic tasks with fine-tuning.
Table 15:
The performance evaluation on FIVES. Bold indicates best performance and underline
shows second-best.
Linear Probe
(%)
Fully Fine-tune (%)
Method
AUC
ACC
mAP
AUC
ACC
mAP
CLIP (ICML-21)
81.25
37.00
64.21
88.96
64.00
76.22
KgCoOp (CVPR-23)
81.63
39.50
64.72
89.16
64.50
76.53
RETFound (Nature-23)
88.09
49.00
72.55
92.83
69.50
81.36
FLAIR (MedIA-25)
84.24
43.50
67.17
89.85
66.00
77.14
KeepFIT (MICCAI-24)
82.31
41.00
64.98
90.62
67.00
77.96
RET-CLIP (MICCAI-24)
86.74
47.50
69.35
92.04
68.50
79.89
UniMed-CLIP (arXiv-24)
82.75
41.50
65.46
91.59
68.00
79.23
VisionFM (NEJM AI-24)
85.98
45.00
68.73
93.11
70.50
82.76
MGLL
89.73
52.00
75.32
94.98
72.00
86.34
29

---

Published as a conference paper at ICLR 2026
Table 16:
The performance evaluation on IDRiD (DR). Bold indicates best performance and
underline shows second-best.
Linear Probe ( (% )
Fully Fine-tune (%)
Method
AUC
ACC
mAP
AUC
ACC
mAP
CLIP (ICML-21)
70.83
35.92
34.44
76.74
44.66
44.01
KgCoOp (CVPR-23)
72.34
40.78
36.29
76.81
46.60
44.86
RETFound (Nature-23)
73.51
43.69
37.48
78.14
53.40
47.05
FLAIR (MedIA-25)
74.62
45.63
39.16
78.82
55.34
48.54
KeepHIT (MICCAI-24)
71.81
37.86
35.37
77.13
48.54
45.32
RET-CLIP (MICCAI-24)
78.18
52.43
47.42
79.32
56.31
49.42
UniMed-CLIP (arXiv-24)
77.39
49.51
45.75
80.72
58.25
52.84
VisionFM (NEJM AI-24)
77.52
51,46
46.22
79.95
57.28
51.38
MGLL
80.28
58.25
51.19
82.57
60.19
54.30
Table 17: The performance evaluation on IDRiD (DME). Bold indicates best performance and underline shows second-best.
Linear Probe
(%)
Fully Fine-tune (%)
Method
AUC
ACC
mAP
AUC
ACC
mAP
CLIP (ICML-21)
71.81
77.67
56.67
73.34
76.70
57.87
KgCoOp (CVPR-23)
60.33
65.05
48.86
62.78
66.02
50.11
RETFound (Nature-23)
65.92
69.90
52.05
69.26
72.82
55.39
FLAIR (MedIA-25)
62.85
66.99
50.31
64.52
68.93
51.83
KeepFIT (MICCAI-24)
68.71
70.87
54.76
69.03
71.84
54.97
RET-CLIP (MICCAI-24)
64.13
67.96
51.28
70.14
73.79
55.75
UniMed-CL.D (arXiv-24)
70.59
74.76
56.04
71.81
75.73
56.85
VisionFM (NEJM AI-24)
73.53
78.64
59.98
77.95
79.61
62.23
MGLL
78.41
79.61
64.72
86.17
80.58
67.80
Table 18: The performance evaluation OIA-DDR. Bold indicates best performance and underline shows second-best.
Linear Probe (
Fully Fine-tune (%)
Method
AUC
ACC
mAP
AUC
ACC
mAP
CLIP (ICML-21)
73.30
55.41
39.21
85.29
71.75
49.91
KgCoOp (CVPR-23)
72.09
53.68
37.87
80.39
65.47
45.92
RETFound (Nature-23)
80.77
61.13
46.94
85.96
72.12
52.21
FLAIR (MedIA-25)
82.48
63.36
48.09
85.54
70.63
50.27
KeepFIT (MICCAI-24)
72.68
54.46
38.72
79.42
64.03
44.53
RET-CLIP (MICCAI-24)
77.43
58.18
43.31
83.68
68.82
48.05
UniMed- EniMed-CLIP (arXiv-24)
74.67
56.19
40.18
81.23
66.69
46.88
VisionFM (NEJM AI-24)
78.85
59.35
44.27
84.25
69.46
48.77
MGLL
86.28
72.09
50.92
88.85
73.13
56.67
Table 19: The performance evaluation on ADAM. Bold indicates best performance and underline shows second-best.
Linear Probe (%)
Fully Fine-fune (%
Method
AUC
ACC
mAP
AUC
ACC
mAP
CLIP (ICML-21)
52.41
76.50
25.05
86.70
83.00
65.48
KgCoOp (CVPR-23)
52.13
74.75
24.36
84.42
82.75
60.54
RETFound (Nature-23)
59.34
78.25
33.53
88.02
83.50
65.86
FLAIR (MedIA-25)
63.82
79.50
39.27
90.16
84.75
66.91
KeepFIT (MICCAI-24)
56.97
77.75
29.88
74.88
82.00
52.26
RET-CLIP (MICCAI-24)
77.48
82.25
53.76
93.27
86.25
79.92
UniMed-CLIP (arXiv-24)
79.33
82.50
55.63
90.64
85.25
69.22
VisionFM (NEJM AI-24)
73.56
81.75
51.42
92.43
85.50
75.89
MGLL
90.02
85.00
62.40
96.30
90.00
90.08
30

---

Published as a conference paper at ICLR 2026
Table 20: The performance evaluation on PALM. Bold indicates best performance and underline
shows second-best.
Linear Probe ( (%)
Fully Fine-tune (%)
Method
AUC
ACC
mAP
AUC
ACC
mAP
CLIP (ICML-21)
93.94
91.75
94.62
99.51
96.00
99.58
KgCoOp (CVPR-23)
87.94
88.25
88.11
94.21
92.00
94.81
RETFound (Nature-23)
92.02
90.50
92.87
95.75
93.00
96.39
FLAIR (MedIA-25)
89.42
89.25
90.03
94.88
92.25
95.24
KeepFIT (MICCA1-24)
88.21
88.50
88.46
93.34
91.50
94.18
RET-CLIP (MICCAI-24)
95.25
92.50
95.89
98.67
95.25
98.42
UniMed-CLIP (arXiv-24)
90.31
89.75
91.12
96.82
93.75
97.02
VisionFM (NEJM AI-24)
97.12
94.25
97.45
97.73
94.50
97.84
MGLL
99.66
96.00
99.72
99.72
95.75
99.76
Table 21: The performance evaluation on REFUGE. Bold indicates best performance and underline shows second-best.
Linear Probe (%)
Fully
Fine-tune (%)
Method
AUC
ACC
mAP
AUC
ACC
mAP
CLIP (ICML-21)
65.33
88.75
15.68
86.96
93.00
58.37
KgCoOp (CVPR-23)
60.69
86.25
11.52
81.45
90.75
49.07
RETFound (Nature-23)
79.67
90.50
45.39
89.02
93.50
63.84
FLAIR (MedIA-25)
70.59
89.25
27.82
85.67
92.25
56.82
KeepFIT (MICCAI-24)
63.04
88.25
14.15
84.89
91.50
55.78
RET-CLIP (MICCAI-24)
84.59
91.25
55.06
90.46
93.75
70.45
UniMed-CLL CLIP (arXiv-24)
61.25
87.50
12.87
83.11
91.00
52.84
VisionFM (NEJM AI-24)
73.21
89.50
33.42
86.13
92.50
57.68
MGLL
92.42
94.50
75.65
93.90
94.75
80.99
Table 22: The performance evaluation on RIM-ONE, Bold indicates best performance and underline shows second-best.
Linear Probe ( (%)
Fully Fine-tune ( (%)
Method
AUC
ACC
mAP
AUC
ACC
mAP
CLIP (ICML-21)
65.96
66.44
54.11
88.38
82.88
84,00
KgCoOp (CVPR-23)
74.34
73.97
63.45
90.39
84.25
85.88
RETFound (Nature-23)
89.79
83.56
83.92
94.22
86.99
90.35
FLAIR (MedIA-25)
81.83
79.45
70.14
94.93
88.36
92.41
KeepFIT (MICCAI-24)
67.35
67.81
55.24
89.91
83.56
85.22
RET-CLIP (MICCAI-24)
84.42
82.19
79.85
92.58
86.30
89.19
UniMed-CLIP (arXiv-24)
69.87
70.55
58.43
83.37
81.51
78.14
VisionFM (NEJM AI-24)
72.97
72.60
61.86
91.27
84.93
87.21
MGLL
94.39
87.67
86.68
97.05
89.73
94.97
Table 23: The performance evaluation on RFMiD. Bold indicates best performance and underline shows second-best.
Linear Probe ( (%)
Fully Fine-tune ( (%)
Method
AUC
ACC
MAP
AUC
ACC
mAP
CLIP (ICML-21)
44.66
92.53
7.28
65.10
92.86
17.31
KgCoOp (CVPR-23)
50.82
92.21
7.96
70.36
92.28
21.49
RETFound (Nature-23)
60.16
92.55
16.37
84.62
93.48
50.21
FLAIR (MedIA-25)
56.11
92.38
14.85
75.43
92.62
23.24
KeepFIT (MICCAI-24)
51.48
92.19
10.24
81.52
93.09
42.39
RET-CLIP (MICCAI-24)
58.94
92.46
16.02
86.12
93.92
51.27
UniMed-CLIP (arXiv-24)
53.44
92.25
13.68
72.59
92.57
22.54
VisionFM (NEJM AI-24)
63.38
92.59
17.84
82.78
93.17
48.92
MGLL
79.62
92.84
34.08
92.83
95.48
64.99
31

---

Published as a conference paper at ICLR 2026
D.2 ZERO-SHOT COMPARISONS ACROSS MEDICAL AND NATURAL DOMAINS
As summarized in Table 24 we evaluate the zero-shot classification performance of MGLL across three representative datasets to demonstrate its strong generalization ability across diverse modal- ities. On the COVIDx dataset Waang et al. (020), MGLL achieves 39.0% accuracy, outperform- ing strong medical vision-language baselines such as CheXAgent [Chen et al. (2024b) (34.3%) and MedVersa Zhou et al. (2024) (35.5%),
well recent contrastive methods including FG- Xie et al. (2025), MGCA Wang et al. (2022)), RetiZero Wang et al. (2025), MAVL[Phaal et al. 20244. and Ark+ Ma al. (2025). For anatomical recognition on the CT-based MNNSST dataset Yang et al. (2023), MGLL again surpasses FG-CLIP Xie et al. (2025) with a significant margin (52.7% % vs. 47.9%) Moreover, MGLL achieves the best performance on the natural image dataset CC3M Sharma et al. (2018) (evaluated on ImageNet), outperforming FG-CLIP Xie et al. (2025) with an accuracy of 23.5%. These results collectively highlight MGLL's flexibility and uni- versal applicability across both medical and natural image domains in zero-shot settings.
Comparisons of Zero-Shot classification on MGLL and others methods. * denotes using Table 24:
published pretrained model.
Method
Pretrain Data
Downstream Data
ACC (%)
FG-CLIP (ICML-25)
CC3M CC3M
ImageNet ImageNet
21.4
MGLL
23.5
FG-CLIP (ICML-25)
PMC-OA
OrganAMNNITT
47.9
MGLL
PMC-OA
OrganAMNIST
52.7
CheXAgent (Arxiv-24)
COVIDx
34.3
MedVersa (Arxiv-24)
COVIDx
35.5
FG-CLIP (ICML-25)
MIMIC-CXR
COVIDx
36.3
MGCA (NIPS-22)
MIMIC-CXR
COVIDx
37.3
RetiZero (Nat. Com-25)
MIMIC-CX CX-R
COVIDx
35.8
MAVL (CVPR-24)
MIMIC-CXC
COVIDx
37.0
Ark+ (Nature-25)
MIMIC-CXR
COVIDx
37.8
MGLL
MIMIC-CXR
COVIDx
39.0
D.3 PERFORMANCE EVALUATION ON REGION SEGMENTATION
We have evaluated MGLL region segmentation tasks, and the results reported in Table 25 Medical image segmentation is inherently challenging due to the subtle differences between adjacent pixels of heterogeneous classes. We compared MGLL with several excellent methods such as ALuRI et al. (2021), RadFRad et al. (202)), LAVi et al. 2022), UniLSegLiu et al. 22024. and STPNe(Shan al. (2025). By incorporating multi-level semantic alignment, MGLL en- hances the model's 's language- guided spatial understanding and achieves the best performance among compared methods.
Comparisons of COVID-19 lesion segmentation between MGLL and others methods on Table 25:
COVID-Xray dataset Degerli et (20TT).
Method
Dataset
Dice (%)
IoU(%)
GLoRIA (ICCV-21)
COVID-Xray
79.94
70.68
CLIP (ICML-21)
COVID-Xray
79.81
70.66
LAVT (CVPR-22)
COVID-Xray
79.28
69.89
UniLSeg (CVPR-24)
COVID-Xray
79.99
70.29
STPNet (TIP-25)
COVID-Xray
80.63
71.42
MGLL
COVID-Xray
81.69
73.06
32

---

Published as a conference paper at ICLR 2026
D.4 MORE DETAILED RESULTS MGLL IN MLLMS
To evaluate MGLL's impact on multimodal large language models' diagnostic capabilities, we con- duct comprehensive multiple-choice evaluations across 25 distinct ophthalmological conditions as shown in Table 27 Table 34 This expanded analysis provides more insights into MGLL's perfor- mance beyond the ten primary conditions highlighted in previous experiments.
Table 26: Brief summary of recent iision-langual models.
Model Name
Key Fentures and Deseription
InstructBI. Dai et al.
2023
Combines vision and language with instraction tuning enable versatile zero-shot performance across tasks. multimodal model designed
Mini-Gemin
lightweight ght
inference strong performance.
Qwen- VI or al.
1123
Supports multimodal reasoning with strong focus on Chinese visioe-langunge understanding. Achieves strong cross-tnodal aligamcat and geacralization across image-text beachmarks.
InterVil nen cl
21246
LLav
Integrates CLIP and LLMMA for open-ended visual question answering and dialogue. Adapts LLaVA for medical vision-language tasks including medical image question ans answering.
LLaVA-Ma MaaLrr sl 2023
Med-Flaming Moor dta..
2023
Extends Flamiago the medical domain with few-shot learning capubilities. Uses bidircctional multimodal modeling to enlance multi-turn visual-language interactions.
Janus-ProChau al.
2025
Our detailed analysis reveals that MGLL integration yields significant improvements across all seven multimodal architectures. InstructBLIIP demonstrates a 9.76% overall accuracy improve- ment (55.17% to 64.94%), with particularly notable enhancements in challenging conditions such as Retinitis (11.11% to 44.44%) and Media Haze (16.13% to 45. 1.6).. These improvements high- light MGLL's capacity to enhance feature extraction for complex ophthalmological pathologies. MGLL integration showcases substantial performance gains across multiple diagnostic tasks and architectures. Qwen- VL with MGLL integration exhibits a 6.58% overall improvement (76.80% to 83.39%), with remarkable advances in low-prevalence conditions including Astigmatic Refractive Error (0.00% 50.00%) and Hypertensive Retinopathy (0.00% to 66.67%) Even high-perforning models benefit significantly, InternVI. achieves 5.19% improvement with MGLL integration, enhancing diagnostic accuracy particularly for conditions such as Optic Disc Edema (45.45% to 63.64%) and Central Retinal Vein Occlusion (36.36% to 63.64%). LLaVA exhibits similarly bust baseline performance (85.22%), yet MGLL integration yields a 5.55% improvement, achiev- ing near-perfect accuracy in several categories including No Diabetic Macular Edema (90.91% to 100.00%) and miscellaneous conditions (90.00% to 100.00%). The most dramatic improvements occur in medical-specializee models. Med-Flamingo demonstrates substantial 21.76% improve- ment (49. 17% to 70.94%), with particularly significant gains in Glaucoma (24.07% to 61.1 (1%) and Diabetic Retinopathy (36.91% 80.54%). Similarly, LLaVA-Medd shows 20.78% improvement (56.47% to 77.25%), with exceptional gains in AMD (16.37% to 58.48%) and Diabetic Retinopathy (26. 85.85% % to 77.18%) Across evaluated models, we observe consistent patterns of improvement that highlight MGLL's particular efficacy with complex retinal conditions, vascular pathologies, and conditions requiring fine-grained feature discrimination. The results demonstrate that MGLL pro- vides substantial benefits regardless of the underlying model architecture. The multiple-choice eval- uation framework presented models with standardized diagnostic queries. Some prompt examples of the multiple-choice evaluation benchmark are as follows:
Question 1: "What is the most reasonable diagnosis? A. Glaucoma B. Drusen C. Chorioretinitis D. Hypertensive Retinopathy Answer with the option's 's letter from the given choices directly."
Answer 1: A.
Question 2: "What diagnosis is most likely? A. Central Serous Retinopathy B. Media Haze C. Diabetic Retinopathy D. Age-related Macular degeneration Answer with the option's 's letter from the given choices directly."
Answer 2: D.
Question 3: "What diagnosis is most probable? A. Optic Disk Cupping B. Mild Non-Proliferatin Diabetic Retinopathy C. Central Serous Retinopathy D. Central Retinal Vein Occlusion Answer with the option's 's letter from the given choices directly."
Answer 3:
33

---

Published as a conference paper at ICLR 2026
Table 27: Comparison of multiple-choice accuracy with MGLL in InstructBLIP on the multiple-
choice evaluation benchmark.
Label Name
AMD
BRVO
Cataract
Chorioretinitis
CN
CRVO
CSR
InstructBLIP BIP Dai et al.
2023
80.17%
0.00%
50.00%
80.00%
0.00%
75.00%
45.45%
0.00%
MGLL
83.63%
50.00%
%
85.00%
33.33%
75.00%
63.64%
28.57
Diabetic Retinopathy
Drusen
Glaucoma
Health
HR
DMB
MH
Myopia
No AMD
76.51%
36.67%
59.30%
51.57%
0.00%
63.79%
16.
44.25%
54.34%
82.55%
50.00%
65.43%
60.22%
33.33%
74.14%
45.16%
52.65%
70.10%
Glaucoma
DME
ODC
ODP
Retinitis
Tessellation
Overall
41.67%
63 33.644
65 63.66%
27.27%
50.00%
58. 58.00%
11.11%
41.67%
55.17%
58.33%
72.73%
75.00%
70.00%
44.44%
58.33%
64.94% (9.76% 1)
Table 28: Comparison of multiple-choice accuracy with MGLL in Mini-Gemini on the multiple- choice evaluation benchmark. Label
BRVO
Cataract
Chorioretinitis
CRVO
CSR
Mini-Gemini Li ct al. 2024
76.61%
0.00%
43.75%
85.00%
0.00%
25.00%
27.27%
14.29%
MGLL
82.46%
0.00%
56.25%
85.00%
66.67
50.00%
54.55%
42.86%
Diahetic Retinopathy
Drusen
Glaucoma
Health
HR
DME
MH
Myopia
No AMD
79.87%
23.33%
67. 90%
61.46%
60.34%
58.41%
63.02%
84.56%
46.67 %
72.22%
68.99 %
33.33%
65.52%
58.06%
64.16%
72.99%
No Glaucoma
No DME
ODC
ODE
ODP
Other
Retinitis
Tessellation
Overall f
66.67%
72.73%
62.50%
36. 36.36%
50.00%
66 66.00%
33.33%
33.33%
62.65%
75.00%
81.82 %
78.13%
45.45%
50.00% %
74.00%
55.56%
41.67%
70.58 % (7.93% 1)
Table 29: Comparison of multiple-choice accuracy with MGLL in Qwen VL on the multiple-choie
evaluation benchmark.
Label Name
AMD
AR
BRVO
Cataract
Chorioretinitis
CN
CRVO
CSR
Qwen- -L. Bai al. 2023
81.87%
0.00%
43.75%
75.00%
0.00%
25.00%
9.09%
28.57%
-MGLL
85.96%
50.00%
62.50%
80.00%
33.33 %
75.00%
36.36%
42.86%
Diabetic Retinopathy
Drusen
Glaucoma
Health
HR
DME
MH
Myopia
No AMD
80.54%
26.67%
78.40%
79,89%
0.00%
84.48%
54.84%
76.55%
82,965
89.93%
43.33%
87.04%
85.39%
66.67%
89.66
70.97%
80.97%
87.14%
No Glaucoma
No DMB
ODC
ODE
ODP
Other
Retinitis
Tessellation
Overall 1
25.00%%
72.73%
$6.25 1
27.27%
22.22%%
25.00%
76.80%
83.33 %
72.73%
68.75%
54.55%
100,00%
86.00 %
33.33%
41.67%
83.39 (6.58% ')
Table 30: Comparison multiple-choice accuracy with MGLL InternLL on the multiple-choice evaluation benchmark.
Label Name
AMD
AR
BRVO
Cataract
Chorieretinitis
CN
CRVO
CSR
Intern Chen et 2024a
81.29%
0.00%
37.50%
85.00%
0.00%
25.00%
36.36%
71.43%
MGLL
86.55%
50.00%
56.25%
90.00%
0.00%
50,00%
63.64%
71.43%
Diabetic Retinopathy
Drusen
Glaucoma
Health
DME
MH
Myopia
No AMD
94.63%
43.33%
89.51%
85.73%
33.33%
87.93%
64.52%
88.05%
85.85%
96.64%
53.33%
90.74%
91.46%
66.67% %
94.83%
67.74%
91.15%
90.68% %
No Glaucoma
No DME
ODC
ODE
ODP
Other
Retinitis
Tessellation
Overall
91.67%
81 82%
87.500 87.50%
45.45%
50.00%
90.00%
44.44%
66.67%
84.33%
100.00%
90.91%
93.75%
63.64%
50.00%
96.00%
55.56%
75.00%
89.52% (5.19%) 1>
Table 31: Comparison of multiple-choice accuracy with MGLL in LLaVA on the multiple-choiee
evaluation benchmark.
Label Name
AMD
BRVO
Cataract
Chorioretinitis
CRVO
CSR
LLaVA Liu al. 2023
83.04%
0.00%
50.00%
90.00%
0.00%
25.00%
9,09%
42.86%
MGG.
84.80%
100.00%
68.75% %
90.00% %
33.33%
25.00%
45.45%
57.14%
Diabetic Retinopathy
Drusen
Glaucoma
Health
HR
DME
MH
Myopia
No AMD
87.25%
36.67%
91.36%
88.65%
0.00%
93. 10%
48.39%
88. 58.50%
90.68%
93.96%
50.00%
91.98% %
33.33%
96.55%
61.29%
90.71 %
96.78%
No Glancoma
No DME
ODC
ODE
ODP
Other
Retinitis
Tessellation
Overall ?
100.00%
90.91%
62.50%
18.1%%
50.00%
90.00%
44.44%
58.33%
85.22%
100.00%
100.00
62.50%
45.45% %
100.00%
100.00%
66.67
66.67 %
90.77% (5.55%
34

---

Published as a conference paper at ICLR 2026
Table 32: Comparison of multiple-choice accuracy with MGLL in LLaVA-Med the multiple-
choice evaluation benchmark.
Label Name
AMD
AR
BRVO
Cataract
Chorioretinitis
CN
CRVO
CSR
LLaVA-MedLi et 2023
16.37%
100.00%
31.25%
15.00%
0.00%
0.00%
45.45%
42.86%
MGLL
58.48%
100.00%
50.00%
65.00%
66.67%
25.00%
63.64%
57.14%
Diabetic Retinopathy
Drusen
Glaucoma
Health
DME
MH
Myopia
AMD
26.85%
16.67%
25.31%
91 46%
33.33%
16.67%
25.81%
23.89%
66.56%
97.42%
55.17%
78.46%
No Glaucoma
No DME
ODC
ODE
ODP
Other
Retinitis
Tessellation
Overall
16.67%
36.36%
21.88%
27.27%
0.00%
28.00%
33.33%
16.67%
56.47%
41.67%
72.73%
40.63%
63.64 %
50,00 %
62.00% %
44.44
58.33%
77.25 255 (20.78% 1)
Table 33: Comparison of multiple-choice accuracy with MGLL in Med-Flamingo the multiple-
choice evaluation benchmark.
Label Name
AMD
AR
BRVO
Cataract
Chorioretinitis
CN
CRVO
CSR
Pacemnaam Dlamingn/Moor al. 2023
25.73%
100.00%
31.25%
30.00%
0.00%
25.00%
63.64%
57.14%
MGLL
69.01%
100.00%
56.25%
75.00%
33.33%
50.00%
72.73%
71.43%
Diabetic Retinopathy
Drusen
Glancoma
Health
HR
DME
MH
Myopia
No AMD
36.91%
20.00%
24.07%
74.16%
33.33%
24.14%
22.58%
18.58%
52.73%
80.54%
43.33%
61.11%
83.37%
66.67%
51.72%
54.84%
45.58%
69.13%
No Glanenma
No DME
ODC
ODE
ODP
Other
Retinitis
Tessellation
Overall 1
16.67%
36. 36.36%
43.75%
36.36%
0.00%
28.00%
22.225
8.33%
49.17%
50.00%
63.64 %
59.38%
63.64%
50.00%
70.00%
44.44%
33.33%
70.94% (21.76%% 1)
Table 34: Comparison of multiple-choice accuracy with MGLL in Janus-Pro on the multiple-choiee
evaluation benchmark.
Label Name
AMD
BRVO
Cataract
Chorioretinitis
CRVO
CSR
Janue Pro Pro(Chen Chen et al. 2025
88.30%
50.00%
56.25%
75.00%
33.33%
25.00%
36.36%
42,8649
MGLL
90.64%
100.00%
62.50%
85.00%
66.67%
75.00%
63.64%
71.43%
Diabetic Retinopathy
Drusen
Glaucoma
Health
HR
DME
MH
Myopia
No AMD
93.29%
40.00%
90.74%
96.40%
33.33%
62.07%
58.06%
87.17%
92.28%
96.64%
53.33%
95.06%
96.63%
66.67%
70.69
67.74%
90.27%
94.21
No Glaucoma
No DME
ODC
ODE
ODP
Other
Retinitis
Tessellation
Overall f
58.33%
72.73% 90.91 %
81.25%
36.36 36.36%
50.00%
82.00%
33.33% 55.56%
58.33%
88.54%
83.33 5
87.50 %
54.55%
50.00%
42.00 %
75.00 %
91.85% (3.31% 1)
35

---

Published as a conference paper at ICLR 2026
What the probable diagnosis?
A. Hypertensive Retinopathy
Optic Disc Pallor
Central Serous Retinopathy
Choroidal Neovascularization
Answer with the option's letter from the given choices directly.
Ground Truth: D
InstructBLIP: D
Mini-Gemini:
Qwen-VL: C
InternVL: C
InstructBLIP MGLL:
Mini-Gemini MGLL: D
Qwen-VL + MGLL: D
InternVL MGLL: D
Med-Flamingo: A
LLaVA-Med: A
LLaVA: C
lanus-Pro:
Med-Flamingo MGLL:
LLaVA-Med MGLL:
LLaVA MGLL: A
Janus-Pro MGLL: D
What diagnosis is most probable? Arteriosclerotic Retinopathy
B. Chorioretinitis
Retinitis
D. Healthy
Answer the option's letter from the given choices directly.
Ground Truth: B
InstructBLIP: A
Mini-Gemini: A
Qwen-VLL D
InternVL:
InstructBLIP + MGLL:
Mini-Gemini MGLL:
Qwen-VL + MGLL:
InternVL + MGLL:
Med-Flamingo: C
LLaVA-Med: C
LLaVA: LLaVA MGLL: B
Janus- Janus-Pro MGLL: B
Med-Flamingo MGLL:
LLaVA-Med MGLL:
Figure 5: Case Studies (Top: Case 1, Bottom: Case 2) Demonstrating MGLL Integration Impact on
Diagnostic Accuracy of Different Multimodal Large Langue Models (MLLMs).
36

---

Published as a conference paper at ICLR 2026
The Fig. 5 presents two representative case studies demonstrating the diagnostic impact of MGLL integration across multiple MLLMs.
Case displays a fundus image with characteristic features of choroidal neovascularization (CNV), including a well-defined yellowish lesion with surrounding subretinal hemorrhage in the macula. Only InstructBLIP correctly identifies this pathology in its baseline configuration, whereas five mod- els with MGLL integration provide accurate diagnoses, demonstrating MGLL's capacity to enhance the detection of vascular abnormalities.
Case 2 exhibits subtle inflammatory changes consistent with chorioretinitis, characterized by chori- oretinal infiltrates against background of mild vitreous haze, which is condition none of the base- line models correctly identify. Following MGLL integration, six models accurately diagnose this in- flammatory condition, with responses shifting from incorrect options (Arteriosclerotic Retinopathy, Retinitis, or Healthy) to the correct identification.
D.5 MORE CAMS ON RETINAL FUNDUS DATASETS
We present additional class activation maps (CAMs) from CLIP and MGLL on downstream retinal datasets in Fig. These images include cases of diabetic retinopathy, diabetic macular edema, and glaucoma. Through both linear probing and full fine-tuning approaches, consistetty demon- strates more precise lesion localization than CLIP, specifically highlighting pathological features rather than producing diffuse, non-specific activations. In diabetic retinopathy cases spanning mild to proliferative stages, MGLL accurately identifies microaneurysmss, hemorrhages, venous bead- ing, and neovascularization sites. While in diabetic macular edema, effectively localizes retinal thickening and exudate formation with activation intensity proportional to disease severity. For glau- coma, MGLL, appropriately focuses on optic disc abnormalities, cup enlargement, and neural rim thinning-critical diagnostic markers often missed by CLIP, which tends to highlight anatomical landmarks regardless of pathological relevance. These findings demonstrate MGLL's advantages for ophthalmological applications, offering more robust performance for clinical feature detection that supports diagnostic confidence.
D.6 ABLATION STUDY ON WEIGHT FACTORS OF LOss
To investigate the impact of different weight factors in our composite loss function, we conducted an extensive ablation study using the RFMiD dataset as shown in Table 35 Our loss function in- corporates three components with corresponding weight factors (011, 02 and as), and the results demonstrate that weight selection significantly influences model performance across all metrics. Compared to the baseline CLIP model, all our weight configurations show substantial improve- ments, with the optimal configuration (a, = 0.5, 02 1.0, and d's = 1.0) achieving the best performance in both linear probing (79. 62.%% AUC, 92.84% ACC, 34.08% mAP) and full fine-tuning (92.83% AUC, 95.48% ACC, 64.99% mAP) scenarios. Notably, reducing the weight of the first component improved performance, while reducing either the second third component weights resulted in performance degradation, suggesting that the information captured by these components particularly valuable for medical image classification tasks and should be emphasized during training. These findings highlight the importance of appropriate loss weighting in multi-component objective functions and provide empirical evidence for the optimal configuration selection.
Table 35: Ablations of Weight Factors on RFMiD, Bold indicates best performance and underline shows second-best.
Linear Probe ( (%)
Fully Fine-tune (%)
Method
AUC
ACC
mAP
AUC
ACC
mAP
CLIP Radford et al. (2021)
44.66
92.53
7.28
65.10
92.86
17.31
01 T.O, 0gg 1.0, 03: 1.0
79.29
92.83
33.82
92.51
95.35
64.57
aj: 0.5, og: 1.0, ag: 1.0
79.62
92.84
34.08
92.83
95.48
64.99
oj: 1.0, 0g: 0.5, ag: 1.0
78.11
92.79
32.42
91.46
94.99
63.28
0n: 1.0, 02! 1.0, ag: 0.5
78.85
92.80
33.39
92.01
95.18
63.77
37

---

Published as a conference paper at ICLR 2026
D.7 ABLATION STUDIES ON THE TEMPERATURE COEFFICIENT (T)
We have conducted the ablation studies of temperature coefficient and observe that the performance first improves and then drops as the temperature coefficient т increases, as shown in Table 33) A smaller sharpens the similarity distribution, enhancing discrimination but causing training insta- bility. Conversely, a larger produces smoother gradients but weakens alignment. The best results achieved when r = 0.07, which provides good balance between discriminative alignment and stable optimization.
Table 36: Ablations of temperature coefficient (r)
on MIDRC-XR-Portahle
Linear Probe (%)
Fully Fine-tune (%)
Method
AUC
ACC
mAP
AUC
ACC
mAP
CLIP, 0,07
71.43
78.22
22.31
91.83
90.08
83.94
MGLL, 0.05
83.55
88.91
30.49
99.60
98.67
89.71
MGLL, T 0.20
81.89
87.02
28.93
97.89
97.26
87.92
MGLL, T 0.550
79.52
86.19
27.74
95.53
94.57
85.95
MGLL, == =0.07
83.86
89.06
30.62
99.75
98.80
89.87
D.8 PERFORMANCE ON DATASETS WITH ARTIFICIALLY INTRODUCED NOISE
We evaluated the robustness MGLL under varying levels of artificially introduced noise, where 10%-30% of granularity labels were randomly removed. As shown in Table [37] even with 30% missing labels, MGLL achieves AUCs of 79.61% (linear probing) and 96.74% (full fine-tuning), which remain substantially higher than CLIP trained with complete labels (71.43% and 91.83%. respectively). These results demonstrate that MGLL maintains strong robustness against incomplete or noisy granularity supervision.
Table 37: Ablations of missing granularity labels MIDRC-XR-Portabb
Linear Probe (%)
Fully Fine-tune (%)
Method
AUC
ACC
mAP
AUC
ACC
mAP
CLIP, No Missing
71.43
78.22
22.31
91.83
90.08
83.94
MGLL, 10% Missing
82.58
88.15
29.97
99.31
98.39
89.30
MGLL, 20% Missing
81.14
87.25
28.86
98.62
97.95
88.74
MGLL, 30% Missing
79.61
86.23
27.75
96.74
96.02
87.28
MGLL, No Missing
83.86
89.06
30.62
99.75
98.80
89.87
D.9 PERFORMANCE ON DATASETS WITH MIXING GRANULARITY LEVELS
We have evaluated MGLL I. on datasets with mixed granularity levels, and the results are reported in Table 38 Specifically, the pretraining dataset randomly divided into two subsets of equal size: Set with two levels of granularity and Set with a single level. MGLL achieves comparable per- formance across both subsets and their combination, demonstrating its robustness to heterogeneous annotation structures and its applicability to real-world scenarios with mixed- -iaeedggraularrty data.
Table 38: Ablations of mixing granularity levels on MIDRC-XR CEIeC-X-PPortabll The dataset is randomly divided into two subsets of equal size for the ablation study, Set A (50% data) and Set B (50% data).
Linear Probe (%)
Fully Fine-tune (%)
Study Desc.
Series Desc.
AUC
ACC
mAP
AUC
ACC
mAP
Set A
Set B
78.93
85.47
27.04
94.81
93.58
85.14
Set A
Set A Set B
80.25
86.46
27.97
96.21
94.87
86.72
Set + Set B MGLL (Ours)
Set B
80.92
87.07
28.63
97.03
96.56
87.45
83.86
89.06
30.62
99.75
98.80
89.87
38

---

Published as a conference paper at ICLR 2026
E DISCUSSION AND FUTURE WORK
Our investigation into Multi-Granular Language Learning (MGLL) reveals several important in- sights about vision-language alignment in complex domains. The consistent performance improve- ments across various medical imaging datasets demonstrate that hierarchical textual information substantially enhances visual understanding, particularly when images correspond multiple clin- ical findings at different levels of specificity. The ablation studies confirm that performance gains scale with both the number of granularity levels and the quality of input data, suggesting that MGLL effectively leverages the complementary information contained in multi-granule textual descrip- tions.
While MGLL achieves simultaneous multi-label and cross-granularity alignment without additional computational cost, further optimization could potentially improve its generality. Future work should explore several directions: (I) extending MGLL to incorporate multimodal inputs beyond images and text, such as patient metadata or temporal information; (2) investigating domain adap- tation techniques to improve generalization to unseen medical conditions or imaging modalities; and (3) exploring the integration of MGLL with large language models to generate more nuanced textual descriptions multiple granularities. Additionally, applying MGLL to other domains with inherently hierarchical structures, such as satellite imagery or scientific visualization, could fur- ther validate its broader applicability beyond medical imaging. These extensions would strengthen MGLL's position as generalizable framework for improved multimodal understanding.
39

---

Published as a conference paper at ICLR 2026
Linear Probing
Fully Finetuning
MLM Non- Non-Prolieaiie Prohieeuiie
Disberic Reinogatit
Moderat Non-Troliieatis Retinopotty
Sevtre Now- NoneeNNo-leennee Prolifemti Reinopatiy
Prolifcative Reinupathy
Giancoma
MEN Dalvete
Mild Diobenie
Sever Dalvele
Sever Dibbele
Sevees Divlotee
Original
CL.PP
MGLL
CLIP
MGLL.
Figure 6: More Class Activation Maps from CLIP and Proposed MGLL.
40


Reviews:
[{"review_id": "jO1gsLui7U", "reviewer": "Reviewer_o8AL", "summary": "This paper presents a vision-language pretraining method for fundus and chest X-ray images. The method considers the multi-granularity of medical disease labels and improves the contrastive learning framework with multi-label and cross-granularity alignment. Experiments are conducted on public fundus and chest X-ray images.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper addresses an important and clinically relevant topic, multi-granular vision-language representation learning for medical imaging, which has strong potential to improve medical AI systems by aligning visual features with diagnostic text at different levels of detail.", "weaknesses": "- Although the paper mentions multi-label alignment as part of its motivation, it remains unclear how labels are defined within the framework, and what the exact relationship is between multi-label and multi-granularity. Does a “label” refer to a textual description at a specific granularity level?\n\n- Ambiguity in the Definition of Text Features in Point-wise Loss. It is not explicitly stated whether T_j refers to a single label at a given granularity level, or a concatenation of multiple labels from different granularity levels, similar to the definition used elsewhere in the paper (e.g., T_i). Moreover, it is unclear what M\\ specifically represents in Equation (4). These details are important for accurately understanding the methodology presented in the paper.\n\n- While the authors impose KL-based consistency across granularities, they simultaneously learn granularity-specific alignments using soft-CLIP. This duality is not reconciled theoretically or empirically. There is no investigation into whether enforcing uniformity across granularities harms the model’s ability to capture granularity-specific patterns.\n\n- The idea of improving the multi-granularity of vision-language alignment is not new and has been explored in previous works. It is unclear how the proposed method differs from or advances beyond previous approaches, such as [R1][R2].\n\n- The paper lacks comparison with closely related multi-granularity alignment methods such as MGCA [R1]\n\n- Both retinal fundus and chest X-ray image domains have seen many vision-language models proposed in recent years. However, the paper lacks a comprehensive comparison with these existing methods, such as [R1-R4].\n\n- The experiments focus solely on disease classification tasks, without evaluating the method on more challenging dense prediction tasks such as critical region segmentation or detection, which are also commonly used in the evaluation of pretraining methods.\n\nReferences:\n[R1] Wang F, Zhou Y, Wang S, Vardhanabhuti V, Yu L. Multi-granularity cross-modal alignment for generalized medical visual representation learning. NeurIPS, 2022.\n[R2] Wang M, Lin T, Lin A, Yu K, Peng Y, Wang L, Chen C, Zou K, Liang H, Chen M, Yao X. Enhancing diagnostic accuracy in rare and common fundus diseases with a knowledge-rich vision-language model. Nature Communications. 2025.\n[R3] Phan VM, Xie Y, Qi Y, Liu L, Liu L, Zhang B, Liao Z, Wu Q, To MS, Verjans JW. Decomposing disease descriptions for enhanced pathology detection: A multi-aspect vision-language pre-training framework. CVPR 2024.\n[R4] Ma D, Pang J, Gotway MB, Liang J. A fully open AI foundation model applied to chest radiography. Nature. 2025 Jun 11:1-1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "34KsImsTzU", "reviewer": "Reviewer_HXxg", "summary": "This paper proposed MGLL, a novel multi-grandularity contrastive learning framework for medical visual language pre-training. Different from vanilla CLIP, MGLL conducts contrastive learning through both multi-label and multi-granular optimization. It proposed three new contrastive losses designed for data with multi-granular annotation, showing an improved performance on various tasks.", "questions": "1. The reviewer is also curious about the training cost. How much extra training time does MGLL need compared with the normal CLIP?\n\n2. It seems that the X-ray data and Fundus data use different granularity; will this influence the final performance? How sensitive is the model to this choice?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "1. The paper has contributed two large-scale image-text pair datasets for fundus and X-ray images, providing multi-granular annotation, which will be very helpful to the field.\n\n2. According to the evaluation and derivation, the proposed losses (soft CLIP loss, point-wise BCE loss, and KL loss) help improve the model's performance on multiple downstream evaluations, showing a uniform improvement against baselines.\n\n3. The ablation experiment is especially detailed, providing strong support to the model design.\n\n4. Code is provided in the supplement.", "weaknesses": "My major concern is the clear formatting issue. The paper has clearly adjusted the vertical space between the section and sub-section titles, gaining more space for their content. Table 1 and Figure 4 overlap with each other. The reviewer believes that this is a violation of the conference policy, which suggests \"Do not change any aspects of the formatting parameters in the style files. In particular, do not modify the width or length of the rectangle the text should fit into...\" Considering that, I am afraid I cannot recommend acceptance regardless of the content of the paper, unless there is clear evidence that suggests otherwise (modifying vertical space is fine, or proving the vertical space is not modified).\n\nBesides the format issue, the reviewer still has concerns about the proposed method. Listed below:\n\n1. In equation (2), the soft CLIP loss is defined based on both $l_\\{ik\\}$ and $l_\\{ki\\}$, where the second term (if I understand correctly) refers to text-to-image loss. But this is clearly wrong, since $i$ is for the $i$-th image/text in the batch, but $k$ is the $k$-th label across different granularities, and they are clearly un-interchangeable. According to Figure 2, the soft CLIP loss should be computed over 3 dimensions: image batch, text batch, and granularity. So, the reviewer guessed there might be a missing index in this equation.\n\n2. The point-wise BCE loss optimizes as a multi-label classification loss. However, it is weird that $M$ now serves as the number of **categories**, rather than the number of different **granularities**, and they can't be the same value. Since for the $i$-th image, it is always paired with all its $M_i$ annotations, as they just refer to different granularity. This might just be a typo, but it would still be better to clarify these two values.\n\n3. As for the KL loss, there is no explanation on how $z$ is obtained, which is the \"predicted logits\". The reviewer assumes this refers to the logits for each category/class of the image. But it still uses $m$ as an index, which makes it very confusing.\n\n4. According the section 3.2, the proposed method relies on category-wise prediction (both point-wise BCE loss and KL loss), which means it can only be applied to data with a known number of classes. And this is a fundamental weakness compared with CLIP, since it loses the capability and flexibility for unseen classes. One needs to carefully design the granularity and classification classes before applying them to new data. This also contradicts the claim of \"plug-and-play\" in the abstract; the proposed method only fits with specific datasets rather than general data.\n\n5. Additionally, the results in section 4.3 and table 2 are also confusing. To replace the vision encoder in the MLLM, one needs to fine-tune the MLLM to adapt to a new vision feature space. However, there is no discussion or clarification about this. If the MLLMs with a new vision encoder are fine-tuned while the original ones are not, then it will be an unfair comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oJNFbvTDtw", "reviewer": "Reviewer_Bhm2", "summary": "This paper proposes a novel medical vision-language pre-training framework, Multi-Granular Language Learning (MGLL), designed to achieve both multi-label and cross-granularity image-text alignment. Building upon CLIP, MGLL introduces three key components: soft-label contrastive loss, point-wise binary supervision loss, and smooth KL divergence loss, to enhance cross-granularity consistency. The authors construct two large-scale medical image-text datasets, MGLL-Fundus (fundus images) and MGLL-Xray (X-ray images), for model pre-training. Experiments demonstrate that MGLL significantly outperforms existing state-of-the-art methods on 11 downstream tasks and effectively boosts the performance of multimodal large language models (MLLMs).", "questions": "1. How is the multi-granular text supervision generated?\n2. Is there a potential risk of trivial solutions in the smooth KL divergence?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The overall contribution is clear and highly practical, offering significant utility for addressing the multi-level semantic complexity inherent in medical images.\n2. Experiments are comprehensive and results are significant:\n2.1. Covers multiple datasets (Fundus and X-ray) and 11 downstream tasks;\n2.2. Validates various application scenarios, including linear probing, full fine-tuning, and integration with MLLMs;\n2.3. Significantly outperforms existing CLIP variants on the majority of tasks, demonstrating strong experimental consistency.", "weaknesses": "1. The details of dataset construction are insufficient. \n2. The innovation leans more towards a compositional approach, rather than proposing entirely new learning principles or optimization mechanisms.\n3. Lacks ablation studies on the temperature coefficient (τ).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MA1UYirn7p", "reviewer": "Reviewer_pAv1", "summary": "The paper proposes MGLL (Multi-Granular Language Learning) which is a training method that extends CLIP to jointly handle multi-label supervision and text granularities in medical imaging. It initroduces three complementary losses: soft-CLIP, point-wise BCE, and smooth KL. The model trained with MGLL outperforms prior CLIP-style baselines across benchmarks and also boost downstream MLLM performance.", "questions": "1) Can the authors provide experimental results showing MGLL's performance on datasets with artificially introduced noise, such as 10%–30% missing granularity labels? \n\n2)  Can the authors provide experimental results showing MGLL's performance on datasets with mixing granularity levels? For example, some samples have 2 level of granularity level, while others have 1 level.\n\n3) Have clinical experts validated MGLL's alignments or its class activation maps for clinical relevance?\n\n4) Can the authors provide specific metrics (e.g., training time, GPU memory usage) comparing MGLL to CLIP or other baselines on the MGLL-Fundus dataset?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The study introduces a triple loss function to for training medical image-text to capture multi-granular level meaning, which standard CLIP does not handle.\n\n2) The study curates two multi-granular datasets, MGLL-Funds and MGLL-Xray.\n\n3) The paper explain the motivation for each term of the MGLL with empirical analyses.\n\n4) The paper provides strong, consistent grains across benchmarks in both linear-probe and full fine-tune settings.\n\n5) MGLL can be easily adopt in other medical domains that have hierarchical labels or reports.", "weaknesses": "1) The study does not provide report on the overlap between MGLL-Fundus (aggregates from 49 sources) and target datasets. Possible risky for data leakage.\n\n2) The paper claims MGLL is computationally efficient due to no additional encoders, but provides no evidence such as training time, memory usage, compared to baselines like CLIP.\n\n3) The paper does not provide clinical relevant metrics such as F1, or sensitive, specificity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)